Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

We have very large database table that I need to select the value of the max id. Given the table size SELECT MAX(id) FROM tableA will not perform well enough.

Table definition (with columns omitted for clarity here)

CREATE TABLE [dbo].[TableA](
    [id] [numeric](19, 0) NOT NULL,
    [timeSampled] [datetime] NOT NULL
 CONSTRAINT [PK_TableA] PRIMARY KEY NONCLUSTERED 
(
    [id] DESC,
    [timeSampled] DESC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)

The table is partitioned by month on the timesampled, and because we switch partitions out to another table for archiving data the timeSampled column must be included in the Primary Key.

So, the PK is

[id] DESC,
[timeSampled] DESC

The table contains rows for May 2013 and June 2013, and the ids increment in date order (i.e. the max id is in June)

When I select the top 1 id from the table as follows

SELECT top 1 id
FROM dbo.TableA
WITH (INDEX (PK_TableA)) 

I don't get the max id, instead I get the max id from May (i.e. the very last entry in the May partition).

Any ideas why this is the case? Basically seems to be picking the MAX id from a partition, not max across all partitions

share|improve this question
5  
TOP 1 without ORDER BY is undefined. Have you actually tried SELECT MAX(id) FROM tableA and determined that it does not perform well? If you have how many partitions do you have and what does the plan for that look like? In principle it ought to be able to use the index in each partition to get the single MAX quickly then find the MAX of those. Do you not get that plan? –  Martin Smith Jun 14 '13 at 9:32
    
This might be a heavy change, but can't you make the PK index clustered, once that ids and date order increment accordingly? It would most probably speed up your SELECT MAX(id) FROM tableA enough –  user16484 Jun 14 '13 at 10:45
    
Martin Smith, unfortunately the max query does perform slowly. The table is filling at a rate of 3-6 million rows per day and will store data for 6 months before we archive it off. –  Adrian Pillinger Jun 14 '13 at 10:49
1  
@MartinSmith, we have discovered a trick to specify the partition to select max from.... SELECT MAX(id) AS maxId FROM dbo.TableA WHERE $partition.TableAPartitionFunction_Monthly](timeSampled) = partitionNumber And this seems to perform well and correctly identify the partition to use. –  Adrian Pillinger Jun 14 '13 at 10:50
1  
@MartinSmith Itzik Ben-Gan wrote a very clear explanation and workaround here –  Paul White Jun 19 '13 at 16:13

1 Answer 1

up vote 4 down vote accepted

Basically seems to be picking the MAX id from a partition, not max across all partitions

Writing TOP (1) without an ORDER BY clause to define which row is 'top' means the query processor is logically free to return any row from the set. The query plan selected by the optimizer happens to return a particular row (highest id from the first partition) but you cannot rely on this, even if it were a useful result.

Whenever you use TOP you should always specify an ORDER BY at the same scope to produce deterministic behaviour - unless you really do not care which row(s) come back.

Given the table size SELECT MAX(id) FROM tableA will not perform well enough

The optimizer is lacking some logic to transform a scalar MAX or MIN aggregate over a partitioned index to a global aggregate over per-partition aggregates. Itzik Ben-Gan explains the limitation and provides a general workaround in this article.

If the highest partition number is known and guaranteed not to change, the workaround to specify a literal partition using the $partition function will work, though it may fail in a non-obvious way if the partitioning strategy changes in future.

This 'solution' works by eliminating all but one partition, resulting in a simple seek on one partition of the index.

Adding an order by id does not improve performance for some reason

The same optimizer limitation broadly applies to TOP (1) ... ORDER BY. The ORDER BY makes the result deterministic, but does not help produce a more efficient plan in this particular case (but see below).

Implied Index Keys

Your index is on id DESC, timeSampled DESC. In SQL Server 2008 and later, partitioning introduces an extra implied leading key on $partition ASC (it is always ascending, it is not configurable) making the full index key $partition ASC, id DESC, timeSampled DESC.

Since id and timeSampled increase together (though there is nothing in the schema to guarantee this) you could rewrite the query as TOP (1) ... ORDER BY $partition DESC, id DESC. Unfortunately, the DESC keys on your index and ASC implied leading key $partition means the index could not be used to scan just one row from the index in order.

If your index keys were instead id ASC, timeSampled ASC the whole index key would be $partition ASC, id ASC, timeSampled ASC. This all-ASC index could be scanned backward, returning just the first row in key order. This row would be guaranteed to have the highest id value in the highest-numbered partition. Given the (unenforced) relationship between id and partition id, this would produce the correct result with an optimal execution plan that reads just a single row.

This 'solution' lacks integrity because the id-timeSampled relationship is not enforced, and you probably do not want to rebuild the nonclustered primary key anyway. Nevertheless, I mention it because it may enhance your understanding of how partitioning interacts with indexes.

share|improve this answer
1  
Thank you for your excellent and very comprehensive answer. That has really helped me understand what is going on and why our attempts had failed. Thank you. –  Adrian Pillinger Jun 24 '13 at 7:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.