How to select from a table without including repeated column values?

Question

In a previous question How to merge data sets without including redundant rows? I asked about filtering redundant historical data during import, but @DavidSpillett correctly replied that I couldn't do what I was trying to do.

Instead of filtering the table during import, I now want to create a view on the table that returns only records where the price has changed.

Here's the original scenario rephrased to suite this question:

We have a table of historical prices for items. The table contains rows where the same price is recorded for multiple dates. I want to create a view on this data which only shows price changes over time, so if a price changes from A to B I want to see it, but if it "changes" from B to B then I don't want to see it.

Example: if the price yesterday was $1, and the price today is $1, and there were no other price changes, then the price today can be inferred from the price yesterday so I only need the record from yesterday.

Example (http://sqlfiddle.com/#!3/c95ff/1):

Table data:

Effective            Product  Kind  Price
2013-04-23T00:23:00  1234     1     1.00
2013-04-24T00:24:00  1234     1     1.00 -- redundant, implied by record 1
2013-04-25T00:25:00  1234     1     1.50
2013-04-26T00:26:00  1234     1     2.00
2013-04-27T00:27:00  1234     1     2.00 -- redundant, implied by record 4
2013-04-28T00:28:00  1234     1     1.00 -- not redundant, price changed back to 1.00

Expected view data:

Effective            Product  Kind  Price
2013-04-23T00:23:00  1234     1     1.00
2013-04-25T00:25:00  1234     1     1.50
2013-04-26T00:26:00  1234     1     2.00
2013-04-28T00:28:00  1234     1     1.00

My initial attempt used ROW_NUMBER:

SELECT
    Effective,
    Product,
    Kind,
    Price
FROM
(
    SELECT
        History.*,
        ROW_NUMBER() OVER
        (
            PARTITION BY
                Product,
                Kind,
                Price
            ORDER BY
                Effective ASC
        ) AS RowNumber
    FROM History
) H
WHERE RowNumber = 1
ORDER BY Effective

Which returned:

Effective               Product  Kind  Price
2013-04-23T00:23:00     1234     1     1.00
                                             -- not 2013-04-24, good
2013-04-25T00:25:00     1234     1     1.50
2013-04-26T00:26:00     1234     1     2.00
                                             -- not 2013-04-27, good
                                             -- not 2013-04-28, bad

I tried searching for a similar question/answer but it's hard to work out how to phrase the search, an example is worth a lot of words.

Any suggestions appreciated. Thanks

Answer 1

SQLfiddle

SELECT H.*
FROM History AS H
OUTER APPLY
(
    SELECT TOP (1)
        H2.Price
    FROM History AS H2
    WHERE
        H2.Product = H.Product
        AND H2.Kind = H.Kind
        AND H2.Effective < H.Effective
    ORDER BY
        H2.Effective DESC
) AS X
WHERE
    NOT EXISTS (SELECT X.Price INTERSECT SELECT H.Price);

This is a good execution plan for the small number of rows given in the question. For a larger table, the ideal index for this query is:

CREATE UNIQUE INDEX [dbo.History Product, Kind, Effective]
ON dbo.History (Product, Kind, Effective DESC)
INCLUDE (Price);

That index is essentially the clustered index keys in a more helpful order. Depending on how the table is used for other queries, it might be better to replace the clustered index instead of creating this new index.

Do not use batch estimated cost percentages to compare different queries. This is not a valid comparison in general. The costs are always optimizer estimates, and not intended to be used this way. Check actual performance metrics (elapsed time, I/Os, CPU usage, memory usage) by all means, but do not put your faith in the percentages.

Whether this query or the one based on ROW_NUMBER is more efficient depends on the distribution of the data, and other factors. Each has their advantages and disadvantages. This query has advantages when there are many rows for each (product, kind) combination.

Answer 2

SELECT  *
FROM   (SELECT TOP 1 *
        FROM    History
        ORDER BY Effective ASC) AS f

UNION ALL

SELECT  *
FROM    History AS a
WHERE   a.Price <> (SELECT TOP 1 Price
                    FROM    History AS b
                    WHERE   b.Effective < a.Effective 
                    ORDER BY Effective DESC) 

ORDER BY Effective ASC ;

Test at SQL-Fiddle

Answer 3

Thanks to @JonSeigel for directing me to Query (duration) different between data value change. I'm posting a modified version of his answer here because my problem was slightly different (needed PARTITION BY in the ROW_NUMBER) so maybe someone will find the variant helpful. If you find this answer helpful please vote up the Jon's answer because I wouldn't have got to this without it.

In my fairly basic timing tests on a table containing ~3M records this solution is about 3x faster than @AK_'s answer.

This is the query:

WITH a AS
(
  SELECT
    History.*,
    ROW_NUMBER() OVER (PARTITION BY Product, Kind ORDER BY Effective ASC) AS RN
  FROM History
)
SELECT
  a1.Effective,
  a1.Product,
  a1.Kind,
  a1.Price
FROM a a1
LEFT OUTER JOIN a a2
  ON a2.RN = a1.RN - 1
    AND a2.Product = a1.Product
    AND a2.Kind = a1.Kind
WHERE
  (a1.Price <> a2.Price)
  OR (a2.RN IS NULL)

For a larger number of rows:

The sorts can be eliminated using the index:

CREATE UNIQUE INDEX [dbo.History Product, Kind, Effective]
ON dbo.History (Product, Kind, Effective)
INCLUDE (Price);

Answer 4

What is needed is a LEAD/LAG solution implemented in 2008. Unfortunately it didn't become available until 2012. I found a couple really good implementations over on SQLAuthority. A comparison can be made between the LEAD and current value and row returned where they are different.

Answer 5

For Sql Server 2012 using Lag()

;with h as(
SELECT  Effective,
    Product,
    Kind,
    Price, 
    lag (price, 1, 0) OVER (PARTITION BY product,kind order by Effective) as prevprice
FROM History )
select Effective, Product, Kind, Price
 from h where (prevprice <> price or prevprice = 0.00)