5

In a previous question How to merge data sets without including redundant rows? I asked about filtering redundant historical data during import, but @DavidSpillett correctly replied that I couldn't do what I was trying to do.

Instead of filtering the table during import, I now want to create a view on the table that returns only records where the price has changed.

Here's the original scenario rephrased to suite this question:

We have a table of historical prices for items. The table contains rows where the same price is recorded for multiple dates. I want to create a view on this data which only shows price changes over time, so if a price changes from A to B I want to see it, but if it "changes" from B to B then I don't want to see it.

Example: if the price yesterday was $1, and the price today is $1, and there were no other price changes, then the price today can be inferred from the price yesterday so I only need the record from yesterday.

Example (http://sqlfiddle.com/#!3/c95ff/1):

Table data:

Effective            Product  Kind  Price
2013-04-23T00:23:00  1234     1     1.00
2013-04-24T00:24:00  1234     1     1.00 -- redundant, implied by record 1
2013-04-25T00:25:00  1234     1     1.50
2013-04-26T00:26:00  1234     1     2.00
2013-04-27T00:27:00  1234     1     2.00 -- redundant, implied by record 4
2013-04-28T00:28:00  1234     1     1.00 -- not redundant, price changed back to 1.00

Expected view data:

Effective            Product  Kind  Price
2013-04-23T00:23:00  1234     1     1.00
2013-04-25T00:25:00  1234     1     1.50
2013-04-26T00:26:00  1234     1     2.00
2013-04-28T00:28:00  1234     1     1.00

My initial attempt used ROW_NUMBER:

SELECT
    Effective,
    Product,
    Kind,
    Price
FROM
(
    SELECT
        History.*,
        ROW_NUMBER() OVER
        (
            PARTITION BY
                Product,
                Kind,
                Price
            ORDER BY
                Effective ASC
        ) AS RowNumber
    FROM History
) H
WHERE RowNumber = 1
ORDER BY Effective

Which returned:

Effective               Product  Kind  Price
2013-04-23T00:23:00     1234     1     1.00
                                             -- not 2013-04-24, good
2013-04-25T00:25:00     1234     1     1.50
2013-04-26T00:26:00     1234     1     2.00
                                             -- not 2013-04-27, good
                                             -- not 2013-04-28, bad

I tried searching for a similar question/answer but it's hard to work out how to phrase the search, an example is worth a lot of words.

Any suggestions appreciated. Thanks

0

5 Answers 5

7

SQLfiddle

SELECT H.*
FROM History AS H
OUTER APPLY
(
    SELECT TOP (1)
        H2.Price
    FROM History AS H2
    WHERE
        H2.Product = H.Product
        AND H2.Kind = H.Kind
        AND H2.Effective < H.Effective
    ORDER BY
        H2.Effective DESC
) AS X
WHERE
    NOT EXISTS (SELECT X.Price INTERSECT SELECT H.Price);

Execution plan

This is a good execution plan for the small number of rows given in the question. For a larger table, the ideal index for this query is:

CREATE UNIQUE INDEX [dbo.History Product, Kind, Effective]
ON dbo.History (Product, Kind, Effective DESC)
INCLUDE (Price);

Execution plan 2

That index is essentially the clustered index keys in a more helpful order. Depending on how the table is used for other queries, it might be better to replace the clustered index instead of creating this new index.

Do not use batch estimated cost percentages to compare different queries. This is not a valid comparison in general. The costs are always optimizer estimates, and not intended to be used this way. Check actual performance metrics (elapsed time, I/Os, CPU usage, memory usage) by all means, but do not put your faith in the percentages.

Whether this query or the one based on ROW_NUMBER is more efficient depends on the distribution of the data, and other factors. Each has their advantages and disadvantages. This query has advantages when there are many rows for each (product, kind) combination.

0
5
SELECT  *
FROM   (SELECT TOP 1 *
        FROM    History
        ORDER BY Effective ASC) AS f

UNION ALL

SELECT  *
FROM    History AS a
WHERE   a.Price <> (SELECT TOP 1 Price
                    FROM    History AS b
                    WHERE   b.Effective < a.Effective 
                    ORDER BY Effective DESC) 

ORDER BY Effective ASC ;

Test at SQL-Fiddle

Execution plan

4
  • 2
    I think it needs a UNION ALL to include the first row in the table (which will not be returned, due to the compare to NULL), but it should work as the OP wants. Commented Jul 30, 2013 at 15:10
  • @ypercube good point....
    – AK_
    Commented Jul 30, 2013 at 15:12
  • Thanks, that did the trick and is surprisingly quick. Can you please edit your answer to include the UNION ALL that @ypercube suggested, then I'll accept this answer. Adding the union will make it easier for future generations who might find their way here :)
    – WileCau
    Commented Jul 31, 2013 at 3:32
  • The results in the SQL-Fiddle don't include the value for 2013-04-27 when the price changes to $2 for Kind 1. The top part of the union needs to consider Product and Kind. I modified the SQL-Fiddle.
    – WileCau
    Commented Jul 31, 2013 at 5:56
4

Thanks to @JonSeigel for directing me to Query (duration) different between data value change. I'm posting a modified version of his answer here because my problem was slightly different (needed PARTITION BY in the ROW_NUMBER) so maybe someone will find the variant helpful. If you find this answer helpful please vote up the Jon's answer because I wouldn't have got to this without it.

In my fairly basic timing tests on a table containing ~3M records this solution is about 3x faster than @AK_'s answer.

This is the query:

WITH a AS
(
  SELECT
    History.*,
    ROW_NUMBER() OVER (PARTITION BY Product, Kind ORDER BY Effective ASC) AS RN
  FROM History
)
SELECT
  a1.Effective,
  a1.Product,
  a1.Kind,
  a1.Price
FROM a a1
LEFT OUTER JOIN a a2
  ON a2.RN = a1.RN - 1
    AND a2.Product = a1.Product
    AND a2.Kind = a1.Kind
WHERE
  (a1.Price <> a2.Price)
  OR (a2.RN IS NULL)

Execution plan

For a larger number of rows:

Execution plan 2

The sorts can be eliminated using the index:

CREATE UNIQUE INDEX [dbo.History Product, Kind, Effective]
ON dbo.History (Product, Kind, Effective)
INCLUDE (Price);

Execution plan 3

2

What is needed is a LEAD/LAG solution implemented in 2008. Unfortunately it didn't become available until 2012. I found a couple really good implementations over on SQLAuthority. A comparison can be made between the LEAD and current value and row returned where they are different.

2
  • Thanks for the link, I hadn't heard of LEAD/LAG. Unfortunately we're stuck with 2008 for now so I'll definitely go through those solutions to see if/where we can use them.
    – WileCau
    Commented Jul 31, 2013 at 3:38
  • 1
    Can you expand your answer to show how LEAD/LAG can be used to solve the problem? Future visitors to the site may well be using SQL Server 2012 and will appreciate it.
    – Paul White
    Commented Jul 31, 2013 at 9:56
2

For Sql Server 2012 using Lag()

;with h as(
SELECT  Effective,
    Product,
    Kind,
    Price, 
    lag (price, 1, 0) OVER (PARTITION BY product,kind order by Effective) as prevprice
FROM History )
select Effective, Product, Kind, Price
 from h where (prevprice <> price or prevprice = 0.00)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.