1

I have been asked to review a process that sadly is highly inefficient but we are held in the short term to the design.

We currently generate based on a range a set of rows inside a table to hold future receipts. The ReceiptData table for every Receipt id has its rows generated first and as the receipts are use the next is populated. Again this process for the short term is in stone.

When a customer adds a receipt we then need to generate a million rows within the receiptData table that will then be used in the future.

DECLARE @tempT TABLE 
( 
    number INT 
) 
DECLARE @recId AS INT -- represents an id a parent table id
DECLARE @receiptStart AS INT --represents where in the sequence to begin normmaly 1 but could be 1500 etc.
DECLARE @receiptEnd AS INT -- represents then end in relation to the start normally 999999 but could be 456789, defined by customer

SET @recId = 12
SET @receiptStart = 120
SET @receiptEnd = 999999

    DECLARE @iCounter AS INT = @receiptStart

WHILE ( @iCounter <= @receiptEnd) 
  BEGIN 
      INSERT INTO @tempT 
      VALUES      (@iCounter) 
      SET @iCounter += 1 
  END 

INSERT receiptdata 
       (recid, 
        receiptnum, 
        negativereceipt) 
SELECT @recID, 
       Cast(number AS VARCHAR), 
       0 
FROM   @tempT 
WHERE  NOT EXISTS ( SELECT  1 
                    FROM    receiptdata 
                    WHERE   ffid = @recId
                            AND receiptnum = number
                  )

I ran this and it took over 13 minutes to process...there has to be a better way. The initial population of @tempT takes over 30 seconds; the insert into ReceiptData takes the majority of the time.

The annoying piece of this is the receiptnum within the receiptData table is a varchar. while we plan out a future design we are stuck with all of this in the short term.

Any suggestions? it it rather straight forward but insanely slow. we usually go with ranged from 1 to 999999 for receiptStart and receiptend.

2
  • 3
    You have a bunch of undeclared variables in your example code. Can you provide a complete example that we can run without access to your system? That will make it easier for people to help you. Also, can you clarify which part of the procedure is taking the most time? You said that it took over 13 minutes to process but a 30 second step took the majority of the time? How is that possible?
    – Joe Obbish
    Commented Aug 15, 2017 at 18:12
  • 1
    An explanation of the relationship between recId and ffid would seem to be in order as well. Why do you need to check if there's an existing record where ffid matches your newly-entered recId and the record number is the one you're trying to add?
    – RDFozz
    Commented Aug 15, 2017 at 18:30

2 Answers 2

2

As far as I can tell from the limited information provided in your question, you don't really need to test for the existence of each and every newly generated ReceiptNum. If you ensure each @recId is unique when generated, you can be certain the combination of @recId and ReceiptNum will be unique. If you are in fact trying to ensure each and every ReceiptNum is unique, regardless of which @recID it's associated with, you can pre-allocate the 1,000,000 receipt numbers each time a @recId is created.

If your system allows duplicate ReceiptNum values for each unique @recId, you could really benefit from creating a clustered index based on ffid and ReceiptNum... Take this example:

USE tempdb;

DROP TABLE IF EXISTS dbo.Receipts;
GO
CREATE TABLE dbo.Receipts
(
    RecID int NOT NULL
    , RecNum varchar(10) NOT NULL
    , SomeData varchar(2000) NOT  NULL
        DEFAULT (CRYPT_GEN_RANDOM(1000))
    , CONSTRAINT PK_Receipts
        PRIMARY KEY CLUSTERED 
        (RecID, RecNum)
) ON [PRIMARY]
WITH (DATA_COMPRESSION = PAGE);
GO

The table above will only allow rows to be inserted where the combination of RecID and ReceiptNum are unique, by virtue of the PRIMARY KEY CLUSTERED (RecID, RecNum) clause.

Next, I'll create a procedure to insert 1,000,000 rows into the table. This procedure does not check if each RecID/ReceiptNum already exists in the table since it cannot, by virtue of the table's primary key:

DROP PROCEDURE IF EXISTS dbo.InsertReceiptBlanks;
GO
CREATE PROCEDURE dbo.InsertReceiptBlanks
(
    @RecID int
)
AS
BEGIN
    ;WITH src AS 
    (
        SELECT v.Num
        FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9))v(Num)
    )
    , MillionRows AS
    (
        SELECT RecID = @RecID
            , RecNum = s1.Num + (s2.Num * 10) + (s3.num * 100) + (s4.num * 1000)
            + (s5.num * 10000) + (s6.Num * 100000)
        FROM src s1
            CROSS JOIN src s2
            CROSS JOIN src s3
            CROSS JOIN src s4
            CROSS JOIN src s5
            CROSS JOIN src s6
    )
    INSERT INTO dbo.Receipts (RecID, RecNum)
    SELECT RecID
        , RecNum
    FROM MillionRows;
END
GO

For fun, we'll create a stored procedure that does check to see if the RecID/ReceiptNum combination exists, prior to doing the insert.

DROP PROCEDURE IF EXISTS dbo.InsertReceiptBlanks_WithNotExists;
GO
CREATE PROCEDURE dbo.InsertReceiptBlanks_WithNotExists
(
    @RecID int
)
AS
BEGIN
    ;WITH src AS 
    (
        SELECT v.Num
        FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9))v(Num)
    )
    , MillionRows AS
    (
        SELECT RecID = @RecID
            , RecNum = s1.Num + (s2.Num * 10) + (s3.num * 100) + (s4.num * 1000)
            + (s5.num * 10000) + (s6.Num * 100000)
        FROM src s1
            CROSS JOIN src s2
            CROSS JOIN src s3
            CROSS JOIN src s4
            CROSS JOIN src s5
            CROSS JOIN src s6
    )
    INSERT INTO dbo.Receipts (RecID, RecNum)
    SELECT RecID
        , RecNum
    FROM MillionRows
    WHERE NOT EXISTS (
        SELECT 1
        FROM dbo.Receipts r
        WHERE r.RecID = MillionRows.RecID
            AND r.RecNum = MillionRows.RecNum
        );
END
GO

Here I'm inserting 5,000,000 rows into the table; each execution of the stored proc takes about 30 seconds total.

EXEC dbo.InsertReceiptBlanks @RecID = 1;
EXEC dbo.InsertReceiptBlanks @RecID = 2;
EXEC dbo.InsertReceiptBlanks @RecID = 3;
EXEC dbo.InsertReceiptBlanks @RecID = 4;
EXEC dbo.InsertReceiptBlanks @RecID = 5;

Here I'm executing the WithNotExists procedure to insert 1,000,000 rows into the table, which takes around 40 seconds on my machine, with the primary key constraint in place.

EXEC dbo.InsertReceiptBlanks_WithNotExists @RecID = 6;

Next we remove the primary key, and re-run the WithNotExists proc:

ALTER TABLE dbo.Receipts
DROP CONSTRAINT PK_Receipts;

The table will now allow us to insert non-unique RecID/ReceiptNum combinations.

EXEC dbo.InsertReceiptBlanks_WithNotExists @RecID = 7;

Running the proc took 1 minute 18 seconds; three times longer than the table with the primary key constraint on RecID/ReceiptNum.

If you need to ensure each and every ReceiptNum is unique, I'd recommend inserts into the table are based upon the RecID value, as in:

DROP PROCEDURE IF EXISTS dbo.InsertReceiptBlanks_UniqueReceiptNum;
GO
CREATE PROCEDURE dbo.InsertReceiptBlanks_UniqueReceiptNum
(
    @RecID int
)
AS
BEGIN
    ;WITH src AS 
    (
        SELECT v.Num
        FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9))v(Num)
    )
    , MillionRows AS
    (
        SELECT RecID = @RecID
            , RecNum = s1.Num + (s2.Num * 10) + (s3.num * 100) + (s4.num * 1000)
            + (s5.num * 10000) + (s6.Num * 100000)
        FROM src s1
            CROSS JOIN src s2
            CROSS JOIN src s3
            CROSS JOIN src s4
            CROSS JOIN src s5
            CROSS JOIN src s6
    )
    INSERT INTO dbo.Receipts (RecID, RecNum)
    SELECT RecID
        , (RecID * 1000000) + RecNum
    FROM MillionRows;
END

We'll add a unique index on the ReceiptNum column, and test the proc above:

TRUNCATE TABLE dbo.Receipts;

ALTER TABLE dbo.Receipts ADD CONSTRAINT PK_Receipts PRIMARY KEY CLUSTERED (RecNum);

EXEC dbo.InsertReceiptBlanks_UniqueReceiptNum @RecID = 1;

Since it's the 1,000,000 inserts that are killing performance, you might also be interested in allocating receipt numbers as they are required; I've written a reliable and scalable answer showing how to do that here.

0
2

To generate the 1 million rows for @tempT you could use a cartesian product of the master..spt_values table, like such:

declare @tempT table (number int)

declare @maxno int
select  @maxno = 1000000

insert into @tempT
select top(@maxno) row_number() over(order by v1.number)
from   master..spt_values v1,
       master..spt_values v2

select min(number),max(number),count(*) from @tempT

(No column name) (No column name) (No column name)
---------------- ---------------- ----------------
               1          1000000          1000000

Takes about 2 seconds to run this dbfiddle

NOTE: master..spt_values should have at least a few thousand records so a 2-way cartesian product should be sufficient to generate 1 million rows; if you need to go a lot higher you could provide an additional ,master..sp_values v3.

You could likely push this cartesian product into your insert receptdata / select query thus eliminating the requirement for the @tempT table variable, ymmv ...


The next (much bigger) performance issue is likely the 1 million not exists() operations.

Without knowing a bit more about the relationship between @recId and receiptdata.ffid it's kinda hard to address this query, eg, do you really need to test all 1 million @tempT rows against receiptdata?

Also, do you have an index on receiptdata the optimizer can use for the not exists() operations?

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.