Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

One of our production systems has exhibited a problem with a single insertion statement into a temp table from our user database. When I comment out the insert/select, the stored proc in question runs in a timely fashion, so I am confident in my isolation of the problem.

The series of stored procs invoked basically grinds to a halt when I un-comment the insert/select in question. I cannot see anything in Top Transactions by Age in tempdb or any of our user databases. I do not see anything in Activity Monitor that deviates from Activity Monitor's information when the database is "at rest", other than CPU being flatlined at ~20%.

The behavior is as follows: when I setup and then execute the reproduction case, upon arrival at the insert/select in question, I see an SOS_SCHEDULER_YIELD and there is an ENCRYPTION_SCAN. About five hours later, I will see processing of our stored procedure resume and the activity will complete (I put quick and dirty log statements around every distinct operation).

I also replaced the variables in the select portion of the insert with the values as executed, and ran the selection query itself, and it returned in five seconds.

The user database in question has FALSE as its encryption enabled value, as does tempdb. The operation in question happens over about 65k rows of data, and I have tried it with only 1k rows, and the behavior persisted, although the time it took was far less.

A single user database is the only instance of this behavior. I have reproduced it locally over a backup of that user databse. We have about 70 other users of the software that do not exhibit this problem.

Given the above information, my question is, why is processing of our stored procedures stopping? As it is probably optimistic to expect a precise answer, what is the correct step to debugging this? Perhaps there is something in one of the DMVs like dm_tran_locks, dm_exec_requests, dm_tran_database_transactions, dm_os_schedulers, dm_exec_sessions that, while they have provided me with some information, I am not interpreting or understanding the output in a way that points toward a solution.

Below is the insert/select in question:

INSERT INTO #TS_EVENT_DATA 
        (   EVENT_FK,
            EVENT_TYPE_CR_FK,
            EVENT_ENTITY_CLASS_CR_FK,
            userDatabase_ID,
            DATA_NAME_FK,               
            IMPORT_JOB_FK,              
            PRODUCT_STRUCTURE_FK,
            ORG_ENTITY_STRUCTURE_FK,
            ENTITY_CLASS_CR_FK,
            ENTITY_DATA_NAME_FK,
            ENTITY_STRUCTURE_FK,                
            DATA_SET_FK,
            DATA_TYPE_CR_FK,
            ORG_IND,
            TABLE_NAME,
            NET_VALUE1_NEW,
            NET_VALUE2_NEW,
            NET_VALUE3_NEW,
            NET_VALUE4_NEW,
            NET_VALUE5_NEW,
            NET_VALUE6_NEW,                                                     
            NET_VALUE1_CUR,
            NET_VALUE2_CUR,
            NET_VALUE3_CUR,
            NET_VALUE4_CUR,
            NET_VALUE5_CUR,
            NET_VALUE6_CUR,                         
            PERCENT_CHANGE1,
            PERCENT_CHANGE2,
            PERCENT_CHANGE3,
            PERCENT_CHANGE4,
            PERCENT_CHANGE5,
            PERCENT_CHANGE6,                            
            VALUE_UOM_CODE_FK,
            ASSOC_UOM_CODE_FK,
            VALUES_SHEET_NAME,
            UOM_CONVERSION_FACTOR,
            END_DATE_CUR,                           
            END_DATE_NEW,   
            EFFECTIVE_DATE_CUR,                                     
            EVENT_EFFECTIVE_DATE,
            EVENT_ACTION_CR_FK,
            EVENT_STATUS_CR_FK, 
            EVENT_CONDITION_CR_FK,
            EVENT_SOURCE_CR_FK,
            EVENT_PRIORITY_CR_FK,
            RESULT_TYPE_CR_FK,
            TABLE_ID_FK,
            BATCH_NO,
            IMPORT_BATCH_NO,
            RULES_FK,
            RECORD_STATUS_CR_FK,                
            UPDATE_TIMESTAMP
             )  
   SELECT
            A.EVENT_ID,                    
            A.EVENT_TYPE_CR_FK,
            A.EVENT_ENTITY_CLASS_CR_FK,
            A.userDatabase_ID,
            A.DATA_NAME_FK,             
            A.IMPORT_JOB_FK,
            A.PRODUCT_STRUCTURE_FK,
            A.ORG_ENTITY_STRUCTURE_FK,
            A.ENTITY_CLASS_CR_FK,
            A.ENTITY_DATA_NAME_FK,
            A.ENTITY_STRUCTURE_FK,
            A.DATA_SET_FK,
            A.DATA_TYPE_CR_FK,
            A.ORG_IND,
            A.TABLE_NAME,
            A.NET_VALUE1_NEW,
            A.NET_VALUE2_NEW,
            A.NET_VALUE3_NEW,
            A.NET_VALUE4_NEW,
            A.NET_VALUE5_NEW,
            A.NET_VALUE6_NEW,                                                   
            A.NET_VALUE1,  
            A.NET_VALUE2,
            A.NET_VALUE3,
            A.NET_VALUE4,
            A.NET_VALUE5,
            A.NET_VALUE6,                       
            CASE ISNULL (A.NET_VALUE1, 0 )
             WHEN 0 THEN 0  
             ELSE ( ( A.NET_VALUE1_NEW  - A.NET_VALUE1) / A.NET_VALUE1 )
             END,
            CASE ISNULL (A.NET_VALUE2, 0 )
             WHEN 0 THEN 0  
             ELSE ( ( A.NET_VALUE2_NEW  - A.NET_VALUE2 ) / A.NET_VALUE2 )
             END,
            CASE ISNULL (A.NET_VALUE3, 0 )
             WHEN 0 THEN 0  
             ELSE ( ( A.NET_VALUE3_NEW  - A.NET_VALUE3 ) / A.NET_VALUE3 )
             END,
            CASE ISNULL (A.NET_VALUE4, 0 )
             WHEN 0 THEN 0  
             ELSE ( ( A.NET_VALUE4_NEW  - A.NET_VALUE4 ) / A.NET_VALUE4 )
             END,
            CASE ISNULL (A.NET_VALUE5, 0 )
             WHEN 0 THEN 0  
             ELSE ( ( A.NET_VALUE5_NEW  - A.NET_VALUE5 ) / A.NET_VALUE5 )
             END,
            CASE ISNULL (A.NET_VALUE6, 0 )
             WHEN 0 THEN 0  
             ELSE ( ( A.NET_VALUE6_NEW  - A.NET_VALUE6 ) / A.NET_VALUE6 )
             END,
        A.VALUE_UOM_CODE_FK,
        A.ASSOC_UOM_CODE_FK,
        A.VALUES_SHEET_NAME,
        ( SELECT CASE ISNULL ( A.VALUE_UOM_CODE_FK, 0 ) 
        WHEN 0 THEN 1
        ELSE
        CASE ISNULL ( A.ASSOC_UOM_CODE_FK, 0 ) 
        WHEN 0 THEN 1
        ELSE
        ( ISNULL (
        ( SELECT 
        ( SELECT ISNULL (
           (SELECT uc.primary_qty
            FROM userDatabase.UOM_CODE uc WITH (NOLOCK)
            WHERE uc.UOM_CODE_ID = A.VALUE_UOM_CODE_FK  ), 1 )
        )
        /
        ISNULL (   (SELECT uc.primary_qty
                    FROM userDatabase.UOM_CODE uc WITH (NOLOCK)
                    WHERE uc.UOM_CODE_ID = A.ASSOC_UOM_CODE_FK )
                 , ISNULL ( (SELECT uc.primary_qty
                             FROM userDatabase.UOM_CODE uc WITH (NOLOCK)
                             WHERE uc.UOM_CODE_ID = A.VALUE_UOM_CODE_FK) , 1 ) 
                 )
        ) , 1 ) )
        END END
        )  AS UOM_CONVERSION_FACTOR,
            A.END_DATE,
            A.END_DATE_NEW, 
            A.EFFECTIVE_DATE,                                       
            A.EVENT_EFFECTIVE_DATE,
            A.EVENT_ACTION_CR_FK,
            A.EVENT_STATUS_CR_FK,   
            A.EVENT_CONDITION_CR_FK,
            A.EVENT_SOURCE_CR_FK,
            A.EVENT_PRIORITY_CR_FK,
            A.RESULT_TYPE_CR_FK,
            A.SHEET_RESULTS_ID,
            A.BATCH_NO,
            A.IMPORT_BATCH_NO,
            A.RULES_FK,
            A.RECORD_STATUS_CR_FK,              
            @L_SYSDATE

FROM ( SELECT
            ED.EVENT_ID,                    
            DS.EVENT_TYPE_CR_FK,
            DS.ENTITY_CLASS_CR_FK AS EVENT_ENTITY_CLASS_CR_FK,
            ED.PRODUCT_STRUCTURE_FK AS userDatabase_ID,
            ED.DATA_NAME_FK,
            ED.IMPORT_JOB_FK,           
            ED.PRODUCT_STRUCTURE_FK,
            CASE ISNULL ( DS.ORG_IND, 0 )
            WHEN 0 THEN ISNULL ( ED.ORG_ENTITY_STRUCTURE_FK, 1 )
            ELSE ED.ORG_ENTITY_STRUCTURE_FK         
            END AS ORG_ENTITY_STRUCTURE_FK,
            DS.ENTITY_STRUCTURE_EC_CR_FK AS ENTITY_CLASS_CR_FK,
            DN.ENTITY_DATA_NAME_FK,
            ED.ENTITY_STRUCTURE_FK,                                                                                 
            DN.DATA_SET_FK,
            DS.DATA_TYPE_CR_FK,
            DS.ORG_IND,
            DS.TABLE_NAME,
            ED.NET_VALUE1_NEW,
            ED.NET_VALUE2_NEW,
            ED.NET_VALUE3_NEW,
            ED.NET_VALUE4_NEW,
            ED.NET_VALUE5_NEW,
            ED.NET_VALUE6_NEW,                                                      
            SR.NET_VALUE1,
            SR.NET_VALUE2,
            SR.NET_VALUE3,
            SR.NET_VALUE4,
            SR.NET_VALUE5,
            SR.NET_VALUE6,  
        ED.VALUE_UOM_CODE_FK,               
        ( SELECT TOP 1 PUC.UOM_CODE_FK
          FROM userDatabase.PRODUCT_UOM_CLASS PUC WITH (NOLOCK)
          WHERE ( PUC.DATA_NAME_FK = DN.UOM_CLASS_DATA_NAME_FK
          AND     PUC.PRODUCT_STRUCTURE_FK = ED.PRODUCT_STRUCTURE_FK
          AND   (  (   DS.ORG_IND = 1
                   AND PUC.ORG_ENTITY_STRUCTURE_FK = ED.ORG_ENTITY_STRUCTURE_FK )
                  OR PUC.ORG_ENTITY_STRUCTURE_FK = 1 ) 
          AND ISNULL ( PUC.ENTITY_STRUCTURE_FK, -999 ) = ISNULL ( ED.ENTITY_STRUCTURE_FK, -999 )   )
        ) AS ASSOC_UOM_CODE_FK,
        ED.VALUES_SHEET_NAME,
            SR.END_DATE,
            ED.END_DATE_NEW,
            SR.EFFECTIVE_DATE,              
            ED.EVENT_EFFECTIVE_DATE, 
            CASE WHEN ED.EVENT_ACTION_CR_FK = 59
            THEN 59
            ELSE
            CASE WHEN SR.SHEET_RESULTS_ID IS NULL
            THEN 51
            ELSE 52
            END
            END  AS EVENT_ACTION_CR_FK,
            ED.EVENT_STATUS_CR_FK,  
            ED.EVENT_CONDITION_CR_FK,
            ED.EVENT_SOURCE_CR_FK,
            ED.EVENT_PRIORITY_CR_FK,
            ISNULL ( ED.RESULT_TYPE_CR_FK, 711 ) AS RESULT_TYPE_CR_FK,                  
            SR.SHEET_RESULTS_ID,
            ED.BATCH_NO,
            ED.IMPORT_BATCH_NO,
            ED.RULES_FK,
            ED.RECORD_STATUS_CR_FK          
 FROM SYNCHRONIZER.EVENT_DATA ED WITH (NOLOCK)
    INNER JOIN userDatabase.DATA_NAME DN WITH (NOLOCK)
     ON ( DN.DATA_NAME_ID = ED.DATA_NAME_FK )
    INNER JOIN userDatabase.DATA_SET DS WITH (NOLOCK)
     ON ( DS.DATA_SET_ID = DN.DATA_SET_FK )
    LEFT JOIN marginmgr.SHEET_RESULTS SR WITH (NOLOCK)
     ON ( SR.DATA_NAME_FK = ED.DATA_NAME_FK
     AND  ISNULL ( SR.PRODUCT_STRUCTURE_FK, 0 ) = ISNULL ( ED.PRODUCT_STRUCTURE_FK, 0 )
     AND  ISNULL ( SR.ORG_ENTITY_STRUCTURE_FK, 0 ) = ISNULL ( ED.ORG_ENTITY_STRUCTURE_FK, 1 )
     AND  ISNULL ( SR.ENTITY_STRUCTURE_FK, 0 ) = ISNULL ( ED.ENTITY_STRUCTURE_FK, 0 )  
    )                                                    
    WHERE 1 = 1 
    AND  EVENT_STATUS_CR_FK = 88        
    AND (   
           (    ISNULL ( @in_event_fk, -999 ) = -999
            AND ISNULL ( ED.BATCH_NO, -999 ) = ISNULL ( @in_batch_no, -999 )
            AND ISNULL ( ED.import_job_fk, -999 ) = ISNULL (@in_import_job_fk, -999 )
            AND isnull ( ED.event_priority_cr_fk, -999 ) = isnull (@in_event_priority_cr_fk, -999)
            AND ISNULL ( ds.table_name, 'NULL DATA' )  = ISNULL ( @in_table_name, 'NULL DATA' ) )
        OR  ED.EVENT_ID = ISNULL (@in_event_fk, -999 ) 
        )
    AND   (   @in_data_name_fk = -999           
          OR  ED.data_name_fk = @in_data_name_fk )          
) A 

Results of sp_configure:

access check cache bucket count 0   65536   0   0
access check cache quota    0   2147483647  0   0
Ad Hoc Distributed Queries  0   1   0   0
affinity I/O mask   -2147483648 2147483647  0   0
affinity mask   -2147483648 2147483647  0   0
affinity64 I/O mask -2147483648 2147483647  0   0
affinity64 mask -2147483648 2147483647  0   0
Agent XPs   0   1   1   1
allow updates   0   1   0   0
backup compression default  0   1   0   0
blocked process threshold (s)   0   86400   0   0
c2 audit mode   0   1   0   0
clr enabled 0   1   1   1
common criteria compliance enabled  0   1   0   0
contained database authentication   0   1   0   0
cost threshold for parallelism  0   32767   5   5
cross db ownership chaining 0   1   0   0
cursor threshold    -1  2147483647  -1  -1
Database Mail XPs   0   1   1   1
default full-text language  0   2147483647  1033    1033
default language    0   9999    0   0
default trace enabled   0   1   1   1
disallow results from triggers  0   1   0   0
EKM provider enabled    0   1   0   0
filestream access level 0   2   0   0
fill factor (%) 0   100 0   0
ft crawl bandwidth (max)    0   32767   100 100
ft crawl bandwidth (min)    0   32767   0   0
ft notify bandwidth (max)   0   32767   100 100
ft notify bandwidth (min)   0   32767   0   0
index create memory (KB)    704 2147483647  0   0
in-doubt xact resolution    0   2   0   0
lightweight pooling 0   1   0   0
locks   5000    2147483647  0   0
max degree of parallelism   0   32767   0   0
max full-text crawl range   0   256 4   4
max server memory (MB)  128 2147483647  5120    5120
max text repl size (B)  -1  2147483647  65536   65536
max worker threads  128 65535   0   0
media retention 0   365 0   0
min memory per query (KB)   512 2147483647  1024    1024
min server memory (MB)  0   2147483647  128 128
nested triggers 0   1   1   1
network packet size (B) 512 32767   4096    4096
Ole Automation Procedures   0   1   0   0
open objects    0   2147483647  0   0
optimize for ad hoc workloads   0   1   0   0
PH timeout (s)  1   3600    60  60
precompute rank 0   1   0   0
priority boost  0   1   0   0
query governor cost limit   0   2147483647  0   0
query wait (s)  -1  2147483647  -1  -1
recovery interval (min) 0   32767   0   0
remote access   0   1   1   1
remote admin connections    0   1   0   0
remote login timeout (s)    0   2147483647  10  10
remote proc trans   0   1   0   0
remote query timeout (s)    0   2147483647  600 600
Replication XPs 0   1   0   0
scan for startup procs  0   1   0   0
server trigger recursion    0   1   1   1
set working set size    0   1   0   0
show advanced options   0   1   1   1
SMO and DMO XPs 0   1   1   1
transform noise words   0   1   0   0
two digit year cutoff   1753    9999    2049    2049
user connections    0   32767   0   0
user options    0   32767   0   0
xp_cmdshell 0   1   1   1

Link to XML of Exec Plan (too large to embed).

share|improve this question
1  
Perhaps some further information such as the results of EXEC sp_configure and information about your tempdb configuration (# of files, size of files), along with configuration of the machine in question might help. When you repro this, are you repro'ing it on a totally separate machine? Also, do you have SQL Server 2012 SP1 CU7 installed? –  Max Vernon Dec 12 '13 at 21:56
1  
@MaxVernon I have one tempdb file and one templog file. Initial sizes are 6G and 1G, respectively, with unlimited autogrowth occurring in 10% increments. I am repro'ing on my development machine with a copy of the user database in question taken directly from the initial failure case found in production. My machine is a Win7 x64 Dell latitude with 8G of RAM and I am allocating 5G (max mem) to the database instance. I will now attempt to repro with 2012 SP1 CU7. –  Robert Gannon Dec 12 '13 at 22:26
2  
Can you post the execution plan XML please. I might be reading it wrong late in the day but that looks like a three deep nested correlated sub-query. Probably just bad query, bad plan, pegging a single core. –  Mark Storey-Smith Dec 12 '13 at 22:51
    
@MarkStorey-Smith I will do so upon completion of CU7 install and repro, which I am in the middle of now. I am curious, in the past this has been an issue from time to time, however I usually see CPU truly pegged, not flatlined ~20%, and I usually see the offending query in both Activity Monitor and the user database's Top Transactions by Age. Finally, attempting execution earlier today it returned 65k rows in 5 seconds. –  Robert Gannon Dec 12 '13 at 23:09
2  
@RobertGannon - I assume you probably came across this already. –  Martin Smith Dec 23 '13 at 20:05

1 Answer 1

You won't only see ENCRYPTION_SCAN resource in your wait list when Encryption (like TDE) is used.

Certain operations will take a shared lock on this resource to make sure the database is not being encrypted during the operation.

The moment you would encrypt a user database with TDE, the tempdb will also be encrypted (otherwise, you would have security risk when User data is used in temp db).

Therefore, some operations will take a shared lock on ENCRYPTION_SCAN in Tempdb to prevent Tempdb from getting encrypted.

Here are two examples:

BULK INSERT

IF object_id('tempdb..##NumberCreation') IS NOT NULL
    drop table ##NumberCreation
GO

--create temp table to hold numbers
create table ##NumberCreation (C int NOT NULL);
GO

-- CREATE Numbers by using trick from Itzik -> http://sqlmag.com/sql-server/virtual-auxiliary-table-numbers 
WITH L1 AS ( SELECT 1 as C UNION SELECT 0 ),
    L2 AS ( SELECT 1 as C FROM L1 CROSS JOIN L1 as B ),
    L3 AS ( SELECT 1 as C FROM L2 CROSS JOIN L2 as B ),
    L4 AS ( SELECT 1 as C FROM L3 CROSS JOIN L3 as B ),
    L5 AS ( SELECT 1 as C FROM L4 CROSS JOIN L4 as B ),
    L6 AS ( SELECT 1 as C FROM L5 CROSS JOIN L5 as B),
    Nums as (SELECT ROW_NUMBER() OVER (ORDER BY C) as C FROM L6) 
insert ##NumberCreation(C)
SELECT TOP 500000 C
FROM Nums

The above code will generate 500k records in a global temp table, you can export these with the following commands. If you run this from SSMS, make sure you are in SQLCMD mode:

--Export
!!bcp ##NumberCreation out "E:\SQLServer\Backup\test\export.dat" -T -n

--format file
!!bcp ##NumberCreation format nul -T -n  -f "E:\SQLServer\Backup\test\export.fmt"

Make sure to choose a directory where SQL Server service account has write permissions and if you run this from SSMS, run it locally on the SQL Server.

Next thing is to start a bulk insert loop. While the loop is running, open a second screen and start running sp_lock untill you see the ENCRYPTION_SCAN shared lock in DB_ID 2 (Which is Tempdb).

The bulk import loop:

BEGIN
    IF OBJECT_ID('tempdb..#Import') IS NOT NULL
        DROP TABLE #Import ;

    CREATE TABLE #Import (C INT) ;
    BULK INSERT #Import
    FROM 'E:\SQLServer\Backup\test\export.dat' WITH (FORMATFILE='E:\SQLServer\Backup\test\export.fmt', FIRSTROW=1, TABLOCK) ;
END
GO 500 --run it 500 times

See the result of sp_lock in second window:

enter image description here

SORT IN TEMPDB

With the same Temp table in place start this very simple loop:

SELECT * from #Import order by C
go 50

It will produce the following Execution plan:

enter image description here

(Make sure that #Import is actually populated, since depending on when you stopped the previous bulk import loop, it could be empty!)

Again, run sp_lock in a second window until you see the ENCRYPTION_SCAN Resource popping up:

enter image description here

Now you know, why this resource wait is showing up. It could be very well that this is not your problem. I'd just wanted to point out the other reasons that make ENCRYPTION_SCAN show up. The reason for your query slowdown might be something else. I'll leave improving your query plan up to the query plan experts on this site ;-) However, could you post the actual execution plan as well instead of just the estimated plan?

share|improve this answer
    
So the ENCRYPTION_SCAN is a bit of a red herring then. It is an S lock. So theory is that during the 5 hour period where the OP sees the ENCRYPTION_SCAN it is in fact just busy doing the insert. Not blocked by it? –  Martin Smith Jan 10 at 14:00
    
Yes, I would think so. Might be good to see cummulated waits for that spid during the run, using extended events? To confirm that the user is in fact waiting for something else? What troubles me is the "A single user database is the only instance of this behavior.......70 other users of the software that do not exhibit this problem." Could it be a bad plan? (I don't see any parameters in this part of the query) We would need the actual plan.. –  Edward Dortland Jan 10 at 14:49
    
@EdwardDortland I will try to get the actual query plan posted. Re:Martin Smith, I am still confused why I would not see the insert in top transactions by age report for tempdb or our user database, although red herring would describe that. In the reproduction database, I re-wrote the where clause to use or statements instead of isnull, and in testing so far (+deployed early last week) the insert is running in a matter of minutes with no ENCRYPTION_SCAN. –  Robert Gannon Jan 10 at 15:03
    
@RobertGannon would be really nice to see the differences between actual plan of the original query and the reworked one. –  Edward Dortland Jan 10 at 15:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.