Tell me more ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

We have built a C# .NET system that can be used to create data warehouses. This system takes selected databases and run a script against these databases to create a combined database/warehouse.

Now, I have three databases to be compiled into a single database and I am copying two tables from each (table [XI] and table [XII] - which have a one to many relationship, but have no constraints set up at the time of the copy/INSERT INTO). The figures for the script to run and the relevant sizes for each table are below:

The executed script consists of 30 SQL queries.

  1. DatabaseA:

    Table [XI]  29,026 Rows (size 20,128Kb).
    Table [XII] 531,958 Rows (size 50,168Kb).
    Time taken for entire script: 1.51s.
    
  2. DatabaseB:

    Table [XI]  117,877 Rows (size 17,000Kb).
    Table [XII] 4,000,443 Rows (size 512,824Kb).
    Time taken for entire script: 2.04s.
    

These both run fine and fast. The next is almost exactly the same size as the first but takes 40x as long!

  1. DatabaseC:

    Table [XI]  29,543 Rows (size 20,880Kb).
    Table [XII] 538,302 Rows (size 68,000Kb).
    Time taken for entire script: 44.38s.
    

I cannot work out why this is taking so long. I have used SQL Server Profiler and the Performance Monitor, but I cannot nail-down the reason for this massive change in performance.

The query being used to do the update is dynamic and is shown at the bottom of this question - it is large due the explicit reference to the required columns. My question is; what could be causing this inordinate increase in execution time?

Any clues would be greatly appreciated.

SQL:

DECLARE @DbName NVARCHAR(128);
SET @DbName = (SELECT TOP 1 [DbName] 
               FROM [IPACostAdmin]..[TmpSpecialOptions]);
DECLARE @FilterSql NVARCHAR(MAX);
SET @FilterSql = (SELECT TOP 1 [AdditionalSQL] 
                  FROM [IPACostAdmin]..[TmpSpecialOptions]);
DECLARE @SQL NVARCHAR(MAX);
DECLARE @SQL1 NVARCHAR(MAX);
DECLARE @SQL2 NVARCHAR(MAX);
SET @SQL1 = 
    'INSERT INTO [' + @DbName + ']..[Episode] 
        ([Fields1], ..., [FieldN])'; 
SET @SQL2 = 
'SELECT 
     [Fields1], ..., [FieldN] 
FROM [B1A] ' + @FilterSql + ';'; 
SET @SQL = @SQL1 + @SQL2;
EXEC(@SQL);
GO

Note: I am splitting the dynamic SQL into @SQL1 and @SQL2 for clarity. Also note that I have not shown all columns due to space and the fact that it would largely be redundant.

Edit1.

1. The databases are on the same server.

2. The database files, including logs are in the same directory on the same drive.

3. There are no primary/foriegn keys or constraints set up on the source databases (DatabaseA/B/C) or the data warehouse database at the time of this INSERT INTO.

Edit2. I have ran the above query in management studio and it took 5s!?

Edit3. I have added a temporary CLUSTERED INDEX in a hope that this would assist this query, this has not helped either. I have also tried with WITH(TABLOCK) to attempt to avoid excessive write to the log - no joy! :'[

share|improve this question
Please don't cross-post across sites. If the question needs to be here then request a migration from the source site (SO). – JNK Sep 24 '12 at 14:56
Why? There are people on different sites with different skill sets. I admit I did not mean to post this on ServerFault that was my bad. However, unless this is actually against SO's policies - I don't see why this is so frowned upon. Thanks. – Killercam Sep 24 '12 at 14:59
It's against policies to post the same question on multiple sites. Basically ask it once, and if it needs to be somewhere else we can move it. There will be a pointer on the source site that sends users to the new site, but if you ask it TWICE then you get two versions of it which is problematic for a lot of reasons. – JNK Sep 24 '12 at 15:02
Okay. Understood. I will take down this question... – Killercam Sep 24 '12 at 15:11
I already took care of this one, no worries! If you want the SO question moved here (which we would accept and it would be a good fit here) then just flag it there for migration. – JNK Sep 24 '12 at 15:14

closed as not constructive by JNK Sep 24 '12 at 14:55

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or specific expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, see the FAQ for guidance.

Browse other questions tagged or ask your own question.