3

I have an ssis package which loads 1.2 million records of data from oracle into SQL SERVER 2008R2(with 32 GB RAM) dataware house table once every week. (truncating and loading)

This process is taking 5 hours to complete the load of 1.2 million records.

If any body did the same thing,i wanted to know their average time to complete the load.

I am new to ssis. Please advice me. Is 5 hours an optimum time ??

Note: the destination table is de-normalized and no indexes are there on it because it is in the dataware house.

Improve this question
8
  • Try on the DBA stack. Not programming related - also, more of a "poll".
    – pst
    Commented Nov 17, 2012 at 1:21
  • 4
    This question cannot be answered in its current form. What is the average row length of each record (hence the size of the data). How is the data being transferred from Oracle to SQL Server? How fast are the disks that you're writing to? Is it a decent IO speed for the amount of data that you're dealing with?
    – Philᵀᴹ
    Commented Nov 17, 2012 at 2:13
  • What wait types do you see for your instance while this package is running?
    – user507
    Commented Nov 17, 2012 at 2:15
  • 2
    Take your existing SSIS package. Remove your destination and instead route rows into a Row Count Transformation. Run the package a few times and observe how long it takes just to extract the data and the number of rows extracted. That's your maximum possible throughput given your current configuration. Compare that to your 5 hour run time. Too many variables to get a proper answer though. Besides structure, what's the IOPS on your storage subsystem; how's your network configured; etc, etc.
    – billinkc
    Commented Nov 17, 2012 at 2:44
  • What recovery model is the destination database in? This seems unreasonably slow. Post the table definition for the destination table. It sounds like either your table is really wide, you're in the full recovery model, you have some heavy transformations in the data flow task, the destination table has a clustered index which is forcing the data to be sorted before insert, or a mixture of the above. Additionally, disk speed or network throughput could be part of the issue. Alot of unknowns here.
    – brian
    Commented Nov 17, 2012 at 3:00

2 Answers 2

Reset to default
6

Time taken would depend on

  • Row length
  • Network Bandwidth
  • Available resources on the source DB server
  • And finally on data volume, as in number of rows

To put things in perspective - I recently did a similar migration. A table with 40 columns ( a mix on nvarchar and numeric) and 3 million rows of data took 35 minutes, where as another table with 7.5 million rows of data ( again a mix of nvarchar and numeric), but with 104 columns took more than 7 hours.

Conversely, a third table with 18 million rows of data, 6 columns, all numeric took 14 minutes. It is impossible to answer this question with Yes or No, with the information provided.

Raj

Improve this answer
1
  • Thanks a ton for your suggestions!! I found the reason!! I changed the DataAccessMode of Destination from "Table or view" TO "Table or View-fast load" and the data load completed in 20 minutes.
    – oOo Testing1 oOo
    Commented Nov 26, 2012 at 22:14
1

Can you please post the schema of the source and destination tables? How wide are the rows and are the clustered index keys identical? 5 hours could or could not be optimal, it really depends on:

How fast it can read the data from the source Some factors that affect this are read performance hits on the source:

  • logical and physical fragmentation
  • locking/blocking
  • disk architecture
  • How much data is being pushed through.

Then on the transport layer of the data/package you need to see:

  • network bandwidth/quality

The SSIS package then would try to use bulk inserts assuming your table is properly setup for that. http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server

Then you'd be confined to the weakest link on the destination:

  • DiskIO

  • blocking/locking

  • bad clustering key

You should clear and analyze your waitstats, view the wait stats associated with the load queries, and also read into how to speed up SSIS packages from some blogs such as: http://www.simple-talk.com/sql/reporting-services/using-sql-server-integration-services-to-bulk-load-data/

Improve this answer
5
  • 1
    It says no indexes in the question!
    – Philᵀᴹ
    Commented Nov 17, 2012 at 2:14
  • I think he is referring to indexes on the source
    – Raj
    Commented Nov 17, 2012 at 2:45
  • Phil, I was asking for clustered index on the source, but destination might have a clustered index, which isn't your typical "index", but whether or not if the table is ordered or just a heap. It makes a big difference sometimes. That was less than 10% of the overall answer and it would be good to know all that info. Of course there's many ways to solve a problem, so I'm curious what anyone else might have in mind! – SQL-Learner 1 hour ago
    – Ali Razeghi - AWS
    Commented Nov 17, 2012 at 5:25
  • 1
    Oracle does not have clustered indexes. The thing that is equivalent is an "index organized table" but that requires a special create table statement.
    – user1822
    Commented Nov 17, 2012 at 9:05
  • Thanks a ton for your suggestions!! I found the reason!! I changed the DataAccessMode of Destination from "Table or view" TO "Table or View-fast load" and the data load completed in 20 minutes.
    – oOo Testing1 oOo
    Commented Nov 26, 2012 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.