Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have an ssis package which loads 1.2 million records of data from oracle into SQL SERVER 2008R2(with 32 GB RAM) dataware house table once every week. (truncating and loading)

This process is taking 5 hours to complete the load of 1.2 million records.

If any body did the same thing,i wanted to know their average time to complete the load.

I am new to ssis. Please advice me. Is 5 hours an optimum time ??

Note: the destination table is de-normalized and no indexes are there on it because it is in the dataware house.

share|improve this question

migrated from stackoverflow.com Nov 17 '12 at 1:52

This question came from our site for professional and enthusiast programmers.

    
Try on the DBA stack. Not programming related - also, more of a "poll". –  pst Nov 17 '12 at 1:21
4  
This question cannot be answered in its current form. What is the average row length of each record (hence the size of the data). How is the data being transferred from Oracle to SQL Server? How fast are the disks that you're writing to? Is it a decent IO speed for the amount of data that you're dealing with? –  Phil Nov 17 '12 at 2:13
    
What wait types do you see for your instance while this package is running? –  Shawn Melton Nov 17 '12 at 2:15
2  
Take your existing SSIS package. Remove your destination and instead route rows into a Row Count Transformation. Run the package a few times and observe how long it takes just to extract the data and the number of rows extracted. That's your maximum possible throughput given your current configuration. Compare that to your 5 hour run time. Too many variables to get a proper answer though. Besides structure, what's the IOPS on your storage subsystem; how's your network configured; etc, etc. –  billinkc Nov 17 '12 at 2:44
    
What recovery model is the destination database in? This seems unreasonably slow. Post the table definition for the destination table. It sounds like either your table is really wide, you're in the full recovery model, you have some heavy transformations in the data flow task, the destination table has a clustered index which is forcing the data to be sorted before insert, or a mixture of the above. Additionally, disk speed or network throughput could be part of the issue. Alot of unknowns here. –  brian Nov 17 '12 at 3:00

2 Answers 2

Time taken would depend on

  • Row length
  • Network Bandwidth
  • Available resources on the source DB server
  • And finally on data volume, as in number of rows

To put things in perspective - I recently did a similar migration. A table with 40 columns ( a mix on nvarchar and numeric) and 3 million rows of data took 35 minutes, where as another table with 7.5 million rows of data ( again a mix of nvarchar and numeric), but with 104 columns took more than 7 hours.

Conversely, a third table with 18 million rows of data, 6 columns, all numeric took 14 minutes. It is impossible to answer this question with Yes or No, with the information provided.

Raj

share|improve this answer
    
Thanks a ton for your suggestions!! I found the reason!! I changed the DataAccessMode of Destination from "Table or view" TO "Table or View-fast load" and the data load completed in 20 minutes. –  oOo Testing1 oOo Nov 26 '12 at 22:14

Can you please post the schema of the source and destination tables? How wide are the rows and are the clustered index keys identical? 5 hours could or could not be optimal, it really depends on:

How fast it can read the data from the source Some factors that affect this are read performance hits on the source:

  • logical and physical fragmentation
  • locking/blocking
  • disk architecture
  • How much data is being pushed through.

Then on the transport layer of the data/package you need to see:

  • network bandwidth/quality

The SSIS package then would try to use bulk inserts assuming your table is properly setup for that. http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server

Then you'd be confined to the weakest link on the destination:

  • DiskIO

  • blocking/locking

  • bad clustering key

You should clear and analyze your waitstats, view the wait stats associated with the load queries, and also read into how to speed up SSIS packages from some blogs such as: http://www.simple-talk.com/sql/reporting-services/using-sql-server-integration-services-to-bulk-load-data/

share|improve this answer
1  
It says no indexes in the question! –  Phil Nov 17 '12 at 2:14
    
I think he is referring to indexes on the source –  Raj Nov 17 '12 at 2:45
    
Phil, I was asking for clustered index on the source, but destination might have a clustered index, which isn't your typical "index", but whether or not if the table is ordered or just a heap. It makes a big difference sometimes. That was less than 10% of the overall answer and it would be good to know all that info. Of course there's many ways to solve a problem, so I'm curious what anyone else might have in mind! – SQL-Learner 1 hour ago –  Ali Razeghi Nov 17 '12 at 5:25
1  
Oracle does not have clustered indexes. The thing that is equivalent is an "index organized table" but that requires a special create table statement. –  a_horse_with_no_name Nov 17 '12 at 9:05
    
Thanks Horse, that's good to know! –  Ali Razeghi Nov 17 '12 at 18:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.