Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

To summarize the specifics: We need to stage approximately 5 million rows into a vendor (Oracle) database. Everything goes great for batches of 500k rows using OracleBulkCopy (ODP.NET), but when we try to scale up to 5M, the performance starts slowing to a crawl once it hits the 1M mark, gets progressively slower as more rows are loaded, and eventually times out after 3 hours or so.

I suspect it's related to a primary key on the table, but I've been trawling the Oracle forums and Stack Overflow for information and a lot of what I'm reading contradicts that (also, a lot of posts seem to contradict each other). I'm hoping that somebody can set the record straight on some closely-related questions about the process:

  1. Does the OracleBulkCopy class use conventional or direct-path loading? Is there some way I can confirm this, one way or another?

  2. Assuming it does use direct-path loading: Is it true that Oracle automatically sets all indexes to unusable during the load and puts them back online afterward? I've read several statements to this effect but again, cannot confirm it.

  3. If #2 is true, then should it make any difference what indexes are on the table before I initiate a bulk copy operation? If so, why?

  4. Related to #3, is there any practical difference, in general, between bulk loading with an unusable index vs. actually dropping the index before the load and recreating it afterward?

  5. If #2 is not correct, or if there are some caveats I'm not understanding, then would it make any difference to explicitly make the index unusable before the bulk load, and then explicitly rebuild it afterward?

  6. Is there anything else, other than index builds, that could cause a bulk copy operation to grow progressively slower as more and more records are added? (Maybe something to do with logging, although I would expect that bulk operations are not logged?)

  7. If there really is no other way to get the performance up to snuff aside from dropping the PK/index first, what steps can I take to make sure that the index doesn't completely disappear, i.e. if the connection to the database is lost in the middle of the process?

share|improve this question
    
Side note: The data being copied is already in sorted order according to the PK, which is the only index on the table. –  Aaronaught Oct 26 '11 at 23:59
    
Are you using a DataReader to read the data from the source? –  bernd_k Oct 27 '11 at 5:10
    
@bernd_k: No, loading entirely from memory. It's definitely not the source that's the problem. –  Aaronaught Oct 30 '11 at 15:25

1 Answer 1

up vote 6 down vote accepted

A few more days of reading and experimentation and I was able to (mostly) answer a lot of these:

  1. I found this buried in the ODP.NET documentation (ironically not in the OracleBulkCopy docs):

    The ODP.NET Bulk Copy feature uses a direct path load approach, which is similar to, but not the same as Oracle SQL*Loader. Using direct path load is faster than conventional loading (using conventional SQL INSERT statements).

    So it appears that it does use direct path.

  2. This I was able to verify by doing a large bulk copy operation and getting the index properties from SQL Developer. The index did appear as UNUSABLE while the bulk copy was in progress. However, I've also discovered that OracleBulkCopy.WriteToServer will refuse to run if the index starts in an UNUSABLE state, so clearly there's more going on here, because if it were as simple as disabling and rebuilding the index then it shouldn't care about initial state.

  3. It does make a difference specifically if the index is also a constraint. Found this little gem in the documentation linked above:

    Enabled Constraints
    During an Oracle bulk copy, the following constraints are automatically enabled by default:

    • NOT NULL
    • UNIQUE
    • PRIMARY KEY (unique-constraints on not-null columns)

    NOT NULL constraints are checked at column array build time. Any row that violates the NOT NULL constraint is rejected.

    UNIQUE constraints are verified when indexes are rebuilt at the end of the load. The index is left in an Index Unusable state if it violates a UNIQUE constraint.

    The documentation is a little hazy on what happens during the load, especially with primary keys, but one thing is absolutely certain - it behaves differently with a primary key vs. without one. Since the OracleBulkCopy will happily allow you violate index constraints (and punt the index into UNUSABLE state when it's done), my hunch is that it's building the PK index during the bulk copy but simply not validating it until afterward.

  4. I'm not sure whether the difference observed is within Oracle itself or just a quirk of the OracleBulkCopy. The jury's still out on this one.

  5. OracleBulkCopy will throw an exception if an index is initially in the UNUSABLE state, so it's really a moot point.

  6. If there are other factors, indexes (and especially PK indexes) are still the most important, as I found out by:

  7. Creating a global temporary table with the same schema (using CREATE AS), then bulk copying into the temporary table, and finally doing a plain old INSERT from the temp table into the real table. Since the temp table has no index, the bulk copy happens very fast, and the final INSERT is also fast because the data is already in a table (I haven't tried the append hint yet, since a 5M row table-to-table copy already takes less than 1 minute).

    I'm not yet sure of the potential ramifications of (ab)using the temporary table space this way, but so far it hasn't given me any trouble, and it's much safer than the alternative by way of preventing corruption of either the rows or indexes.

    The success of this also pretty clearly demonstrates that the PK index is the problem, as that is the only practical difference between the temp table and the permanent table - both started with zero rows during the performance tests.

Conclusion: Don't bother trying to bulk copy more than around 100k rows into an indexed Oracle table using ODP.NET. Either drop the index (if you don't actually need it) or "preload" the data into a different (non-indexed) table.

share|improve this answer
    
I'm not sure about checking primary key constraints. I happed to bulk insert the same data into an Oracle table 2 times and Select * shows 2 the duplicated rows. In that state Delete is not possible, but truncate table helps to get back to a clean state. –  bernd_k Oct 30 '11 at 15:31
    
@bernd_k: Delete is not possible because the index is UNUSABLE. That is a result of the constraint check which happens at the end of the bulk copy. –  Aaronaught Oct 30 '11 at 15:34
    
I have a PowerShell skript running, calling bulkcopy into an Oracle database from an SQL Server data reader, all target tables with primary Keys and I have no problems with tables with up to 205278 rows. But I'm very careful to first full the master tables before filling the details tables. I did not remove any of the other index on the table and I have no problems when the tables are initially empty. –  bernd_k Oct 30 '11 at 15:59
    
@bernd_k: Yes, I didn't have too many problems at that volume either (see my last paragraph). It's when you reach into the millions that it gets awful. Also there may be a difference if you empty the table sometime after each bulk copy (this one doesn't get emptied, it gets appended to, and you know how indexes get slower as they get larger). –  Aaronaught Oct 30 '11 at 17:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.