Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I am considering a model where I use PostgreSQL COPY to copy data from a file into a table. I was wondering what kind of performance to expect on high-end hardware. An interval in MB/s would be nice so I can start estimating. If you just know the answer on other databases, I would be interested to compare.

share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

This is a bit hard to tell, what is high-end hardware?

Using Oracle on EMC storage 60MB/s is certainly possible. Using that same old Oracle on exadata can be slightly faster, 300MB/s and sometimes higher, again, depending on the configuration. A full rack is faster than a half rack ...

share|improve this answer
add comment

@David, you are almost certainly going to be IO bound on your database server, with the following (commonly understood) caveats:

  1. If your target for the COPY uses FUNCTIONs or TRIGGERs or FOREIGN KEYs, expect slower performance
  2. If you have already created INDEXes before you COPY your data, the data will be added to the INDEX, which will slow things down.

The fastest way to COPY load data in to PostgreSQL is via COPY in to a table that has no INDEXes and has fsync = off, and checkpoint_segments = 50 (or at least some value that isn't the default of 3) in postgresql.conf. I prefer the following strategy:

  1. BEGIN
  2. `DROP INDEX my_tbl_idx;
  3. COPY ...
  4. CREATE INDEX my_tbl_idx ON my_tbl
  5. COMMIT
  6. ANALYZE my_tbl

I explicitly add the ANALYZE after the COMMIT because I don't typically want to wait for pg_autovacuum to start.

share|improve this answer
    
While fsync = off can speed up the load indeed, it is highly recommended to turn it on again after the data has been loaded to avoid data corruption. –  a_horse_with_no_name Jun 1 '11 at 6:12
    
Absolutely. synchronous_commit = off is the preferred & recommended way of getting the speed benefits of fsync = off without sacrificing the integrity of your database. –  Sean Jun 1 '11 at 15:17
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.