1

I am considering a model where I use PostgreSQL COPY to copy data from a file into a table. I was wondering what kind of performance to expect on high-end hardware. An interval in MB/s would be nice so I can start estimating. If you just know the answer on other databases, I would be interested to compare.

3

This is a bit hard to tell, what is high-end hardware?

Using Oracle on EMC storage 60MB/s is certainly possible. Using that same old Oracle on exadata can be slightly faster, 300MB/s and sometimes higher, again, depending on the configuration. A full rack is faster than a half rack ...

1

@David, you are almost certainly going to be IO bound on your database server, with the following (commonly understood) caveats:

  1. If your target for the COPY uses FUNCTIONs or TRIGGERs or FOREIGN KEYs, expect slower performance
  2. If you have already created INDEXes before you COPY your data, the data will be added to the INDEX, which will slow things down.

The fastest way to COPY load data in to PostgreSQL is via COPY in to a table that has no INDEXes and has fsync = off, and checkpoint_segments = 50 (or at least some value that isn't the default of 3) in postgresql.conf. I prefer the following strategy:

  1. BEGIN
  2. `DROP INDEX my_tbl_idx;
  3. COPY ...
  4. CREATE INDEX my_tbl_idx ON my_tbl
  5. COMMIT
  6. ANALYZE my_tbl

I explicitly add the ANALYZE after the COMMIT because I don't typically want to wait for pg_autovacuum to start.

  • While fsync = off can speed up the load indeed, it is highly recommended to turn it on again after the data has been loaded to avoid data corruption. – a_horse_with_no_name Jun 1 '11 at 6:12
  • Absolutely. synchronous_commit = off is the preferred & recommended way of getting the speed benefits of fsync = off without sacrificing the integrity of your database. – Sean Jun 1 '11 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.