I am considering a model where I use PostgreSQL COPY to copy data from a file into a table. I was wondering what kind of performance to expect on high-end hardware. An interval in MB/s would be nice so I can start estimating. If you just know the answer on other databases, I would be interested to compare.
This is a bit hard to tell, what is high-end hardware?
Using Oracle on EMC storage 60MB/s is certainly possible. Using that same old Oracle on exadata can be slightly faster, 300MB/s and sometimes higher, again, depending on the configuration. A full rack is faster than a half rack ...
@David, you are almost certainly going to be IO bound on your database server, with the following (commonly understood) caveats:
- If your target for the
FOREIGN KEYs, expect slower performance
- If you have already created
INDEXes before you
COPYyour data, the data will be added to the
INDEX, which will slow things down.
The fastest way to
COPY load data in to PostgreSQL is via
COPY in to a table that has no
INDEXes and has
fsync = off, and
checkpoint_segments = 50 (or at least some value that isn't the default of 3) in
postgresql.conf. I prefer the following strategy:
- `DROP INDEX my_tbl_idx;
CREATE INDEXmy_tbl_idx ON my_tbl
I explicitly add the
ANALYZE after the
COMMIT because I don't typically want to wait for
pg_autovacuum to start.