Derby slows down after 1.2 million records

Question

I'm using a Derby database, mass insertions slow down to 1/4 to 1/6 the speed once there are 1.2 million records (about 6 GB database size, one major table, other tables are tiny). Is that normal? Is there something I can do to tweak it and make it run faster? I compressed the main table.

Questions

• Should I consider other databases?
• How could I compare Derby's performance, with, say, MySQL in this kind of scenario?
• Are there such comparisons already?

(Running on Windows 7-based server R2, 16GB RAM, 8-core machine).

Answer 1

DISCLAIMER : Not a Derby Expert

DERBY

There are options you can set to increase data pages for Derby

For a mass load of table prodtable, you may want to consider

• create temptable just like prodtable, but with no indexes
• load data into the temptable
• rebuild indexes on temptable
• rename prodtable to zaptable
• rename temptable to prodtable
• drop zaptable

MySQL

If you switch over to MySQL, you could do wonderful things to performance tune data processing

• Start with using only InnoDB
• You can configure InnoDB to
  • hyperthread read threads
  • hyperthread write threads
  • access multiple CPUs/Cores
  • setup partitions buffer pools

Here are some of my past posts on configuring InnoDB in MySQL for multicore performance

• Is the CPU performance relevant for a database server? (Apr 26, 2012)
• Multi cores and MySQL Performance (Sep 20, 2011)
• Possible to make MySQL use more than one core? (Sep 12, 2011)
• How do you tune MySQL for a heavy InnoDB workload? (Feb 12, 2011)

While I may be a little biased towards MySQL, make sure you have Derby's caching configured to accommodate mass INSERTs. That way, you need not perform a major data migration if Derby can be made to handle large transactions.