0

I have 15 Amazon AWS EC2 t1.micro instances which simultaneously populate Amazon RDS MySQL d2.m2.xlarge database with data using large INSERTs (40000 rows in query).

The queries are sent continuously. The table is INNODB, two INT columns, there is index for both columns. CPU utilization of RDS instance is about 30% during data receiving.

When I have one EC2 instance, the speed is in orders of magnitude faster than I run 15 instances simultaneously. In light of this, the 15-instances group works slower and slower until the speed becomes totally unsatisfactory.

How can I optimize performance for this process?

UPDATE

My SHOW CREATE TABLE result is the following:

CREATE TABLE `UserData` (
 `uid` int(11) NOT NULL,
 `data` int(11) NOT NULL,
 PRIMARY KEY (`uid`,`data`),
 KEY `uid` (`uid`),
 KEY `data` (`data`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

I need 2 indexes because it is necessary for me to fetch data by uid and by data value.

I insert data with INSERT INTO UserData (uid, data) VALUES (1,2),(1,3),(1,10),... with 40000 (uid,data) pairs.

15 parallel instances insert ~121 000 000 rows in 2 hours, but I am sure that it can be much more faster.

Improve this question
4
  • Please present the SHOW CREATE TABLE your_table output. In particular, is there an AUTO_INCREMENT column? Which PRIMARY KEY? What is the order by which rows are inserted?
    – Shlomi Noach
    May 13, 2013 at 5:10
  • @ShlomiNoach Thank you for reply, I have updated the post
    – zavg
    May 13, 2013 at 12:42
  • Are your INSERTs enclosed into a transaction? START TRANSACTION; INSERT ... and COMMIT;
    – Cristian Porta
    May 13, 2013 at 18:50
  • @crisrian-porta Not, but I use bulk inserts with 40000 values in one query.
    – zavg
    May 13, 2013 at 19:04

1 Answer 1

Reset to default
2

Two hints for you:

  1. The KEY uid is redundant, because it is covered by the PRIMARY KEY

  2. 40,000 rows at a time might make for too large a transaction. Although these are very small rows (two INTs) this may cause the transaction to go to disk, depending on your settings. I usually go with around 1,000 rows at a time (I go as low as 100 and as high as 10,000). Please try doing 40 * 1,000 and see if this works better for you than 1 * 40,000

Improve this answer
3
  • Ok, thank you for advice. I will try your technique and measure the performance increase. Currently I am using such optimizations: 1) I have dropped all indexes while writing to db, 2) I have set AUTOCOMMIT=0; and COMMIT after very large sequence of INSERTs (about 1500000 rows), 3) I am using increased innodb_buffer_pool_size and bulk_insert_buffer_size, 4) also I changed t1.micro instances to m1.small and m2.xlarge to m1.xlarge for better IO performance
    – zavg
    May 17, 2013 at 21:09
  • number 2) sound like an anti-optimization to me. And how do you mean you've dropped all indexes while writing? Do you mean you're ALTERing the table before and after the INSERT? If so - don't do it.
    – Shlomi Noach
    May 19, 2013 at 17:00
  • I insert data to the table without indexes and add them after all INSERTs. then I don't write to this table.
    – zavg
    May 19, 2013 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.