I have a cluster of asynchronous workers that will be extracting text from PDFs and inserting them into a single MySQL table with no relations. This data will then be slurped into a Solr server via a DataImportHandler.

Each worker will parse a PDF containing 50-1000 pages and insert between 1000-18000 rows of data over eight text columns. Seven are varchar(255) and one is a longtext with an average length of possibly 500 characters.

Each worker will take anywhere from one to three minutes to complete the task. The insert occurs via Django ORM's bulk_insert operation.

So, at worst case, I can assume about 270K rows inserted per minute to this table. Can I expect any configuration issues with a default config server?

Hardware is a standard EC2 T2.small instance with a 100GB SSD EBS volume.

  • insert operations are usually faster as they have to write the rows, it may also depend upon the indexes you have as more are the indexes more would be the cost as insert would required to be sorted against those indexes. If you face any issue do post create table for problematic tables to get more insight for the issue – Nawaz Sohail Nov 1 '16 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.