Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I am running a Perl script (using DBI) which reads from raw files from a hard disk, and updates MySQL database (which is on a separate SSD). My performance is rather slow (1000 files processed in 30-60 seconds), but I can not find the bottleneck.

CPU. Network, Disk and Memory are all rather unused. I am running Windows 7 64bit on an i7 machine (8 cores). MariaDB is version 10.0.10. My database is around 78G in size with 5M entries, all tables properly indexed.

Perfmon confirms this showing total CPU 6%, Network 0%, HD Disk 140 I/O/s, Memory 10%. None of the CPU cores are used more than 3-4%.

I have experimented with changing all these mysql variables with no success: innodb_use_global_flush_log_at_trx_commit innodb_buffer_pool_instances global.max_connections innodb_thread_sleep_delay global.innodb_io_capacity global.innodb_sync_spin_loops innodb_flush_log_at_trx_commit wait_timeout

Mysqltuner did not report anything of interest, except for: Data in InnoDB tables: 78G (Tables: 12) Total fragmented tables: 10 Query cache is disabled Thread cache is disabled InnoDB data size/buffer pool: 78.1G/2.0G

Perl profiling showed that majority of time is taken up by DBI::st::execute (invoking sql).

I have also tried disabling firewall and virus scanner too - no difference.

share|improve this question

migrated from stackoverflow.com Jul 19 at 7:18

This question came from our site for professional and enthusiast programmers.

    
Can you give the code of your script? –  nightfox79 Jul 18 at 20:22
    
Are you using prepared statements in your $sth->execute calls? –  friedo Jul 18 at 20:39
    
140 IOPS is pretty high for a mechanical drive. That's probably your bottleneck. –  AKHolland Jul 18 at 21:15
    
Windows 7 64bit -- i think thats your problem. Running what you have is not ideal to be running on a Desktop. Now if you were using a Linux Server like Ubuntu or Redhat on your system youl see a speed gain between 25 to 50 % across the board. –  Tasos Jul 19 at 6:29
    
Sorry for not answering before. I was on an extended vacation. –  Shalabajzer Aug 13 at 1:06

1 Answer 1

You do not provide much information about your system or program, but here are very general pieces of advise to improve the writing performance of your system:

  • Number one reason why MySQL can be slow is because it has inappropriate queries: are tables indexed correctly? Do you have a fast and flexible table design? Are you using memory efficiently? Partitioning? Do your queries involve too many disk IOPS? You should answer those questions first, then try to tune your server parameters.

  • Profile MySQL in the same way that you did with Perl. Low CPU usage doesn't mean there isn't a problem, as actually, it could be a symptom of a bottleneck. There are several tools available to do that, but one is integrated inside the server itself (performance_schema).

  • Augment your buffer pool (innodb_buffer_pool_size). It is the main buffer system used by InnoDB. You will probably not be able to cover your entire dataset (78 GB), but it probably can be higher than 2GB -you have a lot of free memory. The rule of the thumb is making it 80% of your physical memory, but your mileage may vary. Make sure that you are not swapping in any case.

  • Augment your transaction log (innodb_log_file_size). The default is too small in most versions, and all writes have to be modify this file. If it is not large enough, you will get a bottleneck on your writes.

  • Reduce durability (innodb_flush_log_at_trx_commit = 2). As you are writing from disk files, your database crashing may not affect too much your data durability (corruption is not affected when tuning this parameter, innodb is always consistent with disk). Allowing more operations to be buffered in memory is dangerous for banking operations, but maybe you can get by in your particular workload, making the performance much better.

  • innodb_log_buffer_size. Large transactions can become a bottleneck, specially involving large blob-like fields. Make sure you monitor Innodb_log_waits and configure this variable accordingly.

  • Logs - Make sure that you disable or limit their output (general log, slow log, binary log). The binary log is not usually recommended to be disabled -it is useful for disaster recovery and replication-, but in extreme cases, with lots of (large) updates it can be a performance penalty.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.