Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I want to performance test (aka bake-off) MySQL server rpm against some other forks such as Percona server, MariaDB, and possibly some others. I'm hoping that by asking this question I can better understand the methodology behind setting up a proper performance test. I planned on using sysbench to run my actual test, but I'm open to anything.

  1. What steps should I be taking to ensure that the test results in a apples-to-apples comparison and that only the RDBMS is the variant?
  2. Where do I get started?
  3. How do I evaluate results?
  4. What advice can you give me?
share|improve this question
    
I added a link to the MONyog White Paper in my answer !!! –  RolandoMySQLDBA Jun 19 '11 at 22:17
    
This question deserves a +1 becasue I am sure many have thought about this along the line since MySQL and its forks look all the same to most people. A side-by-side comparison from an individual's point-of-view would be a little more refreshing than seeing big companies present charts always showing their version of MySQL as the best. This could encourage more comparison shopping for MySQL Databases. –  RolandoMySQLDBA Jun 19 '11 at 23:55
    
I updated my answer with some suggested metrics !!! –  RolandoMySQLDBA Jun 20 '11 at 17:18
    
Thank you, kind sir !!! –  RolandoMySQLDBA Jul 6 '11 at 2:02
1  
Randy might want to check out someone's MariaDB 5.2 vs Percona 5.1/55 vs MySQL 5.0/5.1/5.5 sysbench results at vbtechsupport.com/606 and vbtechsupport.com/657 to some ideas. –  p4guru Aug 28 '11 at 9:17
show 2 more comments

2 Answers

up vote 7 down vote accepted

Here is the first thing I thought about

  1. Setup 3 DB servers (identical HW/OS configs) each installed with

    • MySQL
    • Percona
    • MariaDB
  2. Get a Fourth Server installed with MONyog (eval version lasts 30 days)

  3. Register the 3 DB servers in MONyog

  4. Use the Performance Metrics Charts in MONyog or Set Up Your Own Charts in MONyog

  5. Use SysBench against all 3 DB Servers at your discretion

This is just a basic outline. Believe me, you can use Monyog right out of the box. Check out my testimonial at MonYog's Website (Look for my name on the page). I eventaully convinced my company to purchase the Utlimate Version outright after my testimonial, which they watched with me as it happened.

Caveat : Do no get Monyog until you have the 3 DB servers and Sysbench Ready to get the best 30-day usage of MonYog.

UPDATE 2011-06-20 13:18 EDT

If you keep the InnoDB Buffer Pool Small (default for innodb_buffer_pool_size is 8M) and keep the binary logs disabled (do no include log-bin in my.cnf), then these metrics should be measured

Many of these are default graphs in MONyog's Monitors/Advisors Section, and more graphs can be created to check other desired MySQL metrics. MONyog also lets you export the graphs as CSV data.

UPDATE 2011-10-12 14:17 EDT

Percona performed a bake-off among multiple releases of MySQL

  • MySQL 4.1
  • MySQL 5.0
  • MySQL 5.1 (with built-in InnoDB)
  • MySQL 5.1 with InnoDB-plugin
  • MySQL 5.5
  • MySQL 5.6

All tests were performed with MySQL unconfigured. The results?

  • MySQL 4.1 performs the best single-threaded
  • MySQL 5.1 with InnoDB plug-in scales on multiple cores better than 5.1 InnoDB built, 5.5 and 5.6

What do I get out of this? Your must configure MySQL 5.5/5.6 to get multicore enhancements engaged.

share|improve this answer
    
You can use all this without MonYog,for benchmarking purposes. Sysbench delivers some transactions persecond or operations per second. –  Vladislav Vaintroub Jun 20 '11 at 15:51
    
What about the documentation and analysis side of this? What metrics matter? –  randy melder Jun 20 '11 at 15:55
    
@rolandoMySQLDBA: +1 for the excellent answer as usual –  ursitesion Mar 17 at 13:07
add comment

This should be very helpful to you as this is the defacto industry benchmark for databases, TPC-C, http://www.tpc.org/tpcc/spec/tpcc_current.pdf

Even if you do not use execute this benchmark, the information contained within the test definition document should provide you with tons of insight into setting up your own benchmark for testing your database options.

share|improve this answer
    
Nice, thanks! +1 –  randy melder Sep 26 '11 at 20:12
1  
Percona has an implementation of tpcc for mysql variants; code.launchpad.net/~percona-dev/perconatools/tpcc-mysql –  Sean McMillan Apr 29 '13 at 13:32
    
Very useful Sean: Thank you for updating this thread with the Percona reference –  James Pulley Apr 30 '13 at 13:15
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.