I would like to suggest something radical. I got this idea from StarTrek : Deep Space 9 (Call to Arms)
A minefield was set up around a wormhole to prevent the Dominion
Forces and the Jemhadar from passing through. Each explosive mine was
armed with a replicator that allowed a mine to recreate another mine
after it explodes. Hence, the minefield stays up indefinitely.
With that DS9 analogy I bring your an interesting idea.
You set up 2 initial read slaves with MySQL Slave with innodb disabled
This is optional. My preference is an all-MyISAM slave to do reads because it is faster for reads than InnoDB for small datasets. Should you choose to go with InnoDB, make sure you relax the ACID compliance with this
innodb_flush_log_at_trx_commit = 0
In the event of a crash, just destroy the Slave and spin up a new one
We'll call the Slaves S0 and S1
Here is something else: have this in /etc/my.cnf in S0
innodb_max_dirty_pages_pct = 0;
innodb_fast_shutdown = 0
These will help S0 shutdown fast with completely flushed data.
The following is what you must script in the replicator process
When you need to generate a new slave (we will call it S2), here is what you must do
STEP 01) On S0, run
service mysql stop
STEP 02) Install the same version of MySQL that S0 has into S2
STEP 03) On S0,
scp /etc/my.cnf S2:/etc/.
STEP 04) On S2, you need to change the
server-id of /etc/my.cnf in S2 to a Unique Value (suggestion : use the 2nd,3rd, and 4th octet [without the dots] of the private IP of S2)
STEP 05) On S2, either remove or comment out the
innodb_max_dirty_pages_pct = 0 from /etc/my.cnf
STEP 06) On S0, run
rsync -av /var/lib/mysql S2:/var/lib/mysql (Note: If you have to spin up 5 Slaves, run the 5 rsyncs at this point)
STEP 07) On S2,
service mysql start (MySQL Replication start immediately where S0 had left off)
STEP 08) On S0, run
service mysql start
Once you create the replicator script, you can use it spin up MySQL on new Slaves.
Meanwhile, S1 is available for SELECT queries and continues replicating.
S0 is used to recreate a new Slave.
If you are doing this in conjunction with spinning up Amazon EC2 or some other Cloud DB Servers, check with your System Administrators on any Linux commands/API that allows you to spin up a DB Server. Then, you apply the replicator to the newly generated DB Server. Even better, you can incorporate the Db Server creation API in your Replicator Script.
If you have to spin up dozens or hundreds of Slaves, all you need to do is have 10 Replicator Slave Servers (S0 - S9) and have 10 copies of the replicator script operator on different Replicator Slaves.