Stress Test to Saturate Memory for MySQL InnoDB

Question

I'm investigating windows 32bit memory problems (the 2G limit) and am after a test which will max out MySQL's memory including InnoDB Buffer Pool, per connection memory and/or any other uses of memory.

Perhaps a query I could use for mysqlslap?

Update

I thought other people would benefit from a simple way to load test mysql and this was an easy question where lots of people would have "tools". They would help my obscure situation and I could use to investigate our real problem.

We have our configuration tuned, and it was fine for months but recently we eventually get mysqld: Out of memory (Needed 220428 bytes) and MySQL will crash minutes later.

I'm not researching your usual "whoops I allocated too much to my buffer pool" problem. That memory is allocated at start up and improper setting causes immediate shutdown. I'm really after the other 20%. Just creating the maximum number of connections with a high buffer pool hasn't proven to reproduce the problem so I want to stress everything. I am not ruling out stressing the buffer pool in case we are actually suffering a MySQL bug relating to it.

Here is an information dump for people interested:

• /3GB to overcome 2GB for mysql - we haven't set /LARGEADDRESSAWARE but we do have /3GB and /PAE with 8gb memory

• We are at risk of low System Page Table Entries but last we saw over 18,000 (not a lot but enough, it stayed at this level at last crash)

• Mysql keep fixing memory related errors in their change logs

• Mysql 5.1 let you put the inndb buffer pool in Address Windowing Extensions (i.e. use PAE) with innodb_buffer_pool_awe_mem_mb but MySQL 5.5 has no such option

I'm no authority on any of the above I just want to be able to produce high memory activity so we can A) reproduce the problem and then B) solve it in the medium term before moving off 32bit, which is the real solution but it won't happen tomorrow.

Update 2

Turns out Mysql 5.5's mysqld.exe is already built to be /LARGEADDRESSAWARE but cannot set the innodb engine size equal to or greater than 1600M. Connections do use the extra memory from /3GB (proven with mysqlslap).

vcvars32.bat
dumpbin.exe /headers mysqld.exe
Microsoft (R) COFF/PE Dumper Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file mysqld.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        4E6A3CEF time date stamp Sat Sep 10 04:21:03 2011
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             122 characteristics
                   Executable
                   Application can handle large (>2GB) addresses
                   32 bit word machine

My conclusion. If you don't use /3GB or otherwise set connection memory + innodb buffer memory greater than available memory you are bound to eventually get mysqld: Out of memory... although mysql seems to do it's best to reject connections to avoid this problem.

Take home message, test that you can actually achieve max_connections with mysqlslap and the rest comes down to maintaining OS memory (and other resources like free system PTE).

Answer 1

From Windows Platform Limitations in the MySQL 5.5 Reference Manual:

On Windows 32-bit platforms, it is not possible by default to use more than 2GB of RAM within a single process, including MySQL.

What's to investigate? We already know what happens when you max the memory: Really Bad Things™.

If there's a remote chance that your setup will periodically require this much memory, 32-bit Windows is the wrong platform.

Almost universally, the biggest memory consumer is of course the InnoDB Buffer Pool, but there's no need for any queries to test this... you don't need any activity at all, or any tables, because the entire amount of memory declared for innodb_buffer_pool_size is allocated immediately when the server starts. The buffer pool never grows, never shrinks, never changes, ever. The number of free pages changes, but those are not "free" from the operating system's perspective -- they're still just as allocated, merely marked as containing nothing of interest within InnoDB.

If the operating system refuses to allocate the amount of memory provisioned for the buffer pool, MySQL will simply refuse to start.

This is illustrated here, where the OP mistakenly believed that the server was crashing and restarting "because" memory couldn't be allocated for the buffer pool, but was in fact crashing for a Linux-specific reason but then refusing to restart because the system would not allocate the total amount of memory required for the pool, due to overuse of available memory by something else... but this allocate-all-at-startup behavior for the InnoDB buffer pool is not platform-specific.

So, you should be able to set this value near the max and then find that taking the server process over the edge should not require very much additional effort at all. But I'm still not sure what the point is.

As you realize, MySQL uses memory for a variety of different purposes, several of which are dynamically-sized, definable on a per-connection basis, and allocated on demand, which makes it virtually impossible to provision a server based on limiting memory usage to some worst-case scenario absolute value, yet expecting that server to be able to handle its typical load efficiently.

The simplest illustration of this is the fact that you can obviously reduce the theoretical maximum memory utilization of a given instance by restricting the maximum number of simultaneous client connections... but any given application needs a certain number of available connections to perform efficiently, and if that number is below your target value, then you're not really solving anything -- it just feels like you have.

I say, either your server has enough memory for the workload, or it doesn't. If it doesn't, then attempting to "tune" your way out of potential trouble is unlikely to offer much in the way of solutions.

---

Some ideas on how to easily generate demand for more memory...

SELECT * FROM large_table ORDER BY non_indexed_column;

SELECT * FROM large_table WHERE non_indexed_column = some_value;

SELECT * FROM large_table WHERE some_column LIKE '%a_freqent_match%';

Queries like these could trigger the allocation of a sort buffer, a read buffer and/or a random read buffer, which should make a new request to the OS for the memory that buffer requires.

---

Simple solution:

DELIMITER $$

DROP PROCEDURE IF EXISTS `test`.`eat_memory_until_server_crashes` $$
CREATE PROCEDURE `test`.`eat_memory_until_server_crashes`()
BEGIN

-- this procedure is intended to eat as much memory as it can
-- it creates a series of consecutively-numbered session variables as large
-- as your configuration will allow them to be.

-- do not run this unless you intend to crash your server

-- also, do not run from a gui tool -- use the mysql command line client:

-- mysql> CALL test.eat_memory_until_server_crashes;

-- if you kill the query or thread before the server crashes, 
-- the memory consumed will be returned to the OS

  DECLARE counter INT DEFAULT 0;
  LOOP
    SET counter = counter + 1;
    SET @qry = CONCAT('SET @crash_me_',counter,' := REPEAT(\'a\', @@max_allowed_packet)');
    SELECT counter, @qry;
    PREPARE hack FROM @qry;
    EXECUTE hack;
    DEALLOCATE PREPARE hack;
    -- adjust timing or remove this entirely depending on how quickly you want this to happen
    DO SLEEP(0.1);
  END LOOP;

END $$

DELIMITER ;

Inspiration for this: Schwartz, Baron; Zaitsev, Peter; Tkachenko, Vadim (2012-03-05). High Performance MySQL: Optimization, Backups, and Replication (Kindle Location 12194). OReilly Media - A. Kindle Edition.

Answer 2

There is way to make the InnoDB constantly flush.

First here are some internals on the InnoDB Buffer Pool:

InnoDB manages the pool as a list, using a variation of the least recently used (LRU) algorithm. When room is needed to add a new block to the pool, InnoDB evicts the least recently used block and adds the new block to the middle of the list. This “midpoint insertion strategy” treats the list as two sublists:

At the head, a sublist of “new” (or “young”) blocks that were accessed recently.

At the tail, a sublist of “old” blocks that were accessed less recently.

This algorithm keeps blocks that are heavily used by queries in the new sublist. The old sublist contains less-used blocks; these blocks are candidates for eviction.

The LRU algorithm operates as follows by default:

3/8 of the buffer pool is devoted to the old sublist.

The midpoint of the list is the boundary where the tail of the new sublist meets the head of the old sublist.

When InnoDB reads a block into the buffer pool, it initially inserts it at the midpoint (the head of the old sublist). A block can be read in because it is required for a user-specified operation such as an SQL query, or as part of a read-ahead operation performed automatically by InnoDB.

Accessing to a block in the old sublist makes it “young”, moving it to the head of the buffer pool (the head of the new sublist). If the block was read in because it was required, the first access occurs immediately and the block is made young. If the block was read in due to read-ahead, the first access does not occur immediately (and might not occur at all before the block is evicted).

As the database operates, blocks in the buffer pool that are not accessed “age” by moving toward the tail of the list. Blocks in both the new and old sublists age as other blocks are made new. Blocks in the old sublist also age as blocks are inserted at the midpoint. Eventually, a block that remains unused for long enough reaches the tail of the old sublist and is evicted.

By default, blocks read by queries immediately move into the new sublist, meaning they will stay in the buffer pool for a long time. A table scan (such as performed for a mysqldump operation, or a SELECT statement with no WHERE clause) can bring a large amount of data into the buffer pool and evict an equivalent amount of older data, even if the new data is never used again. Similarly, blocks that are loaded by the read-ahead background thread and then accessed only once move to the head of the new list. These situations can push frequently used blocks to the old sublist, where they become subject to eviction.

Given this information, the idea is to get the InnoDB Buffer Pool to flash as frequently as possible. Here are the options you need to set:

• innodb_max_dirty_page_pct : The main thread in InnoDB tries to write pages from the buffer pool so that the percentage of dirty (not yet written) pages will not exceed this value. This contributes to dirty pages being flushed frequently when Write IO slows down.
• innodb_old_blocks_pct : Specifies the approximate percentage of the InnoDB buffer pool used for the old block sublist. Setting this to 95 (the max value) makes InnoDB evict 95% of dirty pages out of the InnoDB Buffer Pool. Even neighbor pages get flushed.
• innodb_old_blocks_time : Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before it can be moved to the new sublist. If the value is 0, a block inserted into the old sublist moves immediately to the new sublist the first time it is accessed, no matter how soon after insertion the access occurs. If the value is greater than 0, blocks remain in the old sublist until an access occurs at least that many ms after the first access. For example, a value of 1000 causes blocks to stay in the old sublist for 1 second after the first access before they become eligible to move to the new sublist.

For a Windows machine with 4G of RAM and a dual core processor, set these values in my.ini:

[mysqld]
innodb_buffer_pool_size=1536M # limited by 32 bit
innodb_buffer_pool_instances=2
innodb_max_dirty_pages_pct=0
innodb_old_blocks_pct=95
innodb_old_blocks_time=50

and then restart mysql.

What about the the stress test itself? All you may need is a very large mysqldump to import and let the InnoDB Buffer Pool go crazy. If you have a small mysqldump, create 100 database and load the mysqldump into each of them, sequentially or in parallel.

Would you like sample data?

cd
rm -rf SampleData
mkdir SampleData
cd SampleData
wget http://downloads.mysql.com/docs/menagerie-db.tar.gz
wget http://downloads.mysql.com/docs/sakila-db.tar.gz
wget http://downloads.mysql.com/docs/world.sql.gz
tar zxf menagerie-db.tar.gz
tar zxf sakila-db.tar.gz
gzip -d world.sql.gz
cd menagerie-db
mv *.sql ..
mv *.txt ..
cd ..
rmdir menagerie-db
mkdir sakila-db
cd sakila-db
mv sak* ..
cd ..
rmdir sakila-db
rm -f *.tar.gz

Here is what you can do:

• Make 100 databases
• Load the SQL data into each of those databases
• mysqldump --all-datbasaes into a single text file (call it MySQL100Data.sql)
• Reboot the Windows Machine
• start mysql (if it does not autostart)
• Load MySQL100Data.sql into mysql

Here is another test:

• Reboot the Windows Machine
• Make 100 databases
• Load the SQL data into each of those databases in parallel

In either case, You can monitor RAM while MySQL is getting beat down...

Give it a Try !!!

CAVEAT

This will have the side effect of Disk IO due to flushing the InnoDB Buffer Pool.

Answer 3

Test you haven't over allocated memory

The innodb buffer pool is preallocated at startup. But connection memory is not, trying achieving max_connections with mysqlslap (e.g 500)

mysqlslap -uroot -ppassword -h192.168.1.1 --auto-generate-sql --concurrency=500 --number-of-queries=2000 

If you are ok you will get

 mysqlslap: Error when connecting to server: 1040 Too many connections

If not

mysqlslap: Error when connecting to server: 1135 Can't create a new thread (errno -1); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

Churn your buffer pool

This is what I've come up with so far... it works, it generated the mysql error we saw (buffer pool + connection memory > allocatable memory => out of memory). But only once.

Find your biggest table, or tables

SELECT CONCAT(table_schema, '.', table_name),
       CONCAT(ROUND(table_rows / 1000000, 2), 'M')                                    rows,
       CONCAT(ROUND(data_length / ( 1024 * 1024 * 1024 ), 2), 'G')                    DATA,
       CONCAT(ROUND(index_length / ( 1024 * 1024 * 1024 ), 2), 'G')                   idx,
       CONCAT(ROUND(( data_length + index_length ) / ( 1024 * 1024 * 1024 ), 2), 'G') total_size,
       ROUND(index_length / data_length, 2)                                           idxfrac
FROM   information_schema.TABLES
ORDER  BY data_length + index_length DESC
LIMIT  10;

Dump them to file

mysqldump -uroot -p somedb bigtable othertable > bigtable.sql

Make and run this batch file

 loadbigtables.bat 4

loadbigtables.bat:

@ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION

SET BACKGROUNDJOBS=%1
IF [%1]==[] SET BACKGROUNDJOBS=0

ECHO.
ECHO %0 with %BACKGROUNDJOBS% parallel background jobs
ECHO.

FOR /L %%i IN (0, 1, %BACKGROUNDJOBS%) DO (
    ECHO %DATE% %TIME% Create database stress%%i
    IF NOT EXIST stress%%i_bigtable.sql COPY bigtable.sql stress%%i_bigtable.sql
    mysql -uroot -ppassword -e "DROP DATABASE IF EXISTS stress%%i; CREATE DATABASE stress%%i"
)
ECHO %DATE% %TIME% Create database stress0

CMD /C "timeout /T 3 /NOBREAK" > nul

ECHO %DATE% %TIME% Start...
FOR /L %%i IN (1, 1, %BACKGROUNDJOBS%) DO (
    START /MIN CMD /C "mysql -uroot -ppassword stress%%i -e ^"\. stress%%i_bigtable.sql^" "
)
mysql -uroot -ppassword stress0 -e "\. stress0_bigtable.sql" & DEL stress0_bigtable.sql

CMD /C "timeout /T 10 /NOBREAK" > nul

FOR /L %%i IN (1, 1, %BACKGROUNDJOBS%) DO (
    ECHO %DATE% %TIME% Drop database stress%%i
    START /MIN CMD /C "mysql -uroot -ppassword -e ^"DROP DATABASE IF EXISTS stress%%i;^" "
)
mysql -uroot -ppassword -e "DROP DATABASE stress0;"

ECHO %DATE% %TIME% Done.

It isn't perfect but you can use it along side mysqlslap which consumes connections

mysqlslap -uroot -ppassword --auto-generate-sql --concurrency=500 

Improve as you see fit

Answer 4

Since you are on Windows, EatMem is pretty good for draining all of the RAM (temporarily):

See the MSDN Magazine article Test Run: Stress Testing by James McCaffrey.