Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I'm investigating windows 32bit memory problems (the 2G limit) and am after a test which will max out MySQL's memory including InnoDB Buffer Pool, per connection memory and/or any other uses of memory.

Perhaps a query I could use for mysqlslap?

Update

I thought other people would benefit from a simple way to load test mysql and this was an easy question where lots of people would have "tools". They would help my obscure situation and I could use to investigate our real problem.

We have our configuration tuned, and it was fine for months but recently we eventually get mysqld: Out of memory (Needed 220428 bytes) and MySQL will crash minutes later.

I'm not researching your usual "whoops I allocated too much to my buffer pool" problem. That memory is allocated at start up and improper setting causes immediate shutdown. I'm really after the other 20%. Just creating the maximum number of connections with a high buffer pool hasn't proven to reproduce the problem so I want to stress everything. I am not ruling out stressing the buffer pool in case we are actually suffering a MySQL bug relating to it.

Here is an information dump for people interested:

I'm no authority on any of the above I just want to be able to produce high memory activity so we can A) reproduce the problem and then B) solve it in the medium term before moving off 32bit, which is the real solution but it won't happen tomorrow.

Update 2

Turns out Mysql 5.5's mysqld.exe is already built to be /LARGEADDRESSAWARE but cannot set the innodb engine size equal to or greater than 1600M. Connections do use the extra memory from /3GB (proven with mysqlslap).

vcvars32.bat
dumpbin.exe /headers mysqld.exe
Microsoft (R) COFF/PE Dumper Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file mysqld.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        4E6A3CEF time date stamp Sat Sep 10 04:21:03 2011
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             122 characteristics
                   Executable
                   Application can handle large (>2GB) addresses
                   32 bit word machine

My conclusion. If you don't use /3GB or otherwise set connection memory + innodb buffer memory greater than available memory you are bound to eventually get mysqld: Out of memory... although mysql seems to do it's best to reject connections to avoid this problem.

Take home message, test that you can actually achieve max_connections with mysqlslap and the rest comes down to maintaining OS memory (and other resources like free system PTE).

share|improve this question
add comment

4 Answers

up vote 1 down vote accepted

On Windows 32-bit platforms, it is not possible by default to use more than 2GB of RAM within a single process, including MySQL. -- http://dev.mysql.com/doc/refman/5.5/en/limits-windows.html

What's to investigate? We already know what happens when you max the memory: Really Bad Things™.

If there's a remote chance that your setup will periodically require this much memory, 32-bit Windows is the wrong platform.

Almost universally, the biggest memory consumer is of course the InnoDB Buffer Pool, but there's no need for any queries to test this... you don't need any activity at all, or any tables, because the entire amount of memory declared for innodb_buffer_pool_size is allocated immediately when the server starts. The buffer pool never grows, never shrinks, never changes, ever. The number of free pages changes, but those are not "free" from the operating system's perspective -- they're still just as allocated, merely marked as containing nothing of interest within InnoDB.

If the operating system refuses to allocate the amount of memory provisioned for the buffer pool, MySQL will simply refuse to start.

This is illustrated here, where the OP mistakenly believed that the server was crashing and restarting "because" memory couldn't be allocated for the buffer pool, but was in fact crashing for a Linux-specific reason but then refusing to restart because the system would not allocate the total amount of memory required for the pool, due to overuse of available memory by something else... but this allocate-all-at-startup behavior for the InnoDB buffer pool is not platform-specific.

So, you should be able to set this value near the max and then find that taking the server process over the edge should not require very much additional effort at all. But I'm still not sure what the point is.

As you realize, MySQL uses memory for a variety of different purposes, several of which are dynamically-sized, definable on a per-connection basis, and allocated on demand, which makes it virtually impossible to provision a server based on limiting memory usage to some worst-case scenario absolute value, yet expecting that server to be able to handle its typical load efficiently.

The simplest illustration of this is the fact that you can obviously reduce the theoretical maximum memory utilization of a given instance by restricting the maximum number of simultaneous client connections... but any given application needs a certain number of available connections to perform efficiently, and if that number is below your target value, then you're not really solving anything -- it just feels like you have.

I say, either your server has enough memory for the workload, or it doesn't. If it doesn't, then attempting to "tune" your way out of potential trouble is unlikely to offer much in the way of solutions.


Update:

Some ideas on how to easily generate demand for more memory...

SELECT * FROM large_table ORDER BY non_indexed_column;

SELECT * FROM large_table WHERE non_indexed_column = some_value;

SELECT * FROM large_table WHERE some_column LIKE '%a_freqent_match%';

Queries like these could trigger the allocation of a sort buffer, a read buffer and/or a random read buffer, which should make a new request to the OS for the memory that buffer requires.


Update 2:

Simple solution:

DELIMITER $$

DROP PROCEDURE IF EXISTS `test`.`eat_memory_until_server_crashes` $$
CREATE PROCEDURE `test`.`eat_memory_until_server_crashes`()
BEGIN

-- this procedure is intended to eat as much memory as it can
-- it creates a series of consecutively-numbered session variables as large
-- as your configuration will allow them to be.

-- do not run this unless you intend to crash your server

-- also, do not run from a gui tool -- use the mysql command line client:

-- mysql> CALL test.eat_memory_until_server_crashes;

-- if you kill the query or thread before the server crashes, 
-- the memory consumed will be returned to the OS

  DECLARE counter INT DEFAULT 0;
  LOOP
    SET counter = counter + 1;
    SET @qry = CONCAT('SET @crash_me_',counter,' := REPEAT(\'a\', @@max_allowed_packet)');
    SELECT counter, @qry;
    PREPARE hack FROM @qry;
    EXECUTE hack;
    DEALLOCATE PREPARE hack;
    -- adjust timing or remove this entirely depending on how quickly you want this to happen
    DO SLEEP(0.1);
  END LOOP;

END $$

DELIMITER ;

Inspiration for this: Schwartz, Baron; Zaitsev, Peter; Tkachenko, Vadim (2012-03-05). High Performance MySQL: Optimization, Backups, and Replication (Kindle Location 12194). OReilly Media - A. Kindle Edition.

share|improve this answer
    
You have explained exactly the logic we are working within, I just didn't want to go into these details.The problem is we set the buffer pool to 75% of 2GB and calculated the remaining memory (mostly connection based as you mention) and thought we were fine. Ran for two months no problems. Now MySQL will "Out of memory" every few days and die. My testing cannot repeat this and I want to be able too, hence the question. Added to the mix we are using /3GB but have not edited the mysqld.exe binary to be /LARGEADDRESSAWARE –  KCD Mar 21 '13 at 4:38
    
Oh and we have 8GB ram and /PAE (yes it is foolish to run 32bit!), so this is a MySQL memory problem or a System Page Table Entry problem however we seem to be "ok" at 18k free. The real solution is a 64bit OS but we have just a few more months of this to go... anyway you may see why I decided it best just to ask for good test rather than a solution :) –  KCD Mar 21 '13 at 4:44
    
Yes, I can see why :) but having a sense of the bigger picture does clarify things quite a bit. –  Michael - sqlbot Mar 21 '13 at 5:01
    
Your answer is understandable especially given my rep on dba. How about the new update –  KCD Mar 21 '13 at 5:26
1  
I had a moment of inspiration and wrote a stored procedure that should do exactly what you wanted, quickly and effectively. It's a bit late to the game for this question but it accomplishes the purpose. –  Michael - sqlbot Mar 22 '13 at 1:42
show 2 more comments

There is way to make the InnoDB constantly flush.

First here are some internals on the InnoDB Buffer Pool:

InnoDB manages the pool as a list, using a variation of the least recently used (LRU) algorithm. When room is needed to add a new block to the pool, InnoDB evicts the least recently used block and adds the new block to the middle of the list. This “midpoint insertion strategy” treats the list as two sublists:

At the head, a sublist of “new” (or “young”) blocks that were accessed recently.

At the tail, a sublist of “old” blocks that were accessed less recently.

This algorithm keeps blocks that are heavily used by queries in the new sublist. The old sublist contains less-used blocks; these blocks are candidates for eviction.

The LRU algorithm operates as follows by default:

3/8 of the buffer pool is devoted to the old sublist.

The midpoint of the list is the boundary where the tail of the new sublist meets the head of the old sublist.

When InnoDB reads a block into the buffer pool, it initially inserts it at the midpoint (the head of the old sublist). A block can be read in because it is required for a user-specified operation such as an SQL query, or as part of a read-ahead operation performed automatically by InnoDB.

Accessing to a block in the old sublist makes it “young”, moving it to the head of the buffer pool (the head of the new sublist). If the block was read in because it was required, the first access occurs immediately and the block is made young. If the block was read in due to read-ahead, the first access does not occur immediately (and might not occur at all before the block is evicted).

As the database operates, blocks in the buffer pool that are not accessed “age” by moving toward the tail of the list. Blocks in both the new and old sublists age as other blocks are made new. Blocks in the old sublist also age as blocks are inserted at the midpoint. Eventually, a block that remains unused for long enough reaches the tail of the old sublist and is evicted.

By default, blocks read by queries immediately move into the new sublist, meaning they will stay in the buffer pool for a long time. A table scan (such as performed for a mysqldump operation, or a SELECT statement with no WHERE clause) can bring a large amount of data into the buffer pool and evict an equivalent amount of older data, even if the new data is never used again. Similarly, blocks that are loaded by the read-ahead background thread and then accessed only once move to the head of the new list. These situations can push frequently used blocks to the old sublist, where they become subject to eviction.

Given this information, the idea is to get the InnoDB Buffer Pool to flash as frequently as possible. Here are the options you need to set:

  • innodb_max_dirty_page_pct : The main thread in InnoDB tries to write pages from the buffer pool so that the percentage of dirty (not yet written) pages will not exceed this value. This contributes to dirty pages being flushed frequently when Write IO slows down.
  • innodb_old_blocks_pct : Specifies the approximate percentage of the InnoDB buffer pool used for the old block sublist. Setting this to 95 (the max value) makes InnoDB evict 95% of dirty pages out of the InnoDB Buffer Pool. Even neighbor pages get flushed.
  • innodb_old_blocks_time : Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before it can be moved to the new sublist. If the value is 0, a block inserted into the old sublist moves immediately to the new sublist the first time it is accessed, no matter how soon after insertion the access occurs. If the value is greater than 0, blocks remain in the old sublist until an access occurs at least that many ms after the first access. For example, a value of 1000 causes blocks to stay in the old sublist for 1 second after the first access before they become eligible to move to the new sublist.

For a Windows machine with 4G of RAM and a dual core processor, set these values in my.ini:

[mysqld]
innodb_buffer_pool_size=1536M # limited by 32 bit
innodb_buffer_pool_instances=2
innodb_max_dirty_pages_pct=0
innodb_old_blocks_pct=95
innodb_old_blocks_time=50

and then restart mysql.

What about the the stress test itself? All you may need is a very large mysqldump to import and let the InnoDB Buffer Pool go crazy. If you have a small mysqldump, create 100 database and load the mysqldump into each of them, sequentially or in parallel.

Would you like sample data?

cd
rm -rf SampleData
mkdir SampleData
cd SampleData
wget http://downloads.mysql.com/docs/menagerie-db.tar.gz
wget http://downloads.mysql.com/docs/sakila-db.tar.gz
wget http://downloads.mysql.com/docs/world.sql.gz
tar zxf menagerie-db.tar.gz
tar zxf sakila-db.tar.gz
gzip -d world.sql.gz
cd menagerie-db
mv *.sql ..
mv *.txt ..
cd ..
rmdir menagerie-db
mkdir sakila-db
cd sakila-db
mv sak* ..
cd ..
rmdir sakila-db
rm -f *.tar.gz

Here is what you can do:

  • Make 100 databases
  • Load the SQL data into each of those databases
  • mysqldump --all-datbasaes into a single text file (call it MySQL100Data.sql)
  • Reboot the Windows Machine
  • start mysql (if it does not autostart)
  • Load MySQL100Data.sql into mysql

Here is another test:

  • Reboot the Windows Machine
  • Make 100 databases
  • Load the SQL data into each of those databases in parallel

In either case, You can monitor RAM while MySQL is getting beat down...

Give it a Try !!!

CAVEAT

This will have the side effect of Disk IO due to flushing the InnoDB Buffer Pool.

share|improve this answer
    
The idea to dump and reload databases helped me reproduce one of the two cases I was testing so I have awarded you the bounty. However I wasn't really after configuration changes to make buffer pool churn –  KCD Mar 22 '13 at 1:36
    
Thank you for very bounty !!! –  RolandoMySQLDBA Mar 22 '13 at 1:41
add comment

Since you are on Windows, EatMem is pretty good for draining all of the RAM (temporarily) http://msdn.microsoft.com/en-us/magazine/cc163613.aspx

share|improve this answer
add comment

Test you haven't over allocated memory

The innodb buffer pool is preallocated at startup. But connection memory is not, trying achieving max_connections with mysqlslap (e.g 500)

mysqlslap -uroot -ppassword -h192.168.1.1 --auto-generate-sql --concurrency=500 --number-of-queries=2000 

If you are ok you will get

 mysqlslap: Error when connecting to server: 1040 Too many connections

If not

mysqlslap: Error when connecting to server: 1135 Can't create a new thread (errno -1); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

Churn your buffer pool

This is what I've come up with so far... it works, it generated the mysql error we saw (buffer pool + connection memory > allocatable memory => out of memory). But only once.

Find your biggest table, or tables

SELECT CONCAT(table_schema, '.', table_name),
       CONCAT(ROUND(table_rows / 1000000, 2), 'M')                                    rows,
       CONCAT(ROUND(data_length / ( 1024 * 1024 * 1024 ), 2), 'G')                    DATA,
       CONCAT(ROUND(index_length / ( 1024 * 1024 * 1024 ), 2), 'G')                   idx,
       CONCAT(ROUND(( data_length + index_length ) / ( 1024 * 1024 * 1024 ), 2), 'G') total_size,
       ROUND(index_length / data_length, 2)                                           idxfrac
FROM   information_schema.TABLES
ORDER  BY data_length + index_length DESC
LIMIT  10;

Dump them to file

mysqldump -uroot -p somedb bigtable othertable > bigtable.sql

Make and run this batch file

 loadbigtables.bat 4

loadbigtables.bat:

@ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION

SET BACKGROUNDJOBS=%1
IF [%1]==[] SET BACKGROUNDJOBS=0

ECHO.
ECHO %0 with %BACKGROUNDJOBS% parallel background jobs
ECHO.

FOR /L %%i IN (0, 1, %BACKGROUNDJOBS%) DO (
    ECHO %DATE% %TIME% Create database stress%%i
    IF NOT EXIST stress%%i_bigtable.sql COPY bigtable.sql stress%%i_bigtable.sql
    mysql -uroot -ppassword -e "DROP DATABASE IF EXISTS stress%%i; CREATE DATABASE stress%%i"
)
ECHO %DATE% %TIME% Create database stress0

CMD /C "timeout /T 3 /NOBREAK" > nul

ECHO %DATE% %TIME% Start...
FOR /L %%i IN (1, 1, %BACKGROUNDJOBS%) DO (
    START /MIN CMD /C "mysql -uroot -ppassword stress%%i -e ^"\. stress%%i_bigtable.sql^" "
)
mysql -uroot -ppassword stress0 -e "\. stress0_bigtable.sql" & DEL stress0_bigtable.sql

CMD /C "timeout /T 10 /NOBREAK" > nul

FOR /L %%i IN (1, 1, %BACKGROUNDJOBS%) DO (
    ECHO %DATE% %TIME% Drop database stress%%i
    START /MIN CMD /C "mysql -uroot -ppassword -e ^"DROP DATABASE IF EXISTS stress%%i;^" "
)
mysql -uroot -ppassword -e "DROP DATABASE stress0;"

ECHO %DATE% %TIME% Done.

It isn't perfect but you can use it along side mysqlslap which consumes connections

mysqlslap -uroot -ppassword --auto-generate-sql --concurrency=500 

Improve as you see fit

share|improve this answer
    
You have introduced another factor that invalidates the speculation that churning the buffer pool is relevant to what you are experiencing. The memory required to handle the large extended inserts mysqldump generates by default is more likely to be the explanation, and I would speculate that this same process with a .sql file generated with --skip-extended-insert would probably not lead to the same degree of reproducibility even though the net effect on the buffer pool would be the same. –  Michael - sqlbot Mar 22 '13 at 1:51
    
That sounds logical to me. I wonder what was so unique that mysql actually died rather than simply threw a few out of memory exceptions –  KCD Mar 22 '13 at 2:50
    
Did you get a stack trace from the crash? From looking at the source, there are a lot of conditions where an out of memory condition is handled gracefully, by throwing back an error. If that error condition, in turn, ultimately results in memory being freed as things leading up to the error were torn down, then the server might continue happily... so my assumption would be that you were eventually unable to allocate memory for something more critical than the buffers required by the test proc. –  Michael - sqlbot Mar 22 '13 at 3:09
    
Yeah that sounds right. There has never been a stack trace and it had happened a handful of times. No crashes since allowing /3GB per process such that our configuration doesn't exceed actual memory. In this setup our OS problem should be solved with correct use of /userva to increase free system PTEs - nothing to do with mysql –  KCD Mar 25 '13 at 2:11
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.