MySQL how to release memory used up by SELECT

Question

I executed an INSERT IGNORE combined with SELECT from large table. The SELECT took up 16GB of memory and went into SWAP before I terminated it with CTRL+C.

INSERT IGNORE INTO sid(sid) 
SELECT sid FROM data WHERE sid != "";

The data contains 64 million rows.

Data type of sid is VARCHAR(13). There were 28.2M records that matched the SELECT (I ran it afterwards). That results in around 350MB of data that needs to be SELECTed and INSERTed.

Is it possible to release memory allocated by MySQL? The query was terminated and there is no longer a need for that memory. Why is it still lingering?

Here are some config details:

max_allowed_packet=1024M

# Memory Table

max_heap_table_size=8192M

Everything else is default.

MySQL version 5.5.25, using MyISAM

Inserts into ndbcluster 7.2.7

NDB Config

SharedGlobalMemory=256M
DataMemory=2048M
IndexMemory=512M

My box runs 1/3 SQL nodes, management node. 2 other servers run data nodes.

Answer 1

You could run RESET QUERY CACHE to eliminate collected result sets in the query cache. You could also be more aggressive and drop the query cache and recreate it a different size. For example, to set the query_cache_size to 512M, do the following

SET GLOBAL query_cache_size = 0;
SELECT SLEEP(30);
SET GLOBAL query_cache_size = 1024 * 1024 * 512;

Unfortunately, you can't release the MyISAM Key Buffer and InnoDB Buffer Pool in the same manner. You would have to resize them in /etc/my.cnf (or my.ini for Windows) and restart mysql.

Reading your comments, I can confidently say that MyISAM does not cache data pages. The query cache is the only thing I can think of that grabs and holds result sets. If you are loading in bulk, there should be a lot of query cache invalidations going on.

I am not Cluster Certified, but I will take shot at saying this: Since you are using MySQL Cluster, you may want to try shutting down any mysqld processes or having the cluster disconnect data nodes. Also, be aware of the transmission speed that data hops around the Cluster.

Answer 2

Let's review...

• 16GB of RAM.
• max_heap_table_size = 8G -- that says you are letting each process create a MEMORY table that is half the size of RAM! Not safe. Likely to lead to swapping.
• Query Cache is irrelevant because the SELECT's resultset is bigger than you have allocated (16MB) for the QC. Don't make it bigger, it won't help.
• MyISAM's key_buffer_size can be changed dynamically. But it is irrelevant. Your SELECT will do either a table scan or an index scan (if data has an index starting with sid). The key_buffer is not used for a table scan. For an index scan, it need be big enough for only one block. Bigger will not help. Too big will crowd RAM.
• You have not said what allocations are made for the NDB table sid. This is likely to be in RAM. Furthermore, you have not said how many nodes that sid is spread across.
• You have not said whether MyISAM and NDB are running on the same box.