MySQL InnoDB crash post-mortem

Question

MySQL crashed on me this morning.

With the exception of the standard MySQL included databases everything I use is InnoDB.

I attempted to restart the MySQL daemon but it failed twice.

I then restarted the entire server and MySQL started correctly and has been working well since.

The mysqld log file for the initial crash contains the following:

120927 10:21:05 mysqld_safe Number of processes running now: 0
120927 10:21:06 mysqld_safe mysqld restarted
120927 10:21:12 [Note] Plugin 'FEDERATED' is disabled.
120927 10:21:12 InnoDB: The InnoDB memory heap is disabled
120927 10:21:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120927 10:21:12 InnoDB: Compressed tables use zlib 1.2.3
120927 10:21:12 InnoDB: Using Linux native AIO
120927 10:21:13 InnoDB: Initializing buffer pool, size = 4.0G
InnoDB: mmap(4395630592 bytes) failed; errno 12
120927 10:21:13 InnoDB: Completed initialization of buffer pool
120927 10:21:13 InnoDB: Fatal error: cannot allocate memory for the buffer pool
120927 10:21:13 [ERROR] Plugin 'InnoDB' init function returned error.
120927 10:21:13 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
120927 10:21:13 [ERROR] Unknown/unsupported storage engine: InnoDB
120927 10:21:13 [ERROR] Aborting

120927 10:21:13 [Note] /usr/libexec/mysqld: Shutdown complete

120927 10:21:13 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

When trying to restart the daemon the mysqld log file contains:

120927 10:43:44 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120927 10:43:44 [Note] Plugin 'FEDERATED' is disabled.
120927 10:43:44 InnoDB: The InnoDB memory heap is disabled
120927 10:43:44 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120927 10:43:44 InnoDB: Compressed tables use zlib 1.2.3
120927 10:43:44 InnoDB: Using Linux native AIO
120927 10:43:44 InnoDB: Initializing buffer pool, size = 4.0G
InnoDB: mmap(4395630592 bytes) failed; errno 12
120927 10:43:44 InnoDB: Completed initialization of buffer pool
120927 10:43:44 InnoDB: Fatal error: cannot allocate memory for the buffer pool
120927 10:43:44 [ERROR] Plugin 'InnoDB' init function returned error.
120927 10:43:44 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
120927 10:43:44 [ERROR] Unknown/unsupported storage engine: InnoDB
120927 10:43:44 [ERROR] Aborting

120927 10:43:44 [Note] /usr/libexec/mysqld: Shutdown complete

120927 10:43:44 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

After the server restart the mysqld log file contains:

120927 10:46:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120927 10:46:11 [Note] Plugin 'FEDERATED' is disabled.
120927 10:46:11 InnoDB: The InnoDB memory heap is disabled
120927 10:46:11 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120927 10:46:11 InnoDB: Compressed tables use zlib 1.2.3
120927 10:46:11 InnoDB: Using Linux native AIO
120927 10:46:11 InnoDB: Initializing buffer pool, size = 4.0G
120927 10:46:11 InnoDB: Completed initialization of buffer pool
120927 10:46:12 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
120927 10:46:12  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
120927 10:46:15  InnoDB: Waiting for the background threads to start
120927 10:46:16 InnoDB: 1.1.8 started; log sequence number 57665645675
120927 10:46:16 [Note] Event Scheduler: Loaded 0 events
120927 10:46:16 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.21-cll'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Atomicorp

I have never had to try to decypher a crash MySQL log file.

I am using version: 5.5.21-cll MySQL Community Server (GPL) by Atomicorp

Any ideas on where I should start?

UPDATE: From @Michael-sqlbot's recommendation, I pulled the syslog and found this:

Sep 27 10:20:58 ip-97-74-197-181 kernel: pcscd invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
Sep 27 10:21:00 ip-97-74-197-181 kernel:
Sep 27 10:21:00 ip-97-74-197-181 kernel: Call Trace:
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff800c9f35>] out_of_memory+0x8e/0x2f3
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff8002dfc7>] __wake_up+0x38/0x4f
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff8000f67d>] __alloc_pages+0x27f/0x308
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff80017a84>] cache_grow+0x139/0x3c7
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff8005be28>] cache_alloc_refill+0x138/0x188
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff8000ad2e>] kmem_cache_alloc+0x6c/0x76
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff80012877>] getname+0x25/0x1c2
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff8001a04b>] do_sys_open+0x17/0xbe
Sep 27 10:21:00 ip-97-74-197-181 kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Sep 27 10:21:00 ip-97-74-197-181 kernel:
Sep 27 10:21:11 ip-97-74-197-181 kernel: Mem-info:
Sep 27 10:21:20 ip-97-74-197-181 kernel: Node 0 DMA per-cpu:
Sep 27 10:21:27 ip-97-74-197-181 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 27 10:21:38 ip-97-74-197-181 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 27 10:21:49 ip-97-74-197-181 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 27 10:21:49 ip-97-74-197-181 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 27 10:21:49 ip-97-74-197-181 kernel: cpu 2 hot: high 0, batch 1 used:0
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 2 cold: high 0, batch 1 used:0
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 3 hot: high 0, batch 1 used:0
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 3 cold: high 0, batch 1 used:0
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 DMA32 per-cpu:
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 0 hot: high 186, batch 31 used:60
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 0 cold: high 62, batch 15 used:57
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 1 hot: high 186, batch 31 used:139
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 1 cold: high 62, batch 15 used:61
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 2 hot: high 186, batch 31 used:47
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 2 cold: high 62, batch 15 used:57
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 3 hot: high 186, batch 31 used:52
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 3 cold: high 62, batch 15 used:53
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 Normal per-cpu:
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 0 hot: high 186, batch 31 used:29
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 0 cold: high 62, batch 15 used:17
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 1 hot: high 186, batch 31 used:178
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 1 cold: high 62, batch 15 used:52
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 2 hot: high 186, batch 31 used:22
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 2 cold: high 62, batch 15 used:59
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 3 hot: high 186, batch 31 used:71
Sep 27 10:21:52 ip-97-74-197-181 kernel: cpu 3 cold: high 62, batch 15 used:54
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 HighMem per-cpu: empty
Sep 27 10:21:52 ip-97-74-197-181 kernel: Free pages:       41728kB (0kB HighMem)
Sep 27 10:21:52 ip-97-74-197-181 kernel: Active:1031140 inactive:970428 dirty:0 writeback:0 unstable:0 free:10432 slab:4277 mapped-file:801 mapped-anon:1993003 pagetables:11636
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 DMA free:10096kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9700kB pages_scanned:0 all_unreclaimable? yes
Sep 27 10:21:52 ip-97-74-197-181 kernel: lowmem_reserve[]: 0 2965 8015 8015
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 DMA32 free:24424kB min:4236kB low:5292kB high:6352kB active:1544164kB inactive:1428756kB present:3037024kB pages_scanned:7185900 all_unreclaimable? yes
Sep 27 10:21:52 ip-97-74-197-181 kernel: lowmem_reserve[]: 0 0 5050 5050
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 Normal free:7208kB min:7212kB low:9012kB high:10816kB active:2580172kB inactive:2453052kB present:5171200kB pages_scanned:12935183 all_unreclaimable? yes
Sep 27 10:21:52 ip-97-74-197-181 kernel: lowmem_reserve[]: 0 0 0 0
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep 27 10:21:52 ip-97-74-197-181 kernel: lowmem_reserve[]: 0 0 0 0
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 DMA: 6*4kB 3*8kB 4*16kB 4*32kB 4*64kB 5*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 2*4096kB = 10096kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 DMA32: 24*4kB 3*8kB 1*16kB 1*32kB 1*64kB 3*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 24424kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 Normal: 0*4kB 13*8kB 8*16kB 0*32kB 19*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 7208kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: Node 0 HighMem: empty
Sep 27 10:21:52 ip-97-74-197-181 kernel: 9391 pagecache pages
Sep 27 10:21:52 ip-97-74-197-181 kernel: Swap cache: add 5745145, delete 5744809, find 81873079/82270945, race 0+63
Sep 27 10:21:52 ip-97-74-197-181 kernel: Free swap  = 0kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: Total swap = 2096472kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: Free swap:            0kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: 2359296 pages of RAM
Sep 27 10:21:52 ip-97-74-197-181 kernel: 324458 reserved pages
Sep 27 10:21:52 ip-97-74-197-181 kernel: 21388 pages shared
Sep 27 10:21:52 ip-97-74-197-181 kernel: 336 pages swap cached
Sep 27 10:21:52 ip-97-74-197-181 kernel: Out of memory: Killed process 3044, UID 27, (mysqld).

Answer 1

I have good news, and bad news. The good news is, your filesystem and mysql are most likely fine... but check /var/log/syslog or equivalent to see what else was happening on your system before 10:21:05.

When the first message you posted was logged, your mysql server had already died.

120927 10:21:05 mysqld_safe Number of processes running now: 0

So, assuming you didn't overlook anything in the mysql error log, I'm going to say it didn't crash and die -- it was actually killed.

When mysqld_safe (which is a wrapper, not the server itself) realized the server wasn't running, and that the server hadn't terminated gracefully, it restarted it for you...

120927 10:21:06 mysqld_safe mysqld restarted

...then the server daemon logged some normal startup messages...

120927 10:21:12 [Note] Plugin 'FEDERATED' is disabled.
120927 10:21:12 InnoDB: The InnoDB memory heap is disabled
120927 10:21:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120927 10:21:12 InnoDB: Compressed tables use zlib 1.2.3
120927 10:21:12 InnoDB: Using Linux native AIO

...but when mysqld asked the operating system to allocate 4GB of memory for the InnoDB buffer pool...

120927 10:21:13 InnoDB: Initializing buffer pool, size = 4.0G

...the kernel said "no."

InnoDB: mmap(4395630592 bytes) failed; errno 12

Checking the kernel source to be sure:

#define ENOMEM      12  /* Out of memory */

Yeah. So, every message below the "failed; errno 12" line should be disregarded -- they're all side-effects of this one.

But again, all of these things happened after the first crash.

My best guess is that an extreme low memory condition caused your kernel to originally kill mysqld in an attempt to stablize the system.

Naturally, whatever caused the memory shortage was gone after the reboot. The mysql server was able to allocate 4GB for the InnoDB buffer pool, and all should be good until whatever caused you to run out of memory causes it again.

First guess: apache child processes run amok.

Answer 2

This happened to me recently and this thread has been invaluable for helping me understand what's happening.

I'm running a LAMP stack on Ubuntu 14.04 with 1GB Ram. My server kept crashing due to web traffic spikes. For me, fiddling with the mysql config file only extended the amount of time I'd experience another random crash. To test and fix this, I ultimately used Apache's ab tool:

ab -n 100 -c 10 http://gastonia.com/

10 concurrencies (-c) was ok, so incremented that until I hit 30 - bam - crash. I backed off until I found a number that was safe, and then I adjusted Apache's ServerLimit directive:

ServerLimit 20

After that I could change -c to any number I wanted to, and I have yet to experience another crash.

Hopeully that is helpful to anyone experience the same problem.

Answer 3

I have had the same problem as originally posted plus recurring InnoDB corruptions at random intervals after mysterious Mysql_safe restarts .

I was usually able to restart Mysql by first stopping Apache.

After reading this post i looked in my syslog logs and i found : kernel: Out of memory: Kill process 19468 (mysqld) score 256 or sacrifice child With same timestamp as the mysterious mysql crashes.

I also matched it to bursts of traffic and stacks of httpd(apache) processes. I reduced the innoDB pool size, increased swap size a bit and capped max number of apache processes just in case.

Answer 4

when you've got

Sep 27 10:21:52 ip-97-74-197-181 kernel: Swap cache: add 5745145, delete 5744809, find 81873079/82270945, race 0+63
Sep 27 10:21:52 ip-97-74-197-181 kernel: Free swap  = 0kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: Total swap = 2096472kB
Sep 27 10:21:52 ip-97-74-197-181 kernel: Free swap:            0kB

try to detect what processes eating your memory and swap:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less

then kill em all:

ps -ef | grep eatingprocess |  grep -v grep  |  awk '{print $2}' | xargs kill -9