As a result of increasing PHP_Memory Limit our wordpress data base has apparently crashed and been corrupted. Our host, Rackspace, attempted crash recovery but all attempts failed.
What commands or steps are needed to get the database restored to a working state? There are no recent backups of the database, so the solution will need to be a fix of the corruption.
This is the error:
160201 9:59:18 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. ... InnoDB: Serious error! InnoDB is trying to free page 7288 InnoDB: though it is already marked as free in the tablespace! InnoDB: The tablespace free space info is corrupt. InnoDB: You may need to dump your InnoDB tables and recreate the whole InnoDB: database! InnoDB: Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery.
This is what happened before the error, here told by Rackspace in a support ticket:
1) MySQL was not running, so I started it. I do see some indication in /var/log/mariadb/mariadb.log that the server has had memory constraints; the last time it was started (January 27), it failed to start with this message:
160127 15:12:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
I restarted Apache to clear plenty of free memory today, then MySQL started. So we probably need to look at the Apache MaxRequestWorkers setting. It is presently 40, but you don't have enough memory to set this so high. Let me go on to problem 2...
2) Your PHP memory_limit was 256 MB, which is quite high for a WordPress site, but the site was still refusing to load with the memory error you reported. I raised the system-wide limit to 320 MB and then the site loaded. This is unusually high for a WordPress site, so you should evaluate why your site is using so much memory (it is likely one of your plugins), and try to reduce your usage.
Let's say your site can function with 256 MB of memory for each request to WordPress. We need some memory for the OS and for MySQL, let's conservatively allocate 25% of memory to that. You have a 4 GB server. We're allocating 1 GB for the OS and MySQL, so that means you can only handle 12 simultaneous Apache connections (12 x 256 MB = 3 GB). Setting MaxRequestWorkers to 40 is going to lead to a situation where your server runs out of memory when you get a lot of traffic. In that situation, the Linux out-of-memory killer will kick in and quite possibly kill off MySQL.
To make your server more stable, you are going to have to do one or more of these items:
- Lower MaxRequestWorkers.
- Make WordPress more efficient in terms of memory usage.
- Move MySQL off-server (either on another server or use a cloud DB) so it won't contend with Apache for memory.
- Upgrade your server to increase the memory.
To try and make your server more stable, I have lowered MaxRequestWorkers to 20 for now, and also set MaxConnectionsPerChild to 200 so that Apache child processes won't live too long (they tend to consume more memory the longer they live).
Let me know if you need any more information.