Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I'm hosting a few sites on linode and have encountered a strange problem. For the last two years the site has been running perfectly fine, and now randomly all the websites on the server go to the "site offline check the settings.php" page. No changes have been made to the website at all recently.

when i try go do mysqld stop it says busy for a while then finally says failed, while doing start results in an instant fail.

Many of the threads I googled suggest the hostname is off in the settings.php but since the website has been up for two years I don't think this can be the case.

I haven't had to troubleshoot mySQL before, but if i remember correctly the log file I should be looking at is Hostname.err. In there, it has a large chain of errors concerning InnoDB. The one listed as fatal error says:

InnoDB: Fatal error: cannot allocate memory for the buffer pool

I would appreciate any suggestions, and if there are log files that would help let me know.

Edit: Requested information. 1) When I examine the CNF, it appears that all lines involving innoDB are commented with #. This means that it has always been this way as I have not modified it.

2)mysql Ver 14.14 Distrib 5.5.14, for Linux (i686) using readline 5.1

3) It looks like there is a giant jump in the IO rate every time I try to start the database / before the websites die. How do i go about clearing those cache tables if they have become too unmanageable and is there any risk of losing anything?

Third Edit (Now with datadir):

How much RAM is on the VM ? 512MB

How much space does datadir have ?

Size: 20G Used: 19G Available 0 Use Percent: 100%

What is the size of ibdata1 ( ls -lh /var/lib/mysql/ibdata1)? 114M

FYI: If anyone looks here with the same problem in the future, My issue was the the binary log files had consumed my entire disk space. See here http://systembash.com/content/mysql-binary-log-file-size-huge/

share|improve this question
    
Since its running from 2 years, probably all cache tables have grown in size and there might be a heavy load on mysql. Check your Disk read/writes for mysql. When it reached 100% usually mysql will go down after few minutes –  2-Stroker Apr 5 '13 at 15:35
2  
Please post the following: 1) All the InnoDB settings in my.cnf, 2) What version of MySQL are you running (Run mysql --version in Linux), 3) All hardware limits in the VM. –  RolandoMySQLDBA Apr 5 '13 at 15:38
    
How much RAM is on the VM ? How much space does datadir have ? What is the size of ibdata1 ( ls -lh /var/lib/mysql/ibdata1)? –  RolandoMySQLDBA Apr 5 '13 at 17:03
1  
Please run df -h /var/lib/mysql –  RolandoMySQLDBA Apr 5 '13 at 17:44
add comment

migrated from drupal.stackexchange.com Apr 5 '13 at 15:23

This question came from our site for Drupal developers and administrators.

1 Answer

up vote 1 down vote accepted

OMG !!! Look at the output of df -h

Size: 20G Used: 19G Available 0 Use Percent: 100%

YOU HAVE NO DISK SPACE !!!

Get more diskspace freed up or have linode increase diskspace.

According to MySQL 5.0 Certification Study Guide

enter image description here Page 408,409 Section 29.2 Bulletpoint 11 says:

If you run out of disk space while adding rows to a MyISAM table, no error occurs. The server suspends the operation until space becomes available, and then completes the operation.

In reality, mysqld is not dead. It is waiting for free diskspace to continue whatever it normally does. Even during startup, if mysqld makes a temp table, it is a MyISAM table. No space left on the disk? No problem. MyISAM just patiently waits for you to make free space.

If you have an error file like /var/log/mysql/log, clear its contents like this:

echo -n > /var/log/mysqld.log

If the error file in /var/lib/mysql/my-machine.err then do this:

echo -n > /var/lib/mysql/my-machine.err

Give it a Try !!!

share|improve this answer
    
Thank you, I've got the websites back online by allocating a few extra MB to the disk image that I had in case of a problem like this. Now that its back up I'll see if I can delete a few things to free up disk space. –  Joseph Adamski Apr 5 '13 at 23:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.