0

I recently moved my entire websites from a dedicated CentOS5 server to a CentOS 6.5 on Amazon EC2 with the same cPanel/WHM version. WHM 11.44.1 (build 19) However after angry calls from clients I find mysql service to have stopped and it easily turns back on with a /etc/init.d/mysql start. I noticed the problem after using the transfer tool to move files, it got stuck so I manually killed the process and updated the mysql records to tell WHM the transfer is over. I have created the following in my /etc/my.cnf

[mysqld]
default-storage-engine=MyISAM
innodb_file_per_table=1
sync_binlog=0
local-infile=0
max_allowed_packet=268435456
open_files_limit=6644
innodb_buffer_pool_size=134217728
log-error=/tmp/mysql.errors

the content of /tmp/mysql.errors is:

141031 18:40:43 mysqld_safe Number of processes running now: 0
141031 18:40:43 mysqld_safe mysqld restarted
2014-10-31 18:40:44 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --ex$
2014-10-31 18:40:44 27621 [Note] Plugin 'FEDERATED' is disabled.
2014-10-31 18:40:44 27621 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-10-31 18:40:44 27621 [Note] InnoDB: The InnoDB memory heap is disabled
2014-10-31 18:40:44 27621 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-10-31 18:40:44 27621 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-10-31 18:40:44 27621 [Note] InnoDB: Using Linux native AIO
2014-10-31 18:40:44 27621 [Note] InnoDB: Using CPU crc32 instructions
2014-10-31 18:40:44 27621 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-10-31 18:40:44 27621 [Note] InnoDB: Completed initialization of buffer pool
2014-10-31 18:40:44 27621 [Note] InnoDB: Highest supported file format is Barracuda.
2014-10-31 18:40:44 27621 [Note] InnoDB: The log sequence numbers 17149979 and 17149979 in ibdata fi$
2014-10-31 18:40:44 27621 [Note] InnoDB: Database was not shutdown normally!
2014-10-31 18:40:44 27621 [Note] InnoDB: Starting crash recovery.
2014-10-31 18:40:44 27621 [Note] InnoDB: Reading tablespace information from the .ibd files...
2014-10-31 18:40:44 27621 [Note] InnoDB: Restoring possible half-written data pages
2014-10-31 18:40:44 27621 [Note] InnoDB: from the doublewrite buffer...
2014-10-31 18:40:44 27621 [Note] InnoDB: 128 rollback segment(s) are active.
2014-10-31 18:40:44 27621 [Note] InnoDB: Waiting for purge to start
2014-10-31 18:40:44 27621 [Note] InnoDB: 5.6.17 started; log sequence number 17283325
2014-10-31 18:40:44 27621 [Note] Server hostname (bind-address): '*'; port: 3306
2014-10-31 18:40:44 27621 [Note] IPv6 is available.
2014-10-31 18:40:44 27621 [Note]   - '::' resolves to '::';
2014-10-31 18:40:44 27621 [Note] Server socket created on IP: '::'.
2014-10-31 18:40:44 27621 [Note] Event Scheduler: Loaded 0 events
2014-10-31 18:40:44 27621 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.17'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
141031 18:41:54 mysqld_safe A mysqld process already exists

Output of cat /var/log/messages | grep sql Oct 31 18:27:54 vs110 PAM-hulk[24729]: error logging into hulkd: 400 Unable to connect to database backend: Failed to connect to mysql db: cphulkd: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) at /usr/local/cpanel/Cpanel/DBI.pm line 43.#012

Oct 31 18:40:43 vs110 kernel: [25680]     0 25680     2826       87   0       0             0 mysqld_safe
Oct 31 18:40:43 vs110 kernel: [25870]   498 25870   335852   115975   0       0             0 mysqld
Oct 31 18:40:43 vs110 kernel: Out of memory: Kill process 25870 (mysqld) score 456 or sacrifice child
Oct 31 18:40:43 vs110 kernel: Killed process 25870, UID 498, (mysqld) total-vm:1343408kB, anon-rss:463840kB, file-rss:60kB

Oct 31 18:48:19 vs110 kernel: Killed process 27621, UID 498, (mysqld) total-vm:1341340kB, anon-rss:465824kB, file-rss:168kB

Oct 31 18:49:12 vs110 kernel: Out of memory: Kill process 29486 (mysqld) score 448 or sacrifice child
Oct 31 18:49:12 vs110 kernel: Killed process 29486, UID 498, (mysqld) total-vm:1341348kB, anon-rss:455948kB, file-rss:8kB
Improve this question
5
  • 1
    The problem occurred before the first line of error log that you posted. That line is a symptom. The rest is recovery. cat /var/log/messages | grep kernel. The usual suspect is a web server running on the same machine, and the kernel is killing mysql when the web server starts making excessive demands for more and more memory.
    – Michael - sqlbot
    Oct 31, 2014 at 21:47
  • So how do I let mysql have the memory it needs?
    – Neo
    Oct 31, 2014 at 22:43
  • You are right it seems that wat from this: kernel: Out of memory: Kill process 25870 (mysqld) score 456 or sacrifice child. Is this a configuration issue or did I pick the wrong EC2 instance?
    – Neo
    Oct 31, 2014 at 22:47
  • 2
    You'll need to monitor the server. It's likely not mysql that is the memory hog, it just looks, to the kernel, like a good candidate for killing, when memory pressure is high on the system as a whole. A certain popular major name web server that forks off multiple processes serving one concurrent request each is known for causing this condition under load. Before upgrading the instance class, consider that a more stable alternative is keeping database and web server on separate instances. There are other questions and answers here that should give some helpful hints on this topic.
    – Michael - sqlbot
    Oct 31, 2014 at 23:42
  • Thank you so much at least now I know where the problem is, this whole time I thought there was something wrong with MySQL.
    – Neo
    Oct 31, 2014 at 23:58

0

Reset to default

Browse other questions tagged or ask your own question.