4

We are using mysql master-master (version 5.5.32) for an API manager called Layer7. The 2 hosts holding the instances are in the same ESX

From time to time we have mysql doing a clean shutdown for now reason

We check the cronjobs and they are none We checked the logins to the hosts or the instance during this time and there were none

other than manually shutdown, shutdown from a script or app or shutdown due to a kill what could cause clean shutdown in a master-master config

here is the log of the host where the instance stopped

Thank you

Regards

150605 18:59:32 [Note] /usr/sbin/mysqld: Normal shutdown

150605 18:59:32 [Note] Event Scheduler: Purging the queue. 0 events
150605 18:59:32 [Note] Error reading relay log event: slave SQL thread was killed
150605 18:59:32 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
150605 18:59:32 [Note] Slave I/O thread killed while reading event
150605 18:59:32 [Note] Slave I/O thread exiting, read up to log 'ssgbin-log.000351', position 91210654
150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360878  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360872  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360871  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360864  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360860  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360859  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360858  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360857  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360855  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360854  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360853  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360851  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360849  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360848  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360847  user: 'gateway'

150605 18:59:34 [Warning] /usr/sbin/mysqld: Forcing close of thread 21360846  user: 'gateway'

150605 18:59:36  InnoDB: Starting shutdown...
150605 18:59:36  InnoDB: Waiting for 76 pages to be flushed
150605 18:59:37  InnoDB: Shutdown completed; log sequence number 5077358548454
150605 18:59:37 [Note] /usr/sbin/mysqld: Shutdown complete

150605 18:59:37 mysqld_safe mysqld from pid file /var/lib/mysql/mtllnl7.mtl.cbc.ca.pid ended
150605 18:59:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150605 18:59:38 [Note] Plugin 'FEDERATED' is disabled.
150605 18:59:38 [Note] Plugin 'partition' is disabled.
150605 18:59:38 InnoDB: The InnoDB memory heap is disabled
150605 18:59:38 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150605 18:59:38 InnoDB: Compressed tables use zlib 1.2.3
150605 18:59:38 InnoDB: Using Linux native AIO
150605 18:59:38 InnoDB: Initializing buffer pool, size = 512.0M
150605 18:59:38 InnoDB: Completed initialization of buffer pool
150605 18:59:38 InnoDB: highest supported file format is Barracuda.
150605 18:59:38  InnoDB: Waiting for the background threads to start
150605 18:59:39 InnoDB: 5.5.32 started; log sequence number 5077358548454
150605 18:59:39 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150605 18:59:39 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
150605 18:59:39 [Note] Server socket created on IP: '0.0.0.0'.
150605 18:59:39 [Note] Slave SQL thread initialized, starting replication in log 'ssgbin-log.000351' at position 91210654, relay log '/var/lib/mysql/ssgrelay-bin.000635' position: 854358
150605 18:59:39 [Note] Slave I/O thread: connected to master 'repluser@mtllnl7.mtl.cbc.ca:3307',replication started in log 'ssgbin-log.000351' at position 91210654
150605 18:59:39 [Note] Event Scheduler: Loaded 0 events
150605 18:59:39 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.32-enterprise-commercial-advanced-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Enterprise Server - Advanced Edition (Commercial)
150605 19:05:09 [Note] Slave: received end packet from server, apparent master shutdown:
150605 19:05:09 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'ssgbin-log.000351' at position 94453851
150605 19:05:09 [ERROR] Slave I/O: error reconnecting to master 'repluser@mtllnl7.mtl.cbc.ca:3307' - retry-time: 10  retries: 86400, Error_code: 2003
150605 19:05:29 [Note] Slave: connected to master 'repluser@mtllnl7.mtl.cbc.ca:3307',replication resumed in log 'ssgbin-log.000351' at position 94453851
Improve this question
3
  • change the password of users with SHUTDOWN privilege and you'll find out. MySQL also exits normally when receives TERM signal, too.
    – akuzminsky
    Jun 17, 2015 at 18:31
  • changing the password was the first thing we did. only one user had the shutdown privilege. but still.
    – Philippe Courtois
    Jun 17, 2015 at 18:33
  • then it's a TERM thing
    – akuzminsky
    Jun 17, 2015 at 18:34

1 Answer 1

Reset to default
2

The SHUTDOWN privilege can only be used when running mysqladmin

Taking away SHUTDOWN grant will stop users from running mysqladmin shutdown.

Your Actual Problem

The error log you are presenting in the question is a restart of mysqld.

150605 18:59:32 [Note] /usr/sbin/mysqld: Normal shutdown
.
.
.
150605 18:59:36  InnoDB: Starting shutdown...
150605 18:59:36  InnoDB: Waiting for 76 pages to be flushed
150605 18:59:37  InnoDB: Shutdown completed; log sequence number 5077358548454
150605 18:59:37 [Note] /usr/sbin/mysqld: Shutdown complete

150605 18:59:37 mysqld_safe mysqld from pid file /var/lib/mysql/mtllnl7.mtl.cbc.ca.pid ended
150605 18:59:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

mysqladmin cannot start or restart mysqld.

Someone or something is running service mysql restart

First run the history

# history | grep restart

If nothing comes up in conjunction with the restarts you see, you need to locate a cronjob on the DB Server that is performing service mysql restart. In some cases, it might be a cronjob on a remote server that no one remembers.

Improve this answer
1
  • I did an history | grep restart and I see one. I know we did use it to restart the service. for the crontab and script none on the server have the service msql stop or start or restart. for the app server, this is something ill have to check now
    – Philippe Courtois
    Jun 18, 2015 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.