4

We have a AWS MySQL RDS instance which is about 1.7T in size. Sometimes it becomes unresponsive and no operations can be performed.

  1. CPU utilization, Write IOPS, read IOPS, queue depth, write throughput, write latency and read latency drops to zero.
  2. Number of connections get piled up.
  3. "Show engine innodb status" hangs
  4. Lot of queries (around 25 for each) by rdsadmin which are in hang state.

    SELECT count(*) from mysql.rds_replication_status WHERE action = 'reset slave' and master_host is NULL and master_port is NULL GROUP BY action_timestamp,called_by_user,action,mysql_version,master_host,master_port ORDER BY action_timestamp LIMIT 1;
    
    SELECT NAME, VALUE FROM mysql.rds_configuration; 
    
  5. After sometime, instance gets rebooted automatically with following error.

    MySQL restart initiated to address MySQL induced log backup issues. Note that as part of this resulution, a DB Snapshot will be performed after MySQL completes restarting.

enter image description here

4
  • 1
    I'd be tempted to open a ticket with Amazon
    – Philᵀᴹ
    Apr 18, 2017 at 9:00
  • Unfortunately we haven't enabled aws support yet. Apr 18, 2017 at 9:37
  • See if you can do SHOW PROCESSLIST; when it becomes unresponsive.
    – Rick James
    Apr 18, 2017 at 16:01
  • Yes, I was able to. It shows lot of queries which are in "Opening tables" state plus, many system queries as mentioned at 4th bullet point. Apr 19, 2017 at 5:02

1 Answer 1

1

We were able to resolve this issue by upgrading the instances to 5.6.34.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.