3

We have a AWS MySQL RDS instance which is about 1.7T in size. Sometimes it becomes unresponsive and no operations can be performed.

  1. CPU utilization, Write IOPS, read IOPS, queue depth, write throughput, write latency and read latency drops to zero.
  2. Number of connections get piled up.
  3. "Show engine innodb status" hangs
  4. Lot of queries (around 25 for each) by rdsadmin which are in hang state.

    SELECT count(*) from mysql.rds_replication_status WHERE action = 'reset slave' and master_host is NULL and master_port is NULL GROUP BY action_timestamp,called_by_user,action,mysql_version,master_host,master_port ORDER BY action_timestamp LIMIT 1;
    
    SELECT NAME, VALUE FROM mysql.rds_configuration; 
    
  5. After sometime, instance gets rebooted automatically with following error.

    MySQL restart initiated to address MySQL induced log backup issues. Note that as part of this resulution, a DB Snapshot will be performed after MySQL completes restarting.

enter image description here

  • 1
    I'd be tempted to open a ticket with Amazon – Philᵀᴹ Apr 18 '17 at 9:00
  • Unfortunately we haven't enabled aws support yet. – Sajeeva Lakmal Apr 18 '17 at 9:37
  • See if you can do SHOW PROCESSLIST; when it becomes unresponsive. – Rick James Apr 18 '17 at 16:01
  • Yes, I was able to. It shows lot of queries which are in "Opening tables" state plus, many system queries as mentioned at 4th bullet point. – Sajeeva Lakmal Apr 19 '17 at 5:02
0

We were able to resolve this issue by upgrading the instances to 5.6.34.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.