AWS MySQL RDS instance becomes unresponsive and getting restarted automatically

Question

We have a AWS MySQL RDS instance which is about 1.7T in size. Sometimes it becomes unresponsive and no operations can be performed.

1. CPU utilization, Write IOPS, read IOPS, queue depth, write throughput, write latency and read latency drops to zero.
2. Number of connections get piled up.
3. "Show engine innodb status" hangs

4. Lot of queries (around 25 for each) by rdsadmin which are in hang state.

   SELECT count(*) from mysql.rds_replication_status WHERE action = 'reset slave' and master_host is NULL and master_port is NULL GROUP BY action_timestamp,called_by_user,action,mysql_version,master_host,master_port ORDER BY action_timestamp LIMIT 1;
   
   SELECT NAME, VALUE FROM mysql.rds_configuration; 
   

5. After sometime, instance gets rebooted automatically with following error.

   MySQL restart initiated to address MySQL induced log backup issues. Note that as part of this resulution, a DB Snapshot will be performed after MySQL completes restarting.

Answer 1

We were able to resolve this issue by upgrading the instances to 5.6.34.