In a production environment, our Master MySQL 5.6.26 stopped responding.
Our business handles about 1500 transactions per minute but there are times that nothing gets processes for many seconds (18 seconds this time).
We are logging show full processlist output every few seconds and we find the usual queries that normally took a fraction of a second holding for many seconds. But no indication of why.
In the past, we have an issue with the storage provider that has almost a second of latency and everything fell apart, but now is not the case, normal 5 to 20 milliseconds latency. Discusses here: how to detect infrastructure problems?
It happened once a few weeks ago and I found "waiting for Query cache lock" as the root cause, so we moved this frequent updated table to another system.
Now is happening more frequently, but nothing changed. The slow log shows queries that normally takes a fraction of seconds taken many seconds while this occurs, but before and after they behave just fine as usual.
I have the same feeling that, again, is something outside the database, but I don't know where to look.
What should I look?