So… we’re hitting a weird case the last week.
Our percona 8.0.31 server on ubuntu 22.04 (128G RAM) becomes unresponsive every day at the same time.
The symptoms are a bit weird too. Our usual connection count is about 30-40 connections. Then at 12:30 this starts to increase, or rather the reported number starts to increase. For the brief time we can monitor this, we can tell there are not any more active connections than usual.
Every day, at 12:30 exactly, ‘show processlist’ becomes unresponsive. Queries run for a few minutes more, but at 12:34 there’s nothing left to do but a hard kill. You can still connect to mysql at 12:30, but nothing works. (It just stalls.)
You’d say: easy - check the logs, crontabs, events, incoming connections, for anything that starts 12:30.
We did that:
- There are no long running queries
- There are no events running at 12:30
- There are no cronjobs running at 12:30
- There are no external connections (other then our usual webapplication) that execute at that time.
- The logs show absolutely nothing. Mysql log shows nothing, syslog shows nothing.
What strikes me as weird is that ‘show processlist’ just stalls and never returns anything. Server does not crash or report anything.
This is a new server that has been migrated from another server running 8.0.30 just fine. Server is an ubuntu 22.04 server with 128G ram and pretty much nothing else runs on that machine. Mysql is limited to 70G, but tops out at 40G anyway.
Our (unfortunate) next step would be to downgrade, but that requires a lot of work and has no garantuees to work.
Is there anything I can enable on mysql or server that would add any useful logging?
This is a bare metal server. Nobody else has access.
Procceslist at 1230 shows nothing unusual.
Percona techs advise to gather a core dump when it occurs again. And we will enable the geneal log at 12:29.
SHOW FULL PROCESSLIST;Also have the slowlog turned on; use pt-query-digest after across 12:30. If it is not obvious, show it to us.
SHOW GLOBAL STATUS LIKE 'Threads_running';(at 12:29)
GLOBAL STATUS; there may be some clues in them. How much RAM do you have? mysql.rjweb.org/doc.php/mysql_analysis#tuning