1

So… we’re hitting a weird case the last week.

Our percona 8.0.31 server on ubuntu 22.04 (128G RAM) becomes unresponsive every day at the same time.

The symptoms are a bit weird too. Our usual connection count is about 30-40 connections. Then at 12:30 this starts to increase, or rather the reported number starts to increase. For the brief time we can monitor this, we can tell there are not any more active connections than usual.

Every day, at 12:30 exactly, ‘show processlist’ becomes unresponsive. Queries run for a few minutes more, but at 12:34 there’s nothing left to do but a hard kill. You can still connect to mysql at 12:30, but nothing works. (It just stalls.)

You’d say: easy - check the logs, crontabs, events, incoming connections, for anything that starts 12:30.

We did that:

  • There are no long running queries
  • There are no events running at 12:30
  • There are no cronjobs running at 12:30
  • There are no external connections (other then our usual webapplication) that execute at that time.
  • The logs show absolutely nothing. Mysql log shows nothing, syslog shows nothing.

What strikes me as weird is that ‘show processlist’ just stalls and never returns anything. Server does not crash or report anything.

This is a new server that has been migrated from another server running 8.0.30 just fine. Server is an ubuntu 22.04 server with 128G ram and pretty much nothing else runs on that machine. Mysql is limited to 70G, but tops out at 40G anyway.

Our (unfortunate) next step would be to downgrade, but that requires a lot of work and has no garantuees to work.

Is there anything I can enable on mysql or server that would add any useful logging?

Clarifications

This is a bare metal server. Nobody else has access.

Procceslist at 1230 shows nothing unusual.

Percona techs advise to gather a core dump when it occurs again. And we will enable the geneal log at 12:29.

8
  • Is this a virtual machine or a bare metal server? Do you manage it, or is it a service managed by someone else? You say "pretty much nothing else runs on that machine", yet you only allocate about half of its memory to MySQL; why do you waste the rest?
    – mustaccio
    Feb 24 at 15:27
  • Sure sounds like a backup provided by your Cloud service.
    – Rick James
    Feb 25 at 1:36
  • At 12:29, capture SHOW FULL PROCESSLIST; Also have the slowlog turned on; use pt-query-digest after across 12:30. If it is not obvious, show it to us.
    – Rick James
    Feb 25 at 1:39
  • 30-40 connections says nothing. What is the value of SHOW GLOBAL STATUS LIKE 'Threads_running'; (at 12:29)
    – Rick James
    Feb 25 at 1:40
  • "can't connect" -- Connect at 12:20 and don't disconnect. Another diagnostic: Provide the VARIABLES and GLOBAL STATUS; there may be some clues in them. How much RAM do you have? mysql.rjweb.org/doc.php/mysql_analysis#tuning
    – Rick James
    Feb 25 at 1:43

1 Answer 1

1

We have identified that what causes this crash. We removed the offending line friday, since then the service has not crashed. We re-introduced the line again today at another time - and to be sure mysql became completely unresponsive exactly at that time.

The code itself is too absurd, too simple to cause a complete mysql failure, yet it does.

This is the code, that ran in an non-interactive shell:

echo -n '' | nc {$host} 3306 | cut -d~ -f1

The bug has been acknowledged and fixed by percona:

https://jira.percona.com/plugins/servlet/mobile#issue/PS-8660

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.