I've got a long running process on a headless server. It is a C program that monitors a modem for Caller Id and hangs up on telemarketers. On startup the program opens a connection to a MariaDB 10.3 database on the local machine.

Each morning at 4:00 AM a separate C program fetches the latest data file from the FTC. It includes about 100,000 complaints from the previous day. I add them to the database, so a blacklist table adds 50,000 to 60,000 rows. The current table has about 600,000 rows.

At 9:00 AM when the calls start my queries fail with the message shown below. It is a debugging trace of the C program. It includes Error: MySQL server has gone away.

I checked dmesg and /var/log but there is no indication of the reason or the problem. I can't find it logged anywhere. It also appears the OOM killer did not kill the MariaDB server.

# cat /var/log/mariadb/mariadb.log
# dmesg | grep -iE 'maria|mysql|oom'

How can I begin to troubleshoot this problem?

I can find lots of posts regarding MySQL server has gone away (apparently this is a chronic problem that has never been fixed). For example, here and here. They are dev-2-dev "try this" answers and lack troubleshooting to determine the root cause.

Debug: ===== new call =====
Debug: ring count: 1
Debug: modem_read
Debug: Read 68:
"DATE = 0308
TIME = 0936
NMBR = 4103174745
Debug: Ring: 0, Caller Id: 1
Debug: call_read_name
Debug: name: SPAM? PASADENA
Debug: call_read_number
Debug: Number: 4103174745
Debug: database_number_is_whitelist
Error: Failed to prepare statement, Error: MySQL server has gone away
Info: 4103174745 not found in whitelist
Debug: database_number_is_blacklist_lcl
Error: Failed to prepare statement, Error: MySQL server has gone away
Info: 4103174745 not found in local blacklist
Debug: database_number_is_blacklist_ftc
Error: Failed to prepare statement, Error: MySQL server has gone away
Info: 4103174745 not found in FTC blacklist
  • MariaDB gone away errors are pretty much the same as the mysql ones. Given it looks you have small queries I suspect this is timeout. Does your C program maintain long connections? What aspects of gone away have you investigated and why have you eliminated them? – danblack Mar 9 at 1:46
  • Thanks @dan. The MySQL page does the same damn thing the developers do on Stack Overflow. The page skips to "try this" without any troubleshooting to determine root cause. If MySQL is killing connections I expect them to be logged somewhere for troubleshooting purposes. Does MySqL really fail to log the failure anywhere? – user141074 Mar 9 at 1:56
  • increase max_allowed_packet to the maximum allowed (its harmless). If that doesn't work perhaps answering question about connection time. Are you running mariadb/you app on a different machine with a stateful firewall between them on connections that could got idle for some time? How – danblack Mar 9 at 22:19
  • Additional information request. Post on pastebin.com and share the links. A) complete (not edited) my.cnf or my.ini Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) complete MySQLTuner report F) last 500 lines of error log before incident, include restart log AND Optional very helpful information, if available includes - htop OR top OR mytop for most active apps, ulimit -a for a linux/unix list of limits, iostat -xm 5 3 for IOPS by device and core/cpu count, for server tuning analysis. – Wilson Hauck Mar 20 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy