1

We are a free/open source project. We are having problems determining why MariaDB 5.5.58 on CentOS 7 is not starting. The start problems happened after we enabled CentOS Software Collections (SCL) and performed a MediaWiki upgrade.

Lack of useful error information is making it hard for us to troubleshoot the issue. I found CentOS's Issue 9519, /usr/libexec/mariadb-prepare-db-dir fails (and Mariadb won't start - Failed at step EXEC spawning), but the permissions on / seem OK (see below). We also have the PID file discussed in Red Hat's Issue 1082018, systemctl start mariadb.service wouldn't start.

The result of running /usr/libexec/mariadb-prepare-db-dir manually with set -x as root is shown below. I don't know how systemd is running it (I have not found the commands to print the information yet).

We rent our VM so there is no dmesg or syslog to inspect. It is empty.

If anyone could provide more information on the cause of the failure or additional steps to troubleshoot, then we would be grateful.


MariaDB Conf

# grep -v '#' /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

!includedir /etc/my.cnf.d

And then:

# echo "" > /var/log/mariadb/mariadb.log

Other Directories

# ls -al /
total 64
dr-xr-xr-x 20 root root 4096 Dec 23 15:46 .
dr-xr-xr-x 20 root root 4096 Dec 23 15:46 ..

# ls -al /var/run/mariadb/
total 0
drwxr-xr-x  2 mysql mysql  40 Dec 23 15:46 .
drwxr-xr-x 26 root  root  680 Dec 23 18:26 ..

Data Directory

# ls -al /var/lib/mysql
total 35060
drwxr-xr-x  5 mysql mysql     4096 Dec 23 15:45 .
drwxr-xr-x 30 root  root      4096 Dec 23 18:52 ..
-rw-rw----  1 mysql mysql    16384 Dec 23 15:45 aria_log.00000001
-rw-rw----  1 mysql mysql       52 Dec 23 15:45 aria_log_control
-rw-rw----  1 mysql mysql 35651584 Dec 23 15:45 ibdata1
-rw-rw----  1 mysql mysql  5242880 Dec 23 15:45 ib_logfile0
-rw-rw----  1 mysql mysql  5242880 Dec 23 15:31 ib_logfile1
drwx------  2 mysql mysql     4096 Sep 18  2016 mysql
drwx------  2 mysql mysql    16384 Dec 23 01:14 my_wiki
drwx------  2 mysql mysql     4096 Sep 18  2016 performance_schema

Attempt to Start

# ls /usr/libexec/mariadb-prepare-db-dir
/usr/libexec/mariadb-prepare-db-dir

# systemctl start mariadb.service
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

mariadb.log

# cat /var/log/mariadb/mariadb.log
#

systemctl status

# systemctl -l status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2017-12-23 18:29:28 EST; 23s ago
  Process: 14537 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=1/FAILURE)

Dec 23 18:29:28 ftpit systemd[1]: Starting MariaDB database server...
Dec 23 18:29:28 ftpit systemd[1]: mariadb.service: control process exited, code=exited status=1
Dec 23 18:29:28 ftpit systemd[1]: Failed to start MariaDB database server.
Dec 23 18:29:28 ftpit systemd[1]: Unit mariadb.service entered failed state.
Dec 23 18:29:28 ftpit systemd[1]: mariadb.service failed.
Oct 27 19:05:23 ftpit systemd[1]: Stopping MariaDB database server...
Oct 27 19:05:28 ftpit systemd[1]: Stopped MariaDB database server.
Nov 16 12:48:44 ftpit systemd[1]: Stopping MariaDB database server...
Nov 16 12:48:54 ftpit systemd[1]: Stopped MariaDB database server.

journalctl -xe

# journalctl -xe
-- Subject: Unit sysinit.target has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sysinit.target has finished shutting down.
Nov 16 12:48:57 ftpit systemd[1]: Stopping System Initialization.
-- Subject: Unit sysinit.target has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sysinit.target has begun shutting down.
Nov 16 12:48:57 ftpit systemd[1]: Stopped target Swap.
-- Subject: Unit swap.target has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit swap.target has finished shutting down.
Nov 16 12:48:57 ftpit systemd[1]: Stopping Swap.
-- Subject: Unit swap.target has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit swap.target has begun shutting down.

Here is the result of running /usr/libexec/mariadb-prepare-db-dir manually with set -x added to the script. Honestly, I can't tell if this is expected or not.

# /usr/libexec/mariadb-prepare-db-dir
+ get_mysql_option mysqld datadir /var/lib/mysql
+ '[' 3 -ne 3 ']'
++ tail -n 1
++ sed -n 's/^--datadir=//p'
++ /usr/bin/my_print_defaults mysqld
+ result=/var/lib/mysql
+ '[' -z /var/lib/mysql ']'
+ datadir=/var/lib/mysql
+ get_mysql_option mysqld_safe log-error /var/log/mariadb/mariadb.log
+ '[' 3 -ne 3 ']'
++ sed -n 's/^--log-error=//p'
++ tail -n 1
++ /usr/bin/my_print_defaults mysqld_safe
+ result=/var/log/mariadb/mariadb.log
+ '[' -z /var/log/mariadb/mariadb.log ']'
+ errlogfile=/var/log/mariadb/mariadb.log
+ get_mysql_option mysqld socket /var/lib/mysql/mysql.sock
+ '[' 3 -ne 3 ']'
++ tail -n 1
++ sed -n 's/^--socket=//p'
++ /usr/bin/my_print_defaults mysqld
+ result=/var/lib/mysql/mysql.sock
+ '[' -z /var/lib/mysql/mysql.sock ']'
+ socketfile=/var/lib/mysql/mysql.sock
+ SERVICE_NAME=
+ '[' x = x ']'
+ SERVICE_NAME=mysqld.service
++ sed 's/^User=//'
++ systemctl show -p User mysqld.service
+ myuser=
+ '[' x = x ']'
+ myuser=mysql
++ sed 's/^Group=//'
++ systemctl show -p Group mysqld.service
+ mygroup=
+ '[' x = x ']'
+ mygroup=mysql
++ dirname /var/log/mariadb/mariadb.log
+ '[' '!' -e /var/log/mariadb/mariadb.log -a '!' -h /var/log/mariadb/mariadb.log -a x/var/log/mariadb = x/var/log ']'
++ dirname /var/log/mariadb/mariadb.log
+ errlogdir=/var/log/mariadb
+ '[' -d /var/log/mariadb ']'
+ '[' -e /var/log/mariadb/mariadb.log -a '!' -w /var/log/mariadb/mariadb.log ']'
+ fuser /var/lib/mysql/mysql.sock
+ export LC_ALL=C
+ LC_ALL=C
+ should_initialize /var/lib/mysql
++ list_datadir /var/lib/mysql
++ grep -v -e '^lost+found$' -e '\.err$' -e '^.bash_history$'
++ ls -1A /var/lib/mysql
+ test -z 'aria_log.00000001
aria_log_control
ib_logfile0
ib_logfile1
ibdata1
my_wiki
mysql
performance_schema'
+ '[' -d /var/lib/mysql/mysql/ ']'
+ echo 'Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.'
Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
++ basename /usr/libexec/mariadb-prepare-db-dir
+ echo 'If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.'
If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
+ exit 0
  • start MariaDB manually and paste the error message on screen mysqld_safe --defaults-file=/etc/my.cnf. When there is no error message have a look at strace and manual start. – Ivanov Dec 24 '17 at 8:12
  • Has MariaDB been running since Sep 2016? Did it die reacently? Was anything upgraded (disk, MariaDB, etc) recently? – Rick James Dec 28 '17 at 22:43
0

Determine cause of failed MariaDB start on CentOS 7

We found the cause of the problem. It seems the Linux OOM killer is killing the service when the VM runs out of memory.

Aggravating circumstances include a VM provided by GoDaddy. The machine has 1 GB of RAM and no virtual memory. We cannot upgrade RAM and we cannot configure a swap file.

We attempted to tune the VM performance for Apache and PHP, including web server parameters and caches. Unfortunately none of us are experts in this area so we are not sure how effective the changes were. In fact, given the problem reoccurs our attempts seems to have failed.

We are going to provide a cron job that (1) restarts the service if it is not running; and (2) repairs the database on a restart. (2) is important because we see corrupt tables on occasion after a kill.

For those in a similar situation, here are the commands for CentOS7 and MySQL:

  • Check service status

    systemctl status mariadb.service

  • Repair the database

    mysqlcheck --all-databases --auto-repair --user=XXX --password=YYY


Here is the script:

# cat /etc/cron.hourly/1restart-mysql.cron
#!/usr/bin/env bash

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

RUNNING=$(systemctl status mariadb.service | grep -i -c "running")
if [[ "$RUNNING" -eq 0 ]]; then
    systemctl restart mariadb.service
    mysqlcheck --all-databases --auto-repair --user=XXX --password=YYY
fi

And:

# ls -Al /etc/cron.hourly
total 8
-rwxr-xr-x 1 root root 362 Apr 13  2018 0yum-hourly.cron
-rwxr-xr-x 1 root root 429 Oct 13 17:33 1restart-mysql.cron

And here is how we determined the cause. We missed it originally because our first step was to bounce the server to restore service. Later we would check dmesg and it would be empty. We did not find this information in other logs.

# dmesg
[51000952.633919] Out of memory in UB 5299: OOM killed process 692 (mysqld) score 0 vm:1689248kB, rss:101160kB, swap:0kB
[51000954.240514] Out of memory in UB 5299: OOM killed process 18692 (httpd) score 0 vm:484768kB, rss:27528kB, swap:0kB
[51021879.542687] Out of memory in UB 5299: OOM killed process 20755 (mysqld) score 0 vm:1684804kB, rss:96828kB, swap:0kB
[51031927.106440] Out of memory in UB 5299: OOM killed process 26157 (mysqld) score 0 vm:1687764kB, rss:92504kB, swap:0kB
[51031927.991303] Out of memory in UB 5299: OOM killed process 27789 (httpd) score 0 vm:484616kB, rss:28828kB, swap:0kB
[51031928.375203] Out of memory in UB 5299: OOM killed process 27999 (httpd) score 0 vm:484596kB, rss:28632kB, swap:0kB
[51031928.851050] Out of memory in UB 5299: OOM killed process 27722 (httpd) score 0 vm:482796kB, rss:26124kB, swap:0kB
[51038506.759513] Out of memory in UB 5299: OOM killed process 28865 (mysqld) score 0 vm:1683916kB, rss:92152kB, swap:0kB
[51038508.175325] Out of memory in UB 5299: OOM killed process 30527 (mysqld) score 0 vm:680280kB, rss:69276kB, swap:0kB
[51047405.296807] Out of memory in UB 5299: OOM killed process 21185 (mysqld) score 0 vm:1682140kB, rss:95504kB, swap:0kB
[51047406.815046] Out of memory in UB 5299: OOM killed process 21994 (mysqld) score 0 vm:413284kB, rss:30480kB, swap:0kB
[51094257.662248] Out of memory in UB 5299: OOM killed process 2751 (mysqld) score 0 vm:1678112kB, rss:95196kB, swap:0kB
[51097918.747681] Out of memory in UB 5299: OOM killed process 3689 (mysqld) score 0 vm:1683916kB, rss:91052kB, swap:0kB
[51098957.885464] Out of memory in UB 5299: OOM killed process 4833 (mysqld) score 0 vm:1683620kB, rss:91232kB, swap:0kB
[51099129.993959] Out of memory in UB 5299: OOM killed process 13167 (mysqld) score 0 vm:1686284kB, rss:95464kB, swap:0kB
[51099131.831039] Out of memory in UB 5299: OOM killed process 13255 (httpd) score 0 vm:482484kB, rss:25508kB, swap:0kB
[51099132.017423] Out of memory in UB 5299: OOM killed process 13215 (httpd) score 0 vm:480240kB, rss:22188kB, swap:0kB
[51099132.211569] Out of memory in UB 5299: OOM killed process 13216 (httpd) score 0 vm:478200kB, rss:20832kB, swap:0kB
[51099132.390457] Out of memory in UB 5299: OOM killed process 13240 (httpd) score 0 vm:478040kB, rss:20900kB, swap:0kB
[51099132.505624] Out of memory in UB 5299: OOM killed process 13245 (httpd) score 0 vm:478036kB, rss:20404kB, swap:0kB
-1

This issue might be already solved but I'm still giving one more solution to this just in case someone comes looking for an answer.

Starting the mariadb server creates issues sometimes if the install is done after uninstalling any previously installed version.

Here is the specific error when you need to clean the data-

Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.

If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.

In this case try removing all files from "/var/lib/mysql"

cd /var/lib/mysql
rm -rf *
systemctl start mariadb

This will clean all old data and will start mariadb.

  • You SHOULD NOT remove "/var/lib/mysql" which contains original DB file before everything is working. – Cheung Sep 2 at 6:29
  • That's why I've added "This will clean all old data and will start mariadb.", it's basically for people who are on fresh install and installed it incorrectly and just trying to remove everything and start fresh. – Arun Kumar 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy