MariaDB does not start after restoring incremental backups with mariabackup

Question

OS: CentOS 7

MariaDB: mariadb Ver 15.1 Distrib 10.4.12-MariaDB, for Linux (x86_64) using readline 5.1

---

I am creating a full backup every 7 days on a replication slave, and an incremental backup every hour on top of that.

This script is used to create backups: run-mariabackup.sh

To restore the backup I am using this process.

However, after restoring the backup MariaDB does not start.

systemd: Starting MariaDB 10.4.12 database server...
mysqld: [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB) starting as process 11363 ...
mysqld: [ERROR] mysqld: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
mysqld: [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
mysqld: [ERROR] Plugin 'Aria' init function returned error.
mysqld: [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
mysqld: [Note] InnoDB: Using Linux native AIO
mysqld: [ERROR] InnoDB: Operating system error number 13 in a file operation.
mysqld: [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
mysqld: [ERROR] InnoDB: os_file_get_status() failed on './ibdata1'. Can't determine file permissions
mysqld: [ERROR] Plugin 'InnoDB' init function returned error.
mysqld: [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
mysqld: [Note] Plugin 'FEEDBACK' is disabled.
mysqld: [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
mysqld: [ERROR] Failed to initialize plugins.
mysqld: [ERROR] Aborting
systemd: mariadb.service: main process exited, code=exited, status=1/FAILURE

Of course I have already done chown -R mysql:mysql /var/lib/mysql prior to that, so this error puzzled me. After this error I tried chmod -R 777 /var/lib/mysql to see if there is some permission that was missing, but starting MariaDB again provided me with the same error.

Here are the exact steps I took to restore the backup:

cd /home/mysqlbackup/
for i in $(find . -name backup.stream.gz | grep '2020-03-12_12-08-44' | xargs dirname); do
        mkdir -p $i/backup;
        zcat $i/backup.stream.gz | mbstream -x -C $i/backup/;
done
mariabackup --prepare --target-dir base/2020-03-12_12-08-44/backup/ --user backup --password mypass
mariabackup --prepare --target-dir base/2020-03-12_12-08-44/backup/ --user backup --password mypass --incremental-dir incr/2020-03-12_12-08-44/2020-03-12_13-24-20/backup/
mariabackup --prepare --target-dir base/2020-03-12_12-08-44/backup/ --user backup --password mypass --incremental-dir incr/2020-03-12_12-08-44/2020-03-12_13-54-25/backup/
mariabackup --prepare --target-dir base/2020-03-12_12-08-44/backup/ --user backup --password mypass --incremental-dir incr/2020-03-12_12-08-44/2020-03-12_14-00-02/backup/
mariabackup --prepare --target-dir base/2020-03-12_12-08-44/backup/ --user backup --password mypass --incremental-dir incr/2020-03-12_12-08-44/2020-03-12_15-00-01/backup/
mariabackup --prepare --target-dir base/2020-03-12_12-08-44/backup/ --user backup --password mypass --incremental-dir incr/2020-03-12_12-08-44/2020-03-12_16-00-01/backup/
systemctl stop mariadb
mv /var/lib/mysql/ /var/lib/mysql_bak/
mariabackup --copy-back --target-dir base/2020-03-12_12-08-44/backup/ --user backup --password mypass --datadir /var/lib/mysql/
chown -R mysql:mysql /var/lib/mysql/
systemctl start mariadb

Answer 1

When the permissions look correct, but you're still getting Errcode: 13 "Permission denied", then the next thing to check is whether a security system is causing the problem. On CentOS that system is SELinux. To check if it's running in enforcing mode, execute this:

sestatus

You can do setenforce 0 to temporarily change it to permissive mode.

To avoid the problem:

When you initially install MariaDB, the SELinux security context should be set correctly for all the files. However, when you do mv /var/lib/mysql/ /var/lib/mysql_bak/, then you remove it. I think you can get it back by simply executing restorecon -Rv /var/lib/mysql.

For reference, read this page in the MariaDB Knowledgebase: Security-Enhanced Linux with MariaDB