I am having problems when a MariaDB replication slave wants to upgrade its tables after a server was patched. This happens consistently with any sort of OS patching.
The system has several slave connections to several masters.
Question:
How can I get a MariaDB slave system to start automatically after the OS was patched and rebooted?
Symptoms:
# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
# systemctl status mariadb
● mariadb.service - MySQL server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2021-10-13 01:18:21 EDT; 3min 10s ago
Process: 30780 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, status=1/FAILURE)
Process: 30772 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, status=0/SUCCESS)
/var/tmp/mysql-protected.Luqvvt # grep ERROR log_upgrade_run.err
2021-10-13 1:26:32 139921358805120 [ERROR] Failed to open the relay log './zamv00010-relay-bin-ghn1.000007' (relay_log_pos 702)
2021-10-13 1:26:32 139921358805120 [ERROR] Could not find target log during relay log initialization
2021-10-13 1:26:32 139921358805120 [ERROR] Initialized Master_info from 'master-ghn1.info' failed
2021-10-13 1:26:32 139921358805120 [ERROR] Failed to open the relay log './zamv00010-relay-bin-ghn2.000007' (relay_log_pos 662)
2021-10-13 1:26:32 139921358805120 [ERROR] Could not find target log during relay log initialization
2021-10-13 1:26:32 139921358805120 [ERROR] Initialized Master_info from 'master-ghn2.info' failed
2021-10-13 1:26:32 139921358805120 [ERROR] Failed to open the relay log './zamv00010-relay-bin-ghna.000046' (relay_log_pos 234543216)
2021-10-13 1:26:32 139921358805120 [ERROR] Could not find target log during relay log initialization
2021-10-13 1:26:32 139921358805120 [ERROR] Initialized Master_info from 'master-ghna.info' failed
2021-10-13 1:26:32 139921358805120 [ERROR] Failed to open the relay log './zamv00010-relay-bin-ghnb.000195' (relay_log_pos 556)
2021-10-13 1:26:32 139921358805120 [ERROR] Could not find target log during relay log initialization
2021-10-13 1:26:32 139921358805120 [ERROR] Initialized Master_info from 'master-ghnb.info' failed
2021-10-13 1:26:32 139921358805120 [ERROR] Failed to initialize multi master structures
2021-10-13 1:26:32 139921358805120 [ERROR] Aborting
The supposedly missing relay logs do in fact exist:
# ls -la /var/lib/mysql/zamv00010-relay-bin-ghna.000046
-rw-rw---- 1 mysql mysql 234543239 Oct 13 01:26 /var/lib/mysql/zamv00010-relay-bin-ghna.000046
but they do so in /var/lib/mysql (which is where they belong...). But the mysqld started by mysql-systemd-helper is looking for them somwehere else. I am not sure where, maybe in /var/tmp/mysql-protected.Luqvvt.
Now, this is easy to fix manually: Just start mysqld manually, run mysql_upgrade manually, done. However, it is very annoying to have to do this after every OS patching round (of which these servers have many and where the prople responsible for them insist on rebooting every time). It also seems like a bug to me since t seems that it can simply not work with replication enabled. But maybe I am missing something...
EDIT Since this was requested in a comment:
MariaDB [(none)]> SHOW VARIABLES LIKE 'relay_log%';
+-----------------------+----------------+
| Variable_name | Value |
+-----------------------+----------------+
| relay_log | |
| relay_log_basename | |
| relay_log_index | |
| relay_log_info_file | relay-log.info |
| relay_log_purge | ON |
| relay_log_recovery | OFF |
| relay_log_space_limit | 524288000 |
+-----------------------+----------------+
journalctl -n 100 mariadb.service).SHOW VARIABLES LIKE 'relay_log%';