0

I have a galera cluster with 2 mariadb nodes running. I know that this isn't best practise but I hadn't a chance to do it in another way.

I had a crash and needed to reinitiate the cluster on one node using galera_new_cluster.

The first node a is now running fine, is accessible and the cluster size shows 1 correctly.

But now when I try to start mariadb on the second node b I got the following:

    mariadb.service - MariaDB 10.3.13 database server
    Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
    Active: failed (Result: timeout) since Tue 2019-12-03 17:19:01 CET; 3h 51min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/

    Dec 03 17:17:31 node-b.my.domain systemd[1]: Starting MariaDB 10.3.13 database server...
    Dec 03 17:19:01 node-b.my.domain systemd[1]: mariadb.service start-pre operation timed out. Terminating.
    Dec 03 17:19:01 node-b.my.domain systemd[1]: Failed to start MariaDB 10.3.13 database server.
    Dec 03 17:19:01 node-b.my.domain systemd[1]: Unit mariadb.service entered failed state.
    Dec 03 17:19:01 node-b.my.domain systemd[1]: mariadb.service failed.

While this process is trying to start mariadb it fills up my /tmp very quickly.

I tried to enable error logging unsing this in server.cnf:

    log-error=/var/log/mariadb.log

nothing was logged, I even gave chmod 777 to the file, but still no luck.

I dont know what to do next.

the last times I needed to do galera_new_cluster the second node was starting and connecting to the cluster just fine.

here is the config of node a:

    [galera]
    # Mandatory settings
wsrep_on=ON
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0
datadir=/var/lib/mysql
innodb_log_file_size=100M
innodb_file_per_table
innodb_flush_log_at_trx_commit=2
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://172.17.175.200,172.17.175.201"
wsrep_cluster_name='galera_cluster'
wsrep_node_address='172.17.175.200'
wsrep_node_name='node-a'
wsrep_sst_method=rsync
wsrep_sst_auth=username:password

and node b:

wsrep_on=ON
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0
datadir=/var/lib/mysql
innodb_log_file_size=100M
innodb_file_per_table
innodb_flush_log_at_trx_commit=2
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://172.17.175.200,172.17.175.201"
wsrep_cluster_name='galera_cluster'
wsrep_node_address='172.17.175.201'
wsrep_node_name='node-b'
wsrep_sst_method=rsync
wsrep_sst_auth=username:password

grastate.dat on node a:

version: 2.1
uuid:    da45dbcb-15b9-11e9-b1fc-e7c9d7353265
seqno:   -1
safe_to_bootstrap: 1

grastate.dat on node b:

version: 2.1
uuid:    00000000-0000-0000-0000-000000000000
seqno:   -1
safe_to_bootstrap: 0

I checked file permissions on /tmp already and they are fine.

thank you for any hints or possible next steps.

3
  • I guess you could try to see what the mariadb.service unit tries to do at "start-pre" and why it might time out. – mustaccio Dec 4 '19 at 14:59
  • @mustaccio is pretty much on track - mariadb.com/kb/en/library/systemd/#ssts-and-systemd increase the timeout. Work to fix this automaticly probably needs a newer that systemd-240 version - jira.mariadb.org/browse/MDEV-15607 – danblack Dec 5 '19 at 3:51
  • Thanks for the hints. I got this from command as the failed pre: ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR='/usr/bin/galera_recovery'; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=killed, signal=TERM) I haven't changed this, its default. – Duffkess Dec 5 '19 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.