Mariadb Galera Cluster using Xtradb backup is not working

Question

I am trying out the first test of MariaDB Galera cluster setup without success.

This is the error in the log

innobackupex: Created backup directory /tmp
tar: -: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now
innobackupex: 'tar chf -' returned with exit code 2.
innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 4882
        main::backup_file_via_stream('/tmp', 'backup-my.cnf') called at /usr//bin/innobackupex line 4931
        main::backup_file('/tmp', 'backup-my.cnf', '/tmp/backup-my.cnf') called at /usr//bin/innobackupex line 4955
        main::write_to_backup_file('/tmp/backup-my.cnf', '# This MySQL options file was generated by innobackupex.\x{a}\x{a}# T...') called at /usr//bin/innobackupex line 3762
        main::write_backup_config_file('/tmp/backup-my.cnf') called at /usr//bin/innobackupex line 3689
        main::init() called at /usr//bin/innobackupex line 1566
innobackupex: Error: Failed to stream '/tmp/backup-my.cnf': 2 at /usr//bin/innobackupex line 4882.

Joiner my.cnf

#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]
datadir=/var/lib/mysql
log_error=/etc/log/mysql/mysql.err
log_bin=/etc/log/mysql/mysql-bin.log
slow_query_log_file=/etc/log/mysql/mysql-slow.log
# this is only for the mysqld standalone daemon
[mysqld]
user=mysql
innodb_log_file_size=100M

# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://1.1.1.1 #master
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
wsrep_cluster_name=rms
wsrep_node_address=1.1.1.2
wsrep_node_name=slave
wsrep_sst_method=xtrabackup
wsrep_sst_auth=sst:sst
# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.0 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.0]

Donor my.cnf

#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]
datadir=/var/lib/mysql
log_error=/etc/log/mysql/mysql.err
log_bin=/etc/log/mysql/mysql-bin.log
slow_query_log_file=/etc/log/mysql/mysql-slow.log

wsrep_sst_receive_address=1.1.1.2
# this is only for the mysqld standalone daemon
[mysqld]
user=mysql
innodb_log_file_size=100M
#
# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
wsrep_cluster_name=rms
wsrep_node_address=1.1.1.1
wsrep_node_name=master
wsrep_sst_method=xtrabackup
wsrep_sst_auth=sst:sst

# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.0 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.0]

I have been trying this without success from past 10 hours, please any help is appreciated. Using CentOS 6

When I execute this in donor it's working fine [to test xtrabackup] innobackupex --user=sst --password=sst /tmp/

Answer 1

In my experience, it seems one failed run can leave fragments of config that block future runs.

Case in reference: Xbstream rather than tar is used, although the traceback is consistent in that it halts on writing to backup-my.cnf.

First run fails because my_print_defaults is reading from the wrong file, and this file was stored as the backup.

Once the --defaults-file flag is added to innobackupex, suddenly there is output on the other side of the xbstream.

xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 17 - File exists)
xbstream: failed to create file.

I believe this is an innobackupex safety feature and tar may be making the same checks as xbstream. I would have liked to comment on this rather than answer it but lack the reputation.