mariabackup from remote mariadb10.3 galera | '/var/lib/mysql/' (Errcode: 2 "No such file or directory")

Question

The following setup applies:

• 3 node galera cluster in docker
• backup container that should run mariabackup to get the data out of the galera cluster
• network is fine. backup container can ping galera
• mariadb/mariabackup version 10.3
• the backup container only has mariabackup installed.

The command that i used

mariabackup --backup --host=<service_name> --port=3306 --user=root --password=<pass> --target-dir=/backup

When trying to backup from galera-cluster from within the backup-container i get the following error

00] 2019-09-15 23:38:34 Connecting to MySQL server host: mdb_mariadb, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
[00] 2019-09-15 23:38:34 Using server version 10.3.18-MariaDB-1:10.3.18+maria~bionic-log
[00] 2019-09-15 23:38:34 Warning: option 'datadir' points to nonexistent directory '/var/lib/mysql/'
[00] 2019-09-15 23:38:34 Warning: MySQL variable 'datadir' points to nonexistent directory '/var/lib/mysql/'
[00] 2019-09-15 23:38:34 Warning: option 'datadir' has different values:
  '/var/lib/mysql/' in defaults file
  '/var/lib/mysql/' in SHOW VARIABLES
mariabackup based on MariaDB server 10.3.18-MariaDB debian-linux-gnu (x86_64)
[00] 2019-09-15 23:38:34 uses posix_fadvise().
mariabackup: Can't change dir to '/var/lib/mysql/' (Errcode: 2 "No such file or directory")
[00] 2019-09-15 23:38:34 my_setwd() failed , /var/lib/mysql/

What i tried.

• researching various gists for clues
• researching the mariadb scripts provided by the server install
• trying different option combinations, but always the same error.

Question:

• Has anyone made remote backups and can tell me how you succeeded?
• Is the approach wrong and is there a better way to make hot backups once per day and incremental during the day?

Answer 1

Wrong approach - mariabackup isn't designed to work remotely. See also MDEV-15075.

So you need to run mariabackup locally on one of the nodes in the cluster.

To prevent the backup from slowing down the cluster, you can prevent the node from engaging Flow Control and allow it to fall behind the other nodes by doing:

SET GLOBAL wsrep_desync=ON;  

Once the backup is completed, you can do:

SET GLOBAL wsrep_desync=OFF;

(See wsrep_desync in the MariaDB knowledge-base.)

Obviously, don't direct any database traffic at the node while you're running the backup.