Why setting up new galera cluster with mariabackup as sst starts but all other nodes failed with same error?

Question

I did a fresh reinstallation of mariadb-server on all the nodes (I removed using sudo apt purge mariadb-*)

I started the first node using sudo galera_new_cluster it went fine and is still running. but other nodes threw this error:

● mariadb.service - MariaDB 10.3.27 database serverLoaded: 
loaded (/lib/systemd/system/mariadb.service; 
enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2020-12-19 20:23:19 IST; 2min 9s ago
Docs: man:mysqld(8)
Process: 7089 
ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)Process: 7090 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)Process: 7092 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=\cd /usr/bin/..; /usr/bin/galera_recovery\; [ $? -eq 0 ] && s`
    Process: 7330 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)Main PID: 7330 (code=exited, status=1/FAILURE)
    Status: "MariaDB server is down"
    Dec 19 20:22:53 phl-pi-3 systemd[1]: Starting MariaDB 10.3.27 database server...
    Dec 19 20:22:59 phl-pi-3 sh[7092]: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
    Dec 19 20:22:59 phl-pi-3 mysqld[7330]: 2020-12-19 20:22:59 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1-log) starting as process 7330 ...
    Dec 19 20:22:59 phl-pi-3 mysqld[7330]: 2020-12-19 20:22:59 0 [Warning] Could not increase number of max_open_files to more than 16384 (request: 32186)
    Dec 19 20:23:19 phl-pi-3 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Dec 19 20:23:19 phl-pi-3 systemd[1]: mariadb.service: Failed with result 'exit-code'.
    `Dec 19 20:23:19 phl-pi-3 systemd[1]: Failed to start MariaDB 10.3.27 database server.``

this is my galera config:

[mysqld]#mysql settings
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_size=0query_cache_type=0
bind-address=0.0.0.0
#galera settings
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="test_cluster"
wsrep_cluster_address=gcomm://192.168.0.15,192.168.0.16,192.168.0.12,10.8.0.6
wsrep_node_address="192.168.0.15"
wsrep_sst_method=mariabackup
wsrep_sst_donor=192.168.0.16

all other nodes have same galera config except different wsrep_address and dont have wsrep_sst_donor set.

And the other server config is as below:

$ cat 50-server.cnf
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
## See the examples of server my.cnf files in /usr/share/mysql/
## this is read by the standalone daemon and embedded servers
[server]
skip_name_resolve = 1
# this is only for the mysqld standalone daemon
[mysqld]
transaction_isolation = READ-COMMITTED
binlog_format = ROW
innodb_large_prefix=on
innodb_file_format=barracuda
innodb_file_per_table=1
innodb_io_capacity=4000
# * Basic Settings
user = mysqlpid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
bind-address = 0.0.0.0
# * Fine Tuning
key_buffer_size = 16
Mmax_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam_recover_options = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
## * Query Cache Configuration
#query_cache_limit = 1M
query_cache_type = 1
query_cache_limit = 2M
query_cache_min_res_unit = 2k
query_cache_size = 64M
## * Logging and Replication
## Error log - should be very few entries.
#log_error = /var/log/mysql/error.log
server-id = 16
log_bin = mariadb_bin
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = exclude_database_name
innodb_buffer_pool_size = 128M
innodb_buffer_pool_instances = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 32M
innodb_max_dirty_pages_pct = 90
# For generating SSL certificates you can use for example the GUI tool "tinyca".
## ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
## Accept only connections using the latest and most secure TLS protocol version.
# ..when MariaDB is compiled with OpenSSL:
# ssl-cipher=TLSv1.2
# ..when MariaDB is compiled with YaSSL (default in Debian):
# ssl=on
## * Character sets## MySQL/MariaDB default is Latin1, but in Debian we rather default to the full
# utf8 4-byte character set. See also client.cnf
#character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
tmp_table_size= 64Mmax_heap_table_size= 64M
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 1

all other nodes have same as above except different server-id