2

I tried to deal with this issue for three days now, but unfortunately without success.

I have a Galera cluster (Node1 - donor, Node2 - joiner, Node3 - arbitrator) which was working flawlessly until I decided to update mariadb package.

Details: Node1 and Node2 are physical servers so I cannot revert a snapshot or similar, Node3 is VM and all of them are running Debian 9.

The first issue occurred right after doing the update (apt upgrade) and the error that I got was:

2018-08-10 12:01:53 139701369419520 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.7' --datadir '/var/lib/mysql/'   --parent '48395'  '' : 2
2018-08-10 12:01:53 139701369419520 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2018-08-10 12:01:53 139712303212736 [ERROR] WSREP: SST failed: 2 (No such file or directory)
2018-08-10 12:01:53 139712303212736 [ERROR] Aborting
2018-08-10 12:01:53 139701377812224 [Warning] WSREP: 0.0 (Node1): State transfer to 1.0 (Node2) failed: -22 (Invalid argument)
2018-08-10 12:01:53 139701377812224 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.
mariadb.service: Main process exited, code=killed, status=6/ABRT
Failed to start MariaDB 10.2.16 database server.
mariadb.service: Unit entered failed state.
mariadb.service: Failed with result 'timeout'.

The cluster is still alive and keeping quorum with 2 nodes (Node1 and Node3). A lot of googling and different ideas, including purge of all mariadb packages and:

If I run a clean installation of mysql on the server, without syncing set up, it starts fine. In case I enable the syncing, I get the same error as before. If, after this I want to start mysql again without syncing, I get the following error:

2018-08-10 11:57:01 140195030619904 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2018-08-10 11:57:01 140195030619904 [Note] InnoDB: Cannot open '/var/lib/mysql/ib_buffer_pool' for reading: No such file or directory
2018-08-10 11:57:01 140262652463296 [Note] Plugin 'FEEDBACK' is disabled.
2018-08-10 11:57:01 140262652463296 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2018-08-10 11:57:01 140262652463296 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2018-08-10 11:57:01 140262652463296 [Note] Server socket created on IP: '::'.
mariadb.service: Main process exited, code=exited, status=1/FAILURE
Failed to start MariaDB 10.2.16 database server.
mariadb.service: Unit entered failed state.
mariadb.service: Failed with result 'exit-code'

My config from /etc/mysql/my.cnf is:

[client]
port        = 3306
socket      = /var/run/mysqld/mysqld.sock

default-character-set   = utf8mb4

[mysqld_safe]
socket      = /var/run/mysqld/mysqld.sock
nice        = 0

[mysqld]
user        = mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port        = 3306
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking

skip-name-resolve
log_bin_trust_function_creators = 1
character-set-server    = utf8mb4

event_scheduler     = 1
innodb_mirrored_log_groups  = 1
max_seeks_for_key   = 18446744073709551615
max_write_lock_count    = 18446744073709551615

plugin-load         = sql_errlog.so
sql_error_log_rotate        = 1
sql_error_log_rotations     = 9
sql_error_log_size_limit    = 1M
sql_error_log_rate      = 1
sql_error_log_filename      = sql_errors.log

max_connections     = 1000
connect_timeout     = 10
wait_timeout        = 28800
max_allowed_packet  = 64M

thread_cache_size       = 128
sort_buffer_size    = 4M
bulk_insert_buffer_size = 16M
tmp_table_size      = 32M
max_heap_table_size = 32M

myisam_recover_options = BACKUP
key_buffer_size     = 128M
table_open_cache    = 400
myisam_sort_buffer_size = 512M
concurrent_insert   = 2
read_buffer_size    = 2M
read_rnd_buffer_size    = 1M
query_cache_limit       = 128K
query_cache_size        = 0
query_cache_type        = OFF
log_warnings        = 2
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
log_slow_verbosity  = query_plan

default_storage_engine  = InnoDB
innodb_log_file_size    = 128M
innodb_buffer_pool_size = 60G
innodb_log_buffer_size  = 8M
innodb_file_per_table   = 1
innodb_open_files   = 400
innodb_io_capacity  = 400
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit=2
innodb_file_format = Barracuda

[galera]

wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://192.168.1.6,192.168.1.7,192.168.1.8 # Node1, Node2, Node3 (arbitrator)

wsrep_cluster_name="galera_cluster"
wsrep_node_name='Node2'
wsrep_node_address="192.168.1.7"
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=galera_sst:secret
wsrep_provider_options="gcache.size = 10G"

binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

innodb_doublewrite=1
wsrep_retry_autocommit=3

[sst]
compressor="gzip"
decompressor="gzip -dc"

[mysqldump]
quick
quote-names
max_allowed_packet  = 16M

[isamchk]
key_buffer      = 16M

!includedir /etc/mysql/conf.d/

Logs on Node1 or Node3 are not showing anything useful.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.