1

I have a very minimal, fresh out of the box setup for MariaDB with Galera clustering. My master node of the cluster is able to run, and is reporting that there is only one node attached to the cluster, the master itself. When I try attaching the second node to the cluster I am receiving a state received error and the process errors out, and fails. The configuration on the master looks like this:

[mariadb]
wsrep_cluster_address=gcomm://
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1

This is in /etc/my.cnf.d/zabbix_cluster.cnf. The slave node looks similar, except it has the name of the Master node in it. When I run service mysql restart on the slave node, the output says that MySQL was start successfully, but when I do pgrep mysql it returns nothing. Upon examination of the /var/log/mysql/error.log I initialized, it says that there is a state receive error, and will never receive state. The output looks like this: 

130806 10:10:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130806 10:10:15 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.uZwsHWfH6y
130806 10:10:17 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
130806 10:10:17 [Warning] option 'general_log': boolean value '/var/log/mysql/mysqld.log' wasn't recognized. Set to OFF.
130806 10:10:17 [Warning] option 'slow_query_log': boolean value '/var/log/mysql-slow-queries.log' wasn't recognized. Set to OFF.
130806 10:10:17 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
130806 10:10:17 InnoDB: The InnoDB memory heap is disabled
130806 10:10:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130806 10:10:17 InnoDB: Compressed tables use zlib 1.2.3
130806 10:10:17 InnoDB: Using Linux native AIO
130806 10:10:17 InnoDB: Initializing buffer pool, size = 128.0M
130806 10:10:17 InnoDB: Completed initialization of buffer pool
130806 10:10:17 InnoDB: highest supported file format is Barracuda.
130806 10:10:17  InnoDB: Waiting for the background threads to start
130806 10:10:18 Percona XtraDB (http://www.percona.com) 1.1.8-29.3 started; log sequence number 1598129
130806 10:10:18 [Note] Plugin 'FEEDBACK' is disabled.
130806 10:10:18 [Note] Event Scheduler: Loaded 0 events
130806 10:10:18 [Note] WSREP: Read nil XID from storage engines, skipping position init
130806 10:10:18 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
130806 10:10:18 [Note] WSREP: wsrep_load(): Galera 23.2.4(r147) by Codership Oy <[email protected]> loaded succesfully.
130806 10:10:18 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130806 10:10:18 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130806 10:10:18 [Note] WSREP: Passing config to GCS: base_host = 10.162.111.109; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130806 10:10:18 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130806 10:10:18 [Note] WSREP: Start replication
130806 10:10:18 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
130806 10:10:18 [Note] WSREP: protonet asio version 0
130806 10:10:18 [Note] WSREP: backend: asio
130806 10:10:18 [Note] WSREP: GMCast version 0
130806 10:10:18 [Note] WSREP: (4e646cee-feaa-11e2-0800-10aa5e70a57b, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
130806 10:10:18 [Note] WSREP: (4e646cee-feaa-11e2-0800-10aa5e70a57b, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
130806 10:10:18 [Note] WSREP: EVS version 0
130806 10:10:18 [Note] WSREP: PC version 0
130806 10:10:18 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer 'zabbixcrt02:'
130806 10:10:19 [Note] WSREP: declaring 21415a01-fea8-11e2-0800-7061deb24ae4 stable
130806 10:10:19 [Note] WSREP: Node 21415a01-fea8-11e2-0800-7061deb24ae4 state prim
130806 10:10:19 [Note] WSREP: view(view_id(PRIM,21415a01-fea8-11e2-0800-7061deb24ae4,8) memb {
        21415a01-fea8-11e2-0800-7061deb24ae4,
        4e646cee-feaa-11e2-0800-10aa5e70a57b,
} joined {
} left {
} partitioned {
})
130806 10:10:19 [Note] WSREP: gcomm: connected
130806 10:10:19 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130806 10:10:19 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130806 10:10:19 [Note] WSREP: Opened channel 'my_wsrep_cluster'
130806 10:10:19 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
130806 10:10:19 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
130806 10:10:19 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.29-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_23.7.3.rXXXX
130806 10:10:19 [Note] WSREP: STATE EXCHANGE: sent state msg: 4eb2dd53-feaa-11e2-0800-5d7a774f5dbf
130806 10:10:19 [Note] WSREP: STATE EXCHANGE: got state msg: 4eb2dd53-feaa-11e2-0800-5d7a774f5dbf from 0 (ceszabbixcrt02)
130806 10:10:19 [Note] WSREP: STATE EXCHANGE: got state msg: 4eb2dd53-feaa-11e2-0800-5d7a774f5dbf from 1 (ceszabbixcrt03)
130806 10:10:19 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 7,
        members    = 1/2 (joined/total),
        act_id     = 0,
        last_appl. = -1,
        protocols  = 0/4/2 (gcs/repl/appl),
        group UUID = bcb32946-fea7-11e2-0800-32db11e867f1
130806 10:10:19 [Note] WSREP: Flow-control interval: [23, 23]
130806 10:10:19 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
130806 10:10:19 [Note] WSREP: State transfer required:
        Group state: bcb32946-fea7-11e2-0800-32db11e867f1:0
        Local state: 00000000-0000-0000-0000-000000000000:-1
130806 10:10:19 [Note] WSREP: New cluster view: global state: bcb32946-fea7-11e2-0800-32db11e867f1:0, view# 8: Primary, number of nodes: 2, my index: 1, protocol version 2
130806 10:10:19 [Warning] WSREP: Gap in state sequence. Need state transfer.
130806 10:10:21 [Note] WSREP: Prepared SST request: mysqldump|10.162.111.109:3306
130806 10:10:21 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130806 10:10:21 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2
130806 10:10:21 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bcb32946-fea7-11e2-0800-32db11e867f1): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():442. IST will be unavailable.
130806 10:10:21 [Note] WSREP: Node 1 (zabbixcrt03) requested state transfer from '*any*'. Selected 0 (zabbixcrt02)(SYNCED) as donor.
130806 10:10:21 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
130806 10:10:21 [Note] WSREP: Requesting state transfer: success, donor: 0
130806 10:10:24 [Warning] WSREP: 0 (zabbixcrt02): State transfer to 1 (zabbixcrt03) failed: -2 (No such file or directory)
130806 10:10:24 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.
130806 10:10:24 [Note] WSREP: gcomm: terminating thread
130806 10:10:24 [Note] WSREP: gcomm: joining thread
130806 10:10:24 [Note] WSREP: gcomm: closing backend
130806 10:10:25 [Note] WSREP: view(view_id(NON_PRIM,21415a01-fea8-11e2-0800-7061deb24ae4,8) memb {
        4e646cee-feaa-11e2-0800-10aa5e70a57b,
} joined {
} left {
} partitioned {
        21415a01-fea8-11e2-0800-7061deb24ae4,
})
130806 10:10:25 [Note] WSREP: view((empty))
130806 10:10:25 [Note] WSREP: gcomm: closed
130806 10:10:25 [Note] WSREP: /usr/sbin/mysqld: Terminated.
130806 10:10:25 mysqld_safe Number of processes running now: 0
130806 10:10:25 mysqld_safe WSREP: not restarting wsrep node automatically
130806 10:10:25 mysqld_safe mysqld from pid file /var/lib/mysql/zabbixcrt03.pid ended

I'm not sure why this is happening, or what it means. I see it can connect, but there is something that is not transferring from the Master to the Slave. What should I look for/at or do?

I also have made sure that /var/lib/mysql/ was owned by mysql:mysql and it is. the permissions on the directory are: 755

Improve this question
1
  • It has been suggested that for some reason I don't have a user on either box able to doing this. I need to either create a DBReplication user, or use something else like rsync. Any comments on that?
    – FilBot3
    Commented Aug 7, 2013 at 13:26

3 Answers 3

Reset to default
1
  1. What needed to be done to achieve clustering with a minimal config file was, take out all the wsrep statements until I was ready.
  2. Then start up MySQL, and create a user inside the database for the clustering/replication on both servers.
  3. Then add the line wsrep_sst_auth=<dbuser>:<passwd> on both servers.
  4. Now restart the Master instance of MariaDB first, wait until that is up, then restart the Slave instance of the MariaDB server. N

Now when you issue SHOW STATUS LIKE 'wsrep_%' inside mariadb, it should show 2 nodes in the cluster.

Improve this answer
0

I got this error after a clean installation. Somehow i think something messed up the MariaDB data directory (/var/lib/mysql). The solution was to delete the content of /var/lib/mysql on all nodes and run mysql_install_db --user=mysql. Be aware that this will delete all your existing data.

Improve this answer
0

It is not recommended to use mysqldump for SST, this could take a long time if you have a large database. i suggest changing to wsrep_sst_method=rsync or xtrabackup-v2. also set wsrep_sst_auth=<dbuser>:<passwd> to a user that exist in all nodes.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.