Mysql InnoDB Cluster

Question

I'm trying to setup mysql innodb cluster. I have couple of Digital Ocean (DO) instances running centos 7.

I have installed Mysql Ver 8.0.21 for Linux on x86_64 (MySQL Community Server - GPL) on the DO instances, alone with Mysql-Shell Ver 8.0.21.

I am able to configure all instances (dba.configureInstance('instance')), ready to be added to the cluster. I have created a cluster on one node (dba.createCluster('cluster_name')). I get following message for cluster.status()

  > cluster.status()
  {
    "clusterName": "myCluster",
    "defaultReplicaSet": {
    "name": "default",
    "primary": "139.59.1.215:3306",
    "ssl": "REQUIRED",
    "status": "OK_NO_TOLERANCE",
    "statusText": "Cluster is NOT tolerant to any failures.",
    "topology": {
        "139.59.1.215:3306": {
            "address": "139.59.1.215:3306",
            "mode": "R/W",
            "readReplicas": {},
            "replicationLag": null,
            "role": "HA",
            "status": "ONLINE",
            "version": "8.0.21"
        }
    },
    "topologyMode": "Single-Primary"
},
"groupInformationSourceMember": "139.59.1.215:3306"

}

Now, when I try to add other instances to this cluster (dba.addInstance('other_instance')) and selecting default clone option, I'm getting the following error from the primary node shell.

ERROR: Unable to start Group Replication for instance '139.59.21.251:3306'. Please check the MySQL server error log for more information.
  Cluster.addInstance: Group Replication failed to start: MySQL Error 3092 (HY000): 139.59.21.251:3306: 
  The server is not configured properly to be an active member of the group. 
  Please see more details on error log. (RuntimeError)

And when I check the the log file of the secondary node that I'm trying to add, I see the following error messages

[System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
[System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
[Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv4 localhost address to the whitelist. It is mandatory that it is added.'
[System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine instance.'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine instance.'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine instance.'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine instance.'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
[ERROR] [MY-011640] [Repl] Plugin group_replication reported: 'Timeout on wait for view after joining group'
[ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member is leaving a group without being on one.'

I have made sure that the secondary instance is configured correctly with the command (dba.checkInstanceConfiguration('instance')) and I'm getting status: ok message

 > dba.checkInstanceConfiguration('[email protected]:3306')
   Validating MySQL instance at 139.59.21.251:3306 for use in an InnoDB cluster...

   This instance reports its own address as 139.59.21.251:3306

   Checking whether existing tables comply with Group Replication requirements...
   No incompatible tables detected

   Checking instance configuration...
   Instance configuration is compatible with InnoDB cluster

   The instance '139.59.21.251:3306' is valid to be used in an InnoDB cluster.

   {
      "status": "ok"
   }

I'm kinda stuck at this point and would really appreciate any help on this issue. Thank you :)

Edit: I have also made changes in my.cnf file. For the primary node, I have added -

report_host=139.59.1.215
group_replication_ip_whitelist=139.59.1.215,139.59.21.251
skip-name-resolve

Answer 1

The issue is caused by having SELinux enabled, blocking the needed connections between XCom and GCS, part of a newly introduced feature in 8.0.14. Basically, SELinux needs to allow all connections to MySQL in loopback interfaces.

1. Allow all users to connect to MySQL. In Linux command line execute : setsebool -P mysql_connect_any 1
2. restart mysqld