MySQL Group Replication - Offline Node with cluster Metadata 1.0.1

Question

I have a production MySQL Group Replication setup out of three Nodes. The cluster has Metadata Version 1.0.1. Today, one node went down an during rejoin phase, it says, it cannot connect to the cluster.

When trying to dba.rejoinInstance('');, the MySQL Shell says, it won't do it, because it will only make changes to a cluster with Metadata 2.0.0.

ERROR: Unable to start Group Replication for instance 'O18-SQL-NBG:3306'. Please check the MySQL server error log for more information. Cluster.rejoinInstance: Group Replication failed to start: MySQL Error 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log. (RuntimeError)

The logfile says:

[MY-011521] [Repl] Plugin group_replication reported: 'Member version is incompatible with the group.'

When trying to upgrade Metadata on the cluster to 2.0.0, it says, it won't do it either, because one Member of the cluster is not available.

Dba.upgradeMetadata: This operation requires all the cluster members to be ONLINE (RuntimeError)

When trying to rescan() the cluster, it says, it won't it, because my metadata is out of date and it decided to only do read-only operations on this cluster:

Cluster.rescan: Operation not allowed. No cluster change operations can be executed because the installed metadata version 1.0.1 is lower than the version required by Shell which is version 2.0.0. Upgrade the metadata to remove this restriction. See \? dba.upgradeMetadata for additional details. (RuntimeError)

I have several versions of MySQL Shell available (8.0.17, 8.0.19 and 8.0.20). My primary objective is - guess what - to get the faulty node back up into the cluster. What are my options?

Answer 1

First of all, you are mixing some concepts. You are using MySQL InnoDB cluster which uses MySQL Group Replication as the replication technology.

I see that you have upgraded MySQL Shell version's and most probably you also upgraded the version of the MySQL servers of your cluster. The error you see when running cluster.rejoinInstance() is not related to Metadata but rather related to the incompatibilities of different MySQL versions in the Cluster. More info here: https://dev.mysql.com/doc/refman/8.0/en/group-replication-online-upgrade-combining-versions.html

How did you upgrade your instances? There's a procedure to follow. I'd recommend you to read the documentation regarding rolling upgrades: https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-cluster-upgrade.html

Regarding your scenario, my advice is to:

1. Use an older Shell version to remove the instance you're trying to remove, using cluster.removeInstance(). The version must be 8.0.18 or lower.

2. Using the latest MySQL Shell (8.0.21), upgrade the Metadata schema using dba.upgradeMetadata().

You will then have an operational cluster with the latest metadata schema version and you can then proceed to do the rolling upgrade of your instances and add them to the cluster as intended.

Cheers, Miguel