0

I first asked this question on Stack Overflow but I woke up this morning to the realization that it belongs over here.

I'm setting up an InnoDB Cluster using mysqlsh. This is in Kubernetes, but I think this question applies more generally.

When I use cluster.configureInstance() I see messages that includes:

This instance reports its own address as node-2:3306

However, the nodes can only find each other through DNS at an address like node-2.cluster:3306. The problem comes when adding instances to the cluster; they try to find the other nodes without the qualified name. Errors are of the form:

[GCS] Error on opening a connection to peer node node-0:33061 when joining a group. My local port is: 33061.

It is using node-n:33061 rather than node-n.cluster:33061.

If it matters, the "DNS" is set up as a headless service in Kubernetes that provides consistent addresses as pods come and go.It's very simple, and I named it "cluster" to created addresses of the form node-n.cluster. The cluster itself is a statefulset that numbers nodes sequentially. I don't want to cloud this question with detail I don't think matters, however, as surely other configurations require the instances in the cluster to use DNS as well.

I thought that setting localAddress when creating the cluster and adding the nodes would solve the problem. Indeed, after I added that to the createCluster options, I can look in the database and see

| group_replication_local_address                     | node-0.cluster:33061                 |

Yet when I look at the topology, it seems that the localAddress option has no effect whatsoever (at this point I don't see what it does at all):

{
    "clusterName": "mycluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "node-0:3306", 
        "ssl": "REQUIRED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures.", 
        "topology": {
            "node-0:3306": {
                "address": "node-0:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": null, 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.29"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "node-0:3306"
}

And adding more instances continues to fail with the same communication errors.

Notable perhaps that I cannot set localAddress to node-0.cluster:3306; I get a message that that port is already in use. I tried 3307 but as before it had no effect.

How do I convince each instance that the address it needs to advertise is different? I will try other permutations of the localAddress setting, but it doesn't look like it's intended to fix the problem I'm having. How do I reconcile the address the instance reports for itself with the address that's actually useful for other instances to find it?

Edit to add: Maybe it is a Kubernetes thing? Kubernetes sets the hostname of the pod to match the name of the pod. If so, how do I override it?

There has to be a way to tell the cluster to use something other than the machines' hostnames for discovering each other.

1 Answer 1

0

Apparently this value has to be set at startup. The option for my setup was

--report-host=${HOSTNAME}.cluster

when starting the MySQL instances resolved the issue.

Specifically for Kubernetes, an example is at https://github.com/adamelliotfields/kubernetes/blob/master/mysql/mysql.yaml

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.