Can Cassandra nodes running in 2 different DCs on K8s behind NAT on private subnet form a cluster?

Question

Is there anyway Cassandra nodes running in two different data centers behind NAT, on private subnet can form a single cluster?

Use case:

1. For an on premise Geo-Diverse setup, we needs to run Cassandra spanning across two data centers in two regions
2. Each setup in these two data centers are running in two separate Kubernetes cluster and Cassandra nodes are also running in respective K8s cluster
3. All the K8s nodes are having public IP, but as you know the Cassandra nodes are running as pods and hence has a private IP
4. We wants to use Cassandra's native replication (compared to disk/volume replication or Back-restore options), so we want to form a single Cassandra cluster across all the nodes in these two setup
5. Couple of things we are exploring is - either expose each Cassandra pod on a public IP using metallb backed load balancer service or expose each Cassandra on the host IP using hostPort (Since we have anti affinity, only one Cassandra instance will run on each host)

Problem

Now, we are facing a problem when we configure listen/broad-cast/seed addresses...

1. Broad cast address is the address on which that Cassandra node can be reached - So this can be the public IP on which the Cassandra node is exposed. No issue here
2. Seed address - We have configured the public-address of two Cassandra nodes from Geo1 and Geo2 each in the seed list
3. "listen_address" must be a specific address configured on the network interface where Cassandra process can bind. So looks like the only choice for this is the respective pod's private IP. But there is also another constraint - if a node is seed-node, its "listen_address" must be same as the address mentioned in the seed list. We can not configure private ip in seed list because that is not reachable from other nodes

Is there anyway to solve this? Any other suggestion for deploying Cassandra in K8s cluster in geo-redundant setup?

EDIT

We tried three approaches for routable IP and all three are working in normal cases - hostNetwork, hostPort and assigning an metallb based LB-service-IP for each Cassandra pod. Though the hostPort/hostNetwork are not encouraged by K8s best practices, in our scenario, we are good with it

My question was specific about a point in datastax documentation which states "If the node is a seed node, listen_address must match an IP address in the seeds list. Otherwise, gossip communication fails because it doesn't know that it is a seed". This can not be satisfied on both hostPort and LB-service approaches as the pod-ip/listen-address is from CNI's private address space which gets proxied by hostPort/LB-IP sitting in front of it

In the screen shots below, you can see for both these approaches broad_cast address is different from pod-ip/listen_address which is not routable. Since its not routable, it can not be in the seed_list. We were wondering if the constraint mentioned in the documentation is specific to that scenario of new DC getting added or if that has a larger impact?

Cluster with LB Service IP

Cluster with hostPort

Answer 1

At the Cassandra layer (setting aside Kubernetes for a minute), there are three minimum criteria required for C* nodes to be able to form a C* cluster:

1. They must have the same C* cluster_name.
2. They must have a common seed(s) in the seeds list.
3. They must be able to gossip with each other (shared network segment).

In Kubernetes regardless of where they are deployed, the same rules apply -- same cluster name, have a common seed and network connectivity between the DCs.

In your case, this isn't a question of "how to configure cassandra.yaml" but whether there is network connectivity between those two data centres. More importantly, are the pod IPs routable outside their respective K8s clusters? If you cannot answer "yes" to that question then it won't work.

Have a look at this K8ssandra blog post by Alexander Dejanovski, specifically at the section where he talks about Routable Pod IPs. This should give you a good idea on where you need to focus your investigation. Cheers!