Is there anyway Cassandra nodes running in two different data centers behind NAT, on private subnet can form a single cluster?
- For an on premise Geo-Diverse setup, we needs to run Cassandra spanning across two data centers in two regions
- Each setup in these two data centers are running in two separate Kubernetes cluster and Cassandra nodes are also running in respective K8s cluster
- All the K8s nodes are having public IP, but as you know the Cassandra nodes are running as pods and hence has a private IP
- We wants to use Cassandra's native replication (compared to disk/volume replication or Back-restore options), so we want to form a single Cassandra cluster across all the nodes in these two setup
- Couple of things we are exploring is - either expose each Cassandra pod on a public IP using metallb backed load balancer service or expose each Cassandra on the host IP using hostPort (Since we have anti affinity, only one Cassandra instance will run on each host)
Now, we are facing a problem when we configure listen/broad-cast/seed addresses...
- Broad cast address is the address on which that Cassandra node can be reached - So this can be the public IP on which the Cassandra node is exposed. No issue here
- Seed address - We have configured the public-address of two Cassandra nodes from Geo1 and Geo2 each in the seed list
- "listen_address" must be a specific address configured on the network interface where Cassandra process can bind. So looks like the only choice for this is the respective pod's private IP. But there is also another constraint - if a node is seed-node, its "listen_address" must be same as the address mentioned in the seed list. We can not configure private ip in seed list because that is not reachable from other nodes
Is there anyway to solve this? Any other suggestion for deploying Cassandra in K8s cluster in geo-redundant setup?
We tried three approaches for routable IP and all three are working in normal cases - hostNetwork, hostPort and assigning an metallb based LB-service-IP for each Cassandra pod. Though the hostPort/hostNetwork are not encouraged by K8s best practices, in our scenario, we are good with it
My question was specific about a point in datastax documentation which states "If the node is a seed node, listen_address must match an IP address in the seeds list. Otherwise, gossip communication fails because it doesn't know that it is a seed". This can not be satisfied on both hostPort and LB-service approaches as the pod-ip/listen-address is from CNI's private address space which gets proxied by hostPort/LB-IP sitting in front of it
In the screen shots below, you can see for both these approaches broad_cast address is different from pod-ip/listen_address which is not routable. Since its not routable, it can not be in the seed_list. We were wondering if the constraint mentioned in the documentation is specific to that scenario of new DC getting added or if that has a larger impact?