Cassandra node fails to reconnect to the cluster on Kubernetes with Istio

Question

I am intending to create a multi-datacenter Cassandra cluster with two datacenters distributed between two Kubernetes clusters. The cluster interconnection is configured by the means of Istio multicluster multi-primary setup. To provide an intercluster service discovery the DNS capture is enabled on both Istio control plains. The cassandra nodes on a cluster are configured as Kuberntes StatefulSet.

The caveat is that StatefulSet unique domain names for each pod in the set are not detected and resolved by Istio DNS capture. I have bypassed this by inserting an init container before the actual container with Cassandra, which creates a regular Kubernetes service, leading to the exact pod in a StatefulSet and have declared this service name as a broadcast_address in Cassandra configuration.

This setup works almost perfectly: the cluster is created, connection between remote datacenters is established and the data gets distributed between clusters.

The problem appears if the pod with Cassandra node is deleted and recreated. In such case the node cannot reconnect to the cluster. In the log I can see that the node tries to connect to the IP addresses of the custom services, created by init container and fails. But if the custom service is deleted before restarting the custom node (getting new IP address), the node successfully reconnects the cluster.

My possible guesses about the cause of the problem are:

1. Cassandra problem: for some reason the node cannot identify itself in a proper way and reconnect to the cluster
2. Istio problem: the Envoy proxy on querying the existing service returns a modified response, which, for some reason cannot be properly identified by Cassandra

Answer 1

Based on the symptoms you described, I suspect that the new pod is unable to reach the seeds so it cannot join the cluster which indicates a problem with the network configuration of your Kubernetes cluster.

However, I will admit that without minimal sample configuration + replication steps, I am not able to know for certain that is the cause of the issue.

As a general suggestion, you should consider using a seed service (if you are not already using it) so that the IP list are dynamically generated whenever pods are deleted/added.

For what it's worth, the networking section of this K8ssandra.io tutorial might be of interest to you. For a bit of background, K8ssandra is our opinionated deployment of Cassandra on Kubernetes with all the admin tools already bundled in:

• Reaper for automated repairs
• Medusa for backups and restores
• Metrics Collector for monitoring with Prometheus + Grafana
• Traefik templates for k8s cluster ingress
• Stargate.io - a data gateway for connecting to Cassandra using REST API, GraphQL API, JSON/Doc API and gRPC.

K8ssandra and all its components are all open-source and so free to use. Cheers!