Best Practices for Managing Concurrent Client Connections in Cassandra

Question

Hello Cassandra Community,

We've been experiencing occasional spikes in the number of client connections to our Cassandra cluster, particularly during high-volume API request periods. We're using persistent connections, and we've noticed that the number of connections can increase significantly during these spikes.

We're considering using the following Cassandra parameters to manage concurrent client connections:

native_transport_max_concurrent_connections: This parameter sets the maximum number of concurrent client connections allowed by the native transport protocol. Currently, it's set to -1, indicating no limit.

native_transport_max_concurrent_connections_per_ip: This parameter sets the maximum number of concurrent client connections allowed per source IP address. Like the previous parameter, it's also set to -1.

We're thinking of using these parameters to limit the maximum number of connections from a single IP address, especially to prevent overwhelming the database during spikes in API requests that should be handled by our SOA team exclusively.

Are these parameters suitable for production use, and would implementing restrictions on concurrent connections per IP be considered a good practice in managing Cassandra clusters?

Any insights or recommendations would be greatly appreciated.

Thank you!

Naman Kaushik

Answer 1

Capping the maximum concurrent connections by setting native_transport_max_concurrent_connections and native_transport_max_concurrent_connections_per_ip might seem like it's a way to deal with the load spikes but in my experience, the effect is only temporary. What you'll find is that you're simply delaying the inevitable and will be back to where you started.

Nodes will stop accepting new client connections once the limit is reached. But then what happens is that the clients keep retrying, repeatedly trying to reconnect. The connection requests build up on the nodes until such point that the nodes are bombarded with reconnection requests.

To use an analogy, think of customers lining up to pay at the registers. When all the registers are full, you can tell customers they are no longer allowed to queue but they still have to pay so they'll continue to try queueing at one of the full registers. You may try to stop them from queueing but it doesn't solve the underlying issue which is all the registers are full.

The real solution is to open up more registers to deal with the high volume of customers. The same applies to your cluster -- you need to increase the capacity to deal with the spike in traffic by adding more nodes. Cheers!