Why are connections to Cassandra nodes timing-out during low traffic periods?

Question

Why do Cassandra nodes report timeouts in the logs during low traffic periods?

The system.log on nodes report ConnectTimeoutException between nodes or "socket closed" errors.

Answer 1

Background

This article discusses an issue where a firewall configured with an idle connection timeout can close connections to local nodes and nodes in other data centres during low traffic periods.

Cause

The TCP keepalive may not be appropriately configured for your environment. The default idle connection timeout is usually set to 2 hours (7200 seconds) which means keepalive probes are not sent often enough for the firewall's idle connection timeout.

Solution

To prevent connections between nodes from timing out, set the following network kernel settings:

$ sudo sysctl -w \
  net.ipv4.tcp_keepalive_time=60 \
  net.ipv4.tcp_keepalive_probes=3 \
  net.ipv4.tcp_keepalive_intvl=10

These settings will keep the connection alive by sending 3 probes every 60 seconds with a 10-second gap between each probe. Cheers!