Cassandra 4.0.5 nodes fails to start when replacing dead node

Question

Trying to replace a down node the new nodes fail to start, using property replace_address.

WARN  [Messaging-EventLoop-3-2] 2023-08-16 14:18:58,565 NoSpamLogger.java:95 - /xxx.xxx.xxx.xxx:7000->/yyy.yyy.yyy.yyy:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
INFO  [Messaging-EventLoop-3-2] 2023-08-16 14:19:23,910 NoSpamLogger.java:92 - /xxx.xxx.xxx.xxx->/yyy.yyy.yyy.yyy:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.ConnectTimeoutException: connection timed out: /xxx.xxx.xxx.xxx:7000
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:576)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

xxx.xxx.xxx.xxx - IP of down node
yyy.yyy.yyy.yyy - IP of new node

NO other ERROR/WARNING appears in logs. The node goes into UJ state, but never joins the ring.

While this doesn't always happen , we are seeing this increased behaviour since upgrading from 3.11.9 to 4.0.5.

Configurations are all fine, as to mitigate this we terminate the node and spawn a new one with same configs.

Answer 1

I've seen that a few times in the past, it typically points to trouble with internode connectivity over port 7000, if the port is open then include more of the log if you can, often we'll see other errors that point to some other problem. For example, I've seen this if the seeds list was incorrect, so check the cassandra.yaml and ensure the seeds list doesn't include the node being replaced nor the new node replacing it anywhere. If that doesn't help, see what other issues are in the log around this warning.