0

Trying to replace a down node the new nodes fail to start, using property replace_address.

WARN  [Messaging-EventLoop-3-2] 2023-08-16 14:18:58,565 NoSpamLogger.java:95 - /xxx.xxx.xxx.xxx:7000->/yyy.yyy.yyy.yyy:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
INFO  [Messaging-EventLoop-3-2] 2023-08-16 14:19:23,910 NoSpamLogger.java:92 - /xxx.xxx.xxx.xxx->/yyy.yyy.yyy.yyy:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.ConnectTimeoutException: connection timed out: /xxx.xxx.xxx.xxx:7000
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:576)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

xxx.xxx.xxx.xxx - IP of down node
yyy.yyy.yyy.yyy - IP of new node

NO other ERROR/WARNING appears in logs. The node goes into UJ state, but never joins the ring.

While this doesn't always happen , we are seeing this increased behaviour since upgrading from 3.11.9 to 4.0.5.

Configurations are all fine, as to mitigate this we terminate the node and spawn a new one with same configs.

2
  • How are you deploying your nodes? It looks like the new node cannot reach the cluster on port 7000. Is it possible that sometimes security configs for the new node don't propagate right away? That's the only reason I can think of there a terminate/redeploy of a replacement node would fix anything.
    – Aaron
    Aug 19, 2023 at 14:44
  • We are on GCP cloud and and using K8s for managing infra, so chances are minimal of having security issues as the same settings are working for other nodes. It might be we are hitting some edge case scenario as in the bug CASSANDRA-16877 (this was fixed in 4.0.1 version) and we are at 4.0.5.
    – anshul
    Aug 29, 2023 at 12:02

1 Answer 1

1

I've seen that a few times in the past, it typically points to trouble with internode connectivity over port 7000, if the port is open then include more of the log if you can, often we'll see other errors that point to some other problem. For example, I've seen this if the seeds list was incorrect, so check the cassandra.yaml and ensure the seeds list doesn't include the node being replaced nor the new node replacing it anywhere. If that doesn't help, see what other issues are in the log around this warning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.