Getting "SSLException: Connection reset" from remote nodes

Question

I see these errors in system logs for the passive DC nodes, where the nodes in the Error messages are from the other DC which is active. In active DC I don't see such errors.

Could this be connectivity issues ? Is this pointing to something serious issue with my cluster?

How can I debug my problem?

ERROR [MessagingService-Incoming-/123.12.13.1] 2024-02-19 10:07:57,692 CassandraDaemon.java:244 - Exception in thread Thread[MessagingService-Incoming-/123.12.13.1,5,main]
java.io.IOError: javax.net.ssl.SSLException: Connection reset
ERROR [MessagingService-Incoming-/123.12.13.2] 2024-02-19 10:07:57,692 CassandraDaemon.java:244 - Exception in thread Thread[MessagingService-Incoming-/123.12.13.2,5,main]
java.io.IOError: javax.net.ssl.SSLException: Connection reset

Answer 1

I am no expert, but related articles (https://stackoverflow.com/questions/5507878/ssl-connection-reset) seem to indicate that a good place to start troubleshooting would be to verify that everything is working properly with SSL/TLS. Try verifying that your SSL configurations are set to work at the proper protocol level and view this article under "TLS/SSL Encryption"

https://cassandra.apache.org/doc/stable/cassandra/operating/security.html

Answer 2

This is a serious error you shouldn't ignore.

The MessagingService handles communication between nodes (internode communication), both for dispatching messages (outbound) and routing inbound messages (from other nodes).

The SSLException for inbound messages means that the remote nodes don't have the correct credentials to communicate with the local nodes so you'll need to verify the server encryption options you've configure on all nodes in the cluster. Cheers!