How do I resolve failed hints replay?

Question

I'm getting below error.

HintedHandOffManager.java:486 - Failed replaying hints to /172.16.54.16; \
  aborting (0 delivered), error : Operation timed out - received only 0 responses.

Can someone suggest me how to resolve the above error.

Answer 1

The error message you posted indicates that the node ran into issues while trying to replay hints to a replica (node 172.16.54.16).

Hints are mutations (INSERTs, UPDATEs, DELETEs) which are stored by a coordinator because a replica did not acknowledge the write. The replica could have been down or unresponsive for whatever reason. When the replica comes back online, coordinators which have hints stored will start replaying the mutations to the replica in case it missed the writes. For more info, see Hinted Handoff: repair during write path.

In your case, the hint handoff (replay) timed out because the replica was unresponsive/down/unavailable (for whatever reason).

You need to address the underlying issue for why the replica is unresponsive. You need to review the logs on that node to determine the cause.

The main problems you need to rule out are (a) hardware failures, (b) operating system issues, and (c) network interruptions. Once you've ruled these out, the most common cause of nodes becoming unresponsive is when they are overloaded.

If your cluster is regularly getting overloaded, you should consider increasing the capacity by adding more nodes. Cheers!