Cassandra 4.0.6 repair returned "/IP failed the prepare phase for incremental repair session"

Question

We recentally migrated to Cassandra 4.0.6 from 3.11.6 on Dev environment. During the testing of this version of cassandra, I tried running incremental repair on a table in keyspace and it failed. In system logs,these log messeges are present.

WARN  [AntiEntropyStage:1] 2022-09-19 21:53:19,166 CoordinatorSession.java:172 - /IP failed the prepare phase for incremental repair session 5f3c0880-3837-11ed-bf8e-2d64975764cb
INFO  [CompactionExecutor:5931] 2022-09-19 21:53:19,168 CompactionManager.java:797 - [repair #5f3c0880-3837-11ed-bf8e-2d64975764cb] Completed anticompaction successfully
INFO  [AntiEntropyStage:1] 2022-09-19 21:53:19,169 CoordinatorSession.java:266 - Incremental repair session 5f3c0880-3837-11ed-bf8e-2d64975764cb failed
INFO  [AntiEntropyStage:1] 2022-09-19 21:53:19,169 CoordinatorSession.java:266 - Incremental repair session 5f3c0880-3837-11ed-bf8e-2d64975764cb failed
INFO  [CompactionExecutor:5931] 2022-09-19 21:53:19,169 LocalSessions.java:830 - Prepare phase for incremental repair session 5f3c0880-3837-11ed-bf8e-2d64975764cb completed
ERROR [AntiEntropyStage:1] 2022-09-19 21:53:19,169 RepairRunnable.java:178 - Repair 5f3c0880-3837-11ed-bf8e-2d64975764cb failed:
java.lang.RuntimeException: Incremental repair session 5f3c0880-3837-11ed-bf8e-2d64975764cb has failed
        at org.apache.cassandra.repair.consistent.CoordinatorSession.fail(CoordinatorSession.java:272)
        at org.apache.cassandra.repair.consistent.CoordinatorSessions.handleFailSessionMessage(CoordinatorSessions.java:97)
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:189)
        at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
        at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

Incremental repair work fine on other keyspaces. Please help in debugging the issue.

Answer 1

As this log entry shows:

WARN  [AntiEntropyStage:1] 2022-09-19 21:53:19,166 CoordinatorSession.java:172 - \
    /IP failed the prepare phase for incremental repair session 5f3c0880-3837-11ed-bf8e-2d64975764cb

the prepare phase failed on one of the nodes causing the repair session to fail.

You will need to review the logs on the reported node to determine why the repair failed. Depending on the cause, you might need to do further investigation and eventually take appropriate remediation action. Cheers!