Repair on a 3-node Cassandra cluster returns error "Got negative replies from endpoints"

Question

I have a 3 node cluster that is running as side-car in K8. One of the node went down and there was data update while it was down.

I tried to run repair, nodetool repair sourcekeyspace priceconfig to repair the table, but get an error.

I tried to run repair on all 3 servers and all of them are giving error.

What can I do to fix the problem?

NODE 1

itemdisplaytypename | sizebucket | pricetier | dbdateupdated                   | ruleid                               
        Matte Photo |      GIANT |      BEST | 2022-08-14 09:26:24.822000+0000 | e4b2ec36-b031-4361-8399-a9bdc949c860 

NODE 3

itemdisplaytypename | sizebucket | pricetier | dbdateupdated | ruleid 
        Matte Photo |      GIANT |      BEST |          null |   null 

ERROR MESSAGE

-- > NODE 1
[2022-09-01 00:57:32,093] Repair command #3 failed with error Got negative replies from endpoints [xx.xx.3.84]
[2022-09-01 00:57:32,093] Repair command #3 finished with error
error: Repair job has failed with the error message: [2022-09-01 00:57:32,093] Repair command #3 failed with error Got negative replies from endpoints [xx.xx.3.84]

--> NODE 2
[2022-09-01 01:01:17,887] Repair command #2 failed with error Got negative replies from endpoints [xx.xx.3.84]
[2022-09-01 01:01:17,887] Repair command #2 finished with error
error: Repair job has failed with the error message: [2022-09-01 01:01:17,887] Repair command #2 failed with error Got negative replies from endpoints [xx.xx.3.84]

 --> NODE 3
[2022-09-01 01:02:53,575] Repair command #8 failed with error Got negative replies from endpoints [xx.xx.1.72, xx.xx.2.167]
[2022-09-01 01:02:53,575] Repair command #8 finished with error

Answer 1

This repair error:

Got negative replies from endpoints [...]

is quite serious. Not only did the repair command not receive an error from one of the replicas (endpoints) -- it didn't get a response at all which indicates that the node is either down or unresponsive.

The most common cause (but not always) of this repair failure is a schema disagreement, i.e. the schema do not match across all nodes. This can happen if someone made a schema change while the node was down.

Ultimately, you will need to review the logs on the problematic endpoint(s) to determine the cause of the problem. Cheers!