2

I have a 3 node cluster that is running as side-car in K8. One of the node went down and there was data update while it was down.

I tried to run repair, nodetool repair sourcekeyspace priceconfig to repair the table, but get an error.

I tried to run repair on all 3 servers and all of them are giving error.

What can I do to fix the problem?

NODE 1

itemdisplaytypename | sizebucket | pricetier | dbdateupdated                   | ruleid                               
        Matte Photo |      GIANT |      BEST | 2022-08-14 09:26:24.822000+0000 | e4b2ec36-b031-4361-8399-a9bdc949c860 

NODE 3

itemdisplaytypename | sizebucket | pricetier | dbdateupdated | ruleid 
        Matte Photo |      GIANT |      BEST |          null |   null 

ERROR MESSAGE

-- > NODE 1
[2022-09-01 00:57:32,093] Repair command #3 failed with error Got negative replies from endpoints [xx.xx.3.84]
[2022-09-01 00:57:32,093] Repair command #3 finished with error
error: Repair job has failed with the error message: [2022-09-01 00:57:32,093] Repair command #3 failed with error Got negative replies from endpoints [xx.xx.3.84]

--> NODE 2
[2022-09-01 01:01:17,887] Repair command #2 failed with error Got negative replies from endpoints [xx.xx.3.84]
[2022-09-01 01:01:17,887] Repair command #2 finished with error
error: Repair job has failed with the error message: [2022-09-01 01:01:17,887] Repair command #2 failed with error Got negative replies from endpoints [xx.xx.3.84]

 --> NODE 3
[2022-09-01 01:02:53,575] Repair command #8 failed with error Got negative replies from endpoints [xx.xx.1.72, xx.xx.2.167]
[2022-09-01 01:02:53,575] Repair command #8 finished with error

1 Answer 1

2

This repair error:

Got negative replies from endpoints [...]

is quite serious. Not only did the repair command not receive an error from one of the replicas (endpoints) -- it didn't get a response at all which indicates that the node is either down or unresponsive.

The most common cause (but not always) of this repair failure is a schema disagreement, i.e. the schema do not match across all nodes. This can happen if someone made a schema change while the node was down.

Ultimately, you will need to review the logs on the problematic endpoint(s) to determine the cause of the problem. Cheers!

2
  • Hi Erick, You are correct that there was a node down when the schema was change. However, the schema is the same now across the 3 nodes, but the data is not. All the 3 nodes are up at the moment. That is when I tried to repair with data inconsistency with "nodetool repair", but is failing when I tried to run on it on each node. The challenge is how do I rectify the inconsistent data? Regards, Tommy
    – tommy
    Sep 2, 2022 at 15:41
  • Like I said, you need to review the logs on the problematic nodes to figure out what's going on. Cheers! Sep 2, 2022 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.