2

We have Cassandra cluster of 3 nodes in the same rack. Now as part of regular maintenance activities on the server, I want to stop and start once instance of the Cassandra on one node at a time.

We are using replication factor of 3 and Consistency factor of 1.

Server will be down for more than 4 hours. So when I will make the node up and respective Cassandra instance start, do I need to manually run the repair Or Cassandra will take care itself to replicate the data updated, created, deleted during these 4 hours. If we have to manually run the repair on Cassandra, can I run it on stopped Cassandra instance otherwise it will start accepting the traffic and return old information to the clients.

If I have to manually run the repair, please suggest the steps so that when instance is started it has up to data.

1 Answer 1

2

Several points to address here.

Server will be down for more than 4 hours.

Important to note, but by default each node can store hints for up to 3 hours.

Or Cassandra will take care itself to replicate the data updated, created, deleted during these 4 hours.

Maybe if you could limit the outage window to less than 3 hours. I suppose the other alternative here, would be to increase the max_hint_window_in_ms from 3 hours to 4 hours (on each node). Then you'd probably be ok.

Otherwise, the answer here is that if a node is down for longer than 3 hours, you will miss data because the hints will have stopped collecting at that point. And then yes, you will need to run repair once the node is back.

If we have to manually run the repair on Cassandra, can I run it on stopped Cassandra instance otherwise it will start accepting the traffic and return old information to the clients.

No, the Cassandra process needs to be running for data to be streamed to it.

Depending on how much data is on it, one thing you could try would be to wipe the node and join it brand new. The bootstrap process would stream the data, and not accept client requests until complete.

Now as part of regular maintenance activities on the server, I want to stop and start once instance of the Cassandra on one node at a time.

So will this be a regular practice? Just curious.

I highly recommend setting up a server running Cassandra Reaper. This will allow you to run scheduled repairs on all of your clusters, without having to mess around with cron or SSH.

Without Reaper, you'll need to SSH out to the node and run nodetool repair. If you have been running regular incremental repairs (default), that is all you should need to do. If you've been using full repairs, then pass the --full option.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.