3

We have a 3 node Cassandra cluster with RF = 2. The read and write consistency is set to ONE. We are using Vnodes as well. Lets label these nodes as N1, N2 and N3. Let's say that N3 goes down. I was under the impression that whenever a node goes down, the other nodes would store hints and whenever N3 comes up, the hints would be sent to N3, thereby ensuring that the data is consistent across replicas. However, as I was going through the docs, I came across the parameter '' which defaults to 3 hours. So, if a node is dead for more than 3 hours, it is considered permanently dead and no hints are stored. So far, so good.

So, my understanding now is that if a node is down for say 10 hours, then the hints for the first 3 hours would be transferred to this node when it comes back up, but the writes for this 7 hour duration would be lost for this node. Moreover, if a read query is fired for a particular token range, and since this node is also elligible to serve the read requests for a token range, it would return null instead of the actual data that is stored in some other node. Is my understanding correct? What, then, should be done?

1

Your understanding is correct that beyond the default 3 hours, hints are no longer stored. This is to avoid overloading the other alive nodes for prolonged period of time.

If any node happens to be down for a longer period than the hint_window, its absolutely necessary to rebuild the node or even remove all the files from its data directory and let it bootstrap as a brand new node. You may have to use replace_address if you are bringing up the same node.

0

If you are using default hinted handoff period which is 3 hour and your node was down for more then that than no hints will be stored for that node. Suppose it came online after 10 hours as you mentioned then you must run full repair as soon as it come online however if it was down for more than 2-3 days, repairs would have lots of data to stream and can cause some surprise problems so consider doing rebuilding/bootstrap after removing all data/saved_caches and commit logs file from the down node. Having said that with consistency ONE it is highly possible to lose data, if you query the node before repair finish it may return stale or null as data was never written to it neither propagated via repairs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.