2

Due to disk underuse, I am considering reducing the size of a Cassandra cluster (v 3.11.13) from 12 to 9 nodes (from 4 to 3 nodes per rack). The cluster is therefore spread across 3 racks and has a single "big" keyspace with RF=3.
I plan to

  1. scale up the nodes to handle the future new workloads, assuming that the load will still be properly balanced after the downsizing
  2. run nodetool decommission on one node on every single rack
  3. Rebalance the data? Here I'm a bit confused, as I'm using Vnodes, I thought I don't need to handle data rebalancing, however, in the doc of decommission, it states that the data is passed to the next node in the ring, and not shared by all the remaining nodes, I'd like a clarification on that.
  4. I read that a nodetool cleanup is required afterwards, is this really useful?
  5. Delete the nodes (Kubernetes pods in my case)

Thanks for your help

1 Answer 1

1

it states that the data is passed to the next node in the ring

Yeah, I don't think that's correct (anymore?). That statement is probably from the days when a Cassandra node was responsible for only one token range.

What happens on a decommission, is that the cluster token ranges are recalculated based on the new (smaller) size of the cluster. nodetool decommission then streams the data to the nodes now responsible for its token ranges.

I read that a nodetool cleanup is required afterwards, is this really useful?

So nodetool cleanup essentially checks the node to ensure that it only has data which matches its token ranges. If it finds data it's not responsible for, it removes it. I'm not sure that's terribly necessary after a decommission. In this case, running it shouldn't hurt anything.

Basically, during the nodetool decommission process, run a nodetool netstats (on the decommissioning node) occasionally. You'll see the data streams heading to their new nodes. The new nodes are the ones to keep an eye on for any weirdness or sudden increase in disk usage.

Do you know if it is possible to rollback / cancel a decommission if something goes wrong on the other nodes or on the clients?

Yes. The decommission process is designed to work that way. You can just stop the decommissioning node, restart it, and everything should continue as normal.

I've seen a strange behaviour since the decommission, regularly some cassandra nodes start massively outputting those logs OutboundTcpConnection.java:570 - Cannot handshake version with /10.208.58.4 the issue is that that ip matches a non cassandra pod, it's like that ip is a leftover of an old pod, how can I solve it?

Yep, this issue isn't uncommon in a Kubernetes paradigm. The cluster will "remember" the IPs of old nodes for up to 72 hours after they've been removed or decommissioned...and sometimes just won't "forget" about them for longer than that. You can get rid of it by running:

nodetool assassinate 10.208.58.4
9
  • Additional question, is it worth doing a repair before and/or after the process? Knowing that I never make a repair on this cluster which is active
    – Fos
    Sep 16, 2022 at 9:03
  • 1
    @Fos A repair would help to make sure all the replicas are current. It'd be a good idea to run afterward, just to make sure all replicas were streamed where they need to be.
    – Aaron
    Sep 16, 2022 at 13:22
  • 1
    Thank you Aaron for your precious help!
    – Fos
    Sep 19, 2022 at 14:47
  • Do you know if it is possible to rollback / cancel a decommission if something goes wrong on the other nodes or on the clients? I can't find resources on that.
    – Fos
    Sep 20, 2022 at 9:44
  • 1
    @Fos edit made! Basically, run a nodetool assassinate 10.208.58.4
    – Aaron
    Sep 23, 2022 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.