2

I have a Cassandra cluster consisted of 21 nodes (each node has 4 TB Cassandra data volume) and I need to replace the nodes (migration from Ubuntu 18.04 to 22.04)

I wonder if it's possible to re-attach data volume from existing instance to a new one without negative consequences? I was thinking about following plan:

  1. Drain (nodetool drain) the old Cassandra server and turn it off;
  2. Detach data volume and attach it to new server;
  3. Launch the new server;
  4. Repeat the following steps for the rest of the Cassandra nodes.

Are there any risks in doing that, the only thing that will be changed is the private IP address of the node.

3 Answers 3

2

The only problem I see with doing this, is that the system keyspace tables will know that they are a part of the old node and that could cause issues. Therefore, you may want to rm -rf the data/data/system directory after you stop the original Cassandra node.

As you're moving the data physically, you won't need to worry about streaming it to the remaining nodes in the cluster. Then, move the drive to the new node and start, while setting the JVM flag (in the cassandra-env.sh file) -Dcassandra.replace_address_first_boot=<dead_node_ip> to the new IP address. That will ensure that the "new" node (with the same drive) will get the same token ranges to match up with the data on the drive.

As the new node will already have data, you shouldn't need to stream it. So I'd also set auto_bootstrap: false in the cassandra.yaml file.

I'd recommend trying this out beforehand in a dev/stage environment, just to make sure that the approach works. You should also have a look through the official docs on replacing a node, as that will probably help.

2
  • If you delete system.local, C* won't know which token(s) the node owns so it will allocate new token(s) on startup. Thoughts? 🙂 Commented Feb 8 at 2:21
  • @ErickRamirez correct, but it will get its token assignments from Gossip, and rebuild system.local with that info.
    – Aaron
    Commented Feb 8 at 13:01
1

I agree with Aaron's response, but want to add a note on the fact that Cassandra doesn't allow you to replace a node with auto_bootstrap: false.

The reason for this is that there's an assumption that "new" replacement nodes will bootstrap with no data, which is not the case here, and will only serve read requests once all data finishes streaming.

The workaround for this guardrail is to use, in addition to the JVM flag -Dcassandra.replace_address_first_boot=<dead_node_ip>, also the flag -Dcassandra.allow_unsafe_replace=true.

1
  • The "replace address" flag is unnecessary in this scenario since the node is not getting replaced. For all intents and purposes, C* considers it to be the same node since the data directory is intact and its details are in the system.local table. The only difference is that the node will have a new IP. Cheers! Commented Feb 8 at 2:23
1

Yes, the procedure you outlined is standard practice for replacing hardware of existing Cassandra nodes.

Assuming all other Cassandra configuration properties are equal, Cassandra should start as usual with the only difference being that you'll see a log entry to say that Cassandra has detected a new IP address and that the new IP will be mapped to the node's host ID in the system.local table.

The only thing I would add to your procedure is to either:

  • set cluster_name: REPLACED in cassandra.yaml, or
  • completely uninstall Cassandra from the server.

These precautions should prevent the old server from rejoining the cluster if someone accidentally reboots/restarts the server. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.