1

I have cluster cassandra-2.2.19-1 with 4 nodes. I want to move one of the nodes to a new server. Is it possible to just rsync the cassandra directories, put out the old node and start a new one. Will this method work?

2 Answers 2

1

Yes, if you rsync the directories (data and config, mainly), stop the old node and start the new one, it shouldn't be able to tell the difference.

The real question is, can the new node have the same IP address as the old? If not, then you might be able to make that work by specifying a replacement address in the conf/cassandra-env.sh:

JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_old_node
1

Using rsync to populate the disk on the new node is going to be problematic because the data directories are not static and are constantly mutating with new SSTables from memtable flushes (new writes) and compaction tasks.

The recommended procedure for replacing nodes in a cluster is the "replace address" method. This operation "swaps" out a node and replaces it with a new server.

Here are the steps:

Preparation

Step A1 - Perform a rolling repair, one node at a time, by running the following on ALL nodes in ALL DCs:

$ nodetool repair -pr

Step A2 - Install Cassandra on the new server.

Step A3 - Configure Cassandra on the new server.

Replace method

Step B1 - Stop Cassandra on the existing node (node to be replaced).

Step B2 - Reconfigure Cassandra on the existing node to prevent it from rejoining the cluster by accident. For example, set the following properties in cassandra.yaml:

cluster_name: 'REPLACED'
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "127.0.0.1"

Step B3 - On the new server, add the following line to the end of conf/cassandra-env.sh:

JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=IP_of_existing_node

Step B4 - Confirm that auto_bootstrap: false is NOT set in cassandra.yaml because it will prevent the node from requesting data from other replicas in the cluster.

Step B5 - Start Cassandra on the new node.

Monitor the progress of the startup process with nodetool netstats and watching the system.log.

When the startup has completed successfully, the Cassandra process will be listening for CQL client connections on port 9042 (default) and you will see a log entry similar to:

... Starting listening for CQL clients on ...

Cleanup

Step C1 - When Cassandra has started successfully on the new node, remove the replace_address flag from conf/cassandra-env.sh added in step B3 above. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.