How do I migrate data in tables to a new Cassandra cluster?

Question

How do I migrate application tables in an existing Cassandra cluster to a new cluster?

The new cluster has a different topology and token assignments to the source cluster.

Answer 1

Preparation

On the source cluster, take a snapshot of the relevant keyspaces using the nodetool snapshot command. For example:

$ nodetool snapshot <keyspace_name>

Here is an example where I take a snapshot of the community keyspace.

STEP B1 - Create a snapshot:

$ nodetool snapshot community
Requested creating snapshot(s) for [community] with snapshot name [1591083719993] and options {skipFlush=false}
Snapshot directory: 1591083719993

The directory 1591083719993 name is a Unix timestamp for when the snapshot was created and is equivalent to June 2, 2020 7:41am GMT. There is one table called users in my example keyspace and the snapshot is located in the following directory structure:

data/
  community/
    users-6140f420a4a411ea9212efde68e7dd4b/
      snapshots/
        1591083719993/
          manifest.json
          mc-1-big-CompressionInfo.db
          mc-1-big-Data.db
          mc-1-big-Digest.crc32
          mc-1-big-Filter.db
          mc-1-big-Index.db
          mc-1-big-Statistics.db
          mc-1-big-Summary.db
          mc-1-big-TOC.txt
          schema.cql

For more info, see Taking a snapshot in Cassandra.

Tooling

Taking a snapshot needs to be carried out on all nodes in the cluster. It is preferable if you create them in parallel to make it simpler for you to identify the snapshot folders.

To achieve this, I recommend using tools you already have in your environment. If you are already using orchestration tools like Ansible, create the snapshots in parallel by running the command on all nodes simultaneously. Similarly, you can also script the restore operation so you can execute it in parallel using Ansible.

If you are not using orchestration tools, consider using Cluster SSH (cssh) or Parallel SSH (pssh) so you can run commands simultaneously on all nodes in your cluster.

Cloning a table

PREPARATION - Create the application keyspace and table schema on the destination cluster. If necessary, use the schema.cql file in the snapshots folder as a guide.

Once the keyspace and table schema has been created, follow the procedure below to restore the tables.

STEP 1 - Copy the snapshot to a temporary location so that the SSTable files are located in a directory with keyspace_name/table_name. For example:

$ cp -p \
    data/community/users-6140f420a4a411ea9212efde68e7dd4b/snapshots/1591083719993/* \
    /path/to/community/users/.

STEP 2 - Load the data files to the destination cluster with the utility as follows:

$ sstableloader -d dest_node_ip1, dest_node_ip2 /path/to/community/users/

STEP 3 - Repeat steps 1 & 2 on the next node in the source cluster until the snapshots on ALL nodes have been loaded to the destination cluster.

Repeat the steps above on each table that you want to clone to the destination cluster.

For more info, see Cassandra bulk loader. Cheers!

---

For instructions on cloning application data to an "identical" cluster, see How do I restore Cassandra snapshots to another cluster with identical configuration?.