0

ENV Details:

  1. GKE (Google Kubernetes cluster)
  2. Snitch Google cloud snitch. https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/architecture/archSnitchGoogle.html
  3. 140 nodes with 4 RACKs (4 statefulsets in 4 AZ's in one region)
  4. Each node size is 5TB and cluster size is 700TB.

Followed Procedure to restore DB: I restored a database size 450TB in K8s cassandra from source to target using nodetool refresh by copying sstables. Each node has around 5 to 6 TB data. The approach I have taken:

  1. Build a new cluster with same number of nodes like source 140 nodes.
  2. Created a new disks from source disk snapshots(cloud) and attached those disks to new nodes.
  3. Cleared system tables.
  4. Started C* as a Target cluster.
  5. Created tables manually which are matching with source cluster
  6. Copied stables from old to newly created(UUID) tables. 7 Ran nodetool refresh

Observation: Nodetool status showing same size as source for each node, cfstats also matching the tables size on both target and source.

Post migration issue:

  1. To reduce cluster size I started decommissioning one of the node and it completed in 10 minutes. Expecting to see atleast 8 hours based on previous source cluster maintenance.

  2. Since decommission was not expected to run only for 10 minutes I tried adding a new node to cluster which suppose to copy data approximately 5-6 TB but it only copied 2 TB.

  3. As an another check I just tried "Nodetool cleanup" on another node which is part of the cluster in target and it reduced data from 6 TB to 2 TB. Same cleanup dint reduce any data on source node.

  4. After restore whole cluster is repaired on target using reaper tool and no issues with repair but decommission, cleanup and bootstrap are the problem.

Improve this question

1 Answer 1

Reset to default
0

I suspect that the underlying issue is that your procedure is incorrect. For the "refresh method" to work, the source node(s) and corresponding target node(s) must have identical configuration down to the token(s) assigned to them as I've explained in How do I restore Cassandra snapshots to another cluster with identical configuration?.

When you copy SSTables to a node that has a different token ownership to the source node, the partitions in those SSTables which don't belong to the target node get thrown away when you run nodetool cleanup.

Since the nodes are deployed in a Kubernetes cluster, I assume that you don't have control over the token assignments which means that the pods would have been assigned random tokens.

In a situation where the configuration is not identical between the source and target clusters, you cannot use the refresh method. The correct procedure is to load the data using sstableloader.

If you're interested, I have documented the detailed steps in How do I migrate data in tables to a new Cassandra cluster?. Cheers!

Improve this answer
5
  • Thanks for your quick response. What if I also copy and start target nodes with INITIAL_TOKENS copied from source and the restore schema and run nodetool refresh? Looking at the link you posted seems like have the same process. The only difference would be instead of taking snapshots I will be mounting disk snapshots where the data files will already be there but I will copy them to new UUID directory and run nodetool refresh.
    – Sai
    Aug 31 at 4:20
  • Sure, but how do you plan to set the initial_token for the pods in a Kubernetes cluster? 🙂
    – Erick Ramirez
    Aug 31 at 4:38
  • Plan:1. run pod in a loop without starting cassandra on target 2. Delete all data inclusing system tables. 3. Run a configmap to update initial_token on all nodes by copying from source 4. Start nodes 5. Create schema 6. Copy data files to new UUID and 7. Nodetoolrefresh
    – Sai
    Aug 31 at 4:53
  • in terms of configuration, everything is same except the cluster_name which we want to keep separate from source and target because they both are in same GKE cluster and in future if in any case they both interact each other then the cluster_name should avoid it. Hope this is fine to make it work? This we already tested as I mentioned in description except the tokens that we did not copy from source which is why the issue is happening while decommission or bootstrapping.
    – Sai
    Aug 31 at 13:33
  • Update: We have created a script that will read from an external file for tokens data for each pod and replace the value in cassandra.yaml for INITIAL_TOKENS which will restart the nodes with manually assigned tokes. With this the clone worked as expected. Able to read complete data. All 3 cases such as decommission, bootstrap and cleanup were working as expected. Thanks for your help. Appreciate it if you could check my previous questions and respond.
    – Sai
    Sep 5 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.