- GKE (Google Kubernetes cluster)
- Snitch Google cloud snitch. https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/architecture/archSnitchGoogle.html
- 140 nodes with 4 RACKs (4 statefulsets in 4 AZ's in one region)
- Each node size is 5TB and cluster size is 700TB.
Followed Procedure to restore DB: I restored a database size 450TB in K8s cassandra from source to target using nodetool refresh by copying sstables. Each node has around 5 to 6 TB data. The approach I have taken:
- Build a new cluster with same number of nodes like source 140 nodes.
- Created a new disks from source disk snapshots(cloud) and attached those disks to new nodes.
- Cleared system tables.
- Started C* as a Target cluster.
- Created tables manually which are matching with source cluster
- Copied stables from old to newly created(UUID) tables. 7 Ran nodetool refresh
Observation: Nodetool status showing same size as source for each node, cfstats also matching the tables size on both target and source.
Post migration issue:
To reduce cluster size I started decommissioning one of the node and it completed in 10 minutes. Expecting to see atleast 8 hours based on previous source cluster maintenance.
Since decommission was not expected to run only for 10 minutes I tried adding a new node to cluster which suppose to copy data approximately 5-6 TB but it only copied 2 TB.
As an another check I just tried "Nodetool cleanup" on another node which is part of the cluster in target and it reduced data from 6 TB to 2 TB. Same cleanup dint reduce any data on source node.
After restore whole cluster is repaired on target using reaper tool and no issues with repair but decommission, cleanup and bootstrap are the problem.