2

We started with the first approach listed below using sstableloader. We have a blocker now. Please find below the details of all the work we have done. Looking for suggestions on how to move the data. Thanks!

  1. Sstableloader : AWS JSON data dump is transformed to SSTables by a Spark app and written to the local disk of spark nodes. They are then copied over to the on-prem C* cluster using sstableloader tool. We have a working PoC.

    Blocker discovered and posted a SO question: data is being copied over 'replica' number of times from AWS. Assuming, we have 3 replicas, AWS egress would cost thrice of initially estimated number.

    Solution we're thinking of exploring: Copying the SSTables from AWS to one of the servers on-prem and then running sstableloader tool over there. Cons: too many moving parts if the solution is broken down & executed on 2 datacenters, reliability concerns.

  2. Nodetool : This is another tool similar to sstableloader using JMX internally to copy SSTables over to Cassandra. We just started exploring. Need to check if this copies 'replica' times or once.

  3. JMX Bulk Load SSTables: We have working code using StorageServiceMBean.bulkLoad()

    Con: JMX bulk loader expects the SSTables to be present on the C* nodes. This will probably bring down the utilization of C* node disks for actual data storage to 25%, because, we need to make sure to leave some disk for OS/system stuff, compaction, for swap space and SSTables to be loaded to the cluster by JMX Bulk loader

  4. Conventional data population using Spark-C* integrations: We have the code ready. Planning to check if copied once or 'replica' number of times. But found this to be slower than bulk-load, without any tuning.

    Con: We are anticipating that there will be a significant amount of overhead because of compaction, when TBs of data is written to C* in a short span of time

  5. Dsbulk tool : Planning to explore other options before getting here. PARTITION_KEY based batching might help. Unsure how better this would be when compared to (4).

0

2 Answers 2

2

How many sstables do you copy to Cassandra? and how many nodes in the cluster? Depending on the number, option 2 might be good, but you'd essentially copy every sstable to every node, then run 'nodetool refresh keyspace table' for all tables involved. If your number of nodes is greater than the number of replicas though, this can be a pain because you're putting all the data on all nodes, then you'd want to come through with a 'nodetool cleanup' after running the refresh.

4 and 5 can also work well, partition level batching is the default in DSBulk, but you definitely can't avoid compactions with that much data. What kind of resources do the nodes have in terms of disks and cpu cores? What compaction strategy is in use? Probably the best thing to do would be to tune for larger flush sizes in Cassandra and make sure compaction had enough resources to keep up without completely overloading the nodes, but these can be viable options.

1

I have done the following with a very small cluster N=3, RF=3. My thought was that each node (or rack) has 100% of the data in this arrangement. Therefore, I could copy the data from a single node or rack into a new cluster of N=3 along with the schema and use various nodetool repair and cleanup commands to re-distribute the data.

I did get this to actually work, but it was only an experiment a couple of years ago, and I'm suggesting it here to create discussion.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.