1

We currently have 8 nodes in the old DC and will have 8 nodes in the new DC. All nodes are running Cassandra 3.11, but will be upgraded to 4 after the DC switch. RF is 3 everywhere but should be increased to 5. When we start up the nodes in the new DC, nothing will be replicated for now, we will change that at another time.

I am currently planning to decommission an old DC and have read this very good article to get a rough overview of the tasks involved.

There is a reference to another article about equal token distribution in new clusters.

Since our current DC (which is to be dismantled) is actually not optimally balanced, I would like to avoid this in the new DC.

The second article describes how to build a completely new cluster and I was wondering if this could be applied to my data centre switch?

So, in the new DC, can I bootstrap the three seed nodes beforehand using initial_tokens, create a reference keyspace with RF of my choice and then bootstrap the other nodes using allocate_tokens_for_keyspace to get a well-balanced new DC, or do I need to consider something different in my case as it is not a new cluster I am bootstrapping but just another DC?

1 Answer 1

2

Because you don't plan to utilize any of the keyspace data in the existing DC in the early life of the new DC, I'd probably just bootstrap the initial seed node to the DC, then once online, create a keyspace with a Replication Factor of 5 if that's what you plan to use, then bootstrap the remaining nodes specifying your newly created keyspace with allocate_tokens_for_keyspace. You can always use 'nodetool rebuild' to pull data to the new DC later.

Something to keep in mind though, is that if you use a Replication Factor of 5, then you'll want to maintain that for all user created keyspaces to avoid uneven distribution. Seeing that 3 is far more common than 5 for many reasons, I'm curious why you plan to use 5? And how many user-created keyspaces will the new DC ultimately end up with? Also, how many nodes will the new DC have and approximately how much data do you anticipate per node?

4
  • Thanks for your answer! I would when I turn on the replication of all user keyspace, of course set them directly with RF5 to the new one. ALTER KEYSPACE my_ks WITH replication = { 'class': 'NetworkTopologyStrategy', '<old_dc_name>': '3', '<new_dc_name>': '5' With RF3 and 8 nodes and reads and writes with quorum we can only lose one node. With RF5 we can lose 2 nodes. So if we have to do maintenance on one node and shut it down, no other node can fail without production impact. I see this as very problematic. Currently we have 8 Nodes with around 60GB of data. Commented Nov 4, 2022 at 15:05
  • We will also have 8 nodes in the new dc but maybe extend this DC with more nodes in the future. We have currently 4 keyspaces and all of them runs with RF 3. But as i already said, we want to reduce the risk to go down in production with RF5 in the new DC. Commented Nov 4, 2022 at 15:19
  • And regarding bootstrapping the initial seed node: If I bootstrap this node without the initial_token setting, won't this unbalance of tokens for the new DC become a problem again since I don't perform predictive token allocation on the initial node? Commented Nov 4, 2022 at 15:24
  • If consistency matters then reading/writing at LOCAL_QUORUM is good, and yes, you can have two replicas down with an RF of 5 as opposed to one, but is it worth the extra cost, administration, and overhead of streaming to two additional replicas to gain that benefit? Just want to be sure that it is. If the cluster has a tendency to be overloaded, then you'll likely need additional nodes to keep up with demand. If that all makes good sense then great, but wanted to double check.
    – Paul
    Commented Nov 4, 2022 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.