1

Background and Observations:

I have a 3 node cluster that is running in production (version 3.11). For the migration I first tried creating 1 new node with cassandra 4.1.2 version and added these nodes to the existing cluster. To add these node I had to run -Dcassandra.skip_schema_check=true command else the new node was failing on startup.

Once I was able to add these nodes in the existing cluster I observed that the data is not streaming to the new node from version 3 to version 4 node.

The Idea here was that once the data is replicated to version 4 nodes we can shut down the version 3 nodes one by one and that would be a smooth migration.

The document for migration from 3.x to 4.x here suggests that the migration should be done node by node and I should first take down one node and then setup cassandra 4 on that node and then use the same data files to run cassandra 4 and then do the same for all nodes.

Question:

My question is that if I go ahead with the approach mentioned in the datastax document there will be a time when version 4 and version 3 will coexist while my read/write load will still be coming to the cluster. During this time the data transfer between 2 different versioned nodes won't be happening, so the write request going to version 4 node will not be replicated to version 3 node and vice versa.

This will cause the data inconsistency until all nodes are migrated and they start communicating with each other. We assume that the data will be consistent eventually here but if there is any issue that occurs during this process we won't be having any way to resolve the inconsistent data in production.

So what steps I can follow to make sure that there is no data inconsistency and if something goes wrong there is some easier way to correct the data.

And also I wanted to understand why the 1st approach mentioned above was not supported by cassandra as it seems a easier approach for a developer from migration stand point.

Improve this question
2

2 Answers 2

Reset to default
2

There are several users running Cassandra and upgrading them without any downtime to their app for many decades.

For the migration I first tried creating 1 new node with cassandra 4.1.2 version and added these nodes to the existing cluster.

This is not the recommended approach. If you're worried that your existing cluster will not take the load and your apps aren't using the correct consistency levels for reads / writes when there is a node that will be taken down for restarting post binary upgrades (from 3.x to 4.x that is), then you need to first scale-out your cluster by adding new nodes with the same exact 3.x versions as the other nodes in the cluster and then do the upgrade one by one to 4.x. Another approach is to figure out a low/no traffic period and perform the upgrade process one node at a time first by doing the binary upgrade of 3.x to 4.x on all nodes this cluster and then finally invoking the sstable upgrade process to convert existing SSTables to the newer version format.

For example, a cluster whose nodes are running Cassandra 3.11.15 is considered fully upgraded once all of its nodes have been successfully upgraded to Cassandra 4.1.3. With only a few exceptions, the cluster as a whole continues to work as though it were on the earlier version of Cassandra until all of the nodes in the cluster are upgraded.

During this time the data transfer between 2 different versioned nodes won't be happening, so the write request going to version 4 node will not be replicated to version 3 node and vice versa.

This is not correct understanding. The coordinator will ensure it will satisfy the write consistency as per how it is configured at the application client driver. So, if your client app has a write consistency level of LOCAL_QUORUM on a table whose keyspace has a replication factor set to 3, it will satisfy at least 2 replicas nodes owns the data copy prior to acknowledging the client as the write is successful (even if one of the replica is on 3.x and the other one is on the 4.x version respectively).

There is a fantastic upgrade guide here for 3.x to 4.x that you should familiarise and follow for what you are trying to achieve here.

If you need additional hand holding, you could always get a Luna subscription.

Improve this answer
4
  • So I added 3 new nodes in the cluster with same 3.x version as that of the and these new nodes got the data from the other nodes on startup and then I stopped one of the new nodes and setup cassandra4.1.2 in that and in cassandra.yaml I added data_file_directories: this key with pointing to cassandra3.x data folder and on startup it gave me this error There are nodes in the cluster with a different schema version than us, from which we did not merge schemas. Am I missing something here?
    – Saurabh Singh
    Sep 12 at 12:43
  • Without the exact steps that you performed with the binary upgrade, full stracktraces, logs files + config files -- this is not going to be an effective mechanism to go via a Q&A chat to fix this issue for sure.
    – Madhavan
    Sep 13 at 17:02
  • Thanks for all the help @Madhavan. I was able to migrate to the new cluster with the suggested approach. For the above comment I pointed to the incorrect path for data files which caused the above error. Once I pointed to the correct data folder of cassandra3 it worked as expected
    – Saurabh Singh
    Sep 14 at 18:23
  • Great Saurabh and glad you're now fully upgraded to the latest and greatest of the 4.x release trains! 5.x is right around the corner and should be super awesome with a LOT of features there!
    – Madhavan
    Sep 15 at 16:55
1

You cannot add a Cassandra 4.x node to an existing Cassandra 3.11 cluster because all operations which require streaming such as bootstrapping, decommissioning, or repairs will not work.

The streaming functionality is not compatible between major Cassandra versions which is the reason why we recommend temporarily disabling repairs until all nodes have been upgraded.

Similarly, schema formats are also not compatible between major versions. You will not be able to make any schema changes while the clusters are running in mixed versions and this approach is invalid since it's not intended for what you want to achieve:

I had to run -Dcassandra.skip_schema_check=true command else the new node was failing on startup.

The new 4.1 node failed to start because it is unable to join a C* 3.11 cluster -- they are not compatible. You can only do an in-place node upgrade, meaning you have to upgrade the nodes already in the cluster instead of adding a new node with a newer version of Cassandra.

You need to perform the upgrade one node at a time so the database stays online for the duration of the upgrade and at most only one node is down while it is being upgraded.

The high level steps for upgrading a cluster are:

Step 1 - For the first seed node, upgrade Cassandra to the latest 3.11 release.

Step 2 - Repeat above for the next seed node until all seeds are upgraded, then all the other nodes in the cluster.

Step 3 - Install C* 4.1 on the first seed node.

Step 4 - Configure Cassandra on the node. Do NOT copy the old configuration files like cassandra.yaml because they will not be compatible with the new version so you have to start with fresh C* 4.1 configuration files.

Step 5 - Repeat above for the next seed node until all seeds are upgraded, then all the other nodes in the cluster.

WARNING - You need to familiarise yourself with all the upgrade steps between C* 3.11 and C* 4.1 documented in the following NEWS.txt:

It is important that you follow EACH upgrade step in EACH patch release or risk unexpected outages or worse, data loss. Cheers!

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.