Background and Observations:
I have a 3 node cluster that is running in production (version 3.11). For the migration I first tried creating 1 new node with cassandra 4.1.2 version and added these nodes to the existing cluster. To add these node I had to run
-Dcassandra.skip_schema_check=true command else the new node was failing on startup.
Once I was able to add these nodes in the existing cluster I observed that the data is not streaming to the new node from version 3 to version 4 node.
The Idea here was that once the data is replicated to version 4 nodes we can shut down the version 3 nodes one by one and that would be a smooth migration.
The document for migration from 3.x to 4.x here suggests that the migration should be done node by node and I should first take down one node and then setup cassandra 4 on that node and then use the same data files to run cassandra 4 and then do the same for all nodes.
My question is that if I go ahead with the approach mentioned in the datastax document there will be a time when version 4 and version 3 will coexist while my read/write load will still be coming to the cluster. During this time the data transfer between 2 different versioned nodes won't be happening, so the write request going to version 4 node will not be replicated to version 3 node and vice versa.
This will cause the data inconsistency until all nodes are migrated and they start communicating with each other. We assume that the data will be consistent eventually here but if there is any issue that occurs during this process we won't be having any way to resolve the inconsistent data in production.
So what steps I can follow to make sure that there is no data inconsistency and if something goes wrong there is some easier way to correct the data.
And also I wanted to understand why the 1st approach mentioned above was not supported by cassandra as it seems a easier approach for a developer from migration stand point.