Adding new nodes, nodetool rebuild returns "Unable to find sufficient sources for streaming range"

Question

I have the cassandra 4.0.1 cluster with 3 nodes (datacenter1) and we have decided that we need to add the second datacenter (datacenter2), I have done all the steps described in this article

First of all, we have added the nodes one by one with the auto_bootstrap: false option in the cassandra.yaml configuration for all three new nodes.

The replication factor was rf=o for datacenter2 during adding/bootstraping process. Then we have changed from rf=0 to rf=3 for system_auth, system_traces, system_distributed keyspaces and my user keyspace which is named big_data for datacenter2 through ALTER KEYSPACE.

All the nodes have joined to cluster successfully, but there is one strange situation there:

We cannot do the rebuild process from datacenter1 and cassandra tells me the following:

nodetool: Unable to find sufficient sources for streaming range Full(/192.168.1.141:7000,(9143760283360780305,-9086318401025854203]) in keyspace system_auth  

...and I saw the same print on the next new nodes. What can I do with it?

Schema version is the same on all nodes.
If I perform full repair all the data will appear on all new nodes, but the rebuild process still doesn't work properly in this case.

I need your advice.

Answer 1

There is a good chance that you missed a step when adding the new DC.

The high level steps are:

1. Switch system_distributed, system_auth, system_traces and application keyspaces to NetworkTopologyStrategy with replicas only in the existing DC.
2. Repair the keyspaces.
3. Add nodes to the new DC with auto_bootstrap: false.
4. After all new nodes are running, alter the keyspaces to add replicas in the new DC.
5. Build the nodes in the new DC with:

$ nodetool rebuild -- source_dc

If you performed step (4) without having done steps (1) and (2) then you'll run into the problem you reported above.

Just go back, re-do the steps again and you should be fine. Cheers!