0

I've a very interesting question, is it possible that cassandra won't start rebuild process for any repaired keyspace? if we have few in unrepaired state using this option: nodetool rebuild --DATACENTER -ks KEYSPACE_NAME.

We saw the strange experiment yesterday, if there are few unrepaired keyspaces into the our cluster, and we've decided to change for example replication strategy for some keyspace which have been already repaired and using this command: nodetool rebuild --DATACENTER -ks KEYSPACE_NAME - this command fell with: Unable to find sufficient sources for streaming range, but if we start the command without --DATECENTER: nodetool rebuild -ks KEYSPACE_NAME - data will be appear on the all new nodes, instead only one, can we use this approach or is it bad idea ?.

Improve this question
3
  • 1
    Your question doesn't make sense because it's missing crucial details. Please update it with (1) keyspace definition, (2) full nodetool rebuild command you're running, (3) output of nodetool status, (4) full error message, (5) full stack trace, and (6) Cassandra version. Cheers!
    – Erick Ramirez
    Commented May 17, 2023 at 8:02
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
    – Community Bot
    Commented May 17, 2023 at 8:24
  • How many datacenters are in your cluster, and what are their names?
    – Aaron
    Commented May 17, 2023 at 14:38

1 Answer 1

Reset to default
0

The rebuild operation doesn't have any relation to repairs.

nodetool rebuild will populate data for token ranges on a node depending on whether the data has been streamed before or not (through bootstrap or rebuild).

Cassandra stores the "state" of token ranges of a node in the system.available_ranges table. If a node has been bootstrapped or rebuilt successfully, it will store the available ranges for each keyspace so it knows which data it can serve for read requests.

Furthermore, this error:

Unable to find sufficient sources for streaming range

would have been due to you removing the source DC from replication so Cassandra could not rebuild the new DC because there are no longer any replicas in the source DC you specified.

When you removed the source DC from the rebuild command, it just streamed the data from the other nodes in the new DC which is bad because the DC does not have data, by definition it is new.

In any case, your "experiment" is invalid. Repair is not a recommended way to transfer data to a new node. If it was, we wouldn't have a need for rebuild.

Think of your situation where you have 2 DCs with 3 replicas in each DC. This means a total of 6 replicas (2 DCs x 3 replicas). Repair is very expensive because it compares the data between 6 replicas to determine which data is missing on the node where you ran it.

Since the node is new, we already know it doesn't have data so doing the comparison with the 5 other replicas in the cluster is a waste of resources. All that is required is to request the data from another replica which is exactly what rebuild is designed to do. Cheers!

Improve this answer
8
  • Eric thanks so much for your advanced explanation, it's very useful and helpful information, but i still can't understand few things, I've decided to re-done my experiment, kill datacenter2 from ring via decommission on each node, clean all the nodes from datacenter2 and remove all data from cassandra /data directories, then altered all needed keyspaces (3 system + 2 user keyspaces) and remove any information about datacenter2
    – jetjo
    Commented May 19, 2023 at 9:00
  • Then i've done repair --full for each keyspace, then re-added all 3 nodes for datacenter2 (with auto_bootstrap: false) option in my cassandra.yaml, then I've ALTERED again with datacenter2 for (3 system + 2 user keyspaces) and then i tried to rebuild for single keyspace only, i've chosen for system_auth and again i saw this: nodetool -h localhost -u admin -pw my_password rebuild --datacenter1 -ks system_auth nodetool: Unable to find sufficient sources for streaming range Full(/192.168.1.141:7000,(9143760283360780305,-9110058574097929690]) in keyspace system_auth
    – jetjo
    Commented May 19, 2023 at 9:00
  • I checked the cassandra schema via nodetool describecluster and it the same for each node, and in the debug.log i didn't find any interesting information or error or exception.
    – jetjo
    Commented May 19, 2023 at 9:04
  • maybe is it possible that cassandra can keep any inrormation about previous datacenter2 ? and it must be clean before i will try to re-add with the same name as the new
    – jetjo
    Commented May 19, 2023 at 9:07
  • You can't run a rebuild on system_auth because you need it to run nodetool. The correct procedure is to repair the it when you change its replication. Cheers!
    – Erick Ramirez
    Commented May 22, 2023 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.