1

I recently added a new node to my 3 node 4.0.5 cluster because the nodes were approaching 50% disk usage. I expected the cluster to balance itself but this hasn't happened (yet?). The nodes are similar so I have num_tokens=256 for each. I am using the Murmur3Partitioner. How can I balance the data? I am going to be adding another node soon. Thanks.

# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens  Owns (effective)  Host ID                               Rack
UN  xxx.xxx.xxx.105  2.65 TiB   256     72.9%             afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
UN  xxx.xxx.xxx.253  2.6 TiB    256     73.9%             e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
UN  xxx.xxx.xxx.24   93.82 KiB  256     80.0%             c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
UN  xxx.xxx.xxx.107  2.65 TiB   256     73.2%             ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1

# nodetool netstats
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed   Dropped
Large messages                  n/a         0          71754         0
Small messages                  n/a         0        8398184        14
Gossip messages                 n/a         0        1303634         0

# nodetool ring
Datacenter: dc1
==========
Address               Rack        Status State   Load            Owns                Token
                                                                                     9189523899826545641
xxx.xxx.xxx.24        rack4       Up     Normal  93.82 KiB       79.95%              -9194674091837769168
xxx.xxx.xxx.107       rack1       Up     Normal  2.65 TiB        73.25%              -9168781258594813088
xxx.xxx.xxx.253       rack2       Up     Normal  2.6 TiB         73.92%              -9163037340977721917
xxx.xxx.xxx.105       rack3       Up     Normal  2.65 TiB        72.88%              -9148860739730046229
xxx.xxx.xxx.107       rack1       Up     Normal  2.65 TiB        73.25%              -9125240034139323535
xxx.xxx.xxx.253       rack2       Up     Normal  2.6 TiB         73.92%              -9112518853051755414
xxx.xxx.xxx.105       rack3       Up     Normal  2.65 TiB        72.88%              -9100516173422432134
...
4
  • I have just discovered that my cluster is no longer returning rows for many reads. Help! Commented Apr 3, 2023 at 3:03
  • Any specific reason for having such a high token value (i.e. num_tokens=256) as opposed to the default of 16?
    – Madhavan
    Commented Apr 4, 2023 at 21:18
  • 1
    I turns out that I made the new node a seed node so it didn't bootstrap. Commented Apr 5, 2023 at 4:21
  • I think 256 was the default when I first setup the cluster with 4.0beta3. issues.apache.org/jira/browse/CASSANDRA-13701 Commented Apr 6, 2023 at 5:22

1 Answer 1

2

This isn't a rebalancing issue. Assuming that the new node is the one with only 94KB on it, that indicates to me that there there was a mistake in the way that the node was added so it failed to bootstrap data from replicas:

UN  xxx.xxx.xxx.24  93.82 KiB  256  80.0%  c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4

This can happen for various reasons but mainly for these scenarios:

  • node is configured to not bootstrap (auto_bootstrap: false in cassandra.yaml), or
  • node is configured as a seed node (in cassandra.yaml).

In both of these scenarios, the node will not request data from other nodes and immediately join the ring so it doesn't have any data.

To force the node to request data from replicas, run a rebuild operation on it:

$ nodetool rebuild

As a side issue, I noted you placed the node in rack4 so your ring now has 4 racks. If your application keyspace(s) is configured with a replication factor of 3, the data will not be distributed equally among the 4 racks and load on the nodes will be skewed.

The recommendation is to have racks which are multiples of the replication factor so for RF:3 in the DC, you should deploy 3 or 6 or 9 racks with the same number of nodes per rack so that data will be distributed evenly across nodes. If it is not possible to maintain racks which are multiples of RF, we recommend you deploy all nodes in a single logical rack regardless of the nodes' physical or geographical rack placement. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.