5

After getting some inconsistent query results we noticed that two nodes are claiming ownership of two tokens (-5930834448474026108 and 1653676441334745669). When a node comes up we flip between consistent and inconsistent query results and we see this in the logs.

TokenMetadata.java:226 - Token -5930834448474026108 changing ownership from /172.17.32.251 to /172.17.33.62
TokenMetadata.java:226 - Token 1653676441334745669 changing ownership from /172.17.32.251 to /172.17.33.62

Or the reverse

TokenMetadata.java:226 - Token -5930834448474026108 changing ownership from /172.17.33.62 to /172.17.32.251
TokenMetadata.java:226 - Token 1653676441334745669 changing ownership from /172.17.33.62 to /172.17.32.251

This is the view of peers from a seed node on the cluster.

cqlsh> select peer, tokens from system.peers where tokens contains '-5930834448474026108' and tokens contains '1653676441334745669' ALLOW FILTERING;
    
 peer          | tokens
---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 172.17.32.251 |     {'-3586663459313641829', '-4235041598543029843', '-5390242841269357826', '-5930834448474026108', '-8760593756478543069', '1653676441334745669', '166599935335281141', '2082376249768721299', '2737611629421780150', '3505320720095561779', '5463123014169021365', '6208992122760287054', '753682649578647219', '7718045837568216134', '8541991172278593633', '9138457739984968252'}
  172.17.33.62 | {'-1187962964387570247', '-1890723219453312357', '-3797875287284457587', '-4763268132960963434', '-5264847899607678948', '-5930834448474026108', '-6918838058130066386', '-7528285457674952405', '-7572391200230517', '-815483085963289954', '-8212901179232502774', '1653676441334745669', '4217161315982965657', '7284498335104901513', '8107381942324557264', '8959725380439260138'}

And here we need the view of nodetool status. Each node is set with num_tokens: 16. Notice how 172.17.32.251 only has 14 tokens in this view but 16 items in the cql query above.

nodetool status
Datacenter: us-east-1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.32.17   2.95 TiB   16           27.2%             7b26a987-12bb-4149-98cb-65c1945b611a  a
UN  172.17.32.34   2.96 TiB   16           25.0%             94abcef1-5d8f-49b5-ba35-b2a099c082f6  a
UN  172.17.34.103  3.11 TiB   16           25.1%             ce361749-0133-497d-b687-6cb741df2646  c
UN  172.17.34.87   3.18 TiB   16           25.6%             8687ba37-8e22-49ae-a85d-089f11e2a21d  c
UN  172.17.32.103  3.19 TiB   16           27.0%             37a619e2-0e2c-4aa0-bf15-e189eb265b77  a
UN  172.17.33.9    2.97 TiB   16           24.6%             79ce1b7e-0ba2-48d8-ab07-bdd08d67048e  b
UN  172.17.33.11   3.08 TiB   16           24.9%             bb9c7716-d66a-4464-bfa8-5a161fd45816  b
UN  172.17.32.251  2.53 TiB   14           20.8%             0de45700-70f5-454b-a191-c62faae99682  a
UN  172.17.33.219  3.22 TiB   16           25.8%             5067ced7-4823-4a71-9c7d-c70030530430  b
UN  172.17.34.127  3.04 TiB   16           24.5%             7557204f-0b47-4058-85a4-5fa45c2baf6f  c
DN  172.17.33.62   2.8 TiB    16           24.7%             b5bcb2e4-f2c8-49bc-9363-1393d495ca38  b
UN  172.17.34.238  3.08 TiB   16           24.8%             e56190d4-64f3-4cea-959f-a1bc76b88833  c

This is a production cluster which has been running for almost a year now. We did not use any manual token allocation to setup this cluster.

What's the best way to fix the token allocation issues?

1 Answer 1

3

Did you happen to bootstrap nodes in parallel? I've seen this happen when attempting to bootstrap nodes too quickly. The best way to fix it now is to decommission one of the nodes from the cluster, assassinate the node once decommission has finished, then clear the data and bootstrap it back to the cluster. With your data load, that's not a fast operation to complete, but it's the best way to fix the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.