2

Background: Currently, we were running one instance of cassandra. One day we rebooted that server where cassandra resides and we started experiencing issues with the database. The num_token value changed from 256 to 512. It is my understanding that should not change on its own. It will only be changed if you change the number in the cassandra config file. However, I changed the num_token from 256 to 512 in the casssandra config file. After that changed was made everything was running smooth for a while. We applied some OS patches on that server and then restarted the server. After rebooting the server at that particular time we started seeing in the log file that the num_token value had changed again. It changed from 512 to like 1536. Then it changed again from 1536 3072. When it reached 3072 I could no longer make the change in the cassandra configuration file. The log file stated that 1536 was the max number that a node could be set to.

The thing that is most concerning is why the num_tokens changed in the first place. Is the database corrupted? I am not sure.

Solution: In order to resolved this issue we restored our VM back to prior instance before we reached 3072 num_token. Due to the fact that I feel that at any minute if we reboot the num_tokens will go from 1536 to 3072 and then we will be dead in the water again. In order to resolved this issue I wanted to create a new node to then form a cluster. I was hoping I could add the new node and then set the num_token to 256 on the new node. At that point the data from the original node (node 1) would sync to the new node (node 2).

I added the IP address of node 1 to the seeds line in the cassandra config file on node 2 (new node).

Is that the correct thing to do? Do I need to add the IP address of node 2 (new node) to the seeds line of cassandra config file to the original node (node 1)?

When I run the command "nodetool status" on node 2 (new node), I see both nodes listed. It shows that the new node is UJ and it show node 1 as DN (not sure why it says DN because it is saying UN on node 1) When I run the command "nodetool status" on node 1, I only seed the node 1 listed in the nodes. Here it shows node 1 as UN

Is this the expected behavior?

The sync process of extremely show. Should it take days to sync about 900 GB data to the new node or is something not setup correctly?

1
  • What do you see when you execute: SELECT tokens FROM system.local;. If that value is corrupted, that's where you might look for the root of the problem. The result should be just a (Java) set of token range numbers. If it's anything other than that, you could be in trouble.
    – Aaron
    Dec 3, 2022 at 15:17

1 Answer 1

2

I've never seen num_tokens increase after a node has been bootstrapped properly, it definitely sounds like something strange is occurring. You should also not have the node going DN in the status while the next node is joining, otherwise it won't properly join the cluster. It sounds like there could be a resource issue and potentially a network issue, but I have a few questions.

Which version of Cassandra are you running? How much memory/cpu do the nodes have? Which ports do you open between the nodes

That amount of time to sync does feel excessive, but it largely depends on the hardware and network speeds. I'd also prefer you set num_tokens to 8 instead of those high values and keep data distribution even by utilizing allocate_tokens_for_keyspace as described here (available in 3.0 and later): https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html

As for the seeds, you'll list node1 as itself when bootstrapping it since it's the first to come online. For node2, you'll also just list node1 as a seed, you don't want to include the IP of the local node you are bootstrapping unless it is the first to join the cluster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.