Cassandra nodetool status output is inconsistent across nodes

Question

I have a cassandra cluster with only two nodes (EC2 instances) on the same region. And I inserted 70GB of data into a table of a keyspace with replication factor of 1.

The data is distributed fine between my nodes. The first one has around 30GB of data and the second one has around 40GB.

ubuntu@cassandra1:~/storage/cassandra_data/data$ du -sh .
30G .

ubuntu@cassandra2:~/storage/cassandra_data/data$ du -sh .
40G .

However, when I use nodetool status command, each node claims that it owns around 99% of my data, even though my replication factor is 1.

Here is the output of the command on my first node:

ubuntu@cassandra1:~/apache-cassandra-4.0.5/bin$ nodetool status accounts
Datacenter: eu-central-1
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens  Owns (effective)  Host ID                               Rack
UN  172.31.8.94   39.85 GiB  16      1.1%              63a6fc88-41c5-42f3-8782-c67be4df70e5  eu-central-1c
UN  172.31.8.187  29.97 GiB  16      98.9%             e1de2f4c-1a7e-4329-80f1-91b47cde9d43  eu-central-1c

And this is its output on my second node:

ubuntu@cassandra2:~/apache-cassandra-4.0.5/bin$ nodetool status accounts
Datacenter: eu-central-1
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens  Owns (effective)  Host ID                               Rack
UN  172.31.8.94   39.85 GiB  16      99.1%             63a6fc88-41c5-42f3-8782-c67be4df70e5  eu-central-1c
UN  172.31.8.187  29.97 GiB  16      0.9%              e1de2f4c-1a7e-4329-80f1-91b47cde9d43  eu-central-1c

More information about my cluster:

• cassandra_version: 4.0.5
• endpoint_snitch: Ec2Snitch
• partitioner: org.apache.cassandra.dht.ByteOrderedPartitioner

Update:

cqlsh> describe accounts;

CREATE KEYSPACE accounts WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}  AND durable_writes = true;

CREATE TABLE accounts.transaction_by_signature (
    signature blob,
    slot int,
    time bigint,
    transaction blob,
    PRIMARY KEY (signature, slot)
) WITH CLUSTERING ORDER BY (slot ASC)
    AND additional_write_policy = '99p'
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND cdc = false
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '16', 'class': 'org.apache.cassandra.io.compress.ZstdCompressor'}
    AND crc_check_chance = 1.0
    AND default_time_to_live = 0
    AND extensions = {}
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair = 'BLOCKING'
    AND speculative_retry = '99p';

Answer 1

I suspect that the replication settings for your various keyspaces are all over the place so the output is meaningless unless you supply the keyspace name.

We recommend that you include the keyspace when running the command to get a better understanding of the data ownership. For example:

$ nodetool status app_ks

If you think that the output is still incorrect after specifying the status of a keyspace, please update your original question with the schema of the keyspace and I'd be happy to review it then update my answer.

As a side note, ByteOrderedPartitioner is not recommended and has been deprecated several years ago. Unless you are running a very old cluster that you've recently upgraded to C* 4.0, we recommend that you use the default Murmur3Partitioner. Cheers!