0

I have a Cassandra cluster with 3 nodes and a replication factor of 3. A colleague of mine moved the data for each of the nodes from one disk to the other.

It is possible that something wrong happened in the process.

When I run the query below:

select min(dttime),max(dttime),sum(odo)/1000.0 from module where dttime >= '2022-08-01 01:00' and dttime <= '2022-08-23 17:00' and imei=358899051143710 ;

I get different results at different times.

I got three different results at 3 different times, e.g

cassandra@cqlsh:gps> select min(dttime),max(dttime),sum(odo)/1000.0 from module where dttime >= '2022-08-01 01:00' and dttime <= '2022-08-23 17:00' and imei=358899051143710 ;

 system.min(dttime) | system.max(dttime) | system.sum(odo) / 1000.0
--------------------+--------------------+--------------------------
               null |               null |                        0


 system.min(dttime)              | system.max(dttime)              | system.sum(odo) / 1000.0
---------------------------------+---------------------------------+--------------------------
 2022-08-01 01:00:13.632000+0000 | 2022-08-01 06:11:13.163000+0000 |                   92.471


 system.min(dttime)              | system.max(dttime)              | system.sum(odo) / 1000.0
---------------------------------+---------------------------------+--------------------------
 2022-08-01 01:00:13.632000+0000 | 2022-08-23 16:54:13.686000+0000 |                 8712.734

A further observation is that if one particular node is down, I always get the last result.

What is likely to be the cause of the inconsistency, and if any remedy exists?

I have tried nodetool repair and scrub, but without success.

2 Answers 2

1

moved the data for each of the nodes from one disk to the other.

Was a nodetool refresh run afterward?

if one particular node is down, I always get the last result.

That sounds like there's an inconsistent replica.

I have tried nodetool repair and scrub, but without success.

Not sure which parameters were used for nodetool repair, but that's what needs to happen here. I would run a nodetool repair -pr for that table on all nodes.

The other thing is that you could try using a higher level of consistency when querying. Perhaps running at LOCAL_QUORUM would help with this.

2
  • @ypercubeᵀᴹ regardless of the number of nodes or replicas, the app can be configured to query at a consistency level of ONE, QUORUM or ALL.
    – Aaron
    Sep 23, 2022 at 20:34
  • I deleted the the folder in /var/lib/cassandra/data. Now I don't have the inconsistency problem. With or without quorum, I get correct the correct reults. I had run nodetool repair on all 3 nodes earlier. I am now running nodetool repair -pr on all nodes. On the rogue node, nodetool compactionstats shows 0 tasks pending, while on the others, it produces an output. On the rogue node, I see 57GB of data (this would be since I deleted the data) , while on the others I see 1.5TB. This is to be expected, but data size is no increasing on the node. How do I know that nodetoool -pr is working?
    – ranban282
    Sep 26, 2022 at 7:16
0

Inconsistent results are due to replicas being out-of-sync. At some point, your cluster was overloaded and couldn't keep up with the writes so some nodes (replicas) were dropping mutations.

It is quite easy to verify if this is the case if you turn TRACING on:

cqlsh> TRACING ON;

and using a consistency level of ALL:

cqlsh> CONSISTENCY ALL;

then re-run your query. You should see activities that indicate a digest mismatch(es) and read-repair(s) kicking in.

As Aaron already pointed out, you can fix the inconsistencies by running a rolling repair on the table, one node at a time:

$ nodetool repair -pr -- ks_name table_name

Avoid overloading the cluster so the nodes don't drop mutations and you won't run into this issue. Otherwise, consider increasing the capacity of your cluster by adding more nodes. Cheers!

3
  • What happens if we run repair simultaneously on all nodes?
    – ranban282
    Sep 26, 2022 at 7:47
  • Btw the cause is definitely not overloading - as I mentioned someone moved data incorrectly or missed a command. Corrupt data in the disk could also be a reason.
    – ranban282
    Sep 26, 2022 at 7:50
  • Repairs are CPU and IO intensive because (1) they need to read a lot of data on disk, and (2) compute the inconsistencies to determine what needs to get fixed. If you run them in parallel, it could overload your cluster and affect the user experience. Cheers! Sep 27, 2022 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.