How does Cassandra achieve strong consistency with failed writes if there is no rollback and no two phase commit?

Question

Looking for concrete evidence (documentation or source code) on the behavior of cassandra 3.0+ on the following situation

Write of key1, value1 is requested with consistency level of QUORUM but only N replica responded success where N < QUORUM

What happen to those N nodes that just updated key1? Do they get rollback?

In cassandra documentation https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlTransactionsDiffer.html

if using a write consistency level of QUORUM with a replication factor of 3, Cassandra will replicate the write to all nodes in the cluster and wait for acknowledgement from two nodes. If the write fails on one of the nodes but succeeds on the other, Cassandra reports a failure to replicate the write on that node. However, the replicated write that succeeds on the other node is not automatically rolled back.

It mentions if some write failed and did not satisfied consistency level, coordinator will return failure, but the data will persist on nodes that have write succeeded

But this means strong consistency can never be achieved even if R + W > number of replica as official. documentation suggested

https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlAboutDataConsistency.html

Consider the following situation

replica number = 5
consistency level write = 3
consistency level read = 3

If a write is attempted , but one nodes succeeds , coordinator will return failure, but that one node will not rollback, so you need a consistency level of 5 in order to achieve strong consistentcy

The documentation has conflicting information

What am I getting wrong here?

Answer 1

I couldn't find much in the official documentation to support my point as concrete evidence, due to lack of detail on the full write path, but I can share my 2 cents on the topic.

I think I understand where you see the gap in consistency.

First let's establish that when a client reads with consistency level (CL) of LOCAL_QUORUM with replication factor (RF) 5, the Cassandra coordinator for that request will only contact 3 Cassandra replicas for a read (out of 5) - 1 to return the actual data to the coordinator, and the other 2 to return a digest of the data for a consistency check.

However, when a client writes with CL LOCAL_QUORUM against the same keyspace, the Cassandra coordinator will send the mutation to all reachable replicas. What the CL for mutations implies, in this example LOCAL_QUORUM, is that 3 replicas must succeed in completing the write request locally within the write timeout period (default 2 sec).

In other words if the coordinator doesn't receive an acknowledged write from at least 3 replicas within 2 seconds, the write is reported as failed to the client. Although in the background the coordinator will attempt to replicate it to all replicas in the cluster, with no rollbacks. If you have repairs running and tuned to gc_grace_seconds, eventually all 5 replicas will be consistent.

If a replica is unavailable, the coordinators can store write requests in the form of hints for a default period of 3h. As soon as unreachable nodes are back up, if hints didn't expire yet, the coordinator will send the previously failed mutations.

It should be noted that unreachable nodes will miss mutations past the 3h hint window. Then there are 2 possible ways to retrieve dropped mutations on previously unreachable nodes:

• Read Repairs (passive) - If a coordinator detects a digest mismatch during a read with CL > 1, it will repair the mismatched data before returning the results to the client, in what's called a blocking read repair.
• Anti-entropy Repairs (active) - Repairs are an essential operation in live Cassandra clusters that actively scan, hash, and compare ranges of data between replicas, encompassing the full dataset, and repair the data via streaming whenever inconsistencies are found.

Answer 2

Asking how Cassandra achieves strong consistency for failed writes is a moot point -- because they failed.

If the consistency for a write request is not met then by definition, the request failed. Anything that follows from that failed request is irrelevant.

For the second part of your question, Cassandra does not have a rollback mechanism. When a coordinator sends the mutation (write) to all the replicas, it waits for an acknowledgement from the replicas within write_request_timeout. If a coordinator doesn't hear back from the replica, it is impossible for it to know whether the replica succeeded persisting the mutation. This is an inherent problem in a distributed architecture.

The coordinator returns an UNAVAILABLE exception to the client when not enough replicas respond (consistency level is not met). The driver (client) will handle this exception depending on how it is configured. For example, the Java driver's DefaultRetryPolicy is to retry the write request on another coordinator by picking the next node in the query plan.

If the retry also fails then you as the developer have to decide how your app handles the write failure. Think of the scenario where your app issues a DELETE if your business rules require a rollback.

In your example, your assertion that setting the consistency to N replicas is incorrect since it won't achieve anything that QUORUM does. If any (or all) the replicas don't acknowledge the write then it is still marked as failed. I think your mistake is conflating consistency with rollback when they are separate mechanisms in the write path. Cheers!