Deletions getting omitted, data supposedly deleted returned in CQL queries

Question

We are facing an issue on one of our production systems where after we delete the data the data doesn't seem to get deleted. We have a Get call just after the delete call. The data shows up.

Versions

cassandra : 3.11.6
gocqlx : v2 v2.1.0

1. Client Settings: LocalQuorum
2. Number of Nodes : 3
3. All 3 nodes up and running for weeks.
4. Inserts were done few days earlier. So there is good amount of time difference between Inserts and Deletes and Inserts have made through successfully.

The Delete Call :

q := s.session.Query(stmt, names).BindStruct(*customModel)
err := q.ExecRelease()

We do check the error and it is Nil. There are no exceptions during that time either on the client side or server side.

The Get Call :

q := s.session.Query(stmt, names).BindStruct(*customModel)
err := q.GetRelease(customModel)

This returns the data successfully.

We do have these two options enabled.

1. https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__commitlog_sync

   batch - Send ACK signal for writes after the commit log has been flushed to disk. Each incoming write triggers the flush task.

2. only_purge_repaired_tombstones

This does not happen for all the delete operations. For many of them, the delete seems to go through. This does not seem to be timing-related and the successful and unsuccessful ones are spread out.

CASSANDRA-15690 Single partition queries can mistakenly omit partition deletions and resurrect data

I am trying to go through this PR and ticket. If you have any suggestions please do let me know.

The table structure is the following

CREATE KEYSPACE cycling WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

CREATE TABLE cycling.rider (
    uuid text,
    created_at timestamp,
    PRIMARY KEY (uuid, created_at)
) WITH CLUSTERING ORDER BY (created_at DESC)
    AND WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4', 'only_purge_repaired_tombstones': 'true'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

Answer 1

This is likely the problem:

AND gc_grace_seconds = 3600

The default setting is 864000, which is 10 days. This setting of 3600 is one hour, and that's WAY to little time to wait before marking the tombstones to be collected.

The recommendation here is to set this period to be greater than the scheduled repair interval. The idea being that most teams setup their cluster to repair weekly or every 7 days (which is why the default is 10 days).

In my experience, 4 days (345600) is the lowest that gc_grace_seconds can be set to without causing issues.

Answer 2

CASSANDRA-15690 (fixed in Cassandra 3.11.7) is a good find and it does appear that it applies to your situation given that the cluster is running C* 3.11.6.

However I will point out that in my experience working with enterprises on hundreds of clusters, I have discovered on multiple occasions that what is claimed to be deleted does in fact have a newer version of the data. I don't know how you determine that the data is supposed to be deleted but if you have a record of when the DELETE was issued, I would suggest that you at least compare it to the WRITETIME() of the data.

Finally, I support @Aaron's response that a really low GC grace is very problematic and almost always not the right choice. I suspect there is another problem in your data model that you think will be solved by a low GC grace but you end up creating more problems as a result. Cheers!