Cassandra fully expired SSTables with timestamps in the future not getting deleted

Question

I am struggling with a interesting problem of Cassandra nodes just using up more and more memory on disk for no apparent reason. The schema is an append only data model of event data that should be deleted after a fairly short period of time (24h). The table is using a TTL of 24 hours and we are using TWCS with a 1 day window size.

After some investigation into sstableexpiredblockers and doing a fair amount of sstabledump I found out that there are a lot of old sstables (2022-10-05 being the oldest) still around with records that have a liveness_info like this:

"liveness_info" : { "tstamp" : "2250-12-20T13:58:36.059597Z", "ttl" : 604800, "expires_at" : "2022-03-27T14:55:13Z", "expired" : true },

As you can see, it says expired: true, but the tstamp is in the future.

What could cause this future tstamp? My current suspicion is that these timestamps keep the data from being deleted - as these SSTables even survive user-triggered compactions. All the nodes run ntp correctly and we don't do any timestamp overrides on insert (we use gocql for writing). Cassandra is running version 3.11.12

I also tried querying the data (using the PK from sstabledump) and I get no results - so somehow Cassandra considers the data expired, it's just not being deleted from disk - and that blocks other sstables from being deleted.

The whole record looks like this:

"rows" : [
      {
        "type" : "row",
        "position" : 4671471,
        "clustering" : [ "0f67db7e-e0e0-495c-95a1-5038befe09a7", 58 ],
        "liveness_info" : { "tstamp" : "2250-12-20T13:58:36.059597Z", "ttl" : 604800, "expires_at" : "2022-03-27T14:55:13Z", "expired" : true },
        "cells" : [
          { "name" : "data", "deletion_info" : { "local_delete_time" : "2022-03-20T14:55:13Z" }
          }
        ]
      }

Has anyone seen this before and if so any ideas what could cause this and how to fix it?

Answer 1

Unsurprisingly, I haven't come across this issue before. I've tried to come up with possible scenarios that would cause the timestamp to be in the future and the only way I could think of is when it has been explicitly set with USING TIMESTAMP.

You will need to dig a bit deeper to determine the possibility that data wasn't inserted incorrectly that has lead to this. Cheers!

Answer 2

I've found the solution and wanted to share it here for others to find:

Protocol v3 supports using the client-supplied timestamps as the primary timestamp. This enables consistency even with out-of-order processing on the server (eg. under congestion or just through network latency) instead of the server-time of when the command was received.

The gocql driver uses client side timestamps by default through the DefaultTimestamp setting and that caused the timestamp issues in our application. Since we are using auto-scaled instances it seems sometimes an instance would come up and start writing while still in the process of receiving NTP updates. Those inserts would then be all over the place and cause these far-future timestamps in Cassandra.

Since our use-case/schema has no race conditions where last-write-wins would be relevant we disabled the DefaultTimestamp setting in gocql and since then our TWCS has been behaving correctly in deleting old SSTables.