I've been struggling with how specifically TWCS works lately. In our workflow we have some tables containing time-series data with a mostly static TTL. As such, a decision was made to use TWCS as the compaction strategy for this. There are, however, unfortunately some of these tables which appear to be taking up more space than usual, i.e. they appear to be blocked by the oldest SSTables. We have already pinpointed one potential root cause. There is a scenario in which a TTL rewrite may occur on an existing record, e.g. diminishing the TTL of this existing record. In this scenario, it makes sense that this record, which would reside in multiple SSTables, with higher and lower TTL, should block other SSTables from being compacted. Indeed, after a while these updated records appeared to TTL out and subsequently all was unblocked. However, there are also tables which are growing rapidly, for which the sstableblockers tool reveals no blocking tables.
Therefore I took to reading some articles about this. Some articles I refer to are e.g. https://thelastpickle.com/blog/2016/12/08/TWCS-part1.html or https://www.redshots.com/cassandra-twcs-must-have-ttls/. Especially the latter caused suspicion in me as to whether a TTL change between "new" records would also have a blocking effect. The latter article specifically mentions The deletion of the whole sstable when filled with TTL's will only happen if it is the oldest sstable, therefore once one cannot be deleted, none of the subsequent ones can be deleted either.
We consider the following scenario with TWCS:
We write e.g. 100 records with a TTL of 5000 which all end up in the same SSTable, then subsequently, we write 100 new records with a different TTL of 100, written to a new column (but maybe the same partition).
Based on my tests I presume that the first 100 records to "TTL out" will not release any disk space until the SSTable containing the records with oldest TTL expires.
However, I wasn't entirely confident about this, especially since both SSTables essentially contain nothing that should be shadowing each other. As such, I don't believe there should logically be a reason for these tables to not be removed from disk. Furthermore, I also considered the possibility of older TTL'ed and newer TTL'ed data being mixed in the oldest SSTable. I presumed this would also behave the same. In general, questions remained. As such I wanted to validate this conclusion.
In short: with TWCS, is removal from disk of a new record with lower TTL going to be blocked by any older record with higher TTL residing in a different SSTable?
