Timeline for Cassandra fully expired SSTables with timestamps in the future not getting deleted
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 31, 2023 at 6:47 | comment | added | Tigraine | Thanks for the help here Erick - we finally figured out what the issue was - I've added an answer for others to find here. | |
| Feb 22, 2023 at 5:35 | comment | added | Erick Ramirez |
Maximum timestamp: 4270051401267032369 = 24 Apr 2105 or 82 years in the future. Have you checked (a) the time + (b) the TZ on (c) ALL nodes + (d) ALL app servers? Do they all have NTP configured?
|
|
| Feb 20, 2023 at 8:43 | comment | added | Tigraine | So it happened again and I am pretty much absolutely sure there is nothing writing with USE TIMESTAMP right now. Still sstableblockers now shows a bunch of blocked tables because of this: gist.github.com/Tigraine/ec06bc66e1a099e4dca1078d5aca5cf0 Really can't understand how this could even happen but also it feels very random. Is there maybe some way a incorrectly set timestamp on clients can act up like this? | |
| Feb 16, 2023 at 9:17 | comment | added | Tigraine | The only explanation I could think of is that the application doing the inserting had a mi-configured timestamp and the driver code somehow implicitly uses the client time instead of the Cassandra server time. Also interestingly enough - two tables that had issues with TWCS not cleaning up both started exhibiting the issue on Oct 5th. | |
| Feb 16, 2023 at 9:16 | comment | added | Tigraine | Hi Eric, I thought the same and have been investigating the code in question, but there is really no way we were ever inserting data incorrectly or USING TIMESTAMP. There is exactly one application inserting data here and I can't figure out how such a tstamp could have happened 🤷♀️ I've fixed the one table now by simply truncating the data - but really at a loss how it could have happened. | |
| Feb 16, 2023 at 5:24 | history | answered | Erick Ramirez | CC BY-SA 4.0 |