Startup fails post upgrade with "AssertionError: null" in o.a.c.schema.TableMetadata$CompactTableMetadata.getCompactValueColumn()

Question

I am trying to upgrade my 3 node Cassandra cluster from version 3.11.13 to 4.1.5.

After start the upgrade node I get this error:

ERROR [main] 2024-06-20 08:48:23,708 CassandraDaemon.java:915 - Exception encountered during startup
java.lang.AssertionError: null
        at org.apache.cassandra.schema.TableMetadata$CompactTableMetadata.getCompactValueColumn(TableMetadata.java:1515)
        at org.apache.cassandra.schema.TableMetadata$CompactTableMetadata.<init>(TableMetadata.java:1350)
        at org.apache.cassandra.schema.TableMetadata$Builder.build(TableMetadata.java:747)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:943)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:898)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:857)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836)
        at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132)
        at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:285)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:759)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:893)

I found this issue Upgrading Cassandra 3.11.8 to 4.1.0 returns "AssertionError: null" on startup while loading the schema and the advice was to remove COMPACT STORAGE.

There is only one COMPACT STORAGE in the system keyspace:

DESCRIBE TABLE system.hints

CREATE TABLE system.hints (
    target_id uuid,
    hint_id timeuuid,
    message_version int,
    mutation blob,
    PRIMARY KEY (target_id, hint_id, message_version)
) WITH COMPACT STORAGE
    AND CLUSTERING ORDER BY (hint_id ASC, message_version ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = '*DEPRECATED* hints awaiting delivery'
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'enabled': 'false', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 0
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 3600000
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

Unfortunately I cannot drop it:

cqlsh> ALTER TABLE system.hints DROP COMPACT STORAGE;
Unauthorized: Error from server: code=2100 [Unauthorized] message="system keyspace is not user-modifiable."

I can't even downgrade the node back to 3.11.13:

ERROR [main] 2024-06-20 08:38:07,157 CassandraDaemon.java:803 - Detected unreadable sstables /var/lib/cassandra/data/system/local/nb-244-big-Statistics.db,/var/lib/cassandra/data/system/local/nb-244-big-Data.db,/var/lib/cassandra/data/system/local/nb-244-big-Summary.db,/var/
lib/cassandra/data/system/local/nb-244-big-CompressionInfo.db,/var/lib/cassandra/data/system/local/nb-244-big-Index.db,/var/lib/cassandra/data/system/local/nb-244-big-Filter.db, please check NEWS.txt and ensure that you have upgraded through all required intermediate version
s, running upgradesstables

The table system.hints is not present on my others cassandra cluster where the upgrade went smoothly.

Does anybody have any ideas how the get out of this situation?

Answer 1

After reviewing the stack trace, I don't believe the issue relates to the system.hints table.

The exception is thrown in TableMetadata$CompactTableMetadata.getCompactValueColumn() when it tries to retrieve the a column in a table which has COMPACT STORAGE. I suspect the error is due to another compact table that has been updated to have a regular CQL column (versus a compact/Thrift column) which is invalid.

A compact (Thrift) table cannot have a CQL column added to it so it is no longer allowed in an ALTER TABLE statement (CASSANDRA-14564).

You will need to review the cluster schema again for compact tables other than system.hints. Unfortunately, I'm not aware of a solution for this other than dropping then recreating the table while still in C* 3.11.

On a side note, you cannot downgrade the binaries on a node once it has been running with a newer version of Cassandra since it will generate SSTables in the new format which is not readable by the older version. Cheers!