0

I am running cassandra in 3 node cluster with replication factor 3. It was working fine for 2 days and all of a sudden it went down. I am beginner to cassandra database. I went through documentation to understand why it is happening . I could not get it .Below are the logs.

INFO  [main] 2020-02-17 11:52:34,270 CassandraDaemon.java:473 - Hostname: QCS7
INFO  [main] 2020-02-17 11:52:34,271 CassandraDaemon.java:480 - JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.8.0_152
INFO  [main] 2020-02-17 11:52:34,274 CassandraDaemon.java:481 - Heap size: 998.438MiB/998.438MiB
INFO  [main] 2020-02-17 11:52:34,275 CassandraDaemon.java:486 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4815808(4702K) committed = 4849664(4736K) max = 251658240(245760K)
INFO  [main] 2020-02-17 11:52:34,275 CassandraDaemon.java:486 - Metaspace Non-heap memory: init = 0(0K) used = 18448024(18015K) committed = 19005440(18560K) max = -1(-1K)
INFO  [main] 2020-02-17 11:52:34,275 CassandraDaemon.java:486 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2277864(2224K) committed = 2490368(2432K) max = 1073741824(1048576K)
INFO  [main] 2020-02-17 11:52:34,275 CassandraDaemon.java:486 - Par Eden Space Heap memory: init = 214827008(209792K) used = 98821656(96505K) committed = 214827008(209792K) max = 214827008(209792K)
INFO  [main] 2020-02-17 11:52:34,276 CassandraDaemon.java:486 - Par Survivor Space Heap memory: init = 26804224(26176K) used = 0(0K) committed = 26804224(26176K) max = 26804224(26176K)
INFO  [main] 2020-02-17 11:52:34,276 CassandraDaemon.java:486 - CMS Old Gen Heap memory: init = 805306368(786432K) used = 0(0K) committed = 805306368(786432K) max = 805306368(786432K)

INFO  [pool-3-thread-1] 2020-02-17 11:53:24,320 AutoSavingCache.java:262 - Harmless error reading saved cache /var/lib/cassandra/saved_caches/KeyCache-e.db
java.io.IOException: Corrupted key cache. Key length of 83886081 is longer than maximum of 65535
    at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:481) ~[apache-cassandra-3.11.2.jar:3.11.2]
    at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:219) ~[apache-cassandra-3.11.2.jar:3.11.2]
    at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:164) [apache-cassandra-3.11.2.jar:3.11.2]
    at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:160) [apache-cassandra-3.11.2.jar:3.11.2]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_152]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
INFO  [pool-3-thread-1] 2020-02-17 11:53:24,320 AutoSavingCache.java:173 - Completed loading (21 ms; 560 keys) KeyCache cache

INFO  [main] 2020-02-17 11:55:35,746 MessagingService.java:753 - Starting Messaging Service on /172.30.55.233:7000 (br-provider)
INFO  [main] 2020-02-17 11:55:35,827 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED
WARN  [main] 2020-02-17 11:56:06,859 Gossiper.java:1425 - Unable to gossip with any seeds but continuing since node is in its own seed list
INFO  [main] 2020-02-17 11:56:06,863 StorageService.java:707 - Loading persisted ring state
INFO  [main] 2020-02-17 11:56:06,869 StorageService.java:825 - Starting up server gossip
INFO  [main] 2020-02-17 11:56:07,546 TokenMetadata.java:479 - Updating topology for /172.30.55.233
INFO  [main] 2020-02-17 11:56:07,546 TokenMetadata.java:479 - Updating topology for /172.30.55.233
3
  • 1
    check Cassandra JIRA for that message, also - you're on 3.11.2, the latest one is 3.11.6, so it maybe makes sense to upgrade
    – Alex Ott
    Feb 17, 2020 at 10:35
  • can you point to link on jira ..that would help
    – unnamed-bull
    Feb 17, 2020 at 11:19

1 Answer 1

0

This seems to be an issue with the values for column_index_cache_size_in_kb and key_cache_size_in_mb in your cassandra.yaml file. As explained here, the database will stop when the total size of all the index data for a partition is greater than the value defined in column_index_cache_size_in_kb.

If the tuning of these values don't fix the issue, you may consider using larger instances for the nodes

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.