2

I have noticed that using nodetool snapshot on a Cassandra node causes heavy load (100% on two cores) for around 10-30 seconds. The snapshots are around 1GB in size. During that time connections to that node time out. How do I prevent this?

EDIT:

Cassandra version 2.1.1

Compression on all tables is LZ4Compressor.

Compaction strategy is STCS.

Incremental backups are off.

Compaction throughput limited to 16MB/s.

Most settings left on default.

Here a snippet of the relevant logs during a backup:

INFO  [CompactionExecutor:49922] 2017-08-21 00:15:16,672 CompactionTask.java:252 - Compacted 4 sstables to [/mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-56311,].  447 bytes to 173 (~38% of original) in 52ms = 0.003173MB/s.  3 total partitions merged to 2.  Partition merge counts were {1:1, 2:1, }
INFO  [CompactionExecutor:49922] 2017-08-21 00:15:16,674 ColumnFamilyStore.java:856 - Enqueuing flush of compactions_in_progress: 1360 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:61136] 2017-08-21 00:15:16,674 Memtable.java:324 - Writing Memtable-compactions_in_progress@399278616(170 serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:61136] 2017-08-21 00:15:16,696 Memtable.java:363 - Completed flushing /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-56312-Data.db (194 bytes) for commitlog position ReplayPosition(segmentId=1482099209638, position=32331738)
INFO  [CompactionExecutor:49922] 2017-08-21 00:15:16,697 CompactionTask.java:136 - Compacting [SSTableReader(path='/mnt/cassandra/data/mycompany_social/comment_notification_preferences-61806e80d6cf11e688f0cf4dda6df27f/mycompany_social-comment_notification_preferences-ka-1799-Data.db'), SSTableReader(path='/mnt/cassandra/data/mycompany_social/comment_notification_preferences-61806e80d6cf11e688f0cf4dda6df27f/mycompany_social-comment_notification_preferences-ka-1800-Data.db'), SSTableReader(path='/mnt/cassandra/data/mycompany_social/comment_notification_preferences-61806e80d6cf11e688f0cf4dda6df27f/mycompany_social-comment_notification_preferences-ka-1801-Data.db'), SSTableReader(path='/mnt/cassandra/data/mycompany_social/comment_notification_preferences-61806e80d6cf11e688f0cf4dda6df27f/mycompany_social-comment_notification_preferences-ka-1798-Data.db')]
INFO  [CompactionExecutor:49922] 2017-08-21 00:15:22,674 ColumnFamilyStore.java:856 - Enqueuing flush of compactions_in_progress: 180 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:61135] 2017-08-21 00:15:22,674 Memtable.java:324 - Writing Memtable-compactions_in_progress@542498536(0 serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:61135] 2017-08-21 00:15:22,704 Memtable.java:363 - Completed flushing /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-56313-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1482099209638, position=32335294)
INFO  [CompactionExecutor:49922] 2017-08-21 00:15:22,713 CompactionTask.java:252 - Compacted 4 sstables to [/mnt/cassandra/data/mycompany_social/comment_notification_preferences-61806e80d6cf11e688f0cf4dda6df27f/mycompany_social-comment_notification_preferences-ka-1803,].  1,671,656 bytes to 1,671,470 (~99% of original) in 6,015ms = 0.265010MB/s.  40,290 total partitions merged to 40,290.  Partition merge counts were {1:40290, }
INFO  [CompactionExecutor:49922] 2017-08-21 00:15:22,717 ColumnFamilyStore.java:856 - Enqueuing flush of compactions_in_progress: 1514 (0%) on-heap, 0 (0%) off-heap
  • Is there any additional information in the log files? – hot2use Aug 18 '17 at 12:18
  • to create a snapshot, cassandra creates hard links to existing sstables. this is usually a rapid procedure and shouldn't stress the CPU in that fashion. Do some of your column families have a lot of sstables? What compaction strategy are you using? Can you rule out that the node was doing GC? – LHWizard Aug 18 '17 at 13:06
  • The logs mention that SSTables are being compacted and iotop shows a lot of flushing going on. Is this because the compaction doesn't happen regularly enough? I am taking the backups every 4 hours. Inbetween I can see compactions going on. – Hubert Grzeskowiak Aug 21 '17 at 1:43
  • @LHWizard how much is "lot" of sstables? – Hubert Grzeskowiak Aug 21 '17 at 2:13
  • @hot2use pls see updated question – Hubert Grzeskowiak Aug 21 '17 at 2:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.