0

While Replacing node with dead node getting below error. Can you please let me know the resolution.

-bash-4.2$ cassandra
-bash-4.2$ OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V
CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
INFO  [main] 2023-05-22 15:20:35,695 YamlConfigurationLoader.java:93 - Configuration location: file:/apps/opt/cassandra/cassandra/conf/cassandra.yaml
INFO  [main] 2023-05-22 15:20:36,249 Config.java:555 - Node configuration:[allocate_tokens_for_keyspace=null; allow_extra_insecure_udfs=false; allow_insecure_udfs=false; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true; cache_load_timeout_seconds=30; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=/apps/opt/cassandra/data/cdc_raw; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/apps/opt/cassandra/data/commitlogs; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@4278a03f; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_drop_compact_storage=false; enable_materialized_views=true; enable_sasi_indexes=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=<REDACTED>; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; force_new_prepared_statement_behaviour=false; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/apps/opt/cassandra/data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=null; listen_interface=eth0; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_flush_in_batches_legacy=true; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=-2147483648; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=16; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; repair_session_max_tree_depth=18; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=null; rpc_interface=eth0; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/apps/opt/cassandra/data/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.31.86.170}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_on_duplicate_row_detection=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@147ed70f; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  [main] 2023-05-22 15:20:36,250 DatabaseDescriptor.java:381 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  [main] 2023-05-22 15:20:36,251 DatabaseDescriptor.java:439 - Global memtable on-heap threshold is enabled at 242MB
INFO  [main] 2023-05-22 15:20:36,251 DatabaseDescriptor.java:443 - Global memtable off-heap threshold is enabled at 242MB
WARN  [main] 2023-05-22 15:20:36,256 DatabaseDescriptor.java:503 - Small commitlog volume detected at /apps/opt/cassandra/data/commitlogs; setting commitlog_total_space_in_mb to 2044.  You can override this in cassandra.yaml
WARN  [main] 2023-05-22 15:20:36,257 DatabaseDescriptor.java:530 - Small cdc volume detected at /apps/opt/cassandra/data/cdc_raw; setting cdc_total_space_in_mb to 1022.  You can override this in cassandra.yaml
WARN  [main] 2023-05-22 15:20:36,443 DatabaseDescriptor.java:579 - Only 6.071GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  [main] 2023-05-22 15:20:36,470 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO  [main] 2023-05-22 15:20:36,471 DatabaseDescriptor.java:781 - Back-pressure is disabled with strategy org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}.
INFO  [main] 2023-05-22 15:20:36,686 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
INFO  [main] 2023-05-22 15:20:36,940 JMXServerUtils.java:253 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
INFO  [main] 2023-05-22 15:20:36,948 CassandraDaemon.java:490 - Hostname: ip-172-31-88-213.ec2.internal
INFO  [main] 2023-05-22 15:20:36,957 CassandraDaemon.java:497 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_372
INFO  [main] 2023-05-22 15:20:36,962 CassandraDaemon.java:498 - Heap size: 970.000MiB/970.000MiB
INFO  [main] 2023-05-22 15:20:36,962 CassandraDaemon.java:503 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4556224(4449K) committed = 4587520(4480K) max = 251658240(245760K)
INFO  [main] 2023-05-22 15:20:36,969 CassandraDaemon.java:503 - Metaspace Non-heap memory: init = 0(0K) used = 22798512(22264K) committed = 23298048(22752K) max = -1(-1K)
INFO  [main] 2023-05-22 15:20:36,970 CassandraDaemon.java:503 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2604048(2543K) committed = 2830336(2764K) max = 1073741824(1048576K)
INFO  [main] 2023-05-22 15:20:36,971 CassandraDaemon.java:503 - Par Eden Space Heap memory: init = 83886080(81920K) used = 83886080(81920K) committed = 83886080(81920K) max = 83886080(81920K)
INFO  [main] 2023-05-22 15:20:36,971 CassandraDaemon.java:503 - Par Survivor Space Heap memory: init = 10485760(10240K) used = 10485760(10240K) committed = 10485760(10240K) max = 10485760(10240K)
INFO  [main] 2023-05-22 15:20:36,972 CassandraDaemon.java:503 - CMS Old Gen Heap memory: init = 922746880(901120K) used = 1966320(1920K) committed = 922746880(901120K) max = 922746880(901120K)
INFO  [main] 2023-05-22 15:20:36,976 CassandraDaemon.java:505 - Classpath: /apps/opt/cassandra/cassandra/conf:/apps/opt/cassandra/cassandra/build/classes/main:/apps/opt/cassandra/cassandra/build/classes/thrift:/apps/opt/cassandra/cassandra/lib/airline-0.6.jar:/apps/opt/cassandra/cassandra/lib/antlr-runtime-3.5.2.jar:/apps/opt/cassandra/cassandra/lib/apache-cassandra-3.11.15.jar:/apps/opt/cassandra/cassandra/lib/apache-cassandra-thrift-3.11.15.jar:/apps/opt/cassandra/cassandra/lib/asm-5.0.4.jar:/apps/opt/cassandra/cassandra/lib/caffeine-2.2.6.jar:/apps/opt/cassandra/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/apps/opt/cassandra/cassandra/lib/commons-cli-1.1.jar:/apps/opt/cassandra/cassandra/lib/commons-codec-1.9.jar:/apps/opt/cassandra/cassandra/lib/commons-lang3-3.1.jar:/apps/opt/cassandra/cassandra/lib/commons-math3-3.2.jar:/apps/opt/cassandra/cassandra/lib/compress-lzf-0.8.4.jar:/apps/opt/cassandra/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/apps/opt/cassandra/cassandra/lib/concurrent-trees-2.4.0.jar:/apps/opt/cassandra/cassandra/lib/disruptor-3.0.1.jar:/apps/opt/cassandra/cassandra/lib/ecj-4.4.2.jar:/apps/opt/cassandra/cassandra/lib/guava-18.0.jar:/apps/opt/cassandra/cassandra/lib/HdrHistogram-2.1.9.jar:/apps/opt/cassandra/cassandra/lib/high-scale-lib-1.0.6.jar:/apps/opt/cassandra/cassandra/lib/hppc-0.5.4.jar:/apps/opt/cassandra/cassandra/lib/jackson-annotations-2.13.2.jar:/apps/opt/cassandra/cassandra/lib/jackson-core-2.13.2.jar:/apps/opt/cassandra/cassandra/lib/jackson-databind-2.13.2.2.jar:/apps/opt/cassandra/cassandra/lib/jamm-0.3.0.jar:/apps/opt/cassandra/cassandra/lib/javax.inject-1.jar:/apps/opt/cassandra/cassandra/lib/jbcrypt-0.4.jar:/apps/opt/cassandra/cassandra/lib/jcl-over-slf4j-1.7.25.jar:/apps/opt/cassandra/cassandra/lib/jctools-core-1.2.1.jar:/apps/opt/cassandra/cassandra/lib/jflex-1.6.0.jar:/apps/opt/cassandra/cassandra/lib/jna-4.2.2.jar:/apps/opt/cassandra/cassandra/lib/joda-time-2.4.jar:/apps/opt/cassandra/cassandra/lib/json-simple-1.1.jar:/apps/opt/cassandra/cassandra/lib/libthrift-0.9.2.jar:/apps/opt/cassandra/cassandra/lib/log4j-over-slf4j-1.7.25.jar:/apps/opt/cassandra/cassandra/lib/logback-classic-1.2.9.jar:/apps/opt/cassandra/cassandra/lib/logback-core-1.2.9.jar:/apps/opt/cassandra/cassandra/lib/lz4-1.3.0.jar:/apps/opt/cassandra/cassandra/lib/metrics-core-3.1.5.jar:/apps/opt/cassandra/cassandra/lib/metrics-jvm-3.1.5.jar:/apps/opt/cassandra/cassandra/lib/metrics-logback-3.1.5.jar:/apps/opt/cassandra/cassandra/lib/netty-all-4.0.44.Final.jar:/apps/opt/cassandra/cassandra/lib/ohc-core-0.4.4.jar:/apps/opt/cassandra/cassandra/lib/ohc-core-j8-0.4.4.jar:/apps/opt/cassandra/cassandra/lib/reporter-config3-3.0.3.jar:/apps/opt/cassandra/cassandra/lib/reporter-config-base-3.0.3.jar:/apps/opt/cassandra/cassandra/lib/sigar-1.6.4.jar:/apps/opt/cassandra/cassandra/lib/slf4j-api-1.7.25.jar:/apps/opt/cassandra/cassandra/lib/snakeyaml-1.26.jar:/apps/opt/cassandra/cassandra/lib/snappy-java-1.1.1.7.jar:/apps/opt/cassandra/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/apps/opt/cassandra/cassandra/lib/ST4-4.0.8.jar:/apps/opt/cassandra/cassandra/lib/stream-2.5.2.jar:/apps/opt/cassandra/cassandra/lib/thrift-server-0.3.7.jar:/apps/opt/cassandra/cassandra/lib/jsr223/*/*.jar::/apps/opt/cassandra/cassandra/lib/jamm-0.3.0.jar
INFO  [main] 2023-05-22 15:20:36,976 CassandraDaemon.java:507 - JVM Arguments: [-Xloggc:/apps/opt/cassandra/logs/gc.log, -Dcassandra.replace_address=172.31.86.37, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -Xloggc:/apps/opt/cassandra/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms979M, -Xmx979M, -Xmn100M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/apps/opt/cassandra/cassandra/conf/hotspot_compiler, -javaagent:/apps/opt/cassandra/cassandra/lib/jamm-0.3.0.jar, -XX:HeapDumpPath=/apps/opt/cassandra/logs/cassandra-1684768834-pid5323.hprof, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/apps/opt/cassandra/cassandra/lib/sigar-bin, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/apps/opt/cassandra/logs, -Dcassandra.storagedir=/apps/opt/cassandra/cassandra/data]
INFO  [main] 2023-05-22 15:20:37,201 NativeLibrary.java:176 - JNA mlockall successful
WARN  [main] 2023-05-22 15:20:37,202 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations
WARN  [main] 2023-05-22 15:20:37,203 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
INFO  [main] 2023-05-22 15:20:37,205 SigarLibrary.java:44 - Initializing SIGAR library
INFO  [main] 2023-05-22 15:20:37,234 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN  [main] 2023-05-22 15:20:37,239 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
INFO  [main] 2023-05-22 15:20:37,526 QueryProcessor.java:121 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO  [main] 2023-05-22 15:20:38,809 ColumnFamilyStore.java:432 - Initializing system.IndexInfo
INFO  [main] 2023-05-22 15:20:39,977 ColumnFamilyStore.java:432 - Initializing system.batches
INFO  [main] 2023-05-22 15:20:39,997 ColumnFamilyStore.java:432 - Initializing system.paxos
INFO  [main] 2023-05-22 15:20:40,029 ColumnFamilyStore.java:432 - Initializing system.local
INFO  [SSTableBatchOpen:1] 2023-05-22 15:20:40,086 BufferPool.java:251 - Global buffer pool is enabled, when pool is exhausted (max is 242.000MiB) it will allocate on heap
INFO  [main] 2023-05-22 15:20:40,538 CacheService.java:103 - Initializing key cache with capacity of 48 MBs.
INFO  [main] 2023-05-22 15:20:40,550 CacheService.java:125 - Initializing row cache with capacity of 0 MBs
INFO  [main] 2023-05-22 15:20:40,557 CacheService.java:154 - Initializing counter cache with capacity of 24 MBs
INFO  [main] 2023-05-22 15:20:40,558 CacheService.java:165 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  [main] 2023-05-22 15:20:40,592 ColumnFamilyStore.java:432 - Initializing system.peers
INFO  [main] 2023-05-22 15:20:40,617 ColumnFamilyStore.java:432 - Initializing system.peer_events
INFO  [main] 2023-05-22 15:20:40,666 ColumnFamilyStore.java:432 - Initializing system.range_xfers
INFO  [main] 2023-05-22 15:20:40,687 ColumnFamilyStore.java:432 - Initializing system.compaction_history
INFO  [main] 2023-05-22 15:20:40,719 ColumnFamilyStore.java:432 - Initializing system.sstable_activity
INFO  [main] 2023-05-22 15:20:40,741 ColumnFamilyStore.java:432 - Initializing system.size_estimates
INFO  [main] 2023-05-22 15:20:40,757 ColumnFamilyStore.java:432 - Initializing system.available_ranges
INFO  [main] 2023-05-22 15:20:40,768 ColumnFamilyStore.java:432 - Initializing system.transferred_ranges
INFO  [main] 2023-05-22 15:20:40,783 ColumnFamilyStore.java:432 - Initializing system.views_builds_in_progress
INFO  [main] 2023-05-22 15:20:40,802 ColumnFamilyStore.java:432 - Initializing system.built_views
INFO  [main] 2023-05-22 15:20:40,813 ColumnFamilyStore.java:432 - Initializing system.hints
INFO  [main] 2023-05-22 15:20:40,830 ColumnFamilyStore.java:432 - Initializing system.batchlog
INFO  [main] 2023-05-22 15:20:40,840 ColumnFamilyStore.java:432 - Initializing system.prepared_statements
INFO  [main] 2023-05-22 15:20:40,852 ColumnFamilyStore.java:432 - Initializing system.schema_keyspaces
INFO  [main] 2023-05-22 15:20:40,869 ColumnFamilyStore.java:432 - Initializing system.schema_columnfamilies
INFO  [main] 2023-05-22 15:20:40,880 ColumnFamilyStore.java:432 - Initializing system.schema_columns
INFO  [main] 2023-05-22 15:20:40,900 ColumnFamilyStore.java:432 - Initializing system.schema_triggers
INFO  [main] 2023-05-22 15:20:40,910 ColumnFamilyStore.java:432 - Initializing system.schema_usertypes
INFO  [main] 2023-05-22 15:20:40,926 ColumnFamilyStore.java:432 - Initializing system.schema_functions
INFO  [main] 2023-05-22 15:20:40,938 ColumnFamilyStore.java:432 - Initializing system.schema_aggregates
INFO  [main] 2023-05-22 15:20:40,953 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  [main] 2023-05-22 15:20:41,168 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  [main] 2023-05-22 15:20:41,230 ColumnFamilyStore.java:432 - Initializing system_schema.keyspaces
INFO  [main] 2023-05-22 15:20:41,262 ColumnFamilyStore.java:432 - Initializing system_schema.tables
INFO  [main] 2023-05-22 15:20:41,287 ColumnFamilyStore.java:432 - Initializing system_schema.columns
INFO  [main] 2023-05-22 15:20:41,313 ColumnFamilyStore.java:432 - Initializing system_schema.triggers
INFO  [main] 2023-05-22 15:20:41,338 ColumnFamilyStore.java:432 - Initializing system_schema.dropped_columns
INFO  [main] 2023-05-22 15:20:41,363 ColumnFamilyStore.java:432 - Initializing system_schema.views
INFO  [main] 2023-05-22 15:20:41,386 ColumnFamilyStore.java:432 - Initializing system_schema.types
INFO  [main] 2023-05-22 15:20:41,407 ColumnFamilyStore.java:432 - Initializing system_schema.functions
INFO  [main] 2023-05-22 15:20:41,423 ColumnFamilyStore.java:432 - Initializing system_schema.aggregates
INFO  [main] 2023-05-22 15:20:41,449 ColumnFamilyStore.java:432 - Initializing system_schema.indexes
INFO  [main] 2023-05-22 15:20:41,466 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO  [ScheduledTasks:1] 2023-05-22 15:20:41,687 TokenMetadata.java:527 - Updating topology for all endpoints that have changed
INFO  [main] 2023-05-22 15:20:42,102 StorageService.java:773 - Populating token metadata from system tables
INFO  [main] 2023-05-22 15:20:42,180 StorageService.java:780 - Token metadata:
INFO  [pool-4-thread-1] 2023-05-22 15:20:42,343 AutoSavingCache.java:174 - Completed loading (6 ms; 4 keys) KeyCache cache
INFO  [main] 2023-05-22 15:20:42,380 CommitLog.java:169 - Replaying /apps/opt/cassandra/data/commitlogs/CommitLog-6-1684768148735.log
INFO  [main] 2023-05-22 15:20:42,438 CommitLog.java:171 - Log replay complete, 0 replayed mutations
INFO  [main] 2023-05-22 15:20:42,439 StorageService.java:773 - Populating token metadata from system tables
INFO  [main] 2023-05-22 15:20:42,471 StorageService.java:780 - Token metadata:
INFO  [main] 2023-05-22 15:20:42,640 QueryProcessor.java:174 - Preloaded 0 prepared statements
INFO  [main] 2023-05-22 15:20:42,647 StorageService.java:791 - Cassandra version: 3.11.15
INFO  [main] 2023-05-22 15:20:42,648 StorageService.java:792 - Thrift API version: 20.1.0
INFO  [main] 2023-05-22 15:20:42,648 StorageService.java:793 - CQL supported versions: 3.4.4 (default: 3.4.4)
INFO  [main] 2023-05-22 15:20:42,648 StorageService.java:795 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
INFO  [main] 2023-05-22 15:20:42,716 IndexSummaryManager.java:87 - Initializing index summary manager with a memory pool size of 48 MB and a resize interval of 60 minutes
INFO  [main] 2023-05-22 15:20:42,744 MessagingService.java:750 - Starting Messaging Service on /172.31.88.213:7000 (eth0)
INFO  [main] 2023-05-22 15:20:42,777 StorageService.java:681 - Gathering node replacement information for /172.31.86.37
INFO  [main] 2023-05-22 15:20:42,824 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED
INFO  [HANDSHAKE-/172.31.86.170] 2023-05-22 15:20:42,857 OutboundTcpConnection.java:561 - Handshaking version with /172.31.86.170
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any peers
java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1603)
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1526)
        at org.apache.cassandra.service.StorageService.prepareForReplacement(StorageService.java:682)
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:958)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:837)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:786)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786)
ERROR [main] 2023-05-22 15:21:43,862 CassandraDaemon.java:803 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1603)
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1526)
        at org.apache.cassandra.service.StorageService.prepareForReplacement(StorageService.java:682)
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:958)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:837)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:786)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786)
INFO  [StorageServiceShutdownHook] 2023-05-22 15:21:43,889 HintsService.java:210 - Paused hints dispatch
WARN  [StorageServiceShutdownHook] 2023-05-22 15:21:43,896 Gossiper.java:1728 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2023-05-22 15:21:43,897 MessagingService.java:985 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/172.31.88.213] 2023-05-22 15:21:43,898 MessagingService.java:1346 - MessagingService has terminated the accept() thread
INFO  [StorageServiceShutdownHook] 2023-05-22 15:21:44,203 HintsService.java:210 - Paused hints dispatch

Thanks!

1
  • 2
    Please do not just paste a "wall of text" and expect someone to sort it out for you. We just need the relevant part of the error message(s). Secondly, make sure it's properly formatted.
    – Aaron
    Commented May 22, 2023 at 16:24

2 Answers 2

2

This is your issue:

INFO  [HANDSHAKE-/172.31.86.170] 2023-05-22 15:20:42,857 OutboundTcpConnection.java:561 - Handshaking version with /172.31.86.170
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any peers
java.lang.RuntimeException: Unable to gossip with any peers

172.31.86.170 is likely specified in your seed list and is not able to be reached.

I advise having a look through the docs on replacing a node. Specifically, to make sure that -Dcassandra.replace_address_first_boot=<dead_node_ip> is getting set in the cassandra-env.sh.

Also, it doesn't look like an IP address is being set for this node. That's likely a problem.

0

This looks like a networking issue. The new node cannot connect to the seed node you configured:

seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.31.86.170}

You need to make sure that there is network connectivity between this node (172.31.88.213) and the seed node, making sure that there is bi-directional access on gossip port 7000.

If there is a firewall between the nodes, you need to allow TCP traffic going both ways.

Also make sure that Cassandra is running on the seed node. If 172.31.86.170 is the dead node then you won't be able to use it as a seed so you have to pick another node. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.