- new cluster, DSE 6.7.4, C* 126.96.36.1994
- backed UP data, which was made by Opscenter snapshot backups to remote S3 bucket (1.5TB per each node)
- restoring data to new cluster needs to use sstableloader, because new C* nodes have different ids than nodes which was backed up, so new tokens must by calculated.
Detailed info about new cluster settings:
- resources for nodes were temporarily increased form 10CPU, 60GB RAM to 24CPU, 100GB RAM. Also GCE SSD persistent disks were sized to gain maximum throughput in write operations
- throttled DSE stream throughput at 30MB/s
- disabled noderepair
- nodesync: rate_in_kb: 64 #originaly 8192
- compaction_throughput_mb_per_sec: 1
- concurrent_compactors: 1
- memtable_allocation_type: offheap_objects
- memtable_space_in_mb: 2048
- max_heapsize_mb: 32786
- streaming_keep_alive_period_in_secs: 72000
- streaming_socket_timeout_in_ms: 86400000 #tested, but dse 6.7.4 won’t start up with this setting
- dse_cassandra_streaming_connections_per_host: 3 #originaly 1
- after series of failed restores a running of C* node settings to these presented above I found that restore fails on restoring large tables (190GB)
- at high level Opsecenter reports following message: [tablename]: Failed to run sstableloader because it could not communicate with some nodes. Please verify that all nodes in your cluster are currently healthy.
- If i check status of cassandra nodes, they are perfectly good, no outage at all. Logs at system.log showed following messages
Streaming error occurred on session with peer xxx.xxx.xxx.xxx java.net.SocketException: End-of-stream reached at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)`
Resource utilization during restore was reasonable, you may see it bellow at print screens from one node. At fist data was transferred from AWS S3, after that sstableloader begun loading it to nodes. Restore failed at 17:35.
This error happens one each node in cluster, one node at time. After last node failure Opsceter reports, that restore procedure failed.
Any idea what might cause this behaviour?