Starting point:

  • new cluster, DSE 6.7.4, C*
  • backed UP data, which was made by Opscenter snapshot backups to remote S3 bucket (1.5TB per each node)
  • restoring data to new cluster needs to use sstableloader, because new C* nodes have different ids than nodes which was backed up, so new tokens must by calculated.

Detailed info about new cluster settings:

  • resources for nodes were temporarily increased form 10CPU, 60GB RAM to 24CPU, 100GB RAM. Also GCE SSD persistent disks were sized to gain maximum throughput in write operations
  • throttled DSE stream throughput at 30MB/s
  • disabled noderepair
  • nodesync: rate_in_kb: 64 #originaly 8192
  • compaction_throughput_mb_per_sec: 1
  • concurrent_compactors: 1
  • memtable_allocation_type: offheap_objects
  • memtable_space_in_mb: 2048
  • max_heapsize_mb: 32786
  • streaming_keep_alive_period_in_secs: 72000
  • streaming_socket_timeout_in_ms: 86400000 #tested, but dse 6.7.4 won’t start up with this setting
  • dse_cassandra_streaming_connections_per_host: 3 #originaly 1

Observed failures:

  • after series of failed restores a running of C* node settings to these presented above I found that restore fails on restoring large tables (190GB)
  • at high level Opsecenter reports following message: [tablename]: Failed to run sstableloader because it could not communicate with some nodes. Please verify that all nodes in your cluster are currently healthy.
  • If i check status of cassandra nodes, they are perfectly good, no outage at all. Logs at system.log showed following messages

Streaming error occurred on session with peer xxx.xxx.xxx.xxx java.net.SocketException: End-of-stream reached at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)`

Resource utilization during restore was reasonable, you may see it bellow at print screens from one node. At fist data was transferred from AWS S3, after that sstableloader begun loading it to nodes. Restore failed at 17:35.

Network utilization Overal CPU usage CPU usage per core Memmory usage Disk read/write throughput Disk read/write latencies

This error happens one each node in cluster, one node at time. After last node failure Opsceter reports, that restore procedure failed.

Any idea what might cause this behaviour?

  • I would recommend to submit a ticket to DataStax support... – Alex Ott Aug 13 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.