1

I have a 3 node Cassandra 4.0beta3 cluster that is filling up and I need to know when I should add a couple more nodes. How much data is safe to keep on one node?

The nodes have 7.2T of disk each (log is on a different disk). Between 3.5T and 4T of free space each. The largest file under /var/lib/cassandra/data/ is 858G (approx 800G on the other nodes).

Thanks.

Improve this question

2 Answers 2

Reset to default
1

So the best practice is not to let each node fill more than 50% of the disk. That's because (Size Tiered) compaction can use double the space of the SSTable files that it's running on. Therefore compaction running on your 850GB file will temporarily and abruptly use another 850GB of disk (during the process).

Also makes sense to increase nodes by a factor of your RF. So if you have 3 nodes, I'd recommend going to 6 for even data distribution.

I'd also recommend upgrading to the latest 4.0 version. Just me, but I wouldn't run a beta version in prod.

I could put bigger disks in the machines. Is that a good idea?

Sure, that would do it, too. However, the best, recommended practice is not to go too much more than 2TB per Cassandra node. I do know of folks who have gone beyond that, and run just fine. But anything that requires streaming data between nodes (ex: repair, adding or remove nodes) takes a looooooong time.

Improve this answer
4
  • 1
    Tx. I used the beta in prod because I needed the zstd compression. I has worked perfectly for 2 years and I am very nervous about upgrading to 4.0 :( If it ain't broke don't fix it and all that.
    – David Tinker
    Mar 14 at 18:41
  • 1
    Actually, you should upgrade to 4.1 already!
    – Madhavan
    Mar 14 at 23:06
  • On a related note: I could put bigger disks in the machines. Is that a good idea? The cluster isn't very busy so I don't need more CPU, just more space.
    – David Tinker
    Mar 22 at 15:58
  • @DavidTinker Edit made!
    – Aaron
    Mar 22 at 16:02
0

Our general recommendation is to maintain a node density of about 1TB of data per node for optimal performance.

But just as important, the 1TB node density is the sweet spot when it comes to operational management. The main benefits are:

  1. Backups take a reasonable time to complete.
  2. It is less expensive to repair a node which has 500GB-1TB of data compared to a node which has 2TB, for example.
  3. It is faster to replace nodes with less data -- imagine the impact of streaming 2TB of data to bootstrap a replacement node during peak traffic.
  4. Similarly, adding/decommissioning nodes with less data is more manageable.

Of course, your miles may vary. There are situations where having super-dense nodes are valid but I would argue that if I were in the hot-seat during an outage, I would want to stack the odds in my favour such that "smaller" nodes will take me less than time to recover than dense nodes.

As a side note, I also wouldn't run a beta-release in production. There have been 12 releases in the 4.0 version since 4.0-beta3 which amount to roughly 400 fixes for known issues. That is a lot of known bugs which have been patched since. Not taking advantage of those fixes is just way too much risk in my opinion. Cheers!

Improve this answer
2
  • Thanks. I did upgrade to 4.0.5 before starting the expansion. That was the highest version available in apt for Ubuntu 18.04.
    – David Tinker
    Apr 13 at 15:28
  • I should add that the cluster is running on leased real hardware. So getting new machines isn't instant. So adding having to expand the cluster less often is nice.
    – David Tinker
    Apr 13 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.