How to reduce Backpressure rejection while receiving WRITES and increase throughput limit of cluster

Question

After we rebuild a down node we could see some other nodes are going down and its prompting "Backpressure rejection while receiving WRITES". Nodes are behaving as down and its continuing for sometime. We want to reduce this behaviour and want to know how to redduce this. Its not an error but a symptom. How to deal with this.

Answer 1

Write backpressure happens when the cluster cannot keep up with the write throughput coming in. Fortunately, there are a few ways to deal with this.

The usual way to solve write backpressure is to add more nodes to handle the throughput. However, this only works if the write workload is using a wide range of partition keys. If the writes are only going to select number of partition keys, it might be necessary to re-model the table.

It's quite possible that the nodes were using too many resources to stream to the rebuilt node that they couldn't keep up with their normal write workloads. Of course, this also means that things should settle down once the rebuild is complete.

Another strategy is to put an event streaming product in between the app and Cassandra. Something like Apache Pulsar or Apache Kafka usually does the trick here. This way the Pulsar/Kafka topic absorbs the write pressure, allowing the level of write throughput into Cassandra to be throttled down. The important point here is that you'll need to write a consumer process to pull the data off of the topic and write it into Cassandra.

increasing resources/changing any parameter

You might have a look at the native_transport_rate_limiting_enabled setting in the cassandra.yaml. I'd definitely recommend reading the docs on it, first. But it might help.

The problem is that performance tuning to handle write backpressure often ends up moving the problem someplace else. It may help, but I suspect that it would be a temporary fix.