What is replica filtering protection in Cassandra?

Question

There are instances where operators see the following warning in the Cassandra logs:

WARN ... Replica filtering protection has cached over 2000 rows during query \
  [SELECT ... FROM ... WHERE ...]. (See 'cached_replica_rows_warn_threshold' in cassandra.yaml.)

Or in the worst case, an error:

ERROR ... Replica filtering protection has cached over 32000 rows during query \
  [SELECT ... FROM ... WHERE ...]. (See 'cached_replica_rows_fail_threshold' in cassandra.yaml.)

What do these log entries mean?

Answer 1

Replica filtering protection is a mechanism that ensures results from stale replicas for read consistency levels higher than ONE or LOCAL_ONE do not violate the required consistency. This applies to (a) secondary index queries, or (b) queries using ALLOW FILTERING.

You are getting the warning because your nodes are out-of-sync, most likely because they are overloaded and cannot keep up with writes so have lots of dropped mutations. For details, see CASSANDRA-8272 and CASSANDRA-15907.

You need to repair the problematic table to synchronise the replicas. You should also consider increasing the capacity of your cluster by adding more nodes so you don't end up with thousands of dropped mutations.

As a side note, you should not use ALLOW FILTERING without restricting the query to a single partition. Otherwise, it will not perform well and its behaviour can be unpredictable. Cheers!

---

Republished from DataStax Community.