cassandra-stress shows low read ops when node down (blocked on firewall)

Question

cassandra version: 3.11.13
openjdk version "1.8.0_342"
I did tests on small cluster 2 nodes, but also on a cluster with 6 nodes. Result is same.

I'm doing simple test

cassandra-stress read n=1000000 cl=LOCAL_ONE no-warmup -schema "replication(strategy=NetworkTopologyStrategy,dc1=2)" -mode native cql3  -rate threads=32  -node 192.168.1.131

Test1 shutdown one node

service cassandra stop

all looks fine - after short period of time cassandra-stress shows that op/s restored back. No any problems, everything as I expected.

Test2 Instead of shutdown a process, I blocking network on one node, like this

sudo iptables -P OUTPUT DROP && sudo iptables -P INPUT DROP

then I see extremely low op/s in output exactly for 15 minutes, after this time op/s restored.

timing:
17:10 - blocked traffic on one node
17:25 - read ops restored themself

In a same time other nodes reporting that node is DN exactly after blocking on firewalls.
please help me to understand what happening.

Answer 1

It isn't clear to me exactly what is unexpected in the scenario you described and what you need help with.

I suspect you're running a test on a very small cluster that probably only has 2 or 3 nodes. When you bring a node down, it is expected that the throughput will drop since you have reduced the capacity of the cluster.

If you need a bit more detail, then you will need to provide more information in your original question including details of your cluster configuration + steps on how to replicate your issue. Cheers!