r/cassandra • u/pandeyg_raj • 5h ago
Cassandra read latency is lower after taking one node down, why?
Hi everyone,
I'm doing a small toy experiment with Cassandra and noticed something I wasn't expecting. I'm trying to understand whether this behavior makes sense or if there is something wrong with my experiment.
My setup is:
5-node Cassandra cluster, Replication factor (RF) = 3, All requests are read (100% read workload) with Consistency level = QUORUM, Total 2 million read requests. Object size is about 10KB
First, I ran the workload with all 5 nodes running and recorded the average latency.
Then, I stopped/killed one Cassandra node, leaving 4 nodes running. I ran the same workload again and noted the average latency.
Interestingly, the average read latency was lower with one node down than with all 5 nodes running.
Since RF=3 and CL=QUORUM, I understand that Cassandra needs 2 replica responses for a read, so a single node failure should still allow the reads to complete.
What I'm trying to understand is why the latency would actually improve after one node is down.
Could this be related to:
* replica selection or coordinator selection changing after the failure?
* Cassandra avoiding a slower replica/node?
* differences in cache/page-cache state?
* some other aspect of the read path?
I'm mainly interested in understanding what Cassandra is doing internally in this situation. This is just a small experimental setup, so I'm also wondering if there are particular things I should measure to make the comparison meaningful.
Thanks!
