Why does reading from system.local in Cassandra 4 involve 3 nodes when replication is LocalStrategy?

Question

After upgrading to Cassandra 4, I noticed the increased latency of this very simple query:

cqlsh> SELECT * FROM system.local;

Here is the tracing:

 activity                                                                                                                        | timestamp                  | source       | source_elapsed | client
---------------------------------------------------------------------------------------------------------------------------------+----------------------------+--------------+----------------+--------------
                                                                                                              Execute CQL3 query | 2024-04-09 07:14:11.599000 | 10.252.241.8 |              0 | 10.252.241.8
                                                     READ_REQ message received from /10.252.241.8:7000 [Messaging-EventLoop-3-3] | 2024-04-09 07:14:11.599000 |  10.254.27.3 |             16 | 10.252.241.8
                                                                         Executing single-partition query on roles [ReadStage-2] | 2024-04-09 07:14:11.599001 |  10.254.27.3 |            127 | 10.252.241.8
                                                                                      Acquiring sstable references [ReadStage-2] | 2024-04-09 07:14:11.599002 |  10.254.27.3 |            189 | 10.252.241.8
                                         Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones [ReadStage-2] | 2024-04-09 07:14:11.599003 |  10.254.27.3 |            215 | 10.252.241.8
                                                            Key cache hit for sstable 3ger_0vam_55eds2f0apl1n3stpf [ReadStage-2] | 2024-04-09 07:14:11.599004 |  10.254.27.3 |            236 | 10.252.241.8
                                                                         Merged data from memtables and 1 sstables [ReadStage-2] | 2024-04-09 07:14:11.600000 |  10.254.27.3 |            287 | 10.252.241.8
                                                               Parsing SELECT * FROM system.local; [Native-Transport-Requests-1] | 2024-04-09 07:14:11.600000 | 10.252.241.8 |            100 | 10.252.241.8
                                                                            Read 1 live rows and 0 tombstone cells [ReadStage-2] | 2024-04-09 07:14:11.600001 |  10.254.27.3 |            309 | 10.252.241.8
                                                                               Preparing statement [Native-Transport-Requests-1] | 2024-04-09 07:14:11.600001 | 10.252.241.8 |            165 | 10.252.241.8

I hid most of the lines, but the tracing is pretty long and involves 3 nodes. Given that the system keyspace uses LocalStrategy, I thought that the query will use the local data only.

cqlsh> describe keyspace system; 

CREATE KEYSPACE system WITH replication = {'class': 'LocalStrategy'}  AND durable_writes = true;

And it indeed behaves the way I expected in Cassandra 3: reads from the local node only, low latency.

Am I missing something? Maybe there is some special setting for this new behavior? Thanks

Answer 1

It seems like you misunderstood the trace output. The read request on system.local only involved the local node (10.252.241.8). Here are the relevant details:

 activity                                                          | source       
-------------------------------------------------------------------+--------------
                                                Execute CQL3 query | 10.252.241.8
 Parsing SELECT * FROM system.local; [Native-Transport-Requests-1] | 10.252.241.8
                 Preparing statement [Native-Transport-Requests-1] | 10.252.241.8

All the other activity in the trace relate to an authentication check because you have it enabled on the cluster:

 activity
---------------------------------------------------------
 Executing single-partition query on roles [ReadStage-2]

You stated that 3 nodes were contacted which indicates to me that (1) the system_auth keyspace has a replication factor of 3, and (2) you are using the default cassandra superuser.

When you authenticate with the default cassandra role, the read request to validate the credentials are performed with a consistency level of QUORUM so it needed to contact 2 other replicas (in addition to the local node). For this reason, we recommend ONLY using the default superuser to provision new roles (including other superusers). Once you have provisioned new roles, disable the default superuser and never use it again. Cheers!