2

Cassandra works in docker containers.
Cassandra version: 3.11.10.
Cassandra-driver-core: 3.10.2.
Cassandra restart policy: unless-stopped.

Cassandra cluster: 3 nodes.
Keyspace replication factor: {'class': 'NetworkTopologyStrategy', 'upcloud_pl_waw1': 2'}.
Consistency Level: LOCAL_QUORUM.
Seeds: 1 node.

Test 1:
Firstly the test creates a table and then tries to save 4_000 records. In the meantime, there is a 1-minute power outage of one Cassandra node. When the node is up test keeps going with the saving process. After all, when you count all the records in the table, you don't have 4000 records but fewer.

Test 2:
Firstly the test creates a table and then tries to save 4_000 records. In the meantime, there is a 1-minute power outage of one Cassandra node. When the node is down you check how many records have been saved in the test log, then you count all the records with one of the working nodes. The number seems to be ok. When the crashed node is up again some records disappear.

Question:
In the first test I expected 4_000 records.
In the second test I expected that the records wouldn't disappear.
Why the records are lost though you have replication factor 2 and consistency level LOCAL_QUORUM?

import static java.util.Arrays.asList;

import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.CodecRegistry;
import com.datastax.driver.core.ConsistencyLevel;
import com.datastax.driver.core.ExecutionInfo;
import com.datastax.driver.core.QueryOptions;
import com.datastax.driver.core.Session;
import com.datastax.driver.core.policies.ConstantReconnectionPolicy;
import com.datastax.driver.core.querybuilder.QueryBuilder;
import com.datastax.driver.extras.codecs.date.SimpleTimestampCodec;
import java.util.Random;
import java.util.concurrent.atomic.AtomicInteger;
import lombok.extern.slf4j.Slf4j;
import org.junit.Before;
import org.junit.Test;

@Slf4j
public class CassandraITest {

    public static final long CASSANDRA_RECONNECT_INTERVAL_MS = 1_000;
    public static final String CASSANDRA_HOST = "cassandra1.platform,cassandra2.platform,cassandra3.platform";
    public static final String KEYSPACE = "CONSISTENCY_TEST_KEYSPACE";
    public static final String TABLE_NAME = "TEST_TABLE";

    private final AtomicInteger counter = new AtomicInteger(0);
    private Session session;

    @Before
    public void before() {
        session = Cluster.builder()
                .addContactPoints(CASSANDRA_HOST.split(","))
                .withCodecRegistry(
                        new CodecRegistry().register(SimpleTimestampCodec.instance))
                .withReconnectionPolicy(new ConstantReconnectionPolicy(CASSANDRA_RECONNECT_INTERVAL_MS))
                .withQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM))
                .build()
                .connect();

        createTable();
    }

    @Test
    public void shouldSaveAllRecords() {
        for (int i = 0; i < 4_000; i++) {
            execute(insertIntoStatement(i));
        }
    }

    private void execute(String insert) {
        try {
            session.execute(insert);
            log.info("Already executed {}", counter.incrementAndGet());
        } catch (Exception e) {
            log.error("Execution exception", e);

            try {
                Thread.sleep(100);
            } catch (InterruptedException interruptedException) {
                interruptedException.printStackTrace();
            }
            execute(insert);
        }
    }

    private String insertIntoStatement(int id) {
        return QueryBuilder.insertInto(KEYSPACE, TABLE_NAME)
                .values(
                        asList(
                                "id",
                                "name",
                                "timestamp"),
                        asList(
                                id,
                                "John" + new Random().nextInt(10),
                                System.currentTimeMillis()))
                .toString();
    }

    private ExecutionInfo createTable() {
        return session.execute(
                String.format(
                        "CREATE TABLE IF NOT EXISTS %s.%s (\n"
                                + "id bigint,\n"
                                + "name text,\n"
                                + "timestamp timestamp,\n"
                                + "primary key ((name),timestamp,id)\n"
                                + ")\n"
                                + "WITH\n"
                                + "CLUSTERING ORDER BY (timestamp ASC, id ASC)",
                        KEYSPACE,
                        TABLE_NAME)
        ).getExecutionInfo();
    }

}

2 Answers 2

2

Your test appears to be invalid. Cassandra doesn't lose data when one of the replicas is unavailable -- it's just that the replica which was down hasn't received the write.

I suspect you are running a SELECT COUNT(*) in cqlsh for verification. By default, cqlsh (or any client) runs query with a consistency of ONE. When that replica that was down comes back online, it doesn't have the data so it returns nothing which would explain why your counts are inconsistent.

In any case, 2 replicas isn't a good choice when querying with a consistency higher than *ONE because quorum of 2 replicas is also 2. This means that RF=2 cannot survive any node outage. For this reason, our recommendation which applies to almost all cases is (1) to configure 3 replicas for each DC, and (2) use LOCAL_QUORUM consistency for both reads and writes.

As a side note, counting may work for very small clusters with very small tables but it's unreliable and can affect the performance of your cluster. For details, see my explanation in Why is COUNT() bad in Cassandra?. Cheers!

3
  • Now I see that my test can survive such an outage. When I have RF=2 and I use LOCAL_QUORUM for reads and writes, I lose no data. I can't only write or read during the outage. My system survives such a situation. Is there another danger (losing data or losing the entire node) of such a configuration? Commented Jan 30, 2023 at 14:34
  • 1
    That's a misconception -- the write(s) was in fact UNsuccessful because it was not written at LOCAL_QUORUM but just one replica. One replica does not constitute high-availability and you shouldn't consider it as "survival" because if that replica goes down for whatever reason (e.g. HW failure), your data is gone. As I said, your test is invalid. Cheers! Commented Jan 30, 2023 at 23:54
  • Ok, I see. Thank you. Commented Feb 2, 2023 at 20:15
2

'upcloud_pl_waw1': 2

Why the records are lost though you have replication factor 2 and consistency level LOCAL_QUORUM?

Because quorum of 2 == 2. This configuration does not allow you to lose a node and continue normal operations. If a write happens and it cannot reach a quorum of its replicas, the write operation fails. And with 3 nodes, RF=2, and writing at *_QUORUM, one node being down for a minute would cause a non-trivial amount of write operations to be lost.

Either the RF should be increased to 3, or the operational consistency level should be lowered to *_ONE.

4
  • Actually, I realized that the records weren't lost. It depends on which node you query and on the consistency level. Previously I had the writer consistency level LOCAL_QUORUM, and the reader consistency level ONE. There were different query results on some nodes. When I changed the reader consistency level to LOCAL_QUORUM, I realized that the records aren't lost on all the nodes. I have all data. Question: Is it safe to have replication factor: 2, the reader and writer consistency level LOCAL_QUORUM and three nodes? Commented Jan 27, 2023 at 14:40
  • In such a case I'm not able to save records when any node is down, but when I receive a save exception I can try to repeat the saving procedure. Commented Jan 27, 2023 at 14:40
  • @KonradBączyński I would say “no” to that. High Availability is one of Cassandra’s main features, and using QUORUM w/ RF=2 defeats that.
    – Aaron
    Commented Jan 27, 2023 at 14:58
  • Ok, I see. Thank you. Commented Feb 2, 2023 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.