How is Cassandra data stored physically on disk?

Question

I am learning cassandra architecture and was going through how write and read is done. What I learned is that on a write the MemTable(sorted by partition key) and commitLog(append only) are written onto. Then after some time, MemTable will be flushed to the disk in a SS table and commitLog will be purged.

What I am able to visualize is that the data is being stored as a row in MemTable and SSTable. But I also know that Cassandra is a columnar DB and the columns which are not required are not touched and only selected columns are fetched. As we are storing rows, how only selected columns are touched on disk or memory and not the whole row? If we are not storing as rows, then how data looks like on disk? I am bit confused.

Answer 1

Cassandra is a columnar DB

No, it is not. Cassandra is a partitioned row store.

the columns which are not required are not touched and only selected columns are fetched. As we are storing rows, how only selected columns are touched on disk or memory and not the whole row?

So the underlying file itself is read, and the columns named in a query can be pulled back because the column name/value pairs are stored inside the rows. That makes it easy for Cassandra to just parse out whatever you ask for.

how data looks like on disk?

Consider the a table for weather sensor data with the following PRIMARY KEY definition:

PRIMARY KEY ((city, month), recorded_time)

If I select data for a particular partition, I get this:

> SELECT * FROm weather_sensor_data WHERE city='Minneapolis, MN' AND month=202210;

 city            | month  | recorded_time                   | temp
-----------------+--------+---------------------------------+------
 Minneapolis, MN | 202210 | 2022-10-17 11:30:00.000000+0000 |    1
 Minneapolis, MN | 202210 | 2022-10-17 11:25:00.000000+0000 |    1
 Minneapolis, MN | 202210 | 2022-10-17 11:20:00.000000+0000 |    2
 Minneapolis, MN | 202210 | 2022-10-17 11:15:00.000000+0000 |    2
 Minneapolis, MN | 202210 | 2022-10-17 11:00:00.000000+0000 |    2

(5 rows)

If I look at this partition in the underlying SSTable file, I see this:

% ./sstabledump ../../data/data/stackoverflow/weather_sensor_data-4b0472504e5611edbce23d839e1d28ce/nb-1-big-Data.db
    [
  {
    "partition" : {
      "key" : [ "Minneapolis, MN", "202210" ],
      "position" : 193
    },
    "rows" : [
      {
        "type" : "row",
        "position" : 232,
        "clustering" : [ "2022-10-17 11:30:00.000Z" ],
        "liveness_info" : { "tstamp" : "2022-10-17T20:17:02.876617Z" },
        "cells" : [
          { "name" : "temp", "value" : 1.0 }
        ]
      },
      {
        "type" : "row",
        "position" : 254,
        "clustering" : [ "2022-10-17 11:25:00.000Z" ],
        "liveness_info" : { "tstamp" : "2022-10-17T20:16:59.438630Z" },
        "cells" : [
          { "name" : "temp", "value" : 1.0 }
        ]
      },
      {
        "type" : "row",
        "position" : 276,
        "clustering" : [ "2022-10-17 11:20:00.000Z" ],
        "liveness_info" : { "tstamp" : "2022-10-17T20:16:51.304506Z" },
        "cells" : [
          { "name" : "temp", "value" : 2.0 }
        ]
      },
      {
        "type" : "row",
        "position" : 298,
        "clustering" : [ "2022-10-17 11:15:00.000Z" ],
        "liveness_info" : { "tstamp" : "2022-10-17T20:16:42.069043Z" },
        "cells" : [
          { "name" : "temp", "value" : 2.0 }
        ]
      },
      {
        "type" : "row",
        "position" : 320,
        "clustering" : [ "2022-10-17 11:00:00.000Z" ],
        "liveness_info" : { "tstamp" : "2022-10-17T20:16:36.085526Z" },
        "cells" : [
          { "name" : "temp", "value" : 2.0 }
        ]
      }
    ]
  },

Answer 2

The data in SSTables is laid out based on the table's schema, that is, it is laid out exactly in the order that the application reads it. Let me illustrate with an example.

Here is a simple table that contains email addresses for users:

Name	Type	Email
alice	home	alice@mail.com
alice	work	alice@acme.com
bob	work	bob@thebuilder.co
charli	home	charlix@coolmail.com
charli	other	sixteen@influenc.er

One of the application queries is to retrieve the list of emails for a given user. We want to partition the data by the users' names and store their emails in rows. This is the resulting schema for this query:

CREATE TABLE users_by_name (
    name text,
    type text,
    email text,
    PRIMARY KEY(name, type)
)

In this schema:

• the records will be retrieved by the user's name, and
• emails are clustered by type.

For the name = 'alice' partition, the data in the SSTable is laid out like this:

| partition_key='alice' | type='home' | email='alice@mail.com' | type='work' | email='alice@acme.com' |

You should be able to see that the data is laid out in the exact order that the application will be reading it, one row after another. The storage layout is optimised for the application query and is the reason reads are very fast in Cassandra.

Unlike other storage engines, there is just one disk seek required once the partition's index (disk offset) has been determined. Cassandra doesn't have to jump from one part of the disk to another to retrieve the rows in the partition since all the required data is stored (clustered) next to each other. This is the biggest reason you need to get the data model right -- design a table for each application query -- so that the table is optimised for reads.

Note that I've over-simplified it to make it easy to visualise. In reality, there are other metadata stored with each partition including the partition index, row index, write time, and expiry (TTL).

For a better understanding of the data flow, see How Cassandra reads data.

As a side note, Cassandra stores data in partitioned rows so it is row-oriented for OLTP workloads. Columnar databases are designed for analytics workloads and are completely different. Cassandra tables used to be called "column families" and this came up recently in another question so I can see why there's some confusion. Cheers!