QuasarDB cluster stuck in IOWAIT on AWS EC2 instances

Question

I am experiencing a system freeze in IOWAIT on my QuasarDB cluster.

I am continuously ingesting data at 10 GBit/s using the QuasarDB batch API. After a couple of hours, I have all the qdb threads in IOWAIT and the whole system is unresponsive.

Before the system becomes unresponsive, I observe a spike of activity.

When the qdb is in the IOWAIT state, the whole system seems frozen: I can log in, but copying files is extremely slow.

I initially thought that I was over my limit of IOPS, but I have 8,000 IOPS provisioned, and the spikes are below 1,000 IOPS.

The system is not paging out, and the volumes are not full.

The cluster is 4 AWS EC2 c5n.18xlarge nodes, data is stored on mounted EBS GP3 volumes.

Any clue about what I should look at to find the root cause of the problem would be welcomed.

Answer 1

(Disclaimer: I work at QuasarDB)

Because you are using the newer gp3 EBS volume types, and you are reporting a system-wide instability of the system (rather than just the QuasarDB process), I believe this is related to the EBS volume types, rather than something related to QuasarDB.

We have observed stability issues with gp3 volumes under high pressure ourselves, with a system getting stuck. To the best of our understanding, this is related to an EBS host-level driver. GP3 instability have been report by others as well.

What I would recommend is to use the older GP2 EBS volume types, and see if that solves the issue. If you would still like to achieve high-throughput, you can combine multiple GP2 EBS volumes using mdraid as recommended by AWS themselves.