2

I am experiencing a system freeze in IOWAIT on my QuasarDB cluster.

I am continuously ingesting data at 10 GBit/s using the QuasarDB batch API. After a couple of hours, I have all the qdb threads in IOWAIT and the whole system is unresponsive.

Before the system becomes unresponsive, I observe a spike of activity.

When the qdb is in the IOWAIT state, the whole system seems frozen: I can log in, but copying files is extremely slow.

I initially thought that I was over my limit of IOPS, but I have 8,000 IOPS provisioned, and the spikes are below 1,000 IOPS.

The system is not paging out, and the volumes are not full.

The cluster is 4 AWS EC2 c5n.18xlarge nodes, data is stored on mounted EBS GP3 volumes.

Any clue about what I should look at to find the root cause of the problem would be welcomed.

1 Answer 1

1

(Disclaimer: I work at QuasarDB)

Because you are using the newer gp3 EBS volume types, and you are reporting a system-wide instability of the system (rather than just the QuasarDB process), I believe this is related to the EBS volume types, rather than something related to QuasarDB.

We have observed stability issues with gp3 volumes under high pressure ourselves, with a system getting stuck. To the best of our understanding, this is related to an EBS host-level driver. GP3 instability have been report by others as well.

What I would recommend is to use the older GP2 EBS volume types, and see if that solves the issue. If you would still like to achieve high-throughput, you can combine multiple GP2 EBS volumes using mdraid as recommended by AWS themselves.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.