0

I want to set up a database in a high durability set-up on Azure. I've previously relied on DB-as-a-service offerings, but can't do that in this case, so I'd like your feedback on the plan below. Is this enough to ensure reliable storage of data?

  1. An Azure Web App takes in metric data from the web, does some minor processing and sampling, and sends the data in batches to VM2.
  2. VM2 runs the Clickhouse database, and stores data on an Azure Managed Disk
  3. Some periodical job takes snapshots of the disk using Clickhouse built-in backup functionality and stores them to cold storage

The periodical backup is meant to mitigate human error, i.e. accidentally running "DROP TABLE xx" on the wrong data.

The big question is if managed disks are an acceptable substitute for database replication, to ensure data durability. Azure Managed Disks are advertised as being very durable forms of storage, with built in triple-redundant replication. They are advertised as good for database use. It seems that this should be enough to take away any concerns of data loss due to hardware failure. Is this correct? Do you see any potential problems with this?

The recovery plan is that if VM2 fails, some monitoring process catches this and spins up a new VM2 instance attached to the same managed disk. The Web App similarly restarts if it fails.

I understand that this setup isn't high-availability, if a VM fails there will be some window of time before it is able to store new data. This is acceptable to me. But I want to ensure that data that gets stored will not be lost, i.e. is durably stored with very high probability. Is this enough to ensure that? Do you see any problems?

1 Answer 1

2

But I want to ensure that data that gets stored will not be lost, i.e. is durably stored with very high probability.

Yes. It could happen, but it's an extremely low-probability event, and there's many other low-probability events that are more likely. It's vastly more likely that your data will be lost due to a guest OS driver problems, human error, or malware than Azure would lose data on a disk.

See:

LRS provides at least 99.999999999% (11 nines) durability of objects over a given year.

Azure Storage redundancy

And whatever that means, you can bet that a lot of things would have to go very wrong to lose a disk.

If you are happy recovering from backups for those other scenarios, you should be fine.`

3
  • Thanks. That probability of data loss is certainly low enough by a good margin. What I'm concerned with is whether or not a Managed Disk will behave towards the OS exactly like a locally attached disk, or if there is extra risk introduced there. How would you assess the probability that data loss happens because Clickhouse makes an assumption about the disks behavior, that would be true for all locally attached disks, but is false for a Managed Disk? Can we be reasonably sure that the managed disk will behave towards the OS in a way that is fully compatible with the behavior of a local disk? Commented Apr 5, 2021 at 12:34
  • Azure disks behave like local or attached disks. To be safe you should disable write caching on the disk. Applications can request that writes "write through" any cache, and most databases should do this. But in case clickhouse doesn't, it's safter to disable write caching. Commented Apr 5, 2021 at 12:37
  • Excellent. Thanks for the help David! Commented Apr 5, 2021 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.