How does a database conciliate data to files on distributed filesystems?

Question

The question title might be a little misleading, but I was wondering what happens internally in a database implementation when the file used for storage are placed on a distributed filesystem (like Gluster or Ceph).

I would imagine that the database implementation keeps some data in memory and will conciliate to disk periodically. Does the database instance assume that it is the only one writing to the storage file and use this for optimizations somehow? Can someone elaborate and suggest some good materials on this topic?

My previous understanding was that this wouldn't work, but after a quick search on Google it seems that not only it works but it is fairly common (I'm not talking about explicit replication features).

Answer 1

In simple terms, the DBMS doesn't care (or in some cases even know) if the file system where its files are located is distributed or not. In a high availability scenario with shared (distributed) storage an active-passive setup is most commonly used, where only one instance of the DBMS at a time is running and has access to the database files. It's the responsibility of the cluster manager (software or human) to control DBMS instance startup and failover.

In a shared data cluster, such as Oracle RAC or Db2 pureScale, there are multiple instances of the DBMS running concurrently and accessing the same set of database files, and a special component of the cluster manages concurrent access to the files by the DBMS processes.

That "special component" is usually tightly integrated with the DBMS itself but the actual implementations differ wildly. Oracle Clusterware plays this role in an Oracle RAC; Cluster Cache Facility in Db2 pureScale; both are tightly integrated with their respective underlying shared file systems.

I'm not aware of any open source shared-data cluster databases; they all tend to choose shared-nothing architectures, which are somewhat simpler to implement and more tolerant towards low-end hardware.

Answer 2

but I was wondering what happens internally in a database implementation when the file used for storage are placed on a distributed filesystem (like Gluster or Ceph).

The db thinks you are not the smartest. Performance goes down the drain. Nothing gained.

Does the database instance assume that it is the only one writing to the storage file and use this for optimizations somehow?

Not somewhow. Every server system I know assumes, hard, it is the only one writing on the file.

Can someone elaborate and suggest some good materials on this topic?

RTFM. There probably also are books on database server design CS students are supposed to learn ;)

My previous understanding was that this wouldn't work, but after a quick search on Google it seems that not only it works but it is fairly common

20 years working with databases (not Postgres, oracle and sql server mostly) i have not ONCE seeing a distributed file system used as storage in the real world. I would say that is how common this is. Not sure what you looked up in google. But then maybe I deal more in the high performance sector where "network to storage" - unless a high speed SAN, and even then - would make people laugh. Even 10 gigbabit is only 1 gigabyte per second, which is quite low for IO for a dedicated high end server.