question - I'm working on a project that involves heavy computation and will continually grow in terms of the database it works with. I want to build a small cluster (NUCs I'm thinking) the run a private cloud to run this off of. The code really will only need the CPUs for calculations, and the database will need the disk space so this architecture seems like a good way to get the most out of the hardware and easily add additional nodes as needed (as the database grows and/or more models require more computation).

I'm confident on my coding end for this and I've basically already built the database structure in Postgres - what I'm struggling with is whether or not I can scale this database out horizontally or not. I don't need full replication (though partial would be nice so one failed node doesn't result in data not being available), I just really want to be able to scale out the size of this thing and easily add hardware and/or more VMs as needed. I've seen stuff like pg-pool and pg_shard - not sure if that's what I need - just want to be able to take my existing database structure and spin up additional servers with additional disk space to house the additional data as needed. Any ideas?

  • 1
    Magic? There's nowhere near enough info here. Does the data naturally shard/partition, such that there are few or no inter-dependencies when sharded on a particular key, and you can do your computation without referring to other shards much or at all? i.e. is this data set, for this computation, embarrassingly parallel or close? Unless it is, do not assume sharding/horizontal scaling is best, crosstalk between nodes and node management quickly eats up the benefits, plus the complexity is huge and current tooling sucks. Look into postgres-XL and CitusDB if you must do this. Commented Dec 17, 2015 at 1:49


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.