Say I have 1 type of record, object, with title/slug/description fields, but I have trillions of them. Well, I start off with 1,000 records on day one, by end of year one I have 1M records, by year 2 1B records and by year 5 1T records (who cares really what the growth is like). What is a sharding scheme which would allow for the limitless growth of such a table, without requiring rewriting the shard table data when we reach certain thresholds.

For example, using a hash key bucketing system for sharding won't solve the problem. You might start with 2 shards for the 1 table, and bucket hash keys > x into table 1 and < x into table 2. Then you run out of space and need to add shard 3. Now you can't use the same hashing function, or you need to redistribute all your data from shards 1 and 2 now evenly across shards 1 2 and 3 and can then use the same hashing function again. So that won't work.

Instead, what will allow for adding 1 table at a time, yet you don't have to redistribute the records like that? That also ideally doesn't cause a bottleneck or central point of failure, such as having a lookup table for where to have things go. But if that is the only solution, that is good to know too.

One naïve solution would be to account for the possibility of your growth from the beginning, and create let's say 10,000 shards up front. But that is a waste for a while until you get more data, so would like to avoid that if possible too.

Essentially what I'm beginning to imagine, is, start with 2 empty cups/glasses. Fill it up, when it reaches capacity start filling up cup 2, and add cup 3. When cup 2 fills, start filling cup 3 and add cup 4, etc. But what would make this possible, it doesn't seem easily possible?

That wouldn't work, because given an ID you wouldn't know what table to search for the record!

  • care to add what DBMS you're about to use?
    – vautee
    Jan 15 at 12:45
  • I am going to use postgres.
    – Lance
    Jan 15 at 12:50
  • Does this answer your question? Can you help me with my capacity planning?
    – djdomi
    Jan 16 at 12:54
  • 1
    I think a relational database isn't the best choice for this. But the sharding/partitioning scheme will depends greatly on what you intend to do with it. Do you need fast retrieval? Do you need to get rid of old data quickly? Are you going to retrieve single values (by primary key) all the time? Are you going to retrieve "ranges" of the data? Do you need any aggregates on the data? Is the data mostly read-only or will the existing values be changed a lot? Jan 17 at 13:20


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.