Timeline for Large (>22 trillion items) geospatial dataset with rapid (<1s) read query performance
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 11, 2016 at 8:05 | vote | accept | Azwok | ||
| Jul 11, 2016 at 9:49 | comment | added | usr | I'm not sure you need any spatial features at all. You can compute all of that in the app. You just need the ability to query all data for a rectangle. This can be done by manually splitting the globe into a grid (or multiple resolution grids). Your DB does not need to support spatial I think. | |
| Jul 11, 2016 at 9:48 | comment | added | usr | Note, that you can run multiple shards on the same mongo instance. You can "overshard". That way you can balance the servers out. | |
| Jul 11, 2016 at 9:45 | comment | added | usr | Blobs are trivial to use. The complexity will arise from you needing to implement database features such as serialization, queries, transactions, backups, HA, DA. This is all doable but maybe not wise. Maybe you can store the blobs in a Postgres table. That automates all of that except serialization and query. Perf could be better than blob storage and maybe it's even cheaper. Blobs and VMs are not charged by cost, they have a nice margin (proof: my local webhoster charges 3-5x less for the same compute power than cloud does. This implies high cloud margins). | |
| Jul 11, 2016 at 9:35 | comment | added | Azwok | The sharding by region is the approach I have been looking at with MongoDB, and with the timely release of MongoDB Atlas, I am currently leaning in that direction (using pre-computed aggregated values). At the moment I am unsure how many replica/shard servers I would need, so costing may become an issue. Your proposal of using BLOB storage is also interesting, and you are the second person to propose it. However, using BLOBs is completely new to me, so I need to read into it further, any useful sources you know of? Thanks for the response. | |
| Jul 9, 2016 at 19:52 | history | answered | usr | CC BY-SA 3.0 |