Timeline for Large (>22 trillion items) geospatial dataset with rapid (<1s) read query performance
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 11, 2016 at 9:15 | comment | added | Azwok | I agree pre-calculated aggregates is looking very likely the way to go. The calculated averages at the highest zoom are not averaged over an area, it is the average of the values over time at 1 location. Only as it zooms out will I have separate tables/collections that will average areas to ensure no query/tile has too many location points within in it (max of 50,000-200,000). The maximum resolution of any tile is 256x256 pixels. | |
| Jul 9, 2016 at 21:40 | history | edited | ConcernedOfTunbridgeWells | CC BY-SA 3.0 |
added 217 characters in body
|
| Jul 9, 2016 at 21:40 | comment | added | ConcernedOfTunbridgeWells | If you are querying calculated average values, how many discrete locations are you taking samples at - i.e. what's the resolution of the actual bitmap at the highest level of zoom? | |
| Jul 9, 2016 at 21:36 | comment | added | ConcernedOfTunbridgeWells | I think that pre-calculating aggregates is the key - your temporal calculations can still be batched. This is how OLAP systems get fast query performance and you will probably need to take this sort of approach. Especially relevant if you can live with data that's a day old for your queries. | |
| Jul 9, 2016 at 11:14 | comment | added | Azwok | The read queries can have a bit of a delay (a day or two), so batch processing is a valid option. At any given location, a new value will only be added every 6 days at the fastest (the next satellite pass). The output on the map is not just the latest value, it is calculated based on the whole history of values at that location, e.g. it's average, or gradient, or a custom function. For more zoomed out levels, I'm already working on a clustering/pyramid structure so that I will have a table/collection with averaged values so that no tile (query) will have > 200,000 (or 50,000) location items. | |
| Jul 8, 2016 at 16:01 | history | answered | ConcernedOfTunbridgeWells | CC BY-SA 3.0 |