-2

I'm going to have a very large data set (a few hundred terabytes big eventually), but it is very well structured, relatively simple data involving lat, long points (which is why I want some GIS compatibility).

From what I can tell PostGIS can only handle 32TB or so, and I'm unsure what Oracle Spatial can scale to. Amazon's DynamoDB can go up to petabyte scale, but I've read complaints about how it can become rather complicated and be avoided without having a good reason to use it. And using NoSQL for such structured data seems wrong, but I can't find other alternatives for this size. As far as costs go, yeah it's going to be expensive but let's assume that's not a huge problem. Retrieval speed for spatial and temporal queries is the main deciding factor.

New contributor
Dylan is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

closed as off-topic by Evan Carroll, mustaccio, Marco, Michael Green, McNets Feb 12 at 11:37

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Shopping list question - questions about which tool, library, product or resource you should use are off-topic here because they quickly become obsolete and often are just about the preferences of the answerer. If you have an issue with or a question about a specific tool, please revise your question to conform to that scope." – Evan Carroll, Marco, McNets
If this question can be reworded to fit the rules in the help center, please edit the question.

3

You might want to consider doing some Requirements Engineering for your project. When you have all the requirements listed, then you will want to write it up in a document and have companies like Oracle, Microsoft, IBM or others offer you a solution.

I don't think an open-source solution will work for the amount of data you are expecting without having adequate consultants in the background. The exception to the rule being that you are willing to go through some painful "learning-by-doing" experiences. This can be a viable solution if your "customers" (internal or external) are willing to live with some down-time during the various growth phases of the project.

I guess any database could potentially scale to some amount of data depending on the framework you would have to build around it to meet your requirements.

There are cases of large database systems which are outlined in the following documents/links. Some of the databases are reduced in size based on features the RDBMS supply:

Microsoft SQL Server

PostgreSQL

PostGIS

  • Introduction to PostGIS (PostGIS)

    GlobeXplorer is a web-based service providing online access to petabytes of global satellite and aerial imagery. GlobeXplorer uses PostGIS to manage the metadata associated with the imagery catalogue, so queries for imagery first search the PostGIS catalogue to find the location of the relevant images, then pull the images from storage and return them to the client. In building their system, GlobeXplorer tried other spatial databases but eventually settled on PostGIS because of the great combination of price and performance it offers.

Oracle Spatial and Graph

So basically the limitations are probably to be found more within the hardware you have available than within the software used. But that depends.

Not the answer you're looking for? Browse other questions tagged or ask your own question.