You might want to consider doing some Requirements Engineering for your project. When you have all the requirements listed, then you will want to write it up in a document and have companies like Oracle, Microsoft, IBM or others offer you a solution.
I don't think an open-source solution will work for the amount of data you are expecting without having adequate consultants in the background. The exception to the rule being that you are willing to go through some painful "learning-by-doing" experiences. This can be a viable solution if your "customers" (internal or external) are willing to live with some down-time during the various growth phases of the project.
I guess any database could potentially scale to some amount of data depending on the framework you would have to build around it to meet your requirements.
There are cases of large database systems which are outlined in the following documents/links. Some of the databases are reduced in size based on features the RDBMS supply:
Microsoft SQL Server
- Introduction to PostGIS (PostGIS)
GlobeXplorer is a web-based service providing online access to petabytes of global satellite and aerial imagery. GlobeXplorer uses PostGIS to manage the metadata associated with the imagery catalogue, so queries for imagery first search the PostGIS catalogue to find the location of the relevant images, then pull the images from storage and return them to the client. In building their system, GlobeXplorer tried other spatial databases but eventually settled on PostGIS because of the great combination of price and performance it offers.
Oracle Spatial and Graph
So basically the limitations are probably to be found more within the hardware you have available than within the software used. But that depends.