We have the typical problem of Twitter's timeline tweets, or Facebook profile's news: we need to show the users only the new information (storing them in a fast db), and put all the old ones somewhere else (but able to load them on request in a reasonable time).
The reason for moving the old ones is both economic (we don't need and don't want tens of GB accessed rarely occupying space on a fast production db) and for performance (tens of GB of extra information gives much overhead for indexes and queries).
Now we have all the information, recent and old, on a MongoDB hosted on MongoHQ. Currently everything is on its own specific database, that contains only one big collection. I would like to have only the most recent ones in the fast production db, and to take the older ones, such as the data older than let's say one month, and moving them in another database, or to another storage. I would set up a nightly job that will do the stuff so to maintain only recent data on the MongoDB. I don't want to manage the database by myself now, so I was looking for an hosted solution.
I think we have two options for putting old data:
- storing the data on another db hosted in the cloud, cheaper than the Mongo hosting we currently use as primary db, such as Amazon's DynamoDB, or Google Big Query. Going for this would be to transform our MongoDB collection to be saved in another format, and then to transform it back when we need to read it again
- setting up "our" server, buying for example a package for MongoDB hosted on EC2. Going for this means adding simply another db to our infrastructure, but it is a quite expensive solution as those services provide high performances database, and we don't need that.
I think that (2) is the easy solution to implement, but (1) would be much cheaper giving that the data will be rarely accessed and the new db would be used much as a storage facility. In the next months, we will definitely go for (2), as if data and usage increase I think that the best option would be to manage most of the things by ourselves instead of relying to hosting on PaaS or SaaS, but now this solution seems quite expensive.
Do you have any suggestion for a situation like this? Is there a third way that I haven't considered?