I have one object and a lot of data associated with that object, you can think of it as something like facebook timeline where we have user and user has a lot of posts.

I cannot store data in JSON format since editing a JSON file on each insert would make the application really slow and there is a risk of losing data.

I have to store up to one million objects in a short period of time, which means that I'll have tens of requests per second, which means that I cannot retreive, edit, and store a JSON file quickly enough, not to mention that multiple application servers might request the file at the same time and part of the data would be lost.

I could store each object separately and then use them say object_1, object_2 ... but once again I have a problem because I don't know which ID to use as I have multiple inserts at the same time.

Logging all requests and processing them one by one is not an option as the application would be too slow.

Does anyone have any idea on how to design a database to handle this type of problem?

  • I'd suggest adding some additional info such as your RDBMS version, what sort of code you are using to talk to the RDBMS, total size of the data set, availability requirements, and so on. – Eric Higgins Sep 25 '12 at 17:47
  • I'm using "real" key-value database (riak) not RDB as key-value database – Dave Sep 25 '12 at 17:51
  • added riak tag for you – Eric Higgins Sep 25 '12 at 18:07
  • You've never heard of a document database have you? Such as CouchDB, RavenDB or anything similar? – jcolebrand Sep 26 '12 at 21:32
  • You should also look at the raw layout for tweet data, as a JSON object, and tell me if that is something that would work well for you. They obviously handle thousands of tweets per second. – jcolebrand Sep 26 '12 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.