Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I'm working now on the implementation of the following feature: Say I have a collection in mongodb with millions of entries (very big say 100M), each entry is of type

{"uniqId", "properties"}

for example:

 {"uniqId" : "u1" , "properties" : ["p1", "p2"]}

I have a runtime component that should read the data by uniqId, and some batch process that should insert the data into this collection. There is huge amount of reads and very rare updates (once in a week)

The problem is that i'll get always the full file (that could be very big milions of lines) that i should insert.

The solution i think about is as follwing:

  • There will be unique index on "uniqId" field.
  • I'll have a timestamp field on each document
  • I can use upsert operation to override the existed entries
  • Each operation will be a bulk of some valuable amount (say 1000 documents)
  • There will be a batch process that will remove the documents with 'old' timestamp

There is another solution:

  • Each time create new collection and then swap between the new and the old one (but if i'll use sharding i saw that mongo does not support rename in that case)

Is this design seems to be OK? How it can influence on the mongo performance?

I have the mongo with replica set configuration, and i did not used sharding yet.

Appreciate any valuable ideas,

share|improve this question

migrated from stackoverflow.com Mar 2 '12 at 16:03

This question came from our site for professional and enthusiast programmers.

    
Is the file you receive sorted or not? –  Tyler Brock Mar 6 '12 at 17:33
    
The file is not sorted, it also could be couple of files. The total size could be about 100 millions of entries –  user7258 Mar 7 '12 at 19:57
    
I was hoping to clarify a few parts of your question: By "entries" are you referring to the values stored in the array (named "properties") within your document? Or are you referring to the documents in your collection? Further, when you say "I'll always get the full file", are you indicating that you'll always retrieve the entire document including the very large number of values in the "properties" array? Last: can you provide an example query and an example update? Are these arrays growing in size? How large are these arrays? How large are these documents in general? Thanks! –  Barrie Mar 19 '12 at 21:15
add comment

1 Answer 1

If the files are sorted you can just find the id or timestamp of the last inserted document in the database, then only insert the records in the files which have ids that come after that id.

If they are sorted by _id or timestamp then doing this should be trivial.

The solutions you came up with are great as well.

However, be aware that creating entirely new collections will be slower than what i proposed, and upserting will be even slower than that.

share|improve this answer
    
The problem is that i may receive the updated data for the same user, so i can not insert only new entries. In addition the key is userId is free string and of course may be with gaps –  Julias Mar 11 '12 at 12:13
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.