4

I'm working now on the implementation of the following feature: Say I have a collection in mongodb with millions of entries (very big say 100M), each entry is of type

{"uniqId", "properties"}

for example:

 {"uniqId" : "u1" , "properties" : ["p1", "p2"]}

I have a runtime component that should read the data by uniqId, and some batch process that should insert the data into this collection. There is huge amount of reads and very rare updates (once in a week)

The problem is that i'll get always the full file (that could be very big milions of lines) that i should insert.

The solution i think about is as follwing:

  • There will be unique index on "uniqId" field.
  • I'll have a timestamp field on each document
  • I can use upsert operation to override the existed entries
  • Each operation will be a bulk of some valuable amount (say 1000 documents)
  • There will be a batch process that will remove the documents with 'old' timestamp

There is another solution:

  • Each time create new collection and then swap between the new and the old one (but if i'll use sharding i saw that mongo does not support rename in that case)

Is this design seems to be OK? How it can influence on the mongo performance?

I have the mongo with replica set configuration, and i did not used sharding yet.

Appreciate any valuable ideas,

migrated from stackoverflow.com Mar 2 '12 at 16:03

This question came from our site for professional and enthusiast programmers.

  • Is the file you receive sorted or not? – Tyler Brock Mar 6 '12 at 17:33
  • The file is not sorted, it also could be couple of files. The total size could be about 100 millions of entries – user7258 Mar 7 '12 at 19:57
  • I was hoping to clarify a few parts of your question: By "entries" are you referring to the values stored in the array (named "properties") within your document? Or are you referring to the documents in your collection? Further, when you say "I'll always get the full file", are you indicating that you'll always retrieve the entire document including the very large number of values in the "properties" array? Last: can you provide an example query and an example update? Are these arrays growing in size? How large are these arrays? How large are these documents in general? Thanks! – Barrie Mar 19 '12 at 21:15
1

If the files are sorted you can just find the id or timestamp of the last inserted document in the database, then only insert the records in the files which have ids that come after that id.

If they are sorted by _id or timestamp then doing this should be trivial.

The solutions you came up with are great as well.

However, be aware that creating entirely new collections will be slower than what i proposed, and upserting will be even slower than that.

  • The problem is that i may receive the updated data for the same user, so i can not insert only new entries. In addition the key is userId is free string and of course may be with gaps – Julias Mar 11 '12 at 12:13
0

From my experience - unique index on collection with 100++ mln documents drops insert performance to unreasonable 100-300 inserts/second. Tried both Mongo 3.2 and Postgres.

Checking uniqueness in importing application, say java with HashSet might be several times faster, but will consume several dozens of GB (RAM).

Insert speed with 2-4 ordinary indexes of 2000-3000 inserts/second is achievable on single server, so it's up to you to decide is 10-15 hours is fine or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.