8

I am using mongodb to persist a very big-size (90G), which has nearly 40,000,000 items. I read and parse this file and insert all items into mongodb (my programming language is perl, batch_insert, and I map one item to one mongodb document).

Before I insert, I have already pre-created indexes (about 10 index keys). I find that insert speed cannot meet my need (200 to 400 items per second). I know that too many index keys will definitely slow down my insert, especially when the size of collection becomes quite big. So I wonder if I can index them after I have dumped all the data into db.

Can anyone tell me if this way is available, and if it can definitely save my time?

Improve this question
  • Why don't you just try it out? – Mat Dec 22 '13 at 10:05
  • Well,it is 90G data.I have to make a very good research before I dump it to db.A try is not allowed on such big-size data! – wuchang Dec 23 '13 at 8:40
  • 1
    Try with a subset. You'll never know how long it takes on your hardware if you don't test anyway. – Mat Dec 23 '13 at 8:42
7

Yes, you can index them after you have imported (there will then only be the default _id index on the collection). This is also recommended because the resulting indexes will be more compact and more efficient (for similar reasons foreground vs background indexing is preferred if you can afford to do it). It will take some time to complete though, especially with 10 indexes to build.

To build after the import, simply do not define any indexes until after your import is complete, then use the ensureIndex() command to create the required indexes afterwards (with the usual caveat that such index creation will be resource intensive). For more information:

http://docs.mongodb.org/manual/core/index-creation/

Improve this answer
2

I was having the same problem with a big collection. After more than two days of heavy importing data process what i suggest to do is the next in order to get the collection up and running as soon as possible:

  1. Create the collection empty without index and import just the data.
  2. Select the single field indexes and create them.
  3. Create all the multiple field indexes if you have. (This was specially slow for me having indexes with more than 5 fields, but with single or two fileds are pretty fast).
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.