0

I have a mongo database collection, that I want to modify. I would like to add new indices (text and geo) to the existing data, and remove unnecessary fields. What would be the best way to do this? There are about 20 million records in my data set.

closed as unclear what you're asking by RolandoMySQLDBA, ypercubeᵀᴹ, Philᵀᴹ, James Anderson, RLF Mar 14 '16 at 12:55

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Why don't you forget the number of records for now and concentrate on just ONE simple set of index + Val? Forget about what type the Val is and how complicated it might be in your case. Upsert it and than think about 2 or 2 bil – ninjabber Mar 12 '16 at 0:59
3

Your title and question have slightly different goals (copy vs. modify).

To copy a collection between databases you can use the cloneCollection method:

https://docs.mongodb.org/manual/reference/method/db.cloneCollection/

Once you have a collection in the new database that you want to modify you are able to create all the indexes you want to the existing data.

Some of the considerations to be aware of based on the larger data size of the collection is whether you want to block other activities for the creation of the index. By sending the creation to the "background", you can prevent this:

https://docs.mongodb.org/manual/reference/method/db.collection.createIndex/#options

The other issue, you might want to be aware of, is that index builds in the background generally take longer to build. One option is, presumably if you are running a replica set, is to build the index on one node at a time, after you remove each node from the replica set. You can build the indexes once the node is out, and then put it back into the replica set, and move on to the next node.

As far as removing unnecessary fields, you can do that as you go, and just process through the data in batches. Even if you do it in one query, if it dies, MongoDB will still maintain all the data you already updated, so it can pick up where it left off removing fields that exist in the remaining documents.

Not the answer you're looking for? Browse other questions tagged or ask your own question.