0

I have a kubernetes cluster on bare metal with 3 machines having 8Gb of ram each. All my apps and a mongodb 4.0.9 replicaset runs on it. There is an import program that:

  1. Download a 8Gb mongo dump from external source and restore it in fresly created database A. There is one collection whitout indexes.
  2. Fully browse the restored collection with one find({}) and 1M next().
  3. Emit amqp messages for each doc, then messages are stored in mongo database B, C, D.
  4. Drop the database A.

Part 2 use a lot of ram.

How should I configure mongo database A (and let database B normal) to reduce at max the footprint of the import operation? (Because more important tasks are running in the cluster)

For example, can I configure mongo to not create cache for the temporary collection?

7
  • 1
    Welcome to the site, what is the MongoDB version(x,y,z)? – Md Haidar Ali Khan Apr 12 '19 at 21:44
  • Are you trying the mongorestore operation consumes more RAM ? – Mani Apr 12 '19 at 21:44
  • Thanks you. I edited answer – Slim Apr 12 '19 at 21:54
  • With the current description it sounds like you have a program running on a local machine that connects to a remote 3-member replica set. If that isn't correct, can you clarify how your program relates to MongoDB? For example, are you downloading a backup from a replica set and restoring to a local mongod instance? Also, are you doing any operations on the data (such as aggregation) when you browse the collection or just fetching all documents? Lastly, what is your goal as far as restoring, reading, and then dropping the collection. Is that to verify a backup? – Stennie Apr 12 '19 at 23:03
  • Thanks for your interest Stennie. I have added precisions. – Slim Apr 13 '19 at 6:30
0

Starting in MongoDB 3.4, the default WiredTiger internal cache size is the larger of either:

  • 50% of (RAM - 1 GB),
  • or 256 MB.

For example, In your case from the 8GB of RAM the WiredTiger cache will use 3.5GB of RAM (0.5 * (8 GB - 1 GB) = 3.5 GB).

This can be reduced to for example to 1GB by using the following admin command only when the mongorestore operation happens.

db.adminCommand( { "setParameter": 1, "wiredTigerEngineRuntimeConfig": "cache_size=1G"})

and revert this back to 3.5 GB once the restoration is completed.

Note:

By reducing the wired tiger cache size will increase the time to restore the data.

3
  • Thanks you many, but I don't wan't to change all the replica set since others database are'in it. I just want the temporary collection fetchto be quiet and take not a lot of ressources. – Slim Apr 13 '19 at 6:52
  • Resizing the WiredTiger cache at runtime may have signficant performance impacts, particularly with a replica set deployment. I would not recommend this as a general optimisation strategy for a production environment: it would be better to set engineConfig.cacheSizeDB as a configuration option on startup. – Stennie Apr 15 '19 at 0:26
  • Thanks @Stennie for your inputs. This helped me with how to approach a problem differently. – Mani Apr 15 '19 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.