0

I'm using mongo as a database log. I store on it all the logs of an application, so I have a high rate of inserts, for example 100 inserts per second. To accomplish that I developed a part of the application dedicated to store in mongo the data. this app only uses a mongo connector.

But the memory consuption is incredible high, I mean after start to use mongo the heap memory has increased in about 300MB and the CPU takes over 90%. Anyone knows how I can decrease this memory consuption or why it is happening?. I've tried to use Mysql and the heap memory keeps normal (only is increased in 20MB) and Cpu doesn't increment as mongo situation.

The data to store is very simple. in fact I used the following json for separate test with the same results.

{testData:'something i'}

Is there any way to create a datasource with mongo?

  • It sounds like it is creating a lot of garbage adding burden on the GC. BTW Mb = mega-bit, MB = mega-byte. – Peter Lawrey May 6 '14 at 13:20
  • Is the memory consumed by MongoDB itself or by the program which accesses it? – Philipp May 6 '14 at 13:26
  • @NeilLunn I'm not sure if the problem is on programming side or DB side. I think that I can improve both things... – UserMan May 6 '14 at 13:39
  • Then where is the code that could be possibly causing the problem? Without it there is nothing to diagnose and answer and therefore this is off-topic. Unless you can add that to your question that is. – Neil Lunn May 6 '14 at 13:42
2

see this link. https://groups.google.com/forum/#!topic/mongodb-user/w7G1xRy3TZQ

the bottom line is Mongo caches indexes and data in available RAM

| improve this answer | |
  • 100 writes per second would result in something like 3kB/s and 180kB between flushes. Since there is only the _id_ index (as far as we know) and the write concern is unknown, too, so the data itself is small. The index however may become quite big pretty fast, using the mentioned 300MB after roughly 36 hours of continuous insertion. – Markus W Mahlberg May 22 '14 at 15:08
1

What WriteConcern do you use?

If you use WriteConcern.UNACKNOWLEDGED (default) or ERRORS_IGNORED, your writes will not be blocked, and the server will buffer all data to try to keep up. If you send data faster that it can write to disk, it will need to buffer all in memory. Use WriteConcern.JOURNALED to avoid this.

If you have a replica-set, you need to use WriteConcern.MAJORITY for these kinds of usage. I had a similar issue where the primary node acknowledged everything, but the replica's could not follow, resulting in the operations log filling up until it ran out of disk space.

Greets.

| improve this answer | |
0

Its all about how you design your database try to use normalization or embedded documents here is reference doc here is the link

go through it you can understand the data modeling concepts.

| improve this answer | |
  • The data is very simple, above an example. – UserMan May 6 '14 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy