0

This problem might look strange. Please bear with me :)

It's an application(NodeJS - MongoDB) helping with storing and viewing of LoRa Packets.

Background:

  • It's a CentOS 7 on Azure Instance.
  • MongoDB v3.4.10
  • Node v6
  • Mongoose 4.11.4
  • Several collections with indexes information in the application
  • Collections are created day wise automatically. Simple NodeJS logic.
  • Older days collections are deleted after x days. For example, After 5 days, today's collection will be dropped. -- Configuration based.
  • This setup is installed at more than 10 clients.

Problem:

  • The indexes for collections are being dropped automatically by Mongo.
  • There are no logs of index drops
  • This happens only on one index where data storage space is only 30GB and per day data in-flow is more than 5-6 GB. Older days data is purged every day automatically. Only 5 days data is kept.

What I tried:

  • Checked other instances. No sign of such problem.
  • Manually created indexes for newly generated collections once, but these also get deleted.
  • Added more verbosity to logs using db.setLogLevel(5,'index') but logs do not mention of any index being deleted automatically.
  • I can see that indexes are created in the logs.

MonogDB Logs for Indexes. I can see similar logs for each new collection being generated. However, in some time the indexes vanish.

2019-03-25T00:15:44.307+0000 I INDEX    [conn69] build index on: LoraEventLogs.logs_2019d82 properties: { v: 2, key: { event: 1, dev: 1 }, name: "event_1_dev_1", ns: "LoraEventLogs.logs_2019d82", background: true }
2019-03-25T00:15:45.002+0000 I -        [conn70]   Index Build (background): 3956400/5454162 72%
2019-03-25T00:15:47.000+0000 I -        [conn69]   Index Build (background): 305200/5779245 5%
2019-03-25T00:15:48.001+0000 I -        [conn70]   Index Build (background): 4271000/5454162 78%
2019-03-25T00:15:50.005+0000 I -        [conn69]   Index Build (background): 647500/5779245 11%
2019-03-25T00:15:51.000+0000 I -        [conn70]   Index Build (background): 4611000/5454162 84%
2019-03-25T00:15:53.001+0000 I -        [conn69]   Index Build (background): 961300/5779245 16%
2019-03-25T00:15:54.001+0000 I -        [conn70]   Index Build (background): 4950600/5454162 90%
2019-03-25T00:15:56.001+0000 I -        [conn69]   Index Build (background): 1314300/5779245 22%
2019-03-25T00:15:57.001+0000 I -        [conn70]   Index Build (background): 5279600/5454162 96%
2019-03-25T00:15:58.680+0000 I INDEX    [conn70] build index done.  scanned 5454162 total records. 48 secs

Appreciate, if you can point me to some direction. This might look like an opinion based question but I don't have anywhere to go other than SO.

I suspect that the space might be an issue here. Since 30 GB is very less to keep all of the data and when there are only few GBs or MBs remaining.

  • Mongo decides to delete the indexes in order to free up some space. Or
  • Due to less space for Mongodb Housekeeping, indexes gets corrupted and get deleted.
  • But, since there is nothing in logs, I am not sure if this is the case.

Any pointer will help. Increasing the space is, unfortunately not an option right now.

Thanks Guys!!

Note: I had previously put this question to stackoverflow.com and was suggested to move it here. Here is the last conversation.

- So the basic premise here is "The indexes for collections are being dropped automatically by Mongo.". Do you have any proof of that? What are your methods of investigation to prove that MongoDB is the problem and not something in your own code or possibly another external process with access to the database? I may be very possible that index builds are "failing", but it's a big leap from failing to deleted automatically. – Neil Lunn 9 mins ago
- Also, unless you can actually narrow this down to "some code" which is actually responsible, then you're "off-topic". As the excerpt on the MongoDB tag here says "Questions about server administration can be asked on http://dba.stackexchange.com.". And the maintenance of indexes ( as opposed to the coding and logic of a routine to create indexes ) is a "database administration" question. – Neil Lunn 6 mins ago 
- I suspected this before. Investigated and found that the same codebase is at other clients where it works perfectly. All indexes present. I can see index build success for all collections. – Ravi Kumar Gupta 5 mins ago    
- Suspect and proof are two different things. I suggest you "narrow it down" and where you still think it's a "MongoDB server issue", then ask on dba.stackexchnage.com as you are really supposed to. Stack Oveflow is the site for "programming based questions", and that is not what this is. – Neil Lunn 3 mins ago
- The only difference is the space. This client has only 30GB of SSD while other have more than that. 250GB of SSD and MongoDB taking around 120GB of space for data. All indexes present there as well. :( – Ravi Kumar Gupta 3 mins ago    Edit    
- Thanks Neil. Will ask there. – Ravi Kumar Gupta 3 mins ago    Edit    Delete
  • Do you find any information about the deletion of the index in the log file? – Mani Mar 29 at 21:18
  • MongoDB will not automatically drop indexes to free up disk space or delete corrupt indexes. Index drops are more likely to be the result of application code or user action. Can you clarify the storage usage and free space? It's not clear if you have 30GB total for your dbPath or 30GB free. As for pointers: index creation & drops should be logged by default. A drop will be logged as a COMMAND; try grepping for deleteIndexes. – Stennie Mar 30 at 0:00
  • The log output you've included is from two concurrent background index builds, but doesn't include the start of the build for conn70. Can you add the missing log line to confirm the namespace and spec for the second index? – Stennie Mar 30 at 0:00
  • Mani and @Stennie 1. There was nothing in the logs for deleting any index. 2. Total space on the db path is 30GB. This is an additional SSD. OS is installed on other disk. – Ravi Kumar Gupta Apr 2 at 17:36
  • So total space is 30GB. Do you have any monitoring in place for free space (and if so, how low does free space get)? Have you adjusted any logging options (for example using quiet mode which will suppress connection, replication, and drop events which are normally logged by default)? Can you also address my earlier comment about the missing log information for conn70? – Stennie Apr 2 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.