0

I have been using mongo 2.6.0 for some time on 64-bit centos (with journaling enabled). One fine day when we were testing to insert a document to one of the collections, we started seeing data corruption issues. When I ran db.collection.validate() from mongos, the response had pointed to one of the clusters that had an index corruption. I restarted all the nodes in the cluster but continue to see the issue. Here is the logs on the start up of one of the nodes

2015-09-02T15:35:43.940-0700 ***** SERVER RESTARTED *****
2015-09-02T15:35:44.004-0700 [initandlisten] MongoDB starting : pid=2017        port=27017 dbpath=/opt/ebrary/mongodb/data 64-bit host=mongod01-f.ebrary.com
2015-09-02T15:35:44.005-0700 [initandlisten] db version v2.6.0
2015-09-02T15:35:44.005-0700 [initandlisten] git version:  1c1c76aeca21c5983dc178920f5052c298db616c
2015-09-02T15:35:44.005-0700 [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-09-02T15:35:44.005-0700 [initandlisten] allocator: tcmalloc
2015-09-02T15:35:44.005-0700 [initandlisten] options: { config: "/etc/mongod.conf", processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSet: "mongod-f-shard01" }, security: { keyFile: "****" }, storage: { dbPath: "****" }, systemLog: { destination: "file", logAppend: true, path: "****" } }
2015-09-02T15:35:44.024-0700 [initandlisten] journal dir=/****/journal
2015-09-02T15:35:44.024-0700 [initandlisten] recover : no journal files present, no recovery needed
2015-09-02T15:35:46.641-0700 [initandlisten] waiting for connections on port 27017
2015-09-02T15:35:46.728-0700 [rsStart] replSet I am mongod01-f.xxx.com:27017
2015-09-02T15:35:46.731-0700 [rsStart] replSet STARTUP2
2015-09-02T15:35:46.732-0700 [rsSync] replSet SECONDARY
2015-09-02T15:35:46.732-0700 [rsMgr] replSet info electSelf 0
2015-09-02T15:35:46.807-0700 [initandlisten] connection accepted from 10.10.25.47:51612 #1 (1 connection now open)
2015-09-02T15:35:47.733-0700 [rsMgr] replSet PRIMARY
2015-09-02T15:36:19.931-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "xxx", key: "xxx" }
2015-09-02T15:36:46.623-0700 [clientcursormon] mem (MB) res:66 virt:794147
2015-09-02T15:36:46.623-0700 [clientcursormon]  mapped (incl journal view):793464
2015-09-02T15:36:46.623-0700 [clientcursormon]  connections:1
2015-09-02T15:36:46.623-0700 [clientcursormon]  replication threads:32
2015-09-02T15:44:34.983-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsmon", key: "xxx" }
2015-09-02T15:45:29.948-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsmon", key: "xxx" }
2015-09-02T15:46:11.005-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsmon", key: "xxx" }
**2015-09-02T15:46:11.103-0700 [conn1] core-mongo Assertion failure isOk() src/mongo/db/storage/extent.h 80**
2015-09-02T15:46:11.164-0700 [conn1] core-mongo 0x11bd301 0x115f819 0x114395e 0xee2ec0 0x8ae1e9 0xa28f3f 0xa1cfda 0xa1df4e 0xa1f806 0xd4c6a7 0xb96382 0xb98962 0x76b76f 0x117367b 0x7f20c68e8a51 0x7f20c5c8d9ad          /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11bd301]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x115f819]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x114395e]
 /usr/bin/mongod(_ZNK5mongo13ExtentManager9getExtentERKNS_7DiskLocEb+0x60) [0xee2ec0]
/usr/bin/mongod(_ZNK5mongo10Collection11storageSizeEPiPNS_16BSONArrayBuilderE+0x779) [0x8ae1e9]
 /usr/bin/mongod(_ZN5mongo7DBStats3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2df) [0xa28f3f]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa1cfda]

/usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xc5e) [0xa1df4e] /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa1f806] /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x2307) [0xd4c6a7] /usr/bin/mongod() [0xb96382] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x442) [0xb98962] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x76b76f] /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4fb) [0x117367b] /lib64/libpthread.so.0(+0x7a51) [0x7f20c68e8a51] /lib64/libc.so.6(clone+0x6d) [0x7f20c5c8d9ad]

2

This error means the data files MongoDB are corrupted.

You should run mongod with --repair option to fix them.

mongod --repair

After finishing repairing, stop it and try start mongod again.

| improve this answer | |
  • Journaling was enabled. In spite of that, I am unable to recover. I thought with journaling enabled, the data is guaranteed to have a safe point and journaling would let you recover to the last consistent state. I am not seeing that happening. Is repair the only option? – Satheesh Akkinepally Sep 3 '15 at 1:14
  • 1
    In log file it says: no journal files present, no recovery needed, so no journal file found. However, as you have a secondary node in that replica set, you can try to empty data on that primary node and restore data from the secondary (in case the secondary is not corrupted docs.mongodb.org/master/tutorial/resync-replica-set-member) – anhlc Sep 3 '15 at 3:06
  • not sure what happened and why it says that. Also this is our test environment and have no replicas for the shards. It's a single node replica of 6 shards. I am trying to find the root cause of all of this and trying to ensure mongo db is reliable enough. There is no sign of any other fil system corruption of any sort. – Satheesh Akkinepally Sep 3 '15 at 3:18
  • 1
    v2.6.0 has tons of bugs which makes impossible to figure out what happened. I suggest you to move to latest v2.6.11 – Antonios Sep 3 '15 at 22:37
  • Thank You for the replies. I will schedule to migrate to mongo 2.6.11 asap. – Satheesh Akkinepally Sep 4 '15 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.