Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

help me, production server is down every 15 min, i don't know how can fix it without lock

Tue May 28 14:26:17.803 [conn7] Assertion: 10334:BSONObj size: 0 (0x00000000) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO
0xdcf361 0xd90a1b 0xd90f5c 0x6eaf99 0x81119c 0xa7bbdf 0xa7f48a 0xa78c81 0xa7e03e 0xa8114e 0x9f44d4 0x9f57e2 0x6e747a 0xdbbb7e 0x7f77078bee9a 0x7f7706bceccd
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd90a1b]
 /usr/bin/mongod() [0xd90f5c]
 /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5a9) [0x6eaf99]
 /usr/bin/mongod(_ZN5mongo11BtreeCursor7currentEv+0x5c) [0x81119c]
 /usr/bin/mongod(_ZNK5mongo21ResponseBuildStrategy7currentEbPNS_13ResultDetailsE+0x11f) [0xa7bbdf]
 /usr/bin/mongod(_ZN5mongo20OrderedBuildStrategy11handleMatchEPNS_13ResultDetailsE+0xaa) [0xa7f48a]
 /usr/bin/mongod(_ZN5mongo20QueryResponseBuilder8addMatchEv+0xb1) [0xa78c81]
 /usr/bin/mongod(_ZN5mongo23queryWithQueryOptimizerEiRKSsRKNS_7BSONObjERNS_5CurOpES4_S4_RKN5boost10shared_ptrINS_11ParsedQueryEEES4_RKNS_12ChunkVersionERNS7_10scoped_ptrINS_25PageFaultRetryableSectionEEER$
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1b3e) [0xa8114e]
 /usr/bin/mongod() [0x9f44d4]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x392) [0x9f57e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9a) [0x6e747a]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdbbb7e]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f77078bee9a]enter code here
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7706bceccd]
Tue May 28 14:26:17.807 [conn7] assertion 10334 BSONObj size: 0 (0x00000000) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO ns:cache.topics query:{ slug: "automobiles/autos-for-s$
Tue May 28 14:26:17.807 [conn7]  ntoskip:0 ntoreturn:-1
share

migration rejected from stackoverflow.com Jun 28 '13 at 17:30

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as too broad by Nick Chammas, dezso, Mark Storey-Smith, RolandoMySQLDBA, Paul White Jun 28 '13 at 17:30

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

    
sorry for my english –  user2430406 May 28 '13 at 23:28
1  
I'd recommend you pay for 10gen support for these types of issues. 10gen.com/products/mongodb-subscriptions –  WiredPrairie May 29 '13 at 0:14

1 Answer 1

This kind of error can be caused by many things, from a bad hard disk to network issues, to memory errors, to bugs like this one. It's impossible to determine what might be the cause without detailed logs and diagnostics which are beyond the scope of a question here. However, I can give some general advice about how to get back to a healthy state.

1. Use a backup or a secondary to restore the data

If you have this running as part of a replica set (which you should in production), then my first recommendation would be to use one of the secondaries as a known "good" data source and work from that. Even if you manage to run repair on this database, your data consistency will be suspect - it is approximately equivalent to running fsck on a Linux filesystem, it will get the system back to working but makes no guarantees about the data on the system.

2. Make sure you are not getting hit by a known bug

In general, make sure you are on the latest release of your branch, and preferably it should be one of the two most recent stable branches (as of writing this: 2.2 and 2.4). In particular, if you are one of the versions listed as effected in SERVER-7511, I would recommend an immediate upgrade.

3. Drop and recreate indexes

Occasionally, if corruption has occurred, it can be an issue with the index only, and so dropping indexes and re-creating them can restore a database to functionality with a minimal chance of compromising actual data.

4. The repair command

Finally we get to the last resort if you do not have a good copy from elsewhere and the issue is not with your indexes, the repair command.

If repair is not working from the command line or as an option when you start mongod, then your repair method of last resort would be using mongodump --repair. The mongodump repair is more aggressive in terms of how it will walk the files and extents and attempt to dump out all data it can find - sometimes even this will fail.

Note: In general you will need ~2x the disk space that your database currently occupies to do a repair. The repair mechanism rewrites all the files from scratch in a different location (defaults to a temp folder, see the linked manual page). A mongodump repair may output many times the data included in the original database, depending on the nature of the issue.

In terms of how quickly a repair will run, it will depend on several factors:

  1. How much data you have
  2. How fast your disk is (imagine copying the entire database to another folder as an estimate)
  3. The nature of the corruption (see the note above about the amount of data that mongodump can produce

Because of the variables here, and the variance in performance of different systems, there is no easy way to estimate the time taken to repair a database in general, even if you know the size.

share
    
thank you for help –  user2430406 May 29 '13 at 21:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.