0

I have a 3 server replica set deployed by Googles 1 Click Deploy functionality, the problem is I created an index on a particularly large collection (100,000,000 documents) and it consumed the whole hard drive and caused Mongo to crash and stay crashed.

I have deleted a 30GB log file and freed up some other space and tried to restart the mongo process on both the primary and secondary servers but the secondary keeps falling into fatal even after getting it to rollback.

I've run rs.reconfig() and it seemed to start the rollback process but it just fell back into FATAL and now it won't budge unless I restart the process and it goes in circles.

To make matters worse, the data (due to connection/training issues) sits only on this busted instance and is not spread across the replica.

I've tried to also run rs.slaveOk() and remove the offending index but I can't query without getting told I can't and obviously when it's in ROLLBACK I can't query for anything because of the lock.

Is there any way I can remove the offending index, repair and resync to fix this while it's in this state?

Thanks.

2

If your data exist on a node then this node is named Primary from now on and the other 2 secondaries.

1) Start the Primary as standalone and drop the index.

2) Take a backup of your data just in case

3) Stop secondaries and delete their data directory.

4) Start the Primary as a member of the replica set.

5) Start the first secondary. This will cause an initial sync of the data, when the member join the set as secondary start the other secondary.

  • This is actually what I did in the end although I'm running into an issue where the process is being killed assumedly by the server.. got signal 15 (Terminated), will terminate after current cmd ends – Dave Mackintosh Dec 29 '14 at 0:42
  • What you get on mongodb logs before "signal 15 (Terminated)"? – Antonios Dec 29 '14 at 17:14
  • Thanks for asking, it's resolved now. I was tailing the wrong log for the operation. I was tailing mongod.log and not mongodb.log – Dave Mackintosh Dec 29 '14 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.