I have a 3 server replica set deployed by Googles 1 Click Deploy functionality, the problem is I created an index on a particularly large collection (100,000,000 documents) and it consumed the whole hard drive and caused Mongo to crash and stay crashed.
I have deleted a 30GB log file and freed up some other space and tried to restart the mongo process on both the primary and secondary servers but the secondary keeps falling into fatal even after getting it to rollback.
rs.reconfig() and it seemed to start the rollback process but it just fell back into
FATAL and now it won't budge unless I restart the process and it goes in circles.
To make matters worse, the data (due to connection/training issues) sits only on this busted instance and is not spread across the replica.
I've tried to also run
rs.slaveOk() and remove the offending index but I can't query without getting told I can't and obviously when it's in
ROLLBACK I can't query for anything because of the lock.
Is there any way I can remove the offending index, repair and resync to fix this while it's in this state?