Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have a Mongo replica set with a few secondaries. A box, which hosts a secondary instance, crashed and lost the database.

I started the secondary Mongo instance again and now it's stuck in STARTUP2 for more than 12 hours. Does it make sense ? The docs say Mongo should be in STARTUP2 for a short period of time before entering the RECOVERING state

What does STARTUP2 exactly mean ? Is it copying the database from the primary ? How can I verify it (assuming the Mongo is running in Linux) ?

share|improve this question

1 Answer 1

The STARTUP2 state means the node cannot vote. A member of a RS enters this state once the MongoD process completes loading it's configuration. In this state, the member has created threads to handle the internal replication operations but it has yet to change state to Recovering and onwards from that to Secondary (see the [state and their details in the docs]).

If your node has been in this state for more than a brief period then you are encountering some strange behaviour. This is pretty much impossible to analyse without the logs to determine why it is stuck. Running rs.status() and db.printSlaveReplicationInfo() will give you some details on the local picture on the node.

The normal approach to resolve this would be to shutdown the node, wipe its data files (those files in the dbpath), and restart it. This will restart the initial sync process and it should move to SECONDARY. If it gets stuck in STARTUP2 again, you'll need to look at the logs to gather more information as to why - there are a range of causes but one that can happen is a flaky network or some local resource contention.

share|improve this answer
    
Thanks. We removed the data and restarted the Mongo. It is still in STARTUP2. It looks like the Mongo is working. It is consuming CPU and as I see in db.stats the database is growing. The log says that some objects cloned. I am still looking for possible causes of this problem. –  Michael Feb 12 at 14:07
1  
If this is still an issue, you may just want to do a copy from another node (see this procedure - docs.mongodb.org/manual/tutorial/resync-replica-set-member/…). If you can attach the logs highlights and details on which version you are using, it might point to a cause but equally this is unusual behaviour. Have you tried pinging between the nodes to see what the network latency is like ? –  eoinbrazil Feb 12 at 14:32
    
Mongo 2.4.6 ping between the hosts is Ok. –  Michael Feb 12 at 14:34
    
What are the ping times like as it may be intermittent networking issues ? In this case, it is much easier if you could add some of the log outputs as this is non-standard behaviour and the logs are the prime source of truth when trying to determine what exactly is occurring. –  eoinbrazil Feb 12 at 17:39
    
I am afraid I cannot show the logs here. However I noticed it tries to connect to another secondary member, which is down. Can it be the cause of the problem ? –  Michael Feb 12 at 19:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.