14

I have a Mongo replica set with a few secondaries. A box, which hosts a secondary instance, crashed and lost the database.

I started the secondary Mongo instance again and now it's stuck in STARTUP2 for more than 12 hours. Does it make sense ? The docs say Mongo should be in STARTUP2 for a short period of time before entering the RECOVERING state

What does STARTUP2 exactly mean ? Is it copying the database from the primary ? How can I verify it (assuming the Mongo is running in Linux) ?

12

The answer of eoinbrazil is partly incorrect. A new Node can be in STARTUP2 for a long time. The link the posted says:

Each member of a replica set enters the STARTUP2 state as soon as mongod finishes loading that member’s configuration, at which time it becomes an active member of the replica set. The member then decides whether or not to undertake an initial sync. If a member begins an initial sync, the member remains in STARTUP2 until all data is copied and all indexes are built. Afterwards, the member transitions to RECOVERING.

I'm administering a 700 GB collection and, when I add a new node the STARTUP2 state remains well over 24 hours. But you can still see if there is something happening, by watching if the database grows. You can see the size of the database on the new node with

show databases

or you can also observe the data directory, to see if it is still growing. (on linux with the commands ls, df, du, iotop, etc ....)

| improve this answer | |
  • 1
    show databases fails with not master and slaveOk=false – JDPeckham Dec 12 '19 at 16:19
  • By looking at the logs you can see the progress. For example it will show something like: [rsSync] Index Build: 2538000/22982417 11% – Daniel Benedykt Jan 24 at 14:36
4

The STARTUP2 state means the node cannot vote. A member of a RS enters this state once the MongoD process completes loading it's configuration. In this state, the member has created threads to handle the internal replication operations but it has yet to change state to Recovering and onwards from that to Secondary (see the [state and their details in the docs]).

If your node has been in this state for more than a brief period then you are encountering some strange behaviour. This is pretty much impossible to analyse without the logs to determine why it is stuck. Running rs.status() and db.printSlaveReplicationInfo() will give you some details on the local picture on the node.

The normal approach to resolve this would be to shutdown the node, wipe its data files (those files in the dbpath), and restart it. This will restart the initial sync process and it should move to SECONDARY. If it gets stuck in STARTUP2 again, you'll need to look at the logs to gather more information as to why - there are a range of causes but one that can happen is a flaky network or some local resource contention.

One point to note is that whilst an initial sync is underway the node will remain in STARTUP2 so depending on the amount of data being synced this could be a considerable amount of time (potentially days).

| improve this answer | |
  • Thanks. We removed the data and restarted the Mongo. It is still in STARTUP2. It looks like the Mongo is working. It is consuming CPU and as I see in db.stats the database is growing. The log says that some objects cloned. I am still looking for possible causes of this problem. – Michael Feb 12 '14 at 14:07
  • 1
    If this is still an issue, you may just want to do a copy from another node (see this procedure - docs.mongodb.org/manual/tutorial/resync-replica-set-member/…). If you can attach the logs highlights and details on which version you are using, it might point to a cause but equally this is unusual behaviour. Have you tried pinging between the nodes to see what the network latency is like ? – eoinbrazil Feb 12 '14 at 14:32
  • Mongo 2.4.6 ping between the hosts is Ok. – Michael Feb 12 '14 at 14:34
  • What are the ping times like as it may be intermittent networking issues ? In this case, it is much easier if you could add some of the log outputs as this is non-standard behaviour and the logs are the prime source of truth when trying to determine what exactly is occurring. – eoinbrazil Feb 12 '14 at 17:39
  • I am afraid I cannot show the logs here. However I noticed it tries to connect to another secondary member, which is down. Can it be the cause of the problem ? – Michael Feb 12 '14 at 19:27
2

The easiest way to figure out what is going on in your scenario is to tail the server log on what is going on with the following command:

# This will filter all noise out and leave you only with the "replication" information
tail /var/log/mongodb/mongod.log -f -n 100000 | grep repl

The above command will output what is going on with the synchronisation process.

In case there is nothing there, then use the following:

# This will filter out all "ACCESS" entries and you will be able to more easily dig out what the problem is.
tail /var/log/mongodb/mongod.log -f -n 10000 | grep -v ACCESS

P. S. A member in STARTUP2 state can actually vote (according to docs).

| improve this answer | |
1

One possible cause is that your secondary become "stale" as stated here .

When you are resyncing a member, make sure the RS is not under heavy load.

| improve this answer | |
1

STARTUP2 state could be due to not enough disc space. Well, since there is no where to sync, it can only stay @ STARTUP2 state.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.