Secondary and Arbiter stuck in Startup

Question

I'm using MongoDB version 3.0.0. I'm trying to setup mongodb replication on our machine. Replication was initially setup but due to some changes on the VM the entire thing crashed. When I tried to set it up again, the secondary and Arbiter became stuck in StartUp Mode.

In the mongoDB conf file I have set

replSet=ReplicaSet1

I added 2 machines using the command

rs.add("10.235.96.12:27017")
rs.add("10.235.96.12:27017")

But after this on the primary when I do a rs.status() secondary and arbiter are still shown in StartUp

ReplicaSet1:PRIMARY> rs.status()
{
    "set" : "ReplicaSet1",
    "date" : ISODate("2015-07-31T04:45:57.260Z"),
    "myState" : 1,
    "members" : [
            {
                    "_id" : 0,
                    "name" : "BOSPROD9:27017",
                    "health" : 1,
                    "state" : 1,
                    "stateStr" : "PRIMARY",
                    "uptime" : 63104,
                    "optime" : Timestamp(1438257913, 1),
                    "optimeDate" : ISODate("2015-07-30T12:05:13Z"),
                    "electionTime" : Timestamp(1438254975, 2),
                    "electionDate" : ISODate("2015-07-30T11:16:15Z"),
                    "configVersion" : 7,
                    "self" : true
            },
            {
                    "_id" : 1,
                    "name" : "10.235.96.12:27017",
                    "health" : 1,
                    "state" : 0,
                    "stateStr" : "STARTUP",
                    "uptime" : 62663,
                    "optime" : Timestamp(0, 0),
                    "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                    "lastHeartbeat" : ISODate("2015-07-31T04:45:56.520Z"),
                    "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                    "pingMs" : 0,
                    "configVersion" : -2
            },
            {
                    "_id" : 2,
                    "name" : "10.235.96.13:27017",
                    "health" : 1,
                    "state" : 0,
                    "stateStr" : "STARTUP",
                    "uptime" : 60043,
                    "lastHeartbeat" : ISODate("2015-07-31T04:45:55.786Z"),
                    "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                    "pingMs" : 0,
                    "configVersion" : -2
            }
    ],
    "ok" : 1
}

When I try to do a rs.conf on the Secondary or Arbiter I get the message

2015-07-31T05:23:07.927+0000 E QUERY    Error: Could not retrieve replica set config: {
    "info" : "run rs.initiate(...) if not yet done for the set",
    "ok" : 0,
    "errmsg" : "no replset config has been received",
    "code" : 94
}

I did try multiple times to clean all the local.0, local.1 files in all the machines and also removed all the journal files from the journal folder as well. Still I'm getting the same issue. Can someone tell me what is it I'm doing wrong here?

I have set replset on all the members. Tried starting in standalone mode and it works.

Answer 1

From "BOSPROD9", try to connect with the mongoshell to the other serves:

$ mongo --host 10.235.96.12 --port 27017

$ mongo --host 10.235.96.13 --port 27017

(Telnet is not the same.) If this doesn't work, it might be firewall or BindIP.

Check bind_ip (should be 0.0.0.0, change in mongodb.conf is it's 127.0.0.1):

$ netstat -nap | grep :27017 | grep LISTEN
tcp        0      0 0.0.0.0:27018           0.0.0.0:*               LISTEN      -     

Try to look at the log-files on 10.235.96.12 and 10.235.96.13, why they are stuck. Did they receive the configuration?

Try to reconfigure this way:

mongo> var cfg = {_id:"ReplicaSet1",members:[{_id:0, host:"BOSPROD9:27017"},{_id:1, host:"10.235.96.12:27017"},{_id:2, host:"10.235.96.13:27017",arbiterOnly:true}]};
mongo> rs.reconfig(cfg);

Solution:

Use hostnames instead of ip's: give all servers a hostname, update the hosts-files and use hostnames in the rs-configuration. It seems that mongodb advices not to use ip addresses, but hostnames.

Answer 2

I had the same issue when i cloned a VM to make the arbiter. Similar to what aldwinaldwin said, I had to change the ip address in this line in my /etc/hosts file "192.168.1.103 mongodb-arbiter.domain.local mongodb1-arbiter" to match the ip address of the server.

Answer 3

As stated in aldwinaldwin's answer, check the connectivity of all the members

if all the connectivity works well, may be you have to make a initial sync

Answer 4

In my case it was SSL issue, in mongod logs, it was ssl handshake error for heartbeat sent to primary node.

Answer 5

I just had this issue (using Mongo 4.2.1) and in my case the connectivity between the nodes was OK but the problem was that the new node (the one stuck in STARTUP state) was actually not able to reach himself though it's hostname. For example:

mongo new.node.hostname

Was working OK from the primary and other nodes, but not from the new node itself (because of some NAT issue with the router). So the solution in my case was to make this hostname resolve to localhost on the machine by updating /etc/hosts to add the hostname:

127.0.0.1   localhost.localdomain localhost new.node.hostname

After that the local connection was working and the status automatically switched to "STARTUP2" and started the initial sync.

Answer 6

In my case, with MongoDB 4.4, the issue was a missing authentication on the new Arbiter instance. The following log entry in /var/log/mongodb/mongodb.log was revealing enough:

{"t":{"$date":"2021-09-16T18:38:39.348+00:00"},"s":"I",  "c":"ACCESS",   "id":20249,   "ctx":"conn1223","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","speculative":false,"principalName":"__system","authenticationDatabase":"local","remote":"192.168.15.184:42234","extraInfo":{},"error":"AuthenticationFailed: It is not possible to authenticate as the __system user on servers started without a --keyFile parameter"}}

After copying the same keyfile from the other members of the ReplicaSet, setting the same ownership and permissions and enabling it in the config (/etc/mongod.conf):

security:
  keyFile: /var/lib/mongodb/keyfile

... MongoDB on the Arbiter node was restarted. It then successfully joined the ReplicaSet.