MongoDB replica has no primary

Question

I have an existing MongoDB setup (part of a Graylog installation) where I'm attempting to configure a replica set with a primary, secondary and arbiter. I had the primary and arbiter configured successfully but then for some reason my primary has become the secondary and the application is unable to start now and also isn't allowing me to add in any other nodes in the replica set.

my_mongodb_0:SECONDARY> rs.status()
{
    "set" : "my_mongodb_0",
    "date" : ISODate("2019-01-02T19:50:19.291Z"),
    "myState" : 2,
    "term" : NumberLong(12),
    "heartbeatIntervalMillis" : NumberLong(2000),
    "members" : [
        {
            "_id" : 1,
            "name" : "node2:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 9182,
            "optime" : {
                "ts" : Timestamp(1545938968, 1),
                "t" : NumberLong(12)
            },
            "optimeDate" : ISODate("2018-12-27T19:29:28Z"),
            "configVersion" : 166709,
            "self" : true
        },
        {
            "_id" : 2,
            "name" : "arbiter1:27017",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "lastHeartbeat" : ISODate("2019-01-02T19:50:18.868Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : NumberLong(0),
            "lastHeartbeatMessage" : "Connection refused",
            "configVersion" : -1
        }
    ],
    "ok" : 1
}

...

my_mongodb_0:SECONDARY> rs.add( { host: "node1:27017", force: true } )
{
    "ok" : 0,
    "errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the \"force\" argument to override",
    "code" : 10107
}

Ideally I'd like to add the other node but with the state that it's in now not sure if I can still do this or if I have to revert it to a stand-alone setup first and configure it again. I know the primary election can't occur now and I haven't found a way to force the primary but just wanted to see if anyone else had any ideas.

I'm guessing now it matters what order you add the DB servers and the arbiter but I haven't been able to find a definitive answer to this...

Answer 1

I had the primary and arbiter configured successfully but then for some reason my primary has become the secondary

The issue is that electing or maintaining a primary requires a strict majority (n/2+1) from your configured voting replica set members. A replica set with only two voting members does not provide any fault tolerance for a primary since votes from both members are required to achieve a strict majority.

Since your arbiter is currently "not reachable/healthy" from node2's perspective, node2 no longer has enough votes to remain primary and will step down to become a secondary.

The best way to resolve this would be to fix the communication issue between node2 and your arbiter before adding a third member.

Forced reconfiguration

rs.add( { host: "node1:27017", force: true } )

The force option is only available with rs.reconfig(), so would be ignored in this usage example. If you pass a document as the host parameter for rs.add(host), any options (such as priority or tags) only apply to that individual replica set member.

If you can't get your arbiter back online for some reason, you could follow the forced reconfiguration tutorial to remove the arbiter from your replica set config.

As noted in the tutorial, forced reconfiguration is intended to be used as a last resort rather than routine admin:

The force option forces a new configuration onto the member. Use this procedure only to recover from catastrophic interruptions. Do not use force every time you reconfigure. Also, do not use the force option in any automatic scripts and do not use force when there is still a primary.

I'm guessing now it matters what order you add the DB servers and the arbiter

The order isn't significant aside from adding an initial data-bearing node so the replica set has a primary. However, you do need to ensure you have a majority of voting members online and able to communicate with each other.

Answer 2

I encounter this same error every time I try to connect from a new IP address without first adding the new IP to my list of accepted IP addresses.

Network Access > IP Access List

on cloud.mongodb.com will let you see and update your allowed IP addresses, including granting timed temporary access.