Mongodb replica stuck at STARTUP

Question

I have 4 Servers:
Mongo1(with secondary and arbiter), Mongo2(primary).
Mongo3, Mongo4

Mongo1 and Mongo2 were initiated and successfully running for months.
(1 and 2 + arbiter are connected over the private network)

I'm trying to add new replica members, Mongo3 and Mongo4 to the current replica set(connecting over the public network).

All 4 servers were able to connect with telnet and mongo-cli.
All 4 mongo.conf and mongo-keyfile are the same.

Added new replica with rs.add("mongo-3-public-ip:port");rs.add("mongo-4-public-ip:port");
Mongo3 and Mongo4 not are stuck in STARTUP

rs.status() output on Primary:

                {
                        "_id" : 4,
                        "name" : "mongo-4-ip:port",
                        "health" : 1,
                        "state" : 0,
                        "stateStr" : "STARTUP",
                        "uptime" : 896,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2022-11-04T11:28:11.195Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(13),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -2
                }

rs.status() output on Newly added Mongo3 and Mongo4:

{
        "ok" : 0,
        "errmsg" : "no replset config has been received",
        "code" : 94,
        "codeName" : "NotYetInitialized"
}

mongo.conf

# mongod.conf

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile

# network interfaces
net:
  port: mongo-port
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.

security:
 authorization: enabled
 keyFile: /opt/mongo/mongo-keyfile

replication:
  replSetName: replicaset-name
  oplogSizeMB: 35000

Logs in primary:

2022-11-04T02:03:07.509+0000 I  NETWORK  [LogicalSessionCacheRefresh] Starting new replica set monitor for redateddata/mongo-1-ip:port,mongo-2-ip:port,mongo-3-ip:port,mongo-4-ip:port
2022-11-04T02:03:07.522+0000 I  NETWORK  [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for replicaset-name is replicaset-name/mongo-1-ip:port,mongo-2-ip:port,

Nothing else related to Mongo3 and Mongo4. no errors or anything in Mongo 3 and Mongo 4.

Mongo version: 4.2.5

(Please, comment if more details are needed)

Answer 1

The problem is that those two first servers are using private addresses. How you think that Mongo3 and 4 can connect servers what uses private addresses? All your servers must be on the same network, is that private or public (internet). All servers must have possibility to connect to each other.

Then there is one minor fault too. Location of arbiter. It should NEVER be on the same server where "other" node of the same replica set is (if you have multiple RS's, then arbiter can be on one of the "other" RS node). If that node goes down or get network partitioned, your RS loose two voting members.

Arbiter don't use (basically) any space or CPU power, it can be located to any other machine, like one of your application servers.