MongoDB replica set - Error in heartbeat request

Question

I have created mongo db replica set in my local as below:

    mongod --port 27021 --dbpath /cm1/mongodb/rs0-1 --replSet rs0 --smallfiles --oplogSize 128.

I have started all 3 (1 Primary, 2 secondary) instances with mongo --port 27022.

Replica set is working and data is flowing from Primary to secondaries.

If I shutdown one of the mongodb instances the remaining 2 instances are throwing errors. Logs as below:

2017-07-25T13:01:24.904+0530 I REPL     [ReplicationExecutor] Error in heartbeat request to localhost:27021; HostUnreachable: Connection refused
2017-07-25T13:01:24.904+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to localhost:27021
2017-07-25T13:01:24.906+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to localhost:27021 - HostUnreachable: Connection refused
2017-07-25T13:01:24.906+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to localhost:27021 due to failed operation on a connection
2017-07-25T13:01:24.906+0530 I REPL     [ReplicationExecutor] Error in heartbeat request to localhost:27021; HostUnreachable: Connection refused
2017-07-25T13:01:24.906+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to localhost:27021
2017-07-25T13:01:24.907+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to localhost:27021 - HostUnreachable: Connection refused
2017-07-25T13:01:24.907+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to localhost:27021 due to failed operation on a connection
2017-07-25T13:01:24.907+0530 I REPL     [ReplicationExecutor] Error in heartbeat request to localhost:27021; HostUnreachable: Connection refused
2017-07-25T13:01:26.908+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to localhost:27021
2017-07-25T13:01:26.910+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to localhost:27021 - HostUnreachable: Connection refused
2017-07-25T13:01:26.910+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to localhost:27021 due to failed operation on a connection
2017-07-25T13:01:26.910+0530 I REPL     [ReplicationExecutor] Error in heartbeat request to localhost:27021; HostUnreachable: Connection refused
2017-07-25T13:01:26.910+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to localhost:27021
2017-07-25T13:01:26.911+0530 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to localhost:27021 - HostUnreachable: Connection refused

Is there something missing for replica while shutdown the instance?

Answer 1

Actually no! If you shut down one node from your replica set, other nodes, of course, start complaining that they cannot connect that missing node. All nodes are always "talking" to each other.

And error message tells that, that missing node was in the port 27021.