We have a Mongodb 4.0 database, running a replica set with 3 members. Total database size is about 800GB. All of the members are running in docker containers. Docker shows that the master container has read total 60GB data and written 165GB in the past 5 days. One of the secondary members has read and written 33 + 33GB data in the past 5 days. The last (secondary) member has read 68.9TB (!!!) data since it was started. The average I/O transfer speed for that "rebellious" member is 200 MB/sec. Basically, it uses all available I/O bandwidth, 24/7.
The service log of that member is full of messages like this:
NETWORK [LogicalSessionCacheRefresh] Starting new replica set monitor for rs0/mongo-backup:27017,mongo-master:27017,mongo-replica:27017
This member is used for backup only, nothing is running queries on it.
How is that possible? Why is it reading at 100% speed all the time? What are these messages? Apart from these messages, we don't see anything useful.
UPDATE: If I restart the container, then it goes back normal for a while. But after running for some days, it starts this reading again, and the only way to fix it is to restart the container. All members in the replica set are running mongodb 4.0.6.
Some strange messages from the logs from yesterday:
2019-03-04T04:33:51.101+0100 I ASIO [Replication] Ending connection to host mongo-replica:27017 due to bad connection status; 15 connections to that host remain open 2019-03-04T04:33:51.099+0100 I REPL_HB [replexec-3980] Error in heartbeat (requestId: 4950505) to mongo-replica:27017, response status: NetworkInterfaceExceededTimeLimit: timed out
But otherwise, the replication is working fine.