The problem appeared after updating the binaries to version 4.2. When I start the mongo service, the service fails to start and the log shows this exception before it starts the shutdown procedure:
2020-08-07T14:58:26.205+0300 I STORAGE [initandlisten] exception in initAndListen: Location40415: BSON field 'MinValidDocument.oplogDeleteFromPoint' is an ...
Interesting.. It should not be possible... From the source:
￼ type: timestamp
￼ optional: true # This field only exists on 3.4 upgrade
￼ description: "The timestamp of the first oplog entry in a batch when we are writing
￼ oplog entries to ...
Compact the collection in your staging environment. If index size decreases, space is taken up by overhead.
Compare the size of compacted collection & index to the size of both pre compaction, identify the overhead amount/percentage.
If you login directly to that shard (and not mongos), you can drop that collection. After you have logged in to that shard, you can check with db.collection.find() command that there is no data what you want to save, before you drop it.
Depending on your OS, there's multiple ways to modify the maxopenfile of MongoDB.
On Debian and Ubuntu, in /lib/systemd/system/mongod.service there's the LimitNOFILE argument which can be increased if needed.
You have more infos on other OS on https://docs.mongodb.com/manual/reference/ulimit/
Replication is a risk mitigation strategy, amongst other purposes. If you understand those risks, and the consequences should they eventuate, and that's all acceptable in your context then by all means run multiple replicas on the same hardware.
For a dev environment I can imagine this would be OK. A hardware failure would affect one person only. Recovery is ...
I've discovered MongoShake (https://github.com/alibaba/MongoShake) which is an open source Golang application which does cluster to cluster replication and has been made by Alibaba.
The documentation is focused on its internal work/behaviour, and can be discouraging but you have to look at the example configuration file instead which describe each ...