question on mongodb oplog usage

Question

One of my secondary on a PSS replica set (v4.0.4) always goes out of sync after it tries to rollback automatically after local.oplog.rs collection cleans up on the secondary.

Question

Though I've configured 2GB of oplog size and I've 500GB of unused space on the filesystem (on all 3 nodes of the replica set) still oplog size actually being used is only 1GB from local db which has oplog.rs collection.

Why is it so ? and how we can make to full use of oplog and I believe after 4.0 oplog can grow more than the configured size but is not happening.

2021-12-20T18:42:15.065+0000 I REPL     [rsBackgroundSync] Rollback using 'recoverToStableTimestamp' method.
2021-12-20T18:42:15.065+0000 I REPL     [rsBackgroundSync] Scheduling rollback (sync source: <NODE1>:<port>)
2021-12-20T18:42:15.065+0000 I ROLLBACK [rsBackgroundSync] transition to ROLLBACK
2021-12-20T18:42:15.066+0000 I REPL     [rsBackgroundSync] transition to ROLLBACK from SECONDARY

rs<port>:PRIMARY> rs.printReplicationInfo()
configured oplog size:   2048MB
log length start to end: 1139secs (0.32hrs)
oplog first event time:  Tue Dec 21 2021 04:06:52 GMT+0000 (UTC)
oplog last event time:   Tue Dec 21 2021 04:25:51 GMT+0000 (UTC)
now:                     Tue Dec 21 2021 04:25:51 GMT+0000 (UTC)

rs<port>:PRIMARY> show dbs
XXXXXXXXX   13.788GB
admin       0.000GB
config      0.001GB
local       1.076GB
rs<port>:PRIMARY>

What's the meaning of I STORAGE [WT OplogTruncaterThread: local.oplog.rs] WiredTiger record store oplog truncation finished What's the triggering point of Oplogs truncation.

Answer 1

configured oplog size shows the max size of the data in the oplog, while show dbs shows the current storage size on disk. The latter is compressed.

That replication info output shows that the oplog contains just under 19 minutes worth of data. If the node has been running longer than that, this likely means it reached the 2GB data size and began removing the oldest entries.

To check the actual sizes, run the following from a mongo shell:

rs.secondaryOk()
var stats = db.getSiblingDB("local").oplog.rs.stats();
print("Data Size:", stats.size);
print("Maximum Size:", stats.maxSize);
print("Size on Disk:", stats.storageSize);