0

When does old/first oplog entry in the mongo db primary get removed? I could not find much info when that would happen or if it even gets removed as it is a special circular collection.

My assumption would be that, when an oplog entry is synced with all secondaries, it must get removed?

My use case - in a replica set's primary, I perform mongoimport twice a day, that upserts 290k rows, which causes an oplog window less than 1 hour alert, that auto resolves in few minutes. I wonder if I should increase oplog size - and by how much?

Config:

MongoDB 4.4 Atlas
M10 2GB RAM 10GB Storage
1000 IOPS
1500 max connections
Up to 5 Gigabit network performance
Improve this question

1 Answer 1

Reset to default
2

When does old/first oplog entry in the mongo db primary get removed?

Before version 4.4, oplog size is fixed so Mongodb removes the entries as it fills up. Starting from version 4.4, you can specify a minimum retention period in hours where an oplog gets removed if it passes the specified time period or if the oplog reaches the maximum size[1].

should increase oplog size - and by how much?

On page 250 of MongoDB The Definitive Guide, a possible problem to your scenario is stated as, "can use a great deal of oplog space without a corresponding increase in data size or disk use". Therefore to answer this question accurate, we need three more pieces of information on your system:

  1. What is your storage engine?
  2. How much physical memory and free disk space do you have?
  3. What is the OS?
Improve this answer
4
  • added the info in the question
    – Arun S R
    Apr 23 at 3:54
  • If you are using Wiredtiger storage engine, then oplog takes 5% of free disk space which at most would be 500000 KB which would be more than 290k, but this value would decrease as you do more writes into the database.
    – Khoa Bui
    Apr 23 at 4:13
  • What is the suggestion for this case to keep oplog window more than 1Hr? The 290k rows currently take up 500MB by themselves. Upsert updates almost 80% of the fields of the object. And the free space is almost 6GB. This upsert through import creates 15GB/HR oplog/hr at peak and it reduces to normal amount in 2 mins.
    – Arun S R
    Apr 23 at 4:29
  • Again referring to MongoDB a definitive guide, they recommend the rule of thumb to be if your writes at 1KB/min then oplog fills up at 1KB/min. So then you can allocate the size of the oplog file to be: 500MB for the 290K row + however many operations you expect to have. Let's say we want to increase the peak then, I think the new oplog size should be 790 MB. Hope this helps @Arun
    – Khoa Bui
    Apr 23 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.