0

We have a mongodb 4.2 (stable branch) replica set with three members. It was working for several months without problems. Yesterday one of the secondary members started to do very strange things.

I can see this in the log:

2021-11-14T12:47:33.690+0100 I  ELECTION [replexec-5] Scheduling catchup takeover at 2021-11-14T12:48:03.690+0100
2021-11-14T12:47:33.955+0100 I  REPL     [replexec-5] Canceling catchup takeover callback
2021-11-14T12:47:36.386+0100 I  ELECTION [replexec-5] Scheduling catchup takeover at 2021-11-14T12:48:06.386+0100
2021-11-14T12:47:37.971+0100 I  REPL     [replexec-3] Canceling catchup takeover callback
2021-11-14T12:47:45.137+0100 I  ELECTION [replexec-3] Scheduling catchup takeover at 2021-11-14T12:48:15.137+0100
2021-11-14T12:47:45.159+0100 I  REPL     [repl-writer-worker-133] applied op: CRUD { ts: Timestamp(1636890464, 16), t: 805, h: 0, v: 2, op: "i", ns: "mydb.www.user_view_events", ui: UUID("698007b9-599a-41ad-97b1-6bebef831a08"), wall: new Date(1636890464676), lsid: { id: UUID("9512b937-bc4d-4ebc-84e0-26e21548578b"), uid: BinData(0, 6399AB0DAC62F20BFC466753B10FB58FB7E692BEC952C69B84D997021794D1F8) }, txnNumber: 5182, stmtId: 0, prevOpTime: { ts: Timestamp(0, 0), t: -1 }, o: { _id: ObjectId('6190f761d9c972221fee6e86'), cluster: "kifu", data: { event: "exercise", exercise_event: "evaluate", evaluation_result: { success: false, solution: [ "falanx"], successPercent: 0.0 }, published: true, evaluationTimeStamp: ""Nov 14, 2021, 12:47:45 PM"" }, eventDate: new Date(1636890465013), instance: "tomcat01",  } }, took 117ms
2021-11-14T12:47:46.003+0100 I  REPL     [replexec-3] Canceling catchup takeover callback
2021-11-14T12:47:47.929+0100 I  ELECTION [replexec-3] Scheduling catchup takeover at 2021-11-14T12:48:17.929+0100
2021-11-14T12:47:48.010+0100 I  REPL     [replexec-3] Canceling catchup takeover callback
2021-11-14T12:47:51.025+0100 I  ELECTION [replexec-3] Scheduling catchup takeover at 2021-11-14T12:48:21.025+0100
2021-11-14T12:47:52.032+0100 I  REPL     [replexec-3] Canceling catchup takeover callback
2021-11-14T12:47:53.821+0100 I  ELECTION [replexec-3] Scheduling catchup takeover at 2021-11-14T12:48:23.821+0100
2021-11-14T12:47:54.039+0100 I  REPL     [replexec-5] Canceling catchup takeover callback
2021-11-14T12:47:59.480+0100 I  ELECTION [replexec-3] Scheduling catchup takeover at 2021-11-14T12:48:29.480+0100
2021-11-14T12:48:00.062+0100 I  REPL     [replexec-3] Canceling catchup takeover callback
2021-11-14T12:48:02.377+0100 I  ELECTION [replexec-5] Scheduling catchup takeover at 2021-11-14T12:48:32.377+0100
2021-11-14T12:48:04.076+0100 I  REPL     [replexec-5] Canceling catchup takeover callback

This never stops. I suspect that one schedule/cancel pair happens once for every heartbeat. As you can see, the member also replicates CRUD operations from the primary. All members have the same OS version and the same NTP client setup (sytem clocks are in sync).

"Catchup takeover" usually means that the secondary thinks that it is ahead of the primary. This should rarely happen. I have no explanation. We could delete the whole member and recreate it. But we only have three members in this set, and we don't know what is causing the problem. So deleting the member is too risky (it would leave us with just to replicas for several hours).

We have tried to restart this member, but it did not help.

Any idea what is causing it, and how to fix it?

1
  • Stop the member, drop entire dbPath and restart again. Yes, for a period a time you will have only 2 members available, but is this a problem? I think, that's the situation you also have already now! What is the status of this member? Nov 15, 2021 at 9:23

1 Answer 1

0

We could shutdown all replica set members at the same time and quickly restart all of them. This solved the problem. But this was not the best solution, because we had to stop all related services and have some downtime.

The alternative solution would have been dropping that member and add a new member. It would have been zero downtime, but it would have been higher risk.

I'm not sure if I can try out if anybody else posts an answer, but since my "solution" is not the best, please feel free to do so - it may help others.

I'm also not sure what was causing the problem, and I'm affraid I'll never know.

2
  • Why would it have been higher risk to drop-and-recreate the member?
    – Hannah Vernon
    Nov 16, 2021 at 13:20
  • If you are so concerned about an outage (on the other hand you can stop the entire application), then consider to deploy a 5-member Replica Set. Nov 16, 2021 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.