1

I have a Replica Set with 3 members (1 primary and 2 secondaries), and one of the secondaries was working just fine and thrown the following error

> 2018-03-12T11:03:54.868-0400 E REPL [repl writer worker 13] writer
> worker caught exception: :: caused by :: 84 key/value already in index
> on: { ts: Timestamp 1520867034000|8, h: 287037468373256260, v: 2, op:
> "u", ns: "Sitecore_analytics_PROD.Interactions", o2:
> 
> { _id: BinData(3, EFC3DD1A15B75442986344FDE9CC71EB) } , o: { $set:
> 
> { ContactVisitIndex: 2 } } } 2018-03-12T11:03:54.868-0400 I - [repl
> writer worker 13] Fatal Assertion 16360 2018-03-12T11:03:54.868-0400 I
> - [repl writer worker 13] 
> ***aborting after fassert() failure

After that the MongoDB cannot start anymore, and keep throwing the same error!

How can I determine the root cause of it? Is there a way to prevent? How to solve it?

Improve this question
7
  • 2
    I notice the question is tagged with mongodb-3.0, but can you confirm the full version of MongoDB server? (x.y.z)
    – Stennie
    Commented Apr 6, 2018 at 7:19
  • The full version is 3.0.14
    – Vinicius Deschamps
    Commented Apr 6, 2018 at 9:57
  • 1
    The fatal exception from repl writer worker indicates an issue applying an entry from your replication oplog. Since this problem cannot correct itself, the fatal assertion will repeat on startup and require administrative intervention to resync this member. The source of such an issue can be difficult to track down, but you may be able to find some clues in the logs for this secondary. Perhaps there was an unclean shutdown or local administrative action that caused the data files to diverge.
    – Stennie
    Commented Apr 7, 2018 at 1:49
  • 1
    Note: MongoDB 3.0 reached end of life in Feb 2018, so your best option to avoid future issues would be upgrading to a newer (and supported) version of MongoDB. I suspect you are pinned to an older version of MongoDB as recommended for use with your version of Sitecore, but I believe their matrix of supported versions is based on what they've tested with rather than what will actually work. You could review Compatibility Changes in MongoDB 3.2 and test in a staging environment to see if this has any impact on Sitecore.
    – Stennie
    Commented Apr 7, 2018 at 1:52
  • 1
    This specific issue may be unrelated to MongoDB 3.0, but there have been many improvements in stability and performance since MongoDB 3.0 that make upgrading strongly recommendable. Supported release series also continue to get bug fixes & security releases.
    – Stennie
    Commented Apr 7, 2018 at 1:57

2 Answers 2

Reset to default
1

Do initial sync for that problematic secondary node. Just delete files from dbpath and start mongod. Node will do initial sync automatically. Other and faster way (if you can) is copy snapshot of dbpath of existing node (can be primary or secondary) and start mongod.

The problem here is that there is already entry at unique index and system tries to apply opLog line and crashes to that error.

Improve this answer
4
  • Is it possible to know why this happened?
    – Vinicius Deschamps
    Commented Apr 6, 2018 at 9:37
  • That is probably impossible... This is situation what should not exists. Log files may or may not have information about when and how this situation has developed.
    – JJussi
    Commented Apr 6, 2018 at 11:02
  • At least a way to avoid?
    – Vinicius Deschamps
    Commented Apr 6, 2018 at 12:50
  • Probably, after initial sync, you will never see it again... If your hardware works fine.
    – JJussi
    Commented Apr 7, 2018 at 7:16
1

As i am able to see from your log files you are getting the error like ***aborting after fassert() failure. As young DBA Blog here or MongoDB Jira here, here and here. Without more information its difficult to determine the exact source of the issue.

As @alernerdev describe here I hope you have data backups? Bring down your faulty secondaries, delete the data on one of them, and bring it back up (just one) -- this will start the initial re-sync process. If all is well, at this point you will have 1 primary and 1 secondary. Before you do this, verify that the size of your oplog window is large enough to complete the initial sync process. Once this is all back to normal, do the same for your second Secondary.

For your further ref here and here

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.