I have a primary node and second node of mongodb in Production , and a third Node which is the arbiter , after an unplanned shutdown on production last week , the Arbiter node refuse to start mongodb service , I can start the primary and second node , but the arbiter don't start and I get the error below:
> > 2020-03-17T12:33:45.634+0100 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
about to fork child process, waiting until server is ready for connections.
forked process: 31746
ERROR: child process failed, exited with error number 51
To see additional information in this output, start without the "--fork" option.
someone knows how to solve this issue ?? I have repaired the Database , below the output of reparation
> 2020-03-19T14:10:18.381+0100 I STORAGE [initandlisten] Verify succeeded on uri table:collection-6--6199099221025852735. Not salvaging.
2020-03-19T14:10:18.384+0100 I STORAGE [initandlisten] Repairing collection local.replset.election
2020-03-19T14:10:18.651+0100 I STORAGE [initandlisten] Verify succeeded on uri table:collection-0--1928215778901914464. Not salvaging.
2020-03-19T14:10:18.813+0100 I INDEX [initandlisten] build index on: local.replset.election properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.replset.election" }
2020-03-19T14:10:18.813+0100 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:18.913+0100 I STORAGE [initandlisten] Repairing collection local.replset.minvalid
2020-03-19T14:10:18.953+0100 I STORAGE [initandlisten] Verify succeeded on uri table:collection-2-5553576768946172852. Not salvaging.
2020-03-19T14:10:19.058+0100 I INDEX [initandlisten] build index on: local.replset.minvalid properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.replset.minvalid" }
2020-03-19T14:10:19.058+0100 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.112+0100 I STORAGE [initandlisten] Repairing collection local.replset.oplogTruncateAfterPoint
2020-03-19T14:10:19.121+0100 I STORAGE [initandlisten] Verify succeeded on uri table:collection-0-5553576768946172852. Not salvaging.
2020-03-19T14:10:19.228+0100 I INDEX [initandlisten] build index on: local.replset.oplogTruncateAfterPoint properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.replset.oplogTruncateAfterPoint" }
2020-03-19T14:10:19.228+0100 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.279+0100 I STORAGE [initandlisten] Repairing collection local.startup_log
2020-03-19T14:10:19.310+0100 I STORAGE [initandlisten] Verify succeeded on uri table:collection-2-7122844289700170543. Not salvaging.
2020-03-19T14:10:19.395+0100 I INDEX [initandlisten] build index on: local.startup_log properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2020-03-19T14:10:19.395+0100 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.417+0100 I STORAGE [initandlisten] Repairing collection local.system.replset
2020-03-19T14:10:19.418+0100 I STORAGE [initandlisten] Verify succeeded on uri table:collection-0--6199099221025852735. Not salvaging.
2020-03-19T14:10:19.906+0100 I INDEX [initandlisten] build index on: local.system.replset properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.system.replset" }
2020-03-19T14:10:19.906+0100 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.935+0100 I STORAGE [initandlisten] Repairing collection local.system.rollback.id
2020-03-19T14:10:19.957+0100 I STORAGE [initandlisten] Verify succeeded on uri table:collection-4-5553576768946172852. Not salvaging.
2020-03-19T14:10:20.183+0100 I INDEX [initandlisten] build index on: local.system.rollback.id properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.system.rollback.id" }
2020-03-19T14:10:20.183+0100 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:20.612+0100 W ASIO [initandlisten] No TransportLayer configured during NetworkInterface startup
2020-03-19T14:10:20.625+0100 I STORAGE [initandlisten] finished checking dbs
2020-03-19T14:10:20.642+0100 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
2020-03-19T14:10:20.642+0100 I STORAGE [initandlisten] WiredTigerKVEngine shutting down
2020-03-19T14:10:21.351+0100 I STORAGE [initandlisten] shutdown: removing fs lock...
2020-03-19T14:10:21.351+0100 I CONTROL [initandlisten] now exiting
2020-03-19T14:10:21.351+0100 I CONTROL [initandlisten] shutting down with code:0
but after I restart mongodb I get the same error message and this in the log :
>
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] MongoDB starting : pid=116383 port=27017 dbpath=/DATA/APPS/mongodb/data 64-bit host=serverhostname
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] db version v4.0.3
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] git version: 7ea530946fa7880364d88c8d8b6026bbc9ffa48c
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] allocator: tcmalloc
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] modules: none
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] build environment:
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] distmod: rhel70
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] distarch: x86_64
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] target_arch: x86_64
2020-03-19T19:16:51.586+0100 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1,server adress", port: 27017 }, processManagement: { fork: true, pidFilePath: "/DATA/APPS/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, replication: { replSetName: "rs0" }, security: { authorization: "enabled", keyFile: "/DATA/APPS/mongodb/mongo_key" }, storage: { dbPath: "/DATA/APPS/mongodb/data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/DATA/LOGS/mongodb/mongod.log" } }
2020-03-19T19:16:51.603+0100 I STORAGE [initandlisten] Detected data files in /DATA/APPS/mongodb/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-03-19T19:16:51.603+0100 I STORAGE [initandlisten]
2020-03-19T19:16:51.603+0100 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-03-19T19:16:51.603+0100 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-03-19T19:16:51.603+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7421M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-03-19T19:16:52.518+0100 I STORAGE [initandlisten] WiredTiger message [1584641812:518076][116383:0x7feaccaacb00], txn-recover: Main recovery loop: starting at 382/232960
2020-03-19T19:16:52.646+0100 I STORAGE [initandlisten] WiredTiger message [1584641812:646156][116383:0x7feaccaacb00], txn-recover: Recovering log 382 through 383
2020-03-19T19:16:52.745+0100 I STORAGE [initandlisten] WiredTiger message [1584641812:745035][116383:0x7feaccaacb00], txn-recover: Recovering log 383 through 383
2020-03-19T19:16:52.809+0100 I STORAGE [initandlisten] WiredTiger message [1584641812:809321][116383:0x7feaccaacb00], txn-recover: Set global recovery timestamp: 5e6929cc00000e81
2020-03-19T19:16:52.982+0100 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1583950284, 3713)
2020-03-19T19:16:52.982+0100 I STORAGE [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1583950284, 3713) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1583950284, 3713)
2020-03-19T19:16:53.920+0100 I STORAGE [initandlisten] Starting OplogTruncaterThread local.oplog.rs
2020-03-19T19:16:53.920+0100 I STORAGE [initandlisten] The size storer reports that the oplog contains 36328267 records totaling to 12368122855 bytes
2020-03-19T19:16:53.920+0100 I STORAGE [initandlisten] Sampling from the oplog between May 20 18:48:40:3841 and May 22 02:52:14:1 to determine where to place markers for truncation
2020-03-19T19:16:53.920+0100 I STORAGE [initandlisten] Taking 993 samples and assuming that each section of oplog contains approximately 365578 records totaling to 124462684 bytes
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 19:00:48:2660
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 19:12:51:3577
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 19:26:41:3667
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 19:38:50:2598
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 19:52:40:3379
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 20:04:49:1908
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 20:18:39:5892
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 20:30:49:1088
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 20:42:49:2575
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 20:56:39:3160
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 21:08:49:208
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 21:22:39:4765
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 21:34:40:10236
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 21:46:49:13951
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 22:00:40:1683
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 22:12:49:7749
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 22:24:49:10803
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 22:38:40:2549
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 22:50:49:12279
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 23:04:40:1035
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 23:16:48:5083
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 23:28:50:4431
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 23:42:40:11228
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 20 23:54:50:6299
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 00:08:40:6946
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 00:20:49:3604
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 00:34:39:1317
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 00:46:41:1695
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 00:58:50:11007
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 01:14:50:6893
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 01:30:40:4324
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 01:44:40:10600
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 01:56:49:16198
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 02:10:40:5712
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 02:22:49:2435
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 02:36:39:426
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 02:48:40:15917
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 03:00:50:7329
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 03:14:40:4516
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 03:26:49:2391
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 03:40:39:5006
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 03:52:41:6451
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 04:04:50:7206
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 04:18:40:8152
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 04:30:49:7674
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 04:44:40:19647
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 04:56:50:5726
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 05:10:40:11301
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 05:22:50:2862
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 05:36:40:894
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 05:48:49:8675
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 06:02:40:1593
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 06:14:40:4706
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 06:26:40:8794
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 06:38:50:640
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 06:52:40:2345
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 07:04:48:2055
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 07:16:40:8903
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 07:28:49:9618
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 07:42:39:1518
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 07:52:49:15625
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 08:06:40:4749
2020-03-19T19:16:54.035+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 08:18:49:5554
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 08:30:50:14374
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 08:44:40:14151
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 08:56:50:2952
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 09:10:40:7715
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 09:22:50:67
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 09:36:39:2319
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 09:48:49:1851
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 10:02:39:2805
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 10:14:40:3046
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 10:26:49:3759
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 10:40:39:3475
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 10:52:41:5094
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 11:04:49:2245
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 11:16:50:11889
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 11:30:40:12766
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 11:42:50:4428
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 11:56:39:5304
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 12:08:40:14920
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 12:20:50:5391
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 12:34:40:12954
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 12:44:49:14072
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 13:02:40:5198
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 13:26:50:4251
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 13:52:50:213
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 14:16:49:357
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 14:42:40:10459
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 15:02:49:9824
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 16:20:17:10859
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 17:27:14:7292
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 18:33:33:8149
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 19:25:26:1530
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 21:53:20:4576
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 21 22:45:57:4236
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 22 00:23:46:5235
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 22 01:03:17:21001
2020-03-19T19:16:54.036+0100 I STORAGE [initandlisten] Placing a marker at optime May 22 02:38:44:1429
2020-03-19T19:16:54.095+0100 I CONTROL [initandlisten]
2020-03-19T19:16:54.095+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2020-03-19T19:16:54.095+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2020-03-19T19:16:54.095+0100 I CONTROL [initandlisten]
2020-03-19T19:16:54.095+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2020-03-19T19:16:54.095+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2020-03-19T19:16:54.095+0100 I CONTROL [initandlisten]
2020-03-19T19:16:54.380+0100 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/DATA/APPS/mongodb/data/diagnostic.data'
2020-03-19T19:16:54.385+0100 I REPL [initandlisten] Rollback ID is 3
2020-03-19T19:16:54.385+0100 F - [initandlisten] Invariant failure !stableTimestamp || stableTimestamp->isNull() || appliedThrough.isNull() || *stableTimestamp == appliedThrough.getTimestamp() Stable timestamp Timestamp(1583950284, 3713) does not equal appliedThrough timestamp { ts: Timestamp(1558493534, 1), t: 13 } src/mongo/db/repl/replication_recovery.cpp 261
2020-03-19T19:16:54.385+0100 F - [initandlisten]
***aborting after invariant() failure
2020-03-19T19:16:54.428+0100 F - [initandlisten] Got signal: 6 (Aborted).
0x5625c5504321 0x5625c5503539 0x5625c5503a1d 0x7feacb2325d0 0x7feacae8c207 0x7feacae8d8f8 0x5625c3b38daf 0x5625c3e1d772 0x5625c3e57edc 0x5625c3e587b7 0x5625c3abebae 0x5625c3ba47d6 0x5625c3b3ada9 0x7feacae783d5 0x5625c3ba298f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5625C312C000","o":"23D8321","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5625C312C000","o":"23D7539"},{"b":"5625C312C000","o":"23D7A1D"},{"b":"7FEACB223000","o":"F5D0"},{"b":"7FEACAE56000","o":"36207","s":"gsignal"},{"b":"7FEACAE56000","o":"378F8","s":"abort"},{"b":"5625C312C000","o":"A0CDAF","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"5625C312C000","o":"CF1772","s":"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE"},{"b":"5625C312C000","o":"D2BEDC","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE"},{"b":"5625C312C000","o":"D2C7B7","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE"},{"b":"5625C312C000","o":"992BAE"},{"b":"5625C312C000","o":"A787D6","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"5625C312C000","o":"A0EDA9","s":"main"},{"b":"7FEACAE56000","o":"223D5","s":"__libc_start_main"},{"b":"5625C312C000","o":"A7698F"}],"processInfo":{ "mongodbVersion" : "4.0.3", "gitVersion" : "7ea530946fa7880364d88c8d8b6026bbc9ffa48c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.10.1.el7.x86_64", "version" : "#1 SMP Thu Feb 7 07:12:53 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "5625C312C000", "elfType" : 3, "buildId" : "0D41CD3EED00141900686FE4469C27B9210D4B9A" }, { "b" : "7FFFE969C000", "elfType" : 3, "buildId" : "63023E5151C4D5BBDB7F6F85A6F95A6B95DC780E" }, { "b" : "7FEACC650000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "85F01551F029D8E8DFAAE6739F31FFB96D27610E" }, { "b" : "7FEACC437000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C444AE61E7CBB716FD9C18A0B46A7FE8F4FCF3E5" }, { "b" : "7FEACBFD5000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3593FA778645A59EA272DBBB59D318C60940E792" }, { "b" : "7FEACBD63000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AEF5E6F2240B55F90E9DF76CFBB8B9D9F5286583" }, { "b" : "7FEACBB5F000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "7FEACB957000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7FEACB655000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "7FEACB43F000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "179F202998E429AA1215907F6D4C5C1BB9C90136" }, { "b" : "7FEACB223000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "7FEACAE56000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "DD9A6BA0D81C91F5CA7DBB4A1AC58319CC26DD5A" }, { "b" : "7FEACC8B9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" }, { "b" : "7FEACAC23000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "F4123103FB2318594448C44E47091DD68D1C78C0" }, { "b" : "7FEACA9F9000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "964423B6B0ED1FD8FA32E0B8E825A5D120D872F1" }, { "b" : "7FEACA7A7000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "2E28F6A705F2ECEA8460D4716D5D1C24B5DDA5E4" }, { "b" : "7FEACA580000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "8D0B4010959C321022DF9CE239277A9D7B34A76A" }, { "b" : "7FEACA253000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "F5A64BB37FA3972E545EF459A51310F0AB56FA56" }, { "b" : "7FEACA023000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "E0705772325A52C3372FFFB8BDE5F786E2E200D6" }, { "b" : "7FEAC9E1F000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "24835AAB32F52DF9EFACEBA6DE7E9134B302F877" }, { "b" : "7FEAC9C1A000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "C08A53AF54D3BDB4C2E924CCA4D0754F976FCF65" }, { "b" : "7FEAC99DC000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "775C8FB1A2D708968742CBFAF1436F5AB9AB3CC9" }, { "b" : "7FEAC978F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "BCC30853830CD911E58700591830DF51ABCBD7BA" }, { "b" : "7FEAC94A6000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "45BAB0BB455BDFA960FDA22E4124CF17B67CC930" }, { "b" : "7FEAC9273000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A9B3906192687CC45D483AE3C58C8AF745A6726A" }, { "b" : "7FEAC906F000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "C77BC26CE4D420861BAEBCC075C418BD9311BB5C" }, { "b" : "7FEAC8E60000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "8832509D0687D79342E29FC6FEC587EA85C04CF4" }, { "b" : "7FEAC8C0B000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "FC68D1DA42FB89A81E025368BCA66E5CD1AF82B6" }, { "b" : "7FEAC89F5000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7FEAC87E5000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "94B3BCB669126166B77CDCE6092679A6AA2004C8" }, { "b" : "7FEAC85E1000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8CA73C16CFEB9A8B5660015B9223B09F87041CAD" }, { "b" : "7FEAC83C4000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "9AF2AD92DADE046C6260DCCF02846BF78ABC658C" }, { "b" : "7FEAC819D000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7FEAC7F66000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "740CAD898E29E1F3B73A323CCEC4A7C88911647F" }, { "b" : "7FEAC7D04000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "F5B144F9F5D9BE451C80211B34DB2CE348E039B6" }, { "b" : "7FEAC7B01000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "B758881F4B6AF6C28C07A1A57713CBD2144628D4" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5625c5504321]
mongod(+0x23D7539) [0x5625c5503539]
mongod(+0x23D7A1D) [0x5625c5503a1d]
libpthread.so.0(+0xF5D0) [0x7feacb2325d0]
libc.so.6(gsignal+0x37) [0x7feacae8c207]
libc.so.6(abort+0x148) [0x7feacae8d8f8]
mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x5625c3b38daf]
mongod(_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE+0x6B2) [0x5625c3e1d772]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x54C) [0x5625c3e57edc]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE+0x247) [0x5625c3e587b7]
mongod(+0x992BAE) [0x5625c3abebae]
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x10D6) [0x5625c3ba47d6]
mongod(main+0x9) [0x5625c3b3ada9]
libc.so.6(__libc_start_main+0xF5) [0x7feacae783d5]
mongod(+0xA7698F) [0x5625c3ba298f]
----- END BACKTRACE -----
