1

I have a primary node and second node of mongodb in Production , and a third Node which is the arbiter , after an unplanned shutdown on production last week , the Arbiter node refuse to start mongodb service , I can start the primary and second node , but the arbiter don't start and I get the error below:

> > 2020-03-17T12:33:45.634+0100 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
about to fork child process, waiting until server is ready for connections.
forked process: 31746
ERROR: child process failed, exited with error number 51
To see additional information in this output, start without the "--fork" option.

someone knows how to solve this issue ?? I have repaired the Database , below the output of reparation

>  2020-03-19T14:10:18.381+0100 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-6--6199099221025852735. Not salvaging.
2020-03-19T14:10:18.384+0100 I STORAGE  [initandlisten] Repairing collection local.replset.election
2020-03-19T14:10:18.651+0100 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0--1928215778901914464. Not salvaging.
2020-03-19T14:10:18.813+0100 I INDEX    [initandlisten] build index on: local.replset.election properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.replset.election" }
2020-03-19T14:10:18.813+0100 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:18.913+0100 I STORAGE  [initandlisten] Repairing collection local.replset.minvalid
2020-03-19T14:10:18.953+0100 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-2-5553576768946172852. Not salvaging.
2020-03-19T14:10:19.058+0100 I INDEX    [initandlisten] build index on: local.replset.minvalid properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.replset.minvalid" }
2020-03-19T14:10:19.058+0100 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.112+0100 I STORAGE  [initandlisten] Repairing collection local.replset.oplogTruncateAfterPoint
2020-03-19T14:10:19.121+0100 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-5553576768946172852. Not salvaging.
2020-03-19T14:10:19.228+0100 I INDEX    [initandlisten] build index on: local.replset.oplogTruncateAfterPoint properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.replset.oplogTruncateAfterPoint" }
2020-03-19T14:10:19.228+0100 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.279+0100 I STORAGE  [initandlisten] Repairing collection local.startup_log
2020-03-19T14:10:19.310+0100 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-2-7122844289700170543. Not salvaging.
2020-03-19T14:10:19.395+0100 I INDEX    [initandlisten] build index on: local.startup_log properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2020-03-19T14:10:19.395+0100 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.417+0100 I STORAGE  [initandlisten] Repairing collection local.system.replset
2020-03-19T14:10:19.418+0100 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0--6199099221025852735. Not salvaging.
2020-03-19T14:10:19.906+0100 I INDEX    [initandlisten] build index on: local.system.replset properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.system.replset" }
2020-03-19T14:10:19.906+0100 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:19.935+0100 I STORAGE  [initandlisten] Repairing collection local.system.rollback.id
2020-03-19T14:10:19.957+0100 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-4-5553576768946172852. Not salvaging.
2020-03-19T14:10:20.183+0100 I INDEX    [initandlisten] build index on: local.system.rollback.id properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.system.rollback.id" }
2020-03-19T14:10:20.183+0100 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2020-03-19T14:10:20.612+0100 W ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
2020-03-19T14:10:20.625+0100 I STORAGE  [initandlisten] finished checking dbs
2020-03-19T14:10:20.642+0100 I STORAGE  [WTOplogJournalThread] oplog journal thread loop shutting down
2020-03-19T14:10:20.642+0100 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2020-03-19T14:10:21.351+0100 I STORAGE  [initandlisten] shutdown: removing fs lock...
2020-03-19T14:10:21.351+0100 I CONTROL  [initandlisten] now exiting
2020-03-19T14:10:21.351+0100 I CONTROL  [initandlisten] shutting down with code:0

but after I restart mongodb I get the same error message and this in the log :

>

 2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] MongoDB starting : pid=116383 port=27017 dbpath=/DATA/APPS/mongodb/data 64-bit host=serverhostname
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] db version v4.0.3
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] git version: 7ea530946fa7880364d88c8d8b6026bbc9ffa48c
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] modules: none
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] build environment:
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten]     distmod: rhel70
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten]     distarch: x86_64
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten]     target_arch: x86_64
2020-03-19T19:16:51.586+0100 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1,server adress", port: 27017 }, processManagement: { fork: true, pidFilePath: "/DATA/APPS/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, replication: { replSetName: "rs0" }, security: { authorization: "enabled", keyFile: "/DATA/APPS/mongodb/mongo_key" }, storage: { dbPath: "/DATA/APPS/mongodb/data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/DATA/LOGS/mongodb/mongod.log" } }
2020-03-19T19:16:51.603+0100 I STORAGE  [initandlisten] Detected data files in /DATA/APPS/mongodb/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-03-19T19:16:51.603+0100 I STORAGE  [initandlisten] 
2020-03-19T19:16:51.603+0100 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-03-19T19:16:51.603+0100 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-03-19T19:16:51.603+0100 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7421M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-03-19T19:16:52.518+0100 I STORAGE  [initandlisten] WiredTiger message [1584641812:518076][116383:0x7feaccaacb00], txn-recover: Main recovery loop: starting at 382/232960
2020-03-19T19:16:52.646+0100 I STORAGE  [initandlisten] WiredTiger message [1584641812:646156][116383:0x7feaccaacb00], txn-recover: Recovering log 382 through 383
2020-03-19T19:16:52.745+0100 I STORAGE  [initandlisten] WiredTiger message [1584641812:745035][116383:0x7feaccaacb00], txn-recover: Recovering log 383 through 383
2020-03-19T19:16:52.809+0100 I STORAGE  [initandlisten] WiredTiger message [1584641812:809321][116383:0x7feaccaacb00], txn-recover: Set global recovery timestamp: 5e6929cc00000e81
2020-03-19T19:16:52.982+0100 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1583950284, 3713)
2020-03-19T19:16:52.982+0100 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1583950284, 3713) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1583950284, 3713)
2020-03-19T19:16:53.920+0100 I STORAGE  [initandlisten] Starting OplogTruncaterThread local.oplog.rs
2020-03-19T19:16:53.920+0100 I STORAGE  [initandlisten] The size storer reports that the oplog contains 36328267 records totaling to 12368122855 bytes
2020-03-19T19:16:53.920+0100 I STORAGE  [initandlisten] Sampling from the oplog between May 20 18:48:40:3841 and May 22 02:52:14:1 to determine where to place markers for truncation
2020-03-19T19:16:53.920+0100 I STORAGE  [initandlisten] Taking 993 samples and assuming that each section of oplog contains approximately 365578 records totaling to 124462684 bytes
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 19:00:48:2660
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 19:12:51:3577
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 19:26:41:3667
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 19:38:50:2598
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 19:52:40:3379
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 20:04:49:1908
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 20:18:39:5892
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 20:30:49:1088
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 20:42:49:2575
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 20:56:39:3160
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 21:08:49:208
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 21:22:39:4765
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 21:34:40:10236
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 21:46:49:13951
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 22:00:40:1683
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 22:12:49:7749
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 22:24:49:10803
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 22:38:40:2549
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 22:50:49:12279
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 23:04:40:1035
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 23:16:48:5083
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 23:28:50:4431
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 23:42:40:11228
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 20 23:54:50:6299
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 00:08:40:6946
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 00:20:49:3604
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 00:34:39:1317
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 00:46:41:1695
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 00:58:50:11007
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 01:14:50:6893
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 01:30:40:4324
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 01:44:40:10600
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 01:56:49:16198
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 02:10:40:5712
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 02:22:49:2435
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 02:36:39:426
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 02:48:40:15917
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 03:00:50:7329
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 03:14:40:4516
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 03:26:49:2391
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 03:40:39:5006
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 03:52:41:6451
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 04:04:50:7206
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 04:18:40:8152
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 04:30:49:7674
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 04:44:40:19647
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 04:56:50:5726
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 05:10:40:11301
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 05:22:50:2862
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 05:36:40:894
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 05:48:49:8675
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 06:02:40:1593
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 06:14:40:4706
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 06:26:40:8794
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 06:38:50:640
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 06:52:40:2345
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 07:04:48:2055
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 07:16:40:8903
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 07:28:49:9618
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 07:42:39:1518
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 07:52:49:15625
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 08:06:40:4749
2020-03-19T19:16:54.035+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 08:18:49:5554
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 08:30:50:14374
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 08:44:40:14151
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 08:56:50:2952
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 09:10:40:7715
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 09:22:50:67
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 09:36:39:2319
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 09:48:49:1851
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 10:02:39:2805
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 10:14:40:3046
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 10:26:49:3759
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 10:40:39:3475
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 10:52:41:5094
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 11:04:49:2245
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 11:16:50:11889
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 11:30:40:12766
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 11:42:50:4428
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 11:56:39:5304
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 12:08:40:14920
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 12:20:50:5391
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 12:34:40:12954
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 12:44:49:14072
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 13:02:40:5198
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 13:26:50:4251
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 13:52:50:213
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 14:16:49:357
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 14:42:40:10459
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 15:02:49:9824
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 16:20:17:10859
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 17:27:14:7292
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 18:33:33:8149
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 19:25:26:1530
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 21:53:20:4576
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 21 22:45:57:4236
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 22 00:23:46:5235
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 22 01:03:17:21001
2020-03-19T19:16:54.036+0100 I STORAGE  [initandlisten] Placing a marker at optime May 22 02:38:44:1429
2020-03-19T19:16:54.095+0100 I CONTROL  [initandlisten] 
2020-03-19T19:16:54.095+0100 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2020-03-19T19:16:54.095+0100 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2020-03-19T19:16:54.095+0100 I CONTROL  [initandlisten] 
2020-03-19T19:16:54.095+0100 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2020-03-19T19:16:54.095+0100 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2020-03-19T19:16:54.095+0100 I CONTROL  [initandlisten] 
2020-03-19T19:16:54.380+0100 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/DATA/APPS/mongodb/data/diagnostic.data'
2020-03-19T19:16:54.385+0100 I REPL     [initandlisten] Rollback ID is 3
2020-03-19T19:16:54.385+0100 F -        [initandlisten] Invariant failure !stableTimestamp || stableTimestamp->isNull() || appliedThrough.isNull() || *stableTimestamp == appliedThrough.getTimestamp() Stable timestamp Timestamp(1583950284, 3713) does not equal appliedThrough timestamp { ts: Timestamp(1558493534, 1), t: 13 } src/mongo/db/repl/replication_recovery.cpp 261
2020-03-19T19:16:54.385+0100 F -        [initandlisten] 

***aborting after invariant() failure


2020-03-19T19:16:54.428+0100 F -        [initandlisten] Got signal: 6 (Aborted).
 0x5625c5504321 0x5625c5503539 0x5625c5503a1d 0x7feacb2325d0 0x7feacae8c207 0x7feacae8d8f8 0x5625c3b38daf 0x5625c3e1d772 0x5625c3e57edc 0x5625c3e587b7 0x5625c3abebae 0x5625c3ba47d6 0x5625c3b3ada9 0x7feacae783d5 0x5625c3ba298f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5625C312C000","o":"23D8321","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5625C312C000","o":"23D7539"},{"b":"5625C312C000","o":"23D7A1D"},{"b":"7FEACB223000","o":"F5D0"},{"b":"7FEACAE56000","o":"36207","s":"gsignal"},{"b":"7FEACAE56000","o":"378F8","s":"abort"},{"b":"5625C312C000","o":"A0CDAF","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"5625C312C000","o":"CF1772","s":"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE"},{"b":"5625C312C000","o":"D2BEDC","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE"},{"b":"5625C312C000","o":"D2C7B7","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE"},{"b":"5625C312C000","o":"992BAE"},{"b":"5625C312C000","o":"A787D6","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"5625C312C000","o":"A0EDA9","s":"main"},{"b":"7FEACAE56000","o":"223D5","s":"__libc_start_main"},{"b":"5625C312C000","o":"A7698F"}],"processInfo":{ "mongodbVersion" : "4.0.3", "gitVersion" : "7ea530946fa7880364d88c8d8b6026bbc9ffa48c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.10.1.el7.x86_64", "version" : "#1 SMP Thu Feb 7 07:12:53 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "5625C312C000", "elfType" : 3, "buildId" : "0D41CD3EED00141900686FE4469C27B9210D4B9A" }, { "b" : "7FFFE969C000", "elfType" : 3, "buildId" : "63023E5151C4D5BBDB7F6F85A6F95A6B95DC780E" }, { "b" : "7FEACC650000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "85F01551F029D8E8DFAAE6739F31FFB96D27610E" }, { "b" : "7FEACC437000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C444AE61E7CBB716FD9C18A0B46A7FE8F4FCF3E5" }, { "b" : "7FEACBFD5000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3593FA778645A59EA272DBBB59D318C60940E792" }, { "b" : "7FEACBD63000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AEF5E6F2240B55F90E9DF76CFBB8B9D9F5286583" }, { "b" : "7FEACBB5F000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "7FEACB957000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7FEACB655000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "7FEACB43F000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "179F202998E429AA1215907F6D4C5C1BB9C90136" }, { "b" : "7FEACB223000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "7FEACAE56000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "DD9A6BA0D81C91F5CA7DBB4A1AC58319CC26DD5A" }, { "b" : "7FEACC8B9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" }, { "b" : "7FEACAC23000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "F4123103FB2318594448C44E47091DD68D1C78C0" }, { "b" : "7FEACA9F9000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "964423B6B0ED1FD8FA32E0B8E825A5D120D872F1" }, { "b" : "7FEACA7A7000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "2E28F6A705F2ECEA8460D4716D5D1C24B5DDA5E4" }, { "b" : "7FEACA580000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "8D0B4010959C321022DF9CE239277A9D7B34A76A" }, { "b" : "7FEACA253000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "F5A64BB37FA3972E545EF459A51310F0AB56FA56" }, { "b" : "7FEACA023000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "E0705772325A52C3372FFFB8BDE5F786E2E200D6" }, { "b" : "7FEAC9E1F000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "24835AAB32F52DF9EFACEBA6DE7E9134B302F877" }, { "b" : "7FEAC9C1A000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "C08A53AF54D3BDB4C2E924CCA4D0754F976FCF65" }, { "b" : "7FEAC99DC000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "775C8FB1A2D708968742CBFAF1436F5AB9AB3CC9" }, { "b" : "7FEAC978F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "BCC30853830CD911E58700591830DF51ABCBD7BA" }, { "b" : "7FEAC94A6000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "45BAB0BB455BDFA960FDA22E4124CF17B67CC930" }, { "b" : "7FEAC9273000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A9B3906192687CC45D483AE3C58C8AF745A6726A" }, { "b" : "7FEAC906F000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "C77BC26CE4D420861BAEBCC075C418BD9311BB5C" }, { "b" : "7FEAC8E60000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "8832509D0687D79342E29FC6FEC587EA85C04CF4" }, { "b" : "7FEAC8C0B000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "FC68D1DA42FB89A81E025368BCA66E5CD1AF82B6" }, { "b" : "7FEAC89F5000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7FEAC87E5000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "94B3BCB669126166B77CDCE6092679A6AA2004C8" }, { "b" : "7FEAC85E1000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8CA73C16CFEB9A8B5660015B9223B09F87041CAD" }, { "b" : "7FEAC83C4000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "9AF2AD92DADE046C6260DCCF02846BF78ABC658C" }, { "b" : "7FEAC819D000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7FEAC7F66000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "740CAD898E29E1F3B73A323CCEC4A7C88911647F" }, { "b" : "7FEAC7D04000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "F5B144F9F5D9BE451C80211B34DB2CE348E039B6" }, { "b" : "7FEAC7B01000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "B758881F4B6AF6C28C07A1A57713CBD2144628D4" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5625c5504321]
 mongod(+0x23D7539) [0x5625c5503539]
 mongod(+0x23D7A1D) [0x5625c5503a1d]
 libpthread.so.0(+0xF5D0) [0x7feacb2325d0]
 libc.so.6(gsignal+0x37) [0x7feacae8c207]
 libc.so.6(abort+0x148) [0x7feacae8d8f8]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x5625c3b38daf]
 mongod(_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE+0x6B2) [0x5625c3e1d772]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x54C) [0x5625c3e57edc]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE+0x247) [0x5625c3e587b7]
 mongod(+0x992BAE) [0x5625c3abebae]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x10D6) [0x5625c3ba47d6]
 mongod(main+0x9) [0x5625c3b3ada9]
 libc.so.6(__libc_start_main+0xF5) [0x7feacae783d5]
 mongod(+0xA7698F) [0x5625c3ba298f]
-----  END BACKTRACE  -----
0

It's an arbiter, it has no data so you could do a remove arbiter from the primary then yum remove mongo, then blow the whole folder away and start over.

  • it's not me who configured it at the first , so I don't know what this will impact on the other nodes , and even if it's the arbiter , it contains huge files in Data repository , don't what is it – AbdelAziz ESSAADI Mar 20 at 15:04
  • Yes, the data directory has many files and a size of some hundred MByte. Nevertheless you can delete it without any concern. – Wernfried Domscheit Mar 20 at 18:44
  • I'm waitin the Storage and backup team to tell me if they have a backup of the server to restore it from a specific data , I think it's better – AbdelAziz ESSAADI Mar 21 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.