DummyD3 is node3 of 3 node replication cluster. DummyD2 is primary and DummyD1 is Secondary (which is in sync with primary).
I tried deleting the data under data path and did rsync from other secondary node back to back twice to copy the updates while 1st was running in 2nd rsycn and started the deamon.
rsync -asvh Dummy1:/mongodb/data/dbinstance/* /mongodb/data/dbinstance
rsync -asvh Dummy1:/mongodb/data/dbinstance/* /mongodb/data/dbinstance
rm /mongodb/data/dbinstance/mongod.lock
rm /mongodb/data/dbinstance/mongod.pid
su - mongoinstance -c "/usr/bin/mongod --config /etc/mongo/dbinstance/mongodb.conf &"
Ran above command in screen from problematic node.
~# mongod --version
db version v4.0.4
I'm here looking for solution that if some has solution to help me fixing this issue.
root@DummyD3:~# tail /mongodb/files/errorlogs/dbinstance/mongodb.log
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55c2e748caa1]
mongod(+0x23D9CB9) [0x55c2e748bcb9]
mongod(+0x23DA19D) [0x55c2e748c19d]
libpthread.so.0(+0x11390) [0x7fa295382390]
libc.so.6(gsignal+0x38) [0x7fa294fdc428]
libc.so.6(abort+0x16A) [0x7fa294fde02a]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c2e5ac1ea7]
mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x413) [0x55c2e5bb8eb3]
mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0x2D1) [0x55c2e5b90161]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x63F) [0x55c2e5b9611f]
mongod(+0xAC6149) [0x55c2e5b78149]
mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x552) [0x55c2e62613a2]
mongod(+0x993251) [0x55c2e5a45251]
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x10DB) [0x55c2e5b2d94b]
mongod(main+0x9) [0x55c2e5ac3a39]
libc.so.6(__libc_start_main+0xF0) [0x7fa294fc7830]
mongod(_start+0x29) [0x55c2e5b2bab9]
----- END BACKTRACE -----