Unable to start mongod daemon and it looks its crashing

Question

DummyD3 is node3 of 3 node replication cluster. DummyD2 is primary and DummyD1 is Secondary (which is in sync with primary).

I tried deleting the data under data path and did rsync from other secondary node back to back twice to copy the updates while 1st was running in 2nd rsycn and started the deamon.

rsync -asvh Dummy1:/mongodb/data/dbinstance/* /mongodb/data/dbinstance
rsync -asvh Dummy1:/mongodb/data/dbinstance/* /mongodb/data/dbinstance
rm /mongodb/data/dbinstance/mongod.lock
rm /mongodb/data/dbinstance/mongod.pid
su - mongoinstance -c "/usr/bin/mongod --config /etc/mongo/dbinstance/mongodb.conf &"

Ran above command in screen from problematic node.

~# mongod --version
db version v4.0.4

I'm here looking for solution that if some has solution to help me fixing this issue.

root@DummyD3:~# tail /mongodb/files/errorlogs/dbinstance/mongodb.log 

 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55c2e748caa1]
 mongod(+0x23D9CB9) [0x55c2e748bcb9]
 mongod(+0x23DA19D) [0x55c2e748c19d]
 libpthread.so.0(+0x11390) [0x7fa295382390]
 libc.so.6(gsignal+0x38) [0x7fa294fdc428]
 libc.so.6(abort+0x16A) [0x7fa294fde02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c2e5ac1ea7]
 mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x413) [0x55c2e5bb8eb3]
 mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0x2D1) [0x55c2e5b90161]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x63F) [0x55c2e5b9611f]
 mongod(+0xAC6149) [0x55c2e5b78149]
 mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x552) [0x55c2e62613a2]
 mongod(+0x993251) [0x55c2e5a45251]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x10DB) [0x55c2e5b2d94b]
 mongod(main+0x9) [0x55c2e5ac3a39]
 libc.so.6(__libc_start_main+0xF0) [0x7fa294fc7830]
 mongod(_start+0x29) [0x55c2e5b2bab9]
-----  END BACKTRACE  -----

Answer 1

Wernfried Domscheit said:

You don't need to copy any data in a replicat set. Just remove entire directory and start the service. The replicat set will trigger an inital sync and copies all data. In case you like to take a copy from dbPath you should stop the service or at least lock it with db.fsyncLock() (and of course unlock afterwards with db.fsyncUnlock())

db.fsyncLock() & db.fsyncUnlock() did work and I fixed the chronic replication.