-1

I deleted a bunch of records from my mongodb database running on Ubuntu 18.04 and I am trying to free up the space on disk with the 'compact' command from the console using: db.runCommand( { compact: 'mycollection' } ). After a few seconds mongod crashes and I get this message in the console:

2020-06-27T22:06:11.514+0000 I NETWORK  [js] DBClientConnection failed to receive message from 127.0.0.1:27017 - HostUnreachable: Connection closed by peer
2020-06-27T22:06:11.514+0000 E QUERY    [js] Error: error doing query: failed: network error while attempting to run command 'compact' on host '127.0.0.1:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:168:1
@(shell):1:1
2020-06-27T22:06:11.516+0000 I NETWORK  [js] trying reconnect to 127.0.0.1:27017 failed
2020-06-27T22:06:11.516+0000 I NETWORK  [js] reconnect 127.0.0.1:27017 failed failed

In the mongod log there is this:

2020-06-27T22:05:43.810+0000 I COMMAND  [conn3] compact mydb.mycollection begin, options: paddingMode: NONE validateDocuments: 1
2020-06-27T22:06:10.904+0000 E STORAGE  [conn3] WiredTiger error (16) [1593295570:904147][2674:0x7f49d818f700], WT_SESSION.compact: __compact_worker, 302: compaction halted by eviction pressure: Device or resource busy Raw: [1593295570:904147][2674:0x7f49d818f700], WT_SESSION.compact: __compact_worker, 302: compaction halted by eviction pressure: Device or resource busy
2020-06-27T22:06:10.911+0000 F -        [conn3] Invariant failure: ret resulted in status UnknownError: 16: Device or resource busy at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1516
2020-06-27T22:06:10.911+0000 F -        [conn3]

***aborting after invariant() failure


2020-06-27T22:06:10.966+0000 F -        [conn3] Got signal: 6 (Aborted).
 0x555e80508161 0x555e80507379 0x555e8050785d 0x7f49eabba890 0x7f49ea7f5e97 0x7f49ea7f7801 0x555e7eaca48c 0x555e7ebadf1a 0x555e7f261730 0x555e7f0c994b 0x555e7ff612d6 0x555e7ff67ce9 0x555e7eb802f3 0x555e7eb821d9 0x555e7eb83111 0x555e7eb6e87a 0x555e7eb7b52a 0x555e7eb761e7 0x555e7eb79a01 0x555e7fd525e2 0x555e7eb743d0 0x555e7eb77515 0x555e7eb75927 0x555e7eb7626d 0x555e7eb79a01 0x555e7fd52b45 0x555e8045ff24 0x7f49eabaf6db 0x7f49ea8d888f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"555E7E0A1000","o":"2467161","s":"_ZN5mongo15printStackTraceERSo"},{"b":"555E7E0A1000","o":"2466379"},{"b":"555E7E0A1000","o":"246685D"},{"b":"7F49EABA8000","o":"12890"},{"b":"7F49EA7B7000","o":"3EE97","s":"gsignal"},{"b":"7F49EA7B7000","o":"40801","s":"abort"},{"b":"555E7E0A1000","o":"A2948C","s":"_ZN5mongo24invariantOKFailedWithMsgEPKcRKNS_6StatusERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j"},{"b":"555E7E0A1000","o":"B0CF1A","s":"_ZN5mongo21WiredTigerRecordStore7compactEPNS_16OperationContextEPNS_25RecordStoreCompactAdaptorEPKNS_14CompactOptionsEPNS_12CompactStatsE"},{"b":"555E7E0A1000","o":"11C0730","s":"_ZN5mongo14CollectionImpl7compactEPNS_16OperationContextEPKNS_14CompactOptionsE"},{"b":"555E7E0A1000","o":"102894B","s":"_ZN5mongo10CompactCmd9errmsgRunEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERS8_RNS_14BSONObjBuilderE"},{"b":"555E7E0A1000","o":"1EC02D6","s":"_ZN5mongo23ErrmsgCommandDeprecated3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"555E7E0A1000","o":"1EC6CE9","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE"},{"b":"555E7E0A1000","o":"ADF2F3"},{"b":"555E7E0A1000","o":"AE11D9"},{"b":"555E7E0A1000","o":"AE2111","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"555E7E0A1000","o":"ACD87A","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"555E7E0A1000","o":"ADA52A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"555E7E0A1000","o":"AD51E7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"555E7E0A1000","o":"AD8A01"},{"b":"555E7E0A1000","o":"1CB15E2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"555E7E0A1000","o":"AD33D0","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"555E7E0A1000","o":"AD6515","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"555E7E0A1000","o":"AD4927","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"555E7E0A1000","o":"AD526D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"555E7E0A1000","o":"AD8A01"},{"b":"555E7E0A1000","o":"1CB1B45"},{"b":"555E7E0A1000","o":"23BEF24"},{"b":"7F49EABA8000","o":"76DB"},{"b":"7F49EA7B7000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.16", "gitVersion" : "2a5433168a53044cb6b4fa8083e4cfd7ba142221", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "5.3.0-1028-aws", "version" : "#30~18.04.1-Ubuntu SMP Mon Jun 22 15:48:21 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "555E7E0A1000", "elfType" : 3, "buildId" : "5A1022F045BA03A727AF3D468C844C20A2BB3DC8" }, { "b" : "7FFCFE5EE000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "DEBB4FBE352F8720D664E93DEF884ED0CB8F5148" }, { "b" : "7F49EC0FC000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "7AC7E2A143E253DF17105A3AFD3B63C9C419C802" }, { "b" : "7F49EBEE1000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F49EBA16000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "812108064C0E61DF52271B3147248999619E7AFF" }, { "b" : "7F49EB789000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "881B3F4FDA206DF3909B141DC7410A08C2DD4B90" }, { "b" : "7F49EB585000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F49EB37D000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F49EAFDF000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F49EADC7000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "679F3AE11120EC7C483BC9295345D836F5C104F7" }, { "b" : "7F49EABA8000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F49EA7B7000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F49EC37B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F49EA592000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F49EA375000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "EE6E9462BA2491F4EE8C4E52C3323274A9366614" }, { "b" : "7F49EA159000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F49E9F4B000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F49E9D00000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F49E9AAE000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "F406CC6CE3F33FC19238B7AC4A37D831F1BF0F06" }, { "b" : "7F49E98A0000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "8331348F98447F4D00BF7469EB6E2C27318B3EAC" }, { "b" : "7F49E9683000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F49E9305000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7F49E8F9F000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "1954013F9BBF9FF3A328A8E516481A2BBD8658EB" }, { "b" : "7F49E8D6B000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F49E8B35000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F49E88B4000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F49E85DE000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F49E83AC000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F49E81A8000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "17107881DF65C66B4C6D38CAB37C285FA44663BD" }, { "b" : "7F49E7F9D000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F49E7D82000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "6E98533B96F674F77C1BD83AA3565D974D9C4372" }, { "b" : "7F49E7B41000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F49E7812000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F49E75FF000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F49E73FB000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F49E71F2000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F49E6F65000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F49E6CC3000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F49E6A8D000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F49E6877000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F49E666F000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F49E6446000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F49E6237000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F49E5FED000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F49E5CE4000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "1E52E04B0722C6CFB8E534FB8DA45DDF95B46C19" }, { "b" : "7F49E5AAC000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x555e80508161]
 mongod(+0x2466379) [0x555e80507379]
 mongod(+0x246685D) [0x555e8050785d]
 libpthread.so.0(+0x12890) [0x7f49eabba890]
 libc.so.6(gsignal+0xC7) [0x7f49ea7f5e97]
 libc.so.6(abort+0x141) [0x7f49ea7f7801]
 mongod(_ZN5mongo24invariantOKFailedWithMsgEPKcRKNS_6StatusERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j+0x0) [0x555e7eaca48c]
 mongod(_ZN5mongo21WiredTigerRecordStore7compactEPNS_16OperationContextEPNS_25RecordStoreCompactAdaptorEPKNS_14CompactOptionsEPNS_12CompactStatsE+0xBA) [0x555e7ebadf1a]
 mongod(_ZN5mongo14CollectionImpl7compactEPNS_16OperationContextEPKNS_14CompactOptionsE+0x1A0) [0x555e7f261730]
 mongod(_ZN5mongo10CompactCmd9errmsgRunEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERS8_RNS_14BSONObjBuilderE+0x96B) [0x555e7f0c994b]
 mongod(_ZN5mongo23ErrmsgCommandDeprecated3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0x46) [0x555e7ff612d6]
 mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x555e7ff67ce9]
 mongod(+0xADF2F3) [0x555e7eb802f3]
 mongod(+0xAE11D9) [0x555e7eb821d9]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x3C1) [0x555e7eb83111]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3A) [0x555e7eb6e87a]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x555e7eb7b52a]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x555e7eb761e7]
 mongod(+0xAD8A01) [0x555e7eb79a01]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x555e7fd525e2]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x150) [0x555e7eb743d0]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB55) [0x555e7eb77515]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x555e7eb75927]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x555e7eb7626d]
 mongod(+0xAD8A01) [0x555e7eb79a01]
 mongod(+0x1CB1B45) [0x555e7fd52b45]
 mongod(+0x23BEF24) [0x555e8045ff24]
 libpthread.so.0(+0x76DB) [0x7f49eabaf6db]
 libc.so.6(clone+0x3F) [0x7f49ea8d888f]
-----  END BACKTRACE  -----

I tried searching online using the error messages but couldn't find anything relevant. Does anyone know what might be going on?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.