3

I have MongoDB 5.0.11 on Ubuntu 20.04 server inside of Virtualbox6.1. It has been humming along for months, but suddenly the mongod.service has refused to start.

root@test:~# systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: failed (Result: core-dump) since Tue 2022-09-06 06:48:19 HKT; 2s ago
       Docs: https://docs.mongodb.org/manual
    Process: 2952 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=dumped, signal=ABRT)
   Main PID: 2952 (code=dumped, signal=ABRT)

Sep 06 06:48:19 test.example.test systemd[1]: Started MongoDB Database Server.
Sep 06 06:48:19 test.example.test systemd[1]: mongod.service: Main process exited, code=dumped, status=6/ABRT
Sep 06 06:48:19 test.example.test systemd[1]: mongod.service: Failed with result 'core-dump'.

Verified that all of the permissions are correct against using my prod server with the same profile as the base.

/var/lib/mongodb
/var/log/mongodb
/tmp/mongodb-27017.sock

Are owned by mongodb user.

/etc/mongod.conf
is owned by root

ADDITIONAL INFO
The output of coredumpctl info per the suggestion/answer/reply below is:

           PID: 19220 (mongod)
           UID: 140 (mongodb)
           GID: 149 (mongodb)
        Signal: 6 (ABRT)
     Timestamp: Fri 2022-09-16 15:01:52 HKT (41s ago)
  Command Line: /usr/bin/mongod --config /etc/mongod.conf
    Executable: /usr/bin/mongod
 Control Group: /system.slice/mongod.service
          Unit: mongod.service
         Slice: system.slice
       Boot ID: de41eab6e854443eb761bc75c05c8bed
    Machine ID: eeb050f6814e4a8bae5526de9fd99eac
      Hostname: test.example.test
       Storage: /var/lib/systemd/coredump/core.mongod.140.de41eab6e854443eb761bc75c05c8bed.19220.1663311712000000000000.lz4
       Message: Process 19220 (mongod) of user 140 dumped core




                Stack trace of thread 19220:
                #0  0x00007f23f99e200b raise (libc.so.6 + 0x4300b)
                #1  0x00007f23f99c1859 abort (libc.so.6 + 0x22859)
                #2  0x0000555c58c4972d _ZN5mongo25fassertFailedWithLocationEiPKcj (mongod + 0x13e372d)
                #3  0x0000555c58737bd2 _ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1133 (mongod + 0xed1b>
                #4  0x0000555c58f52d23 __eventv (mongod + 0x16ecd23)
                #5  0x0000555c5874a1a4 __wt_panic_func (mongod + 0xee41a4)
                #6  0x0000555c58743c35 __wt_turtle_read.cold.7 (mongod + 0xeddc35)
                #7  0x0000555c58f1a384 __wt_turtle_validate_version (mongod + 0x16b4384)
                #8  0x0000555c58ecdc42 wiredtiger_open (mongod + 0x1667c42)
                #9  0x0000555c58e796f9 _ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_ (mongod + 0x16136f9)
                #10 0x0000555c58e85318 _ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb (mongod + 0x161f3>
                #11 0x0000555c58e5b591 _ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createEPNS_16OperationContextERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE (mo>
                #12 0x0000555c59c4ccb9 _ZN5mongo23initializeStorageEngineEPNS_16OperationContextENS_22StorageEngineInitFlagsE (mongod + 0x23e6cb9)
                #13 0x0000555c58dc4b3f _ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1929 (mongod + 0x155eb3f)
                #14 0x0000555c58dc74bf _ZN5mongo11mongod_mainEiPPc (mongod + 0x15614bf)
                #15 0x0000555c58c5c94e main (mongod + 0x13f694e)
                #16 0x00007f23f99c3083 __libc_start_main (libc.so.6 + 0x24083)
                #17 0x0000555c58dc18ee _start (mongod + 0x155b8ee)
                
                Stack trace of thread 19235:
                #0  0x00007f23f9ba07d1 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf7d1)
                #1  0x0000555c5b5b2127 _ZN5mongo11ClockSource21waitForConditionUntilERNS_4stdx18condition_variableENS_20BasicLockableAdapterENS_6Date_tEPNS_8WaitableE (mongo>
                #2  0x0000555c5963d5b2 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_18PeriodicRunnerImpl15PeriodicJobImpl4_runEvEUlvE0_JELi0E>
                #3  0x0000555c5b90505f execute_native_thread_routine (mongod + 0x409f05f)
                #4  0x00007f23f9b99609 start_thread (libpthread.so.0 + 0x8609)
                #5  0x00007f23f9abe133 __clone (libc.so.6 + 0x11f133)

                Stack trace of thread 19234:
                #0  0x00007f23f9ba07d1 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf7d1)
                #1  0x0000555c5b5b2127 _ZN5mongo11ClockSource21waitForConditionUntilERNS_4stdx18condition_variableENS_20BasicLockableAdapterENS_6Date_tEPNS_8WaitableE (mongo>
                #2  0x0000555c5963d5b2 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_18PeriodicRunnerImpl15PeriodicJobImpl4_runEvEUlvE0_JELi0E>
                #3  0x0000555c5b90505f execute_native_thread_routine (mongod + 0x409f05f)
                #4  0x00007f23f9b99609 start_thread (libpthread.so.0 + 0x8609)
                #5  0x00007f23f9abe133 __clone (libc.so.6 + 0x11f133)
                
                Stack trace of thread 19233:
                #0  0x00007f23f9ba0376 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf376)
                #1  0x0000555c5b90175c _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (mongod + 0x409b75c)
                #2  0x0000555c5b516566 _ZN5mongo10ThreadPool4Impl13_consumeTasksEv (mongod + 0x3cb0566)
                #3  0x0000555c5b516dac _ZN5mongo10ThreadPool4Impl17_workerThreadBodyERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE (mongod + 0x3cb0dac)
                #4  0x0000555c5b517350 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool4Impl25_startWorkerThread_inlockEvEUlvE2_JELi>
                #5  0x0000555c5b90505f execute_native_thread_routine (mongod + 0x409f05f)
                #6  0x00007f23f9b99609 start_thread (libpthread.so.0 + 0x8609)
                #7  0x00007f23f9abe133 __clone (libc.so.6 + 0x11f133)
                
                Stack trace of thread 19232:
                #0  0x00007f23f9ba0376 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf376)
                #1  0x0000555c5b90175c _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (mongod + 0x409b75c)
                #2  0x0000555c5b5b132f _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_27BackgroundThreadClockSource17_startTimerThreadEvEUlvE_J>
                #3  0x0000555c5b90505f execute_native_thread_routine (mongod + 0x409f05f)
                #4  0x00007f23f9b99609 start_thread (libpthread.so.0 + 0x8609)
                #5  0x00007f23f9abe133 __clone (libc.so.6 + 0x11f133)
                
                Stack trace of thread 19231:
                #0  0x00007f23f99e31a2 __sigtimedwait (libc.so.6 + 0x441a2)
                #1  0x0000555c59aa1b0f _ZN5mongo12_GLOBAL__N_122signalProcessingThreadENS_13LogFileStatusE (mongod + 0x223bb0f)
                #2  0x0000555c59aa2655 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IPFvNS3_13LogFileStatusEEJRS7_ELi0EEET_DpOT0_EUlvE_EEEEE6_M_run>
                #3  0x0000555c5b90505f execute_native_thread_routine (mongod + 0x409f05f)
                #4  0x00007f23f9b99609 start_thread (libpthread.so.0 + 0x8609)
                #5  0x00007f23f9abe133 __clone (libc.so.6 + 0x11f133)

I can find no reason why it would just quit... 🤔🤔🤔

Error log output:

{"t":{"$date":"2022-09-07T06:42:52.028+08:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"-","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2022-09-07T06:42:52.033+08:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2022-09-07T06:42:52.041+08:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2022-09-07T06:42:52.045+08:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-09-07T06:42:52.046+08:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2022-09-07T06:42:52.204+08:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-09-07T06:42:52.204+08:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2022-09-07T06:42:52.204+08:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2022-09-07T06:42:52.204+08:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2022-09-07T06:42:52.207+08:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":155263,"port":27017,"dbPath":"/var/lib/mongodb","architecture":"64-bit","host":"test.example.test"}}
{"t":{"$date":"2022-09-07T06:42:52.207+08:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.12","gitVersion":"79cfcdd83eb6f64e164a588d0daf9bb873328b45","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2022-09-07T06:42:52.207+08:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2022-09-07T06:42:52.207+08:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"security":{"authorization":"enabled"},"storage":{"dbPath":"/var/lib/mongodb","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"path":"/var/log/mongodb/mongod.log"}}}}
{"t":{"$date":"2022-09-07T06:42:52.208+08:00"},"s":"E",  "c":"NETWORK",  "id":23024,   "ctx":"initandlisten","msg":"Failed to unlink socket file","attr":{"path":"/tmp/mongodb-27017.sock","error":"Operation not permitted"}}
{"t":{"$date":"2022-09-07T06:42:52.208+08:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":40486,"file":"src/mongo/transport/transport_layer_asio.cpp","line":1019}}
{"t":{"$date":"2022-09-07T06:42:52.208+08:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

Also, the error message seems to have mysteriously changed:

root@test:~# systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2022-09-07 06:42:52 HKT; 35s ago
       Docs: https://docs.mongodb.org/manual
    Process: 155263 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=14)
   Main PID: 155263 (code=exited, status=14)

Sep 07 06:42:51 test.example.test systemd[1]: Started MongoDB Database Server.
Sep 07 06:42:52 test.example.test systemd[1]: mongod.service: Main process exited, code=exited, status=14/n/a
Sep 07 06:42:52 test.example.test systemd[1]: mongod.service: Failed with result 'exit-code'.

Anyone know how I can get it running again?

Update
I got mongodb to fully function for about 2 minutes, then it broke again. I have no idea as to the cause for the MongoDB to "core dump" for 2 days, then suddenly change error message with nothing actual being done by me...

What I finally did to get it going again for two minutes was:
chown -R mongodb:mongodb /var/lib/mongodb and chown mongodb:mongodb /tmp/mongodb-27017.sock

Broken Again
The solution above is temporary. It is now right back to "core dumping" with the original error above...

baffled 😧😧

Improve this question
7
  • Did you check the logfile in /var/log/mongodb? Try to start it directly with /usr/bin/mongod --config /etc/mongod.conf
    – Wernfried Domscheit
    Sep 6, 2022 at 15:12
  • File /tmp/mongodb-27017.sock should exist only when mongod is running. Try to delete it.
    – Wernfried Domscheit
    Sep 6, 2022 at 15:13
  • thanks for the feedback ... starting directly with /usr/bin/mongod --config /etc/mongod.conf = core dump... tried deleting /tmp/mongodb-27017.sock and restarting = core dump
    – yupthatguy
    Sep 6, 2022 at 15:53
  • And what do you get in the log file?
    – Wernfried Domscheit
    Sep 6, 2022 at 16:38
  • Sorry about that.. I'm Asia.. it was bed time.... posted info above.
    – yupthatguy
    Sep 6, 2022 at 22:57

2 Answers 2

Reset to default
0

I recommend you try to get at least basic core dump in such case. Easiest way on ubuntu would probably be:

apt install systemd-coredump

After attempt to start Mongo, you can then provide at least basic info with:

coredumpctl info

I hit this question as I have been dealing with same problem, with important difference. It happened to me just after upgrade from 4.4 to 5. It never was working on 5.

So at least for couple people which will have same problem as me, here is relevant coredumpctl info:

❯ coredumpctl info
           PID: 7264 (mongod)
           UID: 111 (mongodb)
           GID: 118 (mongodb)
        Signal: 4 (ILL)
     Timestamp: Thu 2022-09-15 14:01:55 UTC (1h 17min ago)
  Command Line: /usr/bin/mongod --config /etc/mongod.conf
    Executable: /usr/bin/mongod
 Control Group: /system.slice/mongod.service
          Unit: mongod.service
         Slice: system.slice
       Boot ID: 9e99b117fb294c91b79ee40b727265f8
    Machine ID: 2fe1195218684d8094c050113dd34aa3
      Hostname: buxton
       Storage: /var/lib/systemd/coredump/core.mongod.111.9e99b117fb294c91b79ee40b727265f8.7264.1663250515000000.lz4
       Message: Process 7264 (mongod) of user 111 dumped core.

                Stack trace of thread 7264:
                #0  0x000055b099ae898a _ZN8tcmalloc7SizeMap4InitEv (mongod)
                #1  0x000055b099af12a7 _ZN8tcmalloc6Static14InitStaticVarsEv (mongod)
                #2  0x000055b099af2cf7 _ZN8tcmalloc11ThreadCache10InitModuleEv (mongod)
                #3  0x000055b099af2e8d _ZN8tcmalloc11ThreadCache22CreateCacheIfNecessaryEv (mongod)
                #4  0x000055b099b9c4f5 _ZN8tcmalloc24allocate_full_malloc_oomEm (mongod)
                #5  0x00007f2c28b3080f n/a (libc.so.6)
                #6  0x00007f2c28b30ab1 bindtextdomain (libc.so.6)
                #7  0x00007f2c27d91bb7 n/a (libgpg-error.so.0)
                #8  0x00007f2c292f937a n/a (ld-linux-x86-64.so.2)
                #9  0x00007f2c292f9476 n/a (ld-linux-x86-64.so.2)
                #10 0x00007f2c292eb0ca n/a (ld-linux-x86-64.so.2)

When googling I hit this thread which get me to right track (in my case). We had one MongoDB node on VM which had been placed on older hardware with no AVX CPU instruction support.

And MongoDB requires AVX instruction support on x86_64 since 5.0.

If you didn't change Virtualbox settings between your attempts, I belive this isn't relevant to your case as MongoDB 5 doesn't even start without AVX. And it did for you.

Improve this answer
2
  • thanks for the reply... still no clue as to fixing this
    – yupthatguy
    Sep 16, 2022 at 7:11
  • I recommend to submit issue at Mongo, they seem quite reponsive and would IMHO be much faster in helping you when you provide all details.
    – Kepi
    Sep 21, 2022 at 21:15
0

I had the same problem

But this post can help you: Setting up MongoDB v5.0 on Ubuntu 20: “core-dump: STATUS 4/ILL”

The problem It's basically a CPU downgrade, where it doesn't support some instructions required by MongoDB 5.0 (which are AVX and SSE).

In Stennie_X words (ex-MongoDB Community Team):

Some options to consider are:

  • Try installing the MongoDB 4.4 server packages (they target older CPU microarchitectures)
  • Upgrade your VM or hosting to support the required server extensions
  • Compile MongoDB 5.0 from source using older architecture tags 135 (this may be tricky given your virtual CPU seems to have limited instruction support)
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.