0

My Mongodb has disappeared 2 times, after 2 weeks, over the weekend within a 4 week time period.

I had a snapshot to recover from but I can't keep on recovering my db every 2 weeks. In an ec2 instance where I hosted Mongo and the API in the same place, this is not an issue.

Side note - does anyone know if it is possible to completely disable db.dropDatabase() or db.collection.drop()?

Below is my mongod.conf file

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /data
  logpath: /log/mongod.log
  journal:
    enabled: true
#  engine:
#  mmapv1:
   wiredTiger:
    prefixCompression: true
# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0
#  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.

Any ideas for how this happens or anyone who has ever solved a similar problem would be greatly appreciated.

var/log/mongodb/mongodb.log

2018-02-26T17:14:09.799+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] MongoDB starting : pid=# port=port dbpath=/var/lib/mongo 64-bit host=ip-*
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] db version v3.6.3
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] modules: none
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] build environment:
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten]     distmod: amazon
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-02-26T17:14:09.987+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-02-26T17:14:09.988+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1382M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten]
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten]
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten]
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten]
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-02-26T17:14:10.692+0000 I CONTROL  [initandlisten]
2018-02-26T17:14:10.693+0000 I STORAGE  [initandlisten] createCollection: admin.system.version with provided UUID: 038fb561-1163-46cb-bcae-b68ddebb4081
2018-02-26T17:14:10.700+0000 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 3.6
2018-02-26T17:14:10.703+0000 I STORAGE  [initandlisten] createCollection: local.startup_log with generated UUID: c994079b-4e9b-416f-91b4-4cbc60e2c118
2018-02-26T17:14:10.710+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2018-02-26T17:14:10.710+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2018-02-26T17:14:21.788+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:54282 #1 (1 connection now open)
2018-02-26T17:14:21.788+0000 I NETWORK  [conn1] received client metadata from 127.0.0.1:54282 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "CentOS Linux release 7.4.1708 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-693.11.6.el7.x86_64" } }
2018-02-26T17:14:23.882+0000 I NETWORK  [conn1] end connection 127.0.0.1:54282 (0 connections now open)
2018-02-26T17:19:10.711+0000 I STORAGE  [thread2] createCollection: config.system.sessions with generated UUID: b181a7ff-d0e5-456e-bee5-ff7e40f32d0a
2018-02-26T17:19:10.739+0000 I INDEX    [thread2] build index on: config.system.sessions properties: { v: 2, key: { lastUse: 1 }, name: "lsidTTLIndex", ns: "config.system.sessions", expireAfterSeconds: 1800 }
2018-02-26T17:19:10.739+0000 I INDEX    [thread2]        building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2018-02-26T17:19:10.740+0000 I INDEX    [thread2] build index done.  scanned 0 total records. 0 secs
2018-02-26T17:28:02.551+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:54292 #2 (1 connection now open)
2018-02-26T17:28:02.551+0000 I NETWORK  [conn2] received client metadata from 127.0.0.1:54292 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "CentOS Linux release 7.4.1708 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-693.11.6.el7.x86_64" } }
2018-02-26T17:33:15.093+0000 I NETWORK  [conn2] end connection 127.0.0.1:54292 (0 connections now open)
2018-02-26T17:34:20.689+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2018-02-26T17:34:20.689+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2018-02-26T17:34:20.689+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2018-02-26T17:34:20.690+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2018-02-26T17:34:20.692+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2018-02-26T17:34:20.763+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2018-02-26T17:34:20.763+0000 I CONTROL  [signalProcessingThread] now exiting
2018-02-26T17:34:20.763+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

journalctl return these red lines

Failed to create mount unit file /run/systemd/generator/data.mount, as it already exists. Duplicate entry in /etc/fstab? also for log.mount and journal.mount

piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr Error: Driver 'pcspkr' is already registered, aborting...

Improve this question
5
  • 1
    Is whole mongodb (program and databases) disappearing or just databases? After disappear, do still /var/log/mongodb/mongod.log exists? If yes, what that file and /var/log/syslog (or messages) tell about that incident?
    – JJussi
    Mar 14, 2018 at 19:24
  • I added the log as it currently appears, might be incorrect since I did have to restore the data from a backup snapshot
    – Brandon
    Mar 14, 2018 at 21:03
  • 1
    As per the log warning Access control is not enabled for the database, this deployment currently allows unrestricted access to anyone who is able to connect. It also has bindIP set for all interfaces (0.0.0.0), so if this instance has a public IP and no firewall you have opened up your deployment to the internet. That would be one likely reason for data randomly disappearing. I would review tutorials in the MongoDB Security Checklist and make sure you have enabled access control, network encryption, and a firewall.
    – Stennie
    Mar 15, 2018 at 0:17
  • I second with @Stennie, probably someone (person or script) contacted your mongodb instance and gave command to drop your database(s).
    – JJussi
    Mar 15, 2018 at 5:43
  • I have another weird question. After starting the backup, I was only able to run the database with sudo mongod --storageEngine wiredTiger --dbpath /data --bind_ip_all --fork --logpath /log/mongod.log because even though sudo service mongod start said okay, it would fail right after. It currently fails if I just overwrite the functioning start command. Is this because the command has to be cancel first or could there be a deeper issue?
    – Brandon
    Mar 15, 2018 at 13:04

1 Answer 1

Reset to default
1

Your latter problem (start with sudo) is because some of the files or directories are NOT owned by user mongod.

When you state sudo service mongod start mongod process is started with user mongod:mongod and now you have situation that some (or all) of the files/directories:

  • /data (storage.dbPath at config)
  • /log/mongod.log (systemLog.path at config)
  • /var/run/mongod/mongod.pid (processManagement.pidFilePath at config)

are not owned by user mongod

So, do multiple sudo chown -R mongod: XXXX (where XXX is right path/file) will fix this problem and your sudo service mongod restart command will work again.

Improve this answer
4
  • I tried the above on all 3 routes and after doing it to the pid route I received this error when trying to run sudo service mongod restart. sudo: /etc/sudo.conf is owned by uid 997, should be 0 sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set I am really scared because this post says I destroyed my whole system askubuntu.com/questions/452860/…
    – Brandon
    Mar 19, 2018 at 14:16
  • That error says that /etc/sudo.conf should be owned by root user, not anybody else! So, chown root: /etc/sudo.conf will fix that error. Anyway, why you changed owner of /etc/sudo.conf, it has nothing to do with our current mongod ownership problem.
    – JJussi
    Mar 19, 2018 at 15:48
  • I never sudo chown -R etc/sudo.conf, I only used the /data /log /.ipd routes as specified above. It appears that it ruined the pid as well and I could no longer ssh in afterwards. I had an ec2 snapshot and recovered from that afterwards, so in the end I didn't lose anything.
    – Brandon
    Mar 19, 2018 at 15:51
  • It looks like that somehow your chown command got "loose" and changed whole /etc/ -directory (and all sub-directories of it).
    – JJussi
    Mar 19, 2018 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.