23

Okay, so after following the clues given by loicmathieu and jstell, and digging it up a little, these are the things I found out about MongoDB using WiredTiger storage engine. I'm putting it here if anyone encountered the same questions. The memory usage threads that I mentioned, all belonged to 2012-2014, all pre-date WiredTiger and are describing behavior ...


4

Docs You may like to read basic memory concerns for MongoDB and also this brief discussion about checking memory usage. Memory usage overview The command db.serverStatus() (docs) can provide an overview of memory usage, specifically: > db.serverStatus().mem { "bits" : 64, "resident" : 27, "virtual" : 397, "supported" : true } > db.serverStatus()....


4

Community wiki answer: *.wt are binary data files used by the WiredTiger storage engine. Individual files are not usable as a standalone backup. If you want to take a file copy backup of a MongoDB dbPath you need to include all of the files using a Supported File Copy Backup Method. If you have a valid file backup you can use it as the dbPath for another ...


4

I don't think you have a problem here with MongoDB, as jstell told you MongoDB with WiredTiger will use 50% of available memory so if you increase the RAM of your server it will takes more memory. As why it's more than the size of DB + indexes, keep in mind that WiredTiger compress the database on disk and also use snapshot logs to record document changes. ...


3

I wasn't able to use wiredTiger upgrading MongoDB. However, at May 17 I uninstalled MongoDB 2.6 then installed MongoDB 3.0.3 . Immediate after the installation, I added storageEngine=wiredTiger on top of my mongod.conf file. Then I gave sudo service mongod start command and eventually I could. Edit: For fresh installed as directed by official documentation;...


3

Strange indeed. That local.oplog.rs is journaling what is enable (default) at 64 bit systems (32bit default value is false). So, running time for that thread should be same than your mongod uptime. But journaling should not rise CPU usage. When you have high CPU usage, what mongotop and mongostat shows? What is happening in the system? If you look ...


3

The number records how many times two or more threads are trying to update the same document inside the WiredTiger storage engine. It is not persisted between server restarts. This statistic is specific to the WiredTiger storage engine (as it is under the WiredTiger section). Under MongoDB this is known as write conflict, and will be transparently retried ...


2

The WiredTiger.wt file is a crucial metadata file that keeps track of the state of the whole database and all the *.wt files. Hence, the content of this file is deployment specific, and not transferable to other deployments. It is not possible to recreate the content of this file. If you have a known good backup of the whole dbpath, you may be able to ...


2

By default collection stats only include information on the collection data (which uses block-level compression rather than prefix compression). To see index details you need to provide an additional indexDetails option, eg: db.myCollection.stats({indexDetails:true}).indexDetails._id_.creationString You should see "prefix_compression=true" in the index ...


2

Do we need to keep extra memory for some kind of memory mapped files for MongoDB data files The mem.mapped and mem.mappedWithJournal metrics in serverStatus output are only applicable to the MMAPv1 storage engine, so are expected to be 0 with WiredTiger. These metrics predate the storage engine API (and WiredTiger) so will likely be moved to a storage-...


2

I understand correctly, secondary index leaf nodes in MySQL InnoDB engine hold the table primary key value (at least for a unique index). InnoDB tables have a clustered index which determines where data for rows is stored. The clustered index for an InnoDB table is either the PRIMARY KEY (if set), the first unique index with all key columns NOT NULL, or ...


1

WiredTiger does not preallocate data files. Only journal files are pre-allocated (at 100MB per pre-allocation), see journaling process. The disk exhaustion you're seeing is likely due to these journal files. Journal files are optimized for quick writes, and their contents will be persisted in a more permanent manner on a WiredTiger checkpoint which occurs ...


1

Ended up working by using a configuration file when running mongod. Don't know why.


1

If replication doesn't keep up with primary and it's not possible to extend the oplog, I'd try to shard the deployment and... let smaller shards replicate faster or drain the original shard and then remove it from the deployment - this solves you storage engine problem, but leaves the replication as laggy as before


1

What do you think about journalCompressor not snappy? Is this a valid config for DBaaS provider and best for most use cases? In general I would leave MongoDB settings at the default unless variations have proven to be beneficial for your common workloads (i.e. through actual testing and measurement). Snappy compression/decompression typically does not have ...


1

You just kill the process. killall mongod To check that you don't have any mongod process running anymore, you can use ps -ef|grep mongod|grep -v grep or just pgrep mongod. With that latter, if you get pid number (as answer), mongod is still running


1

It looks like the initial slowness is because the EBS volume (which is restored from snapshot) is being initialized during the service startup. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html


1

What version you are using? This is known bug, what has been fixed at versions 3.2.13, 3.4.4, 3.5.6 forward.


1

Since you're using MongoDB 3.4.7 with WiredTiger, there are certain defaults in place if you run MongoDB with the default values (journal is enabled): If newly created, the replica set would use protocol version 1 (see Replica Set Protocol Versions) Default write concern is {w:1} (see Write Concern) With {w:'majority'}, the default is {j:true} if journal is ...


1

The answer to your question: Writing data change to disk. When journaling is on, the first change is written to the journal, then it is written to data blocks.


1

If you perform the recovery by overwriting some collection-*.wt files inside your dbpath, then your database is in an unsupported state. Data loss is quite likely in this case. If you want to restore the database, you would need to restore the dbpath as a whole and not as individual files. Please see MongoDB Backup Methods for instructions and information ...


1

When running a journal disabled cluster, we observed frequent disk checkouts - once about every 3 seconds. This was confusing, as the docs specifically mention that checkpoints are created every 60 seconds. After digging around, we found the following comment on mongo's Jira: ... it is expected that the primary node executes a checkpoint whenever there ...


1

Just to clarify, please be careful about using repairDatabase on a replica set node. repairDatabase is meant to be used to salvage readable data i.e. after a disk corruption, so it can remove unreadable data and let MongoDB start in the face of disk corruption. If this node has an undetected disk corruption and you run repairDatabase on it, this could lead ...


1

Steps for migrating 2.6 to 3.0 on Ubuntu and using the new WiredTiger storage engine: Backup current database: mongodump --out final_2.6_backup Stop mongodb service: service mongodb stop Remove current database files: rm -rf /var/lib/mongodb/* Edit /etc/mongod.conf to use the new storage engine: storageEngine=wiredTiger Start mongodb: service mongodb ...


Only top voted, non community-wiki answers of a minimum length are eligible