Hot answers tagged

111

A Replica-Set means that you have multiple instances of MongoDB which each mirror all the data of each other. A replica-set consists of one Master (also called "Primary") and one or more Slaves (aka Secondary). Read-operations can be served by any slave, so you can increase read-performance by adding more slaves to the replica-set (provided that your client ...


84

To restore a single database you need to provide the path to the dump directory as part of the mongorestore command line. For example: # Backup the training database mongodump --db training # Restore the training database to a new database called training2 mongorestore --db training2 dump/training The --db option for mongodump specifies the source ...


59

Easiest way to find the storage engine being used currently. Inside mongo console type db.serverStatus().storageEngine It returns the storage engine being used currently { "name" : "wiredTiger" } Once it is confirmed that wiredTiger is being used then type db.serverStatus().wiredTiger to get all the configuration details of wiredTiger.


57

In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well. Theory Shard your data. ...


47

Here are several examples of YAML configs for Linux (Windows paths and options are a little different), essentially explicitly setting some defaults and commonly used settings. First, a standalone mongod with the default port, path, journal settings - this would be the type of configuration used for local testing, with a few extras so show the general style:...


46

Add --forceTableScan Example: mongodump --forceTableScan -d database_name -o target_directory http://aug2uag.blogspot.com/2018/08/mongoexport-unrecognized-field-snapshot.html


41

If I was going to put this into SQL Server, I would suggest a table something like: CREATE TABLE tcp_traffic ( tcp_traffic_id bigint constraint PK_tcp_traffic primary key clustered IDENTITY(1,1) , tcp_flags smallint /* at most 9 bits in TCP, so use SMALLINT */ , src_as int /* Since there are less than 2 billion A.S.'s possible, use INT ...


38

If you have locked yourself out then you need to do the following: Stop your MongoDB instance Remove the --auth and/or --keyfile options from your MongoDB config to disable authentication Start the instance without authentication Edit the users as needed Restart the instance with authentication enabled


33

Based on my own benchmarks, find().limit(1) is orders of magnitude faster than findOne(). There is either an error in the MongoDB documentation or a bug in findOne(). findOne() performs more like find().limit(N) where N is the number of documents the query would return. I figured this out while trying to figure out why my simple queries were so slow! ...


31

Found answer here: https://stackoverflow.com/questions/1296358/subtract-days-from-a-date-in-javascript db.system.profile.find({ "timestamp" : { $lt: new Date(), $gte: new Date(new Date().setDate(new Date().getDate()-1)) } })


26

Personally, I prefer the mmapv1 storage engine as of now for three reasons. Reason 1: Maturity It isn't that WiredTiger is immature. But mmapv1 is well understood and battle tested all way up and down, back and forth and above and beyond. WiredTiger has had some serious issues (see http://jira.mongodb.com for details) fairly recently, and I am not willing ...


25

You have answered some of your own questions here, specifically you have a decent idea about the write lock aspect of the equation - 12,000 insert/sec gets you to ~60% write lock. That's a reasonable level to get consistent performance - you will be getting some contention, and some ops will be a little slower, but you really want to start worrying at about ...


23

Okay, so after following the clues given by loicmathieu and jstell, and digging it up a little, these are the things I found out about MongoDB using WiredTiger storage engine. I'm putting it here if anyone encountered the same questions. The memory usage threads that I mentioned, all belonged to 2012-2014, all pre-date WiredTiger and are describing behavior ...


20

The '...AnyDatabase' roles should be added to the admin database for the user: use admin db.createUser({ user: "mongoadmin" , pwd: "mongoadmin", roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase"]}) Any database: Database User Roles Database Administration Roles Admin database: Cluster Administration Roles Backup and ...


19

With the need of 10TB to be backed up this gets a bit complicated. Replicas are no replacement for proper backups While delayed replica set members can provide an relatively easy way to help you with accidental operations, there are no replacement for proper backups, very much like RAID isn't a replacement for file system based backups. Recommendations ...


19

Yes.. Easily! mongo "mongodb://test:testpwd@localhost/test?authSource=admin" MongoDB shell version v3.4.6 connecting to: mongodb://test:testpwd@localhost/test?authSource=admin MongoDB server version: 3.4.6 MongoDB> db test MongoDB> You put your destination DB at end of "base" string and add authentication db to parameter authSource


18

The real reason why you can't do as you ask (limit the memory) is because MongoDB doesn't manage the memory it uses directly - it lets the OS do it. MongoDB just memory maps all its data and then has the OS page it in and out of memory as needed. As a result, there is no direct management of the amount used possible until MongoDB implements this in a ...


18

Sharding partitions the data-set into discrete parts. Replication duplicates the data-set. These two things can stack since they're different. Using both means you will shard your data-set across multiple groups of replicas. Put another way, you Replicate shards; a data-set with no shards is a single 'shard'. A Mongo cluster with three shards and 3 ...


16

All data dumped via mongodump has to be read into memory by the MongoDB server. It's also worth noting that mongodump backs up data and index definitions; the time to restore can also be significantly longer as compared to other approaches since mongorestore will need to recreate any secondary indexes after the data is loaded. As noted in the MongoDB ...


14

This can be a little tricky, but the fact that the MongoDB shell is basically a Javascript interpreter gives us decent options in terms of filtering. Here is the function I use to accomplish this: // kills long running ops in MongoDB (taking seconds as an arg to define "long") // attempts to be a bit safer than killing all by excluding replication related ...


13

Some thoughts.... Typically one does not want to store pieces of tightly interrelated information in different systems. The chances of things getting out of sync is significant and now instead of one problem on your hands you have two. One thing you can do with Mongo though is use it to pipeline your data in or data out. My preference is to keep ...


13

I had a similar issue when I created a user without adding a superuser first. The following steps helped solve the problem: Open the MongoDB configuration file using sudo (sudo vi mongodb.conf). The file can be found in /etc/ folder. Comment "auth = true". Stop the MongoDB service (sudo service mongod stop) Start the MongoDB service (sudo service mongod ...


13

The easy, albeit a bit unsecure way Stop the first secondary Delete the content of it's dbpath Restart the secondary Wait for it to catch up with the primary Repeat process with the second secondary This is a bit unsecure as it is unknown why the secondaries entered the Recovering state. The more secure, but also more intrusive way As above, but stop ...


13

Use the following command to restore mongo db: mongorestore -d dbname dbpath


12

Short version: use a manual reference, not a DBRef. Explanation: There is no particular benefit to the use of a DBRef beyond giving you the collection and database that the referenced document resides in. If you know the reference is only between categories and products, then that is not particularly useful. A DBRef simply contains the _id of the ...


12

Since there is already and answer submitted, and a useful and valid one at that, I do not want to distract from its own usefulness but there are indeed points to raise that go way beyond just a short comment. So consider this "augmentation", which is hopefully valid but primarily in addition to what has already been said. The truth is to really consider "...


12

First of all, you can use a pipe mongodump -h sourceHost -d yourDatabase … | mongorestore -h targetHost -d yourDatabase … This reduces the time, as each document read will more or less instantly be restored on targetHost. However, this has the disadvantage that you might run into problems if the procedure is aborted for some reason, for example of a ...


12

The answer from Kalhara is correct if you know the position of the embedded document in your array, however there is an alternative approach that can be used to update the first matching array element without knowing the position. Assuming you update with a single array in your query criteria, the positional operator ($) can be used to refer to the matching ...


12

You certainly have a difference of version between mongodump vs your mongoDB server ( 3.6 vs 4.0). You can use docker to the rescue : docker run --rm -v $(pwd):/workdir/ -w /workdir/ mongo:4.0 mongodump -h server -d $database --out /workdir/dump/ With docker you can use a precise version of tools cli without install it. You can even specify an alias (...


11

If you are using sharding, then the "load balancer" is the mongos process - actually it is more like a router - it keeps an in-memory copy of the config database and can make decisions based on the shard key. If you mean load balancing across identical replica sets or across the members of the set, then there is a feature request to have mongos handle ...


Only top voted, non community-wiki answers of a minimum length are eligible