I've been playing with a large dataset, consisting of a few tens of billions of rows of small ascii strings. I want to count the number of occurrences of each string in the dataset, and i suspect the unique number of strings is about 1 billion. I've been upserting them into a collection in a MongoDB 4.0 database on a three
32GB RAM machines in a replica set, with the WiredTiger storage engine. Things go pretty ok, until i hit the magical number of
200.000.000 rows. After that number, the insert speed starts to grind to a halt. I bulk upsert a chunk of 5000 strings at a time, and some operations take 1 second, but once in a while, an operation takes as much as 40 seconds or more. Replication lag also starts to shoot up.
db.stats(), i see that the only index,
_id, takes about
8GB of memory; storage size is about
6GB; wired tiger cache size default is half of ram, so about
15GB. All of these add up to almost
29GB. I'm guessing that the oplog also takes a few GB as well.
Am i running out of RAM? I was expecting to be able to load the whole dataset in a shot. I will probably convert the 3 member replica set to 3 shards, as the data is mostly perisable, and i will reduce the wired tiger cache size parameter, but i am curious if i am right about the RAM and the number of documents.