0

I just had an interesting experience with the mongoexport tool and came out with a doubt about the exporting process' disk usage.

I was trying to dump a 50 MB collection to a disk that had 3 GB free disk space. When the dump reached 101 records, it halted and eventually the connection timed out. You can read more about it here.

User @priyanka-kariya suggested I checked disk space, then I freed a total of 13 GB and the dump ran ok again.

So I wondered, how is disk space used in order to complete a dump process?

*Both mongoexport and mongod are version 4.0.2.

EDIT

@Stennie suggested I checked uncompressed data size, so db.requests_PRD.stats() returns:

{
    "ns" : "bubeIO.requests_PRD",
    "size" : 52721003,
    "count" : 60196,
    "avgObjSize" : 875,
    "storageSize" : 15486976,
    "capped" : false,
    [...]
}

The size is actually 50.3 MiB, which is less than I initially thought.

|improve this question
  • What is MongoDB version(x,y,z)? – Md Haidar Ali Khan Feb 9 at 15:05
  • Is 60MB the size of data in the remote deployment or the size of storage used (which includes compression)? Generally you should need enough local free space to store the uncompressed data being exported, which may be significantly larger in JSON. Were you monitoring disk space while doing the export? What O/S are you using? It is possible that another process used up available space (eg for dynamic swap/paging file), however your description in the original question appears to be an error reading data from the remote deployment rather than an error writing to local storage. – Stennie Feb 10 at 20:55
  • Hi @Stennie, I checked and the uncompressed data is acutally 52 MiB large. I did not monitor disk space while exporting, then I can't tell if there was actually enough free disk space to run this and the other hundreds of processes that could be using hd space for whatever reason. You mentioned that 101 documents or 1 MB is the size of the first batch to export. Where can I read more about this in order to understand more about the inner workings of mongoexport? – Victor Valente Feb 11 at 13:39
  • mongoexport is just doing a normal find query, so the behaviour I mention above is the standard expectation for cursor batches. If you want to peek at the actual logic, the mongoexport source (in Go) is available on GitHub. – Stennie Feb 12 at 9:34
  • As per my earlier comment, I suspect your issue is related to a network problem given the small size of data to export relative to free disk space. Freeing more disk space may have been coincidental to resolving the problem, or there could be another process transiently competing for disk space (for example, dynamic swap). You haven't included any details on your O/S yet, but in general having free disk space in the low single digit GBs might be cutting things too fine for modern O/S (especially if you run low on RAM and the O/S needs to grow swap or virtual memory). – Stennie Feb 12 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.