0

There will be total 1.5 billion documents in nearly 100 collections in one database. Need I to separate this database into several? Is there a performance limitation for a single database when the quantity of documents going higher?

2
  • In general, database and collection address grouping of data; that is mostly an application functional needs. Having too many databases and collections (thousands) can be an issue as these require additional storage related resources. MongoDB databases, collections and the indexes are stored in individual files, and more files means more resources. Data frequently used by applications is called as "working set" - this needs to be in memory for efficient access. The data access performance depends upon number/size of documents, memory, indexes, kind of query/sort operations, etc.
    – prasad_
    Mar 31 at 2:55
  • @prasad_ Thanks for the answer. So, can I simply summarize that the number of total documents in a database will tinily affect the performance while the number of collections is not too much?
    – zyeewang
    Mar 31 at 5:03

1 Answer 1

0

Not necessary.

It's more like a hardware problem. Especially memory! Not CPU. So, if you can give that DB server quite much memory, like a few hundred of GB, it will be fine!

Of course, the first question is "how big" is the DB? Especially how big partition of that data is "active". And how fast that DB must serve clients.

5
  • The total size of the DB is about 600GB. There will be 500 documents requested each time, and the time consumption is limited to 3s.
    – zyeewang
    Mar 31 at 2:00
  • I'm a little confused about the word "active". If it means that the size of data requested, it nearly 10MB; If it means that the size of collections being writing and reading, the number of collections is 4, totally about 75m documents, the size is about 80GB
    – zyeewang
    Mar 31 at 2:26
  • Because mongo tries to keep as many as possible documents (and indexes) in the memory, active data is that data that is used regularly. So, if you could have 100GB memory, then DB will serve clients in milliseconds. With 32GB memory, response time should still be under second.
    – JJussi
    Apr 1 at 5:26
  • @zyeewang Effective capacity planning includes both resource consideration (available RAM, I/O, CPU relative to the working set of commonly used data & indexes) and data modelling (efficiently modelling your data to support common usage). I recommend reading the series Building with Patterns: A Summary and A Summary of Schema Design Anti-Patterns and How to Spot Them for some typical approaches.
    – Stennie
    Apr 3 at 5:00
  • As JJussi mentions, total size of data is less important than the working set of actively used data & indexes that should ideally fit in RAM. Swapping data to/from disk is orders of magnitude slower than working with data in RAM. Indexes are compressed on-disk & in-memory; documents are compressed on disk and uncompressed in memory. A 600GB data set could translate to 10s of GBs data working set or could be much larger than 600GB if a significant percentage of total data is routinely loaded into RAM. RAM is also needed for filesystem cache and other processing (eg aggregation).
    – Stennie
    Apr 3 at 5:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.