1

emphasized textI imported essentially the same dataset into two different MongoDB instances, both running the same version (5.0.24). One is a single node, and the other is a replica set managed by a cloud provider.

When I compare the sizes of both databases, there is a discrepancy due exclusively to the index size. The index from the cloud-managed database is significantly larger. What could be influencing this difference?

On local single instance node, index size is ~21GB:

> db.stats(1024 * 1024);

{
  db: 'xxxx',
  collections: 17,
  views: 0,
  objects: 590502520,
  avgObjSize: 3856.4778189430926,
  dataSize: 2171764.250192642,
  storageSize: 260034.8125,
  indexes: 65,
  indexSize: 20994.91015625,
  totalSize: 281029.72265625,
  scaleFactor: 1048576,
  fsUsedSize: 9600854.74609375,
  fsTotalSize: 10984287.98046875,
  ok: 1
}

On replicaset managed database from cloud, index size is ~44GB:

{
  db: 'xxxx',
  collections: 17,
  views: 0,
  objects: 596353154,
  avgObjSize: 3880.8902943607136,
  dataSize: 2207165.8776950836,
  storageSize: 269442.23046875,
  indexes: 65,
  indexSize: 44406.3515625,
  totalSize: 313848.58203125,
  scaleFactor: 1048576,
  fsUsedSize: 462518.69921875,
  fsTotalSize: 593613.01171875,
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1707255303, i: 758 }),
    signature: {
      hash: Binary.createFromBase64("wvvT6KAesYNFw19TxpvjQUXBx+4=", 0),
      keyId: Long("7324449712621223941")
    }
  },
  operationTime: Timestamp({ t: 1707255303, i: 758 })
}

When I look at index detail cache from a specific index that is way larger on cloud instance (71MB vs 182MB), I can see a lot of difference between the local single instance and cloud instance:

> db.myCollection.stats({indexDetails:true, indexDetailsName: 'xxx'}).indexDetails.xxx.cache;

On local single instance node:

{
  'bytes currently in the cache': 1742,
  'bytes dirty in the cache cumulative': 0,
  'bytes read into cache': 661,
  'bytes written from cache': 0,
  'checkpoint blocked page eviction': 0,
  'checkpoint of history store file blocked non-history store page eviction': 0,
  'data source pages selected for eviction unable to be evicted': 0,
  'eviction gave up due to detecting an out of order on disk value behind the last update on the chain': 0,
  'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update': 0,
  'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update after validating the update chain': 0,
  'eviction gave up due to detecting out of order timestamps on the update chain after the selected on disk update': 0,
  'eviction gave up due to needing to remove a record from the history store but checkpoint is running': 0,
  'eviction walk passes of a file': 0,
  'eviction walk target pages histogram - 0-9': 0,
  'eviction walk target pages histogram - 10-31': 0,
  'eviction walk target pages histogram - 128 and higher': 0,
  'eviction walk target pages histogram - 32-63': 0,
  'eviction walk target pages histogram - 64-128': 0,
  'eviction walk target pages reduced due to history store cache pressure': 0,
  'eviction walks abandoned': 0,
  'eviction walks gave up because they restarted their walk twice': 0,
  'eviction walks gave up because they saw too many pages and found no candidates': 0,
  'eviction walks gave up because they saw too many pages and found too few candidates': 0,
  'eviction walks reached end of tree': 0,
  'eviction walks restarted': 0,
  'eviction walks started from root of tree': 0,
  'eviction walks started from saved location in tree': 0,
  'hazard pointer blocked page eviction': 0,
  'history store table insert calls': 0,
  'history store table insert calls that returned restart': 0,
  'history store table out-of-order resolved updates that lose their durable timestamp': 0,
  'history store table out-of-order updates that were fixed up by reinserting with the fixed timestamp': 0,
  'history store table reads': 0,
  'history store table reads missed': 0,
  'history store table reads requiring squashed modifies': 0,
  'history store table truncation by rollback to stable to remove an unstable update': 0,
  'history store table truncation by rollback to stable to remove an update': 0,
  'history store table truncation to remove an update': 0,
  'history store table truncation to remove range of updates due to key being removed from the data page during reconciliation': 0,
  'history store table truncation to remove range of updates due to out-of-order timestamp update on data page': 0,
  'history store table writes requiring squashed modifies': 0,
  'in-memory page passed criteria to be split': 0,
  'in-memory page splits': 0,
  'internal pages evicted': 0,
  'internal pages split during eviction': 0,
  'leaf pages split during eviction': 0,
  'modified pages evicted': 0,
  'overflow pages read into cache': 0,
  'page split during eviction deepened the tree': 0,
  'page written requiring history store records': 0,
  'pages read into cache': 1,
  'pages read into cache after truncate': 0,
  'pages read into cache after truncate in prepare state': 0,
  'pages requested from the cache': 0,
  'pages seen by eviction walk': 0,
  'pages written from cache': 0,
  'pages written requiring in-memory restoration': 0,
  'the number of times full update inserted to history store': 0,
  'the number of times reverse modify inserted to history store': 0,
  'tracked dirty bytes in the cache': 0,
  'unmodified pages evicted': 0
}

On cloud instance:

{
  'bytes currently in the cache': 16222,
  'bytes dirty in the cache cumulative': 31577447,
  'bytes read into cache': 98960667,
  'bytes written from cache': 13993066,
  'checkpoint blocked page eviction': 0,
  'checkpoint of history store file blocked non-history store page eviction': 0,
  'data source pages selected for eviction unable to be evicted': 14,
  'eviction gave up due to detecting an out of order on disk value behind the last update on the chain': 0,
  'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update': 0,
  'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update after validating the update chain': 0,
  'eviction gave up due to detecting out of order timestamps on the update chain after the selected on disk update': 0,
  'eviction gave up due to needing to remove a record from the history store but checkpoint is running': 0,
  'eviction walk passes of a file': 129817,
  'eviction walk target pages histogram - 0-9': 126113,
  'eviction walk target pages histogram - 10-31': 3704,
  'eviction walk target pages histogram - 128 and higher': 0,
  'eviction walk target pages histogram - 32-63': 0,
  'eviction walk target pages histogram - 64-128': 0,
  'eviction walk target pages reduced due to history store cache pressure': 0,
  'eviction walks abandoned': 222,
  'eviction walks gave up because they restarted their walk twice': 128180,
  'eviction walks gave up because they saw too many pages and found no candidates': 0,
  'eviction walks gave up because they saw too many pages and found too few candidates': 0,
  'eviction walks reached end of tree': 257975,
  'eviction walks restarted': 0,
  'eviction walks started from root of tree': 128797,
  'eviction walks started from saved location in tree': 1020,
  'hazard pointer blocked page eviction': 8,
  'history store table insert calls': 0,
  'history store table insert calls that returned restart': 0,
  'history store table out-of-order resolved updates that lose their durable timestamp': 0,
  'history store table out-of-order updates that were fixed up by reinserting with the fixed timestamp': 0,
  'history store table reads': 0,
  'history store table reads missed': 0,
  'history store table reads requiring squashed modifies': 0,
  'history store table truncation by rollback to stable to remove an unstable update': 0,
  'history store table truncation by rollback to stable to remove an update': 0,
  'history store table truncation to remove an update': 0,
  'history store table truncation to remove range of updates due to key being removed from the data page during reconciliation': 0,
  'history store table truncation to remove range of updates due to out-of-order timestamp update on data page': 0,
  'history store table writes requiring squashed modifies': 0,
  'in-memory page passed criteria to be split': 0,
  'in-memory page splits': 0,
  'internal pages evicted': 3319,
  'internal pages split during eviction': 0,
  'leaf pages split during eviction': 150,
  'modified pages evicted': 1001,
  'overflow pages read into cache': 0,
  'page split during eviction deepened the tree': 0,
  'page written requiring history store records': 0,
  'pages read into cache': 9268,
  'pages read into cache after truncate': 0,
  'pages read into cache after truncate in prepare state': 0,
  'pages requested from the cache': 267909,
  'pages seen by eviction walk': 194127,
  'pages written from cache': 1337,
  'pages written requiring in-memory restoration': 823,
  'the number of times full update inserted to history store': 0,
  'the number of times reverse modify inserted to history store': 0,
  'tracked dirty bytes in the cache': 0,
  'unmodified pages evicted': 9224
}

Can this explain the size difference between those two index ?

8
  • Are all of the indexes defined exactly the same in both cases?
    – J.D.
    Commented Feb 6 at 22:14
  • Yes, initial database setup and index creation are exactly the same. Index are created via my Spring application and mongo java driver. Commented Feb 6 at 22:45
  • Are you sure the database design is the same and data is at least very similar? You have a difference of around 6 Million documents and have a look at the huge difference on fsUsedSize/fsTotalSize Commented Feb 7 at 8:29
  • Did you compare the indexes one by one? Try this: db.getCollectionInfos({ type: 'collection' }).filter(x => !['system.views'].includes(x.name)).map(x => { return { [x.name]: db.getCollection(x.name).stats(1024 * 1002).indexSizes } }) Commented Feb 7 at 8:46
  • 1
    Could it be a different WiredTiger configuration? Commented Feb 7 at 16:31

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.