High page fault, slow query/ insert during background index creation in MongoDB

Question

I was trying to create two compound indexes in background in mongodb. As per mongo doc and other sources, background index creation should not affect read/write.
But severe performance issue was observed. Page fault jumped to 180/s from 4/s, queries that takes 300ms to complete, started taking more than 2 seconds.
Db size : 40GB
Documents : 40 million
Previous index size : 24GB
Compound index1 : {field1 : 1, field2 : -1}
Compound index2 : {field3 : 1, field4 : -1}

DB stats : { "db" : "response", "collections" : 4, "objects" : 42253780, "avgObjSize" : 1008.5294166817738, "dataSize" : 42614180096, "storageSize" : 43648786160, "numExtents" : 44, "indexes" : 14, "indexSize" : 22113921536, "fileSize" : 79352037376, "nsSizeMB" : 16, "extentFreeList" : { "num" : 53, "totalSize" : 5861306368 }, "dataFileVersion" : { "major" : 4, "minor" : 22 }, "ok" : 1 }

Free -m :
total used free shared buffers cached Mem: 64314 63944 370 1 867 56077 -/+ buffers/cache: 7000 57314 Swap: 3999 434 3565

Answer 1

Index builds can impact the performance of your database. While background index builds will not block read or writes, it is not correct to say that "index creation should not affect [the performance of] read/write".

Building an index will require paging the entire collection through memory; this is cause of the page faults being seen. You will likely want to only build a single index at a time to reduce the impact of this process.