2

Machine Specs: 12 Cores 64GB Ram

Mongo Setup: Mongo Version 4.0.5, XFS file system (Flash SAN backed), 1 replica set with 2 replicas, 10GB network all local (sub 1ms latency)

Operation Being Performed: Upsert, document size is about 200 bytes in total. Keyed off of the _id field

Indices: _ID, {Location:2dSphere, Heading: 1}, TTL

Collection sizes range from 2k - 50k docs.

We are seeing that 80th% upserts complete in about 1 ms, however we consistently seeing cases where they take 3-8 seconds.

During peak times we're running about 4k upserts a second and these jumps are making things difficult.

From what I can tell reading the output below, its taking this long waiting to acquire a database lock. The question is what can I do to eliminate these long running upserts? If you need more info I can provide, hopefully Ive done enough homework though.

An

{
    "op" : "update",
    "ns" : "RGC.MyRgcs_42_-73",
    "command" : {
        "q" : {
            "_id" : "3431363530353935333A2D3732373639323633353A53"
        },
        "u" : {
            "Rgc" : {
                "Address" : {
                    "StreetAddress" : "Taras Schevchenko Expressway  (CT-9)",
                    "City" : "New Britain",
                    "State" : "CT",
                    "Zip" : "06051",
                    "County" : "Hartford",
                    "Country" : "United States",
                    "SPLC" : null,
                    "CountryPostalCodeFilter" : 0,
                    "AbbreviationFormat" : 0
                },
                "Coords" : {
                    "Lat" : "41.650595",
                    "Lon" : "-72.769264",
                    "SpeedLimitOption" : null,
                    "Region" : 0
                },
                "Region" : 4,
                "Label" : "",
                "PlaceName" : "",
                "TimeZone" : "EST",
                "SpeedLimitInfo" : {
                    "Speed" : "65",
                    "SpeedType" : "2",
                    "SpeedTypeString" : "Premium",
                    "LinkIds" : [],
                    "RoadClass" : "Interstate",
                    "Units" : "MPH"
                },
                "ConfidenceLevel" : "Exact",
                "DistanceFromRoad" : 0.0,
                "Errors" : [],
                "CrossStreet" : null
            },
            "Location" : {
                "type" : "Point",
                "coordinates" : [ 
                    -72.7692635, 
                    41.6505953
                ]
            },
            "_id" : "3431363530353935333A2D3732373639323633353A53",
            "Heading" : 4,
            "ExpireAt" : ISODate("2019-01-19T15:45:37.982Z")
        },
        "multi" : false,
        "upsert" : true
    },
    "keysExamined" : 0,
    "docsExamined" : 0,
    "nMatched" : 0,
    "nModified" : 0,
    "upsert" : true,
    "keysInserted" : 3,
    "numYield" : 0,
    "locks" : {
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(1)
            }
        },
        "Database" : {
            "acquireCount" : {
                "w" : NumberLong(1)
            },
            "acquireWaitCount" : {
                "w" : NumberLong(1)
            },
            "timeAcquiringMicros" : {
                "w" : NumberLong(4178085)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "w" : NumberLong(1)
            }
        }
    },
    "millis" : 4181,
    "planSummary" : "IDHACK",
    "execStats" : {
        "stage" : "UPDATE",
        "nReturned" : 0,
        "executionTimeMillisEstimate" : 0,
        "works" : 2,
        "advanced" : 0,
        "needTime" : 1,
        "needYield" : 0,
        "saveState" : 0,
        "restoreState" : 0,
        "isEOF" : 1,
        "invalidates" : 0,
        "nMatched" : 0,
        "nWouldModify" : 0,
        "nInvalidateSkips" : 0,
        "wouldInsert" : true,
        "fastmodinsert" : false,
        "inputStage" : {
            "stage" : "IDHACK",
            "nReturned" : 0,
            "executionTimeMillisEstimate" : 0,
            "works" : 1,
            "advanced" : 0,
            "needTime" : 0,
            "needYield" : 0,
            "saveState" : 0,
            "restoreState" : 0,
            "isEOF" : 1,
            "invalidates" : 0,
            "keysExamined" : 0,
            "docsExamined" : 0
        }
    },
    "ts" : ISODate("2019-01-19T15:45:42.164Z"),
    "client" : "10.154.18.136",
    "allUsers" : [],
    "user" : ""
}
  • Mongo Version 4.0.5 - it said v4 in the question, I've updated it to be more specific. – Wjdavis5 Jan 21 at 17:26
  • Hi Wjdavis5, Is it possible for your requirement to upsert in batches, if so try it, as v4.0.5 (considering you are using WiredTiger) has document level locking, if a query is trying to update same document or even updating concerned index it will lock the document and hence will take more time, smaller batches might work faster. – Priyanka Kariya Feb 8 at 9:07
  • We do batch inserts as best we can given the setup. We run a batch upsert every 500ms second or once the batch queue reaches N entries, which ever comes sooner. During peak times we're processing 4k msgs / second so we typically see modest size batches. – Wjdavis5 Feb 12 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.