2

Machine Specs: 12 Cores 64GB Ram

Mongo Setup: Mongo Version 4.0.5, XFS file system (Flash SAN backed), 1 replica set with 2 replicas, 10GB network all local (sub 1ms latency)

Operation Being Performed: Upsert, document size is about 200 bytes in total. Keyed off of the _id field

Indices: _ID, {Location:2dSphere, Heading: 1}, TTL

Collection sizes range from 2k - 50k docs.

We are seeing that 80th% upserts complete in about 1 ms, however we consistently seeing cases where they take 3-8 seconds.

During peak times we're running about 4k upserts a second and these jumps are making things difficult.

From what I can tell reading the output below, its taking this long waiting to acquire a database lock. The question is what can I do to eliminate these long running upserts? If you need more info I can provide, hopefully Ive done enough homework though.

An

{
    "op" : "update",
    "ns" : "RGC.MyRgcs_42_-73",
    "command" : {
        "q" : {
            "_id" : "3431363530353935333A2D3732373639323633353A53"
        },
        "u" : {
            "Rgc" : {
                "Address" : {
                    "StreetAddress" : "Taras Schevchenko Expressway  (CT-9)",
                    "City" : "New Britain",
                    "State" : "CT",
                    "Zip" : "06051",
                    "County" : "Hartford",
                    "Country" : "United States",
                    "SPLC" : null,
                    "CountryPostalCodeFilter" : 0,
                    "AbbreviationFormat" : 0
                },
                "Coords" : {
                    "Lat" : "41.650595",
                    "Lon" : "-72.769264",
                    "SpeedLimitOption" : null,
                    "Region" : 0
                },
                "Region" : 4,
                "Label" : "",
                "PlaceName" : "",
                "TimeZone" : "EST",
                "SpeedLimitInfo" : {
                    "Speed" : "65",
                    "SpeedType" : "2",
                    "SpeedTypeString" : "Premium",
                    "LinkIds" : [],
                    "RoadClass" : "Interstate",
                    "Units" : "MPH"
                },
                "ConfidenceLevel" : "Exact",
                "DistanceFromRoad" : 0.0,
                "Errors" : [],
                "CrossStreet" : null
            },
            "Location" : {
                "type" : "Point",
                "coordinates" : [ 
                    -72.7692635, 
                    41.6505953
                ]
            },
            "_id" : "3431363530353935333A2D3732373639323633353A53",
            "Heading" : 4,
            "ExpireAt" : ISODate("2019-01-19T15:45:37.982Z")
        },
        "multi" : false,
        "upsert" : true
    },
    "keysExamined" : 0,
    "docsExamined" : 0,
    "nMatched" : 0,
    "nModified" : 0,
    "upsert" : true,
    "keysInserted" : 3,
    "numYield" : 0,
    "locks" : {
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(1)
            }
        },
        "Database" : {
            "acquireCount" : {
                "w" : NumberLong(1)
            },
            "acquireWaitCount" : {
                "w" : NumberLong(1)
            },
            "timeAcquiringMicros" : {
                "w" : NumberLong(4178085)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "w" : NumberLong(1)
            }
        }
    },
    "millis" : 4181,
    "planSummary" : "IDHACK",
    "execStats" : {
        "stage" : "UPDATE",
        "nReturned" : 0,
        "executionTimeMillisEstimate" : 0,
        "works" : 2,
        "advanced" : 0,
        "needTime" : 1,
        "needYield" : 0,
        "saveState" : 0,
        "restoreState" : 0,
        "isEOF" : 1,
        "invalidates" : 0,
        "nMatched" : 0,
        "nWouldModify" : 0,
        "nInvalidateSkips" : 0,
        "wouldInsert" : true,
        "fastmodinsert" : false,
        "inputStage" : {
            "stage" : "IDHACK",
            "nReturned" : 0,
            "executionTimeMillisEstimate" : 0,
            "works" : 1,
            "advanced" : 0,
            "needTime" : 0,
            "needYield" : 0,
            "saveState" : 0,
            "restoreState" : 0,
            "isEOF" : 1,
            "invalidates" : 0,
            "keysExamined" : 0,
            "docsExamined" : 0
        }
    },
    "ts" : ISODate("2019-01-19T15:45:42.164Z"),
    "client" : "10.154.18.136",
    "allUsers" : [],
    "user" : ""
}
  • Mongo Version 4.0.5 - it said v4 in the question, I've updated it to be more specific. – Wjdavis5 Jan 21 at 17:26
  • Hi Wjdavis5, Is it possible for your requirement to upsert in batches, if so try it, as v4.0.5 (considering you are using WiredTiger) has document level locking, if a query is trying to update same document or even updating concerned index it will lock the document and hence will take more time, smaller batches might work faster. – Priyanka Kariya Feb 8 at 9:07
  • We do batch inserts as best we can given the setup. We run a batch upsert every 500ms second or once the batch queue reaches N entries, which ever comes sooner. During peak times we're processing 4k msgs / second so we typically see modest size batches. – Wjdavis5 Feb 12 at 13:45

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.