I want to write python3 code to check and insert missing data. My MongoDB collection documents have one field named "height" which is BTC block number. I want to traverse in a range start to the latest block number and check which number is missing from range. The number which is missing I want to insert that. Can somebody help me with the logic? I have MongoDB version 4.


Assuming your numbers are sequential (which appears to be the case for BTC height values), you can use the $range aggregation expression to generate the expected sequence of values for comparison with the existing values.

Here's an example aggregation function using the mongo shell:

function findMissingValues(start, end) {
   var results = db.blocks.aggregate([

      // This stage would benefit from a relevant index like `height: 1`
      { $match: {
         height: {$gte: start, $lte: end}

      // Find existing height values in the given range
      { $group: {
          _id: null,
          heights: {$push: "$height"}

      // Add the expected range (inclusive of start and end values)
      { $addFields: {
          allHeights: { $range: [ start, end + 1]}

      // Find any expected values that are missing from the existing data
      { $project: {
          _id: 0,
          missing: { $setDifference: [ "$allHeights", "$heights" ] }



Sample usage:

> db.blocks.insert([ { height: 550000 }, { height: 550002 } ])

> findMissingValues(550000,550005)
{ "missing" : [ 550001, 550003, 550004, 550005 ] }

With the missing blocks identified, you can now retrieve and insert the relevant BTC data.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.