How to find all invalid document based on jsonSchema validator?

Question

One of the new feature in MongoDB 3.6 is jsonSchema validator. I can create a collection with an validator like:

{
  "validator": {
    "$jsonSchema": {
      "properties": {
        "name": {
          "bsonType": "string",
          "description": "name"
        },
        "age": {
          "bsonType": "int",
          "minimum": 10,
          "maximum": 30
        }
      },
      "required": ["name", "age"],
      "bsonType": "object"
    }
  }
}

But what if the collection already existed with some documents, that don't fullfill the validation schema? How can I found out all invalid documents based on the validator?

Answer 1

You can use the $jsonSchema query operator to find relevant documents.

For example, using the mongo shell:

// Retrieve the current validator for the "students" collection
var schema = db.getCollectionInfos({ name: "students" })[0].options.validator

// Find any documents not matching the schema
db.students.find({ $nor: [schema] })

Similarly, you can use the $jsonSchema query operator to confirm expected matches before adding or updating the validator for a collection:

var schema = {
  $jsonSchema: {
    properties: {
      name: {
        bsonType: "string",
        description: "name",
      },
    },
    required: ["name"],
    bsonType: "object",
  },
}

// Find documents matching schema
db.students.find(schema)

Answer 2

As MongoDB documented here To specify JSON Schema validation, use the $jsonSchema operator in your validator expression.

For example, the following example specifies validation rules using JSON schema:

db.createCollection("students", {
   validator: {
      $jsonSchema: {   --- jsonSchema validation
         bsonType: "object",
         required: [ "name", "year", "major", "gpa" ],
         properties: {
            name: {
               bsonType: "string",
               description: "must be a string and is required"
            },
            gender: {
               bsonType: "string",
               description: "must be a string and is not required"
            },
            year: {
               bsonType: "int",
               minimum: 2017,
               maximum: 3017,
               exclusiveMaximum: false,
               description: "must be an integer in [ 2017, 3017 ] and is required"
            },
            major: {
               enum: [ "Math", "English", "Computer Science", "History", null ],
               description: "can only be one of the enum values and is required"
            },
            gpa: {
               bsonType: [ "double" ],
               minimum: 0,
               description: "must be a double and is required"
            }
         }
      }
   }
})

Or As I am able to see in your code , the jsonSchema validation.

"validator" : {
                "$jsonSchema" : {  ---jsonSchema
                    "properties" : {
                        "name" : {
                            "bsonType" : "string",
                            "description" : "name"
                        },
                        "age" : {
                            "bsonType" : "int",
                            "minimum" : 10,
                            "maximum" : 30
                        }
                    },
                    "required" : [
                        "name",
                        "age"
                    ],
                    "bsonType" : "object"
                }
            }

In addition to JSON Schema validation, MongoDB supports validation with query filter expressions using the query operators, with the exception of $near, $nearSphere, $text, and $where.

but what if the collection already existed with some documents don't fullfill the validation creterial?

Validation occurs during updates and inserts. When you add validation to a collection, existing documents do not undergo validation checks until modification.

For Existing Documents

The validationLevel option determines which operations MongoDB applies the validation rules:

1. If the validationLevel is strict (the default), MongoDB applies validation rules to all inserts and updates.
2. If the validationLevel is moderate, MongoDB applies validation rules to inserts and to updates to existing documents that already fulfill the validation criteria. With the moderate level, updates to existing documents that do not fulfill the validation criteria are not checked for validity.

For example, create a contacts collection with the following documents:

db.contacts.insert([
   { "_id": 1, "name": "Anne", "phone": "+1 555 123 456", "city": "London", "status": "Complete" },
   { "_id": 2, "name": "Ivan", "city": "Vancouver" }
])

Issue the following command to add a validator to the contacts collection:

db.runCommand( {
   collMod: "contacts",
   validator: { $jsonSchema: { --- jsonSchema validation
      bsonType: "object",
      required: [ "phone", "name" ],
      properties: {
         phone: {
            bsonType: "string",
            description: "must be a string and is required"
         },
         name: {
            bsonType: "string",
            description: "must be a string and is required"
         }
      }
   } },
   validationLevel: "moderate"   --- validationLevel moderate , With the moderate level, updates to existing documents that do not fulfill the validation criteria are not checked for validity
} )

The contacts collection now has a validator with the moderate validationLevel:

If you attempted to update the document with _id of 1, MongoDB would apply the validation rules since the existing document matches the criteria. In contrast, MongoDB will not apply validation rules to updates to the document with _id of 2 as it does not meet the validation rules.

To disable validation entirely, you can set validationLevel to off.

How can I found out all invalid documents based on the validator?

For Accept or Reject Invalid Documents

The validationAction option determines how MongoDB handles documents that violate the validation rules:

1. If the validationAction is error (the default), MongoDB rejects any insert or update that violates the validation criteria.
2. If the validationAction is warn, MongoDB logs any violations but allows the insertion or update to proceed.

For example, create a contacts2 collection with the following JSON Schema validator:

db.createCollection( "contacts2", {
   validator: { $jsonSchema: {  ---jsonSchema validation
      bsonType: "object",
      required: [ "phone" ],
      properties: {
         phone: {
            bsonType: "string",
            description: "must be a string and is required"
         },
         email: {
            bsonType : "string",
            pattern : "@mongodb\.com$",
            description: "must be a string and match the regular expression pattern"
         },
         status: {
            enum: [ "Unknown", "Incomplete" ],
            description: "can only be one of the enum values"
         }
      }
   } },
   validationAction: "warn"  --- validationAction is warn, MongoDB logs any violations but allows the insertion or update to proceed.
} )

With the warn validationAction, MongoDB logs any violations but allows the insertion or update to proceed.

For example, the following insert operation violates the validation rule:

db.contacts2.insert( { name: "Amanda", status: "Updated" } )

However, since the validationAction is warn only, MongoDB only logs the validation violation message and allows the operation to proceed:

2017-12-01T12:31:23.738-0500 W STORAGE  [conn1] Document would fail validation collection: example.contacts2 doc: { _id: ObjectId('5a2191ebacbbfc2bdc4dcffc'), name: "Amanda", status: "Updated" }

Important Note : You cannot specify a validator for collections in the **admin**, local, and config databases.

You cannot specify a validator for system.* collections.

For your further reference with jsonSchema validation here, here and here