{"id":103514,"date":"2024-09-13T20:39:26","date_gmt":"2024-09-13T20:39:26","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=103514"},"modified":"2024-11-14T22:03:29","modified_gmt":"2024-11-14T22:03:29","slug":"working-with-schema-validation-in-mongodb","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/databases\/nosql\/mongodb\/working-with-schema-validation-in-mongodb\/","title":{"rendered":"Working with Schema Validation in MongoDB"},"content":{"rendered":"

This article is part of Robert Sheldon's continuing series on Mongo DB. To see all of the items in the series, click here<\/a>.<\/strong><\/p>\n\n

In the previous article in this series, I introduced you to schema validation in MongoDB. I described how you can define validation rules on a collection and how those rules validate document inserts and updates. In this article, I continue the discussion by explaining how to apply validation rules to documents that already exist in a collection. Before you start in on this article, however, I recommend that you first review the previous article for an introduction into schema validation.<\/p>\n

The examples in this article demonstrate various concepts for working with schema validation, as it applies to existing documents. I show you how to find documents that conform and don\u2019t conform to the validation rules, as well as how to bypass schema validation when inserting or updating a document. I also show you how to update and delete invalid documents in a collection. Finally, I explain how you can use validation options to override the default schema validation behavior when inserting and updating documents.<\/p>\n

Note: For the examples in this article, I used the same MongoDB Atlas and MongoDB Compass environments I used for the previous articles in this series. Refer to the first article for details about setting up these environments. Because this article builds off the previous article, I\u2019ve also used the same hr<\/code> database and candidates<\/code> collection for the examples. If you\u2019ve already deleted the database and collection, you can simply re-create them. The goal is to start with an empty collection with no validation rules defined.<\/p>\n

Searching for invalid documents in a MongoDB collection<\/h2>\n

When you define schema validation rules on a collection that already contains documents, some of those documents might not adhere to the new rules. In which case, you might want to search your collection to determine which documents conform to the rules and which ones do not.<\/p>\n

In this section, I demonstrate how to find valid and invalid documents. If you want to try out the examples, you should create the hr<\/code> database and candidates<\/code> collection, if they don\u2019t already exist.<\/p>\n

The examples in this article were developed for MongoDB Shell. You can use the version of Shell integrated into MongoDB Compass, or you can use the version available through your system\u2019s command-line utility. I like using the version in Compass because I can quickly review my inserted and updated documents.<\/p>\n

If your collection contains any documents, delete those and then delete any validation rules. To remove validation rules, open MongoDB Shell, switch to the hr<\/code> database, and run the following runCommand<\/code> statement:<\/p>\n

db.runCommand( { collMod: \"candidates\", validator: {} } );<\/pre>\n
The runCommand<\/code> method calls the collMod<\/code> database command, which lets you modify options on the specified collection. In this case, the command calls the validator<\/code> method, just like you saw in the previous article when defining validation rules. This time, however, the method\u2019s argument is an empty document (curly brackets), which means that no rules will be defined. If rules already exist, they will be removed.<\/p>\n
When you run this statement, you should receive an OK message that indicates the statement successfully executed. Of course, you could have dropped and re-created the collection, rather than deleting the documents and schema rules. It\u2019s up to you which approach you take, although it can be useful to know how to remove validation rules.<\/p>\n
Once you have a clean slate, you should run the following insertMany<\/code> statement, which adds four simple documents to the candidates<\/code> collection:<\/p>\n
db.candidates.insertMany([\n  { _id: 101,\n    \"name\": \"Drew\", \n    \"dob\": \"1973-9-12\" },\n  { _id: 102,\n    \"name\": \"Parker\", \n    \"dob\": new Date(\"1982-12-2\") },\n  { _id: 103,\n    \"name\": \"Harper\", \n    \"dob\": \"1967-3-25\" },\n  { _id: 104,\n    \"name\": \"Darcy\", \n    \"dob\": new Date(\"1999-5-18\") }\n]);<\/pre>\n
The dob<\/code> values in these documents are intentionally defined with different data types\u2014two as String<\/code> values and two as Date<\/code> values. The difference in types will help to demonstrate how to find valid and invalid documents, as you\u2019ll see shortly. When you run the insertMany<\/code> statement, it should return the following results, which indicate that the documents have been added to the collection:<\/p>\n
{\n  acknowledged: true,\n  insertedIds: {\n    '0': 101,\n    '1': 102,\n    '2': 103,\n    '3': 104\n  }\n}<\/pre>\n
After you add the documents, you can run the following runCommand<\/code> statement to define a simple set of schema validation rules:<\/p>\n
db.runCommand( { collMod: \"candidates\",\n  validator: {\n    $jsonSchema: {\n      bsonType: \"object\",\n      required: [ \"name\", \"dob\" ],\n      properties: {\n        \"name\": { bsonType: \"string\" },\n        \"dob\": { bsonType: \"date\" }\n      }\n    }\n  }\n});<\/pre>\n
The statement calls the validator<\/code> method, which in turn calls the $jsonSchema<\/code> operator. The operator defines a JSON Schema object that contains the validation rules. Notice that the properties<\/code> element specifies that the dob<\/code> field must be the date<\/code> type. The rest of the statement\u2019s components work just like you saw in the previous article. If you have any questions about what I\u2019ve done here, refer to that article.<\/p>\n
Although the documents and validation rules that we\u2019ve added to the collection are relatively simple, they are enough to demonstrate how to find valid and invalid documents. The same principles apply no matter how complex your documents or validation rules.<\/p>\n
When working with a collection\u2019s schema validation, there might be times when you want to view or retrieve the validation rules themselves. For this, you can use the getCollectionInfos<\/code> database method, specifying the validator<\/code> option, as shown in the following statement:<\/p>\n
db.getCollectionInfos( \n  { name: \"candidates\" } )[0].options.validator;<\/pre>\n
The getCollectionInfos<\/code> method lets you view different options about the specified collection, in this case, candidates<\/code>. To retrieve the validation rules, you must specify the validator<\/code> option as it is constructed here. The statement should return the following $jsonSchema<\/code> object:<\/p>\n
{\n  '$jsonSchema': {\n    bsonType: 'object',\n    required: [ 'name', 'dob' ],\n    properties: { name: { bsonType: 'string' }, dob: { bsonType: 'date' } }\n  }\n}<\/pre>\n
Together, the runCommand<\/code> method and validator<\/code> option make it easy to retrieve a collection\u2019s validation rules so you can see how they\u2019ve been defined. You can also use this approach to find valid and nonvalid documents.<\/p>\n
To search for valid documents, you need to include the JSON Schema object as an argument to the find<\/code> collection method. A good way to do this is to assign the object to a variable and then pass the variable in as an argument to the find<\/code> method. For example, the following let<\/code> statement assigns the JSON Schema object returned by the getCollectionInfos<\/code> method to the schema1<\/code> variable:<\/p>\n
let schema1 = \n  db.getCollectionInfos( \n    { name: \"candidates\" } )[0].options.validator;<\/pre>\n
After you define the variable, you can pass it into the find<\/code> statements that you run within the same MongoDB Shell session. Before I demonstrate how to do this, however, I want to first show you how to assign the JSON Schema object definition directly to the variable, without using the getCollectionInfos<\/code> method. In this way, you can check for valid and invalid documents before you define validation rules on a collection.<\/p>\n
To assign the JSON Schema object directly to the variable, use the $jsonSchema<\/code> operator to define the object, using this as the variable value:<\/p>\n
let schema1 =\n{\n  $jsonSchema: {\n    bsonType: \"object\",\n    required: [ \"name\", \"dob\" ],\n    properties: {\n      \"name\": { bsonType: \"string\" },\n      \"dob\": { bsonType: \"date\" }\n    }\n  }\n};<\/pre>\n
Regardless of which approach you use to define your variable, you can easily verify its contents by running the following print<\/code> command:<\/p>\n
print(schema1);<\/pre>\n
The command should return the $jsonSchema<\/code> object, as shown in the following results:<\/p>\n
{\n  '$jsonSchema': {\n    bsonType: 'object',\n    required: [ 'name', 'dob' ],\n    properties: { name: [Object], dob: [Object] }\n  }\n}<\/pre>\n
After you define the schema1<\/code> variable you can use it in your find<\/code> statements to search for valid and invalid documents. For example, the following find<\/code> statement searches the candidates<\/code> collection for documents that conform to the JSON Schema object in the schema1<\/code> variable:<\/p>\n
db.candidates.find( schema1 );<\/pre>\n
The statement returns the following results, which include the documents with the _id<\/code> values of 102<\/code> and 104<\/code>:<\/p>\n
{\n  _id: 102,\n  name: 'Parker',\n  dob: 1982-12-02T08:00:00.000Z\n}\n{\n  _id: 104,\n  name: 'Darcy',\n  dob: 2003-07-02T07:00:00.000Z\n}<\/pre>\n
If you want to search for the documents that do not conform to the validation rules, you can use the $nor<\/code> logical operator, along with the schema1<\/code> variable:<\/p>\n
db.candidates.find( { $nor: [ schema1 ] } );<\/pre>\n
This time, the find<\/code> statement returns only the invalid documents, which include the documents with the _id<\/code> values of 101<\/code> and 103<\/code>:<\/p>\n
{\n  _id: 101,\n  name: 'Drew',\n  dob: '1973-9-12'\n}\n{\n  _id: 103,\n  name: 'Harper',\n  dob: '1967-3-25'\n}<\/pre>\n
As you can see, the value for the dob<\/code> field in the results is a String<\/code> value in both cases, which violates the validation rules. The field must take a Date<\/code> value to conform to the rules. Except for this issue, the documents conform to the validation rules in all other ways.<\/p>\n
Bypassing schema validation in a MongoDB collection<\/h2>\n
At times, you might want to add a document to a collection that violates the validation rules. For example, you might need to restore data from an archive file that includes invalid documents, which you want to preserve them in their original state.<\/p>\n
As you saw in the previous article, MongoDB will return an error when you try to insert a document that violates the validation rules. For instance, MongoDB returns an error if you try to run the following insertOne<\/code> statement:<\/p>\n
db.candidates.insertOne(\n  {\n    _id: 105,\n    \"name\": \"Carey\", \n    \"dob\": \"2003-7-2\"\n  }\n);<\/pre>\n
The statement attempts to insert a document whose dob<\/code> field is a String<\/code> value, which violates the schema we defined above. However, you can override this behavior by including the bypassDocumentValidation<\/code> option in your insertOne<\/code> statement, as in the following example:<\/p>\n
db.candidates.insertOne(\n  {\n    _id: 105,\n    \"name\": \"Carey\", \n    \"dob\": \"2003-7-2\"\n  },\n  { bypassDocumentValidation: true }\n);<\/pre>\n
The bypassDocumentValidation<\/code> option takes a Boolean as its value. When the option is set to true<\/code>, MongoDB ignores the validation rules and inserts the document into the collection. The option works the same way for an insertMany<\/code> statement.<\/p>\n
When you include the bypassDocumentValidation<\/code> option in an insertOne<\/code> or insertMany<\/code> statement, the option applies only when you run the statement. Once you insert the document into the collection, you cannot update it in a way the violates the validation rules. For example, if you try to run the following updateOne<\/code> statement, MongoDB will return an error message stating that the document failed validation:<\/p>\n
db.candidates.updateOne(\n  { _id : 105 },\n  { $set: { \"dob\" : \"2003-7-3\" } }\n);<\/pre>\n
However, the updateOne<\/code> method, as well as the updateMany<\/code> method, also support the bypassDocumentValidation<\/code> option, as in the following example:<\/p>\n
db.candidates.updateOne(\n  { _id : 105 },\n  { $set: { \"dob\" : \"2003-7-3\" } },\n  { bypassDocumentValidation: true }\n);<\/pre>\n
When you run this statement, MongoDB ignores the validation rules, updates the document, and returns a message indicating that one document has been modified.<\/p>\n
As with inserting data, the bypassDocumentValidation<\/code> option applies only to the updateOne<\/code> or updateMany<\/code> statements when you run them. If you later try to update the statement, it must once again conform to the validation rules or include the bypassDocumentValidation<\/code> option.<\/p>\n
Updating invalid documents in a MongoDB collection<\/h2>\n
By default, MongoDB applies validation rules whenever you try to insert or update a document, unless you include the bypassDocumentValidation<\/code> option in your statements. However, MongoDB does not apply those rules to the documents that already exist in the collection at the time you define the validation rules, which means your collection could contain invalid documents.<\/p>\n
In some cases, you might want to update the invalid documents to bring them into conformance with the validation rules or to mark them in some way as being invalid. Before you update the documents, however, you can first search for them so you can assess what you have. For this, you can again use the find<\/code> method, along with the $nor<\/code> operator and JSON Schema object variable.<\/p>\n
You can use the schema1<\/code> variable you defined earlier as long you\u2019re working in the same user session. If you\u2019ve reconnected to MongoDB since running those earlier statements, you\u2019ll need to redefine the variable before you can use it, in which case, you should again run the following let<\/code> statement:<\/p>\n
let schema1 = \n  db.getCollectionInfos( \n    { name: \"candidates\" } )[0].options.validator;<\/pre>\n
Once you\u2019ve defined the variable, you can then run your find<\/code> statement:<\/p>\n
db.candidates.find( { $nor: [ schema1 ] } );<\/pre>\n
The statement returns the following results, which indicate that the candidates<\/code> collection contains three invalid documents:<\/p>\n
{\n  _id: 101,\n  name: 'Drew',\n  dob: '1973-9-12'\n}\n{\n  _id: 103,\n  name: 'Harper',\n  dob: '1967-3-25'\n}\n{\n  _id: 105,\n  name: 'Carey',\n  dob: '2003-7-3'\n}<\/pre>\n
To update the invalid documents, you can use $nor<\/code> operator and schema1<\/code> variable within an updateMany<\/code> statement as part of the statement\u2019s filter. For example, the following updateMany<\/code> statement uses the operator and variable to change the dob<\/code> field in the invalid documents:<\/p>\n
db.candidates.updateMany(\n  { $nor: [ schema1 ] },\n  [ { $set: { \"dob\": { $toDate: \"$dob\" } } } ]\n);<\/pre>\n
The statement first searches for the invalid documents\u2014using the $nor<\/code> operator and the schema1<\/code> variable\u2014and then uses the $toDate<\/code> method to set the field\u2019s data type to Date<\/code>. When you run the statement, you should receive a message indicating that the three documents were modified. If you were to examine the documents, you would find that the dob<\/code> field in each document is now defined with the Date<\/code> data type.<\/p>\n
In this case, the updateMany<\/code> statement modifies all invalid documents because it assumes that the dob<\/code> value needs to be updated in every one of those documents. However, there might be times when you need to refine your statement\u2019s filter to target only a subset of the invalid documents, as in the following example:<\/p>\n
db.candidates.updateMany(\n  { $nor: [ schema1 ], \"dob\": { $type : \"string\" } },\n  [ { $set: { \"dob\": { $toDate: \"$dob\" } } } ]\n);<\/pre>\n
This time, the updateMany<\/code> statement looks for documents that are invalid documents and<\/em> whose dob<\/code> field has a String<\/code> value. If there are documents that violate the validation rules in other ways, they will not be updated.<\/p>\n
Deleting invalid documents in a MongoDB collection<\/h2>\n
You can use the same logic\u2014the $nor<\/code> operator and schema1<\/code> variable\u2014when deleting invalid documents. For example, suppose you insert the following document into the candidates<\/code> collection:<\/p>\n
db.candidates.insertOne(\n  {\n    _id: 106,\n    \"name\": \"Avery\", \n    \"dob\": \"1993-11-24\"\n  },\n  { bypassDocumentValidation: true }\n);<\/pre>\n
Notice that the document includes the bypassDocumentValidation<\/code> option, allowing the document to be added even if it violates the validation rules, which it does.<\/p>\n
After you insert the document, you might decide to delete it, along with any other invalid documents in the collection. For this, you can use a deleteMany<\/code> statement, once again incorporating the $nor<\/code> operator and schema1<\/code> variable, as in the following example:<\/p>\n
db.candidates.deleteMany( { $nor: [ schema1 ] } );<\/pre>\n
When you run the statement, MongoDB should return a message indicating that one document has been deleted (assuming you\u2019ve been following along with the examples). You can verify the deletion by running the following find<\/code> statement:<\/p>\n
db.candidates.find( { $nor: [ schema1 ] } );<\/pre>\n
The statement should now return no results because the candidates<\/code> collection no longer contains invalid documents. At this point, the collection should include only five documents, all of them in conformance with the validation rules. In other words, the dob<\/code> field in each document is now defined with the Date<\/code> data type.<\/p>\n
Setting the schema validation action and level<\/h2>\n
The examples in this article, as well as those in the previous article, demonstrated how validation rules can affect your insert and update operations. As you have seen, you can insert documents only if they conform to those rules, unless you specify the bypassDocumentValidation<\/code> option. Likewise, you can update documents only if the updated documents will conform to the rules.<\/p>\n
However, this is only the default behavior. You can actually control how inserts and updates are handled when defining your validation rules. To implement these controls, you must include one or both of the following options when defining your validation rules:<\/p>\n