Kbaran: /* Data models ⌘ */

2021-04-12T22:45:44Z

Data models ⌘

New page

[[Category:MongoDB]]

{{Copyright Notice}}

<div class="noprint">
<slideshow style="nobleprog" headingmark="⌘" incmark="…" scaled="false" font="Trebuchet MS" >
;title: MongoDB for Developers
;author: Kamil Baran
</slideshow>
</div>

{{:Basics of MongoDB}}

== Querying ⌘==
 

=== Select ⌘===
* find, findOne
* parameters: query, fields, limit, skip, sort, batchSize, options

<syntaxhighlight lang="JavaScript">
> db.people.find()
> db.people.find({"Nationality" : "Great Britain"})
> db.people.find({"Nationality" : "Great Britain", "occupation" : "actor"})
> db.people.find({"_id" : ObjectId("545a36577907b2a255b156c4")})
> db.people.findOne({}, {"Name" : 1, "Nationality" : 1})
> db.people.findOne({}, {"Name" : 1, "Nationality" : 1, "_id" : 0})
> db.people.find().limit(3).skip(5).sort({"BirthYear" : -1})
</syntaxhighlight>
 

=== Query Criteria ⌘===
* $lt, $lte, $gt, $gte, $eq, $ne
* $in, $nin, $or (whenever possible use $in instead of $or)
* $not, $mod
* handling null value, $exists
* Perl Compatible Regular Expressions are allowed

<syntaxhighlight lang="JavaScript">
> db.people.find({Nationality : "USA", BirthYear : {$lte : 1950}})
> db.people.find({Nationality : "USA", BirthYear : {$gte : 1940, $lte : 1950}})
> db.people.find({Nationality : {$in : ["Belgium", "Israel"]}})
> db.people.find({$or : [{Nationality : "Belgium"}, {Nationality : "Israel"}]})
> db.people.find({BirthYear : {$mod : [10, 0]}})
> db.people.find({BirthYear : {$not : {$mod : [10, 0]}}})
> db.people.find({Nationality : null})
> db.people.find({Nationality : {$exists : 1, $eq : null}})
> db.people.find({Name : /east/i})
> db.people.find({Name : /^A/})
> db.people.find({Occupation : /^actor$/i}, {Occupation:1, Name:1, _id:0})
> db.people.find({Name : /^cli.*.wood$/i}, {Name:1, _id:0})
> db.people.find({Name : /^clint eastw..d$/i}, {Name:1, _id:0})
</syntaxhighlight>
 

=== Query Criteria for Arrays ⌘===
* behave in the same way as scalars
* $all, $size (can not be combined with other criterias)
* $slice, $-operator, * $elemMatch

<syntaxhighlight lang="JavaScript">
> db.people.find({Occupation : "Actor"}, {Name : 1, Occupation : 1})
> db.people.find({Occupation : ["Actress"]})
> db.people.find({Occupation : {$all : ["Actress", "Producer"]}})
> db.people.find({"Occupation.1" : "Producer"})
> db.people.find({Occupation : {$size : 3}})
> db.people.find({Name: "Sean Connery"}, {Movie : {$slice : -1}})
> db.people.find({"Movie.Title" : "Never Say Never Again"})
> db.people.find({"Movie.Title" : "Never Say Never Again"}, {"Movie.$" : 1})
</syntaxhighlight>
 

=== Query on Embedded Document ⌘===
* query for entire document
* query for selected key/value pairs
* $elemMatch
* watch out for the dots (.) in keys names

<syntaxhighlight lang="JavaScript">
> db.people.findOne({"BirthPlace" : {"Country" : "United Kingdom, Scotland", "City" : "Edinburgh"}})
> db.people.findOne({"BirthPlace" : {"City" : "Edinburgh", "Country" : "United Kingdom, Scotland"}})
> db.people.findOne({"BirthPlace.City" : "Edinburgh", "BirthPlace.Country" : "United Kingdom, Scotland"})
>
> db.people.find({"Movie._id" : ObjectId("545a5f167907b2a255b156c7"), "Movie.Title" : "Never Say Never Again"})
> db.people.find({"Movie" : {"$elemMatch" : {"_id" : ObjectId("545a5f167907b2a255b156c7"), "Title" : "Never Say Never Again"}}})
</syntaxhighlight>
 

=== Using $where ⌘===
* $where allows to run arbitrary JavaScript as part of a query
* powerful, slow, only for trusted users, can not use indexes

<syntaxhighlight lang="JavaScript">
> db.test_where.insert([{"ordered" : 10, "sent" : 0}, {"ordered" : 10, "sent" : 5}, {"ordered" : 10, "sent" : 10}])
> db.test_where.find({"$where" : function() {
... for (var current in this) {
... for (var other in this) {
... if (current != other && this[current] == this[other]) {
... return true;
... }
... }
... }
... return false;
... }});
</syntaxhighlight>
 

=== Cursors ⌘===
* MongoDB returns results from find using a cursor
* the lifetime of the cursor, immortal cursor
* creating cursors in shell: '''var res = db.test.find()'''
* use '''next''' to iterate through results
* use '''hasNext''' to check if there is another document

<syntaxhighlight lang="JavaScript">
> var cursor = db.people.find()
> while (cursor.hasNext()) {
... doc = cursor.next();
... print(doc.Name+" was born in "+doc.BirthYear);
... }
Johnny Depp was born in 1963
Sean Connery was born in 1930
</syntaxhighlight>
 

* there is '''forEach''' as an alternative for '''while'''

<syntaxhighlight lang="JavaScript">
> var cursor = db.people.find()
> cursor.forEach(function(doc) {
... print(doc.Name+" was born in "+doc.BirthYear);
... })
Johnny Depp was born in 1963
Sean Connery was born in 1930
</syntaxhighlight>
 

=== Query Options ⌘===
* .limit(), .skip(), .sort()
* pagination and avoiding large skips
* $maxScan, $showDiskLoc, $min, $max
* '''.snapshot()''' for consistent results in large collections

<syntaxhighlight lang="JavaScript">
> db.people.find().limit(3).skip(5).sort({"BirthYear" : -1})
> db.people.find({}, {"Nationality" : 1, "Name" : 1}).sort({"Nationality" : -1, "Name" : 1})
> db.people.find()._addSpecial("$maxScan", 5)
> db.people.find()._addSpecial("$showDiskLoc", true)
> db.people.find().min({"BirthYear" : 1975}).max({"BirthYear" : 1990})
</syntaxhighlight>
 

== Data Manipulation ⌘==
 

=== Insert ⌘===
* '''insert''' is the basic method to add documents
* _id will be created automatically if not present in document
* batch insert (pass an array of documents, max 48MB at once, faster)
* insert validation
** only basic structure, size (16MB), adds _id
** most drivers do additional tests (size, non-UTF-8 strings, unrecognised data types) before sending data to MongoDB
** use Object.bsonsize(doc) to see the BSON size of doc in bytes

<syntaxhighlight lang="JavaScript">
> db.people.insert({"_id" : 123, "Name" : "Al Pacino"})
> db.people.insert({"Name" : "Sean Connery"})
> db.people.insert([{"Name" : "Meryl Streep"}, {"Name" : "Clint Eastwood"}, {"Name" : "Penélope Cruz"}])
</syntaxhighlight>
 

=== Remove ⌘===
* remove expects a query document as a parameter
* if query document is empty it will delete all documents from the collection but not entire collection
* there is no way to recover removed documents
* drop is a very fast alternative when removing all documents (require to recreate all indexes)

<syntaxhighlight lang="JavaScript">
> db.people.stats()
> db.people.remove({"Nationality" : "Great Britain"})
> db.people.remove({})
> db.people.drop()
</syntaxhighlight>
 

=== Update ⌘===
* update expects two parameters:
** a query document to locate documents to be updated
** a new document or document that describes how found documents will be changed
* conflicting updates - the last update will "win"
* using _id for update criteria will be faster and safer than other fields

<syntaxhighlight lang="JavaScript">
> person = db.people.findOne({"Name" : "Sean Connery"})
> person.Occupation = "Actor"
> db.people.update({"Name" : "Sean Connery"}, person)
> db.people.findOne({"Name" : "Sean Connery"})
</syntaxhighlight>
 

=== Update Modifiers ⌘===
* useful when changing only specified fields of a document
* used for complex update operations (altering, adding, removing keys, manipulating arrays or embedded documents)
* if key does not exist it will be created
* _id field cannot be changed
* '''$inc''', '''$set''', '''$unset''', '''$push''', '''$pull''', '''$each''', '''$slice''', '''$sort''', '''$pop''', '''$ne''', '''$addToSet''', '''$''' (positional array modificator)

<syntaxhighlight lang="JavaScript">
> db.people.findOne({"Name" : "Sean Connery"})
> db.people.update({"Name" : "Sean Connery"}, {"$inc" : {"Age" : 1}})
> db.people.findOne({"Name" : "Sean Connery"})
</syntaxhighlight>
 

=== Upsert ⌘===
* upsert is a special type of update
* advantage: the same code for create and update
* it will create a document if there is no document that matches the update criteria
* new document will combine update data and criteria
* '''$setOnInsert''' modifier
* '''.save()''' shell helper

<syntaxhighlight lang="JavaScript">
> db.websites.update({"url" : "nobleprog.pl"}, {"$inc" : {"pageviews" : 1}})
> db.websites.update({"url" : "nobleprog.pl"}, {"$inc" : {"pageviews" : 1}}, true)
> db.websites.update({"url" : "nobleprog.pl"}, {"$inc" : {"pageviews" : 1}}, {upsert : true})
> db.websites.update({"url" : "nobleprog.pl"}, {"$inc" : {"pageviews" : 1}, "$setOnInsert" : {"instertDate" : new Date()}}, {upsert : true})
> db.websites.find().pretty()
</syntaxhighlight>
 

=== Updating Multiple Documents ⌘===
* by default only first document will be changed
* default behaviour may change in the future
* db.runCommand({getLastError : 1})

<syntaxhighlight lang="JavaScript">
> db.people.insert([{"Name" : "Jack Nicholson", "Nationality" : "USA"}, {"Name" : "Bruce Willis", "Nationality" : "RFN"}, {"Name" : "Morgan Freeman", "Nationality" : "USA"}])
> db.people.update({"Nationality" : "USA"}, {"$set" : {"Nationality" : "United States of America"}})
> db.people.update({"Nationality" : "USA"}, {"$set" : {"Nationality" : "United States of America"}}, false, true)
> db.people.update({"Nationality" : "USA"}, {"$set" : {"Nationality" : "United States of America"}}, {multi : true})
</syntaxhighlight>
 

=== Two Phase Commits ⌘===
* http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/
 

{{:Indexes in MongoDB}}

== Aggregation ⌘==
 

=== Single Purpose Aggregation ⌘===
* count, distinct, group

<syntaxhighlight lang="JavaScript">
> db.people.count()
> db.people.count({"Occupation" : "Producer"})
>
> db.people.distinct("Movie.Title")
> db.runCommand({"distinct" : "people", "key" : "Movie.Title"})
>
> db.people.group({
... "key" : {"Nationality" : 1},
... //"$keyf" : function(curr) { return {"Nationality" : curr.Nationality.toLowerCase()}; },
... "initial" : {"BirthYear" : 0},
... "reduce" : function(curr, result) {
... if (curr.BirthYear > result.BirthYear) {
... result.BirthYear = curr.BirthYear;
... result.Nationality = curr.Nationality;
... }
... },
... "condition" : {"Nationality" : {"$ne" : null}},
... "finalize" : function(result) {
... result[result.Nationality] = result.BirthYear;
... delete result.Nationality;
... delete result.BirthYear;
... }
... })

</syntaxhighlight>
 

=== Aggregation Pipelines ⌘===
* aggregation is a pipeline
* some pipeline operators: $match, $project, $group, $sort, $limit, $skip, $unwind, $lookup, $out
* $group and $sort needs to collect all documents
* results were limited to 16MB, now unlimited
* memory usage limited to 100MB, allowDiskUse

<syntaxhighlight lang="JavaScript">
> db.people.aggregate([{"$project" : {"Nationality" : 1}},
... {"$group" : {"_id" : "$Nationality", "count" : {"$sum" : 1}}},
... {"$sort" : {"count" : -1}},
... {"$limit" : 3}
... ])
>
> db.people.aggregate([{"$project" : {"Country" : "$Nationality"}}])
> db.people.aggregate([{"$project" : {"Movie" : 1, "_id" : 0}}, {"$unwind" : "$Movie"}])
> db.people.aggregate([{"$project" : {"Movie" : 1, "_id" : 0}}, {"$unwind" : "$Movie"}, {$out:"aggregated"}])
> db.people.aggregate([...], {{allowDiskUse: true, explain: true}})
</syntaxhighlight>
 

==== $project Expressions ⌘====
* mathematical: $add, $subtract, $multiply, $divide, $mod
* date: $year, $month, $week, $dayOfMonth, $dayOfWeek, $dayOfYear, $hour, $minute, $second
* string: $substr, $concat, $toLower, $toUpper
* comparison: $cmp, $strcasecmd, $eq, $ne, $gt, $gte, $lt, $lte
* logical: $and, $or, $not, $cond, $ifNull

<syntaxhighlight lang="JavaScript">
> db.people.aggregate([{"$match" : {"BirthDate" : {"$gte" : new Date("1970-01-01")}}}, {"$project" : {"BirthDate" : {"$year" : "$BirthDate"}}}])
> db.people.aggregate([{"$project" : {"Nationality" : {"$ifNull" : ["$Nationality", "unknown"]}}}])
</syntaxhighlight>
 

==== $group Expressions ⌘====
* arithmetical: $sum, $avg,
* extreme: $min, $max, $first, $last
* array: $addToSet, $push

<syntaxhighlight lang="JavaScript">
> db.people.aggregate([{$group:{_id:"$Nationality", peopleCount:{$sum:1}}}])
> db.people.aggregate([{$group:{_id:"$BirthYear", people:{$addToSet:"$Name"}}}])
</syntaxhighlight>
 

==== Views ====
* new in version 3.4
* views are always read-only
* views are based on existing collections or other views

<syntaxhighlight lang="JavaScript">
> db.createView(<view>, <source>, <pipeline>, <collation>)
> db.createView("peopleByBirthYear", "people", [{$group:{_id:"$BirthYear", people:{$addToSet:"$Name"}}}])
</syntaxhighlight>
 

=== Map-Reduce ⌘===
* powerful and flexible tool for aggregation
* MR uses JavaScript as a "query language"
* not very fast
* arguments: mapreduce, map, reduce, finalize, keeptemp, out, query, sort, limit, scope, verbose

<syntaxhighlight lang="JavaScript">
> map = function () {
... for (var key in this) {
... emit(key, {count : 1});
... }}
>
> reduce = function (key, emits) {
... total = 0;
... for (var i in emits) {
... total += emits[i].count;
... }
... return {"count" : total};
... }
>
> db.runCommand({"mapreduce" : "people", "map" : map, "reduce" : reduce, "out" : "mapreduce_output_collection"})
> db.mapreduce_output_collection.find()
</syntaxhighlight>
 

== Data models ⌘==
* normalization vs. denormalization (referencing vs. embedding)
* few things to consider:
** writes vs. reads
** immediate consistency vs. eventual consistency
** one-to-many vs. one-to-few
** document growth

{| class="wikitable sortable"
|-
! Embedding is better for... !! References are better for
|-
| small subdocuments || large subdocuments
|-
| data that does not change regularly || volatile data
|-
| when eventual consistency is acceptable || when immediate consistency is necessary
|-
| documents that grow by a small amount || documents that grow by a large amount
|-
| data that you'll often need to perform a second query to fetch || data that you'll often exclude from results
|-
| fast reads || fast writes
|-
|}

* more resources:
** docs.mongodb.com/manual/core/data-modeling-introduction/
** developer.mongodb.com/article/mongodb-schema-design-best-practices/

MongoDB for Developers - Revision history

Kbaran: /* Data models ⌘ */