0%

MongoDB for Node.js Developers

Posted on Edited on In Disqus:

MongDB version 3.6

all the materials

Week 1 Introduction

What is MongoDB?

MongoDB as many things. Most of features and functionality of MongoDB rest on the fact that it’s a document database. When discussing MongoDB data models, queries to MongoDB, and data stored in MongoDB, we use JSON or Javascript Object Notation. This JSON document provides an example that illustrates the type of data MongoDB stores as a single record. Important advantages of this design are that developer team can design data models to support common data access pattern.

For example, a team building a news website can design a model so that the most viewd pages may require only a single query to the database and can do this an elegant way that is fully supported by MongoDB. In this example, we will combine author, tags and coments to enable rendering a news article with a single database query.

In contrast to relational database design, which will requrie sevreal joins across tables, or an ugly means of storing multiple values in a single table fields.

Since MongoDB is not predicated on joining data from multiple tables, it is much easier to distribute, and shard, data across multiple servers. This means, in MongoDB, there a wide variety of options when considering database deployment from many, inexpensive commodity machine to a few, larger or powerful servers. MongoDB natively supports scaling out through sharding features, so in a way that is abstracted away from application logic. So developers can build their application in a way that is agnositc about the deployment model used. Whether a single node, or a few nodes, or hundreds of nodes are used makes no difference from the perspective of the application. In contrast, joins and multiple table transactions are difficult to do in parallel. Your best option is usually scaling up, acquiring increasingly expensive hardware so that your data can be served from a single server. While not ture, joins, multiple data tables are not missed in MongDB database because the schema design capabilities support models that require atomic reads and writes only to individual documents.

In summary, MongDB enables developer to design data models that make senses for their applications. That is, those that efficiently support common data access patterns.

As we’ll see as we move through this course, mongodb is designed to support agile software engineering practices and meet the scalability and performance needs of modern applications.

Overview of Building an App with MongoDB

Installing MongoDB on Mac

  • download the mongoDB from official web site

  • unzip and set the environment variables in the .bash_profile

    1
    2
    # MongoDB
    export PATH="/Users/allen/Documents/DevEnvir/mongodb-osx-x86_64-enterprise-3.6.2/bin/:$PATH"

  • configure a place to store data

    1
    2
    mkdir -p <the path to the place you store data, e.g. /data/db>
    chmod 777 <the path to the place you store data, e.g. /data/db>

  • startup the database with the dbpath 

    1
    ➜  ~ mongod --dbpath <the path to the place you store data, e.g. /data/db>
    • Tips
      You can put it into a script.

  • startup mongo shell to connect the mongodb server

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    ➜  ~ mongo
    MongoDB shell version v3.6.2
    connecting to: mongodb://127.0.0.1:27017
    MongoDB server version: 3.6.2
    Server has startup warnings:
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten]
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten]
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server.
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
    2018-01-22T16:35:49.495+1100 I CONTROL  [initandlisten]

    or you haven’t startup database server, so use –nodb

    1
    2
    ➜  ~ mongo --nodb
    MongoDB shell version v3.6.2

JSON

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "headline" : "Apple Reported Fourth Quarter Revenue Today",
  "date" : "2015-10-27T22:35:21.908Z",
  "views" : 1132,
  "author" : {
    "name" : "Bob Walker",
    "title" : "Lead Business Editor"
  },
  "published" : true,
  "tags" : [ 
    "AAPL", 
    { "name" : "city", "value" : "Cupertino" },
    [ "Electronics", "Computers" ]
  ]
}

types: string, data, number, boolean, array, object

feature: nesting objects or other types

BSON

hello_world.bson

1
2
3
4
5
6
// JSON
   { "hello" : "world" }

// BSON
"\x16\x00\x00\x00\x02hello\x00 
\x06\x00\x00\x00world\x00\x00"

Intro to Creating and Reading Documents

CREATE READ UPDATE DELETE

We can use mongo shell to do these things. Use help command to learn.

Usual Commands

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
MongoDB Enterprise > help
	db.help()                    help on db methods
	db.mycoll.help()             help on collection methods
	sh.help()                    sharding helpers
	rs.help()                    replica set helpers
	help admin                   administrative help
	help connect                 connecting to a db help
	help keys                    key shortcuts
	help misc                    misc things to know
	help mr                      mapreduce

	show dbs                     show database names
	show collections             show collections in current database
	show users                   show users in current database
	show profile                 show most recent system.profile entries with time >= 1ms
	show logs                    show the accessible logger names
	show log [name]              prints out the last segment of log in memory, 'global' is default
	use <db_name>                set current database
	db.foo.find()                list objects in collection foo
	db.foo.find( { a : 1 } )     list objects in foo where a == 1
	it                           result of the last line evaluated; use to further iterate
	DBQuery.shellBatchSize = x   set default number of items to display on shell
	exit                         quit the mongo shell
1
2
3
4
5
6
7
8
9
MongoDB Enterprise > show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
m101    0.000GB
m102    0.000GB
pcat    0.000GB
test    0.000GB
video   0.000GB
1
2
3
MongoDB Enterprise > use video
switched to db video
MongoDB Enterprise >
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
MongoDB Enterprise > db.movies.insertOne({"title": "Jaws","year": 1975, "imdb": "tt0073195"})
{
	"acknowledged" : true,
	"insertedId" : ObjectId("5a657dc2ab42be5abd87963a")
}
MongoDB Enterprise > db.movies.insertOne({"title": "Mad Max 2: The Road Warrior","year": 1981, "imdb": "tt0082694"})
{
	"acknowledged" : true,
	"insertedId" : ObjectId("5a657e17ab42be5abd87963b")
}
MongoDB Enterprise > db.movies.insertOne({"title": "Raiders of the Lost Ark","year": 1981, "imdb": "tt0082971"})
{
	"acknowledged" : true,
	"insertedId" : ObjectId("5a657e4dab42be5abd87963c")
}
1
2
3
4
MongoDB Enterprise > db.movies.find()
{ "_id" : ObjectId("5a657dc2ab42be5abd87963a"), "title" : "Jaws", "year" : 1975, "imdb" : "tt0073195" }
{ "_id" : ObjectId("5a657e17ab42be5abd87963b"), "title" : "Mad Max 2: The Road Warrior", "year" : 1981, "imdb" : "tt0082694" }
{ "_id" : ObjectId("5a657e4dab42be5abd87963c"), "title" : "Raiders of the Lost Ark", "year" : 1981, "imdb" : "tt0082971" }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
MongoDB Enterprise > db.movies.find().pretty()
{
	"_id" : ObjectId("5a657dc2ab42be5abd87963a"),
	"title" : "Jaws",
	"year" : 1975,
	"imdb" : "tt0073195"
}
{
	"_id" : ObjectId("5a657e17ab42be5abd87963b"),
	"title" : "Mad Max 2: The Road Warrior",
	"year" : 1981,
	"imdb" : "tt0082694"
}
{
	"_id" : ObjectId("5a657e4dab42be5abd87963c"),
	"title" : "Raiders of the Lost Ark",
	"year" : 1981,
	"imdb" : "tt0082971"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// not specify, return all
MongoDB Enterprise > db.movies.find({}).pretty()
{
	"_id" : ObjectId("5a657dc2ab42be5abd87963a"),
	"title" : "Jaws",
	"year" : 1975,
	"imdb" : "tt0073195"
}
{
	"_id" : ObjectId("5a657e17ab42be5abd87963b"),
	"title" : "Mad Max 2: The Road Warrior",
	"year" : 1981,
	"imdb" : "tt0082694"
}
{
	"_id" : ObjectId("5a657e4dab42be5abd87963c"),
	"title" : "Raiders of the Lost Ark",
	"year" : 1981,
	"imdb" : "tt0082971"
}
1
2
3
4
5
6
7
8
MongoDB Enterprise > db.movies.find({"title": "Jaws"}).pretty()
{
	"_id" : ObjectId("5a657dc2ab42be5abd87963a"),
	"title" : "Jaws",
	"year" : 1975,
	"imdb" : "tt0073195"
}
MongoDB Enterprise >
1
2
3
4
5
6
7
8
9
10
11
12
13
MongoDB Enterprise > db.movies.find({"year": 1981}).pretty()
{
	"_id" : ObjectId("5a657e17ab42be5abd87963b"),
	"title" : "Mad Max 2: The Road Warrior",
	"year" : 1981,
	"imdb" : "tt0082694"
}
{
	"_id" : ObjectId("5a657e4dab42be5abd87963c"),
	"title" : "Raiders of the Lost Ark",
	"year" : 1981,
	"imdb" : "tt0082971"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
MongoDB Enterprise > var c = db.movies.find()
MongoDB Enterprise > c.hasNext()
true
MongoDB Enterprise > c.next()
{
	"_id" : ObjectId("5a657dc2ab42be5abd87963a"),
	"title" : "Jaws",
	"year" : 1975,
	"imdb" : "tt0073195"
}
MongoDB Enterprise > c.next()
{
	"_id" : ObjectId("5a657e17ab42be5abd87963b"),
	"title" : "Mad Max 2: The Road Warrior",
	"year" : 1981,
	"imdb" : "tt0082694"
}
MongoDB Enterprise > c.next()
{
	"_id" : ObjectId("5a657e4dab42be5abd87963c"),
	"title" : "Raiders of the Lost Ark",
	"year" : 1981,
	"imdb" : "tt0082971"
}
MongoDB Enterprise > c.hasNext()
false
MongoDB Enterprise >

Installing Node.js

download from here and install

Hello World on Node.js

  • Hello World on Node.js

    app.js

    1
    console.log("Hello, World");
    1
    2
    3
    ➜  hello_world_nodejs node app.js 
    Hello, World
    ➜  hello_world_nodejs

  • HTTP on Node.js

    app.js

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    // Source: howtonode.org/hello-node

    // Load the http module to create an http server.
    var http = require('http');

    // Configure our HTTP server to respond with Hello World to all requests.
    var server = http.createServer(function (request, response) {
    response.writeHead(200, {"Content-Type": "text/plain"});
    response.end("Hello World\n");
    });

    // Listen on port 8000, IP defaults to 127.0.0.1
    server.listen(8000);

    // Put a friendly message on the terminal
    console.log("Server running at http://127.0.0.1:8000/");
    1
    2
    ➜  hello_world_http node app.js
    Server running at http://127.0.0.1:8000/

Introduction to npm

node’s pacakge manager, install npm from here

create a package.json by npm init, then
npm install consolidate –save
npm install express –save
npm install mongodb –save

finally, we got package.json as follow,

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "name": "intro_to_npm",
  "version": "0.0.0",
  "description": "npm introduction",
  "main": "app.js",
  "dependencies": {
        "consolidate": "~0.13.1",
        "express": "~4.13.3",
        "mongodb": "~2.1.3"
  },
  "author": "Shaun Verch",
  "license": "BSD"
}

app.js

1
2
3
var express = require('express'),
    cons = require('consolidate'),
    mongodb = require('mongodb');

usual commands:

1
2
3
4
5
6
7
8
# use package.json in the current directory to install pacakges
npm install

# install the package globally
npm install <package-name> -g

# install the package for this project
npm install <package-name> --save

Intro to the Node.js Driver

What does it mean in driver?

Driver is a library which handles a lot of things such as opening a socket.

app.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');

var url = 'mongodb://localhost:27017/video';

MongoClient.connect(url, function(err, db) {

    assert.equal(null, err);
    console.log("Successfully connected to server");

    // Find some documents in our collection
    db.collection('movies').find({}).toArray(function(err, docs) {

        // Print the documents returned
        docs.forEach(function(doc) {
            console.log(doc.title);
        });

        // Close the DB
        db.close();
    });

    // Declare success
    console.log("Called find()");
});
1
npm install mongodb --save

Hello World using Express

app.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
var express = require('express'),
    app = express();

app.get('/', function(req, res){
    res.send('Hello World');
});

app.use(function(req, res){
    res.sendStatus(404); 
});

var server = app.listen(3000, function() {
    var port = server.address().port;
    console.log('Express server listening on port %s', port);
});
1
npm install express --save

Hello World using Templates

app.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
var express = require('express'),
    app = express(),
    engines = require('consolidate');

app.engine('html', engines.nunjucks);
app.set('view engine', 'html');
app.set('views', __dirname + '/views');

app.get('/', function(req, res) {
    res.render('hello', { name : 'Templates' });
});

app.use(function(req, res){
    res.sendStatus(404); 
});

var server = app.listen(3000, function() {
    var port = server.address().port;
    console.log('Express server listening on port %s', port);
});

views/hello.html

1
<h1>Hello, {{name}}!</h1>

All Together Now

app.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
var express = require('express'),
    app = express(),
    engines = require('consolidate'),
    MongoClient = require('mongodb').MongoClient,
    assert = require('assert');

app.engine('html', engines.nunjucks);
app.set('view engine', 'html');
app.set('views', __dirname + '/views');

MongoClient.connect('mongodb://localhost:27017/video', function(err, db) {

    assert.equal(null, err);
    console.log("Successfully connected to MongoDB.");

    app.get('/', function(req, res){

        db.collection('movies').find({}).toArray(function(err, docs) {
            res.render('movies', { 'movies': docs } );
        });

    });

    app.use(function(req, res){
        res.sendStatus(404);
    });
    
    var server = app.listen(3000, function() {
        var port = server.address().port;
        console.log('Express server listening on port %s.', port);
    });

});

package.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
    "name": "all_together_now",
    "version": "0.1.0",
    "description": "Using the MongoDB driver together with route handling and templates",
    "main": "app.js",
    "dependencies": {
        "consolidate": "~0.13.1",
        "express": "~4.13.3",
        "mongodb": "~2.1.3",
        "nunjucks": "~2.2.0"
    },
    "author": "Shannon Bradshaw",
    "license": "0BSD"
}

views/movies.html

1
2
3
4
5
6
7
8
9
10
11
12
<head>
<style>
body { font-family: 'Helvetica', 'Arial', sans-serif; }
</style>
</head>

<h1>Movies</h1>
{% for movie in movies %}
  <li><a href="http://www.imdb.com/title/{{ movie.imdb }}">{{ movie.title }}, {{ movie.year }}</a></li>
{% else %}
  <li>No movies found.</li>
{% endfor %}

Express: Handling GET Requests

app.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
var express = require('express'),
    app = express(),
    engines = require('consolidate');

app.engine('html', engines.nunjucks);
app.set('view engine', 'html');
app.set('views', __dirname + '/views');

// Handler for internal server errors
function errorHandler(err, req, res, next) {
    console.error(err.message);
    console.error(err.stack);
    res.status(500).render('error_template', { error: err });
}

app.get('/:name', function(req, res, next) {
    var name = req.params.name;
    var getvar1 = req.query.getvar1;
    var getvar2 = req.query.getvar2;
    res.render('hello', { name : name, getvar1 : getvar1, getvar2 : getvar2 });
});

app.use(errorHandler);

var server = app.listen(3000, function() {
    var port = server.address().port;
    console.log('Express server listening on port %s.', port);
});

package.json

1
2
3
4
5
6
7
8
9
10
11
12
13
{
    "name": "get_express",
    "version": "0.1.0",
    "description": "Get parameters in express",
    "main": "app.js",
    "dependencies": {
        "express": "~4.13.3",
        "consolidate": "~0.13.1",
        "nunjucks": "~2.2.0"
    },
    "author": "Shannon Bradshaw",
    "license": "0BSD"
}

views/error_template.html

1
<h1>Error: {{error}}</h1>

views/hello.html

1
2
3
4
5
<h1>Hello, {{name}}, here are your GET variables:</h1>
<ul>
    <li>{{getvar1}}</li>
    <li>{{getvar2}}</li>
</ul>

https://localhost:3000/shaun?getvar1=variable one&getvar2=variable two

Express: Handling POST Requests

app.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
var express = require('express'),
    app = express(),
    engines = require('consolidate'),
    bodyParser = require('body-parser');

app.engine('html', engines.nunjucks);
app.set('view engine', 'html');
app.set('views', __dirname + '/views');
app.use(bodyParser.urlencoded({ extended: true })); 

// Handler for internal server errors
function errorHandler(err, req, res, next) {
    console.error(err.message);
    console.error(err.stack);
    res.status(500).render('error_template', { error: err });
}

app.get('/', function(req, res, next) {
    res.render('fruitPicker', { 'fruits' : [ 'apple', 'orange', 'banana', 'peach' ] });
});

app.post('/favorite_fruit', function(req, res, next) {
    var favorite = req.body.fruit;
    if (typeof favorite == 'undefined') {
        next('Please choose a fruit!');
    }
    else {
        res.send("Your favorite fruit is " + favorite);
    }
});

app.use(errorHandler);

var server = app.listen(3000, function() {
    var port = server.address().port;
    console.log('Express server listening on port %s.', port);
});

package.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
    "name": "get_express",
    "version": "0.1.0",
    "description": "Get parameters in express",
    "main": "app.js",
    "dependencies": {
        "express": "~4.13.3",
        "body-parser": "~1.14.2",
        "consolidate": "~0.13.1",
        "nunjucks": "~2.2.0"
    },
    "author": "Shannon Bradshaw",
    "license": "0BSD"
}

views/error_template.html

1
<h1>Error: {{error}}</h1>

views/fruitPicker.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<!doctype HTML>
<html>
  <head><title>Fruit Picker</title></head>
  <body>
     <form action="/favorite_fruit" method="POST">
        <p>What is your favorite fruit?</p>
        {% for fruit in fruits %}
          <p>
            <input type="radio" name="fruit" value="{{fruit}}"/>{{fruit}}
          </p>
        {% endfor %}
        <input type="submit" value="Submit"/>
     </form>
  </body>
</html>

https://localhost:3000/shaun?getvar1=variable one&getvar2=variable two

Intro to the Course Project

Homework 1.1

Please refer here.

Change directory into hw1

Use mongorestore to restore the dump into your running mongod. Do this by opening a terminal window (mac) or cmd window (windows) and navigating to the directory so that the dump directory is directly beneath you. Now type:

mongorestore dump

Note you will need to have your path setup correctly to find mongorestore.

Now, using the Mongo shell, perform a find() on the collection called hw1_1 in the database m101. There is one document in this collection. Please provide the value corresponding to the “answer” key (without the surrounding quotes) from the document returned.

Answer:

Hello from MongoDB!

hw1_1.metadata.json

1
{ "indexes" : [ { "v" : 1, "key" : { "_id" : 1 }, "ns" : "m101.hw1_1", "name" : "_id_" } ] }

hw1_1.bson

1
2
3
4
3600 0000 075f 6964 0051 e452 4ef3 651c
651a 4233 1c02 616e 7377 6572 0014 0000
0048 656c 6c6f 2066 726f 6d20 4d6f 6e67
6f44 4221 0000

Homework 1.2

Please refer here.

Your assignment for this part of the homework is to install the mongodb driver for Node.js and run the test application.To do this, first download the hw1-2.zip from Download Handout link, uncompress and change into the hw1-2 directory:

cd hw1-2

Use mongorestore to restore the dump into your running mongod. Do this by opening a terminal window (mac) or cmd window (windows) and navigating to the directory so that the dump directory is directly beneath you. Now type:

mongorestore dump

Note you will need to have your path setup correctly to find mongorestore.

Then install the dependencies.

npm install

This should create a “node_modules” directory. Now run the application to get the answer to hw1-2:

node app.js

If you have the mongodb driver installed correctly, this will print out the message ‘Answer: ‘ followed by some additional text. Enter that additional text in the box below.

Enter answer here:

I like kittens

Homework 1.3

Please refer here.

Your assignment for this part of the homework is to install the mongodb driver for Node.js, Express, and other required dependencies and run the test application. This homework is meant to give you practice using the “package.json” file, which will include some of the code that we provide.

To do this, first download the hw1-3.zip from Download Handout link, uncompress and change into the hw1-3 directory:

cd hw1-3

Use mongorestore to restore the dump into your running mongod. Do this by opening a terminal window (mac) or cmd window (windows) and navigating to the directory so that the dump directory is directly beneath you. Now type:

mongorestore dump

Note you will need to have your path setup correctly to find mongorestore.

Then install all the dependencies listed in the ‘package.json’ file. Calling ‘npm install’ with no specific package tells npm to look for ‘package.json’:

npm install

This should create a “node_modules” directory with all the dependencies. Now run the application to get the answer to hw1-3:

node app.js

If you have all the dependencies installed correctly, this will print the message ‘Express server listening on port 3000’. Navigate to ‘localhost:3000’ in a browser and enter the text that is displayed on that page in the text box below.

Enter answer here:

Hello, Agent 007.

Homework 1.4

Please refer here.

Please see attached for a simple solution to this problem.

Week 2 Mongo Shell, Query Operators, Update Operators and a Few Commands

Creating Documents

insertOne()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
MongoDB Enterprise > db.moviesScratch.insertOne({ "title": "Rocky", "year": "1976", "imdb": "tt0075148"});
{
	"acknowledged" : true,
	"insertedId" : ObjectId("5a67ac35cd0e4d1d379d8a8a")
}
MongoDB Enterprise > db.moviesScratch.insertOne({ "_id": "tt0075148", "title": "Rocky", "year": "1976" });
{ "acknowledged" : true, "insertedId" : "tt0075148" }
MongoDB Enterprise > db.moviesScratch.find().pretty()
{
	"_id" : ObjectId("5a67ac35cd0e4d1d379d8a8a"),
	"title" : "Rocky",
	"year" : "1976",
	"imdb" : "tt0075148"
}
{ "_id" : "tt0075148", "title" : "Rocky", "year" : "1976" }

_id could be customized or created by MongoDB

insertMany()

  • ordered
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
MongoDB Enterprise > db.moviesScratch.insertMany(
...     [
...         {
...     "imdb" : "tt0084726",
...     "title" : "Star Trek II: The Wrath of Khan",
...     "year" : 1982,
...     "type" : "movie"
...         },
...         {
...     "imdb" : "tt0796366",
...     "title" : "Star Trek",
...     "year" : 2009,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt0084726",
...     "title" : "Star Trek II: The Wrath of Khan",
...     "year" : 1982,
...     "type" : "movie"
...         },
...         {
...     "imdb" : "tt1408101",
...     "title" : "Star Trek Into Darkness",
...     "year" : 2013,
...     "type" : "movie"
...         },
...         {
...     "imdb" : "tt0117731",
...     "title" : "Star Trek: First Contact",
...     "year" : 1996,
...     "type" : "movie"
...         }
...     ]
... );
{
	"acknowledged" : true,
	"insertedIds" : [
		ObjectId("5a67ae87cd0e4d1d379d8a8b"),
		ObjectId("5a67ae87cd0e4d1d379d8a8c"),
		"tt0084726",
		ObjectId("5a67ae87cd0e4d1d379d8a8d"),
		ObjectId("5a67ae87cd0e4d1d379d8a8e")
	]
}

However, if there an duplicated key ‘tt0084726’, it only inserted first two and stop report errors because we set ordered true, which means the order of items we insert cannot be changed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
MongoDB Enterprise > db.moviesScratch.drop()
true
MongoDB Enterprise > db.moviesScratch.insertMany(
...     [
...         {
...     "_id" : "tt0084726",
...     "title" : "Star Trek II: The Wrath of Khan",
...     "year" : 1982,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt0796366",
...     "title" : "Star Trek",
...     "year" : 2009,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt0084726",
...     "title" : "Star Trek II: The Wrath of Khan",
...     "year" : 1982,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt1408101",
...     "title" : "Star Trek Into Darkness",
...     "year" : 2013,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt0117731",
...     "title" : "Star Trek: First Contact",
...     "year" : 1996,
...     "type" : "movie"
...         }
...     ],
...     {
...         "ordered": true
...     }
... );
2018-01-24T08:53:58.692+1100 E QUERY    [thread1] BulkWriteError: write error at item 2 in bulk operation :
BulkWriteError({
	"writeErrors" : [
		{
			"index" : 2,
			"code" : 11000,
			"errmsg" : "E11000 duplicate key error collection: video.moviesScratch index: _id_ dup key: { : \"tt0084726\" }",
			"op" : {
				"_id" : "tt0084726",
				"title" : "Star Trek II: The Wrath of Khan",
				"year" : 1982,
				"type" : "movie"
			}
		}
	],
	"writeConcernErrors" : [ ],
	"nInserted" : 2,
	"nUpserted" : 0,
	"nMatched" : 0,
	"nModified" : 0,
	"nRemoved" : 0,
	"upserted" : [ ]
})
BulkWriteError@src/mongo/shell/bulk_api.js:369:48
BulkWriteResult/this.toError@src/mongo/shell/bulk_api.js:333:24
Bulk/this.execute@src/mongo/shell/bulk_api.js:1177:1
DBCollection.prototype.insertMany@src/mongo/shell/crud_api.js:314:5
@(shell):1:1
  • unordered

‘unordered’ is to solve the above problem. It can jump over the inserted item with errors.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
MongoDB Enterprise > db.moviesScratch.drop()
true
MongoDB Enterprise > db.moviesScratch.insertMany(
...     [
...         {
...     "_id" : "tt0084726",
...     "title" : "Star Trek II: The Wrath of Khan",
...     "year" : 1982,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt0796366",
...     "title" : "Star Trek",
...     "year" : 2009,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt0084726",
...     "title" : "Star Trek II: The Wrath of Khan",
...     "year" : 1982,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt1408101",
...     "title" : "Star Trek Into Darkness",
...     "year" : 2013,
...     "type" : "movie"
...         },
...         {
...     "_id" : "tt0117731",
...     "title" : "Star Trek: First Contact",
...     "year" : 1996,
...     "type" : "movie"
...         }
...     ],
...     {
...         "ordered": false
...     }
... );
2018-01-24T09:03:41.130+1100 E QUERY    [thread1] BulkWriteError: write error at item 2 in bulk operation :
BulkWriteError({
	"writeErrors" : [
		{
			"index" : 2,
			"code" : 11000,
			"errmsg" : "E11000 duplicate key error collection: video.moviesScratch index: _id_ dup key: { : \"tt0084726\" }",
			"op" : {
				"_id" : "tt0084726",
				"title" : "Star Trek II: The Wrath of Khan",
				"year" : 1982,
				"type" : "movie"
			}
		}
	],
	"writeConcernErrors" : [ ],
	"nInserted" : 4,
	"nUpserted" : 0,
	"nMatched" : 0,
	"nModified" : 0,
	"nRemoved" : 0,
	"upserted" : [ ]
})
BulkWriteError@src/mongo/shell/bulk_api.js:369:48
BulkWriteResult/this.toError@src/mongo/shell/bulk_api.js:333:24
Bulk/this.execute@src/mongo/shell/bulk_api.js:1177:1
DBCollection.prototype.insertMany@src/mongo/shell/crud_api.js:314:5
@(shell):1:1

The _id Field

_id must be in each document. It is a unique primary key.

MongoDB will create ObjectID for _id value if we don’t supply one.

ObjectId(12-BYTE HEX STRING) = DATE + MAC_ADDR + PID + COUNTER

Reading Documents

restore data from all the materials

1
2
3
4
5
6
7
➜  video mongorestore --db video movieDetails.bson 
2018-01-24T09:22:57.135+1100	checking for collection data in movieDetails.bson
2018-01-24T09:22:57.137+1100	reading metadata for video.movieDetails from movieDetails.metadata.json
2018-01-24T09:22:57.188+1100	restoring video.movieDetails from movieDetails.bson
2018-01-24T09:22:57.254+1100	no indexes to restore
2018-01-24T09:22:57.254+1100	finished restoring video.movieDetails (2295 documents)
2018-01-24T09:22:57.254+1100	done

  • filter {field : value}

    1
    2
    MongoDB Enterprise > db.movieDetails.find({ rated : "PG-13"} ).count()
    153
1
2
MongoDB Enterprise > db.movieDetails.find({ rated : "PG-13", year : 2009 } ).count()
8


- nested structure - object, use dot nation

    
1
2
MongoDB Enterprise > db.movieDetails.find( { "tomato.meter" : 100 } ).count()
8



    do not forget the double quote


- nested structure - array, order is a matter

    Equality Matches on Arrays  
     - ON THE ENTIRE ARRAY  
     - BASED ON ANY ELEMENT  
     - BASED ON A SPECIFIC ELEMENT  
     - MORE COMPLEX MATCHES USING OPERATOR (DISCUSSED IN ANOTHER LESSON)  

    
1
2
3
4
5
6
7
# ON THE ENTIRE ARRAY  
MongoDB Enterprise > db.movieDetails.find( { "writers" : ["Ethan Coen", "Joel Coen"] } ).count()
1
MongoDB Enterprise >
MongoDB Enterprise > db.movieDetails.find( { "writers" : ["Joel Coen", "Ethan Coen"] } ).count()
0
MongoDB Enterprise >



    do not forget brackets for the array

    
1
2
3
# BASED ON ANY ELEMENT 
MongoDB Enterprise > db.movieDetails.find( { "actors" : "Jeff Bridges"  } ).count()
4



    
1
2
3
# BASED ON A SPECIFIC ELEMENT
MongoDB Enterprise > db.movieDetails.find( { "actors.0" : "Jeff Bridges"  } ).count()
2
  • cursors
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
MongoDB Enterprise > var c = db.movieDetails.find();
MongoDB Enterprise > var doc = function() { return c.hasNext() ? c.next() : null;}
MongoDB Enterprise > c.objsLeftInBatch()
101
MongoDB Enterprise > doc()
{
	"_id" : ObjectId("5692a50724de1e0ce2dfdc4e"),
	"title" : "Be Cool",
	"year" : 2005,
	"rated" : "PG-13",
	"released" : ISODate("2005-03-04T05:00:00Z"),
	"runtime" : 118,
	"countries" : [
		"USA"
	],
	"genres" : [
		"Comedy",
		"Crime",
		"Music"
	],
	"director" : "F. Gary Gray",
	"writers" : [
		"Elmore Leonard",
		"Peter Steinfeld"
	],
	"actors" : [
		"John Travolta",
		"Uma Thurman",
		"Vince Vaughn",
		"Cedric the Entertainer"
	],
	"plot" : "Disenchanted with the movie industry, Chili Palmer tries the music industry, meeting and romancing a widow of a music executive on the way.",
	"poster" : "http://ia.media-imdb.com/images/M/MV5BMTUxNDc4MzAyNV5BMl5BanBnXkFtZTYwMzQ1NDc2._V1_SX300.jpg",
	"imdb" : {
		"id" : "tt0377471",
		"rating" : 5.6,
		"votes" : 58073
	},
	"tomato" : {
		"meter" : 30,
		"image" : "rotten",
		"rating" : 4.6,
		"reviews" : 169,
		"fresh" : 50,
		"consensus" : "Be Cool is tepid, square, and lukewarm; as a parody of the music business, it has two left feet.",
		"userMeter" : 42,
		"userRating" : 2.8,
		"userReviews" : 212417
	},
	"metacritic" : 37,
	"awards" : {
		"wins" : 1,
		"nominations" : 13,
		"text" : "1 win & 13 nominations."
	},
	"type" : "movie"
}
MongoDB Enterprise > c.objsLeftInBatch()
100

  • projection { field : 1 or 0}

    1 - include

    0 - exclude

    a light way to reduce the returned result… limit the returned fields 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    MongoDB Enterprise > db.movieDetails.find( { rated: "PG" }, { title : 1, _id : 0} )
    { "title" : "I Am David" }
    { "title" : "E.T. the Extra-Terrestrial" }
    { "title" : "Star Wars: Episode IV - A New Hope" }
    { "title" : "Star Wars: Episode VI - Return of the Jedi" }
    { "title" : "Star Wars: Episode I - The Phantom Menace" }
    { "title" : "Star Wars: Episode II - Attack of the Clones" }
    { "title" : "Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb" }
    { "title" : "Zathura: A Space Adventure" }
    { "title" : "The Adventures of Tintin" }
    { "title" : "The Adventures of Sharkboy and Lavagirl 3-D" }
    { "title" : "The Adventures of Buckaroo Banzai Across the 8th Dimension" }
    { "title" : "The Adventures of Rocky & Bullwinkle" }
    { "title" : "The Truman Show" }
    { "title" : "Dead Poets Society" }
    { "title" : "Treasure Planet" }
    { "title" : "Planet 51" }
    { "title" : "The Karate Kid" }
    { "title" : "The Karate Kid, Part II" }
    { "title" : "Honey I Blew Up the Kid" }
    { "title" : "Diary of a Wimpy Kid" }
    Type "it" for more

Comparision Operators

here

1
2
3
4
5
6
7
8
9
10
11
MongoDB Enterprise > db.movieDetails.find( { runtime : { $gt : 90} } ).count()
1004
MongoDB Enterprise > db.movieDetails.find( { runtime : { $gt : 90, $lte : 120 } } ).count()
742
MongoDB Enterprise > db.movieDetails.find( { "tomato.meter" : { $gte : 95 } , runtime : { $gt : 180 } }, { title : 1, _id : 0} )
{ "title" : "Lagaan: Once Upon a Time in India" }
{ "title" : "The Godfather: Part II" }
MongoDB Enterprise > db.movieDetails.find({ rated : { $ne : "UNRATED" } }, { title : 1, _id : 0 }).count()
2263
MongoDB Enterprise > db.movieDetails.find({ rated : { $in : ["R", "PG-13"]  } }, { rated : 1, title : 1, _id : 0 }).count()
367

Element Operators

here

  • $exists 

    1
    2
    MongoDB Enterprise > db.movieDetails.find( { "tomato.meter": { $exists: true} }, { "tomato.meter" : 1} ).count()
    362

  • $type

    load data

    1
    2
    3
    4
    5
    6
    7
    video mongorestore --db video moviesScratch.bson 
    2018-01-24T10:16:29.102+1100	checking for collection data in moviesScratch.bson
    2018-01-24T10:16:29.103+1100	reading metadata for video.moviesScratch from moviesScratch.metadata.json
    2018-01-24T10:16:29.153+1100	restoring video.moviesScratch from moviesScratch.bson
    2018-01-24T10:16:29.219+1100	no indexes to restore
    2018-01-24T10:16:29.219+1100	finished restoring video.moviesScratch (8 documents)
    2018-01-24T10:16:29.219+1100	done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
MongoDB Enterprise > db.moviesScratch.find()
{ "_id" : "tt0084726", "title" : "Star Trek II: The Wrath of Khan", "year" : 1982, "type" : "movie" }
{ "_id" : "tt0117731", "title" : "Star Trek: First Contact", "year" : 1996, "type" : "movie" }
{ "_id" : "tt0796366", "title" : "Star Trek", "year" : 2009, "type" : "movie" }
{ "_id" : "tt1408101", "title" : "Star Trek Into Darkness", "year" : 2013, "type" : "movie" }
{ "_id" : ObjectId("56945e5ebcb8793f83332495"), "imdb" : "tt0084726", "title" : "Star Trek II: The Wrath of Khan", "year" : 1982, "type" : "movie" }
{ "_id" : ObjectId("56945e5ebcb8793f83332496"), "imdb" : "tt0796366", "title" : "Star Trek", "year" : 2009, "type" : "movie" }
{ "_id" : ObjectId("56945e5ebcb8793f83332497"), "imdb" : "tt1408101", "title" : "Star Trek Into Darkness", "year" : 2013, "type" : "movie" }
{ "_id" : ObjectId("56945e5ebcb8793f83332498"), "imdb" : "tt0117731", "title" : "Star Trek: First Contact", "year" : 1996, "type" : "movie" }
MongoDB Enterprise > db.moviesScratch.find({ "_id" : { $type : "string"} })
{ "_id" : "tt0084726", "title" : "Star Trek II: The Wrath of Khan", "year" : 1982, "type" : "movie" }
{ "_id" : "tt0117731", "title" : "Star Trek: First Contact", "year" : 1996, "type" : "movie" }
{ "_id" : "tt0796366", "title" : "Star Trek", "year" : 2009, "type" : "movie" }
{ "_id" : "tt1408101", "title" : "Star Trek Into Darkness", "year" : 2013, "type" : "movie" }

Logical Operators

here

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
MongoDB Enterprise > db.movieDetails.find({ $or : [ {"tomato.meter" : { $gt : 95 } }, { "metacritic" : { $gt : 88 } } ] }, {"tomato.meter" : 1, "metacritic": 1}).limit(3).pretty()
{
	"_id" : ObjectId("5692a50f24de1e0ce2dfdc5c"),
	"tomato" : {
		"meter" : 97
	},
	"metacritic" : 81
}
{
	"_id" : ObjectId("5692a52a24de1e0ce2dfdc97"),
	"tomato" : {
		"meter" : 98
	},
	"metacritic" : 94
}
{
	"_id" : ObjectId("5692a52f24de1e0ce2dfdca2"),
	"tomato" : {
		"meter" : 100
	},
	"metacritic" : 85
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
MongoDB Enterprise > db.movieDetails.find({ $and : [ {"tomato.meter" : { $gt : 95 } }, { "metacritic" : { $gt : 88 } } ] }, {"tomato.meter" : 1, "metacritic": 1}).limit(3).pretty()
{
	"_id" : ObjectId("5692a52a24de1e0ce2dfdc97"),
	"tomato" : {
		"meter" : 98
	},
	"metacritic" : 94
}
{
	"_id" : ObjectId("569190cd24de1e0ce2dfcd63"),
	"tomato" : {
		"meter" : 99
	},
	"metacritic" : 96
}
{
	"_id" : ObjectId("5692a15324de1e0ce2dfcf93"),
	"tomato" : {
		"meter" : 98
	},
	"metacritic" : 99
}

Regex Operator

here

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
MongoDB Enterprise > db.movieDetails.find( { "awards.text" : { $regex : /^Won.*/ } } , { "awards.text" : 1 })
{ "_id" : ObjectId("5692a52a24de1e0ce2dfdc97"), "awards" : { "text" : "Won 4 Oscars. Another 46 wins & 32 nominations." } }
{ "_id" : ObjectId("5692a54d24de1e0ce2dfdcd5"), "awards" : { "text" : "Won 1 Oscar. Another 2 wins & 3 nominations." } }
{ "_id" : ObjectId("569190cb24de1e0ce2dfcd52"), "awards" : { "text" : "Won 10 Oscars. Another 18 wins & 11 nominations." } }
{ "_id" : ObjectId("569190cc24de1e0ce2dfcd59"), "awards" : { "text" : "Won 6 Oscars. Another 38 wins & 27 nominations." } }
{ "_id" : ObjectId("569190cd24de1e0ce2dfcd65"), "awards" : { "text" : "Won 7 Oscars. Another 55 wins & 85 nominations." } }
{ "_id" : ObjectId("569190d024de1e0ce2dfcd7a"), "awards" : { "text" : "Won 1 Oscar. Another 9 wins & 16 nominations." } }
{ "_id" : ObjectId("569190d124de1e0ce2dfcd86"), "awards" : { "text" : "Won 2 Oscars. Another 16 wins & 22 nominations." } }
{ "_id" : ObjectId("569190d124de1e0ce2dfcd87"), "awards" : { "text" : "Won 2 Oscars. Another 4 wins & 5 nominations." } }
{ "_id" : ObjectId("569190d124de1e0ce2dfcd8b"), "awards" : { "text" : "Won 1 Oscar. Another 41 wins & 48 nominations." } }
{ "_id" : ObjectId("569190d224de1e0ce2dfcd8d"), "awards" : { "text" : "Won 1 Oscar. Another 17 wins & 18 nominations." } }
{ "_id" : ObjectId("5692c8fc24de1e0ce2dfe27d"), "awards" : { "text" : "Won 1 Primetime Emmy. Another 3 nominations." } }
{ "_id" : ObjectId("5692c8fc24de1e0ce2dfe27f"), "awards" : { "text" : "Won 1 Primetime Emmy. Another 1 nomination." } }
{ "_id" : ObjectId("5692a14024de1e0ce2dfced7"), "awards" : { "text" : "Won 3 Oscars. Another 24 wins & 15 nominations." } }
{ "_id" : ObjectId("5692a14124de1e0ce2dfcee1"), "awards" : { "text" : "Won 1 Oscar. Another 9 wins & 5 nominations." } }
{ "_id" : ObjectId("5692a14424de1e0ce2dfcef7"), "awards" : { "text" : "Won 1 Oscar. Another 4 wins & 13 nominations." } }
{ "_id" : ObjectId("5692a14d24de1e0ce2dfcf4d"), "awards" : { "text" : "Won 2 Oscars. Another 10 wins & 8 nominations." } }
{ "_id" : ObjectId("5692a14e24de1e0ce2dfcf56"), "awards" : { "text" : "Won 2 Oscars. Another 32 wins & 26 nominations." } }
{ "_id" : ObjectId("5692a14f24de1e0ce2dfcf62"), "awards" : { "text" : "Won 1 Oscar. Another 11 wins & 51 nominations." } }
{ "_id" : ObjectId("5692a15024de1e0ce2dfcf73"), "awards" : { "text" : "Won 1 Oscar. Another 24 wins & 81 nominations." } }
{ "_id" : ObjectId("5692a15024de1e0ce2dfcf74"), "awards" : { "text" : "Won 2 BAFTA Film Awards. Another 17 wins & 34 nominations." } }
Type "it" for more

Array Operators

here

1
2
3
4
5
MongoDB Enterprise > db.movieDetails.find( { genres: { $all: ["Comedy", "Drama", "Crime"] } }, { genres : 1 } ).pretty().count()
8
MongoDB Enterprise > db.movieDetails.find( { genres: { $all: ["Comedy", "Crime", "Drama", ] } }, { genres : 1 } ).pretty().count()
8
MongoDB Enterprise >
1
2
MongoDB Enterprise > db.movieDetails.find( { countries : { $size : 1 } } ).count()
1915

assume there is 

1
2
3
4
5
boxOffice: [ { "country": "USA", "revenue": 41.3 },
             { "country": "Australia", "revenue": 2.9 },
             { "country": "UK", "revenue": 10.1 },
             { "country": "Germany", "revenue": 4.3 },
             { "country": "France", "revenue": 3.5 } ]
1
2
3
4
5
6
 
# this one match those records with country UK or revenue greater than 15 in boxoffice
db.movieDetails.find({ boxOffice: { country: "UK", revenue: { $gt: 15 } } })

# this one match those records with the whole element with country UK and revenue greater than 15
db.movieDetails.find({ boxOffice: {$elemMatch: { country: "UK", revenue: { $gt: 15 } } } })

Updating Documents

update operator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
db.movieDetails.updateOne({
  title: "The Martian"
}, {
  $set: {
    poster: "http://ia.media-imdb.com/images/M/MV5BMTc2MTQ3MDA1Nl5BMl5BanBnXkFtZTgwODA3OTI4NjE@._V1_SX300.jpg"
  }
})

db.movieDetails.updateOne({
  title: "The Martian"
}, {
  $set: {
    "awards": {
      "wins": 8,
      "nominations": 14,
      "text": "Nominated for 3 Golden Globes. Another 8 wins & 14 nominations."
    }
  }
});

/*
 * Updates are used to correct errors and, over time, keep our data current. 
 * For movie data, much of what's there is static: directors, authors and the 
 * like. Other content such as reviews and ratings will need to be updated as 
 * users take action. We could use $set for this purpose, but that's an error 
 * prone approach. It's too easy to do the arithmetic incorrectly. Instead, we 
 * have a number of operators that support numeric updates of data: 
 * $min, $max, $inc, $mul. Let's look at an example using $inc to update reviews.
 */

db.movieDetails.updateOne({
  title: "The Martian"
}, {
  $inc: {
    "tomato.reviews": 3,
    "tomato.userReviews": 25
  }
})

var reviewText1 = [
  "The Martian could have been a sad drama film, instead it was a ",
  "hilarious film with a little bit of drama added to it. The Martian is what ",
  "everybody wants from a space adventure. Ridley Scott can still make great ",
  "movies and this is one of his best."
].join()
db.movieDetails.updateOne({
  title: "The Martian"
}, {
  $push: {
    reviews: {
      rating: 4.5,
      date: ISODate("2016-01-12T09:00:00Z"),
      reviewer: "Spencer H.",
      text: reviewText1
    }
  }
})

var reviewText2 = [
  "i believe its ranked high due to its slogan 'Bring him Home' there is nothi",
  "ng in the movie, nothing at all ! Story telling for fiction story !"
].join()

var reviewText3 = [
  "This is a masterpiece. The ending is quite different from the book - the mo",
  "vie provides a resolution whilst a book doesn't."
].join()

var reviewText4 = [
  "There have been better movies made about space, and there are elements of t",
  "he film that are borderline amateur, such as weak dialogue, an uneven tone,",
  " and film cliches."
].join()

var reviewText5 = [
  "This novel-adaptation is humorous, intelligent and captivating in all its v",
  "isual-grandeur. The Martian highlights an impeccable Matt Damon, power-stac",
  "ked ensemble and Ridley Scott's masterful direction, which is back in full ",
  "form."
].join()

var reviewText6 = [
  "A declaration of love for the potato, science and the indestructible will t",
  "o survive. While it clearly is the Matt Damon show (and he is excellent), t",
  "he supporting cast may be among the strongest seen on film in the last 10 y",
  "ears. An engaging, exciting, funny and beautifully filmed adventure thrille",
  "r no one should miss."
].join()

var reviewText7 = [
  "The Martian could have been a sad drama film, instead it was a hilarious fi",
  "lm with a little bit of drama added to it. The Martian is what everybody wa",
  "nts from a space adventure. Ridley Scott can still make great movies and th",
  "is is one of his best."
].join()

db.movieDetails.updateOne({
  title: "The Martian"
}, {
  $push: {
    reviews: {
      $each: [{
        rating: 0.5,
        date: ISODate("2016-01-12T07:00:00Z"),
        reviewer: "Yabo A.",
        text: reviewText2
      }, {
        rating: 5,
        date: ISODate("2016-01-12T09:00:00Z"),
        reviewer: "Kristina Z.",
        text: reviewText3
      }, {
        rating: 2.5,
        date: ISODate("2015-10-26T04:00:00Z"),
        reviewer: "Matthew Samuel",
        text: reviewText4
      }, {
        rating: 4.5,
        date: ISODate("2015-12-13T03:00:00Z"),
        reviewer: "Eugene B",
        text: reviewText5
      }, {
        rating: 4.5,
        date: ISODate("2015-10-22T00:00:00Z"),
        reviewer: "Jens S",
        text: reviewText6
      }, {
        rating: 4.5,
        date: ISODate("2016-01-12T09:00:00Z"),
        reviewer: "Spencer H.",
        text: reviewText7
      }
      ]
    }
  }
})


db.movieDetails.updateOne({
  title: "The Martian"
}, {
  $push: {
    reviews: {
      $each: [{
        rating: 0.5,
        date: ISODate("2016-01-13T07:00:00Z"),
        reviewer: "Shannon B.",
        text: "Enjoyed watching with my kids!"
      }],
      $position: 0,
      $slice: 5
    }
  }
})


// Could do this, but it's probably the wrong semantics.
db.movieDetails.updateMany({
  rated: null
}, {
  $set: {
    rated: "UNRATED"
  }
})


// Better to do this.
db.movieDetails.updateMany({
  rated: null
}, {
  $unset: {
    rated: ""
  }
})


var detail = {
  "title": "The Martian",
  "year": 2015,
  "rated": "PG-13",
  "released": ISODate("2015-10-02T04:00:00Z"),
  "runtime": 144,
  "countries": [
    "USA",
    "UK"
  ],
  "genres": [
    "Adventure",
    "Drama",
    "Sci-Fi"
  ],
  "director": "Ridley Scott",
  "writers": [
    "Drew Goddard",
    "Andy Weir"
  ],
  "actors": [
    "Matt Damon",
    "Jessica Chastain",
    "Kristen Wiig",
    "Jeff Daniels"
  ],
  "plot": "During a manned mission to Mars, Astronaut Mark Watney is presumed" +
  " dead after a fierce storm and left behind by his crew. But Watney has" +
  " survived and finds himself stranded and alone on the hostile planet." +
  " With only meager supplies, he must draw upon his ingenuity, wit and " +
  "spirit to subsist and find a way to signal to Earth that he is alive.",
  "poster": "http://ia.media-imdb.com/images/M/" +
  "MV5BMTc2MTQ3MDA1Nl5BMl5BanBnXkFtZTgwODA3OTI4NjE@._V1_SX300.jpg",
  "imdb": {
    "id": "tt3659388",
    "rating": 8.2,
    "votes": 187881
  },
  "tomato": {
    "meter": 93,
    "image": "certified",
    "rating": 7.9,
    "reviews": 280,
    "fresh": 261,
    "consensus": "Smart, thrilling, and surprisingly funny, The Martian offers" +
    " a faithful adaptation of the bestselling book that brings out the best " +
    "in leading man Matt Damon and director Ridley Scott.",
    "userMeter": 92,
    "userRating": 4.3,
    "userReviews": 104999
  },
  "metacritic": 80,
  "awards": {
    "wins": 8,
    "nominations": 14,
    "text": "Nominated for 3 Golden Globes. Another 8 wins & 14 nominations."
  },
  "type": "movie"
};


db.movieDetails.updateOne({
  "imdb.id": detail.imdb.id
}, {
  $set: detail
}, {
  upsert: true
});


db.movies.replaceOne({
    "imdb": detail.imdb.id
  },
  detail);


db.reviews.insertMany([{
  rating: 2.5,
  date: ISODate("2015-10-26T04:00:00Z"),
  reviewer: "Matthew Samuel",
  text: "There have been better movies made about space, and there are" +
  " elements of the film that are borderline amateur, such as weak dialogue," +
  " an uneven tone, and film cliches."
}, {
  rating: 5,
  date: ISODate("2015-12-18T09:00:00Z"),
  reviewer: "Jarrad C",
  text: "The Martian Review: There are some movies you know going into" +
  " them that they're going to be great. The Martian was not only great but" +
  " exceeded all my expectations as well. A return to form from director " +
  "Ridley Scott (Alien, Blade Runner) who directed last year's abysmal " +
  "'Exodus: Gods and Kings (which I adeptly gave 1/5 because I was happy that" +
  " it ended), The Martian soars to the heights of a great space epic. " +
  "In fact the thing that I loved about the Martian more than Alfonso Curon's" +
  " 'Gravity' and Christopher Nolan's 'Interstellar' is its optimism." +
  " Botanist Mark Watney played by an incredible Matt Damon, plays his " +
  "character to a tee without depressing or alienating viewers. It is " +
  "through this lens that we see plausible science (well somewhat plausible)" +
  " succeed and the audience truly root for Mark Watney to be rescued by his" +
  " Ares III crew creating a palpitating suspense during the rescue launch." +
  " That aside the supporting cast is impeccable, literally every single" +
  " character feels important and is given enough screen-time to shine," +
  " from Jessica Chastain's Commander Lewis to Chitwetel Ejiofer's Vincent" +
  " Kapoor. Also to be highly commended is the diversity of this cast- where" +
  " every character is simply just equal- just the way it should be in every film."
}, {
  rating: 3,
  date: ISODate("2015-12-13T03:00:00Z"),
  reviewer: "hunterjt13",
  text: "An astronaut/botanist is stranded on Mars and must rely upon " +
  "ingenuity to survive. It's hard to divorce my opinions about this film " +
  "from my opinions of the book, for after the film I thought that they had" +
  " adapted the book but left out all the good parts. The cut that bleeds " +
  "most is the final chapter, which gives the whole story a philosophical" +
  " significance that is both poignant and naively inspiring. Nevertheless," +
  " what remains is a fine tale of how science, wit, perseverance, and " +
  "intelligence are greater weapons than anything else. Mark Watney's humor" +
  " survives in the film, even if the film strips him of his vulgarity, all" +
  " in the service of obtaining a PG-13 rating (really, we must reexamine " +
  "why one 'fuck' corrupts thirteen- to seventeen-year-olds so much more than " +
  "ten, twenty, or 100 'fucks'). Mark Watney's intelligence survives the film" +
  " version, even if the film's story presents him with fewer obstacles to " +
  "'science the shit out of,' all in the service of giving director Ridley " +
  "Scott the opportunity to show beautiful Martian sunsets in the time " +
  "allowed. That Mark Watney character is quite a survivor. Overall, " +
  "please watch the film - it's good and it's values are inspiring - but read the book too."
}, {
  rating: 4.5,
  date: ISODate("2015-12-13T03:00:00Z"),
  reviewer: "Eugene B",
  text: "This novel-adaptation is humorous, intelligent and captivating in" +
  " all its visual-grandeur. The Martian highlights an impeccable Matt Damon," +
  " power-stacked ensemble and Ridley Scott's masterful direction, which is back in full form."
}, {
  rating: 3.5,
  date: ISODate("2015-10-10T03:00:00Z"),
  reviewer: "Kevin M. W",
  text: "Personable sci-fi flick from Ridley Scott that updates the old " +
  "Robinson Crusoe stuck on a desert isle (and how he learns to survive) " +
  "trope. First off, make that a deserted planet. Mars. And the background " +
  "and the hope this film presents is grounded on is our quest to reach beyond" +
  " the limits of Earth's gravity (and so it loudly parades that we'll " +
  "overcome every adversity). Now there's science here but let's not get too" +
  " excited. The emphasis is interestingly on the personalities involved, " +
  "interestingly because while the characters are the primary focus of this " +
  "tale on one hand, on the other hand they are only the flimsiest of " +
  "constructs with zero time spent 'getting to know them'. We only watch while" +
  " they react to danger. Who they are or who they might be outside of these" +
  " situations, we never get to see. This momentary drawback weakens the film, " +
  "but only by a bit. Its worth a watch, Jessica Chastain and Jeff Daniels " +
  "leading a deep bench of secondary players."
}, {
  rating: 4.5,
  date: ISODate("2015-10-13T03:00:00Z"),
  reviewer: "Drake T",
  text: "Equal parts fun, smart and thrilling. The Martian is a survival " +
  "story that doesn't ramp up melodrama but instead chooses to humanize " +
  "Watney's struggle with entertaining laughs. A love letter from Ridley to " +
  "fans of the book, it's simply flawless in pacing, dialogue and " +
  "characterization. The only element missing is visual style, there's " +
  "nothing iconic to look at that'll let us recall the film in years to come." +
  " Still it's beautifully directed and perfectly acted, an exciting ride " +
  "through and through. Most definitely a piece of hard fiction that'll leave" +
  " a mark for 2015 rivaling the success of genre-similar 'Interstellar' of " +
  "2014 yet completely different in approach of tone."
}, {
  rating: 4.5,
  date: ISODate("2015-10-22T00:00:00Z"),
  reviewer: "Jens S",
  text: "A declaration of love for the potato, science and the indestructible" +
  " will to survive. While it clearly is the Matt Damon show (and he is " +
  "excellent), the supporting cast may be among the strongest seen on film " +
  "in the last 10 years. An engaging, exciting, funny and beautifully filmed " +
  "adventure thriller no one should miss."
}, {
  rating: 2.5,
  date: ISODate("2015-09-17T02:00:00Z"),
  reviewer: "FiLmCrAzY",
  text: "This movie has everything, excitement, drama, emotion, humour, " +
  "strong characters led by Damon but ultimately fails to capture my attention!"
}, {
  rating: 3.5,
  date: ISODate("2015-09-21T11:00:00Z"),
  reviewer: "Sanjay R",
  text: "I really like how this film relied more on story than on action " +
  "and special effects. Goddard manages to keep the screenplay interesting, " +
  "and sometimes funny, throughout the entire 2 hours and 14 minutes. One of " +
  "my pet peeves in movies, however, is an isolated man talking to himself " +
  "and explaining all of his actions as if he knows he is being watched by an" +
  " audience and is obligated to amuse them. I know that sounds a bit like " +
  "saying 'Hey Matt Damon, be less entertaining,' but what I am really saying" +
  " is that I wish the film's entertainment came from a more natural place. " +
  "Other than my personal bete noire, the film is very well directed, acted " +
  "and written. A broad range of moviegoers will enjoy it, I would recommend " +
  "it to anyone."
}, {
  rating: 3.5,
  date: ISODate("2015-10-15T11:00:00Z"),
  reviewer: "Emile T"
}, {
  rating: 3.5,
  date: ISODate("2015-10-14T11:00:00Z"),
  reviewer: "Carlos M",
  text: "With a great 3D that explores very well the red landscapes using " +
  "mostly a large depth of field, this smart science fiction also knows how " +
  "to use exposition in its favor and works so well due to its delightful " +
  "sense of humor and efficient moments of tension when it needs to be tense."
}, {
  rating: 4.5,
  date: ISODate("2015-10-11T03:00:00Z"),
  reviewer: "Adriel L",
  text: "Out-standing!"
}, {
  rating: 3.5,
  date: ISODate("2015-10-05T03:00:00Z"),
  reviewer: "Pierluigi P",
  text: "It doesn't take itself too seriously, which translates in an " +
  "intelligent, immersive and charming adventure of survival. Untouched by " +
  "sappiness and thoughtful in execution. A triumph for Mr. Scott."
}, {
  rating: 4,
  date: ISODate("2015-09-17T03:00:00Z"),
  reviewer: "Flutie A",
  text: "Shades of 'Cast Away' & 'Apollo 13'. Really enjoyed it!"
}, {
  rating: 4.5,
  date: ISODate("2015-10-01T03:00:00Z"),
  reviewer: "KJ P",
  text: "Director Ridley Scott has hit the point in his life where it is as " +
  "unpredictable as the weather, as to how good his films will be. I can " +
  "gladly say, 'The Martian' is not only his best film in a very long time, " +
  "but it is also one os 2015's best overall pictures. After presumed dead " +
  "and left on Mars by his crew, botanist Mark Watney must use his scientific" +
  " skills to savour/grow food to last him up to four years. Filled with " +
  "unexpected humour, this film is able to balance a light-hearted tone while" +
  " digging deep into the emotion when necessary. This film had me chuckling " +
  "quite a bit, and I give full credit to Drew Goddard d Andy Weir who penned " +
  "the script. This witty sic-fi rescue mission is also as intense as it needs" +
  " to be, ramping up to the conclusion. From the visuals to the directing, " +
  "from the writing to the cinematography, and from the acting to the " +
  "believability of this film, it really is one of the biggest achievements " +
  "of the year by far. I was enthralled from start to finish. Although I have " +
  "to fault the film for ending a bit too abruptly, I can forgive that for " +
  "being the only true thing to bother me. In the end, 'The Martian' is a " +
  "damn fine piece of filmmaking. Fantastic!"
}, {
  rating: 4,
  date: ISODate("2015-11-03T09:00:00Z"),
  reviewer: "Sheldon C",
  text: "THE MARTIAN is a success because it presents its sci-fi scenarios " +
  "with care and plausibility, it keeps the plot simple and moving, and it " +
  "anchors itself with a strong lead actor in a character who is both " +
  "relatable and charming, resulting in a film that is accordingly thrilling," +
  " visually dazzling, and surprisingly funny. Although the narrative itself " +
  "does not necessarily present anything new - we've seen this scenario " +
  "before - and the story is quite predictable, Ridley Scott's latest benefits" +
  " from both a strong script that keeps momentum flowing and an especially " +
  "nuanced performance by Damon; it carries a much different tone than other " +
  "recent space dramas and for that, it is justified and comes highly recommended."
}, {
  rating: 5,
  date: ISODate("2016-01-12T09:00:00Z"),
  reviewer: "Mike S",
  text: "At no point in time did I have any doubt this movie would end " +
  "positively. Humor, ingenuity, and a beautiful adventure sums this film up," +
  " thanks to Matt Damon's impressive performance. The man's willingness to " +
  "survive with overall optimism gets him home."
}, {
  rating: 0.5,
  date: ISODate("2016-01-12T07:00:00Z"),
  reviewer: "Yabo A.",
  text: "i believe its ranked high due to its slogan 'Bring him Home' there " +
  "is nothing in the movie, nothing at all ! Story telling for fiction story !"
}, {
  rating: 5,
  date: ISODate("2016-01-12T09:00:00Z"),
  reviewer: "Kristina Z.",
  text: "This is a masterpiece. The ending is quite different from the book " +
  "- the movie provides a resolution whilst a book doesn't."
}, {
  rating: 4.5,
  date: ISODate("2016-01-12T09:00:00Z"),
  reviewer: "Spencer H.",
  text: "The Martian could have been a sad drama film, instead it was a " +
  "hilarious film with a little bit of drama added to it. The Martian is " +
  "what everybody wants from a space adventure. Ridley Scott can still make " +
  "great movies and this is one of his best."
}])

Homework 2.1

Which of the choices below is the title of a movie from the year 2013 that is rated PG-13 and won no awards? Please query the video.movieDetails collection to find the answer.

NOTE: There is a dump of the video database included in the handouts for the “Creating Documents” lesson. Use that data set to answer this question.

1
2
3
MongoDB Enterprise > db.movieDetails.find({rated:"PG-13", "awards.wins" : 0, year : 2003}, {title : 1, "_id" : 0}).pretty()
{ "title" : "Ek Aur Ek Gyarah: By Hook or by Crook" }
MongoDB Enterprise >

Homework 2.2

Using the video.movieDetails collection, which of the queries below would produce output documents that resemble the following. Check all that apply.

NOTE: We are not asking you to consider specifically which documents would be output from the queries below, but rather what fields the output documents would contain.

1
2
3
{ "title" : "P.S. I Love You" }
{ "title" : "Love Actually" }
{ "title" : "Shakespeare in Love" }

db.movieDetails.find({}, {title: 1, _id: 0})

db.movieDetails.find({}, {title: 1})

db.movieDetails.find({title: “”}, {title: 1})

db.movieDetails.find({}, {title})

db.movieDetails.find({year: 1964}, {title: 1, _id: 0})

db.movieDetails.find({title: “Muppets from Space”}, {title: 1})

The key to answering this question is in understanding how projection works in find() queries. Specifically, we’re looking for queries that include projection documents such as the following.

{title: 1, _id: 0}

Homework 2.3

Using the video.movieDetails collection, how many movies list “Sweden” second in the the list of countries.

NOTE: There is a dump of the video database included in the handouts for the “Creating Documents” lesson. Use that data set to answer this question.

1
2
3
MongoDB Enterprise > db.movieDetails.find({ "countries.1" : "Sweden"} ).count()
6
MongoDB Enterprise >

Answer
For this you want to use dot notation to specify the second element of the array as the key against which to match.

db.movieDetails.find({ "countries.1": "Sweden" })

Homework 2.4

How many documents in our video.movieDetails collection list just the following two genres: “Comedy” and “Crime” with “Comedy” listed first.

NOTE: There is a dump of the video database included in the handouts for the “Creating Documents” lesson. Use that data set to answer this question.

1
2
3
MongoDB Enterprise > db.movieDetails.find( {genres: ["Comedy", "Crime"]} ).count()
20
MongoDB Enterprise >

For this question we need to do an equality query on an entire array as follows:

db.movieDetails.find({ "genres": ["Comedy", "Crime"] }).count()

Homework 2.5

As a follow up to the previous question, how many documents in the video.movieDetails collection list both “Comedy” and “Crime” as genres regardless of how many other genres are listed?

NOTE: There is a dump of the video database included in the handouts for the “Creating Documents” lesson. Use that data set to answer this question.

1
2
3
MongoDB Enterprise > db.movieDetails.find( {genres: { $all : ["Comedy", "Crime"] }} ).count()
56
MongoDB Enterprise >

Answer
The following is one way of answering this question:

db.movieDetails.find({ "genres": {$all: ["Comedy", "Crime"] } })

Homework 2.6

Suppose you wish to update the value of the “plot” field for one document in our “movieDetails” collection to correct a typo. Which of the following update operators and modifiers would you need to use to do this?

Answer: $set

Challenge Problem : Arrays with Nested Documents

Quiz: Challenge Problem: Arrays with Nested Documents
This problem is provided as a supplementary learning opportunity. It is more challenging that the ordinary homework. It is ungraded. We do not ask you submit an answer.

Suppose our movie details documents are structured so that rather than contain an awards field that looks like this:

1
2
3
4
5
"awards" : {
    "wins" : 56,
    "nominations" : 86,
    "text" : "Won 2 Oscars. Another 56 wins and 86 nominations."
}

they are structured with an awards field as follows:

1
2
3
4
5
6
7
8
9
10
11
12
"awards" : {
    "oscars" : [
        {"award": "bestAnimatedFeature", "result": "won"},
        {"award": "bestMusic", "result": "won"},
        {"award": "bestPicture", "result": "nominated"},
        {"award": "bestSoundEditing", "result": "nominated"},
        {"award": "bestScreenplay", "result": "nominated"}
    ],
    "wins" : 56,
    "nominations" : 86,
    "text" : "Won 2 Oscars. Another 56 wins and 86 nominations."
}

What query would we use in the Mongo shell to return all movies in the video.movieDetails collection that either won or were nominated for a best picture Oscar? You may assume that an award will appear in the oscars array only if the movie won or was nominated. You will probably want to create a little sample data for yourself in order to work this problem.

HINT: For this question we are looking for the simplest query that will work. This problem has a very straightforward solution, but you will need to extrapolate a little from some of the information presented in the “Reading Documents” lesson.

Answers:

db.movieDetails.find({"awards.oscars.award": "bestPicture"})

Challenge Problem: Updating Based on Multiple Criteria

This problem is provided as a supplementary learning opportunity. It is more challenging that the ordinary homework. It is ungraded. We do not ask you submit an answer.

Write an update command that will remove the “tomato.consensus” field for all documents matching the following criteria:

The number of imdb votes is less than 10,000.
The year for the movie is between 2010 and 2013 inclusive.
The tomato.consensus field is null.
How many documents required an update to eliminate a “tomato.consensus” field?

NOTE: There is a dump of the video database included in the handouts for the “Creating Documents” lesson. Use that data set to answer this question.

Answer

You can arrive at the answer here in a couple of different ways, either of which provide some good learning opportunities. The key is realizing that you need to report on the number of documents that actually required an update to remove the tomato.consensus field. You can do this either by ensuring that you filter for only those documents that do not contain a tomato.consensus field or by recognizing that only 13 documents were actually modified by your update.

Using the first approach, you can issue the following command.

1
2
3
4
5
db.movieDetails.updateMany({ year: {$gte: 2010, $lte: 2013},
                             "imdb.votes": {$lt: 10000},
                             $and: [{"tomato.consensus": {$exists: true} },
                                    {"tomato.consensus": null} ] },
                           { $unset: { "tomato.consensus": "" } });

In response, you will receive the following:

{ "acknowledged" : true, "matchedCount" : 13, "modifiedCount" : 13 }

Using the second approach, you can issue a simpler command, but one that is not precise about what needs to be updated.

1
2
3
4
db.movieDetails.updateMany({ year: {$gte: 2010, $lte: 2013},
                             "imdb.votes": {$lt: 10000},
                             "tomato.consensus": null },
                           { $unset: { "tomato.consensus": "" } });

In response, you will receive the following:

{ "acknowledged" : true, "matchedCount" : 204, "modifiedCount" : 13 }

Note that while the query portion of the update matches 204 documents, only 13 documents actually required an update.

Week 3 The Node.js Driver

basic： package.json 

1
2
3
4
5
6
7
8
9
10
11
{
    "name": "nodejsDriverFindAndCursors",
    "version": "0.1.0",
    "description": "Using the MongoDB driver to read documents using find",
    "main": "app.js",
    "dependencies": {
        "mongodb": "~2.1.3"
    },
    "author": "Shannon Bradshaw",
    "license": "0BSD"
}

find() and Cursors in the Node.js Driver

companies.json from https://www.crunchbase.com/

startup the database

1
2
chmod 777 /Users/allen/Documents/Code/workspace08/data/db
mongod --dbpath /Users/allen/Documents/Code/workspace08/data/db

import data

1
2
3
4
➜  findAndCursorsInNodeJSDriver mongoimport --db crunchbase -c companies companies.json

2018-01-29T08:53:45.899+1100	connected to: localhost
2018-01-29T08:53:48.550+1100	imported 18801 documents

verify

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
➜  ~ mongo localhost/crunchbase
MongoDB shell version v3.6.2
connecting to: mongodb://localhost:27017/crunchbase
MongoDB server version: 3.6.2
Server has startup warnings:
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten]
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten]
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server.
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2018-01-29T08:50:47.624+1100 I CONTROL  [initandlisten]
MongoDB Enterprise > show dbs
admin             0.000GB
config            0.000GB
create_lesson_db  0.000GB
crunchbase        0.033GB
local             0.000GB
m101              0.000GB
m102              0.000GB
pcat              0.000GB
test              0.000GB
video             0.001GB
MongoDB Enterprise > db
crunchbase
MongoDB Enterprise > show collections
companies
MongoDB Enterprise > db.companies.find().count()
18801

cursor.toArray() vs cursor.forEach(function)

toArray returns all documents, and then processes them
forEach do the iteration of returning and processing part of documents each time

cursor.toArray()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");

    var query = {"category_code": "biotech"};

    db.collection('companies').find(query).toArray(function(err, docs) {

        assert.equal(err, null);
        assert.notEqual(docs.length, 0);
        
        docs.forEach(function(doc) {
            console.log( doc.name + " is a " + doc.category_code + " company." );
        });
        
        db.close();
        
    });

});

cursor.forEach(function)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");

    var query = {"category_code": "biotech"};

    var cursor = db.collection('companies').find(query);

    cursor.forEach(
        function(doc) {
            console.log( doc.name + " is a " + doc.category_code + " company." );
        },
        function(err) {
            assert.equal(err, null);
            return db.close();
        }
    );

});

Projection in the Node.js Driver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");

    var query = {"category_code": "biotech"};
    // exclude or include
    var projection = {"name": 1, "category_code": 1, "_id": 0};

    var cursor = db.collection('companies').find(query);
    cursor.project(projection);

    cursor.forEach(
        function(doc) {
            console.log(doc.name + " is a " + doc.category_code + " company.");
            console.log(doc);
        },
        function(err) {
            assert.equal(err, null);
            return db.close();
        }
    );

});

Quiz

For the space marked with a TODO comment below, write ONE LINE OF CODE that will cause only the name and number_of_employees fields to be returned in query results. To simplify, please do not assign your projection document to a variable as we did in the lesson. Instead, just type the correct projection document directly into the call to the appropriate method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");

    var query = {"founded_year": 2010};

    var cursor = db.collection('companies').find(query);

    /* TODO: Write your line of code here. */

    cursor.forEach(
        function(doc) {
            console.log(doc.name + " has " + doc.number_of_employees + " employees.");
        },
        function(err) {
            assert.equal(err, null);
            return db.close();
        }
    );

});

Answers cursor.project({"name":1, "number_of_employees":1, "_id":0})

The CrunchBase Dataset

Please refer to fackbook.json and diggCompany.json

Query Operators in the Node.js

based on range of years and number of employees 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
var MongoClient = require('mongodb').MongoClient,
    commandLineArgs = require('command-line-args'), 
    assert = require('assert');


var options = commandLineOptions();

MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = queryDocument(options);
    var projection = {"_id": 1, "name": 1, "founded_year": 1,
                      "number_of_employees": 1, "crunchbase_url": 1};

    var cursor = db.collection('companies').find(query, projection);
    var numMatches = 0;

    cursor.forEach(
        function(doc) {
            numMatches = numMatches + 1;
            console.log( doc );
        },
        function(err) {
            assert.equal(err, null);
            console.log("Our query was:" + JSON.stringify(query));
            console.log("Matching documents: " + numMatches);
            return db.close();
        }
    );

});


function queryDocument(options) {

    console.log(options);
    
    var query = {
        "founded_year": {
            "$gte": options.firstYear,
            "$lte": options.lastYear
        }
    };

    if ("employees" in options) {
        query.number_of_employees = { "$gte": options.employees };
    }
        
    return query;
    
}


function commandLineOptions() {

    var cli = commandLineArgs([
        { name: "firstYear", alias: "f", type: Number },
        { name: "lastYear", alias: "l", type: Number },
        { name: "employees", alias: "e", type: Number }
    ]);
    
    var options = cli.parse()
    if ( !(("firstYear" in options) && ("lastYear" in options))) {
        console.log(cli.getUsage({
            title: "Usage",
            description: "The first two options below are required. The rest are optional."
        }));
        process.exit();
    }

    return options;
    
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
➜  queryOperatorsInNodeJSDriver node app.js

Usage

  The first two options below are required. The rest are optional.

Options

  -f, --firstYear number
  -l, --lastYear number
  -e, --employees number

➜  queryOperatorsInNodeJSDriver
➜  queryOperatorsInNodeJSDriver node app.js -f 2004 -l 2005 -e 2000
Successfully connected to MongoDB.
{ firstYear: 2004, lastYear: 2005, employees: 2000 }
{ _id: 52cdef7c4bab8bd675297d8e,
  name: 'Facebook',
  crunchbase_url: 'http://www.crunchbase.com/company/facebook',
  number_of_employees: 5299,
  founded_year: 2004 }
{ _id: 52cdef7c4bab8bd675297ea4,
  name: 'Webkinz',
  crunchbase_url: 'http://www.crunchbase.com/company/webkinz',
  number_of_employees: 8657,
  founded_year: 2005 }
Our query was:{"founded_year":{"$gte":2004,"$lte":2005},"number_of_employees":{"$gte":2000}}
Matching documents: 2

$regex in the Node.js Driver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
// app.js
var MongoClient = require('mongodb').MongoClient,
    commandLineArgs = require('command-line-args'), 
    assert = require('assert');


var options = commandLineOptions();


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = queryDocument(options);
    var projection = projectionDocument(options);

    var cursor = db.collection('companies').find(query);
    cursor.project(projection);
    
    var numMatches = 0;

    cursor.forEach(
        function(doc) {
            numMatches = numMatches + 1;
            console.log( doc );
        },
        function(err) {
            assert.equal(err, null);
            console.log("Our query was:" + JSON.stringify(query));
            console.log("Matching documents: " + numMatches);
            return db.close();
        }
    );

});


function queryDocument(options) {

    console.log(options);
    
    var query = {};

    if ("overview" in options) {
        query.overview = {"$regex": options.overview, "$options": "i"};
    }
    
    return query;
    
}


function projectionDocument(options) {

    var projection = {
        "_id": 0,
        "name": 1,
        "founded_year": 1,
        "overview": 1
    };

    return projection;
}


function commandLineOptions() {

    var cli = commandLineArgs([
        { name: "overview", alias: "o", type: String }
    ]);
    
    var options = cli.parse()
    if (Object.keys(options).length < 1) {
        console.log(cli.getUsage({
            title: "Usage",
            description: "You must supply at least one option. See below."
        }));
        process.exit();
    }

    return options;
    
}
1
2
3
4
5
6
7
8
➜  regexOperatorInNodeJSDriver node app.js -o "facebook.*chatting"
Successfully connected to MongoDB.
{ overview: 'facebook.*chatting' }
{ name: 'VisualSage',
  founded_year: 2008,
  overview: '<p>VisualSage produces a <a href="http://www.crunchbase.com/company/facebook" title="Facebook">Facebook</a> desktop application that visualizes and provides functionality for your Facebook network. The application is built upon Microsoft&#8217;s new technology WPF. </p>\n\n<p>VisualSage provides easy access to many Facebook&#8217;s primary functions, including Facebook notification, message, poke, profile, and photo browsing. The application provides chatting functionality, which allows you to instant communicate with your friends on Facebook. You can also arrange and optimize your network by grouping your friends to subnets.</p>\n\n<p>VisualSage is a software division of Minesage.</p>' }
Our query was:{"overview":{"$regex":"facebook.*chatting","$options":"i"}}
Matching documents: 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
// app-milestones.js
var MongoClient = require('mongodb').MongoClient,
    commandLineArgs = require('command-line-args'), 
    assert = require('assert');


var options = commandLineOptions();


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = queryDocument(options);
    var projection = projectionDocument(options);

    var cursor = db.collection('companies').find(query);
    cursor.project(projection);
    
    var numMatches = 0;

    cursor.forEach(
        function(doc) {
            numMatches = numMatches + 1;
            console.log( doc );
        },
        function(err) {
            assert.equal(err, null);
            console.log("Our query was:" + JSON.stringify(query));
            console.log("Matching documents: " + numMatches);
            return db.close();
        }
    );

});


function queryDocument(options) {

    console.log(options);
    
    var query = {};

    if ("overview" in options) {
        query.overview = {"$regex": options.overview, "$options": "i"};
    }

    if ("milestones" in options) {
        query["milestones.source_description"] =
            {"$regex": options.milestones, "$options": "i"};
    }

    return query;
    
}


function projectionDocument(options) {

    var projection = {
        "_id": 0,
        "name": 1,
        "founded_year": 1
    };

    if ("overview" in options) {
        projection.overview = 1;
    }

    if ("milestones" in options) {
        projection["milestones.source_description"] = 1;
    }

    return projection;
}


function commandLineOptions() {

    var cli = commandLineArgs([
        { name: "overview", alias: "o", type: String },
        { name: "milestones", alias: "m", type: String }
    ]);
    
    var options = cli.parse()
    if (Object.keys(options).length < 1) {
        console.log(cli.getUsage({
            title: "Usage",
            description: "You must supply at least one option. See below."
        }));
        process.exit();
    }

    return options;
    
}
1
2
3
4
5
➜  regexOperatorInNodeJSDriver node app-milestones.js -o "facebook" -m "chatting"
Successfully connected to MongoDB.
{ overview: 'facebook', milestones: 'chatting' }
Our query was:{"overview":{"$regex":"facebook","$options":"i"},"milestones.source_description":{"$regex":"chatting","$options":"i"}}
Matching documents: 0

Dot Notation in the Node.js Driver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
//app.js
var MongoClient = require('mongodb').MongoClient,
    commandLineArgs = require('command-line-args'), 
    assert = require('assert');


var options = commandLineOptions();


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = queryDocument(options);
    var projection = {"_id": 0, "name": 1, "founded_year": 1,
                      "number_of_employees": 1, "ipo.valuation_amount": 1};

    var cursor = db.collection('companies').find(query, projection);
    var numMatches = 0;

    cursor.forEach(
        function(doc) {
            numMatches = numMatches + 1;
            console.log( doc );
        },
        function(err) {
            assert.equal(err, null);
            console.log("Our query was:" + JSON.stringify(query));
            console.log("Matching documents: " + numMatches);
            return db.close();
        }
    );

});


function queryDocument(options) {

    console.log(options);
    
    var query = {
        "founded_year": {
            "$gte": options.firstYear,
            "$lte": options.lastYear
        }
    };

    if ("employees" in options) {
        query.number_of_employees = { "$gte": options.employees };
    }
    
    if ("ipo" in options) {
        if (options.ipo == "yes") {
            query["ipo.valuation_amount"] = {"$exists": true, "$ne": null};
        } else if (options.ipo == "no") {
            query["ipo.valuation_amount"] = null;
        }               
    }
    
    return query;
    
}


function commandLineOptions() {

    var cli = commandLineArgs([
        { name: "firstYear", alias: "f", type: Number },
        { name: "lastYear", alias: "l", type: Number },
        { name: "employees", alias: "e", type: Number },
        { name: "ipo", alias: "i", type: String }
    ]);
    
    var options = cli.parse()
    if ( !(("firstYear" in options) && ("lastYear" in options))) {
        console.log(cli.getUsage({
            title: "Usage",
            description: "The first two options below are required. The rest are optional."
        }));
        process.exit();
    }

    return options;
    
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
➜  dotNotationInNodeJSDriver node app.js
Usage

  The first two options below are required. The rest are optional.

Options
  -f, --firstYear number
  -l, --lastYear number
  -e, --employees number
  -i, --ipo string➜  dotNotationInNodeJSDriver
➜  dotNotationInNodeJSDriver node app.js -f 2004 -l 2008 -e 2000 -i yesSuccessfully connected to MongoDB.{ firstYear: 2004, lastYear: 2008, employees: 2000, ipo: 'yes' }{ name: 'Facebook',  number_of_employees: 5299,
  founded_year: 2004,
  ipo: { valuation_amount: 104000000000 } }
{ name: 'Groupon',  number_of_employees: 10000,  founded_year: 2008,  ipo: { valuation_amount: 12800000000 } }
Our query was:{"founded_year":{"$gte":2004,"$lte":2008},"number_of_employees":{"$gte":2000},"ipo.valuation_amount":{"$exists":true,"$ne":null}}
Matching documents: 2

Dot Notation on Embedded Documents in Arrays

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
// app-conuntries.js
var MongoClient = require('mongodb').MongoClient,
    commandLineArgs = require('command-line-args'), 
    assert = require('assert');


var options = commandLineOptions();


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = queryDocument(options);
    var projection = {"_id": 0,
                      "name": 1,
                      "offices.country_code": 1,
                      "ipo.valuation_amount": 1};

    var cursor = db.collection('companies').find(query, projection);
    var numMatches = 0;

    cursor.forEach(
        function(doc) {
            numMatches = numMatches + 1;
            console.log( doc );
        },
        function(err) {
            assert.equal(err, null);
            console.log("Our query was:" + JSON.stringify(query));
            console.log("Matching documents: " + numMatches);
            return db.close();
        }
    );

});


function queryDocument(options) {

    console.log(options);
    
    var query = {
        "founded_year": {
            "$gte": options.firstYear,
            "$lte": options.lastYear
        }
    };

    if ("employees" in options) {
        query.number_of_employees = { "$gte": options.employees };
    }
    
    if ("ipo" in options) {
        if (options.ipo == "yes") {
            query["ipo.valuation_amount"] = {"$exists": true, "$ne": null};
        } else if (options.ipo == "no") {
            query["ipo.valuation_amount"] = null;
        }               
    }

    if ("country" in options) {
        query["offices.country_code"] = options.country;
    }
    
    return query;
    
}


function commandLineOptions() {

    var cli = commandLineArgs([
        { name: "firstYear", alias: "f", type: Number },
        { name: "lastYear", alias: "l", type: Number },
        { name: "employees", alias: "e", type: Number },
        { name: "ipo", alias: "i", type: String },
        { name: "country", alias: "c", type: String }
    ]);
    
    var options = cli.parse()
    if ( !(("firstYear" in options) && ("lastYear" in options))) {
        console.log(cli.getUsage({
            title: "Usage",
            description: "The first two options below are required. The rest are optional."
        }));
        process.exit();
    }

    return options;
    
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
➜  dotNotationInNodeJSDriver node app-countries.js
Usage

  The first two options below are required. The rest are optional.

Options
  -f, --firstYear number
  -l, --lastYear number
  -e, --employees number
  -i, --ipo string  
  -c, --country string
➜  dotNotationInNodeJSDriver
➜  dotNotationInNodeJSDriver node app-countries.js -f 2004 -l 2008 -e 200 -i yes -c America
Successfully connected to MongoDB.
{ firstYear: 2004,
  lastYear: 2008,  employees: 200,  ipo: 'yes',
  country: 'America' }
Our query was:{"founded_year":{"$gte":2004,"$lte":2008},"number_of_employees":{"$gte":200},"ipo.valuation_amount":{"$exists":true,"$ne":null},"offices.country_code":"America"}
Matching documents: 0
➜  dotNotationInNodeJSDriver node app-countries.js -f 2004 -l 2008 -e 200 -i yes -c IRLSuccessfully connected to MongoDB.{ firstYear: 2004,  lastYear: 2008,  employees: 200,  ipo: 'yes',
  country: 'IRL' }
{ name: 'Facebook',
  offices:   [ { country_code: 'USA' },     { country_code: 'IRL' },     { country_code: 'USA' } ],
  ipo: { valuation_amount: 104000000000 } }
Our query was:{"founded_year":{"$gte":2004,"$lte":2008},"number_of_employees":{"$gte":200},"ipo.valuation_amount":{"$exists":true,"$ne":null},"offices.country_code":"IRL"}
Matching documents: 1

Sort, Skip, and Limit in the Node.js Driver

paging uses these things

mongodb always does this sequence - sort first, then skip , finally limit

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
// app-sort.js
var MongoClient = require('mongodb').MongoClient,
    commandLineArgs = require('command-line-args'), 
    assert = require('assert');


var options = commandLineOptions();


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = queryDocument(options);
    var projection = {"_id": 0, "name": 1, "founded_year": 1,
                      "number_of_employees": 1};

    var cursor = db.collection('companies').find(query);
    cursor.project(projection);
    //cursor.sort({founded_year: -1});
    cursor.sort([["founded_year", 1], ["number_of_employees", -1]]);
        
    var numMatches = 0;

    cursor.forEach(
        function(doc) {
            numMatches = numMatches + 1;
            console.log(doc.name + "\n\tfounded " + doc.founded_year +
                        "\n\t" + doc.number_of_employees + " employees"); 
        },
        function(err) {
            assert.equal(err, null);
            console.log("Our query was:" + JSON.stringify(query));
            console.log("Matching documents: " + numMatches);
            return db.close();
        }
    );

});


function queryDocument(options) {

    var query = {
        "founded_year": {
            "$gte": options.firstYear,
            "$lte": options.lastYear
        }
    };

    if ("employees" in options) {
        query.number_of_employees = { "$gte": options.employees };
    }
    
    return query;
    
}


function commandLineOptions() {

    var cli = commandLineArgs([
        { name: "firstYear", alias: "f", type: Number },
        { name: "lastYear", alias: "l", type: Number },
        { name: "employees", alias: "e", type: Number }
    ]);
    
    var options = cli.parse()
    if ( !(("firstYear" in options) && ("lastYear" in options))) {
        console.log(cli.getUsage({
            title: "Usage",
            description: "The first two options below are required. The rest are optional."
        }));
        process.exit();
    }

    return options;
    
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
➜  sortSkipLimitInNodeJSDriver node app-sort.js
Usage

  The first two options below are required. The rest are optional.Options  -f, --firstYear number  -l, --lastYear number  -e, --employees number

➜  sortSkipLimitInNodeJSDriver
➜  sortSkipLimitInNodeJSDriver node app-sort.js -f 2004 -l 2009 -e 1500
Successfully connected to MongoDB.Facebook        founded 2004        5299 employees
ReachLocal
        founded 2004
        1700 employeesUCWeb        founded 2004
        1700 employees
Webkinz
        founded 2005
        8657 employees
Zoho
        founded 2005
        1600 employees
Gemalto        founded 2006        12000 employeesSpotify
        founded 2006
        5000 employees
Veeam Software
        founded 2006
        1500 employees
Thomson Reuters
        founded 2008
        50000 employees
Thomson Reuters
        founded 2008
        50000 employees
Groupon
        founded 2008
        10000 employees
WJT Global Solutions
        founded 2008
        2000 employees
Our query was:{"founded_year":{"$gte":2004,"$lte":2009},"number_of_employees":{"$gte":1500}}
Matching documents: 12
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
// app-sortSkipLimit.js
var MongoClient = require('mongodb').MongoClient,
    commandLineArgs = require('command-line-args'), 
    assert = require('assert');


var options = commandLineOptions();


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = queryDocument(options);
    var projection = {"_id": 0, "name": 1, "founded_year": 1,
                      "number_of_employees": 1};

    var cursor = db.collection('companies').find(query);
    cursor.project(projection);
    cursor.limit(options.limit);
    cursor.skip(options.skip);
    cursor.sort([["founded_year", 1], ["number_of_employees", -1]]);
        
    var numMatches = 0;

    cursor.forEach(
        function(doc) {
            numMatches = numMatches + 1;
            console.log(doc.name + "\n\tfounded " + doc.founded_year +
                        "\n\t" + doc.number_of_employees + " employees"); 
        },
        function(err) {
            assert.equal(err, null);
            console.log("Our query was:" + JSON.stringify(query));
            console.log("Documents displayed: " + numMatches);
            return db.close();
        }
    );

});


function queryDocument(options) {

    console.log(options);
    
    var query = {
        "founded_year": {
            "$gte": options.firstYear,
            "$lte": options.lastYear
        }
    };

    if ("employees" in options) {
        query.number_of_employees = { "$gte": options.employees };
    }
    
    return query;
    
}


function commandLineOptions() {

    var cli = commandLineArgs([
        { name: "firstYear", alias: "f", type: Number },
        { name: "lastYear", alias: "l", type: Number },
        { name: "employees", alias: "e", type: Number },
        { name: "skip", type: Number, defaultValue: 0 },
        { name: "limit", type: Number, defaultValue: 20000 }
    ]);
    
    var options = cli.parse()
    if ( !(("firstYear" in options) && ("lastYear" in options))) {
        console.log(cli.getUsage({
            title: "Usage",
            description: "The first two options below are required. The rest are optional."
        }));
        process.exit();
    }

    return options;
    
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
➜  sortSkipLimitInNodeJSDriver node app-sortSkipLimit.js

Usage

  The first two options below are required. The rest are optional.

Options

  -f, --firstYear number
  -l, --lastYear number
  -e, --employees number
  --skip number
  --limit number

➜  sortSkipLimitInNodeJSDriver
➜  sortSkipLimitInNodeJSDriver node app-sortSkipLimit.js -f 2004 -l 2009 -e 1500 --skip 3 --limit 2
Successfully connected to MongoDB.
{ skip: 3,
  limit: 2,
  firstYear: 2004,
  lastYear: 2009,
  employees: 1500 }
Webkinz
        founded 2005
        8657 employees
Zoho
        founded 2005
        1600 employees
Our query was:{"founded_year":{"$gte":2004,"$lte":2009},"number_of_employees":{"$gte":1500}}
Documents displayed: 2

Quiz

Suppose you have a MongoDB collection called school.grades that is composed solely of these 20 documents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{"_id": 1, "student": "Mary", "grade": 45, "assignment": "homework"}
{"_id": 2, "student": "Alice", "grade": 48, "assignment": "homework"}
{"_id": 3, "student": "Fiona", "grade": 16, "assignment": "quiz"}
{"_id": 4, "student": "Wendy", "grade": 12, "assignment": "homework"}
{"_id": 5, "student": "Samantha", "grade": 82, "assignment": "homework"}
{"_id": 6, "student": "Fay", "grade": 89, "assignment": "quiz"}
{"_id": 7, "student": "Katherine", "grade": 77, "assignment": "quiz"}
{"_id": 8, "student": "Stacy", "grade": 73, "assignment": "quiz"}
{"_id": 9, "student": "Sam", "grade": 61, "assignment": "homework"}
{"_id": 10, "student": "Tom", "grade": 67, "assignment": "exam"}
{"_id": 11, "student": "Ted", "grade": 52, "assignment": "exam"}
{"_id": 12, "student": "Bill", "grade": 59, "assignment": "exam"}
{"_id": 13, "student": "Bob", "grade": 37, "assignment": "exam"}
{"_id": 14, "student": "Seamus", "grade": 33, "assignment": "exam"}
{"_id": 15, "student": "Kim", "grade": 28, "assignment": "quiz"}
{"_id": 16, "student": "Sacha", "grade": 23, "assignment": "quiz"}
{"_id": 17, "student": "David", "grade": 18, "assignment": "exam"}
{"_id": 18, "student": "Steve", "grade": 14, "assignment": "homework"}
{"_id": 19, "student": "Burt", "grade": 90, "assignment": "quiz"}
{"_id": 20, "student": "Stan", "grade": 92, "assignment": "exam"}

Assuming the variable db holds a connection to the school database in the following code snippet.

1
2
3
4
var cursor = db.collection("grades").find({});
cursor.sort({"grade": -1});
cursor.skip(4);
cursor.limit(2);

Which student’s documents will be returned as part of a subsequent call to toArray()?
Katherine
Stacy

insertOne() and insertMany() in the Node.js Driver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Twitter Developer Documentation is here:
https://dev.twitter.com/overview/documentation

Documentation on the streaming API is here:
https://dev.twitter.com/streaming/overview

Documentation on the REST API is here:
https://dev.twitter.com/rest/public

To use any of the Twitter APIs you will need access tokens. The simplest means of acquiring access tokens is described here:
https://dev.twitter.com/oauth/overview/application-owner-access-tokens

The Twitter API client library for Node.js that I used in the lessons is found here:
https://www.npmjs.com/package/twitter

Note that you can place your access tokens in a separate file (.env) and use the following package to load them.
https://www.npmjs.com/package/dotenv

The package.json file for this lesson contains the dependencies for the twitter and dotenv packages. See the applications in the handouts for examples of how to use. The documentation for the twitter and nodenv packages provides details on setting up your tokens as environment variables, loading them, and using them to access the twitter API.

package.json 

1
2
3
4
5
6
7
8
9
10
11
12
13
{
    "name": "social",
    "version": "0.1.0",
    "description": "Demonstrating promises",
    "main": "app.js",
    "dependencies": {
        "dotenv": "~1.2.0",
        "mongodb": "~2.1.3",
        "twitter": "~1.2.5"
    },
    "author": "Shannon Bradshaw",
    "license": "0BSD"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
//app.js
var MongoClient = require('mongodb').MongoClient,
    Twitter = require('twitter'),
    assert = require('assert');

require('dotenv').load();
var twitterClient = new Twitter({
    consumer_key: process.env.TWITTER_CONSUMER_KEY,
    consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
    access_token_key: process.env.TWITTER_ACCESS_TOKEN_KEY,
    access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
});


MongoClient.connect('mongodb://localhost:27017/social', function(err, db) {

    assert.equal(null, err);
    console.log("Successfully connected to MongoDB.");

    twitterClient.stream('statuses/filter', {track: "marvel"}, function(stream) {
        stream.on('data', function(status) {
            console.log(status.text);
            db.collection("statuses").insertOne(status, function(err, res) {
                console.log("Inserted document with _id: " + res.insertedId + "\n");
            });
        });
 
        stream.on('error', function(error) {
            throw error;
        });
    });

});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
//app-insertOne.js
var MongoClient = require('mongodb').MongoClient,
    Twitter = require('twitter'),
    assert = require('assert');

require('dotenv').load();
var twitterClient = new Twitter({
    consumer_key: process.env.TWITTER_CONSUMER_KEY,
    consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
    access_token_key: process.env.TWITTER_ACCESS_TOKEN_KEY,
    access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
});


MongoClient.connect('mongodb://localhost:27017/social', function(err, db) {

    assert.equal(null, err);
    console.log("Successfully connected to MongoDB.");

    twitterClient.stream('statuses/filter', {track: "marvel"}, function(stream) {
        stream.on('data', function(status) {
            console.log(status.text);
            db.collection("statuses").insertOne(status, function(err, res) {
                console.log("Inserted document with _id: " + res.insertedId + "\n");
            });
        });
 
        stream.on('error', function(error) {
            throw error;
        });
    });

});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
//app-insertMany.js
var MongoClient = require('mongodb').MongoClient,
    Twitter = require('twitter'),
    assert = require('assert');


require('dotenv').load();
var client = new Twitter({
    consumer_key: process.env.TWITTER_CONSUMER_KEY,
    consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
    access_token_key: process.env.TWITTER_ACCESS_TOKEN_KEY,
    access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
});


MongoClient.connect('mongodb://localhost:27017/social', function(err, db) {

    assert.equal(null, err);
    console.log("Successfully connected to MongoDB.");

    var screenNames = ["Marvel", "DCComics", "TheRealStanLee"];
    var done = 0;

    screenNames.forEach(function(name) {

        var cursor = db.collection("statuses").find({"user.screen_name": name});
        cursor.sort({ "id": -1 });
        cursor.limit(1);

        cursor.toArray(function(err, docs) {
            assert.equal(err, null);
            
            var params;
            if (docs.length == 1) {
                params = { "screen_name": name, "since_id": docs[0].id, "count": 10 };
            } else {
                params = { "screen_name": name, "count": 10 };
            }
            
            client.get('statuses/user_timeline', params, function(err, statuses, response) {
                
                assert.equal(err, null);
                
                db.collection("statuses").insertMany(statuses, function(err, res) {

                    console.log(res);
                    
                    done += 1;
                    if (done == screenNames.length) {
                        db.close();
                    }
                    
                });
            });
        })
    });
});

deleteOne() and deleteMany() in the Node.js Driver

package.json 

1
2
3
4
5
6
7
8
9
10
11
{
    "name": "nodejsDriverFindAndCursors",
    "version": "0.1.0",
    "description": "Using the MongoDB driver to read documents using find",
    "main": "app.js",
    "dependencies": {
        "mongodb": "~2.1.3"
    },
    "author": "Shannon Bradshaw",
    "license": "0BSD"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
app-deleteOne.js
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = {"permalink": {"$exists": true, "$ne": null}};
    var projection = {"permalink": 1, "updated_at": 1};

    var cursor = db.collection('companies').find(query);
    cursor.project(projection);
    cursor.sort({"permalink": 1})

    var numToRemove = 0;

    var previous = { "permalink": "", "updated_at": "" };
    cursor.forEach(
        function(doc) {

            if ( (doc.permalink == previous.permalink) && (doc.updated_at == previous.updated_at) ) {
                console.log(doc.permalink);

                numToRemove = numToRemove + 1;

                var filter = {"_id": doc._id};

                db.collection('companies').deleteOne(filter, function(err, res) {

                    assert.equal(err, null);
                    console.log(res.result);

                });

            }
            
            previous = doc;
            
        },
        function(err) {

            assert.equal(err, null);

        }
    );

});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
app-deleteMany.js
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');


MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {

    assert.equal(err, null);
    console.log("Successfully connected to MongoDB.");
    
    var query = {"permalink": {$exists: true, $ne: null}};
    var projection = {"permalink": 1, "updated_at": 1};

    var cursor = db.collection('companies').find(query);
    cursor.project(projection);
    cursor.sort({"permalink": 1})

    var markedForRemoval = [];

    var previous = { "permalink": "", "updated_at": "" };
    cursor.forEach(
        function(doc) {

            if ( (doc.permalink == previous.permalink) && (doc.updated_at == previous.updated_at) ) {
                markedForRemoval.push(doc._id);
            }

            previous = doc;
        },
        function(err) {

            assert.equal(err, null);

            var filter = {"_id": {"$in": markedForRemoval}};

            db.collection("companies").deleteMany(filter, function(err, res) {

                console.log(res.result);
                console.log(markedForRemoval.length + " documents removed.");

                return db.close();
            });
        }
    );

});

Homework: 3.1

When using find() in the Node.js driver, which of the following best describes when the driver will send a query to MongoDB?

When we call a cursor method passing a callback function to process query results

Homework: 3.2

Suppose you have a MongoDB collection called school.grades that is composed solely of these 20 documents: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{"_id": 1, "student": "Mary", "grade": 45, "assignment": "homework"}
{"_id": 2, "student": "Alice", "grade": 48, "assignment": "homework"}
{"_id": 3, "student": "Fiona", "grade": 16, "assignment": "quiz"}
{"_id": 4, "student": "Wendy", "grade": 12, "assignment": "homework"}
{"_id": 5, "student": "Samantha", "grade": 82, "assignment": "homework"}
{"_id": 6, "student": "Fay", "grade": 89, "assignment": "quiz"}
{"_id": 7, "student": "Katherine", "grade": 77, "assignment": "quiz"}
{"_id": 8, "student": "Stacy", "grade": 73, "assignment": "quiz"}
{"_id": 9, "student": "Sam", "grade": 61, "assignment": "homework"}
{"_id": 10, "student": "Tom", "grade": 67, "assignment": "exam"}
{"_id": 11, "student": "Ted", "grade": 52, "assignment": "exam"}
{"_id": 12, "student": "Bill", "grade": 59, "assignment": "exam"}
{"_id": 13, "student": "Bob", "grade": 37, "assignment": "exam"}
{"_id": 14, "student": "Seamus", "grade": 33, "assignment": "exam"}
{"_id": 15, "student": "Kim", "grade": 28, "assignment": "quiz"}
{"_id": 16, "student": "Sacha", "grade": 23, "assignment": "quiz"}
{"_id": 17, "student": "David", "grade": 5, "assignment": "exam"}
{"_id": 18, "student": "Steve", "grade": 9, "assignment": "homework"}
{"_id": 19, "student": "Burt", "grade": 90, "assignment": "quiz"}
{"_id": 20, "student": "Stan", "grade": 92, "assignment": "exam"}

Assuming the variable db holds a connection to the school database in the following code snippet.

1
2
3
4
var cursor = db.collection("grades").find({});
cursor.skip(6);
cursor.limit(2);
cursor.sort({"grade": 1});

Which student’s documents will be returned as part of a subsequent call to toArray()?

Seamus, Bob

Homework: 3.3

This application depends on the companies.json dataset distributed as a handout with the findAndCursorsInNodeJSDriver lesson. You must first import that collection. Please ensure you are working with an unmodified version of the collection before beginning this exercise.

To import a fresh version of the companies.json data, please type the following:

mongoimport --drop -d crunchbase -c companies companies.json

If you have already mongoimported this data you will first need to drop the crunchbase database in the Mongo shell. Do that by typing the following two commands, one at a time, in the Mongo shell:

use crunchbase
db.dropDatabase()

The code in the attached handout is complete with the exception of the queryDocument() function. As in the lessons, the queryDocument() function builds an object that will be passed to find() to match a set of documents from the crunchbase.companies collection.

For this assignment, please complete the queryDocument() function as described in the TODO comments you will find in that function.

Once complete, run this application by typing:

node buildingQueryDocuments.js

When you are convinced you have completed the application correctly, please enter the average number of employees per company reported in the output. Enter only the number reported. It should be three numeric digits.

As a check that you have completed the exercise correctly, the total number of unique companies reported by the application should equal 42.

If the grading system does not accept the first solution you enter, please do not make further attempts to have your solution graded without seeking some help in the discussion forum.

Enter answer here:

169

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');


var allOptions = [
    {
        firstYear: 2002,
        lastYear: 2016,
        city: "Palo Alto"
    },
    {
        lastYear: 2010,
        city: "New York"
    },
    {
        city: "London"
    }
];

var numQueriesFinished = 0;
var companiesSeen = {};

for (var i=0; i<allOptions.length; i++) {
    var query = queryDocument(allOptions[i]);
    queryMongoDB(query, i);
}


function queryMongoDB(query, queryNum) {

    MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {
        
        assert.equal(err, null);
        console.log("Successfully connected to MongoDB for query: " + queryNum);
        
        var cursor = db.collection('companies').find(query);
        
        var numMatches = 0;
        
        cursor.forEach(
            function(doc) {
                numMatches = numMatches + 1;
                if (doc.permalink in companiesSeen) return;
                companiesSeen[doc.permalink] = doc;
            },
            function(err) {
                assert.equal(err, null);
                console.log("Query " + queryNum + " was:" + JSON.stringify(query));
                console.log("Matching documents: " + numMatches);
                numQueriesFinished = numQueriesFinished + 1;
                if (numQueriesFinished == allOptions.length) {
                    report();
                }
                return db.close();
            }
        );
        
    });
    
}


function queryDocument(options) {

    console.log(options);
    
    var query = {
        "tag_list": {"$regex": 'social-networking' , "$options": "i"}  /* TODO: Complete this statement to match the regular expression "social-networking" */        
    };

    if (("firstYear" in options) && ("lastYear" in options)) {
        /* 
           TODO: Write one line of code to ensure that if both firstYear and lastYear 
           appear in the options object, we will match documents that have a value for 
           the "founded_year" field of companies documents in the correct range. 
        */
        query.founded_year = { "$gte": options.firstYear, "$lte": options.lastYear };
    } else if ("firstYear" in options) {
        query.founded_year = { "$gte": options.firstYear };
    } else if ("lastYear" in options) {
        query.founded_year = { "$lte": options.lastYear };
    }

    if ("city" in options) {
        /* 
           TODO: Write one line of code to ensure that we do an equality match on the 
           "offices.city" field. The "offices" field stores an array in which each element 
           is a nested document containing fields that describe a corporate office. Each office
           document contains a "city" field. A company may have multiple corporate offices. 
        */
        // {"offices.city": "Menlo Park"}
        query["offices.city"] = options.city
    }
        
    return query;
    
}


function report(options) {
    var totalEmployees = 0;
    for (key in companiesSeen) {
        totalEmployees = totalEmployees + companiesSeen[key].number_of_employees;
    }

    var companiesList = Object.keys(companiesSeen).sort();
    console.log("Companies found: " + companiesList);
    console.log("Total employees in companies identified: " + totalEmployees);
    console.log("Total unique companies: " + companiesList.length);
    console.log("Average number of employees per company: " + Math.floor(totalEmployees / companiesList.length));
}

Homework: 3.4

In completing this exercise, you will find the “Logical Operators” lesson from Chapter 2 of this course helpful as a refresher on the $or operator.

This application depends on the companies.json dataset distributed as a handout with the “find() and Cursors in the Node.js Driver” lesson. You must first import that collection. Please ensure you are working with an unmodified version of the collection before beginning this exercise.

To import a fresh version of the companies.json data, please type the following:

mongoimport --drop -d crunchbase -c companies companies.json

If you have already mongoimported this data you will first need to drop the crunchbase database in the Mongo shell. Do that by typing the following two commands, one at a time, in the Mongo shell:

use crunchbase

db.dropDatabase()

The code attached is complete with the exception of the queryDocument() function. As in the lessons, the queryDocument() function builds an object that will be passed to find() to match a set of documents from the crunchbase.companies collection.

For this assignment, please complete the queryDocument() function as described in the TODO comments you will find in that function.

Once complete, run this application by typing:

node overviewOrTags.js

When you are convinced you have completed the application correctly, please enter the average number of employees per company reported in the output. Enter only the number reported. It should be two numeric digits.

As a check that you have completed the exercise correctly, the total number of unique companies reported by the application should equal 194.

If the grading system does not accept the first solution you enter, please do not make further attempts to have your solution graded without seeking some help in the discussion forum.

Enter answer here:

48

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
var MongoClient = require('mongodb').MongoClient,
    assert = require('assert');

var allOptions = [
    {
        overview: "wiki",
    },
    {
        milestones: "CMO"
    }
];

var numQueriesFinished = 0;
var companiesSeen = {};

for (var i=0; i<allOptions.length; i++) {
    var query = queryDocument(allOptions[i]);
    queryMongoDB(query, i);
}

function queryMongoDB(query, queryNum) {

    MongoClient.connect('mongodb://localhost:27017/crunchbase', function(err, db) {
        
        assert.equal(err, null);
        console.log("Successfully connected to MongoDB for query: " + queryNum);
        
        var cursor = db.collection('companies').find(query);
        
        var numMatches = 0;
        
        cursor.forEach(
            function(doc) {
                numMatches = numMatches + 1;
                if (doc.permalink in companiesSeen) return;
                companiesSeen[doc.permalink] = doc;
            },
            function(err) {
                assert.equal(err, null);
                console.log("Query " + queryNum + " was:" + JSON.stringify(query));
                console.log("Matching documents: " + numMatches);
                numQueriesFinished = numQueriesFinished + 1;
                if (numQueriesFinished == allOptions.length) {
                    report();
                }
                return db.close();
            }
        );
        
    });
    
}

function queryDocument(options) {

    var query = {};

    if ("overview" in options) {
        /*
           TODO: Write an assignment statement to ensure that if "overview" appears in the 
           options object, we will match documents that have the value of options.overview 
           in either the "overview" field or "tag_list" field of companies documents.

           You will need to use the $or operator to do this. As a hint, "$or" should be the
           name of the field you create in the query object.

           As with the example for options.milestones below, please ensure your regular
           expression matches are case insensitive.

           I urge you to test your query in the Mongo shell first and adapt it to fit
           the syntax for constructing query documents in this application.
        */
        query["$or"] = [ {"overview": {"$regex": options.overview, "$options": "i"}}, { "tag_list": {"$regex": options.overview, "$options": "i"}} ]
    }

    if ("milestones" in options) {
        query["milestones.source_description"] =
            {"$regex": options.milestones, "$options": "i"};
    }

    return query;
    
}


function report(options) {
    var totalEmployees = 0;
    for (key in companiesSeen) {
        totalEmployees = totalEmployees + companiesSeen[key].number_of_employees;
    }

    var companiesList = Object.keys(companiesSeen).sort();
    console.log("Companies found: " + companiesList);
    console.log("Total employees in companies identified: " + totalEmployees);
    console.log("Total unique companies: " + companiesList.length);
    console.log("Average number of employees per company: " + Math.floor(totalEmployees / companiesList.length));
}

Week 4: Schema Design

MongoDB Schema Design

Application Driven Schema

Relational database schema design conforms 3rd normal. In mongodb, it could be different. There are some problems we should think about such as which data is used together and not read or written all the time.

The key is to get the data access pattern

When design mongodb schema, these things should be known,

  • rich documents
  • pre jon/embed data
  • no mongo joins
  • no constraints
  • atomic operation
  • no declared schema

Quiz

What’s the single most important factor in designing your application schema within MongoDB?

  • making the design extensible
  • making it easy to read by a human
  • ✔︎matching the data access patterns ofyour application
  • keeping the data in third normal form

Relational Normalization

This diagram depicts a database design violates 3rd normalization.

Goals of Nomalization

  • free the database of modification anomalies (will avoid it in MongoDB)
  • minimize redesign when extending (very flexiable in MongoDB, can add attributes and values without changing existing the database)
  • avoid bias toward any particular access pattern (not be worried about in MongoDB)

Modelinig a Blog in Documents

Posts

1
2
3
4
5
6
7
8
9
{
	_id : " ",
	title : " ",
	author : " ",
	content: " ",
	comments: [{author:"", content:"", date:""}]
	tags: ["", ""]
	date: date()
}

Author

1
2
3
4
5
6
{
	_id: "",
	name: "",
	email: "",
	password: ""
}

Quiz

Given the document schema that we proposed for the blog, how many collections would need to be accessed to display a blog post with its comments and tags?

1

Answer

One document is all that is needed

That’s not to say that you couldn’t break it up into multiple documents, or that there are no advantages in doing so in some circumstances, but there will also be disadvantages; we’ll be talking about this more in other lessons on schema design and data modeling. But only one document is needed.

Living Without Constraints

In traditional relational database design we use foreign key constrain to make sure data consistent when data is updated.

In Mongodb，we don’t have foreign key constraints and use embeding document to do the same thing. We call it pre-join.

Quiz

What does Living Without Constraints refer to?

Keeping your data consistent even though MongoDB lacks foreign key constraints

Living Without Transactions

MongoDB only supports atomic operations. It relies on the pre-join features.

  1. restructure to only a single document
  2. implement lock, critical section, test set
  3. tolerate, which means accepting someone gets the information several seconds earlier than others

Which of the following operations operate atomically within a single document? Check all that apply.

  • ✔︎Update
  • ✔︎findAndModity
  • ✔︎$addToSet (within an update)
  • ✔︎$push within an update

One to One Relations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

* design relation:

Employee: Resume   - 1 : 1

Building: Floor Plan - 1 : 1
 
Patient: Medical History  - 1 : 1

Employee: Resume model it in several ways


Employee {  					Resume {
id : 20,						id : 30,
name : "Andrew",				jobs : ,
resume : 30,					education : ,
								employee : " "
}								}
								
1.	separate, but embed each other's id

Employee  { 					Resume {
_id : 20						_id : 30
name : "Andrew"				jobs : []
resume : 30			
}								employee : 20
								}					
2. embed the whole document to another one

Employee   {					

_id : 20						

name : "Andrew"				

resume : {
	_id : 30
	jobs : []
	Education : []
	employee : 20
}	

...
}											

* comparison
	
If separated, there will be inconsistency(atomic operation) if you are not careful.						
If embeded, there will be working set problem because it may be vary large.


* when designed, consider:

1. the size of the document (>16MB) or the future growth of size

2. consistency 

3. convenience


**Quiz**

What's a good reason you might want to keep two documents that are related to each other one-to-one in separate collections? Check all that apply.

- To enforce foreign key constraints
- Because the combined size of the documents would be larger than 16MB

Please remember MongoDB hasn't foreign key.

**Lecture Notes ** 

Read this note after you watch the video. It's also worth noting that you might decide to keep the resume separate from the employee document if the resume needs to exist when no employee exists for that resume. That is, perhaps you first get a resume and they later become an employee, or not.

One to Many Relations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
city : people   1 ：many

1. 1 to many

people {
	name ： “Andrew”
	city ： “NYC”

}

city {
	_id : "NYC",
	____: 
}

true linking

2. 1 to few

blogpost : comments  1 : few  e.g. 1 to 10

posts {
	name : "",
	comments : [
	 	___,
	 	___,
	 	___,
	]
}

**Quiz**

When is it recommended to represent a one to many relationship in multiple collections?

whenever the many is large

Many to Many Relations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
books : anthors         if few  ： few

books : anthors

books{
	_id : 12
	title ： “”
	authors : [27]
}

authors{
	_id : 27
	author_name : ""
	books:[12,7,8]
}

students : teachers

students {					teachers {

 teachers : [] 					students : []
}								}

Multikeys

multikey indexes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
MongoDB Enterprise > use test
switched to db test
MongoDB Enterprise > db.teachers.insert({_id : 0,  name:"Allen", students: [0, 1]})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.teachers.insert({_id : 1,  name:"Tome", students: [0, 1]})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.teachers.insert({_id : 2,  name:"Frank", students: [0, 2]})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.students.insert({_id:0, name:"C", teachers:[0, 1]})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.students.insert({_id:1, name:"B", teachers:[0, 1, 2]})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.students.insert({_id:2, name:"A", teachers:[0, 1]})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.students.createIndex({teachers : 1})
{
	"createdCollectionAutomatically" : false,
	"numIndexesBefore" : 1,
	"numIndexesAfter" : 2,
	"ok" : 1
}
MongoDB Enterprise > db.students.find(teachers: {$all:[0, 1]} )
2018-02-03T11:52:23.266+1100 E QUERY    [thread1] SyntaxError: missing ) after argument list @(shell):1:25
MongoDB Enterprise > db.students.find({teachers: {$all:[0, 1]} })
{ "_id" : 0, "name" : "C", "teachers" : [ 0, 1 ] }

{ "_id" : 1, "name" : "B", "teachers" : [ 0, 1, 2 ] }

{ "_id" : 2, "name" : "A", "teachers" : [ 0, 1 ] }

Benefits of Embedding

Embedding means colocate the data together so that scanning disk could be more efficient.

  • good
    • improved read Performance
    • one round trip to the DB

Trees

find parents of a category, so it is like a tree

1
2
3
4
5
category{
	id: 7
	category_name: "outdoors"
	ancestors：[3, 5, 7]
}

Given the following typical document for a e-commerce category hierarchy collection called categories

1
2
3
4
5
6
{
  _id: 34,
  name: "Snorkeling",
  parent_id: 12,
  ancestors: [12, 35, 90]
}

db.categories.find({acncestors： 34})

When to denormalize

1
2
3
1 ：1   				    embed
1 : Many 				embed （from the many to the one）
many : many 			links objectid in the array

Homework: 4.1

Please review the data model for the Crunchbase companies data set. The document from this collection for Facebook is attached in the handout for convenience. Documents in this collection contain several array fields including one for “milestones”.

Suppose we are building a web site that will display companies data in several different views. Based on the lessons in this module and ignoring other concerns, which of the following conditions favor embedding milestones (as they are in the facebook.json example) over maintaining milestones in a separate collection. Check all that apply.

Note: Schema design is as much an art as a science. If you get the answer wrong on your first attempt. Please visit the forum to discuss with your fellow students.

  • ✔︎The number of milestones for a company rarely exceeds 10 per year
  • Milestones will never contain more than 15 fields
  • An individual milestone entry will always be smaller than 16K bytes
  • ✔︎One frequently displayed view of our data displays company details such as the “name”, “founded_year”, “twitter_username”, etc. as well as milestones.
  • Some of the milestone fields such as “stoneable_type” and “stoneable” are frequently the same from one milestone to anpther.

I haven’t got the correct answer, but the principle to support embeding should be the milestones has to be small and viewd often.

Homework: 4.2

Suppose you are working with a set of categories defined using the following tree structure. “Science” is a sub-category of “Books”; “Chemistry” and “Physics” are sub-categories of “Science”; and “Classical Mechanics” and “Quantum Mechanics” are sub categories of “Physics”.

1
2
3
4
5
6
Books
    Science
        Chemistry
        Physics
            Classical Mechanics
            Quantum Mechanics

For this tree, each node is represented by a document in a collection called categories. What kind of schema will make it possible to find() all descendant documents of a category using a single query. 

1
2
3
4
5
6
db.categories.insertOne({"_id": "Quantum Mechanics", "ancestors": ["Books", "Science", "Physics"], "parent": "Physics"})
db.categories.insertOne({"_id": "Classical Mechanics", "ancestors": ["Books", "Science", "Physics"], "parent": "Physics"})
db.categories.insertOne({"_id": "Physics", "ancestors": ["Books", "Science"], "parent": "Science"})
db.categories.insertOne({"_id": "Chemistry", "ancestors": ["Books", "Science"], "parent": "Science"})
db.categories.insertOne({"_id": "Science", "ancestors": ["Books"], "parent": "Books"})
db.categories.insertOne({"_id": "Books", "ancestors": [], "parent": null})

Homework: 4.3

Suppose you are working with a library catalog system containing collections for patrons, publishers, and books. Book documents maintain a field “available” that identifies how many copies are currently available for checkout. There is also a field “checkout” that holds a record of all patrons that are currently borrowing a copy of the book. For example, the document below indicates that the library owns four copies of “Good Book”. Three are currently available for checkout. One has been checked out by patron “33457”.

1
2
3
4
5
6
7
8
9
{
    _id: 123456789,
    title: "Good Book",
    author: [ "Sam Goodman", "Mike Smith" ],
    published_date: ISODate("2010-09-24"),
    publisher_id: "Smith Publishing",
    available: 3,
    checkout: [ { patron_id: "33457", date: ISODate("2012-10-15") } ]
}

Can make atomic updates as books are checked out or turned in.

Reference

Learn MongoDB from MongoDB - M101JS: MongoDB for Node.js Developers