Kbaran: /* Storage engine */

2018-09-07T12:03:45Z

Storage engine

New page

[[Category:MongoDB]]

{{Copyright Notice}}

{{:Basics of MongoDB}}

== Single-server Configuration and Deployment ==

=== Configuration File Options ===
* YAML-based configuration file format (since 2.6)
** http://docs.mongodb.org/manual/reference/configuration-options/
 

<syntaxhighlight lang="bash">
mongod --config /etc/mongodb.conf
mongod -f /etc/mongodb.conf
</syntaxhighlight>
 

<syntaxhighlight lang="yaml">
net:
port: 27017
bindIp: 127.0.0.1
operationProfiling:
mode: slowOp
slowOpThresholdMs: 10
storage:
dbPath: c:\data\db
wiredTiger:
engineConfig:
cacheSizeGB: 1
systemLog:
destination: file
path: c:\data\logs\mongodb.log
security:
authorization: enabled
keyFile: c:\data\config\keyfile.txt
replication:
replSetName: training
oplogSizeMB: 128
</syntaxhighlight>
 

=== Storage engine ===
* A storage engine is the part of a database that is responsible for managing how data is stored on disk.
** in other words, it's an interface between database and hardware
* MMAPv1 Storage Engine
** historically first storage engine
** collection level locking
** in-place updates
** power of 2 sized document allocations
* WiredTiger Storage Engine
** new in version 3.0
** default since 3.2
** improved performance in most use cases
** document level locking
** compression (for data and indexes)
* In-Memory (experimental)
* RocksDB, HDFS, FusionIO (under development)
 

==== MMAPv1 ====
* --storageEngine mmapv1
* locks:
** multiple readers, single writer lock
** shared resources: data, metadata (indexes, journal)
* lock levels:
** database level locking (2.2 - 2.6)
** collection level locking (3.0)
* journal - write-ahead transaction log
** all operations are written to journal
** and after that are applied to data files
* data on disk is raw BSON, directly mapped into virtual memory
* document allocations
** no padding
** padding factor (automatic, manual)
** power of 2 sized allocations (from 32B to 2MB)
 

==== WiredTiger ====
* --storageEngine wiredTiger
* data is stored in B-Trees (similar to B-Trees MMAPv1 uses for indexes)
** initially documents get written into un-used regions
** and then merged with the rest of the data in background later
* WT uses two caches:
** WT cache (half of RAM by default)
** operating system cache
* it uses a write-ahead transaction log in combination with checkpoints to ensure data persistence
* MongoDB will commit a checkpoint to disk
** 60 seconds after the end of previous checkpoint or
** when there is to much dirty data in WT cache (2 gigabytes of data)
* document level locking
** WT has no locks but good concurrency protocols
** writes should scale with the number of threads
* compression (can be set for each collection separately)
** snappy (fast, balance storage efficiency and processing requirements)
** zlib (higher compression rates at the cost of more CPU)
** off
* WiredTiger options
* how to test your own data?
 

<syntaxhighlight lang="JavaScript">
db.createCollection( "email", { storageEngine: { wiredTiger: { configString: 'block_compressor=zlib' }}})
</syntaxhighlight>
 

=== Authentication and Authorization ===
* MongoDB employs Role-Based Access Control
* MongoDB supports
** password-based authentication (SCRAM-SHA-1, MONGODB-CR)
** x.509 certificates
** LDAP proxy (enterprise edition)
** Kerberos (enterprise edition)
* authentication is disabled by default
** to turn it on use --auth switch,
** create at least one superuser account before
* user belongs to a database and must be authenticated in that database
* users created in admin or local database can perform operations on all databases
* commonly used roles: read, readWrite, dbAdmin, userAdmin, dbOwner
* http://docs.mongodb.org/manual/core/authorization/
 

<syntaxhighlight lang="JavaScript">
> use admin
> db.createUser({user : "ubuntu", pwd : "NobleProg", roles : ["root"]})
</syntaxhighlight>

<syntaxhighlight lang="bash">
$ mongo -u ubuntu -p --authenticationDatabase admin
$ mongo localhost/admin -u ubuntu -p
</syntaxhighlight>

<syntaxhighlight lang="JavaScript">
> use admin
> db.auth("ubuntu", "NobleProg")
> use test
> db.createUser({user: "testuser", pwd: "NobleProg", roles: ["dbOwner"]})
> db.getSiblingDB("admin").system.users.find().pretty()
> db.getUser("testuser")
> db.getRole("dbOwner")
> db.getRole("dbOwner", {showPrivileges: true})

> db.grantRolesToUser("testuser", [{role: "read", db: "dept"}])
> db.grantRolesToUser("testuser", [{role: "readWrite", db: "dept"}])
> db.revokeRolesFromUser("testuser", [{role: "read", db: "dept"}])
> db.grantRolesToUser("testuser", [{role: "dbAdmin", db: "dept"}])

> db.dropUser("testuser")
> db.logout()
</syntaxhighlight>

<syntaxhighlight lang="JavaScript">
//how to create a custom role
> use admin
> db.auth("ubuntu", "NobleProg")
> db.createUser({user: "monitoringuser", pwd: "NobleProg", roles: []})
> db.createRole({role: "statsWatcher", privileges: [{resource: {"anyResource": true}, actions: ["serverStatus"]}], roles: []})
> db.getRole("statsWatcher", {showPrivileges: true})
> db.grantRolesToUser("monitoringuser", [{role: "statsWatcher", db: "admin"}])
</syntaxhighlight>
 

=== Monitoring MongoDB ===
* mongotop
* mongostat
* MongoDB's Web Console on port +1000
** enable '''rest''' and '''httpinterface'''
* MongoDB Monitoring Service (MMS)
** plus Munin for monitoring hardware
 

<syntaxhighlight lang="JavaScript">
> db.serverStatus()
> db.runCommand("serverStatus")
</syntaxhighlight>
<syntaxhighlight lang="JavaScript">
> rs.status()
> db.runCommand("replSetGetStatus")
</syntaxhighlight>
<syntaxhighlight lang="JavaScript">
> db.currentOp()
> db.stats()
> db.collection.stats()
</syntaxhighlight>
 

=== Profiler ===
* can be turned on database level or for entire server
* profiling levels:
** 0 - no profiling
** 1 - only includes "slow" operations
** 2 - includes all operations
 

<syntaxhighlight lang="JavaScript">
> db.getProfilingLevel()
> db.setProfilingLevel(1, 100)
> db.system.profile.find().sort({millis : -1}).pretty()
</syntaxhighlight>
 

{{:Indexes in MongoDB}}

{{:Replication in MongoDB}}

{{:Backup and Restore in MongoDB}}

== Other Administrative Tasks ==
* Preheating data
** loading everything into memory
** loading selected collections only
<syntaxhighlight lang="javascript">
> db.runCommand({ touch: "collectionName", data: true, index: true })
</syntaxhighlight>
** loading specific indexes
** loading recently created documents
** replay application usage
 

* Compacting
<syntaxhighlight lang="javascript">
> db.runCommand({"compact" : "collectionName", "paddingFactor" : 1.5})
</syntaxhighlight>
 

* Repairing data
** use --repair and optionally --repairpath
<syntaxhighlight lang="javascript">
> db.repairDatabase()
</syntaxhighlight>

MongoDB for Administrators - Revision history

Kbaran: /* Storage engine */