Deployment Alvin Richards alvin@10gen.

com

So you’re ready to deploy !

some points to consider

Agenda
• A word on performance • Sizing Your Hardware • Software

• memory

/ cpu / disk io

• Installing MongoDB / Upgrades • EC2 Notes • Security • Backup • Durability • Upgrading • Monitoring • Scaling out

• os / filesystem

A Word on Performance
• Ensure your queries are being executed correctly

• Enable profiling • db.setProfilingLevel(n)
• n=1: slow operations, n=2: all operations

• Viewing profile information
• db.system.profile.find({info: /test.foo/})
•http://www.mongodb.org/display/DOCS/Database+Profiler

• Query execution plan:

•db.xx.find({..}).explain()
•http://www.mongodb.org/display/DOCS/Optimization • Make sure your Queries are properly indexed.

• Working set should be as much in memory as possible, but

Sizing Hardware: Memory

• your whole data set doesn’t have to
•Memory Mapped files

• Maps Files on Filesystem to Virtual Memory
• Not Physical RAM

• Page Faults - not in memory - from disk - expensive
• Indices

• Part of the regular DB files
• Consider Warm Starting your Database

Sizing Hardware: CPU
• MongoDB uses multiple cores

• For working-set queries, CPU usage is minimal • Generally, faster CPU are better
• Aggregation, Full Tablescans

•Makes heavy use of CPU / Disk •Instead of counting / computing:
• cache / precompute • Map Reduce

• Currently Single threaded •Can be run in parallel across shards. • This restriction may be eliminated, investigating options

Sizing Hardware: I/O
• Disk I/O determines performance of non-working set queries • More Faster Disks = Better • Raid 10 - Stripe + Mirror • improved write performance
• survive single disk failure • double storage needs

• Raid 5 or 6 • 1 or 2 additional disks required for parity
• Flash • survive 1 or 2 disk failures • not all implementation created equally

• Expensive, getting cheaper • Significantly reduced seek time, increased IO throughput • Poor random writes & sequential reads

MongoStat
• Tool that comes with MongoDB • Shows

• counters for I/O, time spent in write lock, ...

IOStat
iostat  -­‐x  2 iostat  -­‐w  1
disk0 KB/t tps MB/s 12.83 3 0.04 11.12 75 0.81 4.00 3 0.01 disk1 KB/t tps 2.01 0 0.00 0 0.00 0 MB/s 0.00 0.00 0.00 disk2 KB/t tps 12.26 2 0.00 0 0.00 0 cpu MB/s 0.02 0.00 0.00 load average us sy id 1m 5m 15m 11 5 83 0.35 0.26 0.25 60 24 16 0.68 0.34 0.28 60 23 17 0.68 0.34 0.28

avg-cpu:

%user 0.00

%nice %system %iowait 0.00 7.96 29.85 rrqm/s 0.00 0.50 wrqm/s 0.00 4761.19 r/s 0.00 6.47

%steal 0.50 w/s 0.00 837.31

%idle 61.69 rsec/s wsec/s avgrq-sz avgqu-sz 0.00 0.00 0.00 0.00 75.62 43681.59 51.86 38.38 await 0.00 42.33 svctm 0.00 0.46 %util 0.00 38.41

Device: sda1 sda2

Monitor  disk  transfers  :   >  200  -­‐  300  Mb/s  on  XL  EC2,    but  your  mileage  may  vary CPU  usage >  30  %  during  normal  operations  

OS
• For production: Use a 64bit OS

• 32bit has 2G limit • Clients can be 32 bit
• MongoDB supports (little endian only):

• Linux, FreeBSD, OS X • Windows • Solaris (joyent)

Filesystem
• All data, namespace files stored in data directory

• Possible to create symbolics links • Better to aggregate IO across disks
•File Allocation

Filesystem
• Logfiles:

• --logpath <file> • Rotate:

• db.runCommand(“logRotate”) • kill -SIGUSR1 <mongod pid> • killall -SIGUSR1 mongod

• Does not work for ./mongod > <file>
• MongoDB is filesystem-neutral:

• ext3, ext4, XFS are most used • ext4 / XFS preferred (posix_fallocate())

• improved performance for file allocation

MongoDB Version Policy
• Production: •Development
run even numbers

• 1.4.x, 1.6.x, 1.8.x •1.5.x, 1.7.x

• Critical bugs are back ported to even versions

Installing MongoDB
• Installing from Source

• Requires Scons, C++ compiler, Boost libraries, SpiderMonkey,
PCRE

• Installing from Binaries (easiest)

• curl -O http://downloads.mongodb.org/_os_/_version_
• Upgrading database

• Install new version of MongoDB • Stop previous version • Start new version
•In case of database file changes,

•mongodump / mongorestore

EC2 Notes
• Default storage instance is EXT3

• For best performance, reformat to EXT4 / XFS • Use recent version of EXT4
• Use Striping (using MDADM or LVM) aggregates I/O

•This is a good thing
• EC2 can experience spikes in latency

• 400-600mS •This is a bad thing

More EC2 Notes

• EBS snapshots can be used for backups • S3 can be used for longer term backups • Use Amazon availability zones

• EBS can disappear

• High Availability • Disaster Recovery

• Mongo supports basic security • We encourage to run mongoDB in a safe environment • Authenticates a User on a per Database basis • Start database with --auth • Admin user stored in the admin database
use admin db.addUser("administrator", "password") db.auth(“administrator”, “password”)

Security

• Regular users stored in other databases
use personnel db.addUser("joe", "password") db.addUser(“fred”, “password”, true)

Backup
• Typically backups are driven from a slave • Eliminates impact to client / application traffic to master

Backup

•Two main Strategies • Filelock + fsync

• mongodump / mongorestore • Filesystem backup / snapshot

mongodump
• binary, compact object dump • each consistent object is written • not necessarily consistent from start to finish

• unless you lock database: • db.runCommand({fsync:1,lock:1})
• mongorestore to restore database

• database does not have to be up to restore

Filesystem Backup
• MUST

• fsync - flushes buffers to disk • lock - blocks writes
db.runCommand({fsync:1,lock:1})

• Use file-system / LVM / storage snapshot • unlock
db.$cmd.sys.unlock.findOne();

Database Maintenance
• When doing a lot of updates or deletes

• occasional database compaction might be needed
• indices and datafiles

• db.repair()
• With replica sets

• Rolling: start up node with --repair param

Durability
What failures do you need to recover from? • Loss of a single database node? • Loss of a group of nodes?

Durability - Master only
• Write acknowledged
when in memory on master only

Durability - Master + Slaves
• W=2 • Write acknowledged when

in memory on master + slave • Will survive failure of a single node

Durability - Master + Slaves + fsync
• W=n • Write acknowledged when in
memory on master + slaves • Pick a “majority” of nodes • fsync in batches (since it blocking)

Slave delay
• Protection against app
faults • Protection against administration mistakes • Slave runs X amount of time behind

read
shard1
rep_a1 rep_b1 rep_c2

Scale out
shard2
rep_a2 rep_b2 rep_c2

shard3
rep_a3 rep_b3 rep_c3
mongos  /   config  server mongos  /   config  server mongos  /   config  server

write

Monitoring
• mongostat • Built in UI, use --rest and port #

+1000 • Plugins for: • Munin, Cacti, Nagios, Zabbix... • Hosted options • Server Density, Cloudkick...

• Primary function: • Measure stats over time • Tells you what is going on with
your system

Management
• Mongo shell • 3rd party tools
• • • • • • • • • • Fang of Mongo Futon4Mongo Mongo3 MongoHub MongoVUE Mongui Myngo Opricot PHPMoAdmin RockMongo

download at mongodb.org

We’re Hiring !
alvin@10gen.com
conferences,  appearances,  and  meetups
http://www.10gen.com/events

http://bit.ly/mongo>  

Facebook                    |                  Twitter                  |                  LinkedIn
@mongodb

http://linkd.in/joinmongo

Sign up to vote on this title
UsefulNot useful