Professional Documents
Culture Documents
AWS NoSQL MongoDB
AWS NoSQL MongoDB
January 2012
MongoDB on AWS
Miles Ward January 2012
Page 1 of 20
January 2012
Table of Contents
Table of Contents ........................................................................................................................................ 2 Introduction ................................................................................................................................................ 3 What is NoSQL? ........................................................................................................................................... 3 NoSQL on AWS ............................................................................................................................................ 3 Running NoSQL Systems on Amazon Elastic Compute Cloud ......................................................................... 4
Performance ..........................................................................................................................................................4 Durability and Availability ......................................................................................................................................5 Elasticity and Scalability .........................................................................................................................................5 Configuration .........................................................................................................................................................5 Pricing....................................................................................................................................................................6
Page 2 of 20
January 2012
Introduction
Amazon Web Services (AWS) is a flexible, cost-effective, easy-to-use cloud computing platform. NoSQL software packages are widely deployed in the AWS cloud. Running your own NoSQL data store on Amazon Elastic Cloud Compute (Amazon EC2) is a great scenario for users whose application requires the unique benefits of structured storage software optimized for high-performance operations on large datasets. This whitepaper will help you understand one of the most popular NoSQL options available with the AWS cloud computing platformthe open source application MongoDB. We provide an overview of general best practices that apply to all major NoSQL options, and we examine important MongoDB implementation characteristics such as performance, durability, and security. We pay particular attention to identifying features that support scalability, highavailability, and fault-tolerance.
What is NoSQL?
NoSQL is a popular name for a subset of structured storage software that is designed with the intention of delivering increased optimization for high-performance operations on large datasets, at the expense of strict ACID (atomicity, consistency, isolation, and durability) compliance and, as the name implies, native querying in the SQL syntax. NoSQL software is typically easy to use by developers, tolerant of scale by way of horizontal distribution, and reflects optimization around narrower workload definitions. There are three major categories of NoSQL applications today: Key-value stores: like Cassandra, Riak, and Project Voldemort Graph databases: like Neo4j, DEX, and InfiniteGraph Document stores: like MongoDB, CouchDB, and BaseX
These NoSQL applications are either separate open source software (OSS) projects or commercial closed-source projects. The applications are written in different languages; they expose different interfaces; and they implement different optimizations.
NoSQL on AWS
Amazon Web Services (AWS) provides an excellent platform for running many advanced data systems in the cloud. Some of the unique characteristics of the cloud provide strong benefits for NoSQL workloads. In many ways AWS infrastructure is similar to local, physical infrastructure; some minor differences apply to all major NoSQL systems. A general understanding of those differences can assist greatly in making good architecture decisions for your system. Amazon Web Services has native NoSQL services that do not require direct administration and offer usage-based pricing. Consider these options as possible alternatives to building your own system with an OSS or a commercial NoSQL application. Amazon SimpleDB is a highly available, scalable, and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests, and Amazon SimpleDB does the rest. Amazon Simple Storage Service (Amazon S3) provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of websites. All services aim to maximize benefits of scale and to pass those benefits on to developers.
Page 3 of 20
January 2012
Performance
The performance of a NoSQL system on Amazon Elastic Compute Cloud (Amazon EC2) depends on many factors, including the Amazon EC2 instance type, the number and configuration of Amazon Elastic Block Store (Amazon EBS) volumes, the configuration of the software, and the application workload. We encourage users to benchmark their actual application on several Amazon EC2 instance types and storage configurations in order to select the most appropriate configuration. To increase the performance of your system you need to know which of the servers resources is the performance constraint. If CPU or memory limits your system performance you can scale up the memory, compute, and network resources available to the software by choosing a larger Amazon EC2 instance type. Remember, 32-bit Amazon Machine Images (or AMIs) cant run on 64-bit instances, so if you anticipate needing the higher performance instance types, start with 64-bit instances. Also, remember that changing an existing instance to a different size requires a stop/start cycle. If your performance is disk I/O limited, you might want to make changes to the configuration of your disk resources. Remember that Amazon EBS volumes, the persistent block storage available to Amazon EC2 instances, are connected via the network. An increase in network performance can have a significant impact on aggregate performance, so be sure to choose the appropriate instance size. To scale up random I/O performance, you can increase the number of EBS volumes as a ratio of EBS storage (6 x 100 GB EBS volumes versus 1 x 600 GB EBS volume), and if aggregation is required, use software RAID (redundant array of independent disks) 0 (disk striping) across multiple EBS volumes to increase single logical volume total IOPS (input output operations per second). Remember, utilizing RAID striping reduces operational durability of the logical volume by a degree inversely proportional to the number of EBS volumes in the stripe set. A single EBS volume can provide approximately 100 IOPS, and single instances with arrays of 10 or more attached EBS disks can often reach 1,000 IOPS sustained. To take advantage of additional attached EBS disks, evaluate the network load to ensure that your instance size is sufficient to provide the network bandwidth required. For sequential disk access, ephemeral disks are somewhat higher performance and dont impact your network connectivity. Some customers use ephemeral disks in conjunction with Amazon Elastic Block Store to conserve network bandwidth and EBS I/O for more write-heavy operations. In many cases, the constraint is more abstract. In general, users can use the same system performance tuning options in the Amazon EC2 environment that they use in a physical server environment. Users can also scale the total performance of a NoSQL system by scaling horizontally across multiple servers with sharding, caching, and synchronous and asynchronous replication strategies.
Page 4 of 20
January 2012
No approach to structured storage is complete without using the available options in each system for application-level redundancy.
Configuration
To create a NoSQL structured data store on Amazon EC2, you can launch an instance from an AMI that has the base operating system you want, then install the application software using the standard operating system software install processes. Remember, theres no DVD drive in the cloud, so users must download the required software. Many customers use the AWS Management Console to upload setup files to Amazon S3 from their workstations.
Page 5 of 20
January 2012
After your application is installed and configured on Amazon EC2, you interact with your service via its exposed interface, either by exposing that interface over the web (or a VPN) to your location, or remotely accessing the server via SSH, NX, VNC, or RDP. You can work with an application on Amazon EC2 just as you would with an on-premises software application. Youll need to configure the instances security group to allow traffic on the port used by the NoSQL application. All ports are disabled by default, so make sure you open up the ports you need. The Amazon EBS volumes that serve as the root drives of Amazon EC2 instances are often durable, and they are designed to survive the physical failure of the Amazon EC2 instance host or an individual EBS host hardware failure. However, you might have complex operating system and application configurations that should be preserved by rebundling your AMI. When you re-bundle your AMI, subsequent (or additional) launches of Amazon EC2 instances will include all of your configuration changes. For directions, go to Creating Amazon EBS-backed AMIs in the Amazon Elastic Cloud Compute User Guide.
Pricing
Eliminating the need to manage hardware and having the ability to quickly provision capacity to scale your NoSQL structured storage makes the AWS cloud the optimum solution for running NoSQL systems. The AWS cloud not only gives you flexibility when designing a solution, which well cover in greater detail in coming sections of this paper, but also makes it cost-efficient to operate it. Depending upon the architecture of your NoSQL solution, there are a few factors you should account for when estimating the cost of delivery: 1. Compute: Amazon EC2 instance hours purchased either in the on-demand, Reserved (Light, Medium or Heavy Utilization) or Spot models for the computational load of the system. 2. Storage: Amazon EBS volumes for your storage (both actual storage capacity and I/O to disk). 3. Data Transfer: Network transit (particularly intra-Availability Zone networking charges for the configurations that span across multiple availability zones (Replica-set). 4. Backups: Amazon S3 storage for snapshots of configured root (EBS) volumes. Feel free to leverage tools like AWS Simple Monthly Calculator to estimate the cost of your projected workloads.
Page 6 of 20
January 2012
Basic tips
MongoDB is updated frequently; www.mongodb.org is a critical resource for the latest code enhancements, documentation, and news on this open source software (OSS) project. Use 64-bit instances only; MongoDB can only store about 2.5 GB of data in a 32-bit system. For more information go to http://blog.mongodb.org/post/137788967/32-bit-limitations. Use XFS or Ext4. These file systems support I/O suspend and write-cache flushing which is critical for multi-disk consistent snapshots. Use EBS in RAID. Use RAID 0 or 10 for the data volume, RAID 1 for configdb. Snapshot regularly for backups. Be careful with large objects. MongoDB currently restricts documents to 16 MB. If you need larger, use GridFS within Mongo. For more information, go to http://www.mongodb.org/display/DOCS/GridFS. Raise file descriptor limits. The default of 1024 on most systems wont work for production-scale workloads. For more information go to http://www.mongodb.org/display/DOCS/Too+Many+Open+Files. Use MongoDB 2.0 or greater. Significant feature enhancements in the version 2.0 release include security in sharded environments and other critical elements. Turn on authentication. For more information see http://www.mongodb.org/display/DOCS/Security+and+Authentication#SecurityandAuthenticationReplicaSetandShardingAuthentication. Turn off atime and diratime when you mount the data volume. This reduces I/O overhead, which isnt useful to MongoDB. Dont use large virtual memory (VM) pages. For more information see, http://linuxgazette.net/155/krishnakumar.html.
Page 7 of 20
January 2012
9. Alternatively, you can install MongoDB via a package manager, in order to do so create a file at
/etc/yum.repos.d/10gen.repo
12. Edit the /etc/mongodb.conf file to increase oplogSize, enable journaling, and target our data and log directories.
oplogSize = 8000 journal = true dbpath = /data/db logpath = /data/db/logs
13. Use the AWS Management Console to allow ingress from your application servers via appropriate ports by creating rules in your Amazon EC2 security group. (The default port is 27017.) 14. Start mongod.
$ ./mongodb-xxxxxxx/bin/mongod
15. Optionally, you can have mongod start automatically every time the server boots.
#$ chkconfig mongod on
Page 8 of 20
January 2012
Architecture
The design of your MongoDB installation on Amazon EC2 depends on the scale at which you want to operate. Are you experimenting with the framework on your own for a private project? Its likely that youll install mongod (the running MongoDB application) on the same machine as the rest of your software, and have all the mongod file structure colocated with the rest of your application on the root volume of the instance. This scale is fine for developer experimentation, however for high- scale applications more complex topologies provide higher performance and fault tolerance. MongoDB has two separate constructs for multi-node topologies, which are often combined in the highest-performance systems: replica sets and sharded replica sets. Replica sets are an asynchronous cluster replication technology, and sharding is an automatic data distribution system. Increasing the number of instances in a replica set provides horizontal scalability for read performance and fault-tolerance. Increasing the number of shards (each one being a replica set) allows you to distribute distinct data to separate mongod processes to provide horizontal scalability for write performance.
Routing to the appropriate shard is provided by the mongos process, which allows seamless access to your distributed dataset. If you set slaveOkay in your client layer (either in your driver or in your direct queries), mongos will route read-traffic to the secondary nodes in each shard's replica set.
Building Blocks
mongod Mongod is the primary mongo application, responsible for the structured storage system. Mongod will be running on each node of the cluster you deploy. mongos Mongos is a routing and coordination process that makes the mongod nodes in the cluster look like a single system. Mongos processes have no persistent state; rather, they pull their state from the config server on startup. Any changes that occur on the config servers are propagated to each mongos process. Mongos processes may be run on the shard servers themselves, but they are lightweight enough to exist on each application server. Many mongos processes can be run simultaneously since these processes do not coordinate between one another.
Page 9 of 20
January 2012
config server Config server is a mongod process used to synchronously replicate the state information of a sharded environment. In a sharded environment, config servers store the metadata of the cluster. Each config server is a mongod process that stores metadata. Although config server can run standalone, any production deployment should have three config servers that have copies of the same metadata (for data safety and high availability (HA)). The config server mongod process is fairly lightweight and can be run on t1.micro instances; however, it does carry critical data that should be aggressively protected. Many customers scale out their MongoDB server infrastructure following these steps along the growth curve .For your config server we recommend that you use DNS for high availability. To change the IP address of config servers you may need to take the cluster down or change the location to which the host name points. An easier approach is to have DNS records for each config server instance so that you can change the locations easily. This introduces a dependency on your DNS solution. Amazon Route 53 provides a highly available, high-performance DNS service inside of Amazon EC2 thats very easy to set up, or you can run a BIND or other DNS server on our network. The config server is only required in sharded deployments. If you are just running a replica set, there is no need for config servers.
Production Designs
So, given these building blocks, how would a system scale to accommodate growing load over time? The following steps and diagrams guide you through a sample production design process that scales as the load grows. Step 1: Separate the mongod server from the application tier. In this setup theres no need for a config server, or mongos since there is no sharding structure for your applications to navigate.
Page 10 of 20
January 2012
Step 2: For replica sets, use mongos for distributing read load to slaves. Note: Mongos replication is asynchronous; its critical to use getLastError in your queries with w : 2 or w : majority; See extra detail here. Use an odd number of MongoDB instances, in this example we use three, but you can use more if you need higher read performance. For high availability these replicas should be placed in different Availability Zones.
Step 2a: If your desired regions only support two Availability Zones, consider replication to the next nearest region, shifting to five replicas (two in one Availability Zone, and three in the other). Step 2b: Alternatively, you could place an arbiter instance in the Availability Zone with the fewest instance members of the replica set. Having an arbiter instance in place allows failover to occur when a majority of replicas (2/3 in this example) are lost.
Page 11 of 20
January 2012
Step 3: Create a sharded cluster of replica sets. In this example three config servers are running on separate t1.micro EC2 instances.
Step 4: Scale to a multi-region system that uses a time-delayed replica for warm backup.
Page 12 of 20
January 2012
new fleet of instances from backups of production instances, take a small maintenance window to sync the new more powerful fleet, and then return to service. Designing Storage to Support MongoDB The pattern of scaling the instance footprint for a MongoDB cluster is designed to deliver additional process memory to your workload. Youll want to design your infrastructure carefully to avoid storage bottlenecks. So, it is critical to design the storage subsystem that supports your MongoDB in a way that correctly leverages your available resources. Minimum scale: Use Amazon Elastic Block Store (Amazon EBS). Amazon EBS has significant write cache, excellent random I/O performance, and provides enhanced durability in comparison to the ephemeral disks on the instance. If youre going to use ephemeral disks, on instances which expose more than a single volume, be sure to mirror those volumes (using RAID 1) to enhance operational durability. Remember, even though theyre mirrored, if the instance is lost or terminated, youll lose all of your data.
Medium scale: Optimize for high durability by using RAID 10 on EBS volumes. We suggest using four volumes for your /data/db file, a separate file (potentially a RAID 1 mirror) for your journal, and another separate file for your log.
Page 13 of 20
January 2012
Large scale: Move up to higher bandwidth instance types (m1.xlarge, c1.xlarge, m2.4xlarge), and increase the size of the RAID 10 to be an 8-volume set.
Extra-Large scale: Heres where the pattern changes. At large scale you should have many replicas and as a result, the burden of data durability moves to the application tier. We recommend that you increase storage subsystem performance at the expense of durability by removing the network/performance overhead of the RAID 10 and switch to using only a RAID 0 stripe set. This is especially true if your MongoDB workload is largely very small average writes (4-8 KB). In this situation, due to the complexity of taking snapshots of large multi-volume stripe sets, we recommend adding another replica using a single EBS volume, with time-delayed replication for easier durable backups to Amazon Simple Storage Service (Amazon S3) via snapshot.
Page 14 of 20
January 2012
Web Scale: Shard! If the limit is aggregate write performance, increase the number of shards in your cluster. If the limit is read performance, look at increasing the number of replicas within your shards. Read performance at these scales can be limited if the working dataset isnt in memory, so make sure you have sufficient shard breadth to keep the working set in memory.
If youre located in our US East region, we offer Cluster Compute nodes, which have more bandwidth for communications with the EBS system; CC1 and CC2 nodes would make excellent primary nodes, particularly when paired with a large number of EBS volumes (more than 16).
Creating a Replica Set Setting up asynchronous replication in MongoDB is a three-step process: provisioning the requisite instances and storage, installing and starting the mongod processes, and then formally initiating the set.
Page 15 of 20
January 2012
1. Start three identical EC2 instances, each with enough disks for the medium scale recommendation discussed earlier.
$ ec2-run-instances ami-1b814f72 -b /dev/sdf=:80 -b /dev/sdg=:80 -b /dev/sdh=:80 -b /dev/sdi=:80 -g smongo -m -t m2.2xlarge -z us-east-1a $ ec2-run-instances ami-1b814f72 -b /dev/sdf=:80 -b /dev/sdg=:80 -b /dev/sdh=:80 -b /dev/sdi=:80 -g smongo -m -t m2.2xlarge -z us-east-1b $ ec2-run-instances ami-1b814f72 -b /dev/sdf=:80 -b /dev/sdg=:80 -b /dev/sdh=:80 -b /dev/sdi=:80 -g smongo -m -t m2.2xlarge -z us-east-1c
2. Follow the basic installation instructions discussed earlier, with the following alteration to the step 14, start mongod:
$ mongod --replSet thenameofyourreplicaset --port 27017 --dbpath /db/data
3. When you run the above set, the node should output an error message saying that the replica set isnt up and running yet. You can use DNS to swap out individual replica set hosts; so create three DNS entries in Amazon Route53 (or your DNS system of choice). For more information go to Creating, Changing, and Deleting Resource Record Sets in the Amazon Route 53 Developer Guide. Alternatively, you could host the MongoDB instances inside the Amazon Virtual Private Cloud (Amazon VPC), where you have control over static instance IP addresses. 4. Start the mongo shell (on any one of the replica hosts), create the configuration for the replica set, and initiate the replication.
$ Mongo $ > config = {_id: 'foo', members: [ {_id: 0, host: 'replicaset1.mongohosts.mysite.com:27017'}, {_id: 1, host: 'replicaset2.mongohosts.mysite.com:27017'}, {_id: 2, host: 'replicaset3.mongohosts.mysite.com:27017'}]} $ > rs.initiate(config);
You can also pass a priority:value flag in for each node. This flag controls the priority by which nodes will be elected to the master in a node failure condition. This is useful for electing a time-delayed node for running backups, or for spanning regions. This way, if one of the nodes failed in the primary region with two nodes, that the other node in that region (not the distant disaster recovery (DR) node) would be elected the primary node. More details are available from MongoDB at: http://www.mongodb.org/display/DOCS/Replica+Set+Tutorial and http://www.mongodb.org/display/DOCS/Data+Center+Awareness
Creating a Sharded and Replicated MongoDB Cluster Sharded systems are typically built on top of replicated systems, to support high availability of the architecture and read performance scalability.
Page 16 of 20
January 2012
1. Start additional hosts for mongod, using the --shardsvr flag to enable sharding, and an _id name to distinguish each replica set. This changes the default port to 27018. For example, you could have three shards with three replicas (a primary and two slaves) each, for a total of nine EC2 instances. 2. Confirm that youve started config servers, using the --configsvr flag to enable the config server process. This will default the port to 27019. These config servers can be distributed among the mongod servers at random, or they run on standalone instances (t1.micro). 3. Install mongos routers on each application host, starting them with the --configdb flag, passing the DNS names and ports for all three of the config server instances. 4. Enable sharding on the database. This enables the shardcollection command below.
db.runCommand( { enablesharding : "<dbname>" } );
5. Shard the relevant collections. The key is the object in the collection that will be indexed and used to distribute chunks of items across your cluster. Once a collection is sharded it cannot be un-sharded without a backup/restore cycle.
db.runCommand( { shardcollection : "<namespace>",key : <shardkeypatternobject> });
Operations
Backup
AWS strongly recommends use of Mongo version 2.0.1 or later and the --journal option for MongoDB on Amazon EC2. We also recommend placing the journal on Amazon EBS volumes distinct from the default dbpath. The actual mechanics of backup are greatly simplified by the Amazon S3 snapshot capability native to Amazon EBS. To perform a backup 1. Flush and lock the mongod by using the fsync and the lock commands.
> db.runCommand({fsync:1,lock:1}); { "info" : "now locked", "ok" : 1 }
2. Create a snapshot of each EBS volume you want to use to hold data, journals, or oplogs for mongod, using the volume numbers on your specific instance. You can find these in the AWS Management Console or using the ec2-describe-instances command.
$ ec2-create-snapshot -d thedescriptionofyourbackup vol-11111111
3. After the ec2-create-snapshot command returns the Pending state for the snapshot, it is safe to resume operations on mongod. Unlock the database and resume processing.
> db.$cmd.sys.unlock.findOne(); { "ok" : 1, "info" : "unlock requested" }
Page 17 of 20
January 2012
Note: For the duration of a snapshot (until its listed as Completed) there will be a variable negative impact to Amazon Elastic Block Store disk performance. If you are operating near maximum capacity, you should take your system snapshots on a time-delayed replica.
Restore
To restore from a backup 1. Create an Amazon EBS volume from each snapshot used to back up your data. This could be more than one volume. You can find the snapshot IDs and create the volumes by using the AWS Management Console or by using the ec2-describe-snapshots command.
$ ec2-create-volume --availability-zone us-east-1a --snapshot snap-12345678
2. Attach the volumes to the instance. Remember, if youre restoring a RAID set to replace the volumes in the same order for easiest re-creation of the RAID volume in the OS.
$ ec2-attach-volume --instance i-aaaaaaa --device /dev/sdp vol-01010101
3. Make a directory into which you can mount the file system.
$ sudo mkdir -p /data/db/ $ sudo chown `id -u` /data/db
Monitoring
AWS provides robust monitoring of Amazon EC2 instances, Amazon EBS volumes, and other services via the Amazon CloudWatch service. Amazon CloudWatch can send an alarm, via SMS or email, when user-defined thresholds are reached on individual Amazon Web Services. For example, you can set an alarm to warn of excessive storage throughput. To set this alarm, go to Send Email Based on Storage Throughput Alarm in the Amazon CloudWatch Developer Guide. Another approach would be to write a custom metric to Amazon CloudWatch, for example current free memory on your instances, and to alarm or trigger automatic responses off of those measures. 10gen, the company behind MongoDB, has also released MongoDB Monitoring Service (MMS), a free, cloud-based product for monitoring and managing MongoDB installations. More details on this service are available at http://www.10gen.com/mongodb-monitoring-service.
Security
Network
To access MongoDB resources from other Amazon EC2 instances or from remote servers, you need to configure your security group to permit ingress over the appropriate TCP ports. Like any network service, these ports should be
Page 18 of 20
January 2012
opened conservatively; for example, open them only to your corporate office or to other authenticated machines that require access. Default TCP port numbers for MongoDB processes are as follows: A standalone mongod process: 27017 mongos : 27017 shard server (mongod --shardsvr) : 27018 config server (mongod --configsvr) : 27019 web statistics page for mongod : add 1000 to port number (28017, by default). Most statistics pages in the HTTP UI are unavailable unless the --rest option is specified. To disable the "home" stats page use --nohttpinterface. (If you use this port, you should secure it. However, the information on the stats home page is read-only regardless.) Ports can be opened to all computers in the same Amazon EC2 security group, another one, to a specific IP address, or a CIDR IP address range. Security group parameters can be changed at any time. For extra security, some users use automation scripts to open these ports when needed and close them when MongoDB resources are not in use. The best practice in a multi-tier architecture is to permit operational access to the mongod tier only from servers in the security groups that require access, and control access only from known administrative IP addresses.
Authentication
To enable authentication, run the mongod process with the - - auth option. You need to add a user to the admin collection before starting the server, or add the first user from the localhost interface. > use admin > db.addUser("theadmin", "anadminpassword") To authenticate a replicated or sharded mongod installation 1. Create a key file (Base64, 6+ characters, no greater than 1 KB) that can be copied to each server in the set. 2. Modify this file's permissions to be readable only by the appropriate user. 3. Start each server in the cluster (including all replica set members, all config servers, and all mongos processes) with the --keyFile /path/to/file option.
Page 19 of 20
January 2012
Summary
The AWS cloud provides a unique platform for any NoSQL application, including MongoDB. With capacities that can meet dynamic needs, cost that is based on use, and easy integration with other AWS products like Amazon CloudWatch, the AWS cloud enables you to run a variety of NoSQL applications without having to manage the hardware yourself. In addition, management features inherent in NoSQL systems can be leveraged in the cloud. This paper provides specific information on MongoDB, including hardware configurations, architecture designs, high availability and disaster recovery approaches, and security details. For more information on Amazon AWS, see http://aws.amazon.com.
Page 20 of 20