Utilizing Amazon S3 and EC2

in Rails

Jonathan Weiss, Peritor Wissensmanagement GmbH

RailsConf Europe Berlin, 2007
Starting Point

One machine:

• Apache

• Ruby / Rails

• MySQL

2
Worst Case Populariy

3
A Difficult Path

4
Problem: Backup

• High availability

• Redundancy

• Very big data sets

5
Problem: File System

• Important files have to be accessed by many servers

• NFS / Samba not practical

6
Problem: Spontaneous Traffic

7
Problem: Load Fluctuation

8
Don‘t reinvent the wheel!

9
Amazon Web Services

Simple Storage Service - S3 Elastic Compute Cloud - EC2 E-Commerce S..

Redundant Data Store Virtual server per hour …….

$0,15 per GB data per month $0,10 per CPU hour
$0,10 - $0,20 per GB traffic $0,10 - $0,20 per GB traffic

10
S3 - Simple Storage Service

• Redundant storage - as much as you like

• max. 5 GB per object

• Organized in „Buckets“

• Web Service API for uploads

• Downloads via
• Web Service
• HTTP / HTTPS
• BitTorrent

11
S3 - Buckets

S3

• Unique over all S3

• Contains many
MyBucket_name www.peritor.com
key-value-metadata tupel
foto_5.jpg site/screen.css
• Cannot contain other buckets!
backups/januar.zip site/script.js
• Key can contain „/“ fotos/2007/001.png FreeBSD-6.2.iso

12
S3 with AWS::S3
Upload

Download

13
EC2 - Elastic Compute Cloud

• Based on XEN virtualization

• On demand virtual servers - controlled with Web Service API

• 1,7 GHz
• 1,75 GB RAM
• 160 GB local file system (not persistent!)

• Use your favorite Linux distro (Linux 2.6.16),

Amazon Machine Images (AMI) are stored on S3

• ACLs for hosts/ports access control

14
EC2 Tools
List availabe images

Start a new instance

Login with SSH

Shutdown instance

15
amazon-ec2 gem
Setup

Usage

16
And now?

How does this solve

my problems?

17
S3 - Backup

• s3sync.rb • S3Browser

• Brackup • Firefox S3 Organizer extension

• Jungle Disk •…

• S3 FUSE

• s3DAV

• Duplicity

18
s3sync.rb
Backup

Restore

19
S3 Asset Host

20
S3 Asset Host

Setup DNS

Rails configuration

21
S3 Asset Host

welcome.rhtml template

Output

22
S3 - Authenticated User Data

23
attachment_fu Rails plugin

Setup

24
attachment_fu Rails plugin

Upload

Download

25
On-Demand Computing with EC2

Time based, e.g. with cron

26
On-Demand Computing with EC2

Load based, e.g. with Monit

27
On-Demand Computing with EC2

28
EC2 for extra capacity

29
Load Balancer / Proxy

Example mod_proxy_blancer:

• Talks to multiple backend servers

(Mongrel)

• Central Proxy/Load-Balancer
configuration that has knowledge
about nodes

• Typically proxy restart on config

change

30
Swiftiply
Swiftiply Proxy:

• Multiple backend clients have a

persistent connection to the backend
port

• Proxy listens on cluster port for

requests and forwards them

No re-configuration

31
Swiftiply Proxy

Installation Configuration

Start

32
Swiftiplied Mongrel

• Gem plugin that patches Mongrel

• Transforms Mongrel in Swiftiply client

• Experimental

Start

or

33
EC2 on Demand before Swiftiply

34
EC2 on Demand with Swiftiply

35
EC2 on Demand with Swiftiply

36
Ressources

• Amazon Web Services

http://aws.amazon.com

• Swiftiply
http://swiftiply.swiftcore.org

• Attachment_fu
http://svn.techno-weenie.net/projects/plugins/attachment_fu/

• Elastic Rails
http://www.elasticrails.com

• Capazon
http://capazon.rubyforge.org

• RightScale
http://www.rightscale.com

37
Peritor Wissensmanagement GmbH
Lenbachstraße 2
12157 Berlin
Internet: www.peritor.com Telefon: +49 (0)30 69 40 11 94
Email: info@peritor.com Telefax: +49 (0)30 69 40 11 95

© Peritor Wissensmanagement GmbH - All Rights Reserved

38