SCALABLE DATA STORAGE

GETTING YOU DOWN?
TO THE CLOUD!
Web 2.0 Expo SF 20#
Mike Maloe, Mike P~cmnko, D~ek Smoh, Paul Læhrop
THE CAST
MIKE MALONE
INFRASTRUCTURE ENGINEER
@MJMALONE
MIKE PANCHENKO
INFRASTRUCTURE ENGINEER
@MIHASYA
DEREK SMITH
INFRASTRUCTURE ENGINEER
@DSMITTS
PAUL LATHROP
OPERATIONS
@GREYTALYN
SIMPLEGEO
We originally began as a mobile
gaming startup, but quickly
discovered that the location services
and infrastructure needed to support
our ideas didn’t exist. So we took
matters into our own hands and
began building it ourselves.
Mæt Ga#ig~
CSO & co-founder
Joe Stump
CTO & co-founder
THE STACK
storage
queues
AWS
ELB
data centers
... ...
api servers
index
Apache Cassandra
reads
wri tes
auth/proxy
record
storage
geocoder
reverse
geocoder
GeoIP
pushpin
HTTP
gnop
www
AWS
RDS
BUT WHY NOT
POSTGIS?
DATABASES
WHAT ARE THEY GOOD FOR?
DATA STORAGE
Durably persist system state
CONSTRAINT MANAGEMENT
Enforce data integrity constraints
EFFICIENT ACCESS
Organize data and implement access methods for efficient
retrieval and summarization
Data independence shields clients from the details
of the storage system, and data structure
DATA INDEPENDENCE
LOGICAL DATA INDEPENDENCE
Clients that operate on a subset of the attributes in a data set should
not be affected later when new attributes are added
PHYSICAL DATA INDEPENDENCE
Clients that interact with a logical schema remain the same despite
physical data structure changes like
• File organization
• Compression
• Indexing strategy
TRANSACTIONAL RELATIONAL
DATABASE SYSTEMS
HIGH DEGREE OF DATA INDEPENDENCE
Logical structure: SQL Data Definition Language
Physical structure: Managed by the DBMS
OTHER GOODIES
They’re theoretically pure, well understood, and mostly
standardized behind a relatively clean abstraction
They provide robust contracts that make it easy to reason
about the structure and nature of the data they contain
They’re ubiquitous, battle hardened, robust, durable, etc.
These terms are not formally defined - they’re a
framework, not mathematical axioms
ACID
ATOMICITY
Either all of a transaction’s actions are visible to another transaction, or none are
CONSISTENCY
Application-specific constraints must be met for transaction to succeed
ISOLATION
Two concurrent transactions will not see one another’s transactions while “in flight”
DURABILITY
The updates made to the database in a committed transaction will be visible to
future transactions
ACID is a sort-of-formal contract that makes it
easy to reason about your data, and that’s good
ACID HELPS
IT DOES SOMETHING HARD FOR YOU
With ACID, you’re guaranteed to maintain a persistent global
state as long as you’ve defined proper constraints and your
logical transactions result in a valid system state
At PODC 2000 Eric Brewer told us there were three
desirable DB characteristics. But we can only have two.
CAP THEOREM
CONSISTENCY
Every node in the system contains the same data (e.g., replicas are
never out of date)
AVAILABILITY
Every request to a non-failing node in the system returns a response
PARTITION TOLERANCE
System properties (consistency and/or availability) hold even when
the system is partitioned and data is lost
CAP THEOREM IN 30 SECONDS
CLIENT SERVER REPLICA
CAP THEOREM IN 30 SECONDS
CLIENT SERVER REPLICA
wroe
CAP THEOREM IN 30 SECONDS
CLIENT SERVER REPLICA
wroe
aplicæe
CAP THEOREM IN 30 SECONDS
CLIENT SERVER REPLICA
wroe
aplicæe
ack
CAP THEOREM IN 30 SECONDS
CLIENT SERVER REPLICA
wroe
aplicæe
ack
aoeptm
CAP THEOREM IN 30 SECONDS
CLIENT SERVER REPLICA
wroe
ænim
FAIL!
UNAVAILABu!
CAP THEOREM IN 30 SECONDS
CLIENT SERVER REPLICA
wroe
FAIL!
aoeptm
wCæSkTmT!
Certain aspects of ACID encourage (require?)
implementors to do “bad things”
ACID HURTS
Unfortunately, ANSI SQL’s definition of isolation...
relies in subtle ways on an assumption that a locking scheme is
used for concurrency control, as opposed to an optimistic or
multi-version concurrency scheme. This implies that the
proposed semantics are ill-defined.
Joseph M. Hellerstein and Michael Stonebraker
Anatomy of a Database System
BALANCE
IT’S A QUESTION OF VALUES
For traditional databases CAP consistency is the holy grail: it’s
maximized at the expense of availability and partition
tolerance
At scale, failures happen: when you’re doing something a
million times a second a one-in-a-million failure happens every
second
We’re witnessing the birth of a new religion...
• CAP consistency is a luxury that must be sacrificed at scale in order to
maintain availability when faced with failures
A distributed system must also manage the
network - if it doesn’t, the client has to
NETWORK INDEPENDENCE
CLIENT APPLICATIONS ARE LEFT TO HANDLE
Partitioning data across multiple machines
Working with loosely defined replication semantics
Detecting, routing around, and correcting network and
hardware failures
WHAT’S WRONG
WITH MYSQL..?
TRADITIONAL RELATIONAL DATABASES
They are from an era (er, one of the eras) when Big Iron was
the answer to scaling up
In general, the network was not considered part of the system
NEXT GENERATION DATABASES
Deconstructing, and decoupling the beast
Trying to create a loosely coupled structured storage system
• Something that the current generation of database systems never
quite accomplished
UNDERSTANDING
CASSANDRA
APACHE CASSANDRA
A DISTRIBUTED STRUCTURED STORAGE SYSTEM
EMPHASIZING
Extremely large data sets
High transaction volumes
High value data that necessitates high availability
TO USE CASSANDRA EFFECTIVELY IT HELPS TO
UNDERSTAND WHAT’S GOING ON BEHIND THE SCENES
APACHE CASSANDRA
A DISTRIBUTED HASH TABLE WITH SOME TRICKS
Peer-to-peer architecture with no distinguished nodes, and
therefore no single points of failure
Gossip-based cluster management
Generic distributed data placement strategy maps data to nodes
• Pluggable partitioning
• Pluggable replication strategy
Quorum based consistency, tunable on a per-request basis
Keys map to sparse, multi-dimensional sorted maps
Append-only commit log and SSTables for efficient disk utilization
NETWORK MODEL
DYNAMO INSPIRED
CONSISTENT HASHING
Simple random partitioning mechanism for distribution
Low fuss online rebalancing when operational requirements
change
GOSSIP PROTOCOL
Simple decentralized cluster configuration and fault detection
Core protocol for determining cluster membership and
providing resilience to partial system failure
Improves shortcomings of modulo-based hashing
CONSISTENT HASHING
1
2
3
Lsh(alice) % 3
=> 23 % 3
=> 2
With modulo hashing, a change in the number of
nodes reshuffles the entire data set
CONSISTENT HASHING
1
2
3
4
Lsh(alice) % 4
=> 23 % 4
=> 3
Instead the range of the hash function is mapped to a
ring, with each node responsible for a segment
CONSISTENT HASHING
Lsh(alice) => 23
0
42 84
When nodes are added (or removed) most of the data
mappings remain the same
CONSISTENT HASHING
Lsh(alice) => 23
0
42
64
84
Rebalancing the ring requires a minimal amount of
data shuffling
CONSISTENT HASHING
Lsh(alice) => 23
0
32
64
96
GOSSIP
DISSEMINATES CLUSTER MEMBERSHIP AND
RELATED CONTROL STATE
Gossip is initiated by an interval timer
At each gossip tick a node will
• Randomly select a live node in the cluster, sending it a gossip message
• Attempt to contact cluster members that were previously marked as
down
If the gossip message is unacknowledged for some period of
time (statistically adjusted based on the inter-arrival time of
previous messages) the remote node is marked as down
REPLICATION
REPLICATION FACTOR Determines how many
copies of each piece of data are created in the
system
Lsh(alice) => 23
0
32
64
96
RF=3
CONSISTENCY MODEL
DYNAMO INSPIRED
QUORUM-BASED CONSISTENCY
Lsh(alice) => 23
0
32
64
96
W=2
wroe
aad
R=2
W + R > N
Cos<t~t
TUNABLE CONSISTENCY
WRITES
ZERO
DON’T BOTHER WAITING FOR A RESPONSE
ANY
WAIT FOR SOME NODE (NOT NECESSARILY A
REPLICA) TO RESPOND
ONE
WAIT FOR ONE REPLICA TO RESPOND
QUORUM WAIT FOR A QUORUM (N/2+1) TO RESPOND
ALL WAIT FOR ALL N REPLICAS TO RESPOND
TUNABLE CONSISTENCY
READS
ONE
WAIT FOR ONE REPLICA TO RESPOND
QUORUM WAIT FOR A QUORUM (N/2+1) TO RESPOND
ALL WAIT FOR ALL N REPLICAS TO RESPOND
CONSISTENCY MODEL
DYNAMO INSPIRED
wroe
W=2
READ REPAIR
HINTED HANDOFF
ANTI-ENTROPY
fail
CONSISTENCY MODEL
DYNAMO INSPIRED
wroe
W=2
READ REPAIR Asynchronously checks replicas during
reads and repairs any inconsistencies
HINTED HANDOFF
ANTI-ENTROPY
aad + fix
CONSISTENCY MODEL
DYNAMO INSPIRED
wroe
READ REPAIR
HINTED HANDOFF Sends failed writes to another node
with a hint to re-replicate when the failed node returns
ANTI-ENTROPY
aplica
CONSISTENCY MODEL
DYNAMO INSPIRED
* !ck *
READ REPAIR
HINTED HANDOFF Sends failed writes to another node
with a hint to re-replicate when the failed node returns
ANTI-ENTROPY
apair
CONSISTENCY MODEL
DYNAMO INSPIRED
READ REPAIR
HINTED HANDOFF
ANTI-ENTROPY Manual repair process where nodes
generate Merkle trees (hash trees) to detect and
repair data inconsistencies
apair
DATA MODEL
BIGTABLE INSPIRED
SPARSE MATRIX it’s a hash-map (associative array):
a simple, versatile data structure
SCHEMA-FREE data model, introduces new freedom
and new responsibilities
COLUMN FAMILIES blend row-oriented and column-
oriented structure, providing a high level mechanism
for clients to manage on-disk and inter-node data
locality
DATA MODEL
TERMINOLOGY
KEYSPACE A named collection of column families
(similar to a “database” in MySQL) you only need one and
you can mostly ignore it
COLUMN FAMILY A named mapping of keys to rows
ROW A named sorted map of columns or supercolumns
COLUMN A <name, value, timestamp> triple
SUPERCOLUMN A named collection of columns, for
people who want to get fancy
DATA MODEL
{
“users”: {
“alice”: {
“city”: [“St. Louis”, 1287040737182],
“name”: [“Alice”, 1287080340940],
},
...
},
“locations”: {
},
...
}
column family
key
columns
(name, value, timestamp)
row
{
“users”: {
“alice”: {
“city”: [“St. Louis”, 1287040737182],
“name”: [“Alice”, 1287080340940],
},
...
},
...
}
IT’S A DISTRIBUTED HASH TABLE
WITH A TWIST...
COLUMNS IN ARE STORED TOGETHER ON ONE NODE,
IDENTIFIED BY <keyspace, key>
alice
bob
column family
key
columns
sD3b
3e8A
HASH TABLE
SUPPORTED QUERIES
EXACT MATCH
RANGE
PROXIMITY
ANYTHING THAT’S NOT
EXACT MATCH
COLUMNS
SUPPORTED QUERIES
EXACT MATCH
RANGE
PROXIMITY
{
“users”: {
“alice”: {
“city”: [“St. Louis”, 1287040737182],
“friend-1”: [“Bob”, 1287080340940],
“friend-2”: [“Joe”, 1287080340940],
“friend-3”: [“Meg”, 1287080340940],
“name”: [“Alice”, 1287080340940],
},
...
}
}
friends
LOG-STRUCTURED MERGE
MEMTABLES are in memory data structures that
contain newly written data
COMMIT LOGS are append only files where new
data is durably written
SSTABLES are serialized memtables, persisted to
disk
COMPACTION periodically merges multiple
memtables to improve system performance
CASSANDRA
CONCEPTUAL SUMMARY...
IT’S A DISTRIBUTED HASH TABLE
Gossip based peer-to-peer “ring” with no distinguished nodes and no
single point of failure
Consistent hashing distributes workload and simple replication
strategy for fault tolerance and improved throughput
WITH TUNABLE CONSISTENCY
Based on quorum protocol to ensure consistency
And simple repair mechanisms to stay available during partial system
failures
AND A SIMPLE, SCHEMA-FREE DATA MODEL
It’s just a key-value store
Whose values are multi-dimensional sorted map
ADVANCED CASSANDRA
- A case study -
SPATIAL DATA IN A DHT
A FIRST PASS
THE ORDER PRESERVING PARTITIONER
CASSANDRA’S PARTITIONING
STRATEGY IS PLUGGABLE
Partitioner maps keys to nodes
Random partitioner destroys locality by hashing
Order preserving partitioner retains locality, storing
keys in natural lexicographical order around ring
alice
bob
sam
a z
a
m
h u
ORDER PRESERVING PARTITIONER
EXACT MATCH
RANGE
PROXIMITY
?
On a single dimension
SPATIAL DATA
IT’S INHERENTLY MULTIDIMENSIONAL
1 2
1
2
x
2, 2
DIMENSIONALITY REDUCTION
WITH SPACE-FILLING CURVES
1 2
4 3
Z-CURVE
SECOND ITERATION
Z-VALUE
x
14
GEOHASH
SIMPLE TO COMPUTE
Interleave the bits of decimal coordinates
(equivalent to binary encoding of pre-order
traversal!)
Base32 encode the result
AWESOME CHARACTERISTICS
Arbitrary precision
Human readable
Sorts lexicographically
01101
e
DATA MODEL
{
“record-index”: {
“9yzgcjn0:moonrise hotel”: {
“”: [“”, 1287040737182],
},
...
},
“records”: {
“moonrise hotel”: {
“latitude”: [“38.6554420”, 1287040737182],
“longitude”: [“-90.2992910”, 1287040737182],
...
}
}
}
key
<geohash>:<id>
BOUNDING BOX
E.G., MULTIDIMENSIONAL RANGE
1 2
4 3
Giæe 2 b 3
Giæe 4 b 5
Giæe stuff æ bæmæg box!
SPATIAL DATA
STILL MULTIDIMENSIONAL
DIMENSIONALITY REDUCTION ISN’T PERFECT
Clients must
• Pre-process to compose multiple queries
• Post-process to filter and merge results
Degenerate cases can be bad, particularly for nearest-neighbor
queries
Z-CURVE LOCALITY
Z-CURVE LOCALITY
x
x
Z-CURVE LOCALITY
x
x
Z-CURVE LOCALITY
x
x
o
o
o
o
o
o
o
THE WORLD
IS NOT BALANCED
Credit: C. Mayhew & R. Simmon (NASA/GSFC), NOAA/NGDC, DMSP Digital Archive
TOO MUCH LOCALITY
1 2
4 3
SAN FRANCISCO
TOO MUCH LOCALITY
1 2
4 3
SAN FRANCISCO
TOO MUCH LOCALITY
1 2
4 3
SAN FRANCISCO
I’m sad.
TOO MUCH LOCALITY
1 2
4 3
SAN FRANCISCO
I’m sad.
I’m b~m.
Me bo.
Let’s pky xbox.
A TURNING POINT
HELLO, DRAWING BOARD
SURVEY OF DISTRIBUTED P2P INDEXING
An overlay-dependent index works directly with nodes of the
peer-to-peer network, defining its own overlay
An over-DHT index overlays a more sophisticated data
structure on top of a peer-to-peer distributed hash table
ANOTHER LOOK AT POSTGIS
MIGHT WORK, BUT
The relational transaction management system (which we’d
want to change) and access methods (which we’d have to
change) are tightly coupled (necessarily?) to other parts of
the system
Could work at a higher level and treat PostGIS as a black box
• Now we’re back to implementing a peer-to-peer network with failure
recovery, fault detection, etc... and Cassandra already had all that.
• It’s probably clear by now that I think these problems are more
difficult than actually storing structured data on disk
LET’S TAKE A STEP BACK
EARTH
EARTH
EARTH
EARTH
EARTH, TREE, RING
DATA MODEL
{
“record-index”: {
“layer-name:37.875, -90:40.25, -101.25”: {
“38.6554420, -90.2992910:moonrise hotel”: [“”, 1287040737182],
...
},
},
“record-index-meta”: {
“layer-name:37.875, -90:40.25, -101.25”: {
“split”: [“false”, 1287040737182],
}
“layer-name: 37.875, -90:42.265, -101.25” {
“split”: [“true”, 1287040737182],
“child-left”: [“layer-name:37.875, -90:40.25, -101.25”, 1287040737182]
“child-right”: [“layer-name:40.25, -90:42.265, -101.25”, 1287040737182]
}
}
}
DATA MODEL
{
“record-index”: {
“layer-name:37.875, -90:40.25, -101.25”: {
“38.6554420, -90.2992910:moonrise hotel”: [“”, 1287040737182],
...
},
},
“record-index-meta”: {
“layer-name:37.875, -90:40.25, -101.25”: {
“split”: [“false”, 1287040737182],
}
“layer-name: 37.875, -90:42.265, -101.25” {
“split”: [“true”, 1287040737182],
“child-left”: [“layer-name:37.875, -90:40.25, -101.25”, 1287040737182]
“child-right”: [“layer-name:40.25, -90:42.265, -101.25”, 1287040737182]
}
}
}
DATA MODEL
{
“record-index”: {
“layer-name:37.875, -90:40.25, -101.25”: {
“38.6554420, -90.2992910:moonrise hotel”: [“”, 1287040737182],
...
},
},
“record-index-meta”: {
“layer-name:37.875, -90:40.25, -101.25”: {
“split”: [“false”, 1287040737182],
}
“layer-name: 37.875, -90:42.265, -101.25” {
“split”: [“true”, 1287040737182],
“child-left”: [“layer-name:37.875, -90:40.25, -101.25”, 1287040737182]
“child-right”: [“layer-name:40.25, -90:42.265, -101.25”, 1287040737182]
}
}
}
SPLITTING
IT’S PRETTY MUCH JUST A CONCURRENT TREE
Splitting shouldn’t lock the tree for reads or writes and failures
shouldn’t cause corruption
• Splits are optimistic, idempotent, and fail-forward
• Instead of locking, writes are replicated to the splitting node and the
relevant child[ren] while a split operation is taking place
• Cleanup occurs after the split is completed and all interested nodes are
aware that the split has occurred
• Cassandra writes are idempotent, so splits are too - if a split fails, it is
simply be retried
Split size: A Tunable knob for balancing locality and distributedness
The other hard problem with concurrent trees is rebalancing - we
just don’t do it! (more on this later)
THE ROOT IS HOT
MIGHT BE A DEAL BREAKER
For a tree to be useful, it has to be traversed
• Typically, tree traversal starts at the root
• Root is the only discoverable node in our tree
Traversing through the root meant reading the root for every
read or write below it - unacceptable
• Lots of academic solutions - most promising was a skip graph, but
that required O(n log(n)) data - also unacceptable
• Minimum tree depth was propsed, but then you just get multiple hot-
spots at your minimum depth nodes
BACK TO THE BOOKS
LOTS OF ACADEMIC WORK ON THIS TOPIC
But academia is obsessed with provable, deterministic,
asymptotically optimal algorithms
And we only need something that is probably fast enough
most of the time (for some value of “probably” and “most of
the time”)
• And if the probably good enough algorithm is, you know... tractable...
one might even consider it qualitatively better!
- THE ROOT -
A HOT SPOT AND A SPOF
We have We want
THINKING HOLISTICALLY
WE OBSERVED THAT
Once a node in the tree exists, it doesn’t go away
Node state may change, but that state only really matters
locally - thinking a node is a leaf when it really has children is
not fatal
SO... WHAT IF WE JUST CACHED NODES THAT
WERE OBSERVED IN THE SYSTEM!?
CACHE IT
STUPID SIMPLE SOLUTION
Keep an LRU cache of nodes that have been traversed
Start traversals at the most selective relevant node
If that node doesn’t satisfy you, traverse up the tree
Along with your result set, return a list of nodes that were
traversed so the caller can add them to its cache
TRAVERSAL
NEAREST NEIGHBOR
x
o
o
o
o
TRAVERSAL
NEAREST NEIGHBOR
x
o
o
o
o
TRAVERSAL
NEAREST NEIGHBOR
x
o
o
o
o
KEY CHARACTERISTICS
PERFORMANCE
Best case on the happy path (everything cached) has zero
read overhead
Worst case, with nothing cached, O(log(n)) read overhead
RE-BALANCING SEEMS UNNECESSARY!
Makes worst case more worser, but so far so good
DISTRIBUTED TREE
SUPPORTED QUERIES
EXACT MATCH
RANGE
PROXIMITY
SOMETHING ELSE I HAVEN’T
EVEN HEARD OF
DISTRIBUTED TREE
SUPPORTED QUERIES
EXACT MATCH
RANGE
PROXIMITY
SOMETHING ELSE I HAVEN’T
EVEN HEARD OF
M
U
L
n
P
u

D
I
e
N
S
ø
N
S
!
LIFE OF A REQUEST
THE BIRDS ‘N THE BEES
ELB
service service
worker pool worker pool
index
gate gate
cass
THE BIRDS ‘N THE BEES
ELB
gate
service
worker pool
cass
index
THE BIRDS ‘N THE BEES
ELB
gate
service
worker pool
cass
index
load bakøæg; AWS s~vice
THE BIRDS ‘N THE BEES
ELB
gate
service
worker pool
cass
index
load bakøæg; AWS s~vice
auwæicæøn; f~w~dæg
THE BIRDS ‘N THE BEES
ELB
gate
service
worker pool
cass
index
load bakøæg; AWS s~vice
auwæicæøn; f~w~dæg
busæ«s logic - b~ic validæøn
THE BIRDS ‘N THE BEES
ELB
gate
service
worker pool
cass
index
load bakøæg; AWS s~vice
auwæicæøn; f~w~dæg
busæ«s logic - b~ic validæøn
ac~d st~age
THE BIRDS ‘N THE BEES
ELB
gate
service
worker pool
cass
index
load bakøæg; AWS s~vice
auwæicæøn; f~w~dæg
busæ«s logic - b~ic validæøn
ac~d st~age
busæ«s logic - st~age/ææxæg
THE BIRDS ‘N THE BEES
ELB
gate
service
worker pool
cass
index
load bakøæg; AWS s~vice
auwæicæøn; f~w~dæg
busæ«s logic - b~ic validæøn
ac~d st~age
busæ«s logic - st~age/ææxæg
aw«ome sauce f~ qoryæg
ELB

Traffic management

Control which AZs are serving traffic

Upgrades without downtime

Able to remove an AZ, upgrade, test,
replace

API-level failure scenarios

Periodically runs healthchecks on nodes

Removes nodes that fail
GATE

Basic auth

HTTP proxy to specific services

Services are independent of one another

Auth is Decoupled from business logic

First line of defense

Very fast, very cheap

Keeps services from being overwhelmed
by poorly authenticated requests
RABBITMQ

Decouple accepting writes from
performing the heavy lifting

Don’t block client while we write to db/
index

Flexibility in the event of degradation
further down the stack

Queues can hold a lot, and can keep
accepting writes throughout incident

Heterogenous consumers - pass the same
message through multiple code paths easily
AWS
EC2
• Security groups
• Static IPs
• Choose your data center
• Choose your instance type
• On-demand vs. Reserved
ELASTIC BLOCK SUPPORT
• Storage devices that can be anywhere from 1GB to 1TB
• Snapshotting
• Automatic replication
• Mobility
ELASTIC LOAD BALANCING
• Distribute incoming traffic
• Automatic scaling
• Health checks
SIMPLE STORAGE SERVICE
• File sizes can be up to 5TBs
• Unlimited amount of files
• Individual read/write access credentials
RELATIONAL DATABASE SERVICE
• MySQL in the cloud
• Manages replication
• Specify instance types based on need
• Snapshots
CONFIGURATION
MANAGEMENT
WHY IS THIS NECESSARY?
• Easier DevOps integration
• Reusable modules
• Infrastructure as code
• One word: WINNING
POSSIBLE OPTIONS
SAMPLE MANIFEST
# /root/learning-manifests/apache2.pp
package {
'apache2':
ensure => present;
}

file {
'/etc/apache2/apache2.conf':
ensure => file,
mode => 600,
notify => Service[‘apache2’],
source => '/root/learning-manifests/apache2.conf',
}

service {
'apache2':
ensure => running,
enable => true,
subscribe => File['/etc/apache2/apache2.conf'],
}
AN EXAMPLE
TERMINAL
CONTINUOUS
INTEGRATION
GET ‘ER DONE
• Revision control
• Automate build process
• Automate testing process
• Automate deployment
Local code changes should result in production
deployments.
DON’T FORGET TO DEBIANIZE
• All codebases must be debianized
• Open source project not debianized yet, fork the repo and
do it yourself!
• Take the time to teach others
• Debian directories can easily be reused after a simple search
and replace
REPOMAN
HTTPS://GITHUB.COM/SYNACK/REPOMAN.GIT
repoman upload myrepo sg-example.tar.gz
repoman promote development/sg-example staging
HUDSON
MAINTAINING MULTIPLE
ENVIRONMENTS
• Run unit tests in a development environment
• Promote to staging
• Run system tests in a staging environment
• Run consumption tests in a staging environment
• Promote to production
Congratz, you have now just automated yourself
out of a job.
THE MONEY MAKER
Github Plugin
TYING IT ALL TOGETHER
TERMINAL
Flume is a distributed, reliable and available
service for efficiently collecting, aggregating and
moving large amounts of log data.
FLUME
syslog on steriods
DATA-FLOWS
AGENTS
i-192df98
Physical Host
twitter_stream
hdfs_writer
tail
Logical Nodes
collectorSink("hdfs://namenode.sg.com/bogus/", "logs")
tail(“/var/log/nginx/access.log”)
twitter(“dsmitts”, “mypassword”, “url”)
agentSink(35853)
collectorSource(35853)
agentSink(35853)
Source and Sink
RELIABILITY
END-TO-END
STORE ON FAILURE
BEST EFFORT
Custom Decorators
GETTIN’ JIGGY WIT IT
HOW DO WE USE IT?
PERSONAL EXPERIENCES
• #flume
• Automation was gnarly
• Its never a good day when Eclipse is involved
• Resource hog (at first)
MARCH 2011
We’a buildæg kick-~s bols f~ ævaop~s b ææx,
æt~pat, ~ cosume dæa conectm b a locæøn

Sign up to vote on this title
UsefulNot useful