#43 Get More Refcardz! Visit refcardz.

com

Scalability &
CONTENTS INCLUDE:
n
Overview
n
Implementing Scalable Systems

High Availability
n
Caching Strategies
n
Clustering
n
Redundancy and Fault Tolerance
n
Hot Tips and more...
By Eugene Ciurana
Scalability, continued
Overview
the amount of main memory to host more virtual servers in the
Scalability, High Availability, and Performance same hardware.
The terms scalability, high availability, performance, and
mission-critical can mean different things to different Virtual Node 3
organizations, or to different departments within an Virtual Node 2

organization. They are often interchanged and create

Virtual Node 2
confusion that results in poorly managed expectations,
implementation delays, or unrealistic metrics. This Refcard Virtual Node 1
Scales up
provides you with the tools to define these terms so that Virtual Node 1

your team can implement mission-critical systems with well-

understood performance goals. Virtual Node 0
Virtual Node 0

Scalability Dual Core Dual Core

Its the property of a system or application to handle bigger Single Processor
16 MB RAM
Dual Processor
32 MB RAM
amounts of work, or to be easily expanded, in response to
increased demand for network, processing, database access or Figure 2: Virtualization Figure 2 - Virtualization
www.dzone.com

file system resources.

Horizontal scalability
A system scales horizontally, or out, when its expanded by
adding new nodes with identical functionality to existing
ones, redistributing the load among all of them. SOA
systems and web servers scale out by adding more servers to
a load-balanced network so that incoming requests may be
distributed among all of them. Cluster is a common term for
describing a scaled out processing system.
Load Balancer
Scalability & High Availability
High Availability
Availability describes how well a system provides useful
resources over a set period of time. High availability
guarantees an absolute degree of functional continuity within
a time window expressed as the relationship between uptime
and downtime.
A = 100 (100*D/U), D ::= unplanned downtime, U ::= uptime;
D, U expressed in minutes
Uptime and availability dont mean the same thing. A
system may be up for a complete measuring period, but
may be unavailable due to network outages or downtime in

related support systems. Downtime and unavailability are

synonymous.
Node Node Node

Scales Cheat Like a Pro

40+ Professional
out Brought to you by...
#8
Get More Refcarz! Visit refcardz.com

CONTENTS INCLUDE:

Cheat Sheets.
Chain of Responsibility

Design Patterns
n

n
Command Inspired
n
Interpreter by the
n
Iterator GoF
Bestseller
n
Mediator
n
Observer By Jason McDonald
n
Template Method and more...

Load Balancer
Choose your
Chain of Responsibility, continued
ABOUT DESIGN PATTERNS
Use n Multiple objects may handle a request and the handler doesnt have to
When be a specific object.
This Design Patterns refcard provides a quick reference to the n A set of objects should be able to handle a request with the handler
original 23 Gang of Four (GoF) design patterns, as listed in determined at runtime.
n A request not being handled is an acceptable potential outcome.
the book Design Patterns: Elements of Reusable Object-
Oriented Software. Each pattern includes class diagrams, Example Exception handling in some languages implements this pattern. When an
exception is thrown in a method the runtime checks to see if the method
explanation, usage information, and a real world example.

Creational Patterns: Used to construct objects such
that they can be decoupled from their implementing
favorite topics.
has a mechanism to handle the exception or if it should be passed up the
call stack. When passed up the call stack the process repeats until code to
handle the exception is encountered or until there are no more parent
objects to hand the request to.

system.

Structural Patterns: Used to form large object COMMAND Object Behavioral

structures between many disparate objects.

Behavioral Patterns: Used to manage algorithms, Client Invoker

relationships, and responsibilities between objects. ConcreteCommand

+execute( )
Object Scope: Deals with object relationships that can

Spring Configuration
w w w.dzone.com

be changed at runtime.

Node Node Node Node Class Scope: Deals with class relationships that can be
changed at compile time.
Receiver
Command
+execute ( )

Purpose Encapsulates a request allowing it to be treated as an object. This allows

jQuery Selectors
the request to be handled in traditionally object based relationships such
C Abstract Factory S Decorator C Prototype as queuing and callbacks.

S S Facade S Proxy Use You need callback functionality.

Adapter n

When n Requests need to be handled at variant times or in variant orders.

S Bridge C Factory Method B Observer n A history of requests is needed.
n The invoker should be decoupled from the object handling the invocation.

Windows Powershell
C S Flyweight C Singleton
Builder
Example Job queues are widely used to facilitate the asynchronous processing
B B Interpreter B State of algorithms. By utilizing the command pattern the functionality to be
Chain of

Figure 1: Clustering
executed can be given to a job queue for processing without any need
Responsibility B Iterator B Strategy for the queue to have knowledge of the actual implementation it is
invoking. The command object that is enqueued implements its particular

Figure 1 - Clustering
B Command

EJB 3
B Mediator B Template Method algorithm within the confines of the interface the queue is expecting.

S Composite B Memento B Visitor

CHAIN OF RESPONSIBILITY Object Behavioral

successor

Netbeans IDE JavaEditor

Vertical scalability
Design Patterns

<<interface>>
Client Handler

Getting Started with Eclipse

+handlerequest( )

A system scales vertically, or up, when its expanded by adding

ConcreteHandler 1 ConcreteHandler 2

Very First Steps in Flex

+handlerequest( ) +handlerequest( )

Purpose Gives more than one object an opportunity to handle a request by linking
receiving objects together.

processing, main memory, storage, or network interfaces to DZone, Inc. | www.dzone.com

And Many More...
a node to satisfy more requests per system. Hosting services
companies scale up by increasing the number of processors or
Refcardz.com
DZone, Inc. | www.dzone.com
 2
Scalability & High Availability

High Availability, continued Implementing Scalable Systems, continued

Measuring Availability Load Balancing
Vendors define availability as a given number of nines like Load balancing is a technique for minimizing response time
in Table 1, which also describes the number of minutes or and maximizing throughput by spreading requests among two
seconds of estimated downtime in relation to the number of or more resources. Load balancers may be implemented in
minutes in a 365-day year, or 525,600, making U a constant for dedicated hardware devices, or in software. Figure 3 shows
their marketing purposes. how load-balanced systems appear to the resource consumers
as a single resource exposed through a well-known address.
Availability % Downtime in Minutes Downtime per Year Vendor Jargon The load balancer is responsible for routing requests to
90 52,560.00 36.5 days one nine available systems based on a scheduling rule.
99 5,256.00 4 days two nines
Consumer
99.9 525.60 8.8 hours three nines

99.99 52.56 53 minutes four nines R = request

Rn
n = sequence number
99.999 5.26 5.3 minutes five nines Load Balancer
74.0.125.28
99.9999 0.53 32 seconds six nines

Table 1: Availability as a Percentage of Total Yearly Uptime R1 R3 R2

Node Node Node Node

Analysis 192.168.202.55 192.168.202.66 192.168.202.67 192.168.202.69

High availability depends on the expected uptime defined for Figure 3: Load BalancerFigure 3 - Load Balancer
system requirements; dont be misled by vendor figures. The

Scheduling rules are algorithms for determining which server
meaning of having a highly available system and its measurable must service a request. Web applications and services are
uptime are a direct function of a Service Level Agreement. balanced by following round robin scheduling rules. Caching
Availability goes up when factoring planned downtime, such as pools are balanced by applying frequency rules and expiration
a monthly 8-hour maintenance window. algorithms. Applications where stateless requests arrive with
The cost of each additional nine of availability can grow a uniform probability for any number of servers may use a
exponentially. Availability is a function of scaling the systems pseudo-random scheduler. Applications like music stores,
up or out and implementing system, network, and storage where some content is statistically more popular, may use
redundancy. asymmetric load balancers to shift the larger number popular
Service Level Agreement (SLA) requests to higher performance systems, serving the rest of the
SLAs are the negotiated terms that outline the obligations of requests from less powerful systems or clusters.
the two parties involved in delivering and using a system, like: Persistent Load Balancers
System type (virtual or dedicated servers, shared hosting) Stateful applications require persistent or sticky load
Levels of availability balancing, where a consumer is guaranteed to maintain a
Minimum session with a specific server from the pool. Figure 4 shows a
Target sticky balancer that maintains sessions from multiple clients.
Uptime Figure 5 shows how the cluster maintains sessions by sharing
Network data using a database.
Power
Maintenance windows Consumer Consumer Consumer
Serviceability
Performance and metrics
Billing
Sticky Load
SLAs can bind obligations between two internal organizations Balancer
74.0.125.28
(e.g. the IT and e-commerce departments), or between the
organization and an outsourced services provider. The SLA
establishes the metrics for evaluating the system performance, Node Node
192.168.202.55 192.168.202.69
and provides the definitions for availability and the scalability Node
192.168.202.66
Node
192.168.202.67
targets. It makes no sense to talk about any of these topics
unless an SLA is being drawn or one already exists. Figure 4: Sticky Load Balancer
Figure 4 - Sticky Load Balancer

Common Features of a Load Balancer
Implementing Scalable Systems Asymmetric load distribution assigns some servers to handle
a bigger load than others
SLAs determine whether systems must scale up or out. They Content filtering inbound or outbound
also drive the growth timeline. A stock trading system must Distributed Denial of Service (DDoS) attack protection
scale in real-time within minimum and maximum availability Firewalling
levels. An e-commerce system, in contrast, may scale in during Payload switching send requests to different servers
the slow months of the year, and scale out during the retail based on URI, port, and/or protocol
holiday season to satisfy much larger demand. Priority activation adds standing by servers to the pool

DZone, Inc. | www.dzone.com

 3
Scalability & High Availability

Load Balancing, continued Caching Strategies, continued

Rate shaping ability to give different priority to different Application Caching
traffic Implicit caching happens when there is little or no
Scripting reduces human interaction by implementing programmer participation in implementing the caching.
programming rules or actions The program executes queries and updates using its native
SSL offloading hardware-assisted encryption frees web API and the caching layer automatically caches the
server resources requests independently of the application. Example:
TCP buffering and offloading throttle requests to servers Terracotta (http://www.terracotta.org).
in the pool
Explicit caching happens when the programmer
Consumer participates in implementing the caching API and may also
implement the caching policies. The program must import
Load Balancer
the caching API into its flow in order to use it. Examples:
74.0.125.28
memcached (http://www.danga.com/memcached) and
Oracle Coherence (http://coherence.oracle.com).

Node
192.168.202.55
Node
192.168.202.66
Node
192.168.202.67
Node
192.168.202.69 In general, implicit caching systems are specific to a platform
or language. Terracotta, for example, only works with Java and
JVM-hosted languages like Groovy. Explicit caching systems
may be used with many programming languages and across
Session
Data
multiple platforms at the same time. memcached works with
every major programming language, and Coherence works with
Figure 5: Database Sessions
Figure 5 - Database Sessions Java, .Net, and native C++ applications.

Web Caching
Caching Strategies
Web caching is used for storing documents or portions of
documents (particles) to reduce server load, bandwidth usage
Stateful load balancing techniques require data sharing among
and lag for web applications. Web caching can exist on the
the service providers. Caching is a technique for sharing
browser (user cache) or on the server, the topic of this section.
data among multiple consumers or servers that are expensive
Web caches are invisible to the client may be classified in any
to either compute or fetch. Data are stored and retrieved
of these categories:
in a subsystem that provides quick access to a copy of the
frequently accessed data. Web accelerators: they operate on behalf of the
Caches are implemented as an indexed table where a unique server of origin. Used for expediting access
key is used for referencing some datum. to heavy resources, like media files. Content
Consumers access data by checking distribution networks (CDNs) are an
(hitting) the cache first and retrieving example of web acceleration caches;
the datum from it. If its not there Akamai, Amazon S3, Nirvanix are
(cache miss), then the costlier retrieval Begin examples of this technology.
operation takes place and the consumer
Proxy caches: they serve requests to a
or a subsystem inserts the datum to the query update
Query? group of clients that may all have
cache.
Fetch
access to the same resources. They
Write Policy datum from
cache
can be used for content filtering and
The cache may become stale if the Update for reducing bandwidth usage. Squid,
datum in
backing store changes without updating database Apache, ISA server are examples
datum is no
the cache. A write policy for the cache None of this technology.
defines how cached data are refreshed. yes Invalidate cache
Some common write policies include: Query Distributed Caching
Write-through: every write to the datum from
database Caching techniques can be implemented
cache follows a synchronous write to Add or update across multiple systems that serve
datum to cache
the backing store Add datum to
requests for multiple consumers and
Write-behind: updated entries are cache from multiple resources. These are
marked in the cache table as dirty known as distributed caches, like the
and its updated only when a dirty setup in Figure 7. Akamai is an example
datum is requested. Use datum of a distributed web cache. memcached
in app
No-write allocation: only read is an example of a distributed application
requests are cached under the cache.
End
assumption that the data wont
change over time but its expensive
Figure 6: Caching Usage
Figure Pattern
6 - Caching Usage Pattern
to retrieve
DZone, Inc. | www.dzone.com
 4
Scalability & High Availability

Caching Strategies, continued Clustering, continued

High Availability Cluster(Active/Passive): Improve services
Consumer
availability by providing uninterrupted service through
redundant nodes that eliminate single points of failure.
Load Balancer
74.0.125.28 High availability clusters require two nodes at a minimum, a
heartbeat to detect that all nodes are ready, and a routing
mechanism that will automatically switch traffic if the main
Node Node Node Node
192.168.202.55 192.168.202.66 192.168.202.67 192.168.202.69 node fails.
Consumer

Load Balanced Conguration or Datagram

Cache 0 Cache 1 Cache 2 Cache 3 Master

Load Balancer

Database

Node Node Node Node

Figure 7: Distributed Cache
Figure 7 - Distributed Cache

Clustering Load Balancer

A cluster is a group of computer systems that work together

Node Node Node Node
to form what appears to the user as a single system. Clusters
are deployed to improve services availability or to increase Figure 10: Grid Figure 10 - Grid
computational or data manipulation performance. In terms
Grid: Process workloads defined as independent jobs that
of equivalent computing power, a cluster is more cost- dont require data sharing among processes. Storage or
effective than a monolithic system with the same performance network may be shared across all nodes of the grid, but
characteristics. intermediate results have no bearing on other jobs progress
The systems in a cluster are interconnected over high-speed or on other nodes in the grid, such as a Cloudera Map Reduce
local area networks like gigabit Ethernet, fiber distributed data cluster (http://www.cloudera.com).
interface (FDDI), Infiniband, Myrinet, or other technologies.
Consumer

Consumer
Master

Load Balancer
74.0.125.28

Node Node Node Node

Node Node Node Node

192.168.202.55 192.168.202.66 192.168.202.67 192.168.202.69
Node Node Node Node

Figure 8: Load Balancing Cluster

Figure 8 - Load Balancing Cluster
Figure 11: Computational Clusters
Figure 11- Computational Cluster
Load-Balancing Cluster (Active/Active): Distribute the load
among multiple back-end, redundant nodes. All nodes in the Computational Clusters: Exeute processes that require raw
cluster offer full-service capabilities to the consumers and are computational power instead of executing transactional
active at the same time. operations like web or database clusters. The nodes are tightly
coupled, homogeneous, and in close physical proximity. They
often replace supercomputers.
Consumer

Redundancy and fault tolerance

Router
74.0.125.28
Redundant system design depends on the expectation that
any system component failure is independent of failure in the
Active Node heartbeat Failover Node
other components.
192.168.202.55 192.168.202.69
Fault tolerant systems continue to operate in the event of
State Data component or subsystem failure; throughput may decrease but
Cache
overall system availability remains constant. Faults in hardware
replication or clustered database
or software are handled through component redundancy. Fault
Failover
Database
Database tolerance requirements are derived from SLAs. The implemen-
Figure 9 - High Availability Cluster tation depends on the hardware and software components,
Figure 9: High Availability Cluster and on the rules by which they interact.

DZone, Inc. | www.dzone.com

 5
Scalability & High Availability

Redundancy and Fault Tolerance, continued Cloud Computing, continued

Fault Tolerance SLA Requirements Amazon S3
http://media.company.com
Google App Engine
http://www.company.com
No single point of failure redundant components ensure
Datastore Datastore
continuous operation and allow repairs without disruption
of service PNG PNG PNG PNG Node Node Node Node

Fault isolation problem detection must pinpoint the PNG PNG PNG PNG Node Node Node Node

specific faulty component

Fault propagation containment faults in one component
must not cascade to others User

Reversion mode set the system back to a known state

Redundant clustered systems can provide higher availability, Firewall

better throughput, and fault tolerance. The A/A cluster

in Figure 12 provides uninterrupted service for a scalable, services.company.com Enterprise Service Bus

stateless application.
Node Node Node Node
Consumer

Legacy Back-end
Load Balancer
74.0.125.28

Enterprise
Database
Replacement
Node Node Node Node
Node
192.168.202.55 192.168.202.66 192.168.202.67 192.168.202.69 Figure 15 - Cloud Computing Conguration
192.168.202.53
Figure 14: Cloud Computing Configuration
Figure 12: A/A Full Tolerance andFault
Figure 12 - A/A Recovery
Tolerance and Recovery
Cloud Services Types
Some stateful applications may only scale up; the A/P cluster Web services Salesforce
in Figure 13 provides uninterrupted service and disaster com, USPS, Google Maps
recovery for such an application. A/A configurations provide Service platforms Google App Engine, Amazon Web
failure transparency. A/P configurations may provide failure Services (EC2, S3, Cloud Front), Nirvanix, Akamai,
transparency at a much higher cost because automatic failure MuleSource
detection and reconfiguration are implemented through a
feedback control system, which is more expensive and trickier
Fault Detection Methods
Fault detection methods must provide enough information
to implement.
to isolate the fault and execute automatic or assisted failover
action. Some of the most common fault detection methods
include:

Built-in Diagnostics
Protocol Sniffers
Sanity Checks
Watchdog Checks

Criticality is defined as the number of consecutive faults

reported by two or more detection mechanisms over a fixed
time period. A fault detection mechanism is useless if it
reports every single glitch (noise) or if it fails to report a real
fault over a number of monitoring periods.

Figure 13: A/P Fault Tolerance and Recovery System Performance

Enterprise systems most commonly implement A/P fault
tolerance and recovery through fault transparency by diverting
services to the passive system and bringing it on-line as soon
as possible. Robotics and life-critical systems may implement
probabilistic, linear model, fault hiding, and optimization
control systems instead.
Cloud Computing
Cloud computing describes applications running on
distributed, computing resources owned and operated by a
third-party.
End-user apps are the most common examples. They utilize
the Software as a Service (SaaS) and Platform as a Service
(PaaS) computing models.
Performance refers to the system throughput under a
particular workload for a defined period of time. Performance
testing validates implementation decisions about the system
throughput, scalability, reliability, and resource usage.
Performance engineers work with the development and
deployment teams to ensure that the systems non-functional
requirements like SLAs are implemented as part of the system
development lifecycle. System performance encompasses
hardware, software, and networking optimizations.
Performance testing efforts must begin at the
Hot same time as the development project and
Tip continue through deployment

DZone, Inc. | www.dzone.com

 6
Scalability & High Availability

System Performance, continued System Performance, continued

The performance engineers objective is to detect bottlenecks Software Testing Tools
early and to collaborate with the development and deployment There are many software performance testing tools in the
teams on eliminating them. market. Some of the best are released as open-source
System Performance Tests software. A comprehensive list of those is available from:
Performance specifications are documented along with the http://www.opensourcetesting.org/performance.php
SLA and with the system design. Performance troubleshooting These include Java, native, PHP, .Net, and other languages
includes these types of testing: and platforms.
Endurance testing- identifies resource leaks under the
continuous, expected load. Staying Current
Load testing determines the system behavior under a Do you want to know about specific projects and cases where
specific load.
Spike testing shows how the system operates in scalability, high availability, and performance are the hot
response to dramatic changes in load. topic? Join the scalability newsletter:
Stress testing identifies the breaking point for the http://eugeneciurana.com/scalablesystems
application under dramatic load changes for extended
periods of time.

AB O U T T HE A U T H O R RE C O MME N D E D B O O K
Eugene Ciurana Developing with Google App Engine
Eugene Ciurana is an open-source evangelist who introduces Google App Engine, a platform
specializes in the design and implementation of mission- that provides developers and users with
critical, high-availability large scale systems. As Director the infrastructure that Google itself uses for
of Systems Infrastructure, he and his team designed and developing and deploying massively scalable
built a 100% SOA and cloud system that enables millions applications. Using Python as the primary
of Internet-ready educational and handheld products and services. As chief programming tool, Developing with Google
liaison between Walmart.com Global and the ISD Technology Council, he led App Engine makes it easy to implement
the official adoption of Linux and other open-source technologies at Walmart scalability and high performance features like
Stores Information Systems Division. Hes also designed high performance distributed databases, clustering, stateless
systems for major financial institutions and many Fortune 100 companies in applications, and sophisticated data caching.
the United States and Europe.
Publications Web site
n Developing with the Google App Engine http://eugeneciurana.com BUY NOW
n Best Of Breed: Building High Quality Systems, books.dzone.com/books/google-app-engine
Within Budget, On Time, and Without Nonsense
n The Tesla Testament: A Thriller

Bro
ugh
t to
you
by...
Professional Cheat Sheets You Can Trust
tt erns Exactly what busy developers need:
n Pa
ld
ona

sig son
McD
simple, short, and to the point.
De
Ja
By

#8
ired
Insp e
by th
GoF ller con
tinu
ed
ndle
r doe
snt
have
to

James Ward, Adobe Systems

ity,
r
E: e ha ndle
d th
LUD Best
se
ns ibil st an ith th
e ha

Upcoming Titles Most Popular

IN C que st w
ility spo
re
TS e le a req
ue
ome.
EN nsib
of R
nd le a
ay ha outc
NT spo sm hand tial hen
an
CO f Re in ject le to oten
Cha
o . le p .W
le ob bject be ab tern ethod
m

in tab
Cha d ultip ecific o should cep pat
this if the m up the
man
n M
an ac
z.co

n
sp s ents
be a ject ime. d is see passed

Download Now
Com reter of ob at runt handle plem ks to de to

RichFaces Spring Configuration

n
Use se t im ec b e co
rp
n A ined ges ch ld til t
Inte Whe
n
erm being ngua ime shou peats un paren
not e la the runt or if it
tor det s re ore
a rd

...
n
uest som tion proces e no m
Itera tor ore req
n A g in metho
d
cep
dm ndlin
e e ar

Agile Software Development jQuery Selectors

dia e ex ack th
n
a ther
Me rver d an the n ha rown in ndle th ll st until
re f c

tho e to ptio th
Exce ion is sm to ha up th tered or
e ca
n

Ob
se Me S renc ral

Refcardz.com
plate RN refe listed in mp
le ce pt ni
echa n pa ssed co un avio
Beh
TTE
n
ex
is en
ick
BIRT Windows Powershell
am
Tem Exa he .
A s has ack. W tion uest to ject
NP a qu s, a ct- cep Ob
it

n
st q
IG e s e rn b je call e ex e re
vid patt able O le th nd th
DES
! V is

pro s, hand s to ha
UT ard ign us ram .
des diag
JSF 2.0 Dependency Injection with EJB 3
ct
ABO refc oF) f Re le obje
erns ts o lass exa
mp oke
r
Patt ur (G lemen es c Inv
arz

n F o d rl d h
Des
ig go
f s: E inclu al w
o suc
Th is G a n
l 23 sign Pa
tt e rn e rn
patt , and a
re je ts
t ob mentin
c g AN
D
Adobe AIR Netbeans IDE JavaEditor
c

a h c M M
ef

in c u
orig kD
e
re.
Ea tion ons
tr ple CO ma
nd
boo Softwa rma to c their im om
BPM&BPMN Getting Started with Eclipse
re R

the info ed Clie

nt
cre
teC
d
ente on, us
age : Us d from Con nd
Ori r ns ct () ma
ti atte uple bje cute Com )
lana al P e deco eo +exe
Flex 3 Components Very First Steps in Flex
s
Mo

( low
e x p o n la rg cute is al ch
ati an b rm ts. +exe . Th
bject nship
s su
Cre y c fo je c an o
t th
e
ed
to ob ms, d as ed rela
tio
Get

tha : Us arate rith eate as

s te m.
tt e r ns n y disp
g e algo je c ts. e r
be tr ject b
it to lly ob
sy Pa a b g
ana
iv
ral en m en o Rece
win
allo traditi
ona
to m betwe
s.
ctu twe sed
uest
req ndled in nt or
der
Stru s be
stru
cture ttern
l Pa d respo
s: U
nsib
ilitie
s

nsh
ips
that
can psu
Enca quest
the
re
late
sa

and
e ha
to b llbacks
ca
.

nalit
y.
riant
times
or in
varia

ling
the
invo
catio

ing
n.
DZone, Inc. ISBN-13: 978-1-934238-46-2
ra o pose uing nctio led at va hand
io n ti ct cess be
av ,a la P ur
as q
ue
ack
fu je pro
Beh nships t re
1251 NW Maynard
e nd ob

rela
tio
with
ob je c
s th
at c
a n b ed
You ne s need
st
callb
to
que
b e ha eded.
ne
sts is couple n
d fro m the

e th
ynch
e as the fu
ro nous nality
nctio
itho
to
y ne
ut an is
ed
ISBN-10: 1-934238-46-5
eals
it
ship Reque y of re de ar
ld be litat pattern ssing w tation articul
e: D tion faci

50795
or
e. Use
n

A hist shou d ce en p
ed to mman for pro implem ents its ting.
cop runtim rela type
ker
Cary, NC 27513
n
S s Whe invo y us
n
s co al m pec
jec t at cla Pro
to Th e
w id el th e ue ue
ac tu
d im
p le ex
are utilizing a job q of the ue is
Ob ged with
n
C ues ueue e que
han eals me.
y dge enq
que s. B en to th
e c : D P roxy Job orithm be giv knowle that is terface
b e ti
cop
g n ct
pile r S in
rato er mp
le of al ed ca to have d obje of the
ss S at com serv
888.678.0399
Exa ut
Deco nes
an
.com

Ob exec e queue comm

Cla d S B th e confi
nge de leto
n for
king
. Th
ithin
the
cha tory Faca od Sing invo hm w

DZone communities deliver over 4 million pages each month to ract

Fac S
Meth C algo
rit

919.678.0300
one

Abst tory
C C
Fac
B
State
t
pte
r eigh y
teg
Ada Flyw Stra od
z

Meth
more than 2 million software developers, architects and decision
S S r B
w. d

e rp rete plate
ridg
Refcardz Feedback Welcome
B B
Inte Tem
S B
er tor or
ww

Build Itera Visit

$7.95

C B r B

makers. DZone offers something for everyone, including news, diato

f
in o ral
refcardz@dzone.com
ty Me
Cha nsibili avio
B o B
ento Beh
Resp m ject
d Me Ob
m man B

tutorials, cheatsheets, blogs, feature articles, source code and more.

C o
Y
9 781934 238462
B

NSIB
ILIT
S
Com
po si te

P O succ
ess
or Sponsorship Opportunities
RES
DZone is a developers dream, says PC Magazine.
F
CH
AIN
O
ace
>> sales@dzone.com
terf r
<<in andle
H st ( )

Copyright 2009 DZone, Inc. All rights reserved.

Clie
nt
No part of this publication
dle
r2
d lere
que
may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical,
+ha
n
Version 1.0
Han
photocopying, or otherwise, without prior written permission Cof the
oncr
ete publisher.
que
st ( ) Reference: Developing with Google App Engine, Eugene Ciurana, APress, 2009
s

re
ndle
+ha
ern

1 ki ng
ler by lin .c o
m
and uest one
teH req z
cre () le a w.d
Con uest hand | ww