FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN ENTERPRISE SOFTWARE DEVELOPMENT

Scalability
eMag Issue 11 - April 2014

Interview with Raffi Krikorian

on Twitter’s Infrastructure
Raffi Krikorian, Vice President of Platform Engineering at Twitter,
gives an insight on how Twitter prepares for unexpected traffic peaks
and how system architecture is designed to support failure. PAGE 4

INTERVIEW: ADRIAN COCKCROFT ON HIGH AVAILABILITY,

BEST PRACTICES, AND LESSONS LEARNED IN THE CLOUD P. 9
TO EXECUTION PROFILE OR TO MEMORY PROFILE? THAT IS THE QUESTION. P. 12
VIRTUAL PANEL: USING JAVA IN LOW LATENCY ENVIRONMENTS P. 16
RELIABLE AUTO-SCALING USING FEEDBACK CONTROL P. 25
 Contents

Interview with Raffi Krikorian on Twitter’s Infrastructure Page 4

Raffi Krikorian, Vice President of Platform Engineering at Twitter, gives an insight on how
Twitter prepares for unexpected traffic peaks and how system architecture is designed to
support failure.

Interview: Adrian Cockcroft on High Availability,

Best Practices, and Lessons Learned in the Cloud Page 9
Netflix is a widely referenced case study for how to effectively operate a cloud application
at scale. While their hyper-resilient approach may not be necessary at most organizations,
Netflix has advanced the conversation about what it means to build modern systems. In this
interview, InfoQ spoke with Adrian Cockcroft who is the Cloud Architect for the Netflix
platform.

To Execution profile or to Memory Profile?

That is the question. Page 12
There are times when memory profiling will provide a clearer picture than execution
profiling to find execution hot spots. In this article Kirk Pepperdine talks through some
indicators for determining when to use which kind of profiler.

Virtual Panel: Using Java in Low Latency Environments Page 16

Java is increasingly being used for low latency work where previously C and C++ were the
de-facto choice. InfoQ brought together four experts in the field to discuss what is driving the
trend, and some of the best practices when using Java in these situations.

Reliable Auto-Scaling using Feedback Control Page 25

Philipp K. Janert explains how to reliably auto-scale systems using a reactive approach based
on feedback control which provides a more accurate solution than deterministic or rule-
based ones.

Interview with Raffi Krikorian on
Twitter’s Infrastructure
by Xuefeng Ding
Twitter’s Raffi Krikorian gives insight on how the company prepares for unexpected
traffic peaks and how system architecture is designed to support failure.
InfoQ: Hi, Raffi. Would you please introduce yourself
to the audience and the readers of InfoQ?
Raffi: Sure. My name is Raffi Krikorian. I’m the
vice-president of platform engineering at Twitter.
We’re the team that runs basically the backend
infrastructure for all of Twitter.
InfoQ: With the help of Castle in the Sky, Twitter
created a new peak tweets-per-second record. How
does Twitter deal with unpredictable peak traffic?
Raffi: What you’re referring to is the Castle in the
Sky event, which is what we call it internally. That
was a television show that aired in Tokyo. We set
our new record of around 34,000 tweets a second
coming into Twitter during that event. Normally,
Twitter experiences something on order of 5,000 to
10,000 tweets a second, so this is pretty far out of
our standard operating bounds. I think it says a few
things about us. I think it says how Twitter reacts to
the world at large, like things happen in the world and
they get reflected on Twitter.
So the way that we end up preparing for something
like this is really years of work beforehand. This type
of event could happen at any time without real notice.
So we do load tests against the Twitter infrastructure.
Contents Page 4
Scalability / eMag Issue 11 - April 2014
We’d run those on the order of every month – I don’t
know what the exact schedule is these days – and
then we do analyses of every single system at Twitter.
When we build architecture and systems at Twitter,
we look at the performance of all those systems
on a weekly basis to really understand what the
theoretical capacity of the systems looks like,
right now on a per-service basis, and then we try
to understand what the theoretical capacity looks
like overall. From that, we can decide whether we
have the right number of machines in production
at any given time or whether we need to buy more
computers, and we can have a rational conversation
on whether or not the system is operating efficiently.
So if we have certain services, for example, that
can only take half the number of requests a second
as other services, we should look at those and
understand architecturally: are they performing
correctly or do we need to make a change?
So for us, the architecture to get to something like the
Castle in the Sky event is a slow evolutionary process.
We make a change, we see how that change reacts
and how that change behaves in the system, and we
make a decision on the slow-rolling basis of whether
or not this is acceptable to us. We make a tradeoff,

like do we buy more machinery or do we write new
software in order to withstand this?
While we never have experienced an event like Castle
in the Sky before, some of our load tests have pushed
us to those limits already so we were comfortable
when it happened in real life. We’re like, “Yes, it
actually worked.”
InfoQ: Are there any emergency plans in Twitter? Do
you practice for unusual times, such as shut down
some servers or switches?
Raffi: Yeah. We do two different things, basically, as
our emergency planning – maybe three, depending
on how you look at it. Every system is carefully
documented for what would turn it on and what
would turn it off. We have what we call “runbooks” for
every single system so we understand what we would
do in an emergency. We’ve already thought through
the different types of failures. We don’t believe we’ve
thought through everything but we think we’ve
documented at least the most common ones and we
understand what we need to do.
Two, we’re always running tests against production,
so we understand what the system would look like
when we hit it really hard and we can practice. So we
hit it really hard and teams on call might get a page or
something, and we can try to decide whether or not
we do need to do something differently and how to
react to that.
And third, we’ve taken some inspiration from Netflix.
Netflix has what they call their Chaos Monkey, which
kills machines in production. We have something
similar to that within Twitter that helps make sure
that we didn’t accidentally introduce a single point of
failure somewhere. We can randomly kill machines
within the data center and make sure that the service
doesn’t see a blip while that’s happening.
All this requires us to have excellent transparency
with respect to the success rate of all the different
systems. We have a massive board. It’s a glass wall
with all these graphs on it that show us what’s going
on within Twitter. And when these events happen,
we can see in an instant whether or not something is
changing, whether it would be traffic to Twitter or a
failure within a data center, so that we can react to it
as quickly as we can.
Contents Page 5
Scalability / eMag Issue 11 - April 2014
InfoQ: How do you isolate the broken module in the
system? When something goes wrong, what’s your
reaction at the first moment?
Raffi: The way that Twitter is architected these days
is that a failure should stay relatively constrained to
the feature in which the failure occurred. Of course,
the deeper you get down the stack, the bigger the
problem becomes. So if our storage mechanisms
all of a sudden have a problem, a bunch of different
systems would show something going wrong. For
example, if someone made a mistake on the Web site,
it won’t affect the API these days.
The way that we know that something is going wrong
again is by being able to see the different graphs
of the system. We have alerts set up over different
thresholds on a service-by-service basis. So, if the
success rate of the API fell below some number, a
bunch of pagers immediately go off; there’s always
someone on call for every single service at Twitter
and they can react to that as quickly as they can.
Our operations team and our network command
center will also see this and might try some really
rudimentary things, the equivalent of “should we
turn it off and on again and see what happens?”
Meanwhile, the actual software developers on a
second track try to understand what is going wrong
with the system. So, operations is trying to make sure
the site comes back as quickly as it can while software
development is trying to understand what actually
went wrong and to determine whether we have a bug
that we need to take care of.
So this is how we end up reacting. But, like I said,
the architecture at Twitter keeps failure fairly well
constrained. If we think it’s going to propagate or
we think that, for example, the social graph is having
a problem, the social-graph team will then start
immediately notifying everyone else just in case they
should be on alert for something going wrong.
One of our strengths these days, I like to say jokingly,
is emergency management: what we do in a case of
disaster because it could happen at any time. My
contract with the world is that Twitter will be up so
you don’t have to worry about it.
InfoQ: The new architecture helps a lot in stability
and performance. Could you give us a brief
introduction to it?

Raffi: Sure. When I joined Twitter a couple of years
ago, we ran the system on what we call the monolithic
codebase. Everything you had to do with the
software at Twitter was in one codebase that anyone
could deploy, anyone could touch, anyone could
modify. That sounds great. In theory, that’s actually
excellent. It means that every developer in Twitter is
empowered to do the right thing.
In practice, however, there’s a balancing act.
Developers then need to understand how everything
actually works in order to make change. And in
practical realities, the concern I would have is that
with the speed at which Twitter is writing new
code, people don’t give deep thought into places
they haven’t seen before. I think this is standard in
the way developers write software. It’s like, “I don’t
understand what I fully need to do to make this
change, but if I change just this one line it probably
gets the effect I want.” I’m not saying that this is a bad
behavior. It’s a prudent and expedient behavior. But
this means that technical debt builds up when you do
that.
So what we’ve done is we’ve taken this monolithic
codebase and broken it up into hundreds of different
services that comprise Twitter. This way, we can
have actual real owners for every single piece of
business logic and every single piece of functionality
at Twitter. There’s actually a team responsible for
managing photos for Twitter. There’s another team
who manages the URLs for Twitter. There are now
subject experts throughout the company, and you
could consult them when you want to make a feature
change that would change something – for example,
where URLs work.
Breaking up the codebase in all these different ways
and having subject-matter experts also allows things
that we’ve spoken about: isolation for failure and
isolation for feature development. If you want to
change the way tweets work, you only have to change
a certain number of systems. You don’t have to
change everything in Twitter anymore, so we can have
good isolation both for failure and for development.
InfoQ: What’s the role of Decider in the system?
Raffi: Decider is one of our runtime configuration
mechanisms at Twitter. What I mean by that is
we can turn off features and software in Twitter
without doing a deployment. Every single service at
Twitter is constantly looking to the Decider system
Contents
Scalability / eMag Issue 11 - April 2014
for the current runtime values of Twitter. How
that practically maps is, for example, the Discover
homepage has a Decider value that wraps it, and that
Decider value tells Discover whether it’s on or off
right now.
So I can deploy Discover into Twitter and have it
deployed in the state that Decider says it should be.
We don’t get an inconsistent state. The Discover page,
or any feature at Twitter, runs across many machines.
You don’t want to get in the inconsistent state where
some of the machines have the feature and some of
them don’t. So we can deploy it in the off state using
Decider and then when it is on all the machines that
we want it to be on, we can turn it on across the data
center by flipping a Decider switch.
This also gives us the ability to do a percentage-
based control. I can say that now that it’s on all of
the machines, I only want 50% of users to get it. I can
actually make that decision as opposed to it being a
side effect of the way that things are being deployed
in Twitter. This allows us to have runtime control over
Twitter without having to push code. Pushing code is
a dangerous thing; the highest correlation to failure in
a system like ours, not just Twitter but any big system,
is software-development error. This way we can
deploy software in a relatively safe way because it’s
off. Turn it on really slowly, purposefully, make sure
it’s good, and then ramp it up as fast as I want.
InfoQ: How does Twitter push code online? Would
you please share the deployment process with us?
For example, how many different stages? Do you
choose daily pushing or weekly pushing or both?
Raffi: Twitter deployment, because we have this
services architecture, is up to the control of every
individual team. So the onus is on the team to make
sure that when they’re deploying code, everyone
that may be affected by it should know that you’re
doing it, and the network control center should also
know what you’re doing so they have a global view of
the system. But it’s really up to every single team to
decide when and if they want to push.
On average, I would say teams have a bi or tri-
weekly deployment schedule. Some teams deploy
every single day; some teams only deploy once a
month. But the deployment process looks about the
same to everybody: you deploy into a developing
environment. This is so developers can hack on it
quickly, make changes, look at the product manager,
Page 6

look at the designer, and make sure it does the right
thing. Then we deploy into what we call the “Canary
system” within Twitter, which means that it’s getting
live production traffic but we don’t rely on its results
just yet. So it’s just basically loading it to make sure it
handles it efficiently, and we can look at the results
that would have returned and inspect them to make
sure that it did what we thought it would do given live
traffic.
Our testing scenarios may not have covered all the
different edge cases that the live traffic gets, so it’s
one way we learn what the real testing scenarios
should look like. After we go into Canary, we deploy
at dark, then we slowly start to ramp it up to really
understand what it looks like at the scale. That ramp-
up could take anywhere from a day to a week. We’ve
had different products that we’ve ramped to 100%
in the course of a week or two. We’ve added other
products that we’ve ramped up to 100% in the course
of minutes.
Again, it’s really up to the team. And each team is
responsible for their feature, is responsible for their
service. So it’s their call on how they want to do it, but
those stages of development – Canary, dark reading,
ramp up by Decider – is the pattern that everyone
follows.
InfoQ: There are huge amounts of data in Twitter.
You must have some special infrastructure (such as
Gizzard and Snowflake) and methods to store the
data, to even process them in real time.
Raffi: That’s really two different questions, I
think. There is how do we ingest all this data that’s
coming into Twitter because Twitter is a real-time
system with latency for a tweet to get delivered in
milliseconds to Twitter users. And then there’s the
second question of what we do with all that data.
For the first one, you’re right; we have systems like
Snowflake, Gizzard, and things like that to handle
tweet ingestion. Tweets are only one piece of data
that comes into Twitter, obviously. We have things
like favorites. We have retweets. We have people
sending direct messages. People change their avatar
images, their background images, and things like that.
People click on URLs and load Web pages. These are
all events that are coming into Twitter.
Contents
Scalability / eMag Issue 11 - April 2014
So we begin to ingest all this and log them so we can
do analysis. It’s a pretty hard thing. We actually have
different SLAs depending on what kind of data comes
in. Tweets, we measure in milliseconds. In order to get
around database locking, for example, we developed
Snowflake, which can generate unique IDs for us
incredibly quickly and do it decentralized so that we
don’t have a single point of failure in generating IDs
for us.
We have Gizzard, which handles data flowing in and
shards it as quickly as possible so that we don’t have
hot spots on different clusters in the system. It tries to
probabilistically spread the load so that the amount of
data coming in doesn’t overload the databases. Again,
tweets go through very fast on the system.
Logs of, for example, people clicking on things or
viewing tweets, have their SLA measured in minutes
as opposed to milliseconds. Those go into completely
different pipeline. Most of it is based on Scribe these
days. So, those slowly trickle through, get aggregated,
get collected and get jumped into HDFS so we can
analyze them later.
For long-term retention, all of the data, whether it be
real-time or not, ends up in HDFS and that’s where we
run massive MapReduce and Hadoop jobs to really
understand what’s going on in the system.
So, we try to achieve a balance of what needs to be
taken care of right now, especially given the onslaught
of data we have, and where do we put things because
this unclogged data accumulates very fast. If Twitter
sees 400 million tweets a day and has been running
for a couple of years now… you can imagine the size of
our corpus. HDFS handles all that for us, and we can
run these massive MapReduce jobs that way.
InfoQ: Twitter is an amazing place for engineers.
What’s the growing path of an engineer in Twitter?
How would one become a successful geek like you?
Raffi: Well, I can’t say I’m a successful engineer since
I don’t write software anymore. I started at Twitter as
an engineer, and I’ve risen into this position of running
a lot of engineering these days.
Twitter has a couple of different philosophies and
mentalities around it, but we have a career path for
engineers that basically involves tackling harder and
Page 7
 Scalability / eMag Issue 11 - April 2014

harder and harder problems. We would like to say
that it doesn’t actually matter how well the feature
you built does. In some cases, it does. But we really
like the level of technical thought and technical merit
you’ve put into the project you work on.
So growth in Twitter is done very much through a
peer-based mechanism. To be promoted from one
level to the next level at Twitter requires consensus.
It requires a bunch of engineers at that higher level to
agree that, yes, you’ve done the work needed in order
to get to this level at Twitter.
of expand your horizons within Twitter and a way to
safely decide whether or not you want to go and try
something new.
We invest in our engineers because, honestly, they’re
the backbone of the company. The engineers build the
thing that we all thrive on within Twitter, and that the
world uses, so I give them as many opportunities as I
can in order to try different things and to geek out in
lots of different ways.

To help with that, managers make sure projects go

to engineers that are looking for big challenges.
Engineers can move between teams. They’re not
stuck on the tweet team, for example, or a timeline
team. If an engineer says, “I want to work on the
mobile team because that’s interesting. I think
there’s career growth for me,” my job as a person
that manages a lot of this is to make that possible.
You can do almost whatever you want within Twitter.
I tell engineers what my priorities are in running
engineering and what the company’s priorities are in
user growth or money or features to build. And then
engineers should flow to the projects that they think
they can make the biggest impact on.
ABOUT THE INTERVIEWEE
Raffi Krikorian is vice-president of
platform engineering at Twitter. His
teams manage the business logic, scalable
delivery, APIs, and authentication of
Twitter's application. His group helped
create the iOS 5 Twitter integration as
well as the The X Factor Twitter voting
mechanism.

On top of that, I run a small university within Twitter

that we call Twitter University. It’s a group of people READ THIS INTERVIEW
whose whole job is training. For example, if an ONLINE ON InfoQ
engineer wants to join the mobile team but is a back-
end Java developer, we’re say, “Great. We’ve created
a training class so you can learn Android engineering
or iOS engineering and you can take a one weeklong
class that will get you to the place that you’ve
committed to with that codebase, and then you can
join that team for real.” This gives you a way to sort

Contents Page 8
 Scalability / eMag Issue 11 - April 2014

Interview: Adrian Cockcroft on

High Availability, Best Practices, and
Lessons Learned in the Cloud
by Richard Seroter

Netflix is a widely referenced case study for how to effectively operate a cloud
application at scale. While their hyper-resilient approach may not be necessary at
most organizations – and the jury is out on that assumption – Netflix has advanced
the conversation about what it means to build modern systems. InfoQ spoke with
Adrian Cockcroft, who is the cloud architect for the Netflix platform.
InfoQ: What does “high availability 101” look like architects build out circuit breakers and employ
for a new Netflix engineer? How do they learn best other techniques for preventing cascading failures
practices, and what are the main areas of focus? in systems?

Cockcroft: We run an internal “boot camp” every few
months for new engineers. The most recent version
is a mixture of presentations about how everything
works and some hands-on work making sure that
everyone knows how to build code that runs in
the cloud. We use a version of the Netflix OSS RSS
Reader as a demo application.
InfoQ: Are there traditional Web-development
techniques or patterns that you often ask engineers
to “forget” when working with cloud-scale
distributed systems?
Cockcroft: Sticky session-based programming
doesn’t work well so we make everything request
scoped, and any cross-request information must be
stored in memcached using our EVcache mechanism
(which replicates the data across zones).
InfoQ: You and others at Netflix have spoken at
length about expecting failures in distributed
systems. How do you specifically recommend that
Cockcroft: The problem with dependencies between
services is that it rapidly gets complicated to keep
track of them, and it’s important to multi-thread calls
to different dependencies, which gets tricky when
managing nested calls and responses. Our solution to
this is based on the functional reactive pattern that
we’ve implemented using RxJava, with a backend
circuit-breaker pattern wrapped around each
dependency using Hystrix. To test that everything
works properly under stress, we use Latency Monkey
to inject failures and high latency into dependent
service calls. This makes sure we have the timeouts
and circuit breakers calibrated properly, and
uncovers any “unsafe” dependencies that are being
called directly, since those can still cause cascading
failures.
InfoQ: Netflix OSS projects cover a wide range of
services including application deployment, billing,
and more. Which of these projects do you consider
most indispensable to your team at Netflix, and
why?

Contents Page 9

Cockcroft: One of our most powerful mechanisms
and somewhat overlooked Netflix OSS projects is
the Zuul gateway service. This acts as an intelligent
routing layer, which we can use for many purposes:
handling authentication; geographic and content
aware routing; scatter/gather of underlying services
into a consistent external API; etc. It’s dynamically
programmable, and can be reconfigured in seconds.
In order to route traffic to our Zuul gateways, we
need to be able to manage a large number of DNS
endpoints with ensemble operations. We’ve built
the Denominator library to abstract away multiple
DNS vendor interfaces to provide the same high
levels of functionality. We have found many bugs and
architectural problems in the commonly used DNS-
vendor-specific APIs, so as a side effect we have been
helping fix DNS management in general.
InfoQ: Frameworks often provide a useful
abstraction on top of complex technology. However,
are there cases where an abstraction shields
developers from truly understanding something
more complex, but useful?
Cockcroft: Garbage collection lets developers
forget about how much memory they are using and
consuming. While it helps them quickly write code,
the sheer volume of garbage and number of times
data is copied from one memory location to another
is not usually well understood. While there are some
tools to help (we open-sourced our JVM GCviz tool),
it’s a common blind spot. The tuning parameters for
setting up heaps and garbage-collection options are
confusing and are often set poorly.
InfoQ: Netflix is a big user of Cassandra, but is there
any aspect of the public-facing Netflix system that
uses a relational database? How do you think that
modern applications should decide between NoSQL
and relational databases?
Cockcroft: The old Netflix DVD-shipping service
still runs on the old code base on top of a few large
Oracle databases. The streaming service has all
its customer-request-facing services running on
Cassandra, but we do use MySQL for some of our
internal tools and non-customer-facing systems
such as the processes that we use to ingest metadata
about new content. If you want to scale and be
highly available, use NoSQL. If you are doing rapid
continuous delivery of functionality, you will
eventually want to denormalize your data model and
give each team its own data store so they can iterate
Contents Page 10
Scalability / eMag Issue 11 - April 2014
their data models independently. At that point, most
of the value of a unified relational schema is gone
anyway.
InfoQ: Can you give us an example of something
at Netflix that didn’t work because it was too
sophisticated and made you opt for a simpler
approach?
Cockcroft: There have been cases where teams
decided that they wanted to maintain strong
consistency, so they invented complex schemes that
they thought would also keep their services available,
but this tends to end up with a lot of downtime, and
eventually a much simpler and more highly available
model takes over. There is less consistency guarantee
with the replacement, and perhaps we had to build
a data-checking process to fix things up after the
event if anything goes wrong. A lot of Netflix outages
around two years ago were due to an attempt to keep
a datacenter system consistent with the cloud, and
cutting the tie to the datacenter so that Cassandra
in the cloud became the master copy made a big
difference.
InfoQ: How about something you built at Netflix
that failed because it was too simple?
Cockcroft: Some groups use Linux load-average as a
metric to tell if their instances are overloaded. They
then want to use this as an input to autoscaling. I
don’t like this because load-average is time-decay
weighted so it’s slow to respond, and it’s non-linear
so it tends to make autoscaler rules over-react. As
a simple rule, total (user+system) CPU utilization
is a much better metric, but it can still react too
slowly. We’re experimenting with more sophisticated
algorithms that have a lot more inputs, and hope
to have a Netflix Tech Blog post on this issue fairly
soon (keep watching http://techblog.netflix.com for
technology discussion and open source project
announcements).
InfoQ: How do you recommend that developers
(at Netflix and other places) set up appropriate
sandboxes to test their solutions at scale? Do you
use the same production-level deployment tools
to push to developer environments? Should each
developer get their own?
Cockcroft: Our build system delivers into a test
AWS account that contains a complete running set
of Netflix services. We automatically refresh test
 Scalability / eMag Issue 11 - April 2014

databases from production backups every weekend

(overwrite the old test data). We have multiple
“stacks” or tagged versions of specific services that
are being worked on, and ways to direct traffic by
tags is built into Asgard, our deployment tool. There’s
a complete integration stack that is intended to be ABOUT THE INTERVIEWEE
close to production availability but reflect the next Adrian Cockcroft has had a long career
version of all the services. Each developer has their working at the leading edge of technology.
own tagged stack of things they are working on, Before joining Battery in 2013, Adrian
that others will ignore by default, but they share
helped lead Netflix’s migration to a
the common versions. We re-bake AMIs from a test
large scale, highly available public-cloud
account to push to production account with a few
changes to environment variables. There is no tooling architecture and the open sourcing of
support to build an AMI directly for the production the cloud-native NetflixOSS platform.
account without launching it in test first. Prior to that at Netflix he managed a team
working on personalization algorithms
InfoQ: Given the size and breadth of the Netflix and service-oriented refactoring. He
cloud deployment, how and when do you handle graduated from The City University,
tuning and selection of the ideal AWS instance size
London with a Bsc in Applied Physics and
for a given service? Do you run basic performance
Electronics, and was named one of the
profiling on a service to see if it’s memory-bound,
I/O bound, or CPU bound, and then choose the right top leaders in Cloud Computing in 2011
type of instance? At what stage of the service’s and 2012 by SearchCloudComputing
lifecycle do you make these assessments? magazine. He can usually be found on
Twitter @adrianco.
Cockcroft: Instance type is chosen primarily based
on memory need. We’re gradually transitioning
where possible from the m2 family of instances to
the m3 family, which have a more modern CPU base READ THIS INTERVIEW
(Intel E5 Sandy Bridge) that runs Java code better. ONLINE ON InfoQ
We then run enough instances to get the CPU we
need. The only instances that are I/O intensive are
Cassandra, and we use the hi1.4xlarge for most of
them. We’ve built a tool to measure how efficiently
we use instances, and it points out the cases where a
team is running more instances than they need.

Contents Page 11

To Execution-Profile or to Memory-
Profile? That Is the Question
by Kirk Pepperdine
I recently had a group of developers troubleshoot a
problem-riddled application from my performance
workshop. After dispensing with a couple easy wins,
the group was faced with a CPU that was running
very hot. The group reacted in exactly the same
way that I see most teams do when faced with a hot
CPU; they fired up an execution profiler hoping that
it would help them sort things out. In this particular
case, the problem was related to how the application
was burning through memory. Now, while an
execution profiler can find these problems, memory
profilers will paint a much clearer picture. My group
had somehow missed a key metric that was telling
them that they should have been using a memory
profiler. Let’s run through a similar exercise here so
that we can see when and why it is better to use a
memory profiler.
Profilers work by either sampling top of stack
or instrumenting the code with probes, or a
combination of both. These techniques are good
at finding computations that happen frequently
or take a long time. As my group experienced, the
information gathered by execution profilers often
correlates well with the source of the memory
inefficiency. However, it points to an execution
problem which can sometimes be confusing.
The code found in Listing 1 defines the method
API findByName(String,String). The problem here
isn’t so much in the API itself but more in how the
method treats the String parameters. The code
Contents Page 12
Scalability / eMag Issue 11 - April 2014
concatenates the two strings to form a key that is
used to look up the data in a map. This misuse of
strings is a code smell in that it indicates that there
is a missing abstraction. As we will see, that missing
abstraction is not only at the root of the performance
problem, but adding it also improves the readability
of the code. In this case, the missing abstraction is a
CompositeKey<String,String>, a class that wraps the
two strings and implements both the equals(Object)
and hashCode() methods.
public class CustomerList {
private final Map customers = new
ConcurrentHashMap();
public Customer addCustomer(String
firstName, String lastName) {
Customer person = new
Customer(firstName, lastName);
customers.put(firstName + lastName,
person);
return person;
}
public Customer findCustomer(String
firstName, String lastName) {
return (Customer) customers.
get(firstName + lastName);
}
Listing 1. Source for CustomerList
Another downside to the style of API used in this
example is that it will limit scalability because of
the amount of data the CPU is required to write to
 Scalability / eMag Issue 11 - April 2014

memory. In addition to the extra work to create the

data, the volume of data being written to memory by
the CPU creates a back pressure that will force the
CPU to slow down. Though this bench is artificial in
how it presents the problem, it’s a problem that is
not so uncommon in applications using the popular
logging frameworks. That said, don’t be fooled into
thinking only String concatenation can be at fault.
Memory pressure can be created by any application
that is churning through memory, regardless of the
underlying data structure.
The easiest way to determine if our application
is burning through memory is to examine the
garbage-collection (GC) logs. GC logs report on
heap occupancy before and after each collection.
Subtracting occupancy after the previous collection
from the occupancy before the current collection
yields the amount of memory allocated between
collections. If we do this for many records, we can
get a pretty clear picture of the application’s memory
needs. Moreover, getting the needed GC log is both
cheap and, with the exception of a couple of edge
cases, will have no impact on the performance of
your application. I used the flags -Xloggc:gc.log
and -XX:+PrintGCDetails to create a GC log with a
sufficient level of detail. I then loaded the GC log file
into Censum, jClarity’s GC-log analysis tool.
Chart 1. Allocation rates
In this chart, we can see that the allocation rates
initially start out at about 2.75 GB per second. The
laptop that I used to run this benchmark under
ideal conditions can sustain an allocation rate of
about 4 GB per second. Thus this value of 2.75
GB/s represents a significant portion of the total
memory bandwidth. In fact, the machine is not able
to sustain this rate as is evidenced by the drop over
time in allocation rates. While your production
servers may have a larger capacity to consume
memory, it is my experience that any machine trying
to maintain object-creation rates greater than 500
MB per second will spend a significant amount of
time allocating memory. It will also have a very
limited ability to scale. Since memory efficiency is the
overriding bottleneck in our application, the biggest
wins will come from making it more memory efficient.

Table 1. Summary of garbage-collection activity
Execution Profiling
It should go without saying that if we’re looking
to improve memory efficiency, we should be using
a memory profiler. However, when faced with a
hot CPU, our group decided that they should use
execution profiling, so let’s start with that and see
where it leads. I used the NetBeans profiler running in
VisualVM in its default configuration to produce the
profile in Chart 2.

Censum provides a whole host of statistics (see

Table 1), of which we’re interested in the “Collection
Type Breakdown” (at the bottom). The “% Paused”
column (the sixth column in Table 1) tells us that the
total time paused for GC was 0.86%. In general, we’d
like GC pause time to be less than 5%, which it is.
The number suggests that the collectors are able to
reclaim memory without too much effort. Keep in
mind, however, that when it comes to performance, a
single measure rarely tells you the whole story. In this
case, we need to see the allocation rates and in Chart
1 we can see just that.
Chart 2. Execution profile

Contents Page 13

Looking at the chart, we can see that outside of the
Worker.run() method, most of the time is spent in
CustomerList.findCustomer(String,String). If the
source code were a bit more complex, you could
imagine it being difficult to understand why the
code is a problem or what you should do to improve
performance. Let’s contrast this view with the one
presented by memory profiling.
Memory Profiling
Ideally, I would like my memory profiler to show me
how much memory is being consumed and how many
objects are being created. I would also like to know
the causal execution paths – that is, the path through
the source code that is responsible for churning
through memory. I can get these statistics using the
NetBeans profiler once again running in VisualVM.
However I will need to configure the profiler to collect
allocation stack traces. This configuration can be seen
in Figure 1.
Figure 1. Configuring NetBeans memory profiler
Note that the profiler will not collect for every
allocation but only for every 10th allocation. Sampling
in this manner should produce the same result as if
you were capturing data from every allocation but
with much less overhead. The resulting profile is
shown in Chart 3.
Chart 3. Memory profile
The chart identifies char[] as the most popular object.
Having this information, the next step is to take a
Contents Page 14
Scalability / eMag Issue 11 - April 2014
snapshot and then look at the allocation stack traces
for char[]. The snapshot can be seen in Chart 4.
Chart 4. char[] allocation stack traces
The chart shows three major sources of char[]
creation of which one is opened up so that you can see
the details. In all three cases, the root can be traced
back to the firstName + lastName operation.
It was at this point that the group tried to come up
with numerous alternatives. However, none of the
proposed solutions were as efficient as the code
produced by the compiler. It was clear that to have
the application run faster, we were going to have
to eliminate the concatenation. The solution that
eventually solved the problem was to introduce a Pair
class that took the first and last name as arguments.
We called this class CompositeKey as it introduced
the missing abstraction. The improved code can be
seen in Listing 2.
public class CustomerList {
private final Map customers = new
ConcurrentHashMap();
public Customer addCustomer(String
firstName, String lastName) {
Customer person = new
Customer(firstName, lastName);
customers.put(new
CompositeKey(firstName, lastName),
person);
return person;
}
public Customer findCustomer(String
firstName, String lastName) {
return (Customer) customers.get(new
CompositeKey(firstName, lastName));
}
}
Listing 2. Improved implementation using CompositeKey
abstraction
 Scalability / eMag Issue 11 - April 2014

CompositeKey implemented both hashCode() and

equals(), thus eliminating the need to concatenate
the strings together. While the first benchmark
completed in ~63 seconds, the improved version ABOUT THE AUTHOR
ran in ~21 seconds, a threefold improvement. The Kirk Pepperdine has worked in high-
garbage collector ran four times, making it impossible performance and distributed computing
to get an accurate picture, but the application for nearly 20 years. Since 1998, Kirk has
consumed in aggregate just under 3 GB of data as
been working all aspects of performance
apposed to the more than 141 GB consumed by the
and tuning in each phase of a project
first implementation.
life cycle. In 2005, he helped author
Two Ways to Fill a Water Tower the foremost Java-performance tuning
A colleague of mine once said that you can fill a workshop that has been presented to
water tower one teaspoon at a time. This example hundreds of developers worldwide.
proves that you certainly can. However, it’s not the Author, speaker, consultant, Kirk was
only way to fill the tower; you could also run a large recognized in 2006 as a Java Champion
hose to fill it very quickly. In those cases, it’s unlikely
for his contributions to the Java
that an execution profiler would pick up on the
community. He was the first non-Sun
problem. However the garbage collector will see the
allocation and the recover and certainly the memory employee to present a technical lab at
profiler will see the allocation in sheer byte count. JavaONE, an achievement that opened
In one application where these large allocations the opportunity for others in the industry
predominated, the development team had exhausted to do so. He was named a JavaONE
the vast majority of the gains they were going to get Rockstar in 2011 and 2012 for his talks
by using an execution profiler, yet they still needed to on garbage collection. You can reach him
squeeze more out of the app. At that point, we turned
by e-mail at kirk@kodewerk.com or on
on the memory profiler and it exposed one allocation
Twitter @kcpeppe.
hotspot after another, and with that information
we were able to extract a number of significant
performance gains. What that team learned is that
not only was memory profiling giving them the
right view, it was giving them the only view into the READ THIS ARTICLE
problem. This is not to say that execution profiling ONLINE ON InfoQ
isn’t productive. What it is saying is that sometimes
it’s not able to tell you where your application is
spending all of its time – and in those cases, getting a
different perspective on the problem can make all the
difference in the world.

Contents Page 15

Virtual Panel: Using Java in
Low-Latency Environments
by Charles Humble
Java is increasingly being used for low-latency work where previously C and C++
were the de facto choices.
InfoQ brought together four experts in the field to discuss what is driving the trend
and some of the best practices when using Java in these situations.
The Participants
Peter Lawrey is a Java consultant interested
in low-latency and high-throughput systems.
He has worked for a number of hedge funds,
trading firms, and investment banks.
Martin Thompson is a high-performance and
low-latency specialist, with over two decades
working with large-scale transactional and
big-data systems, in the automotive, gaming,
financial, mobile, and content-management
domains.
Todd L. Montgomery is vice-president of
architecture for Informatica Ultra Messaging
and the chief designer and implementer of
the 29West low-latency messaging products.
Andy Piper recently joined Push Technology
as chief technology officer, from Oracle.
Contents
Scalability / eMag Issue 11 - April 2014
The Questions
Q1: What do we mean by low latency? Is it the
same thing as real time? How does it relate to high-
performance code in general?
Lawrey: A system with a measured latency
requirement that is too fast to see. This could be
anywhere from 100 ns to 100 ms.
Montgomery: Real time and low latency can be
quite different. The majority view on real time would
be determinism over pure speed with very closely
controlled, or even bounded, outliers. However, low
latency typically implies that pure speed is given
much higher priority and some outliers may be,
however slightly, more tolerable. This is certainly
the case when thinking about hard real time. One of
the key prerequisites for low latency is a keen eye
for efficiency. From a system view, this efficiency
must permeate the entire application stack, the
OS, and the network. This means that low-latency
systems have to have a high degree of mechanical
sympathy to all those components. In addition, many
of the techniques that have emerged in low-latency
systems over the last several years have come from
high-performance techniques in OSs, languages,
Page 16

VMs, protocols, other system-development areas,
and even hardware design.
Thompson: Performance is about two things:
throughput, i.e. units per second, and response time,
otherwise know as latency. It is important to define
the units and not just say something should be fast.
Real time has a very specific definition and is often
misused. Real time is to do with systems that have
a real-time constraint from input event to response
time regardless of system load. In a hard real-time
system, if this constraint is not honored then a total
system failure can occur. Good examples are heart
pacemakers or missile control systems.
With trading systems, real time tends to have a
different meaning in that the system must have high
throughput and react as quickly as possible to an
event, which can be considered low latency. Missing
a trading opportunity is typically not a total system
failure so you cannot really call this real time.
A good trading system will have a high quality of
execution for which one aspect is to have a low-
latency response with little deviation in response
time.
Piper: Latency is simply the delay between decision
and action. In the context of high-performance
computing, low latency has typically meant that
transmission delays across a network are low or
that the overall delays from request to response
are low. What defines “low” depends on the context
– low latency over the Internet might be 200 ms
whereas low latency in a trading application might
be 2 μs. Technically, low latency is not the same
as real time – low latency typically is measured as
percentiles where the outliers (situations in which
latency has not been low) are extremely important
to know about. With real time, guarantees are made
about the behavior of the system – so instead of
measuring percentile delays, you are enforcing a
maximum delay. You can see how a real-time system
is also likely to be a low-latency system, whereas the
converse is not necessarily true. Today, however, the
notion of enforcement is gradually being lost so that
many people now use the terms interchangeably.
If latency is the overall delay from request to
response then it is obvious that many things
contribute to this delay – CPU, network, OS,
application, even the laws of physics! Thus low-
Contents Page 17
Scalability / eMag Issue 11 - April 2014
latency systems typically require high-performance
code so that software elements of latency can be
reduced.
Q2: Some of the often-cited advantages of using
Java in other situations include access to the rich
collection of libraries, frameworks, application
servers, and so on, and also the large number of
available programmers. Do these advantages apply
when working on low-latency code? If not, what
advantages does Java have over C++?
Lawrey: If your application spends 90% of the time
in 10% of your code, Java makes optimizing that 10%
harder, but writing and maintaining 90% of your code
easier, especially for teams of mixed ability.
Montgomery: In the capital markets, especially
algorithmic trading, there are a number of factors
that come into play. Often, the faster an algorithm
can be put into the market, the more advantage
it has. Many algorithms have a shelf life and
quicker time to market is key in taking advantage
of that. With the community around Java and the
options available, it can definitely be a competitive
advantage, as opposed to C or C++ where the options
may not be as broad for the use case. Sometimes,
though, pure low latency can rule out other concerns.
I think the current difference in performance
between Java and C++ is so close that it’s not a
black-and-white decision based solely on speed.
Improvements in GC techniques, JIT optimizations,
and managed runtimes have made traditional Java
weaknesses with respect to performance into some
very compelling strengths that are not easy to ignore.
Thompson: Low-latency systems written in Java
tend to not use third-party or even standard libraries
for two major reasons. Firstly, many libraries have
not been written with performance in mind and often
do not have sufficient throughput or response time.
Secondly, they tend to use locks when concurrent,
and they generate a lot of garbage. Both of these
contribute to highly variable response times, due to
lock contention and garbage collection respectively.
Java has some of the best tooling support of any
language, which results in significant productivity
gains. Time to market is often a key requirement
when building trading systems, and Java can often
get you there sooner.

Piper: In many ways the reverse is true: writing good
low-latency code in Java is relatively hard since
the developer is insulated from the guarantees of
the hardware by the JVM itself. The good news is
that this is changing. Not only are JVMs constantly
getting faster and more predictable but developers
are now able to take advantage of hardware
guarantees through a detailed understanding of
the way that Java works – in particular, the Java
memory model - and how it maps to the underlying
hardware. (Indeed, Java was the first popular
language to provide a comprehensive memory
model that programmers could rely on. C++ only
provided one later on.) A good example is the lock-
free, wait-free techniques that Martin Thompson
has been promoting and that our company, Push,
has adopted into its own development with great
success. Furthermore, as these techniques become
more mainstream, we are starting to see their
uptake in standard libraries (e.g. the Disruptor) so
that developers can adopt the techniques without
needing such a detailed understanding of the
underlying behavior.
Even without these techniques, the safety
advantages of Java (memory management, thread
management, etc.) can often outweigh the perceived
performance advantages of C++, and of course JVM
vendors have claimed for some time that modern
JVMs are often faster than custom C++ code because
of the holistic optimizations that they can apply
across an application.
Q3. How does the JVM support concurrent
programs?
Lawrey: Java has had built-in multi-threading
support from the start and high-level concurrency
support standard for almost 10 years.
Montgomery: The JVM is a great platform for
concurrent programs. The memory model allows a
consistent model for developers to utilize lock-free
techniques across hardware, which is a great plus for
getting the most out of the hardware by applications.
Lock-free and wait-free techniques are great for
creating efficient data structures, something we very
desperately need in the development community. In
addition, some of the standard library constructs for
concurrency are quite handy and can make for more
resilient applications. With C++11, certain specifics
aside, Java is not the only one with access to a lot of
Contents Page 18
Scalability / eMag Issue 11 - April 2014
these constructs. And the C++11 memory model is a
great leap forward for developers.
Thompson: Java (1.5) was the first major language
to have a specified memory model. A language-
level memory model allows programmers to reason
about concurrent code at an abstraction above the
hardware. This is critically important, as hardware
and compilers will aggressively reorder our code to
gain performance, which has visibility issues across
threads. With Java, it is possible to write lock-free
algorithms that when done well can provide some
pretty amazing throughput at low and predictable
latencies. Java also has rich support for locks.
However, when locks are contended the operating
system must get involved as an arbitrator with huge
performance costs. The latency difference between
a contended and uncontended lock is typically three
orders of magnitude.
Piper: Support for concurrent programs in Java start
with the Java Language Specification itself – the
JLS describes many Java primitives and constructs
that support concurrency. At a basic level, this is
the java.lang.Thread class for the creation and
management of threads and the synchronized
keyword for the mediation of access to shared
resources from different threads. On top of this,
Java provides a whole package of data structures
optimized for concurrent programs (java.util.
concurrent) from concurrent hash tables to task
schedulers to different lock types. One of the biggest
areas of support, however, is the Java memory model
(JMM) that was incorporated into the JLS as part
of JDK 5. This provides guarantees around what
developers can expect when dealing with multiple
threads and their interactions. These guarantees
have made it much easier to write high-performance,
thread-safe code. In the development of Diffusion,
we rely very heavily on the JMM in order to achieve
the best possible performance.
Q4. Ignoring garbage collection for a moment,
what other Java-specific techniques (things that
wouldn’t apply if you were using C++) are there for
writing low-latency code? I’m thinking here about
things like warming up the JVM, getting all your
classes into permgen to avoid I/O, Java-specific
techniques for avoiding cache misses, and so on.
Lawrey: Java allows you to write, test, and profile
your application with limited resources more
effectively. This gives you more time to ensure you

cover the entire “big picture”. I have seen many C/
C++ projects spend a lot of time drilling down to the
low level and still ending up with longer latencies end
to end.
Montgomery: That is kind of tough. The only obvious
one would be warm up for JVMs to do appropriate
optimizations. However, some of the class and
method call optimizations that can be done via
class-hierarchy analysis at runtime are not possible
currently in C++. Most other techniques can also
be done in C++ or, in some cases, don’t need to be
done. Low-latency techniques in any language often
involve what you don’t do that can have the biggest
impact. In Java, there are a handful of things to avoid
that can have undesirable side effects for low-latency
applications. One is the use of specific APIs, such as
the Reflection API. Thankfully, there are often better
choices for how to achieve the same end result.
Thompson: You mention most of the issues in your
question. :-) Basically, Java must be warmed up to
get the runtime to a steady state. Once in this steady
state, Java can be as fast as native languages and in
some cases faster. One big Achilles heel for Java is
lack of memory-layout control. A cache miss is a lost
opportunity to have executed ~500 instructions on
a modern processor. To avoid cache misses, we need
control of memory layout and then we must access
it in a predictable fashion to avoid cache misses. To
get this level of control, and reduce GC pressure, we
often have to create data structures in direct byte
buffers or go off heap and use Unsafe. Both of these
allow for the precise layout of data structures. This
need could be removed if Java introduced support
for arrays of structures. This does not need to be a
language change and could be introduced by some
new intrinsics.
Piper: The question seems to be based on false
premises. At the end of the day, writing a low-latency
program is very similar to writing other programs
where performance is a concern; the input is code
provided by a developer (whether C++ or Java),
which executes on a hardware platform with some
level of indirection inbetween (e.g. through the
JVM or through libraries, compiler optimizers,
etc. in C++). The fact that the specifics vary makes
little difference. This is essentially an exercise in
optimization and the rules of optimization are, as
always:
1. Don’t.
Contents Page 19
Scalability / eMag Issue 11 - April 2014
2. Don’t yet (for experts only).
And if that does not get you where you need to be:
1. See if you actually need to speed it up.
2. Profile the code to see where it’s actually spending
its time.
3. Focus on the few high-payoff areas and leave the
rest alone.
Now, of course, the tools you would use to achieve
this and the potential hotspots might be different
between Java and C++, but that’s just because they
are different. Granted, you might need to understand
in a little more detail than your average Java
programmer would what is going on but the same
is true for C++, and, of course, by using Java there
are many things you don’t need to understand so
well because they are adequately catered for by the
runtime. In terms of the types of things that might
need optimizing – these are the usual suspects of
code paths, data structures, and locks. In Diffusion,
we have adopted a benchmark-driven approach
where we are constantly profiling our application and
looking for optimization opportunities.
Q5. How has managing GC behavior affected the
way people code for low latency in Java?
Lawrey: There are different solutions for different
situations. My preferred solution is to produce so
little garbage that it no longer matters. You can cut
your GCs to less than once a day.
At this point, the real reason to reduce garbage is
to ensure you are not filling your CPU caches with
garbage. Reducing the garbage you are producing
can improve the performance of your code by two to
five times.
Montgomery: Most of the low-latency systems
I have seen in Java have gone to great lengths to
minimize or even try to eliminate the generation
of garbage. As an example, avoiding the use of
Strings altogether is not uncommon. Informatica
Ultra Messaging (UM) itself has provided specific
Java methods to cater to the needs of many users
with respect to object reuse and avoiding some
usage patterns. If I had to guess, the most common
implication has been the prevalent use of object
reuse. This pattern has also influenced many other

non-low-latency libraries such as Hadoop. It’s a
common technique now within the community to
provide options or methods for users of an API or
framework to utilize them in a low or zero garbage
manner.
In addition to the effect on coding practices, there is
also an operational impact for low-latency systems.
Many systems will take some, shall we say, creative
control of GC. It’s not uncommon to only allow GC
to occur at specific times of the day. The implications
on application design and operational requirements
are a major factor in controlling outliers and gaining
more determinism.
Thompson: Object pools are employed or, as
mentioned in the previous response, most data
structures need to be managed in byte buffers or off
heap. This results in a C style of programming in Java.
If we had a truly concurrent garbage collector then
this could be avoided.
Piper: How long is a piece of java.lang.String?
Sorry, I’m being facetious. The truth is that some
of the biggest changes to GC behaviour have come
about through JVM improvements rather than
through programmers’ individual coding decisions.
HotSpot, for instance, has come an incredibly long
way from the early days when you could measure
GC pauses in minutes. Many of these changes have
been driven by competition – it used to be that
BEA JRockit behaved far better than HotSpot from
a latency perspective, creating much lower jitter.
These days, however, Oracle is merging the JRockit
and HotSpot codebases precisely because the gap
has narrowed so much. Similar improvements have
been seen in other, more modern, JVMs such as
Azul’s Zing and in many cases, developer attempts
to improve GC behavior have actually had no net
benefit or made things worse.
However, that’s not to say that there aren’t things
that developers can do to manage GC – for instance,
by reducing object allocations through either pooling
or using off-heap storage to limit memory churn. It’s
still worth bearing in mind, however, that these are
problems that JVM developers are also very focused
on, so it still may well be either not necessary to do
anything at all or easier to simply buy a commercial
JVM. The worst thing you can do is prematurely
optimize this area of your applications without
knowing whether it is actually a problem or not,
since these kinds of techniques increase application
Contents Page 20
Scalability / eMag Issue 11 - April 2014
complexity through the bypass of a very useful Java
feature (GC) and therefore can be hard to maintain.
Q6. When analyzing low-latency applications,
are there any common causes or patterns you see
behind spikes or outliers in performance?
Lawrey: Waiting for I/O of some type. CPU
instruction or data-cache disturbances. Context
switches.
Montgomery: In Java, GC pauses are beginning to be
well understood and, thankfully, we have better GCs
that are available. System effects are common for all
languages though. OS scheduling delay is one of the
many causes behind spikes. Sometimes it is the direct
delay and sometimes it is a knock-on effect caused by
the delay that is the real killer. Some OSs are better
than others when it comes to scheduling under heavy
load. Surprisingly, for many developers the impact
that poor application choices can make on scheduling
is something that often comes as a surprise and is
often hard to debug sufficiently. Of a related note
is the delay inherent from I/O and contention that
I/O can cause on some systems. A good assumption
to make is that any I/O call may block and will block
at some point. Thinking through the implications
inherent in that is very often key. And remember,
network calls are I/O.
There are a number of network-specific causes for
poor performance to cover as well. Let me list the key
items to consider.
• Networks take time to traverse. In WAN
environments, the time it takes to propagate data
across the network is non-trivial.
• Ethernet networks are not reliable; it is the
protocols on them that provide reliability.
• Loss in networks causes delay due to
retransmission and recovery as well as second-
order effects such as TCP head-of-line blocking.
• Loss in networks can occur on the receiver side
due to resource starvation in various ways when
UDP is in use.
• Loss in networks can occur within switches
and routers due to congestion. Routers and
switches are natural contention points and when
contended for, loss is the tradeoff.

• Reliable network media, like InfiniBand, trade off
loss for delay at the network level. The end result
of loss causing delay is the same, though.
To a large degree, low-latency applications that make
heavy use of networks often have to look at a whole
host of causes of delay and additional sources of
jitter within the network. Beside network delay, loss
is probably a high contender for the most common
cause of jitter in many low-latency applications.
Thompson: I see many causes of latency spikes.
Garbage collection is the one most people are
aware of but I also see a lot of lock contention,
TCP-related issues, and many Linux-kernel issues
related to poor configuration. Many applications
have poor algorithm design that does not amortize
the expensive operations like I/O and cache misses
under bursty conditions and thus suffers queuing
effects. Algorithm design is often the largest cause
of performance issues and latency spikes in the
applications I’ve seen.
Time to safepoint (TTS) is a major consideration
when dealing with latency spikes. Many JVM
operations require all user threads to be stopped by
bringing them to a safepoint. Safepoint checks are
typically performed on method returns. The need
for safepoints can be anything from revoking biased
locks or some JNI interactions and de-optimizing
code, through to many GC phases. Often, the time
taken to bring all threads to a safepoint is more
significant than the work to be done. The work is
then followed by the significant costs in waking
all those threads to run again. Getting a thread to
safepoint quickly and predictably is often not a
considered or optimized part of many JVMs, e.g.
object cloning and array copying.
Piper: The most common cause of outliers are
GC pauses, however the most common cure for
GC pauses is GC tuning rather than actual code
changes. For instance, simply changing from the
parallel collector that is used by default in JDK 6
and JDK 7 to the concurrent mark-sweep collector
can make a huge difference to stop-the-world GC
pauses that typically cause latency spikes. Beyond
tuning, another thing to bear in mind is the overall
heap size being used. Very large heaps typically put
more pressure on the garbage collector and can
cause longer pause times – often simply eliminating
memory leaks and reducing memory usage can make
Contents
Scalability / eMag Issue 11 - April 2014
a big different to the overall behavior of a low-
latency application.
Apart from GC, lock contention is another major
cause of latency spikes, but this can be rather
harder to identify and resolve due to its often non-
deterministic nature. It’s worth remembering also
that any time the application is unable to proceed,
it will yield a latency spike. This could be caused by
many things, even things outside the JVM’s control
– e.g. access to kernel or OS resources. If these kinds
of constraint can be identified then it is perfectly
possible to change an application to avoid the use of
these resources or to change the timing of when they
are used.
Q7. Java 7 introduced support for Sockets Direct
Protocol (SDP) over InfiniBand fabric. Is this
something you’ve seen exploited in production
systems yet? If it isn’t being used, what other
solutions are you seeing the wild?
Lawrey: I haven’t used it for Ethernet because it
creates quite a bit of garbage. In low-latency systems,
you want to minimize the number of network hops
and usually it’s the external connections that are
the only ones you cannot remove. These are almost
always Ethernet.
Montgomery: We have not seen this that much. It
has been mentioned, but we have not seen it being
seriously considered. Ultra Messaging is used as
the interface between SDP and the developer using
messaging. SDP fits much more into a (R)DMA access
pattern than a push-based usage pattern. Turning a
DMA pattern into a push pattern is possible, but SDP
is not that well-suited for it, unfortunately.
Thompson: I’ve not seen this used in the wild. Most
people use a stack like OpenOnload and network
adapters from the likes of Solarflare or Mellanox.
At the extreme I’ve seen RDMA over InfiniBand
with custom lock-free algorithms accessing shared
memory directly from Java.
Piper: Oracle’s Exalogic and Coherence products
have used Java and SDP for some time so in that
sense we’ve seen usage of this feature in production
systems for some time also. In terms of developers
actually using the Java SDP support directory rather
than through some third-party product, no, not so
much – but if it adds business benefit then we expect
this to change. We ourselves have made used of
Page 21

latency-optimized hardware (e.g. Solarflare 10GbE
adapters) where the benefits are accrued from
kernel-driver installation rather than specific Java
tuning.
Q8. Perhaps a less Java-specific question, but
why do we need to try and avoid contention? In
situations where you can’t avoid it, what are the
best ways to manage it?
Lawrey: For ultra-low latency, this is an issue, but for
multi-microsecond latencies, I don’t see it as an issue.
In situations where you can’t avoid it, be aware of and
minimize the impact of any resource contention.
Montgomery: Contention is going to happen.
Managing it is crucial. One of the best ways to deal
with contention is architecturally. The “single-writer
principle” is an effective way to do that. In essence,
just don’t have the contention, assume a single writer
and build around that base principle. Minimize the
work on that single write and you would be surprised
what can be done.
Asynchronous behavior is a great way to avoid
contention. It all revolves around the principle of
“always be doing useful work”.
This also normally turns into the single-writer
principle. I often like a lock-free queue in front of
a single writer on a contended resource and use a
thread to do all the writing. The thread does nothing
but pull off a queue and do the writing operation in a
loop. This works great for batching as well. A wait-
free approach on the enqueue side pays off big here
and that is where asynchronous behavior comes into
the play for me from the perspective of the caller.
Thompson: Once we have contention in an
algorithm, we have a fundamental scaling bottleneck.
Queues form at the point of contention and Little’s
law kicks in. We can also model the sequential
constraint of the contention point with Amdahl’s
law. Most algorithms can be reworked to avoid
contention from multiple threads or execution
contexts, giving a parallel speed up, often via
pipelining. If we really must manage contention on
a given data resource then the atomic instructions
provided by processors tend to be a better
solution than locks because they operate in user
space without ever involving the kernel. The next
generation of Intel processors (Haswell) expands on
these instructions to provide hardware transactional
Contents Page 22
Scalability / eMag Issue 11 - April 2014
memory support for updating small amounts of data
atomically. Unfortunately, Java is likely to take a long
time to offer such support directly to programmers.
Piper: Lock contention can be one of the biggest
performance impediments for low-latency
applications. Locks in themselves don’t have to
be expensive and in the uncontended case, Java
synchronized blocks perform extremely well.
However, with contended locks, performance can
fall off a cliff – not just because a thread holding a
lock prevents another thread that wants the same
lock from doing work, but also because simply the
fact that more than one thread is accessing the
lock makes the lock more expensive for the JVM
to manage. Obviously, avoidance is key, so don’t
synchronize stuff that doesn’t need it – remove locks
that are not protecting anything, reduce the scope
of locks that are protecting, reduce the time that
locks are held, don’t mix the responsibilities of locks,
etc. Another common technique is to remove multi-
threaded access – instead of giving multiple threads
access to a shared data structure, updates can be
queued as commands with the queue being tended
by a single thread. Lock contention then simply
comes down to adding items to the queue, which
itself can be managed through lock-free techniques.
Q9. Has the way you approach low-latency
development in Java changed in the past couple of
years?
Lawrey: Build a simple system that does what you
want. Profile it as end-to-end as possible. Optimize
and/or rewrite where you measure the bottlenecks
to be.
Montgomery: Entirely changed. Ultra Messaging
started in 2004. At the time, the thought of using
Java for low latency was just not a very obvious
choice. But a few certainly did consider it. And more
and more have ever since. Today I think the landscape
is totally changed. Java is not only viable, it may be
the predominant option for low-latency systems. It’s
the awesome work done by Martin Thompson and
[Azul Systems’] Gil Tene that has really propelled this
change in attitude within the community.
Thompson: The main change over the past few
years has been the continued refinement of lock-
free and cache-friendly algorithms. I often have fun
getting involved in language shootouts that just keep
proving that the algorithms are way more important

than the language to performance. Clean code that
displays mechanical sympathy tends to give amazing
performance, regardless of language.
Piper: Java VMs and hardware are constantly
changing, so low-latency development is always
an arms race to stay in the sweet spot of target
infrastructure. JVMs have also gotten more robust
and dependable in their implementation of the Java
memory model and concurrent data structures
that rely on underlying hardware support, so that
techniques such as lock-free/wait-free have moved
into the mainstream. Hardware also is now on a
development track of increasing concurrency based
on increasing execution cores, so that techniques
that take advantage of these changes and minimize
disruption (e.g. by giving more weight to avoiding
lock contention) are becoming essential to
development activities.
In Diffusion, we have now got down to single-digit
microsecond latency all on stock Intel hardware
using stock JVMs.
Q10. Is Java suitable for other performance-
sensitive work? Would you use it in a high-
frequency trading system, for example, or is C++
still a better choice here?
Lawrey: For time to market, maintainability, and
support from teams of mixed ability, I believe Java
is the best. The space for C or C++ between where
ABOUT THE PANELISTS
Peter Lawrey is a Java consultant
interested in low-latency and high-
throughput systems. He has works
for a number of hedge funds, trading
firms, and investment banks. Peter
is third for Java on StackOverflow,
his technical blog gets 120K page
views per month, and he is the
lead developer for the OpenHFT
project on GitHub. The OpenHFT
project includes Chronicle, which
supports up to 100 million persisted
messages per second. Peter offers
a free hourly sessions on different
low-latency topics twice a month to
the Performance Java User’s Group.
Contents
Scalability / eMag Issue 11 - April 2014
you would use Java and FPGAs or GPUs is getting
narrower all the time.
Montgomery: Java is definitely an option for most
high-performance work. For HFT, Java already has
most everything needed. There is more room for
work, though: more intrinsics is an obvious one. In
other domains, Java can work well, I think. Just like
low latency, I think it will take developers willing to
try to make it happen, though.
Thompson: With sufficient time, I can make a C/
C++/ASM program perform better than Java, but
there is not that much in it these days. Java is often
the much quicker delivery route. If Java had a good
concurrent garbage collector, control of memory
layout, unsigned types, and some more intrinsics for
access to SIMD and concurrent primitives then I’d be
a very happy bunny.
Piper: I see C++ as an optimization choice. Java
is by far the preferred development environment
from a time-to-market, reliability, higher-quality
perspective, so I would always choose Java first and
then switch to something else only if bottlenecks
are identified that Java cannot address. It’s the
optimization mantra all over again.
Todd L. Montgomery is vice-president of
architecture for the Messaging Business Unit
of 29West, now part of Informatica. As the
chief architect of Informatica’s Messaging
Business Unit, Todd is responsible for the design
and implementation of the Ultra Messaging
product family, which has over 170 production
deployments within the financial services sector.
In the past, Todd has held architecture positions
at TIBCO and Talarian, as well as research and
lecture positions at West Virginia University.
He has contributed to the IETF and performed
research for NASA in various software fields.
With a deep background in messaging systems,
reliable multicast, network security, congestion
control, and software assurance, Todd brings
a unique perspective tempered by 20 years of
practical development experience.
Page 23
 Scalability / eMag Issue 11 - April 2014

Martin Thompson is a high-

performance and low-latency
specialist, with experience gained
over two decades working with
large-scale transactional and
big-data domains, including
automotive, gaming, financial,
mobile, and content management.
He believes mechanical sympathy
- applying an understanding of
the hardware to the creation
of software - is fundamental
to delivering elegant, high-
performance, solutions. Martin
was the co-founder and CTO of
LMAX, until he left to specialize in
helping other people achieve great
performance with their software.
The Disruptor concurrent
programming framework is just one
example of what his mechanical
sympathy has created.

Andy Piper recently joined the

Push Technology team as chief
technology officer. Previously
a technical director at Oracle
Corporation, Andy has over 18
years experience working at
the forefront of the technology
industry. In his role at Oracle,
Andy led development for Oracle
Complex Event Processing (OCEP)
and drove global product strategy
and innovation. Prior to Oracle,
Andy was an architect for the
WebLogic Server Core at BEA
Systems, a provider of middleware
infrastructure technologies.

READ THIS ARTICLE

ONLINE ON InfoQ

Contents

Reliable Auto-Scaling
Using Feedback Control
by Philipp K. Janert
Introduction
When deploying a server application to production,
we need to decide on the number of active server
instances to use. This is a difficult decision, because
we usually do not know how many instances will
be required to handle a given traffic load. As a
consequence, we are forced to use more, possibly
significantly more, instances than actually required in
order to be safe. Since servers cost money, this makes
things unnecessarily expensive.
In fact, things are worse than that. Traffic is rarely
constant throughout the day. If we deploy instances
with peak traffic in mind, we basically guarantee that
most of the provisioned servers will be underutilized
most of the time. In particular, in a cloud-based
deployment scenario where instances can come
and go at any moment, we should be able to realize
significant savings by having only as many instances
active as are required to handle the load at any
moment.
One approach to this problem is to use a fixed
schedule, in which we somehow figure out the
required number of instances for each hour of
the day. The difficulty is that such a fixed schedule
can not handle random variations: if for some
reason traffic is 10% higher today than yesterday,
the schedule will not be capable of providing the
additional instances that are required to handle the
unexpected load. Similarly, if traffic peaks half an
hour early, a system based on a fixed schedule will
not be able to cope.
Contents Page 25
Scalability / eMag Issue 11 - April 2014
Instead of a fixed (time-based) schedule, we might
consider a rule-based solution: we have a rule that
specifies the number of server instances to use for
any given traffic intensity. This solution is more
flexible than the time-based schedule but it still
requires us to predict how many servers we need for
each traffic load. And what happens when the nature
of the traffic changes — as may happen, for example,
if the fraction of long-running queries increases?
The rule-based solution will not be able to respond
properly.
Feedback control is a design paradigm that is fully
capable of handling all these challenges. Feedback
works by constantly monitoring some quality-of-
service metric (such as the response time), then
making appropriate adjustments (such as adding
or removing servers) if this metric deviates from its
desired value. Because feedback bases its control
actions on the actual behavior of the controlled
system, it is capable of handling even unforeseen
events, such as traffic that exceeds all expectations.
Moreover, and in contrast to the rule-based solution
sketched earlier, feedback control requires very little
a priori information about the controlled system.
The reason is that feedback is truly self-correcting:
because the quality-of-service metric is monitored
constantly, any deviation from the desired value is
spotted and corrected immediately, and this process
repeats as necessary. To put it simply: if the response
time deteriorates, a feedback controller will simply
activate additional instances, and if that does not
help, it will add more. That’s all.

Feedback control has long been a standard method
in mechanical and electrical engineering, but it does
not seem to be used much as a design concept in
software architecture. As a paradigm that specifically
applies in situations of incomplete information
and random variation, it is rather different than
the deterministic, algorithmic solutions typical of
computer science.
Although feedback control is conceptually simple,
deploying an actual controller to a production
environment requires knowledge and understanding
of some practical tricks in order to work. In this
article, we will introduce the concepts and point out
some of the difficulties.
Nature of a Feedback Loop
The basic structure of a feedback loop is shown in the
figure. On the right, we see the controlled system.
Its output is the relevant quality-of-service metric.
The value of this metric is continuously supplied to
the controller, which compares it to its desired value,
which is supplied from the left. (The desired value
of the system’s output metric is referred to as the
“setpoint”.) Based on the two inputs of the desired
and the actual value of the quality-of-service metric,
the controller computes an appropriate control
action for the controlled system. For instance, if the
actual value of the response time is worse than the
desired value, the control action might consist of
activating a number of additional server instances.
The figure shows the generic structure
of all feedback loops. Its essential components are
the controller and the controlled system. Information
flows from the system’s output via the return path to
the controller, where it is compared to the setpoint.
Given these two inputs, the controller decides on an
appropriate control action.
So, what does a controller actually do? How does it
determine what action to take?
Contents Page 26
Scalability / eMag Issue 11 - April 2014
To answer these questions, it helps to remember
that the primary purpose of feedback control is to
minimize the deviation of the actual system output
from the desired output. This deviation can be
expressed as “tracking error”:
error = actual - desired
The controller can do anything it deems suitable
to reduce this error. We have absolute freedom in
designing the algorithm — but we will want to take
knowledge of the controlled system into account.
Let’s consider again the data-center situation. We
know that increasing the number of servers reduces
the average response time. So, we can choose a
control strategy that will increase the number of
active servers by one whenever the actual response
time is worse than its desired value (and decrease
the server count in the opposite case). But we can
do better than that, because this algorithm does not
take the magnitude of the error into account, only its
sign. Surely, if the tracking error is large, we should
make a larger adjustment than when the tracking
error is small. In fact, it is common practice to let the
control action be proportional to the tracking error:
action = k error
The value of k is some number.
With this choice of control algorithm, large
deviations lead to large corrective actions, whereas
small deviations lead to correspondingly smaller
corrections. Both aspects are important. Large
actions are required in order to reduce large
deviations quickly but it is also important to let
control actions become small if the error is small —
only if we do this does the control loop ever settle to
a steady state. Otherwise, the behavior will always
oscillate around the desired value, an effect we
usually wish to avoid.

We said earlier that there is considerable
freedom in choosing a particular algorithm for the
implementation of the feedback controller but it is
usually a good idea to keep it simple. The magic of
feedback control lies in the loopback structure of
the information flow, not so much in a particularly
sophisticated controller. Feedback control incurs a
more complicated system architecture in order to
allow for a simpler controller.
One thing, however, is essential: the control action
must be applied in the correct direction. In order to
guarantee this, we need to have some understanding
of the behavior of the controlled system. Usually, this
is not a problem: we know that more servers means
better response times and so on. But it is a crucial
piece of information that we must have.
Implementation Issues
Thus far, our description of feedback control has
been largely conceptual. However, when attempting
to turn these high-level ideas into a concrete
realization, some implementation details need to be
settled. The most important of these concerns the
magnitude of the control action that results from a
tracking error of a given size. (If we use the formula
given earlier, this amounts to choosing a value for the
numerical constant, k.)
The process of choosing specific values for the
numerical constants in the controller implementation
is known as “controller tuning”. Controller tuning
is the expression of an engineering tradeoff: if we
choose to make relatively small control actions,
then the controller will respond slowly and tracking
errors will persist for a long time. If, on the other
hand, we choose to make rather large control actions,
then the controller will respond much faster, but
at risk of over-correcting and incurring an error
in the opposite direction. If we let the controller
make even larger corrections, it is possible for the
control loop to become unstable. If this happens,
the controller tries to compensate each deviation
with an ever-increasing sequence of control actions,
swinging wildly from one extreme to the other
while increasing the magnitude of its actions all the
time. Instability of this form is highly detrimental to
smooth operations and therefore must be avoided.
The challenge of controller tuning therefore amounts
to finding control actions that are as large as possible
without making the loop unstable.
Contents Page 27
Scalability / eMag Issue 11 - April 2014
A first rule of thumb in choosing the size of control
actions is to work backwards: given a tracking error
of a certain size, how large would a correction need
to be to eliminate this error entirely? Remember that
we do not need to know this information precisely:
the self-correcting nature of feedback control
assures that there is considerable tolerance in
choosing values for the tuning parameters. But we do
need to get at least the order of magnitude right. In
other words, to improve the average query response
time by 0.1 seconds, do we need to add roughly one
server, 10 servers, or 100?
Some systems are slow to respond to control
actions. For instance, it may take several minutes
before a newly requested (virtual) server instance
is ready to receive requests. If this is the case, we
must take this lag or delay into account: while the
additional instances spin up, the tracking error will
persist, and we must prevent the controller from
requesting further and further instances. Otherwise,
we will eventually have way too many active servers
online! Systems that do not respond immediately
pose specific challenges and require more care but
systematic methods exist to tune such systems.
(Basically, one first needs to understand the duration
of the lag or delay before using specialized plug-in
formulas to obtain values for the tuning parameters.)
Special Considerations
We must keep in mind that feedback control is
a reactive control strategy: things must first go
out of whack, at least a little, before any corrective
action can take place. If this is not acceptable,
feedback control might not be suitable. In practice,
this is usually not a problem: a well-tuned feedback
controller will detect and respond even to very small
deviations and generally keep a system much closer
to its desired behavior than a rule-based strategy or
a human operator would.
A more serious concern is that no reactive control
strategy is capable of handling disturbances that
occur much faster than it can apply its control
actions. For instance, if it takes several minutes to
bring additional server instances online, we will not
be able to respond to traffic spikes that build up
within a few seconds or less. (At the same time, we
will have no problem handling changes in traffic that
build up over several minutes or hours.) If we need
to handle spiky loads, we must either find a way to
speed up control actions (for instance, by having

servers on hot standby) or employ mechanisms that
are not reactive (such as message buffers).
Another question that deserves some consideration
is the choice of the quality-of-service metric to
be used. Ultimately, the only thing the feedback
controller does is to keep this quantity at its desired
value, hence we should make sure that the metric we
choose is indeed a good proxy for the behavior that
we want to maintain. At the same time, this metric
must be available, immediately and at all times. (We
cannot build an effective control strategy on some
metric that is only available after a significant delay,
for instance.)
A final consideration is that this metric should not
be too noisy, because noise tends to confuse the
controller. If the relevant metric is naturally noisy,
then it usually needs to be smoothed before it can
be used as control signal. For instance, the average
response time over the last several requests provides
a better signal than just the response time of the
most recent request. Taking the average has the
effect of smoothing out random variations.
Summary
Although we have introduced feedback control here
in terms of data-center autoscaling, it has a much
wider area of applicability: wherever we need to
maintain some desired behavior, even in the face of
uncertainty and change, feedback control should
be considered an option. It can be more reliable
than deterministic approaches and simpler than
rule-based solutions, but it requires a novel way of
thinking and knowledge of some special techniques
to be effective.
Further Reading
This short article can only introduce the basic notions
of feedback control. More information is available
on my blog and in my book on the topic (Feedback
Control for Computer Systems. O’Reilly, 2013).
Contents
Scalability / eMag Issue 11 - April 2014
ABOUT THE AUTHOR
Philipp K. Janert provides consulting
services for data analysis and
mathematical modeling, drawing on
his previous careers as physicist and
software engineer. He is the author
of the best-selling Data Analysis with
Open Source Tools (O’Reilly), as well as
Gnuplot in Action: Understanding Data
with Graphs (Manning Publications).
In his latest book, Feedback Control for
Computer Systems, he demonstrates
how the same principles that govern
cruise control in your car also apply
to data-center management and
other enterprise systems. He has
written for the O’Reilly Network, IBM
developerWorks, and IEEE Software.
He holds a Ph.D. in theoretical physics
from the University of Washington.
Visit his company’s Web site.
READ THIS ARTICLE
ONLINE ON InfoQ
Page 28