The InfoQ eMag / Issue #87 / November 2020

Edge Cloud

Is Edge Computing a Deploying Edge Cloud State at the Edge:

Thing? Solutions without an Interview with
Sacrificing Security Peter Bourgon

FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT

 InfoQ @ InfoQ InfoQ InfoQ

Edge Cloud

IN THIS ISSUE

Is Edge Computing a Thing? 06 Edge Computing and Flow

Evolution 22
Deploying Edge Cloud
Solutions without Sacrificing
Security 11 State at the Edge: an
Interview with Peter Bourgon 27

The Modern Edge 17

PRODUCTION EDITOR Ana Ciobotaru / COPY EDITORS Lawrence Nyveen & Susan Conant / DESIGN Dragos Balasoiu
GENERAL FEEDBACK feedback@infoq.com / ADVERTISING sales@infoq.com / EDITORIAL editors@infoq.com
CONTRIBUTORS
Simon Crosby  Sam Bocetta
is the CTO of SWIM.AI. Previously, Simon
was a co-founder and CTO of Bromium, a
security technology company. At Bromium,
Simon built a highly secure virtualized
system to protect applications. Prior to
Bromium, Crosby was the co-founder and
CTO of XenSource before its acquisition by
Citrix, and later served as the CTO of the
Virtualization and Management Division at
Citrix. Previously, Crosby was a principal
engineer at Intel. Crosby was also the
founder of CPlane, a network-optimization
software vendor. Simon Crosby was named
as one of InfoWorld’s Top 25 CTOs.
Zack Bloom
is the Head of Developer Marketing at
Cloudflare. He is the creator of open source
projects which have gathered over forty
thousand stars on Github.
is a former security analyst, having spent
the bulk of his as a network engineer for the
Navy. He is now semi-retired, and educates
the public about security and privacy
technology. Much of Sam’s work involved
penetration testing ballistic systems. He
analyzed our networks looking for entry
points, then created security-vulnerability
assessments based on my findings. Further,
he helped plan, manage, and execute
sophisticated “ethical” hacking exercises to
identify vulnerabilities and reduce the risk
posture of enterprise systems used by the
Navy (both on land and at sea).
James Urquhart
is a Global Field CTO with VMware Tanzu.
Mr. Urquhart brings over 25 years of
experience in distributed applications
development, deployment, and operations,
focusing on software as a complex adaptive
system, cloud native applications and
platforms, and automation. Mr. Urquhart
has also written and spoken extensively
about software agility and the business
opportunities it affords.

Richard Seroter
is Director of Outbound Product
Management at Google Cloud, with
a master’s degree in Engineering
from the University of Colorado. He’s
also an instructor at Pluralsight,
the lead InfoQ.com editor for cloud
computing, a frequent public speaker,
the author of multiple books on
software design and development,
and a former 12-time Microsoft MVP
for cloud. As Director of Outbound
Product Management at Google
Cloud, Richard leads a team focused
on products and customer success
for app modernization (e.g. Anthos).
Richard maintains a regularly updated
blog on topics of architecture and
solution design and can be found on
Twitter as @rseroter.
A LETTER FROM
THE EDITOR
“Edge” isn’t a new thing to most
technologists. However, if you
think it just relates to Inter-
net-of-Things (IoT), it’s time
to take a fresh look. Whether
you’re at a startup to established
enterprise, edge computing has
become more relevant. Why?
You’re generating data in more
places: SaaS systems, public
clouds, remote devices, on-prem-
ises, and partner data centers.
You’re also creating systems that
execute logic on mobile phones
or vehicles, then at the CDN, and
finally in your application. These
distributed systems solve one
set of problems, and can create
another set of problems. At InfoQ,
we sought to learn more from the
people building and evaluating
these edge-powered systems.
In “Is Edge Computing a Real
Thing”, industry leader Simon
Crosby proposes that edge
computing requires us to rethink
how we process data. Specifical-
ly, Crosby lays out an approach
powered by stateful, real-time
streaming, and what’s needed to
make that a reality.
Extending your solutions to the
edge might involve rethinking
your security strategy. In “De-
ploying Edge Cloud Solutions
without Sacrificing Security”,
Sam Bocetta first explores the
security challenges with an edge
solution. Then he assesses the
“interconnected processes” that
you’ll need to consider when
building a secure edge solution
at scale.
“The Modern Edge”, from Zack
Bloom at Cloudflare, looks at
practical examples of how com-
panies are using edge com-
puting. He explains how edge
computing powers JavaScript
package management system
npm. That’s a big use case, but
Bloom also tells about two “reg-
ular” companies that use edge
 The InfoQ eMag / Issue #87 / November 2020
computing in their own ways to serve
their customers.

Industry vet James Urquhart sees

edge as increasingly essential, in a
thoughtful piece entitled “Edge Com-
puting and Flow Evolution.” Urquhart
draws parallels between distributed
computing and real-world systems—
think cities, electrical grids, and the
body’s circulatory system.

Finally, it’s important to look at where

the next innovations may be. In an
interview with Fastly’s Peter Bourgon,
we learn about state processing at
the edge. Bourgon proposes a handful
of ideas that are designed to respect
the unique challenges of distributed
data processing, while stripping away
unnecessary complexity.

This series of articles touches on

many of the key aspects of designing
and delivering a solution that uses
edge computing. We hope you enjoy
it, and that it sparks new ideas and
debates with your colleagues.

5
The InfoQ eMag / Issue #87 / November 2020
Is Edge Computing a Thing?
by Simon Crosby, CTO of SWIM.AI.
 Enterprise architects are
comfortable using public cloud
services to help quickly build
applications. The (REST API
+ stateless microservice +
database) model for cloud-
native apps is a pattern that
has been key to scaling the
cloud (any server will do) and
increased use of abstractions
such as Kubernetes further
simplifies the operational
deployment and management
of microservices.  But there’s a
class of applications for which
today’s cloud-native stacks are
an awkward fit.   the requirements for these apps
6
I’m not referring to lift+shift
legacy applications that are
expensive to run in the cloud
(and will take forever to re-
write).  Instead, my focus is a
class of applications that are
increasingly important to en-
terprise customers that analyze
streaming data – from produc-
tion, products, supply chain
partners, employees and more
- to deliver real-time insights
to help drive the business.  In-
creasingly these apps are put
into the “edge computing” cat-
egory.    In this piece I’ll dissect
and (hopefully) convince you
that, for streaming data apps
at least, the “edge” is not a
place.  Instead it’s a new way to
compute on streaming data. 
There are many uses for ad-
vanced computing embedded
in next-gen products “at the
edge”, from cars to compres-
sors.  The engineers that devel-
op them will use the best CPUs,
ML acceleration, hypervisors,
Linux and other technology
to build vertically integrated
solutions: Better cars, com-
pressors and drones.   Is this
“edge computing”?   Sure, but
not in a generic sense – these
are tightly integrated solutions
rather than computing systems

that could be used for a broad
set of applications - interesting
but only narrowly.   But what
about the data that these and
other connected products pro-
duce?  “Smart” devices (with a
CPU and lots to say) are being
connected at a rate of about 2M/
hour, and there already are a bil-
lion mobile devices in our hands
- so increasingly networks are
becoming flooded with stream-
ing data.  Applications that drink
from this firehose are becoming
common, and we need tools to
help devs create them.
The term “edge computing”
implies a generic capability that
is different from cloud comput-
ing.  While there are often re-
quirements such as data volume
reduction, latency or security/
compliance concerns that dictate
an on-prem component of an
application, other than these,
does edge computing have
unique requirements?  It does:
Real-time analysis of streaming
data demands that we kick the
REST + database habit.  But there
is nothing that is unique to the
physical edge.  This is great news
because it means that “edge
applications” can run on cloud
infrastructure, or on prem.  “Edge
computing” is definitely a thing,
but it’s about processing stream-
ing data from the edge, as op-
posed to running the application
at the physical edge.   Edge ap-
plications that process streaming
data from real world things have
to:  stack is needed for any solution
Continuously and statefully
analyze new data: Stateless
microservices can only update or
consume from a database.  That
means their performance is
dominated by database response
times.  Often many database
accesses will be required to
compute a contextual result,
making performance non-real-
time.  Stateful processing is key.
Always have an answer: The
“store then analyze” approach
needs to become “analyze, react,
then (maybe) store”, without
database roundtrips.  Storing raw
data for later analysis is slow
and batch focused, and moreover
most raw data is only ephemeral-
ly useful.  must load (code and) the previ-
Process all data: Sub-sampling is
inadequate.  The software stack
needs to process every event,
on-the-fly, and share insights in
real-time.
Analyze and visualize in context:
An event is of value only in the
context in which it was gener-
ated, so the interrelationship
between event sources is key, at
a granular level.
These requirements for stream
processing software are inde-
pendent of “where” the solution
runs. If the data sources are
fixed, then it makes sense to
co-locate some compute nearby
– particularly to deal with data
reduction. But a cloud-hosted
The InfoQ eMag / Issue #87 / November 2020
that processes data from mobile
devices.   
OK, what’s different?   Whereas
cloud computing builds on a
powerful triumvirate: Stateless
APIs and micro-services, and
stateful databases, streaming
applications need different infra-
structure abstractions: 
“Things” are stateful: Whereas
the stateless model of the web
serves traditional cloud services
well because it lets any server
(including “serverless” Lamb-
da / Functions) process an event,
this has a massive drawback
when  processing streaming
data:  For each new event, an app
ous state from a database, com-
pute a new value, and store the
new state back in the database. 
This bounds performance to the
round-trip time to the database,
which inevitably is a million times
slower than the CPU.  We need a
computing architecture that can
deal with real world state chang-
es - “things” change state often
and changes to one thing propa-
gate to others.    
Algorithms must be adapted for
boundless data:  Algorithms that
compute analytics, learn or pre-
dict must be adapted to deal with
boundless data – computing on
every new event.
Context is vital: Events aren’t
meaningful in isolation.  Re-
al-world relatedness of things
7
 such as containment, proximity
The InfoQ eMag / Issue #87 / November 2020
and adjacency are key in applica-
tions that reason about events in
context.  So, edge computation
is necessarily “in (a dynamical-
ly changing) data graph” rather
than “over (a pre-built) graph”.
A key observation is that “things”
in the real-world change state
concurrently and independent-
ly, and their state changes are
context specific – based on the
state of (other things in) their
environment.  It is the state
changes in things that are critical
for applications, and not raw
data.  Moreover, whereas da-
tabases are good at capturing
relationships that are fixed (bus-
es have engines) they are poor at
capturing dynamically computed
relationships (a truck with bad
braking behavior is near an in-
spector).  Real-world relation-
ships between data sources are
fluid, and based on computed
relationships such as bad brak-
ing behavior, the application
should respond differently.
Finally, effects of changes are
immediate, local and contextual
(the inspector is notified to stop
the truck).   The dynamic nature
of relationships suggests a graph
database – and indeed a graph of
related “things” is what is  need-
ed.  But in this case, to satisfy the
need to process continuously, the
graph itself needs to be fluid and
computation must occur “in the
graph”.
In summary: Edge computing
at any scale demands state-
8
ful processing, and not the
usual stateless microservice
model that relies on REST and
databases.  But rather than
make engineers scratch their
heads and learn something new,
we have to provide an approach
that is easy to adopt – using
familiar dev and devops meta-
phors to make it easy for engi-
neers to quickly deliver stateful
applications that offer real-time
insights at scale.  Below we
discuss a powerful “stream-
ing-native” application platform
called  SwimOS, an Apache 2 li-
censed platform loosely modeled
on the actor model, that builds
graphs of linked actors to state-
fully analyze sources of stream-
ing data in real-time.  Developers
create simple object-oriented
applications in Java or JavaS-
cript.  Streaming data builds an
automatically scaled-out graph
of stateful, concurrent objects -
called web agents – actors that
are effectively “digital twins” of
data sources.  even correlation.  Linked agents
Each web agent actively process-
es raw data from a single source
and keeps its state in memory
(as well as persisting its state
in the background to protect
against failures).  Web agents are
thus stateful, concurrent digital
twins of real-world data sourc-
es that at all times mirror the
relevant state of the real-world
things they represent.  The dia-
gram shows web-agents created
as digital twins of sensors in a
smart city environment.  
Web agents link to each oth-
er based on computed context
based on changes in the data,
dynamically building a graph that
reflects real-world relationships
like proximity, containment, or
see each other’s state changes
in real-time.  Agents can con-
currently compute on their own
state and that of agents they are
linked to.  They analyze, learn and
predict, and continuously stream
enriched insights to UIs, bro-
kers, data lakes and enterprise
applications. 
The diagram shows that the
sensors at an intersection link to
the intersection digital twin.  In-
tersections link to their neigh-
bors, enabling them to see state
changes in real-time.   The links

are dynamically computed: mem-
bership/containment and prox-
imity are used to build the graph
using insights computed by the
digital twins from their own data.
SwimOS benefits from in-memo-
ry, stateful, concurrent computa-
tion that yields several orders of
magnitude performance im-
provement over database-centric
analysis simply because all state
is in-memory and web agents
can compute immediately when
data arrives or a contextual state
change in another linked agent
occurs.  Streaming implementa-
tions of key analytical, learning
and prediction algorithms are
included in SwimOS, but it is also
easy to interface to existing plat-
forms such as Spark.     the calculations that they use
The InfoQ eMag / Issue #87 / November 2020
SwimOS moves analysis, learning
and prediction into the dynam-
ically constructed graph of web
agents.  An application is a thus
a dynamically constructed graph
built from data.  This dramatical-
ly simplifies application creation:
The developer simply describes
the objects and their inputs, and
to determine how to link.  When
9
 data arrives, SwimOS creates a
The InfoQ eMag / Issue #87 / November 2020
web agent for each data source,
each of which independently and
concurrently computes as data
flows over it, and in turn streams
its insights – such as predictions
or analytical insights. Each is
responsible for consuming raw
data from its real-world sibling,
and links dynamically to other
agents based on computed rela-
tionships.  As data flows over the
graph, each web agent computes
insights using its own state and
that of other agents to which it is
linked.    streaming – a link is effect a
You can see SwimOS in action in
an application that consumes
over 4TB per day of data from the
traffic infrastructure in Palo Alto,
CA, to predict the future state of
each intersection (click on the
blue dots to see the predicted
phases, and the colors for down-
timers of each light).   This
application runs in many cities in
the USA, and delivers predictions,
on a per intersection basis, via an
Azure hosted API to customers
that need to route their vehicles
through each city.  The source
code for the application is part of
the SwimOS GitHub repo. Start-
ing with SwimOS is easy – the
site is complete with docs and
tutorials.
In SwimOS the graph is a living,
dynamically changing struc-
ture. In the example earlier:
(a truck with bad braking behav-
ior is near an inspector) the rela-
tionship is dynamically computed
and ephemeral; the link between
10
the inspector and the truck is
thus also ephemeral, built when
the truck enters a geo-fence
around the inspector, and broken
when it leaves.
An application in SwimOS is a
dynamic graph of linked web
agents that continuously con-
currently compute as data flows
over the graph.  As each web
agent modifies its own state,
that state is immediately visi-
ble to linked web agents in the
graph.  This is achieved through
streaming API, and takes the
form of a URI – just like a REST
API call.   SwimOS uses a pro-
tocol called WARP, which runs
over web-sockets, to synchronize
state and deliver streamed in-
sights. The key difference is this:
each web agent transforms its
own raw data into state chang-
es, and those state changes are
streamed to linked web agents in
the graph.  They in turn compute
based on their own states, and
the states of agents to which
they are linked.   needing to access a data-
A SwimOS application is built
from the data, given a simple ob-
ject definition for each source.  A
single app is easy to build.  But
there is another benefit in this
approach: If an application is re-
used in multiple sites, no chang-
es are required.  For example,
data in a smart city application is
used to build an application that
predicts future traffic behavior in
the city.  But note that there is no
specific model required for each
city.  Instead, only the local con-
text for each intersection is used,
by the digital twin of the intersec-
tion itself, to learn and predict.  In
each city, the app for the city is
built from the data, without need-
ing to change a line of code.
A word about transformations: In
traditional analytical stacks – for
example using Spark – the data
is transported to the application,
which is forced to transform
data to state, for each thing, and
save it to a database, before the
analytical app can operate on
the state of the real world sourc-
es.  SwimOS transforms raw data
to the relevant state of the source
in each web agent.  The data-to-
state transformation is done by
each digital twin and the graph
of web agents is a mirror of the
states of the real-world sourc-
es.  Then, each digital twin can
compute locally, at high resolu-
tion, using its own state and the
states of linked web agents.  This
proceeds concurrently, at CPU
and memory speed, without
base.  The performance increase
is huge, and the reductions of
infrastructure required are com-
mensurate.  We frequently find
that a SwimOS application uses
less than 5% of the infrastructure
of a database-centric implemen-
tation of the same application.  
Finally, SwimOS digital twins are
real-time streaming mirrors of
real-world things – things in the
edge environment.  They com-
pute and stream insights on the
fly.  Visualization tools are vital
for such applications.  Swi-
mOS includes a set of JS and
typescript bindings that enable
developers to quickly devel-
op browser-based UIs that
update in real-time, simply
because they subscribe to the
streamed insights computed
by web agents.
In summary: Edge Computing
is definitely a thing, but the
computing need not occur
at the edge.  Instead what is
needed is an ability to com-
pute (anywhere) on streaming
data from large numbers of
dynamically changing devices,
in the edge environment.  This
in turn demands an architec-
tural pattern for stateful, dis-
tributed computing.  SwimOS
is an example of a stateful,
real-time platform for appli-
cations that process real-time
streaming data.
The InfoQ eMag / Issue #87 / November 2020
TL;DR
• “Edge” is not a place,
but rather, a new way to
compute on streaming
data.
• Real-time analysis of
streaming data demands
that we kick the REST +
Database habit.
• Cloud computing is
built on stateless mi-
croservices and stateful
databases, whereas
edge requires stateful
“things”, boundless data
streams, and dynamic
context.
• SwimOS is an example
of a stateful, real-time
platform for applications
that process real-time
streaming data
11
The InfoQ eMag / Issue #87 / November 2020
Deploying Edge Cloud Solutions without
Sacrificing Security
by Sam Bocetta, Security Researcher
Edge cloud was a major topic of
debate at RSA this year. Multi-
ple panels were devoted to the
subject, and even in those that
weren’t the utility of edge solu-
tions was often raised. At the
same time, however, a tension
was apparent: operations and
dev staff were quick to stress the
performance gains of edge cloud
infrastructures, and cybersecurity
pros raised concerns about the
12
security implications of these
same architectures.
This tension has been apparent
for a while. In their 2020 Outlook
report, Carbon Black pointed to a
bit of a rift between IT and secu-
rity teams regarding resource al-
location in cloud edge structures.  ular is projected to account for
Edge, Cloud, and Edge Cloud
In order to see the challeng-
es involved in deploying edge
cloud solutions whilst retaining
strong security, it’s worth re-
minding yourself why edge cloud
solutions such as Software-as-
a-Service (SaaS) were initially
developed. Such solutions are
becoming virtually commonplace,
to the point that SaaS in partic-
almost all of the software needs
for 86% of companies within the
next two years (it already is being
used to a lesser extent by 90% of
companies right now).
Some security firms will tell you
that their edge cloud SaaS solu-
tions are only designed to ensure
security, but that’s not quite true.
In reality, edge cloud systems
were developed with one simple
factor in mind: bandwidth. That’s
why, for instance, the Open Glos-
sary of Edge Computing, an open
source effort led by the Linux
Foundation’s LF Edge group, de-
fines edge cloud systems primar-
ily in terms of performance: “by
shortening the distance between
devices and the cloud resources
that serve them,” the glossa-
ry explains, “and also reducing
network hops, edge computing
mitigates the latency and band-
width constraints of today’s
Internet, ushering in new classes
of applications.”
In other words, as the number of
IoT devices connected to net-
works began to increase expo-
nentially around five or so years
ago, many systems engineers
found that their cloud providers
were not keeping up with the
increased computing load. The
solution was to insert another
level of processing between de-
vices and cloud storage provid-
ers, and thereby reduce the data
loads that cloud services had to
process. Only later, in fact, were
edge cloud systems thought
about as a tool to secure the
devices they interact with. And
that’s primarily why cybersecuri-
ty pros don’t trust them.
These concerns are well illus-
trated by the ongoing worry that
the most high-profile example of
edge cloud systems – automated
vehicles – can be easily hacked.
The ease with which data used
by autonomous vehicles can be
accessed and manipulated has
been a concern for years, and as
a result many of the security pro-
tocols used in ege cloud systems
have been designed, primarily, to
protect autonomous vehicles.  incur a performance loss, but at
The Challenges
There are a few problems with
edge cloud solutions from a
security perspective. Some are
technical, and some relate to
the way in which these ser-
vices are used within a typical
organization.
Architecture
First, let’s think about the struc-
ture of edge cloud systems. In
most implementations, edges are
within organizations’ comput-
ing boundaries, and so they will
be protected by a wide variety
of tools that focus on perimeter
scanning and intrusion detection.
However, that›s not quite the
whole story: in most systems,
there will also be a tunnel
between the edge straight to
cloud storage. 
Sending data from the edge to
the cloud in a secure way is fairly
straightforward, because orga-
nizations will control the infra-
structure that is used to encrypt
and verify it. The problem arises
when the cloud needs to send
data back to the edge for pro-
The InfoQ eMag / Issue #87 / November 2020
cessing. The challenge here is to
ensure that this data is authenti-
cated and verified, and is there-
fore safe to enter into an organi-
zations’ internal systems. 
Fragmentation
First, and most obviously, edge
cloud systems fragment data.
Having each device connected
directly to cloud services might
least this data is centralized, and
can be covered by a single cloud
security policy. Because edge
cloud servers – almost by defi-
nition – need to be connected
to many different devices, they
represent a nightmare when it
comes to securing these same
connections.
Fragmentation is not only a prob-
lem when it comes to protecting
data, though. With a growing
number of IoT devices running
via edge processing, each needs
to be authenticated and follow a
privacy policy that allows net-
work admins to keep control of
their data. The edge cloud model
makes it inherently difficult to
apply global privacy policies to
each device, since each is com-
municating independently.
Physical Security
A third issue with edge cloud
systems is that locking down
physical access to these devices
can be a challenge. The devices
typically used in edge cloud in-
frastructures are designed, after
all, to be portable, and as such
13
 are more susceptible to physical
The InfoQ eMag / Issue #87 / November 2020
tampering than standard data
devices.
An example of this is the “micro
data centers” that many telecom-
munications providers are now
making use of. These centers
sometimes sit at the base of cell
towers, and pre-process data be-
fore feeding either back to con-
sumer devices or into corporate
data systems. Micro data centers
like this can dramatically improve
the performance of cell networks,
but they are also vulnerable to
physical tampering.
Sprawl
All of these issues are com-
pounded by the tendency of edge
cloud systems to grow beyond
the boundaries they were origi-
nally designed to operate within.
In large organizations, building
edge cloud functionality can be
an invitation for other engineers,
from other parts of your organi-
zation, to shift their computing
14
demands to your edge cloud
system.  that they provide.
Overcoming this challenge re-
quires a dual approach. On the
one hand, management needs to
be made aware of the limitations
of edge cloud systems, both in
terms of computing power and
security, in order to prevent many
new devices being connected to
them. Secondly, engineers should
design edge cloud systems with
a view to the future, and make
sure that the security that is built
into these systems is easily un-
derstandable for other employees
working with them.
User Error
The problem of “sprawl” is relat-
ed to another: that many IT pro-
fessionals simply don’t take IoT
device security seriously. Despite
the well-documented security is-
sues that these devices present,
many people simply don’t realize
the level of connectivity – and
the level of cybersecurity risk –
 
In this context, it is all too
common for IoT devices to be
connected to networks (and
connected together) in poorly se-
cured horizontal structures. Not
only does this make them more
susceptible to attack, but it also
allows intruders a huge degree of
lateral movement once they are
inside IoT networks. 
So Why Use Edge Cloud?
Given all these security risks,
and given that a recent study by
Tech Republic found that two-
thirds of IT teams considered
edge computing as more of a
threat than an opportunity, it›s
worth wondering why we need
edge cloud solutions at all.
This is, in fact, a very pertinent
question, because some
analysts have argued that the
advent of 5G networks, coupled
with the increased computing
power of contemporary IoT
devices, means that most of the
processing currently done by
edge cloud systems can now be
done by devices themselves.
That doesn’t seem to be born
out by the facts, though. In a
recent report Futuriom writes
that 5G will actually be a catalyst
for edge-compute technology.
“Applications using 5G technol-
ogy will change traffic demand
patterns, providing the biggest
driver for edge computing in mo-
bile cellular networks,” the report
states. 
In other words: whilst connec-
tion and cloud technologies are
developing rapidly, demand for
them is increasing even faster.
This is a particular problem when
it comes to managing online
backup services, because with-
out proper oversight a cloud edge
system can end up undermining
the integrity of backup policies
implemented by individual teams.
Put simply, companies can’t
afford to give up their cloud-
based systems or devices.
Cloud connected VoIP systems
can save businesses 70% of their
total phone bills on average, and
companies that turn to cloud
computing to fulfill their software
needs have seen major increases
in productivity to help drive busi-
ness growth. At the same time,
the bandwidth available to these
same companies lags far behind
the amount of data they need to
process on the cloud. Edge cloud
computing, in this context, seems
like an obvious choice.
Ensuring Security
This means that, for now, security
teams are stuck with edge cloud
solutions, and will have to work
out how to harden them further
against cyberattack. Crucial to
this attempt will be the deploy-
ment of perimeter scanning
systems that are able to analyse
not just standard network data,
but a huge variety of other forms
of data such as that produced by
embedded IoT devices. Overcom-
ing the security challenges posed
by edge cloud systems can be
broken into a number of intercon-
nected processes.
Decentralization and Resilience
First, it’s worth pointing out that
one of the features that makes
edge cloud infrastructures so
hard to secure – the fragmen-
tation of data – can also make
them more resilient. This is
because, as Proteus Duxbury, a
transformation expert at PA Con-
sulting, said recently, “instead
of one or two or even three data
centers, where if they’re close
enough together that, say, a big
storm could impact them all, you
have distributed data and com-
pute on the edge, which makes it
much more resilient to malicious
and nonmalicious events.”  Full Spectrum Security
In some ways, then, pushing
data to the edge can mean that
attacks on organizations are less
effective, because they are not
able to compromise a centralized
data storage system which holds
every piece of sensitive data.
On the other hand, and as seen
above, this fragmentation can
make the application of global
security measures more difficult.
The IoT and Encryption
Another issue that is raised by
the widespread adoption of edge
cloud systems is the security of
the IoT itself. Concerns about the
security of IoT devices are not
new, of course: it has long been
noted that the design of these
devices prioritizes connectivity
over security. However, in
The InfoQ eMag / Issue #87 / November 2020
traditional cloud systems, the
processing required to run
these devices can be managed
centrally. As IoT devices begin
to utilize edge cloud solutions,
this exposes them to increased
threats.
The most commonly suggested
solution to this problem is to in-
crease the security of IoT devic-
es themselves. However, at the
moment many embedded devic-
es lack the computing power to
encrypt data before sending it to
either cloud or edge cloud sys-
tems. As a result, network engi-
neers have been forced to rely on
other forms of security. 
Securing edge cloud systems
is ultimately a problem of scale
rather than of essence. Secu-
rity professionals already have
access to many of the tools that
are required to protect these
systems, but will need to huge-
ly extend their reach in order to
protect data on the edge. 
In fact, in many ways securing
edge cloud systems requires
network engineers to return to
the basic principles of network
security, but then to apply them
outside the systems that they
directly manage. These elements
include:
• Perimeter scanning tech-
niques that use encrypted
tunnels, firewalls, and access
15
 control policies to protect the identity of the entity, real time tication and verification steps
The InfoQ eMag / Issue #87 / November 2020

data held in edge cloud context and security/compliance on every connected device, at a
systems. policies.” At a basic level, SASE frequency that ensures that the
combines SD-WAN, SWG, CASB, data being handled stays secure
• Securing applications running
ZTNA and FWaaS as core abil- whilst not affecting global net-
on the edge in the same ways
ities, with the ability to identify work performance. 
that applications running
sensitive data or malware and
within your organization are
the ability to decrypt content at Pushing The Edge
already secured.
line speed, with continuous mon- Whilst cloud edge computing
• Upgrading threat detection itoring of sessions for risk and offers many opportunities, it also
capabilities so that intrusion trust levels.  comes with challenges. To make
can be detected not just in matters worse, these challenges
relation to cloud or in-house Though SASE is still a new ap- come at a time when security
systems, but also for the proach, Gartner has high hopes teams are struggling to keep up
edge. for the new technology. They with other developments – the
predict that, by 2024, at least necessity to go multi-cloud
• Automated patching that
40% of enterprises will have in whilst still using cloud-na-
allows network managers
place strategies to adopt this tive tools, and becoming involved
to trust that both software
approach. in DevSecOps migrations.
and firmware automatically
receives security updates.
The End Of Zero Trust? For that reason, a major deter-
Securing edge cloud systems mining factor in the security of
Secure Access Service Edge
also involves overturning some edge cloud systems will be the
(SASE)
basic misconceptions about speed at which they are de-
All of these approaches and tools
threat hunting. Namely, it might ployed by businesses. Though
have been combined by Gartner
be that the increased popularity edge cloud can offer significant
into a new category of hardware
of edge cloud solutions overturns gains in terms of performance, it
and services that are specifically
another piece of received wis- will not replace traditional cloud
designed to improve edge cloud
dom, the superiority of the zero models where these are currently
security. In 2019, the firm coined
trust model. Many of these new working well. 
a new term – Secure Access
systems, or at least the devices
Service Edge (SASE) – to define
that they interface with, will be In some ways, this removes
these systems.
extremely difficult to bring into some of the pressure on security
single sign-on and user access teams, who can afford to design
Gartner has defined SASE as a
control processes.  each edge cloud system with
combination of multiple existing
security in mind at the earliest
technologies. The new paradigm,
Instead, ensuring security in edge possible stage. On the other
they say, «combines network
cloud solutions might require hand, as edge cloud systems
security functions (such as SWG,
a more pragmatic approach, in grow in importance, security pro-
CASB, FWaaS and ZTNA), with
which individual networks are fessionals will be in the unenvi-
WAN capabilities (i.e., SDWAN)
segmented and protected indi- able position of having to secure
to support the dynamic secure
vidually. This, in turn, requires cloud, edge cloud, and in-house
access needs of organizations.
that networks be configured to system simultaneously. 
These capabilities are delivered
automatically perform authen-
primarily aaS and based upon

16
As with any piece of new tech-
nology, the level of security of
cloud edge solutions is unlikely to
become apparent anytime soon.
But that doesn’t mean that we
shouldn’t put in place tools and
processes to protect these sys-
tems as far as is practical.  it comes to fragment-
The InfoQ eMag / Issue #87 / November 2020
TL;DR
• Edge cloud systems face
security issues when
ing data, locking down
physical access, and the
tendency of edge cloud
systems to grow beyond
the boundaries of what
they were originally de-
signed to operate in.
• Most security systems
are stuck with edge
cloud systems, and
need to figure out how
to harden them against
attack rather than aban-
don them
• Overcoming the secu-
rity challenges of edge
cloud security systems
will really come down to
decentralization, en-
cryption, and utilizing
full spectrum security
measures
• A big determining factor
for the security of edge
cloud systems will be
the speed at which busi-
nesses deploy them
17
The InfoQ eMag / Issue #87 / November 2020
The Modern Edge
by Zack Bloom, Head of Developer Marketing at Cloudflare
Too often we hear fanciful stories
of the purpose of ‘edge com-
puting’. I was raised on tales of
self-driving cars (which probably
should be able to drive with-
out an internet connection),
AI-driven security cameras, and
oil drilling platforms which can’t
process their own data. These
use-cases are figurative and
fanciful, and while they may exist
someday, they aren’t a real part
of our lives and jobs. To combat
18
that I thought it would be fun
to share some features of how
interesting companies are using
edge computing in ways you
might not expect to solve real
problems and help build a better
Internet.
npm at the Edge
npm was acquired by GitHub a
few weeks ago. What you might
not know is every npm package
you download is powered by
edge computing! 
“We can say pretty confidently
there’s about 11 or 12 million Ja-
vaScript developers in the world.
And we know because that’s how
many people are using npm.” -
Isaac Schlueter, Creator of npm
As is common with many ser-
vices, npm began by carefully di-
viding the world into ‘static’ files
and ‘dynamic’ responses like npmjs.org searches.
Static files could be cached and delivered by a
CDN, while dynamic responses had to travel all
the way to a central origin to be responded to.
In the words of Isaac Schlueter, creator of npm:
“Once you have a CDN, you have a really high
degree of control over [how files are delivered],
especially if you’re talking about GETs and espe-
cially if you’re talking about fetches of downloads
of archives that can be long lived… that’s just how
NPM is architected.”
Unfortunately using a CDN is not without conse-
quences. Static files can’t be customized based
on who is viewing them or what that viewer is
looking for. Similarly, dynamic responses are slow
as they may have to travel halfway around the
world and hard to scale as they rely on inflexi-
ble infrastructure. npm was built on JavaScript,
but they found themselves with a full time engi-
neer devoted to writing not JS, but bespoke CDN
configuration languages. As their service became
more powerful with features like npm for Enter-
prise, it became very necessary to deliver different
packages to different users, something a CDN was
not very good at.
npm uses Cloudflare Workers which run JavaS-
cript in Cloudflare’s points of presence around
Figure 1
the world. Workers takes the V8 JavaScript and
The InfoQ eMag / Issue #87 / November 2020
WebAssembly engine which runs as a part of Goo-
gle Chrome and runs it on Cloudflare’s network
of thousands of servers in hundreds of locations
around the world. As V8 doesn’t need to launch
a dedicated process or container (or Kuberne-
tes pod) for each customer’s code, it’s possible
to perform serverless computing at grand scale
without the ‘cold starts’ which plague traditional
serverless systems. Code running in a Worker
starts in single-digit milliseconds, and requires
as little as one tenth the memory overhead of a
Node.js process.
Inside their Worker code npm makes decisions
about whether a package is private or not. For
private packages they ensure that the user has
the appropriate authentication token (using the
standardized WebCrypto API) before delivering the
file.
To understand the transformation, let’s consider
performing a similar type of authentication on a
request using VCL, the configuration language
of the Varnish Cache used by several CDNs,
versus being able to write code with JavaScript
and Cloudflare Workers. The VCL code has been
abridged for space:
19
 VCL Authentication Example Using JavaScript with Cloudflare Workers
The InfoQ eMag / Issue #87 / November 2020

sub vcl_recv {
/* unset state tracking header to avoid async function handle(request) {
client sending it */
// Make an authentication request that
if (req.restarts == 0) { is identical to the
unset req.http.X-Authed; // original request, but a GET with no
body.
}
let authUrl = new URL(request.url)
authUrl.pathname = “/authenticate”
if (!req.http.X-Authed) {
let authResponse = await fetch(au-
/* stash the original URL and Host
thUrl, {
for later */
...request,
set req.http.X-Orig-URL = req.url;
method: “GET”,
/* set the URL to what the auth back-
end expects */ body: null
set req.url = “/authenticate”; })
/* Auth requests won’t be cached, so
pass */
if (authResponse.status === 200) {
return(pass);
// Client is authenticated.
}
return fetch(request)
} else if (authResponse.status ===
if (req.http.X-Authed == “true”) { 401) {
/* we’re authed, so proceed with the // Authentication server wants cre-
request */ dentials from the client.
/* reset the URL */ return authResponse
set req.url = req.http.X-Orig-URL; } else {
} else { // Every other response from the au-
thentication server becomes 403.
/* the auth backend refused the re-
quest, so 403 the client */ return new Response(null, {
error 403; status: 403,
} statusText: “Forbidden”,
})
#CDN recv }
}
...etc...
}
 

20
As a programmer, I certainly find
the code easier to decipher. This
means edge computing isn’t just
relevant for esoteric use-cases,
it’s relevant for each and every
CDN requirement developers
have. When compared to deploy-
ing similar code in a central loca-
tion using something like Node.
JS, deploying to a CDN offers
a massive scale advantage for
npm. Any site being powered by
a CDN is already being served by
thousands of servers, this means
any code they write is already
horizontally scaled to handle
their hundreds of thousands of
requests per second.
Building Your Own Edge
Anycart looks like any other
grocery delivery service. When
you dig deeper you find one big
exception: it’s very fast. Not the
delivery, the website! It’s also
growing very quickly. (See Figure
1, page 19)
This growth isn’t coming from
paid ads or marketing, it’s com-
ing purely from the speed of
the site translating into a better
experience and better SEO than
its competitors.
CEO, Rafa Sanches, has built a
unique setup which eschews any
public cloud provider. Instead,
he leases dedicated servers in
dozens of locations around the
world. In fact, he has over 30
dedicated points of presence for
his application, and seven more
for his database, entirely built
and managed by him.
I will never, never, never, nev-
er, never, NEVER build on the
cloud - Rafa
To Rafa, the public cloud is ex-
ceptionally expensive. He is able
to get dedicated machines so
affordably that his entire hosting
bill is under $2000 a month, for a
complex service with more than
200k users a day. The secret to
getting such cheap hardware
is being able to always accept
the best (cheapest) deal. He can
afford that because he doesn’t
need to worry about the avail-
ability of any one of his instanc-
es. He uses MySQL with Galera
Cluster to arrange all seven of
his database points of presence
into a master-master replication
configuration. That configuration
means he can write to any one of
the instances and have it appear
everywhere.
Careful programming has en-
sured that his system expects all
processing to be done asynchro-
nously, so replication lag which
would be an issue is minimized.
Batch jobs are processed on
a 48-core machine which his
team only pays $180 a month to
run (the same machine on AWS
would be over $3000). His users
are also generally only touching
their own records (like their cart
and their recipes), meaning con-
flict resolution is rarely an issue
between replicas. This system
means his users get blazing fast
reads and writes from anywhere
on earth, and he gets to sleep
well knowing it would take quite
the act of god to take his data-
The InfoQ eMag / Issue #87 / November 2020
base offline.
Similarly, his application serv-
ers are run on thirty dedicated
machines in different corners of
the world. Cloudflare global load
balancing is used to map his
incoming web traffic to the near-
est host, and to perform health
checks to remove hosts which
might be offline.
Many of us make performance
decisions based on ‘best prac-
tices’, generally accepted min-
imums required to build some-
thing of quality. For Rafa though,
things are more simple: “How can
we build a user experience that
is so good that if Google doesn’t
put us first, they’re doing a bad
job.” As an example of what that
looks like, they don’t use the in-
dustry standard tool for frontend
programming: React. It’s simply
too slow. 
Outsmarting Performance
What can cleansing herbs teach
you about performance? They
taught Shalom Volchok that
e-commerce platforms are not
ready for the modern web. Sha-
lom’s family ran a major herbal
product company that wanted
to perform A/B tests. An A/B
test is where a web developer
deploys two variants of a page,
shows each variant to a portion
of visitors, and tests that perform
better. Unfortunately for them,
every method of doing A/B tests
with existing systems delivered
subpar performance. In fact, ev-
21
 ery e-commerce vendor they tried deployed directly to a CDN, but
The InfoQ eMag / Issue #87 / November 2020

simply didn’t have the capability
to both do things dynamically and
with performance.
So Shalom founded Outsmartly,
a startup e-commerce platform
which is built on the edge. Like
many new ideas, they had to build
the tools to build their prod-
uct. Namely, they constructed a
Cloudflare Workers-based hosting
platform which allows for vari-
ants to be deployed and served
directly from the edge. Features
like multiple site variants being
deployed at once and integration
with most of the modern web
landscape (React, Angular, Vue)
all had to be built from scratch.
With their platform they are able
to host sites which load in tens of
milliseconds almost anywhere on
Earth, but which can have many
of the capabilities of a traditional
web server.
TL;DR
actually support the tools mod-
ern businesses rely on. It goes to
show you what technology can do
when it meets a problem ready to
• Edge computing isn’t
be solved.
just for esoteric applica-
tions, it’s a tool for every
The Future
web developer
The future of web development
and computing is going to look a • It will take time for
lot like the present. Hard working edge-computing-level
and passionate people will build performance to become
amazing things, and they will a best practice
often not get the credit they de-
• npm uses Cloudflare
serve. I believe in that future that
Workers to power
websites will be faster for those
every single package
people who are most disenfran-
downloaded
chised by the Internet of today.
Platforms will be more powerful • Edge computing allows
and have less moving pieces, ‘static’ files to not be
less APIs, and less languages. static anymore, which
Ultimately edge computing isn’t might eliminate the CDN
going to solve someone else’s as we know it
problems, it will solve yours!

The dream location would be to

render a full application within the
CDN’s cache — effectively that’s
exactly what Cloudflare Work-
ers enables for our platform. For
large ecommerce companies, this
page load improvement can equal
millions of dollars in incremen-
tal revenue. The performance is
really that astonishing. — Shalom
Volchok, co-founder Outsmartly

They now can deliver static

websites that support an unlim-
ited number of A/B tests (and the
statistically relevant analytics
to go with them). These sites
are just as fast as any static site

22

Edge Computing and Flow Evolution
by James Urquhart Global Field CTO with VMware Tanzu
As every enterprise finds their
software portfolio merging into
a single, large scale distributed
computing architecture, we are
also seeing that architecture
evolve. “New” patterns seem to
be trending that are purported to
accommodate the sheer scale of
digitizing corporate operations.
For some, the rise of edge com-
puting would be one of the great
changes to the way computing
gets done since cloud computing
replaced private data centers.
But the truth is edge computing
is an easily predictable pattern as
digital operations scale, especial-
ly for geographically distributed
businesses, such as national re-
tail chains, insurance brokerages,
or even modern manufacturing
supply chains. There is science
that explains these patterns,
coming from the field of complex
adaptive systems. It is probably
worth understanding the basics
of this science, as it will help
you make better decisions about
what to run centrally, and what to
run “on the edge”.
First, let’s take a second to look
at this amorphous beast we call
edge computing. Normally my
next sentence would be some-
The InfoQ eMag / Issue #87 / November 2020
thing like “What exactly is edge
computing?”, but having been
through the cloud wars, I know
better than to launch yet another
debate on definitions and se-
mantics. Instead, “the edge” can
be seen as the set of deployment
locations that are not in the data
center. This is good enough for
our purposes.
Now, let’s look at what complex-
ity science can tell us about the
evolution of the edge. Geoffrey
West is a theoretical physicist
and distinguished professor at
the Santa Fe Institute, which
has been the epicenter of com-
23
 plexity science for the last three
The InfoQ eMag / Issue #87 / November 2020
decades. West studied the ways
systems handle flow: the move-
ment of shared resources be-
tween the agents of the system.
Think blood vessels delivering
oxygen to cells, the electric grid
delivering electrons to machines
and appliances, or even city
infrastructure delivering passen-
gers to homes and businesses.
What West noticed is that these
systems seem to evolve in very
analogous patterns. Our circula-
tory system, for example, delivers
blood throughout our body using
a system that has huge central
blood vessels which deliver blood
to smaller vessels, that in turn
do the same, until you reach the
capillaries that are tiny, but serve
very specific sets of cells.
Our electric grid has evolved such
that large, central generators
feed a massive core transmis-
sion infrastructure (sometimes
operating at tens of thousands of
volts!), which in turn feeds more
localized transmission infra-
structure (working at hundreds
or thousands of volts), which
again passes through step-down
transformers at the neighbor-
hood level to become the local
standard voltage that runs in our
homes. (Even our homes are one
further step-down point as the
incoming current is controlled
and distributed to allow the same
utility lines to serve dozens of
different devices simultaneously.
24
Cities are especially interest-
ing to West (and to me, frankly),
as vehicular infrastructure has
evolved into a very similar pat-
tern. Interstates bring traffic to
highways and major thorough-
fares, which in turn feed local
streets and neighborhood lanes.
Air infrastructure has major inter-
national hubs feeding smaller
national airports which may even
feed tiny regional airports. The
shipping industry has massive
ships that feed railroads and gi-
ant semi-trucks that in turn feed
local distribution warehouses
that feed (often smaller) trucks
that feed local stores.
This pattern of large, core “trunk”
flows with “limbs, branches, and
leaves” is incredibly common in
complex systems that handle
flow. So much so, in fact, that
West and his colleagues dis-
covered there are mathematical
patterns in the way these sys-
tems are structured. While some
elements scale very quickly, and
others scale very slowly relative
to the system as a whole, all
scale with an exponential trend
known as a “power law”.
Power laws and limits
A power law is defined as follows
by Wikipedia:
In statistics, a power law is
a functional relationship between
two quantities, where a relative
change in one quantity results in
a proportional relative change in
the other quantity, independent
of the initial size of those
quantities: one quantity varies as
a power of another. For instance,
considering the area of a square
in terms of the length of its side,
if the length is doubled, the area
is multiplied by a factor of four.
In flow systems, components
have a power law relationships
with each other, The nature of
those relationships can tell us a
lot about how systems will con-
tinue to evolve over time, Sys-
tems where key resources scale
at a power law less than one
with respect to the infrastructure
depended on to deliver those
resources will hit limits that pre-
clude further growth.
In mammals, metabolism in-
creases with a ¾ power law rela-
tive to size. Thus, a larger animal
has a higher overall metabolic
rate (the energy used to maintain
cellular function per unit of time)
than its smaller counterparts,
but the difference is not directly
proportional. As the animal gets
larger, the change in metabolic
rate gets lower for each addi-
tional kilogram. So, on a per-cell
basis, the metabolism actually
gets lower in larger animals than
in smaller animals, but because
the number of cells is so much
higher, it adds up to a larger met-
abolic rate.
Combine this with the interesting
fact that all mammals have the
same average number of heart-
beats in their lifespan. Tiny ani-
mals that live short lives have to
beat their tiny heart much faster
to maintain blood flow through
their blood vessels. Large an-
imals can ship more blood a
longer distance with more effi-
ciency (thanks to a bigger pump),
and thus the heart doesn’t have
to beat as often to maintain
the same blood pressure at the
capillaries (where oxygen and
carbon dioxide are exchanged
with the cells). However, there is
a limit there, too, as the physics
of animal blood flow hit a wall at
about the size of an elephant for
land animals, or a blue whale for
ocean dwelling mammals.
Interestingly, the slower metabol-
ic rate per cell in larger animals
explains another limit. Ever won-
der why large animals live longer
than small ones? Well, that slow
cellular metabolic rate means
cells in large animals wear out
slower than those in smaller
animals, thus contributing to a
longer overall life span. (There
are obviously many details that
I am leaving out, but read West’s
book if you are curious to know
more.)
Power Laws and Computing
So, let’s apply this line of thinking
to another system in which a re-
source (data) must flow between
agents (computers) at great
scale. The Internet is an amazing
example of such a system, and
the application portfolio (and
supporting infrastructure) for the
average Fortune 2000 company
is a great subset. As the scale of
these systems increases, how
might we predict the patterns
of computing architecture will
change with it?
We can start with the simple fact
that two fundamental elements
of distributed computing aren’t
changing anytime soon: the
time-per-instruction-executed on
modern CPUs (which has levelled
out in the last decade or so), and
the speed of light. Anything we
do to scale distributed computing
will be limited by these con-
stants. This means that a) the
only way to gain more computing
power is to add more processors,
and b) operational latency will
naturally suffer if you increase
geographic distance between
processors.  vices that depend most on each
Thus, as computing power grows,
the performance gain of each
additional GB/s will result in
a smaller gain in performance
than the last. At some point, the
gain in processing power will
largely be displaced by the loss
in network performance, and the
application won’t be able to scale
effectively any further.
Edge and Data Centers Are Both
Essential
How do you address that? Well,
the first obvious answer might be
to reduce the distance between
compute nodes that depend on
each other. The more “chatter”
between nodes, the less network
distance there should be between
the two (ideally). So centralizing
everything in data centers should
be perfect, right?
It turns out that some of the
The InfoQ eMag / Issue #87 / November 2020
highest volume of chatter in a
distributed application is be-
tween the end nodes (things like
mobile phones, laptops, and even
digital sensors) and the serv-
ers that deliver data and user
interfaces to them. For latency
sensitive applications, having ev-
eryone globally reach one or two
data centers results in some per-
centage of the population having
less than optimal performance.
OK, so distribute everything to
computers as close as possible
to those end nodes, you might
think. Well, now you have the
problem that the backend ser-
other are talking over longer
distances (and through more net-
work hops, etc.). This may make
the entire system less efficient.
There is a third option, however.
Place as much of the system
that directly interfaces with end
nodes in computers that can
be placed as close to those end
nodes as possible. Then, place
the services (and data) that are
heavily dependent on one anoth-
er in data centers (hopefully also
distributed, but not as extensive-
ly). This creates a “trunk, limb,
branch” model that enables the
direct interaction between end
nodes to be handled by local
edge nodes, but the exchange of
shared data, events, and other
interactions between services to
be optimized in central comput-
ing locations (data centers).
25
 This mimics what happens in
The InfoQ eMag / Issue #87 / November 2020
nature in very real ways, and
is why I believe that the future
architectures of enterprise and
commercial systems will follow
this pattern (though it will morph
a little from this strict interpreta-
tion). We’ll see massive growth
in edge computing because
of it (though anyone using an
application or content delivery
network is already doing it). And,
you’ll see data center business
continue, though more and more
of that business is likely to move
26
to public cloud providers (with a
few, very large scale exceptions).
While I assume that just about
any company with distributed
physical operations has some
form of edge computing today,
I would be surprised if more
than 30% or so actually have an
edge computing strategy. We all
should, however, as this is a nat-
ural evolution of the world’s great
digital complex adaptive system,
Internet-based computing.
TL;DR
• Edge computing rep-
resents a major new
architectural option for
enterprise computing,
but its rapid adoption
should not be surprising.
• Geoffrey West of the
Santa Fe Institute wrote
a book, Scale: The Uni-
versal Laws of Growth,
Innovation, Sustainabil-
ity, and the Pace of Life
in Organisms, Cities,
Economies, and Com-
panies, that explains the
evolution of flow-centric
systems like distributed
computing
• Key elements of distrib-
uted computing, namely
processing power and
network performance
scale at a power law of
less than one relative to
one another
• The limitations of these
key elements at scale
make edge computing a
logical addition to data
centers and end-user
computing for software
at scale
 SPONSORED ARTICLE
Apps Are Becoming Distributed,
What about Your Infra?
You’ve read how modern compa-
nies are disrupting their legacy
peers by pursuing a data-driven
culture…
• Chick-fil-A’s edge use case
allows them to transform
their fast food restaurants;
throwing jet fuel on top of
their already impressive
growth.
• An industry-first, soft-
ware-only mobile infrastruc-
ture by Rakuten Mobile
• Tesla Autopilot functional-
ity getting ever so close to
Level 5, with new features
and enhancements pushed to
vehicles almost bi-weekly
There are a myriad of examples,
but the common thread is this:
these are software-first com-
panies in their categories. And
why are they software first? They
are data-driven.
Their secret sauce is the con-
tinuous pursuit to process and
extract insights from data to
drive their business decisions
and operational efficiencies. So
if that’s the recipe, why aren’t
incumbents doing it?
DISTRIBUTED DATA & APPS...
OH MY!
THE TREND
• Data is getting increasingly
distributed in the enterprise,
and for multiple reasons,
including:
Please read the full-length version of this article here.
The InfoQ eMag / Issue #87 / November 2020
• Digital transformation of
business processes wher-
ever they’re needed — at the
branch, the factory, the store…
even the car or plane
• The distribution of workloads
and clusters themselves
due to modern app architec-
ture and deployment trends,
including micro-services,
containers and multi-cloud
As a result, distributed apps and
data in the enterprise typically
require the following:
• Real-time responses
• Hyper-localization aka...edge 
• Near-zero downtime
• Intelligent management and
security
To address the above, there
are technological and opera-
tional shifts required. Those
changes can be summarized
in this figure .
27
The InfoQ eMag / Issue #87 / November 2020
State at the Edge: an Interview
with Peter Bourgon
by Richard Seroter, Director of Outbound Product Management at Google Cloud
At this year’s QCon London, Fast-
ly’s Peter Bourgon did a well-re-
ceived talk about the challenges
of state management in
distributed systems. Specifically,
he talked about how an
architecture and communication
model for a global-scale edge
platform.
It was a talk that addressed a
wide-range of topics. Is there
a central source of truth for the
data? How should data be syn-
chronized across the system?
What’s the right data structure to
28
store state? InfoQ reached out to
Peter to further explore a handful
of areas that we think are inter-
esting to our readers.  can meet those expectations.
InfoQ: Do you think “edge” is the
next widely adopted architec-
ture for modern systems? Or is
it an important, but small niche
suitable for a specific subset of
systems? And if so, which ones?
Bourgon: I don’t think that “edge”
is an architecture in itself, but
rather it’s a component of an ar-
chitecture. Users expect applica-
tions to be fast, and the speed of
light imposes some unavoidable
physical constraints on how we
Past a certain point, the only way
to decrease latency and improve
experiences is to move your
application, in whole or in part,
physically closer to your users.
And edge platforms are the way
to do that.
It’s true that not all systems
will take the same value from
extending their state and logic
out of the datacenter. There’s
definitely some re-thinking,
re-factoring, and re-architecting
work involved, and that’s always
an engineering decision of costs
and benefits. But, looking to the
future, I think that the role of the
edge in system design is going to
get bigger and more important.
InfoQ: You say in your talk that
a “general purpose database”
isn’t a fit for stateful edge sys-
tems. Can you explain a bit why
someone wouldn’t want a sin-
gle centralized database for an
edge system? Is it entirely about
latency?
Bourgon: I think latency is the
biggest reason. If you’re ex-
tending your system out of the
datacenter and toward the edge,
at least part of the reason is
presumably that the round-trip
costs are too high otherwise. Us-
ers experience those costs with
static assets, with application
logic, and also with state, so if
your transactions always go back
to your origin, you’re not taking
full advantage of the architecture.
But there’s another part, too,
which is related to consistency.
With a centralized database, it’s
relatively easy to express trans-
actions against a single, logical,
coherent global “truth”. Because
all the parts of the system aren’t
separated by very much distance,
you can perform the necessary
communication for that transac-
tion quickly, and stay within la-
tency budgets. But if you balkan-
ize your state all over the world,
making a traditional transaction
becomes cost-prohibitive. You
have to allow users to manipulate
state locally, without establish-
ing global consensus, and this
means opting in to non-tradition-
al, typically eventually consistent,
data systems.
InfoQ: In your talk, you go into
some depth on conflict-free rep-
licated data types (CRDTs). Can
you explain more about these,
why they make sense for a state-
ful edge architecture, and how/
what data is stored?
Bourgon: Arguably the hardest
part of distributed systems is
dealing with faults. Computers
are ephemeral, networks are
unreliable, topologies change
— the fallacies of distributed
computing are well-known, and
accommodating them tends
to dominate the engineering
effort of successful systems.
And if your system is managing
state, things get much more
difficult: maintaining a useful
consistency model for users
requires extremely careful
coordination, with stronger
consistency typically demanding
commensurate effort. This
inevitably corresponds to more
bugs and less reliability.
CRDTs, or conflict-free replicated
data types, are a relatively novel
state primitive that give us a way
to skirt around a lot of this com-
plexity. I think of them as careful-
ly constructed data types, each
combined with a specific set of
operations. Over-simplifying, if
The InfoQ eMag / Issue #87 / November 2020
you make sure the operations
are associative, commutative,
and idempotent, then CRDTs
allow you to apply them in any
order, including with duplicates,
and get the same, deterministic
results at the end. Said another
way, CRDTs have built-in conflict
resolution, so you don’t have
to do that messy work in your
application. Formally, they exhibit
something called strong eventual
consistency.
This property, by itself, means
any system built with CRDTs
can have absolutely trivial fault
management. Try to send the op-
erations, in any order and without
any coordination, to the nodes
that need them. If there’s a prob-
lem, just try again later. That’s it.
As long as the operations even-
tually get where they need to go,
the system is guaranteed to be
correct. By choosing a smarter
state primitive, we can build a
much simpler and more reliable
system.
CRDTs enable a lot of cool use
cases, like offline-first docu-
ment editing that can sync up
automatically. In the context of
edge state, they let us keep our
transactions local to the point of
presence, to meet our latency re-
quirements, while still being able
to share state globally.
Of course these benefits don’t
come for free. For one thing, it’s
not always easy, or even obvi-
ous, how to model your data as
29
 CRDTs. Simple-seeming opera-
The InfoQ eMag / Issue #87 / November 2020
tions like delete can have fiend-
ishly complex CRDT equivalents.
Also, the sometimes messy de-
tails of multiple parallel versions
of state tend to surface in the
APIs of these systems, and ap-
plications have to adapt to deal
with them, which isn’t always
easy. Most significantly, a single
byte of usable, logical state in a
CRDT requires many more bytes
of actual memory, which can
quickly render the economics of
these systems infeasible.
Like any technology put to pro-
ductive use, engineering compro-
mises, trade-offs, and optimiza-
tions are required to make CRDTs
viable.
InfoQ: You made the argument
that synchronous calls for state
synchronization might be the
better option than an asynchro-
nous, event-driven one. Why is
that?
Bourgon: This was sort of a mi-
nor point, but an important one,
related to the implementation of
distributed systems. It’s essen-
tially impossible to build confi-
dence in the safety and reliability
of large-scale systems like this
without being able to run the
system under deterministic test,
or simulation. And it’s essentially
impossible to simulate a system
unless each component can be
modeled as a plain, determinis-
tic state machine. It’s certainly
possible to get these properties
using asynchronous events as
30
the messaging pattern, but ex-
perience has taught me that it’s
significantly easier with a syn-
chronous RPC-style approach.
Similar to how CRDTs eliminate
entire classes of complexity
related to fault handling, syn-
chronous RPCs eliminate entire
classes of complexity related to
queueing theory. They get you
automatic backpressure, they let
you take advantage of the queues
that already exist at various lay-
ers of the operating system and
network stack, and, importantly,
they make it a lot easier to build
deterministic components.
InfoQ: You concluded your
talk by saying that “consensus
rounds, or leader election, or
distributed locks, or distributed
transactions” are dead ends,
and that large scale systems are
going to use simple communica-
tion, and be eventually consis-
tent. Tell us more about why you
think that’s the case.
Bourgon: I think the whole
of human technological
achievement has been an
exercise in creating, reifying,
and extending abstractions.
The human capacity for
understanding and managing
complexity is essentially fixed; in
order to make more and greater
things possible, we have to
use abstractions to “wall off”
domains of complexity behind
simpler models that can be
more easily understood and built
upon. For example, application
developers today don’t need
to know how their network
interfaces translate binary
data to electrical signals over
copper, or ethernet frames to
binary, or datagrams to ethernet
frames, and so on — the OSI
model’s abstractions enable
the developer to spend their
complexity budget on a much
higher level. It’s basically stood
the test of time, so I think it’s a
pretty good set of abstractions.
Developers have also relied
on the abstraction of a single,
global truth in their data layer
since essentially the very first
database. It’s understandable
and productive, so that makes
sense. But I think it’s a bit like the
Newtonian model for physics: it
works, until it doesn’t. At really
large scale, Newtownian physics
no longer predicts real-world
behavior, so we have to switch
to the more complex Relativistic
model. Similarly, when we start
extending our data layer across
large physical distances, I believe
the abstraction of a single, global
truth begins to leak. An incredible
amount of engineering effort is
required to maintain the illusion
of atomicity, using techniques
like the ones I listed. At some
point, the developers working in
and around this layer of abstrac-
tion are going to exhaust their
complexity budget.
Typical RDBMS systems and
common distributed systems
techniques are incredibly useful
and productive up to a certain
scale. But past that scale, the

complexity required to prop up
the illusion of global atomicity
becomes unreliable, unproduc-
tive, and ultimately unjustifiable.
While there’s an upfront cost to
thinking about multiple parallel
universes of state in your appli-
cation, I believe biting that bullet
offsets orders of magnitude
more hidden complexity in the
alternate, leaky abstraction. And
I believe that, eventually, we’re
going to realize it’s the only way
Peter Bourgon
s currently leading re-
search and development
on a global infrastructure
for state at the edge at
Fastly, a CDN and edge
cloud platform. He is
the author of Go kit, the
preeminent toolkit for
microservices in Go; and
several large-scale coordi-
nation-avoiding distributed
systems, including Roshi
(stream index) and OK Log
(log aggregation).
to build reliable, large-scale dis-
tributed systems.
For what it’s worth, this isn’t ex-
actly novel thinking. The natural
world is full of incredibly complex
systems whose behaviors are
emergent from simple primitives
and rules. My favorite example is
probably how groups of fireflies
manage to synchronize their
lights — no leader election or
consensus rounds involved.
The InfoQ eMag / Issue #87 / November 2020
TL;DR
• Latency is the biggest
reason that a gener-
al purpose database
isn’t a good fit for edge
architectures.
• CRDTs, or conflict-free
replicated data types, are
a relatively novel state
primitive that give us a
way to skirt around a lot
of the complexity around
consistency.
• Similar to how CRDTs
eliminate entire classes
of complexity related
to fault handling, using
synchronous remote
procedure calls for syn-
chronization eliminate
entire classes of com-
plexity related to queue-
ing theory.
• Typical RDBMS systems
and common distributed
systems techniques are
incredibly useful and
productive up to a cer-
tain scale, and after that,
the complexity becomes
unjustifiable.
31
InfoQ @ InfoQ InfoQ InfoQ

Curious about
previous issues?
The InfoQ eMag / Issue #83 / March 2020 The InfoQ eMag / Issue #77 / October 2019 The InfoQ eMag / Issue #81 / January 2020

Service Mesh Taming Complex Microservices:

Ultimate Guide Systems in Production Testing, Observing,
and Understanding
@emilywithcurls

Service Service Mesh Exploring the An Engineer’s Sustainable Operations Testing in Tyler Treat on
12 Microservices Obscuring
Mesh Implementations (Possible) Future of Guide to a Good in Complex Systems with Production—Quality Microservice
Testing Techniques Complexity
Features and Products Service Meshes Night’s Sleep Production Excellence Software, Faster Observability

FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT
FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT

This eMag aims to answer To tame complexity and its This eMag takes a deep
pertinent questions for effects, organizations need dive into the techniques and
software architects and a structured, multi-pronged, culture changes required
technical leaders, such as: human-focused approach, to successfully test,
what is a service mesh?, that: makes operations observe, and understand
do I need a service mesh?, work sustainable, centers microservices.
and how do I evaluate the decisions around customer
different service mesh experience, uses continuous
offerings? testing, and includes chaos
engineering and system
observability. In this
eMag, we cover all of these
topics to help you tame the
complexity in your system.