OVHcloud’s CEO AWS’s spotty pricing Genomic treasures

A European When cloud costs Mining with

approach to data change Wellcome Sanger

Issue 34 • November 2019

datacenterdynamics.com

Plus: Show preview

and highlights for
>London

Supplement
The Telco Story
5G cometh Head to space Sell it all
Readying yourself for The skies are filling with Telcos are lining up to sell
a data deluge new opportunities their data centers
 R

High Rate & Extreme Life High Rate Batteries for

Critical Power Data Center Applications

RELIABLE
Narada HRL and HRXL Series batteries deliver! Engineered for exceptional
long service life and high rate discharges, Narada batteries are the one to
choose. Narada can provide solutions to meet any Critical Power need.

ISO9001/14001/TL9000 Certified Quality

Backed by Industry Leading Warranties
Narada...Reliable Battery Solutions
www.mpinarada.com - email : ups@mpinarada.com - MPI Narada - Newton, MA
Tel: 800-982-4339
 ISSN 2058-4946

Contents
November 2019
6 News

12
Police raid NATO bunker used as
illegal data center

12 Globalization and its disconnects

Nations are shutting down the Internet
in an effort to quell protests. But they
risk fracturing the web as great powers

49 disentangle, fundamentally changing

the net as we know it

Industry interview
20 Michel Paulin, OVHcloud
“We do believe that our roots
are in Europe and in openness.
Transparency, GDPR and open
source are European values. When
we go to Asia and Australia, do we

20
have to change these key DNA
values? I don’t think so.“

22 Uptime on downtime

59 26 Why outages have vastly different

impacts on different companies

23 The Telco supplement

A special supplement delving into all
things telco - from 5G and Edge, to
the CORD initiative, to space-based
Internet. Plus, learn about fiber’s
renaissance and why telcos are selling
off their data centers

39 Amazon’s spotty pricing

AWS said they were making pricing
smoother. It ended up more expensive
and less predictable

46 Virginia’s land dilemma

When space is at a premium, perhaps
it is time to look elsewhere

49 Building a home for AI

Prepare for high-density workloads of
the future, today

34 51 The London show preview

What to watch and where to go

59 Mining genomes
We head to the Wellcome Sanger
Institute’s data center

63 Storage wars
Batteries still rule when you want to
keep charged

63 66 Do I even have to say it?

Stop arguing about scientific facts

Issue 34 • November 2019 3

From the Editor
The Internet wasn't
meant to be like this
T
he Internet was going
to change everything.
We'd have frictionless
access to any goods,
cutting waste and
freeing up resources.
We'd have anonymous virtual worlds
to dream up utopias. And online access
to information would blitz conspiracy
theories and educate us all.
Instead, it seems like things are
the same or worse. Social media has
empowered the far right, e-commerce
has powered inequality.
Block protesters
from the Internet, and
it may increase
violence, not reduce it
The great disconnect. Efforts by
nation states to intervene with online
power have turned out to be either
futile, or sometimes worse than the
problem they perceive.
Some countries have protocols in
place to limit access to the Internet.
China has the most hardline control
over its Net, but the results are mixed.
To take one example, some
states have the ability to shut down
communications completely to
hamper the activity of protesters or
political opponents.
Even if you think that's a good idea,
there's a problem. Research suggests
protesters without the Internet may
actually be more likely to turn violent.
Sebastian Moss found plenty of
surprises in his investigation of state
control over the Internet (p12).
Dive deeper
Follow the story and find out more about DCD products that can further expand your knowledge.
Each product is represented with a different icon and color, shown below.
Events Intelligence
4 DCD Magazine • datacenterdynamics.com
Meet the team
55PB
Global Editor
Peter Judge
@Judgecorp
Deputy Editor
Sebastian Moss
@SebMoss
Reporter
Alex Alley
SEA Correspondent
Paul Mah
GDPR versus the CLOUD Act? The @PaulMah
US has caused controversy with its Brazil Correspondent
expectation that it should have access Tatiane Aquim
@DCDFocuspt
to private communications of citizens
elsewhere in the world. Head of Design
Dot McHugh
The European GDPR guarantees
Amount of Designer
online privacy of individuals. But it's
storage the Mandy Ling
not just a political issue: for OVHcloud, Wellcome
the leading European cloud player, it's a Head of Sales
Sanger Martin Docherty
business model. Institute says it Conference
CEO Michel Paulin (p20) told us the has in its data Director, Global
world needs a cloud provider outside center... but Rebecca Davison
of the increasingly intrusive US and it's growing at Conference
Chinese regimes. 30 percent per Director, NAM
year Kisandka Moses
Curing illness with DNA data. It was Chief Marketing Officer
a pleasure to step away from politics Dan Loosemore
and visit the Wellcome Sanger Institute,
Head Office
to see a data center in harmony with
scientific research (p59). DatacenterDynamics
102–108 Clifton Street
Data center manager Simon Binley
London EC2A 4HW
faces unprecedented demand, as the +44 (0) 207 377 1907
Institute's genetic sequencers generate
petabytes of DNA data. But his spend
comes from a budget that also saves
lives through genomic research - and
any money he saves enables more of
that research.
The Institute's facility is evolving
in tandem with the Institute itself, and
Binley only takes the upgrades that really
support its work.
I'm looking forward to joining
Simon on stage at DCD>London on 5-6
November, to tell the story. PEFC Certified
This product is
from sustainably
managed forests and
controlled sources
Elsewhere this issue we explore the PEFC/16-33-254 www.pefc.org
most revolutionary frontiers of the
telecoms world in a supplement (p23). © 2019 Data Centre Dynamics Limited All
rights reserved. No part of this publication
We also hear about AWS pricing may be reproduced or transmitted, in
any form or by any means, electronic,
discrepancies (p39), look in detail at mechanical, photocopying, recording
DCD>London, and round up the most Peter Judge or otherwise, or be stored in any
retrieval system of any nature, without
vital news in the field (p6). DCD Global Editor prior written permission of Data Centre
Dynamics Limited. Applications for
written permission should be directed
bit.ly/DCDMagazine to the editorial team at editorial@
datacenterdynamics.com. Any views or
opinions expressed do not necessarily
represent the views or opinions of Data
Centre Dynamics Limited or its affiliates.
Disclaimer of liability: Whilst every effort
has been made to ensure the quality and
accuracy of the information contained in
this publication at the time of going to
press, Data Centre Dynamics Limited and
its affiliates assume no responsibility as
to the accuracy or completeness of and,
to the extent permitted by law, shall not
be liable for any errors or omissions or
any loss, damage or expense incurred by
reliance on information or any statement
contained in this publication. Advertisers
are solely responsible for the content
of the advertising material which they
submit to us and for ensuring that the
material complies with applicable laws.
Data Centre Dynamics Limited and its
affiliates are not responsible for any error,
omission or material. Inclusion of any
advertisement is not intended to endorse
any views expressed, nor products or
Debates Training Awards CEEDA services offered, nor the organisations
sponsoring the advertisement.
Connect to maximum density

with faster installations.

As the popularity of cloud computing and big data
grows, the demands for high-speed transmission and
Visit www.corning.com/emea/en/dc-solutions
data capacity are greater than ever before. Address
to learn more about how to overcome cabling
your most challenging data centre concerns with our
high-fiber-count MTP® trunks, a preterminated solution challenges in larger networks operating at
offering increased density, easier cable management, higher speeds.
and enabling reduced installation time.

© 2019 Corning Optical Communications. LAN-2472-BEN / September 2019

 Whitespace
News
Whitespace: The biggest data center
news stories of the last three months
Police raid illegal NATO bunker
The German data center was hosting all sorts of crimes
A “criminally operated data center” in a
former NATO bunker was shut down after it
was discovered hosting sites for child porn,
drug dealing, and botnets.
More than 600 police officers stormed
the ‘CyberBunker’ data center in Traben-
Trarbach, western Germany, where they
seized roughly 200 servers.
Seven people were arrested at a local
restaurant and in Schwalbach, outside
Frankfurt.
13 other people aged between 20 to
59 are under investigation. None of the
suspects were at the data center when the
arrests took place.
Acquired in 2013, the former NATO
facility was bought from the Office for
Geoinformation of the Bundeswehr, by an
unidentified Dutchman - the chief suspect.
The Dutchman, now 59, upgraded the
bunker “to make it available to clients,
according to our investigations, exclusively
for illegal purposes,” regional criminal
police chief Johannes Kunz said.
“I think it’s a huge success... that we
were able at all to get police forces into the
bunker complex, which is still secured at
the highest military level.”
When the bunker was purchased
6 DCD Magazine • datacenterdynamics.com
in 2013, the buyer was not identified
but said that he was also involved with
CyberBunker, the alleged operator of a
Dutch data center in its own Cold War
bunker.
In 2013, the now-defunct data
center company ‘Bunker Infra’ claimed
CyberBunker was using images of its
bunker but was not based there.
CyberBunker previously said it would
host “services to any Web site ‘except
child pornography and anything related
to terrorism.’” The company’s website has
been seized by the German police.
The location of the Traben-Trarbach
facility matches that of Calibour, a
company that said it operated a NATO-
bunker based secure data center. Its website
is now also unavailable. The CEO and MD
of Calibour, Herman-Johan Xennt, claimed
to own CyberBunker as of 2010.
The cases are still developing, and
there are as yet no formal identifications
or charges. While 200 servers were seized,
some reports suggest that there could be
as many as 2,000 at the facility. Kunz told
reporters the data analysis could take years.
bit.ly/NATOdiversifies
NEWS IN BRIEF
Equinix’s $1bn hyperscale joint
venture with GIC set to begin
Equinix has completed the formation of
a $1bn joint venture with its Singaporean
sovereign wealth fund partner, GIC.
The unnamed JV will develop hyperscale-
focused data centers in Europe.
LinkedIn to move to Microsoft
Azure in multi-year effort
After acquiring professional networking
site LinkedIn for $26.2bn back in 2016,
Microsoft is set to shift the company to its
Azure cloud service.
Cray wins $600m
supercomputer contract for
National Nuclear Security
Administration
El Capitan, is expected to be delivered
in late 2022, and run classified nuclear
weapons simulations. It will have a peak
performance of 1.5 exaflops.
Apple to sell Irish land after
data center site construction
fails to get off the ground
The proposed location of the $1bn
Derrydonnell data center in Athenry,
County Galway, is up for sale.
Five years of protests and court battles
delayed the project, culminating in the tech
giant throwing in the towel.
Trans-Saharan and Tatweer to
deploy Libyan data center
The small prefabricated facility will be
deployed in Tripoli next year. “There are
multiple challenges, mainly the political
unrest, electricity instability & infrastructure,”
Ehab Elghariani, Trans-Sahara’s DC unit
manager, told DCD. “We are willing to
overcome the latter two challenges and
minimize the impact of the first.”
Share-online.biz shut down
in police raids across three EU
countries
Police from Germany, France, and Holland
have conducted raids on multiple data
centers in a “crackdown” on an illegal
hosting site. Share-online.biz, the largest
file hosting site in Germany, was taken
down in raids led by Cologne prosecutor
Christoph Hebbecker and the Cybercrime
Nordrhein-Westfalen (ZAC NRW) division.
 China’s data centers
primarily coal powered
According to a study by Greenpeace
and the North China Electric Power
University, China’s data center
industry was responsible for 99
million tonnes (109m US tons) of
CO2 emissions in 2018.
The regional data center
industry consumed 161TWh of
READ MORE electricity in 2018.
Understand The facilities - which spanned
the deeper US- 150 million square meters (1.6bn
China struggle, sq ft) in 2017 - mostly relied on
p12 grid energy, with coal providing
73 percent of power used by the
US sanctions hit China’s tech firms data centers in 2018.
Without serious changes,
The companies have been blacklisted by the US amid claims of human carbon emissions are expected to
rights violations against the muslim minority in China rise drastically.

Eight tech companies are among the
28 Chinese public security bureaus and
companies on The US Commerce Department’s
“Entity List,” essentially blocking them from
doing business with American firms.
The blacklist is purportedly over their
involvement in human rights violations against
Muslim minorities in Xinjiang.
SenseTime, the world’s most valuable
artificial intelligence startup (at least before
the ban), the large AI company Megvii, and
facial recognition firm Yitu Technologies were
among those put on the Entity List.
Also listed were surveillance companies
Hikvision, Zhejiang Dahua Technology
Co, voice recognition company iFlytek, bit.ly/ChokeOnThis
cybersecurity group Meiya Pico and nanotech
firm Yixin Science and Technology.
Hikvision, one of the world’s largest security
camera makers, could be among the hardest
hit, with its servers likely impacted.
A Commerce Department spokesperson told
reporters the ban was unrelated to the US and
Chinese trade negotiations despite the talks
resuming on the same day.
It follows a similar blacklist against
supercomputing companies Sugon, Hygon,
and telecoms giant Huawei.
bit.ly/SenseCrime

Mission Critical Training

Colocation/MTDC
Managing training within organizations that provide “Infrastructure
as-a-Service” is complicated given the variety of learning
requirements needed, but it is essential to maintain the
competitive edge. DCPRO has a flexible approach to workforce Head over to
development and we even work with you to ensure our materials www.dcpro.training
fit your bespoke requirements exactly. for more

“DCPRO’s training courses are always informative and interactive. The trainers are very experienced and
knowledgable. I recommend these courses not only to the operations team, but to anyone who works at a data
center to understand the criticality of running a data center,” Charlene Gomez | Digital Realty
 Whitespace

“A profound sense of sadness and loss:”

Oracle co-CEO Mark Hurd passes away
The CEO lauded as one of the ‘greatest salesmen in
Silicon Valley’ has died at 62

Oracle co-CEO Mark Hurd has passed
away. His death was announced in
a company-wide email by Oracle
Chairman Larry Ellison on October 18.
Back in September, 62-year old Hurd
began a leave of absence for unspecified
health-related reasons.
It was understood co-CEO Safra Catz
and Oracle founder Ellison would assume
his responsibilities during this time.
Ellison wrote in the email: “It is with a
profound sense of sadness and loss that
I tell everyone here at Oracle that Mark
Hurd passed away early this morning.
“Mark was my close and irreplaceable
friend and trusted colleague.
“I know that many of us are
inconsolable right now, but we are left
with memories and a sense of gratitude…
that we had the opportunity to get to
know Mark, the opportunity to work with
him… and become his friend.”
It’s likely the company will look for a
new partner soon to replace Hurd since
Ellison has reportedly grown to appreciate
the dual-CEO system.
One option is Jeff Henley, Oracle’s vice
chairman, and former CFO.
According to Bloomberg, Ellison once
mentioned Don Johnson, head of Oracle’s
cloud infrastructure division, and Steve
Miranda, head of Oracle’s applications
unit as possible replacements to Hurd.
As of yet, no announcements have
been made.
Appointed by then-CEO Ellison in
2010, Hurd was named president of Oracle
Corporation alongside Safra A. Catz.
In 2014, he and Catz were named joint
CEOs when Ellison stepped down.
bit.ly/SiliconValleysGreatestSalesman

We’re putting the

power in data.
Temporary power and battery solutions for
data centres.
Our temporary power and battery solutions will provide
your data centre with the power you need, when you
need it, for as long as you need it. Bridging any gap in
demand while you’re building a permanent off-grid
solution. Aggreko put the power into data.

Tell us what
8 DCD you
Magazine need 0333 016 3475
• datacenterdynamics.com aggreko.com
 Banking services
across Mexico
down due to Prosa
outage
An outage at a data center
in August brought much of
Mexico’s banking services
offline, with customers
unable to make purchases
or withdraw cash.
Electronic transaction
services firm Prosa said
that an electrical fault at
its data center in Santa Fe,
Mexico City, was to blame.
It impacted customers
of Banorte, HSBC, Invex,
Santander, Scotiabank,
and Banjército.
NERSC shuts down amid blackout The company said
at the time: “We want to
California goes dark after PG&E cuts power over risk of wildfires inform you that today we
are having an outage on
On October 9, as utility PG&E cut power to hundreds petaflops Cori supercomputer. our Santa Fe data center.
of thousands of Californians in an effort to reduce NERSC’s HPC systems are used by 7,000 scientists “The management
the risk of wildfires, supercomputers were forced to working on various research projects. team and the entire IT
shut down. “Our users run large scale climate models, they and Innovation team are
The National Energy Research Scientific run large scale simulations of exploding stars, they working as a priority in
Computing Center, part of the Lawrence Berkeley run large scale simulations of a fusion model,” Katie resolving this incident.”
National Lab, turned off its supercomputers as power Antypas, division deputy and data department head It took several hours for
went out. at NERSC, told DCD earlier this year. services to start to resume,
“PG&E has informed us that they will definitely be NERSC detailed research it is working on and hours more for cards
cutting power to the Berkeley Lab campus sometime simulating the ‘Camp Fire’ wildfire that last year killed to work.
between 12:01 am (Pacific) and noon Wednesday. 86 people and burned more than 150,000 acres. The outage comes at a
Berkeley Lab is closed effective Wednesday, October With PG&E trying to avoid a Camp Fire scenario time Mexico wants to cut
9 at 12:01 am. NERSC will continue to operate until reoccurring, such work had to be paused, as were down on cash and move
power is cut by PG&E,” user engagement group leader other research projects studying the impact of to electronic banking
Rebecca Hartman-Baker said in an email to NERSC anthropogenic climate change. systems.
users ahead of the cut. Systems returned online on October 12.
All of the high-performance computing facilities bit.ly/TakingThePeso
at NERSC had to be shut down, including the 30 bit.ly/NERSCgetsNerfed

Software outage knocks 500 stocks

off London Stock Exchange
The London Stock Exchange staggered when it opened on
August 16 after a “technical software issue” caused its longest
outage in eight years.
489 stocks were unable to trade for an hour and forty
minutes after London started trading at 8am, including those in
the FTSE 100 and 250 indexes.
An LSE spokeswoman refused to rule out whether its trading
software was at fault.
The LSE handles £5bn ($6bn) worth of trading each day and
is currently in talks to acquire Refinitiv, a financial markets data
and infrastructure firm, for $27bn.
A similar technical error caused an outage in June 2018. The
LSE also suffered outages in 2011 and 2009.

bit.ly/StockShocker

Issue 34 • November 2019 9

 Whitespace

US military to
acquire three Cray
supercomputers for
$71m
The US Air Force will
deploy a Cray Shasta
supercomputer, while the
Army Research Lab (ARL)
and the US Army Engineer
Research and Development
Center (ERDC) will each
deploy a Cray CS500.
The contracts are worth
more than $71m.
The Air Force’s $25m
DoD awards controversial $10bn
system will be acquired
by the Air Force Life Cycle
JEDI cloud contract to Microsoft
Management Center in Azure goes to war
partnership with Oak Ridge
National Laboratory. After delays, legal fights, employee protests, and Amazon, once the front runner for JEDI, is
Named HPC11, it will be an intervention by President Trump, the US thought to be considering a legal challenge.
used for meteorology to Department of Defense have awarded the long- With the President already known to be
help the US Air Force and discussed JEDI cloud contract. negatively inclined towards Amazon CEO Jeff
Army operate in numerous Microsoft will provide its services for the Bezos - due to his ownership of The Washington
theaters. Joint Enterprise Defense Infrastructure cloud Post - and with the frequent Fox News-
The other Cray CS500 in a deal that could last 10 years and be worth as watching world leader tuning into segments
supercomputer will be much as $10bn. about possible JEDI corruption by Amazon,
deployed by the US Army’s “This contract will address critical and there were several rumors throughout the year
Engineer Research and urgent unmet warfighter requirements for that he would intervene to stop AWS winning
Development Center modern cloud infrastructure at all three JEDI. In Holding The Line: Inside Trump’s
(ERDC). classification levels delivered out to the tactical Pentagon with Secretary Mattis, author Guy
ERDC manages the DoD edge,” the Department of Defense said in a Snodgrass claimed that Trump called Mattis in
Supercomputing Resource statement. the summer of 2018 and directed him to “screw
Center (DSRC) at Vicksburg, “The DoD will rigorously review contract Amazon” out of a chance to bid on JEDI.
Mississippi. performance prior to the exercise of any options,”
DSRC typically the Department said. bit.ly/ExpectToSeeThisInCourt
operates two or more
supercomputers on an
average four-year life cycle. Peter’s military factoid
Work underway at the
site includes research into Palantir is developing an $800m Distributed Common
nanotechnology. Ground System (DCGS-A) to act as the Army’s primary system
to track troop movements, enemies, weather and more
bit.ly/CrayCrayCray

US Army buys $12m IBM shipping container

supercomputer The IBM system is housed in a could be placed near to the theater
shipping container with on-board
uninterruptible power supply, chilled
water cooling, and fire suppression
systems.
The HPC-in-a-Container is
designed to be deployable to the
tactical edge; with deployment
opportunities to remote locations
“currently being explored and
evaluated.”
It is unlikely that the system would
be deployed on a battlefield itself, but
of war.
It will be deployed at the US Army
Combat Capabilities Developmental
Command Army Research Laboratory
DoD Supercomputing Resource
Center at Aberdeen Proving Ground,
Maryland, later this year.
The system is capable of six
petaflops of single precision
performance.
bit.ly/ComesAlreadyWrapped

10 DCD Magazine • datacenterdynamics.com

 READ MORE EU biological data unit moves into Kao
Learn about
genomics and campus, London
data centers,
p59 EMBL-EBI takes 1.5MW for genomic sequencing and other
bioinformatics work

The European Bioinformatics Institute else in smaller quantities called “cells” or

(EMBL-EBI) has signed up for 1.5MW
of data center capacity at Kao Data, the
science-focused campus being built in
North London.
Cambridge-based EMBL-EBI has
taken a substantial chunk of capacity at
Kao Data One, the first of four 8.8MW data
centers scheduled to be built in the Kao
campus, within the Harlow Enterprise
Zone, close to the M11 motorway between
London and Cambridge.
EMBL-EBI already holds some 270
petabytes of biological data, and its job
is to make it available to the scientific
community.
Kao Data One is the first of four
projected 8.8MW data centers.
The company plans to sell capacity
either a whole building at a time, or
2.2MW suites (a quarter of a building), or
“pods.”
The Institute took space in Kao Data
so its data center engineers could have
easy access to the equipment, saving on
operational expenditure.
EMBL-EBI’s data storage demands are
growing daily, and it could scale quickly
into TS02 if required in future, according
to the company.
“The biological data we store and
share through our data resources are
used by life science researchers all over
the world to power new discoveries,”
said Steven Newhouse, head of technical
services, EMBL-EBI.
“As such, data center space, physical
security and infrastructure availability
were critical in our decision-making.”
bit.ly/NewhousesNewHouse

WHEN REPUTATION RELIES ON UPTIME.

Solving your data center power challenges are our priority. Our engineering teams can create a
vertically integrated solution for your facility. No matter the fuel type, our trusted reputation and
unrivaled product support allow you to maintain uptime with backup power that never stops.

DEMAND CAT® ELECTRIC POWER.

© 2019 Caterpillar. All Rights Reserved. CAT, CATERPILLAR, LET’S DO THE WORK, their respective logos, “Caterpillar Yellow”, the “Power Edge” and Cat
“Modern Hex” trade dress as well as corporate and product identity used herein, are trademarks of Caterpillar and may not be used without permission.
 Cover Feature | The Great Disconnect

Governments are shutting down the Internet, using

digital sieges to quell unrest, and threatening the
Balkanization of the web. Sebastian Moss reports

Sebastian Moss
Deputy Editor

30 DCD Magazine
12 Supplement
• datacenterdynamics.com
• datacenterdynamics.com
E
very time web traffic suddenly
drops in a particular country,
an alert goes off in Cloudflare’s
headquarters in California.
“It could be that there's
something wrong with one of
our points of presence, or that something’s
wrong with the connectivity,” John Graham-
Cumming, the web infrastructure company’s
CTO, told DCD. “Our first reaction is ‘did we
break something?’ And we want to be able to
fix it.”
The alert is repeating again and again,
but there’s nothing Cloudflare can do. The
outages are real, but nothing is broken - it’s
an intentional disconnection. “We actually
have an internal chat room called Internet
Shutdown Tracking, because we see these
things happening pretty regularly.”
One moment a country is part of the
Internet, a piece of the whole. The next,
darkness, a nation winking out of digital
existence, unmoored and alone.
“The thing to note about state-sponsored
cut-offs of the Internet is how widespread
they are,” Graham-Cumming said. “Even
this year, there’s been Venezuela, Sudan,
Indonesia, Sri Lanka, and Ethiopia, to name
but a few. The Democratic Republic of Congo
shut down the Internet completely for 20
days - it's a long list of shutdowns, and some
of them are quite large.”
Authoritarian regimes are the most
common abusers of this power, forcing
telecommunications companies, which are
often state-owned, to shut down operations.
But it is in a young democracy that the worst
offender is found.
INDIA'S SHUTDOWNS
“There’s been quite a lot of advocacy around
some of the shutdowns that happened in
Africa, which were largely on a national
level and executed by central authorities,”
censorship and connectivity researcher Jan
Rydzak told DCD.
“There were surprisingly few people
focusing on the number one country in
which the most shutdowns were taking
place: India. Since 2012, the country has had
approximately 350 cases of shutdowns of
various kinds at various levels. This is just
orders of magnitude more than any other
country in the world.”
That count, ever-growing, has
been carefully tallied by the Software
Freedom Law Center, India, which began
tracking the outages in lieu of any official
announcements.
“In 2012, we started noticing - in addition
to website and content blocking - complete
blanket shutdowns of the net in certain
areas,” Mishi Choudhary, ‘SFLC.in’ founder
and human rights lawyer, said. “It started
with around three shutdowns. By 2014, it was
still in the single digits. Then in 2015, we saw
a spike in the number of instances to around
14.
“Last year, we had the highest numbers
we've ever seen - 134.”
Choudhary’s figures are on the
conservative side, she noted. Only outages
that the center can confirm as intentional are
included, with media reports and tips used
as a starting point, backed up by SFLC’s legal
challenges to individual states, demanding
information.
India’s federal government has a long
process for deciding whether to initiate
a shutdown, with checks and balances in
place, but the country’s 28 states are not
bound by the stipulations of the Information
Technology Act. “What the states started
doing is use the Police Power, a very different
statute,” Choudhary said.
In cases of civil unrest, police in mostly
northern and western states are turning to
shutdowns to try to defuse the situation. “It
can become a checklist for the police when
they do their job: They think ‘first shut the
Internet down, that will stop this viral spread
of messaging, and then go and control the
situation on the ground.’”
This approach can be tempting for those
in power, Choudhary admitted. “India has a
long history of communal riots exacerbated
by our colonial masters. Social media
has really amplified the messaging and
aggravated the situation, and made it very,
very easy for a large group of people to be
able to receive messages and assemble in
one place.
“As much as I am a free speech advocate,
as much as I love the Internet, I am not blind
to the fact that if a police commissioner or
magistrate who thinks there are going to
be 2,000 people [converging] that want to
kill each other because of some WhatsApp
message, they may want to shut it down.”
The problem is there’s no data to confirm
that shutting down the network actually
helps law enforcement agencies do their job,
or stops the messaging, Choudhary said.
“It's not that the riots didn't happen before
these tools were there. People use phones,
there was word of mouth, people live in areas
together, there are ghettos. There's a lot that
goes on in a riot.”
Issue 34 Edge
∞ November 2019 13
Supplement 31
 Cover Feature | The Great Disconnect
Working at Stanford’s Global Digital
Policy Incubator, Rydzak has tried to
analyze whether the shutdowns were
actually effective. His paper ‘Of Blackouts
and Bandhs: The Strategy and Structure of
Disconnected Protest’ took SLFC’s Indian
outage data, as well as datasets on protests
(their location, length, who was involved,
and whether it was violent), to see how the
shutdowns changed their nature.
“Essentially, I was trying to look at
whether the protesters’ strategy changes
with a lack of access to information and
communication,” he said. “In an information
vacuum, does information travel differently,
and does it lead to different outcomes for
protests?”
In non-violent protests, the results proved
“very ambiguous and inconsistent,” Rydzak
said. “Shutdowns are sometimes effective
against peaceful demonstrations, but it's
by no means guaranteed. It's practically no
different from a coin toss.”
It was in violent protests that Rydzak saw
a real difference. “Shutdowns are followed by
an escalation in violent protests. It's a very
strong effect that doesn't just refer to the
first day that a riot takes place, but to several
subsequent days as well.
“People will always find a conduit for
protest. Social media is just a platform for
people to vent their frustration and anger.
This is just a hypothesis, but it's possible that
anger that is normally spilled out on social
media can spill out into the streets instead.”
THE SIEGE ON KASHMIR
The majority of intentional outages last up
to 72 hours, short disruptions across a state,
or sometimes a smaller area. Other times
the blackout is widespread, long-lasting and
total. Rydzak calls these events Digital Sieges.
“What's happening in Kashmir is
14 DCD Magazine • datacenterdynamics.com
absolutely a siege. It's just unprecedented.
Landlines are not usually affected. But in
this case, the government decided to take
no risks and just cut off all communication
completely. It's a siege in every sense of the
word; not only a militarized siege, but also a
siege of all forms of communication. You'd
be hard-pressed to find an equally extreme
example even among the hundreds of
shutdowns that we've seen so far.”
The longest shutdown ever recorded
happened in the state of Jammu and
Kashmir in 2016, lasting 133 days. Now, in
October, the state is offline again, but this
time it’s different. “This is the first time
that phones were completely shut down -
landlines, mobiles and Internet - everything,”
Dr. Mudasir Firdosi, a Kashmiri psychiatrist
and writer based in London, told DCD.
In Jammu and Kashmir - a troubled state
with high levels of unrest, an insurgency
movement, and regular terrorist attacks
- darkness has prevailed since August
4th. “Even in the modern world today, it
is possible to isolate a large population of
eight million people and not let them talk,”
Choudhary said. “Again, there are national
security reasons for it, we can’t deny them -
but it has a real impact.”
72 days into the siege, a small opening
was allowed. On October 15, phone calls
from ‘postpaid’ contract cell phones were
let through, while calls from the more
commonly used top-up phones remain
blocked. “I believe the reason for that is when
you take a postpaid connection in Kashmir,
they verify your identity, they know who you
are,” Firdosi said.
With limited connection resumed, the
Kashmiri diaspora is finally able to connect
with loved ones in the state. In some cases,
the news has been dire - relatives have
learned of illnesses or deaths; funerals have
been missed, weddings delayed.
“This has got so many costs,” Firdosi
said. “We are living in the Internet age,
students are sitting at home, people have to
fill in forms for jobs or for higher education.
Businesses are run on the Internet.
Everything is down.”
“People at some places didn’t get
$2.4bn
The cost of
shutdowns
in 2015
immediate medical help, resulting in deaths
and aggravation of terminal illnesses,”
Aakash Hassan, Kashmir correspondent at
CNN-News18, told DCD.
For journalists such as Hassan, the
situation is fraught with danger. “There have
been multiple cases of journalists being
detained and even injured while covering
stories. One photojournalist was injured with
pellets - this is the physical aspect,” he said.
“[But the] intangibility of this clampdown
has affected reporters the most, because
they are the ones who have to write about it
and get news out… We have been provided
a facilitation center by the administration
where we can use the Internet for around
half-an-hour in 24 hours. Each day, we bring
our stories and have to wait in line to file.
There has never been a time when journalists
were so disempowered.”
Fear prevails. No one knows who could be
listening. Firdosi recalled conversations with
doctors in Kashmir who were granted limited
mobile access: “When I start asking them
how the situation is, they just start saying,
‘oh, the weather is good.’ They don’t talk
about it. People are afraid.”
The distress is not limited to the region.
Firdosi and colleagues are studying the
impact of the disconnection on Kashmiris
living abroad. “We have a survey with around
450 responses,” he said. “Though we can’t
diagnose people on surveys, it just gives an
indication, but 88 percent showed abnormal
scores pointing to cases of depression or
anxiety.”
Using the ‘Hospital Anxiety and
Depression Scale,’ more than 90 percent
scored high on the ‘frightened feeling as if
something bad is about to happen’ section
of the survey. “It’s the not knowing,” Firdosi
said. “It has taken over our lives - I am at
work right now, and I am still thinking about
it.”
So far, these outages have primarily
impacted less influential areas. “If something
were to happen in Delhi, Bombay, or Calcutta,
the noise would be heard and ricochet all
around the world,” Choudhary said.
This is partly because they are seats of
power, globally recognized regions deeply
integrated with the wider world. It may also
be because these are areas with higher levels
of Internet penetration, where a shutdown
would have a far more profound impact.
“One of the things which we've always
struggled with is that, because of the Digital
India initiative, so many of the services
are now going online,” Choudhary said.
“I'm supposed to pay my taxes online,
I'm supposed to keep all my important
documents with the government online;
after demonetization, we’re expected to go
cashless completely. Almost my entire life is
going to be online.
 “And then I'm handing over the power
of that kill switch to these police guys who
have not even thought about these things
in a nuanced way. And, unfortunately,
constitutional rights and laws mean nothing
to them.”
HONG KONG ON THE BRINK
We may soon find out what a shutdown of
a highly-connected global financial center
could look like. Over in Hong Kong, the
threat of disconnection is growing. On
October 4, Hong Kong chief executive Carrie
Lam enacted the colonial-era Emergency
Regulations Ordinance, which allows the
government to “make any regulations
whatsoever” that it considers to be in the
“public interest,” if faced with “an occasion
of emergency or public danger.” This would
include communications shutdowns.
"Any such restrictions,
however slight
originally, would start
the end of the open
Internet of
Hong Kong"
“The use of the Internet for both the [2014]
Umbrella Protests and the current protests is
vital,” Nathan Law, Hong Kong politician and
activist, told DCD.
“In terms of the current protest, we use
online platforms to generate ideas, making
our protest more fluid and more influential.
Using the Internet can also make us better
at reaching the international community.
The protester can participate in the agenda…
broadcasting our message rather than letting
the media interpret everything.”
Law, founding chair of the Hong Kong
youth activist group Demosistō, said the
movement already assumes it could be under
digital surveillance: “We don’t talk about
sensitive issues through social networks or
online communication software. If we have
to do so, we will use a more secure app like
Signal.”
Protesters are also “preparing for the
possible shutdown of the Internet,” Law said.
For instance, they have apps that don’t need
the Internet to work, like FireChat, which
uses wireless mesh networking to enable
smartphones to communicate directly. But
being taken offline would still stymie the
movement, he admitted.
“It is important that we retain a free flow
of information and a channel to reach the
world,” he said. “If you take a look at the
situation in China, it is like a black box, it is
difficult for the activists to be connected and
for the outside world to understand what is
happening inside. So it is very important for
us to remain open.”
Others in Hong Kong are concerned,
including members of the business
community. In August, as rumors of
impending digital censorship spread,
the Hong Kong Internet Service
Providers Association sent out an urgent
statement: “Technically speaking, given
the complexity of the modern Internet,
including technologies like VPN, cloud and
cryptography, it is impossible to effectively
and meaningfully block any services, unless
we put the whole Internet of Hong Kong
behind [a] large scale surveillance firewall.
“Therefore, any such restrictions,
however slight originally, would start the
end of the open Internet of Hong Kong, and
would immediately and permanently deter
international businesses from positing their
businesses and investments in Hong Kong.”
The association added: “Hong Kong is
the largest core node of Asia’s optical fiber
network and hosts the biggest Internet
exchange in the region, and it is now home
to 100+ data centers operated by local and
international companies, and it transits 80
percent+ of traffic for mainland China. All
these successes rely on the openness of
Hong Kong’s network.”
A Chinese approach to the Internet would
mark a radical shift for Hong Kong, which -
at the time of publication - has a relatively
free and open network. “China is the prime
example of a preventive regime,” Rydzak said.
“Instead of reacting to protests, they
try to smother them in advance. They
are operating under the assumption that
criticism and protest born on the Internet
can spill over into the streets, so nipping it in
the bud is their priority.”
A WEB OF ITS OWN
China’s Internet is unlike anything else.
“The sophistication of the infrastructure and
the censorship system in China is much
more superior to anything that we've seen,”
Professor Christopher Leberknight, online
censorship researcher at Montclair State
University, told DCD.
“China has gotten it down to ‘we can
block specific keywords, we can block
specific pages of a website.’ They also
have a huge army of people that are just
looking at blogs, websites, and if there's
something that's a little bit ambiguous, then
the information doesn't get posted. It sits in
limbo for 24 hours.
“And then there are individuals that
Threats from the sea
With the vast majority of
intercontinental data transfer occuring
in cables under the oceans, some fear
that submarine cables could be easy
targets for malicious actors.
In August, the UK banned the
export of submarines to Russia,
citing the threat. "This additional
control is a consequence of Russia
developing certain capabilities
- including the ability to track,
access and disrupt undersea
communication cables," the
International Trade Department’s
export-control unit said.
"These activities represent a risk
to our national security and the new
control is intended to mitigate this
risk."
In 2015, a Russian Defense
Ministry-owned news channel
said that Russia can “both cut the
special communication cables on
the ocean floor and scan the signals
they carry.”
Earlier this year, Russia's AS-12
Losharik submarine caught fire.
US officials claim the vessel
was designed to tamper with
submarine cables.
actually read the content and make the
decision as to whether or not to publish it."
Circumvention tools in the country are
hard to use, because “in China, you can't use
encryption-based technologies unless you
register with the authorities. If you try to
transmit unregistered encryption-based data,
the packets just get dropped,” Leberknight
said.
Beyond the software and man-power
required to run the Great Firewall, the nation
- with the fervor of a technocratic regime -
made sure its Internet was built in its image.
"Most developed nations have a large
number of non-domestic carriers with a
presence in-country. This means that foreign
telecoms are interconnected with local
and other international carriers at physical
locations (Internet Exchanges) within
these countries," Dave Allen, Oracle's VP of
business operations and strategy, said in a
research report.
"China is different: there are no observable
foreign carriers with a presence in China’s
borders. The general trend globally is that
countries - both developed and developing -
are becoming increasingly connected. China,
on the other hand, has had no meaningful
foreign telecom presence over our many
years of historical data.
Issue 34 ∞ November 2019 15
 Cover Feature | The Great Disconnect
"Nevertheless, we know Chinese citizens
can still connect with the global public
Internet, subject to the restrictions placed
on them by the Great Firewall. China’s
connections to the rest of the global Internet
just aren’t in China. They are in Western
Europe and the United States, along with a
few other locations."
This offers a crucial advantage for a
censorial regime, Mohit Lad, CEO of network
monitoring company ThousandEyes, told
DCD: “If you can concentrate all your traffic
to a certain set of points, then you can
technically have the ability to inspect every
single packet that goes through there and be
able to apply rules and so on.
“The interesting part about China is they
built it very early in the Internet's rise. And
as a result they've built it, they've scaled it,
they've tuned it. And they are able to handle
the kind of volume that they see through
their firewall at scale.”
Other states are envious, but may struggle
to achieve the same level of control over
their network, Lad said. “If you think about
countries like Russia, it's going to be pretty
challenging, because it's at a scale where
you can't just turn it on; it's a very different
problem.”
But that doesn’t mean they're not trying.
BUILDING RUNET
“There are a lot of small ISPs [in Russia] who
have this transitional traffic flow, and these
lines are still working,” Ilona Stadnik, a cyber
security researcher at Saint Petersburg State
University, told DCD.
“[Most] Russian traffic still goes inside the
territory and just one percent goes out, but
these lines are still working,” Stadnik said.
“And if you issue an order that now just one
state network operator, Rostelecom, will be
able to move traffic abroad, it won't work -
the authority would have to go and dig out all
the lines that are going outside Russia. It will
take time, and it's not feasible.”
That may change, however, with
incoming laws that impose strict demands
on network operators that “are so high
that they will probably be deprived of their
business, and will have to sell it because the
expenses will be too high,” Stadnik said. “This
could lead to the absorption of small ISPs
and network operators by the biggest one.
So the number of independent, unknown
transborder lines will be reduced.”
Stadnik could not say whether this was an
intentional result of Russian policy or a side
effect, but the outcome is clear: “Just change
the market itself, and then it will be easier to
control.”
Russia has also - perhaps more
aggressively than any other state - sought to
shut down the Internet of other nations. The
country is linked to numerous distributed
16 DCD Magazine • datacenterdynamics.com
denial-of-service attacks on foreign territories,
although attribution can always be tricky.
BRINGING DOWN YOUR ENEMIES
“You've got a situation where outages are
technopolitical,” Martin Rudd, CTO of cyber
security and government infrastructure
company Telesoft, told DCD. ”You can use an
outage to enforce an aim, whether the goal
is espionage, sabotage or theft. You can use
that outage against either that nation-state
or against a multinational competitor or
multinational organization.”
In 2007, following the removal of a statue
of a Soviet soldier, a series of coordinated
DDoS attacks battered the tiny Baltic state of
Estonia. The attacks grew in intensity, hitting
government, banking, and media sites,
among others, and threatened to bring the
nation to a standstill.
"When the 2007 cyber
attacks happened, it was
kind of like ‘this is the
real deal’"
“Basically we closed off Estonia from
abroad, the Internet became an intranet in
Estonia. We could still operate it, except that
there were no [outside] connections - you
don't see CNN or BBC or whatever you need
to use, but you can still use the services
inside,” Cybernetica CEO Oliver Väärtnõu
told DCD.
Väärntõu’s company, best known for
developing Estonia’s ‘X-Road’ network layer
and the Internet voting system that has
allowed Estonia to become a highly advanced
digital nation, is all too aware that attacks by
nation states could happen again.
“When the 2007 cyber attacks happened,
it was kind of like ‘this is the real deal.’ So we
looked at how we can actually get over this,
what are our vulnerabilities, etc. It’s about
not only creating systems using the secure
software development process, but also how
you develop and refine the architecture of
the e-government.”
The country is actively preparing for the
worst. In 2017, Estonia announced plans
to build a data center in Luxembourg to
store crucial government and citizen data
as a backup. More data centers in different
countries were planned, Väärtnõu said, but
the process has been slow.
“Today, we are not fully in this position
where we can say that if Estonia is
completely shut down, then the government
will continue in cyberspace after being taken
over. It is a very fancy thing to say that our
government is backed up to Luxembourg, or
in the future to whatever country it is - but
we also have to bear in mind that when we
create the infrastructure inside that country
that it's fully resilient, that we have the
failovers, and that everything is being copied.”
The idea of a digital nation
unencumbered by the threat of physical
attack remains a dream. The nightmare
of an attack on sovereign soil is still a
terrifying possibility. And, despite Estonia
and Cybernetica’s efforts to improve cyber
security, there’s little one can do against
certain events. “If there is no electricity, I
think then we're going back to the Stone
Age,” Väärntõu said.
Taking out a larger nation may prove
trickier. "I'm not of the belief that one single
attack could take down America's Internet
at all," Winn Schwartau, the cyber security
researcher who in 1991 warned Congress of the
threat of an 'Electronic Pearl Harbor,' told DCD.
“You'd have to cut too damn many wires.”
Motivated by nothing more than their
business interests, companies in the US have
helped strengthen the Internet, pushing
for resiliency, backups and redundant
connections.
It’s also not clear if an adversary would
want to take America out. “It's much more
profitable to keep it going, because of social
media and the access it gives you,” Mark
Carney, pentester and security researcher for
Security Research Labs, said.
“So motivated and intelligent attackers
think ‘okay, there is now such redundancy
that we can't bring it down, however, there's
such connectivity that we can influence that
in a way where we can have an effect.’”
Russia, meanwhile, maintains that it too
could face attacks on it own network. In
April, the country passed the controversial
‘Internet isolation’ bill “providing for the
safe and sustainable functioning” of Russia’s
Internet, that by November is meant to allow
the state the ability to cut itself off from the
wider web.
“We should be afraid of [an] external kill
switch - this is how it is explained to us,”
Stadnik said. “This is a unique discourse
that the Russian authorities have, nobody in
the world is really talking about an external
shutdown. This is a story that can be really
kind of favorable for other countries to pull.”
Many are concerned that the law has
more to do with Russia controlling its own
territory than any real fear of foreign attacks.
“This August, there were documented
shutdowns of the mobile Internet during the
protest in Moscow. I think they are trying
to test how far they can go with this, and to
what extent they can execute this, and to see
how people react to it.
“The trend is obvious,” Stadnik said.
“The government really wants to keep track
of what's happening in the information
sphere, what's inside the traffic. But the
only problem is that the amount of such
data is so enormous that even if you pass
this regulation, you won't find enough
equipment to do so.”
Unable to build the tech on its own,
Russia appears to be turning to China
for help, with a number of technological
agreements signed over the past three years.
This October, the two nations signed a joint
treaty aimed at tackling “illegal Internet
content,” which lacked specifics, but might
include China sharing some of its Great
Firewall hardware and software.
As for Russia’s Internet isolation law, it’s
light on technical detail, partly for security
reasons, and partly because "our legislators
don't know in detail how everything works,
because for them the Internet is more like a
telephone," Stadnik said.
"You have people in the legislative bodies
that don't have enough technical expertise
to make such laws. And they are sometimes
very wrong, sometimes very illogical. But
it doesn't prevent legislators from moving
them forward. That's how it works in Russia."
Key aspects of the law remain shrouded
in mystery, but the country is barreling
ahead nonetheless. In September, Alexander
Zharov, head of the federal communications
regulator Roskomnadzor, said that
“equipment is being installed on the networks
of major telecom operators” to allow RuNet to
be separated from the greater Internet.
"Countries are always threatening to cut
themselves off," Professor Milton Mueller,
author of Will the Internet Fragment? and one
of the founders of the Internet Governance
Project, told DCD.
"But when you look more closely at those
proposals, they really are not about cutting
themselves off from the Internet. Other than
in emergency situations, that's really pretty
stupid and self-destructive.”
Instead of a complete separation of a
nation’s Internet, “the number one fear
for me would be the Balkanization of the
Internet,” Schwartau said.
Mueller agreed: “Countries are trying to
align national boundaries with their digital
economies as much as they can. That's kind
of pushing the stone uphill, because it's
just not the way the Internet is constructed.
What's interesting now is that it's gone
beyond the network layer into equipment
and software, where we're discovering
how damaging that decoupling and that
disintegration will be.”
GREAT POWERS DIVERGE
He fears what the deteriorating relationship
between China and the US will do to the
Internet. “There was hope that China would
gradually become aware of the degree to
which their power and their wealth relies on
openness, interconnection, and trade with
the rest of the world,” Mueller said.
“And they couldn't play the game both
ways: they can't shield themselves from
foreigners, and at the same time expect
foreigners to be open to them.”
But, while he conceded “there was some
need for a confrontation or systemic change
in the way [China does] things,” Mueller
believes that “President Trump went about it
in a very, very terrible way.”
Tariffs, sanctions on globally-focused
companies, and restrictions on trade in
technology will only make the divide worse,
he argued. “What the US is doing is using our
dominance of chips to shut the Chinese out
of high technology, because they have this
nostalgic sense that the technological and
economic dominance that the US enjoyed
after the decline of the Soviet Union is going
to last forever.
“And they think that they can stunt
China's growth and keep it subordinate
indefinitely by cutting them off from US
technology. All that's going to do is make
China develop its own chips and its own
advanced services in a way that is not
integrated with the West. I would much
rather have China buying US chips and
having technology flowing both ways, than
to have these fragmented blocks around
these technological superpowers.”
The idea of a sovereign digital state in
charge of all the bytes within its borders is
growing in popularity, with data residency
laws spreading across the globe. "What we're
seeing is that all those data centers around
the world are becoming sovereign assets,”
Telesoft’s Rudd said.
“The boundaries in cyber are so blurred,
stuff like banking, finance and the stock
market - are they part of the sovereign state,
while being commercially owned?”
For the normal day-to-day operation of
the Internet, which country a data center
resides in is mostly irrelevant. Latency, grid
infrastructure, and cooling costs are all
important, but an average user in the UK will
not notice a difference between a facility in
Denmark or Belgium, for example.
However, when things become strained,
the location of data can be crucial. Imagine
a scenario where hackers attack a “national
telecoms operator in a particular country that
is using a cloud provider,” Rudd said. “Do you
really want the data [on the attack] which is
being brought up from that network to be
classified by a third party in another country?
“That's the same data which is used to
optimally train the AI algorithms in defense
for anomaly detection or machine learning.
It tells you who is attacking what and how.”
Conversely, if nations require data to be
stored in a country of origin, information on
how an attack happened may be withheld
from a multinational business. Suddenly, it is
unable to fully study the intrusion attempts
it has suffered in certain regions, and is less
secure as a result.
We don’t know how this will play out; it is
not clear how much further the Internet will
Balkanize. In a world where authoritarianism
is on the rise and democracies are
weakening, the future appears bleak.
“I think the norms and the levels of
cooperation that are happening across
North America and Europe are going to
move towards a more integrated system,”
Mueller said. “And then we're seeing the rest
of the world, the more authoritarian world,
detaching from that and possibly creating a
bipolar information technology world.”
The West, to be clear, is not without
sin. It exports the tools of surveillance
and censorship and is experimenting
with small shutdowns of its own, on San
Francisco’s BART network and the London
Underground. It also surveils its people, and
blocks certain websites.
“It is very difficult for the Western world to
complain about things at this point, because
we're putting so much pressure on the social
media companies to censor things,” Mueller
said. “So what basis do we have to say that
China should have this freewheeling open
social media environment?
“And what happens immediately is that
they pick up on that and say, ‘Oh look, you
just did this. Why can't we do that?’”
It may be, Mueller said, that the only way
to overcome the nationalistic issues of the
Internet is to overcome the nationalistic
issues of the world. “It's a big ask,” he added.
“I just don't think that's gonna happen.”
Issue 34 ∞ November 2019 17
 XXXXXXXX

For more than a decade, the DCD Awards platform

has showcased stories of innovation and cutting-
edge design globally, with our APAC Awards
adding a regional focus over the past six years. Our
entrants have shared examples of best practice and
innovation, as the region gears up for a major new
era of digitalization.

1 The Edge Data Center

Project of the Year
Winner: GPX India Pvt Ltd
Project: GPX Mumbai Data Center
2 The Multi Tenant Data
Center Design Award
Winner: AirTrunk
Project: AirTrunk SYD1
3 The Enterprise Data
Center Design Award
Winner: DoIT Government of
Rajasthan in conjunction with
Sterling and Wilson
Project: State Data Center,

4 Hybrid IT Project of the

Year 5 The Energy Smart
Award
Jaipur

Winner: China Life Insurance

Company
Project: China Life Hybrid IT,
Winner: Huawei
Technologies Co. Ltd
Project: iCooling at the
6 Operations Team
of the Year
Winner: Datacom
Shanghai Langfang Cloud Data Center
Project: New Zealand Rebuild
and Expansion Project Teams

7 Data Center Manager of

the Year
Winner: Rathish Mani, Digital Realty
8 Data Center Construction
Team of the Year
Winner: Bangladesh National
Data Center in conjunction with
ZTE Corporation
Project: The National Data
Center Construction Team
9 Business Leader
of the Year
Winner: Jeremy Deutsch,
President Asia Pacific at
Equinix

10 Outstanding Industry Contribution

Winner: Jacqueline Chan, Director at DSCO Group

18 DCD Magazine • datacenterdynamics.com

 >Awards | 2019

Public Vote: Best Mainstream

Press Coverage of the Data
Center Industry NEW!
Everyone deserves to understand a subject as important as data centers. When the press gets it
right, governments will regulate the sector better, and make better economic decisions about it,
while the public will engage constructively with the facilities they live alongside.
This Year's public vote category will recognize journalists and publications which helped the data
center community by delivering useful information about the sector to a broader audience.

www.dcdawards.global/best-mainstream-press
Public
voting
closes
Sponsored by: Nov 2
See we
5!
b link
above

Charity Partner
 CEO | In focus
Vive la différence!
In a world dominated by US and Chinese providers, does the world
need a European cloud player to preserve privacy? OVHcloud’s CEO,
Michel Paulin, talks to Peter Judge
O
VHcloud is the only global
cloud provider based in
Europe, and that simple fact
is guiding its next steps,
says the CEO, Michel Paulin.
The company backs open
standards and choice, and it champions
European privacy measures against the state-
based intervention of the US and China.
He believes the world needs a European
alternative to interventionist states and the
AWS monopoly.
The company is not well known outside
France, but turned 20 this year. The OVH
hosting business was founded in 1999 by
20 DCD Magazine • datacenterdynamics.com
Polish-born entrepreneur Octave Klaba. It
is still often seen as a web hoster, but has
a thriving public cloud, and hosted private
cloud business, served from 30 data centers
on four continents (in the US, Canada,
Singapore, Australia, and five European
countries).
In 2018, Klaba appointed Paulin as CEO,
and at this year’s annual OVH Summit in
Paris, the company announced a new name:
OVHcloud. “It's a way to demonstrate to a
market where we are not well known, that
we are a cloud provider,” he told us at the
summit. “And we are not to be perceived as a
small web hosting company any more.”
For more
high-level
insights, check
out our C-Level
Summit on
31 March
Peter Judge
Global Editor
The new name and new CEO may
underline the company’s message, but
there’s no major change in direction. Paulin
comes to the cloud from a different sector:
in the telecoms world, he managed the IPO
of Neuf Cegetel and its merger with the telco
SFR, later becoming SFR’s CEO. But he’s not
there to change things.
Paulin is adapting to OVHcloud’s world,
rather than the other way round, and says
the cloud is different to telecoms: “There’s
much less regulation. And it is very, very new
technology.”
Telecoms is slower, he says: “Even 5G uses
IP technology, and fiber is nearly 50 years
old. We're not talking about pure software like
Kubernetes, and Hadoop servers. Telecoms
took 15 years to go from GSM to 2G. In the
cloud the rhythm is days, or weeks.”
Klaba remains as chairman of the board,
and the two have daily discussions: “Octave
has a vision, and we are trying to execute as
fast as we can.” Klaba’s role as a visionary - a
“geek in a good way” - is clearly important.
The Summit is designed to demonstrate
OVHcloud’s status and continuity with OVH.
Three thousand delegates (and maybe a
few thousand more over the web) enjoy a
keynote bristling with senior speakers. The
French Digital Economy Minister Cédric
O says business needs a European cloud,
vice-admiral Arnaud Coustilliere from the
Ministry of Defence says everyone needs
private data. They are joined by senior
executives from Capgemini and Deloitte
- and finally, in an OVH tradition, Klaba
performs a musical number with two friends.
At the Summit, Paulin tells us his mission
is to broaden the company’s appeal, and raise
the profile of its cloud services, countering
the widespread perception that it is primarily
a hoster, and increasing awareness of what
it’s actually doing.
“We are sometimes a ‘best kept secret,’”
Paulin says. Not everyone knows that 70
percent of OVHcloud’s business comes
from the cloud, he says. “Of the world’s top
ten cloud providers, all the others are US or
Chinese, and there is one Japanese operator.
We are the European cloud provider with a
worldwide presence.”
And that cloud business is diverse.
OVHcloud aims to offer an alternative to
AWS and the other giants, and Paulin says
it’s always had a more mature vision of the
reality of cloud. “We do PaaS (platform-as
-a-service) and IaaS (infrastructure-as-a-
service) offerings somewhat similar to the
AWS behemoth, but also include bare metal,
containers, and private cloud.”
Other people have overestimated the
power of public cloud: “Five years ago, people
expected public cloud would solve all the
problems for everybody, that it would be the
Holy Grail, and everything would be perfect.
Octave always said no - it would work, but it
would have some problems. The solution is
hybrid cloud.”
Early on, OVH launched a product
for hosted private cloud based on SDDC
(software-defined data centers), to use
alongside public cloud: “You can put
the appropriate workload with the right
architecture, to be sure that you have the
same SLAs, the same resilience and the same
latency - as if it was on your own premises.”
Like those hybrid clouds, OVHcloud’s
business itself isn’t monolithic, like the
offerings of big players like AWS and Azure.
It needs partners: “We do need an ecosystem,
and we don’t want to be monolithic. We rely
on a lot of partners when our solutions go to
the market. Because when you are three you
can go faster than alone, and when you are
100 you can even accelerate.”
There’s one drawback to this. If the
customer only sees the partner’s brand,
doesn’t OVHcloud remain a “best kept
secret?”
“This is something we are aware of,” says
Paulin. “Some of our customers don't want
to say our name,” he admits. Some PaaS and
SaaS customers and hosters want to maintain
the illusion that they have their own facilities.
For instance, giant hoster GoDaddy is
understood to be an OVH customer, but as
both offer hosting, they are also competitors.
In some countries (like the UK) the
majority of partners don’t promote the
OVHcloud name, says Paulin: “So we are
under the radar.” But in future he believes
that OVHcloud’s unique approach will
become something that people will actively
want to share.
"A French politician
says there's a war
between GDPR and the
CLOUD Act"
The company’s distinctive hardware is
definitely winning customers, he says: “We
don't have air-conditioning. We do water
cooling. It is much more eco-friendly - for
the same level of servers, we use 10-50
percent less energy than a classical data
center. Our PUE [power usage effectiveness]
is 1.09; the latest generation of air cooling is
at 1.2 and many facilities are at 1.6.”
Water cooling makes things simpler in
the long run, he says: “It's complicated to
monitor, it's complicated to design. But
because you have fewer elements, you use
fewer components. For example, there is no
air cooler. It’s less expensive in capex, less
expensive in opex, and less expensive to
manage.”
It also saves on building expenses. The
liquid cooled racks don’t need traditional
contained aisles and raised floors, so they
can be installed rack by rack in ordinary
warehouse space: “We don't need to have
white rooms, filtering and freon. We don't
have all those ugly words.”
OVH hosts cloud services based on
the OpenStack open source platform,
and VMware’s vCloud - both of which
were heavily promoted in the early
2000s, as public cloud alternatives to the
AWS juggernaut. As those public cloud
contenders fell by the wayside, OVHcloud
remains as an exception.
The company has doubled down on
that role, adding more unusual services,
including a cloud service which offers
Kubernetes containers implemented on
bare metal servers, instead of the virtual
machines (VMs) favored by other providers.
This gives customers more performance,
but is more of a management headache for
OVHcloud, CTO Alain Fiocco explained at
the same OVHcloud event.
In 2017, OVH picked up VMware’s
failing public cloud effort vCloud Air, and
consolidated the underlying hardware
into its own data centers. This gave it a US
footprint, and Octave Klaba spent the first
half of 2019 living in Dallas to understand
how to integrate the US-heritage business
into a European organization.
Will OVHcloud have to modify its
European attitudes as it becomes more
international? No, says Paulin. If anything,
the European DNA is even more important:
“We do believe that our roots are in Europe
and openness - transparency, GDPR and
open source are European values.
“When we go to Asia and Australia, do
we have to change these key DNA values?
I don't think so,” he says. If anything,
adherence to the European GDPR gives
OVHcloud an edge over US-based firms that
have signed the CLOUD Act.
“The CLOUD Act is a legal tool which
gives access to your data if you're on
Amazon, Google or Microsoft. So companies
are a little bit afraid that without notice,
someone can ask for access to their data.”
OVHcloud actually offers compliance
with the CLOUD Act as an option, since
some firms may require it. Its US subsidiary
is based in the States, so it has been
established as a strictly separate entity,
bound by the Act.
He’s surprised that the UK has effectively
signed up to the CLOUD Act with a new
treaty, and amused at the US reaction to
the bilateral agreement. “In the US, some
organizations said 'Oh, this is a shame. We
gave the right to the UK to have access to
some data in the US'." This is "bizarre," he
says, since that's exactly what the US has
demanded from the rest of the world.
Paulin thinks there’s more opportunity in
holding out an alternative: “The government
in France and the European Commission
asked us to think about what we need to
do to protect European data. The French
government doesn't want other states or
jurisdictions or companies to have access
without any limitation.”
“A politician in France said there's a war
between GDPR and the CLOUD Act,” Paulin
says. “I don't know if it's a war. But I think it
is a debate.”
Issue 34 ∞ November 2019 21
 Outages | Andy Lawrence
Creeping criticality
Why do some industries and organizations suffer more serious,
high profile outages than others? Andy Lawrence explains
I
n June 2019, the US General
Accounting Office (GAO) issued
a report on the IT resilience of US
airlines. The GAO wanted to better
understand if the all-too-frequent
IT outages and resultant chaos
passengers face have any common causes
and, if so, how they could be addressed. Since
then, the UK-owned carrier British Airways
suffered its second big outage in two years,
once again stranding tens of thousands of
passengers and facing heavy costs.
The GAO report didn’t uncover much
new: in some cases, the airlines needed
better testing, a little more redundancy here
and there, and some improved processes.
But despite suspicions of under-investment,
there was nothing systemic wrong. The
outages had varied causes. They were often
avoidable in hindsight, but not predictable.
But there is still an undeniable pattern.
Our own analysis of three years of
public, media-reported outages shows
that two industries, airlines and retail
financial services, do appear to suffer
from significantly more, highly disruptive
(category 4 and 5), high profile outages than
other industries.
To be clear: these businesses do not
necessarily have more outages, but rather
they suffer a higher number of highly
disruptive outages, and as a result, get more
negative publicity when there is a problem.
Cloud providers are not far behind.
Why is this? The reasons may vary, but
these businesses very often offer services
on which large numbers of people depend,
for which almost any interruption causes
immediate losses and negative publicity, and
in which it may not be easy to get back to the
status quo.
Another trait that seems to set these
businesses apart is that their almost complete
dependence on IT is relatively recent (or
they may be a new IT service or industry).
They may not yet have invested to the same
levels as, for example, an investment bank,
stock exchange or a power utility. In these
last examples, the mission-critical nature of
the business has long been clear, they are
probably regulated, and so have investments
and processes fully in place.
22 DCD Magazine • datacenterdynamics.com
Organizations have long conducted
business impact analyses, and there are
various methodologies and tools available
to help carry these out. Uptime Institute has
been researching this area, particularly to
see how organizations might specifically
address the business impact of failures in
digital infrastructure. One simple approach
is to create a “vulnerability” rating for each
application/service, with scores attributed
across a number of factors. Some of our
thinking - and this is not comprehensive - is
outlined below:
"Outages were often
avoidable in hindsight,
but not predictable"
Profile. Certain industries are consumer
facing, large scale or have a very public
brand. A high score in this area means even
small failures - Facebook’s outages are a good
example - will have a big public impact.
Failure sensitivity. Sensitive industries
are those for which an outage has immediate
and high impact. If an investment bank can’t
trade, planes can’t take off or clients can’t
access their money, the sensitivity is high.
Recoverability. Organizations that can
take a lengthy time to restore normal service
will suffer more seriously from IT failures.
The costs of an outage may be multiplied
many times over if the recovery time is
lengthy. For example, airlines may find it
takes days to get all planes and crews in the
right location to restore normal operations.
Regulatory/compliance. Failures in the
certain industries either must be reported
or will attract attention from regulators.
Emergency services (e.g., 911, 999, 112), power
companies and hospitals are good examples
… and this list is growing.
Platform dependence. Organizations
whose customers include service
providers - such as software-as-a-service;
infrastructure-as-a-service; and colocation,
Andy Lawrence
Uptime Institute
hosting and cloud-based service providers -
will not only breach service level agreements
but also lose paying clients. (There are many
examples of this.)
One of the challenges of carrying out
assessments is that the impact of any
particular service or application failing is
changing, in two ways.
First, in most cases, it is increasing,
along with the IT dependence of all
businesses and consumers. And second, it is
becoming more complicated and harder to
determine accurately, largely because of the
interdependence of many different systems
and applications, intertwined to support
different processes and services. There may
even be a logarithmic hockey stick curve,
with the impact of failures growing rapidly
as more systems, people and businesses are
involved.
Looked at like this, it is clear that certain
organizations have become more vulnerable
to high impact outages than they were a
year or two previously, because while the
immediate impact on sales/revenue may
not have the changed, the scale, profile
or recoverability may have. It may be that
airlines, which only two years ago could
board passengers manually, can no longer
do so without IT; similarly, retail banking
customers used to carry sufficient cash or
checks to get themselves a meal and get
home. Not anymore. These organizations
now have a very tricky problem: How do
they upgrade their infrastructure, and their
processes, to a level of mission criticality for
which they were not designed?
All this raises a further tricky question
that Uptime Institute is researching: Which
industries, businesses or services have
become (or will become) critical to the
national infrastructure - even if a few years
ago they certainly were not (or they are
not currently)? And could regulation help?
We are seeking partners to help with this
research. Organizations are not the only
ones struggling with these questions -
governments are as well.
More information on this topic is available to
members of the Uptime Institute Network
bit.ly/UptimeInstituteNetwork
 > Telco | Supplement

Sponsored by

™
INSIDE

The 5G promise The Telco Edge The big sell-off

> A new generation of > Fixed and mobile network > Telcos are selling data centers
connectivity means a new world operators shift virtual workloads by the dozen to pay mounting
of opportunities to the Edge piles of debts

Contents
26 A competitive Edge for telcos:
Shifting virtual workloads
to the Edge has multiple
advantages
28 5
 G, Edge and the
revolution: A lot of 5G
predictions sound like
hype. Here is the reality
30 A
 dvertorial: Getting up to
speed on 5G strategy and
micro data centers
32 F
 lood the world with fiber:
If we can lower the cost to
deploy fiber, we could
connect the planet
34 R
 outers in the sky:
Terrestrial fiber has ruled the
world. Now satellites
are challenging it
36 T
 he telco sell-off:
Telcos thought they'd
be great at colo
services. Now they
are moving out
28
32
Hailing on all frequencies
T
elecoms is an equal
partner with data
centers in the digital
infrastructure which is
vital to our lives. Either
without the other
would much less useful.
Telecoms has been seen as
a slow-moving sector compared
with the digital world and the cloud
(see comments by OVHcloud CEO
Michel Paulin, on p20, for instance).
That is now changing. The
vast and rapid shifts which are
continuously remaking the data
center sector, are shaking up the
telecoms sector, transforming the
ways in which operators interact
with customers, transmit data, and
choose their battles.
This supplement takes a look at
the biggest of those changes.
A competitive Edge
Data center resources are being
directed to the edge of the network,
to support emerging applications
like the IoT (p26).
This move depends on having
access to fast networks to transmit
the data those applications need.
So Edge is as much an issue for
telecoms as it is for facilities.
There's an unprecedented
crossover in the moves to
deliver the hardware that these
applications need. Standards for
telecoms facilities are emerging
from the data center world
5G brings a revolution
The fifth generation of mobile
services (5G) is more than just a
new and faster kind of network. It's
data-driven, and delivered through
small cells.
36
5G services will be
fundamentally data driven, and
Telco Supplement
Sponsored by
will drive a new dependence on
resources embedded into the
network.
Using satellites
Fiber has been seen as the best way
to distribute telecoms services.
After the losses, limits and power
demands of copper, glass fiber
sparked a revolution when it began
to displace metal wires in long
distance communications.
But now satellites are seeing
an unprecedented comeback. It
turns out that low earth orbit (LEO)
constellations can take a lot of data.
And even though the route up to
space and back down seems like
a long way to go, there are fewer
hops, and the light travels faster in
the vacuum of space than it does in
glass (p34).
So expect to be offered satellite-
based offerings for applications like
back up and recovery
Funding fiber
Despite that, fiber roll-outs
continue apace. As well as the
obvious international submarine
cable connections, projects are
aiming to find new ways to fund
fiber for areas where cost is a
barrier (p32).
Seeing sense
Finally, this rush of exciting
telecoms activity shouldn't make
operators feel invulnerable. They
are still recovering from a rash of
over-investment in data centers in
the early years of this decade (p36).
As telcos shift their data center
assets to the specialists who can
handle them, we are seeing an
object lesson on focus.
Telcos need to think on their
feet, and move smartly.
Telco Supplement 25
A competitive
Edge for Telcos
Shifting virtual workloads into the Edge gives both fixed and mobile
network operators multiple advantages. Martin Courtney reports
A
s a new approach
to architecting
telecommunications networks,
there is no doubt that Edge
computing has significant
potential to change the way
that carriers and service providers deliver
a range of services to their business
and consumer customers. But with the
technology at such an early stage of its
development, specific use cases are still
under development - particularly as telcos
work out the best way to exploit software-
defined networking (SDN) and network
functions virtualization (NFV) to drive
down their own infrastructure costs and
streamline provisioning, configuration and
management processes.
There is still debate over what the
network Edge actually is, and no strict
definition to clear up the confusion. Most
see it as smaller data center hosting/
processing facilities located closer to the
end user, but others feel it could incorporate
local workloads running on customer
premise equipment and other points of
presence with local (LANs) rather than wide
area networks (WANs). Instead of a location,
the Edge defines only a workload hosted at
some indistinct node within the provider or
customer infrastructure.
According to network and
telecommunications equipment
manufacturer Cisco, the point of the
Edge is threefold: to deliver lower latency
to the end device to benefit application
performance and improve the quality of
the experience; implement Edge offloading
for greater network efficiency; and perform
computations that augment the capabilities
of devices and reduce network transport
costs. To that end, much of the ongoing
innovation has so far focused on the
enablement of fifth generation (5G) cellular
networks.
Indeed most mobile operators agree that
cost efficient 5G service delivery is simply
unfeasible without the deployment of some
form of Edge data hosting and processing
26 DCD Supplement • datacenterdynamics.com
capability to override the need to transmit
and crunch large volumes of information via
centralized data centers and the core.
But Cisco’s imperatives apply equally
to applications and services delivered over
wired broadband connections as they do
to 5G links, as much to Edge infrastructure
workloads in cable broadband and gigabit-
capable passive optical networks (GPON)
access as to the 5G radio access network
(RAN). As such, fixed line carriers and
service providers too are looking at where
Edge computing solutions can help them
deliver wired broadband connectivity -
and the range of IP-based voice and data
services which that supports - to customers
previously accessed via local loop telephone
exchanges.
"Edge computing will
process data to facilitate
services as close to the
user as possible"
US telco AT&T, long at the vanguard of
SDN/NFV adoption, is currently working
to convert some of its estimated 4,700
telephone exchanges into mini data centers.
The fixed and mobile network giant is close
to achieving its goal of virtualizing 75 percent
of its infrastructure by 2020, having already
deployed SDN enabled broadband access
(SEBA) to deliver superfast fiber broadband
services to consumers and businesses in
US cities such as Irving and Atlanta. SEBA is
a set of open networking components that
virtualize the software to run optical network
terminals (ONTs) and optical network units
(ONUs) on fiber networks, though it can be
extended to other types of network including
fixed wireless and Gfast that use copper
cabling.
In the UK, BT is extending core network
Martin
Courtney
Contributor
functions currently hosted within five to
ten of its exchanges to around 100 metro
locations. BT’s own Network Cloud will
evolve to reduce data and application
latency, again initially for 5G applications
and services. But, once the infrastructure
is in place, it can be used for a variety of
different functions, including broadband, IP
telephony, and unified communications as a
service (UCaaS) provision to customers.
BT currently has around 1,200 local
exchanges in the UK which serve as a first
point of aggregation, more of which could
be migrated to Edge facilities to meet the
needs of different cloud hosted services in
the future.
5G cell towers are another proposed
location for Edge compute resources in the
base of 5G cell towers which can also be
used to accommodate fixed line operators’
equipment. Defined by the Open Network
Foundation (ONF), the Central Office Re-
architected as a Datacenter (CORD) initiative
combines NFV, SDN and commodity clouds
to bring cost efficiency and cloud agility to
the Telco Central Office (UK parlance the
local telephone exchange), allowing them
to dynamically configure new services for
residential, enterprise and mobile customers
in real time.
As voice as well as data traffic becomes
IP enabled, routing and switching functions
can be virtualized, making them easier
to provision, configure and manage
remotely. It is envisaged that the reference
implementation of CORD will be built from
commodity servers and white-box switches
defined by the Open Compute Project (OCP)
which are cheaper to buy than proprietary
telecommunications hardware for example,
alongside disaggregated access technologies
(vOLT, vBBU, vDOCSIS), and open source
software (OpenStack, ONOS, XOS).
Elsewhere, the European
Telecommunications Standards Institute
(ETSI) multi-access Edge computing (MEC)
specification was designed to promote the
 The Telco Edge

convergence of mobile base stations and

IT and telecommunications networking,
ostensibly to support anticipated new
business cases around video analytics,
location services, IoT, augmented reality,
data caching and optimized local content
distribution (what used to be known as a
content delivery network - CDN).
Those use cases were defined specifically
with 5G in mind, but as software overlays
devolved from the underlying network, there
is no reason why they cannot be applied
equally to wired broadband connections too
(multi-access is included in the acronym for
a reason). Similarly, Edge routers, designed
to process data collected from thousands
of different devices and end users, already
provide various interfaces to both wired and
radio-based transmission technologies and
communication standards - everything from
5G and WiFi to Bluetooth and Ethernet.

Edge can compete with cloud, after a

fashion. Telcos like AT&T and BT need to be
able to deliver fast, reliable hosted voice and
data services. They are a crucial element of
commercial cloud strategies, but the cloud is
delivered from centralized facilities, a sector
where the telcos have failed.
IT giants like Amazon Web Services,
Microsoft, Google, IBM and others have won
in the enterprise space by investing heavily
in building their own hyperscale facilities.
After finally admitting defeat, AT&T sold
off its core data center assets to Brookfield
Infrastructure for $1.1bn earlier this year,
following similar divestitures by other
telcos (see p36).

Having a distributed compute

infrastructure at their disposal gives telcos
something the cloud service providers do
not, and would find very difficult to obtain
for themselves: dedicated Edge hosting
and processing facilities closer to the
customer which are better able to support
a range of latency sensitive applications for
business customers. Those could include
everything from infrastructure- (IaaS),
platform- (PaaS), network- (NaaS) and unified
communication-as-a-service (UCaaS) to
industrial IoT (IIoT) and high definition
video capture (e.g. CCTV surveillance
and consumer retail applications), the
provision of which (telcos hope) could be
supplemented by value added systems
integration and managed services contracts.
And building out their Edge facilities
puts telcos in a prime position to make
themselves indispensable to bigger cloud
providers when it comes to delivering more
latency sensitive services and applications
to their own customers - a potential
market carve up that plays to both sides’
strengths and reach.

Telco Supplement 27
5G, the Edge and the
service revolution
A lot of 5G predictions sound like hype.
Vlad-Gabriel Anghel explains the reality
I
n mainstream media during 2019, the
term 5G has been increasingly seen
and touted as the future for mobile
communications and data processing.
But infrastructure industry giants have
been hard at work for quite some time
getting ready to tackle the challenges that
come with the vast amounts of possibilities
that 5G will allow.
It is tempting to see 5G as an incremental
step up from 4G/LTE, but 5G is exponentially
better. It is capable of reaching speeds of up
to 20Gbps and supporting up to a million
devices per square kilometer (that’s a lot of
IoT devices) while providing an alleged 1ms
latency. 5G is ultimately the true foundation
for the Internet of Things.
Since the emergence of IoT devices,
network limitations have placed numerous
boundaries in terms of real-life use cases,
while fields like HPC applications have seen
28 DCD Supplement • datacenterdynamics.com
limitations on data handling requirements.
A ridesharing app can only reach out
to an AI prediction algorithm within a data
center so many times per minute, and the
same applies for other types of apps that
rely on data processing in the cloud. This is
true regardless of where the data is obtained
from: Ultimately, network constraints will not
allow for a fully seamless and instantaneous
end user experience if the processing is
centralized in the cloud.
The digital infrastructure industry has
proposed a solution to this in the form of
Edge computing - a way to make distributed
systems more efficient by taking out parts
from a centralized core and making them
available closer to the data source or the
Edge.
In simpler terms, this means storage,
data services and computing power being
redistributed accordingly based on their
Vlad-Gabriel
Anghel
Contributor
service or function. The ones that benefit
from lower latency are moved closer to the
Edge, while the rest remain at the core.
Edge computing can reduce latency by
placing critical resources close to the end
users and increases resiliency as it creates
alternate data transmission routes. It does,
however, fragment the system and that can
pose a risk in terms of both physical and
logical security, and because it relies on
additional hardware it requires a significant
upfront investment. In practice, the current
capabilities of Edge computing are far from
being able to support the innovative use
cases envisioned in one form or another for
several decades. It all comes down to network
latency and availability.
5G can go a long way in removing these
constraints, because it effectively increases
the capability of the network edge. The
demand for data storage and processing

power there will tremendously increase.
If 5G delivers on that promise of 1ms
latency and one million devices per square
kilometer, it will reshape a lot of industries,
right down to their best practices and design
standards.
For example, the majority of distributed
system architects have been limited in their
design choices by bandwidth and latency
considerations. If 5G brings these barriers
down then, instead of monitoring 30 sensors
in real time, systems might manage 1,000, if
this would bring a competitive or strategic
advantage.
The same goes for mobile apps. Instead
of querying a cloud endpoint every minute,
why not every second? As these design
choices evolve, it will have a massive impact
on the digital infrastructure supporting these
systems, and businesses will need to embrace
HPC technologies and design strategies.
Essentially, 5G will rely heavily on high-
performance computing elsewhere.
Current mobile networks are capable
to a certain extent of providing services
for technologies like autonomous cars,
drones and weather forecasting, but these
applications will be truly unlocked by the use
of 5G.
What gives 5G network their tremendous
speeds and bandwidth is their technology.
They operate on what is known as millimeter
waves - radio signals with a frequency
between 30GHz and 300GHz (4G operates
between 1GHz and 5GHz). These have less
range than shorter wavelengths, so the area
previously covered by one 4G transmission
tower must now be covered by a multitude
of smaller, inexpensive 5G antennas fixed to
buildings and streetlights.
Telco Edge data centers are the first to see
this change and keeping up will be tricky.
Just revisiting the field of autonomous cars -
with their numbers on the rise, telemetry data
will be gathered by multiple 5G antennas on a
continuous basis. To analyze this in real time,
and keep all the cars in lane and on the road,
will require high-performance computing
through storage and AI predictive services.
This raises considerations about deploying
the proper HPC equipment in an efficient
and sustainable manner. With 4G, devices
connect on a one to one basis: the cellphone
connects to the telecoms tower; the tower
connects to another tower and so on. 5G
will allow devices to connect to multiple
antennas and this presents the possibility of
the utopian scenario of 100 percent reliability.
However, operators will need to embrace HPC
technologies like distributed file systems,
in-memory data grids etc. and implement
proper design methodologies as traffic scales
exponentially to millions of writes per second.
The possibilities and requirements of 5G
will reshape how data centers are built and
operated and current service providers will
need an overhaul on their infrastructure
in order to keep up. The backend services
supporting 5G will need to be much more
Current mobile communication
frequency (long wavelength)
Wavelength
Extremely high frequency
mmWave (short wavelength)
scalable than the previous 4G equipment,
however these services will most likely be
born out of already existing cloud native tools
and technologies which are the center point
of building scalable cloud services.
Furthermore, storage and computing
power will shift towards the Edge, as close
to the end users as possible. As mentioned
before, these will need to be designed and
deployed through an HPC methodology.
With data increasing vastly, the need for
data analytics and management will also
increase. This will happen again through HPC
technologies like object stores, distributed
databases and file systems. Operators will
require these tools and technologies in order
to streamline the deployment, management
and scalability of larger data volumes. As the
data moves from the secure core cloud data
5G is coming
center to the network Edge, new security
issues arise. The more fragmented a system
is, the harder it becomes to provide proper
security. This need is further underlined
by the critical nature of the applications
which are likely to use 5G services, such
as connected traffic control systems,
autonomous cars, drones and the like.
The arrival of 5G has propelled a
diversification exercise throughout the
data center industry, and the landscape is
changing. Businesses like Vapor IO and
EdgeConneX are attempting to create a new
ecosystem of Edge modular data centers
and have predicted tremendous growth for
this sector due to 5G. Meanwhile, already
established players within the data center
sector could need to shift their focus towards
deploying micro data centers, while making
additional investments in already existing
data centers and colocation facilities to keep
up with the upcoming demand.
This should not be seen as blocker in the
expansion of these businesses but rather a
necessary step in fully releasing the potential
of mobile technology and communications.
Ultimately, 5G is evolving to become what
is known as a general-purpose technology
(GPT) - a type of technology that has the
ability to drive fundamental change across
the entire global economy. Previous examples
of GPTs have been the printing press, the
automobile and the steam engine. As data
center owners and operators come to grips
with the needs and challenges of 5G and
adjust their infrastructure and facilities
accordingly, 5G end user numbers will soar,
making it the latest and most impactful GPT
to date.
With the number of IoT devices connected
to the Internet expected to reach the order
of billions in the near future, it is through
5G that all of these devices will be able to
interconnect and exchange data, more
quickly and more reliably than before.
It does, however, require data center
owners to first find huge investments
while managing stakeholders’ expectations
on short term ROI. Deployment of these
methodologies will need precise and careful
planning and implementation.
The industry as a whole is not there yet,
still a lot needs to be done in order for 5G to
become the network and service revolution it
aspires to be, but it seems, at least for now, it
is on the right path.
The future is around the corner!
Telco Supplement 29
Advertorial: Schneider Electric
Getting up to Speed on 5G
Strategy and Micro
Data Centers
Greg Jones, Schneider Electric's VP of Strategy and Offer Management for the Cloud &
Service Provider Segment talks to Steven Carlini, VP of Innovation and Data Center
E
verywhere I go, it seems that
people are talking about 5G.
I wanted to find out more
about the specific role of
micro data centers in 5G
so I sat down with Steven
Carlini, our Vice President of Innovation
and Data Centers. Steven is responsible
for developing integrated solutions and
communicating the value proposition for
Schneider Electric’s data center segment. I
knew he’d have a wealth of timely insights
on the topic and he didn’t disappoint. Here
is part of our conversation.
What is your definition of micro data
centers? I would classify micro data centers
as two or less IT racks, where a massive
amount of computing power or storage can
be managed. Today, we have micro data
centers as small as 6U that can hang on a
wall or even be put in a ceiling! I see micro
data centers as a critical extension of cloud
data center architectures to reduce latency
and add redundancy in a hybrid cloud
environment. These micro data centers
are a key building block. However, because
they are spread out all over the place,
they do present challenges in the form of
troubleshooting, maintenance, and repair.
Also, energy usage becomes a critical
operating expense at scale. For example,
let’s say you have 2,000 sites and 10 KW per
site – that’s 20MW of power! That equates
to roughly $20 million in electric bills a year
(operating at average efficiency). This is
why the efficiency of micro data centers is
top of mind for a lot of companies.
Is Schneider Electric investing in micro
data centers and/or infrastructure
changes? For Schneider Electric’s Secure
Power Division, edge data centers
including micro data centers are a top
priority. Many projections show that the
market for local edge micro data centers
will approach or exceed the market for
hyperscale mega data centers. Historically
speaking, the data center market has
been cyclical between centralized and
distributed. I see it coming again, but it
may move to a more balanced architecture
as core and edge become an integrated
architecture.
Why are micro data centers important?
Applications and operations are moving
closer to the user or the data on the edge
and micro data centers are handling many
business functions. Think of the hotels
that solely rely on local data centers for
coding digital room keys and managing
reservations. In the future, many hotels
are considering facial recognition for
completely automated experiences. As
the world gets more automated, it will rely
more and more on micro data centers.
In a 5G architecture, micro data centers
are essential. And, local clusters are
necessary to meet 5G performance targets.
But we have the speed of light limitation
of 300 million meters per second. So, with
less than 1 ms of latency spec for 5G, the
maximum distance is less than 200 miles
round trip – and that’s a theoretical best
case. We are dealing with many carriers
and they are laying out their clusters in
circles with a much smaller radius than
100 miles, especially in densely populated
areas.
Can you provide some background on
5G micro data centers? Let’s start with
the two main enablers of 5G: one is the
new radio access network (RAN) and the
other is the data center architecture. 5G
uses microwaves and millimeter waves
at very high spectrum so the signal does
not go far. An easy way to think of it is the
three-house rule: to operate 5G you need
In a 5G architecture,
micro data centers are
essential. And, local
clusters are necessary
to meet 5G performance
targets
an antenna for every three houses. That is
how close together they need to be. And,
let’s go back to the speed of light at 300M
meters per second. The only way to achieve
the required latency of less than 1 ms is to
build local clusters that will include micro
data centers. In a recent 5G test in Chicago,
a 4K movie was downloaded in 20 minutes
using 4G and 19.5 minutes using 5G. Why
did that happen? It was because the only
5G portion on the connection was from the
small cell hanging on the light pole to the
phone – about 100 feet. The movie was in
a data center many miles away. In the near
future, micro data centers will serve the
function of mobile edge computing (MEC).
They will have traditional telco functions,
like call routing, and also IT functions, like
content delivery. For example, that movie
could have been stored in the small cell and
downloaded almost instantly.
What will 5G micro data centers do for
energy savings? This is a lively discussion
topic as Schneider and our industry have
been focused on energy efficiency for
larger data centers over the years. Our
goal is to make the micro data centers as
efficient as the hyperscale data centers with
no increased OpEx. Schneider, like many
companies, has carbon neutral goals and
edge data centers are a big part of those
goals. We know that globally, millions of
units will be needed to support 5G. As we
 Advertorial: Schneider Electric

discussed, the energy use is being shifted

from the core to the edge, and it’s a top-of- Leveraging Cell Sites for Mobile Edge Computing
mind issue to address energy efficiency.
For 20 MW, 2,000 10kW micro data centers, 4.5-5G requires computational power in closer proximity to users - creating
a 20 percent efficiency reduction will a unique edge computing opportunity for cell site owners. However, this
roughly cost an extra $4 million per year. transformation is not without its challenges. End-to-end solutions focused
That’s why it’s important to make sure on power, cooling, enclosures, and management software are needed.
the designs of micro data centers include
highest efficiency cooling technologies like
liquid cooling, for example. And we can’t
forget the management and maintenance
aspect. A cloud-based management system
is critical for looking at thousands of
sites. And accurate reporting is absolutely
essential so it’s clear when maintenance
is needed, and it can be performed quickly
and efficiently. This has been an area of
focus for Schneider.

Can you provide updates on Schneider

Electric Secure Power strategy for 2019,
and key focus areas for next year?
Schneider is working across different
stakeholders that are vying for a leadership
spot in the data center architecture needed
for 5G deployment. Telco used to be an
exclusive country club with only a few
members. But 5G, a new technology that
embraces openness, is a public course
where everyone from Tiger Woods to
Happy Gilmore can show up with clubs. In
addition to the carriers and traditional telco
equipment providers, we are dealing with
cable TV service providers, IT companies,
and internet giants. Our strategy is to
enable these companies to add value in
the 5G ecosystem. We are focused on Schneider Electric
integrating the safest enclosures, latest Contact Details
battery technology, most innovative Steve Carlini
cooling, and cloud-based management Steven.Carlini@se.com
systems. Of course, in 5G, as in golf, the
best player will win.
Flood the world with fiber
If we can lower the cost to deploy fiber,
we could connect the planet.
Sebastian Moss reports

Sebastian Moss
Deputy Editor

32 DCD Supplement • datacenterdynamics.com


W
orking in this industry,
discussing the roll-out
of 5G and the impact
of a world full of
connected machines,
it can feel like we have
already solved the basics of the digital age.
But we’ve left half of the planet behind
us. “We've reached four and a half billion
Internet users,” Isfandiyar Shaheen told DCD.
“How do we onboard the next three to four
billion people on the Internet?”
This question, of how to connect billions
and provide a stable connection for billions
more, has consumed the lives of many.
Giant corporations have sunk millions into
ambitious schemes involving balloons,
satellites and huge drones.
Shaheen, founder and CEO of NetEquity,
believes there’s a simpler approach: Fiber.
“When you compare the bandwidth of
fiber to everything else, whether it's a LEO
satellite, microwave, millimeter wave - those
things are not even in the same quadrant. It’s
literally, thousands of times less bandwidth
than a single set of fiber.”
His company believes that it is possible
to reach a vast number of the disconnected
or poorly connected people in the world by
deploying fiber across utility networks. “The
theory I'm working with is saying that if you
follow the electrical grid to deploy fiber, you
can fiberize 80-90 percent of cell towers in
the world,” Shaheen said
His hope is that he can convince utilities
to deploy fiber along their grid infrastructure
as “utilities need fiber to achieve substation
automation,” Shaheen said. “Utilities have
also created a standard called IEC 61 850
that relies on fiber to do key things such
as integrating renewables, integrating
storage, and achieving a hell of a lot more
automation to reduce their line losses, which
will lower operating expenses and through
which they can start lowering their capital
expenditures. Because when a substation
goes digital, its copper footprint goes down
by 80 percent.”
The utilities he hopes to convince are,
however, “mostly bankrupt - they're running
on state subsidies. So the solution I've come
up with is to say to them ‘I will build you a
fiber network on my expense, and you get to
lease it based on whatever the going rate for
dark fiber in your market is.’ It's usually about
60 cents to 70 cents a meter a year.”
“That kind of lease payment makes it
feasible to raise about 80 percent of the
project costs as debt.”
Once the network is underway, then
it becomes easier to turn to telecoms
companies and say “‘hey I am bringing fiber
to your tower, but instead of charging you
whatever the market rate is for this service,
let me make this an opex neutral deal for you.
If you are spending $500 a month per tower
on running microwaves, pay me the same
$500 per tower per month, and I will give you
a fiber connection,’” Shaheen said.
“But I won't give it as dark fiber, I will give
it as a lit service. Because if I give it to you
as dark fiber, you will lock away bandwidth.
My goal is to make bandwidth abundant.
So I would offer you abundant bandwidth,
for the same operating costs of running
microwaves. Now that combination of
selling services to cell towers and getting
these revenues from dark fiber leases from
the utility that makes the whole business
case hold.”
By playing to the pain points of the utility
and the telecoms company, Shaheen hopes
to avoid the aggressive legal fights and turf
wars that have stymied others’ attempts at
creating utility bandwidth.
A few examples of municipal fiber exist,
with Shaheen pointing to efforts of small San
Juan-based Orcas Power & Light Cooperative
(OPALCO), which “deployed fiber at the cost
of about $30 a meter.
“What's been cool about this cooperative
is they published a very nice payback
analysis in one of their board publications.
And that shows a payback period of about 12
“Currently, the dry
powder of infrastructure
funds is sitting at $180
billion”
years. It shows the breakdown of the sources
of savings, which came largely from saving
the number of trips that their guys were
making for repairs, because fiber gave them
visibility across the grid.”
For Shaheen’s plan to work, however,
requires fiber to be deployed at a lot less than
$30 a meter.
“Fiber cables are a commodity, it costs
less than $1 per meter. Yet most deployments
end up costing $30 to $50 a meter, and that's
because all the cost is in labor and right of
way. If there is a way to bypass labor and
right of way, there's no reason you can't
deploy fiber at $4 a meter or around that
benchmark.”
Shaheen believes that there is a way to
avoid these cost barriers. But to achieve a
radical price cut, Shaheen has to rely on an
ambitious secret hardware project by another
company to be pulled off without a hitch.
“What allows me to pull this stunt off is
that I have this relationship with Facebook,”
Shaheen said. “I signed an MoU with
Facebook that gives me access to some
Fiber to the people
technologies that can substantially lower
the cost of fiber deployments. It's not a
completely ready technology, that's the risk
associated with the project.”
Facebook declined to detail the
technology, which is currently under
development. The project is part of
Facebook’s Connectivity division, home
to several projects focused on improving
Internet penetration, including an ill-fated
drone project and the controversial Free
Basics effort.
Shaheen met Facebook representatives
at a Telecom Infra conference, advised the
company as a consultant, and then became
its first ‘Entrepreneur In Residence.’
“I have no funding from them,” Shaheen
clarified. “This is important and by
design, because if I were to have a deeper
relationship with them, then I am subjected
to their bureaucracy - they’re a 60,000
person company. I don't need the entire
company, I just need to talk to a couple of
really solid engineers.”
Those engineers are key to Shaheen’s
strategy, and roll-outs can’t proceed without
the technology working. “My bet isn't so
much on Facebook the company, my bet is
on who I am collaborating with.”
In the meantime, Shaheen is working on
“building a business around the promise,”
he said. “My timeline is such that for the
next year to year and a half, some of the
useful tech that Facebook is working on will
get finalized and will get prepared. During
that period, I want to generate a pipeline of
deals, sign NDAs with utilities, get their grid
maps, turn them into investor documents,
and then start putting together investor
consortiums.”
So far, he says he has “made progress
with utilities in Pakistan and South Africa,”
and is working with a large East Asian
investment group to explore a partnership.
If the technology works and $4 a meter fiber
is possible, Shaheen is convinced he can
sign up more investors: “Currently, the dry
powder of infrastructure funds is sitting at
$180 billion.”
But to pull off that dream of
connecting the world will require a
large chunk of those funds. “There’s
25 million route kilometers of power
lines that are suitable for fiber
deployments and at $4 a meter
that’s $100 billion; at $5 a meter,
that's $125 billion.”
Should he be able to line up
investors, Shaheen hopes to
“flood the world with fiber, make
sure it's not under exclusive
contracts, and create a true
public utility that can that can
help many billions of people access the
Internet at a price that they can afford.”
Telco Supplement 33
Routers in the sky
Terrestrial fiber has ruled the telecoms world for a
generation. Now satellites are promising to match its
performance, says Doug Mohney
G
igabit-speed satellite
broadband promising
performance equal to or
better than terrestrial fiber
is almost here. Newcomer
OneWeb and industry darling
SpaceX are launching satellites in earnest by
the end of 2019 to build global high-speed
networks. Dozens of satellites at a time will
go into space on each rocket with each
mission building towards constellations of
thousands of spacecraft overhead.
With full global coverage expected to be
completed in 2021, OneWeb and SpaceX’s
Starlink project aims to extend broadband
to underserved and unserved regions of the
world, along with more profitable markets
including aviation, maritime, government
and enterprise sectors.
In contrast, LeoSat and Telesat plan to
launch satellites in 2021 specifically designed
for enterprise-class services - “MPLS routers
in the sky” with “fiber-like” performance
according to executives from both firms -
with global service turnup expected by the
end of 2022 to mid-2023. Looming behind
them all is Amazon, with its own ambitions
and needs.
“Our approach to [low earth orbit
broadband] is we’re building a Layer 2
satellite system,” said Erwin Hudson, vice
president of Telesat LEO. “We’re designing
our system to be compatible with MEF
standards from the bottom up to provide
enterprise-quality service with business class
SLAs.”
Appealing to the corporate IT department
is a strategy start-up LeoSat is finding
success with. “I can explain what we do to a
data guy in five minutes,” chief commercial
officer Ronald van der Breggen said. “We’re
putting a bunch of MPLS routers in the
sky, connecting them with lasers, and you
can use them on any point in the world
to connect. ‘When is that available?’ is the
first question. We have a service which is
unique, resonating with enterprise and
governments.”
Both satellite constellation designs
34 DCD Supplement • datacenterdynamics.com
include optical cross-links between satellites,
a feature providing more speed and security
over traditional fiber. Customers can choose
to set up a connection between sites that
exclusively rides over the satellite network,
bypassing traditional network exchange
points and reducing the number of hops
found with terrestrial connectivity. Laser
light traveling through the emptiness of
space in a straight line between satellites
moves faster than it does traveling through
a glass fiber following a meandering path
under the sea, through cities, and along
railroads, highways, and gas pipelines.
Fewer ground hops mean fewer access
points for interception or disruption,
providing a level of resilience against
the threat of backhoes and other fiber
disruptions.
“Data centers are a
key component of our
network when it is fully
rolled out”
LeoSat and Telesat expect to have their
respective networks ready to deliver service in
the same timeframe, but the companies have
very different financial paths to get there, with
LeoSat facing a bigger hill in front of it.
“It’s no secret, we’ve had difficulties in
raising the equity,” said van der Breggen (see
Note). “We had hoped to close the Series A
right before the summer. Now we’re working
very hard to find additional investors to get
us over the hump. There’s a lot of interest
from venture funds, strategic funds. We need
another investor to line up with what we
already have and get us over that hump.”
LeoSat estimates it will take $3 billion to
build and launch 90 satellites for its initial
network. Satellite providers SKY Perfect JSAT
and Hispasat placed early investments in
LeoSat, but exact amounts have not been
disclosed. The company also has logged
Doug Mohney
Contributor
$2 billion of customer commitments in
Memorandum of Understandings (MoUs),
with a diversified mix of firms including
high-frequency trading markets, oil and gas,
and telecom firms for good measure.
Broadband speed offerings for LeoSat
start at 50Mbps and range up to 1Gbps, with
the most popular requests in the MoU stack
100Mbps. Latency is expected to be in the 20
millisecond (ms) range for a simple trip up
and down between ground and satellite, but
van der Breggen played down fixating on the
simplest of latency examples.
“It’s a meaningless case and only tells you
so much what an individual satellite is able to
do,” van der Breggen said. ”The capabilities of
a service are going to be far more important.
You have to add in all the fiber [involved]
as well as the satellite.” Routing via satellite
optical links between major financial centers
is expected to be significantly faster than
submarine fiber connections, he said,
providing a competitive advantage to banks
and stock trading firms.
In comparison, privately held Telesat
is one of the oldest and largest satellite
operators in the world. The Ottawa, Canada-
based company publishes quarterly and
annual financial reports since it issues
publicly traded debt, providing a transparent
window into its financials. Last year, Telesat
closed around $680 million in revenue and
had a contracted backlog for future services
of nearly $2.8 billion.
Telesat plans to launch 198 satellites for
global coverage in the first half of 2023, with
service available in the polar regions by Q3
2022. An additional 100 satellites will be
added by the end of 2023 for a total of 298
satellites in the initial constellation. User
service speeds are expected to be scalable to
the Gbps range with one-way latency “less
than 50ms” in the same continental region.
Pricing to build the system, including
satellites and ground equipment, and launch
it into orbit hasn’t been finalized with the
constellation expected to cost “several
billions of dollars,” Erwin said. A primary
contractor for the system should be selected

Source: LeoSat
by the end of this year, with Telesat asking
the winning company to build a factory for
its satellites in Canada.
“We intend to finance Telesat LEO with a
combination of cash, equity and debt,” said
Erwin. ”As a leading global satellite operator,
we generate substantial cash flow and are
able to provide a significant amount of
funding for the LEO development ourselves.”
With an established customer base and
a large sales and marketing organization,
Telesat also has the Government of Canada
as an anchor customer, with a commitment
of $500 million over 10 years as a part of
efforts to expand broadband access. Canada
will use Telesat to deliver backhaul services
for ISPs and phone companies in under-
connected communities.
Both LeoSat and Telesat see data centers
as primary partners for delivering network
access. “The short answer is data centers
are a key component of our network when
it is fully rolled out,” van der Breggen stated.
“A large portion of traffic will go into data
centers.”
Multi-tenant facilities such as Equinix are
prime real estate, enabling a gigabit satellite
service provider or a third-party handling
the work within a physical facility to connect
customers to high-speed interconnection
points, existing terrestrial networks, Edge
computing, and other colocated resources.
Telesat plans to establish PoPs at major
Internet and cloud exchange points to
interconnect with customer networks. “Our
network architecture will drive more traffic
to data centers on our global WAN,” Erwin
said.
Telesat also plans to work with data center
operators as well as cloud service providers
to simplify access to “cloud on-ramps” for
its existing enterprise customers. “There’s
about a dozen network access points around
the world where we aggregate three to five
of our earth landing stations at a common
point,” said Erwin. “Regional customers
can connect to our network at that point of
presence.”
However, data centers could be more
than where LeoSat and Telesat spend money
to connect into the rest of the world.
“We’re not really focused on fiber
replacement, but we can provide disaster
recovery if [a data center] lost terrestrial
connectivity,” Erwin said. Telesat can deliver
1Gbps service using a 1m satellite dish, with
higher speeds possible with larger dishes.
Delivering 1.2Gbps to 2Gbps of connectivity
Routers in the sky
using a 1.8m antenna is feasible.
While there has been skepticism in some
circles about the commercial viability of LEO
broadband constellations, LeoSat, OneWeb,
SpaceX, Telesat received some validation
and headache this spring.
Amazon announced Project Kuiper,
its satellite broadband project, this spring
after its ITU spectrum filings become
public. The e-commerce and cloud giant
wants to launch over 3,200 satellites
for internal broadband use as well as to
provide broadband to customers but hasn’t
discussed a timetable for when it will start
putting hardware in the sky or offer services.
Amazon may start launching satellites
by 2023, but it isn’t clear at this point in
time if it will be a direct competitor to the
enterprise-designed services of LeoSat and
Telesat.
Note: Less than 24 hours before
DCD>Magazine went to print, LeoSat
underwent several layoffs, including COO
Ronald van der Breggen and at least two
other executives.
The company did not reply to multiple
requests for comment. Updates to this story
will be posted at datacenterdynamics.com
Telco Supplement 35
 Telco sell-off

T
he news that Telecom Italia
is looking to spin-off 23 of its
data centers and list them on
the stock market is only the
latest in a series of moves,
which are seeing telecoms
service providers backing away from earlier
plans to make a lot of money out of data
center colocation.
It seemed so simple in the early years of
this decade. Data centers were booming,
and they are a service industry based on
infrastructure hardware. To telecoms
operators, it looked like a logical expansion,
and many of them dived into the market.
Ten years on, most of them are exiting.
“Despite many telcos making moves into
the data center and cloud infrastructure
markets, more and more are now realizing
that they would rather concentrate on their
core business and let someone else manage
their data centers,” says Massimo Bandinelli,
marketing manager at Telecom Italia’s
compatriot Aruba.
Many telcos simply bought existing
data center providers, often at high prices.
Verizon, for instance, acquired data center
provider Terremark in 2011 for $1.4 billion.
Eight years later, the company decided that
offering colocation services did not fit with
its business model, and sold off its data
centers to Equinix for $3.5 billion.
Also in the US, AT&T painstakingly
accumulated a network of data centers, only
to sell them off to Brookfield Infrastructure
and other institutional partners for $1.1
billion in 2017. Brookfield relaunched them
as a new data center provider, Evoque.
Also in 2017, CenturyLink sold 57 data
centers for $2.3bn to a consortium that
became another standalone data center
provider, Cyxtera.
It wasn’t a sudden change. DCD first
Peter Judge noticed the phenomenon in 2015, when
Global Editor some smaller telcos unloaded their data
centers. For instance, in that year Arkansas
telco Windstream sold its holding of 14 data
centers to TierPoint for $575m, giving that
provider 179,000 sq ft (17,000 sq m) of space.

The Telco data

Rumors started about the imminent
sales at the telco giants AT&T, Verizon and
CenturyLink back in 2015, but took a couple
of years to come to fruition.
Photography: Sebastian Moss

center sell-off
Telecoms providers thought they’d be great at
colocation data center services. Now they’re
mostly getting out of the game, says Peter Judge
The move took in telcos which had built
out their own data centers, as well as those
which acquired them. Telecom Italia, for
instance, had at least some of its facilities
built by a partner from the telecoms
industry - Ericsson, which is primarily a
network provider.
Also in Europe, Telefónica SA sold off
its 11 colocation data centers. They went
to Asterion Industrial Partners for €550m
($600m). In the UK, BT seems to have been

36 DCD Supplement • datacenterdynamics.com

selling its data centers off one-at-a-time in
deals like a 2015 sale which saw a Tier III
facility near Gatwick go to operator 4D.
The trend extends to younger markets
as well, where the telcos’ data center
investment may have been much more
recent. In Latin America, Mexican telco
Axtel sold three data centers to Equinix for
about $175m.
Indian telco Tata flipped dramatically. In
2013 and 2014, it saw a period of rapid data
center expansion, building or acquiring
44 data centers in India and elsewhere in
"The workforce required
is very different from the
one needed as a telco
operator”
Asia. It then floated them as an independent
subsidiary, Tata Communications Data
Centres (TCDC). However, in 2016, Tata
decided they were more of a liability than
an asset, and sold TCDC to Singapore’s ST
Telemedia for around $650 million.
It’s easy in hindsight to think that
these telcos stumbled into data centers by
mistake, getting into an area they did not
fully understand, where they would face
more focused competition that could run
rings round them.
That analysis is pretty much true, but in
the latter years of this decade, telcos faced
quite a few financial pressures, analysts
have pointed out,
Back in 2015, The Motley Fool’s Adam
Levy suggested US telcos needed money
because they had paid heavily for wireless
spectrum, but were not yet gaining huge
revenues from mobile data. “Both [AT&T
and Verizon] spent heavily in the FCC’s
AWS-3 spectrum auction, acquiring
valuable airwave licenses for their wireless
Windstream Tata Comms
Sells to Sells to
TierPoint ST Telemedia
2015 2016
businesses,” wrote Levy.
Heavily burdened with debt, many telcos
have been looking for ways to raise money
in recent years. And the data center field
has been a good potential source of cash as,
during a period of rapid growth, facilities
are valued very highly.
While selling CenturyLink’s data centers,
company chief executive Glenn Post told
Barclays Capital’s Amir Rozwadowski that a
lot of the motivation was simply that buyers
were prepared to pay big money for those
assets.
And telecoms operators, who have been
burnt in previous market crashes, have been
understandably keen to get ahold of that
cash, ahead of any potential future crash.
“First of all, as to why now is an
opportune time… valuations are obviously
good right now. They can always change,
but we know the market’s good.” Post said,
back in 2015. “We think our cashflow could
be used for investments that can drive
higher returns, and better shareholder value.
So that’s why we’re looking at divesting data
center assets.”
But there are other reasons why data
centers are not such a good fit for telcos as
they might have been once. There have also
been changes in the data center industry
over the last several years, which have
moved data centers ever further from the
comfort zone of telecoms providers.
“The workforce required is very
different from the one needed as a telco
operator,” warns Bandinelli, adding that this
specialization has increased, as data centers
have become more evolved and more
commoditized.
There are also large investments required
to keep up to date, adjusting to industry-
wide regulations, adopting standards,
getting certifications, and moving to
renewable energy.
Alongside this, new business models are
emerging like Edge resources, while at the
other end of the scale, hyperscale providers
are building a market for huge facilities
AT&T CenturyLink Verizon
Sells to Sells to Sells to
Evoque Cyxtera Equinix
2017 2018
which are not cost-effective for a services
company to deliver.
When the sell-off began, Zahl Limbuwala
of data center analytics company Romonet,
now a subsidiary of CBRE, felt that the
telcos may have taken a view that their data
centers were likely to go down in value
because of the investment requirements: “If
your data centers are approaching 10 years
old and have not had a major reinvestment,
you are in for a nasty surprise,” he told DCD
in 2015.
Some organizations have invested
in data centers seeing them as a kind of
commercial real estate with very large
returns. Limbuwala pointed out that they
also had high costs, with a “reinvestment
time” before new investment is needed,
of 10 years, about half that of mainstream
commercial property.
Despite this gloomy picture, there
are some telcos which are apparently
exceptions to this trend. Japan’s NTT has a
thriving data center subsidiary, which has
absorbed RagingWire in the US, NetMagic in
Asia, e-shelter and Gyron in Europe, and is
in the process of forming them into a single
coherent unit.
However, NTT is an exception, and
operates its data centers at arm’s length.
RagingWire CEO Doug Adams contrasts its
approach with that of US telcos like Verizon
and AT&T: “[The US telcos] were very short-
sighted, very quarterly focused,” he said in a
DCD interview earlier this year.
“They were getting their tushes handed
to them by the Equinixes, Digitals and
RagingWires of the world, and they backed
out. I think NTT was extraordinarily
intelligent for doubling down on this
business.”
Whether it is standalone data center
firms, or subsidiaries like NTT Data Centers
operating independently, Bandinelli
believes that the world is shifting towards
pure-play data center providers that are able
to meet market requirements and provide
competent technicians at a lower cost.
Telefónica Axtel Telecom Italia
Sells to Sells to Sells to
Asterion Equinix ???
2019 2020?
Telco Supplement 37
GROWTH
Colocation providers see faster

with lower risk.

EcoStruxure™ for Cloud and Service Providers

brings efficiency to any site, on time.

• Lower total cost of ownership with efficient, sustainable

CapEx and OpEx management.
• Achieve higher operational performance while
reducing energy costs up to 60%.*
Galaxy™ VX UPS
• Get support throughout your data center’s lifecycle —
from design to construction and operation.
EcoStruxure IT
#WhatsYourBoldIdea

se.com/colo

* Based on previous data, 2017. This is not a guarantee of future performance or

performance in your particular circumstances.

©2019 Schneider Electric. All Rights Reserved. Schneider Electric | Life Is On and
EcoStruxure are trademarks and the property of Schneider Electric SE, its subsidiaries,
and affiliated companies. 998_20607840_GMA-US
 Amazon’s spotty pricing

Amazon’s spotty pricing

AWS changed its Spot pricing to be smoother. The result ended up
more expensive, and less transparent. Sebastian Moss reports Sebastian Moss
Deputy Editor

I
n November 2017, AWS changed how at some point, if all of the institutions get probability. It was a great success,” Wolski
it charged for a service. The switch, full, what we want to do is burst from the said. “This went on for a couple of years.”
made suddenly and with little fanfare, Federation into Amazon,” Wolski told DCD. Then in late 2017, something happened.
was touted as a small improvement The group decided to use the Spot market “We saw in the press that Amazon had
- but raised prices and accidentally to maximize cost savings. Jamie Kinney, changed the pricing. At first, I was overjoyed
stymied a government cloud project. AWS senior manager scientific computing, - we thought, wow, this is great. If you
said in a press release at the time: "We are smooth things, the technique that we had
AWS EC2 Spot Instances, launched in 2011, excited to work with the Aristotle team developed should just become much more
have always been something of a gamble. to provide cost-effective and scalable accurate.
Available at a significantly lower price than infrastructure that helps accelerate the time “And we started looking at it, and it didn't
standard EC2 instances, the Spot market to science.” look right. From a mathematical perspective,
allows users to bid for the remaining capacity from a data analysis perspective, it just didn't
in an AWS data center. The more bids, the But as it was university-led scientific look like what the press was saying was
higher the price - or at least, that’s the claim. research, backed with government happening.
While Spot Instances are cheaper, users money, the ‘bursts’ required some level of “Why doesn't this look right? Has
run the risk of the work being terminated if predictability. “Universities do fixed budget something else changed? Is our method
the Spot price exceeds the maximum price resource allocation, you get ‘this’ many wrong?” Wolski’s team scrambled to work
bid by the user, or if the capacity is no longer dollars, and it has to last ’that’ many years,” out what had happened. “We started digging
available. Wolski said. into it, we read everything we could read,
“What you're looking at there is our So Wolski and his team developed an and we started seeing reports from the
attempt to recover the marginal cost of algorithm to predict Spot price changes, popular press about companies that had
that as-yet unused capacity, capacity that and the likelihood that a workload would their own internal algorithm for optimizing
has not yet been sold for demand usage be terminated early. “We would be able to their use of the Spot market. And those
or for reserve instances,” Ian Massingham, say if you bid ‘this’ much, you'll get a day's algorithms were breaking.
AWS director of developer technology and worth of time, guaranteed with 99 percent “We went back in and just did a very
evangelism, told DCD last year.
“So that's essentially what the Spot
market is; it is AWS recovering the marginal
cost of having large amounts of capacity
deployed and unused by customers around
the world.” At the time, however, AWS had
already changed its algorithm - and Amazon
has since declined numerous requests for
comment from DCD.
In the early years, the potential cost
savings from Spot pricing proved enticing
for many, including the US National
Science Foundation. Rich Wolski, professor
of Computer Science at the University of
California, Santa Barbara, was part of a team
building a federated cloud for several US
universities with NSF backing.
The aim of the Aristotle Cloud Federation
was for the institutions to share computing
resources across their data centers. “But

Issue 34 • November 2019 39

careful analysis,” Wolski said. The results,
published in the research paper Analyzing
AWS Spot Instance Pricing (August 2019),
found that prices were higher by an average
of between 37 percent and 61 percent.
But price increases were not the real issue
for Wolski’s team: “If you're doing fixed
budget stuff, that just means you have less
work to do,” he said.
The problem was that it became far
harder to predict which workloads would be
terminated, with the system relying less on
an IMS Engineered Products Brand
STANDARD AND CUSTOM SIZES
40 DCD Magazine • datacenterdynamics.com
auction-like market forces, and instead on a
hidden algorithm to decide costs and when
to end workloads. “This had an impact,”
Wolski said. “It was suddenly unreliable,
people who were depending on the fact that
we can make this prediction could no longer
use it.”
The change crucially shifted Spot
Instances from what it used to be, Wolski
said. “There's no indication that after the
change it's a market at all, it's just retail
pricing - it's dynamically changing retail
pricing. Amazon has full price control of the
Spot Instances.”
“I felt pessimistic,” Professor John
Brevik, who worked on the
original NSF technique
and the subsequent paper,
said. “It's sort of like this
door's been shut on what
was an interesting thing to
figure out - this dynamic
pricing mechanism and
how to predict it. I'll leave
it to the more corporate or
economically astute to infer
why that kind of change
happened.”
But while the 2017 shift
was the final nail in the
coffin for the market system,
Spot Instances have always
relied on hidden algorithms
and an invisible hand to
control pricing, Orna Agmon
Ben-Yehuda told DCD. Her
team at the Israel Institute
of Technology studied
Spot Instances when they
launched in 2011 up to 2013.
5400 lbs load rating
“In 2011 we first showed that during
the first two years of the operation of the
AWS spot instances, 98 percent of the price
traces were consistent with being the result
of an artificial algorithm. This algorithm
computed a reserve price: a price under
which AWS were not willing to rent the
instance.”
Her work discovered “the existence of
several unnatural, artificial characteristics,
which had no economic justification.“
She added: “I would like to stress that the
problem was never that AWS had a reserve
price, or even that they changed it. The
problem was that they declared their prices
were based on supply and demand... and
people believed it, and based their academic
work and economic plans on that.”
There are legitimate reasons for some
control of the market, Steve Fox, CEO of
AWS reseller AutoScalr, said. Users began
to realize that if they bid ridiculously high
prices that forced others to use the standard
On Demand service it would clear the
marketplace. "I only have to pay that high
price for a couple of minutes, and then I'm
back down at the cheap price - and I am
never interrupted."
Fox told DCD: "So it turned into this game
of chicken, where people started bidding
higher and higher and higher, just trying to
keep from getting interrupted. And it got so
bad, where prices were going extraordinarily
high, like 1,000 times. So Amazon put a cap
on it to say prices could never go above 10
times the On Demand price."
For AutoScalr and its customers, the 2017
change did bring about price stabilization
but left the company in the dark as to when
workloads would be terminated. "So now
UL LISTED RACK HEIGHTS UP TO 62 RU

you had a very predictable price that didn't
change very fast, but you never knew when
it was going to go away," Fox said.
Previously, prices would rapidly rise
when more users requested Spot Instances,
and it was obvious that the chance of
being terminated would rise with it. "And
so we had algorithms that would diversify
away from the risk and go to more stable
spot markets, which meant our overall
interruption rate was lower," Fox said.
Now the price change is much slower - "it
looks like it's on the order of days or weeks,
whereas before, it was minutes.
“So the challenge is, when a lot of people
come in and start to use an instance,
“[AWS uses]
unnatural, artificial
characteristics, which
had no economic
justification”
eventually they run out, and the price
doesn't change. But how do they go about
picking who to terminate? That's a big
mystery."
Fox, whose company is a certified AWS
partner, was keen to point out that, despite
the change, AWS Spot Instances are still
cheap - and added that Amazon regularly
makes price cuts across all of its services.
"It's like, yeah, maybe the prices have
crept up a little bit. But it's a dramatic way to
CUSTOM RACKS MANUFACTURED DAILY
Amazon’s spotty pricing
save on cloud compute. Maybe it wasn't as
good a deal as it was a few years ago, but it's
still a good deal."
Customers of AutoScalr, he said,
continue to use to Spot market. "You just
have to lean heavier on diversification, as
opposed to prediction."
But for Wolski’s ‘burst’ method for the
Aristotle Federated Cloud, the change
proved fatal.
"I don't want to ascribe to them a
nefarious purpose here,” Wolski said.
“I think it was more that we were so
far underneath their radar that they
just missed it." He believes that his
technique, which was publicly
available, added value to the Spot
Market, making it better for other
users.
“You have the science community doing
something that might make other people
use Amazon in a more efficient way, and
we're not charging for it,” Wolski said.
“I sense that if the Amazon people had
thought about that, maybe they would have
announced this thing differently, or they
because it's retail pricing, that will be stable."
While he is working on a solution, the
would have contacted us.” whole experience has given Wolski pause.
His team is currently working on a “It was an important lesson for the science
replacement tool that hopes to rank various community. Normally, we buy machines.
cloud companies' instances and compare And when you buy a machine, it's that
them against internal capacity for a given machine until you throw it away.
workload. “It doesn't morph into something else
"So if I'm gonna burst with a retail halfway through its lifetime. If on a Tuesday
product, what's my spend going to be? it's an x86 box, it's going to be an x86 box on
What's say the minimum I have to spend to a Wednesday. But when you buy a service,
get the same power? Or if I want twice as that can happen. It was still called Spot
much power, how much do I have to spend? Instances, it still had the API.
Or if I can wait twice as long and I want half “It's just that on some day, you went to it,
the power, what do I have to spend? And and it behaved completely differently.”
Learn more at amcoenclosures.com/dcd
CUSTOM RACKS CAN SHIP IN 2 WEEKS MADE IN THE USA
Issue 34 • November 2019 41
Introducing the
DEKA SHIELD from

Our exclusive Deka Shield program gives you peace of mind and exclusive
additional warranty protection for your Deka batteries. This benefit is just one offering
of Deka Services, the North American service network operated by East Penn.

Exclusive DEKA SHIELD

Deka Shield is an innovative and exclusive program to provide optimum battery performance and life to your Deka Batteries no matter the
application: Telecom, UPS, or Switchgear. By allowing Deka Services to install and maintain your batteries, your site will receive extended
warranty benefits.
How do I sign up for the Deka Shield program?
 Installation must be completed by Deka Services or a Deka Services approved certified installer
 The application and installation area must be approved by Deka Services prior to installation
 Access for Deka Services to perform annual maintenance during the full warranty period

What coverage does the Deka Shield program provide?*:

 Full coverage labor to replace any  Freight allowance for new product to
defective product installation site
 Full labor to prepare any defective  Full return freight for defective product
* Terms and conditions
product for shipment  Extended warranty apply – please contact us
for additional information.

Extensive DEKA SERVICES

Deka Services provides full service turnkey EF&I solutions across North America. Their scope of services include, but are not limited to:

• Turnkey EF&I Solutions • Battery Capacity Testing • Project and Site Management
• Battery Monitoring • Removal and Recycling • Logistical Support
• Battery Maintenance • Engineering • Installation

All products and services are backed by East Penn, the largest single site
lead battery manufacturer in the world. With over 70 years of manufacturing, battery,
and service expertise let Deka Services be your full scale power solution provider.
 Don't just look to the north
Virginia’s land dilemma
With Northern Virginia bursting at the seams, new submarine fiber
could create new hubs in the south of the Commonwealth. Will
Calvert reports
N
orthern Virginia has arguably
the highest concentration of
data centers on the planet.
In 2018, 115MW of data
center capacity was leased
by companies in the region
alone. This is almost double the 59MW of
capacity absorbed in any other US market
in a single year, according to real estate firm
Jones Lang LaSalle.
But is having that much capacity
flowing through one market something
to be celebrated? Or is it something to
worry about? Many experts think it’s time
to start spreading capacity across the
Commonwealth of Virginia as a whole
instead of letting it concentrate in the north.
Sean Baillie, chief of staff at data center
provider QTS, told DCD: “I’ve lived in
Ashburn for 19 years and we're running out
of easements [permits to dig across land].
We have so much fiber here, and there is
nowhere else to dig.
“There is one main core ring in Ashburn,
and there's a bunch of sub rings that hang
out of it. The main core ring is being
upgraded by a company called Fiberlight,
and they're digging in the median [the
central strip of the road, referred to in some
countries as the central reservation].
“The public rights of way outside are full
and there's nowhere else to put anything.
And as soon as the median fills up, there's
nowhere else to go. So, we’ve got a geography
problem.”
One of the ways to address the diversity
problem is to bring in more fiber from
outside the Commonwealth. Until recently, all
fiber links to Northern Virginia had to come
from other states, with the nearest submarine
cables making landfall in New Jersey.
In recent years, Virginia Beach in the
south of the Commonwealth has become
established as a fiber landing point. Telxius’
Brusa and Marea cables in Virginia Beach
arrived in 2018, offering a shorter journey for
terrestrial fiber connections to Northern and
Southern Virginia, compared with the nearest
East Coast landfall 300 miles north in New
Jersey.
Spanish for tide, Marea was built by
Telxius, the infrastructure arm of the
46 DCD Magazine • datacenterdynamics.com
Telefónica group, and funded by Facebook
and Microsoft. The submarine cable stretches
across the Atlantic from Virginia Beach to
Bilbao in Northern Spain. Marea is capable
of reaching 200 terabits per second (Tbps) of
transmission capacity.
Also built by Telxius, but funded by parent
company Telefónica, Brusa (Brazil-USA)
is a private cable that offers low latency
communication links between the US and
Brazil. The cable also departs from Virginia
Beach, but lands in Rio de Janeiro, Fortaleza,
and San Juan in Puerto Rico.
The need for capacity and the arrival of
submarine fiber in the South of the state is
enabling growth in southern counties like
"We were hit with the
outages when Hurricane
Sandy rolled through the
Northeast"
Henrico and Virginia Beach itself, alongside
the established Northern players like
Loudoun and Prince William County.
The Virginia Beach authorities are
priming the pump with an incentive: sales
tax on power, cooling and IT equipment has
been dropped to 0.4 percent, and cheaper
land prices are also in its favor: earlier in
2019, Digital Realty bought a 13-acre plot in
Ashburn, Northern Virginia for a record $2.14
million per acre.
In Richmond, Henrico County, Facebook
has invested $1.75 billion in a 970,000 sq ft
(90,000 sq m) data center, which already has
a 1.5 million sq ft expansion planned. Just
down the road, QTS opened a 1.3 m sq ft
(120,000 sq m) data center back in 2010.
Henrico County is only 100 miles from
Virginia Beach, so this year QTS opened a
NAP (network access point) there, to offer
peering with the Marea and Brusa cables.
Telxius also connects to other facilities
including Globalinx’s carrier-neutral data
center, virtually next door to the landing site.
Will Calvert
Reporter
Don't miss
out on our new
DCD>Virginia
event in October
2020
Two more cables are on their way: next
year, Google’s Dunant cable, named after
Henry Dunant, founder of the Red Cross and
first recipient of the Nobel Prize, will connect
Virginia Beach to the west coast of France.
And SAEx International is building the South
Atlantic Express cable, which starts in Cape
Town, South Africa.
“Diversity is why we ended up in Henrico
County, Virginia,” said Najam Ahmad, VP
of network engineering at Facebook, at the
launch of the QTS NAP earlier this year.
“Ashburn has got a lot of compute power, but
it then becomes a very large single point of
failure and that is a concern for Facebook.”
There’s one problem says Ahmad: “We
need multiple paths to provide diversity.
The trouble with subsea, as always, is that
if a cable is cut, it might be weeks before it
can be fixed. If something's out for that long,
you have a good chance to pick up a second
failure and cause really massive problems.”
As if in answer to this, Telxius announced
in October that it would connect Virginia
Beach to New Jersey - the first direct fiber
between two landing stations, and a useful
extra route for intercontinental traffic.
Marea’s other funder, Microsoft agrees,
says the company’s director of global network
strategy Frank Rey: “Like any other network
provider or network user, we were hit with
the outages when Hurricane Sandy rolled
through the Northeast of the US in 2012.
Because of this we saw a huge need to bring
additional diversity to the East coast.”

A data center to
transform Bangladesh
To support the growth and digitization of Dhaka, Bangladesh’s
government decided to build a national data center. ZTE
delivered the project on time, to a world-leading standard
B
angladesh needs digital
infrastructure. The population
of the capital Dhaka has
reached 15 million, and that
creates an urgent need to
handle vast quantities of urban
data, both for the city itself and for the digital
consumers that live there.
The Bangladesh government decided to
establish high-level infrastructure, and had the
help of ZTE to get a national data center built,
in the Kaliakoir high tech park, about 50km (30
miles) north of Dhaka.
The facility is a full turnkey project in which
ZTE carried out civil work, building the data
center facility and supplying ICT equipment,
including servers, storage and network
equipment as well as software applications.
The contract was signed in May 2016, with
the facility to be delivered in three years. It was
formally accepted in May 2019, meeting that
goal.
Demanding schedule
Bangladesh’s extreme weather was an issue
during the building. “There are floods in
succession,” N M Zeaul Alam, the Secretary
for ICT in the Bangladesh Ministry of Posts,
explained to DCD. Construction had to be
scheduled for favorable weather windows.
“ZTE has designed a perfect plan and
construction table to overcome the impact of
weather. This year’s floods seem to be better
than previous years, and we are lucky to
achieve our goals to the fullest extent possible.”
Mr Alam is not exaggerating: “It rains so
frequently, that nearly six months of the year
is not suitable for outdoor concrete work,”
explained Tongbing Huo, marketing director
for data center products at ZTE. Even after the
rain clears, the roads can be very, very muddy,
making it difficult to transport equipment, he
told DCD.
Despite this, the project was finished on
time, and to a very high standard. Civil
engineering started in September of 2016, and
was accepted in December 2018. Because the
ICT part of the project was being completed
in parallel, the whole project right up to the
software stack was finished and accepted by
the end of December.
To de-risk the project, the team built in
quality, by choosing top-quality devices and
materials. They were inspected before delivery
and re-inspected after arrival, said Tongbing
Huo: “They will be verified again before and
after the installation.”
The facility has such a crucial role that
Dhaka needs the “best data center in the
world,” said Mr Alam. To ensure the facility
is reliable, the government asked for Uptime
Tier IV accreditation, “the highest data center
certification standard in the world”.
Top reliability
Specifically, the data center construction
must meet the requirements of “shockproof,
flood-proof, explosion-proof performance
for the purpose of risk prevention,” explained
Tonbing Huo. The final building’s resilience is
“even higher than the initial design,” he said.
Uptime accreditation is in two stages.
ZTE’s co-operation got the facility’s design
documents accredited before work started.
The Tier IV demonstration test of the
completed facility was done in December,
2018, and the facility was certified in March
2019. It’s not only the first Tier IV certified
facility in Bangladesh, but the first in the
whole South Asia region.
The Tier IV specification is designed to
deal with issues such as power blackouts,
and these may happen more often than in
comparable facilities, as Bangladesh’s grid is
being developed. This challenge means that
the facility’s diesel generators will be called for
more often and must be maintained carefully.
Advertorial: ZTE
Developing talent
Staff is another key concern, and the
government-backed data center will have a
key role in ensuring that talented personnel
are available, and the nation has a stream of
tech talent for the future. To get excellent data
center talent, the government offered “very
favorable conditions,” said Mr Alam.
The long term plan involves training: “In the
future, the government will vigorously develop
education, train more data center talents,
and do a good job in talent reserve,” said Mr
alam. And ZTE has been playing its part here,
training local staff to work in the data center
field, and providing on-site job training.
“This is what our country and region need,”
said Mr Alam. “I believe our cooperation with
ZTE is a long-term cooperative relationship,
which can bring common interests to both
sides.”
Tongbing Huo echoed that: “The team built
a state-of-the-art data center. At the same
time, they trained a number of experts for
the country. This project will become the
accelerator of the digital transformation of
Bangladesh.”
Award winning
The data center has a potential power
consumption 8500 kVA, and holds more
than 600 cabinets, providing 2000Tbyte of
cloud storage, in an area of 16,000 sq m. It
sits alongside two 920 sq m power buildings,
in a campus area of 28,000 sq m. The
facility includes a power subsystem, HVAC
subsystem,and a fire prevention system.
During the building, the team paid close
attention to the actual performance of the
building and its contents, measuring the
temperature in the cold aisles, and performing
adjustments to make sure that the airflow was
optimal and delivered high efficiency.
The project won this year’s DCD APAC
Award, for Data Center Construction Team
of the Year, with Alam commenting: “We are
honored to win this special glory.”
The Kaliakor data center may be the first
in Bangladesh to achieve Tier IV facility
accreditation, but it won’t be the last. ”I
believe that we can continue to build better
data centers to meet the growing needs of
our country,” said Mr Alam. “Of course, we
will continue to cooperate with ZTE when
conditions permit.”
Contact Details
Huo Tongbing, Data Centre Product
Marketing Director, ZTE Corporation
86-18623012306
huo.tongbing@zte.com.cn
Advertorial: Server Technology
Come visit
us at
DCD>London
exhibition
stand 19
Everything Needs
Power - Powering
the Next Revolution
Finding the right solutions to power the
systems running AI applications is critical
S
mart Cities around the globe are
relying on an ever-expanding
deployment of IoT devices
coupled with forthcoming 5G
wireless infrastructure and edge
computing to bring new levels
of management, coordination, service and
information to their citizenry. The systems
used to bring IoT, 5G, and edge computing
together will require machine-based AI to
provide insight, optimize performance, and
improve efficiency of the systems that make
Smart Cities truly smart.
But the race towards AI adoption across
industries and applications has really only
just begun. Corporations around the globe are
looking to deploy AI in their quest to reduce
headcounts, improve yield, and increase
profits.
Trucking companies use AI to optimize
routing. Hospitals employ AI to improve
diagnosis, read X-rays, and deliver an all-
around better quality of care. Retail and
online merchants utilize AI to recommend
products based on user preference and prior
purchasing trends. Banks implement AI to
detect fraudulent transactions and support
credit scoring. City and state governments
Intelligent PDUs from Server Technology,
such as the recently launched HDOT Cx with
Switched POPs (Per Outlet Power Sensing),
enable edge data center IT managers to
maximize uptime and at the same time
provide a rich data source for the latest AI,
focused on optimizing between minimizing
power consumption and reducing latency.
The industry leading and award-winning
HDOT Cx PDU brings limitless possibilities
to data centers with just one standard PDU
offering the ultimate in flexibility:
• 2 different types of outlets in 1 unit
(accommodates either a C14 or C20 plug)
• The most densely-packed outlets per form
factor
• Easy load balancing with alternating phase
technology
• Available in Smart, Switched and POPS
• Future ready for equipment updates
• Highly customizable in 4 simple steps
• Fast delivery - ships in 10 days
HDOT Cx is available with a variety of
feature sets and price points to suit even the
most demanding of customer applications.
For more information use the below
resources:
• Build Your Own Customized Data Center
PDU bit.ly/STCustomizedPDU
• Powering the Next Revolution - Artificial
Intelligence (AI) Industry Brief download
bit.ly/STAIIndustryBrief
use AI to help determine traffic patterns, Everything Needs Power
improve traffic flow, and deliver improved
services to their citizens. These are but a few
of the many possible areas where AI can and
will be utilized in the future. Core Smart
IoT Cities
Artificial Intelligence Runs on Intelligent
Power AI
For the time being, all AI applications run Edge Cloud
on silicon-based computational hardware,
usually accessed through a public cloud. And
that hardware all requires power, be it DC 5G
(like a battery) or AC (from the grid), to deliver
on the promises of better information and
better control, resulting in improved lifestyles,
improved citizen satisfaction, and greater
Your Power Strategy Experts
efficiency.
Finding the right solutions to power the
systems running AI applications that enable
Smart Cities, 5G networks, IoT, and edge
computing is critical. The vast majority of the
power that fuels them will need to be remotely
monitored and managed through automation
to ensure uptime while maintaining
efficiency and avoiding those expensive truck Contact Details
rolls. When it comes to powering a server Tel: (1) 775 284 2000
rack, data center managers are looking for Email: sales@servertech.com
density, flexibility and longevity. www.servertech.com

Building a
home for AI
Design for the future today, lest you
get left behind, Max Smolaks warns
D
ata center designers and
builders have to stay on top
of the latest developments
in server hardware: the
environments they create
require a massive upfront
investment and are expected to last at least 20
years, so they have to be ready for housing the
IT equipment of the future.
The latest trend in IT workloads that’s set to
impact the way data centers are constructed is
machine learning. The ideas fueling the boom
in artificial intelligence are not new - many
of them were proposed in the 1950s - and the
power of AI is undoubtedly being over-hyped,
but there are plenty of use cases where AI tech
in its current state is already bringing tangible
benefits.
For example, algorithms are much better
than people at securing corporate networks,
able to pick up on anomalies that humans and
their rules-based tools might miss. Algorithms
are also great at analyzing large chunks of
Max Smolaks
Contributor
boring text: lawyers use AI-based software
to scan through case files and contracts,
while universities use something similar to
establish whether a paper was written by the
student who submitted it, or a freelancer hired
online. And finally, there’s plenty of research
to suggest that optical image recognition
will match and even surpass the best human
doctors at spotting signs of disease on
radiology scans.
The variety of machine learning
applications is only going to increase,
introducing radically different demands on
storage performance, network bandwidth and
compute, more akin to something seen in the
world of supercomputers.
According to a recent research report by
Tractica, an analyst firm focused on emerging
technologies, AI hardware sales to cloud and
enterprise data centers will see compound
annual growth rate (CAGR) of almost 35
percent for the next six years, increasing from
around $6.2 billion in 2018 to $50 billion by
AI ready
Verne Global
2025. All this gear will have higher power
densities than traditional IT, and as a result,
higher cooling requirements.
Up until now, most machine learning
tasks have been performed on banks of
GPUs, plugged into as few motherboards as
possible. Every GPU has thousands of tiny
cores that need to be supplied with power,
and on average, a GPU will need more
power than a CPU. Nvidia, the world’s largest
supplier of graphics chips, has defined the
current state of the art in ML hardware with
DGX-2, a 10U box for algorithm training that
contains 16 Volta V100 GPUs along with two
Intel Xeon Platinum CPUs and 30TB of flash
storage. DGX-2 delivers up to two petaflops of
compute, and consumes a whopping 10kW
of power - more than an entire 42U rack of
traditional servers running at an average load.
"If you don't support
water-cooled processors,
you're excluding yourself
from the top end of the
market”
There are two parts to almost any machine
learning project: training and inference.
Inference is easy, you just take a fully
developed machine learning model and apply
it to whatever data you want to manipulate.
This process can run facial recognition on
a smartphone, for example. It’s the training
Issue 34 ∞ November 2019 49
 AI ready

that’s the intensive part - getting the model to
look at thousands of faces to learn what a nose
should look like.
“There are large amounts of compute
used and the training can take days, weeks,
sometimes even months, depending on
the size of the [neural] network, and the
diversity of the data feeding into it and the
complexity of the task that you’re trying to
train the network for,” Bob Fletcher, VP of
strategy at Verne Global, told DCD. Verne runs
a data center campus in Iceland which was
originally dedicated to industrial-scale HPC,
but has recently embraced AI workloads.
The company has also experimented with
blockchain, but who hasn’t done that?
According to Fletcher, AI training is
proving to be much more compute-intensive
than traditional HPC workloads that the
company is used to, like computational fluid
dynamics. “We have people like DeepL [a
machine translation service] in our data center
that are running tens of racks full of GPU-
powered servers; they are running them at
95 percent, the whole time. It sounds like a jet
engine and they have been running like that
for a couple of years now,” he said. What many
people don’t realize is the fact that machine
learning models used in production have
to be re-trained and updated all the time to
maintain accuracy - creating a sustained need
for compute, rather than a one-off spike.
When housing machine learning
equipment, it is absolutely necessary to be
religious about aisle containment and things
like blanking plates. “You go from 3-5kW to
around 10kW per rack for regular HPC, and
if you are going into AI workloads, which
are generally going to be around 15-35kW
air-cooled, then you have to be more careful
about air handling,” Fletcher explained.
“If you look at something like DGX-2, it
blows out 11-12kW of warm air. And it has a
very low temperature spread between input
and output, so the airflow is quite fast. If you
are not thoughtful about positioning, and you
have two of these at the same height, pointing
back to back and maybe 30 inches between
them, they are going to blow hot air at each
other like crazy, and you lose the airflow.
“So you have to place them at different
heights, or you have to use air deflectors, or
you can spread the aisles further apart, but
whatever you do, you’ve got to make sure that
all of the hot air that’s coming out of one of
these devices isn’t going to be banging into
hot air coming out of another.”
The advent of AI hardware in the data
center could at last mark the moment when
water cooling finally becomes necessary.
“One of the real changes from a data center
operator's perspective is direct water-cooled
chips to support some of these applications.
These products were once on roadmaps,
and they are now mainstream - the reason
they are available is because of some of these
workloads around AI,” said Paul Finch, CEO at
Kao Data.
Kao has just opened a wholesale colocation
campus near London, inspired by hyperscale
designs, with 8.8MW of power capacity
available in Phase 1, and 35MW in total. The
project was developed with an eye to the
future - and according to Finch, that future
includes lots of AI. He says that data halls
at Kao have been designed to support up to
70kW per rack.
“Many of these new processors, the real
Ferraris of chip technology, are now moving
"It sounds like a jet engine
and they have been
running like that for a
couple of years now"
towards being water-cooled. If your data
center is not able to support water-cooled
processors, you are actually going to be
excluding yourself from the top end of the
market,” Finch told DCD.
“In terms of the data center architecture,
we have higher floor-to-ceiling heights –
obviously water is far heavier than air, so it's
not just about the weight of IT, it’s going to be
about the weight of the water used to cool the
IT systems. All of that has to be factored into
the building structure and floor loading. We
see immersion cooling as a viable alternative,
it just comes with some different challenges.”
Floors are not the most glamorous data
center component, but even floors have to be
Sean Lie, Cerebras
considered when housing AI gear, since high
density racks are also heavier. Like many of
the recent data center projects, Kao houses
servers on a concrete slab. This makes it
suitable for hyperscale-style pre-populated
racks like those built by the fans of the Open
Compute Project, and enables the facility to
support heights of up to 58U.
And in terms of networking, Fletcher
said AI requires more expensive InfiniBand
connectivity between the servers - classic
Ethernet simply doesn’t have enough
bandwidth to support clusters with dozens
of GPUs. Cables between servers in a
cluster need to be kept as short as possible,
too. “Not only are you looking at cooling
constraints, you are looking at networking
and connectivity constraints to keep the
performance as high as possible,” he said.
In the next few years, hardware for
machine learning workloads will get a
lot more diverse, with all kinds of novel
AI accelerators battling it out in the
marketplace, all invariably introducing
their own acronyms to signify a new class
of devices: these include Graphcore’s
Intelligence Processing Units (IPUs), Intel’s
Nervana Neural Network Processors (NNPs)
and Xilinx’s Adaptive Compute Acceleration
Platforms (ACAPs). Google has its own
Tensor Processing Units (TPUs), but these
are only available in the cloud. American
startup Cerebras Systems recently unveiled
the Wafer Scale Engine (WSE) - a single
chip that measures 8.5 by 8.5 inches and
features 400,000 cores, all optimized for deep
learning.
It’s not entirely clear just how this monster
will fit into a standard rack arrangement, but
it serves as a great example of just how weird
AI hardware of the future might become.

50 DCD Magazine • datacenterdynamics.com

 > London Global Content Partner
18th Annual

5-6 November 2019 // Old Billingsgate

Show Preview See Inside

Audience demographics
and ecosystem

Full 2-day London

conference program

Key themes and

conference producer
highlights

Exhibition floor plan

Preview our 2020 global

events calendar

1800+ 70+ 85+ 20+ 32+ 16

Delegates Sponsors & Speakers Hours of Dedicated Hours of Expert In-depth panel
Exhibitors Networking Thought-leadership discussions &
fireside chats

Headline Sponsor Lead Sponsors Community Partner

www.datacenterdynamics.com Follow us: @dcdevents #dcdlondon

Issue 34 • October 2019 51
 London

London Ecosystem > Insights from

registration data
Registration data gathered from more than 1,400 pre-qualified buyers of data
center products and services in advance of the DCD>London conference 40% 25%
points to the fact that Edge, backup power generation, and thermal <25%
management solutions remain key technology areas heading into 2020. 75-100%
25-50%
44% 50-75% 14%
Major buyside groups
33% 23% 21%
11% We asked the enterprise audience
Systems what percentage of their data
As we approach the 18th annual
Integrators
5% center footprint remains
DCD>London we are excited to bring
Real Estate
Brokers
on-premise
together a stellar line-up of global
industry leaders to share their insights
and experiences across what promises 8%
to be a first-rate conference program. 54% AEC/ Architects
Consulting Advisory
Our content themes for this year’s Engineers
event have been specifically curated

57
to tackle the biggest challenges and
opportunities to shape the future of
the industry. Day One sees us kick
22%
Building
off the big discussion of the global Contractors
climate emergency and the rising
tide of environmentalism that will
see governments and consumers 15%
increasingly hold the industry to Cloud
account. And Day Two develops the Services of Enterprise operators have
edge story and how 5G is about to upgrade projects in the
challenge how we think about data Service
centers. Providers 70% pipeline

New for 2019 are our Challenge Panels, 15% Colo/MTDC

injecting more end-user discussions Telco/ISP

into the agenda, and providing
opportunity to dig into the issues really
keeping the industry awake at night. 68% 23%
We encourage your full engagement
UK Europe
Media/Broadcasting & Entertainment
Govt, Education &

40%
in every aspect of this event - join
the debate, ask questions, challenge 35% Enterprise
assumptions, find out what people are
Research

8%
30%
Financial Services & Insurance

doing that’s at the leading edge, and

Manufacturing

how this might benefit your work and 25%

Rest of
& Industry

world
Energy &

organization.
Utilities

20%
Professional

Healthcare
Construction
& Real Estate

& Pharma
Services

Enjoy the next two days and I hope to 15%

Retail

meet with you at some point. 10% We have service providers

attending from 36 different
Rebecca Davison 5%
countries making this our most
Global Conference Director 0%
5.92% 5.92% 9.87% 3.29% 33.55% 22.70% 4.61% 9.21% 4.28% international event

>Speakers
Adam Pool, Xtralis Brenden Rawle, Equinix Dr Mike Hazas, Lancaster University
Adrien Viaud, Kingston Technology Brian Conroy, Moy Materials Dr Rabih Bashroush, Uptime Institute
Alaa Salama, Google Cara Mascini, EdgeInfra Ed Ansett, i3 Solutions Group
Alex Sharp, Iron Mountain Chester Reid, CyrusOne Emma Fryer, techUK
Ali Moinuddin, Uptime Institute Chris Brown, Uptime Institute Gary Cook, Greenpeace
Amy Daniell, NTT Ltd Chris Downing, Siemens Garry Connolly, Host in Ireland
Andrew Wettern, 1Energy Group Christian Belady, Microsoft George Rockett, DCD
Andy Lawrence, Uptime Insitute Ciarán Forde, Eaton Heather Dooley, Google
Anna Kondratenko, Systemair Dave Johnson, Schneider Electric Ian Lovatt, FNT GmbH
Anthony Robinson, Corning David Hall, Equinix Jack Ke, China Mobile
Astrid Wynne, Techbuyer Dean Nelson, Infrastructure Masons Jack Pouchet, Natron Energy
Atle Haga, Statkraft Deborah Andrews, London South Bank University Jason Simpson, Liberty Global
Avner Papouchado, Server Farm Realty Diarmuid O'Sullivan, PM Group Jeff Omelchuck, Infrastructure Masons
Bill Kleyman, Switch Dr Mark Coughlin, EnerSys Jim Smith, xScale at Equinix

52 DCD Magazine • datacenterdynamics.com

> Top 15 most viewed w
Theme | The Business of Theme | 5G, AI & The
speaker profiles online Data Centers Connected Edge
Access to capital, site selection, energy The digital infrastructure landscape is
1 Noelle Walsh
Microsoft
procurement, and connectivity are just some
of the factors that developers and operators
evolving fast with 5G and IoT technologies
set to ignite exponential data growth and
must consider when planning a new facility. a new edge data center industry. This
This year, we look at how the European year’s conference brings together the
2 Gary Cook
GreenPeace
data center market is adapting to power foremost thinkers on the subject to help
shortages, regulation, economic and political delegates better understand how data
uncertainty. center deployment and capacity models
will radically change.
3 Jason Simpson
Liberty Global 2019 sees the launch of our new Challenge
Panels. Mark Trevor from Cushman & Debating how the data center industry
Wakefield will moderate the “Financially will deal with exponential data growth as
4 Mario Müller
Volkswagen
Challenged Panel”, joined by experts
including John Wilson from Sumitomo
Moore’s Law slows down will be Christian
Belady from Microsoft, Kurtis Bowman of
Mitsui Banking Corporation, Chester GenZ Foundation – DellEMC, Dean Nelson
Reid of CyrusOne, Avner Papouchado of iMasons and Bill Kleyman from Swtich.
5 Rhonda Ascierto
Uptime Institute
of ServerFarm Realty and Romain Le Are we facing a doomsday scenario or a
Mélinaidre from InfraVia Capital to debate new era of creativity?
how the investment model for data centers
is evolving. Rhonda Ascierto from the Uptime Institute
6 Jim Smith
Equinix will share the Institute’s latest research
In another Challenge Panel Kevin Kent of on how artificial intelligence (AI) will
OSU Wexner Medical Center, Sean Moloney enable very smart data centers and power
7 Alaa Salama
Google
of GWLE, and Paul Jennings of Imperial
College London will discuss how the
operational decisions.

modern data center manager continues to Mark Thiele of Ericsson will kick off the
John Wilson span the chasm between IT and facilities, 5G discussion with why 5G is about
8 Sumitomo Mitsui
Banking Corporation
and how the role of the data center manager to challenge how we think about data
is changing in response to Hybrid IT. centers. Mark will examine how 5G will
release high connectivity and change the
9 Dave Johnson
Schneider Electric Producer's Highlight:
technological landscape and why the ‘data
center’ in all its forms - from core to edge,
to micro, to pica - will be at the center.
Major Panel: Fiber, energy,
10wv Christian Belady
Microsoft
politics and demand – What
will the European data center Producer's Highlight:
map look like by 2025? Plenary Panel: The race to build
11 Mark Thiele
Edge Gravity by Ericsson How have cities like Frankfurt, London, the mobile edge - What will it
Amsterdam and Paris (FLAPs) managed
look like? Who will own it? Who
the epic growth rates of the data
will pay for it?
12 Dean Nelson
Infrastructure Masons
center sector and how will government
regulation and public opinion shape 5G will require edge compute, no doubt.
their development going forward? Does But what shape will it take, literally? Will
this convert into ISO container micro data
13 Cara Mascini
EdgeInfra
the Nordic promise of carbon-free
compute offer the answer to absorbing centers? Who will own it? The incumbent
continued growth or is this a pipe dream? telco, the mobile network operators or
Join techUK, the Dutch Data Center will it be the major cloud players, a new
14 Heather Dooley
Google
Association, Basefarm, Statkraft and Host breed of neutral colocation or will it be
a municipal play? So many questions, so
in Ireland as they share the challenges
little time… we ask the Uptime Institute,
they face and debate how the European
wv Ericsson, EdgeInfra, SmartEdge DC,
15 Mike Bennett
Cyxtera
data center map is likely to take shape
over the coming years.
Equinix, and Arcadis what they think.

John Booth, Carbon3IT
John Laban, Open Compute Foundation
John Wilson, Sumitomo Mitsui Banking Corporation
Kevin Brown, Schneider Electric
Kevin Kent, OSU Wexner Medical Center
Kurtis Bowman, Gen-Z Consortium & Dell EMC
Lee Kirby, Salute Mission Critical
Lex Coors, Interxion
Maikel Bouricius, Asperitas
Marc Garner, Schneider Electric
Mario Müller, Volkswagen
Mark Harrop, Arcadis
Mark Howell, Ford
Mark Thiele, EDGE GRAVITY by Ericsson
Mark Trevor, Cushman & Wakefield
Mike Bennett, Cyxtera
Mike Hughes, Schneider Electric
Nick Ewing, EfficiencyIT
Noelle Walsh, Microsoft
Patrik Öhlund, Node Pole
Peter Hannaford, Portman Partners
Paul Jennings, Imperial College London
Peter Hewkin, SmartEdge DC
Petter Tømmeraas, Basefarm
René Kristensen, DEIF
Rhonda Ascierto, Uptime Insitute
Rob Cooper, CS Technology
Robert Thorogood, HDR | Hurley Palmer Flatt
Rod McAllister, Penguin Computing
Romain Le Mélinaidre, InfraVia Capital
Seán Moloney, Great-West Lifeco
Simon Allen, Infrastructure Masons
Simon Binley, Wellcome Sanger Institute
Sophia Flucker, Operational Intelligence
Stephen Lorimer, Keysource
Stewart Grierson, Upnorth Group
Stijn Grove, Dutch Data Center Association
Susanne Baker, techUK
Susanna Kass, UN Sustainable Development Group
Tim Chambers, coolDC
Tobias Spilker, Siemens
Tor Kristian Gyland, Green Mountain

Issue 34 • November 2019 53

 London

Day 1 | Tuesday 5 November

Content themes:
Energy Smart
07:30 Registration open Infrastructure
09:10 Opening remarks: George Rockett, DCD Modernization &
Lifecycle Management
09:20 Hall 1 - Plenary Keynote: A balancing act – delivering cloud at the intersection of sustainability, community
and innovation Noelle Walsh, Microsoft Planning for Hybrid IT

09:40 Hall 1 - Plenary Panel: How should the data center industry now respond to the global climate emergency? 5G, AI & The
Andy Lawrence, Uptime Institute | Noelle Walsh, Microsoft | Dr. Mike Hazas, Lancaster University Connected Edge
Susanna Kass, United Nations Sustainable Development Group | Gary Cook, Greenpeace | Dave Johnson, The Business of
Schneider Electric | Emma Fryer, techUK Data Centers
Moderator: George Rockett, DCD Building at Scale
& Speed
10:40 Coffee Break | Expo | Innovation Stage Presentations | Speed Networking | VIP Brunch Briefings

Hall 1 Hall 2 Hall 3

11:40 Next Generation DCIM: Re-inventing a Top trends impacting data centers today Sustainability: Just a word or something
platform for the brave, new hybrid world Anthony Robinson, Corning more?
Kevin Brown, Schneider Electric Stephen Lorimer, Keysource

12:10 Mastering complexity – A case study on fire Power network transformation – designing China Mobile Case Study: The challenge of
safety and consistency in planning and data and operating the power network as a global expansion
center operations truly connected and intelligent ‘System’ to Jack Ke, China Mobile
Chris Downing, Siemens prepare for the energy transition ahead
Tobias Spilker, Siemens Ciarán Forde, Eaton

12:40 Vertically Challenged: Vertically Challenged: Financially Challenged:

How is the role of the consulting engineer
changing in the era of hyperscale?
Ed Ansett, i3 Solutions
Robert Thorogood, HDR | Hurley Palmer Flatt
Sophia Flucker, Operational Intelligence
Amy Daniell, NTT Ltd
Moderator: Peter Hannaford, Portman
Partners
What conversation should tenants be having How is the investment model for data
with colos about technology adoption? centers evolving?
Mike Bennett, Cyxtera John Wilson, Sumitomo Mitsui Banking
Lex Coors, Interxion Corporation
Rob Cooper, CS Technology Chester Reid, CyrusOne
Ian Lovatt, FNT Avner Papouchado, ServerFarm Realty
Moderator: Dan Loosemore DCD Romain Le Mélinaidre, InfraVia Capital
Moderator: Mark Trevor, Cushman &
Wakefield

13:20 Lunch | Expo | Networking | Innovation Stage Presentations | VIP Lunch Briefings

15:00 Major Panel: What are the barriers to building
at scale and speed and how can the industry
overcome them?
Mike Hughes, Schneider Electric
Alex Sharp, Iron Mountain
Jim Smith, xScale at Equinix
Diarmuid O’Sullivan, PM Group
Moderator: George Rockett, DCD
Major Panel: How will the data center
industry deal with exponential data growth
as Moore’s Law slows down?
Kurtis Bowman, GenZ Consortium & Dell
EMC
Dean Nelson, iMasons
Christian Belady, Microsoft
Bill Kleyman, Switch
Moderator: Sebastian Moss, DCD
Major Panel: Fiber, energy, politics and
demand – What will the European data
center map look like by 2025?
Stijn Grove, Dutch Data Center Association
Petter Tømmeraas, Basefarm
Atle Haga, Statkraft
Garry Connolly, Host in Ireland
Moderator: Emma Fryer, techUK

16:00 Why now is the time for immersion cooling
to go from niche to scale
Maikel Bouricius, Asperitas
Make the global CO2 challenge an
opportunity for your business
Patrik Öhlund, Node Pole
Rightsizing and resilience through
modularity – the next generation of
modular UPS to solve the capacity
challenge
John Booth, Carbon3IT

16:30 Case study: Data center lifecycle
management for the human genome project
Simon Binley, Wellcome Trust
Nick Ewing, EfficiencyIT
Peter Judge, DCD
Panel: The journey towards a circular A Fireside Chat with Google: Are we
economy for the data center industry running out of staff and out of options?
Susanne Baker, techUK Heather Dooley, Google
Astrid Wynn, Techbuyer Rhonda Ascierto, Uptime Institute
Alaa Salama, Google
Rod McAllister, Penguin Computing
Moderator: Deborah Andrews, LSBU

17:15 Isle of Harris Gin Drinks Reception & Networking on Expo Floor - sponsored by in partnership with

17:30 >Awards | Global Finalists Announcement

19:00 Close of Day One

54 DCD Magazine • datacenterdynamics.com

Day 2 | Wednesday 6 November
07:30 Registration open

09:10 Opening remarks: George Rockett, DCD

09:20 Hall 1 - Plenary Keynote: Why is 5G about to challenge how we think about data centers?
Mark Thiele, EDGE GRAVITY by Ericsson

09:40 Hall 1 - Plenary Panel: The race to build the mobile edge – What will it look like? Who will own it? Who will pay for it?
Mark Thiele, EDGE GRAVITY by Ericsson | Cara Mascini, EdgeInfra | Peter Hewkin, SmartEdge DC | Brenden Rawle, Equinix | Mark Harrop, Arcadis
Moderator: Rhonda Ascierto, Uptime Institute

10:40 Coffee Break | Expo | Innovation Stage Presentations | VIP Brunch Briefings

Hall 1 Hall 2 Hall 3

11:30 OREO to ESCO: Innovation in legacy data
center energy strategy – Virgin Media case
study
Jason Simpson, Liberty Global
Stewart Grierson, Upnorth Group
Critical IT facilities at the Edge Resiliency in the age of Cloud: 99.99 red
Mark Howell, Ford flags?
Andy Lawrence, Uptime Institute

12:10 Very smart data centers: How artificial Energy storage technology trends and What upgrades and modifications have
intelligence will power operational implications for mission critical infrastructure the biggest positive impact on your data
decisions Jack Pouchet, Natron Energy center’s energy efficiency?
Rhonda Ascierto, Uptime Institute John Booth, Carbon3IT

12:40 Technologically Challenged: Technologically Challenged: Vertically Challenged:

What role will data centers play in “the grid
of the future”?
David Hall, Equinix
Andrew Wettern, 1Energy Group
Bill Mazzetti, Rosendin Electric
Mohan Gandhi, Sustainable Digital
Infrastructure Alliance (SDIA)
Moderator: Peter Judge, DCD
What’s holding the industry back from mass-
adoption of liquid cooling in the data center as
talk turns to 100kW per rack?
John Laban, Open Compute
Dr Rabih Bashroush, Uptime Institute
Tim Chambers, CoolDC
Chris Brown, Uptime Institute
Moderator: Kisandka Moses, DCD
How is the role of the data center
manager changing in response to Hybrid
IT?
Kevin Kent, OSU Wexner Medical Center
Sean Moloney, GWLE
Paul Jennings, Imperial College London
Moderator: Ali Moinuddin, Uptime Institute

13:20 Lunch | Expo | Networking | Innovation Stage Presentations

Hall 1
15:00 iMasons Global Member Summit
1. End User Summit Read-Outs
2. What industry challenges should the
iMasons membership be tackling in 2020?
Interactive roundtable discussion session
Jeff Omelchuck, iMasons, Simon Allen,
iMasons, Patrik Öhlund, Node Pole
Hall 2
Workshops | Day 1
A deep-dive into the newly released data
center sector energy routemap 12:10 - 13:00 Exec Workshop: An academic
perspective on quantifying energy usage
Emma Fryer, techUK
per IT service.
Dr. Mike Hazas, Lancaster University
Cityside West Room
16:00 - 17:00 Exec Workshop: Why has
the #clickingclean report become so
important to the data center industry?
Gary Cook, Greenpeace
Cityside East Room

17:00 End of conference

VIP Briefings | Day 1 VIP Briefings | Day 2

Workshops | Day 2
12:40 - 13:20 Fireside Chat: Energy Smart
Private Brunch Private Brunch Private Brunch
10:50-11:40am 10:50-11:40am 10:50-11:40am meets Circular Economy – step by step
Cityside East Room Cityside West Room Cityside West Room towards the next-gen sustainable data
center
Susanna Kass, United Nations Sustainable
Private Lunch Private Lunch VIP Briefings by invitation only Development Group | Alaa Salama, Google
1:30-2:40pm 1:30-2:40pm
Cityside East Room Cityside West Room
Cityside West Room

Issue 34 • November 2019 55

 London

Sponsors, Exhibitors & Partners

> Exhibitors
Armstrong Fluid 48
Theme | Planning for Asperitas 65
Bouygues E&S 23
Hybrid IT CBRE 43
China Mobile 46
Effectively managing capacity and Corning 2
operations between on-premise, Cummins 33
colocation and the cloud is an increasingly Datacenter People 30
DCPRO 68
complex task that requires new types
E+I Engineering Ltd 58
of risk assessment and skill sets. This East Penn 36
year’s conference program provides new Eaton 60
perspectives on the intersect between EkkoSense 37
facilities and IT operations in a hybrid EnerSys 49
world. European Data Center Associations Pavilion
Danish Data Center Industry 4
Data Centers By Sweden 18
Producer's Highlight: Dutch Data Center Association 17
Innovation Norway 5

Vertically Challenged: What Finning CAT 45

conversation should tenants FNT GmbH 29
Future Facilities 1
be having with colos about Green Revolution Cooling 3
technology adoption? Hewlett Packard Enterprise 66
InfraNorth
Jacobs
32
7
>Exhibitors
When an organization chooses to Janitza Electronics GmbH 47
colocate their critical data center Kingston Technology 42 DATA CENTRE SOLUTIONS

infrastructure in a 3rd party facility the Legrand 19

Moy Materials 50
lines of who is responsible for what DATA CENTERS
BY SWEDEN
Nlyte Software 25
aspects of the physical infrastructure and Operational Intelligence 8
management systems can get quickly PermAlert 26
get blurred. This panel brings together PM Group 35
industry leaders including Mike Bennett Schneider Electric 55
Siemens 64
of Cyxtera, Lex Coors of Interxion, Rob
Socomec 44
Cooper of CS Technology and Ian Lovatt Starline 28
of FNT, as they explore the challenges Structuretone 54
that both the end-user and colocation Submer 27
operator face in order to get the most Sunbird Software 39
Systemair 31
out of their infrastructure investment and
Tate Europe 52 PermAlert
deliver a win-win approach to energy
TM

Tileflow 53 Division of Perma-Pipe, Inc.

Liquid Leak Detection & Location Systems

efficiency, availability, and capacity Uptime Institute 38

management. Winthrop 56 ®

Xtralis 51

Innovation Stage | Day 1 - Tuesday 5 November 2019

10:50am | A low carbon footprint 11:10am | How can Kingston

>Knowledge Partners
for your data center: Using direct support your technology needs for
and indirect free-cooling units performance in data centers?
Anna Kondratenko, Systemair Adrien Viaud, Kingston Technology

1:25pm | Green roofing the data

11:30 | Batteries - the most critical center; leading in mission critical
component in an evolving data environmental sustainability
center Mark Coughlin, EnerSys Brian Conroy, Moy Materials

2:05pm | Accelerate your data

center capacity planning with the 2:25pm | Optimized power train
Digital Twin for highly resilient data centers
Mark Fenton & Adam Smith Marc Garner, Schneider Electric
Future Facilities

Innovation Stage | Day 2 - Wednesday 6 November 2019

PRO

56 DCD Magazine • datacenterdynamics.com

 Theme | Energy Smart
Infrastructure
From future battery technology to
microgrids and the latest cooling tech
this year’s conference program will
provide data center operators with
expert opinion and guidance on how
to design and operate highly-efficient
and highly-reliable mission critical
environments that support the next
generation of high density IT.
Noelle Walsh from Microsoft will
open the conference with a plenary
keynote on delivering cloud at the
intersection of sustainability, community
and innovation. Noelle will share her
perspective on this unique intersection,
along with incremental and profound
changes she believes the industry will
need to make to deliver on the world’s
digital needs, responsibly.
John Laban of Open Compute, Dr Rabih
Bashroush of Uptime Institute and Tim
Chambers from CoolDC will debate
what’s holding the industry back from
mass-adoption of liquid cooling in the
data center as talk turns to 100kW per
rack.
Participants in the CEDaCI Project, led
by Deborah Andrews of LSBU and joined
by Alaa Salama from Google, will discuss
the journey towards a circular economy
for the data center industry. Hear the
approaches they are taking to minimize
waste, design for repairability, and keep
products and materials in circulation for
as long as possible to ensure sustainable
data center operation.
Producer's Highlight:
Plenary Panel: How should
the data center industry now
respond to the global climate
emergency?
DCD’s George Rockett will be joined
by global experts from across the
digital infrastructure ecosystem to
tackle the increasingly critical issue of
how the data center industry is going
to sustain growth rates that support
exponential data usage in the face of
a global climate emergency. Will this
ever be a sustainable industry? Find
out where the industry goes next from
Microsoft, Greenpeace, Schneider
Electric, Uptime Institute, Lancaster
University, techUK and the United
Nations Sustainable Development
Group as the ‘heat’ turns up.
Check out our new website for the most up to date event details datacenterdynamics.com
Theme | Modernization & Theme | Building at Scale
Lifecycle Management & Speed
Retrofitting existing data centers to It’s crucial for data center developers and
manage modern workloads and maximise operators to reduce time to construction
capex might be less glamorous than especially within the colo and hyperscale
greenfield data center design and build, sectors. This content theme explores
but 60% of attendees say these are priority new greenfield data center design,
projects. This year’s conference brings construction techniques, framework
together operators, technology vendors
agreements and supply chain dynamics.
and design consultants to share best
practice for data center upgrades.
Producer's Highlight:
The Wellcome Genome Campus will be
the focus of a fireside chat with the data
Panel: What are the barriers to
center manager and design consultant.
Simon Binley from the Wellcome Trust building at scale and speed and
and Nick Ewing of EfficiencyIT will look how can the industry overcome
at the journey through the lifecycles of 4 them?
data halls over 15 years – the upgrades,
the challenges and the present-day Not a week goes by without a
deployments for high density HPC new mega-data center build being
environments that are AI-ready. announced somewhere in the world,
often with power consumption cited
Kevin Brown from Schneider Electric in the hundreds of MWs and technical
will address how DCIM failed to live areas the size of many premier league
up to early market hype, and how next football grounds. This expert panel brings
generation DCIM tools may well deliver a together Mike Hughes of Schneider
re-invented platform for the brave, new Electric, Alex Sharp of Iron Mountain,
hybrid world. Jim Smith of xScale at Equinix, and
JDiarmuid O'Sullivan of PM Group
to share their insights on the latest
construction techniques and the many
Producer's Highlight: supply chain challenges faced when
building at scale.
OREO to ESCO: Innovation
in legacy data center energy
strategy – Virgin Media case
study
Virgin Media is on the final phase of a
4-year award winning energy efficiency
project that has seen nearly 200 ’sub-
scale’ legacy telco sites across the UK
receive free air cooling upgrades and
improved monitoring and controls
leading to a 20% improvement in PuEs -
Project OREO. But what's next?
Jason Simpson, leading energy strategy
at Virgin Media’s parent company, Liberty
Global, and Stewart Grierson, CEO of
VM’s key strategic partner in energy
infrastructure, Upnorth Group, will have
a robust, and at times uncomfortable,
debate about unrealistic expectations,
untested technologies, unclear policy
and regulation, aggressive sustainability
commitments and the art of the
possible – audience participation will be
expected!
45%
of the audience are exploring
backup generation, UPS, water
and air cooling solutions
Issue 34 • November 2019 57
 New York
>LATAM

>2020 Event Calendar Digital Week

17-19 March

>New York
31 March - 1 April

>Datacenter-nomics
31 March
New York Energy Smart
>Jakarta
15 April

>Energy Smart
 Stockholm
27-28 April

>Madrid
27 May

>Shanghai
11 June

>San Francisco
31 March - 1 April 2020 27 - 28 April 2020 15-16 June

New York Marriott Marquis The Brewery, Stockholm >Bangalore

More than half of last year’s enterprise
audience was made up of Financial
Services organizations including buyers
from every Fortune 100 bank in NYC with
more than $8.7 trillion in assets. Colo
companies that attended last year’s event
operate 14,700,000 sq.ft of data center
space in the New York Tri-state area alone.
Construction companies that attended last
year’s event had in excess of $13.5 billion
worth of data center revenues in 2018.
And consulting engineers and architects
that attended recorded more than $382
million in fees for data center projects.
DCD>New York is the largest marketplace
of its kind in the United States.
16 July
DCD returns to Stockholm, probably the
most energy aware capital in the world, >Sydney
for the third edition of DCD>Energy
13 August
Smart. This international conference with
600+ attendees, focused on helping
>Santiago
the data center and cloud infrastructure
9 September
industry meet capacity demands
sustainably, connects the builders of digital
infrastructure with the builders of energy >Singapore
infrastructure to have an energy smart 15-16 September
conversation.
>Mexico
The 2020 agenda will explore how circular 30 September
economy thinking can be applied to the
data center industry, how the industry can >Virginia
speed up adoption of the UN sustainability
5-6 October
goals, and the latest energy efficient
technologies that are ready for deployment.
>Mumbai
16 October

>Dallas
New events for 2020 26 October

>São Paulo
NEW NEW 3-4 November
31 March 5-6 Oct
17-19 Mar >London
LATAM Digital >Datacenter-nomics 10-11 November
Week Colocated with >Virginia
DCD>New York >Beijing
3 December

Check out our new website for the most up to date event details >Canada Digital Week
datacenterdynamics.com 8-10 December

58 DCD Magazine • datacenterdynamics.com


Mining genomes
As genomic data emerges at ever-increasing
rates, can the data center at Wellcome
Sanger Institute keep up?
Y
ou could say the Wellcome
Genome Campus, near
Cambridge UK, is the CERN
of bio-sciences. It leads the
world’s efforts to apply genomic
research to benefit human
health, while CERN leads the world’s particle
physics research from its base in Geneva.
The Campus has built up around the
Wellcome Sanger Institute, set up in 1992,
and now includes a rapidly-expanding
cluster of other bio-science and bio-
informatics organizations (see box:
Wellcome to the world of genomics).
Everything on the campus seems new,
and DCD’s visit begins in the Ogilvie
Building (opened 2016) with a graphic
illustration of progress in DNA sequencing.
Some of the Human Genome Project’s early
equipment is on display, which took 13
years to sequence the first reference human
genome. Next to it are subsequent systems,
which do it much quicker (see box: Faster
genomes).
The lights dim, and a video plays on one
wall, illustrating the rapid progress in the
field. Now, we are told fresh genomes are
sequenced every day by banks of dozens of
machines. There’s a glut of data: sequences
for humans, for cancer cells, for parasites,
and for bacteria.
These genomes hold the keys to new
medicine. So far, we have failed to eliminate
malaria, which kills half a million people
a year in Africa. One problem is the
parasite plasmodium falciparum develops
resistance to drugs. Checking the genome
of malaria samples lets health bodies track
that resistance and keep one step ahead,
targeting new drugs where they are needed.
Meanwhile, hospitals live in fear of the
“superbug” MRSA, which is resistant to
standard antibiotics. In fact, analysis of
MRSA’s genome has shown that several
strains can be dealt with by specific
antibiotics. Sequencing and analysis could
save the lives of patients in hospitals struck
by the superbug.
And a full genome sequence could
“We are the single largest
user of sequencing
consumables in the
world"
potentially improve the healthcare an
individual receives. Our genetic fingerprint
influences how likely we are to suffer
specific diseases or conditions. Beyond that,
our genome determines which treatments
will be effective, and which will have side
effects if we succumb to illness.
As the video ends, the wall it’s projected
on slides apart. Through a two-way mirror,
in a bright gleaming laboratory, we see
Genomic treasures
Peter Judge
Global Editor
a bank of NovaSeq 6000s - the latest
sequencing machines from Illumina.
Each is fed a continuous stream of
genetic samples, and each takes just one day
to repeat the task which took the Human
Genome Project 13 years. Between them, they
are pumping out petabytes of genomic data.
Everything that happens at the Wellcome
Sanger campus flows from this firehose.
Keeping up with the data deluge is the
mission of the Wellcome Sanger Institute’s
data center manager, Simon Binley. It’s his
job to share it and make it useful to scientists
on the campus and around the world.
“We are the single largest user of
sequencing consumables in the world,” says
Binley. He’s proud of his facility, but has no
illusions about who is the star of the show:
“Our priority is to make sure the science gets
the grunt it needs to perform world-class
science.”
That goal places unique demands on
the Wellcome Trust’s data center, he says:
“The original sample, that piece of human
tissue, or organic matter, that will be
lost. Eventually, it will decay. So the only
reference we've got to that is the data stored
here. If it is referenced in a paper, we have to
retain the data forever.”
Issue 34 • November 2019 59
 Genomic treasures

To see the data center, Binley walks us to

the Morgan Building, an older space which
opened in 2005, On the lowest floor, there
are four 250 sq m (2,700 sq ft) data halls,
color-coded red, yellow, green, and blue,
which hold some 35,000 computer cores in
400 racks.
The life-cycle of this data center is far
more interesting than even these raw figures
suggest. When it opened in 2005, the Sanger
Institute planned to adapt to technology
changes, by reserving a “fallow” hall. Three
rooms were gradually populated, and the
blue hall left empty, waiting for a new
generation of equipment.
The fallow hall remained empty for a long
while, while the first three halls got some
updates over the data center’s first 14 years.
As a result, the three “legacy” halls have
"We've got to keep the
some quite recent equipment, and they’ve sequencers running. If
recently implemented comprehensive
cloud-based data center infrastructure they dry up, our reason to racks would consume some 750kW, as much
management (DCIM) using Schneider’s as an entire legacy halls. This still leaves
EcoStruxure.
be here goes away" more than 1MW to take up, he says: “We’re
The infrastructure management extends planning to use this for the next 15 years,
beyond the data center to communications and technology is not going to stand still.”
rooms through the campus - and also to the A quick mental sum suggests that he could
crucial sequencers in the Ogilvie building. as the other three halls combined. pretty much fill the room with 30kW racks.
While we were there, Binley pointed out “We needed something that we could The increase in energy demands from
individual UPS systems sitting by each one, start wrapping much heavier workloads the blue hall might be a concern, as the
all under the DCIM control. into,” Binley explains. “We needed a number campus has rather average-quality power.
Binley shows us one of the legacy halls: of racks where up to 30kW could be It’s stuck at the end of the grid, Binley says,
there is conventional air conditioning with accommodated.” and suffers occasional brownouts and
no aisle containment, and the air around is This density demands liquid-cooled outages.
quite chilly as it’s drawn upwards through back-of-rack chillers, says Binley: “These One current proposal to deal with
the racks. coolers can handle 35kW and burst to 40kW,” unreliable grids is to use a microgrid, where
The racks each have about 10kW of load, Binley says. “The temperature is 34 degrees some power is generated locally to increase
and the room totals 750kW, with a PUE (93°F) at the back of the rack, and goes up reliability. The Sanger Institute is on trend
(power usage effectiveness) which Binley is to 50 or 60 (122-140°F) six inches later. The here: for more than seven years it’s had a
bringing down from around 1.8 towards 1.4, back-of-rack coolers take it back to half a combined cooling, heat and power (CCHP)
partly by raising the temperature from 19°C degree below the inlet temperature.” system on the Morgan building’s roof. It
(66°F) to 21°C (70°F). This method allows Binley to tailor the uses natural gas to deliver 2MW of electrical
In the blue hall we feel the difference. cooling in different parts of the room. There power, while capturing waste heat for use in
Engineers are installing racks, but are 25 racks with back-of-rack cooling, the buildings, while also providing energy
equipment there is already in use, and the and half of them were occupied when DCD for cooling systems, so it can deliver about
hall is noticeably warmer than the others. visited. There are also air-cooled rows, 1MW of cooling.
Here more power is available, and more is which now have state-of-the-art aisle This makes good economic sense, since
going where it is needed. This hall has a containment. gas costs around half the price of electricity
potential capacity of 2.2MW, nearly as much Fully populated, the 25 water-cooled per kWh, and it could conceivably provide
primary power, with the grid switched for
backup and office use.
Faster genomes But that’s not how the Wellcome campus
works, says Binley: “We are not the only
IT professionals are used to rapid increases in performance, as solid state circuits improve. essential service on campus. We've got to
Genomic technology has been increasing at even faster rates. The latest NovaSeq 6000 from keep the sequencers running. If they dry
Illumina can deliver 6Tb or 20Tb reads of an entire genome, in less than two days. up, our reason to be here goes away.” So
Overall, the costs fall faster. The last generation of sequencers cost £700,000 ($900,000 the CCHP doesn’t support the data center
for a unit that produced 2Tb a day. The new ones cost £300,000 ($390,000) to half a million directly. It puts 2MW into the campus ring,
and generate 4Tb per day. and increases reliability overall.
While earlier sequencing methods required manual effort and collation, newer systems Another data center efficiency trend
automatically generate results in a form compatible with a library information management fell foul of economics, however. Some
system (LIMS). data centers that are keen to reduce their
The end result is huge quantities of data, which are delivered ready for use, which are environmental footprint are making
vitally important to science. It is up to the Wellcome Sanger Institute’s data center to keep up their waste heat available for use by their
with that flow of data, by using the best affordable IT. neighbors.

60 DCD Magazine • datacenterdynamics.com

 It’s normally thought that liquid cooling
makes this easier, because it delivers waste
heat in a more concentrated and easily
usable form. But it’s not that simple, as the
Sanger Institute found.
The warm air from the legacy halls were
relatively easy to plumb into the CCHP’s heat
reclamation system, but in the blue hall, the
use of liquid cooling means the waste air is
no longer warm enough to consider. “The
heat recycling in the legacy hall is legacy
heat recycling,” he explains. “It supplements
the building‘s heating in the winter. In the
new hall, the air is cooler.”
Meanwhile, it turned out recycling the
heat from the water wasn’t viable, because of
the level of investment required: “This is a £9
million ($11m) room, directly supporting the
science,” he explained. Adding heat reuse
would have added £4m ($4.8m) to the cost,
and saved a much smaller figure.
“That £4m could pay for a number of PhD
students,” he says. “They could work on a
program like eradicating malaria.
“For us, the important thing is to enable
world-class science,” he says. “The Wellcome
Trust is a charity. It gives its results away
and doesn’t get government grants. We
have to be as careful with our money as a
commercial organization.”
All data center managers look to the
future, as the demands on their systems
develop and the technology evolves. For
Binley, there are multiple developments to
follow, as the technology of sequencing
develops in parallel with the IT.
Once again, the sequencers get priority,
and the Institute has to balance the
available resources between doing more
gene sequencing, and ensuring that there’s
enough IT resource to handle the output.
This could have interesting results in
future. As IT evolves into ever more compact
forms, Binley thinks he may be able to
continue to grow the power of the data
center, while shrinking its size.
It could soon be possible to take all three
legacy halls and provide a larger IT resource
in a single hall. In five years, he can imagine
fitting all the resources the campus needs
into two halls.
When that happens, he says, it might be
possible to switch two halls for lab space,
allowing for more sequencers right next to
the data halls.
Why would he be considering this? Those
machines would have the benefit of close
network links to the data center, and they’d
be directly on its protected power supply, no
longer needing remote dedicated UPS.
Once again, it comes back to the primacy
of the research. This could be the largest
biosciences data center in Europe - but it’s
still just there to serve the scientists.
Hear Simon
Binley talk at
DCD>London on
5 November at
16:30
in Hall 1
Wellcome to the world of
genomics
The Wellcome Sanger Institute dates back
to 1992, when it was launched to take
part in the Human Genome Project, the
world’s largest biological science project.
The same year, the European Molecular
Biology Laboratory (EMBL) created the
European Bioinformatics Institute (EMBL-
EBI) - a resource to store and share the
growing number of DNA sequences
emerging from genetic research - and
decided to locate it on the same campus
at Hinxton, near Cambridge.
Few people involved at that time could
have predicted the speed with which the
field would take off. The Human Genome
Project took thirteen years, and completed
its sequence of a single reference human
genome in 2003. Now full genomes can
be sequenced in less than a day, and it is
possible to look for DNA variations within
a single individual.
EMBL-EBI began in 1972 as a paper-
based library for the few DNA sequences
then known, says Steve Newhouse, IT
manager of EMBL-EBI. Now, from its
outstation at Hinxton, the institute handles
the long term archiving of petabytes of
genomic data, which are served freely to
researchers around the world. As well as a
large chunk of one of Binley’s legacy halls,
EMBL-EBI is opening a resource in the Kao
Data Park, just outside London, a short
hop along the M11 motorway.
“As well as genomic data, we've just
started an imaging archive. We also
have protein array data and structural
data,” he tells DCD. “Our archives take in
approximately six petabytes a year.”
As the science exploded, other
organizations have located at the
Wellcome Genome Campus, including
Genomics England, whose 100,000
Genomes Project is paving the way to
personalized medicine on Britain’s NHS.
There is also the BioData Innovation
Center - an incubator for genomics
startups - and a conference center.
Issue 34 • November 2019 61
POWER
DISTRIBUTION
UNITS
ISO 9001: 2015
Quality Assured

Bespoke Offering

Steel Construction

INTELLIGENT PDUs CUSTOMISED TO Quick delivery

YOUR SPECIFICATION
The Olson intelligent remote monitoring PDU is a modular system which comprises of a
main monitor module with a built-in graphical display and keypad. Offering the option to
connect up to four 8x output switching modules allowing a total of 32 16A switched outputs
to be controlled (total 32A).

• Monitor energy consumption • Programmable UPS

• Remote monitor & switching • SNMP enabled
• External temperature & Humidity sensors
FOR MORE INFORMATION
VISIT OLSON.CO.UK

+44 (0)20 8905 7273

sales@olson.co.uk
Designed & manufactured
olson.co.uk in the UK

Batteries lead
the charge
for a better
grid
In future, energy will be stored
in huge concrete towers and
underground compressed air
vaults. For now, batteries are still
the best option for data centers,
says Natron Energy's
Jack Pouchet
T
he increasing deployment of
renewable energy systems
is leading to greater grid
instability, demand for
additional grid services,
and local energy storage.
Data center owners and operators have
an interesting opportunity to add stability
to the grid, add new revenue models for
internal and external clients, and add new
levels of resiliency to their operations all
with ‘new’ energy storage systems. Although
‘new’ is perhaps not much more than new
approaches to existing proven systems -
albeit with innovative chemistries.
When it comes to energy storage, the
news media and corporate board rooms
are enamored with swashbuckling energy
storage plays like ‘I’ll ship you 100MW in 100
days’ and Popular Science cover projects
such as giant concrete blocks, caverns
of compressed air, and nearly perpetual
motion machines. But when we look
behind the curtain, we find that there are
several practical energy storage systems
commercially available today that may align
well with the power profiles required to
sustain our mission critical applications.
Breaking it down - what type of storage
requirements are there? Or how do we
describe our power profile: power (kW),
speed, acceleration, and duration (kWh).
Think of it as ranging from ultracapacitors
for 10 to 30 seconds all the way to months in
duration from pumped hydro. From a power
profile we may need 100 percent of the
available power immediately and for as long
as it will last, as in batteries.
From a storage perspective speed refers
to the rate at which the storage system
reaches full power capacity. Instantaneous
in the case of ultracapacitors and certain
battery chemistries to ten minutes or more
when speaking of gravitational/compressed
storage systems.
Acceleration refers to peak loads/highly
variable load profile and the ability of the
energy storage system to provide bursts of
power as and when needed.
In the data center world, we prefer to
control our destiny. Hence the reliance
on closely coupled batteries and on-site
generators. Although the combination of
new, nonflammable energy storage systems
of varying energy capacities coupled with
alternative energy resources that can be
dispatched in seconds (fuel cells) or minutes
(turbines) may change the standby generator
Storage wars
status quo. More on diesel as a convenient
energy source later.
From a technology perspective we are
seeing renewed interest and innovation in
gravity and pumped/compressed media
systems. This is in large part due to their
respective abilities to store energy in high
capacity - albeit not necessarily in overall
high density. In the case of pumped hydro
(technically gravity for generation) this could
be in GW months.
On a global basis 95 percent of today’s
stored energy is comprised of Pumped Hydro
Electric Storage. Unfortunately, we won’t see
many new large-scale PHES systems due to
challenges from environmentalists and lack
of political willpower.
Large inertia and compressed gas
systems certainly draw media attention.
Their practicality has yet to be demonstrated,
but there are real examples where these
apparently sci-fi concepts have come to life.
For instance, Energy Vault, a Swiss/
Californian startup has proposed a system
to store gravitational energy that does not
require flooding a valley with a dam. The
company’s “energy tower” has a six-armed
crane, which raises and lowers giant
35-tonne concrete blocks in a 35-story tower,
Issue 34 • November 2019 63
 Storage wars
storing and retrieving electrical energy from
the grid. It’s still at an early stage, with a one-
seventh scale prototype but it has had $110
million investment from the Softbank Vision
Fund.
In Australia, the Hydrostore project has
had some AU$9 million from government
sources including the Australian Renewable
Energy Agency (ARENA) scheme, to develop
an underground compressed air storage
facility, at the abandoned Angas Zinc Mine
near Adelaide. The Advanced Compressed
Air Energy Storage (A-CAES) project will cost
$30 million in total, and should be able to
hold 10MWh of energy, and deliver it at a rate
of 5MW, synchronized and regulated to be
compatible with the local electric grid.
The system uses surplus electricity (off
peak or from renewable sources) to compress
air which is stored underground, kept at
pressure by water displaced into a reservoir.
During discharge, the water flows back and
the expanding air turns a turbine to deliver
electricity. To increase efficiency, the heat
released during compression is stored, and
used to warm the air again during expansion.
These kinds of systems have the potential
to store a large amount of energy for perhaps
four to 24 hours. However, their power
profiles are not instantaneous. They do not
go from zero to 100 percent (or more for peak
loads) of capacity quickly and will require
batteries at or near the load for bridging
purposes.
64 DCD Magazine • datacenterdynamics.com
"There is a simple reason
for all the interest in
batteries: they work!"
A tower in the desert could add value in a
blended, smart-grid ecosystem - so long as it
doesn’t shade the solar field or block the wind
turbines. Compressed gas in an abandoned
mine could do the same. Just don’t plan on
one near your data center any time soon.
Coming back down to earth, the vast
majority of venture capital and government
funding (grants to companies and academia
research grants) goes towards R&D
associated with batteries.
This tends to be clustered around a few
hot subject areas including chemistry,
with lithium still top of the list, anode/
cathode chemistry and material science,
break-through approaches such as thermal
batteries applications like EV Fast Charging.
There is a simple reason for all the interest
in batteries: they work!
And this includes the tried-and-true
lead-acid battery, which is not going away
any time soon. The lead-acid battery is well
known, the characteristics and life-cycle well
understood, it can be relatively inexpensive,
and it is the easiest material to recycle with
over 90 percent of the lead in new batteries
derived from recycling. On a global basis,
lead-acid batteries are already providing
somewhere on the order of 15 minutes
backup for four to eight percent of the grid.
Perhaps more when we include legacy telco
and industrial plants with half-hour to four-
hour backup requirements.
Batteries receive the greatest amount
of investment and R&D for energy storage
systems. Most of the focus is on lithium-
based batteries seeking new chemistries
that reduce the use of rare earth and conflict
minerals, reduce the potential for fire and
explosions, improve electron flow through
the cathode and anode (reducing fire and
explosion risks), manufacturability, and
supply chain improvements.
Fortunately, research continues on
promising chemistries including zinc
derivatives (in early commercial production)
and various sodium-ion options such as
Prussian Blue and ceramic carbon (still R&D).
Of these, the Prussian blue sodium-ion
battery exhibits many of the traits we seek
in the data center space - nonflammable, no
thermal runaway conditions, extremely high
power capacity, wide operating temperature
range (-20°C to +45°C negating the need
for special battery room cooling), extremely
fast recharge (eight minutes), and very high
cycle-rates & life with >60,000 cycles a
reasonable expectation.
In summary, the energy storage market
will continue to see significant financial
investment. There will be exciting
announcements of incredible engineering
projects, and headline news of huge battery
plants. In the meantime, the data center
will continue to deploy practical systems
based upon lead-acid, lithium, and sodium
batteries and back them all up with diesel
gen-sets as diesel fuel remains the best
energy source for continuous operations
once the batteries expire.
Jack Pouchet is at Natron Energy
www.fossilfreedata.com
 Power + Cooling

Do I really
have to say it?

L
ast month, on the way to visit a renewable
energy-powered data center in Sweden,
I found myself trapped in a turgid
conversation with someone in this industry.
There is not enough space on this page
to describe the feeling of mounting dread
that overcame me as I slowly realized that this man was
a climate change denier. No, he was not quibbling about
Learn more
whether we should invest more in wind and solar, or
about data center whether nuclear is the best approach - he was denying
efficiency at the need to do anything at all.
DCD>Energy It’s a viewpoint that still exists among some in this
Smart on 27-28 industry. Sometimes it rears its head in the comments
April section of our website. Other times it is muttered by an
audience member of a conference during an energy
efficiency panel. Occasionally it is said during a very long
train ride through Sweden.
This is meant to be an industry of engineers, whose
actions are based on scientific certitude. This is meant
“When building to be an industry of entrepreneurs, whose accurate
forecasts of the future form the basis of robust business
a data center, plans. Hell, this is meant to be an industry of data

we don’t argue professionals.

How are there still people refusing to accept the data?
whether it needs The Intergovernmental Panel on Climate Change’s
multiple reports, created by hundreds of scientists, based
sturdy floors on thousands of research articles, using multiple high-
because gravity resolution supercomputer simulations, across decades of
study, paints a stark picture: Climate change is real, and
is up for debate” we are to blame.
This is not a point to be argued. When building a data
center, we don’t argue about whether it needs sturdy
floors because ‘gravity is up for debate,’ we don’t argue
about connecting it to the grid because ‘there’s no proof
electricity exists,’ and we don’t argue about cooling it
because ‘we don’t know what will happen to this data
center’s climate.’
Now, this is not to say that we should end all
discussion on the matter.
There are tough questions that need answering:
Which renewables should we invest in? Who should be
responsible for deploying energy storage solutions? Are
power purchase agreements enough?
Should Google and Microsoft fund think-tanks and
lobbyist groups that deny climate change? Okay, that last
one isn’t that tough.
We need to start arguing how to actually combat this
problem, not whether there is one. It is too late for us to
pretend that anthropogenic climate change is a myth
dreamed up by scientists for reasons unclear.
Once we accept the data, then the real change
can begin.

66 DCD Magazine • datacenterdynamics.com

DATA DOESN’T DO
DOWNTIME
DEMAND CAT® ELECTRIC POWER

Cat® generating sets and power solutions provide flexible, reliable, quality power
in the event of a power outage; maintaining your operations, the integrity of
your equipment and your reputation.

Learn more at http://www.cat.com/datacentre

© 2019 Caterpillar. All Rights Reserved. CAT, CATERPILLAR, LET’S DO THE WORK, their respective logos, “Caterpillar Yellow”, the “Power Edge” and Cat “Modern Hex”
trade dress as well as corporate and product identity used herein, are trademarks of Caterpillar and may not be used without permission.

Visit Cat Dealer Finning UK at DCD London 2019,

5th - 6th November Stand No. 45
 Still relying solely Switch to automated
on IR scanning? real-time temperature data.

Introducing the
Starline Temperature Monitor

Automated temperature monitoring is the way of the future. The Starline Critical Power Monitor (CPM) now incorporates
new temperature sensor functionality. This means that you’re able to monitor the temperature of your end feed lugs in
real time– increasing safety and avoiding the expense and hassle of ongoing IR scanning. To learn more about the
latest Starline CPM capability, visit StarlineDataCenter.com/DCD.