> Colo Design | Supplement

Future-Proofing

Colo Design
Supplement
Sponsored by
INSIDE

Racks of the future Designing for chaos Safe and secure

> We’re used to the 19in rack. > We’re still designing for a world > It might be mostly security
Some of us are comfortable with that is not on fire. It’s time to accept theater, but protecting your colo
the 21in rack. But why stop there? climate change, and build for it from intruders matters
 First lithium-ion battery
cabinet designed by data
center experts for data
center users.
Cut your total cost of ownership with the Vertiv™ HPL
lithium-ion battery system, a high-power energy
solution with best-in-class footprint, serviceability
and user experience.

Smaller, lighter and lasting up to four times longer than VRLA

counterparts, the newest generation of lithium-ion batteries pay
for themselves in a few years. The Vertiv HPL battery cabinet features
safe and reliable lithium-ion battery modules and a redundant battery
management system architecture with internal power supply. These
features and the cabinet’s seamless integration with Vertiv UPS units
and monitoring systems make it ideal for new deployments or as
replacement for lead-acid alternatives. Plus, its user-friendly display
leads to a best-in-class user experience.

Why choose Vertiv HPL Energy Storage System?

yySafe, reliable backup power that offers a lower total cost

of ownership

yyDesign that is optimized for use with Liebert® UPS units

and for control integration

yyWarrantied 10-year battery module runtime at temperatures up

to 86°F/30°C for improved power usage effectiveness (PUE)

yySuperior serviceability

Visit Vertiv.com/HPL

© 2019 Vertiv Group Corp. All rights reserved. Vertiv™ and the Vertiv logo are trademarks
or registered trademarks of Vertiv Group Corp. While every precaution has been taken to
ensure accuracy and completeness here, Vertiv Group Corp. assumes no responsibility, and
disclaims all liability, for damages resulting from use of this information or for any errors or
omissions. Specifications are subject to change at Vertiv’s sole discretion upon notice.

Contents
4 T
 he future of hardware: Why racks
and cooling aren't a done deal
8 Advertorial: Vertiv lives the dream
10 R
 esilient design: Can your facility go
off-grid and survive a catastrophe?
13 P
 ost human design:
Facilities still need people
14 Colo security: Let the right one in
13
Supplement designed by Dorothy McHugh
10
4 preparation for this is emerging
8 and storms.
14
Designs for the future
A
fter many years
of building data
centers, you might
think that the
innovation was
done, but that isn't
the case. This supplement looks at
how design is changing in the most
important areas.
Racks and cooling are the most
basic elements of any facility, but
they are up for some of the biggest
potential changes (p4).
Niche areas like HPC have the
biggest leeway to rock the boat.
They have higher margins and
higher demands for performance.
Cloud providers with complete
control of their hardware are also
leading the charge. In one French
provider we saw racks shifted to a
horizontal position, and a custom
cooling system designed and built
for one provider alone.
That doesn't seem to make sense
in a world where commoditization
normally drives out diversity.
But what's happening is
that changes in one part of
the ecosystem are driving
developments elsewhere.
Living through disaster has
risen through the priorities of data
center operators, as the likelihood
of freak weather increases, and
political tensions fluctuate and roil.
Apocalyptic as this sounds,
it turns out that a lot of the
right now (p10). Data centers
already have to plan for power
outages, with battery power and
UPS. Now they need to add floods
If they can also arrange
independence from the grid, then
so much the better. A data center
which can support itself this way
is also helping reduce overall
environmental impact. And if it can
Colo Design Supplement
Sponsored by
share some of that strength with
the surrounding grid, then it is very
much part of the solution.
Goodbye to humans? It looked
like lights-out operation might be
coming, allowing IT equipment to
be operated in sealed environments
suited to electronics, not breathing
organisms (p13).
But the customer is still human.
Even for vanity alone, data centers
still need life support in case the
masters should drop by, either to fix
the hardware or impress a client.
For now at least. If Microsoft
succeeds in moving data to the
sea bed, server huggers may finally
decide to stay ashore.
Finally, security. You might
think this remains the same. It's
all about keeping data private and
avoiding breaches which can harm
your organization. But if there is
a radical change to architectures,
then you need to rethink how you
secure the data and the promises
that hold it (p14).
In a nutshell, when virtual and
physical spaces are increasingly
shared, this increases the necessity
to secure physical access. But
ironically, as outsourcers hold
other people's data, the same
process might actually decrease the
likelihood that they will be properly
secured.
Security is never complete, and
nor is the overall design issue. This
supplement updates the story but it
will not be the last word.
This year's design ideas could
be obsolete in a couple of years'
time, and so could some of the
business models driving them.
The common factor will be
the relentless pressure to find
creative solutions to the shifting
requirements of the field.
Colo Design Supplement 3
 Colo Design Supplement | Hardware of the future

FUTURE OF
HARDWARE
You might expect the physical design
of data center hardware to be pretty
standard by now. Don’t be so sure…

"REITs are not the

perfect answer. We are
a large option, not as
constrained a REIr. We
are a large option, not as
constrained a REIT"

Peter Judge
Global Editor

4 DCDMagazine
4 DCD Magazine••datacenterdynamics.com
datacenterdynamics.com
S
ervers have been installed in 19in
racks since before data centers
existed. The technology to cool
the air in buildings has been
developed to a high standard, and
electrical power distribution is a
very mature technology, which has made only
incremental changes in the last few years.
Given all that, you might be forgiven
for thinking that the design of data center
hardware is approaching a standard, and
future changes will be mere tweaks. However
you would be wrong. There are plenty of
radical approaches to racks, cooling and
power distribution around. Some have been
proclaimed for years, others have come
from out of the blue. Not all of them will gain
traction.
Rack revolution
For someone used to racks in rows, a visit
to one of French provider OVH’s cloud
data centers is a disorienting experience.
OVH wants to be the major European cloud
provider - it combines VMware and OpenStack
based platform as a service (PaaS) public
clouds and hosts corporate private clouds for
customers - but goes against the standard fare
of the industry. Instead of standing vertically,
OVH’s racks lie horizontally.
Close to its French facilities in Roubaix,
it has a small factory, which makes its own
rack frames. These “Hori-Racks” are the same
size as normal 48U racks, but configured
completely differently. Inside them, three
small 16U racks sit side-by-side. The factory
pre-populates these “Hori-Racks” with servers
before they are shipped to OVH facilities,
mostly in France, but some in farther corners
of the world.
The reason for the horizontal approach
seems to be speed of manufacturing and ease
of distribution: “It’s quicker to deploy them in
the data centers with a forklift, and stack them”
said OVH chief industrial officer François
Stérin.
Racks are built and tested speedily, with
a just-in-time approach that minimizes
inventory. Three staff can work side by side to
load and test the hardware, and then a forklift,
truck or trailer (or ship) can move the rack
to its destination, in Gravelines, Strasbourg
or Singapore. In the building, up to three
racks are stacked on top of each other, giving
the same density of servers as conventional
racking.
OVH can indulge its passion for exotic
hardware because it sells services at the PaaS
level - it doesn’t colocate customers’ hardware.
Also, it takes a novel approach to cooling
(see later) which allows it flexibility with the
building itself.
You don’t have to go as far as OVH in
changing racks: there are plenty of other
people suggesting new ways to build them.
The most obvious example is the open source
hardware groups like Open Compute Project
(OCP), started by Facebook, and Open19,
started by LinkedIn.
Both operate as “buyers’ clubs,” sharing
custom designs for hardware so that multiple
customers can get the benefits of a big
order for those tweaks - generally aimed at
simplifying the kit, and reducing the amount
of wasted material and energy in the end
product. Traditional racks, and IT hardware,
it turns out, include a lot of wasted material,
from unnecessary power equipment down to
manufacturers’ brand labels.
OCP was launched by Facebook in 2011 to
develop and share standardized OEM designs
for racks and other hardware. The group’s
raison d’etre was that webscale companies
could demand their own customized
hardware designs from suppliers, because
of their large scale. By sharing these designs
more widely it would be possible to spread the
benefits to smaller players - while gathering
suggestions for improved designs from them.
While the OCP’s founders all addressed
giant webscale players, there are signs that
the ideas have spread further, into colocation
spaces. Here, the provider doesn’t have
ultimate control of the hardware in the space,
so it can’t deliver the monolithic data center
architectures envisaged by OCP, but some
customers are picking up on the idea, and the
OCP has issued facility guidelines on making
a place “OCP ready,” meaning that OCP
racks and OCP hardware are welcome and
supported there.
OCP put forward a new design of racks,
which packs more hardware into the same
space as a conventional rack. By using
more of the space within the rack, it allows
equipment that is 21in wide, instead of the
usual 19in. It also allows deeper kit, with an
OpenU measuring 48mm, compared to a
regular rack unit of 44.5mm.
The design also uses DC power, distributed
via a bus bar on the back of the rack. This
approach appeals to monolithic webscale
users such as Facebook, as it allows the
data center to do away with the multiple
power supplies within the IT kit. Instead of
distributing AC power, and rectifying it to DC
in each individual box, it’s done in one place.
Open Rack version 1 used 12V power,
and a 48V power was also allowed in version
2, which also added the option for lithium-
ion batteries within the racks, as a kind of
distributed UPS system.
That was too radical for some, and in 2016
LinkedIn launched the Open19 group, which
proposed mass market simplifications without
breaking the 19in paradigm. Open19 racks are
divided into cages with a simplified power
distribution system, similar to the proprietary
blade servers offered by hardware vendors.
The foundation also shared specifications for
network switches developed by LinkedIn.
“We looked at the 21 inch Open Rack,
and we said no, this needs to fit any 19 inch
Colo Design Supplement 5
 Colo Design Supplement | Hardware of the future
rack,” said Open19 founder Yuval Bachar,
launching the group at DCD Webscale in 2016.
”We wanted to create a cost reduction of 50
percent in common elements like the PDU,
power and the rack itself. We actually achieved
65 percent.”
Just as it launched Open19, LinkedIn
was bought by Microsoft, which was a major
backer of OCP, and a big user of OCP-standard
equipment in data centers for its Azure cloud.
Microsoft offered webscale technology to
OCP, such as in-rack lithium-ion batteries to
provide local power continuity for the IT kit,
potentially replacing the facility UPS.
While the LinkedIn purchase went
through, OCP and Open19 have continued
in parallel, with OCP catering for giant data
centers, and Open19 aiming at smaller
facilities used by smaller companies that
nevertheless, like LinkedIn, are running their
own data centers. Latterly, Open19 has been
6 DCD Magazine • datacenterdynamics.com
focusing on Edge deployments.
In July 2019, however, LinkedIn
announced it no longer planned to run its
own data centers, and would move all its
workloads over to the public cloud - obviously
enough using its parent Microsoft’s Azure
cloud.
Also this year, LinkedIn announced that
its Open19 technology specifications will be
contributed to OCP. It’s possible that OCP and
Open19 specifications may merge in future,
but too early to say. Even if LinkedIn no longer
needs it, the group has more than 25 other
members.
For webscale facilities, OCP is pushing
ahead with a third version of the OCP Rack,
backed by Microsoft and Facebook, and it
seems to be driven by increasing power
density demanded by AI and machine
learning.
“At the component level, we are seeing
power densities of a variety of processors
and networking chips that will be beyond the
ability of air to cool in the near future,” said the
Facebook blog announcing OCP Rack v3. “At
the system level, AI hardware solutions will
continue to drive higher power densities.”
The new version aims to standardize
manifolds for circulating liquid coolant
within the racks, and also heat exchangers
for cabinet doors, and includes the option of
fully immersed systems. It’s not clear what the
detailed specifications are but they will emerge
from OCP’s Rack and Power project, and its
Advanced Cooling Solutions sub-project.
Liquid cooling
For the last couple of decades, liquid cooling
has shown huge promise. Liquid has a much
higher ability to capture and remove heat than
air, but moving liquid through the hardware
in a rack is a big change to existing practices.
So liquid has remained resolutely on the list
of exotic technologies which aren’t actually
worth the extra cost and headache.
Below 20kW per rack, air does the job
cost effectively, and there is no need to bring
liquid into the rack. Power densities are still
generally well below that figure, so most data
centers can easily be built without liquid
cooling. However, there are two possibilities
pushing liquid cooling to the fore.
Firstly GPUs and other specialized
hardware for AI and the like could drive the
power density upwards.
"At the system level, AI
hardware solutions will
continue to drive higher
power densities"
Secondly, for those who implement liquid
cooling there are other benefits. Once it’s
implemented, liquid cooling opens up a lot of
flexibility for the facility. Air-cooled racks are
part of a system which necessarily includes air
conditioning, air handling and containment -
up to and including the walls and floors of the
whole building.
Liquid cooled racks just need an umbilical
connection, and can be stood separately, on a
cement floor, in a carpeted space, or in a small
cabinet.
This can be difficult to present in a retail
colocation space, because it affects the IT
equipment. So unless the end customer
specifically needs liquid cooling, it’s not going
to happen there. But it does play well to the
increasing flexibility of data centers, where
the provider has control of the hardware,
and there isn’t a building-level containment
system.
Small Edge facilities are often micro data
centers without the resources of a full data
center behind them. Other data centers are
being built inside repurposed buildings, often
in small increments. A liquid cooling system
can fit these requirements well.
Early mainframes were water-cooled but
in the modern era, liquid cooling has emerged
from various sources.
Some firms, including Asperitas, Submer
and GRC, completely immerse the racks in
a tub of inert liquid. No energy is required
for cooling, but maintenance is complicated
because the rack design is completely
changed, and servers and switches must be
markets, said Stérin: “To test the market we
don't need to build a giant 100MW mega data
center, we can start with like a 1MW data
center and see how the market works for us.”
With a lot of identical racks, and a
factory at its disposal, OVH has pushed the
technology forward, showing DCD multiple
versions of the concept. OVH technicians
demonstrated a maintenance procedure on
the current version of the cooling technology.
It seemed a little like surgery. First the
tubes carrying the cooling fluid were sealed
with surgical clamps, then the board was
disconnected from the pipes and removed.
Then a hard drive was replaced with an SSD.

"To test the market we

don't need to build a
giant 100MW mega data
center"
lifted from the tub and drained before any
hardware modifications. Iceotope, which now
has backing from Schneider, has a system
which immerses components in trays within
the racks.
Others offer direct circulation, piping liquid
through the heatsinks of energy-hungry
components. This was pioneered by gamers
who wanted to overclock their computers,
and developed rigs to remove the extra
heat produced. Firms like CoolIT developed
circulation systems for business equipment
in racks, but they have been niche products
aimed particularly at supercomputers. They
require changes to the racks, and introduce
a circulatory system, which flows cool water
into the racks and warm water out.
In Northern France, OVH has its own take Already this design is being superseded by its own, on a cement floor, connected to a
on liquid cooling, as well as the racks we saw another one, which uses bayonet joints, so the conventional cooling system. Mainstream
earlier. It builds data centers in repurposed board can be unplugged without the need to vendors such as Vertiv offer modular builds
factories which formerly made tapestry, clamp the tubes. that can be placed on cement floors, while
soft drinks and medical supplies, and liquid Less radical liquid cooling systems are others offer their own take.
cooling allows it to treat these facilities as available, including heat exchangers in the One interesting vendor is Giga Data
shells: building in one go, with a raised floor, cabinet doors, which can be effective at Centers, which claims its WindChill enclosure
and building-level air conditioning OVH levels where air cooling would still be a viable can achieve a PUE of 1.15, even when it is
stacks in racks as needed, connecting them to option. being implemented in small increments
a liquid cooling system. OVH combines this with its circulation within a shell, such as the Mooresville, North
“Our model is that we buy existing system. The direct liquid cooling removes Carolina building where it recently opened a
buildings and retrofit them to use our 70 percent of the heat from its IT equipment, facility.
technology,” OVH chief industrial officer but the other 30 percent still has to go The approach here is to build an air
François Stérin explained. “We can do that somewhere - and is removed by rear-door cooling system alongside the rack, so a large
because we make our own racks with a water heat exchangers. This is a closed loop system, volume of air can be drawn in and circulated.
cooling system that is quite autonomous - with the heat rejected externally. Don’t be lulled into a false sense of
and we also use a heat exchanger door on the Liquid cooling isn’t necessary for a system security. The design of hardware is changing
back of the racks. This makes our racks pretty designed to be installed in a shell. It’s now just as fast as ever, and those building
agnostic to the rest of the building.” quite common to see data centers where colocation data centers need to keep track of
The flexibility helps with changing a contained row has been constructed on developments.

Colo Design Supplement 7

Advertorial: Vertiv
Why the Internet of
Everything Needs the
Internet of Everywhere
A fundamental realignment of data center infrastructure is underway from the core to
the edge. But what does that mean in practical terms? Ubiquity, speed and power will be
critical according to Vertiv and EdgeConneX.
I
t’s no secret that the number
and types of Internet-connected
devices are undergoing a seismic
shift.
The expectation is that data
from billions of newly-connected
Internet of Things (IoT) devices – from
mobile handsets to industrial equipment
– will drive a major change in Internet
architecture: from the core to the edge.
According to a 2018 451 Research survey,
of more than 700 data center operators, 73
percent of respondents expect that up to 75
percent of data center/cloud capacity will
be used to support IoT initiatives by the end
of 2020.
The need to effectively remake the
Internet for a new era of connected devices
and associated two-way data traffic will
create challenges and opportunities for
data center operators (including colocation
providers) and technology suppliers alike.
Two organizations that are deeply
immersed in helping to deliver that
architectural change are EdgeConneX and
Vertiv.
Vertiv, a specialist in critical
infrastructure, and EdgeConneX, a pioneer
in defining and building the edge of the
network, have partnered on dozens of
projects across three continents (North
America, South America and Europe) since
2014.
Those projects include supporting
hyperscale cloud deployments in Chicago,
Dublin and Amsterdam, to multiple network
edge sites extending east to west from
Miami to Seattle, and from north to south
from Toronto to Buenos Aires.
Vertiv and EdgeConneX also have
invested significant time, effort and
resources into understanding what
future data center demand will look like –
especially at the edge. For example, Vertiv
recently released an in-depth report on
the future of the data center industry.
Data Center 2025: Closer to the Edge
included results from a global survey of
more than 800 data center professionals.
According to the report, 20 percent of
survey respondents predicted a 400 percent
increase in the number of edge sites by
2025.
Similarly, EdgeConneX also has been
involved in impactful research initiatives.
For example, it contributed to a recently
published Telecommunications Industry
Association (TIA) Position Paper that
provides an important overview for
defining development, implementation,
and operational aspects of Edge Data
Centers (EDCs).
Vertiv and EdgeConneX recently brought
their unique perspectives together. Matt
Weil, director of offering management
for Vertiv, joined Philip Marangella, chief
marketing officer for EdgeConneX, for an
in-depth DCD Discussion to discuss how
data center infrastructure is evolving, and
the practical considerations for owners and
operators.
Beyond the Data Tsunami
The DCD Discussion came at the issue of
future edge and data center demand from
several angles, but the initial focus was
on trying to understand the size of the
demand and the key drivers. “A lot of people
in the industry point to the volume of data
generated and the volume consumed. We
have referred to those sexy statistics too:
The number of zettabytes per year, what
that is going to be five years from now,
10 years from now, using phrases like
"The Internet of the
last decade was not
architected in the way it
needs to be architected
for the future."
exploding demand and tsunami of data,”
said Vertiv’s Weil. “However, what I think is
interesting is to start to sub-segment that
and not focus on the mega-trend but think
more about the use cases and the things
that are driving that data.”
Weil referenced another recent research
effort from Vertiv in which the company
defined four key edge archetypes it believes
could help the industry develop a more
nuanced view of different edge scenarios
and when they may materialize. “We
need to think about which of these are
happening now, which are happening
soon and which are a little bit more down
the road,” Weil said. “For example, people
point to driverless cars as an edge use case,
but from a mass adoption perspective it is
probably at least five years off, if not more.”
Ubiquity is key
There was agreement from both Weil and
Marangella that supporting those new use
cases and archetypes from a latency and
bandwidth perspective will require Internet
architecture that is ubiquitous and not
just centered around major data center
hubs. “I like the phrase from EdgeConneX
that the ‘Internet of Everything needs the
Internet of Everywhere’ – you need those
points of presence to be ubiquitous,” said
Weil. “And that really requires us, which is

the phase we are going through now, to
rearchitect how that Internet is constructed.
The Internet of the last decade was not
architected in the way it needs to be
architected for the future.”
The need for speed
Another key issue in the Vertiv and
EdgeConneX discussion was the
importance speed of deployment has
and will play in meeting future data
center demand. Marangella explained
that build time is a major competitive
differentiator for EdgeConneX. “Speed is
absolutely critical for us. We sell to the most
demanding cloud, content and networking
service providers in the world and they
have extremely high expectations,” he said.
“Also, consider that we don’t speculatively
build, ‘Here is our site, come to us.’ We
work with the customer to find the optimal
location for them whether it be diverse
power feed, diverse network or what have
you.”
Marangella added that speed is such
a critical issue for EdgeConneX because
the company often competes against
colocation providers that may already
be established in a specific territory.
“A competitor might not have the best
capacity in terms of location, but the
availability of that capacity is here and
now,” he explained. “To solve for that we
need to be very fast in our process in terms
of how we select and build or convert a
particular site and make it live for that
customer.”
Advertorial: Vertiv
EdgeConneX may need to complete The response from the industry needs be
new builds in as little as six months to investment in new approaches to on-site
remain competitive, but often exceeds even power generation, according to Marangella.
that compressed timeframe. “I think our “That is where you need to get creative with
record is three and a half months where some power requirements,” he said. “We
we were able to turn around a data center did that in Dublin where we built our own
for a particular customer, and that is where natural gas power generation plant on-site
having Vertiv as a partner comes in, where to be able to augment the grid power to
that consistent design can be replicated in have adequate MW to be able to support
new markets.” customer requirements.”
Ultimately, it is important to understand
Power availability that keeping up with change is not just about
While innovations in data center design are the edge or just the enterprise, and growth
helping EdgeConneX and other operators of the edge is not some sort of solution to a
to compete more effectively, there are other data center problem. It is a natural evolution
issues that still need to be overcome when of the network to meet changing consumer
it comes to building out future capacity. demand. And more importantly, it is one
Specifically, Marangella sees power piece of an increasingly diverse network
availability as a long-term issue for the quilt that also includes enterprise, colocation
whole industry. and cloud data centers.
“Power is a big mitigating factor for The next-gen network is both hyperscale
data centers going forward, more than the and hyper-local, and the businesses who
design or what is going inside the building,” will succeed in this environment will work
he said. “The issue is especially pronounced with partners who can provide end-to-end
when you look at markets like Europe expertise and solutions. Anything less will
where available grid power is insufficient to prove insufficient in the face of the growing
support the growth of data centers.” data tsunami.
Vertiv Global Solutions
Matt Weil, Director Offering Management
matthew.weil@vertiv.com
(+1) 630 579 5052
 Colo Design Supplement | Resiliency amid chaos

Designing for
RESILIENCY
in a chaotic age
We live in interesting times, and that requires data centers Sebastian Moss
to be designed with extra care, Sebastian Moss reports Deputy Editor

I
n a year where arctic forests burn,
cities experience their first major
floods, and record hurricanes batter
unprotected towns, belief in a stable
world is perhaps a sign of an unstable
mind.
Our planet is becoming an increasingly
dangerous place, threatening not just
people’s lives, but the buildings that support
societies, and the grid that powers it all. And
yet, despite copious evidence showing these
rising risks, many are continuing to operate
with a business as usual mentality.
“It's incredible to me to see that even
data center operators are not considering
that at all,” Ibbi Almufti, head of the risk and
resilience practice at engineering and design
consultancy company Arup, told DCD.
“They're just building like everyone else
builds - it's crazy.”
Data centers are often designed simply
to comply with modern building code
standards, which on the face of it seem to
be written to ensure the construction can
withstand a reasonable amount of disaster.
“But the only thing the [building codes]
care about is life safety. And even when
you're designing to a higher level of code
performance, it's just lower life safety risk.
“It's never looking at functionality-level
performance or anything like that,” Almufti
said. Considering the rising perils, and the
need to maintain data center uptime, data
center design needs to “move the goalposts
from life safety to beyond life safety and look
at limiting downtime and repair costs and
protecting your investment.”
He added: “I will venture to say that if
you're building data centers right now, in
higher natural hazard zones, you most likely
will not be meeting your availability targets
even if you're designing to modern building
codes.”
It’s a question of weighing the cost of
making a data center more resilient versus
the benefit of it not suffering downtime - or,
at least, returning to service faster. “With
data centers, if you're out for a day or two,
you're out millions of dollars,” Almufti said.
“You could just take those millions of dollars
and invest it up front to prevent that from
happening in the first place. Kind of a no
brainer.”
Anthropogenic climate change has made
the weather more savage and unpredictable,
and will soon make it much worse. But it’s
not the only problem.
“Forget about global warming for a
second, if you have a site you built 10
years ago, and now you've got land being
developed [nearby], then all of a sudden,

10 DCD Magazine • datacenterdynamics.com

you’ve got water run-on - that's increasing
your flood hazard,” Almufti said.
In the Midwest of the US, as well as
elsewhere around the world, “there's new
seismicity from the deep well wastewater
injection,” also known as fracking. “This is
man-made induced seismicity,” Almufti said.
“And scientists discover new faults all the
time - I'm a seismic engineer - you will never
see the seismic hazard go down in a location
because of new science, it always goes up.“
Those looking to design for seismic events
could learn from Japan, a nation that has
wrestled with earthquake after earthquake.
After opening a new data center in the
country in 2017, Colt DCS' VP of operations,
Ian Dixon, told DCD: "The tried and trusted
way to make a structure more resistant to the
lateral forces of an earthquake is to tie the
walls, floor, roof together to form a super-
structure. Then have an isolated foundation
that becomes the sub-structure.
“That is the basic premise for our latest
data center. However, in order to nullify the
vibrations and keep the equipment as safe
as the human inhabitants, we also needed to
look at what the building sits on.”
Colt’s Inzai 2 facility sits on a series of
seismic isolators and teflon sliders. “These
isolators are sometimes called a Shake
Table - capable of holding 125 tons per m2,
isolating the whole building from any seismic
forces and allowing the sub-structure and
super-structure to move independently,”
Dixon said. “Thus, permitting the bulk of the
seismic shock/energy to be dissipated by the
sub-structure and reduce the impact on the
super-structure."
That might be excessive for most US data
centers, but the problem is that the majority
of data center operators may no longer be
aware of the risks surrounding their facility,
and what design changes they need to make.
Worsening weather, flood runoff risks,
seismic changes - “a lot of these things just
slip through the cracks,” Almufti said. “You
might have assessed the risk at one point
- probably not - but even if you have, it's
changed a lot in the last 5, 10 or so years.”
Even when a facility itself is protected
from these changes, it may be at the mercy
of surrounding infrastructural weaknesses.
“Our analysis shows that relatively low wind
speeds can cause power outages. This is
"If you're building data
centers in higher natural
hazard zones, you
most likely will not be
meeting your availability
targets"
primarily due to trees falling on power lines
and disrupting the services - they're typically
down for one, two days at a time. It's not an
infrequent event.”
Another problem “is that multiple data
centers may be relying on the same utility
substation, and that substation is essentially
a single point of failure,” Almufti said. “A lot
of the substations that we've seen are in flood
zones, for example.”
That’s why some are turning to microgrids
to give data centers more control over the
power used to keep their data centers online.
"We know that gas can go down, we know
that power can go down; it's about making
the site secure," Arup associate Russell Carr
told DCD.
Microgrids incorporate three key
components: “You've got some generation,
you've got some storage, and you've got a
demand or load,” Carr said. “Generation can
be provided by multiple sources - it can be
variable renewables, such as photovoltaics,
wind, biogas fuel cells, natural gas fuel cells,
natural gas generators, regular generators -
it just needs a system to provide power but
it has to have storage as well, that can be
lithium-ion batteries, flow batteries, various
technologies.”
The decision over which generation and
storage technologies to use is dictated by the
hazards found in the data center’s location, as
Colo Design Supplement 11
 Colo Design Supplement | Resiliency amid chaos
well as local regulatory stipulations.
“Let's say I'm on the East Coast [of the
US],” Carr said. “There’s a lot of overhead
power lines, I might have my backup
generation in gas fuel cells because I know in
Superstorm Sandy the gas system stayed up,
while the electricity system stayed down.
“Whereas, on the West Coast, the disaster
here is an earthquake, gas lines can go out for
a long time, but electricity would typically be
restored back in a week.’” Regulations, such as
air quality laws in California, dictate design:
“Diesel generators can't operate all the time,
gas turbines go through a lot of permitting
because California is trying to phase out gas…
these are things you have to consider.”
Arup recently completed a microgrid
project in California for a confidential client
who wanted to build a multipurpose campus
with several Edge data centers. “Resiliency
was a big driver for this site,” Carr said.
“The client's aim was to be able to operate
indefinitely independent of the grid for their
key loads,” islanding the site off from the
wider grid.
“This client wasn't stupid, they didn't
want to put this in for free - they also wanted
the system to make money, so we looked at
how can we make money [from] the battery
capacity of this site.”
For the campus, the client had 4MW of
Edge data center load in each of the four
buildings, along with around 20MW of
general campus load. “We had 16MW of solar
on site, 6MW of diesel generation, 4MW of
fuel cell generation, and 5MW of batteries at
this site,” Carr said.
Should clouds gather over its solar panels,
the system proactively drops non-critical
loads before the shade cuts photovoltaic
output. Whatever happens to the outside
world, the campus can continue on, an
island unto itself.
But, Carr conceded, such luxurious grid
resilience is only really for smaller data
center deployments: “If we're looking at a
big hyperscale data center, it's harder to put
renewable generation on that data center -
even if you covered the roof with solar, it's a
100MW data center. You're gonna power the
lights.
DCD>Resources When can we afford to deploy solar
for on-grid solutions?
With the reduction of solar panel prices and infrastructure, and the slow rise of utility rates,
this whitepaper analyses whether solar can serve as a power device and share duties to
provide power to the load in On-Grid solutions with reasonable financial returns.
bit.ly/DCDSolarStudies
12
12 DCD
DCDSupplement • datacenterdynamics.com
Magazine • datacenterdynamics.com
“You've got to look at the load of that That fault ruptures every multiple thousand
particular building and the size of that years, but when it does, it's huge. So how do
building, and how to match up generation you design for that?”
and storage to that building.” And, with some risks potentially wiping
Microgrids for larger facilities can still out the region’s population, is there even any
provide financial returns, and help improve point designing for that? Such questions can
grid resilience, but full islanding is unlikely. weigh on someone, and Almufti admits that
Instead, with such sites, it helps that the immersing himself in a world of risks has
companies that operate those data centers changed him.
usually manage numerous facilities. “I'm super risk averse,” he said. “I live in
“It’s about basically combining the San Francisco, and when I walk through
interdependencies of those data centers,” downtown I’m always looking up like 'Oh my
Almufti said. “So, for example, the parent-child god, please don't happen now,' or if I go in
relationships between individual data centers an older building, it’s like ‘get me out of here
and comparing that against the spatial extent right now.’
of a given hazard scenario to understand “It's kind of like living in like The Matrix,
the likelihood of simultaneous outage and no one else sees it, but you see it because you
downtime at multiple data centers.” do it every day. The chances that you're in
a building when the disaster strikes is really
low, but it happens. It happens.”
"Solar flares are kind of
scary, I don't think we
can quantify the risk to
that. That's like a black
swan event"
Such an approach can “inform ideal
separation distances between existing data
centers and new data centers.”
Despite such precautions, no matter how
much one prepares, no matter how strong
one builds, there are still the unknown,
unpredictable dangers. “A major solar flare
could wipe out the entire eastern seaboard,”
Almufti said. “Things would be down for
months - everything, not just the individual
data centers. Solar flares are kind of scary,
I don't think we can quantify the risk to
that. That's like a black swan event, but it
happened 250 years ago, so it's possible.”
There are several such hazards where
“the cost benefit would show that you
shouldn't design for these events, if you're
not impacted for hundreds and hundreds of
years.
“One thing that's lurking out there is the
New Madrid Fault Line in the Memphis area.
Read
Now

The post-human data
center will have to wait
Despite the predictions, it seems that data centers
aren’t yet ready to do without human kindness
T
hree years ago, we heard a lot
about “lights-out” facilities.
Data centers were going to
become fully autonomous and
run without intervention from
people.
There were a whole lot of reasons this
was going to happen. Staff are expensive,
and equipment is reliable. One leader in
promising this was EdgeConneX, a colo
provider specializing in smaller facilities
outside the major data center hubs. “When
you try to operate this small footprint, like a
two-megawatt facility, it’s difficult to man on-
site. It’s just not economically feasible,” he told
a data center event.
At this point, we had had a decade of huge
claims for management and monitoring
systems like DCIM, which could handle the
M&E equipment, while every IT systems
vendor was promising fully virtualized private
cloud systems made up of pools of storage
and servers which could be built and changed
in software, on demand - HPE’s “composable”
data center for instance.
Surely, with all this, it should be possible
to operate a data center remotely. Failed
hardware could be sidelined, and replaced en
masse during an occasional site visit by an
engineer.
Going lights-out promised other benefits
besides staff savings. Automatic processes
could be more reliable we were told -
Experts in failure, like the Ponemon
Institute, regularly list human
error as the top cause of
outages.
Colo Design Supplement | Post-people?
Meanwhile, conditions in an unstaffed
data center could be optimized for the
efficient running of the IT equipment, not
the comfort of the staff. Modern equipment
operates at a greater range of temperatures.
The so-called cold aisle can now be as warm
as 80°F, and the hot aisle can be more than
100°F. Stop cooling data centers for comfort,
and a lot less energy is wasted.
And in these temperatures, if you get the
physical IT work done by robots, then you
We now finally
have a use case for
autonomous IT
resources: The Edge
can eliminate unused space too. Robots can
be designed to operate in narrower spaces,
leaving more room for the IT equipment.
Back then, companies like web hoster
PayPerHost were promising robotic data
centers, while LitBit was going to use AI to
automate the maintenance procedures. And
EdgeConneX had edgeOS, a platform for
remote management.
A peak moment for the lights-out data
center movement came in 2016, when
Microsoft announced that it had operated a
rack of Azure Cloud servers on the sea bed
for three months. Project Natick was sealed
in a pressurized vessel, and operated in a
lights-out manner, because the only way to
physically access it was to haul it up to the
surface.
Natick added another item to the list of
benefit delivered by post-human facilities:
it was filled with a nitrogen
atmosphere: It was
unbreathable, but it
guaranteed there would be
no fires.
After all that excitement,
the world of lights-out
facilities has gone rather
quiet. Which is a shame,
because we now finally
Peter Judge
Global Editor
have a use case which absolutely needs
autonomous IT resources: The Edge.
The over-familiar pitch for the Edge is that,
because new applications like the Internet
of Things require low latency, data center
resources must be available close to the
sources of data, at the “edge” of the network.
Managing a multitude of tiny Edge
resources manually would add so much
overhead, those applications would become
financially impossible, so now is the time for
the lights-out facility to finally arrive.
The trouble is, things have gone quiet.
DCIM products continue to potter along,
making the same promises and not
perceptibly changing. LitBit disappeared
without a trace, and the robot arms are folded
away: PayPerHost is sticking to its knitting:
As far as we can tell, it’s never said anything
more about robots.
EdgeConneX’s King says that small users
don’t mind the idea of autonomous IT, but
large customers don’t trust it: “Fortune 500
companies say 'okay that’s great, but we’re
going to add staff.'”
And, perhaps ironically, those considering
lights-out operation can be dissuaded by
industry efforts to increase reliability. If a
business wants Uptime Institute’s Tier III or
IV levels of reliability, Uptime recommends a
minimum of one to two qualified operators
on site 24 hours per day, 7 days per week, 365
days per year. And it goes without saying,
those staff need oxygen and room to move.
Project Natick is still going, and has
expanded from one rack to twelve racks, in a
sunken box the size of a shipping container,
under the sea near Orkney. But that was a
year ago. Microsoft promised to reveal results
about now, but all it’s saying is that the test
continues, with Natick in use by development
teams.
“The test is ongoing,” Ben Cutler of
Microsoft Research said to DCD. “We’re
extremely pleased with the reliability and
other operational metrics we've seen.”
That’s fine, but anyone thinking about
lights-out operation, will need a little more
than that.
Issue 31 •Colo
January/February 2019 13
Design Supplement
 Colo Design Supplement | Security focus
Data is private, but data centers are often shared.
That’s the issue colocation security methods must
contend with, now and in the future
E
nterprises may want to move
to the cloud, but they still hold
plenty of IT resources. To make
this easier to handle, they are
often moving them to colocation
spaces. A 2017 survey by Vertiv
found that 57 percent of businesses were
planning to increase their data center
outsourcing, while research from Technavio
predicts that the colocation market will grow
at around nine percent through to 2022.
Meanwhile, much of the cloud is actually
held in colocation space, where giants like
AWS use sizable chunks of shared facilities to
fill gaps in their coverage.
For both these customers, the sector
offers a reduction in capital costs, the ability
to scale, and a useful geographical spread.
But there’s a downside: You have to trust the
provider that is looking after your hardware
and your data. You have to know it is kept safe
from potential attackers.
“Data center users really value security,”
says Russell Poole, managing director of
Equinix UK and the Nordics. “It’s the top
requirement of all our customers when they
are looking to deploy with us.”
A high priority for colo providers
A security incident at a colo site is just as
harmful to the customer as if it happened on
a site they owned and operated. But security
failures can be even more embarrassing for
the affected colo provider - as they represent
a failure in its core business.
A Chicago-based colo site hosted by
C I Host was broken into four times in a
two-year period between 2005-2007, with
thieves making off with tens of thousands of
14
14 DCD
DCDMagazine
Magazine••datacenterdynamics.com
datacenterdynamics.com
dollars’ worth of servers. In December 2018,
Australian telecommunications provider
Vocus came under fire when a customer
complained that a door to the facility had
been left wide open and unlocked for
months. As well as theft of the physical
infrastructure, unauthorized access to
servers could allow intruders to steal data
or make changes to the data and processes
running on that hardware.
While both of the above firms are still in
business, failing to keep a location secure
could potentially be ruinous. Even if your
terms of service are watertight when it
comes to liability for security incidents,
the loss of trust could easily lead to an
exodus of customers, especially in a highly
competitive landscape with so many
alternative providers.
“In a world where data breaches can
see a global business go under overnight,
data centers have a critical role to play
in protecting against this,” says Poole.
“The implications of a security breach are
catastrophic for the reputation of not just
the data center company, but any company
that hosts its data within the premises.”
Unique challenges
Colocation sites have all the security
concerns of in-house facilities which are
owned and operated by one organization.
But they have another big set of challenges
because they have multiple users, and
potentially a revolving door of tenants
coming and going to the site at all times.
Externally, minimal signage and
promotion around the location of a site
can reduce the chance of unexpected -
Dan Swing
Contributor
or unwanted - visitors. Perimeter fences,
generic warning signs, and minimal external
entry/exit points will help deter opportunistic
attempts at entry. Guards, barriers,
monitoring such as CCTV and potentially
access controls such as key cards will reduce
the number of people that even make it to the
front doors of the facility.
However, while securing the external
perimeter matters, internally is where much
“If you're targeting a
business that hosts at a
colo, you can take some
space there"
of the focus should be. The extra footfall
compared to an owned and operated data
center means that staff should remain extra
vigilant and more stringent controls should
be in place; employees may become used to
seeing unfamiliar faces in various parts of the
facility performing seemingly innocent work,
but in reality could be an attacker targeting a
customer.
When asked about examples of potential
attack methods on customers renting space
within colocation sites, Holly Grace Williams,
Technical Director at penetration testing firm
Secarma says one way is to simply rent space
within the same facility.
“If you're targeting a business that hosts
at a colo, you can take some space there and
gain access to the premises. Then you can
try and target other people within that space;
if there are exposed ports in cages and you
have a window of time you can probably
insert cables into those ports.”
Because of this, she says, proper
segmentation between customer areas, along
with proper monitoring and trained staff is
key.
“Colocation providers should have man
traps, which only allow one person through at
a time, and in the hosting space they should
have rack segmentation, room segmentation
and solid cages with narrow mesh.”
“You should have anti-tamper
mechanisms so you can detect when a
"The colo provider's
team needs to keep an
eye on anyone working
in rooms"
customer's rack has been opened and
integrate it with a system that tells you
instantly whether that customer's staff are
present. Lock picking and the forcing open of
cages can happen.
“The colo provider's team needs to keep
an eye on anyone working in rooms to
ensure they’re only accessing their own kit
and be there to take action if a rack has been
opened when staff from the tenant company
are not in the building.”
Within the building itself guests
should only be accepted if scheduled and
confirmed with the company, with relevant
documentation and ID in hand. Access
controls such biometrics and key cards
should be in use across the facility, with logs
to keep a history of who went where and
when. Internal monitoring such as CCTV
should also be deployed across the facility,
and manned 24/7.
“When prospective tenants visit a data
center to review its suitability,” advises
Equinix’s Poole, “they should ask themselves
‘how difficult would it be for me to get in here
if I had forgotten my fob or pass?’. The answer
to that should be, ‘impossible’.”
At Equinix sites, he explains, access is by
appointment only and entry is gained via a
series of fob and biometric, hand geometry
readers that recognize handprints and
authority from an encrypted database.
“Once inside, a sign-in procedure and
visual confirmation by trained security
guards ensures that entry is only given to
authorised visitors. Hundreds of security
cameras and hand readers are stationed
throughout the International Business
Exchanges to provide detailed surveillance
and archiving of critical infrastructure areas
and all customer cages.”
All on-site guards and staff should be
well-trained and aware of potential social
engineering attempts are important. All the
controls and defenses in the world will fail if
on-site are willing to hold doors open or flout
regular processes for a smart and determined
attacker. Ensuring that staff are confident
enough to stick to protocol even in the face
of pressure, are unafraid to ask questions
or double-check anything they are unsure
of, and be wary of attempts of manipulation
is key to ensuring attempts at social
engineering are less likely to succeed.
Regular penetration testing conducted
by both the colo provider and the tenants in
order to not only ensure that security controls
are being enacted properly but are also
effective, as well as finding potential gaps or
shortfalls and make improvements. Likewise,
customers should be encouraged to make
their own inspections and ensure the site is
up the standards they expect or require.
“There is a difference between a secure
colo and a very secure colo,” explains
Secarma's Williams, “but most people
wouldn't make that distinction on gut feeling,
they'd make it based on some compliance or
regulatory requirement.”
Issue 31Colo
• January/February 2019 15
Design Supplement
When speed of deployment is critical,
Vertiv has you covered.
Hot-scalable, isolated, power-dense critical infrastructure from the experts in digital
continuity — just in time to meet your unique needs.

With the Vertiv™ Power Module 1000/1200, you can rapidly construct redundant blocks of 1000 or 1200 kVA/kW
critical power infrastructure for your new or existing facility, giving you added capacity without overburdening IT
resources. Plus, the hot-scalable infrastructure ensures business continuity by allowing you to deploy additional
units without taking critical loads offline.

Why choose Vertiv Power Module?

yyHigh power density built around market-leading Liebert® UPS technology

yyEnergy-efficient operation with airflow containment to ensure optimal equipment conditions
yyClose to “plug and play” functionality minimizes site work and speeds deployment
yyHot-scalable units eliminate the need for downtime when boosting power capacity

Vertiv.com/PowerModule

© 2019 Vertiv Group Corp. All rights reserved.