Sponsored by

> Smart Energy | Supplement

INSIDE

Super smarts Server success Demand response

> NREL turns to AI Ops for the > Finding energy savings at the > Alternate strategies for smart
ultimate system rack level batteries

Contents
4 S
 uper smarts Smart Energy: thinking
How NREL uses AI Ops to save
energy how to cut energy use
6 E
 nergy efficient servers
W
Intelligence cuts energy
ork smarter,
use
not harder.
8 E
 +I Advertorial It's a good
Busways to prevent arc flash motto for life,
but also for
10 D
 emand response data center
Data centers can save the utility hardware and infrastructure.
grid but why should they? This special supplement is about
how AI and human intelligence
13 K
 eeping up with 5G's power can be applied right now to digital
demands infrastructure, to save Watts of
5G is set to usher in higher power demand.
data transfer speeds, enabling We've mostly looked inside the
a new wave of computing facility, with a foray into the wider
technologies world.
Common sense can save lives
and equipment lost to arc flash,
according to E+I (p8).
One way to reduce that risk is to
use an intelligent design of busway,
that includes protective housing
and safety features.
A high-performance computing
site at an energy efficiency
8 laboratory was a natural place
to develop the use of AI to make
computing itself more efficient.
The US NREL institution found
out that it takes a huge amount
of data points and a lot of effort
training up algorithms, to start to
making its supercomputers run
more efficiently, and to intervene
when they work less well (p4).
Another finding is that, even
with this level of applied intellect,
it's too early to completely hand
over optimization to AI. We still
need to check the working, or else
the AI's recommendations run the
10 13
risk of being impractical or just
wrong.
Sponsored by
Human intelligence has plenty to
contribute, in redesigning racks,
servers, and cooling systems, to
reduce energy waste.
We may be approaching the
limits of efficiency achievable
through the approach of PUE,
which reduces the energy used in
cooling the facility.
The next step after that is to
concentrate on the energy used in
the IT equipment. Alex Alley tracks
the trends in today's data centers,
which may be pointing to a need
for new ideas in future (p6).
Supply and demand are well
understood in economics, but in
data centers, they are still getting to
know each other (p10).
Data centers want energy, they
want reliability, and they also want
to use renewable power sources. It's
increasingly likely that they won't
be able to get all three, unless they
start work with the utilities.
Renewables are intermittent. At
a certain point, utilities can't switch
on any more solar or wind power,
until sites with stored power start to
share it. That's the hurdle demand-
response is going to address - and
it's a challenge for data centers to
deliver on their promises to enable
green energy.
5G could change everything - but
only if we can actually provide the
electric power it needs (p13).
The history of cellular comms is
a story of continuing change and
improvement. Vlad-Gabriel Anghel
shows how we can power the next
revolution.
Smart Energy Supplement 3
 AI Ops at the scale of exascale
Super smarts
NREL’s highly efficient data center is turning to AI to prepare us for
exascale supercomputers, Sebastian Moss reports
T
he world’s most efficient
data center wants to get
even better, using artificial
intelligence to eke out more
compute power with the same
electrical energy,
Building upon a wealth of data, the
Energy Systems Integration Facility (ESIF)
HPC data center hopes that AI can make its
supercomputers smarter, and prepare us for
an exascale future.
Nestled amongst the research labs of
the National Renewable Energy Laboratory
campus in Colorado, the ESIF had an average
power usage effectiveness (PUE) of just 1.032
in 2017, and currently captures 97 percent of
the waste heat from its supercomputers to
warm nearby office and lab space.
For the last five to ten years, researchers
at NREL have used sensors to try to track
everything happening in the facility, and
within its two systems - the HPE machines
Peregrine and Eagle. This hoard of data has
grown and grown to more than 16 terabytes,
just waiting for someone to use it.
A little under three years ago, Mike
Vildibill - then VP of HPE’s Advanced
Technologies Group - had a problem. He
was in charge of running his company’s
exascale computing efforts, funded by the
Department of Energy.
“We formed a team to do a very deep
analysis and design of what is needed
4 DCD Supplement • datacenterdynamics.com
to build an exascale system that is really
usable and operational in a real world
environment,” Vildibill, now HPE’s VP & GM
of high performance networking, told DCD.
“And it was kind of a humbling experience.
How do we manage, monitor and control
one of these massive behemoth systems?”
Vildibill’s team started with a brute force
approach, he recalled: “We need to manage
and monitor this thing, we have to collect
this much data from each server, every
storage device, every memory device, and
everything else in the data center. We've got
to put it in a database. We've got to analyze
it, and then we’ve got to use that to manage,
monitor and control the system.”
With this approach in mind, the group
did a rough calculation for an exascale
system. “They came back and told me that
they can do it, but that the management
system that has to go next to the exascale
system would have to be the size of the
largest computer in the world [the 200
petaflops Summit system],” he said: “Okay,
so we’ve stumbled across a real problem.”
At the time, Vildibill was also looking
into AI Ops, the industry buzzword for the
application of artificial intelligence to IT
operations. “We realized we needed AI Ops
on steroids to really manage and control -
in an automated manner - a big exascale
system,” he said.
To train that AI, his team needed data -
lots and lots of data. Enter NREL. “They have
Sebastian Moss
Deputy Editor
all this data, not just for the IT equipment,
but for what we call the OT equipment,
the operational technologies, the control
systems that run cooling systems, fans, and
towers, as well as the environmental data.
“We realized that that's what we want to
use to train our AI.”
Armed with a data set with a whopping
150 billion sample points, Vildibill’s team
last year announced a three year initiative
with NREL to train and run an AI Ops
system at ESIF.
“Our research collaboration will span the
areas of data management, data analytics,
and AI/ML optimization for both manual
and autonomous intervention in data center
operations,” Kristin Munch, manager for
the data, analysis and visualization group at
NREL, said.
“We’re excited to join HPE in this
multi-year, multi-staged effort - and we
hope to eventually build capabilities for an
advanced smart facility after demonstrating
these techniques in our existing data
center.”
Vildibill told DCD that the project is
already well underway. “We spent several
months ingesting that data, training our
models, refining our models, and using
their [8 petaflops] Eagle supercomputer to
do that, although in small fractions - we
didn't take the whole supercomputer for a
month, but rather, we would use it for 10
minutes, 20 minutes here and there.
“So we now have a trained AI.”
The system has now progressed to a
stage, Vildibill revealed, that it can “do real
time capturing of the data, put it into a
framework for analytics and storage, and do
the prediction in real time because now we
have it all together.
“We did 150 billion historical data points.
Now we're in a real time model. That’s the
Nirvana of all of this: Real time monitoring,
management and control.”
But, for all its value, data from Eagle
and the outgoing 2.24 petaflops Peregrine
can only get you so far. Exascale systems,
capable of at least 1,000 petaflops, will
produce a magnitude more data.
“The next steps we're doing within
NREL is just to bloat or expand the data that
they're producing,” Vildibill said. “Like for
example, if one sensor gives one data point

every second, we want to go in and tweak
it and have it do a 100 per second. Not that
we need 100 per second, but we're trying to
test the scalability of all the infrastructure in
planning for a future exascale system.”
All this data is ingested and put into
dashboards that humans can (hopefully)
understand. "I could literally tell you 100
things we want on that dashboard, but one
of them is PUE and the efficiency of the data
center as a result."
As an efficiency metric for data centers,
PUE has some detractors, but it’s good
enough for NREL. “That's what NREL cares
about, but we're building this infrastructure
for customers who have requirements that
we don't even yet know,” Vildibill said.
He noted that the system "might do
prediction analysis or anomaly detection,"
and we “can have dashboards that are about
trying to save water. Some geographies like
Australia worry as much about how much
water is consumed by cooling a data center
as they do about how much electricity is
consumed. That customer would want a
dashboard that says, how efficiently they
are using their data center by the metric
of gallons per minute that are being
evaporated into the air.
“Some customers, in metropolitan areas
like New York, are really sensitive to how
much electricity they used during peak
time versus off hours because they've got
to shape their workload to try to minimize
electrical usage during peak times. Every
customer has a different dashboard. That
was the exciting thing about this program.”
It’s still early days though, Vildibill
cautioned, when asked whether the
AI Ops program would be available for
data centers that did not include HPE or
(HPE-owned) Cray equipment. “That's a
very fair question,” he said. “We're really
excited about what we're doing. We're
onto something big, but it's not a beta of a
product. It is still advanced development. So
the question you ask is exactly the very first
question that a product team would and will
ask and that is: 'Okay, Vildibill, you guys are
on something big. We want to productize
it. First question, is it for HPE or is it going
to be a product for everybody?’ And I don't
think that that decision even gets asked
until later in the development process.”
Alphabet’s DeepMind grabbed headlines
in 2016 with the announcement that it had
cut the PUE of Google’s data centers by
15 percent, and expected to gain further
savings. It also said that it would share how
it did it with the wider community, but DCD
understands the company quietly shelved
those plans as the AI program required
customized implementations unique to
Google’s data centers.
“I can tell you this - and I'm putting
pressure on the future product team that's
going to have to make these decisions -
but everything I'm describing is entirely
transferable,” Vildibill said.
“In fact, we envision this being
something that could even be picked up
by the hyperscalers. It would be very ready
for use to manage cloud infrastructure,
in addition to being used by our typical
customers, both HPC and enterprise, that
are running on-premises.
“What I'm driving with this design is
entirely transferable that I think, if it's not,
then you depreciate its value entirely.”
The six layers of AI Ops
For the AI Ops program, Mike Vildibill
breaks the system down into six layers
through which data travels.
First is the data source: "we've got to
be able to ingest historical data, capture
telemetry from servers in real-time, and so
on.”
Then comes the aggregation and
processing of that data: “We pre-process
it to make it usable. Instead of 20 different
formats or log files, we have to unify it into a
format that everyone can understand."
Third is data analytics, followed by AI
and machine learning techniques in the
fourth layer.
In the layer after that, AI begins to
predict and advise the user. For instance, if
the system sees a large number of corrected
errors on a DIMM in a specific rack, it could
recommend that node should be replaced.
Finally, layer number six will provide
automation and control: "This is the big
objective. Instead of simply advising what
a human should do, the system goes in.
For example, it turns off a node, if it has
predicted that it's going to fail.”
That final stage is still a way off, however:
“This program is really touching upon the
first five of those six, and we want to get a
couple of years of a strong prediction and
advice capability under our belt first.”
The current focus is to make the process
behind the first five layers more transparent
to humans.
"I want to be able to make sure
that, by the time a piece of advice or
prediction comes out, we have a very clear
understanding of what data led to that
prediction, so that we can go back and audit
the decisions that are being made," said
Vildibill.
"I don't think we can get all the way to
automatically controlled systems unless
humans can understand the factors that led
to a decision," he concluded.
"We can't just hand it over to the AI
and say 'I don't know what you're doing. I
hope it works out.' I think there's going to
be a lot more research involved before we
really turn the systems over to complete
automated control."
Smart Energy Supplement 5
Efficiency efforts
move into the IT
stack
Data center power consumption is costly - but servers are getting more
efficient, Alex Alley reports
D
ata centers bring together
a large number of servers
in one place, and run
applications on them.
Whether they are enterprise,
colocation or cloud data
centers, they have to operate 24x7 to support
those mission-critical applications so, as
data centers emerged, the first priority was to
build in reliability.
Once the reliability task was done, costs
and efficiency came to the fore. Those early
data centers were over-engineered and over-
cooled to ensure reliability, but it quickly
became apparent that more than half the
energy they consumed went into keeping the
hardware cool, and less than half was actually
used in computation.
Ten years of working on the efficiency
of cooling systems has given us a current
generation of facilities with a power usage
effectiveness (PUE) of 1.2 or less, meaning
more than 80 percent of the power they use
is burnt in the servers themselves.
So now, it’s time to start looking in
more detail at the power used by servers,
as a major component of the energy used
6 DCD Supplement • datacenterdynamics.com
by data centers. In February, the Lawrence
Berkeley National Laboratory co-wrote a
report commissioned by the US Department
of Energy, which revealed some interesting
statistics.
Firstly, the study confirmed an oft-
quoted rule of thumb, that data centers now
consume a small but significant part of global
energy. However, the word on the street has
been cranking up to around two percent,
the DOE report reckons it was closer to one
percent in 2018.
That sounds like a manageable figure,
but it masks areas where data centers have
become a burden. For instance, Ireland is
facing a boom in data center building, and
has a limited ability to grow its grid. The Irish
Academy of Engineering has predicted that
in 2027, 31 percent of all power on the grid
will go to data centers.
Secondly, and more interestingly, the
report shows that this overall figure is not
growing as fast as some had feared.
Over the past decade, things have
dramatically changed. In 2018, data cen­ter
workloads and compute instances increased
more than six-fold compared to 2010, yet
Alex Alley
Reporter
power usage only went up by six percent.
“Increasing data center energy efficiency
is not only an environmentally friendly
strategy but also a crucial way of managing
costs, making it an area that the industry
should be prioritizing,” Jim Hearnden, part of
Dell Technologies’ EMEA data center power
division, told DCD. “Most IT managers are
keen to increase their energy efficiency in
relation to their data center, particularly when
doing so also helps improve performance
and reduce cost.”
It’s clear that data centers have seen huge
efficiency gains - and as one would expect
from the PUE figures, the majority of these
have been in the cooling side of the facility.
But during that same eight year period, server
energy consumption went up by 25 percent.
That’s a substantial increase, although
it’s a much smaller uptick than the six-fold
increase in workloads the study noted.
It’s clear that server are also getting more
efficient, gaining the ability to handle higher
workloads with less power.
Much of this is down to more powerful
processors. We are still in the era of Moore’s
Law, where the number of transistors on a
chip has been doubling every two years, as
predicted by Gordon Moore, the one-time
CEO of Intel.
More transistors on a chip means more
processing power for a given amount of
electrical energy, because more of that
computation can be done within the chip,
using the small power budget of on-chip
systems, without having to amplify the
signals to transmit to neighboring silicon.
Moore’s Law implies that the
computational power of processors should
double every 18 months, without any
increase in electrical energy consumed,
according to an observation by Moore’s
colleague David House in 1975.
As well as in the processors, there’s been
waste energy to be eliminated in all the
components that make up the actual servers
in the data centers.
Supermicro makes “white-label” processor
“Despite people doing
more, they are doing it
with less electric power"
boards used by many large data center
builders, and it has been hard at work to
shave inefficiencies off its servers, according
to Doug Herz, senior director of technical
marketing at the company.
“The data center’s electric power
consumption in the US has started to flatten
off,” he told DCD in an interview. “It’s not
going up that fast due to a number of energy-
saving technologies. Despite people doing
more, they are doing it with less electric
power.”
Supermicro has spotted the part of the
puzzle where it can help: “Manufacturers
have not focused on idle servers and their
cost,” Herz said. “And newer management
software can aid in keeping that
consumption down.”
A five-year old server can use 175W
when it is idle, which is not that much less
than when it is in use. Idle server power
consumption has improved over recent
years, but still Herz estimates that data
centers with idle servers can be wasting a
third or even a half of the power they receive.
Newer management software can balance
workloads, distributing tasks so servers
spend less time idling. “This software is used
not only to monitor the servers in your data
center but also to load balance the servers in
your data center and optimize the electric
power,” Herz said.
“If you have a set amount of workloads
that you have to distribute over a certain
number of servers in your data center, maybe
there are more efficient ways to go about it.
Try optimizing the servers in your data center
so that you're running some of them at full
capacity. And, that way you're able to get
economies of scale.”
Further up the stack, it’s possible to
optimize at a higher level, where the server
power use shades over into the cooling. For
instance, French webscale provider OVH
takes Supermicro boards and customizes
its servers, with specially-adapted racks
and proprietary water cooling systems.
Small watertight pockets are placed on hot
components to conduct heat and transport
it away.
“It makes good business sense,” OVH’s
chief industrial officer, Francois Sterin, told
DCD. “The goal is that our server needs to be
very energy and cost-efficient.”
OVH has around 400,000 servers in
operation, and its process is just as software-
driven as Supermicro’s, Sterin told us: “We
submit a server to a lot of different tests
and environmental tests. This allows us
to measure how much energy the rack is
consuming. The goal is that our server needs
to be energy and cost-efficient.”
It’s clear that energy efficiency is now
top of mind at all levels of the data center
stack. More efficient server chips are being
managed more effectively, and used more
continuously, so they crank out more
operations per Watt of supplied power.
At the same time, those servers are being
cooled more intelligently. Liquid cooling
is ready to reduce the energy demand on
cooling systems, while conventional systems
are being operated at higher temperatures so
less energy is wasted.
We know that Moore’s Law is reaching the
end of its reign, however. Chip densities can’t
go on increasing indefinitely, and delivering
the same rate of increase in performance
power Watt.
If we’ve made cooling as efficient as
possible, and chip efficiency begins to level
out, where will the next efficiency gains be
found? One possibility is in the software
running on those processors: how many
cycles are wasted due to inefficient code?
Another possibility is in the transmission
of power. Between eight and 15 percent of all
the power put into the grid is lost in the long-
distance high-voltage cables that deliver it.
To reduce that would require a shift to a more
localized power source, such as a micro-grid
at the data center.
The data center sector has great needs,
and plenty of ingenuity. The next stage of
the efficiency struggle could be even more
interesting.
Smart Energy Supplement 7
Reducing Data Centre
Arc Flash Risk Through
Innovative Busway Design
It is estimated that more than 30,000 arc flash incidents occur each year,
resulting in an average of 7,000 burn injuries, 2,000 hospitalisations and
400 fatalities.
T
he NFPA 70E, describe
arc flash as “a dangerous
condition associated with
the possible release of energy
caused by an electric arc.”
An arc flash occurs when
a surge of electrical current caused by a
short circuit, flows through the air from
one energised conductor to another. This
results in a release of ‘incident energy’
which is expressed as an explosion of heat
and pressure into the external environment.
Despite the transient nature of arc flash
incidents, they have the potential to reach
temperatures of up to 35,000°F.
Arc Flash Risk in the Data Centre
As the demand for data increases, modern
data centres are now seeking higher power
capacities, higher rack densities and higher
efficiency designs, all of which have an
impact on arc flash risk. As power capacity
and rack densities increase, all things
being equal, so too does the available fault
current. Ironically, the quest for higher
efficiency designs can also increase
the risk of arc flash in the data centre.
Transformers represent one of the highest
losses in electrical power distribution,
however they also provide inductive
and resistive impedance that limits fault
current. To reduce losses, modern data
centres are being designed with fewer,
larger transformers compared to traditional
data centres. However, as this also reduces
electrical impedance within the power
system, the trend of higher efficiency
designs tends to increase the available fault
current and the risk of arc flash. This can
result in consequences such as lost work
time, downtime, fines, medical costs, lost
business, equipment damage; And most
importantly arc flash presents a severe
safety risk to human life.
Intelligent Medium Powerbar – Safety by
Design
E+I Engineering’s iMPB open-channel
busway solution has been manufactured
with the safety of the installer and user as
the number one priority. iMPB is the safest
and most flexible open-channel busway
system on the market due to a range of
in-built features that greatly reduce the
likelihood of arc flash in the data centre.
These features are designed to ensure
continuity of power in in a world that
demands 24/7 uptime, whilst also making
operator safety a priority. Arc flash testing
according to IEC/TR 61641:2014 has been
completed for E+I Engineering’s full iMPB
range.
Protective Housing
iMPB lengths are designed as an open track
system where tap off units can be plugged
in anywhere along the bar. The assembly
is designed to exceed the minimum ‘finger
safe’ requirements of both UL and IEC
standard, this greatly reduces any risk of
any accidental contact and hence reduces
the risk of arc flash. The copper conductors
are fully isolated from the housing using
a certified thermoplastic material, the
insulation has excellent dielectric strength

and is impact resistant. The lengths
are connected using custom designed
thermally and electrically secure joint
packs, that can easily be disassembled and
reassembled.
iMPB offers an enhanced layer of safety
by offering a closure strip that can be fitted
over the area of the busbar that is not
connected to a tap off unit. This increases
the protection to IP3X, further reducing the
risk of an arc flash.
Mechanical MCB Interlock
An MCB Safety Interlock can be integrated
into E+I Engineering’s iMPB product to
prevent the tap off unit being fitted to the
bar while the MCB is in the ‘On’ position.
Similarly, the tap off unit can only be
removed from the busbar when the MCB
is in the ‘Off’ position. The MCB can only
be switched on when the contacts are fully
engaged with the busbar. This provides
users with an extra layer of safety when
fitting/removing tap off boxes from the
busbar.
The mechanical interlock secures the
tap off box to the busbar using high tensile
strength lockable hardware which cannot
be fitted incorrectly. Once fitted to the bar,
the engager handle can be turned. This
lifts the contacts into the busbar and has a
positive lock once fully rotated.
This mechanical connection between
the tap off unit and the busway prior to any
electrical connection, ensure that there
is no risk of an arc flash incident when
installing iMPB tap off boxes to the busbar.
“Ground First, Break Last” Technology
iMPB tap off units are fitted to the busbar
using E+I Engineering’s unique ‘earth first,
break last’ safety feature. Each tap off unit
interlocks onto the distribution length with
a ground strip. This ensures that the ground
is the first point of contact with the busbar
system during installation, achieving a
lower fault current and lower fault clearance
time as excess current will always exit the
busway system through the grounding
strip.
Advertorial | E+I
Hook Operated Tap-Off Units
iMPB can be installed vertically or
horizontally depending on project
requirements. However, it is typically
ceiling mounted above the rack. For safer
operation, E+I Engineering have introduced
a hook operated tap-off unit which can be
switched on and off from the floor using
a simple hook mechanism. This adds a
further level of protection as users have no
direct contact with the tap-off box while it is
energised.
In mission critical environments, which
is constantly demanding higher levels of
power, the dangers of arc flashes can never
be completely eliminated but they can be
controlled. This is done by understanding
where the potential hazards are and taking
steps to mitigate them. In the open channel
iMPB busbar product, E+I Engineering have
implemented a range of safety features to
minimise the risk of an arc-flash incident,
preserving both operator safety and system
efficiency.
 Responding to demand response demand
Demand
response
- is there a
demand?
Data centers can play a role in cutting emissions and reducing the
strain on the grid. But why should they?
I
n a bid to reduce emissions,
renewable sources are being used
where possible - but this creates
new problems for the grid, making
it harder to match generation and
consumption. Data centers could
help to create a balance, through techniques
referred to as “demand response” but so far
it’s proven difficult to enlist their help.
All the world’s economies are attempting
to reduce carbon emissions by increasing
10 DCD Supplement • datacenterdynamics.com
the share of renewable sources in their
electricity generation, and reducing that
provided by fossil fuels. However, there are
two problems with this.
Firstly, apart from hydroelectric power,
renewables are mostly intermittent. Solar
panels and wind turbines only deliver energy
when the sun shines or the wind blows,
and can’t be switched on as required. And
secondly, the fossil fuel-powered capacity
Peter Judge
Global Editor
that is being retired is exactly the steady,
readily available capacity that the grid needs,
providing a continuous baseload, and also
extra flexible capacity as needed.
The electricity grid has to satisfy a
fluctuating demand - and there are two big
factors where long term policies designed
to reduce emissions could actually add to
the burden on the grid. Electricity is being
proposed as a replacement for fossil fuels
in cars and heating. But this will increase
the demand for electricity - and it only
reduces emissions if green electricity can
be increased to match that demand.
To respond to changing supply and
demand, the grid has to become flexible.
According to Mohan Gandhi, a civil
engineer and analyst at the New Bridge
Founders thinktank: “As intermittent
renewables penetrate further into the
generation mix, flexibility becomes an
increasingly important feature of the
electricity system.”
Some of this flexibility is based on
moving electricity to where it is needed,
but transmitting power is costly and
involves losses - and the cables may not
even be there: “Renewables are actually
being built faster than cables can be
laid,” says Gandhi. “In Germany,
wind generation in the north
has grown enormously, but the
interconnection cables between
the north and south are yet to
be built.”
Instead of moving
electricity around, another
approach is to shift demand towards
times when electricity is cheaper or
more available - an approach dubbed
“demand response.”
This can be as simple as offering
consumers a cheaper tariff for night-time
electricity (in the UK often referred to as
Economy 7). In industry, energy use is more
concentrated, and there is potential for
more advanced methods including on-site
generation and stored energy, so industrial
sites can temporarily shift their load
completely off the grid, or even become an
energy source, feeding power into the grid.
“Demand response is often the most
economical form of flexibility because
it requires few new transmission or
distribution investments,” says Gandhi. It
also has a lot of potential: The European
Commission estimates that Europe as a
whole could deliver 100GW of demand
response power (and this is expected to
rise to 160GW by 2030). However, the
European grid is currently only accessing
around 20GW of available demand response
capacity. Globally, Gandhi estimates
that 20 percent of the world's electricity
consumption will be eligible for demand-
response by 2040.
As a sophisticated and significant
electricity user, believed to be using around
one percent of the US grid’s output, digital
infrastructure can play a big role here. “Data
centers, with their real-time management
and workload flexibility, are good
candidates for demand response schemes,”
says Gandhi. “They can ‘shift’ load outside
peak hours, or deliver surplus energy stored
in their batteries and on-site generators to
the grid at times of undersupply.”
It’s been suggested that a group of data
centers could help shift demand by
migrating their loads amongst themselves
to make use of the cheapest and greenest
electricity at a given moment. “Many typical
data center workloads are delay-tolerant,
and could be rescheduled to off-peak
hours,” says Gandhi.
There are drawbacks to this. Firstly,
if a data center is running profitable
workloads, then it costs money to move
them elsewhere, and the most cost-effective
use of that resource is to run it at capacity
as long as possible. And secondly, the
customer who owns the data may need to
ensure that is processed in a given location
to comply with local regulations.
It’s actually possible for data centers
to reduce their power demands without
affecting IT workloads. Research by the
Lawrence Berkeley National Laboratory
(LBNL) found that energy consumption could
be reduced by five percent in five minutes,
and 10 percent in 15 minutes by making
changes such as setting a temporarily higher
air temperature.
Beyond this, demand response
approaches tend to use the facility’s
uninterruptible power supply (UPS). This is
designed to support the data center when
the grid fails: there is an alternative source
of power (usually diesel gensets), and some
energy storage (typically batteries) that
will support the data center while the local
power starts up.
Why not use the batteries, or even switch
to diesel for a few hours, when energy is
expensive? “When power is expensive, you
can use energy from batteries, not the grid,”
says Janne Paananen, technology manager
of energy equipment company Eaton. “This
gives savings in cost of energy. You can do
it yourself.”
Beyond the DIY approach, there are
systems managed by the utilities, which
work in a surprisingly simple way. The grid
frequency in the UK is 50Hz (plus or minus
one percent), but it varies at heavy loads.
The utilities use this to regulate the power
- the grid detects the change in frequency
and uses that to switch on extra capacity.
Because there are industries with their
own generating capacity for backup, the
electricity industry has come up with a
scheme called Firm Frequency Response
(FFR), in which those third party resources
are turned on in response to the same
change in frequency.
Data center UPS systems are designed to
switch on immediately, and can be hooked
into this sort of scheme.
FFR is in operation in Ireland and likely
to come onstream in the UK shortly. Eaton
is working with the FFR scheme in Ireland,
says Paananen. ”With fast frequency
response, you are providing services for
the grid, and getting revenue by providing
those services. Instead of responding to the
cost of energy, you respond to a real-time
signal.”
In Ireland, facilities on the FFR program
get a signal roughly once a month, says
Pannanen. “Normally the frequency
deviation lasts for only a few seconds.” This
is a level of usage that traditional lead-acid
batteries can readily support - and if FFR
takes the place of a scheduled battery test, it
can actually create less stress to the system.
These systems are proven, says
Paananen. On 8 August 2019 in the UK, two
power plants went down, the frequency of
the grid changed, and that signaled various
responses, so numerous factories and
facilities went off-grid.
All this has been possible for years, but
- as with any new idea - the big hurdles
are making it pay, and gaining users’ trust.
Utilities are prepared to offer cheaper
electricity at different times, and even pay
consumers to take themselves off-grid. But
will data centers take them up on this?
Back in 2013, Ciaran Flanagan of ABB
told DCD: “Demand response programs
(DRPs) have not only become a tool for
grid operators to manage demand, but also
a source of revenue for DRP participants.
DRPs are in operation today in many
commercial and industrial sectors but,
ironically, data centers are largely non-
participants, even though they are the
fastest-growing part of the grid’s load.”
A big reason for data centers’ reluctance
is that they make profits from continuous
availability, and sharing support systems
might increase the risk of failure.
“Participating in demand response
programs may reduce the availability or lead
to a higher risk of downtime. This risk is
exacerbated by the potential surrendering
of control to aggregators,” says Gandhi.
“Data centers are typically in the business
of avoiding downtime, minimizing risk and
maximizing availability.”
One operator put it more simply at a
DCD event: “I put in that UPS to support my
load when the data center is browning out.
Why would I share that just when I most
need it?”
“The challenge is educating the
market so they understand it is safe,” says
Paananen, “there are safety features built in.
The UPS will refuse to participate when it is
needed.”
Beyond that, the trouble is that any
revenue from demand-response is small:
“I’m not sure how much the extra revenue
is meaningful for data centers.” He suggests
that the return could be in automated
maintenance with no service charge - at
least that’s something that customers need.
Smart Energy Supplement 11
 Responding to demand response demand
In a colocation facility, an operator’s hands
may be tied. The UPS may be shared by
users, some of whom object to handing
over control to a demand response
scheme. In theory, hyperscale data centers,
with monolithic applications under one
organization’s control, have more freedom.
Paananen hopes that a different
approach in use in Nordic countries may be
a better fit for data centers. In 2017, Eaton
launched an “UPS-as-a-reserve” (UPSaaR)
service with Swedish energy supplier
Fortum, a more flexible approach in which
UPS batteries effectively act as a part of a
“virtual power plant - and get paid around
€50,000 per MW of power allocated to
grid support. Similar schemes are now in
operation in Finland and Norway.
It’s still early days, but Paananen
has faith: “Things are progressing, but
not nearly as fast as people hoped. The
challenge is if you really want to get
commercial benefits from it, you need big
batteries.” Large deals with hyperscalers will
take years to complete, he cautions.
Paananen thinks the Nordic scheme
may be the most promising. Ireland uses
FFR for “containment,” kicking in when the
frequency has dropped significantly. But
the Nordic UPSaaR schemes are more about
fine-tuning or “regulating” the system when
it is slightly out of line.
In Nordic markets, the UPS gets used
more often, for shorter periods, and with a
faster response required: “It can be only 30
12 DCD Supplement • datacenterdynamics.com
seconds, and consumes very little energy.
That is very nice for data center. The UPS
can run for less than its full design load.”
The trouble is that this kind of use
demands more modern batteries: “There’s
a more constant discharge in that scheme.
You would need lithium-ion batteries.”
It also requires critical functions in the
UPS itself: “The UPS needs to understand,
and follow external signals - and make its
own decisions.”
With all that effort, an ironic problem
can be that the utilities aren’t always keen,
says Gandhi, due to the basic laws of
supply and demand. Demand response
programs are often implemented by third-
party aggregators who have agreements
with utilities and consumers, and pool
the demands of a group of customers.
Aggregators can hook into proprietary
control systems like those of Eaton, or
add intelligence to the operation of other
vendors’ equipment.
Demand response systems led by
aggregators help the consumers reduce
consumption, and cut their costs. But
why would utilities relinquish control and
potentially lose revenue?
More subtly, as a third party, the
aggregator masks the real demand from the
utility. “There is no incentive for electricity
suppliers to include aggregators in their
contracts with customers because this
undermines many areas of their business
model,” Ghandi points out.
Some governments are stepping in to
demand the adoption of demand response,
and regulators are getting involved. “The
markets are not yet open in every country,”
says Paananen.
One aggregator that is optimistic is
Upside Energy, a UK operation that was
adopted as a partner for data center demand
response by equipment maker Vertiv.
The pair have made no big wins yet, but
Upside CEO Devrim Celal says “we are
super excited about data centers, and that
business will be increasing significantly in
the next few years.”
In the meantime, though, there’s plenty
of interest from other sectors. “There’s good
activity with cooling and refrigeration, and
behind-the-meter cogeneration,” says Celal.
But the demand response proponents
want data centers on board. Right now,
large players like Google are paying to
have more renewable energy connected to
the grid with power purchase agreements
(PPAs), but there are limits to this, says
Paananen: “At some stage, you may not
be able to use all that renewable energy.
Demand response helps the grid to get
more renewable energy in - so the only
way data centers can get green energy is by
helping the grid to get it.”
It could change perceptions, he goes
on: “Instead of DCs being a problem,
they are part of the solution, and help the
grid to adapt.”
Power efficiency
needs to keep
up with 5G's
demands
5G is set to usher in higher data
transfer speeds, enabling a new wave of
computing technologies. But this will
put new pressure on energy efficiency,
Vlad-Gabriel Anghel finds
I
n 1979, Nippon Telegraph and
Telephone (NTT) launched the first
mobile cellular network in Tokyo,
Japan. An analog system, it is now
referred to as a “first-generation” or
1G network. By 1983, NTT rolled out
coverage throughout the whole of Japan,
while other 1G networks were springing up
in Europe and elsewhere.
Motorola began service in 1983 in the
US - where Bell Labs had proposed such a
network as early as 1947, but dismissed it
as impractical. 1G had a lot of drawbacks
including poor audio quality and limited
coverage. There was no roaming support
between networks, because there were no
standards. However, it was revolutionary
and paved the way for further development
within the sector.
The next iteration, 2G was a big
improvement, using digital radio signals.
It appeared in 1991 in Finland, and was
launched under a standard - GSM - which
promised the possibility of international
roaming. Providing SMS and MMS, the new
technology was adopted widely. Despite
slow speeds and relatively small bandwidth,
2G revolutionized businesses and customers
on a scale never seen before.
3G evolved in the years leading up to
2000, with the aim of standardizing the
network protocols used by vendors thus
providing truly international roaming. NTT
Docomo launched the first 3G network in
2001. 3G offered speeds about four times
Vlad-Gabriel
Anghel
Contributor
higher than the peak possible with 2G, and
supported new protocols and solutions like
VoIP, video conferencing and web access.
This opened the packet switching era
in mobile phone communications. While
functions like Internet access struggled
at first, the launch of the iPhone in 2007
stretched 3G’s capabilities to the limit. It
was clear that 4G would be needed - and
international standards bodies including the
ITU have been working on it since 2002.
In 2009, the 4G Long Term Evolution (4G
LTE) standard got its first run in Sweden and
Norway, and was rolled out over the next few
years. 4G’s speed has enabled multiplayer
gaming on the go, high quality video
streaming and much more. However, the
protocol is plagued by network patchiness in
a lot of regions around the globe with some
suffering extremely low 4G penetration.
Enter 5G
This all paved the way for 5G, which was
being developed from the moment 4G
was delivered and is now already deployed
in certain areas. It promises massive
improvements such as allowing up to a
million devices per square kilometer - which
could revolutionize a lot of sectors.
The latest estimates from IHS Markit see
5G having an economic impact of at least $12
trillion as the focus shifts from connecting
people to information towards connecting
everything to everyone. Plus it promises an
energy efficiency revolution in the field of
Smart Energy Supplement 13
 Building 5G efficiently
mobile communication networks.
The Internet of Things era is now in full
swing, with more devices connected to the
Internet than there are people on Earth.
All these devices send and receive data on
an almost constant basis. While home IoT
applications benefit from a local wireless
network, business and industrial IoT will
require untethered connectivity over long
distances as well as enough bandwidth to
accommodate all these applications at the
same time.
Currently there are so many devices
connecting to both 3G and 4G networks
that these networks are close to breaking
point, and cannot add the growing number
of fresh devices that require perpetual
connectivity. This is where 5G comes
in. It is faster, smarter and more efficient
than its predecessors with a much higher
bandwidth.
Antennas, spectrum, and base stations
Communications networks are increasingly
finding that space, power consumption,
and emissions are major economical and
operational issues. Given the vast increase
in connected devices it is clear that energy
efficiency will be a top concern for operators
wishing to reduce overheads from both
capital and operational expenditure.
So how does 5G fare in terms of power
consumption and efficiency compared to its
predecessors?
5G’s design requirements specify a 90
percent reduction in power consumption
compared to current 4G networks - a figure
which is based on the whole ecosystem,
from the base stations all the way to the
energy used by the client device. This
massive reduction happens through a
combination of updated design practices,
hardware optimizations, newer protocols,
and smart software management of the
underlying infrastructure.
Currently mobile networks use, on
average, anywhere between 15 to 20 percent
14 DCD Supplement • datacenterdynamics.com
of their overall power consumption in actual
data traffic, with the rest being wasted
mostly by keeping the components in a
ready-to-operate state.
5G base stations will be able to go to
sleep when network activity pauses. This
is immensely important as base stations
account for around 80 percent of the power
used by a mobile network.
In current networks, the majority of base
stations may be idle at any moment. Despite
the increased number of devices and higher
data rates, 5G may have more opportunities
to power down base stations. Since 5G has
a higher data transfer rate, data transits
the network more quickly, increasing the
time when the base station is idle. 5G data
packets are more compressed, further
reducing traffic volume.
5G specifies a power
reduction of 90 percent
over 4G - from the base
station all the way
to the client device
The new network also adds a multipath
transport control protocol (MPTCP) which
reduces the need for packet replication and
retransmission and increases reliability
as more network paths are created. This is
possible because 5G uses cheap and efficient
MIMO (multiple input multiple output)
antennas. A MIMO system of antennae can
handle more clients and bigger volumes of
network activity, and increase reliability by
providing more routes for the data to take.
Should one route fail another one can take
its place.
MIMO antennas communicate with
multiple clients using focused beams of
radio waves (“beamforming”). This increases
the channel efficiency along with data
transfer rates and reduces the possibility of
interference. It also focuses radio energy
directly towards the connected device, and
can identify the exact amount of power and
energy required to further reduce energy
consumption for both the base station and
the device itself.
Furthermore, 5G makes use of smaller
network cells, covering a given area with
a larger number of smaller antennas. In
mobile networks, a cell refers to the base
station, its antennas and the physical area
they serve.
5G’s small cells are designed to be
deployed inside large buildings or outside in
highly populated areas. Power consumption
increases with the distance between the
base station and the client device (the
antenna has to “shout”), so these smaller
network cells should be deployed to keep
the communication distance as small as
possible.
Finally, through a combination of
new scheduling algorithms the spectral
efficiency of 5G New Radio is massively
improved over current networks.
In 4G networks, the signal scheduling
included a large number of control and
verification codes at regular intervals which
could consume up to 20 percent of the
network’s energy overhead during higher
frequency transmissions.
In 5G streams, the control and
verifications codes are greatly reduced. This
is because the use of smaller mobile network
cells cuts the communication distance,
reducing the chance of interference or
failure.
The scalability and flexibility of 5G
networks are increased with software
defined networking (SDN) and its related
virtualization technologies. SDN works by
decoupling the control layer of the wider
network from its data plane and merged
into a centralized control plane, which
has access and overview over the whole
network. This means that the hardware
resources can be dynamically assigned by
software to optimize traffic flow.
What does 5G actually deliver?
The 5G design specifies an almost Utopian
technology which promises an energy
efficiency revolution across the whole
mobile communication ecosystem, but
these promises should be taken with a pinch
of salt when analyzing how the technology
will perform once deployed.
The rollout of 5G to the public will be slow,
mostly because of the relatively high initial
investment required. So a full 5G coverage to
the level of current 4G deployment is within
sight but will take a few years. Full coverage is
well beyond that.
For a long time, “legacy” mobile
networks like 3G and 4G will account for
the majority of a base station’s total energy
consumption. The latest figures from Cisco’s
Annual Internet Report 2018 - 2023 further
emphasize this with “the top three 5G
countries in terms of percent of devices and
connections share on 5G will be China (20.7
percent), Japan (20.6 percent), and United
Kingdom (19.5 percent), by 2023.”
Some analysts have argued that there
may in fact be no energy savings when the
technology is rolled out - depending on
one’s definition of energy efficiency across 5G.
For example, the computational power
required to process 5G signals is around four
times that of 4G. On comparable hardware,
one would expect 5G’s data processing
component to have four times the energy
consumption. However, this is only true if
5G is rolled out on the same infrastructure
that 4G currently runs on without adopting
the increased efficiency delivered by
hardware manufacturers.
Mobile operators have been given
significant spectrum for 5G networks,
partly due to the imminent saturation of
previous networks and the rise of industrial
and commercial IoT. However efficient
5G is by design, it will add up to a further
increase in energy used by the total mobile
communication infrastructure. This will
be the case until (and if) previous network
generations are phased out.
This can have a domino effect on the
wider digital infrastructure. The growth of
5G networks comes hand in hand with the
rise and expansion of Edge data centers, so
it seems reasonable to include the energy
use of these facilities when discussing 5G’s
efficiency.
Putting critical resources closer to the
network edge can reduce latency, so 5G plus
Edge facilities may deliver individual client
interactions more effectively. However, if we
factor in the vast increase in data storage,
and processing needs of these applications
over the coming decade, it seems that 5G
will in fact simply enable an even more
power-hungry digital infrastructure
industry than today’s.
This could be potentially offset by
a close collaboration between all the
interconnecting industries, ensuring a
powerful and reliable smart grid backed up
by renewables and proper energy storage
technologies.
Searching for standards
Standards are slow to emerge, as a
consensus is needed between various
parts of the communications and energy
ecosystem. The number of bits transmitted
per Joule of energy expended is now one
of the top metrics used to analyse the
efficiency of a network.
Currently, a cellular site will deliver an
energy efficiency of around 20kbit/Joule,
and some research papers in the field
forecast that 5G could boost this more than
two orders of magnitude to 10Mbit/Joule.
The future looks even more exciting as
technologies are already starting to appear
that introduce novel ways to harness 5G.
In “radio frequency harvesting,” the
energy transmitted over radio waves can be
captured and used on the client device or
parts of the underlying infrastructure.
Since radio frequency signals carry both
information and energy, it is theoretically
possible to harvest some energy and receive
some information from the same input.
This system is known as SWIPT -
simultaneous wireless information and
power transfer. The hardware required for
this is still in development and there’s a rate-
energy tradeoff between the amount of data
and energy derived from a signal.
So SWIPT will never charge your
smartphone wirelessly, but it is a novel
approach which could offset the power
consumption required for data transmission
on the client device.
The rollout of 5G is likely to be closely
linked to the rise of the modular and
containerized “Edge” data center market,
which in turn is driven by the need to make
communication as efficient as it can be in
energy and latency terms, by placing time-
sensitive functionality at the network edge.
It’s clear that energy efficiency will be
key to all future developments in the digital
infrastructure industry.
In the next decade, we will see how 5G’s
energy efficiency - and its impact on the
world - plays out. Right now, the technology
is in its infancy and there is not enough
data to know what the impact its energy
consumption will have in real life.
As every moving part of the industry
becomes more knowledgeable and less
risk averse, 5G could bring in an era of
near instantaneous communication with
a plethora of new industries and markets,
making every effort to minimize their
impact on the world, through collaboration
and smart infrastructure deployment and
management.
The roll out of this Edge-based technology
will bring huge changes to data centers and
other infastructure.
But for that to happen, companies have to
ensure that efficiency is paramount, as they
rush to compete for the sector.
Smart Energy Supplement 15
 INTELLIGENT
MEDIUM POWERBAR
Delivering power safely and
efficiently in mission critical
environments.

E+I Engineering's innovative iMPB

product is an open channel busway
system designed for use in data centres
and other mission critical environments.
E+I engineering have completed iMPB
installation in data centres across the
globe where security and flexibility of
electrical distribution is paramount.

iMPB has been engineered with the

safety of the installer and user in mind.

For more information about our full range of products

please contact us at info@e-i-eng.com

Donegal, Ireland | South Carolina, USA | Ras al-Khaimah, UAE

WWW.E-I-ENG.COM