You are on page 1of 4

2018 IEEE Canadian Conference on Electrical & Computer Engineering (CCECE)

Power Quality Characteristics and Electromagnetic


Compatibility of Modern Data Centres

Alexandre B. Nassif Yang Wang Iraj Rahimi Pordanjani


ATCO Electric University of Alberta AltaLink
Edmonton, Canada Edmonton, Canada Calgary, Canada
nassif@ieee.org yang5@ualberta.ca iraj.rahimi@altalink.ca

Abstract—Data centers have evolved to become large power loads has attracted attention of electric utilities. Some
consumers. Their supporting infrastructure includes large commercial ventures have noticed the opportunity to harvest
HVAC systems, and is in addition to massive computer and bitcoins and constructed massive data centers, which
lighting loads. Nowadays, some of these centers have been built
specifically to mine electronic currency (bitcoins). Their sheer
predominantly contain computer and lighting loads, as well as
sizes exceed those of typical large commercial customers by a associated peripheral equipment, such as Heat, Ventilation and
vast amount. With such massive sizes, their impact on power Air Conditioning (HVAC) systems. While the characteristics
quality has become an important concern and needs to be of the individual devices has been well understood [2], they
addressed by careful design. There is a need for research into suggest issues can be created in a distribution network. This is
application of innovative design and technology for conditioning because computer loads are built with an unfiltered diode-
power supply in a system supplying such data centers. This paper
presents measurements collected at the Point of Common
bridge front-end rectifier and feature large harmonic current
Coupling (PCC) between the electric utility and two large emissions. These have very high harmonic current with total
(20MW and 32MW) data centers. Their impact on harmonic demand distortion (TDD) often in excess of 100% [3].
emissions, interharmonic injection, voltage flicker and imbalance Electromagnetic compatibility standards, covered by IEEE
are analyzed. It is found that individual device standard 519 [4] and Canadian Standards Association (CSA) 61000-3-6
compliance does not translate into aggregate load standard [5], apply to a customer facility, but not to individual
compliance, and utilities need to be cognizant of these challenges
and work with the customer to strategize effective harmonic
equipment (unlike IEC 61000-3-2 [6], the European standard
mitigation. that applies to small appliances). The reason is likely the
potential impact they would have on the cost of small
Keywords—Power Quality; Harmonics; Power Electronic appliances, such as small power supplies and lights. For this
Converters; EMC; Data Centers. reason, computer and lights, which were anticipated to
compose only a small fraction of a distribution feeder, end up
I. INTRODUCTION becoming a massive harmonic source when installed in a large
The recent proliferation of residential and commercial scale data center without adequate mitigation strategy.
power electronic loads reflects modernization. Classic This paper analyzes the characteristics of two large
examples are the replacement of CRT TVs with LED sets, and (>20MW) data centers built in Alberta to mine bitcoins. The
incandescent lighting with LED lights. Power electronics- paper focuses on harmonic emissions, interharmonics (to
based home appliances are predominantly built to operate account for voltage flicker), and voltage imbalance at both
under direct current supply. As a result, most devices have a 60Hz and harmonic frequencies. It also discusses the
front-end bridge rectifier. In most cases, this bridge is built implications of European standard IEC 61000-3-2 should it be
with either diodes or transistors. The former introduces large adopted by the Canadian Standards Association. This paper is
magnitude low frequency harmonic content and behaves like structured as follows. Section II describes the two data centers
constant harmonic current injection, whereas the latter under study, and the measurement method. Section III
typically injects low magnitude low frequency harmonics, but presents the main findings on power quality disturbances.
noticeable high frequency harmonics (>3kHz), and behaves Section IV discusses electromagnetic compatibility standards.
like constant harmonic voltage supply [1]. In both topologies, Section V concludes the work.
front-end passive filtering can be applied to reduce harmonic
II. DATA CENTERS UNDER ANALYSIS
emissions. The application of these built-in filters normally
depends on economies of scale and on customer class. Two major data centers were constructed in an urban area
The sheer size of some concentrations of power electronic of Alberta and connected to two recently upgraded
substations. The PCC was defined as the primary meter

978-1-5386-2410-4/18/$31.00 ©2018 IEEE


2018 IEEE Canadian Conference on Electrical & Computer Engineering (CCECE)

location, with a voltage level of 25kV. Fig. 1. Shows the currents were recorded and the data were processed in Matlab.
generic configuration that applies to both PCCs. In both cases,
A. First Data Centre
the loads were connected through eight (8) Yg-Yg 2.5MVA
transformers with a secondary Neutral Grounding Resistor In the first data centre, it was found that the load demand
(NGR) and voltage transformation 25kV-480V. The NGR was was largely flat, about 14MW. Fig. 3 shows a sample current
used to reduce short-circuit levels for line-to-ground faults at waveform containing the three phases. This figure reveals
the 480V bus, as the PCCs are very close to the substations. current distortion and slight phase asymmetry. Fig. 4 shows
The implication is that the transformers, which were a free the corresponding spectra of the three phases.
path for zero-sequence current, become a high-impedance 500

path, which in turn reduces the zero-sequence harmonic Phase A


Phase B

emissions from the facilities.


Phase C

Current [A]
0

-500
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1
Time [s]

Figure. 3. Sample current waveform for the first data center

Figure. 1. Simple circuit to represent the data centers connection 350


Phase A
300 Phase B

Current Magnitude [A]


Phase C
Both data centres were connected near their feeder heads, 250

very close the substation and at a point of high system 200

150
strength. As a result, the voltages show very little distortion, 100

as illustrated in Fig. 2, which show a sample waveform 50

snapshot at the PCC of data centre #1 (a) and data centre #2 0


0 5 10 15 20 25

(b). In data centre #1, the voltage Total Harmonic Distortion


Harmonic order [p.u.]

(THD) was about 1%, and in data centre #2, the voltage THD Figure. 4. Sample current spectrum for the first data center
was about 2%. It was also found that the total power factor
Fig. 4 reveals the following:
(including harmonics) was about 0.9 in both cases, which is
- The 3rd and 5th harmonics are the highest measured
typical for composite loads of this nature.
harmonics in this system. This is expected.
x 10
4 - The Total Demand Distortion (TDD) is about 13%.
2
Phase A
Phase B
- There is a presence of 2nd harmonic current. This is not
1
Phase C
expected; however, the magnitudes are small.
- There is very small imbalance at 60Hz (less than 1%).
Voltage [V]

0
However, the harmonic imbalance (defined as per [7]),
was calculated to be quite large. For example, the
-1

-2
imbalance at 540 Hz (9th harmonic) is over 16%. This
is attributed to the harmonic diversity effect [8] within
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1
Time [s]

(a) the data centres.


4
x 10
Phase A
2
Phase B
25 0.35
Phase C
1
0.3
Voltage [V]

20

0 0.25
Time [s]

15
0.2
-1
10
0.15
-2
5 0.1
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1
Time [s]
0.05

(b) 0.5 1 1.5


Frequency [Hz]
2 2.5 3
x 10
4

Figure. 2. Background voltage at the 25kV PCC, (a) data centre #1, and (b) Figure. 5. Sample interharmonic spectrum for phase A of the first data center
data centre #2
The interharmonic content of the current waveform was
III. MEASUREMENT RESULTS also analyzed. To obtain interharmonic information for proper
visualization, a 1-second sliding window was applied to allow
Continuous waveform recording was carried out with a
for a 1 Hz frequency resolution. Then, each harmonic
resolution of 1024 samples/cycle, which allow harmonic
frequency ± 4 Hz was erased from the plot to hide the picket
acquisition up to 30,712 Hz (512th harmonic). A portable
fence effect due to harmonic spill-over [9]. The result is
power quality recorder was used. Three phase voltages and
2018 IEEE Canadian Conference on Electrical & Computer Engineering (CCECE)

shown in Fig. 5, which reveals that no significant The interharmonic content of the current waveform was
interharmonic component exists; any frequency component is also analyzed for data centre #2 and shown in Fig. 8. No
of very small magnitude (<< 1 A) and essentially confined significant interharmonic component exists, as any frequency
within all harmonic vicinities, meaning they are likely a result component is of very small magnitude (<< 1 A) and confined
of harmonic spill-over into adjacent frequencies. within harmonic vicinities.
B. Second Data Centre IV. ELECTROMAGNETIC COMPATIBILITY STANDARDS THAT
Like what was found for data centre #1, in the second data APPLY TO DATA CENTRES
centre it was also found that the load demand was largely flat,
about 5MW. The reason the demand is relatively low (as A. Comparison between IEEE 519 and IEC 61000-3-6
compared to the total 20MVA installed capacity), is that this The IEEE 519 [4] and the IEC 61000-3-6 [5] are two most
customer had not ramped out activities yet. Since the load is well-known standards governing the connection of harmonic
scalable, the characteristics of the load can be scaled up to producing customers. While US utilities have broadly adopted
represent its behavior under full load condition. Fig. 6 shows a the IEEE 519, Canadian utilities tend to adopt the IEC 61000-
sample current waveform containing the three phases. This 3-6, often with modifications [10]. This is reflected by the
figure reveals current distortion, and some asymmetry. Fig. 7 adoption of this standard by CSA under the same name.
shows the corresponding spectra of the three phases. The IEEE 519 prescribes voltage and current distortion
limits. Current limits compliance is generally a customer’s
200
Phase A responsibility, whereas voltage limits compliance is generally
150
the utility’s responsibility. This attribution makes sense, as the
Phase B
Phase C
100

customer is better positioned to mitigate its own harmonic


Current [A]

50

-50
0
emissions through the use harmonic filters or other means to
-100 meet the current distortion limits. There is no provision of
-150
harmonic distortion margin for future customers [10].
-200
0 0.01 0.02 0.03 0.04 0.05
Time [s]
0.06 0.07 0.08 0.09 0.1
The IEC 61000-3-6 was established based on two CIGRE
Figure. 6. Sample current waveform for the second data center publications: “Connection of Harmonic Producing
installations in High-voltage Networks with Particular
Reference to HVDC" [11], and "Equipment Producing
140
Phase A
120 Phase B
Harmonics and Conditions Governing Their Connections to
Current Magnitude [A]

Phase C
100

80 the Mains Power Supply" [12]. The objective is to set a


60 voltage distortion compatibility level at any point of a system.
40
The current injection levels from connected customers are in
20

0
turn determined so that the resulting collective voltage
distortions are below the compatibility level. Limits for the
0 5 10 15 20 25
Harmonic order [p.u.]

Figure. 7. Sample current spectrum for the second data center planning level are more restrictive than those for the
compatibility level and are recommended for system design.
Fig. 6 and Fig. 7 reveal that: The IEC standard proposes a three-stage compliance
- The 3rd and 5th harmonics are the highest measured verification approach: Stage 1 accepts small customers
harmonics in this system. This is expected. through a simplified evaluation process. Stage 2 requires
- The Total Demand Distortion (TDD) is about 7%. customers to demonstrate their compliance to the harmonic
- Even-order harmonics are of very small magnitude current injection limits. Stage 3 applies to high distortion cases
- There is some imbalance at 60Hz (about 4%), and the on an exceptional and precarious basis. Stage 2 of the IEC
imbalance at harmonic orders are higher. The 61000-3-6 is analogous to IEEE 519 current limits. A large
imbalance at 7th harmonic is about 8%, and about 26% portion of IEC 61000-3-6 explains how to estimate the current
at 15th order. This can be attributed to the harmonic injection levels for Stage 2. Specific formulas are developed
diversity effect within the data centre. for this purpose so that all customers, including future ones,
share the harmonic absorbing capacity of the supply system.
25
Current limits are estimated, rather than obtained by harmonic
0.2

0.18

20 0.16

0.14
load flow results. As a result, voltage limits might be violated
even if a customer complies with the current limits.
Time [s]

15 0.12

Alternatively, estimated limits might be conservative [10].


0.1
10
0.08

0.06
5
0.04 B. IEC 61000-3-2
0.02
0.5 1 1.5
Frequency [Hz]
2 2.5 3
x 10
4 Both aforementioned standards apply to the PCC of a large
Figure. 8. Sample interharmonic spectrum for phase A for data center #2 customer. In 2001, the IEC developed a document to address
harmonic emissions at the device level. The IEC 61000–3-2
2018 IEEE Canadian Conference on Electrical & Computer Engineering (CCECE)

addresses the harmonic current emission limits for non-linear results suggest harmonic current limit violations. Nowadays,
appliances rated < 16 A. Most appliances in such absorption two analogous standards are adopted in North America to
range take part of the class identified by the standard as class limit harmonic emissions, namely the IEEE 519 [4] and the
D [6]. IEC 61000–3-2, thus, attributes the burden of IEC 61000-3-6 [5]. A European standard IEC 61000-3-2 [6]
complying with emission limits to the individual appliance was developed to address individual appliance harmonic
manufacturer. The IEC 61000–3-2, however, is a European emissions. The adoption of [6] would not have improved the
standard that has not been adopted in North America. There is compliance situation in neither data centre, even if every
no analogous US or Canadian standard. As a result, device individual device be individually compliant. This is because
manufacturers would not be required to comply with this [6] is not effective in a massive collection of non-linear loads,
standard, unless they do so voluntarily, and with an implied
but rather in a blend of non-linear and linear loads, such as the
higher cost. Furthermore, this standard is most effective in
case in typical residential loads. The loads would, however, be
typical residential loads, where a blend of linear and non-
accepted under Stage 3 of [5], as the voltage supply distortion
linear loads exists. Should an individual nonlinear load be
checked against IEEE 519, it would violate the IEEE limits. is well within acceptable limits if connected to a very stiff
system.
C. Application of EMC Standards for the Data Centres
The short-circuit capacities at the substation where the first ACKNOWLEDGMENT
and second data centres were connected are 6,050A and The authors would like to acknowledge the support
7,060A, respectively, at the 25kV PCC. These would result in provided by ATCO electric. The following employees are
ISC/IL < 20 (ISC is the three-phase short-circuit current at the recognized: Dwayne Hoffus, Matthew Chang, Tom
PCC, and IL is the rated load at the PCC), corresponding to the Greenwood-Madsen and Julian Tong.
limits presented in Table I, from IEEE 519 [4].
REFERENCES
TABLE I. IEEE 519 CURRENT HARMONIC LIMITS (<69KV), ISC/IL < 20 [1] R. Alaei, S. A. Khajehoddin, W. Xu, “Control and Experiment of
AC/AC Sparse Modular Multilevel Converter”, IEEE Trans Power Del,
ISC/IL < 11 11”h<17 17”h<23 23”h<35 35”h TDD vol. 32, no. 3, pp. 1527-1534.
< 20 4.0 2.0 1.5 0.6 0.3 5.0
[2] A. B. Nassif, J. Yong, W. Xu, “Indices for Comparative Assessment of
the Harmonic Effect of Different Home Appliances,” Intl Trans on
Fig. 9 shows the currents for both data centres, along with Electrical Energy Systems, vol. 22, issue 5, pp. 632-654, 2012.
distortion limits, suggesting lack of compliance at harmonic [3] J. Yong, A. B. Nassif, W. Xu, “Effect of Voltage Crest Shape on the
Harmonic Amplification and Attenuation of Diode-bridge Converter-
orders 3 and 5, as well as TDD. However, the voltage based Loads,” IET GTD, vol. 5, issue 10, pp. 1033-1041, 2011.
distortion is well within limits (about %1 was measured), due [4] IEEE Std. 519-2014, “IEEE Std 519-2014 (Revision of IEEE Std 519-
to the system strength (near a high capacity substation). This 1992) - IEEE Recommended Practice and Requirements for Harmonic
means that the customer would be accepted under Stage 3 of Control in Electric Power Systems”.
[5], does not adhere to IEEE 519 limits [5] CAN/CSA-C61000-3-6-09 (R2014) - Electromagnetic compatibility
(EMC) - Part 3-6: Limits - Assessment of emission limits for the
connection of distorting installations to MV, HV and EHV power
14 systemsCAN/CSA-C22.3 NO. 9-08 (R2015) - Interconnection of
12 distributed resources and electricity supply systems.
Magnitude [%]

10
8
[6] IEC 61000-3-2:2014 Electromagnetic compatibility (EMC) - Part 3-2:
6
Limits - Limits for harmonic current emissions (equipment input current
4 ” 16 A per phase).
2 [7] P. Pillay and M. Manyage, “Definitions of voltage unbalance,” IEEE
0 Power Eng. Rev. Mag., vol. 5, pp. 50-51, May 2001.
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
42
44
46
48
50

Harmonic Order [p.u.] [8] A. Mansoor, W. M. Grady, A. H. Chowdhury, M. J. Samotyi, “An


investigation of harmonics attenuation and diversity among distributed
Data Centre 1 Data Centre 2 IEEE 519 Limits single-phase power electronic loads”, 1994 IEEE T&D Conf.
Figure. 9. IEEE 519 veryfication for both data centres [9] C. Li, W. Xu, T. Tayjasanant, “Interharmonics: basic concepts and
techniques for their detection and measurement,” Electric Power
Systems Research, Vol 66, no. 1.
V. CONCLUSIONS [10] W. Xu, “Comparisons and comments on harmonic standards IEC 1000-
3-6 and IEEE Std. 519”, 2000 ICHQP, pp. 260-263.
The evolution in power electronic-based devices, such as [11] CIGRE JTF 36.05 Report, “Connection of Harmonic Producing
LED lighting and computer loads, does not imply compliance Installations in High-Voltage Networks with Particular Reference to
HVDC - Guide for Limiting Harmonic Voltage Effects", Electra,
with EMC standards. This paper presented two measurement- no.149, Aug 1993, pp.73-87.
based case studies in two large (>20MW) data centres built to [12] CIGRE WG 36.05 Report, "Equipment Producing Harmonics and
mine electronic currenty. It was found that no interharmonics, Conditions Goveming Their Connection to the Mains Power Supply",
flicker, or imbalance issues exist at the PCCs. However, the Electra, no. 123, March 1989, pp 20-37.

You might also like