2015 IEEE/SICE International Symposium on System Integration (SII)
December 11-13, 2015. Meijo University, Nagoya, Japan
Distribution Control Systems for Data Center
Montri Wiboonrat
College of Graduate Study in Management
Khon Kaen University, Bangkok Campus
Bangkok, Thailand
mwiboonrat@gmail.com, montwi@kku.ac.th
Abstract—System uptime is the crucial objective of data center
operations. Distribution control system (DCS) proposes design
topologies and attributes for critical network, electrical, and
mechanical systems to attain these availability. Data center
infrastructure management (DCIM) is hardware and software
tools for monitor, measure, control, manage performance,
utilization and energy consumption through direct digital
control (DDC) devices. DCIM provides DCS function for all IT-
related equipment (demand side) and facility infrastructure
components (supply side). This paper was derived from
standard changed requirements from Tier Classification which
will be affected data center design in the future. Understanding
integrity constraints are the key to optimize demand side and
supply side through DCIM distribution control system for data
center. This paper purposes autonomous demand and supply
response model through DCS for increasing energy efficiency
and resource utilization while sustains data center operations
and protect critical loads.
Keywords—distribution control systems; direct digital control;
data center; data center infrastructure management
I. INTRODUCTION
Data centers are highly complex things, and complex
things can quickly become unmanageable [4][5][6]. Modular
design allows you to create highly complex systems from
smaller, more manageable building blocks. These smaller
units are more easily defined and can be more easily replicated
[7]. They can also be defined by even smaller units, and you
can take this to whatever level of granularity necessary to
manage the design process. Tier IV fault tolerance system is
designed for autonomous response to a failure and load
demand which is more than just engineering the most
technically correct or best system [1][3]. However, customers
may want is not the best system, they really need is the most
practical system that is focused on and addresses our particular
basics.
Data center facility infrastructure has performed as the
foundation of information technology that facility
infrastructure has been evolutional throughout the last several
years and transformational after the beginning of the 21 st
century. The data center life cycle is around 10-15 years [10].
However, within 5 years technology will be changed, thus
most data center design is flexible and easily upgradable.
Therefore, it is critical to a successful long-term design.
978-1-4673-7242-8/15/$31.00 ©2015 IEEE 789
Strategic planning of data center is not an attempt to
eliminate risk or to forecast what the data center will look like
in fifteen years, it is taking action to understand what risks to
take and what paths will align with the business. Partial
flexibility concept is advantage the design cost-effective. Tier
classification design decision has as impact on the investment
and long-term operations. Designing a cost effective data
center is greatly dependent on the mission critical
applications of the enterprise. Data center design and build
can be classified into 2 concepts, design for scalability or
modular design. For this reason, data center project (DCP)
requires different strategic approach [2]. It is differed from
enterprise to enterprises. Without proper DCP, a design and
build project can experience delays, increased costs and
frustrated team members. Regardless of project size, activities
need to be structured and standardized so everyone is on the
same page and the project outcome is predictable. Strong
project management makes sure nothing slips through the
cracks or any effort is duplicated. When building or
upgrading a data center’s physical infrastructure, project
management provides the foundation for a successful result.
Its purpose is to structure the project into phases, coordinate
the work of the entire team and monitor progress. A typical
team might consist of the end users, consultants, hardware
and software providers, general contractor and more.
The new trend of data center infrastructure management
(DCIM) was just emerging during the past a few years.
DCIM solution provides measurable to improved system
availability, capacity planning for provisioning, energy
efficiency, resource utilization, and productivity for remote
management. DCIM as a tool for monitor, measure, manage
and/or control data center utilization and energy consumption
between demand and supply sides of all IT-related equipment
(such as servers, storage and network switches) and facility
infrastructure components (such as power distribution units
(PDUs) and computer room air conditioners (CRACs).
This paper was derived problems from the new
requirements from Tier Classification [3] subject to; ignoring
main power utility sources (independence/supply); adding
continuous cooling system by Tier 4 needs to provide makeup
water up to 12 hours; and autonomous response to a failure
and critical load. The solution of imply DCIM to adjust
demand and supply response was presented in this paper as a
guideline for compliance with Tier Classification or BISCI-
002 [2].
II. BACKGROUND
A. Tier Classification
The purpose of data center site infrastructure standard is
compliant business needs and requirements for design
professionals, facility installation, and operations with
objectives and effectiveness imply for identifying the
anticipated performance of different data center site
infrastructure design topologies, as illustrated in Figure 1.
2. Where is the best place to place a new server?
3. Do I have sufficient space, power, cooling and
network connectivity to provide my needs for the next
6 months? Next year? Next five years?
4. An event occurred in the data center — what
happened, what services are impacted, where should
the technicians go to resolve the issue?
5. Do I have underutilized resources in my data center?
6. Will I have enough power or cooling under fault or
maintenance conditions?

Small Businesses
99.671% Uptime
28.8 Hours Downtime Per Year
No Redundancy
1 12 Hours Engine-Generator Protection

Medium-size Businesses
99.749% Uptime
22 Hours Downtime Per Year
Partial Redundancy in Power and Cooling
Critical Components
2 12 Hours Engine-Generator Protection

Large Businesses
99.982% Uptime
1.6 Hours Downtime Per Year
N+1 Concurrent Maintenance
12 Hours Engine-Generator Protection
3
Enterprise Corporations
99.995% Uptime
2.4 Minutes Downtime Per Year
2N+1 Fully Redundant with Compartment
and Automatic Fault Tolerance
12 Hours Engine-Generator Protection
4 12 Hours Continuous Cooling

Figure 1. New requrements for Tier classification.

1) Tier 3: Concurrently Maintainable Site Infrastructure

A concurrently maintainable data center requires Figure 2. Tier 3 topology.
redundant critical components and multiple independent
distribution paths serving the critical environment. Only one
distribution path is required to serve the critical load at any
time and twelve hours of on-site fuel storage for ‘N’ capacity
[7], as shown in Figure 2.
2) Tier 4: Fault Tolerant Site Infrastructure
A fault tolerant data center requires multiple, independent,
physically isolated systems (compartment) that provide
redundant capacity components and multiple, independent,
device, active distribution paths simultaneously serving the
critical loads. The redundant critical components and diverse
distribution paths shall be configured such that ‘N’ capacity is
providing power and cooling to the critical loads after any
distributed path failure [7], as demonstrated in Figure 3.
B. Data Center Management Components
DCIM is the convergence of previous generations of
purely facilities-oriented power management, physical asset
management, network management, and financial
management and planning solutions for data centers. There are
many benefits for implementing a DCIM solution. The
primary components of data center management consist of:
Figure 3. Tier 4 topology.
design, operations, monitoring, predictive, and planning, as
depicted in Figure 4. One of the primary benefits of DCIM is
the ability to answer questions such as the following [15]:
1. Where is my data center asset located?

790
 1.5% of condenser water flow to sustain evaporative cooling
process [9].
C. Data Center Infrastrucutre Management (DCIM)
The autonomous response to a failure and load demand is
one of the newest technologies that apply for monitoring and
control, energy efficiency, and resource utilization inside data
center or called “Data Center Infrastructure Management or
DCIM”. DCIM defines as systems collect and manage data
refer to a data center assets, resource usage, and operational
status throughout data center operational lifecycle. This
information is distributed, integrated, analyzed, and applied
over facilities, networking, and systems in purposes that help
managers or operators meet business and service-oriented
aims and optimize the data center performance, as presented in
Figure 5. DCIM activities are performed through functions of
Figure 4. Data center management components [15]. monitoring and alarming, automation and control, modeling
and simulation, software services, and integrations.
III. NEW STANDARDS COMPLIANCE
In this paper, research was concentrated on Tier 3 and Tier
4. Since 2014, the new requirements of Data Center Tier
Classification was changed by ignoring main power utility
sources (independence) and adding continuous cooling system
by Tier 4 needs to provide makeup water up to 12 hours.
Moreover, the infrastructure controls system demonstrates
autonomous response to a failure while sustaining the critical
loads. Therefore, all new data center infrastructure changes
need to comply if data center owners prefer a certification
body from Uptime Institute.
A. Engine Generators Physical Functional Process
• Asset management
Engine generators and their ratings are governed by •
•
Power system
Cooling system
•
•
Monitoring and alarming
Automation and control
•
•
Availability management
Capacity management
International Organization for Standardization (ISO) Standard •
•
IT space
Security
•
•
Modeling and simulation
Software Services,
•
•
Change management
Crisis management
8528-1 [12]. This standard covers reciprocating internal • integrations • Integrations •
•
Financial management,
Integrations
combustion (RIC) engines, alternating current (AC)
generators, and associated systems. Uptime Tier Classification
for Tier III and Tier IV requires only continuous power that Figure 5. The physical layer of data center infrastructure [17].
the maximum power for which an engine-generator is capable
of delivering continuously for a constant load for an unlimited IV. DCIM INTERGRATED TECHNOLOGY
number of hours [8]. Continuous power (COP) means the The integration will provide an increased level of
engine-generator is able to provide power to a non-varying communications between IT Service Management (ITSM) and
load for an unlimited number of hours per year. The average DCIM platforms, providing data center managers with more
power output of the engine-generator is 70-100 percent of the efficient management of data center assets subject to, capacity
rating. The rating is designed to provide 100 percent of the management, automated provisioning, cost management,
rating for 100 percent of the data center operating hours [13]. service catalogs based, analytics and optimization, and
B. Makeup Water availability, as demonstrated in Figure 6.
Since, engine generators are designed for continuous A. DCIM Components and Functions
operations for minimum 12 hours therefore; loss of public The asset management is a key component of DCIM. A
utility power is a design condition rather than a failure mode, data center can contain thousands of assets, from servers,
as the same time, the loss of public utility water must also be storage and network devices to power and cooling
considered as a design condition. This is the critical problem infrastructure equipment. Tracking these assets is an ongoing
for evaporative cooling situations because of the volume of and often monumental task. A DCIM can be hardware-based
water required to maintain functional water level in cooling and/or software-based and is used to monitor a data center or
towers. For assumption each 1 megawatts (MW) of cooling computer room. Direct digital control (DDC) or device
load (approximately 285 refrigeration tons) requires about 855 communication is typically done using SNMP, although some
gallons per minute (gpm) of condenser water flow through the data center monitoring systems can also communicate using
evaporative cooling towers. The estimating evaporation Modbus, IPMI or other protocols, as a part of IEC 61850 [20].
consumes about 1% and 0.5% for drift consumes, thus a This attribute is considered as a real-time monitoring for
source of water is required to replenish, or “makeup” around utilization of hardware component (IT and Infrastructure

791
Systems), as shown in Table I. Moreover, a real-time TABLE II. DCIM SOFTWARE VENDORS [15]
monitoring is not only analyzed capacity planning but also
applied for provisioning and maintenance to reduce mean time
to repair (MTTR) or increasing uptime.

TABLE III. DCIM HARDWARE VENDORS [15]

Figure 6. Integrated DCIM and ITSM infrastructure [16].

TABLE I. DCIM COMPONENT AND FUNCTIOINS

C. Demand and Supply Response through DCIM

Online data collection is a base for DCIM function blocks.
The data is used as input to the other functional areas,
including Asset and Change Management, Environmental
Monitoring, Power and Energy Measuring and Modeling,
Power Management and IT Service and Systems
Management. A data management layer integrates data from
the lower layers to facilitate reporting as well as providing
input to higher level planning, forecasting and optimization
layers, as represented in Figure 7. Figure 8 presents DCIM
distributed systems among facility network, IT management
network, data center network, and enterprise network.

B. DCIM Software and Hardware Vendors

IEC 61850 is a standard for the design of electrical
substation automation [20]. All DCIM vendors must be
complied with this standard. They are many DCIM software
and hardware vendors in the market. However, compatibility
of products is the primary concern when select DCIM
software and equipment rather than investment and Figure 7. Demand and supply response through DCIM [16].
maintenance. Table II and III are listed vendors on this DCIM
business.

792

DCIM DCIM
DCIM
DCIM DCIM
DCIM
Figure 8. DCIM topology [14].
V. ANALYSIS DEMAND AND SUPPLY RESPONSE
Since 2014, the new requirements of Tier Classification
are changed that effect to the new design of data center which
require Tier Certification. This research was investigated on
factors that impact on data center design especially on Tier 4
which were required continuous cooling systems and
autonomous response to a failure and load demand.
A. Continuous Cooling System
Tier 4 is the only one that requires continuous cooling.
Continuous cooling for a chilled water system is generally
accomplished with thermal energy storage (TES) capability.
Secondary pumps and computer room air handlers (CRAHs)
are required to be on UPS. This can be the IT UPS or a
separate, concurrently maintainable and fault tolerant,
mechanical UPS system. If the cooling system is in a
primary-direct configuration, then the primary pumps are
required to be on UPS. Moreover, for chilled water system, it
needs to consider makeup water to compensate the
evaporative cooling process. Continuous cooling for direct
exchange (DX) systems requires both the computer room air
conditioners (CRACs) and the external condensers to be on a
concurrently maintainable and fault tolerant UPS system [11].
Table IV shown about heat load of server power trends or
demand cooling requirements (Racks/Rows/Rooms) for
calculation continuous cooling or cooling supply from
chillers.
TABLE IV. HEAT LOAD OF SERVER POWER TRENDS [18]
management and energy management, with a focus on IT
assets and the physical infrastructure needed to support them,
as presented in Figure 9. Tier 4 requires autonomous response
to failure, which encompasses the ability to detect a
fault, isolate the fault and sustain operations with
alternate systems that mean real-time monitoring and control
will handle on this data before alert and distributed the signal
to data center managers or other involved persons or other
systems such as building automation system (BAS) increase
of fire alarm or water leak. The function of autonomous
response to failure is not only prevented cascading failure
from component to system but also reduced MTTR of
equipment and systems. This is direct function of DCIM
rather than assess management. The simple test of fault
tolerance for fuel is that ‘N’ or the needed amount of fuel for
the data center, is available after any failure. Some
failure scenarios of fuel systems include loss of power, leaks,
failure of components or fire. Leak detection of fuel systems
is crucial to fault tolerance. Fault detection may be achieved
using a variety of methods, but the chosen method must be
integrated with the system response to isolate the fuel leak.
The isolation must limit fuel loss such that the ability of
alternate systems to deliver ‘N’ fuel is not impacted. The
Figure 10 shows typical system architecture used for an
integrated facility management system that integrates with
various network, electrical and mechanical through DDC of
sub system vendors, and providing a complete building
management system (BMS), property management system
(PMS), and energy management system (EMS) functionality.
Figure 9. DCIM supports on demand and supply response [19].

B. DCIM for Autonomous Response

DCIM enables IT professionals to understand energy
consumption from a holistic data center perspective subject to
power utilization (watts), heat rejection (watts), space
occupancy, and future resource provisioning. DCIM toots
integrate facets of system management with building
Figure 10. Example of DCIM system configuration through DDC [21].

793
 VI. DISCUSSION
DCIM has economic value depends on sizing of monitor
and control points in data center (number of racks) and
compliant with Tier Standard. This paper reveals Tier 1-3 no
need to comply with fault tolerance procedure only Tier 4
required as statement “site must be designed and operated to
tolerate the cumulative impact of every site infrastructure
component, system, and distribution path disrupted by failure
[22],” that implied Tier 4 needs DCIM to define, measure,
analyze, improve, and control in operation processes.
However, investment in DCIM increases capital expenditure
(CapEx) and long-term operational expenditure (OpEx) that
reflects on total cost of ownership (TCO) and return on
investment (ROI).
VII. CONCLUSION
Data center design for Tier 3 and Tier 4 Certification
needs to comply with all new Uptime Tier Classification
standard requirements. The three major requirements changed
were; the main power utility sources (independence) is
generators 2(N+1) with continuous type not power utilities;
adding continuous cooling system by Tier 4 needs to provide
makeup water up to 12 hours, both chiller water (needs
makeup water) or direct exchange (DX) models; the
infrastructure controls system demonstrates autonomous
response to a failure while sustaining the critical loads. These
three requirements need to fulfill by data center infrastructure
management (DCIM) with function of demand and supply
response through distribution control system (DCS) function
of DCIM. The objectives of DCIM are brings the entire data
center operation into a unified view while system allows data
center operators and managers to perceive information across
multiple disciplines that enable them to make decisions.
Moreover, DCIM is designed for monitor, measure, manage
and/or control data center utilization and energy consumption
between demand and supply sides to improve power usage
effectiveness (PUE). The intelligent functions of DCIM help
data center operations subject to resource utilization, energy
efficiency, preventive maintenance, improve uptime, reduce
operation costs, and easy to management.
REFERENCES
[1] TIA 942, Telecommunications Infrastructure Standard for Data
Centers, Telecommunictions Industry Association (TIA), 2014.
794
[2] BICSI-002, Data Center Design and Implementation Best Practices,
BICSI 002-2014, 2014.
[3] TS102120-0812, “Data Center Site Infrastructure Tier Standard:
Topology,” Uptime Institute, LLC., 2012.
[4] M. Wiboonrat, “An Optimal Data Center Availability and Investment
Trade-Offs,” Software Engineering, Artificial Intelligence, Networking,
and Parallel/Distributed Computing,” SNPD 2008, The 9th ACIS
International Conference on, pp. 712-719, 2008a.
[5] M. Wiboonrat, “Transformation of system failure life cycle,” Int. J. of
Management Science and Engineering Management, Vol. 4, No. 2., pp.
143-152, 2008b.
[6] M. Wiboonrat, “An Empirical Study on Data Center System Failure
Diagnosis,” The 3rd International Conference on Internet Monitoring
and Protection, pp. 103-108, 2008c.
[7] M. Wiboonrat, “System Reliability of Fault Tolerant Data Center,” The
5th International Conference on Communication Theory, Reliability,
and Quality of Service. Chamonix/ Mont Blanc. France. 29 April- 4
May, 2012.
[8] ATD104100-110, “Accredited Tier Designer Technical Paper Series:
Engine-Generator Ratings,” Uptime Institute, LLC., 2010.
[9] ATD101100-110, “Accredited Tier Designer Technical Paper Series:
Makeup Water,” Uptime Institute, LLC., 2010.
[10] TS106130-0413, “Data Center Site Infrastructure Tier Standard:
Operational Sustainability,” Uptime Institute, LLC., 2013.
[11] UI109110-411, “Accredited Tier Designer Technical Paper Series:
Continuous Cooling,” Uptime Institute, LLC., 2011.
[12] ISO Std. 8528-1:2005, Reciprocating Internal Combustion Engine
Drive Alternating Current, Generating Sets, 2005.
[13] R. Marfell, “Demystifying Generator Set Ratings,” Caterpillar, October
2012.
[14] Decathlon, “Data center infrastructure management: Operational
intelligence for agile business,” ABB Inc., 2013.
[15] D. Cole, “Data Center Konwledge Guide to Data Center Infrastructure
Management (DCIM),” Data Center Knowledge, May 2012.
[16] A. Lawrence and R. Ascierto, “The DCIM marketplance: Suppliers,
products and ecosystems,” 451 Research, 2013.
[17] Raritan, “An Introduction to Data Center Infrastructure Management,”
Raritan Inc., 2010.
[18] ASHRAE, “Datacom Equipment Power Trends and Cooling
Application,” 2nd Edition, 2012.
[19] Siemens, “Leveraging DCIM to Increase Engergy Efficiency and
Savings,” Siemens AG, August 6, 2013.
[20] IEC 61850, Communication Networks and Systems in Substations,
Technical Committee 57, IEC, 2010.
[21] Decathlon, “Data Center Infrastructure Management: Decathlon
System Guide,” ABB Pte. Ltd., January 11, 2015.
[22] W. P. Turner, J. H. Seader, and V. E. Renaud, “Data Center Site
Infrastructure Tier Standard: Topology,” Uptime Institute, LLC., 2012.