energies

Article
Thermal Performance Evaluation of a Data Center
Cooling System under Fault Conditions
Jinkyun Cho 1, * , Beungyong Park 2, * and Yongdae Jeong 2
1 Department of Building and Plant Engineering, Hanbat National University, Daejeon 34158, Korea
2 Energy Division, KCL (Korea Conformity Laboratories), Jincheon 27872, Korea
* Correspondence: jinkyun.cho@hanbat.ac.kr (J.C.); bypark@kcl.re.kr (B.P.)




Received: 12 July 2019; Accepted: 31 July 2019; Published: 3 August 2019


Abstract: If a data center experiences a system outage or fault conditions, it becomes difficult to
provide a stable and continuous information technology (IT) service. Therefore, it is critical to
design and implement a backup system so that stability can be maintained even in emergency
(unforeseen) situations. In this study, an actual 20 MW data center project was analyzed to evaluate
the thermal performance of an IT server room during a cooling system outage under six fault
conditions. In addition, a method of organizing and systematically managing operational stability
and energy efficiency verification was identified for data center construction in accordance with the
commissioning process. Up to a chilled water supply temperature of 17 ◦ C and a computer room air
handling unit air supply temperature of 24 ◦ C, the temperature of the air flowing into the IT server
room fell into the allowable range specified by the American Society of Heating, Refrigerating, and
Air-Conditioning Engineers standard (18–27 ◦ C). It was possible to perform allowable operations for
approximately 320 s after cooling system outage. Starting at a chilled water supply temperature of
18 ◦ C and an air supply temperature of 25 ◦ C, a rapid temperature increase occurred, which is a serious
cause of IT equipment failure. Due to the use of cold aisle containment and designs with relatively
high chilled water and air supply temperatures, there is a high possibility that a rapid temperature
increase inside an IT server room will occur during a cooling system outage. Thus, the backup
system must be activated within 300 s. It is essential to understand the operational characteristics
of data centers and design optimal cooling systems to ensure the reliability of high-density data
centers. In particular, it is necessary to consider these physical results and to perform an integrated
review of the time required for emergency cooling equipment to operate as well as the backup system
availability time.

Keywords: data center; cooling system outage; temperature control; thermal performance evaluation;
under fault condition; CFD thermal model

1. Introduction
Over the past decade, data centers have made considerable efforts to ensure energy efficiency and
reliability, and the size and stability of their facilities have been upgraded because of the enormous
increase in demand [1,2]. Currently, the amount of data to be processed is expanding exponentially
due to the growth of the information technology (IT) industry, and data center construction is on the
rise to meet this demand. If a data center experiences a system outage or fault conditions, it becomes
difficult to a provide stable and continuous IT service, such as internet, banking, telecommunication,
broadcast, etc., and if this situation occurs on a large scale, it can even lead to chaos in the finance
industry, the stock market, telecommunications, and the Internet. Therefore, it has become critical
to design and implement backup and uninterruptible power supply (UPS) systems so that system
stability can be maintained even in emergency situations.

Energies 2019, 12, 2996; doi:10.3390/en12152996 www.mdpi.com/journal/energies

Energies 2019, 12, 2996 2 of 16

As shown in Figure 1, considerable technological infrastructure is required in data centers to

meet even the minimum requirements for Tier 1 system availability (96.671% uptime per year), which
is the basic level. In specialized colocation data centers for business purposes, enormous system
construction costs are required for uninterrupted system operation to ensure Tier 3–4 high availability
(95.995–99.982% uptime) [3]. On the other hand, interest in environmental problems such as global
warming and the demand for low-energy and high-efficiency buildings is increasing, so the use of
excessive facilities in the pursuit of reliability alone must be avoided. Considering these facts, ensuring
stable and efficient performance is an important aspect of data center design and implementation.
Conventional construction methods are focused on tangible results such as the construction period
and financial costs (not quality) necessary to set up the data center infrastructure according to the
design plans, which can make it difficult to verify whether non-IT equipment is demonstrating proper
performance or the design is operating as intended. Furthermore, it is not immediately apparent
when equipment with performance problems is installed, and it can be difficult to detect the problems.
Even if the problems are detected, it is difficult to determine clearly who is responsible between the
major contractor and vendors; thus, a warranty cannot be requested. Data centers strengthen their
stability by having redundant power supply paths, including emergency generators, UPSs, etc. IT
servers require uninterruptible supplies of not only power, but also cooling [4–6]. For this purpose,
central cooling systems are designed to allow for chilled water supply during cooling system outages
by including cooling buffer tanks for stable cooling of IT equipment. If the chillers are interrupted,
the emergency power and cooling system are activated and then the chilled water is supplied again.
Consequently, the mission-critical facility design for the stable operation of data centers leads to huge
cost increases, so careful reviews must be performed from the initial planning stage [7–9]. Considering
that the number of times that such emergency situations occur during the life cycle of a data center
building is very small and the tolerance of IT servers to various operational thermal environments
has improved vastly compared to that in the past due to the development of IT equipment, there is
considerable room for reducing the operating times and capacities of chilled-water storage tanks. In the
global data center construction market, there has been a trend of expanding the commissioning process
to verify designs through expert knowledge on stability and efficiency. The datacenter is equipped
with a UPS and power generator. As the city power goes out, the UPS instantly kick in before starting
the emergency power generator and no service is affected. Once a few seconds have passed to allow
the generator to warm up, the power transfer switches the source of power from city to emergency
generator. Most UPS are installed for IT equipment, not cooling systems. When power outages
occur, the cooling system has critical issues for stable IT environments before supplying emergency
power. This paper proposes a method of organizing and systematically managing operational stability
and energy efficiency verification for data center construction in accordance with the commissioning
process. This research aims to apply the IT environmental principle of maximum output from minimum
input to emergency back-up system for cooling with UPS for chilled water circulation pump and
buffer tank. The available time needed to maintain the IT environment by using the potential cooling
energy of chilled water without a mechanical cooling system, was analyzed. In a case study on a
currently completed 20 MW data center, we assumed an emergency situation in the cooling system
and evaluated the subsequent temperature increases of the chilled water and the indoor temperature
changes in the IT server room. For this purpose, we analyzed the thermal environment of the data
center by employing a computational fluid dynamics (CFD) model and quantitatively examined the
degree to which the environment inside the IT server room was maintained by secondary chilled water
circulation when the central cooling plant was interrupted. The goal was to increase the reliability of
the system by using the analysis results to set an initial cooling response limit time and determine an
appropriate chilled-water storage tank size.
Energies 2019, 12, 2996 3 of 16
Energies 2019, 12, x FOR PEER REVIEW 3 of 16

Figure 1. Data center tier standards for evaluating the quality and reliability of the server hosting
Figure 1. Data center tier standards for evaluating the quality and reliability of the server hosting
ability of a data center [3].
ability of a data center [3].
2. Literature Reviews
2. Literature Reviews
In recent years, a small number of theoretical studies have been conducted on data center cooling
systems In under
recentfault
years, a small including
conditions, number of theoretical
system thermal studies
and energyhave performance,
been conducted system on distribution
data center
cooling systems
optimization, andunder fault conditions,
simulation study. Kummert including system
et al. thermal the
[10] studied andimpact
energy of performance,
the air and system water
distribution optimization, and simulation study. Kummert et al.
temperatures in the chiller system failure by using the TRNSYS simulation. The cooling plant and air [10] studied the impact of the air
and water temperatures in the chiller system failure by using the
temperature levels of data centers have made it possible to evaluate the design of cooling systems in TRNSYS simulation. The cooling
plant and
response to air temperature
system failures by levels
power of outage.
data centers Zavřelhave
et al.made it possible
[11] analyzed to evaluate
a support the design
emergency power of
cooling systems in response to system failures by power outage.
planning by using a building performance simulation and considered keeping the server room at an Zavřel et al. [11] analyzed a support
emergency temperature
appropriate power planning in thebyevent
usingofa abuilding performance
power outage simulation
at the data center. andA case considered
study was keeping
analyzed the
inserver
detailroom at emergency
for the an appropriate temperature
cooling possibility. in Lin
the etevent of adeveloped
al. [12] power outage at the data
a transient center.
real-time A case
thermal
modeling to demonstrate the data center cooling system at the loss of utility power and evaluated thea
study was analyzed in detail for the emergency cooling possibility. Lin et al. [12] developed
ITtransient
environmentreal-time thermal modeling
characteristics to demonstrate
of air temperature rise. Inthe datatocenter
order achieve cooling system at
the necessary the loss of
temperature
utility power and evaluated the IT environment characteristics
control during power outages, the appropriate method is recommend depending on the characteristics of air temperature rise. In order to
of each cooling system. Lin et al. [13] provided a practical strategy to control cooling and proposedis
achieve the necessary temperature control during power outages, the appropriate method
recommend
the main factors depending
related to onthethetransient
characteristics of each
temperature cooling
rise duringsystem. Lin et al.
power supply [13] provided
failure. Moss [14]a
practical strategy to control cooling and proposed the
investigated how quickly an IT facility might heat up and what risk the data center might main factors related to the transient
be at.
temperature rise during power supply failure. Moss [14] investigated
Considering the large energy differences associated with running very cold data center temperatures, how quickly an IT facility
itmight
is notheat up and whatstrategy
a recommended risk thetodata center
extend might be at.
ride-through Considering
time. Complacency the large
couldenergylead todifferences
the belief
associated with running very cold data center temperatures,
that the facility has more time than required to get data center running again, but this can it is not a recommended strategy
be risky. to
extend
Gao ride-through
et al. [15] unveiled time.
a new Complacency
vulnerability could lead to the
of existing databelief
centersthatwith
the aggressive
facility has cooling more time energythan
required to get data center running again, but this can be risky.
saving policies, conducted thermal experiments, and uncovered effective thermal models at the data Gao et al. [15] unveiled a new
vulnerability of existing data centers with aggressive cooling energy
center, rack, and server levels. The results demonstrated that thermal attacks can largely increase the saving policies, conducted
thermal experiments,
temperature and uncovered
of victim servers degrading effective thermal models
their performance at the datanegatively
and reliability, center, rack, and server
impacting on
levels. The results demonstrated that thermal attacks can largely
thermal conditions of neighboring servers causing local hotspots, raising the cooling cost, and increase the temperature of victim
even
servers to
leading degrading their performance
cooling failures. Nada et al.and [16]reliability,
studied, with negatively impacting on
a physical-scaled thermal
data centerconditions
model, the of
neighboring
control servers
of the cold causing
air flow rateslocal
alonghotspots,
the servers raising
for thethepossibility
cooling cost, and even the
of controlling leading to cooling
heterogeneous
failures. Nada et al. [16] studied, with a physical-scaled data center
temperature distributions. Torell et al. [17] conducted data center cost analysis and demonstrated model, the control of the cold air
flow
the rates along
importance the servers for the
of comprehensively possibility
evaluating dataof controlling
centers, including the the
heterogeneous
energy of IT temperature
equipment.
distributions. Torell et al. [17] conducted data center cost analysis
They also discussed the effects of elevated temperatures on server failure. By selecting equipment and demonstrated the importance
with a short restart time, maintaining sufficient back-up cooling capacity, and using heatThey
of comprehensively evaluating data centers, including the energy of IT equipment. storage,also
discussed
power the effects
outages can be of elevatedintemperatures
managed a predictableon way.server
Fewfailure.
technical Byworks
selectinghave equipment
been carried withona data
short
restart time, maintaining sufficient back-up cooling capacity, and
center architecture and its IT load for supply air temperatures, power density, air containment, and using heat storage, power outages
can be managed
right-sizing of coolingin aequipment.
predictableHowever,
way. Few technical
their thermal works have been
performance carried
still needs to be onclarified.
data center
architecture and its IT load for supply air temperatures, power density, air containment, and
right-sizing of cooling equipment. However, their thermal performance still needs to be clarified.
 Energies 2019, 12, x FOR PEER REVIEW 4 of 16
Energies
Energies2019,
2019,12,
12,x 2996
FOR PEER REVIEW 4 of 1616
4 of
3. Major Issues in Data Center Planning
3. Major Issues in Data Center Planning
3. Major
3.1. Issues and
Root Causes in DataScaleCenter
of FaultPlanning
Conditions
3.1. Root Causes and Scale of Fault Conditions
DataCauses
3.1. Root centers and must
Scale ofbeFault adequately
Conditionsfurnished with various backup systems to prepare for
Data centers must be
unplanned outages, and operation training adequately furnished
for facilitywithmanagers
various must backup alsosystems
be conductedto prepare
properly.for If
unplannedData centers must be adequately furnished with various backup systems to prepare for unplanned
adequate outages,
preparations and operation
are not made training for facility
in this regard,managers
it becomes must also be conducted
impossible to provide properly.
stable and If
outages, preparations
adequate and operationare training
not for facility
made in this managers
regard, itmust
becomes also be conducted
impossible to properly.
provide If adequate
stable and
continuous IT services, which means that the data center fails to serve its intended purpose.
preparations
continuous are not made
IT services, which inmeans
this regard,
that it becomes
the data center impossible
fails to serveto provide stable purpose.
its intended and continuous IT
The results of benchmarking 63 data centers that experienced unplanned outages showed that
services, whichof means that the data center fails tothatserve its intended purpose. outages showed that
theThe results
damage costs benchmarking
incurred in 201663were data38% centers higher than experienced
in 2010, and unplanned
the mean damage cost when an
the The
damage results
costs of benchmarking
incurred in 2016 63
were data
38% centers
higher that
than experienced
in 2010, and unplanned
the mean damage outages showed that
outage occurred at one data center increased from $505,502 in 2010 to $740,357 in 2016cost
[18].when
Figurean 2
the damage
outage occurred costs at incurred
one data in 2016 increased
center were 38% from higher than in in
$505,502 2010,
2010 and
to the meanin
$740,357 damage
2016 costFigure
[18]. when 2an
summarizes the root causes and ratios of system outages in the sample of 63 data centers. The major
outage occurred
summarizes the root at one dataand
causes center
ratiosincreased
of system from $505,502the
outages in sample
2010 to $740,357 dataincenters.
2016 [18].TheFigure 2
cause was UPS system failure, which seems to be due toin inadequate of 63
UPS reliability verification major
upon
summarizes
cause the
wasinstallation.
UPS systemToroot causes
failure, and ratios
which of system outages in the sample of 63 data centers. The major
initial resolve thisseemsproblem,to be due it istonecessary
inadequatetoUPS reliability
verify equipmentverification upon
and systems
cause
initial was UPS
installation. system failure, which seems to be due to inadequate UPS reliability verification upon
thoroughly and to To resolve
conduct trialthis
runsproblem,
accordingit toisa necessary
systematic to verifyfrom
process equipment
the initialand
design systems
phase.
initial
thoroughly installation.
and To
to conduct resolvetrial this problem,
runs according it is necessary to verify equipment and systems thoroughly
In addition, cooling system failures accountedtofor a systematic
a large portion process from
of the theconditions.
fault initial design In phase.
the case
Inand to conduct
addition, coolingtrialsystem
runs according
failures to a systematic
accounted for a process
large from the
portion
of system outages due to accidents and mistakes, it is important to provide manuals for possible of initial
the fault design phase. In
conditions. Inthe
addition,
case
cooling
ofsituations system
system outages failures
due tofor accounted
accidents for a large
and mistakes, portion of
it is even the
important fault conditions.
toisprovide In the case of system
and training facility managers, because if there backup manuals
equipment forfor possible
dealing
outages due
situations and totraining
accidents for and mistakes,
facility managers, it is important
because to provide
even if there manuals
is backup forequipment
possible situations
for dealingand
with outages, it is useless if the operational knowledge of the facility manager is poor. Figure 3
training
with for facility
outages, managers,
it is useless because even if knowledge
there is backup equipment for dealingiswith outages, it3is
shows the damage cost as ifa function
the operational
of the duration of the ofunplanned
the facilitysystem
manager outage.poor.
As can Figure
be seen,
useless
shows if the operational knowledge of the facility manager is poor. Figure 3 shows the damage cost as
whenthe andamage
outage cost as a function
occurs, the damage of thecost durationis lower of theif unplanned
the outage system time isoutage. As canbybeaseen,
minimized quick
a
when function of
an outage the duration
occurs, of the
the todamage unplanned cost is system
lower outage.
if the As
outage can be seen,
time is when
minimized an outage occurs,
response. As such, in order create stable data centers, backup systems that can handlebyunexpected
a quick
the damage
response. cost is lower if the outage time is minimized by a quick response. As such, in order to
accidentsAsmust such,beinsystematically
order to createdesigned,stable datatested, centers, and backup
managed systemsfromthatthecan handle
initial unexpected
stage. Even after
create
accidents stable data
must facility centers,
be systematically backup systems
designed, that can
tested, andhandle unexpected
managed accidents must be systematically
construction, managers must be thoroughly educated andfromtrainedthetoinitial
respond stage. Even afterto
immediately
designed, tested,
construction, facility and managedmust
managers frombe the initial stage.
thoroughly Even after
educated and construction,
trained to facilityimmediately
respond managers must to
unplanned outages.
be thoroughly
unplanned outages. educated and trained to respond immediately to unplanned outages.

Figure2.2. Root
Figure Root causes
causes of
of unplanned
unplanned outages
outages [18].
[18].
Figure 2. Root causes of unplanned outages [18].

Figure
Figure3.3. Relationship
Relationship between
between cost
cost and
and duration
duration of
of unplanned
unplannedoutages
outages[18].
[18].
Figure 3. Relationship between cost and duration of unplanned outages [18].
Energies 2019, 12, 2996 5 of 16

3.2. Temperature and IT Reliability

Initially, data centers were designed and operated with a focus on IT equipment stability rather
than energy savings. In the past, the temperatures and humidity levels of data centers were managed
very strictly and kept at 21.5 ◦ C and 45.5%, respectively, to maintain optimal thermal conditions for IT
equipment operation. For years, IT support infrastructure system design had to match the specifications
of high-density power components, including the proper cooling and operating condition ranges of the
equipment. A considerable amount of energy is inevitably required to maintain a constant temperature
and humidity level.
However, due to recent technological advances in the IT field, the heat resistance of such equipment
has improved, and as warnings about energy costs and greenhouse gas emissions have become more
prominent, the environmental specifications of equipment for air cooling have become somewhat more
flexible. As shown in Table 1, an IT server intake temperature of 18–27 ◦ C is recommended, and thus
allowable temperature ranges of 15–32 ◦ C (class 1) and 10–35 ◦ C (class 2) with an allowable relative
humidity range of 20–80% have been specified [19]. The insufficient amount of clear information
about the reliability changes that can occur when IT equipment is operated has been an obstacle
to proposing wider temperature and humidity ranges. Intel [20] conducted the first study on this
topic, using industry standard IT servers to demonstrate that reliability is not physically affected
much by temperature and humidity, which was an unexpected conclusion. Figure 4 presents the
internal component temperature changes of a typical x86 server with a variable-speed fan according
to changes in the server intake air temperature that were obtained by the American Society of
Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE). To reduce the effects of server inlet
temperature increases, the inside of the server is kept at a constant temperature by increasing the server
fan speed. Therefore, the reliability of the server components is not directly affected by the inlet air
temperature. However, there is a trade-off with the energy used by the fan. In server component and
air flow management, changes in the server operating temperature have decisive effects on the stability
of the server, but servers are designed to ease the changes into the operating range. Specifically, the
increases in the data center operating temperature did not have any effects on the power consumption
of the server until the server inlet temperature reached the set range. Figure 5 presents the failure rate
changes for various IT servers and vendors, showing the spread of the variability between devices and
the mean values at specific temperatures. Each data point shows not the actual number of failures,
but rather the relative change in the failure rate in a sample of devices from several vendors. The
device failure rate is normalized to 1.0 at 20 ◦ C. Thus, Figure 6 demonstrates that the failure rate during
continuous operation at 35 ◦ C is 1.6 times higher than that during continuous operation at 20 ◦ C. For
example, if 500 servers are continuously operated in a hypothetical data center and it is assumed that
an average of five servers will normally experience errors when operating at 20 ◦ C, it is expected that
an average of eight servers will experience errors when operating at 35 ◦ C [21].

Table 1. Environmental classes for data centers [19].

Dry Bulb Relative Humidity Dew Point Change Rate

Class
(◦ C) (%) (◦ C) (◦ C/h)
Recommended A1 to A4 18 to 27 5.5 ◦ C (DP) to 60 (RH) N/A N/A
A1 15 to 32 20 to 80 17 5/20
Allowable A2 10 to 35 20 to 80 21 5/20
A3 5 to 40 8 to 85 24 5/20
A4 5 to 45 8 to 90 24 5/20
 Energies 2019, 12, x FOR PEER REVIEW 6 of 16
Energies 2019, 12,
Energies 2019, 12, 2996
x FOR PEER REVIEW 66 of
of 16
16
Energies 2019, 12, x FOR PEER REVIEW 6 of 16

Figure 4. Internal component temperature and fan power versus inlet temperature [20].
Figure 4. Internal component temperature and fan power versus inlet temperature [20].
Figure
Figure 4. Internal component
4. Internal component temperature
temperature and
and fan
fan power
power versus
versus inlet
inlet temperature
temperature [20].
[20].

Figure 5. Relative failure rate versus temperature for volume servers [21].
5. Relative failure rate versus temperature for volume servers [21].
Figure 5.
Figure
Figure 5. Relative failure rate versus temperature for volume servers [21].

Figure 6. Reference data center of the case study.

Figure 6. Reference data center of the case study.
4. Case Study: Data Center Figure
and 6. Reference data center of the case study.
Modeling
Figure 6. Reference data center of the case study.
4. Case Study: Data Center and Modeling
4. Case
4.1. DataStudy:
CenterData Center
Cooling Systemand Modeling
4. Case Study: Data Center and Modeling
4.1. Data Center Cooling System
This is a practice-based learning study in a 20 MW data center project (Figure 6). As shown in
4.1. Data Center Cooling System
4.1. Data
Figure 7, Center
This the Cooling
is acooling
practice-basedSystem
system learning
of the casestudy studyindata a 20center
MW data center project
is composed of central(Figure 6). As
chilled shown in
water-type
This
Figure 7, the
computer is a
room practice-based
cooling learning
system (CRAH)
air handling of the case study
unitsstudy in a
withdata 20 MW data
centercooling
a district center
is composed project
systemofthat (Figure
central 6).
chilled
supplies As shown
water-type
chilled water in
This is a practice-based learning study in a 20 MW data center project (Figure 6). As shown in
Figure
computer
to 7, the
a nine-story cooling
room ITair system
room.of
handling
server the
(CRAH)
In caseunits
total, 43study
CRAHwith data center
a district
units is composed
cooling
(to ensure a system of that
constant central chilledchilled
supplies
temperature water-type
and water
relative
Figure 7, the cooling system of the case study data center is composed of central chilled water-type
computer
humidity) room
to a nine-story air
IT handling
are installed server
on each(CRAH)
room. units 43
In including
floor, total, with n a+district
CRAH unitscooling
1 redundant system
(to units,
ensure anda that
the supplies
constant chilled
rackstemperature
of IT servers water
and
are
computer room air handling (CRAH) units with a district cooling system that supplies chilled water
to a nine-story
relative humidity)
basically IT
installed inare server room.
installed
a cold In total, 43 CRAH
on each floor,structure
aisle containment including units (to
to nallow ensure
+ 1 redundant a
relatively highconstant temperature
units,temperatures
and the racks and
forofthe IT
to a nine-story IT server room. In total, 43 CRAH units (to ensure a constant temperature and
relative
serversair
supply humidity)
are fromare
(SA)basically theinstalled
installed
CRAH units. onin each floor,
aTherefore,
cold including
aisle
fromcontainment n + 1 redundant
the primary of the units,
structure
side to allow
heat and relatively
the racks
exchanger, ofhigh
the district IT
relative humidity) are installed on each floor, including n + 1 redundant units, and the racks of IT
servers
chilled are undergoes
temperatures
water basically installed
for the supplythermal in a from
airexchange
(SA) cold aisle
the
so that CRAH containment
relatively units. structure
Therefore,
high-temperature from to the
allow
chilled relatively
primary
water side of
is supplied highthe
to
servers are basically installed in a cold aisle containment structure to allow relatively high
temperatures
heat
the exchanger,
secondary for the
side. the supply air
district an
Furthermore, (SA)
chilledfrom
emergency the CRAH
waterchilled units.
undergoes Therefore,
water supplythermal from
exchange
is contained the primary
so buffer
within side
that relativelyof the
(storage)
temperatures for the supply air (SA) from the CRAH units. Therefore, from the primary side of the
heat
tanks, exchanger,
for use in casethe
high-temperature district
chilled
an outage water chilled
occurs. water
is supplied
Table toundergoes
2 shows the
thesecondary
supply thermal side.exchange
conditions Furthermore,
of the centralso that relatively
an emergency
cooling system.
heat exchanger, the district chilled water undergoes thermal exchange so that relatively
high-temperature
chilled
Due water
to the naturesupply chilled
of the is
data water
contained is supplied
within buffer
center business, to the secondary
(storage)
the IT server roomtanks, side.
is fullfor Furthermore,
use stage,
at each in case an
butanthis emergency
outage
analysis occurs.
was
high-temperature chilled water is supplied to the secondary side. Furthermore, an emergency
chilled
Table 2water
performed shows supply
on the the is contained
topsupply
five conditions
floors, whichwithinof
were buffer
the put (storage)
central
together cooling tanks,
first for the
system.
during use
Due intocase
initial the an outage
nature
operation of the
phase. occurs.
data
It is
chilled water supply is contained within buffer (storage) tanks, for use in case an outage occurs.
Table
center2business,
normal shows
for thethesupply
the horizontal IT serverconditions
piping room is of
fullthe
in a dedicated at central
each
data stage,cooling
center but system.
this
cooling Due
analysis
system to
towasbe the nature
performed
installed in of
a on the
thedata
loop-type top
Table 2 shows the supply conditions of the central cooling system. Due to the nature of the data
center business,
five floors, which
configuration the
on each IT
were server
put
floor room
together
with is full at each
first during
a redundant stage,
riser tothe but this
initialfor
prepare analysis
operation
emergencies. was
phase. performed on
It is normalstorage
Chilled-water the top
for the
center business, the IT server room is full at each stage, but this analysis was performed on the top
five floors, which were put together first during the initial operation phase. It is normal for the
five floors, which were put together first during the initial operation phase. It is normal for the
Energies 2019, 12, 2996 7 of 16

(buffer) tanks were installed to provide stable chilled water before the emergency power begins to be
supplied in the event of a cooling system outage. It is important to identify an economical and optimal
storage tank size by determining the time range during which the chilled water in the pipes can be
recirculated and used without the cooling system without affecting the IT server operating environment.
For this purpose, the amount of water in the pipes was calculated first. As shown in Table 3, the riser
part calculations were performed by dividing the section by the pipe diameter, and the mechanical plant
room and horizontal pipes of a typical floor were calculated to have the same pipe diameter. The total
Energies 2019, 12, x FOR PEER REVIEW 8 of 16
amount of water in the chilled water pipes was calculated to be 234.3 m3 .

Figure 7. Data center cooling system: (a) secondary chilled water loop; (b) IT server room on a
Figure 7. Data center cooling system: (a) secondary chilled water loop; (b) IT server room on a typical
typical floor.
floor.
Table 2. Information technology (IT) cooling system operating conditions.
The inlet air temperature of each IT server was found to fall within the allowable temperature
range specified in Item ASHRAE. In addition, a CFD simulation CRAH-1 was performed to CRAH-2 find the allowable
chilled water supply Type
temperature range by checking water-sidethe economizer with indirect
SA temperature range ofair-side economizer
the CRAH units
Cooling
and calculating capacity
the (usRT) for it to reach the allowable
time required 43 15
chilled water temperature range. To
the /h)
Air volume 3 31,000 8500
ensure the reliability of (m CFD modeling, the characteristics of the IT equipment must be reflected
Supply air (SA) temperature (◦ C) 20.0 20.0
accurately. There are various
Return air (RA) temperature ( C)
types
◦ of commercial simulation
34.5
software for performing
34.5
analysis. In
this study, 6 Sigma DC was
Chilled water flow rate (LPM)chosen, as it is specially designed
391 to reflect the characteristics
144 of data
centers. This specialized evaluation
Chilled water supply (CWS) temperature ( C) ◦
program is intended
10–14 for the designers of data center
10 mechanical
Chilled water
systems. One return (CWS) temperature
of its analysis modules, the (◦ C)6 Sigma DC 15.5–19.5
Room, is a CFD simulation module 15.5 that can be
used toNumber
performofintegrated
systems (EA/floor)
efficiency reviews of IT server 12 rooms. The special feature31of this module
is that it has various information on and databases for the most important types of IT equipment that
are employed in data centers,Table Water
and it3.can content
create in chilled
accurate water pipes. [22]. The basic module was
IT environments
composed of a minimum of seven cold and hot aisles (area A’ in Figure 7), and the room depth was
Length (m)
less than
Size 15
(mm) m, which is the maximum distance that a CRAHVolume unit can(msupply
3) air Content
Water for cooling.
(m3 ) As
shown in Figure 8, the IT Horizontal Riser
equipment was composed Total
of a rack that could hold a maximum of 42 U
200 the power
servers, and 1850density
(typicalwas
floor)set at 4.0- kW/rack. 1850 58.0
In total, 192 IT server racks were arranged
234.3
according 250to the standards presented
- in ASHRAE.
82 The82air distribution
4.0 method was under floor air
distribution + side wall air return, which is currently the most universal method. The raised floor
height, which affects the air flow distribution, was set at 900 mm, and the ceiling height was set at 3.0
m. The SA temperature of a CRAH unit is an important factor that is closely related to the energy
consumption of the cooling system. An increase in the SA temperature can cause the supply chilled
water temperature to increase, so it is correlated with the cooling plant system. Table 4 shows the
simulation boundary conditions, based on the operation conditions of the IT server. The chilled
Energies 2019, 12, 2996 8 of 16

Table 3. Cont.

Length (m)
Size (mm) Volume (m3 ) Water Content (m3 )
Horizontal Riser Total
350 - 82 82 7.8
400 - 82 82 10.2 234.3
450 - 82 82 13.0
500 440 (mechanical room) 280 720 141.3

4.2. IT Server Room Thermal Model

By using the IT server room thermal model, the SA temperature of the CRAH units and the inlet
air temperature of the IT servers were analyzed.
The inlet air temperature of each IT server was found to fall within the allowable temperature
range specified in ASHRAE. In addition, a CFD simulation was performed to find the allowable
chilled water supply temperature range by checking the SA temperature range of the CRAH units and
calculating the time required for it to reach the allowable chilled water temperature range. To ensure
the reliability of the CFD modeling, the characteristics of the IT equipment must be reflected accurately.
There are various types of commercial simulation software for performing analysis. In this study,
6 Sigma DC was chosen, as it is specially designed to reflect the characteristics of data centers. This
specialized evaluation program is intended for the designers of data center mechanical systems. One
of its analysis modules, the 6 Sigma DC Room, is a CFD simulation module that can be used to perform
integrated efficiency reviews of IT server rooms. The special feature of this module is that it has various
information on and databases for the most important types of IT equipment that are employed in
data centers, and it can create accurate IT environments [22]. The basic module was composed of a
minimum of seven cold and hot aisles (area A’ in Figure 7), and the room depth was less than 15 m,
which is the maximum distance that a CRAH unit can supply air for cooling. As shown in Figure 8, the
IT equipment was composed of a rack that could hold a maximum of 42 U servers, and the power
density was set at 4.0 kW/rack. In total, 192 IT server racks were arranged according to the standards
presented in ASHRAE. The air distribution method was under floor air distribution + side wall air
return, which is currently the most universal method. The raised floor height, which affects the air
flow distribution, was set at 900 mm, and the ceiling height was set at 3.0 m. The SA temperature of
a CRAH unit is an important factor that is closely related to the energy consumption of the cooling
system. An increase in the SA temperature can cause the supply chilled water temperature to increase,
so it is correlated with the cooling plant system. Table 4 shows the simulation boundary conditions,
based on the operation conditions of the IT server. The chilled water temperature was changed to
10–18 ◦ C to respond to CRAH unit SA temperatures of 20–25 ◦ C. The data center environment standard
ranges [19] at which IT servers could operate normally were set at 18–27 ◦ C for the IT server inlet
temperature and 40–60% as the judgment standards, as these are the recommended values for Classes
A1–A4. The division of the operating environment of the IT equipment into four classes was according
to the most reasonable standards required when judging the suitability of a CFD model (Table 1). The
conditions shown here are those of the air flowing into the IT servers for actual cooling, so they do not
have to be equal to the average conditions inside the server room.
In an emergency, it is possible to expand the ranges of allowable conditions, but if the thermal
balance is upset, it does not take much time to exceed the allowable range. Thus, in this study, the
analysis was performed with the recommended conditions.
are the recommended values for Classes A1–A4. The division of the operating environment of the IT
equipment into four classes was according to the most reasonable standards required when judging
the suitability of a CFD model (Table 1). The conditions shown here are those of the air flowing into
the IT servers for actual cooling, so they do not have to be equal to the average conditions inside
Energies 2019, 12, 2996
the
9 of 16
server room.

Figure 8. IT
Figure 8. IT server
server room
room thermal
thermal model
model for
for computational
computational fluid
fluid dynamics
dynamics (CFD)
(CFD) simulation.
simulation.
Table 4. Simulation boundary conditions.
Table 4. Simulation boundary conditions.
Item Value Item Value
Item Value Item Value
Room
Room (m2(m
sizesize ) 2) 544.3
544.3 Raised
Raisedfloor
floorheight
height(m)
(m) 0.8
0.8
Number of cabinets
Number of cabinets 192 EA
192 EA Number
Number of CRAHunit
of CRAH unit 6
6 EAEA
◦ C)
AirAir
flow (m3(m
raterate
flow /h/CRAH)
3/h/CRAH) 31,000
31,000 Supply
Supplyair
airtemperature
temperature((°C) 20–25
20~25
Room height (m) 5.1 False ceiling height (m) 3.0
Room height (m) 5.1 False ceiling height (m) 3.0
Rack IT limit (kW/rack) 4.0 Cooling capacity (kW/CRAC) 150 (43 RT)
Rack IT limit (kW/rack) 4.0 Cooling capacity (kW/CRAC) 150 (43 RT)
Cold aisle/hot aisle Cold aisle containment Chilled water temperature (◦ C) 10–18
Cold aisle/hot aisle Cold aisle containment Chilled water temperature (°C) 10~18

4.3. Simulation Results it is possible to expand the ranges of allowable conditions, but if the thermal
In an emergency,
balance
The is upset,coil
cooling it does
(heatnot take much
exchanger) time toofexceed
capacity a CRAHtheunit
allowable range. Thus,
is determined based in
onthis study, the
the minimum
analysis
inlet waswater
chilled performed with theHence,
temperature. recommended conditions.
if the chilled water supply temperature changes, the cooling
coil capacity changes, and the temperature of the air supplied by the CRAH unit to the IT server
4.3. Simulation
room Results
changes accordingly. Analysis was performed on the air inlet and outlet (SA/RA) temperature
variations with the cold chilled
The cooling coil (heat exchanger) water inlet capacity
temperature of abased
CRAH on the
unittechnical data of the
is determined CRAH
based onunit
the
that was used
minimum in chilled
inlet this study.water According to theHence,
temperature. results,ifallowable
the chilled operation conditions
water supply are achievable
temperature up
changes,
to a chilled water supply temperature of 14 ◦ C, but if chilled water is supplied at a temperature of
the cooling coil capacity changes, and the temperature of the air supplied by the CRAH unit to the IT
15 ◦ C or higher, the cooling coil capacity of the CRAH unit is reduced, and the temperature of the air
server room changes accordingly. Analysis was performed on the air inlet and outlet (SA/RA)
supplied
temperature to the IT server with
variations roomthe willcold
increase,
chilled as water
showninlet
in Figure 9. In thebased
temperature CFD simulation results,data
on the technical the ITof
server inlet air temperature and server room temperature distribution
the CRAH unit that was used in this study. According to the results, allowable operation conditionsas functions of the temperature
of
arethe air supplied
achievable up by to athe CRAH
chilled unit and
water supplythe temperature
chilled water of supply
14 °C,temperature
but if chilled conditions are shownatina
water is supplied
Figures
temperature10 andof11.15The °C most important
or higher, the element
cooling coilin thecapacity
evaluation of the
of the air distribution
CRAH efficiency
unit is reduced, andof the
an
IT server room is the air temperature distribution within the room, particularly
temperature of the air supplied to the IT server room will increase, as shown in Figure 9. In the CFD the inlet air temperature
of the IT server.
simulation Since
results, theanITincrease in theair
server inlet inlet air temperature
temperature is a primary
and server factor in server
room temperature failure, the
distribution as
air distribution efficiency was increased by using cold aisle containment,
functions of the temperature of the air supplied by the CRAH unit and the chilled water supply which involves physical
isolation
temperature of theconditions
air inflow are partshown
of the IT inserver.
Figures Although
10 and the 11.temperature ranges differ
The most important according
element to
in the
the SA temperature,
evaluation of the airthe containment
distribution systemofclearly
efficiency an IT has
servera difference
room is the between the cold and
air temperature hot aisle
distribution
air temperatures because the cold and hot aisles are separated overall.
within the room, particularly the inlet air temperature of the IT server. Since an increase in the inletTemperature increases due
to
airthe recirculation
temperature is aofprimary
outlet air as inlet
factor air of failure,
in server IT servers theare
air adistribution
major causeefficiency
of serverwas failures, and the
increased by
failures
using cold mainly
aisleoccur at the top
containment, of theinvolves
which rack-server. If the
physical server room
isolation of thecooling
air inflowload (i.e.,
part of the
the heat gain
IT server.
from the servers)
Although is less thanranges
the temperature 85% ofdiffer
the cooling
accordingcapacity of the
to the SA CRAH unit, then
temperature, the acceptable
containment operation
system
conditions ◦
clearly hasare achievablebetween
a difference until thethe SAcold
temperature reaches
and hot aisle air approximately
temperatures because24.0 C, the according
cold and to the
hot
design
aisles arecooling capacity,
separated and at
overall. this time theincreases
Temperature chilled water
due to temperature is 17 ◦ C.
the recirculation of As shown
outlet air asin inlet
Figure
air12,
of
if the chilled water temperature exceeds 17 ◦ C, the cooling capacity of the CRAH unit decreases to
less than 80%, and the indoor temperature increases rapidly. This steady-state result was obtained by
analyzing the temperature changes based on the cooling coil capacity of the CRAH unit.
the cooling capacity of the CRAH unit, then acceptable operation conditions are achievable until the
SA temperature reaches approximately 24.0 °C, according to the design cooling capacity, and at this
time the chilled water temperature is 17 °C. As shown in Figure 12, if the chilled water temperature
exceeds 17 °C, the cooling capacity of the CRAH unit decreases to less than 80%, and the indoor
temperature increases rapidly. This steady-state result was obtained by analyzing the temperature
Energies 2019, 12, 2996 10 of 16
changes based on the cooling coil capacity of the CRAH unit.

Figure9.
Figure Supplyair
9.Supply air(SA)
(SA)temperature
temperatureand
andcooling
coolingcapacity
capacityof
ofaacomputer
computerroom
roomair
airhandling
handling(CRAH)
(CRAH)
unit according to chilled water supply temperature.
unit according to chilled water supply temperature.

5. Temperature Control Approaches

5. Temperature Control Approaches
5.1. Temperature Increase over Time After a Cooling System Outage
5.1. Temperature Increase over Time After a Cooling System Outage
After finding the allowable chilled water temperature range by performing CFD analysis, the next
step was tofinding
After thethe
calculate allowable chilled
time to reach water
this temperature
temperature. range byinperforming
The increase the chilled CFD
wateranalysis, the
temperature
next
after step was tosystem
a cooling calculate the was
outage time analyzed
to reach this temperature.
by performing The increase
a system in thetochilled
scale-down water
conduct the
temperature after a cooling system outage was analyzed by performing a system scale-down
calculations for a typical floor. Consequently, the total amount of water inside the chilled water pipes to
conduct
was aroundthe 45
calculations for aon
m3 /floor based typical floor.floor.
the typical Consequently, the the
Table 5 shows totalcooling
amount of water
capacity andinside the
operating
chilled water pipes was around 45 m 3/floor based on the typical floor. Table 5 shows the cooling
conditions of the CRAH unit of the typical floor. Equations (1)–(3) are the basic functions for calculating
capacity and
the rate of operating
change conditions
of the of the
chilled water CRAH unitinofthe
temperature thepipes
typical
andfloor. Equations
the delay time (1)–(3)
after a are the
cooling
basic
system functions
outage. for calculating the rate of change of the chilled water temperature in the pipes and
the delay time after
In detail, a cooling
the total cooling system outage. of CRAH unit (C) is proportionate to the chilled water
coil capacity
flow rate (Q) and temperature differential of chilled water return (TCHR ) and supply (TCHS ). Therefore,
the total chilled water flow rate of CRAH unit is the total cooling coil capacity in inverse proportion to
chilled water temperature differential (∆T). The one cycle operating time of chilled water return and
supply can be the water volume in chilled water pipes (V) divided by the chilled water flow rate. As
shown in Equation (4), if the heat gain of the IT server room is constant, a constant amount of cooling
should be supplied by the cooling coil of the CRAH unit.

C = Q × {ρ × ρ × (TCHR − TCHS ) = Q × {ρ × ρ × ∆T, (1)

C
Q= , (2)
{ρ × ρ × ∆T

V V × {ρ × ρ × ∆T
t= = , (3)
Q C
∆Tn = (TCHR )n − (TCHS )n = 5.5 °C ( f ixed). (4)

Table 5. IT cooling system for a typical floor.

Item CRAH-1 CRAH-2 Total

Cooling capacity (kW) 150 × (11 EA) 50 × (17 EA) 2500
Chilled water flow rate (m3 /min) 0.391 × (11 EA) 0.144 × (17 EA) 6.75
Chilled water content in pipes (m3 ) - 45 m3
Energies 2019, 12, 2996 11 of 16
Energies 2019, 12, x FOR PEER REVIEW 11 of 16

Figure 10.
Figure CFD simulation
10. CFD simulation results
results for
for normal
normal conditions
conditions and
and fault
fault conditions
conditions 11 and
and 2.
2.
Energies 2019, 12, 2996 12 of 16
Energies 2019, 12, x FOR PEER REVIEW 12 of 16

Figure 11. CFD simulation results for fault conditions 3, 4, and 5.

Figure 11. CFD simulation results for fault conditions 3, 4, and 5.

In detail, the total cooling coil capacity of CRAH unit (C) is proportionate to the chilled water
flow rate (Q) and temperature differential of chilled water return (TCHR) and supply (TCHS). Therefore,
the total chilled water flow rate of CRAH unit is the total cooling coil capacity in inverse proportion
to chilled water temperature differential (∆T). The one cycle operating time of chilled water return
and supply can be the water volume in chilled water pipes (V) divided by the chilled water flow rate.
Energies 2019, 12, x FOR PEER REVIEW 13 of 16

As shown in Equation (4), if the heat gain of the IT server room is constant, a constant amount of
Energies
cooling2019, 12, 2996
should be
supplied by the cooling coil of the CRAH unit. 13 of 16

Figure 12. Maximum IT server inlet air temperature according to the SA and chilled water conditions.
Figure 12. Maximum IT server inlet air temperature according to the SA and chilled water
the temperature difference ∆T between the chilled water inlet and outlet temperatures is
conditions.
If
assumed to be constant 5.5 ◦ C, the functions for the chilled water temperature changes can be expressed
Table 5. IT cooling
as Equation (5). Therefore, the circulation n + 1thsystem
chilledfor a typical
water supply floor.
temperature is nth chilled water
supply temperature +5.5 C. Item
◦ The delay time is Equation CRAH-1(6). With these functions,Total
CRAH-2 the point in time at
which the cooling system outage
Cooling occurred
capacity (kW) was set to 150T×0 (11
to calculate
EA) the× (17
50 range
EA)of increase
2500 of the chilled
water temperature due to
Chilled coolflow
water lossrate
over(mtime.
3/min) At this point
0.391 × (11inEA)
time,0.144
out of× (17
the EA)
total amount
6.75 of water (V 2
or 45 m3 ), the amount of chilled
Chilled water water
content at 10(m
in pipes ◦C3) in the chilled water - supply (CHS)45 pipe
m3 excluding the
amount of chilled water at 15.5 ◦ C in the chilled water return (CHR) pipe passing through the CRAH
unit is 22.5 m3 or 50%, and the amount of water at this time is V 1 . In addition, t1 is the time at which
the thermal capacity of V 1 is fully exhausted and the temperature of the entire amount of V 2 reaches
15.5 ◦ C, which was calculated𝐶 to 𝑄 be∁𝜌3.45 𝜌min by 𝑇 applying𝑇 𝑄 ∁𝜌(6).𝜌The∆𝑇,
Equation times t2 and t3 at which(1)
all
◦
of V 2 reaches 21.0 and 26.5 C are 10.36 and 17.27 min, respectively. The chilled water temperature
𝐶
time function for each specific interval can 𝑄be expressed as ,shown in Equation (7). (2)
∁𝜌 𝜌 ∆𝑇
 
f (TCHS )n+1 = (TCHS )n + ∆Tn = (TCHR )n , (5)
𝑉 𝑉 ∁𝜌 𝜌 ∆𝑇
𝑡 , (3)
𝑄 𝐶
V1 × {ρ × ρ × ∆T1 X V2 × {ρ × ρ × ∆Tn
f (tn ) = + , (6)
∆𝑇 𝑇 C 𝑇 n>1 5.5 ℃ 𝑓𝑖𝑥𝑒𝑑 .
C (4)
 
5 4 between
If the temperature difference
f (t) = 6E14x − 2E10xΔT + 2E07x3 +the 1E04x 2
chilled water inlet
+ 0.0436x + 10 and R2 =outlet
0.99 .temperatures(7)is
assumed to be constant 5.5 °C, the functions for the chilled water temperature changes can be
expressed
5.2. Systemas Equation
Response (5).Environment
to IT Therefore, theunder circulation n + 1th chilled water supply temperature is nth
Fault Conditions
chilled water supply temperature +5.5 °C. The delay time is Equation (6). With these functions, the
The results of analyzing the amount of cooling supplied by the CRAH units based on the potential
point in time at which the cooling system outage occurred was set to T0 to calculate the range of
cooling capacity showed that a period of approximately 150 s (2 min 30 s) is required for the temperature
increase of the chilled water temperature due to cool◦ loss over time. At this point in time, out of the
of the chilled water in a pipe to increase from 10 to 14 C. The cooling system selection is implemented
total amount of water (V2 or 45 m3), the amount of chilled water at 10 °C in the chilled water supply
to enable the CRAH unit to maintain a set temperature up to a chilled water supply temperature of
(CHS) pipe excluding the amount of chilled water at 15.5 °C in the chilled water return (CHR) pipe
14 ◦ C for the water-side economizer, and up 3to this point in time, it is in the safe range. Up to a chilled
passing through the CRAH unit is 22.5 m or 50%, and the amount of water at this time is V1. In
water temperature of 14 ◦ C, the maintenance of the IT environment is unaffected by changes in the air
addition, t1 is the time at which the thermal capacity of V1 is fully exhausted and the temperature of
inlet and outlet temperatures (of the CRAH units) caused by changes in the chilled water inlet and
the entire amount of V2 reaches 15.5 °C, which was calculated to be 3.45 min by applying Equation
outlet temperatures (of the cooling coils in the CRAH units), based on the technical data provided by
(6). The times t2 and t3 at which all of V2 reaches 21.0 and 26.5 °C are 10.36 and 17.27 min,
the equipment manufacturer. As for the inlet air temperature of the IT server, which was found in
respectively. The chilled water temperature time function for each specific interval can be expressed
the CFD simulation, 24 ◦ C is the maximum allowable temperature of the supply air from the CRAH
as shown in Equation (7).
units that enables the ASHRAE allowable temperature requirements to be met. Furthermore, the
maximum chilled water supply𝑓temperature𝑇 𝑇
at which the∆𝑇 𝑇
air temperature , can be maintained is 17(5) ◦ C.

The time required to reach this chilled water temperature is around 320 s (5 min 20 s) after a cooling
changes in the chilled water inlet and outlet temperatures (of the cooling coils in the CRAH units),
based on the technical data provided by the equipment manufacturer. As for the inlet air
temperature of the IT server, which was found in the CFD simulation, 24 °C is the maximum
allowable temperature of the supply air from the CRAH units that enables the ASHRAE allowable
temperature requirements to be met. Furthermore, the maximum chilled water supply temperature
Energies 2019, 12, 2996 14 of 16
at which the air temperature can be maintained is 17 °C. The time required to reach this chilled water
temperature is around 320 s (5 min 20 s) after a cooling system outage, during which allowable
operation
system is possible.
outage, Figureallowable
during which 13 showsoperation
the safe and allowable
is possible. operation
Figure periods
13 shows according
the safe to the
and allowable
chilled water
operation temperature.
periods according to the chilled water temperature.

Figure 13. Chilled water temperature as a function of time under fault conditions.
Figure 13. Chilled water temperature as a function of time under fault conditions.
6. Conclusions
6. Conclusions
The temperature conditions and allowable operation times of the chilled water and air sides of a
CRAH The temperature
unit conditionscooling
after an unplanned and allowable operation
system outage times
were of the chilled
analyzed for the water
stableand
andair sides of a
economical
CRAH of
design unit
andafter an unplanned
equipment cooling
selection system
for data outage
center were
cooling analyzed
systems. CFDforanalysis
the stableofand
eacheconomical
stage was
design of and equipment selection for data center cooling systems. CFD
performed to analyze the temperature of the air supplied to the server to remove the heatanalysis of eachgain
stageofwas
the
performed
IT server withto analyze
changestheintemperature
the supply airof the air supplied
temperature andtocooling
the server to remove
capacity theCRAH
of the heat gain
unit,ofand
the
IT server with changes in the supply air temperature and cooling capacity of the
the results were compared to the ASHRAE standard. The changes in the air supply temperature and CRAH unit, and the
results were compared to the ASHRAE standard. The changes in the air supply
cooling capacity of the CRAH unit were predicted according to the changes in the chilled water inlet temperature and
cooling capacity
temperature. of the
Finally, CRAH
the chilledunit were
water predicted according
temperature to thetime
increases over changes
after in the chilled
removal of thewater inlet
IT server
temperature. Finally, the chilled water temperature increases over time after removal
room cooling load were calculated based on the amount of water in the chilled water pipe and the of the IT server
room cooling
thermal capacityload
in were
ordercalculated
to determinebased
the on
safethe amount
and of water
allowable in theperiods.
operation chilled water pipe and the
thermal capacity in order to determine the safe and allowable operation periods.
(1) Up to a chilled water supply temperature of 17 ◦ C and a CRAH unit air supply temperature of
(1) Up◦to a chilled water supply temperature of 17 °C and a CRAH unit air supply temperature of
24 C, the temperature of the air flowing into the IT server fell within the required range set forth
24 °C, the temperature of the air◦ flowing into the IT server fell within the required range set
in the ASHRAE standard (18–27 C). Using a CRAH unit coil capacity of 85%, it was possible to
perform allowable operations for approximately 320 s after cooling system outage.
(2) Starting at a CRAH unit chilled water supply temperature of 18 ◦ C and an air supply temperature
of 25 ◦ C, the coil capacity became smaller than the cooling load, and a rapid temperature increase
occurred, which is a serious cause of IT equipment failure.
(3) Currently, the number of cases in which cold aisle containment and designs with relatively high
chilled water and air supply temperatures are used is increasing. During a cooling system outage,
there is a high possibility that a rapid temperature increase will occur inside the IT server room.
Thus, backup systems must be activated within 300 s. The fixed value is the maximum allowable
time until the cooling system re-starts working.

It is essential to understand the operational characteristics of data centers and design optimal
cooling systems to ensure the reliability of high-density data centers. In particular, it is necessary to
consider these physical results and to perform integrated reviews of the time required for emergency
cooling equipment to operate and the availability time of the chilled water storage tanks. In addition,
integrated safety evaluations must be performed, and the effects of each design element must be
determined in future research.

Author Contributions: Investigation, J.C.; Simulation analysis, J.C.; Formal analysis, B.P. and Y.J.; Literature
review, B.P. and Y.J.; Writing-original draft preparation, J.C.; Writing-review and editing, B.P. and Y.J.
Energies 2019, 12, 2996 15 of 16

Funding: This research was supported by grant of the research fund of the MOTIE (Ministry of Trade, Industry
and Energy) in 2019. Project number: 20182010600010.
Conflicts of Interest: The authors declare no conflict of interest.

Nomenclature
Nomenclature
C cooling coil capacity of CRAH unit (kW)
Q chilled water flow rate (m3 /h)
V water volume in chilled water pipes (m3 )
T chilled water temperature (◦ C)
∆T chilled water temperature differential (◦ C)
Cp specific heat (J/kg◦ C)
ρ density (kg/m3 )
t operating time (s)
Subscripts and Superscripts
CRAH computer room air handling
CHS chilled water supply (in CRAH unit)
CHR chilled water return (in CRAH unit)
n number of data

References
1. Zhang, K.; Zhang, Y.; Liu, J.; Niu, X. Recent advancements on thermal management and evaluation for data
centers. Appl. Therm. Eng. 2018, 142, 215–231. [CrossRef]
2. Garday, D.; Housley, J. Thermal Storage System Provides Emergency Data Center Cooling; Doug Garday and Jens
Housley, Intel Corporation: Santa Clara, CA, USA, 2007.
3. Tunner, W.P.; Seader, J.H.; Brill, K.G. Tier Classifications Define Site Infrastructure Performance; White Paper;
The Uptime Institute, Inc.: NewYork, NK, USA, 2005.
4. Cho, J.; Lim, T.; Kim, B.S. Measurements and predictions of the air distribution systems in high compute
density (Internet) data centers. Energy Build. 2009, 41, 1107–1115. [CrossRef]
5. Nadjahi, C.; Louahlia, H.; Lemasson, S. A review of thermal management and innovative cooling strategies
for data center. Sustain. Comput. Inf. Syst. 2018, 19, 14–28. [CrossRef]
6. ASHRAE TC 9.9. Design Considerations for Datacom Equipment Centers; American Society of Heating
Refrigerating and Air-Conditioning Engineers Inc.: Atlanta, GA, USA, 2005.
7. Hasan, Z. Redundancy for Data Centers. ASHRAE J. 2009, 51, 52–54.
8. Hartmann, B.; Farkas, C. Energy efficient data centre infrastructure—Development of a power loss model.
Energy Build. 2016, 127, 692–699. [CrossRef]
9. He, Z.; Ding, T.; Liu, Y.; Li, Z. Analysis of a district heating system using waste heat in a distributed cooling
data center. Appl. Therm. Eng. 2018, 141, 1131–1140. [CrossRef]
10. Kummert, M.; Dempster, W.; McLean, K. Thermal analysis of a data centre cooling system under fault
conditions. In Proceedings of the 11th IBPSA International Building Performance Simulation Conference,
Glasgow, Scotland, 27–30 July 2009.
11. Zavřel, V.; Barták, M.; Hensen, J.L.M. Simulation of data center cooling system in an emergency situation.
Future 2014, 1, 2.
12. Lin, M.; Shao, S.; Zhang, X.S.; VanGilder, J.W.; Avelar, V.; Hu, X. Strategies for data center temperature control
during a cooling system outage. Energy Build. 2014, 73, 146–152. [CrossRef]
13. Lin, P.; Zhang, S.; VanGilde, J. Data Center Temperature Rise during a Cooling System Outage; Schneider Electric
White Paper; Schneider Electric’s Data Center Science Center: Foxboro, MA, USA, 2013.
14. Moss, D.L. Facility Cooling Failure: How Much Time Do You Have; A Dell Technical White Paper, Dell Data
Center Infrastructure: Round Rock, TX, USA, 2011.
15. Gao, X.; Xu, Z.; Wang, H.; Li, L.; Wang, X. Reduced cooling redundancy: A new security vulnerability in
a hot data center. In Proceedings of the Network and Distributed Systems Security (NDSS) Symposium,
San Diego, CA, USA, 18–21 February 2018.
Energies 2019, 12, 2996 16 of 16

16. Nada, S.A.; Attia, A.M.A.; Elfeky, K.E. Experimental study of solving thermal heterogeneity problem of data
center servers. Appl. Therm. Eng. 2016, 109, 466–474. [CrossRef]
17. Torell, W.; Brown, K.; Avelar, V. The Unexpected Impact of Raising Data Center Temperatures; Schneider Electric
White Paper; Schneider Electric’s Data Center Science Center: Foxboro, MA, USA, 2016.
18. Ponemon Institute. Cost of Data Center Outages, Data Center Performance Benchmark Series; Emerson Network
Power: Columbus, OH, USA, 2016.
19. ASHRAE TC 9.9. Thermal Guideline for Data Processing Environments; American Society of Heating Refrigerating
and Air-Conditioning Engineers, Inc.: Atlanta, GA, USA, 2015.
20. Intel. Reducing Data Center Cost with an Air Economizer; IT@Intel Brief, Intel Information Technology:
Santa Clara, CA, USA, 2008.
21. Strutt, S.; Kelley, C.; Singh, H.; Smith, V. Data Center Efficiency and IT Equipment Reliability at Wider Operating
Temperature and Humidity Ranges; The Green Grid Technical Committee: Oregon, OR, USA, 2012.
22. Cho, J.; Yang, J.; Park, W. Evaluation of air distribution system’s airflow performance for cooling energy
savings in high-density data centers. Energy Build. 2014, 68, 270–279. [CrossRef]

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).