Proceedings of the ASME 2011 Pacific Rim Technical Conference & Exposition on
Packaging and Integration of Electronic and Photonic Systems
July 6-8, 2011,
Proceedings of the ASME 2011 Pacific Rim Technical Conference & Exposition
on of Electronic and Photonic Systems
July 6-8, 2011, Portland, Oregon, USA
LIQUID COOLING NETWORK SYSTEMS FOR ENERGY CONSERVATION
IN DATA CENTERS
Mayumi Ouchi
AIST
Tsukuba, Japan
Masato Fukagaya Haruhiko Ohta
SOHKi Co., Ltd. Kyushu University
Nagoya, Japan Fukuoka, Japan
Masahide Sato
Utsunomiya University
Utsunomiya, Japan
ABSTRACT
Energy consumption in data center has been drastically
increasing in recent years. In data center, server racks are
cooled down by air conditioning for the whole room in a
roundabout way. This air cooling method is inefficient in
cooling and it causes hotspot problem that IT equipments are
not cooled down enough, but the room is overcooled. On the
other hand, countermeasure against the heat of the IT
equipments is also one of the big issues. We therefore proposed
new liquid cooling systems which IT equipments themselves
are cooled down and exhaust heat is not radiated into the server
room. For our liquid cooling systems, three kinds of cooling
methods have been developed simultaneously. Two of them are
direct cooling methods that the cooling jacket is directly
attached to heat source, or CPU in this case. Single-phase heat
exchanger or two-phase heat exchanger is used as cooling
jackets. The other is indirect cooling methods that the heat
generated from CPU is transported to the outside of the chassis
through flat heat pipes, and condensation sections of the heat
pipes are cooled down by liquid. Verification tests have been
conducted by use of real server racks equipped with these
cooling techniques while pushing ahead with five R&D
subjects which constitute our liquid cooling system, which
single-phase heat exchanger, two-phase heat exchanger, high
performance flat heat pipes, nanofluids technology, and plug-in
connector. As a result, the energy saving effect of 50~60%
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/interpack2011/70214/ on 07/07/2017 Terms of Use: http://www.a
InterPACK2011
Portland, Oregon,
on Packaging USA
and Integrati
InterPACK2011
InterPACK2011-52066
IPACK2011-
InterPACK2011-52066
Yoshiyuki Abe
AIST
Tsukuba, Japan
Yasuhisa Shinmoto
Kyushu University
Fukuoka, Japan
Ken-ichi Iimura
Utsunomiya University
Utsunomiya, Japan
comparing with conventional air cooling system was provided
in direct cooling technique with single-phase heat exchanger.
INTRODUCTION
Data center heat loads have increased drastically as more
components are squeezed tighter into densely packed server
racks. Energy consumption is 7 billion kWh in 2009, and it is
expected to increase annually by about 7.0%. In this context,
NEDO (New Energy and Industrial Technology Development
Organization) launched Green IT Project for power saving of
IT equipments and reduction of environmental impact.
The data centers are cooled by chilled air that is supplied
from CRACs (Computer Room Air Conditioning Units).
Typical air circulation system in data centers consists of bottom
air intake for the chilled air supplied from perforated tile on
raised floor plenum, and chilled air flows upward. As shown in
Fig.1, the top part of the server rack is not efficiently cooled
down, and the temperature difference between bottom and top
part of server rack reaches about 30℃. In data centers, a
temperature range of 16 to 24℃ is recommended, so the air
temperature is set to unnecessarily low to keep room
temperature, and it causes high energy consumption in the data
centers. Furthermore, excess heat in server room adversely
effects equipment performance, shortens equipment life-times,
and so on. Therefore, the thermal management in data centers is
1 Copyright © 2011 by ASME
 high-priority issue for not only the energy saving but also the
IT equipments.
To improve the cooling efficiency, hot aisle/cold aisle
approach has been adopted [1]. This method is that every aisle
between rows of rack is bounded with exclusive hot-air outlets
or exclusively cool-air intakes. Likewise, optimization of the
server room is in progress based on the measurements and
prediction of airflow in the data centers [2].
In response, the authors have proposed the new liquid
cooling systems for data centers. This system is of cooling
down the CPUs directly or indirectly by liquid at each server
rack and exhaust heat is controlled intensively using by thermal
cable without radiating into the server room as shown in Fig.2
[3].
The three liquid cooling systems have been developed
simultaneously depending on the customer’s need and the
performance of the computers. This is because about 50 % of
IT customer does not accept intervention of the water to the
server room even though electrically insulating liquid is used in
our system. For the customers who tolerate liquid in server,
direct cooling techniques by single-phase narrow channel heat
exchangers and two-phase narrow heat exchangers are offered.
In addition, for the customers who recuse liquid in the server
racks, indirect liquid cooling technique by using thin heat pipes
is suggested.
Hot air to CRACs
server
Server
DT ~ 30℃
Chilled air from CRACs
Figure 1. Cooling system in server rack for conventional
air cooling technique.
To/From
Cooling network
system
Figure 2. Cooling system in server rack for new advanced
liquid cooling system.
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/interpack2011/70214/ on 07/07/2017 Terms of Use: http://www.a
In this paper, current status of technology development of
five R&D subjects of our new liquid cooling system is shown.
Then, the energy saving effect is demonstrated by verification
tests with real servers.
TECHNOLOGY DEVELOPMENTS
This liquid cooling system is consisted of following five
R&D subjects;
1) Single-phase narrow heat exchanger
2) Two-phase narrow heat exchanger
3) High performance thin heat pipes
4) Nanofluid technology for heat transfer enhancement
5) Plug-in connectors
In this section, current state of these five R&D subjects is
shown.
1) Single-phase narrow heat exchanger
For this system, the single-phase narrow heat exchanger is
expected to be adapted to up to 200W/CPU. To cool down the
CPUs, the heat exchanger is directly attached to CPUs. The
benefit of cooling capacity of single-phase narrow heat
exchanger has been well recognized.
The single-phase narrow heat exchanger is made of copper,
and its size is 7mm in thickness and 88mm x 88mm with 3mm
in depth and 6mm in width channels inside.
2) Two-phase narrow heat exchanger
Figure 3 shows the structure of a two-phase narrow heat
exchanger [4]. The two-phase heat exchanger aims to be
adapted up to 300W/CPU for HPCs or Supercomputers. This
heat exchanger consists of main channel with V-grooved
surface and auxiliary unheated channel to supply the liquid into
the main channel. These channels are separated with porous
metal plates. The area of heating surface is 30mm x 30mm.
Figure 4 shows an example of the experimental results in
horizontal configuration for heat transfer performance of two-
phase narrow heat exchanger with FC-72 as a coolant. The
vertical axis is the temperature of the heating surface and the
horizontal axis is the heat flux. The inlet liquid temperature is
35 ℃ and pressure is 0.15MPa. From this graph, cooling
capacity of 300W can be achieved when the velocity of the
coolant is higher than 0.22m/s.
Demonstration of cooling capacity. Figure 5 shows
the schematic diagram of the test section for the cooling
experiments. Dummy CPU is attached to circuit board, and air
cooling fan (see Fig.5 (a)) or the two-phase narrow heat
exchanger (see Fig.5 (b)) is installed above the dummy CPU. In
this experiment, ceramic heater is used as dummy CPU, and
surface temperature of the circuit board is recorded by
thermography from the bottom of the circuit board. For the
cooling experiment with the two-phase narrow heat exchanger,
the FC-72 is used as a coolant. Flow rate of the coolant is
1.0L/min and inlet temperature is 35℃. Figure 6(a) shows the
surface temperature of the circuit board in the case of air
cooling, and the surface temperature right under the heater
2 Copyright © 2011 by ASME
 exceeds 100℃. In addition, the temperature of the circuit board
is risen in whole. In contrast, by using two-phase narrow heat
exchanger, circuit board right under the heater is cooled to a
temperature below 90℃, and the circuit board temperature is
maintained at the room temperature as shown in Fig.6 (b).

3) High performance thin heat pipes

Thin heat pipes are used to avoid liquid intervention in the
server rack. The heat evolved from CPU is moved by heat
pipes, and the condensation part of the heat pipes is cooled
down with liquid. This cooling system with thin heat pipes is (a) Air cooling.
expected to guarantee cooling capacity up to 100W/CPU.
The authors have focused their attention to working fluid
to enhance heat transfer in the course of liquid/vapor phase
change due to Marangoni effect induced by surface tension
gradient [5]. Instead of pure water, the authors employed the
so-called “self-rewetting” fluid, which is dilute high carbon
alcohol aqueous solution, to improve the thermal performance
of the heat pipes [6]. In addition, the nanofluids are also used
as working fluids. (b) Two-phase heat exchanger.

Figure 5. Schematic diagram of test section for cooling

experiments by air cooling and liquid cooling
with two-phase narrow heat exchanger.

Figure 3. Schematic diagram of narrow channel for two-

phase heat exchanger. (a) Air cooling (heat load: 231W).
90
85 T = 35 °C
in
P = 0.15 MPa
80 in

75 300W
T , °C

70
w

65 u , m/s
in
0.17
60 0.22
100W 200W 0.28
55
0.33
50 (b) Liquid cooling with two-phase narrow heat exchanger.
0 0.1 0.2 0.3 0.4
2 6 (heat load: 278W)
q , W/m [x 10 ]
w

Figure 4. Boiling heat transfer performance of two-phase Figure 6. Surface temperature of the circuit board
narrow heat exchanger by using FC-72 as a recorded by thermometer from under the board.
coolant.

3 Copyright © 2011 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/interpack2011/70214/ on 07/07/2017 Terms of Use: http://www.a

 Thin flat heat pipes. The thin flat heat pipes are made
by flattening out the tube-type heat pipes. Its dimensions are
1.5mm in thickness, 15mm in width, and 200mm in length.
Figure 7 shows an example of thermal performance in
horizontal configuration of the thin flat heat pipes depending
on the working fluids. The vertical axis is thermal resistance
between the evaporation section and the condensation section,
R, and the horizontal axis is the thermal input, Q. All
experiments are carried out in horizontal configuration and in a
vacuum. As compared in Fig. 7, self-rewetting fluid enables
higher heat transfer rate.
Plate-type heat pipes. To aim at simplification of the
cooling system, the prototype plate-type heat pipes were
developed by heat rolling of two copper films. The size of this
plate-type heat pipe is 2mm in thickness, 30mm in width, and
200mm in length. At this stage, the thermal performances of
this plate-type heat pipes with the combination between eight
1 show outstanding stability, and it can be controlled the size and
Water
0.8 Self-rewetting fluid
Nano self-rewetting fluid
0.6
R, K/W
kinds of wick structures and four kinds of working fluids have
been examined. All experiments were carried out in horizontal
configuration. As a result, the thermal transport rate of about
70W is achieved and the thermal resistance of 0.2K/W is
obtained when the temperature at the condensation section
reaches upper limit as shown in Fig. 8. Note that the dryout did
not occur at this condition yet.
4) Synthesis and optimization of nanofluids for heat
transfer enhancement
Benefits of the inclusion of nanoparticles into working
fluids in heat transfer have been extensively demonstrated [7].
Although most of researchers prepare nanofluids by
mechanical mixing of nanoparticles in solvent, the authors have
consistently employed so-called microwave polyol method
which allows for a direct and rapid synthesis of nanofluids by
means thermo-chemical reaction in liquid state assisted by
microwave irradiation [8]. Nanofluids prepared by this method
the shape of nanoparticles by adjusting the synthesis conditions
of the thermo-chemical reaction.
Typical silver nanoparticles, spheres, flakes, and wire are
shown in Fig. 9. From the view point of heat transfer
enhancement, nanoparticles with a larger aspect ratio, such as

nano wire, are considered to be more promising because of a

0.4 larger contribution in the increase in thermal conductivity [9].
At this stage, the shape of nanoparticles can be controlled and
primary thermophysical properties, such as density, surface
0.2 tension, thermal conductivity, viscosity, have been
experimentally evaluated.
0
0 20 40 60 80 100 120 140 160
Q, W
Figure 7. Comparison of the thermal performances of thin
flat heat pipe (1.5mm x 15mm x 200mm)
among three working fluids.

0.5

0.4
(a) Spheres (f=40 nm) (b) Flake

0.3
R, K/W

0.2

0.1

0
0 10 20 30 40 50 60 70 80 (c) Wires (d=70~100 nm, l=7~10 mm)
Q, W
Figure 9. SEM images of typical shapes of silver
Figure 8. Thermal performance of thin flat plate-type heat nanoparticles.
pipe (2 mm x 30 mm x 200 mm).

4 Copyright © 2011 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/interpack2011/70214/ on 07/07/2017 Terms of Use: http://www.a

 5) Plug-in connectors
A prototype plug-in connector for the single-phase heat
exchanger and the thin heat pipes is shown in Fig. 10. Two thin
heat pipes are supported by a copper conductive attachment as
shown in Fig. 10(a). In Fig. 10(b), the assembly of the
conductive attachment and heat pipes is inserted into a narrow
channel single-phase heat exchanger. Figure 11 shows the
thermal resistance between the single-phase heat exchanger and
the copper attachment. The horizontal axis indicates heat input
to the heat pipes by ceramic heater. As shown in Fig. 11, the
thermal resistance is about 0.12K/W, and it gradually decreases
as increasing heat load. However, this experiment is conducted
without thermal grease, so the thermal resistance must be
improved if the contact condition is ameliorated.
Various types of plug-in connectors, both fluidic
connectors and thermal connectors, for liquid/vapor lines, heat
exchangers, manifolds, heat pipes, and so on are necessary to
construct the cooling network system. These connectors must
be drip free structure, and have low thermal resistance.
(a) Before insertion (b) After insertion
Figure 10. Prototype plug-in connector for thin heat pipes
and single-phase narrow heat exchanger.
0.2
0.15
R, K/W
0.1
0.05
0 consumption of servers. As a result, it is expected that energy
0 10 20
30 40 50
Q, W
Figure 11. Thermal performance of plug-in connector
without thermal grease.
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/interpack2011/70214/ on 07/07/2017 Terms of Use: http://www.a
INTEGRATION AND DEMONSTRATION OF COOLING
SYSTEMS
The authors have constructed prototype server racks
equipped with real servers. In this section, the results of the
verification tests of our liquid cooling systems by using real
server racks are shown, especially direct cooling technique
with single-phase narrow heat exchanger and the indirect
cooling technique with high performance thin heat pipes.
Direct liquid cooling technology with single-phase
narrow heat exchangers
Figure 12 gives the cooling unit with single-phase narrow
heat exchangers. The CPUs are Intel Xeon E5540 2.53GHz
80W. CPU utilization of each server is controlled by software,
and the surface temperature of CPU is measured by K-type
thermocouples. FC-72 is used as a coolant and its inlet
temperature and the flow rate is 20 ℃ and 1.0L/min,
respectively. All experiments were carried out in the ambient
temperature of 32℃. Table 1 shows an example of surface
temperature of CPUs when the CPU utilization is 100%. Air
cooling indicates that the CPUs are cooled by only six fans. In
the case of the liquid cooling-1, both six fans and single-phase
narrow heat exchanger are used for cooling of CPUs, and liquid
cooling-2 indicates only the single-phase narrow heat
exchanger is used. By using the liquid cooling system, the
surface temperature of CPUs is much lower than the air cooling
case. At this stage, heat removal rate from CPUs achieved 95%,
which is estimated to correspond to 67% of the total energy
Single-phase narrow heat exchanger
Liquid manifold
directly attached to CPU 2
Fans
Single phase narrow heat exchanger
directly attached to CPU 1
Figure 12. Cooling unit for CPUs in server with single-
phase narrow heat exchanger.
saving effect of 50 to 60% is realized by means of this system
comparing with the conventional air cooling systems by
CRACs.
For heat transfer enhancement, silver nano-water is used as
a coolant instead of water. The concentration of silver
nanoparticles is about 1.3 wt%. Figure 13 gives heat removal
quantity as a function of the coolant temperature for both water
and silver nano-water when the flow rate is 1.1L/min. Heat
removal quantity is estimated from temperature difference
5 Copyright © 2011 by ASME
 between inlet and outlet temperature of the coolant. As a result,
the improvement of the heat removal quantity is seen by using
silver nano-water. Moreover, decreasing trend of the heat
removal quantity is shown with increasing temperature of the
coolant when this experimental condition.

Table 1. Comparison of the surface temperature of CPUs

between air cooling and liquid cooling with
single-phase narrow heat exchangers.
[unit: ℃]
Air cooling Liquid cooling-1 Liquid cooling-2
Fan: 6/6 Fan: 6/6 Fan: 0/6
CPU 1 57 29.4 30.5
CPU 2 58 28.3 30.3
Ave. 57.5 28.8 30.4 (a) 36U server rack (b) Blade server rack
 CPU Utilization: 100% Pentium4 2.66GHz 66W Xeon X5670 2.93GHz
 Coolant: FC-72 CPU 95W x 2 CPUs, Tesla
 Flow rate: 1.0L/min GPU
 Inlet temperature: 20℃
Figure 14. 36U and the blade server racks with single-
phase heat exchanger.
130 Heat exchangers
Thin flat heat pipes Aluminum heat spreader
120 water attached to CPU 2
Heat removal quantity, W

silver nano-water
110
Fan
100

90

80
Aluminum heat spreader attached to CPU 1
70
10 15 20 25 30 Figure 15. Cooling unit for CPUs in server with thin heat
Coolant temperature, °C pipes and heat exchangers.
Figure 13. Comparison of the heat removal quantity Heat exchangers Plate-type flat heat pipes
between water and silver nano-water. Aluminum heat spreader
In addition, we have been working on the verification tests attached to CPU
using server racks that has 36 columns with Pentium 4
2.66GHz 66W CPU for each server, and blade sever, which has
Xeon X5670 2.93GHz 95W x 2, Tesla GPU. These server racks
are shown in Fig. 14.

Indirect liquid cooling technology with high

performance thin heat pipes
For this verification test, the CPUs are Intel Xeon E5540
2.53GHz 80W. HFE-7200 was used as a coolant and its inlet
temperature and flow rate is 14℃ and 0.7L/min, respectively.
As far as air cooling is concerned, the experiment was carried
out at room temperature of 27℃.
Figure 15 shows the cooling unit with thin heat pipes. As
shown in Fig. 15, four heat pipes are used to cool down each
CPU. A single-phase heat exchanger is used in the
Aluminum heat spreader attached to CPU 1
Figure 16. Cooling unit for CPUs in server with plate-type
heat pipes and heat exchangers.
condensation section of the heat pipes. For this system, two
fans are also used to cool the other devices.
Figure 16 shows the cooling unit with plate-type heat
pipes. One CPU is cooled by one plate-type heat pipe, and the

6 Copyright © 2011 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/interpack2011/70214/ on 07/07/2017 Terms of Use: http://www.a

 system is simplified. For this system, three fans are used
subsidiarily.
Table 2 shows an example of the surface temperature of
CPUs when the CPU utilization is 100%. On the averaged
temperature of both CPUs, there is no significant difference
between air cooling and liquid cooling with thin heat pipes. In
contrast, higher temperature is shown for the cooling technique
with plate-type heat pipes. Thereafter, plate-type heat pipes will
be adapted to the blade servers.
the blade server with plate-type heat pipes will be demonstrated
while modifying the property of the heat pipes to suit cooling
capacity we expected.
ACKNOWLEDGMENTS
We would like to thank Mr. Takashi Kitagawa, Kawamura
electric Inc., for technical support for the server racks and the
CPU utilization program. The present work has been conducted
under the Green IT Project sponsored by NEDO.

Table 2. Comparison of the surface temperature of REFERENCES

CPUs between air cooling and liquid cooling [1] Techreport, 2009. Fundamentals of data center power and
with high performance thin flat heat pipes cooling efficiency zones. White Paper #21, The Green
technique. [unit: ℃] Grid, March.
[2] Cho, J., Lim, T., and Kim, B. S., 2009. “Measurements and
Liquid cooling
Air Liquid cooling predictions of the air distribution systems in high compute
Plate-type heat
cooling Thin heat pipes density (Internet) data centers”. Energy and Buildings, 41,
pipes
Fan: 6/6 Fan: 2/6 pp. 1107-1115.
Fan: 3/6
[3] Abe, Y., Fukagaya, M., Kitagawa, T., Ohta, H., Shinmoto,
CPU 1 60.4 56.7 64.7
Y., Sato, M., and Iimura, K., 2009. “Advanced integrated
CPU 2 54.2 58.3 67.8
cooling systems for thermal management in data centers”.
Ave. 57.3 57.5 66.3 Proc. IPACK2009, IPACK2009-89009.
 CPU Utilization: 100% [4] Ohta, H., Shinmoto, Y., Ishikawa, Y., and Ariki, K., 2006.
 Coolant: HFE-72 “High heat flux cooling of large areas by improved liquid
 Flow rate: 0.7L/min supply for flow boiling in narrow channels”. Proc. 13th
 Inlet temperature: 16℃ International Heat Transfer Conference, BOI-32, p.12.
[5] Abe, Y., 2009. “Applications of self-rewetting fluids as a
CONCLUSION working fluid in heat transfer”. Int. J. Transport
New liquid cooling systems for data centers were proposed Phenomena, 11(1), pp.1-18.
for energy saving. To establish the systems, five R&D subjects, [6] Abe, Y., Tanaka, K., Mochizuki, M., Sato, M., di
which are single-phase heat exchanger, two-phase heat Francescantonio, N., and Savino, R., 2006. “Heat pipes
exchanger, high performance thin heat pipes, nanofluids with self-rewetting fluids”. Proc. 8th Int. Heat Pipe
technique, and plug-in connector, have been developed. With Symposium, pp. 76-81.
demonstration of cooling capacity with real servers, the energy [7] Godson, L., Raja, B., Lal, D. M., and Wongwises, S., 2010.
saving effect of 50~60% was expected by means of the direct “Enhancement of heat transfer using nanofluids –An
liquid cooling with single-phase heat exchanger comparing overview”. Renewable and Sustainable Energy Reviews, 14,
with the conventional air cooling system by CRACs. As far as pp. 629-641.
the indirect liquid cooling system with high performance thin [8] Tsuji, M., Nishizawa, Y., Hashimoto, M., and Tsuji, T.,
heat pipes, cooling capacity comparable to the air cooling 2004. “Syntheses of silver nanofilms, nanorods, and
system was obtained as property improvement of heat pipes is nanowires by a microwave-polyol method in the presence
in progress. of Pt seeds and polyvinylpyrrolidone”. Chem. Let. 33(4),
As the next step, quantitative advantage of the present pp. 370-371.
liquid cooling systems over conventional air cooling system [9] Zhang, X., Goo, H., and Fujii, M., 2006. “Measurements of
will be clarified in its very own server room. In the meanwhile, effective thermal conductivity and thermal diffusivity of
integration of this system with two-phase heat exchanger in real nanofluids”. Netsu bussei, 20(1), pp.14-19.
servers will be conducted. Moreover, the verification test for

7 Copyright © 2011 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/interpack2011/70214/ on 07/07/2017 Terms of Use: http://www.a