Applied Energy 89 (2012) 150–155

Contents lists available at ScienceDirect

Applied Energy
journal homepage: www.elsevier.com/locate/apenergy

Computational fluid dynamic investigation of liquid rack cooling in data centres

Ali Almoli a, Adam Thompson a, Nikil Kapur a, Jonathan Summers a, Harvey Thompson a,⇑, George Hannah b
a
Institute of Engineering Thermofluids, Surfaces & Interfaces (iETSI), School of Mechanical Engineering, University of Leeds, Leeds, LS2 9JT, United Kingdom
b
Airedale International Air Conditioning Ltd, Leeds, United Kingdom

a r t i c l e i n f o a b s t r a c t

Article history: Relying on thermal air management in a data centre is becoming less effective as heat densities from the
Received 12 July 2010 Information Technology (IT) equipment continue to rise. Direct liquid cooling is more efficient at trans-
Received in revised form 21 January 2011 ferring the waste heat, but requires liquid loops passing as close as possible to the heat source. A new
Accepted 3 February 2011
Computational Fluid Dynamics (CFD) strategy is developed for data centre scenarios where a liquid loop
Available online 25 February 2011
heat exchanger is attached at the rear of server racks (back doors), which can avoid the need to separate
the cold and hot air streams in traditional hot/cold aisle arrangements. The effectiveness of additional
Keywords:
fans in the back door heat exchangers is investigated using the three-dimensional CFD model of a sim-
Data centres
Liquid cooling
plified three-aisle, six-rack data centre configuration.
Heat transfer Ó 2011 Elsevier Ltd. All rights reserved.
CFD modelling

1. Introduction In this paper we focus on thermal air flow management and

cooling in data centres, issues which are particularly challenging
The rapidly increasing energy demand of data centres is pre- since Computer Room Air Conditioning (CRAC) systems have to
senting industry and governments with an energy supply problem. maintain temperatures and humidity levels in narrow bands in or-
It is clear that the current roadmap of higher density servers with der to avoid catastrophic data losses due to over-heating servers,
no radical changes in computing technology on the horizon will hygroscopic dust or electric discharge failures. Strategies are usu-
lead to year-on-year increases in energy requirements since data ally based on the separation of hot and cold air via a layout of
centre power consumption has doubled in the last 5 years and is hot and cold aisles, but efficiencies can be gained with either of
likely to double again in the next 5 years to over 100 billion the two aisles contained. The CRAC units usually supply cold air
kW h [1]. Within the UK, the issue of thermal management in data into data centres through raised floor tiles and the cold air passes
centres is becoming critical since power supply is becoming re- through the server racks, cools the electronic equipment and
stricted in key data centre locations such as London Docklands, emerges from the back of the servers as a hot air stream. Compu-
The Thames Valley and Manchester [2]. tational Fluid Dynamics (CFD) is increasingly being used to im-
There are two main inefficiencies leading to such enormous en- prove air flow design in such systems [5,6]. The objectives of this
ergy requirements: the Information Technology (IT) hardware inef- paper are to: (i) develop a new CFD modelling strategy to investi-
ficiencies and the cooling requirements, each accounting for gate the effectiveness of liquid loop heat exchangers mounted to
roughly 40% of the total energy usage, with the result that each the rear of the IT server racks, referred to as a back door cooler,
kW h of energy for data processing requires a further kW h for and (ii) apply the CFD model to a simplified three-row, six-rack
cooling. The energy-efficient design of data centres is a truly mul- high-density data centre scenario where the back door cooler
ti-disciplinary problem. IT load inefficiencies can be addressed by either contains a series of fans (active) or no fans (passive), the lat-
improved semiconductor technologies [2] and server virtualisa- ter relying entirely on the fans in the servers to push the air
tion, while cooling of the electronics in data centres can be through the back door cooler.
achieved in a number of ways, by far the most popular at present
being via cold air, but there is a trend to adopt direct liquid cooling
[3] of the servers by, for example tube and fin heat exchangers at- 2. CFD modelling of thermal air flow in data centres
tached to the back of the server rack or, yet to be fully utilised,
dielectric liquid immersion cooling and on-chip spray cooling [4]. 2.1. Mathematical model

Although Computational Fluid Dynamics (CFD) is now used to

⇑ Corresponding author. Tel.: +44(0) 113 343 2136; fax: +44(0)113 343 2150. analyse and design thermal air flows in data centres, they are still
E-mail address: h.m.thompson@leeds.ac.uk (H. Thompson). largely unverified with respect to the accuracy of their thermal

0306-2619/$ - see front matter Ó 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.apenergy.2011.02.003
 A. Almoli et al. / Applied Energy 89 (2012) 150–155 151

predictions for large data centres [6]. In addition, the modelling investigations into the most appropriate turbulence modelling ap-
methodologies adopted for the all-important flow through server proaches are required, however this issue is not pursued here. The
racks have not been described adequately [1], making comparison two transport equations are:
with previous studies extremely difficult. In this paper we briefly  
@k 1 lt 2l
describe a modelling methodology for the thermal air flow in data þ r  ðkUÞ ¼ r  rðkÞ þ t Sij  Sij  e ð3Þ
centres with a particular focus on the representation of the server @t q qk q
racks and back door cooling units.  
Thermal air flows in data centres are usually complex, recircu- @e 1 lt e e2
þ r  ðeUÞ ¼ r  rðeÞ þ C 1e 2lt Sij  Sij  C 2e ð4Þ
lating air flows characterised by a hierarchy of different length @t q qe kq k
scales. A typical Reynolds number, Re, based on an air inlet velocity
with the turbulent viscosity defined via
from the supply vents of 1 m/s and a rack length scale of 2.4 m,
leads to an estimated Re  105 indicating the turbulent flow re- k
2

gime. Most previous CFD studies of data centre air flows have used lt ¼ qC l ð5Þ
e
Reynolds Averaged Navier–Stokes (RANS) models, see e.g. the very
recent study of Cho et al. [1]. The governing continuity and and the Sij terms are the deformation tensor. Five empirical con-
momentum equations, written in the RANS format, are: stants qk, qe, C1e, C2e, and Cl in Eqs. (3)–(5) are set equal to 1, 1.3,
1.44, 1.92, and 0.09 respectively (Boulet et al. [7]).
rU ¼0 ð1Þ The energy equation is also solved and takes the form:
  
@U 1 1 @T v vT 1
þ r  ðUUÞ ¼ r  ðr  qU 0 U 0 Þ þ S ð2Þ þ r  ðTUÞ ¼ r  þ rðTÞ þ SQ ð6Þ
@t q q @t Pr PrT qC p

where r ¼ PI þ lðrðUÞ þ ½rðUÞT Þ is the Newtonian stress tensor where T and v are the temperature and dynamic viscosity respec-
and l is the air viscosity, q its density (defined using the ideal gas tively and Pr is the Prandtl number defined by
law with constant pressure 1.013  105 Pa), U and U 0 are the aver- v k
age and turbulent fluctuation velocity vectors respectively, P is Pr ¼ where a ¼ ; ð7Þ
a qC p
the pressure and I is the unit tensor. The vector S represents the
additional momentum sources, which will be discussed in greater k is the thermal conductivity and Cp is the air’s specific heat transfer
detail below, and the qU 0 U 0 term is the so-called Reynolds stress capacity. The subscript T indicates the turbulent flow and SQ is the
tensor that requires additional model equations. source term of the energy equation, described below.
Following Cho et al. [1], the CFD models developed here use the
standard k–e model where the turbulence is described with two 2.2. Server rack and back door cooler modelling
additional variables k (turbulent kinetic energy) and e (turbulent
dissipation). This model requires the flow to be fully turbulent, The simplified data centre configuration considered is shown in
which may not be the case in all areas of the air flow within the Fig. 1. It consists of two rows of three server racks where cold air is
data centre, but the model is well tested and popular for flows that supplied from six floor vents into a cold aisle, passes the rack
have complex geometries and heat transfer. Clearly, more detailed mounted IT equipment (servers), absorbs the heat generated by

Fig. 1. Plan view of three-aisle, six-rack data centre.

152 A. Almoli et al. / Applied Energy 89 (2012) 150–155

them, and the resultant hot air flows into a hot aisle. Many data defining the following source terms in the momentum and energy
centres recirculate the hot air that flows into the hot aisle through equations respectively:
CRAC units that cool the air before it flows back out into the cold
Q rack
aisle through the supply vents. In this paper we investigate cases S ¼ KU and SQ ¼ ð8Þ
V rack
where liquid loop heat exchanger units are attached to the back
of the servers so that the hot air is cooled before it flows into the where K ij ¼  alij and aij are elements of the permeability tensor. The
‘hot’ aisle. coordinate system and hence the momentum source terms are
Previous CFD studies of data centre air flows that have appeared aligned with the principal axes of the porous medium and therefore
in the literature have provided very little explanation of the way the elements in Kij are zero apart from the diagonal ones, K11, K22
the flow through server racks are modelled. These omissions make and K33 (see [8]). Note that the momentum source term assumed
it very difficult to carry out detailed comparison with previous CFD here is linear; however more complex nonlinear models for porous
studies. media flows [9] can be imposed if necessary.
The modelling strategy adopted here is summarised in Fig. 2. In Eq. (8) a11 is the permeability of the rack in the direction of
The cold aisle is represented by region B, where the inlet air veloc- flow (front to back), Qrack is the rate of heat generation inside the
ity and temperature is specified at each of the supply vents and the rack and Vrack is the volume of the rack. The other two non-zero
governing RANS, continuity, energy, k and e equations are solved terms, a22 and a33 are both set to 1020 m2 to ensure that flow is
simultaneously. The region to the left of region B is composed of primarily through the servers from front to back. In practise, a11
a server rack (rack 1), back door cooler (back door cooler 1) and can be estimated experimentally by measuring the pressure drop
a hot aisle (region A). In practise, rack 1 forms a complex geomet- across the rack for a range of flow rates, but its value would be
rical obstacle to the air passing through it and the processors act as dependent on the type of IT equipment. The rate of energy gener-
sources of thermal energy into the air stream [5]. Previous CFD ation by the IT equipment can be estimated at a particular server
studies of flow through the servers have ignored the small-scale load from manufacturers’ specifications. Results presented here
geometric features and modelled them in a highly averaged sense have used an estimated a11 = 1.8  105 m2, which is calculated
[6]. In the present study we treat the rack as a porous medium by by equating the pressure drop in a Hagen–Poiseuille flow in a pipe

Fig. 2. The solution procedure of the governing equations and the boundary condition.
 A. Almoli et al. / Applied Energy 89 (2012) 150–155 153

Table 1 The hot air emerging from the rear of rack 1 passes into an air
Heat load per rack and the source term. gap within which the same RANS, continuity, energy, k and e equa-
Heat load per rack (kW) Source term per rack (W/m3) tions as in region B, are solved. The air then flows into the back
15 7102.2 door cooler 1, which constitutes a further obstruction to the air
25 11,837 flow and which is again modelled by an additional porous medium
30 14204.5 by adding the source term in Eq. (8). In the illustrative results pre-
sented here, the permeability values, aij, for the back door cooler
are set to the same values as for the racks.
The energy removing properties of the back door cooler are
Table 2
Variation in rack intake temperature as a function of number of mesh cells.
modelled by treating it as a radiator which removes energy from
the air stream at a rate given by
Number of Cells Intake temperature of the rack (K)
68,788 294.1
78,336 294.5 Q_ bdc ¼ mhðjUjÞAðT
_ ref  T air Þ
87,884 295.0
169,442 295.0
417,358 295.0 where m _ is the mass flow rate of air across the back door cooler,
hðjUjÞ and A are its average convective heat transfer coefficient at
airspeed jUj and cross-sectional surface area respectively, Tair is
whose diameter is equal to the hydraulic diameter of a typical 1U the temperature of the air stream and Tref is the set point tempera-
server. The heat load and the source term for each case are shown ture of the back door cooler. In practise the heat transfer coefficient,
below in Table 1. h, for the back door cooler will be a function of air speed through
Each server rack is split into four sections representing a cluster the cooler. The required empirical relationship of the form hðjUjÞ
of rack servers or a blade cluster of approximately 10U. Since the can be determined by using experimental measurements of the dif-
air flow through the rack is aided by server fans at its rear (indi- ference between air on and air off temperatures as a function of air
cated by the left orange line in Fig. 2) this is modelled by a pressure speed. Following Tang et al’s [11] results for a plane fin and tube
drop across the rear rack boundary, which is given as a cubic poly- heat exchanger, in all calculations presented here, the value
nomial in terms of the speed of air flow normal (given by vector n ^) h ¼ 900jUj0:71 is used based on a best-fit against experimental data
to the boundary and is based on a Comair Rotron fan [10], for a passive back door cooler.
Back door cooler 1 is an example of a passive back door cooler
^ and
V ¼U n since the air flow through it is not driven by mechanical means,
DP ¼ 948:4  131:39V þ 16:2V 2  1:154V 3 ðfor the three lower server clustersÞ but relies on the pressure generated by the fans of the IT equip-
DP ¼ 1481:87  164:23V þ 16:2V 2  0:923V 3 ðfor the upper server clusterÞ ment. Back door cooler 2, however, is an example of an active back
door cooler which has additional fans at its rear to increase the
The above fan curve expressions can be adjusted based on fan speed of air flow through the back door cooler. Although this
speed using a relationship between
 2 the fan speed and the static clearly requires additional energy to power the fans, it can be seen
pressure that is given as PP12 ¼ NN12 , where N1 and N2 are the first from the relationship between h and |U| that the increased air flow
and second fan speeds (RPM) respectively and P1 and P2 are the will lead to a higher average heat transfer coefficient, and hence
associated static pressures (Pa) at these fan speeds respectively. higher heat extraction rate, from the air stream.

Fig. 3. Velocity profile (left), colour indicates airflow magnitude with yellow being the highest speed; and temperature profile (right), blue for cool and red for hot for the
active back door cooler at 30 kW. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Table 3
_ CRAC and m
Effect of using back door coolers on the cooling load of CRAC units with six 30 kW racks, where m _ BD are the mass flow rate through the
CRAC and back door respectively.

Configuration Heat load on CRAC Heat load on back door

CRAC 180 kW 0 kW
CRAC + passive m_ CRAC C p ðT OUT  T IN Þ ¼ 14:5 kW m_ BD C p ðT B4  T AFTER Þ ¼ 164:8 kW
CRAC + active m_ CRAC C p ðT OUT  T IN Þ ¼ 11:5 kW m_ BD C p ðT B4  T AFTER Þ ¼ 170:8 kW
154 A. Almoli et al. / Applied Energy 89 (2012) 150–155

Table 4
The intake and exhaust temperatures (T) for both active (black curves) and passive (red curves) back door coolers for different heat loads (kW). (For interpretation of the
references to colour in this figure legend, the reader is referred to the web version of this article.)

Heat dissipation load (kW) Rack intake temperature Rack exhaust temperature
15 kW

25 kW

30 kW

3. Results and discussion racks, where it is heated by the IT loads within the servers and then
passes into the hot aisles at the back of the two rows of servers. Fi-
The CFD model for the server rack in a simplified data centre nally the hot air in the hot aisles is extracted via the outlet vents in
configuration as described above is solved using the commercial the roof of the simplified data centre. Fig. 3 (left) shows the air flow
CFD package, ANSYS FLUENT 12.0.1. Within the CFD mesh, surfaces via colour coded velocity vectors and it is possible to see that the
and volumes are represented using quadrilateral and hexahedral air speeds are greatest in front of the servers where the air is pro-
elements respectively. Results were obtained on five different grid pelled into the room through the flow tiles, and in the case of active
sizes using between 68,788 and 417,358 cells as highlighted in Ta- back doors, the air speeds to the rear of the server racks is in-
ble 2. The table demonstrates that the results are mesh indepen- creased by the additional fans in the back door cooler. Fig. 3 (right)
dent when using a mesh with 87,884 cells as it can be seen that shows the corresponding temperature distribution.
for larger mesh sizes there is no variation in the rack intake tem-
perature. Steady state solutions were obtained using the pressure 3.2. Comparison of passive and active back door coolers
based method for solving the governing equations using the de-
fault relaxation factors. A brief analysis of the effectiveness of both Table 3 shows an example of how the use a back door cooler can
the active and passive back door coolers is presented in Table 3. reduce the cooling load for a CRAC unit. The data shown is for a
case in which each rack adds 30 kW of thermal power into the
3.1. Air flow within the data centre air streams. When CRAC units are used without the assistance of
a back door cooler, all of the heat transferred into the air stream
Air enters the data centre via the floor vents where an inlet tem- has to be removed by the CRAC unit, namely 180 kW (six racks
perature, TIN, is maintained by the CRAC unit supplying cool air to at 30 kW each).
the under floor plenum. The cool air rises in the cold aisle, which is When the passive back door coolers are used, these remove
in front of the server racks and is then drawn through the server approximately 165 kW from the air streams with the result that
 A. Almoli et al. / Applied Energy 89 (2012) 150–155 155

the load on the CRAC units is reduced to only 14.5 kW. Finally, exchanger is attached at the rear of the server racks. This enables
when an active back door cooler is used, the higher heat transfer the potential benefits of liquid back door cooling versus traditional
coefficient that results from higher air speeds through it enables air cooling in terms of reduced load on the CRAC units to be assessed
the load on the CRAC units to be reduced further to only against the need to cool the recirculating water, which could in prin-
11.5 kW. Clearly the precise values of these benefits will depend ciple be achieved by free cooling in cold climates. The additional
critically on the back door cooler specifications. The cost benefit benefits of using active versus passive back door cooling can also
analysis of using liquid cooling via the back door scenarios will be assessed by comparing the further reduced load on the CRAC
need to take account of the need to reduce the temperature of units against the energy needed by the active back door cooler’s
the water before it returns to the rear of the rack, but there are fans. Detailed validation of the CFD methodology is required before
benefits in using liquid in that its specific heat capacity is much it can be used in practical data centre design scenarios.
greater than air and the temperature ranges could make use of free
cooling in certain geographical locations. References
Table 4 shows CFD predictions of the rack intake and exhaust
temperatures for the passive and active back door cooler specifica- [1] Cho J, Lim T, Sean Kim B. Measurements and predictions of the air distribution
tions described earlier, for heat dissipation rates of 15, 25 and systems in high compute density (Internet) data centers. Energy Build
2009;41:1107–15.
30 kW per rack. As expected, the exhaust air temperatures leaving [2] Cushing R, Doherty J. Next generation data centres. MSC Business
the active back door cooler are lower than those associated with Development Ltd.; 2009.
the passive cooler. For the parameters used here, the exhaust tem- [3] Cader T, Westra L, Sorel V, Marquez A. Liquid cooling in data centers. ASHRAE
Trans 2009;115(pt 1):231–41.
peratures with the active back door cooler are typically between 1 [4] Anandan SS, Ramalingam V. Thermal management of electronics: a review of
and 2 °C less than for the passive case with a heat dissipation load literature. Therm Sci 2008;12(2):5–26.
per rack of 30 kW. [5] Choi J, Kim Y, Srebric ASivasubramaniamJ, Wang Q, Lee J. A CDF-based tool for
studying temperature in rack-mounted servers. IEEE Transactions on
Computers 2008;57(8):1129–42.
4. Conclusions [6] Shrivastava SK, Iyengar M, Sammakia BG, Schmidt R, VanGilder JW.
Experimental-numerical comparison for a high-density data center: hot spot
heat fluxes in excess of 500W/ft2. IEEE Trans Compon Packag Technol
Relying solely on air as the primary heat transfer medium in data 2009;32(1):166–72.
centres is becoming increasingly problematical due to inexorable [7] Boulet M, Marcos B, Dostie M, Moresoli C. CFD modeling of heat transfer and
increases in power densities from IT equipment. Although CFD is flow field in a bakery pilot oven. J. Food Eng 2010;97(3):393–402.
[8] Liakopoulos AC. Darcy’s coefficients of permeability as symmetric tensor of
increasingly being used to improve air flow design in data centres, second rank. J Hydrol Sci 1965;10(3):41–8.
few details of the means by which the all-important flow through [9] Bear J. Dynamics of fluids in porous media. Dover Books; 1988.
server racks is modelled have appeared in the literature to date. This [10] Comair rotron company. Fan specification based on Tarzan AC – TNE2A –
020174 (http://www.comairrotron.com).
is an important omission since it makes comparison and validation [11] Tang LH, Zeng M, Wang QW. Experimental and numerical investigation on air-
with other studies difficult. A new CFD methodology has been devel- side performance of fin-and-tube heat exchangers with various fin patterns.
oped for simulating data centre scenarios where a liquid loop heat Exp Therm Fluid Sci 2009;33:818–27.