14. Design Improvement of Water-Cooled Data Centres Using Computational Fluid Dynamics

The increase in the data centre industry in recent years along with High Power Computers (HPCs) energy consumption produced rapid growth on the server’s heat density. Increasing numbers in data centre (DC) number and HPC’s power densities is leading to an increase in high energy demand. The number of internet users is predicted to increase from 3.6 billion to 5 billion between 2018 to 2025 correspondingly, along with the number of internet of things (IoT) connections, expected to increase from 7.5 billion to 25 billion between 2018 to 2025 respectively [1]. In 2018, globally, data centre electricity consumption was estimated to be 198 TWh, which is almost 1% of global electricity demand [2]. Global data centre’s energy consumption is predicted to increase by 15–20% per year in the future [3]. Cooling systems in data centres accounted for a major portion of the total data centre energy demand, which is estimated around 40% of its total energy consumption. Therefore, cooling systems effectiveness signifies the great window for the energy and cost saving. Most data centres are still using traditional air-cooled methods. However, these traditional air-cooling methods are inefficient and expensive to operate. Thus, alternative cooling methods should be considered to cope with rising cooling demand in data centres. Liquid cooling systems are one such solution which are in their early developmental stage. In this article, the use of Computational Fluid Dynamics (CFD) to further improve the design of liquid-cooled systems is discussed by predicting temperature distribution and heat exchanger performance.

HPCs are common in the recent data centres, where air-cooled systems might not be effective in cooling the electronic components of HPC. Therefore, liquid cooled systems are considered a promising solution for high heat density servers. Water-cooled systems have a higher heat transfer rate than the air-cooled systems due to the high specific heat capacity of liquid water [4]. This property of water allows it to operate effectively with lower temperature differences between server air and the coolant (water). Furthermore, water-cooled systems improve the effectiveness of heat removal and thus its energy efficiency by eliminating the need for a second heat exchange loop as required by traditional air-cooled systems [5]. Only a limited amount of published literature exists for liquid-cooled data centre systems, mainly focussing on direct liquid cooling with cold plate heat exchanger and micro channel flow [6‐10]. This study primarily focuses on rear door liquid-cooled system with intermediate air-to-liquid heat exchangers, on which, there even less published literature.

In a liquid-cooled rear door heat exchanger rack, additional fans are placed in the rear door to overcome the greater pressure drop caused by the air-to-liquid cold heat exchanger. These fans draw in cold air from outside through the front door, transfer the heat from the servers to the air and then transfer the heat from the air to a liquid via the intermediate heat exchanger. In modern data centres, several individual server racks are packed into cabinets which have a combined heat density of 30 kW. Almoli et al., suggested that water cooled rear door coolers remove 90% of the server heat effectively with the remaining 10% lost to the surroundings through radiative transfer from the cabinet surfaces [11].

Other authors have attempted CFD modelling of heat transfer inside rack cabinets and could confirm their modelling results by validation with experimental results. A study by Almoli et al. accurately predicted air flow velocities and temperature distribution using ANSYS CFD software which was in close agreement with the experimental results [11]. In this current work, a similar study has been carried out using the ANSYS radiator model as air–water heat exchanger. The current work evaluates the cooling performance of the water-cooled rear door data centre rack at high heat density (40 kW) using similar validated CFD modelling procedure as used by Almoli et al. [11]. However, their results did not consider a radiator model to predict the actual performance of the heat exchanger. So, in this current work, simulations were carried out to evaluate the heat exchanger performance by using analytical calculations which were not considered in previous studies. In this current work, a similar numerical solver was be used with radiator model as air–water heat exchanger. The current work evaluates the cooling performance of the water-cooled rear door data centre rack at high heat density (40 kW) using similar validated CFD solver procedure but with different radiator model referred by Almoli [11].

A typical 40 kW rack cabinet with rear door fans and an intermediate air–liquid heat exchanger is used in the CFD simulations to predict temperature distribution inside the cabinet and the heat exchanger performance. A 3-D model of the rack with rear door cooler has been modelled using ANSYS design modeller. The rack and heat exchanger have been constructed to represent 19-inch 42U data centre rack which has 40 kW cooling capacity. The height (Y), width (X) and length (Z) of the cabinet are 2 m, 0.6 m and 2 m respectively. The server rack is a framework where the server can be mounted. Racks usually have slots to place the servers in their designated place with the help of screws or bolts and nuts. ‘U’ refers to the standard to measure the vertical space in data server racks where 1 U = 1.75 inches. Figure 14.1 illustrates 3-D model of designed rear door unit with heat exchanger holder along with standard 42U data centre server rack and Fig. 14.2 shows the airflow direction as well as the server’s position in the cabinet.

The air flow distribution and temperature difference within the cabinet were governed by the conservation of mass, momentum and energy balance equations in the ANSYS CFD solver. The radiator is placed inside the cabinet and it acts to extract the energy from the servers’ hot air flow. The CFD solver breaks down the original naiver stokes equations into Reynolds averaged naiver stokes equations in order to solve the realistic engineering scenarios in reasonably less computational time [12]. Comprehensive description of the RANS equations can be found in the paper by Hussain et al. [13].

For example, commonly used energy equation of the 3-D RANS model expressed in the equation below, where “q” represents the heat source term and Re and Pr represent Reynolds and Nusselt numbers respectively [14].

The complete CFD Navier–Stokes solver consists of continuity, energy, turbulence and momentum RANS equations. Widely used RANS turbulence model can be categorised into K-epsilon-RNG, K-Omega and Reynolds Stress model. The K-epsilon model was used to minimise the computational time by the solver. Buoyancy effect is used in CFD solver to simulate the air density change within the cabinet due to heating. In this study, the Boussinesq approximation was used to account for the buoyancy force driving the convective motion of the fluid. The density is assumed constant in the governing equations except in the buoyancy term (Boussinesq approximation). The Computational Fluid Dynamics simulations were performed to investigate the temperature distribution and cooling performance across the cabinet in the rear door cooling configuration. Commonly used SIMPLE algorithm with higher order relaxation factor (0.75) is used in the numerical procedure to solve the K-epsilon RANS naiver stokes equation.

The key properties to consider in a water-cooled data centre cabinet are (i) the temperature rise within the cabinet and (ii) heat exchanger properties. ANSYS fluent radiator model solves the transport equation which is a derivative of energy equation for heat removal. In addition, analytical heat transfer equations were used to estimate the heat exchanger performance to determine the cooling load of the cabinet. The temperature rise inside the cabinet is based on the heat load within the cabinet. This was determined by energy balance equation which given in Table 14.1. For this application, an air–liquid fin cooled heat exchanger was used with Darcy smooth approximation for pressure drop using Gnielinski correlation. The turbulence intensity was set at 7%. More comprehensive understanding of the pressure drop, air and liquid side heat transfer properties of the heat exchanger can be found in heat exchanger textbook [16]. Based on the initial analytical calculations, the following boundary conditions were used for the CFD simulations.

The post-processing results from CFD simulations are plotted below.

Figure 14.3 illustrates the iso-surface contours of the temperature upstream the heat exchanger, which indicates the temperature rise within the cabinet reaches the 40 ℃ for 40 kW heat dissipating servers. Figure 14.4 illustrates the iso-surface contours of the temperature downstream heat exchanger. The temperature contours prove that the outlet temperature of the heat exchanger and room temperature are same. This indicates that, 40 kW heat can be removed efficiently by using rear door cooled heat exchanger without changing the room temperature, or in other words, without the need for air conditioning.

Comparative results in numerical and analytical work reveal that air inlet temperature of the server cabinet plays a key role in the temperature rise inside the cabinet. Similar performance was also spotted in the study by Almoli [11]. Figure 14.5 illustrates the temperature across the cabinet in mid Y axis plane upstream and downstream of the heat exchanger. It shows that the air is heated up to 40 ℃ at the inlet and as it passes through the heat exchanger at the rear door, it cools down to the target temperature which is about 23 ℃, the room air temperature.

In Fig. 14.6, the red and blue lines represent the temperatures in the front and back of the heat exchanger respectively. The results indicate that the hot air flow temperature from the server (40 ℃) has been cooled down to 24 ℃ after the heat exchanger as it extracts the energy from the hot mainstream air. This indicates that hot air from the server has been cooled down to the room temperature as it passes through the heat exchanger. The quantitative data from two vertical lines further illustrates the temperature profiles in Figs. 14.3, 14.4 and 14.5 Similar patters were observed in a previous study [11].

In Fig. 14.7, the horizontal lines represent the temperatures in the front and the back of the cabinet at three different heights. Results representing uniform temperature distribution can be obtained for a given air flow velocity and heat exchanger parameters. Also, results indicate sudden temperature drop downstream the heat exchanger in a uniform way, which ensures that temperature after the heat exchanger reaches the room temperature for a given heat exchanger configuration. This eliminates the need for any external cooling or air conditioning in the data centre. The results indicate that the temperature distribution predicted by the CFD solver can be used as a benchmark for future energy efficient water-cooled data centre server rack cabinets for further design improvements.

The following conclusions can be drawn from this study and the future work to further improve the solver algorithm and accuracy of the simulation are identified as follows.