Conjugate Heat Transfer Modeling of a Cold Plate Design for Hybrid-Cooled Data Centers

Abstract

Liquid-cooled servers can be deployed to reduce the energy consumption and environmental footprint of hybrid-cooled data centers. A computational fluid dynamics (CFD) model can bring extremely useful insights and results for thermal simulations of air- and liquid-cooled servers in a single environment. In this study, a conjugate heat transfer (CHT) numerical model is developed and validated with experimental data to simulate heat transfer from the CPU to the air and cold plate considering the effect of thermal paste. The cooling performance of an in-house developed cold plate design is thoroughly investigated via the validated CHT model. A dataset containing one hundred samples of various flow, thermal and workload conditions was generated using the Latin hypercube sampling (LHS) method, which was further utilized in the series of CHT simulations. Finally, a novel empirical equation is proposed for the prediction of heat transfer from the CPU to the air. The accuracy of the proposed equation is confirmed by comparing estimated and simulated results for a test dataset. A thermal analysis of a rack containing air and liquid-cooled servers is performed using the presented approach. The simulation results reveal that the proposed compact model can be used reliably for the thermal simulation of a hybrid-cooled data center.

1. Introduction

The tremendous amount of energy usage and the carbon footprint of data centers were significant challenges for public authorities in the last decade. Liquid-cooled servers can be deployed in a data center to minimize the environmental footprint with the utilization of renewable energy technologies. This can be achieved by reducing the power consumption of cooling devices, since liquid-cooled systems are more efficient than their traditional air-cooled counterparts. In order to enhance the energy efficiency of a data center, air-cooled servers can be replaced by, or transformed into, liquid-cooled servers with the advantage of the open hardware architectures of Open Compute Project (OCP) servers. The thermal structure of a data center can be simulated using a computational fluid dynamics (CFD) model in a realistic way to ensure that the cooling system can effectively keep central processing unit (CPU) temperatures under the maximum value recommended by the manufacturer.

In particular, the thermal simulation of a liquid-cooled server considering heat convection and conduction along the cold plate is crucial to the thermal simulation of a hybrid-cooled data center containing air- and liquid-cooled servers. The thermal performance of a liquid-cooled server is mainly influenced by the design of the cold plate. Sun et al. [1] proposed a new cold plate design focusing on the flow channel, flow impedance and thermal resistance. They proposed an open-loop liquid cooling solution and reported that fan speeds could be minimized using this approach. Bench-level experiments conducted by Ramakrishnan et al. [2] for different liquid flow rates and chip power showed that the thermal resistance of the cold plate decreased with the increasing flow rate. Gao et al. [3] developed a liquid cooling system in which cold plates were used for the cooling of high-density processors and GPU accelerators. On the other hand, compact server models have gained importance in the fast and accurate thermal modeling of data centers. Van Gilder et al. [4] proposed a compact model for the transient modeling of servers. Such a black-box server model successfully estimated the ambient temperature depending on the time varying exhaust temperatures. Pardey et al. [5] reviewed progress in the transient server modeling to date and proposed a compact server model for data centers. Thermal capacitance and efficiency, server mass, server size and their relationship were investigated based on experimental measurements. Ibrahim et al. [6] proposed a compact server model to represent server components in the numerical modeling of a data center. A thermal mass was implemented to the server by adding density and specific heat to the rack. The proposed thermal mass approach led to an improvement in determining the temperature distribution within the data center. Manarseh et al. [7] focused on the proper management of airflow, temperature and energy to eliminate energy consumption using the cooling system and developed a machine learning augmented compact model. The proposed model accurately predicted both the power consumption of the IT equipment and the flow rate, as well as the exhaust air temperature. The power consumption of the compact model was obtained as a function of the CPU usage. It was confirmed with experiments that the maximum error was 5.7% and 11.4% for the outlet temperature and flow rate, respectively.

Deployment of liquid-cooled servers has become widespread for the reduction of power consumption and waste heat in hybrid-cooled data centers. Recirculation effects that may occur in a conventional air-cooled data center can also be eliminated using liquid-cooled servers. Ellsworth and Iyengar [8] performed a series of analyses of liquid-cooled data centers and found that the energy efficiency could be increased using a liquid cooling system, which could save up to 45% of the operational energy at a data center. It was also determined that the use of the waterside economizer at the plant level could reduce the cooling power of the plant by more than 90%. Liquid cooling can be applied to IT (information technology) equipment without direct contact between the heat source and liquid cooler by replacing the air-cooled heat sink with an evaporator or other liquid-cooled heat sinks, such as cold plates and water blocks [9,10]. The coolant distribution unit provides chilled liquid that can be controlled from an external cooling source, connected to electronic devices with an indirect liquid cooling system. Liquid cooling applications are designed to focus mainly on the cooling of the CPUs, since the CPUs are the main source of heat generation in a data center. The remaining components of IT equipment are generally cooled by the air [11,12]. As the coolant used in a liquid cooling system has a higher heat capacity and conductivity than air, the energy efficiency in a data center can be enhanced by allowing higher coolant temperatures and lower airflow rates [13]. Electrically insulated liquid coolers are in direct contact with the electronic components [14]. Zimmermann et al. [15] provided a temperature difference of 15 °C, which was required for the cooling of IT equipment in a data center with a liquid-cooling system. Han et al. [16] designed and analyzed a hybrid micro heat sink for the thermal management of processors in a liquid-cooled data center. High cooling could be achieved with low pumping power. A flow rate of 500 mL/min and a pressure drop of 0.8 kPa, as well as a thermal resistance of 0.13 °C cm²/W could be obtained with the heat sink. It was revealed that the increase of the flow rate to 1 L/min could significantly reduce thermal resistance and the designed heat sink could save cooling energy costs while maintaining high heat removal in the data center. Sridhar et al. [17] modeled a liquid-cooling system with a compact transient thermal model using a micro channel heat sink design in comparison to experimental data obtained from real liquid-cooled integrated circuits. Carbo et al. [18] designed and tested a micro data center with 1.2 kW IT power capacity and experimental studies showed that the maximum outlet water temperature was approximately 50 °C. Rubenstein et al. [19] adopted a hybrid data center approach to reduce energy consumption and developed higher computing density in high-density data centers. The prior literature focused on cooling efficiency by changing the coolant, decreasing the outlet temperature of the server with a certain black-box approach, the effect of the technology for the waste heat reuse, improving cold plate designs, and the placement of components in servers with cold plate technology [20,21,22,23,24,25,26]. Unlike other studies in the literature, this study benefits from a three-dimensional conjugate heat transfer (CHT) model consisting of multi regions for the simulation of heat transfer from the air to the liquid, based on an extensive Latin hypercube sampling (LHS) dataset. Finally, a new method is proposed for the modeling of liquid-cooled servers as air-cooled servers, based on the energy balance between the air, CPU and cold-plate.

In the current work, we focus on the development of a CHT model for the thermal simulation of hybrid-cooled data centers. In particular, we focus on the thermal modeling of a cold plate design implemented to an OCP server for the recovery of waste heat from data centers. A simplified CFD model is presented for the modeling of air- and liquid-cooled servers in a computational framework. The accuracy of the CHT model is validated by comparing numerical results with experimental data for natural and forced convections. An extensive dataset is created using the LHS method depending on the thermal and Intelligent Platform Management Interface (IPMI) data of an operating hybrid-cooled data center. A series of CFD simulations are performed for the generated dataset using the validated CHT model and an empirical equation is proposed based on the CFD results. The model accounts for the inlet temperature and the Reynolds number of air and water entering the server, as well as the power consumption of each CPU. Finally, a thermal analysis of a hybrid rack is performed using the proposed model to demonstrate the application of the present approach for hybrid-cooled data centers.

2. Methods

2.1. Governing Equations

Heat transfer between air, water and the cold plate can be accurately modeled using a multi-region CHT model, where fluid and solid regions are modeled separately. Continuity, momentum and energy equations are solved at the air and water parts considering turbulence and buoyancy effects, and the energy equation is solved at the solid part for heat conduction. Compressible and turbulent flow in the fluid region can be represented with the following unsteady continuity and momentum equations in tensor form using Reynolds averaging:

$$\frac{\partial \rho }{\partial t}+\frac{\partial \rho u_j}{\partial x_j}=0$$

(1)

$$\frac{\partial }{\partial t}\left(\rho u_i\right)+\frac{\partial }{\partial x_j}\left(\rho u_i{u}_j\right)=-\frac{\partial p^{\prime }}{\partial x_i}+\frac{\partial }{\partial x_j}\left(\mu \frac{\partial u_i}{\partial x_j}\right)+\frac{\partial }{\partial x_j}\left(-\mathsf{\rho }\overline{u_i^{\prime }{u}_j^{\prime }}\right)$$

(2)

where $\rho$ is the density, $t$ is the time, $u_i$ is the velocity component in the $x_i$ direction, $p^{\prime }$ is the modified pressure $p^{\prime }=p+\frac{2}{3}\rho k+\frac{2}{3}\left(\mu +{\mu }_t\right)\frac{\partial u_k}{\partial x_k}$, and $\mu$ and ${\mu }_t$ are molecular and turbulent viscosities, respectively. The turbulent stresses in Equation (2) can be calculated from the following Boussinesq approximation:

$$-\rho \overline{u_i^{\prime }{u}_j^{\prime }}={\mu }_t\left(\frac{\partial u_i}{\partial x_j}+\frac{\partial u_j}{\partial x_i}\right)-\frac{2}{3}k{\delta }_{ij}$$

(3)

where $k$ is the turbulent kinetic energy and ${\delta }_{ij}$ is the Kronecker delta. In this study, the $k-\omega$ SST turbulence closure modeled is used for the modeling of turbulence effects in the air and water regions. Two transport equations are solved for the kinetic energy $k$ and specific dissipation rate $\omega$ [27]. Then, the turbulent viscosity can be calculated from the following equation:

$${\mu }_t=\frac{\rho a_{1}k}{max\left(a_1\omega ,S{F}_2\right)}$$

(4)

where $S$ is the strain rate invariant and $a_1$ is the model constant [20]. The energy equation is solved for the compressible flow:

$$\frac{\partial }{\partial t}\left(\rho E\right)+\frac{\partial }{\partial x_i}\left[u_i\left(\rho E+p\right)\right]=\frac{\partial }{\partial x_j}\left[{\lambda }_{eff}\frac{\partial T_f}{\partial x_j}+u_i\frac{\partial }{\partial x_j}\left(\mu \frac{\partial u_i}{\partial x_j}-\mathsf{\rho }\overline{u_i^{\prime }{u}_j^{\prime }}\right)\right]+S_E$$

(5)

where $E$ is the total energy and $S_E$ is the source term. The following energy transport equation is solved for the solid region to consider the temporal change of the enthalpy with respect to the divergence of the heat conducted through the solid:

$$\frac{\partial \left({\rho }_s{h}_s\right)}{\partial t}=\frac{\partial }{\partial x_j}\left({\alpha }_s\frac{\partial T_s}{\partial x_j}\right)+S_i$$

(6)

where subscript $s$ shows the solid part, $h_s$ is the specific enthalpy, ${\alpha }_s$ is the thermal conductivity and $S_i$ is the volumetric heat source. Note that radiation effects are not considered in the present CHT model.

2.2. Numerical Model

Numerical simulations were performed using open source CFD code OpenFOAM 8, which employs a finite volume method for the discretization of the governing equations over a control volume with a PIMPLE algorithm, which is a pressure-based algorithm for velocity and pressure coupling as the combination of the PISO (Pressure Implicit with Splitting of Operator) and SIMPLE (Semi-Implicit Method for Pressure-Linked Equations) algorithms. Convective terms were discretized using a second-order accurate linear-upwind divergence scheme and gradient terms were discretized using a second-order accurate Gauss linear method. A limited Gauss linear method was used for the discretization of Laplacian terms with a limiting coefficient of 0.33 to increase the stability of the numerical solution for a skewed and non-conformal mesh. Thus, the second-order accuracy was satisfied for the minimization of truncation errors arising from the discretization of the non-linear partial differential equations. Coefficients of the discretized equations were stored in the matrices for the implicit solution of the governing equations in a stable way. The diagonal preconditioned conjugate gradient (PCG) and preconditioned bi-conjugate gradient stabilized (PBiCGStab) solvers were used for the approximate solutions of matrices for density, pressure and velocity, respectively. The residuals were set to 1 × 10⁻⁶ for the matrix solutions. Numerical simulations were performed with parallel computing using 23 processors on the Blockheating^© Data Center.

The geometry and mesh of the multi-region domain were created using the Salome software, which is an open source software including several modules such as CAD (computer-aided design), mesh and visualization for engineering applications. Mesh algorithms are available in the Salome platform for various mesh types, such as NETGEN (an automatic mesh generation tool), GMSH (a three-dimensional finite element mesh generator) and MG-Tetra (a commercial mesher). In this study, NETGEN and GMSH algorithms were used to discretize surfaces and volumes depending on the grid requirements. One can generate a single mesh with various cell zones and split the grid into several regions with the splitRegion tool of OpenFOAM to generate a multi-region mesh. However, each region was created separately in this study to create a boundary layer near the solid boundaries of each fluid region.

2.3. Modeling Thermal Paste

Regions are coupled in a multi-region solver applying thermal conditions on the boundaries between neighboring regions. Temperature and heat flux conditions are satisfied on the boundary between fluid and solid regions as $T_f=T_s$ and $Q_f=-Q_s$. Where $T$ and $Q$ are the temperature and heat flux on the boundary, respectively, and subscripts $f$ and $s$ denote the fluid and solid regions, respectively. Using these two conditions, Equation (8) can be obtained. The heat flux vector $q_i$ can be calculated on the boundary from the following equation:

$$q_i=k{n}_i\frac{\partial T}{\partial \mathrm{n}}$$

(7)

where $k$ is the thermal conductivity and $n$ is the surface normal vector. The temperature at the solid boundary can be calculated from the energy balance using the following expression:

$$T_b=\left(\frac{k_s}{\delta n_s}{T}_s^i+\frac{k_f}{\delta n_f}\frac{1}{T_f^i}\right){\left(\frac{k_f}{\delta n_f}+\frac{k_s}{\delta n_s}\right)}^{-1}$$

(8)

where $T_b$ is the temperature at the boundary; $T_f^i$ and $T_s^i$ are temperatures at the internal fluid and solid cells near the solid boundary, respectively; and $\delta n_s$ and $\delta n_f$ are distances from the cell center to the face center of the solid and fluid cells near a shared boundary, respectively. Note that thermal coupling is achieved with Equation (8) between two neighboring regions in a single loop.

As shown in Figure 1, temperatures at the boundaries of the solid and heater regions are not identical when a thermal paste is applied between the solid and the heater to increase the thermal conductivity between the CPU and the cold plate. Thus, the heat flux through the boundary is equal to the heat flux through the layers of the materials between regions:

$$k_s\frac{\partial T}{\partial n}=R∆T$$

(9)

where $R$ and $∆T$ are the contact resistance and temperature difference between solid and heater regions. Thus, the temperature at the boundary of the solid region can be calculated as:

$$T_s^b=\frac{R}{k_s/\delta n_s+R}{T}_h^b+\left(1-\frac{R}{k_s/\delta n_s+R}\right)T_s^i$$

(10)

where $R=k_L/t_L$ is the contact resistance, and $k_L$ and $t_L$ are thermal conductivity and thickness of the material between solid and heater regions, respectively. Finally, the $∆T$ is applied on the boundary as a jump condition as shown in Figure 1b to include the effect of the thermal paste without solving governing equations along the thin layer.

3. Results

3.1. Validation of the CHT Model

The present multi-region solver is validated with experimental data of the external pipe problem [28] shown in Figure 2 for natural and forced convection conditions. Heat is convected from the hot water inside the pipe to the copper, then conducted through the copper pipe and the pipe is cooled down by the surrounding cold air. The present problem is a good test problem, since the heat convection and conduction processes between different fluid and solid regions will be simulated using the present multi-region heat transfer model.

Details of the geometrical, flow and thermal conditions are provided for natural and forced conditions in

Table 1. Parameters of the pipe simulation setup.

Case	Variable	Value	Unit
-	Pipe inner diameter	0.0283	m
	Pipe outer diameter	0.0442	m
	Pipe length	0.0884	m
	Size of air domain	1.55 × 2.65 × 0.0884	m3
Free Convection	Water temperature	315	K
	Water velocity	5	m/s
	Air temperature	300	K
	Air Prandtl number	0.7	-
	Raleigh number	0.86 × 105	-
Forced Convection	Water temperature	330	K
	Water velocity	5	m/s
	Air temperature	300	K
	Air Prandtl number	0.7	-
	Air velocity	0.0489	m/s
	Reynolds number	130	-

. While the surrounding air temperature remains constant at 300 K for natural and forced convections, water temperature increases from 315–330 K in the forced convection. Turbulence effects were not considered in both air and water regions due to the low Reynolds number flows, even for the forced convection. The Sutherland viscosity model is used to model the air viscosity considering thermal conditions. The far boundaries are maintained with ample width to mitigate the impact of boundary conditions on the solution. Boundary conditions selected for the computational model include cyclic boundary conditions for the front and back faces, a zeroGradient boundary condition for the top face, and a noSlip boundary condition for the left, right and cylinder surfaces. In the case of the bottom boundary condition, pressureInletOutletVelocity is chosen for the natural convection scenario, while a velocity of 0.0489 m/s is prescribed in the z direction for the forced convection scenario. The temporal terms are discretized using a first-order accurate implicit Euler scheme.

Four meshes are created with different resolutions and provided in

Table 2. Mesh-independence study for instantaneous and time-averaged data.

Mesh	Number of Cells			Error (%)
	Air Region	Water Region	Copper Region	Instantaneous	Time-Averaged
Mesh 1	77,488	11,344	4400	5.94	4.84
Mesh 2	395,296	17,344	8784	4.43	4.19
Mesh 3	512,320	23,200	11,120	3.18	2.41
Mesh 4	1,503,488	22,288	13,616	2.73	2.70

for each region. Instantaneous and time-averaged results are compared with the experimental data in Figure 3 to obtain a mesh-independent distribution of the thermal field. The variation of the Nusselt number (Nu) with the theta, which is the angle between direction of the flow and the surface, shows that Mesh 3 yields the most accurate results among the tested mesh resolutions for the instantaneous data. The time-averaged results are found to deviate from the experimental data downstream of the cylinder for all meshes, which may be associated with the uncertainty originating from the experimental measurements. Area-averaged values of the Nu are first calculated over the cylinder from the experimental and numerical data. Then, an error is calculated between the numerical and experimental data. Errors calculated from the instantaneous and mean values in Table 2 also show that Mesh 3 can be used reliably for the simulation of the heat transfer over the circular face. The results of Mesh 4 deviate from the experimental data when the theta is greater than 130, since the small eddies captured with the fine mesh result in a fluctuating thermal field on the server.

A two-dimensional surface grid was generated for each region and then extruded along the normal surface to create a three-dimensional mesh. Quadrilaterals were generated to represent surfaces in the water and copper regions, whereas triangles were generated in the air region. As shown in Figure 4, viscous sublayers were added to the air and water regions to capture boundary layer effects accurately. The extruded mesh contains prisms at the far field and hexahedrals near the boundary layers. The maximum non-orthogonality and skewness of the mesh were determined as 35.45° and 0.71, respectively, which are acceptable for the solutions of the gradients and interpolations on the face centers [29]. The number of the cells in the air, copper and water regions are 512,320, 11,120 and 23,200, respectively. The computational domain is decomposed to subdomains using the scotch algorithm in OpenFOAM and each subdomain is solved with a single thread of the processor. The subdomains are communicated to each other using a message passing interface (MPI) during parallel computation [29].

The present unsteady solver converges to the steady-state between two successive time steps using approximately 25 iterations for the selected tolerance of 1 × 10⁻⁶. The initial residual reduces from about 1 × 10⁻³ to the selected tolerance using the highest iteration number for the pressure. The maximum Courant number is set to 0.1 and the time step is adjusted according to the Courant number during unsteady simulations in validation cases. The calculated time step size is observed to vary by approximately 0.0001 s during unsteady CHT simulations, which is the main reason for long simulation durations. Temperature distributions are shown for natural and forced convection cases in Figure 5. The heat transfer from the hot water and the copper pipe to the air generates a plume behind the circular pipe at a certain distance from the pipe due to buoyancy effects. In addition to density difference, the momentum difference in the forced convection resulted in a stronger vortical flow downstream of the pipe. Thus, the present solver can simulate heat transfer between water, pipe and air, as well as density- and momentum-induced vortical flows for natural and forced convections.

The area integrated value of the Nu over the cylinder surface is continuously logged during the simulation and plotted in Figure 6 to capture periodical variation in the surface value. The plot is generated during a periodic stage when the Nu reaches peak to observe peak and trough values in one period. The vortex shedding observed in Figure 5b resulted in a periodic variation of the Nu with a period of T = 0.03 s for the present configuration. The Strouhal number $St=UT/D,$ where $U$ is the inlet velocity and $D$ is the diameter, is then calculated as 0.032. This observation confirms that the present CHT model can capture unsteady flow and thermal effects acting on the cylinder.

Distributions of the Nu on the cylinder are calculated based on the instantaneous data and compared with the experimental measurements for free and forced convections in Figure 7. A function object was developed and incorporated to the open source code for the accurate calculation of the Nu, depending on the local thermal and flow features near the boundary. The function object is publicly distributed in the GitHub repository. The consistency observed between the numerical and the experimental data confirms that the present open source CHT model can accurately calculate convective and conductive heat transfer between different regions, considering the thermo physical features of each region. OpenFOAM files of the validation cases are publicly shared via the public repository Zenodo [30] to enable the reproduction of identical results on different platforms.

3.2. Thermal Simulation of the Cold Plate

The cold plate was designed and applied to the Leopard V1 model of the OCP server using Blockheating^© waste heat recovery from the data center to greenhouses, farms, and industry in a sustainable way. The layout of the Blockheating Data Center is shown in Figure 8. The number of air- and liquid-cooled servers deployed in the data center are 42 (11%) and 326 (89%), respectively, having the same cold plate design. Thus, the present data center is a hybrid-cooled data center, which brings a challenge to the fast thermal simulation of air- and liquid-cooled servers in a single environmental. In order to address this challenge, the validated CHT model is employed for the prediction of the energy balance of the present cold plate used in the whole data center. Otherwise, detailed modeling of the cold plate over the complete model of a data center would increase required computational memory and time enormously.

The geometry of the computational domain and micro-channel design are depicted in Figure 9. Air enters the domain at the inlet patch and leaves the domain at the outlet patch of the rectangular domain. The remaining boundaries are defined as solid walls. Water is maintained by the inlet pipe as a constant flow rate and temperature to the first part of the cold plate and the heated water emerges from the system by an outflow pipe. The serpentine micro-channel shown in Figure 9b yields the distribution of the heat from the CPU to the cold plate to cool down the CPU effectively and to provide hot water at an expected temperature at the outlet in terms of waste heat recovery from the data center. Minor effects of the internal components such as RAMs, peripheral component interconnect (PCI) cards and microchips on the motherboard were excluded from the present CFD model to reduce mesh size and simulation durations, which will be critical in the latter part of this study to perform CFD simulations in a broad range of input data. In this study, CPUs are considered as the main heat sources and an externalWallHeatFluxTemperature boundary condition of OpenFOAM was used to apply heat flux to the temperature field on the CPU. The thermal conductivity and thickness of the thermal paste between the CPU and cold plate are 6 W/m/K and 0.1 mm, respectively.

Figure 10 shows multi-region meshes for air, water and solid regions. Boundary layers were added to the air and water regions to accurately predict adverse pressure gradients and boundary layer separation near the walls. Different mesh features at each region resulted in a non-conformal mesh over the interfaces between regions. The present solver can calculate gradients and fluxes on the interface boundaries in a non-conformal mesh [30]. The multi-region mesh consists of 832,813 cells in the air, 248,570 cells in the copper and 711,974 cells in the water region. The maximum non-orthogonality and skewness of the mesh were determined as 81.74° and 2.91, respectively. A non-orthogonal correction with three steps was applied to the air region to reduce errors arising from non-orthogonality. The non-orthogonal correction is applied to the PISO algorithm of OpenFOAM.

The Reynolds number and temperature of the air flow are fixed to 5460 and 35 °C at the inlet, depending on the operational conditions of the Blockheating^© data center. The Reynolds number is defined at the inlet as:

$$Re=\frac{\rho \left|U\right|D_h}{\mu }$$

(11)

where $Re$ is the Reynolds number, $\rho$ and $\mu$ are the density and viscosity of the fluid depending on the inlet temperature, $\left|U\right|$ is the average velocity of the fluid and $D_h$ is the hydraulic diameter. The density and viscosity of the fluid are calculated as the function of the inlet temperature using empirical equations [31,32]. A constant 120 W heat flux was implemented to each cold plate for the representation of the heat generated by the CPUs. The Reynolds number of the flow and supply temperature are set to 11,000 and 40 °C, respectively, in the water region. The turbulence effects at the air and water flows are calculated using the $k-\omega$ SST turbulence closure model (Equation (4)). The $k-\omega$ SST turbulence model was validated with experimental data in previous studies [33,34]. In order to calculate heat transfer to the air and water during the numerical simulations, an object function was developed and incorporated to the numerical model using the following energy equation:

$$P=\sum _{i=1}^N{\dot{m}}_i{C}_p\left(T_i-T_{inlet}\right)$$

(12)

where $P$ is the amount of heat transferred to the region; $N$ is the number of the faces; ${\dot{m}}_i$, $C_p$ and $T_i$ are the mass flux, specific heat capacity and temperature at the ith face of the outlet boundary, respectively; and $T_{inlet}$ is the temperature at the inlet of the region. In order to accurately capture the viscous boundary layer, boundary layer resolution is added to the grid on the walls in the fluid regions. Distributions of the dimensionless wall distances y⁺ values are shown for the air and water regions in Figure 11. The maximum values of the y⁺ in the air and water regions are 9.47 and 0.26, respectively. This confirms that the present mesh resolution can accurately predict boundary layer effects near the walls.

Figure 12 shows variations of the residuals of the equations with the iteration number during steady-state solutions. The energy variables are observed from the figure to be more sensitive to the iteration number than the flow variables, since achieving energy balance between regions takes longer than achieving steady-state behavior of the flow. Numerical experiments were conducted using higher relaxation factors and less tolerance for the matrix solvers to reduce the residuals of pressure and velocity in Figure 12. While the residuals of the p_rgh, Ux, Uy and Uz reduced to 1.0 × 10⁻⁴, the computational time increased by 2.55 times to achieve a converged solution, which would result in a drastic increase in the duration of a series of numerical simulations. The difference in heat transfer to the air between numerical tests was found to be trivial. Thus, the present convergence parameters are reliable to achieve steady-state results in an acceptable time duration. The flow converges to the steady-state at about 10,000 iterations and residuals are observed to fluctuate around a mean value due to the truncation and round off errors. The maximum iteration number is selected as 20,000 for all cases to mitigate risks associated with the flow and thermal conditions of an arbitrary case created in the LHS data set.

The distributions of velocity magnitude and temperature over the air region are visualized in Figure 13. High local temperatures are observed around the rear cold plate due to a higher temperature field than that of the front cold plate and lower velocities.

Temperature distributions over the water and pipe regions are visualized in Figure 14 along with streamlines through the micro-channels. The maximum temperature observed on the copper pipe is 42.9 °C, which is approximately 3 °C higher than the temperature of the supplied water. Note that the estimated temperature may not reflect the actual CPU temperature since a dye-level thermal analysis is required to consider material properties of the CPU, which is not in the scope of the present study. As can be clearly seen in Figure 14b, the temperature over the second cold plate is much higher than the first cold plate, since the heated water emerging from the first cold plate had less heat-removing capacity along the second cold plate. On the other hand, reduced temperatures near the pipe that connects two cold plates and hot points were observed near the corners of the cold plate.

The total heat generated by two CPUs on the server is set 240 W in the present simulation.

Table 3. Calculated heat transfers from the CPUs to the air and water.

Region	Inlet Temperature (K)	Outlet Temperature (K)	Temperature Jump (K)	Heat (W)
Air	308.15	308.5	0.35	3.54
Water	313.15	315.03	1.88	234.04

shows the heat transfer rates calculated using the developed function objects. Outlet temperatures of the air and water are observed to be higher: approximately 0.35 K and 1.88 K than the inlet temperatures, respectively. This finding proves that the present cold plate design could successfully reduce air temperature exhausted from the servers even for a high IT load. Another important observation is that most of the heat generated by the CPUs is transferred to the water, which satisfies the maximum heat recovery purposes.

3.3. A Compact Model for the Liquid-Cooled Server

The energy generated by the CPUs can be applied to the server as heat sources for the open-box modeling of air-cooled servers. This approach can be applied to liquid-cooled servers if the heat transfer to the air is known, depending on the operational conditions. To this end, as shown in Figure 15, a dataset was created using the LHS method for the generation of 100 samples distributed over the entire ranges of the input variables, such as Reynolds numbers and temperatures of the air and water flows at the server inlet, as well as power consumptions of the front and rear CPUs. The total power consumption of the CPUs is assumed to be transferred to the air and liquid according to the following energy balance:

$$P_{total}=P_{front}+P_{rear}=P_{air}+P_{water}$$

(13)

where $P_{front}$ and $P_{rear}$ refer to the power consumption of the rear and front CPUs, and $P_{air}$ and $P_{water}$ represent the energy transferred to the air and water, respectively.

A series of numerical simulations were performed using the validated CHT model based on the generated dataset, while the rate of heat transfer to the air and water are calculated from successive CFD simulations, which took approximately 15 days using parallel computing on the Blockheating© data center. The following equation is derived based on the correlation of the CFD results, covering an extensive data range with R² = 0.926:

$$P_{air}=c_1\times T_w-c_2\times T_a+c_3\times P_{front}+c_4\times P_{rear}-c_5\times {Re}_w+c_6\times {Re}_a$$

(14)

where $T_w$ and $T_a$ are the temperatures of the water and air as °C at the inlet of the server; ${Re}_a$ and ${Re}_w$ are the Reynolds numbers of the air and water flows, respectively; $P_{front}$ and $P_{rear}$ are the power consumptions of the front and rear CPUs as W, respectively; and empirical coefficients appearing in the equation are $c_1=0.512776$ (W/°C), $c_2=0.593935$ (W/°C), $c_3=0.00691981$, $c_4=0.00927457$, $c_5=0.000040379$ (W) and $c_6=0.000764473$ (W). The power consumption of each CPU may change depending on the workload traffic in a data center and the architecture of the CPU, such as hyper threading. The present solver can calculate the average temperature and mass flux of the air at a face set created at the inlet of the server. Finally, the rate of heat transfer to the air can be estimated using Equation (14) and the estimated power consumption is applied to the liquid-cooled server as a heat source, which enables modeling of a liquid-cooled server as an air-cooled server during numerical simulations. In order to test the accuracy and reliability of the proposed equation, five numerical simulations are performed for different $T_w$ values ranging from from 42 °C to 54 °C, while the remaining parameters are set to $T_a=36.35$ °C, $P_{front}=70$ (W), $P_{rear}=70$ (W) and ${Re}_a$ = 16,811 as constants for the simulations. A comparison of the estimated results from Equation (14) with the simulated results in Figure 16 confirms the reliability of the proposed empirical equation for the present cold plate design.

3.4. Thermal Analysis of a Hybrid Cooled Rack

Thermal analysis of a rack located at an operating data center [35] is performed using the present approach to demonstrate applicability of the proposed compact model in a real case. Figure 17 shows the geometry of the rack, including dimensions of the servers and boundaries such as the inlet, outlet and walls. Air enters the domain with a fixed temperature of 32 °C and a mass flow rate of 1.5 kg/s at the inlet, which is consistent with previous studies [34,35]. A no-slip condition is used for the velocity and wall functions, k and ω, on the walls. The exhaust air heated by the CPUs leaves the domain at the outlet. The power consumptions provided in

Table 4. Power consumptions of the servers.

Server	Power Consumption (W)
Server 1	3720
Server 2	3720
Server 3	3840
Server 4	3720

were assigned to the cell sources, defined in the servers as heat sources in the numerical model. Server 2 was replaced with a liquid-cooled Leopard V1 model of the OCP server to see performance of the proposed approach in a hybrid-cooled rack consisting of air- and liquid-cooled servers.

The power consumptions in Table 4 were implemented with the air-cooled servers as heat sources using the standard OpenFOAM library. However, heat transfer from the CPUs to the air was implemented with the liquid-cooled server using the swak4foam library, which is publicly available as an external library [36] for OpenFOAM. As can be seen in the code snippet in Figure 18, the energy transfer from two CPUs to the air can be calculated using Equation (14) during a transient simulation depending on the Reynolds number and temperature at a face zone created at the inlet of the server.

The time variable in Figure 18 controls switching from air to liquid cooling during a transient simulation. The Server 2 switches from air to liquid cooling at t = 100 s to mimic the opening of the valve at the inlet of the cold plate and the numerical simulation is run during 200 s. This simulation is designed to show efficiency of the present cold plate design in comparison to air cooling. Figure 19 depicts the time variation of the exhaust temperature of Server 2. The exhaust temperature stabilizes at $t\approx 50 \mathrm{s}$ during air cooling and remains constant until t = 100 s due to mass and energy conservation in the numerical model without any stability issues. Then, liquid cooling is activated and the exhaust temperature drops from 41 °C to 34 °C due to fact that most of the energy produced by the CPUs is transferred to the water, although water enters the cold plate at a high temperature which is critical for waste heat recovery.

Figure 20 shows the distribution of the exhaust temperature at the rear of the rack. As can be seen in the figure, the temperature at the rear of the rack reduces due to the effective cooling of the present cold plate design. The present approach can capture the reduction in the air temperature due to the heat transfer from the air to the water.

4. Conclusions

Air- and liquid-cooled servers can be deployed in a hybrid-cooled data center for the enhancement of cooling efficiency and waste heat recovery. The energy transfer from the CPU to the air can be applied to the server as a heat source in the open-box modeling of air-cooled servers. However, the thermal simulation of liquid-cooled servers, considering heat transfer between the air, water and water pipes, as well as the effect of the thermal paste between the CPU and cold plate, is challenging in the thermal management of a hybrid-cooled data center. To address this challenge, a CHT thermal model is developed and validated with experimental measurements from the literature considering the thermal paste. OpenFOAM files of the open source CHT model are publicly shared to increase dissemination of this study.

The validated CHT model is used for the thermal analysis of an in-house developed cold plate design with a serpentine micro-channel for the effective cooling and waste heat valorization, as well as to reduce the power consumption and environmental footprint of data centers. The numerical simulation results revealed that approximately 97% of the heat generated by the CPUs could be transferred to the water with the effect of the serpentine micro-channel design of the present cold plate. An extensive data set was generated using the LHS method based on the operational conditions of a hybrid-cooled data center and thermal simulations were performed using the validated CFD model. Based on the CFD results, an empirical equation is suggested for the prediction of the heat transfer from the CPU to the air with a high correlation. A thermal simulation of a rack consisting of air- and liquid-cooled servers was performed using the proposed equation. In conclusion, the proposed approach can be reliably used in thermal simulations of liquid-cooled servers along with air-cooled servers for simple and fast thermal simulations of hybrid-cooled data centers.