Liquid Cooling Helps AI Data Centers Perform Smarter

In a recent byline for FacilitiesNet, Bill Kosik, P.E., CEM, LEED AP, Mission Critical Sector Leader at HED, examines how liquid cooling is changing the design and operation of high-density data centers. His perspective draws on decades of work in data center HVAC optimization, sustainability, and mission critical infrastructure.

FacilitiesNet published a version of Bill’s article, outlining how hybrid air- and liquid-cooling systems, cooling distribution units, commissioning protocols, and evolving industry standards are shaping the next generation of AI data center design. The full, unabridged version is included below for readers seeking a deeper technical view of the challenges and opportunities ahead.

Read the published piece at FacilitiesNet, or view the full-length article below:

Data Centers: Enhancing Sustainability, Efficiency with Liquid Cooling
https://www.facilitiesnet.com/datacenters/article/Data-Centers-Enhancing-Sustainability-Efficiency-with-Liquid-Cooling–21000

Opportunities for Enhanced Sustainability and Efficiencies Using Liquid Cooling Systems

Setting the Stage – Can Liquid Cooling Systems Used in AI Data Centers Enable Energy Efficiency and Sustainability?

Since the early 2020s, growing societal use of artificial intelligence created the need for a new class of data centers that are designed to handle the immense power and cooling requirements of the computers. As the design and operation for these data centers matures, long-held reliability and maintainability strategies have been expanded to include electricity, water availability, and greenhouse gas (GHG) emissions.

Knowing that the electrical capacity of these data centers can be three or four orders of magnitude greater than modern colocation and hyperscale facilities, designing data centers that are highly efficient will result in lower GHG emissions. These efficiencies will also include dematerialization of the structure and infrastructure systems knowing that liquid cooling systems use less space.

Introduction

Maintaining strict environmental control in data centers continues as one of the primary drivers to ensure optimal performance and reliability. But this challenge is not new: as far back as the 1950s early mainframe computer operators often had difficulties keeping temperatures within acceptable limits, preventing failure and possible shutdown. Since many of these early mainframe computers were used for defense purposes, shutting down one of these computers could compromise strategic military analysis and preparation.

These large mainframe computers evolved during the 1960s and early 1970s from purpose-built, classified systems used by the US military. New mainframes were designed and built by private-sector companies and sold to large corporations to improve efficiency of the back-end systems such as banking, finance, HR, and other applications. Companies that invested in this technology realized cost reductions and improved operational capability due to enhanced data analysis and record retrieval.

As the demand for these business mainframes expanded, the manufacturers continued to optimize the hardware and operating systems, resulting in ongoing improvements. Key elements in achieving performance enhancements came from advances in processing, storage, and networking hardware and operating systems. The advent of solid-state electronics allowed for a much smaller internal footprint and a reduced form factor. But with the miniaturization of hardware components and concentrated electrical loads, the internal heat density of the computer continued to grow, until it became a roadblock to the development of more powerful and versatile computers.

Early Use of Liquid Cooling

At this point in the maturity of these business machines, cooling using water and refrigerants was being used mainly in niche, purpose-built computing systems. As one of the leading business computing manufacturers, IBM analyzed the impacts of using a combination of air and liquid to cool the computers. What IBM concluded is that using liquid cooling not only enables a more effective and reliable cooling method, it also allows for higher electrical densities, resulting in greater computer performance and reliability. Legacy installations of IBM mainframes also included a water tank for cooling. However, more than 50 years later with modern computing power tens of orders of magnitude greater than early systems, the design challenge remains – how to cool high-density computers effectively, economically, and with high reliability.

Air Cooling and Increasing Cabinet Density

As data centers grew and IT equipment loads became more concentrated, supplemental cooling products were introduced to serve localized high-density areas. These unitary systems were typically installed in data centers, next to or above the computer cabinets. These systems are still a useful solution, but as data center design became more sophisticated, including the use of analytical tools such as computational fluid dynamics, it became possible to design facilities that relied only on air without those supplemental units. The threshold for cabinet power density at that time was approximately 10 kW per cabinet using air cooling. Over time that threshold increased, reaching 20 to 25 kW per cabinet and in some cases up to 30 kW.

Additional Strategies for Air-Cooling High-Density Applications

In addition to the solutions for high-density cooling that primarily relied on augmented cooling infrastructure, there were also several strategies based on integrated computer cabinets. The design of these cabinets completely isolated the cooling air supplied from an underfloor plenum from the data center. The air is drawn upwards across the hardware components, allowing for more precise cooling of the computer. The hot air off the computer is discharged into a hot air return plenum above the ceiling. This method is extremely effective since there is no chance of unintentional air circulation. But even these specialized, air-cooled computer cabinets have an upper limit in air cooling.

To cool very high-density loads, the computer’s on-board fans must circulate high volumes of air. This results in significant increases in power demand for the computers due to increased fan power. Also, the heat gain from the computer fans increases the overall cooling load. Another approach using air to cool high-density computing is to reduce the temperature of the supply air from the cooling infrastructure systems. While this will work in certain instances, reducing the supply air temperature will result in increased power consumption of the compressors in the central cooling plant. Finally, using air to cool the computers will typically result in non-uniform air distribution across hardware components, resulting in overheating and possible damage to the components.

Advent of AI and Ultra-High Compute Density Data Centers

What is an AI data center? The term “AI data center” loosely describes data centers that mainly run training and inference for AI compute systems. And while AI data centers share common characteristics with other types of data centers, the power and cooling requirements are far greater. Additionally, computers have significantly different operational characteristics. When in training mode, the AI cluster will run at full capacity for long periods of time. In inference mode, the cluster will vary in power use. The most critical part in designing AI data centers is operational protocol for “instant on” computing, where the computer power will rapidly hit maximum computing capability in a matter of seconds.

Another distinct feature of AI data centers is the ability to work directly with the manufacturer of the graphics processing units (GPUs) and the tensor processing units (TPUs). The manufacturer of these components has a vested interest in making AI data centers successful and since the predominant cooling load is due to these components, the liquid cooling design concepts are part of the manufacturer’s guidance. This level of transparency, especially having access to technology roadmaps, is precedent-setting and not typical for other types of data centers.

As discussed, with the rise of AI data centers, there is a greater focus on cooling system design and performance capability. The ultra-high computer densities seen in AI applications require a fundamental shift in data center design. And while the thermodynamic fundamentals like mass flow and heat transfer properties apply to all types of cooling systems, AI facility-level cooling loads and the electrical density in the data hall exceed those of enterprise, hyperscale, and cryptocurrency mining data centers. For example, an enterprise data center may have computer cabinet density up to 10 kW; hyperscale and cryptocurrency operations potentially up to 30 kW. These data centers are typically air-cooled and rely on pre-manufactured hot-aisle containment (HAC) systems. HACs minimize recirculation of the hot discharge air from the computers with the cold air that cools the computers. It is important to note that, in general, AI data centers will contain air-cooled computers. Currently, most installations require air-cooling for storage and networking equipment. The cooling system design is based on a mixture of air- and liquid-cooled design strategies. The commingling of air and liquid cooling presents challenges in not only the design of the data halls, but that of the central plant and cooling distribution.

Evolution of Cooling Equipment for Ultra-High-Density Applications

Although legacy supercomputer cooling designs cannot be transferred directly to current AI facilities, supercomputing systems still provide an important precedent for liquid cooling in AI applications. One differentiator is that supercomputing sites are usually purpose-built and will operate as long as the systems are able to meet the scientific and research requirements. Supercomputing sites can be located at research universities or national labs and are used for specific government agency purposes.

As an example of how liquid cooling has evolved, in the early 2010s this author was part of a design team for a new supercomputing facility. The system’s computing nodes used a combination of liquid (80%) and air (20%) cooling. Each node was approximately 150 kW. The cooling system design consisted of primary-secondary-tertiary chilled water distribution. The secondary to tertiary chilled water distribution was a closed system using heat exchangers for heat transfer from the computer nodes to the chilled water system. In general, this type of design was common for liquid-cooled computing systems.

Data centers for the private sector, especially AI data centers, will be modular, built with expansion capabilities to meet current and future end-user requirements. This is a critical aspect of private-sector data centers, minimizing initial capital expense with the end user or tenant covering the cost of the build-out. The power and cooling equipment used for these facilities must follow the modularization approach, providing a “just in time” deployment to minimize first and operational costs. (The modularization approach also works well for procuring electricity from a grid operator or utility that requires the data center developer to submit a ramp-up plan that indicates how much power is required in a five-to-ten-year timeframe).

It wasn’t until the arrival of AI and other high-performance computing systems that data center cooling manufacturers started developing equipment specific to liquid cooling. Examples of this are cooling distribution units (CDUs). While the basic operating premise and functionality of the CDUs is not new, the design and packaging of internal components is used to address the specific design requirements of liquid-cooled data centers. CDUs provide heat transfer and pumping capabilities between the facility’s chilled water-cooling system (facility water system – FWS) and the technical liquid that is used to cool the computers, typically via a cold plate heat transfer (technical coolant system – TCS).

After CDUs were introduced to the data center cooling market, they continued to be refined with better control, functionality, and redundancy. As new generations of these enhanced CDUs are released, manufacturers have increased their cooling capacity to accommodate the rapid growth of AI computer cooling requirements. CDUs come in several different capacities ranging from 100 kW to two megawatts (MW). The CDU sizes generally match the power requirements of the AI compute systems, further enabling a modular design.

An example of AI data center power and cooling system modularity:

1. In a data hall, one row of AI clusters will require six megawatts of power.
2. Roughly 90 to 95% will be liquid cooled, the rest using air cooling.
3. One data hall can accommodate approximately 36 MW of AI clusters.
4. One data center can accommodate six data halls, resulting in an overall power and cooling requirement of over 200 MW.

Without these specialized systems, the data center cooling system would be “stick-built”, using equipment from different manufacturers assembled on site, possibly jeopardizing interoperability and equipment delivery schedules.

Data Center Industry Guidance on Liquid Cooling

Over the last two decades, larger and more powerful data centers were required to meet the needs of cloud computing and other internet-based applications. Starting with the Bell-Telcordia design requirements written for telco applications, several industry organizations have published best practice standards and guidelines on how to maintain uptime and high reliability in the design and operations of a data center. While these standards are excellent at discussing reliability or tier levels, best practices in cooling system design were predominantly covered at a high level, without detailed technical information needed by the HVAC practitioner.

However, in the early 2000s ASHRAE published design guidelines and technical analysis of data center cooling systems. This series of publications continued to grow, covering more topics for cooling, and the early guidelines were updated to reflect the advancements in data center design. Many of the guidelines have a strong focus on data center sustainability and energy efficiency. These documents became a foundation for the rapidly evolving power and cooling system requirements.

ASHRAE also published a new energy standard that is specific to data centers, Standard 90.4, “Energy Efficient Data Centers”. The first publication was released in 2013, with updates released on a three-year cycle, with the most recent being released in 2025. While this standard provides guidance on methods and metrics for data center cooling systems, it also covers renewable energy and GHG emissions, providing a broader envelope for sustainability. The standard is written in code-ready language and has been adopted by the International Energy Conservation Code (IECC).

During this time, other organizations were also developing guidelines for ITE and infrastructure systems performance.

1. The Green Grid (TGG), the developer of PUE (and other xUE benchmarking processes) developed white papers and other technical documents aimed at data center energy efficiency and sustainability.
2. Organizations such as the Open Compute Project (OCP) are an important contributor for standards and best practices in data center design.
3. The US Green Building Council (USGBC) developed adaptations to the LEED rating system that provided a framework to achieve energy and sustainability.

As AI data centers started to emerge, these and other organizations continued the development of standards and guidance documents, to include the special requirements for high-density and liquid-cooled data center environments. And as of this writing, the requirements for AI data center power and cooling continue to evolve, requiring strong ongoing industry participation to ensure existing standards and guidelines are updated as AI and other types of data centers evolve to meet the demands of new technologies.

Liquid Cooling Systems: Start-Up and Commissioning

Typically, large-scale AI data centers (also known as AI factories) are purpose-designed and built for supporting dedicated AI processes. These data centers will be built on a site that can accommodate the large footprint of the facility. Other types of AI deployments are on a smaller scale and are often integrated into a multi-tenant, hyperscale data center. This arrangement includes base-building power and cooling systems. The landlord is responsible for reliable deployment of infrastructure that will meet the performance requirements of the tenant. Using air-cooled data halls makes the hand-off from landlord to tenant easier and is well defined.

In this context, it is important to understand the criticality of start-up and commissioning. TCS fluid cleanliness is judged on the requirements for maximum particulate size. Larger particulate size can reduce the heat transfer capability of cold plates, which transfer the heat from the internal computer components to the TCS loop fluid. The design of the cold plates includes microchannels, which enable highly efficient heat transfer. If particulates accumulate in the microchannels, heat transfer capability of the cold plate can be significantly degraded. Hydronic systems in data centers and other commercial buildings do not have this type of cleanliness requirement. Defining testing and commissioning for the TCS fluid (and several other AI-specific instances) is critical to ensure proper and effective system performance.

For multi-tenant data centers, the demarcation is less clear. Some data center owners will use the CDU as the point of demarcation, defining the TCS piping and distribution design as the responsibility of the tenant. And while this isn’t universally true in multi-tenant data centers, the start-up and operation of the TCS loop also provides challenges. The tubing to the liquid-cooled IT equipment must be tested to ensure proper cooling capability. As these systems are commissioned, guidance is available, but this type of documentation is still being developed into industry-wide guidelines, required to establish common benchmarks and uniform testing protocol.

Outlook

Some of the challenges in designing power and cooling systems for AI data centers are yet to be seen. This is especially true for liquid cooling. As in the past, only with time and historical knowledge will the data center industry be able to produce authoritative guidance. And the use of air-cooling in data centers will continue to be the predominant cooling strategy, most likely a mixture of conventional air-cooled cabinets and liquid-cooled systems. Also, as manufacturers expand the availability of AI-specific cooling and power equipment, the industry will have more options for designing modular and adaptable facilities. Finally, industry organizations must continue to release standards and guidelines specific to data centers. These items are a critical part of building standardization and uniformity in AI data center design, start-up, and commissioning of power and liquid cooling systems.