Summer is still an issue, but fun solutions are possible. With the right geology, I think it’s possible to heat an underground volume in the summer and recapture (some of) that heat in the winter. In many, many climates, annual heating costs are far higher than cooling costs, at least if people aren’t stupid with skylights. [0]

[0] As a back-of-the-envelope heuristic, heating or cooling load due to conduction and air exchange is proportional to the difference between indoor and outdoor temperature. Outdoor temperatures of -10F to 30F are not unusual in the winter and are 40-80F away from an indoor temp of 70F. But outdoor temperatures in these climates rarely exceed 95F and are mostly lower in the summer, so that’s 15-25F of cooling. And heat pumps are more efficient at smaller temperature differences.

Radiative heating is an entirely different story.

Aaahh, there’s the catch. “Save resources on cooling by building your data centre somewhere cold, and pollute the surrounding environment by dumping your waste heat wholesale into that!”

Good job Nvidia, I almost thought we had something good there.

I get that they're using liquid coolant at higher than usual temperatures, but why couldn't they do that before? Most of the comparison in the article is for air cooled datacenters but what about other liquid cooled ones?

Surely in all the previous datacenters that have been designed there has been someone doing the math and determining what temperature things need to run at, how much energy it will use, how much heat it all will produce, etc.

edit: just saw this:

>Previous liquid-cooled servers were hybrid: GPUs and CPUs got cold plates, but the rest of the system stayed air-cooled, with finned heat sinks designed to shed heat into moving air. In a fully liquid-cooled server, the cooling for these components needed to be completely redesigned to use liquid.

What's a favorable climate, apart from, obviously, Greenland? The piece is a little light on details on the correlation between outside temperatures and efficiency & cost. It'd be nice to see even a broad-strokes discussion of that.

https://www.nasa.gov/centers-and-facilities/ames/doing-more-...

https://www.nas.nasa.gov/assets/nas/pdf/ModularSupercomputin...

The usual way to cool servers is with air and heatsinks attached to the hot hardware, similar to how your desktop computer or laptop works. As the hardware gets denser and more powerful, you need bigger and bigger heatsinks and cooler air blown over them. At some point you can't make the heatsinks bigger because of space constraints, and you can't blow the air faster (because of noise and efficiency), so you need cooler air. That's when you start running chillers that evaporate water to cool your intake air. This is the huge water consumption that we'd like to avoid.

The next step is, obviously, liquid cooling. Again, this is similar to your fancy gaming desktop. You can dump a lot of heat to a liquid medium through a small heat exchanger inside, where you're space constrained, and you can run the liquid through a gigantic heat exchanger outside, despite the temp delta between your coolant and outside air being pretty small.

This article is about a system that's FULLY liquid cooled — CPUs, GPUs, memory, networking, the whole thing. That's the actual cool part (pun unintended). On top of that, their solution is optimized to be able to run the coolant quite warm — this obviously limits the heat flux at the hardware side, but it allows you to run the outside heat exchangers "dry", i.e. without wasting any water for its latent heat.

Our data center is mostly air-cooled, with 100F hot aisles and chilled water going to heat exchangers in every third rack position, although some of our newer GPU racks use chilled water directly. For most of the year that means we don't need chillers - the warm water return is sufficiently hotter than outdoor temp that heat "flows downhill" with a bit of pumping.

But we use an evaporative cooler to carry that heat away outside, because it's more efficient. If you look up the heat of vaporization of water, that means we evaporate about 10,000 gallons of water per day per megawatt of power dissipated. We're on canal off a large river, and we don't dissipate all that many megawatts, so I believe the water use isn't significant.

Using air as the working fluid to draw heat from your machines has a limitation - humans have to breathe that air whenever they work on the equipment, so the temperature is limited. Once the exterior temperature nears your hot aisle air temp, you either need active A/C to create a heat source hotter than ambient, or an evaporative cooler to lower the "effective" ambient temp to the dew point.

Liquid cooling lets you run your working fluid a lot hotter without killing folks like me who go into the data center, although honestly 45C sounds like an incremental improvement over the 100F places like ours are already running. (although to be fair, the warm water return from the heat exchangers is no doubt somewhat less than 100F) It also lets you run your "cold" side a lot hotter - if you "chill" your water down to 100F (38C) on a hot day, it's still cold enough to carry away a lot of heat at 45C.

(I'm skipping over the fact that there are multiple heat exchanger loops involved - e.g. any system with an evaporative cooler needs a heat exchanger to keep leaves and bird shit outside the building where it belongs)

    Historically, cooling alone has accounted for up to 40% of a data center’s electricity consumption

Or are they for some unfathomable reason using evaporative cooling in data centers?

They’ve made this claim numerous times in the article and I really don’t understand it. The building has tons of water being recirculated through it. That water came from somewhere in the surrounding natural world. How is that 0 water consumption?

That doesn't mean a datacenter doesn't have a very high _initial_ need for water, or to replace _some_ amount of leakage, replacement etc (agreed it will be way lower than say a swamp cooler). For example, they could be using millions of gallons as a sort of "ballast" to keep the water temperature very stable in the short run.

This whole projects depends on the whole stack of components to be able to tolerate 45c from memory, drives, the lighting of the building, the humans ...

It strikes me that building everything around room temperature or slightly chilled air is a strange choice. This is already 290K-300K or so, and now this is suggesting that things run fine at 320K or 330K?

I've wondered why we couldn't just design everything to operate around 200°C and just use free-cooling by pumping ambient air through. Why don't data centres look more like chicken barns? Do things melt? Are there more errors of some other type at high temperatures?

Does that mean the whole server board is running in liquid? If true, how do they do maintenance? Replacing parts must be extremely difficult.

Look at the author of the blog.

https://blogs.nvidia.com/blog/author/joshparker/

Is this NVIDIA lawyer trying to influence the public perception of datacenter for his company?

On the other hand: the heat has to go somewhere. So… where? Datacenters already create a warm microclimate in their vicinity, is that getting even worse?

I'm ignorant to the specifics of how this might work (and whether it'd even be feasible at a cost/logistics level), but it feels far less invasive and far more sustainable as an armchair thought. Curious if anyone has worked on something like this and what sort of gotchas you've found.

[1] https://www.heata.co/

[2] https://thermify.cloud/how-it-works/

Leslie Groves, Oppenheimer

Nvidia has so much money and they can’t afford to pay a human for a day of their time to write a blog post?

I do think ground based centers will have better economics when they can be built though, and this addresses noise and water complaints which are the big 2 regional complaints.

It seems like lots of bottlenecks are getting solved quickly, except for maybe memory.