Comments on: How Liquid Cooling Servers Works with Gigabyte and CoolIT

By: Tom

Tom — Sun, 03 Jul 2022 15:04:45 +0000

Very insightful and informative article.

I just finished re-building my homelab based on a water-cooled Cray CS400-LC rack and the power of water cooling is just amazing.

Certainly on this scale a very interesting topic from an engineering standpoint.

By: Patrick Kennedy

Patrick Kennedy — Mon, 16 May 2022 22:44:50 +0000

In reply to Mark Hemmeyer. Mark, mentioned this a bit, but there is another version (H262-ZL1) that also cools the memory and the NVIDIA-Mellanox adapter, we just kept it easy and did the -ZL0 version that is CPU only. I did not want to have to bring another $5K of gear through customs.

By: Mark Hemmeyer

Mark Hemmeyer — Mon, 16 May 2022 17:40:27 +0000

How about the memory cooling, storage, and network? You still need the fans for that in this system.
With DDR5 memory coming up and higher power there will be a much smaller savings from a cpu only liquid cooled system.
Any chance to publish the system temperatures too?
Thanks!

By: Eric Olson

Eric Olson — Thu, 14 Apr 2022 17:24:26 +0000

With high-density dual-socket 2U4N modules, for example, there seems to be one socket which gets the cooled first and the exhaust of that cooling used to cool the second.

If I understood the plumbing correctly the water which cools the first socket in the setup tested here is then used to cool the second socket. RAM is air cooled but again the air is passed over the first group of DIMMs before cooling the second.

Under load I’d expect there to be a difference in the temperature of the first compared to the second. In my opinion measuring this difference (after correcting for any variations in the chips and sensors) would be an meaningful way to compare the efficiency of the present water cooling to other cooling designs in similar form factors.

By: Nils

Nils — Tue, 12 Apr 2022 14:05:00 +0000

Would have been interesting to know how warm the liquid is going in and out. Could it be used to heat an adjacent office or residential building before going to the chiller?

By: Patrick Kennedy

Patrick Kennedy — Mon, 11 Apr 2022 23:05:39 +0000

In reply to Bill. Bill, it depends, but I think I just did back of the envelope on if you did not even re-chill the water and just passed water through adding heat. It was still less than half the cost of using fans. Even if you use fans, you then have to remove the heat from the data center and in most places that means using a facility chiller anyway.