Comments on: Hybrid 2U2N GPU Workstation-Server Platform Supermicro SYS-210GP-DNR Hands-on

By: Anthony Miller

Anthony Miller — Thu, 27 Jan 2022 12:57:05 +0000

Hey Mr. Will,

Thanks for sharing such useful information with us, But as I can see your post is a little bit old. But still, you’ve successfully managed this article to be a great piece of work.

– Best Regards.

By: Gabriele

Gabriele — Sat, 04 Dec 2021 11:58:19 +0000

@Harvey: In almost any modern hypervisor one can leverage on a technology called Single Root IO Virtualization (SR-IOV) which explicitly does what you are suggesting. Basically it consists on the PCIe device being able to split itself (virtually) in multiple functions (the part of the PCIe address after the dot) you can individually pass through to multiple VMs at the same time.

Then the physical device is capable of scheduling the instructions coming from each PCIe virtual function slot at hardware level, therefore with real low performances penalties. Ofc this can be hammered hard with all the VMs requiring full bandwidth at the same time, but for most applications the overhead is negligible.

This should give you what you need and considering that some of such cards allow for 32-way splitting or more, the consolidation you can get from such approach can be huge.
Have a look on google for “SR-IOV ethernet card with FC” and you should get some great examples. Consider that ESXi sometimes gets a bit picky on hardware support, so maybe you can start the search by “SR-IOV ethernet card with FC ESXi” to further restrict the results.

By: Harvey

Harvey — Wed, 01 Dec 2021 17:57:29 +0000

@fuzzyfuzzyfungus: We use the machines as ESXi hosts, and an average machine will have 2-3 dozen VMs on it (most usually suspended). It’s not possible (at least to my knowledge) to have a single card be shared between multiple machines, so each powered VM needs it’s own dedicated PCI card in passthrough mode. We do make extensive use of switches in our network, as our main datacenter has about 40 tape libraries in it.

And in case you were wondering, the feature of ESXi that lets you assign “virtual WWN Fibre Channel” to a VM, does not work with tape libraries and is storage array only which isn’t particularly useful for us as arrays are a very small part of our work.

By: fuzzyfuzzyfungus

fuzzyfuzzyfungus — Tue, 30 Nov 2021 20:15:58 +0000

@Harvey, is there something about your application(either raw bandwidth or particular devices being very, very, fussy about sharing) that keeps you from just using the fact that fiber channel treats switched fabric as a pretty standard topology?

I would suspect that this thing could support the multiple HBAs, given that GPUs are probably the PCIe devices with the most oddities, along with being fairly demanding; I’m just a little surprised that you would need to stuff the system that full of HBAs rather than using just enough ports to connect to the switch, ideally with some redundancy.

As for the use of these as remote workstations; has the industry coalesced around one option as the de-facto standard for remote keyboard/video/mouse in cases where artifacts and nagging lag just aren’t acceptable(like graphic artists using tens of thousands of dollars worth of GPUs) at rangers greater than ‘just use high quality displayport cables’ covers; or is that still up in the air with multiple contenders?

By: Harvey

Harvey — Tue, 30 Nov 2021 16:52:46 +0000

Probably a silly question, and 100% not what this unit was intended for but:
Is there any reason why instead of loading it with GPU’s I couldn’t load it with (for example) Fibre Channel Cards?
I work in an industry where a machine will have half a dozen fiber channel cards/ports in it to connect to multiple tape libraries and storage arrays, and we have many of these machines. As long as small “extension” cables were passed through from the GPU’s in the middle I don’t see why we couldn’t use one of these to get multiple machines in a smaller form factor.