Workstation

TODO

  • Mobility and Performance
  • Mobility as lifestyle choice (e.g. reduce the number of screens)
  • Simple “hack” type systems (e.g. tablet+tmux)
  • Complex systems (e.g. Multi-GPU)
  • Peripherals (e.g. Docks, DACs, Keyboards)

Multi-GPU and Device Passthrough #

There is a kernel driver subsystem called VFIO (Virtual Function I/O) which allows for the “passthrough” of a device to a virtual machine. The devices are passed through in groups based on the Input-Output Memory Management Unit which connects the DMA-capable I/O bus to the main system memory.

The VFIO driver is an IOMMU/device agnostic framework for exposing direct device access to userspace, in a secure, IOMMU protected environment.

Devices are grouped into topologies, sometimes alone or sometimes with other dependent systems:

For instance, an individual device may be part of a larger multi- function enclosure. While the IOMMU may be able to distinguish between devices within the enclosure, the enclosure may not require transactions between devices to reach the IOMMU.

Some special purpose devices being shared, or being split into logical sub-components via Single Root Input-Output Virtualization.

Given this background, the objective is to have a Linux on the bare metal and a virtual machine with a GPU and USB based hardware passed through to it. The most strait forward way of achieving this is to:

  • Have a motherboard with good IOMMU topologies support
    • significantly easier since mid-2020
    • AM4 x570 chipset have been reported successful
      • allows for choice of “boot GPU” which makes the process easier
  • Have two GPU(s)

For a specific example we’re fortunate enough to use the following:

  • Gigabyte X570 Aorus Master
    • 2.5GbE (used for host) and 1.0GbE (used for guest)
  • AMD Reference W6600 (used for host)
  • AMD Reference 6800XT (used for guest)
    • USB-C onboard

I’ve prepared the BIOS as such, some of the IOMMU settings are a bit buried (in two places):

  • Update BIOS (f35b at time of this posting)
  • Set some BIOS settings:
    • Settings → IO Ports → Initial Display Output → PCIe 1 Slot (W6600)
    • Tweaker → Advanced CPU Settings → SVM Mode → Enabled
    • Settings → Miscellaneous → IOMMU → Enabled
    • Settings → NBIO Common Options → IOMMU → Enabled

Our goal is to passthrough the 6800 XT with its USB-C and the motherboards 1.0GbE adapter to the virtual machine. With Fedora Silverblue we’ll begin:

We run this to enumerate the IOMMU topology on our system:

#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

From this we find the necessary PCI Identifiers for the 1.0GbE and 6800XT:

  • 8086:1539 (1.0 GbE)
  • 1002:73bf (6800XT)
  • 1002:ab28 (6800XT HDMI Audio)
  • 1002:73a6 (6800XT USB)
  • 1002:73a4 (6800XT USB)

We prepare our system to generate it’s own initramfs with the modules we will need:

[agd@chonk ~]$ vim /etc/dracut.conf.d/vfio.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "

Need to enable initramfs, where “In Silverblue, the initramfs is prebuilt and included in the system image”, enabling will apparently create the local initramfs and read the dracut.conf.d we configured above:

[agd@chonk ~]$ rpm-ostree initramfs --enable

Install the necessary hypervisor packages:

[agd@chonk ~]$ rpm-ostree install virt-manager qemu-kvm

Adjust the kernel boot parameters:

[agd@chonk ~]$ rpm-ostree kargs --append="amd_iommu=pt rd.driver.pre=vfio-pci vfio-pci.ids=1002:73bf,1002:ab28,8086:1539 pci-stub.ids=1002:73a6,1002:73a4"
  • We use pci-stub insted of vfio-pci for the USB as it essentially “acts earlier” and prevents the kernel from loading something else and occupying those devices.
  • Often times you will see write ups with video=efifb:off mentioned, but that isn’t required since this chipset/mainboard allows us to choose a specific slot for initial GPU.

Now you can create a VM and passthrough those devices and have an extremely powerful hardware accelerated machine within the machine.

Windows has lmitations on the number of CPU sockets, so you’ll want to specify cores instead of sockets:

<cpu mode="host-passthrough" check="none" migratable="on">
  <topology sockets="1" dies="1" cores="12" threads="1"/>
</cpu>

Video card drivers by AMD incorporate very basic virtual machine detection targeting Hyper-V extensions. Should this detection mechanism trigger, the drivers will refuse to run, resulting in a black screen. If this is the case, it is required to modify the reported Hyper-V vendor ID:

<features>
  ...
  <hyperv>
    ...
    <vendor_id state='on' value='randomid'/>
    ...
  </hyperv>
  ...
</features>