Proxmox PCI-Passthrough

When I originally installed Proxmox some years ago, my initial use case was to experiment with PCI Passthrough, primarily passing a GPU (Graphics Card), to experiment with how much the performance degraded. Now this is my default configuration of a desktop windows PC.

This post details those investigations and also contains a walkthrough on how to setup Proxmox PCI Passthrough on an intel architecture (Please note, AMD is different and I will not be covering it in this post).

Note: This post is largely based upon the excellent Reddit thread “The Ultimate Beginner’s Guide to GPU Passthrough (Proxmox, Windows 10)” and of course the PCI passthrough page from the official Proxmox documentation.

Disclaimer: In no way, shape, or form does this guide claim to work for all instances of Proxmox/GPU configurations. Use at your own risk. I am not responsible if you blow up your server, your home, or yourself. I stole this from the reddit post 🙂

Introduction

This guide assumes that you have a working installation of Proxmox, at the time of writing the current version was 6.2 and note that PCI Passthrough is an experimental feature at this time.

Configure GRUB

First you need to enable IOMMU by editing the kernel commandline.

nano /etc/default/grub

Modify the line GRUB_CMDLINE_LINUX_DEFAULT as follows:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

Or alternatively, this can be achieved using the following command:

sed -i.bak "s/quiet/quiet\ intel_iommu=on/g"  /etc/default/grub

Now we need to update grub

update-grub

VFIO Modules

The following VFIO modules need to be added to your system.

nano /etc/modules

Add the following lines:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Or alternatively, this can be achieved using the following command:

printf "vfio\nvfio_iommu_type1\nvfio_pci\nvfio_virqfd\n" >> /etc/modules

Blacklisting Drivers

We need to stop the host system from using the GPU, therefore we need to blacklist the drivers. I tend to only blacklist the GPU that I am using, in my case NVIDIA:

echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

however you could use either of the following depending on your configuration:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

Adding the GPU to VFIO

First we need to find the address of the the VGA card:

lspci | grep VGA

in my instance, this returns:

03:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2) (prog-if 00 [VGA controller])
08:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) (prog-if 00 [VGA controller])

We are interested in the 03:00. Using this number run the following command:

lspci -n -s 03:00

This will return the GPUs Vendor IDs. There are probably two lines returned, one for the GPU and one for the Audio Bus. In my case I get the following:

03:00.0 0300: 10de:1380 (rev a2)
03:00.1 0403: 10de:0fbc (rev a1)

Now we can add these vendor IDs to the VFIO

echo "options vfio-pci ids=10de:1380,10de:0fbc disable_vga=1" > /etc/modprobe.d/vfio.conf

Remember to replace the Vendor IDs (in bold) with your own.

Finally we run the following commands:

update-initramfs -u
reset

Reboot

You now have Proxmox configured for PCI-Passthrough. I normally reboot at this point, but I am not sure it is totally necessary.

Configuring your Virtual Machine

Open your VMs hardware window and make sure that the bios is set to OVMF (UEFI). When you change this, you will be prompted to add an EFI disk.

2. Make sure the machine type is set to q35 (this is not the default).

3. Change the display to none.

4. Finally add a PCI device for the graphics card. Select All Functions, Primary GPU and PCI Express from the options.

5. Passthrough the USB Keyboard and Mouse. Add USB Vendor/DeviceId entries for your keyboard and mouse.

Note: When you boot the virtual machine, the keyboard and mouse will be assigned to the virtual machine and will no longer be available to the server.

You can confirm that all this has been correctly configured by checking the machine configuration file:

nano /etc/pve/qemu-server/100.conf

In my instance, the following lines are added:

bios: ovmf
efidisk0: zfs-hot:vm-100-disk-1,size=1M
machine: q35
vga: none
hostpci0: 03:00,pcie=1,x-vga=1
usb0: host=046d:c52b,usb3=1
usb1: host=1b1c:0c10,usb3=1

The (incomplete) Script

In most of my posts I normally include a script at the the end that completes all of the functions listed above. In this instance I have not fully automated it (yet), but I am including the following as work in progress. Do not run this.


# enable pci passthrough

# configure grub
intel_iommu=on
sed -i.bak "s/quiet/quiet\ intel_iommu=on/g"  /etc/default/grub
update-grub

# vfio modules
printf "vfio\nvfio_iommu_type1\nvfio_pci\nvfio_virqfd\n" >> /etc/modules

# blacklisting drivers
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

########
# adding gpu to vfio - this section is not autmated.
########
lspci | grep VGA
lspci -n -s 65:00
echo "options vfio-pci ids=10de:1cb2,10de:0fb9 disable_vga=1" > /etc/modprobe.d/vfio.conf
update-initramfs -u
reset

# add entries to the machine config file
cd /etc/pve/qemu-server
nano 100.conf
######
bios: ovmf
efidisk0: zfs-hot:vm-100-disk-2,size=128K
hostpci0: 65:00,x-vga=1,pcie=1
machine: q35
vga: none

Job done.

References:
1. Pci passthrough (Official Proxmox documentation) – https://pve.proxmox.com/wiki/Pci_passthrough
2. The Ultimate Beginner’s Guide to GPU Passthrough (Proxmox, Windows 10) – https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/