The end of the Dual-Boot Era!
Oh yes! No more dual-booting baby! I still cannot believe it! I was really looking forward to it and it’s finally here also for me! I really love working under Linux: everything feels smooth, stable, secure and the power of customization is in your hands! Qualities that you cannot find in Windows. Not up until now at least. Windows are however a necessity: if you are a gamer or your bound to windows-only programs you know what I mean!
In this guide I will show you how you can setup QEMU-KVM to achieve GPU passthrough and ultimately get rid of dual booting. This guide will cover the case of using identical GPUs and is a bit different from the most guides out there which go via the pci-stub module. We will directly use the vfio-pci !
Setup and Installation
Step 1: Enable VT-d/AMD-v
The first thing we need to do is enable VT-d/AMD-v and ensure functionality.
We need to edit /etc/default/grub :
sudo nano /etc/default/grub
Look for the line that says GRUB_CMDLINE_DEFAULT= and append the following:
- Intel:
GRUB_CMDLINE_LINUX_DEFAULT=" ... intel_iommu=on pcie_acs_override=downstream"
- AMD:
GRUB_CMDLINE_LINUX_DEFAULT=" ... amd_iommu=on iommu=pt pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1"
After editing rebuild the grub.cfg file:
sudo grub-mkconfig -o /boot/grub/grub.cfg
Find your current active entry under the /boot/loader/entries/ and edit it, e.g.:
sudo nano /boot/loader/entries/arch.conf
Look for the line that starts with options and append the following:
- Intel:
options root=PARTUUID=be2539d1-41b9-4171-901e-71c7a363be4b rw intel_iommu=on pcie_acs_override=downstream
- AMD:
options root=PARTUUID=be2539d1-41b9-4171-901e-71c7a363be4b rw amd_iommu=on iommu=pt pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1"
-Reboot!
Now, verify that VT-d/AMD-v is enabled:
dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] ACPI: DMAR 0x000000007F2B4D78 0000DC (v01 INTEL SKL 00000001 INTL 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 0.034995] DMAR: Host address width 39
[ 0.034996] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.035001] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ec
which is an example of Intel VT-d and check also if there are any IOMMU groups:
sudo find /sys/kernel/iommu_groups/ -type l
Example output:
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
...
/sys/kernel/iommu_groups/12/devices/0000:01:00.0
/sys/kernel/iommu_groups/12/devices/0000:01:00.1
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/13/devices/0000:02:00.1
If you do’t have an iommu_group folder then VT-d is probably not enabled or something went wrong.
Step 2: Identify the PCI-Bus of the GPU(s)
Now we need to identify the bus-id of the GPU we want to pass-through:
lspci -nnk
...
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102]
Kernel driver in use: nvidia
Kernel modules: snd_hda_intel
...
In my case I have two identical graphics cards – two GTX 750 Ti. I am interesting in passing through the GPU at PCI-bus 02:00.0 . Note that the Audio device at 02:00.1 is the audio bus of the GPU, which we will also pass it through the VM-client.
Step 3: Fixing IOMMU Grouping (if needed)
This step might be needed for some people and for some it does not. To find out, check the IOMMU group of the GPU you want to pass through.
# Either run
sudo find /sys/kernel/iommu_groups/ -type l
# or
ls -lha /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/
If you can see the PCI-bus id of the other GPU card, e.g.:
drwxr-xr-x 2 root root 0 Feb 28 19:41 .
drwxr-xr-x 3 root root 0 Feb 28 19:40 ..
lrwxrwxrwx 1 root root 0 Feb 28 19:41 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root 0 Feb 28 19:41 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
lrwxrwxrwx 1 root root 0 Feb 28 19:41 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Feb 28 19:41 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root 0 Feb 28 19:41 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 Feb 28 19:41 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
To fix this, we are going to download a new kernel that has already been patched to deal with issues like that. We need to download linux-vfio using yaourt:
yaourt -S linux-vfio
Important! During installation you will be asked if you want to only install linux-vfio . Choose NO so it will also install the docs and headers for the kernel!!!
sudo grub-mkconfig -o /boot/grub/grub.cfg
For systemd-boot users we need to create a new entry for our new kernel. Basically copy the entry of your current kernel:
# in my case my current kernel entry is "arch"
sudo cp /boot/loader/entries/arch.conf /boot/loader/entries/arch-vfio.conf
and edit it to point to the new kernel:
sudo nano /boot/loader/entries/arch-vfio.conf
linux /vmlinuz-linux-vfio <-- here initrd /intel-ucode.img initrd
/initramfs-linux-vfio.img <-- and here
options root=PARTUUID=be2539d1-41b9-4171-901e-71c7a363be4b rw intel_iommu=on
and make also sure that at the /boot/loader/loader.conf the timeout value is more than 3 seconds you will have time to choose the desired new entry.
Important note: I hope you are using dmks for your graphics card drivers. If not, you have to obviously re-install the drivers for the new kernel.
Step 4: Configure vfio-pci
If you are using GRUB, all you need to do to override the default kernel module and load vfio-pci
is to configure /etc/default/grub
:
GRUB_CMDLINE_LINUX_DEFAULT=" ... vfio-pci.ids=10de:13c2,10de:0fbb,1b73:1100"
where the ids are from:
sudo lspci -nnk
08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 [GeForce GTX 970] [1462:3160]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
08:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 High Definition Audio Controller [1462:3160]
Kernel driver in use: nvidia
Kernel modules: snd_hda_intel
0a:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
Subsystem: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100]
Kernel driver in use: xhci_hcd
where in this example we pass the NVIDIA card and the USB pci hub.
Now this is the part that differs from most guides out there. And most importantly, with the following method we are able to proceed with our setup even if we have two identical graphics cards. The idea is basically to override the NVDIA driver module with the vfio-pci one. For that, we need to assign as driver the vfio-pci module before the NVIDIA driver module has the chance to take over. Luckily, we can achieve that with the mkinitcpio with two steps:
- Create a script that overrides the graphics card driver
- Load the vfio-pci before any boot hooks
vfio-pci-override script
The following script will use the driver_override feature introduced in kernel v3.16 and make vfio-pci the exclusive driver for that device:
sudo nano /sbin/vfio-pci-override.sh
#!/bin/sh
# 02:00.0 is the pci-bus of my graphics card and 02:00.1 its sound bus
DEVS="0000:02:00.0 0000:02:00.1"
for DEV in $DEVS; do
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done
modprobe -i vfio-pci
And make it executable:
sudo chmod 755 /sbin/vfio-pci-override.sh
vfio-pci module
To load vfio-pci module on boot create a new entry in the :
sudo -s
echo "install vfio-pci /sbin/vfio-pci-override.sh" > /etc/modprobe.d/vfio_pci.conf
The above translates to: To install the vfio-pci module, run the script /sbin/vfio-pci-override.sh , which sets up our driver overrides and then loads the module, ignoring the install option (-i) to prevent a loop.
mkinitcpio.conf
The last thing is to update the ramdisk environment. Edit the mkinitcpio.conf :
sudo nano /etc/mkinitcpio.conf
and make sure the following have been appended in that file:
MODULES="vfio_iommu_type1 vfio_pci"
BINARIES="/sbin/vfio-pci-override.sh"
FILES="/etc/modprobe.d/vfio_pci.conf"
After changes are made to the mkinitcpio configuration file, the image must be regenerated
sudo mkinitcpio -p linux
# or for those that use a patched kernel
sudo mkinitcpio -p linux-vfio
Reboot!
Check with lspci -nnk that your graphics card uses vfio-pci module as its driver:
lspci -nnk
...
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI]
Device [1462:3102]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
...
You’re all set! Now we can move on to installing QEMU.
Step 6: Install QEMU
Install qemu first. Install also rpmextract as we are going to need it
sudo pacman -S qemu rpmextract
Download OVMF
Next we need the UEFI bios called OVMF from here. Look for the edk2.git-ovmf-x64-###.noarch.rpm file.
# Download it
wget https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20160226.b1536.gd2ba6f4.noarch.rpm
# Extract it
rpmextract.sh edk2.git-ovmf-x64-0-20160226.b1536.gd2ba6f4.noarch.rpm
# Copy files to /usr/share/
sudo cp -R usr/share/* /usr/share/
Test Setup (Almost done!)
Now, you are almost done! Check your system with the following script to see if everything is working:
#!/bin/bash
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
qemu-system-x86_64 \
-enable-kvm \
-m 2048 \
-cpu host,kvm=off \
-vga none \
-device vfio-pci,host=02:00.0,multifunction=on \
-device vfio-pci,host=02:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd
Running this as root, you will able to see on the monitor of your second graphics card the UEFI shell! 🙂 Congrats!!! You did it! Now you can install you favorite OS. In the next section I will show you how to install Windows 10!