This post mainly demonstrates my personal virtualization setup. The configuration of my computer (along with some peripherals) is listed below.
- AMD Ryzen 3900XT 12 cores 24 threads
- ASUS TUF GAMING B550M-PLUS (WI-FI)
- 64GB DDR4
- Samsung 860 Pro 512G NVMe
- Intel 760p 2TB NVMe
- NVIDIA RTX 3090
- NVIDIA RTX 2070 Super
- Dell TypeC-Ethernet Dongle
I tried to use unRAID to fulfil my purpose, but I could not do a PCIe passthrough for the Intel 760p drive because there were some glitches with the SM2271 controller. Also, the specific motherboard I used is kind of cheap, so there are only 2 m.2 sockets (x4.CPU + x4.Chipset) and 3 PCIe sockets. The layout is shown in the figure below.
As the manual says that PCIEX1 and PCEIX16_2 share bandwidth and the maximum bandwidth for PCEIX16_2 is PCIe 3.0@x4. PCIEX16_1 can reach up to PCIe 4.0@x16. So I'm going to plug in my RTX 3090 in PCIEX16_1 and plug in 2070 Super in PCIEX16_2. The PCIEX1 will not be used.
Then, of course, we need to switch on IOMMU in UEFI. This is part of what I get when I check the IOMMU grouping. There are literally a bunch of devices in IOMMU Group 15 (I guess that came from the B550M Chipset). But as you can see, the RTX 2070 Super GPU is one of the devices in IOMMU group 15. I cannot do PCIe passthrough for this GPU. (Next time I perhaps will buy a better motherboard and a better CPU with more PCIe lanes).
IOMMU Group 15: 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee] 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb] 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9] 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] [10de:1e84] (rev a1) 04:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1) 04:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1) 04:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1) 05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808] 06:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a) 07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)
Alright, I cannot use unRAID or just passthrough my RTX 2070 Super. But it's not that bad. Because I only use Windows for games at my free time. I can just unbind RTX 3090 from nvidia kernel driver and bind it to the vfio-pci one if I want to use Windows, and vice-versa if I need to run Tensorflow or PyTorch. And most importantly, with unRAID, I need to shut down my Ubuntu, add/remove the RTX 3090 to/in it and then power it on. Now I can use my Ubuntu without interrupting.
Continue reading My Virtualization Setup