kvm 透传显卡至win10虚拟机的方法_服务器知识

环境

 已安装nvidia   显卡  驱动
  操作系统:CentOS Linux release 7.9.2009 (Core)
  内核版本:Linux 5.4.135-1.el7.elrepo.x86_64
  显卡   型号:rtx 6000
  
  kvm版本:
  virsh version
  Compiled against library:   libvirt   4.5.0
  Using library:   libvirt   4.5.0
  Using   API  : QEMU 4.5.0
  Running hypervisor: QEMU 1.5.3

解绑GPU 设备

1.配置GRUB_CMDLINE_LINUX(实测，有效)，查询要解绑的设备，记录设备id号

lspci  -nn|grep   NVIDIA |grep VGA
或者 lspci  -nn|grep   NVIDIA

kvm 透传显卡至win10虚拟机的方法

2.修改 /etc/default/grub文件的GRUB_CMDLINE_LINUX

vi /etc/default/grub 
以下是内容，新增加 GRUB_CMDLINE_LINUX行: intel_iommu=on pci-stub.ids=10de:1e30" 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed "s, release .*$,,g" /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="intel_iommu=on crashkernel=auto spectre_v2=retpoline rhgb quiet rdblacklist=nouveau pci-stub.ids=10de:1e30"
GRUB_DISABLE_RECOVERY="true"

3.更新grub2 conf，执行如下命令

grub2-mkconfig -o /boot/grub2/grub.cfg

4.重启系统

reboot

验证并查看nvidia 显卡

1.执行nvidia-smi ，看不到显卡，证明显卡已经被解绑

# nvidia-smi 
NVIDIA-SMI has failed because it couldn"t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

2.验证内核支持iommu

cat /proc/cmdline | grep iommu 
结果输出: intel_iommu=on

kvm 透传显卡至win10虚拟机的方法

3.验证VT-D是否启用

dmesg |grep -e DMAR -e IOMMU

kvm 透传显卡至win10虚拟机的方法

安装win10虚拟机

qemu-img create -f qcow2  /home/kvm-machines/shaoyan.qcow2  200G
virt-install --name shaoyan --ram 4096 --vcpus=8 --os-type=windows --accelerate --cdrom=/home/kvm_images/win10.iso   --disk path=/home/kvm-machines/shaoyan.qcow2,format=qcow2,bus=ide --bridge=br0  --autostart --vnc --vncport=5910 --vnclisten=0.0.0.0

宿主机安装软件包

yum -y groups install "GNOME Desktop"  virt-manager virt-viewer dejavu-lgc-sans-fonts

显卡加载vfio驱动
1.显卡加载vfio驱动

modprobe vfio
modprobe vfio-pci

2.从服务器卸载1张显卡的4个设备

virsh nodedev-detach pci_0000_b1_00_0
virsh nodedev-detach pci_0000_b1_00_1
virsh nodedev-detach pci_0000_b1_00_2
virsh nodedev-detach pci_0000_b1_00_3

3.验证显卡设备是否加载vfio-pci驱动,加载成功

# lspci -vv -s b1:00.0 | grep driver
        Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.1 | grep driver
        Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.2 | grep driver
        Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.3 | grep driver
        Kernel driver in use: vfio-pci

win10虚拟机添加NVDIA显卡
1.通过远控卡登录服务器透传显卡，执行startx 调出可视化页面

startx

2.添加显卡硬件

kvm 透传显卡至win10虚拟机的方法

3.查看显卡

kvm 透传显卡至win10虚拟机的方法

显卡驱动安装
1.要安装显卡驱动，需要修改虚拟机win10的配置文件否则显卡驱动安装过程中自检会报错ERROR CODE 43，驱动不允许安装在虚拟机上。

#编辑VM XML
virsh edit shaoyan

#修改第一行
<domain type="kvm">
#为
<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">

#找到<features>节点，添加内容
<features>
... 
  <kvm><hidden state="on"/></kvm>
...
</features>

#找到</device>节点，在device后添加内容
<qemu:commandline><qemu:arg value="-cpu"/><qemu:arg value="host,hv_time,kvm=off,hv_vendor_id=null"/>
</qemu:commandline>

#重启VM
virsh destroy shaoyan
virsh start shaoyan

2.附录完整xml 文件如下，如果是拷贝至其他宿主机配置文件的话，只需要保留xml 文件中的第一组hostdev即可，否则会报错

# cat /etc/libvirt/qemu/shaoyan.xml

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit shaoyan
or other application using the libvirt API.
-->

<domain type="kvm">
  <name>shaoyan</name>
  <uuid>e06c6c3b-0a57-43e8-9d6f-02fd1e2aad83</uuid>
  <title>gpu</title>
  <description>gpu</description>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <vcpu placement="static">16</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-rhel7.0.0">hvm</type>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
  </features>
  <cpu mode="host-passthrough" check="none"/>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/data/kvm/shaoyan.qcow2"/>
      <target dev="hda" bus="ide"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <controller type="ide" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <interface type="bridge">
      <mac address="52:54:00:48:b2:5a"/>
      <source bridge="br0"/>
      <model type="e1000"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="vnc" port="5910" autoport="no" listen="0.0.0.0">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <video>
      <model type="vga" vram="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0xb1" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0xb1" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0xb1" slot="0x00" function="0x2"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0xb1" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </memballoon>
  </devices>
</domain>

3.登录win10 虚拟机配置网络

4.安装驱动，选择具体型号进行下载并安装驱动

https://www.nvidia.cn/Download/index.aspx?lang=cn

5.查看显卡设备

kvm 透传显卡至win10虚拟机的方法