KVM PCI passthrough problems after kernel v4.2

KVM PCI passthrough problems after kernel v4.2

[Juhani Rautiainen]

Hi!

I've been using KVM PCI passthrough with a dual-port NIC for over a
year. I just upgraded to v4.3 from v4.1.4 and PCI passthrough stopped
working. I checked with v4.1.9 and it still works there. Version 4.2
seems to point after which my system doesn't work. How can I help to
narrow down the cause of the problem? I checked the intel-iommu.c and
there seem to have been lots of changes between 4.1.9 and 4.2 in
drivers/iommu/intel-iommu.c. My motherboard is Asus P8C WS which has
Intel C216 chipset. CPU is Intel I5-3470. From v4.2 onwards
passthrough fails with:

pci-stub 0000:02:00.0: kvm assign device failed ret -22.

This is grep from dmesg. First is non-working v4.2 then working v4.1.9:

Nov 12 23:14:06 base kernel: Linux version 4.2.0 (root@base) (gcc
version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov 12
23:04:04 EET 2015
Nov 12 23:14:06 base kernel: ACPI: DMAR 0x00000000DA99F4A8 0000B8 (v01
INTEL  SNB      00000001 INTL 00000001)
Nov 12 23:14:06 base kernel: DMAR: Host address width 36
Nov 12 23:14:06 base kernel: DMAR: DRHD base: 0x000000fed90000 flags: 0x0
Nov 12 23:14:06 base kernel: DMAR: dmar0: reg_base_addr fed90000 ver
1:0 cap c0000020e60262 ecap f0101a
Nov 12 23:14:06 base kernel: DMAR: DRHD base: 0x000000fed91000 flags: 0x1
Nov 12 23:14:06 base kernel: DMAR: dmar1: reg_base_addr fed91000 ver
1:0 cap c9008020660262 ecap f010da
Nov 12 23:14:06 base kernel: DMAR: RMRR base: 0x000000da911000 end:
0x000000da922fff
Nov 12 23:14:06 base kernel: DMAR: RMRR base: 0x000000dd000000 end:
0x000000df1fffff
Nov 12 23:14:06 base kernel: DMAR-IR: IOAPIC id 2 under DRHD base
0xfed91000 IOMMU 1
Nov 12 23:14:06 base kernel: DMAR-IR: HPET id 0 under DRHD base 0xfed91000
Nov 12 23:14:06 base kernel: DMAR-IR: Enabled IRQ remapping in xapic mode
Nov 12 23:14:06 base kernel: DMAR: No ATSR found
Nov 12 23:14:06 base kernel: DMAR: dmar0: Using Queued invalidation
Nov 12 23:14:06 base kernel: DMAR: dmar1: Using Queued invalidation
Nov 12 23:14:06 base kernel: DMAR: Setting RMRR:
Nov 12 23:14:06 base kernel: DMAR: Setting identity map for device
0000:00:02.0 [0xdd000000 - 0xdf1fffff]
Nov 12 23:14:06 base kernel: DMAR: Setting identity map for device
0000:00:14.0 [0xda911000 - 0xda922fff]
Nov 12 23:14:06 base kernel: DMAR: Setting identity map for device
0000:00:1a.0 [0xda911000 - 0xda922fff]
Nov 12 23:14:06 base kernel: DMAR: Setting identity map for device
0000:00:1d.0 [0xda911000 - 0xda922fff]
Nov 12 23:14:06 base kernel: DMAR: Prepare 0-16MiB unity mapping for LPC
Nov 12 23:14:06 base kernel: DMAR: Setting identity map for device
0000:00:1f.0 [0x0 - 0xffffff]
Nov 12 23:14:06 base kernel: DMAR: Intel(R) Virtualization Technology
for Directed I/O
Nov 12 23:14:06 base kernel: [drm] DMAR active, disabling use of stolen memory
Nov 12 23:15:31 base kernel: pci-stub 0000:02:00.0: kvm assign device
failed ret -22

Nov 12 23:17:38 base kernel: Linux version 4.1.9 (root@base) (gcc
version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov 12
22:10:21 EET 2015
Nov 12 23:17:38 base kernel: ACPI: DMAR 0x00000000DA99F4A8 0000B8 (v01
INTEL  SNB      00000001 INTL 00000001)
Nov 12 23:17:38 base kernel: dmar: Host address width 36
Nov 12 23:17:38 base kernel: dmar: DRHD base: 0x000000fed90000 flags: 0x0
Nov 12 23:17:38 base kernel: dmar: IOMMU 0: reg_base_addr fed90000 ver
1:0 cap c0000020e60262 ecap f0101a
Nov 12 23:17:38 base kernel: dmar: DRHD base: 0x000000fed91000 flags: 0x1
Nov 12 23:17:38 base kernel: dmar: IOMMU 1: reg_base_addr fed91000 ver
1:0 cap c9008020660262 ecap f010da
Nov 12 23:17:38 base kernel: dmar: RMRR base: 0x000000da911000 end:
0x000000da922fff
Nov 12 23:17:38 base kernel: dmar: RMRR base: 0x000000dd000000 end:
0x000000df1fffff
Nov 12 23:17:38 base kernel: DMAR: No ATSR found
Nov 12 23:17:38 base kernel: IOMMU: dmar0 using Queued invalidation
Nov 12 23:17:38 base kernel: IOMMU: dmar1 using Queued invalidation
Nov 12 23:17:38 base kernel: [drm] DMAR active, disabling use of stolen memory
Nov 12 23:18:55 base kernel: pci-stub 0000:02:00.0: kvm assign device
Nov 12 23:18:55 base kernel: pci-stub 0000:02:00.1: kvm assign device

Best Regards,
Juhani
-- 
Juhani Rautiainen                                   jrauti@iki.fi

Re: KVM PCI passthrough problems after kernel v4.2

[Alex Williamson]

On Thu, 2015-11-12 at 23:36 +0200, Juhani Rautiainen wrote:
> Hi!
> 
> I've been using KVM PCI passthrough with a dual-port NIC for over a
> year. I just upgraded to v4.3 from v4.1.4 and PCI passthrough stopped
> working. I checked with v4.1.9 and it still works there. Version 4.2
> seems to point after which my system doesn't work. How can I help to
> narrow down the cause of the problem? I checked the intel-iommu.c and
> there seem to have been lots of changes between 4.1.9 and 4.2 in
> drivers/iommu/intel-iommu.c. My motherboard is Asus P8C WS which has
> Intel C216 chipset. CPU is Intel I5-3470. From v4.2 onwards
> passthrough fails with:
> 
> pci-stub 0000:02:00.0: kvm assign device failed ret -22.

This happens when kvm calls iommu_attach_device(), which was changed in
v4.2 to fail with -EINVAL (-22) when the device is part of a
non-singleton iommu group.  Legacy kvm device assignment is not iommu
group aware and has been deprecated for some time.  Have you considered
using vfio for device assignment instead?

For reference, here's the comment that made this change:

commit 426a273834eae65abcfc7132a21a85b3151e0bce
Author: Joerg Roedel <jroedel@suse.de>
Date:   Thu May 28 18:41:30 2015 +0200

    iommu: Limit iommu_attach/detach_device to devices with their own group
    
    This patch changes the behavior of the iommu_attach_device
    and iommu_detach_device functions. With this change these
    functions only work on devices that have their own group.
    For all other devices the iommu_group_attach/detach
    functions must be used.
    
    Signed-off-by: Joerg Roedel <jroedel@suse.de>

Re: KVM PCI passthrough problems after kernel v4.2

[Juhani Rautiainen]

Hi!

Thanks a lot for the help. I switched to vfio and got NIC working
again. Now I just have to figure out what I have to change to get this
to persist through the host boots. To tell the truth I couldn't quite
parse from that commit message that my device would stop working:).
The error messages in this area are quite uninformative anyway. KVM
could for instance warn for using deprecated method (kernel does it in
many other places). When I did that configuration, I did it using KVM
PCI passthrough pages and the method there was this deprecated method
(and it still was when I switched to Centos7, because I lost my config
through stupid mistake and had to do it all over again). And I thought
that I've been following this area in LKML but somehow missed the
whole deprecation business:).

Thanks,
-Juhani

On Fri, Nov 13, 2015 at 12:00 AM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Thu, 2015-11-12 at 23:36 +0200, Juhani Rautiainen wrote:
>> Hi!
>>
>> I've been using KVM PCI passthrough with a dual-port NIC for over a
>> year. I just upgraded to v4.3 from v4.1.4 and PCI passthrough stopped
>> working. I checked with v4.1.9 and it still works there. Version 4.2
>> seems to point after which my system doesn't work. How can I help to
>> narrow down the cause of the problem? I checked the intel-iommu.c and
>> there seem to have been lots of changes between 4.1.9 and 4.2 in
>> drivers/iommu/intel-iommu.c. My motherboard is Asus P8C WS which has
>> Intel C216 chipset. CPU is Intel I5-3470. From v4.2 onwards
>> passthrough fails with:
>>
>> pci-stub 0000:02:00.0: kvm assign device failed ret -22.
>
> This happens when kvm calls iommu_attach_device(), which was changed in
> v4.2 to fail with -EINVAL (-22) when the device is part of a
> non-singleton iommu group.  Legacy kvm device assignment is not iommu
> group aware and has been deprecated for some time.  Have you considered
> using vfio for device assignment instead?
>
> For reference, here's the comment that made this change:
>
> commit 426a273834eae65abcfc7132a21a85b3151e0bce
> Author: Joerg Roedel <jroedel@suse.de>
> Date:   Thu May 28 18:41:30 2015 +0200
>
>     iommu: Limit iommu_attach/detach_device to devices with their own group
>
>     This patch changes the behavior of the iommu_attach_device
>     and iommu_detach_device functions. With this change these
>     functions only work on devices that have their own group.
>     For all other devices the iommu_group_attach/detach
>     functions must be used.
>
>     Signed-off-by: Joerg Roedel <jroedel@suse.de>
>
>



-- 
Juhani Rautiainen                                   jrauti@iki.fi