Re: [PATCH 0/2] rust: pci: expose is_virtfn() and reject VFs in nova-core

[Zhi Wang <zhiw@nvidia.com>]

On 1.10.2025 13.32, Danilo Krummrich wrote:
> On Wed Oct 1, 2025 at 3:22 AM CEST, John Hubbard wrote:
>> On 9/30/25 5:29 PM, Alistair Popple wrote:
>>> On 2025-10-01 at 08:07 +1000, John Hubbard <jhubbard@nvidia.com> wrote...
>>>> Post-Kangrejos, the approach for NovaCore + VFIO has changed a bit: the
>>>> idea now is that VFIO drivers, for NVIDIA GPUs that are supported by
>>>> NovaCore, should bind directly to the GPU's VFs. (An earlier idea was to
>>>> let NovaCore bind to the VFs, and then have NovaCore call into the upper
>>>> (VFIO) module via Aux Bus, but this turns out to be awkward and is no
>>>> longer in favor.) So, in order to support that:
>>>>
>>>> Nova-core must only bind to Physical Functions (PFs) and regular PCI
>>>> devices, not to Virtual Functions (VFs) created through SR-IOV.
>>>>
>>>> Add a method to check if a PCI device is a Virtual Function (VF). This
>>>> allows Rust drivers to determine whether a device is a VF created
>>>> through SR-IOV. This is required in order to implement VFIO, because
>>>> drivers such as NovaCore must only bind to Physical Functions (PFs) or
>>>> regular PCI devices. The VFs must be left unclaimed, so that a VFIO
>>>> kernel module can claim them.
>>>
>>> Curiously based on a quick glance I didn't see any other drivers doing this
>>> which makes me wonder why we're different here. But it seems likely their
>>> virtual functions are supported by the same driver rather than requiring a
>>> different VF specific driver (or I glanced too quickly!).
>>
>> I haven't checked into that, but it sounds reasonable.
> 
> There are multiple cases:
> 
> Some devices have different PCI device IDs for their physical and virtual
> functions and different drivers handling then. One example for that is Intel
> IXGBE.
> 
> But there are also some drivers, which do a similar check and just stop probing
> if they detect a virtual function.
> 

Right, it really depends on the hardware design and the intended use
cases, and is therefore device-specific. In networking, for example,
there are scenarios where VFs are used directly on bare metal - such as
with DPDK to bypass the kernel network stack for better performance. In
such cases, PF and VF drivers can end up being quite different and VF 
driver can attach on the baremetal (via pdev->is_virtfn in probe()).

Similarly, in the GPU domain, there are comparable scenarios where VFs
are exposed on bare metal for use cases, like containers.

(I remember Xe driver can be attached to a VF in bare metal for such a
use case.)

For NVIDIA GPUs, VFs are only associated with VMs. So this change
makes sense within this scope.

Z.

> So, this patch series does not do anything uncommon.
> 
>>> I'm guessing the proposal is to fail the probe() function in nova-core for
>>> the VFs - I'm not sure but does the driver core continue to try probing other
>>> drivers if one fails probe()? It seems like this would be something best
>>> filtered on in the device id table, although I understand that's not possible
>>> today.
> 
> Yes, the driver core keeps going until it finds a driver that succeeds probing
> or no driver is left to probe. (This behavior is also the reason for the name
> probe() in the first place.)
> 
> However, nowadays we ideally know whether a driver fits a device before probe()
> is called, but there are still exceptions; with PCI virtual functions we've just
> hit one of those.
> 
> Theoretically, we could also indicate whether a driver handles virtual functions
> through a boolean in struct pci_driver, which would be a bit more elegant.
> 
> If you want I can also pick this up with my SR-IOV RFC which will probably touch
> the driver structure as well; I plan to send something in a few days.