Hello alltogether,

I'm using an intel E810 card on a proxmox/qemu installation. The virtual
functions (VF) from the E810 card are imported as PCI devices into
virtual machines running inside kvm/qemu and are supported by the iavf
driver. 

The iavf driver should use as many interrupt queues as configured during
startup of the hypervisor via
/sys/class/pci_bus/0000:3b/device/0000:3b:01.1/sriov_vf_msix_count (this
is set via udev for all the VF). The hypervisor system is running Debian trixie
pve kernel 6.14.11-3 using the "ice" driver and all virtual machines are
using Debian trixie standard kernel 6.12.48. The virtual machines are
running with at least 4 cores.

This works as expected as long as the virtual machines are not using
iPXE boot (from iPXE.org). iPXE boot obviously uses a function to force
the PF (physical function, i.e. the driver for the real hardware) to
configure only one interrupt queue for the used VF. This can be seen in
the kernel log of the hypervisor while iPXE boot is running.  This
message is logged by the hypervisor if the virtual machine using VF 10
is doing PXE boot.

ice 0000:3b:00.0: VF 10 granted request of 1 queues.

After this this VF number 10 only provides one interrupt queue for a
linux virtual machine using debian trixie (Linux kernel 6.12.48) and
this cannot be solved with tools like ethtool inside the VM. Even if the
VF is later assigned to a different virtual without iPXE boot this "only
one interrupt queue" stays active.

I tracked this down to a patch from 2020:
"[net-next,02/16] iavf: Enable support for up to 16 queues" that removed
the possibility to increase the queue count via ethtool and just uses
the queue count provided by the PF (but in my case that is configured to
"1" by PXE boot that is running before the Linux kernel is started)

I did a manual change (see attached patch, kind of reverts the above
patch) the introduces the possibility to increase the interrupt queue
count manually via ethtool inside the VM.  This worked, the MSI-X
interrupts were established and I got 4 queues as I ordered via "ethtool
-L enp1s0 combined 4" and the MSI-X queues could be seen inside the VM.
This was logged by the hypervisor:

ice 0000:3b:00.0: VF 1 granted request of 4 queues.

This means I've a working example that the functionality of
increasing/setting the queue count is currently missing and how to fix
it. Personally, I think to make it acceptable patch that does everything
"right" automatically some additional work is necessary.  It would be
great if someone could look into this. 

I think it would be a good
solution if the iavf driver could read the MSI-X interrupt count from
the PCI info like this would be done with "real" hardware and configure
this via the VIRTCHNL_OP_REQUEST_QUEUES message to the PF.

Thank you very much.

Regards,

   Carsten Groß

-- 
Carsten Gross            | http://www.siski.de/~carsten/