Re: vPASID capability for VF

[Alex Williamson <alex.williamson@redhat.com>]

On Tue, 9 May 2023 23:41:44 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, May 10, 2023 7:13 AM
> > 
> > On Tue, May 09, 2023 at 10:57:04PM +0000, Tian, Kevin wrote:  
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Wednesday, May 10, 2023 6:44 AM
> > > >
> > > > On Tue, May 09, 2023 at 08:34:53AM +0000, Tian, Kevin wrote:  
> > > > > According to PCIe spec (7.8.9 PASID Extended Capability Structure):
> > > > >
> > > > >   The PASID configuration of the single non-VF Function representing
> > > > >   the device is also used by all VFs in the device. A PF is permitted
> > > > >   to implement the PASID capability, but VFs must not implement it.
> > > > >
> > > > > To enable PASID on VF then one open is where to locate the PASID
> > > > > capability in VF's vconfig space. vfio-pci doesn't know which offset
> > > > > may contain VF specific config registers. Finding such offset must
> > > > > come from a device specific knowledge.  
> > > >
> > > > Why? Can't vfio probe the cap tree and just find a gap to insert a new
> > > > cap? We already mangle the cap list, I'm not sure I see what
> > > > the problem is?
> > > >  
> > >
> > > PCI config space includes not only caps, but also device specific
> > > defined fields. e.g. Intel IGD defines offset 0xfc as a pointer to
> > > OpRegion. I'm sure Alex can give many other examples.  
> > 
> > Do we even expose those over VIFO? I thought in general we blocked of  
> 
> Yes. I did a quick check:
> 
> /*
>  * Default unassigned regions to raw read-write access.  Some devices
>  * require this to function as they hide registers between the gaps in
>  * config space (be2net).  Like MMIO and I/O port registers, we have
>  * to trust the hardware isolation.
>  */
> static struct perm_bits unassigned_perms = {
> 	.readfn = vfio_raw_config_read,
> 	.writefn = vfio_raw_config_write
> };
> 
> vfio_config_do_rw()
> {
> 	...
> 	if (cap_id == PCI_CAP_ID_INVALID) {
> 		perm = &unassigned_perms;
> 		cap_start = *ppos;
> 	} ...
> }
> 
> vfio_config_init()
> {
> 	...
> 	memset(map, PCI_CAP_ID_BASIC, PCI_STD_HEADER_SIZEOF);
> 	memset(map + PCI_STD_HEADER_SIZEOF, PCI_CAP_ID_INVALID,
>               		 pdev->cfg_size - PCI_STD_HEADER_SIZEOF);
> 	...
> }
> 
> > various parts of the config space. I keep seeing patches to unblock
> > parts of config space?
> > 
> > I'd do the reverse and say devices that want to pass parts of their
> > config space should have a special hook to do it and otherwise we
> > should sanitize and block?  
> 
> This then may break backward compatibility. We don't know how
> many devices have such hidden registers so if anyone misses a hook
> immediately it cannot be assigned after we start blocking as default.
> 
> > 
> > eg we already have a hook to pass the opregion
> >   
> > > So it's easy to find the gap between caps, but not easy to know
> > > whether that gap is actually free to use.  
> > 
> > Because, let's face it, this is a horrible thing to do, and the
> > opregion stuff is just ugly as s sin.
> >   
> 
> It's ugly, but that is the reality. :/

Have a peak at the config space of an NVIDIA GPU and tell me which of
those non-zero fields between capabilities are used as well.  Glass
houses...  ;-)

IIRC we originally needed to enable this for a Broadcom NIC that
stuffed device specific registers in un-architected config space. The
capabilities we're {un}hiding are architected things that we know are
unsupported or unsafe, the gaps, just like device specific
capabilities, we're obliged to expose for functionality.  Thanks,

Alex