Re: [PATCH v9 08/18] PCI/P2PDMA: Route Relaxed Ordering Completions directly

[Leon Romanovsky <leon@kernel.org>]

On Tue, Oct 06, 2026 at 05:21:09PM -0500, Bjorn Helgaas wrote:
> On Thu, Oct 01, 2026 at 02:55:16PM +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@nvidia.com>
> > 
> > ACS P2P Completion Redirect leaves Completions carrying the Relaxed
> > Ordering attribute alone. PCIe r7.0 sec 6.12.1.1 redirects only those "that
> > do not have the Relaxed Ordering Attribute bit set", and sec 7.7.12.5
> > describes the enable bit as "applicable only to Completions whose Relaxed
> > Ordering Attribute is clear". P2PDMA reports one answer for every kind of
> > TLP, so a client whose provider returns such Completions is sent through
> > the host bridge for a redirect that never happens to it.
> 
> I guess pci_acs_p2pdma_completion() returned PCI_ACS_P2PDMA_REDIRECT
> even for RO Completions?  But that didn't change the hardware
> behavior -- maybe the caller *thought* Completions were routed through
> the host bridge, but they actually weren't.
> 
> I kind of lost the plot here.  What does the caller do with this
> information?  I don't think a device is *required* to set RO even when
> it is enabled, and it may set RO on some transactions but not others.

The mlx5 patch, which is currently on hold, uses the
PCI_P2PDMA_TLP_RELAXED_CPL TLP flag to dynamically enable or
disable ATS.

https://lore.kernel.org/all/20260920-fix-p2p-acs-v4-0-v7-19-ca0828ab697c@nvidia.com/

Thanks

> 
> > Add enum pci_p2pdma_tlp_flags and let a caller state that property.
> > 
> > Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
> > Tested-by: Tushar Dave <tdave@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> > ---
> >  drivers/pci/p2pdma.c | 6 +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> > index 43e225cc5735..1fadef6d0609 100644
> > --- a/drivers/pci/p2pdma.c
> > +++ b/drivers/pci/p2pdma.c
> > @@ -544,11 +544,15 @@ pci_acs_p2pdma_request(u16 ctrl, unsigned int tlp_flags)
> >  /*
> >   * Decide how a peer-to-peer Completion at an ACS-capable ingress port routes.
> >   * PCIe r7.0 sec 6.12.1.1: no ACS control other than P2P Completion Redirect
> > - * affects a Completion.
> > + * affects a Completion, and that one leaves Completions carrying the Relaxed
> > + * Ordering attribute alone.
> >   */
> >  static enum pci_acs_p2pdma_state
> >  pci_acs_p2pdma_completion(u16 ctrl, unsigned int tlp_flags)
> >  {
> > +	if (tlp_flags & PCI_P2PDMA_TLP_RELAXED_CPL)
> > +		return PCI_ACS_P2PDMA_DIRECT;
> > +
> >  	return ctrl & PCI_ACS_CR ? PCI_ACS_P2PDMA_REDIRECT :
> >  				   PCI_ACS_P2PDMA_DIRECT;
> >  }
> > 
> > -- 
> > 2.55.0
> >