Re: [PATCH v9 02/18] PCI/P2PDMA: Derive routing from directional ACS controls

[Bjorn Helgaas <helgaas@kernel.org>]

On Fri, Oct 23, 2026 at 12:59:14PM -0300, Jason Gunthorpe wrote:
> On Fri, Oct 09, 2026 at 08:43:01PM +0300, Leon Romanovsky wrote:
> > On Wed, Oct 07, 2026 at 03:40:45PM -0500, Bjorn Helgaas wrote:
> > > On Wed, Oct 07, 2026 at 09:32:27AM +0300, Leon Romanovsky wrote:
> > > > On Tue, Oct 06, 2026 at 03:49:47PM -0500, Bjorn Helgaas wrote:
> > > > > On Thu, Oct 01, 2026 at 02:55:10PM +0300, Leon Romanovsky wrote:
> > > > > > From: Leon Romanovsky <leonro@nvidia.com>
> > > > > > 
> > > > > > pci_bridge_has_acs_redir() treats Request and Completion Redirect as
> > > > > > interchangeable. On asymmetric fabrics, a control for only the reverse TLP
> > > > > > direction can unnecessarily force P2PDMA through the host bridge.
> > > > > 
> > > > > Does "the reverse TLP direction" refer to Completions?
> > > > 
> > > > In general, the P2P code treats TLPs flowing from device A to device B the
> > > > same as TLPs flowing from device B to device A.
> > > > 
> > > > However, in the context of this commit message, yes: completions flow in the
> > > > opposite direction from the device's perspective.
> > > 
> > > And I guess asymmetric fabrics must mean fabrics where Request
> > > Redirect and Completion Redirect are not set the same way?
> > 
> > Yes, in some of our systems, deviceA to deviceB is going through
> > different route than deviceB to deviceA.
> 
> It is a little more subtle, I think practical systems try very hard to
> avoid that since many host bridges do not like seeing only half of an
> exchange.
> 
> What we end up with is Relaxed Ordering and Strict Ordering take
> different flows in the fabric.
> 
> > > But I asked about ordering because it sounds like this patch expects
> > > to encounter asymmetric fabrics where Request Redirect and Completion
> > > Redirect may not be set the same way, and the spec implies that
> > > asymmetry may result in ordering violations.
> > 
> > My guess is that the hardware and system architects have ensured this
> > doesn't happen. At least, I haven't received any complaints about this
> > series during internal testing.
> 
> Yes, the ACS combination is selected to avoid ordering issues

Does a user have any way to know whether the configuration is safe?

I'm wondering whether users might use "pci=config_acs=" to configure
ACS so things appear to work as they prefer but may not provide the
ordering guarantees drivers depend on.

If that's a possibility, do we have any notes in dmesg that we are
depending on fabric behavior that the OS can't discover and verify?

Bjorn