Re: [PATCH v6 18/18] RDMA/mlx5: Ask P2PDMA whether ATS takes a direct peer-to-peer route

["Christian König" <christian.koenig@amd.com>]

On 9/18/26 19:05, Jason Gunthorpe wrote:
> On Fri, Sep 18, 2026 at 03:42:28PM +0200, Thomas Hellström wrote:
>>
>> 1) Xe attachment check if pci_p2pdma_distance() returns OK for the
>> path. Then Xe always sets up dma-addresses using dma_map_resource(). 
> 
> Open coding pci_p2pdma_distance() in drivers is a hack. Using
> dma_map_resource() like this was never "allowed".
> 
> We've fixed things so these hacks are not needed, the drivers need to
> move over to things like dma_buf_phys_vec_to_sgt() and the hmm helpers
> to use the DMA API correctly.

That is a completely broken approach as well since it limits the exported resources to addresses the CPU can reach.

Christoph Hellwig is right that drivers should never use that stuff directly, not even through that dma_buf_phys_vec_to_sgt() function.

I need to take a closer look and maybe discuss with Christoph and Thomas how to proceed here.

>> 1a) Let's say the importer (Xe) has ATS enabled, and the exporter is
>> supposed to map dma_addresses.
>> In between there is a switch that allows direct traffic.
>> Now with your patch series, How would the exporter know that Xe has ATS
>> enabled and therefore should return IOVA mappings rather than the BUS
>> addresses representing the shortcut.
> 
> The thing that does the DMA map must be aware of what PCI TLPs the
> PCIe initiator will use to compute the proper P2P mapping mode.
> 
> ATS always needs an IOVA but that still doesn't mean the P2P will
> work.
> 
>> It seems to me that a pci-device settable flag "ATS always enabled"
>> should be enough to fix both issues?
> 
> It should be be per-mapping to support the NIC workflow that isn't a
> global operation.

I just realized what you guys are doing and I'm not sure if the Linux PCI subsystem should support such hacks at all.

Basically from the point of view of the TA the NIC has ATS enabled all the time, but has a per request option to use translated addresses directly without previously translating and caching them using ATS, correct?

If yes than that is extremely questionable behavior, I'm not sure if that is covered by the PCIe spec.

ATS is meant to be an optimization which moves the TLB from the root complex (TA) into the devices at the cost of TLB invalidation complexity. But what you do here is abusing that functionality as far as I can see.

Christian.


> 
> Jason