mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Christian König" <christian.koenig@amd.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Christoph Hellwig <hch@lst.de>,
	Leon Romanovsky <leonro@nvidia.com>,
	Xu Yilun <yilun.xu@linux.intel.com>,
	kvm@vger.kernel.org, dri-devel@lists.freedesktop.org,
	linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org,
	sumit.semwal@linaro.org, pbonzini@redhat.com, seanjc@google.com,
	alex.williamson@redhat.com, vivek.kasireddy@intel.com,
	dan.j.williams@intel.com, aik@amd.com, yilun.xu@intel.com,
	linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org,
	lukas@wunner.de, yan.y.zhao@intel.com, daniel.vetter@ffwll.ch,
	leon@kernel.org, baolu.lu@linux.intel.com,
	zhenzhong.duan@intel.com, tao1.su@intel.com
Subject: Re: [RFC PATCH 01/12] dma-buf: Introduce dma_buf_get_pfn_unlocked() kAPI
Date: Wed, 8 Jan 2025 16:25:54 +0100	[thread overview]
Message-ID: <5a858e00-6fea-4a7a-93be-f23b66e00835@amd.com> (raw)
In-Reply-To: <20250108145843.GR5556@nvidia.com>

Am 08.01.25 um 15:58 schrieb Jason Gunthorpe:
> On Wed, Jan 08, 2025 at 02:44:26PM +0100, Christian König wrote:
>
>>> Having the importer do the mapping is the correct way to operate the
>>> DMA API and the new API that Leon has built to fix the scatterlist
>>> abuse in dmabuf relies on importer mapping as part of it's
>>> construction.
>> Exactly on that I strongly disagree on.
>>
>> DMA-buf works by providing DMA addresses the importer can work with and
>> *NOT* the underlying location of the buffer.
> The expectation is that the DMA API will be used to DMA map (most)
> things, and the DMA API always works with a physaddr_t/pfn
> argument. Basically, everything that is not a private address space
> should be supported by improving the DMA API. We are on course for
> finally getting all the common cases like P2P and MMIO solved
> here. That alone will take care of alot.

Well, from experience the DMA API has failed more often than it actually 
worked in the way required by drivers.

Especially that we tried to hide architectural complexity in there 
instead of properly expose limitations to drivers is not something I 
consider a good design approach.

So I see putting even more into that extremely critical.

> For P2P cases we are going toward (PFN + P2P source information) as
> input to the DMA API. The additional "P2P source information" provides
> a good way for co-operating drivers to represent private address
> spaces as well. Both importer and exporter can have full understanding
> what is being mapped and do the correct things, safely.

I can say from experience that this is clearly not going to work for all 
use cases.

It would mean that we have to pull a massive amount of driver specific 
functionality into the DMA API.

Things like programming access windows for PCI BARs is completely driver 
specific and as far as I can see can't be part of the DMA API without 
things like callbacks.

With that in mind the DMA API would become a mid layer between different 
drivers and that is really not something you are suggesting, isn't it?

> So, no, we don't loose private address space support when moving to
> importer mapping, in fact it works better because the importer gets
> more information about what is going on.

Well, sounds like I wasn't able to voice my concern. Let me try again:

We should not give importers information they don't need. Especially not 
information about the backing store of buffers.

So that importers get more information about what's going on is a bad thing.

> I have imagined a staged approach were DMABUF gets a new API that
> works with the new DMA API to do importer mapping with "P2P source
> information" and a gradual conversion.

To make it clear as maintainer of that subsystem I would reject such a 
step with all I have.

We have already gone down that road and it didn't worked at all and was 
a really big pain to pull people back from it.

> Exporter mapping falls down in too many cases already:
>
> 1) Private addresses spaces don't work fully well because many devices
> need some indication what address space is being used and scatter list
> can't really properly convey that. If the DMABUF has a mixture of CPU
> and private it becomes a PITA

Correct, yes. That's why I said that scatterlist was a bad choice for 
the interface.

But exposing the backing store to importers and then let them do 
whatever they want with it sounds like an even worse idea.

> 2) Multi-path PCI can require the importer to make mapping decisions
> unique to the device and program device specific information for the
> multi-path. We are doing this in mlx5 today and have hacks because
> DMABUF is destroying the information the importer needs to choose the
> correct PCI path.

That's why the exporter gets the struct device of the importer so that 
it can plan how those accesses are made. Where exactly is the problem 
with that?

When you have an use case which is not covered by the existing DMA-buf 
interfaces then please voice that to me and other maintainers instead of 
implementing some hack.

> 3) Importing devices need to know if they are working with PCI P2P
> addresses during mapping because they need to do things like turn on
> ATS on their DMA. As for multi-path we have the same hacks inside mlx5
> today that assume DMABUFs are always P2P because we cannot determine
> if things are P2P or not after being DMA mapped.

Why would you need ATS on PCI P2P and not for system memory accesses?

> 4) TPH bits needs to be programmed into the importer device but are
> derived based on the NUMA topology of the DMA target. The importer has
> no idea what the DMA target actually was because the exporter mapping
> destroyed that information.

Yeah, but again that is completely intentional.

I assume you mean TLP processing hints when you say TPH and those should 
be part of the DMA addresses provided by the exporter.

That an importer tries to look behind the curtain and determines the 
NUMA placement and topology themselves is clearly a no-go from the 
design perspective.

> 5) iommufd and kvm are both using CPU addresses without DMA. No
> exporter mapping is possible

We have customers using both KVM and XEN with DMA-buf, so I can clearly 
confirm that this isn't true.

Regards,
Christian.

>
> Jason


  reply	other threads:[~2025-01-08 15:26 UTC|newest]

Thread overview: 141+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-01-07 14:27 [RFC PATCH 00/12] Private MMIO support for private assigned dev Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 01/12] dma-buf: Introduce dma_buf_get_pfn_unlocked() kAPI Xu Yilun
2025-01-08  8:01   ` Christian König
2025-01-08 13:23     ` Jason Gunthorpe
2025-01-08 13:44       ` Christian König
2025-01-08 14:58         ` Jason Gunthorpe
2025-01-08 15:25           ` Christian König [this message]
2025-01-08 16:22             ` Jason Gunthorpe
2025-01-08 17:56               ` Xu Yilun
2025-01-10 19:24                 ` Simona Vetter
2025-01-10 20:16                   ` Jason Gunthorpe
2025-01-08 18:44               ` Simona Vetter
2025-01-08 19:22                 ` Xu Yilun
     [not found]                   ` <58e97916-e6fd-41ef-84b4-bbf53ed0e8e4@amd.com>
2025-01-08 23:06                     ` Xu Yilun
2025-01-10 19:34                       ` Simona Vetter
2025-01-10 20:38                         ` Jason Gunthorpe
2025-01-12 22:10                           ` Xu Yilun
2025-01-14 14:44                           ` Simona Vetter
2025-01-14 17:31                             ` Jason Gunthorpe
2025-01-15  8:55                               ` Simona Vetter
2025-01-15  9:32                                 ` Christoph Hellwig
2025-01-15 13:34                                   ` Jason Gunthorpe
2025-01-16  5:33                                     ` Christoph Hellwig
2024-06-19 23:39                                       ` Xu Yilun
2025-01-16 13:28                                       ` Jason Gunthorpe
     [not found]                                 ` <420bd2ea-d87c-4f01-883e-a7a5cf1635fe@amd.com>
2025-01-17 14:42                                   ` Simona Vetter
2025-01-20 12:14                                     ` Christian König
2025-01-20 17:59                                       ` Jason Gunthorpe
2025-01-20 18:50                                         ` Simona Vetter
2025-01-20 19:48                                           ` Jason Gunthorpe
2025-01-21 16:11                                             ` Simona Vetter
2025-01-21 17:36                                               ` Jason Gunthorpe
2025-01-22 11:04                                                 ` Simona Vetter
2025-01-22 13:28                                                   ` Jason Gunthorpe
2025-01-22 13:29                                                   ` Christian König
2025-01-22 14:37                                                     ` Jason Gunthorpe
2025-01-22 14:59                                                       ` Christian König
2025-01-23 13:59                                                         ` Jason Gunthorpe
     [not found]                                                           ` <9a36fba5-2dee-46fd-9f51-47c5f0ffc1d4@amd.com>
2025-01-23 14:35                                                             ` Christian König
2025-01-23 15:02                                                               ` Jason Gunthorpe
     [not found]                                                                 ` <89f46c7f-a585-44e2-963d-bf00bf09b493@amd.com>
2025-01-23 16:08                                                                   ` Jason Gunthorpe
2025-01-09  8:09                     ` Christian König
2025-01-10 20:54                       ` Jason Gunthorpe
2025-01-15  9:38                         ` Christian König
2025-01-15 13:38                           ` Jason Gunthorpe
     [not found]                             ` <f6c2524f-5ef5-4c2c-a464-a7b195e0bf6c@amd.com>
2025-01-15 13:46                               ` Christian König
2025-01-15 14:14                                 ` Jason Gunthorpe
     [not found]                                   ` <c86cfee1-063a-4972-a343-ea0eff2141c9@amd.com>
2025-01-15 14:30                                     ` Christian König
2025-01-15 15:10                                       ` Jason Gunthorpe
     [not found]                                         ` <6f7a14aa-f607-45f9-9e15-759e26079dec@amd.com>
2025-01-15 17:09                                           ` Jason Gunthorpe
     [not found]                                             ` <5f588dac-d3e2-445d-9389-067b875412dc@amd.com>
2024-06-20 22:02                                               ` Xu Yilun
2025-01-20 13:44                                                 ` Christian König
2025-01-22  4:16                                                   ` Xu Yilun
2025-01-16 16:07                                               ` Jason Gunthorpe
2025-01-17 14:37                                                 ` Simona Vetter
     [not found]               ` <0e7f92bd-7da3-4328-9081-0957b3d155ca@amd.com>
2025-01-09  9:28                 ` Leon Romanovsky
2025-01-07 14:27 ` [RFC PATCH 02/12] vfio: Export vfio device get and put registration helpers Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 03/12] vfio/pci: Share the core device pointer while invoking feature functions Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 04/12] vfio/pci: Allow MMIO regions to be exported through dma-buf Xu Yilun
2026-05-06  2:35   ` Alexey Kardashevskiy
2026-05-06 13:16     ` Jason Gunthorpe
2026-05-07  7:16       ` Alexey Kardashevskiy
2026-05-11 12:01         ` Jason Gunthorpe
2026-05-11 23:42         ` Alexey Kardashevskiy
2026-05-11 23:56           ` Jason Gunthorpe
2026-05-12  5:49             ` Alexey Kardashevskiy
2025-01-07 14:27 ` [RFC PATCH 05/12] vfio/pci: Support get_pfn() callback for dma-buf Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 06/12] KVM: Support vfio_dmabuf backed MMIO region Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 07/12] KVM: x86/mmu: Handle page fault for vfio_dmabuf backed MMIO Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 08/12] vfio/pci: Create host unaccessible dma-buf for private device Xu Yilun
2025-01-08 13:30   ` Jason Gunthorpe
2025-01-08 16:57     ` Xu Yilun
2025-01-09 14:40       ` Jason Gunthorpe
2025-01-09 16:40         ` Xu Yilun
2025-01-10 13:31           ` Jason Gunthorpe
2025-01-11  3:48             ` Xu Yilun
2025-01-13 16:49               ` Jason Gunthorpe
2024-06-17 23:28                 ` Xu Yilun
2025-01-14 13:35                   ` Jason Gunthorpe
2025-01-15 12:57                     ` Alexey Kardashevskiy
2025-01-15 13:01                       ` Jason Gunthorpe
2025-01-17  1:57                         ` Baolu Lu
2025-01-17 13:25                           ` Jason Gunthorpe
2024-06-23 19:59                             ` Xu Yilun
2025-01-20 13:25                               ` Jason Gunthorpe
2024-06-24 21:12                                 ` Xu Yilun
2025-01-21 17:43                                   ` Jason Gunthorpe
2025-01-22  4:32                                     ` Xu Yilun
2025-01-22 12:55                                       ` Jason Gunthorpe
2025-01-23  7:41                                         ` Xu Yilun
2025-01-23 13:08                                           ` Jason Gunthorpe
2025-01-20  4:41                             ` Baolu Lu
2025-01-20  9:45                             ` Alexey Kardashevskiy
2025-01-20 13:28                               ` Jason Gunthorpe
2025-03-12  1:37                                 ` Dan Williams
2025-03-17 16:38                                   ` Jason Gunthorpe
2025-01-07 14:27 ` [RFC PATCH 09/12] vfio/pci: Export vfio dma-buf specific info for importers Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 10/12] KVM: vfio_dmabuf: Fetch VFIO specific dma-buf data for sanity check Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 11/12] KVM: x86/mmu: Export kvm_is_mmio_pfn() Xu Yilun
2025-01-07 14:27 ` [RFC PATCH 12/12] KVM: TDX: Implement TDX specific private MMIO map/unmap for SEPT Xu Yilun
2025-04-29  6:48 ` [RFC PATCH 00/12] Private MMIO support for private assigned dev Alexey Kardashevskiy
2025-04-29  7:50   ` Alexey Kardashevskiy
2025-05-09  3:04     ` Alexey Kardashevskiy
2025-05-09 11:12       ` Xu Yilun
2025-05-09 16:28         ` Xu Yilun
2025-05-09 18:43           ` Jason Gunthorpe
2025-05-10  3:47             ` Xu Yilun
2025-05-12  9:30               ` Alexey Kardashevskiy
2025-05-12 14:06                 ` Jason Gunthorpe
2025-05-13 10:03                   ` Zhi Wang
2025-05-14  9:47                     ` Xu Yilun
2025-05-14 20:05                       ` Zhi Wang
2025-05-15 18:02                         ` Xu Yilun
2025-05-15 19:21                           ` Jason Gunthorpe
2025-05-16  6:19                             ` Xu Yilun
2025-05-16 12:49                               ` Jason Gunthorpe
2025-05-17  2:33                                 ` Xu Yilun
2025-05-20 10:57                           ` Alexey Kardashevskiy
2025-05-24  3:33                             ` Xu Yilun
2025-05-15 10:29                     ` Alexey Kardashevskiy
2025-05-15 16:44                       ` Zhi Wang
2025-05-15 16:53                         ` Zhi Wang
2025-05-21 10:41                           ` Alexey Kardashevskiy
2025-05-14  7:02                   ` Xu Yilun
2025-05-14 16:33                     ` Jason Gunthorpe
2025-05-15 16:04                       ` Xu Yilun
2025-05-15 17:56                         ` Jason Gunthorpe
2025-05-16  6:03                           ` Xu Yilun
2025-05-22  3:45                         ` Alexey Kardashevskiy
2025-05-24  3:13                           ` Xu Yilun
2025-05-26  7:18                             ` Alexey Kardashevskiy
2025-05-29 14:41                               ` Xu Yilun
2025-05-29 16:29                                 ` Jason Gunthorpe
2025-05-30 16:07                                   ` Xu Yilun
2025-05-30  2:29                                 ` Alexey Kardashevskiy
2025-05-30 16:23                                   ` Xu Yilun
2025-06-10  4:20                                     ` Alexey Kardashevskiy
2025-06-10  5:19                                       ` Baolu Lu
2025-06-10  6:53                                       ` Xu Yilun
2025-05-14  3:20                 ` Xu Yilun
2025-06-10  4:37                   ` Alexey Kardashevskiy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5a858e00-6fea-4a7a-93be-f23b66e00835@amd.com \
    --to=christian.koenig@amd.com \
    --cc=aik@amd.com \
    --cc=alex.williamson@redhat.com \
    --cc=baolu.lu@linux.intel.com \
    --cc=dan.j.williams@intel.com \
    --cc=daniel.vetter@ffwll.ch \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=hch@lst.de \
    --cc=jgg@nvidia.com \
    --cc=kvm@vger.kernel.org \
    --cc=leon@kernel.org \
    --cc=leonro@nvidia.com \
    --cc=linaro-mm-sig@lists.linaro.org \
    --cc=linux-coco@lists.linux.dev \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=lukas@wunner.de \
    --cc=pbonzini@redhat.com \
    --cc=seanjc@google.com \
    --cc=sumit.semwal@linaro.org \
    --cc=tao1.su@intel.com \
    --cc=vivek.kasireddy@intel.com \
    --cc=yan.y.zhao@intel.com \
    --cc=yilun.xu@intel.com \
    --cc=yilun.xu@linux.intel.com \
    --cc=zhenzhong.duan@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®