mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Alex Williamson <alex@shazbot.org>
To: Matt Evans <matt@ozlabs.org>
Cc: "Alex Mastro" <amastro@fb.com>,
	"Leon Romanovsky" <leon@kernel.org>,
	"Jason Gunthorpe" <jgg@nvidia.com>,
	"Christian König" <christian.koenig@amd.com>,
	"Kevin Tian" <kevin.tian@intel.com>,
	"Pranjal Shrivastava" <praan@google.com>,
	"Longfang Liu" <liulongfang@huawei.com>,
	"Mahmoud Adam" <mngyadam@amazon.de>,
	"David Matlack" <dmatlack@google.com>,
	"Björn Töpel" <bjorn@kernel.org>,
	"Sumit Semwal" <sumit.semwal@linaro.org>,
	"Ankit Agrawal" <ankita@nvidia.com>,
	"Alistair Popple" <apopple@nvidia.com>,
	"Vivek Kasireddy" <vivek.kasireddy@intel.com>,
	linux-kernel@vger.kernel.org, linux-media@vger.kernel.org,
	dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org,
	kvm@vger.kernel.org, linux-pci@vger.kernel.org, alex@shazbot.org
Subject: Re: [PATCH v8 8/9] vfio/pci: Support mmap() of a VFIO DMABUF
Date: Tue, 17 Nov 2026 15:33:12 -0800	[thread overview]
Message-ID: <20261117153312.781fd839@shazbot.org> (raw)
In-Reply-To: <df301c78-2745-4c64-a331-c830efb74660@ozlabs.org>

On Wed, 11 Nov 2026 10:48:47 +0000
Matt Evans <matt@ozlabs.org> wrote:

> Hi Alex,
> 
> On 14/10/2026 21:38, Matt Evans wrote:
> > Hi Alexes,
> > 
> > On 14/10/2026 00:13, Alex Williamson wrote:  
> >> On Tue,  6 Oct 2026 20:36:36 +0100
> >> Matt Evans <matt@ozlabs.org> wrote:
> >>  
> >>> A VFIO DMABUF can export a subset of a BAR to userspace by fd; add
> >>> support for mmap() of this fd.  This provides another route for a
> >>> process to map BARs, in which the process can only map a specific
> >>> subset of a BAR represented by the exported DMABUF.
> >>>
> >>> mmap() support enables userspace drivers to safely delegate BAR
> >>> sub-ranges to other processes by sharing a DMABUF fd, without having
> >>> to share the (omnipotent) VFIO device fd with them.
> >>>
> >>> Since the main VFIO BAR mmap() is now DMABUF-aware, the new mmap() 
> >>> reuses
> >>> the existing vm_ops.
> >>>
> >>> The lifecycle of an exported DMABUF remains decoupled from that of the
> >>> device fd it came from, i.e. the device fd could be closed with DMABUF
> >>> VMAs present, meaning a fault on a VMA could happen concurrently with
> >>> vfio_pci_dma_buf_cleanup().
> >>>
> >>> To deal with this scenario, the fault handler now temporarily takes a
> >>> VFIO device registration to ensure the vdev remains valid, and then
> >>> vdev locks can be taken.  
> >>
> >> This sounds like potentially measurable overhead if something is
> >> slamming the fault handler where huge-fault is not supported or active.
> >> Have you considered a fast path for the internal dmabuf where the
> >> lifecycle is constrained?  
> > [snip]
> > 
> > Hmm.  I'll hack up a test to try to measure the relative overhead of VMA 
> > faults before/after (modulo access to meaningful HW).  
> 
> Alex Mastro and I have put together some tests, and measured this 
> overhead on the v8 series.
> 
> https://github.com/evansm7/vfio-playground/tree/master/vfio-barmap-perf
> 
> Alex also generated a similar test with a slightly different method:
>   https://github.com/opsound/vfio-tools/blob/main/vfio_bar_bench.c
> 
> 
> Test method:
> 
> Time accesses to a VFIO mmap'd BAR immediately after the mmap, to 
> measure the 'cold' initial fault time.  Re-time a second pass for which 
> the pagetable should now be populated, and subtract.  This should 
> isolate the cost of the fault (uarch exception, generic pagefault 
> plumbing, VFIO fault handler) from common costs (the MMIO access itself, 
> TLB misses, the loop iterating across the BAR).  Maximise the number of 
> faults by disabling hugepages (per concern above).
> 
> This is done both for a single-threaded case which just iterates across 
> a new BAR mapping, and a multi-threaded contended case.  Rather than 
> just pthreads, this test fork()s to try to avoid seeing contention on 
> per-mm locks (e.g. mmap_lock) and to focus on the VMA/VFIO locking.
> 
> The default test mmaps before forking, meaning there's one shared 
> mapping/DMABUF accessed from both processes.  The fault handler should 
> then contend on dmabuf->resv, vdev->memory_lock, and vdev->dmabuf_lock.
> 
> For comparison/curiosity, the --mt-map-late option mmaps _after_ 
> forking; this leads to a mapping/DMABUF in each process so I expect no 
> contention on dmabuf->resv (but still on memory_lock/dmabuf_lock).
> 
> Alex's test instead runs threads in-process, and goes further with 
> supporting a larger number of threads (though with the caveat that they 
> may also be fighting over intra-process mm locks).  It also supports a 
> mode where threads can access per-thread VMAs (which is comparable to 
> the 'mt-map-late' option of my test).  Both are interesting approaches IMO.
> 
> 
> Findings:
> 
> On the dusty AMD Ryzen 3950X under my desk (targeting a 256MB BAR) I 
> found the following averaged over 200 runs:
> 
>   - A single-threaded/uncontended fault became 5.8% slower
>   - A two-process contended case became 8.7% slower
>    - 0.5% of this is the "early mmap" dmabuf->resv contention, i.e. using
>      mt-map-late (a mapping in each process) reduces by 0.5%.
> 
> Alex's runs on fancier EPYC 9334 machines show a similar (slightly 
> lower) overhead in the 1-2 case, and go on to show fairly high overhead 
> when many threads are involved (baseline times in ns, relative series v8 
> behaviour in percentage change):
> 
>    Threads    One shared mapping   Mapping per thread
>               7.3     v8            7.3     v8
>    --------------------------------------------
>       1       761   +4.5%           777   +2.7%
>       2       869   +6.6%           840   +5.6%
>       8      1465    +78%          1363    +47%
>      16      4152    +97%          4258    +19%
>      32      9927    +92%         10158    +12%
> 
> 
> Now.  The absolute timings of these sub-microsecond faults indicate a 
> realistic application won't see a human-visible change.  Small BARs 
> mapped with 4k pages are still fast to map;  it is reasonable to ask 
> that applications map huge (GBs/TBs) BARs using huge (MB/GB) pages!  Of 
> course, this is just a one-time cost of warming up the VMA, not ongoing.
> 
> 
> Alex, my preference is not to complicate the fault handler with a fast 
> path just yet.  It's easy to think of a contrived scenario where this 
> matters but in middle-of-the-road application behaviour it would be 
> surprising if this was noticeable.  What are your thoughts?
> 
> (In the previous mail I suggested keying off priv->vfile (or a flag 
> thereof), and it seems ~straightforward to skip the extra 
> synchronisation.  If you feel strongly that the fault handler doesn't 
> have enough if() statements yet and needs more paths then we could bring 
> this into v9, but would prefer to follow up later. :)  A fastpath diff 
> appears small but the reasoning overhead is bigger.)

I tend to agree.  IIUC, in the worst case concurrency (that we likely
don't experience), if we assume huge pages, we're looking at low
single digit millisecond total overhead to fault in an entire BAR
(~256 max faults).  Further, if our configuration doesn't force type1 to
fault in the entire BAR for DMA mapping, that overhead is amortized
across every user/guest access.

So yes, it's probably not worth having two different locking semantics,
but we should note that as an intentional decision in the commit log,
potentially even as a comment in the fault handler.  Thanks,

Alex

  reply	other threads:[~2026-11-17 23:33 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-06 19:36 [PATCH v8 0/9] vfio/pci: Add mmap() for DMABUFs Matt Evans
2026-10-06 19:36 ` [PATCH v8 1/9] vfio/pci: Remove DMABUF export dependency on vdev->memory_lock Matt Evans
2026-10-13 23:14   ` Alex Williamson
2026-10-14 18:05     ` Matt Evans
2026-10-06 19:36 ` [PATCH v8 2/9] vfio/pci: Un-revoke DMABUFs in LOW_POWER_ENTRY_WITH_WAKEUP resume Matt Evans
2026-10-06 19:36 ` [PATCH v8 3/9] dma-buf: Provide dma_buf_set_name() Matt Evans
2026-10-07  9:22   ` Christian König
2026-10-12  6:27   ` Leon Romanovsky
2026-10-06 19:36 ` [PATCH v8 4/9] vfio/pci: Add a helper to look up PFNs for DMABUFs Matt Evans
2026-10-13 23:14   ` Alex Williamson
2026-10-14 18:25     ` Matt Evans
2026-10-06 19:36 ` [PATCH v8 5/9] vfio/pci: Add a helper to create a DMABUF for a BAR-map VMA Matt Evans
2026-10-06 19:36 ` [PATCH v8 6/9] vfio/pci: Convert BAR mmap() to use a DMABUF Matt Evans
2026-10-06 19:36 ` [PATCH v8 7/9] vfio/pci: Clean up BAR zap and revocation Matt Evans
2026-10-13 23:13   ` Alex Williamson
2026-10-14 18:07     ` Matt Evans
2026-10-06 19:36 ` [PATCH v8 8/9] vfio/pci: Support mmap() of a VFIO DMABUF Matt Evans
2026-10-13 23:13   ` Alex Williamson
2026-10-14 20:38     ` Matt Evans
2026-11-11 10:48       ` Matt Evans
2026-11-17 23:33         ` Alex Williamson [this message]
2026-10-06 19:36 ` [PATCH v8 9/9] vfio/pci: Revoke a DMABUF on request from userspace Matt Evans
2026-10-13 23:13   ` Alex Williamson
2026-10-14 19:02     ` Matt Evans

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261117153312.781fd839@shazbot.org \
    --to=alex@shazbot.org \
    --cc=amastro@fb.com \
    --cc=ankita@nvidia.com \
    --cc=apopple@nvidia.com \
    --cc=bjorn@kernel.org \
    --cc=christian.koenig@amd.com \
    --cc=dmatlack@google.com \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=jgg@nvidia.com \
    --cc=kevin.tian@intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=leon@kernel.org \
    --cc=linaro-mm-sig@lists.linaro.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=liulongfang@huawei.com \
    --cc=matt@ozlabs.org \
    --cc=mngyadam@amazon.de \
    --cc=praan@google.com \
    --cc=sumit.semwal@linaro.org \
    --cc=vivek.kasireddy@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®