Re: [PATCH v8 8/9] vfio/pci: Support mmap() of a VFIO DMABUF

[Matt Evans <matt@ozlabs.org>]

Hi Alex,

On 14/10/2026 21:38, Matt Evans wrote:
> Hi Alexes,
> 
> On 14/10/2026 00:13, Alex Williamson wrote:
>> On Tue,  6 Oct 2026 20:36:36 +0100
>> Matt Evans <matt@ozlabs.org> wrote:
>>
>>> A VFIO DMABUF can export a subset of a BAR to userspace by fd; add
>>> support for mmap() of this fd.  This provides another route for a
>>> process to map BARs, in which the process can only map a specific
>>> subset of a BAR represented by the exported DMABUF.
>>>
>>> mmap() support enables userspace drivers to safely delegate BAR
>>> sub-ranges to other processes by sharing a DMABUF fd, without having
>>> to share the (omnipotent) VFIO device fd with them.
>>>
>>> Since the main VFIO BAR mmap() is now DMABUF-aware, the new mmap() 
>>> reuses
>>> the existing vm_ops.
>>>
>>> The lifecycle of an exported DMABUF remains decoupled from that of the
>>> device fd it came from, i.e. the device fd could be closed with DMABUF
>>> VMAs present, meaning a fault on a VMA could happen concurrently with
>>> vfio_pci_dma_buf_cleanup().
>>>
>>> To deal with this scenario, the fault handler now temporarily takes a
>>> VFIO device registration to ensure the vdev remains valid, and then
>>> vdev locks can be taken.
>>
>> This sounds like potentially measurable overhead if something is
>> slamming the fault handler where huge-fault is not supported or active.
>> Have you considered a fast path for the internal dmabuf where the
>> lifecycle is constrained?
> [snip]
> 
> Hmm.  I'll hack up a test to try to measure the relative overhead of VMA 
> faults before/after (modulo access to meaningful HW).

Alex Mastro and I have put together some tests, and measured this 
overhead on the v8 series.

https://github.com/evansm7/vfio-playground/tree/master/vfio-barmap-perf

Alex also generated a similar test with a slightly different method:
  https://github.com/opsound/vfio-tools/blob/main/vfio_bar_bench.c


Test method:

Time accesses to a VFIO mmap'd BAR immediately after the mmap, to 
measure the 'cold' initial fault time.  Re-time a second pass for which 
the pagetable should now be populated, and subtract.  This should 
isolate the cost of the fault (uarch exception, generic pagefault 
plumbing, VFIO fault handler) from common costs (the MMIO access itself, 
TLB misses, the loop iterating across the BAR).  Maximise the number of 
faults by disabling hugepages (per concern above).

This is done both for a single-threaded case which just iterates across 
a new BAR mapping, and a multi-threaded contended case.  Rather than 
just pthreads, this test fork()s to try to avoid seeing contention on 
per-mm locks (e.g. mmap_lock) and to focus on the VMA/VFIO locking.

The default test mmaps before forking, meaning there's one shared 
mapping/DMABUF accessed from both processes.  The fault handler should 
then contend on dmabuf->resv, vdev->memory_lock, and vdev->dmabuf_lock.

For comparison/curiosity, the --mt-map-late option mmaps _after_ 
forking; this leads to a mapping/DMABUF in each process so I expect no 
contention on dmabuf->resv (but still on memory_lock/dmabuf_lock).

Alex's test instead runs threads in-process, and goes further with 
supporting a larger number of threads (though with the caveat that they 
may also be fighting over intra-process mm locks).  It also supports a 
mode where threads can access per-thread VMAs (which is comparable to 
the 'mt-map-late' option of my test).  Both are interesting approaches IMO.


Findings:

On the dusty AMD Ryzen 3950X under my desk (targeting a 256MB BAR) I 
found the following averaged over 200 runs:

  - A single-threaded/uncontended fault became 5.8% slower
  - A two-process contended case became 8.7% slower
   - 0.5% of this is the "early mmap" dmabuf->resv contention, i.e. using
     mt-map-late (a mapping in each process) reduces by 0.5%.

Alex's runs on fancier EPYC 9334 machines show a similar (slightly 
lower) overhead in the 1-2 case, and go on to show fairly high overhead 
when many threads are involved (baseline times in ns, relative series v8 
behaviour in percentage change):

   Threads    One shared mapping   Mapping per thread
              7.3     v8            7.3     v8
   --------------------------------------------
      1       761   +4.5%           777   +2.7%
      2       869   +6.6%           840   +5.6%
      8      1465    +78%          1363    +47%
     16      4152    +97%          4258    +19%
     32      9927    +92%         10158    +12%


Now.  The absolute timings of these sub-microsecond faults indicate a 
realistic application won't see a human-visible change.  Small BARs 
mapped with 4k pages are still fast to map;  it is reasonable to ask 
that applications map huge (GBs/TBs) BARs using huge (MB/GB) pages!  Of 
course, this is just a one-time cost of warming up the VMA, not ongoing.


Alex, my preference is not to complicate the fault handler with a fast 
path just yet.  It's easy to think of a contrived scenario where this 
matters but in middle-of-the-road application behaviour it would be 
surprising if this was noticeable.  What are your thoughts?

(In the previous mail I suggested keying off priv->vfile (or a flag 
thereof), and it seems ~straightforward to skip the extra 
synchronisation.  If you feel strongly that the fault handler doesn't 
have enough if() statements yet and needs more paths then we could bring 
this into v9, but would prefer to follow up later. :)  A fastpath diff 
appears small but the reasoning overhead is bigger.)


Thanks,


Matt