From: "David Matlack" <dmatlack@google.com>
To: "Samiullah Khawaja" <skhawaja@google.com>,
"David Matlack" <dmatlack@google.com>
Cc: "Jason Gunthorpe" <jgg@nvidia.com>,
"Alex Mastro" <amastro@fb.com>,
dri-devel@lists.freedesktop.org, iommu@lists.linux.dev,
kvm@vger.kernel.org, linaro-mm-sig@lists.linaro.org,
linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-media@vger.kernel.org, linux-pci@vger.kernel.org,
"Alex Williamson" <alex@shazbot.org>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Christian König" <christian.koenig@amd.com>,
"Josh Hilke" <jrhilke@google.com>,
"Kevin Tian" <kevin.tian@intel.com>,
"Leon Romanovsky" <leon@kernel.org>,
"Logan Gunthorpe" <logang@deltatee.com>,
"Matt Evans" <mattev@meta.com>, "Matt Evans" <matt@ozlabs.org>,
"Mina Almasry" <almasrymina@google.com>,
"Shuah Khan" <shuah@kernel.org>,
"Sumit Semwal" <sumit.semwal@linaro.org>,
"Vipin Sharma" <vipinsh@google.com>
Subject: Re: [RFC PATCH 0/7] vfio: selftests: Add peer-to-peer DMA selftest
Date: Mon, 16 Nov 2026 12:50:24 -0800 [thread overview]
Message-ID: <DMWXK2VTL3XP.MWW7ZG7Z6HUF@google.com> (raw)
In-Reply-To: <avtI--OIZu5E5sma@google.com>
On Mon Nov 16, 2026 at 10:24 AM PST, Samiullah Khawaja wrote:
> On Mon, Nov 16, 2026 at 10:02:56AM -0800, David Matlack wrote:
>>Looks like GPUs and NVMe can do both as well.
>
> AFAIU the memory BARs for GPUs would need initialization. The internal
> VRAM needs to be exposed (mapped) to the BARs.
>
> Do you think we should have a P2P device "setup" step that can be
> implemented for various types of devices?
Yes, I think we need something like that.
I've been prototyping a new optional op in struct vfio_pci_driver_ops:
/*
* Set up a range of device memory, accessible through a BAR,
* that other devices can target with P2P DMA and that the CPU
* can access through mem_bar->vaddr.
*/
int (*mem_bar_setup)(struct vfio_pci_device *device, u64 min_size,
struct vfio_pci_mem_bar *mem_bar);
vfio_pci_find_mem_bar() falls back to calling it when the device does
not match any entry in mem_bar_descs[], sanity checks the returned
range, and then runs the existing memory verification on it.
mem_bar_descs[] stays around for devices like pci-testdev that need no
setup.
For NVIDIA GPUs, I had an LLM work on it and the setup was simpler than
I expected, at least on Blackwell. BAR1 comes out of reset in physical
mode, where BAR1 offset N maps directly to VRAM physical address N
without going through the GPU MMU. The LLM confirmed this on by writing
VRAM through BAR1 and reading it back through the BAR0 PRAMIN window
(and vice versa) at offsets from 256MiB up to 178GiB.
The nv_falcon implementation only has to:
- Wait for FSP to finish booting, which takes a few seconds after the
reset VFIO does when the device is opened.
- Make sure BAR1 is in physical mode.
- Return the range of BAR1 that maps VRAM (the size comes from
NV_USABLE_FB_SIZE_IN_MB), staying clear of firmware reservations at
either end.
With that the P2P test passes with a ConnectX-7 (mlx5 driver) as the
initiator and B200 VRAM as the target.
A couple of open questions:
1. The prototype allows .mem_bar_setup() to be called before .init().
That is enough for GPUs, but as Jason described, mlx5 device memory
has to be allocated from firmware, which needs the command interface
and therefore init(). Supporting mlx5 as a target would mean running
.mem_bar_setup() after .init().
2. nv_falcon does not support Falcon DMA on Blackwell yet, so a B200
can only be a target for now. That requires a driver to be able to
probe a device that cannot do memcpy. I am not sure I want to go
down that road.
prev parent reply other threads:[~2026-11-16 20:50 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 19:24 David Matlack
2026-11-11 19:24 ` [RFC PATCH 1/7] vfio: selftests: Add vfio_pci_find_mem_bar() helper David Matlack
2026-11-11 19:24 ` [RFC PATCH 2/7] vfio: selftests: Verify memory behavior in vfio_pci_find_mem_bar() David Matlack
2026-11-11 19:24 ` [RFC PATCH 3/7] vfio: selftests: Allow multiple BDFs in VFIO_SELFTESTS_BDF David Matlack
2026-11-11 19:24 ` [RFC PATCH 4/7] vfio: selftests: Add vfio_pci_dmabuf_create() helper David Matlack
2026-11-11 19:24 ` [RFC PATCH 5/7] vfio: selftests: Add iommu_map_file() helper David Matlack
2026-11-11 19:24 ` [RFC PATCH 6/7] vfio: selftests: Add vfio_pci_find_p2p_pair() helper David Matlack
2026-11-11 19:24 ` [RFC PATCH 7/7] vfio: selftests: Add P2P DMA selftest David Matlack
2026-11-12 20:35 ` [RFC PATCH 0/7] vfio: selftests: Add peer-to-peer " Alex Mastro
2026-11-12 21:47 ` David Matlack
2026-11-15 23:19 ` Jason Gunthorpe
2026-11-16 18:02 ` David Matlack
2026-11-16 18:16 ` Jason Gunthorpe
2026-11-16 18:24 ` Samiullah Khawaja
2026-11-16 20:50 ` David Matlack [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMWXK2VTL3XP.MWW7ZG7Z6HUF@google.com \
--to=dmatlack@google.com \
--cc=alex@shazbot.org \
--cc=almasrymina@google.com \
--cc=amastro@fb.com \
--cc=bhelgaas@google.com \
--cc=christian.koenig@amd.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=jrhilke@google.com \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=leon@kernel.org \
--cc=linaro-mm-sig@lists.linaro.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=logang@deltatee.com \
--cc=matt@ozlabs.org \
--cc=mattev@meta.com \
--cc=shuah@kernel.org \
--cc=skhawaja@google.com \
--cc=sumit.semwal@linaro.org \
--cc=vipinsh@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®