From: Andrea Righi <arighi@nvidia.com>
To: Vineeth Pillai <vineethrp@google.com>
Cc: Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Eduard Zingerman <eddyz87@gmail.com>,
Kumar Kartikeya Dwivedi <memxor@gmail.com>,
Martin KaFai Lau <martin.lau@linux.dev>,
Song Liu <song@kernel.org>,
Yonghong Song <yonghong.song@linux.dev>,
Jiri Olsa <jolsa@kernel.org>,
Emil Tsalapatis <emil@etsalapatis.com>,
Ihor Solodrai <ihor.solodrai@linux.dev>,
John Fastabend <john.fastabend@gmail.com>,
Tejun Heo <tj@kernel.org>, David Vernet <void@manifault.com>,
Changwoo Min <changwoo@igalia.com>, Josh Don <joshdon@google.com>,
Peilin Ye <yepeilin@google.com>, Zhi Wang <zhiw@nvidia.com>,
Lee Trager <ltrager@nvidia.com>,
Balbir Singh <balbirs@nvidia.com>,
Shameer Kolothum <skolothumtho@nvidia.com>,
Himadri Chhaya-Shailesh <himadrics@protonmail.com>,
NchangRoy <royfru44@gmail.com>,
bpf@vger.kernel.org, sched-ext@lists.linux.dev,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 bpf-next 0/3] bpf: Import external pages into an arena
Date: Sat, 21 Nov 2026 22:49:27 +0100 [thread overview]
Message-ID: <awIR5y9x9sK8j7dB@gpd4> (raw)
In-Reply-To: <CA+HDTgQ0mx0PgXDps2h2kd8TbJsFWOmJGJmDRGEHZgBZ_qSVwQ@mail.gmail.com>
Hi Vineeth,
On Fri, Nov 20, 2026 at 08:45:19AM -0500, Vineeth Pillai wrote:
> Hi Andrea,
>
> Thanks for the v2 and for sharing the scx_pvhost/scx_pvguest demo
> branch. Thought of discussing the possibility of using the pvsched
> transport with the bpf arena framework implemented in this series.
Thanks for looking at this and for the detailed write-up, this is very
helpful. Comments below.
>
> On Tue, Nov 17, 2026 at 5:30 AM Andrea Righi <arighi@nvidia.com> wrote:
> >
> > As discussed in the BPF track and the sched_ext microconference at Linux
> > Plumbers 2026 [1][2], shared BPF arenas can provide a generic channel for
> > bidirectional zero-copy communication between BPF programs running on a host
> > and its guests, for example to exchange scheduling hints between host and
> > guest sched_ext schedulers.
> >
> > v1 [3] exposed a pinned arena as a sized bpffs file and gave the guest its
> > share of it as a NUMA node. Following the review, this version drops that
> > approach and teaches the arena to map pages that it didn't allocate, based on
> > Alexei's prototype [4].
> >
> > Background
> > ==========
> >
> > An arena can only map pages it allocated itself, so its memory can be shared
> > only with the BPF programs and the user space that map the arena.
> >
> > Sharing it with a guest through a NUMA node, as in v1, has two fundamental
> > problems: the node is ordinary guest RAM, so the guest arena's pages go back to
> > the guest's buddy allocator when the guest scheduler exits, while the host can
> > still write to them; and memory block granularity makes it wasteful (16M of
> > guest memory for one shared page) and limits it to a few hundred guests per
> > host scheduler.
> >
> > Design
> > ======
> >
> > The arena gains a single primitive, arena_map_ext_pages(), which maps pages
> > supplied from outside at a free arena address. Each call creates one import,
> > kept in an interval tree with its pages, which belong to their supplier, not to
> > the arena:
> >
> > - bpf_arena_free_pages() over any of them releases the whole import: the
> > pages are unmapped and unpinned, and the range goes back to the arena, so
> > the arena doesn't grow as guests come and go,
> > - they are released when the arena is destroyed,
> > - they are not counted in the arena memory usage,
> > - the arena mapping raises SIGBUS on them (SIGSEGV with
> > BPF_F_SEGV_ON_FAULT): user space accesses them through their supplier.
> >
> > Two sleepable kfuncs for syscall programs supply the pages, one for each side
> > of a host/guest shared arena:
> >
> > int bpf_arena_import_memfd(void *map, void __arena *addr,
> > u32 page_cnt, int fd, u64 off);
> >
> > Host side: pins the pages of a memfd, or of any file on tmpfs or hugetlbfs,
> > with memfd_pin_folios() like udmabuf does. The host hands the same file to the
> > VMM as a memory backend, with no change to the VMM.
> >
> > int bpf_arena_import_dax(void *map, void __arena *addr,
> > u32 page_cnt, u64 src);
> >
> > Guest side: the VMM exposes the file as a virtio-pmem or NVDIMM device, which
> > the guest turns into a device-dax device (/dev/daxX.Y). The kfunc pins the
> > device-dax pages mapped at an address of the calling task, and holds a
> > reference on their dev_pagemap until it releases them.
> >
> > Both sides then see the same pages as arena memory, at an offset known to both
> > of them, so they can agree on a layout in-band, with plain BPF loads and
> > stores, and keep pointers to it in the rest of their arena state.
> >
> > Use case: paravirt scheduling
> > =============================
> >
> > The git tree below has an additional [DEMO] commit, not meant for upstream,
> > that provides a real, tested use case of this functionality: a pair of
> > sched_ext schedulers, scx_pvhost on the host and scx_pvguest in every guest,
> > which share a few pages per VM through their arenas. The host publishes which
> > vCPUs it has preempted, its idle CPUs and every VM's fair share of the host
> > CPUs; the guests publish their demand and the state of their vCPUs. With that,
> > the guests stop queueing work behind preempted vCPUs and fold onto their share
> > of the host CPUs only while that pays off.
> >
>
> As Josh mentioned on v1, we have been developing pvsched [5] -- a
> lightweight guest/VMM/host paravirt transport (with QEMU and Cloud
> Hypervisor support) based on the "guest allocates ordinary RAM, passes
> GPAs via a small PCI BAR + synchronous doorbell, host pins via
> /dev/pvsched" model. We recently added external-policy support so the
> shared pages after a small negotiation header are completely opaque to
> the kernel framework.
>
> Looking at Patch 1, arena_map_ext_pages() (along with import_memfd for
> host/userspace sharing) already takes an arbitrary struct page ** array
> and maps it contiguously into arena VA space. That means for VMs we can
> layer your shared-arena model directly on top of the pvsched transport
> (on both host and guest) and get the best of both approaches:
Agreed, that was the idea behind keeping arena_map_ext_pages() generic:
the arena doesn't care where the pages come from, memfd and device-dax
are just the two suppliers I could use without touching the VMM.
>
> 1. No extra memory backend or devdax/nvdimm stack:
> Instead of a separate 16 MiB pmem file per VM (plus guest ZONE_DEVICE
> memmap overhead, libnvdimm/dax_pmem config, and sysfs rebinding) for
> sharing a few pages at a hardcoded 2 MiB offset, the guest allocates
> ordinary 4 KiB RAM pages (including from the right guest NUMA nodes)
> and registers their GPAs in BAR1. Both sides still map that page
> list as a flat, contiguous virtual slice in their BPF arenas via
> arena_map_ext_pages(), and Patch 2 (import_dax, along with its
> dev_pagemap unbind blocking and mem_hotplug_lock ordering) wouldn't
> be needed for VMs.
Yes, the device-dax path is mostly a way to get the whole machinery
working today with stock QEMU and Firecracker. The memory cost is small:
the backing file is sparse on the host, so only the shared pages (and
the namespace info block) are allocated, and the guest pays ~256 KB for
the memmap of the 16 MiB device. But it does need the extra device, the
nvdimm/dax stack and the namespace setup in the guest, and with a proper
transport the guest side becomes much simpler.
I'd still like to keep the device-dax import as an optional supplier,
since it works with existing VMMs and lets people experiment with shared
arenas right now. It may be deprecated in the future in favor of a
paravirt transport like pvsched/pvshm once that is available. Does that
sound reasonable?
>
> 2. Clean lifetime and synchronous revocation:
> In the guest, pvsched_guest owns the allocated RAM pages across
> guest scheduler reloads, so scx_pvguest exiting never returns shared
> pages to the guest buddy allocator while attached; on driver unbind
> or kexec, a synchronous DOORBELL=0 detaches on the host first. On
> the host, /dev/pvsched detach callbacks know the exact instant a VM
> detaches or exits and can synchronously unmap the imported arena
> PTEs (relying on ex_handler_bpf zeroing loads/dropping stores),
> avoiding stale writes or blocked device unbinds.
Yeah, this is the part I'm missing with device-dax: the dax core doesn't
tell its users that a device is going away, so there's no way to revoke
the pages and the unbind just blocks.
>
> 3. Up-front negotiation, authoritative vCPU identity, and doorbell:
> BAR0 + the page header give an explicit handshake for policy name,
> version, and layout before anything goes live, and /dev/pvsched
> binds pages directly to VMM runner TIDs and KVM vcpu_ids (avoiding
> inotify + /proc/*/maps + thread comm scraping across QEMU, CHV,
> Firecracker, crosvm, etc.). BAR0 also provides a kick doorbell and
> optional KVM boundary hooks when a policy wants event-driven wakeups
> in addition to polling.
Agreed, the vCPU discovery in the demo is just there to avoid touching
the VMM, it's the least robust part and an authoritative binding from
the transport would be much better.
>
> 4. Consumer-agnostic on both ends:
> Decoupling the transport from the consumer lets scx_pvhost <->
> scx_pvguest use shared BPF arenas as in your demo, while also
> allowing mixed setups -- e.g., keeping stock EEVDF in the guest
> (publishing via thin in-kernel hooks without replacing guest task
> scheduling, which you noted in scx-pvsched.md's TODO) consumed by a
> host sched_ext scheduler or kernel module.
Makes sense, and that's the main reason I'd like the arena side to stay
a generic primitive and not grow anything pvsched/VM-specific.
>
> 5. Live migration and Confidential VMs:
> With memfd + arena->kern_vm, host BPF writes go through init_mm and
> aren't tracked by user PTE / KVM dirty logging during pre-copy
> migration, and x-ignore-shared conflicts with VMs whose main RAM
> uses shared memfd (e.g. for vhost-user). With guest-RAM pages +
> pvsched, pages migrate as normal guest RAM: source VMM detaches at
> stop-and-copy (unmapping the arena import and marking pages dirty
> before the final sync), and destination VMM replays BAR1 into
> destination /dev/pvsched before vCPU resume (immediately disabling
> via header status if the destination runs no host scheduler, with
> zero blackout). Guest-allocated pages also map naturally to CoCo VMs
> via set_memory_decrypted() before GPA registration.
Live migration actually works with the scx-pvsched demo already: the
shared pages only hold soft state that both sides can rebuild. Every
time the host picks up a VM it writes a new epoch in the shared header.
If the migration copies the shared pages, the destination host sees its
epoch overwritten by the source host's and writes its own back; if it
doesn't, the guest sees a new epoch and writes its entries again. Either
way the guest is back under the new host's fair share within a second or
two, and if the destination runs no host scheduler, the heartbeat stops
and the guest behaves like a normal VM.
That said, having the transport detach and re-attach the pages
explicitly is cleaner than relying on the epoch, and it would let the
destination host know about the VM before it resumes.
Confidential VMs are a known limitation of v2, and a transport that
registers the pages after set_memory_decrypted() solves that naturally.
>
> On the pvsched side, supporting scx_pvhost/scx_pvguest cleanly would
> just mean:
> - making the pvsched PCI transport + /dev/pvsched arch-independent so
> arm64 works alongside x86_64,
> - allowing an ordered multi-page VM slice in BAR1 (not just 1 page per
> vCPU), and
> - exposing a thin kernel helper around arena_map_ext_pages() /
> arena_unmap_import() in kernel/bpf/arena.c so host and guest kfuncs
> can import and synchronously revoke those pages.
I can add something like this in v3:
int bpf_arena_map_ext_pages(struct bpf_map *map, u64 addr,
struct page **pages, u32 page_cnt,
void (*release)(struct page **pages,
u32 page_cnt, void *priv),
void *priv);
int bpf_arena_unmap_ext_pages(struct bpf_map *map, u64 addr,
void *priv);
The caller keeps ownership of the pages: the arena calls the release
callback exactly once, whether the import goes away through
bpf_arena_unmap_ext_pages(), bpf_arena_free_pages() or arena destruction
(the memfd and dax kfuncs use the same callback).
The unmap is synchronous: on return the PTEs are gone, the TLB is
flushed and the callback has run. The range then reads as zero from BPF
and goes back to the arena. It returns -ENOENT if addr isn't the start
of the caller's import (priv must match) or if it was already released.
Would that work for the /dev/pvsched detach path?
>
> I am working on a PoC using the paravirt transport from pvsched for the
> bpf arena framework in this series. Will share it soon.
Great, looking forward to it!
>
> We are also thinking of extracting the transport side of pvsched into a
> standalone pvshm framework, with pvsched as a consumer of pvshm. I am
> envisioning pvshm as the transport framework for guest/host BPF arena
> imports as well, while pvsched remains a standalone scheduling
> framework supporting kernel-module or BPF-based policies.
I like this split and I think a generic shared memory transport, with
BPF arenas as one consumer and pvsched as another, fits well with
keeping the arena side generic.
Thanks,
-Andrea
>
> Please let me know your thoughts about this.
>
> Thanks,
> Vineeth
>
> [5] https://github.com/pvsched/linux
>
> > On an NVIDIA Vera host (arm64), with 8-vCPU VMs on QEMU or Firecracker, at 2x
> > and 6x overcommit, compared to stock VMs, the paravirt pair is 2.2-3x as fast on
> > hackbench, 1.3-1.9x on a kernel build and serves 1.6-2x the requests of schbench
> > and nginx; tools/sched_ext/scx-pvsched.md in the [DEMO] commit describes the
> > design, the results and how to run it.
> >
> > Known limitations
> > =================
> >
> > - Imports are released whole: freeing part of an import releases all of it.
> > - Imported device-dax pages keep the device in use: unbinding it blocks until
> > the arena releases them. Revoking them would need a notification from the
> > dax core and a way to tell the BPF programs that the memory is gone.
> > - BPF programs have no machine check recovery: on a real NVDIMM, the supplier
> > has to keep bad pages out of the arena.
> > - There is no doorbell: the sched_ext use case polls the shared pages at
> > scheduling events.
> > - Confidential VMs would need the guest to keep the shared pages shared with
> > the host, which the device-dax import doesn't handle yet.
> >
> > References
> > ==========
> >
> > [1] https://lpc.events/event/20/contributions/2429/
> > [2] https://lpc.events/event/20/contributions/2482/
> > [3] https://lore.kernel.org/r/20261010224902.1232178-1-arighi@nvidia.com/
> > [4] https://lore.kernel.org/r/DM6HVNSP3USV.AJSWS289Z3I9@gmail.com/
> >
> > Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-pvsched
> >
> > Changes in v2:
> > - Drop BPF_F_ARENA_NO_FREE, BPF_F_ARENA_EXPORT and the sized bpffs file, and
> > map pages that the arena didn't allocate instead (Alexei Starovoitov)
> > - Import the host's pages from a memfd with memfd_pin_folios(), the VMM uses
> > the same file as a memory backend (Alexei Starovoitov)
> > - Replace the guest NUMA node with a device-dax import, so the guest's pages
> > never go back to its buddy allocator (Alexei Starovoitov, Josh Don)
> > - Release an import with bpf_arena_free_pages(), instead of keeping the arena
> > at its high watermark (Alexei Starovoitov, Josh Don)
> > - Map the shared pages at an offset known to both sides, instead of
> > discovering it out of band (Josh Don)
> > - Track imports in an interval tree instead of a second range tree and a
> > list, so that a release finds its import without allocating
> > - Link to v1: https://lore.kernel.org/r/20261010224902.1232178-1-arighi@nvidia.com/
> >
> > Andrea Righi (3):
> > bpf: Allow importing memfd pages into an arena
> > bpf: Allow importing device-dax pages into an arena
> > selftests/bpf: Test importing pages into an arena
> >
> > kernel/bpf/arena.c | 461 ++++++++++++++++++++-
> > .../selftests/bpf/prog_tests/arena_import.c | 451 ++++++++++++++++++++
> > tools/testing/selftests/bpf/progs/arena_import.c | 93 +++++
> > 3 files changed, 994 insertions(+), 11 deletions(-)
> > create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_import.c
> > create mode 100644 tools/testing/selftests/bpf/progs/arena_import.c
>
prev parent reply other threads:[~2026-11-21 21:49 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 10:28 Andrea Righi
2026-11-17 10:28 ` [PATCH 1/3] bpf: Allow importing memfd " Andrea Righi
2026-11-17 11:14 ` bot+bpf-ci
2026-11-17 10:28 ` [PATCH 2/3] bpf: Allow importing device-dax " Andrea Righi
2026-11-17 11:02 ` bot+bpf-ci
2026-11-17 10:28 ` [PATCH 3/3] selftests/bpf: Test importing " Andrea Righi
2026-11-20 13:45 ` [PATCH v2 bpf-next 0/3] bpf: Import external " Vineeth Pillai
2026-11-21 21:49 ` Andrea Righi [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=awIR5y9x9sK8j7dB@gpd4 \
--to=arighi@nvidia.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=balbirs@nvidia.com \
--cc=bpf@vger.kernel.org \
--cc=changwoo@igalia.com \
--cc=daniel@iogearbox.net \
--cc=eddyz87@gmail.com \
--cc=emil@etsalapatis.com \
--cc=himadrics@protonmail.com \
--cc=ihor.solodrai@linux.dev \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=joshdon@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=ltrager@nvidia.com \
--cc=martin.lau@linux.dev \
--cc=memxor@gmail.com \
--cc=royfru44@gmail.com \
--cc=sched-ext@lists.linux.dev \
--cc=skolothumtho@nvidia.com \
--cc=song@kernel.org \
--cc=tj@kernel.org \
--cc=vineethrp@google.com \
--cc=void@manifault.com \
--cc=yepeilin@google.com \
--cc=yonghong.song@linux.dev \
--cc=zhiw@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®