From: Vipin Sharma <vipinsh@google.com>
To: David Matlack <dmatlack@google.com>
Cc: kexec@lists.infradead.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org, kvm@vger.kernel.org,
linux-mm@kvack.org, linux-kselftest@vger.kernel.org,
linux-pci@vger.kernel.org, ajayachandra@nvidia.com,
alex@shazbot.org, amastro@fb.com, ankita@nvidia.com,
apopple@nvidia.com, bhelgaas@google.com, chrisl@kernel.org,
christian.koenig@amd.com, corbet@lwn.net, graf@amazon.com,
jacob.pan@linux.microsoft.com, jgg@nvidia.com, jgg@ziepe.ca,
jrhilke@google.com, julianr@linux.ibm.com, kees@kernel.org,
kevin.tian@intel.com, leon@kernel.org, leonro@nvidia.com,
lukas@wunner.de, mattev@meta.com, michal.winiarski@intel.com,
parav@nvidia.com, pasha.tatashin@soleen.com, praan@google.com,
pratyush@kernel.org, rananta@google.com, rdunlap@infradead.org,
rientjes@google.com, rodrigo.vivi@intel.com, rppt@kernel.org,
saeedm@nvidia.com, schnelle@linux.ibm.com,
skhan@linuxfoundation.org, skhawaja@google.com,
vivek.kasireddy@intel.com, witu@nvidia.com,
yanjun.zhu@linux.dev, yi.l.liu@intel.com
Subject: Re: [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update
Date: Tue, 17 Nov 2026 22:44:26 -0800 [thread overview]
Message-ID: <20261117220059.GA2250529.vipinsh@google.com> (raw)
In-Reply-To: <auzftbPFO_6XBYVw@google.com>
On Thu, Nov 05, 2026 at 07:57:41PM +0000, David Matlack wrote:
> On 2026-11-02 07:36 AM, Vipin Sharma wrote:
> > + * DMA Memory Ordering Contract
> > + * ============================
> > + *
> > + * Because a preserved noiommu device continues performing DMA using physical
> > + * addresses across kexec, any memory buffer targeted by device DMA must remain
> > + * at the same physical address in both kernels and must not be freed to the
> > + * page allocator while the device can still access it:
> > + *
> > + * * When using a LUO-preserved ``memfd``, userspace **must** preserve the
> > + * ``memfd`` in the **same** LUO session as the VFIO device and **before**
> > + * preserving the VFIO device. LUO finishes files within a session in reverse
> > + * preservation order (LIFO), guaranteeing that if the device is not taken
> > + * over in the new kernel, VFIO's ``finish()`` callback stops bus mastering
> > + * and resets the device before the ``memfd``'s ``finish()`` callback releases
> > + * its folios. Cross-session ordering is the responsibility of userspace.
> > + *
> > + * * After retrieving a preserved ``memfd`` in the new kernel, its folios are
> > + * ordinary unpinned shmem page cache until pinned again; userspace must keep
> > + * the ``memfd`` alive and re-pin/map it before finishing the session, and
> > + * must not rely on a ``memfd`` whose ``LIVEUPDATE_SESSION_RETRIEVE_FD`` call
> > + * failed (failed ``memfd`` retrieval frees its folios immediately).
>
> Most of this section documents LUO and memfd behavior (finish ordering
> within and across sessions, what happens to memfd folios after
> retrieval), which belongs in the LUO and memfd documentation rather than
> here. The VFIO documentation should just state what VFIO requires of
> userspace and what happens if userspace violates it. Something like:
>
> DMA Memory
> ==========
>
> A preserved device keeps performing DMA across kexec to the physical
> addresses it was programmed with before kexec. VFIO does not know
> which memory the device accesses and does not preserve it. Userspace
> must ensure that all memory the device can access stays allocated at
> the same physical address in both kernels until the device is stopped,
> either by userspace after taking it over or by VFIO when the session
> is finished. If userspace violates this, the device can corrupt memory
> that has been freed and reused by either kernel. VFIO cannot detect
> this.
>
> VFIO doesn't stop the device when the session is finished until
> PATCH 11, so either add this section there or drop that clause here and
> add it in PATCH 11.
>
Sounds good. I will remove memfd from here and mention thing what VFIO
expects.
> > + cfg_size = pdev->cfg_size;
> > +
> > + if (cfg_size != PCI_CFG_SPACE_SIZE && cfg_size != PCI_CFG_SPACE_EXP_SIZE)
> > + return -EINVAL;
>
> pdev->cfg_size isn't limited to these two values. PCI quirks set other
> sizes, e.g. quirk_citrine() (0xA0) and quirk_nfp6000() (0x600), and
> vfio-pci handles those devices fine since vconfig is just pdev->cfg_size
> bytes. This check only seems to exist to bound the variable-length
> layout of struct vfio_pci_core_device_ser, which goes away with the
> changes suggested below. Please drop it.
>
> > + total_size = size_add(sizeof(*ser),
> > + size_add(size_mul(2, cfg_size), saved_size));
> > + if (total_size > U32_MAX)
> > + return -EOVERFLOW;
> > +
> > + ser = kho_alloc_preserve(total_size);
> > + if (IS_ERR(ser))
> > + return PTR_ERR(ser);
> > +
> > + if (saved_size) {
> > + void *dst = ser->data + 2 * (size_t)cfg_size;
> > +
> > + err = pci_liveupdate_serialize_saved_state(vdev->pci_saved_state,
> > + dst, saved_size);
> > + if (err) {
> > + kho_unpreserve_free(ser);
> > + return err;
> > + }
> > + }
>
> Please don't embed struct pci_saved_state_ser in VFIO's serialized
> struct. Variable-length serialized structs should be preserved in their
> own allocation, and VFIO should just store the physical address (0 if
> there is no saved state). That way the PCI core owns the size, flags and
> validation of its own ABI, either struct can grow without changing the
> layout of the other, and the offset math here and in PATCH 11 goes away.
> struct pci_ser already references its devices array the same way.
>
> I plan to structure the PCI core helpers for serializing and
> deserializing struct pci_saved_state to require this. The PCI core will
> do the allocation and just return a pointer to the serialized saved
> state. But I just wanted to call it out here.
>
Sounds good, v7 will store physical address returned by new PCI helpers.
I will base my series on top of your PCI saved state series latest
version. Current one is v2 at
https://lore.kernel.org/all/20261106004218.604799-1-dmatlack@google.com/
> > + ser->flags = vdev->reset_works ? VFIO_PCI_SER_RESET_WORKS : 0;
>
> reset_works shouldn't be serialized. Alex's series replaces
> vdev->reset_works with pci_reset_supported() [1], which only looks at
> pdev->reset_methods. Those are set up at enumeration without resetting
> the device, so the next kernel can answer the question itself. And the
> next kernel's answer is the one that matters, since it's the kernel that
> will reset the device.
>
Thanks for bringing it up. I will send that specific patch separately as
discussed in
https://lore.kernel.org/kvm/20261118000511.GC2250529.vipinsh@google.com/
> Please base on that series (or cherry-pick that patch) and drop
> VFIO_PCI_SER_RESET_WORKS, along with the incoming reset_works copy and
> the reset_works argument to vfio_pci_core_liveupdate_reset() in PATCH
> 11, and the copy in vfio_pci_liveupdate_take_over() in PATCH 14. flags
> then has no defined bits, but please keep it (and keep rejecting unknown
> bits) for future features, see below.
>
> > + if (!vdev->vconfig || !vdev->pci_config_map ||
> > + ser->cfg_size != pdev->cfg_size)
> > + return -EINVAL;
>
> ser->cfg_size was set from pdev->cfg_size in preserve(), and
> pdev->cfg_size doesn't change after enumeration, so the cfg_size check
> can't fail.
Yeah, this is redundant.
>
> > + memcpy(ser->data, vdev->vconfig, ser->cfg_size);
> > + memcpy(ser->data + ser->cfg_size, vdev->pci_config_map, ser->cfg_size);
>
> pci_config_map doesn't need to be preserved. PATCH 14 already rebuilds
> it with vfio_config_init() in the next kernel and only uses the
> preserved copy to check that the two are identical.
>
> I assume the intent is to detect changes to vfio-pci's config emulation
> between the two kernels, but the map isn't a good proxy for that. It
> changes for differences that don't matter (e.g. hiding a capability or
> learning the length of a new extended capability), which would fail
> take-over and lose the Live Update on exactly the kernel upgrades Live
> Update is meant for. And it doesn't change for differences that do
> matter, like which bits are virtualized or writable within a capability.
I think it does matter because reads and writes of unitialized
capabilities in VFIO PCI config space goes to hardware directly. So, in
the new kernel if pci_config_map changes it from PCI_CAP_ID_INVALID to
others then those reads/writes can go to vconfig which will contain
stale unitialized data from the old kernel's heap.
This also changes behavior under a running user. Writes that reached
hardware before Live Update can be silently dropped or redirected to
vconfig. See the comment above struct perm_bits unassigned_perms{}
in vfio_pci_config.c
I do agree current patch is not protecting against bits virt/write
changes inside a capability, which is also important.
>
> Preserving vconfig makes vfio-pci's config emulation part of the
> contract between kernels either way. I think changes to the config
> emulation that affect how a preserved vconfig is interpreted need to be
> tied to a userspace opt-in, recorded as a flag in struct
> vfio_pci_core_device_ser. A kernel that doesn't understand the flag then
> rejects the preserved device, and userspace shouldn't opt in while it
> can still roll back to a kernel without the new emulation.
How would a userspace opt-in? New kernel can change a capability visiblity
or virt and write permission of the bits inside a capability. Kernel
cannot know if userspace used the capability or not. Creating a VFIO
device in the new kernel by default results in a opt-in and currently
there is no way for userspace to opt-out unless we add a new
functionality where userspace can disable/enable features something
similar to KVM capabilities.
For now, I think we don't need to worry about it. As this version will
be a base and if any serialized data with flags set received in this
kernel will lead to erroring out restore.
In the next version, I will not preserve pci_config_map. Hopefully,
anyone changing VFIO code doesn't forget to update flags if they change
capability or permission bits (I will write comments at appropriate
places.)
If we want to be very careful, we can do two things in this series:
1. Create a per bit valid mask for the vconfig and compare this between
new and old kernel during restore. In the new kernel, VFIO then takes
each bit from the preserved copy only where the mask says it was
valid, and keeps its own freshly initialized value elsewhere.
2. Create a hash or store virt and writable bits in config space.
Compare it with the new kernel.
Both of the above combined will safeguard against forgotten flag update
in the code.
We don't have to do it but it safeguards against mistakes. Let me know
what you prefer.
>
> > + pci_read_config_word(pdev, PCI_COMMAND, &cmd);
> > + vdev->liveupdate_intx_disabled = !!(cmd & PCI_COMMAND_INTX_DISABLE);
> > + if (!vdev->liveupdate_intx_disabled)
> > + pci_intx(pdev, 0);
>
> This doesn't mask INTx on devices with pdev->broken_intx_masking set,
> either because PCI_COMMAND_INTX_DISABLE isn't writable or because a
> quirk marks INTx masking as broken. vfio-pci normally masks INTx for
> those devices at the interrupt controller, but nothing does that across
> kexec, so the device can still assert INTx in the next kernel before
> anything is ready to handle it. If the line is shared, that can get the
> line disabled as spurious and break the other device.
>
> Please reject those devices in preserve() if they have an INTx pin and
> !pci_intx_mask_supported(pdev), and add that to the Restrictions in the
> DOC. Check the PCI core flag rather than vdev->pci_2_3, since
> vfio_pci.nointxmask=1 clears pci_2_3 even when INTx masking works.
Agree, I will reject those devices in preserve().
>
> > +struct vfio_pci_core_device_ser {
> > + u32 domain;
> > + u16 bdf;
> > + u16 cfg_size;
> > + u32 size;
> > + u32 flags;
> > + u32 saved_state_size;
> > + u32 reserved;
> > + u8 data[];
> > +} __packed;
>
> Once config_map and the saved state are gone, vconfig is the only
> variable-length part left. Please either make it a fixed-length array
> sized for the largest config space (PCI_CFG_SPACE_EXP_SIZE), or preserve
> it in a separate allocation like the saved state. Either way this struct
> becomes fixed-size and data[] goes away.
>
> cfg_size should stay, to record how much of vconfig is valid. The next
> kernel computes pdev->cfg_size on its own and could come up with a
> different value, so it needs to check that ser->cfg_size matches (and
> fits in the vconfig buffer) before using the preserved vconfig. Doing
> that once in retrieve() should be enough.
>
> I understand this will waste some memory today, since each KHO
> allocation must be a multiple of a PAGE_SIZE. But that is a KHO problem
> that can be fixed later.
>
I agree. I will change data to fixed size for vconfig. We will not need
pci_config_map, and pci saved state array.
I am planning to add two flags required and optional similar to your PCI
serialization struct.
Thanks again for the review.
next prev parent reply other threads:[~2026-11-18 6:44 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-02 15:36 [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 01/27] vfio/pci: Factor out the reset logic in VFIO PCI device close path Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 02/27] vfio: Export various helpers from VFIO Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 03/27] liveupdate: Export symbols needed by modules Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 04/27] vfio/pci: Register a file handler with Live Update Orchestrator Vipin Sharma
2026-11-06 22:29 ` Samiullah Khawaja
2026-11-10 19:02 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 05/27] PCI: liveupdate: Add helpers to preserve struct pci_saved_state Vipin Sharma
2026-11-02 23:06 ` Bjorn Helgaas
2026-11-03 0:48 ` David Matlack
2026-11-04 22:55 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 06/27] vfio: Add helpers to identify device files that can be preserved Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update Vipin Sharma
2026-11-05 19:12 ` David Matlack
2026-11-17 21:59 ` Vipin Sharma
2026-11-05 19:57 ` David Matlack
2026-11-18 6:44 ` Vipin Sharma [this message]
2026-11-06 22:35 ` Samiullah Khawaja
2026-11-18 6:55 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 08/27] vfio: Introduce vfio_find_device() helper Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 09/27] vfio: Refactor vfio_device_fops_cdev_open() Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 10/27] vfio: Add API to open cdev device for Live Update restore Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 11/27] vfio/pci: Retrieve preserved device files after Live Update Vipin Sharma
2026-11-06 17:19 ` David Matlack
2026-11-18 7:51 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 12/27] vfio: Enforce preserved devices are retrieved via LIVEUPDATE_SESSION_RETRIEVE_FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 13/27] vfio/pci: Protect incoming preserved devices until they finish Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 14/27] vfio/pci: Take over preserved devices without resetting them Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 15/27] docs: liveupdate: Add documentation for VFIO PCI Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 16/27] vfio: selftests: Build liveupdate library in VFIO selftests Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 17/27] vfio: selftests: Add vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 18/27] vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 19/27] vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 20/27] vfio: selftests: Add iommufd noiommu mode Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 21/27] vfio: selftests: Add helpers to map DMA memory at IOVA == PA in " Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 22/27] vfio: selftests: Add helpers for Live Update DMA memory and sysfs state Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 23/27] vfio: selftests: Add an attach() driver op Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 24/27] vfio: selftests: Test preserving noiommu devices in vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 25/27] vfio: selftests: Add vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 26/27] vfio: selftests: Verify that opening VFIO device fails during Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 27/27] vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-06 22:20 ` [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) David Matlack
2026-11-10 18:45 ` Vipin Sharma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117220059.GA2250529.vipinsh@google.com \
--to=vipinsh@google.com \
--cc=ajayachandra@nvidia.com \
--cc=alex@shazbot.org \
--cc=amastro@fb.com \
--cc=ankita@nvidia.com \
--cc=apopple@nvidia.com \
--cc=bhelgaas@google.com \
--cc=chrisl@kernel.org \
--cc=christian.koenig@amd.com \
--cc=corbet@lwn.net \
--cc=dmatlack@google.com \
--cc=graf@amazon.com \
--cc=jacob.pan@linux.microsoft.com \
--cc=jgg@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=jrhilke@google.com \
--cc=julianr@linux.ibm.com \
--cc=kees@kernel.org \
--cc=kevin.tian@intel.com \
--cc=kexec@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=leon@kernel.org \
--cc=leonro@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-pci@vger.kernel.org \
--cc=lukas@wunner.de \
--cc=mattev@meta.com \
--cc=michal.winiarski@intel.com \
--cc=parav@nvidia.com \
--cc=pasha.tatashin@soleen.com \
--cc=praan@google.com \
--cc=pratyush@kernel.org \
--cc=rananta@google.com \
--cc=rdunlap@infradead.org \
--cc=rientjes@google.com \
--cc=rodrigo.vivi@intel.com \
--cc=rppt@kernel.org \
--cc=saeedm@nvidia.com \
--cc=schnelle@linux.ibm.com \
--cc=skhan@linuxfoundation.org \
--cc=skhawaja@google.com \
--cc=vivek.kasireddy@intel.com \
--cc=witu@nvidia.com \
--cc=yanjun.zhu@linux.dev \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®