Re: [RFC PATCH 0/3] iommu/arm-smmu-v3: Add support for hitless replace of CD entries

[Robin Murphy <robin.murphy@arm.com>]

On 25/09/2026 2:13 am, Samiullah Khawaja wrote:
> This patch series adds support for replacing IOMMU domains with S1 page
> tables hitlessly, that is without disrupting the ongoing DMAs. This is
> one of the dependencies of the Liveupdate IOMMU support for Arm SMMUv3.
> 
> S1 domains are programmed by CD entries in the CD table. During a new
> domain attach these are updated using arm_smmu_write_cd_entry(). Since
> ASID and TTB0 are in different 64-bit words, arm_smmu_write_cd_entry()
> considers it a non-hitless update and sets V=0. This means such an
> update is disruptive and would generate translation faults if there are
> ongoing DMAs.
> 
> Replace the CD entries by using an unused temporary ASID in the
> following sequence to allow hitless replacement of CD entry,
> 
> - Update the CD entry with temporary ASID.
> - Update the CD entry with the target TTB0.
> - Invalidate the temporary ASID.
> - Update the CD entry with target ASID.
> - Invalidate and release the temporary ASID.

Why bother transitioning through a 3rd ASID? The old and new pagetables 
must contain the same translations anyway, so why not just: switch ASID, 
switch TTB, invalidate old ASID?

It's perhaps worth noting that for correctness either way this also very 
much depends on no concurrent map/unmap operations being able to happen 
in either hwpt while the switch is ongoing, but then I guess it already 
depends on the caller guaranteeing that the hwpts hold identical 
translations, which we can't realistically verify at the driver level 
either...

Thanks,
Robin.

> The series adds a vfio selftest that triggers a continuous DMA and does
> the hwpt replace while the DMA is ongoing. Since domain replacement is a
> race with the translation requests, the test does it multiple times to
> capture the small window where IOMMU faults can occur.
> 
> Without this change we can observe the SMMU faults in the kernel logs
> and with it the SMMU faults do not occur.
> 
> This is only tested on qemu with the emulated SMMUv3 and an out of tree
> NVMe vfio selftest driver, not on real hardware. Following are not
> covered and can be done later,
> 
> - S2 domains have the same problem, S2VMID is in STE qword 2 and S2TTB is
>    in qword 3. These need the same handling with a temporary VMID.
> - ATS is not tested, the test environment runs with pci=noats.
> 
> Testing:
> 
> Without Fix:
> 
>    TAP version 13
>    1..2
>    # Starting 2 tests from 2 test cases.
>    #  RUN           vfio_iommufd_replace_hwpt_test.domain_replace.memcpy ...
>    [   33.267749] vfio-pci 0000:00:0a.0: enabling device (0000 -> 0002)
>    [   33.270709] vfio-pci 0000:00:0a.0: resetting
>    [   33.385682] vfio-pci 0000:00:0a.0: reset done
>    [   37.128021] arm-smmu-v3 9050000.smmuv3: event 0x0a received:
>    [   37.131885] arm-smmu-v3 9050000.smmuv3:      0x000000500000000a
>    [   37.134585] arm-smmu-v3 9050000.smmuv3:      0x0000000000000000
>    [   37.136961] arm-smmu-v3 9050000.smmuv3:      0x0000000000000000
>    [   37.139631] arm-smmu-v3 9050000.smmuv3:      0x0000000000000000
>    [   37.142188] arm-smmu-v3 9050000.smmuv3: event: C_BAD_CD client: 0000:00:0a.0 sid: 0x50 ssid: 0x0
> 
>    ...
>    ...
> 
> With Fix:
> 
>    TAP version 13
>    1..2
>    # Starting 2 tests from 2 test cases.
>    #  RUN           vfio_iommufd_replace_hwpt_test.domain_replace.memcpy ...
>    [   33.418253] vfio-pci 0000:00:0a.0: enabling device (0000 -> 0002)
>    [   33.421218] vfio-pci 0000:00:0a.0: resetting
>    [   33.539750] vfio-pci 0000:00:0a.0: reset done
>    #            OK  vfio_iommufd_replace_hwpt_test.domain_replace.memcpy
>    ok 1 vfio_iommufd_replace_hwpt_test.domain_replace.memcpy
>    #  RUN           vfio_iommufd_replace_hwpt_test.noreplace.memcpy ...
>    [   88.416453] vfio-pci 0000:00:0a.0: resetting
>    [   88.531989] vfio-pci 0000:00:0a.0: reset done
>    #            OK  vfio_iommufd_replace_hwpt_test.noreplace.memcpy
>    ok 2 vfio_iommufd_replace_hwpt_test.noreplace.memcpy
> 
> Looking forward to your feedback on this.
> 
> Dependencies:
> The patch series includes a patch from the vfio selftest multi iommu
> series as it is required for the hwpt_replace vfio selftest. It is part
> of this series since the original code doesn't apply cleanly on the
> base.
> https://lore.kernel.org/all/20260505221518.619123-1-skhawaja@google.com/
> 
> Related Work:
> Liveupdate IOMMU patch series:
> https://lore.kernel.org/all/20260921004834.2601285-1-skhawaja@google.com/
> Liveupdate IOMMU feature patchset breakdown:
> https://docs.google.com/document/d/1enDn-uPE9U77U-xHEnzn6HHGKiePSAtMIP8EDU3NO0M
> 
> Samiullah Khawaja (3):
>    vfio: selftests: Add support of creating iommus from iommufd
>    vfio: selftests: Add iommufd hwpt replace test
>    iommu/arm-smmu-v3: Add support for hitless replace of S1 domains
> 
>   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  66 ++++++-
>   tools/testing/selftests/vfio/Makefile         |   1 +
>   .../vfio/lib/include/libvfio/iommu.h          |   2 +
>   .../lib/include/libvfio/vfio_pci_device.h     |   2 +
>   tools/testing/selftests/vfio/lib/iommu.c      |  60 +++++-
>   .../selftests/vfio/lib/vfio_pci_device.c      |  25 ++-
>   .../vfio/vfio_iommufd_hwpt_replace_test.c     | 171 ++++++++++++++++++
>   7 files changed, 316 insertions(+), 11 deletions(-)
>   create mode 100644 tools/testing/selftests/vfio/vfio_iommufd_hwpt_replace_test.c
> 
> 
> base-commit: 3d7783543c2646af69ad65825e810060494bea21