[PATCH v12 00/12] PCI/CXL: Add CXL reset support for Type 2 devices

[Srirangan Madhavan <smadhavan@nvidia.com>]

Hi folks!

This series adds CXL Reset support for CXL Type 2 devices through the
existing PCI reset_method ABI. The reset sequence follows the CXL 4.0
specification [1], including CXL.cache disable, optional cache
writeback, CXL Reset Memory Clear, ResetComplete polling, and ResetError
reporting.

The userspace ABI is the existing PCI reset interface:

    /sys/bus/pci/devices/.../reset_method
    /sys/bus/pci/devices/.../reset

Userspace can select "cxl_reset" in reset_method and then trigger reset
through the existing reset attribute.

CXL Reset is exposed conservatively. The target must be a
single-function CXL Type 2 device with no enabled VFs, CXL.cache and
CXL.mem must be enabled, and CXL Reset Memory Clear must be supported.
The kernel must also have cached HDM state sufficient to validate affected
ranges and restore decoder programming.

For reset through the sysfs reset attribute, the generic PCI path calls
the driver's reset_prepare() callback when provided and disables PCI I/O,
memory decoding, and bus mastering. The CXL reset path then temporarily
reserves each affected HDM range exclusively for the duration of the
reset, synchronizes CPU and device caches, and blocks IOMMU translations
where applicable. It restores HDM decoder state and CXL Device DVSEC
Cache and Mem Enable controls before PCI operation resumes. The path does
not offline memory or revoke mappings; it relies on CXL.mem users retaining
conflicting iomem resources while access remains active.

Following Dan's v6 feedback, this uses the existing PCI reset_method
interface instead of adding a memdev-specific sysfs ABI.

v12 changes from v11 [2]:
- Reject cxl_reset and cxl_bus reset when the PF has enabled VFs,
  including VFs enabled after reset-method probing.
- Require CXL Reset Memory Clear support and request memory clearing so
  the generic PCI reset does not preserve tenant memory.
- Cache the CXL Device DVSEC control register during early PCI discovery
  and restore its CXL.cache and CXL.mem enable bits after CXL Reset and
  CXL bus reset.
- Initialize the early PCI HDM cache only from pci_bus_add_device(),
  removing the later CXL core retry and its lifecycle locking.
- Reapply and verify a cached HDM decoder lock after decoder restoration
  and commit, while leaving surviving locked committed state unchanged.
- Convert temporary early HDM-cache ownership to scoped cleanup.
- Document the reset-safety contract, including the iomem ownership
  requirement and the limits around memory offlining and mapping
  revocation.

Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI
  reset methods.

- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is
  the architectural reset mechanism for those protocols.

- The PCI reset_method ABI lets userspace select CXL Reset before
  falling back to broader bus reset methods.

Change Description:
-------------------

Patch 1: cxl: Move HDM decoder programming helpers
- Introduce a decoder settings snapshot collected under the DPA lock.
- Move low-level settings-based HDM programming into resource.c.

Patch 2: cxl: Make HDM commit helpers available to reset code
- Build generic HDM resource helpers under CONFIG_CXL_RESET.
- Export the lock and commit helpers needed when cxl_core is modular.

Patch 3: cxl: Share HDM decoder decode logic
- Share HDM decoder decode and validation between normal CXL core
  enumeration and early PCI HDM cache setup.
- Decode zero-size committed decoder state without imposing caller policy.

Patch 4: cxl: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.

Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM and CXL Device DVSEC control state during PCI
  enumeration.
- Cache the HDM register locator as BAR-relative metadata and preserve raw
  decoder state for reset restore.
- Keep early cache initialization in the PCI bus-add path only.

Patch 6: cxl: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate
  CXL Reset with Memory Clear, and wait for completion.

Patch 7: cxl: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges, reject busy ranges, and perform CPU
  cache invalidation when a runtime backend is available.

Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices
- Conservatively reject CXL Reset for multifunction devices and PFs with
  enabled VFs.

Patch 9: cxl: Restore CXL state after PCI reset
- Restore cached global and per-decoder HDM state, decoder locks, and CXL
  Device DVSEC protocol enables while the target IOMMU block is active.

Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset-method table ahead of FLR.

Patch 11: Documentation/ABI: Document CXL Reset PCI reset method
- Document the reset method and its safety contract.

Patch 12: PCI/CXL: Restore CXL state after CXL bus reset
- Restore cached CXL HDM and Device DVSEC state after cxl_bus reset.
- Reject multifunction devices and PFs with enabled VFs.

The CPU cache invalidation step depends on
cpu_cache_invalidate_memregion() support for the affected address ranges.
If no runtime provider is available, the kernel emits a warning and
continues after the affected HDM ranges have been reserved.

Example:

    echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
    echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset

Testing:
- Applied all 12 patches sequentially to a fresh v7.3-rc1 source tree
  with CONFIG_CXL_BUS=m and CONFIG_CXL_RESET=y. Every intermediate patch
  passed `make vmlinux modules` and `make M=tools/testing/cxl/`.
- Built and booted the series on an arm64 NVIDIA NVL72 system and
  successfully exercised CXL Reset through the PCI reset sysfs ABI.
- Exercised cxl_bus reset with CXL state restoration on all four GPUs in
  an NVL72 system; all four resets completed successfully.
- Earlier revision testing ran 100 iterations of cxl_reset.

References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/20260902072804.665639-1-smadhavan@nvidia.com/
[3] https://lore.kernel.org/linux-cxl/20260804192958.1823952-1-smadhavan@nvidia.com/
[4] https://lore.kernel.org/linux-cxl/20260709010304.680422-1-smadhavan@nvidia.com/
[5] https://lore.kernel.org/linux-cxl/20260703220508.546528-1-smadhavan@nvidia.com/

Srirangan Madhavan (12):
  cxl: Move HDM decoder programming helpers
  cxl: Make HDM commit helpers available to reset code
  cxl: Share HDM decoder decode logic
  cxl: Cache decoder settings on PCI devices
  cxl: Cache endpoint decoder settings during PCI enumeration
  cxl: Add CXL Device Reset helper
  cxl: Validate HDM ranges before CXL reset
  PCI/CXL: Reject CXL Reset on multifunction devices
  cxl: Restore CXL state after PCI reset
  PCI/CXL: Expose CXL Reset as a PCI reset method
  Documentation/ABI: Document CXL Reset PCI reset method
  PCI/CXL: Restore CXL state after CXL bus reset

 Documentation/ABI/testing/sysfs-bus-pci |   25 +
 drivers/cxl/Kconfig                     |    4 +
 drivers/cxl/core/Makefile               |    2 +-
 drivers/cxl/core/core.h                 |    8 +
 drivers/cxl/core/hdm.c                  |  301 +++----
 drivers/cxl/core/pci.c                  |   22 +-
 drivers/cxl/core/port.c                 |   19 +
 drivers/cxl/core/regs.c                 |    4 +
 drivers/cxl/core/resource.c             | 1400 +++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h                       |   41 -
 drivers/pci/bus.c                       |    2 +
 drivers/pci/pci.c                       |    8 +
 drivers/pci/probe.c                     |    2 +
 include/cxl/cxl.h                       |  120 +-
 include/linux/pci.h                     |    8 +-
 include/uapi/linux/pci_regs.h           |   14 +
 tools/testing/cxl/Kbuild                |    1 -
 17 files changed, 1769 insertions(+), 212 deletions(-)
 create mode 100644 drivers/cxl/core/resource.c


base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
-- 
2.43.0