From: Srirangan Madhavan <smadhavan@nvidia.com>
To: Alison Schofield <alison.schofield@intel.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
Dave Jiang <dave.jiang@intel.com>,
Davidlohr Bueso <dave@stgolabs.net>,
Ira Weiny <iweiny@kernel.org>,
Jonathan Cameron <jic23@kernel.org>,
Vishal Verma <vishal.l.verma@intel.com>,
linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org, Li Ming <ming.li@zohomail.com>,
Joshua Machauer <machaujo@amazon.de>,
Alex Williamson <alex.williamson@redhat.com>,
vsethi@nvidia.com, alwilliamson@nvidia.com,
Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
Vishal Aslot <vaslot@nvidia.com>,
Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
Richard Cheng <icheng@nvidia.com>,
linux-tegra@vger.kernel.org
Subject: Re: [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices
Date: Fri, 20 Nov 2026 00:16:03 -0800 [thread overview]
Message-ID: <1c5b291c-25f1-4936-8601-0660e3627b8e@nvidia.com> (raw)
In-Reply-To: <avyOoQjiD7Uh_3rE@aschofie-mobl2.lan>
On 11/17/26 9:28 AM, Alison Schofield wrote:
>> Hi folks,
>>
>> This series adds CXL Reset for Type 2 devices through the existing PCI
>> reset_method and reset sysfs attributes. It disables CXL.cache, performs
>> supported cache writeback, initiates CXL Reset with Memory Clear, and
>> restores cached HDM decoder and CXL Device DVSEC state before PCI use
>> resumes. Active HDM ranges are reserved and CPU caches invalidated before
>> and after reset. The method remains limited to eligible single-function
>> devices with both CXL.cache and CXL.mem enabled. CXL bus reset restores
>> the same cached state and flushes the host and device caches around the
>> reset.
>>
>> The reset sequence follows the CXL 4.0 specification [4], including
>> CXL.cache disable, optional cache writeback, CXL Reset Memory Clear,
>> ResetComplete polling, and ResetError reporting. Sections 9.7.2 and 9.7.3
>> define the completion timeout encodings and reset access restrictions.
>>
>> The userspace ABI is the existing PCI reset interface:
>>
>> /sys/bus/pci/devices/.../reset_method
>> /sys/bus/pci/devices/.../reset
>>
>> Userspace selects "cxl_reset" in reset_method and triggers the reset
>> through reset. Following Dan's v6 feedback, this uses the PCI interface
>> instead of adding a memdev-specific sysfs ABI.
>>
>> For sysfs reset, the generic PCI path invokes the driver's reset_prepare()
>> callback when provided and disables PCI I/O, memory decoding, and bus
>> mastering. The CXL path reserves affected HDM ranges exclusively and
>> blocks IOMMU translations where applicable. It does not offline memory
>> or revoke mappings: userspace is expected to offline CXL.mem users before
>> reset. The reservation enforces this for claimed ranges by failing the
>> reset while a conflicting resource is claimed [9].
>>
>> The global region lock is held from snapshot through restoration. Memory
>> Clear completion is awaited for the timeout the device advertises, up to
>> 100 seconds, so the lock can be held for that long plus cache flushes.
> Hi Srirangan,
>
> Thanks for all the followups on review feedback. I think you answer some
> of my question above, 'For sysfs reset...' Could you explain the intended
> use cases for this reset method? Like why and when would userspace issue
> a CXL reset. In particular, is reset intended to be available before endpoint
> and region discovery have completed?
>
Hi Alison,
I am looking at these types of use cases: a VMM handing a Type 2
accelerator from one guest to another (the vfio-cxl series), where we
need the device's memory cleared and its CXL state reset and restored,
which FLR doesn't give us. Or an admin or a tool resetting a hung or
errored device through reset_method and reset, without taking down the
whole link. In those cases we need the CXL reset. The CXL 4.0 spec says
FLRs do not affect the CXL.cache and CXL.mem protocols, and that errors
in CXL.cache and CXL.mem logic cannot be recovered by an FLR.
My understanding is that it is intended to be available at enumeration,
before endpoint and region discovery. I can fix the gaps in the address
caching mechanism to make it work.
cxl_reset is listed at device add, like the other reset methods. It can
execute once the HDM cache exists, which is right after enumeration.
> I see how the series saves state for restoration after reset. I don't see
> what ensures discovery has progressed far enough for reset to be safe.
>
Nothing as of v17. Normalized Addressing is recorded after the HDM cache is
filled, so an early reset can treat a range that isn't a HPA as one.
What I'm thinking of for v18 is a check that doesn't wait for the flag:
also reject an enabled decoder whose range isn't inside a platform CXL
window (the "CXL Window" resources cxl_acpi registers). Until cxl_acpi
has registered them nothing matches, so the reset is refused. The flag
check stays as it is.
Both cxl_reset and cxl_bus reach it through cxl_reset_cache_prepare():
cxl_reset_cache_prepare()
cxl_hdm_quiesce_prepare()
cxl_hdm_ranges_collect() per enabled, non-zero-size decoder:
NORMALIZED flag set -> -EOPNOTSUPP (today)
range not in a CXL window -> -EOPNOTSUPP (new)
cxl_hdm_range_add()
cxl_hdm_ranges_request() reserve the ranges
cxl_hdm_ranges_invalidate() flush host caches
So a reset on a range we can't validate is refused before anything is
reserved or flushed. Does this sounds like a good idea?
> I also see DaveJ's cxl_bus deadlock report. My concern is related but
> separate. Even without a deadlock, a reset could hold the global region lock
> for up to 100 seconds plus cache maintenance and restoration, blocking
> discovery and region assembly on unrelated devices. Understanding the intended
> usage would help assess whether that disruption is acceptable. Note that my
> simple concern is a user issuing those cxl_reset's on repeat and blocking CXL
> bringup. Yeah, that's a bad user, but is there any protection against that?
>
> Adding specific questions on patches 11, 13, and 16
>
I rechecked the code for avoiding the global lock. For v18, I think I can reduce
both resets to take the lock briefly twice instead:
prepare: lock -> snapshot, reserve ranges, set reset_pending -> unlock
do reset: no region lock
restore: lock -> re-read the cache, restore, clear reset_pending -> unlock
While reset_pending is set, cxl_decoder_commit() on that device returns
-EBUSY.
This way, the long lock is not held and remaining is scoped to the device being
reset: only commits of that device's decoders return -EBUSY during the reset, so
a region that includes it can't commit until the reset is done. The two brief
windows are still global. I couldn't find a norrower option.
I posted the same proposal with some more details in reply to DaveJ
https://lore.kernel.org/linux-cxl/20261105185917.2667656-1-smadhavan@nvidia.com/T/#mcea903d781c011fe22abf76ddd16d4f78a3c14f3
Please let me know if this could work.
> -- Alison
--
Regards,
Srirangan
prev parent reply other threads:[~2026-11-20 8:16 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 18:58 Srirangan Madhavan
2026-11-05 18:58 ` [PATCH v17 01/19] cxl: Clarify decoder interleave limit Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 02/19] cxl: Share CXL port upstream PCI device lookup Srirangan Madhavan
2026-11-10 13:36 ` Li Ming
2026-11-13 15:50 ` Dave Jiang
2026-11-15 3:25 ` Li Ming
2026-11-05 18:59 ` [PATCH v17 03/19] cxl: Move decoder declarations to shared header Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 04/19] cxl: Embed decoder configuration in a standalone structure Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 05/19] cxl: Introduce endpoint HDM decoder settings Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 06/19] cxl: Move HDM decoder helpers to built-in code Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 07/19] cxl: Share HDM decoder register unpacking Srirangan Madhavan
2026-11-12 21:35 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 08/19] cxl: Move cxl_rwsem into built-in code Srirangan Madhavan
2026-11-12 23:49 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 09/19] cxl: Refresh cached PCI HDM decoder settings Srirangan Madhavan
2026-11-13 0:38 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 10/19] cxl: Build register block helpers into the kernel Srirangan Madhavan
2026-11-13 18:02 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 11/19] PCI/CXL: Cache endpoint HDM state during PCI enumeration Srirangan Madhavan
2026-11-10 14:12 ` Li Ming
2026-11-11 3:54 ` Li Ming
2026-11-17 17:29 ` Alison Schofield
2026-11-20 8:17 ` Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 12/19] PCI/CXL: Add CXL Device Reset sequencing Srirangan Madhavan
2026-11-10 14:29 ` Li Ming
2026-11-11 10:12 ` Lucero Palau, Alejandro
2026-11-05 18:59 ` [PATCH v17 13/19] cxl: Validate and synchronize HDM ranges around reset Srirangan Madhavan
2026-11-17 17:32 ` Alison Schofield
2026-11-20 8:20 ` Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 14/19] PCI/CXL: Reject reset with unsafe function scope Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 15/19] PCI/CXL: Restore CXL state after PCI reset Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 16/19] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-11-17 17:37 ` Alison Schofield
2026-11-20 8:21 ` Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 17/19] PCI/CXL: Restore CXL state after CXL bus reset Srirangan Madhavan
2026-11-13 17:37 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 18/19] PCI/CXL: Flush host and device caches for " Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 19/19] cxl: Add KUnit checks for HDM decoder unpacking Srirangan Madhavan
2026-11-05 19:31 ` [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Alison Schofield
2026-11-16 19:11 ` Dave Jiang
2026-11-20 7:32 ` Srirangan Madhavan
2026-11-17 17:28 ` Alison Schofield
2026-11-20 8:16 ` Srirangan Madhavan [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1c5b291c-25f1-4936-8601-0660e3627b8e@nvidia.com \
--to=smadhavan@nvidia.com \
--cc=alex.williamson@redhat.com \
--cc=alison.schofield@intel.com \
--cc=alwilliamson@nvidia.com \
--cc=bhelgaas@google.com \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=icheng@nvidia.com \
--cc=iweiny@kernel.org \
--cc=jan@nvidia.com \
--cc=jic23@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=machaujo@amazon.de \
--cc=mhonap@nvidia.com \
--cc=ming.li@zohomail.com \
--cc=skancherla@nvidia.com \
--cc=vaslot@nvidia.com \
--cc=vishal.l.verma@intel.com \
--cc=vsethi@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®