From: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
To: Anisa Su <anisa.su887@gmail.com>
Cc: linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev,
linux-kernel@vger.kernel.org, Dave Jiang <dave.jiang@intel.com>,
Alison Schofield <alison.schofield@intel.com>,
Davidlohr Bueso <dave@stgolabs.net>,
Ira Weiny <iweiny@kernel.org>,
Vishal Verma <vishal.l.verma@intel.com>,
Fan Ni <nifan.cxl@gmail.com>, Li Ming <ming.li@zohomail.com>,
Benjamin Cheatham <benjamin.cheatham@amd.com>,
Gregory Price <gourry@gourry.net>,
Richard Cheng <icheng@nvidia.com>,
Anisa Su <anisa.su@samsung.com>
Subject: Re: [PATCH v1 05/14] cxl/mem: Add 20 second timeout for stalled DC event chains
Date: Tue, 17 Nov 2026 03:00:14 +0000 [thread overview]
Message-ID: <20261117030014.6676dd47@jic23-hlaptop> (raw)
In-Reply-To: <20261104230445.66625-6-anisa.su@samsung.com>
On Thu, 5 Nov 2026 08:02:29 +0900
Anisa Su <anisa.su887@gmail.com> wrote:
> A Dynamic Capacity event can span multiple event records grouped
> together by the CXL_DCD_EVENT_MORE flag. Extents are staged in the
> pending list until the last event record ('More'=0) is received, at
> which point the chain is processed. If the device opens such a chain
> (More=1) but never sends the closing record, the staged list sits
> indefinitely.
>
> Add a delayed-work watchdog that drops the chain on timeout: an add
> chain is refused with an empty Add Dynamic Capacity Response (4802h),
> a release chain is left for the device to retry.
>
> The spec sets no limit; 20s is a generous upper bound that may be
> tightened later, and guards against a lost mailbox response or a
> crashed fabric manager.
My gut feeling on this is don't paper over it.
I think there should be a standard recovery flow from crashed fabric
manager and today we may have a hole here that needs closing. That
might be as simple as a zero length dcd extent sent with the more
flag not set to terminate any sequences in flight.
Lost mailbox responses should only happen if something worse
went on so I'd expect some other ras indication.
Anyhow, I'd suggest asking your spec folk to raise this (or poking John
Groves)..
>
> The deadline is fixed when the chain opens rather than extended per
> record, so a device cannot keep a chain alive, and the staged list
> growing, by streaming More=1 records.
>
> Signed-off-by: Anisa Su <anisa.su@samsung.com>
> Tested-by: Wonjae Lee <wj28.lee@samsung.com>
> Tested-by: Junhee Park <jh9934.park@samsung.com>
> Tested-by: Heesoo Kim <habil.kim@samsung.com>
> ---
> drivers/cxl/core/mbox.c | 84 +++++++++++++++++++++++++++++++++++++++++
> drivers/cxl/cxlmem.h | 15 ++++++++
> 2 files changed, 99 insertions(+)
>
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 3acf5ace48db..fea910e6c510 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -873,6 +873,8 @@ static const uuid_t log_uuid[] = {
> [VENDOR_DEBUG_UUID] = DEFINE_CXL_VENDOR_DEBUG_UUID,
> };
>
> +static void cxl_cancel_dcd_chain_work(void *_mds);
> +
> /**
> * cxl_enumerate_cmds() - Enumerate commands for a device.
> * @mds: The driver data for the operation
> @@ -891,6 +893,16 @@ int cxl_enumerate_cmds(struct cxl_memdev_state *mds)
> struct cxl_mem_command *cmd;
> int i, rc;
>
> + /*
> + * The DC chain watchdog sends a mailbox command when it fires. Register
> + * its cancellation here, once the mailbox is usable, so devm runs it
> + * before the mailbox goes away: in cxl_pci the device registers were
> + * mapped before this point and are unmapped after the cancel.
> + */
> + rc = devm_add_action_or_reset(dev, cxl_cancel_dcd_chain_work, mds);
> + if (rc)
> + return rc;
> +
> gsl = cxl_get_gsl(mds);
> if (IS_ERR(gsl))
> return PTR_ERR(gsl);
> @@ -1198,6 +1210,14 @@ static void clear_pending_extents(void *_mds)
> delete_extent_node(pos);
> }
>
> +/*
> + * Bound on how long the host will wait for a device to finish a
> + * multi-record DC event chain (More=1 ... More=0) before refusing the
> + * chain. The spec sets no limit; 20s is a generous upper bound, the
> + * maximum GPF timeout.
> + */
> +#define CXL_DC_CHAIN_TIMEOUT (20 * HZ)
> +
> static const char *cxl_dcd_evt_type_str(u8 type)
> {
> switch (type) {
> @@ -1226,6 +1246,11 @@ static void cxl_dc_chain_abort(struct cxl_memdev_state *mds)
> struct cxl_dc_event_chain *chain = &mds->dc_chain;
> struct device *dev = mds->cxlds.dev;
>
> + lockdep_assert_held(&chain->lock);
> +
> + chain->armed = false;
> + cancel_delayed_work(&chain->timeout_work);
> +
> if (chain->event_type == DCD_ADD_CAPACITY &&
> cxl_send_dc_response(mds, CXL_MBOX_OP_ADD_DC_RESPONSE,
> &chain->pending_extents, 0))
> @@ -1234,6 +1259,43 @@ static void cxl_dc_chain_abort(struct cxl_memdev_state *mds)
> clear_pending_extents(mds);
> }
>
> +static void cxl_dc_chain_timeout(struct work_struct *work)
> +{
> + struct cxl_dc_event_chain *chain = container_of(to_delayed_work(work),
> + struct cxl_dc_event_chain,
> + timeout_work);
> + struct cxl_memdev_state *mds = container_of(chain,
> + struct cxl_memdev_state,
> + dc_chain);
> + struct device *dev = mds->cxlds.dev;
> +
> + guard(mutex)(&chain->lock);
> +
> + /*
> + * handle_dcd_chain_event() cancels this work non-synchronously (a sync
> + * cancel would deadlock on @chain->lock, which the chain-close path
> + * holds), so a callback that already started running can reach here
> + * after its chain has moved on. Abort only if a chain is still armed
> + * AND the timer has not been re-armed since this expiry fired: a new
> + * chain's queue_delayed_work() makes delayed_work_pending() true,
> + * meaning this expiry belongs to a superseded deadline and must not
> + * abort the current chain.
> + */
> + if (!chain->armed || delayed_work_pending(&chain->timeout_work))
> + return;
> +
> + dev_warn(dev, "DC %s chain timed out\n",
> + cxl_dcd_evt_type_str(chain->event_type));
> + cxl_dc_chain_abort(mds);
> +}
> +
> +static void cxl_cancel_dcd_chain_work(void *_mds)
> +{
> + struct cxl_memdev_state *mds = _mds;
> +
> + cancel_delayed_work_sync(&mds->dc_chain.timeout_work);
> +}
> +
> static int stage_pending_extent(struct list_head *pending,
> struct cxl_extent *to_add)
> {
> @@ -1260,6 +1322,8 @@ static int handle_dcd_chain_event(struct cxl_memdev_state *mds,
> struct device *dev = mds->cxlds.dev;
> int rc;
>
> + guard(mutex)(&chain->lock);
> +
> /* Chained records share one event type (CXL r4.0 Table 8-229) */
> if (list_empty(&chain->pending_extents)) {
> chain->event_type = event->event_type;
> @@ -1279,9 +1343,22 @@ static int handle_dcd_chain_event(struct cxl_memdev_state *mds,
>
> if (event->flags & CXL_DCD_EVENT_MORE) {
> dev_dbg(dev, "more bit set; delay processing of extent\n");
> + /* The deadline is fixed when the chain opens, not per record */
> + if (!chain->armed) {
> + queue_delayed_work(system_wq, &chain->timeout_work,
> + CXL_DC_CHAIN_TIMEOUT);
> + chain->armed = true;
> + }
> return 0;
> }
>
> + /*
> + * Chain is closing. Disarm before flushing so a pending watchdog
> + * (queued but blocked on @chain->lock) sees !armed and bails out.
> + */
> + chain->armed = false;
> + cancel_delayed_work(&chain->timeout_work);
> +
> /* Nothing is tracked yet: refuse every add, acknowledge every release */
> if (event->event_type == DCD_ADD_CAPACITY)
> rc = cxl_send_dc_response(mds, CXL_MBOX_OP_ADD_DC_RESPONSE,
> @@ -2021,7 +2098,14 @@ struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev, u64 serial,
>
> mutex_init(&mds->event.log_lock);
> INIT_LIST_HEAD(&mds->dc_chain.pending_extents);
> + mutex_init(&mds->dc_chain.lock);
> + INIT_DELAYED_WORK(&mds->dc_chain.timeout_work, cxl_dc_chain_timeout);
>
> + /*
> + * The watchdog cancel is registered later, by cxl_enumerate_cmds(), so
> + * devm's reverse-order unwind cancels (and waits for) the watchdog
> + * before this list cleanup runs.
> + */
> rc = devm_add_action_or_reset(dev, clear_pending_extents, mds);
> if (rc)
> return ERR_PTR(rc);
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index c88de7cb65d1..52c936ec15a9 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -8,6 +8,8 @@
> #include <linux/uuid.h>
> #include <linux/node.h>
> #include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/workqueue.h>
> #include <cxl/event.h>
> #include <cxl/mailbox.h>
> #include "cxl.h"
> @@ -431,6 +433,11 @@ static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
> * struct cxl_dc_event_chain - staged Dynamic Capacity event chain
> * @pending_extents: extents received so far; flushed when the chain
> * closes (More=0)
> + * @timeout_work: watchdog that fires if a chain is opened with
> + * CXL_DCD_EVENT_MORE but the closing record never arrives
> + * @lock: serialises updates to the chain state against the watchdog
> + * @armed: set when a More=1 chain opens; cleared when the chain closes
> + * (More=0), is aborted by cxl_dc_chain_abort(), or times out.
> * @event_type: DCD event type of the open chain (add or release)
> *
> * The records of one Dynamic Capacity operation can be chained by the
> @@ -439,9 +446,17 @@ static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
> * chain is open at a time. Both Add Capacity and Release Capacity chains
> * are staged here, @event_type says which. Records are staged until
> * More=0, then the chain is processed and responded to as a whole.
> + *
> + * If a chain is opened (More=1) but the device never sends the closing
> + * record, the staged list would otherwise sit indefinitely. @timeout_work
> + * is a defensive watchdog that drops such a chain: an add is refused with
> + * an empty response, a release is held for the device to retry.
> */
> struct cxl_dc_event_chain {
> struct list_head pending_extents;
> + struct delayed_work timeout_work;
> + struct mutex lock; /* chain state, against the watchdog */
> + bool armed;
> u8 event_type;
> };
>
next prev parent reply other threads:[~2026-11-17 3:00 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-04 23:02 [PATCH v1 00/14] DCD CXL Series: Dynamic Capacity extent handling Anisa Su
2026-11-04 23:02 ` [PATCH v1 01/14] cxl/core: Return endpoint decoder information from region search Anisa Su
2026-11-04 23:02 ` [PATCH v1 02/14] cxl/port: Add dynamic capacity options to endpoint decoder mode Anisa Su
2026-11-17 2:22 ` Jonathan Cameron
2026-11-04 23:02 ` [PATCH v1 03/14] cxl/region: Add dynamic DAX region support Anisa Su
2026-11-17 2:32 ` Jonathan Cameron
2026-11-04 23:02 ` [PATCH v1 04/14] cxl/mem: Introduce DC event handlers Anisa Su
2026-11-17 2:54 ` Jonathan Cameron
2026-11-04 23:02 ` [PATCH v1 05/14] cxl/mem: Add 20 second timeout for stalled DC event chains Anisa Su
2026-11-17 3:00 ` Jonathan Cameron [this message]
2026-11-04 23:02 ` [PATCH v1 06/14] cxl/extent: Handle DC Add Capacity events Anisa Su
2026-11-12 3:34 ` Richard Cheng
2026-11-16 21:53 ` Anisa Su
2026-11-04 23:02 ` [PATCH v1 07/14] cxl/mem: Drop misaligned DCD extent groups Anisa Su
2026-11-04 23:02 ` [PATCH v1 08/14] cxl/extent: Validate DC extent partition Anisa Su
2026-11-04 23:02 ` [PATCH v1 09/14] cxl/mem: Enforce tag-group semantics Anisa Su
2026-11-04 23:02 ` [PATCH v1 10/14] cxl/extent: Enforce cross-region tag uniqueness Anisa Su
2026-11-04 23:02 ` [PATCH v1 11/14] cxl/extent: Handle DC Release Capacity events Anisa Su
2026-11-12 7:34 ` Richard Cheng
2026-11-04 23:02 ` [PATCH v1 12/14] cxl/region: Read existing extents on region creation Anisa Su
2026-11-04 23:02 ` [PATCH v1 13/14] cxl/mem: Trace Dynamic capacity Event Record Anisa Su
2026-11-04 23:02 ` [PATCH v1 14/14] Documentation/cxl: Document DCD extent handling Anisa Su
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117030014.6676dd47@jic23-hlaptop \
--to=jonathan.cameron@oss.qualcomm.com \
--cc=alison.schofield@intel.com \
--cc=anisa.su887@gmail.com \
--cc=anisa.su@samsung.com \
--cc=benjamin.cheatham@amd.com \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=gourry@gourry.net \
--cc=icheng@nvidia.com \
--cc=iweiny@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ming.li@zohomail.com \
--cc=nifan.cxl@gmail.com \
--cc=nvdimm@lists.linux.dev \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®