mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Unsung Lee <unsung0317@gmail.com>
To: linux-cxl@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, Davidlohr Bueso <dave@stgolabs.net>,
	Jonathan Cameron <jic23@kernel.org>,
	Dave Jiang <dave.jiang@intel.com>,
	Alison Schofield <alison.schofield@intel.com>,
	Vishal Verma <vishal.l.verma@intel.com>,
	Dan Williams <djbw@kernel.org>, Ira Weiny <iweiny@kernel.org>,
	Li Ming <ming.li@zohomail.com>
Subject: [PATCH] cxl/port: Do not queue a memdev detach when the memdev unbinds itself
Date: Sun, 22 Nov 2026 21:20:14 +0900	[thread overview]
Message-ID: <20261122122014.2194208-1-unsung0317@gmail.com> (raw)

Binding a memdev to cxl_mem right after unbinding it often fails: the
probe returns -EBUSY, or the bind succeeds and the memdev is unbound
again a few milliseconds later. In the second case the bind write
returns success, so nothing tells userspace that the device ended up
unbound.

cxl_endpoint_port_probe() registers schedule_detach() as a devm action
of the endpoint port, so that a memdev whose endpoint port, or a port
above it, goes away is detached. Unbinding cxl_mem deletes the endpoint
port itself (delete_endpoint()), which runs that action too and queues
cxlmd->detach_work although the memdev is already going away.
detach_memdev() later calls device_release_driver(&cxlmd->dev), which
takes the device lock and releases whatever driver is bound at that
point. A bind that runs before the work makes cxl_mem_probe() fail its
work_pending(&cxlmd->detach_work) check with -EBUSY. A bind that is in
progress when the work runs is released by it as soon as it completes:

  bash     echo mem0 > unbind: cxl_mem removes endpoint2 and its decoders
  kworker  detach_memdev(): device_release_driver(mem0) waits for the lock
  bash     echo mem0 > bind: cxl_mem_probe() returns 0
  kworker  releases mem0 again, endpoint2 and its decoders go away

Mark the memdev while delete_endpoint() removes the endpoint port and
skip the detach in schedule_detach() then, dropping the reference the
work would have dropped. A memdev with ->attach still gets the detach,
since there it releases the parent device, which depends on the
memdev being attached. Losing the endpoint port or a port above it
otherwise still detaches the memdev as before.

The problem was found by an automated test harness driven by an LLM
coding assistant. The analysis and the fix are my own.

Fixes: 8dd2bc0f8e02 ("cxl/mem: Add the cxl_mem driver")
Signed-off-by: Unsung Lee <unsung0317@gmail.com>
---
Question: is the detach meant to be queued on a cxl_mem unbind too? For
memdevs with ->attach the patch keeps it, as there it releases the
parent; I have no such device to test.

Reproducer, as root in a QEMU guest with one type3 device
(-M q35,cxl=on,cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=8G
 -object memory-backend-ram,id=vmem0,share=on,size=8G
 -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1
 -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2
 -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0):

  M=mem0; D=/sys/bus/cxl/drivers/cxl_mem
  for i in $(seq 40); do
          [ -e /sys/bus/cxl/devices/$M/driver ] || { echo $M > $D/bind; sleep 1; }
          echo $M > $D/unbind
          echo $M > $D/bind 2>/dev/null; rc=$?
          sleep 1
          [ -e /sys/bus/cxl/devices/$M/driver ] && s=bound || s=unbound
          echo "bind rc=$rc $s"
  done | sort | uniq -c

Without the patch 7/40 and 11/40 fail on cxl/next 32af781935 (2 of
the 11 with the bind returning 0 and mem0 unbound after), 14/40 on
v7.3.1.

With the patch on cxl/next (release and KASAN/lockdep builds) and on
v7.3.1 (KASAN/lockdep): 120/120 stay bound on each kernel, unbinding
the endpoint or host bridge port still unbinds the memdev, and 3
minutes of racing memdev and port unbind/bind gave no warnings.

Not tested: memdevs with ->attach, switches, real hardware. Regions were
only checked on v7.3.1: cxl/next creates no root decoder for QEMU's
CFMWS (restrictions 0x2f, "cannot have both HDM-H and HDM-DB").

 drivers/cxl/core/port.c |  3 +++
 drivers/cxl/cxlmem.h    |  2 ++
 drivers/cxl/port.c      | 14 +++++++++++++-
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index fffcb9954d..4960a288a1 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1514,9 +1514,12 @@ static void delete_endpoint(void *data)
 
 	scoped_guard(device, host) {
 		if (host->driver && !endpoint->dead) {
+			/* @cxlmd is going away, see schedule_detach() */
+			WRITE_ONCE(cxlmd->unbinding, true);
 			devm_release_action(host, cxl_unlink_parent_dport, endpoint);
 			devm_release_action(host, cxl_unlink_uport, endpoint);
 			devm_release_action(host, unregister_port, endpoint);
+			WRITE_ONCE(cxlmd->unbinding, false);
 		}
 		cxlmd->endpoint = NULL;
 	}
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 4cf8a0f6cf..0d566b8e32 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -41,6 +41,7 @@
  * @cdev: char dev core object for ioctl operations
  * @cxlds: The device state backing this device
  * @detach_work: active memdev lost a port in its ancestry
+ * @unbinding: the endpoint goes away because this memdev is being unbound
  * @cxl_nvb: coordinate removal of @cxl_nvd if present
  * @cxl_nvd: optional bridge to an nvdimm if the device supports pmem
  * @endpoint: connection to the CXL port topology for this memory device
@@ -58,6 +59,7 @@ struct cxl_memdev {
 	struct cdev cdev;
 	struct cxl_dev_state *cxlds;
 	struct work_struct detach_work;
+	bool unbinding;
 	struct cxl_nvdimm_bridge *cxl_nvb;
 	struct cxl_nvdimm *cxl_nvd;
 	struct cxl_port *endpoint;
diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
index 96fade0299..092b138827 100644
--- a/drivers/cxl/port.c
+++ b/drivers/cxl/port.c
@@ -26,8 +26,20 @@
  * PCIe topology.
  */
 
-static void schedule_detach(void *cxlmd)
+static void schedule_detach(void *data)
 {
+	struct cxl_memdev *cxlmd = data;
+
+	/*
+	 * A detach queued while @cxlmd unbinds itself would run after it
+	 * and could release the next binding instead. With ->attach the
+	 * detach releases the parent, which still has to happen. Read
+	 * without the host lock when the endpoint port is unbound directly.
+	 */
+	if (READ_ONCE(cxlmd->unbinding) && !cxlmd->attach) {
+		put_device(&cxlmd->dev);
+		return;
+	}
 	schedule_cxl_memdev_detach(cxlmd);
 }
 
-- 
2.34.1


                 reply	other threads:[~2026-11-22 12:20 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122122014.2194208-1-unsung0317@gmail.com \
    --to=unsung0317@gmail.com \
    --cc=alison.schofield@intel.com \
    --cc=dave.jiang@intel.com \
    --cc=dave@stgolabs.net \
    --cc=djbw@kernel.org \
    --cc=iweiny@kernel.org \
    --cc=jic23@kernel.org \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=ming.li@zohomail.com \
    --cc=vishal.l.verma@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®