Re: [PATCH v9 16/19] cxl/region: Read existing extents on region creation

[Davidlohr Bueso <dave@stgolabs.net>]

On Sun, 13 Apr 2025, Ira Weiny wrote:

>Dynamic capacity device extents may be left in an accepted state on a
>device due to an unexpected host crash.  In this case it is expected
>that the creation of a new region on top of a DC partition can read
>those extents and surface them for continued use.
>
>Once all endpoint decoders are part of a region and the region is being
>realized, a read of the 'devices extent list' can reveal these
>previously accepted extents.
>
>CXL r3.1 specifies the mailbox call Get Dynamic Capacity Extent List for
>this purpose.  The call returns all the extents for all dynamic capacity
>partitions.  If the fabric manager is adding extents to any DCD
>partition, the extent list for the recovered region may change.  In this
>case the query must retry.  Upon retry the query could encounter extents
>which were accepted on a previous list query.  Adding such extents is
>ignored without error because they are entirely within a previous
>accepted extent.  Instead warn on this case to allow for differentiating
>bad devices from this normal condition.
>
>Latch any errors to be bubbled up to ensure notification to the user
>even if individual errors are rate limited or otherwise ignored.
>
>The scan for existing extents races with the dax_cxl driver.  This is
>synchronized through the region device lock.  Extents which are found
>after the driver has loaded will surface through the normal notification
>path while extents seen prior to the driver are read during driver load.
>
>Based on an original patch by Navneet Singh.
>
>Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>Reviewed-by: Fan Ni <fan.ni@samsung.com>
>Signed-off-by: Ira Weiny <ira.weiny@intel.com>
>
>---
>Changes:
>[0day: fix extent count in GetExtent input payload]
>[iweiny: minor clean ups]
>[iweiny: Adjust for partition arch]
>---
> drivers/cxl/core/core.h   |   1 +
> drivers/cxl/core/mbox.c   | 109 ++++++++++++++++++++++++++++++++++++++++++++++
> drivers/cxl/core/region.c |  25 +++++++++++
> drivers/cxl/cxlmem.h      |  21 +++++++++
> 4 files changed, 156 insertions(+)
>
>diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
>index 027dd1504d77..e06a46fec217 100644
>--- a/drivers/cxl/core/core.h
>+++ b/drivers/cxl/core/core.h
>@@ -22,6 +22,7 @@ cxled_to_mds(struct cxl_endpoint_decoder *cxled)
>	return container_of(cxlds, struct cxl_memdev_state, cxlds);
> }
>
>+int cxl_process_extent_list(struct cxl_endpoint_decoder *cxled);
> int cxl_region_invalidate_memregion(struct cxl_region *cxlr);
>
> #ifdef CONFIG_CXL_REGION
>diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
>index de01c6684530..8af3a4173b99 100644
>--- a/drivers/cxl/core/mbox.c
>+++ b/drivers/cxl/core/mbox.c
>@@ -1737,6 +1737,115 @@ int cxl_dev_dc_identify(struct cxl_mailbox *mbox,
> }
> EXPORT_SYMBOL_NS_GPL(cxl_dev_dc_identify, "CXL");
>
>+/* Return -EAGAIN if the extent list changes while reading */
>+static int __cxl_process_extent_list(struct cxl_endpoint_decoder *cxled)
>+{
>+	u32 current_index, total_read, total_expected, initial_gen_num;
>+	struct cxl_memdev_state *mds = cxled_to_mds(cxled);
>+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
>+	struct device *dev = mds->cxlds.dev;
>+	struct cxl_mbox_cmd mbox_cmd;
>+	u32 max_extent_count;
>+	int latched_rc = 0;
>+	bool first = true;
>+
>+	struct cxl_mbox_get_extent_out *extents __free(kvfree) =
>+				kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
>+	if (!extents)
>+		return -ENOMEM;
>+
>+	total_read = 0;
>+	current_index = 0;
>+	total_expected = 0;
>+	max_extent_count = (cxl_mbox->payload_size - sizeof(*extents)) /
>+				sizeof(struct cxl_extent);
>+	do {
>+		u32 nr_returned, current_total, current_gen_num;
>+		struct cxl_mbox_get_extent_in get_extent;
>+		int rc;
>+
>+		get_extent = (struct cxl_mbox_get_extent_in) {
>+			.extent_cnt = cpu_to_le32(max(max_extent_count,
>+						  total_expected - current_index)),

s/max/min().

>+			.start_extent_index = cpu_to_le32(current_index),