From: Shaikh Kamaluddin <shaikhkamal2012@gmail.com>
To: Jonathan Cameron <jic23@kernel.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>,
Dave Jiang <dave.jiang@intel.com>,
Alison Schofield <alison.schofield@intel.com>,
Vishal Verma <vishal.l.verma@intel.com>,
Dan Williams <djbw@kernel.org>, Miaohe Lin <linmiaohe@huawei.com>,
Andrew Morton <akpm@linux-foundation.org>,
Vlastimil Babka <vbabka@kernel.org>,
Breno Leitao <leitao@debian.org>, Ira Weiny <iweiny@kernel.org>,
Li Ming <ming.li@zohomail.com>, Richard Cheng <icheng@nvidia.com>,
Naoya Horiguchi <nao.horiguchi@gmail.com>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Brendan Jackman <brendan.jackman@linux.dev>,
Johannes Weiner <hannes@cmpxchg.org>, Zi Yan <ziy@nvidia.com>,
linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-mm@kvack.org, David Hildenbrand <david@kernel.org>,
Oscar Salvador <osalvador@suse.de>,
Kiryl Shutsemau <kas@kernel.org>, Harry Yoo <harry@kernel.org>
Subject: Re: [RFC PATCH 5/5] cxl/region: Register device poison before exposing RAM
Date: Fri, 20 Nov 2026 02:12:36 +0530 [thread overview]
Message-ID: <av9fPGZHfa8yOnq7@acer-nitro-anv15-41> (raw)
In-Reply-To: <20261016233532.51766fc4@jic23-hlaptop>
On Fri, Oct 16, 2026 at 11:35:32PM +0100, Jonathan Cameron wrote:
> On Sat, 10 Oct 2026 21:50:13 +0530
> Shaikh Kamaluddin <shaikhkamal2012@gmail.com> wrote:
>
> > The trigger_poison_list interface may retrieve poison after CXL memory
> > is already online, which is too late to prevent known-bad frames from
> > entering the page allocator.
> >
> > During RAM-region probe, retrieve a poison-list snapshot for each
> > participating endpoint decoder. Validate each record against the
> > decoder's assigned DPA range, translate its poisoned DPA units to HPA
> > and PFN, and collect the affected PFNs in an xarray. Coalesce consecutive
> > PFNs and register the resulting ranges with the pre-online hwpoison
> > registry before creating the DAX region.
> >
> > When Get Poison List is supported, abort region probe if the list is
> > incomplete, malformed, or cannot be translated. If volatile poison-list
> > retrieval is unsupported, preserve the existing behavior and warn that
> > poison cannot be withheld before memory onlining.
> >
> > Signed-off-by: Shaikh Kamaluddin <shaikhkamal2012@gmail.com>
> A few comments inline. I wouldn't rush to revise this though as I think
> the whole concept needs more input from others.
>
Hi Jonathan,
Agreed. There are open comments on the mm side too (3/5 and 4/5), so I
will get the design settled before sending v2.
> Jonathan
>
> > ---
> > drivers/cxl/core/region.c | 322 ++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 322 insertions(+)
> >
> > diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> > index c72524923b38..e3ffa77927df 100644
> > --- a/drivers/cxl/core/region.c
> > +++ b/drivers/cxl/core/region.c
> > @@ -14,6 +14,7 @@
> > #include <linux/string_choices.h>
> > #include <cxlmem.h>
> > #include <cxl.h>
> > +#include <linux/memory-failure.h>
> > #include "core.h"
> > #include "mce.h"
> >
> > @@ -3038,6 +3039,323 @@ int cxl_get_poison_by_endpoint(struct cxl_port *port)
> > return rc;
> > }
> >
> > +/*
> > + * Device-reported poison is gathered at RAM-region probe, before the range
> > + * reaches dax/kmem, and registered with the pre-online registry so the
> > + * frames it covers are withheld when the memory is onlined.
> > + */
> > +static int cxl_poison_store_pfn(struct xarray *pfns, u64 hpa)
> > +{
> > + return xa_err(xa_store(pfns, PHYS_PFN(hpa), xa_mk_value(1),
> > + GFP_KERNEL));
>
> Fix alignment of the line above. Right now it isn't visually clear
> which function call the parameter is for.
>
Will fix.
> > +}
>
> > +static int cxl_poison_record_to_pfns(struct cxl_region *cxlr,
> > + struct cxl_memdev *cxlmd,
> > + const struct cxl_poison_record *record,
> > + struct xarray *pfns)
> > +{
> > + struct cxl_region_params *p = &cxlr->params;
> > + u64 raw = le64_to_cpu(record->address);
> > + u64 units = le32_to_cpu(record->length);
> > + u64 dpa = raw & CXL_POISON_START_MASK;
> > + u8 source = raw & CXL_POISON_SOURCE_MASK;
> > + u64 length, end, step, cur;
> > +
> > + if (!units ||
> > + check_mul_overflow(units, (u64)CXL_POISON_LEN_MULT, &length) ||
> > + check_add_overflow(dpa, length, &end))
> > + return -EPROTO;
> > +
> > + /*
> > + * A corrected error never reaches the poison list, so every record
> > + * is uncorrectable. Only a failure of the device's own media is
> > + * withheld: External poison was written by the host into healthy
> > + * media and a rewrite or Clear Poison recovers it, and Injected
> > + * poison is a test artifact. Withholding either would make a
> > + * recoverable state permanent.
>
> This reflects back to the comment I made in the cover letter.
> Device discovered poison can be ephemeral.
Agreed. If the error was only temporary, the next write would fix it.
But nothing ever writes a withheld frame, so it stays on the list and
we keep withholding it forever.
>I'm also not sure for volatile
> where any poison record would come from if you are doing this for newly
> powered up devices - they might scan whilst initializing but might not as
> that is very very slow to do. Typically poison is found much later by
> patrol scrub (or synchronous read).
A real media fault is still there after a power cycle, which is why
PPR exists, at boot or at runtime. But on a volatile device the poison
list may not remember it: until patrol scrub or a read finds it again,
the list can be empty. So this mainly helps when the device stays
powered while the host restarts, like kexec or a driver re-bind, or
with a device that keeps its error records across a power cycle.
>This may be more interesting for
> persistent memory where the persistence is used, but you aren't addressing
> that as far as I can tell.
>
Agreed, that is the stronger case. Today the CXL driver doesn't feed
the poison list into the nvdimm bad ranges at all, so pmem and devdax
on CXL don't see known-bad locations either. cxl_pmem_ctl() also
returns -ENOTTY for ND_CMD_CLEAR_ERROR, so a bad block couldn't be
cleared by a write even if it were listed.
I will look at both as a separate series: bad ranges from the poison
list at pmem region probe, plus clear-error support. kmem could then
use the same list when a pmem range becomes system RAM.
> > + */
> > + if (source != CXL_POISON_SOURCE_INTERNAL) {
> > + dev_dbg(&cxlr->dev, "%s: dpa %#llx source %u not withheld\n",
> > + dev_name(&cxlmd->dev), dpa, source);
> > + return 0;
> > + }
> > +
> > + /*
> > + * Translate once per unit of HPA contiguity rather than per 64
> > + * bytes. Consecutive DPAs within an interleave granule map to
> > + * consecutive HPAs, and a granule no larger than a page lies within
> > + * one page, so stepping by min(granule, PAGE_SIZE) from an aligned
> > + * start reaches every page the record covers.
> > + */
> > + step = clamp_t(u64, p->interleave_granularity, CXL_POISON_LEN_MULT,
> > + PAGE_SIZE);
> > +
> > + for (cur = ALIGN_DOWN(dpa, step); cur < end; cur += step) {
> > + u64 hpa = cxl_dpa_to_hpa(cxlr, cxlmd, max(cur, dpa));
> > + int rc;
> > +
> > + /* Not mapped by this region; nothing here to withhold. */
> > + if (hpa == ULLONG_MAX)
> > + continue;
> > +
> > + rc = cxl_poison_store_hpa(cxlr, pfns, hpa);
> > + if (rc)
> > + return rc;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int cxl_region_collect_decoder_poison(struct cxl_region *cxlr,
> > + struct cxl_endpoint_decoder *cxled,
> > + struct xarray *pfns)
> > +{
> > + struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
> > + struct cxl_poison_snapshot snapshot;
> > + int rc;
> > +
> > + rc = cxl_poison_snapshot_init(cxlmd, &snapshot);
> > + if (rc == -EOPNOTSUPP) {
> > + dev_info(&cxlr->dev,
> > + "%s: no Get Poison List support, nothing withheld\n",
> > + dev_name(&cxlmd->dev));
> > + return 0;
> > + }
> > + if (rc)
> > + return rc;
> > +
> > + rc = cxl_get_poison_by_decoder(cxled, &snapshot,
> > + CXL_POISON_QUERY_PREONLINE);
> > + if (rc == -EFAULT) {
> > + /*
> > + * The device keeps no poison list for volatile ranges, which
> > + * CXL r3.2 8.2.9.8.4.1 permits. Online the region as today.
>
> Use latest available spec for references (they don't make old ones easy
> to download!)
>
Will update the latest references.
> > + */
> > + dev_info(&cxlr->dev,
> > + "%s: no volatile poison list, nothing withheld\n",
> > + dev_name(&cxlmd->dev));
> too noisy.
>
Will drop it.
> > + rc = 0;
> > + goto out;
> > + }
> > + if (rc)
> > + goto out;
> > +
> > + /*
> > + * Withholding from a partial list would online the region on the
> > + * belief that every bad frame had been named.
> > + */
> > + if (snapshot.truncated ||
> > + (snapshot.device_flags & CXL_POISON_FLAG_OVERFLOW)) {
> > + dev_err(&cxlr->dev, "%s: poison list incomplete\n",
> > + dev_name(&cxlmd->dev));
> > + rc = -EOVERFLOW;
> > + goto out;
> > + }
> > + if (snapshot.device_flags & CXL_POISON_FLAG_SCANNING) {
>
> That one seems wrong. We don't prevent memory from coming online because
> there might be poison to be found.
Agreed, I will drop that check. Anything the scan finds later is handled
like any other poison found at runtime.
>
> > + dev_err(&cxlr->dev, "%s: media scan in progress\n",
> > + dev_name(&cxlmd->dev));
> > + rc = -EBUSY;
> > + goto out;
> > + }
> > +
> > + for (u32 i = 0; i < snapshot.nr_records; i++) {
> > + rc = cxl_poison_record_to_pfns(cxlr, cxlmd,
> > + &snapshot.records[i], pfns);
> > + if (rc)
> > + break;
> > + }
> > +out:
> > + cxl_poison_snapshot_destroy(&snapshot);
> > + return rc;
> > +}
> > +
> > +static unsigned int cxl_count_poison_ranges(struct xarray *pfns)
> > +{
> > + unsigned long index, previous = 0;
> > + unsigned int nr = 0;
> > + void *entry;
> > + bool first = true;
> > +
> > + xa_for_each(pfns, index, entry) {
> > + if (first || index != previous + 1)
> > + nr++;
> > + first = false;
> > + previous = index;
> > + }
> > + return nr;
> > +}
> > +
> > +/*
> > + * The xarray holds each PFN once and xa_for_each() walks them in ascending
> > + * order, so runs of consecutive indices are exactly the sorted,
> > + * nonoverlapping ranges preonline_hwpoison_register() requires.
> > + */
> > +static int cxl_pfns_to_ranges(struct xarray *pfns,
> > + struct preonline_hwpoison_range **ranges_out,
> > + unsigned int *nr_ranges_out)
> > +{
> > + struct preonline_hwpoison_range *ranges;
> > + unsigned long index, previous = 0, run_start = 0;
> > + unsigned int nr_ranges, nr = 0;
> > + void *entry;
> > + bool first = true;
> > +
> > + nr_ranges = cxl_count_poison_ranges(pfns);
> > + if (!nr_ranges) {
> > + *ranges_out = NULL;
> > + *nr_ranges_out = 0;
> > + return 0;
> > + }
> > +
> > + ranges = kvmalloc_array(nr_ranges, sizeof(*ranges), GFP_KERNEL);
>
> kvmalloc_objs()
>
Will switch.
Thanks,
Shaikh Kamal
> > + if (!ranges)
> > + return -ENOMEM;
> > +
> > + xa_for_each(pfns, index, entry) {
> > + if (first) {
> > + run_start = index;
> > + previous = index;
> > + first = false;
> > + continue;
> > + }
> > + if (index == previous + 1) {
> > + previous = index;
> > + continue;
> > + }
> > + ranges[nr++] = (struct preonline_hwpoison_range) {
> > + .start_pfn = run_start,
> > + .nr_pages = previous - run_start + 1,
> > + };
> > + run_start = index;
> > + previous = index;
> > + }
> > + ranges[nr++] = (struct preonline_hwpoison_range){
> > + .start_pfn = run_start,
> > + .nr_pages = previous - run_start + 1,
> > + };
> > +
> > + WARN_ON_ONCE(nr != nr_ranges);
> > + *ranges_out = ranges;
> > + *nr_ranges_out = nr;
> > + return 0;
> > +}
> > +
> > +static void cxl_unregister_device_poison(void *data)
> > +{
> > + preonline_hwpoison_unregister(data);
> > +}
> > +
> > +static int cxl_region_register_device_poison(struct cxl_region *cxlr)
> > +{
> > + struct cxl_region_params *p = &cxlr->params;
> > + struct preonline_hwpoison_range *ranges = NULL;
> > + struct preonline_hwpoison *handle;
> > + unsigned int nr_ranges = 0;
> > + unsigned long nr_frames = 0;
> > + struct xarray pfns;
> > + int rc = 0;
> > +
> > + lockdep_assert_held(&cxl_rwsem.region);
> > + lockdep_assert_held(&cxl_rwsem.dpa);
> > +
> > + /* The region may have been decommitted while the lock was dropped. */
> > + if (p->state < CXL_CONFIG_COMMIT)
> > + return -ENXIO;
> > +
> > + if (test_bit(CXL_REGION_F_NORMALIZED_ADDRESSING, &cxlr->flags)) {
> > + dev_info(&cxlr->dev,
> > + "normalized addressing, poison not translated\n");
> > + return 0;
> > + }
> > +
> > + xa_init(&pfns);
> > +
> > + for (int i = 0; i < p->nr_targets; i++) {
> > + struct cxl_endpoint_decoder *cxled = p->targets[i];
> > +
> > + if (!cxled->dpa_res || !resource_size(cxled->dpa_res))
> > + continue;
> > + rc = cxl_region_collect_decoder_poison(cxlr, cxled, &pfns);
> > + if (rc)
> > + goto out;
> > + }
> > +
> > + if (xa_empty(&pfns))
> > + goto out;
> > +
> > + rc = cxl_pfns_to_ranges(&pfns, &ranges, &nr_ranges);
> > + if (rc)
> > + goto out;
> > +
> > + rc = preonline_hwpoison_register(ranges, nr_ranges, &handle);
> > + if (rc == -EOPNOTSUPP) {
> > + /* No poison tracking in this kernel; nothing to withhold. */
> > + rc = 0;
> > + goto out;
> > + }
> > + if (rc)
> > + goto out;
> > +
> > + for (unsigned int i = 0; i < nr_ranges; i++)
> > + nr_frames += ranges[i].nr_pages;
> > + dev_warn(&cxlr->dev,
> > + "withholding %lu frame(s) reported poisoned by the device\n",
> > + nr_frames);
> > +
> > + rc = devm_add_action_or_reset(&cxlr->dev, cxl_unregister_device_poison,
> > + handle);
> > +out:
> > + kvfree(ranges);
> > + xa_destroy(&pfns);
> > + return rc;
> > +}
>
>
next prev parent reply other threads:[~2026-11-19 20:42 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-10 16:20 [RFC PATCH 0/5] cxl, mm: Keep device-reported poison out of the page allocator Shaikh Kamaluddin
2026-10-10 16:20 ` [RFC PATCH 1/5] cxl/mbox: Add Get Poison List snapshot support Shaikh Kamaluddin
2026-10-16 22:18 ` Jonathan Cameron
2026-11-16 23:45 ` Shaikh Kamaluddin
2026-10-10 16:20 ` [RFC PATCH 2/5] cxl/region: Factor decoder poison-list retrieval Shaikh Kamaluddin
2026-10-16 22:21 ` Jonathan Cameron
2026-11-17 1:03 ` Shaikh Kamaluddin
2026-10-10 16:20 ` [RFC PATCH 3/5] mm/memory-failure: Add a pre-online poison registry Shaikh Kamaluddin
2026-11-12 9:04 ` David Hildenbrand (Arm)
2026-10-10 16:20 ` [RFC PATCH 4/5] mm/page_alloc: Keep pre-online poison out of the buddy allocator Shaikh Kamaluddin
2026-10-15 7:27 ` Richard Cheng
2026-10-22 20:09 ` Shaikh Kamaluddin
2026-11-07 15:32 ` Shaikh Kamaluddin
2026-11-12 9:08 ` David Hildenbrand (Arm)
2026-10-10 16:20 ` [RFC PATCH 5/5] cxl/region: Register device poison before exposing RAM Shaikh Kamaluddin
2026-10-15 7:59 ` Richard Cheng
2026-10-19 20:00 ` Dave Jiang
2026-11-12 3:46 ` Shaikh Kamaluddin
2026-10-16 22:35 ` Jonathan Cameron
2026-11-19 20:42 ` Shaikh Kamaluddin [this message]
2026-10-16 20:42 ` [RFC PATCH 0/5] cxl, mm: Keep device-reported poison out of the page allocator Jonathan Cameron
2026-10-20 1:11 ` Alison Schofield
2026-10-21 20:18 ` Shaikh Kamaluddin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av9fPGZHfa8yOnq7@acer-nitro-anv15-41 \
--to=shaikhkamal2012@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=alison.schofield@intel.com \
--cc=brendan.jackman@linux.dev \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=david@kernel.org \
--cc=djbw@kernel.org \
--cc=hannes@cmpxchg.org \
--cc=harry@kernel.org \
--cc=icheng@nvidia.com \
--cc=iweiny@kernel.org \
--cc=jic23@kernel.org \
--cc=kas@kernel.org \
--cc=leitao@debian.org \
--cc=linmiaohe@huawei.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.com \
--cc=ming.li@zohomail.com \
--cc=nao.horiguchi@gmail.com \
--cc=osalvador@suse.de \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
--cc=vishal.l.verma@intel.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®