Re: [PATCH v6 0/9] dax/hmem, cxl: Coordinate Soft Reserved handling with CXL and HMEM

["Koralahalli Channabasappa, Smita" <skoralah@amd.com>]

On 2/11/2026 10:38 PM, Alison Schofield wrote:
> On Tue, Feb 10, 2026 at 11:49:04AM -0800, Koralahalli Channabasappa, Smita wrote:
>> Hi Alison,
>>
>> On 2/10/2026 11:16 AM, Alison Schofield wrote:
>>> On Tue, Feb 10, 2026 at 06:44:52AM +0000, Smita Koralahalli wrote:
>>>> This series aims to address long-standing conflicts between HMEM and
>>>> CXL when handling Soft Reserved memory ranges.
>>>>
>>>> Reworked from Dan's patch:
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/patch/?id=ab70c6227ee6165a562c215d9dcb4a1c55620d5d
>>>>
>>>> Previous work:
>>>> https://lore.kernel.org/all/20250715180407.47426-1-Smita.KoralahalliChannabasappa@amd.com/
>>>>
>>>> Link to v5:
>>>> https://lore.kernel.org/all/20260122045543.218194-1-Smita.KoralahalliChannabasappa@amd.com
>>>>
>>>> The series is based on branch "for-7.0/cxl-init" and base-commit is
>>>> base-commit: bc62f5b308cbdedf29132fe96e9d591e526527e1
>>>>
>>>> [1] After offlining the memory I can tear down the regions and recreate
>>>> them back. dax_cxl creates dax devices and onlines memory.
>>>> 850000000-284fffffff : CXL Window 0
>>>>     850000000-284fffffff : region0
>>>>       850000000-284fffffff : dax0.0
>>>>         850000000-284fffffff : System RAM (kmem)
>>>>
>>>> [2] With CONFIG_CXL_REGION disabled, all the resources are handled by
>>>> HMEM. Soft Reserved range shows up in /proc/iomem, no regions come up
>>>> and dax devices are created from HMEM.
>>>> 850000000-284fffffff : CXL Window 0
>>>>     850000000-284fffffff : Soft Reserved
>>>>       850000000-284fffffff : dax0.0
>>>>         850000000-284fffffff : System RAM (kmem)
>>>>
>>>> [3] Region assembly failure works same as [2].
>>>>
>>>> [4] REGISTER path:
>>>> When CXL_BUS = y (with CXL_ACPI, CXL_PCI, CXL_PORT, CXL_MEM = y),
>>>> the dax_cxl driver is probed and completes initialization before dax_hmem
>>>> probes. This scenario was tested with CXL = y, DAX_CXL = m and
>>>> DAX_HMEM = m. To validate the REGISTER path, I forced REGISTER even in
>>>> cases where SR completely overlaps the CXL region as I did not have access
>>>> to a system where the CXL region range is smaller than the SR range.
>>>>
>>>> 850000000-284fffffff : Soft Reserved
>>>>     850000000-284fffffff : CXL Window 0
>>>>       850000000-280fffffff : region0
>>>>         850000000-284fffffff : dax0.0
>>>>           850000000-284fffffff : System RAM (kmem)
>>>>
>>>> "path":"\/platform\/ACPI0017:00\/root0\/decoder0.0\/region0\/dax_region0",
>>>> "id":0,
>>>> "size":"128.00 GiB (137.44 GB)",
>>>> "align":2097152
>>>>
>>>> [   35.961707] cxl-dax: cxl_dax_region_init()
>>>> [   35.961713] cxl-dax: registering driver.
>>>> [   35.961715] cxl-dax: dax_hmem work flushed.
>>>> [   35.961754] alloc_dev_dax_range:  dax0.0: alloc range[0]:
>>>> 0x000000850000000:0x000000284fffffff
>>>> [   35.976622] hmem: hmem_platform probe started.
>>>> [   35.980821] cxl_bus_probe: cxl_dax_region dax_region0: probe: 0
>>>> [   36.819566] hmem_platform hmem_platform.0: Soft Reserved not fully
>>>> contained in CXL; using HMEM
>>>> [   36.819569] hmem_register_device: hmem_platform hmem_platform.0:
>>>> registering CXL range: [mem 0x850000000-0x284fffffff flags 0x80000200]
>>>> [   36.934156] alloc_dax_region: hmem hmem.6: dax_region resource conflict
>>>> for [mem 0x850000000-0x284fffffff]
>>>> [   36.989310] hmem hmem.6: probe with driver hmem failed with error -12
>>>>
>>>> [5] When CXL_BUS = m (with CXL_ACPI, CXL_PCI, CXL_PORT, CXL_MEM = m),
>>>> DAX_CXL = m and DAX_HMEM = y the results are as expected. To validate the
>>>> REGISTER path, I forced REGISTER even in cases where SR completely
>>>> overlaps the CXL region as I did not have access to a system where the
>>>> CXL region range is smaller than the SR range.
>>>>
>>>> 850000000-284fffffff : Soft Reserved
>>>>     850000000-284fffffff : CXL Window 0
>>>>       850000000-280fffffff : region0
>>>>         850000000-284fffffff : dax6.0
>>>>           850000000-284fffffff : System RAM (kmem)
>>>>
>>>> "path":"\/platform\/hmem.6",
>>>> "id":6,
>>>> "size":"128.00 GiB (137.44 GB)",
>>>> "align":2097152
>>>>
>>>> [   30.897665] devm_cxl_add_dax_region: cxl_region region0: region0:
>>>> register dax_region0
>>>> [   30.921015] hmem: hmem_platform probe started.
>>>> [   31.017946] hmem_platform hmem_platform.0: Soft Reserved not fully
>>>> contained in CXL; using HMEM
>>>> [   31.056310] alloc_dev_dax_range:  dax6.0: alloc range[0]:
>>>> 0x0000000850000000:0x000000284fffffff
>>>> [   34.781516] cxl-dax: cxl_dax_region_init()
>>>> [   34.781522] cxl-dax: registering driver.
>>>> [   34.781523] cxl-dax: dax_hmem work flushed.
>>>> [   34.781549] alloc_dax_region: cxl_dax_region dax_region0: dax_region
>>>> resource conflict for [mem 0x850000000-0x284fffffff]
>>>> [   34.781552] cxl_bus_probe: cxl_dax_region dax_region0: probe: -12
>>>> [   34.781554] cxl_dax_region dax_region0: probe with driver cxl_dax_region
>>>> failed with error -12
>>>>
>>>> v6 updates:
>>>> - Patch 1-3 no changes.
>>>> - New Patches 4-5.
>>>> - (void *)res -> res.
>>>> - cxl_region_contains_soft_reserve -> region_contains_soft_reserve.
>>>> - New file include/cxl/cxl.h
>>>> - Introduced singleton workqueue.
>>>> - hmem to queue the work and cxl to flush.
>>>> - cxl_contains_soft_reserve() -> soft_reserve_has_cxl_match().
>>>> - Included descriptions for dax_cxl_mode.
>>>> - kzalloc -> kmalloc in add_soft_reserve_into_iomem()
>>>> - dax_cxl_mode is exported to CXL.
>>>> - Introduced hmem_register_cxl_device() for walking only CXL
>>>> intersected SR ranges the second time.
>>>
>>> During v5 review of this patch:
>>>
>>> [PATCH v5 6/7] dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges
>>>
>>> there was discussion around handling region teardown. It's not mentioned
>>> in the changelog, and the teardown is completely removed from the patch.
>>>
>>> The discussion seemed to be leaning towards not tearing down 'all', but
>>> it's not clear to me that we decided not to tear down anything - which
>>> this update now does.
>>>
>>> And, as you may be guessing, I'm seeing disabled regions with DAX children
>>> and figuring out what can be done with them.
>>>
>>> Can you explain the new approach so I can test against that intention?
>>>
>>> FYI - I am able to confirm the dax regions are back for no-soft-reserved
>>> case, and my basic hotplug flow works with v6.
>>>
>>> -- Alison
>>
>> Hi Alison,
>>
>> Thanks for the test and confirming the no-soft-reserved and hotplug cases
>> work.
>>
>> You're right that cxl_region_teardown_all() was removed in v6. I should have
>> called this out more clearly in the changelog. Here's what I learnt from v5
>> review. Correct me if I misunderstood.
>>
>> During v5 review, regarding dropping teardown (comments from Dan):
>>
>> "If we go with the alloc_dax_region() observation in my other mail it means
>> that the HPA space will already be claimed and cxl_dax_region_probe() will
>> fail. If we can get to that point of "all HMEM registered, and all CXL
>> regions failing to attach their
>> cxl_dax_region devices" that is a good stopping point. Then can decide if a
>> follow-on patch is needed to cleanup that state (cxl_region_teardown_all())
>> , or if it can just idle that way in the messy state and wait for userspace
>> to cleanup if it wants."
>>
>> https://lore.kernel.org/all/697aad9546542_30951007c@dwillia2-mobl4.notmuch/
>>
>> Also:
>>
>> "In other words, I thought total teardown would be simpler, but as the
>> feedback keeps coming in, I think that brings a different set of complexity.
>> So just inject failures for dax_cxl to trip over and then we can go further
>> later to effect total teardown if that proves to not be enough."
>>
>> https://lore.kernel.org/all/697a9d46b147e_309510027@dwillia2-mobl4.notmuch/
>>
>> The v6 approach replaces teardown with the alloc_dax_region() resource
>> exclusion in patch 5. When HMEM wins the ownership decision (REGISTER path),
>> it successfully claims the dax_region resource range first. When dax_cxl
>> later tries to probe, its alloc_dax_region() call hits a resource conflict
>> and fails, leaving the cxl_dax_region device in a disabled state.
>>
>> (There is a separate ordering issue when CXL is built-in and HMEM is a
>> module, where dax_cxl may claim the dax_region first as observed in
>> experiments [4] and [5], but that is an independent topic and might not be
>> relevant here.)
>>
>> So the disabled regions with DAX children you are seeing on the CXL side are
>> likely expected as Dan mentioned - they show that CXL tried to claim the
>> range but HMEM got there first. Though the cxl region remains committed, no
>> dax_region gets created for it because the HPA space is already taken.
> 
> Hi Smita,
> 
> The disable regions I'm seeing are the remnants of failed region assemblies
> where HMEM rightfully took over. So the take over is good, but the expected
> view shown way above and repasted below is not what I'm seeing. Case [3]
> is not the same as Case [2], but have a region btw the SR and DAX.
> 
> 
>>>> [2] With CONFIG_CXL_REGION disabled, all the resources are handled by
>>>> HMEM. Soft Reserved range shows up in /proc/iomem, no regions come up
>>>> and dax devices are created from HMEM.
>>>> 850000000-284fffffff : CXL Window 0
>>>>     850000000-284fffffff : Soft Reserved
>>>>       850000000-284fffffff : dax0.0
>>>>         850000000-284fffffff : System RAM (kmem)
>>>>
>>>> [3] Region assembly failure works same as [2].
>>>>
> 
> I posted a patch[1] that I think gets us to what is expected.
> FWIW I do agree with abandoning the teardown all approach. In this
> patch I still don't suggest tearing down the region. It can stay for
> 'forensics', but I do think we should make /proc/iomem accurately
> reflect the memory topology.
> 
> [1] https://lore.kernel.org/linux-cxl/20260212062250.1219043-1-alison.schofield@intel.com/
> 
> -- Alison

Sorry I missed this message. I will go through it.

I think the reason I wasn't seeing regions in /proc/iomem during my 
testing is that I was using both of your test patches together, the fake 
failure in cxl_region_sort_targets() and the cleanup patch that calls 
devm_release_action()->unregister_region() on attach_target() failure in 
cxl_add_to_region(). The second patch removes the region on assembly 
failure, which is why the iomem tree had no region in my case.

You are right, just with faking failure in cxl_region_sort_targets() the 
region will exist in iomem tree.

Thanks
Smita

> 
>>
>> Thanks
>> Smita
>>