[BUG] KHO handover hangs when memmap reserves PMEM

[BUG] KHO handover hangs when memmap reserves PMEM

[Chris Bainbridge]

Hello,

I am reporting a regression in KHO handover.

In the most recent Ubuntu 26.10 beta mini-ISO releases (2026-09-18 onwards),
the first kernel reserves RAM for the downloaded ISO with a memmap directive
such as memmap=<size>!4G, then kexecs a second kernel. On affected kernels, the
first kernel reaches "kexec_core: Starting new kernel", but the second kernel
produces no further output and QEMU spins at 100% CPU. On unaffected kernels,
the second kernel starts and finds /dev/pmem0 with the expected "Persistent
Memory (legacy)" range in /proc/iomem.

A source bisect identified 3f2ad90060f6 ("kho: adopt radix tree for preserved
memory tracking") as the first bad commit; reverting it at the bisected
revision restored the handover. The regression appears related to KHO metadata
being accessed after the receiving kernel's memmap directive changes how the
metadata's physical range is mapped.

#regzbot introduced: 3f2ad90060f6

I have an AI-generated candidate fix that changes 22 files. I have not attached
or published it: although the AI did extensive verification on both x64 and
arm64, I cannot review or explain the complete change, so I am not submitting
it as a fix. If one of you thinks an unreviewed candidate would be useful for
further analysis, please let me know.

Re: [BUG] KHO handover hangs when memmap reserves PMEM

[Pratyush Yadav]

Hi,

On Sat, Oct 03 2026, Chris Bainbridge wrote:

> Hello,
>
> I am reporting a regression in KHO handover.
>
> In the most recent Ubuntu 26.10 beta mini-ISO releases (2026-09-18 onwards),
> the first kernel reserves RAM for the downloaded ISO with a memmap directive
> such as memmap=<size>!4G, then kexecs a second kernel. On affected kernels, the
> first kernel reaches "kexec_core: Starting new kernel", but the second kernel
> produces no further output and QEMU spins at 100% CPU. On unaffected kernels,
> the second kernel starts and finds /dev/pmem0 with the expected "Persistent
> Memory (legacy)" range in /proc/iomem.

Just to confirm, the first kernel boots without memmap= right?

If so, I have seen this before. Michal (+Cc) found this originally in
our downstream test suite.

I think the main problem is that the radix tree pages can be anywhere in
memory. So on the first kernel, they can land in an area that the second
kernel's memmap= reserves. So kho_memory_init() might end up using
memory that memmap= reserved. I'm not sure why exactly the next kernel
doesn't produce any output, but IIRC memmap= memory is not mapped to the
direct map, so accessing those pages likely causes a page fault.

It is not easy to fix. We would need to make memmap= aware of KHO and
skip the radix tree pages, but that is pretty complicated to do for
early boot code. And honestly, I don't think it is worth it either.

So I'd suggest you turn KHO off for the kexec that adds memmap= if you
can.

>
> A source bisect identified 3f2ad90060f6 ("kho: adopt radix tree for preserved
> memory tracking") as the first bad commit; reverting it at the bisected
> revision restored the handover. The regression appears related to KHO metadata
> being accessed after the receiving kernel's memmap directive changes how the
> metadata's physical range is mapped.
>
> #regzbot introduced: 3f2ad90060f6
>
> I have an AI-generated candidate fix that changes 22 files. I have not attached
> or published it: although the AI did extensive verification on both x64 and
> arm64, I cannot review or explain the complete change, so I am not submitting
> it as a fix. If one of you thinks an unreviewed candidate would be useful for
> further analysis, please let me know.

-- 
Regards,
Pratyush Yadav

Re: [BUG] KHO handover hangs when memmap reserves PMEM

[Chris Bainbridge]

On Sun, Oct 04, 2026 at 03:39:25PM +0200, Pratyush Yadav wrote:
> 
> Just to confirm, the first kernel boots without memmap= right?

Yes.

> I think the main problem is that the radix tree pages can be anywhere in
> memory. So on the first kernel, they can land in an area that the second
> kernel's memmap= reserves. So kho_memory_init() might end up using
> memory that memmap= reserved. I'm not sure why exactly the next kernel
> doesn't produce any output, but IIRC memmap= memory is not mapped to the
> direct map, so accessing those pages likely causes a page fault.

That's correct, kho_memory_init() page faults. It happens before the
serial console is available, so there is no output.

Re: [BUG] KHO handover hangs when memmap reserves PMEM

[Pratyush Yadav]

On Sun, Oct 04 2026, Chris Bainbridge wrote:

> On Sun, Oct 04, 2026 at 03:39:25PM +0200, Pratyush Yadav wrote:
>> 
>> Just to confirm, the first kernel boots without memmap= right?
>
> Yes.
>
>> I think the main problem is that the radix tree pages can be anywhere in
>> memory. So on the first kernel, they can land in an area that the second
>> kernel's memmap= reserves. So kho_memory_init() might end up using
>> memory that memmap= reserved. I'm not sure why exactly the next kernel
>> doesn't produce any output, but IIRC memmap= memory is not mapped to the
>> direct map, so accessing those pages likely causes a page fault.
>
> That's correct, kho_memory_init() page faults. It happens before the
> serial console is available, so there is no output.

Yeah, then I think you need an extra kexec with KHO off to enable
memmap=. Or I guess hot-unplug the memory you'd like to memmap= early on
first boot, which should make sure no radix tree pages land there.

Apart from that, I don't have any good ideas to properly solve this
problem.

-- 
Regards,
Pratyush Yadav