* hunting memory corruption bug in 6.18.x
@ 2026-09-25 8:48 Nikola Ciprich
2026-09-25 10:05 ` Lorenzo Stoakes (ARM)
2026-09-25 12:13 ` Lorenzo Stoakes (ARM)
0 siblings, 2 replies; 5+ messages in thread
From: Nikola Ciprich @ 2026-09-25 8:48 UTC (permalink / raw)
To: linux-mm; +Cc: linux-kernel, akpm, david, ljs, nikola.ciprich
Hi,
I've been hunting a weird memory corruption bug for the last few weeks,
without success so far, so I'd like to report it and kindly ask for help.
We first hit it after a live VM migration between two KVM hosts:
suddenly some dynamic libraries in the host appeared to be corrupted:
Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 548: elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed!
(Later we also hit this with libcrypto.so.3 etc.) The files on disk
were OK; the problem seemed to exist only in RAM.
I'm fairly sure this is not hardware related: there were no ECC errors,
and we have since hit this (and similar issues, more on that below) on
multiple machines.
The problems started after we moved from 5.15.x to 6.18.x kernels.
Since then I've spent a lot of time trying to reproduce it on a lab
cluster, and we were able to trigger some corruption after days of
migrating VMs back and forth. At first I suspected the Intel ice driver,
for which I found similar reports, but we saw new problems even after
backporting fixes (and also with Mellanox cards).
So far we've hit three different kinds of problems, which may or may
not be related:
- .so library corruption right after VM migration
- VM crashes (or process crashes inside VMs), possibly related to
migration (those always happened during migration)
- host crashes due to kernel structure corruption (these happened
without any VM migration)
We first hit these problems with 6.18.31; the last crash I saw was
with 6.18.44.
All affected machines use AMD EPYC CPUs and act as KVM hosts; the OS
is AlmaLinux 9.
I suspect two subsystems that have seen a lot of changes:
- transparent hugepages
- NUMA balancing
(but those are just my guesses)
As a safety measure, we've disabled THP and NUMA balancing on all hosts.
I'm aware this is still a very vague report with a lot of guessing,
but my question is: has anybody hit similar problems with 6.18 or
newer kernels?
I see a lot of patches in every stable release, but simply trying
newer kernels doesn't seem efficient here. Deploying them is also
risky, since the hosts have to be emptied by migrating VMs off them
before reboot, and that migration itself may trigger more crashes.
None of the released or queued fixes for 6.18 seem to be directly
related.
I tried running my migration tests on hosts with KASAN enabled, and
also with SLUB debugging, but was never able to reproduce the problem
with those enabled (without them, I was able to hit issues within
days).
I'll start another round of migration tests in the lab, now with
6.18.54-rc1, but I still thought it would be good to report this and
ask here.
last but not least, here's kdump from last crash (this was not related
to any VM migration, but is very similar to another few crashes
we got):
[1924553.414736] Oops: general protection fault, probably for non-canonical address 0xfffffff0c930038: 0000 [#1] SMP NOPTI
[1924553.434800] CPU: 23 UID: 189 PID: 7538 Comm: pacemaker-contr Kdump: loaded Tainted: G E 6.18.44lb9.01 #1 PREEMPT(voluntary)
[1924553.456934] Tainted: [E]=UNSIGNED_MODULE
[1924553.465551] Hardware name: ASUSTeK COMPUTER INC. RS720A-E12-RS12/K14PP-D24 Series, BIOS 2305 11/21/2025
[1924553.484152] RIP: 0010:__d_lookup+0x4a/0xc0
[1924553.492878] Code: ff 48 89 c5 c1 e8 07 48 8d 1c c2 e8 60 8f d1 ff 48 8b 03 48 89 c3 48 83 e3 fe 48 83 f8 01 77 0a eb 2f 48 8b 1b 48 85 db 74 27 <39> 6b 18 75 f3 4c 8d 63 78 4c 89 e7 e8
d5 e1 7c 00 4c 39 6b 10 74
[1924553.525191] RSP: 0018:ff7532a13699fda0 EFLAGS: 00010212
[1924553.534986] RAX: 0fffffff0c930020 RBX: 0fffffff0c930020 RCX: 0000000000000000
[1924553.546679] RDX: ff2e6dbe0d9b6000 RSI: ff7532a13699fe60 RDI: ff2e6d1e4e630d80
[1924553.558367] RBP: 000000000b654440 R08: 0000000000002403 R09: 0000000000000179
[1924553.570026] R10: 000000000000000d R11: 0000000000000000 R12: 0000000001876e5c
[1924553.581601] R13: ff2e6d1e4e630d80 R14: ff7532a13699fe60 R15: 0000000000000000
[1924553.593115] FS: 00007ff8743aaa80(0000) GS:ff2e6e5e94c45000(0000) knlGS:0000000000000000
[1924553.605576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1924553.615611] CR2: 00007ffcd63a5000 CR3: 00000003ee840001 CR4: 0000000000771ef0
[1924553.627022] PKRU: 55555554
[1924553.633869] Call Trace:
[1924553.640353] <TASK>
[1924553.646369] d_lookup+0x27/0x50
[1924553.653366] lookup_dcache+0x1f/0x80
[1924553.660713] lookup_one_qstr_excl+0x1e/0xe0
[1924553.668589] ? preempt_schedule_common+0x2c/0x70
[1924553.676837] filename_create+0xc4/0x160
[1924553.684209] do_mkdirat+0x5a/0x190
[1924553.691050] __x64_sys_mkdir+0x42/0x60
[1924553.698163] do_syscall_64+0x64/0xbf0
[1924553.705145] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[1924553.713533] RIP: 0033:0x7ff8754ff08b
[1924553.720358] Code: 8b 05 91 bd 0f 00 41 bc ff ff ff ff 64 c7 00 16 00 00 00 e9 4f ff ff ff e8 12 f7 01 00 66 90 f3 0f 1e fa b8 53 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d
bd 0f 00 f7 d8 64 89 01 48
[1924553.748552] RSP: 002b:00007ffc05e5c148 EFLAGS: 00000246 ORIG_RAX: 0000000000000053
[1924553.759401] RAX: ffffffffffffffda RBX: 00005623cc0bd513 RCX: 00007ff8754ff08b
[1924553.769760] RDX: 000000000fde421b RSI: 00000000000001c0 RDI: 00005623cc0bd4f4
[1924553.780083] RBP: f49998db0aa753ff R08: 0000000000000004 R09: 0000000000000001
[1924553.790348] R10: 00007ff87587d000 R11: 0000000000000246 R12: 8421084210842109
[1924553.800604] R13: 00005623cc0bd513 R14: 00007ff8755bd740 R15: 000000000fde421b
[1924553.810819] </TASK>
I'll be very very gratefull for any hints here..
with best regards
nikola ciprich
PS: I tried to CC maintainers of suspected subsystems, but those are just my guesses,
so I hope I won't offend anyone.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hunting memory corruption bug in 6.18.x
2026-09-25 8:48 hunting memory corruption bug in 6.18.x Nikola Ciprich
@ 2026-09-25 10:05 ` Lorenzo Stoakes (ARM)
2026-09-25 12:13 ` Lorenzo Stoakes (ARM)
1 sibling, 0 replies; 5+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-09-25 10:05 UTC (permalink / raw)
To: Nikola Ciprich
Cc: linux-mm, linux-kernel, akpm, david, Dave Hansen, Mike Rapoport
+cc Dave, Mike
On Fri, Sep 25, 2026 at 10:48:26AM +0200, Nikola Ciprich wrote:
> Hi,
>
> I've been hunting a weird memory corruption bug for the last few weeks,
> without success so far, so I'd like to report it and kindly ask for help.
>
> We first hit it after a live VM migration between two KVM hosts:
> suddenly some dynamic libraries in the host appeared to be corrupted:
>
> Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 548: elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed!
>
> (Later we also hit this with libcrypto.so.3 etc.) The files on disk
> were OK; the problem seemed to exist only in RAM.
>
> I'm fairly sure this is not hardware related: there were no ECC errors,
> and we have since hit this (and similar issues, more on that below) on
> multiple machines.
>
> The problems started after we moved from 5.15.x to 6.18.x kernels.
I had an AI dig into this.
And to give a quick response before trying to wrangle/check what it said
into a coherent analysis, the TL;DR is it seems to be caused by some CPA
bugs we fixed recently in x86.
This should be fixed in 6.18.53+ could you test again with everything
re-enabled?
I'll reply again with something more detailed.
>
> Since then I've spent a lot of time trying to reproduce it on a lab
> cluster, and we were able to trigger some corruption after days of
> migrating VMs back and forth. At first I suspected the Intel ice driver,
> for which I found similar reports, but we saw new problems even after
> backporting fixes (and also with Mellanox cards).
>
> So far we've hit three different kinds of problems, which may or may
> not be related:
>
> - .so library corruption right after VM migration
> - VM crashes (or process crashes inside VMs), possibly related to
> migration (those always happened during migration)
> - host crashes due to kernel structure corruption (these happened
> without any VM migration)
>
> We first hit these problems with 6.18.31; the last crash I saw was
> with 6.18.44.
>
> All affected machines use AMD EPYC CPUs and act as KVM hosts; the OS
> is AlmaLinux 9.
>
> I suspect two subsystems that have seen a lot of changes:
>
> - transparent hugepages
> - NUMA balancing
>
> (but those are just my guesses)
>
> As a safety measure, we've disabled THP and NUMA balancing on all hosts.
>
> I'm aware this is still a very vague report with a lot of guessing,
> but my question is: has anybody hit similar problems with 6.18 or
> newer kernels?
>
> I see a lot of patches in every stable release, but simply trying
> newer kernels doesn't seem efficient here. Deploying them is also
> risky, since the hosts have to be emptied by migrating VMs off them
> before reboot, and that migration itself may trigger more crashes.
> None of the released or queued fixes for 6.18 seem to be directly
> related.
>
> I tried running my migration tests on hosts with KASAN enabled, and
> also with SLUB debugging, but was never able to reproduce the problem
> with those enabled (without them, I was able to hit issues within
> days).
>
> I'll start another round of migration tests in the lab, now with
> 6.18.54-rc1, but I still thought it would be good to report this and
> ask here.
>
> last but not least, here's kdump from last crash (this was not related
> to any VM migration, but is very similar to another few crashes
> we got):
>
> [1924553.414736] Oops: general protection fault, probably for non-canonical address 0xfffffff0c930038: 0000 [#1] SMP NOPTI
> [1924553.434800] CPU: 23 UID: 189 PID: 7538 Comm: pacemaker-contr Kdump: loaded Tainted: G E 6.18.44lb9.01 #1 PREEMPT(voluntary)
> [1924553.456934] Tainted: [E]=UNSIGNED_MODULE
> [1924553.465551] Hardware name: ASUSTeK COMPUTER INC. RS720A-E12-RS12/K14PP-D24 Series, BIOS 2305 11/21/2025
> [1924553.484152] RIP: 0010:__d_lookup+0x4a/0xc0
> [1924553.492878] Code: ff 48 89 c5 c1 e8 07 48 8d 1c c2 e8 60 8f d1 ff 48 8b 03 48 89 c3 48 83 e3 fe 48 83 f8 01 77 0a eb 2f 48 8b 1b 48 85 db 74 27 <39> 6b 18 75 f3 4c 8d 63 78 4c 89 e7 e8
> d5 e1 7c 00 4c 39 6b 10 74
> [1924553.525191] RSP: 0018:ff7532a13699fda0 EFLAGS: 00010212
> [1924553.534986] RAX: 0fffffff0c930020 RBX: 0fffffff0c930020 RCX: 0000000000000000
> [1924553.546679] RDX: ff2e6dbe0d9b6000 RSI: ff7532a13699fe60 RDI: ff2e6d1e4e630d80
> [1924553.558367] RBP: 000000000b654440 R08: 0000000000002403 R09: 0000000000000179
> [1924553.570026] R10: 000000000000000d R11: 0000000000000000 R12: 0000000001876e5c
> [1924553.581601] R13: ff2e6d1e4e630d80 R14: ff7532a13699fe60 R15: 0000000000000000
> [1924553.593115] FS: 00007ff8743aaa80(0000) GS:ff2e6e5e94c45000(0000) knlGS:0000000000000000
> [1924553.605576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1924553.615611] CR2: 00007ffcd63a5000 CR3: 00000003ee840001 CR4: 0000000000771ef0
> [1924553.627022] PKRU: 55555554
> [1924553.633869] Call Trace:
> [1924553.640353] <TASK>
> [1924553.646369] d_lookup+0x27/0x50
> [1924553.653366] lookup_dcache+0x1f/0x80
> [1924553.660713] lookup_one_qstr_excl+0x1e/0xe0
> [1924553.668589] ? preempt_schedule_common+0x2c/0x70
> [1924553.676837] filename_create+0xc4/0x160
> [1924553.684209] do_mkdirat+0x5a/0x190
> [1924553.691050] __x64_sys_mkdir+0x42/0x60
> [1924553.698163] do_syscall_64+0x64/0xbf0
> [1924553.705145] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [1924553.713533] RIP: 0033:0x7ff8754ff08b
> [1924553.720358] Code: 8b 05 91 bd 0f 00 41 bc ff ff ff ff 64 c7 00 16 00 00 00 e9 4f ff ff ff e8 12 f7 01 00 66 90 f3 0f 1e fa b8 53 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d
> bd 0f 00 f7 d8 64 89 01 48
> [1924553.748552] RSP: 002b:00007ffc05e5c148 EFLAGS: 00000246 ORIG_RAX: 0000000000000053
> [1924553.759401] RAX: ffffffffffffffda RBX: 00005623cc0bd513 RCX: 00007ff8754ff08b
> [1924553.769760] RDX: 000000000fde421b RSI: 00000000000001c0 RDI: 00005623cc0bd4f4
> [1924553.780083] RBP: f49998db0aa753ff R08: 0000000000000004 R09: 0000000000000001
> [1924553.790348] R10: 00007ff87587d000 R11: 0000000000000246 R12: 8421084210842109
> [1924553.800604] R13: 00005623cc0bd513 R14: 00007ff8755bd740 R15: 000000000fde421b
> [1924553.810819] </TASK>
>
> I'll be very very gratefull for any hints here..
Hi, I had a
>
> with best regards
>
> nikola ciprich
>
> PS: I tried to CC maintainers of suspected subsystems, but those are just my guesses,
> so I hope I won't offend anyone.
>
--
Cheers, Lorenzo
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hunting memory corruption bug in 6.18.x
2026-09-25 8:48 hunting memory corruption bug in 6.18.x Nikola Ciprich
2026-09-25 10:05 ` Lorenzo Stoakes (ARM)
@ 2026-09-25 12:13 ` Lorenzo Stoakes (ARM)
2026-09-26 5:58 ` Nikola Ciprich
1 sibling, 1 reply; 5+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-09-25 12:13 UTC (permalink / raw)
To: Nikola Ciprich
Cc: linux-mm, linux-kernel, akpm, david, Mike Rapoport, Dave Hansen,
Pedro Falcato, Kiryl Shutsemau
+cc various
Tl;DR before I dig in, I had an AI dig into the report (as they're
essentially superhuman at this kind of thing so always worth doing), and it
seems the recently fixed CPA bugs are likely to be the underlying cause
here.
EDIT: OK so I spent 2+ hrs analysing this :>))) but hopefully it's useful,
I wanted to make sure what the LLM came up with was vaguely sensible.
It's speculative, but I really think the below is the best explanation for
what you're observing.
And TL;DR is that 6.18.53 should fix it.
On Fri, Sep 25, 2026 at 10:48:26AM +0200, Nikola Ciprich wrote:
> Hi,
>
> I've been hunting a weird memory corruption bug for the last few weeks,
> without success so far, so I'd like to report it and kindly ask for help.
Thanks for the detailed report!
>
> We first hit it after a live VM migration between two KVM hosts:
> suddenly some dynamic libraries in the host appeared to be corrupted:
>
> Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 548: elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed!
>
> (Later we also hit this with libcrypto.so.3 etc.) The files on disk
> were OK; the problem seemed to exist only in RAM.
>
> I'm fairly sure this is not hardware related: there were no ECC errors,
> and we have since hit this (and similar issues, more on that below) on
> multiple machines.
>
> The problems started after we moved from 5.15.x to 6.18.x kernels.
It seems that commit 41d88484c71c ("x86/mm/pat: restore large ROX pages after
fragmentation") is the underlying cause (landed in 6.15).
It impacts CPA or 'Change Page Attributes' which is the means by which direct
mapping page table entries are updated to reflect underlying attribute changes
for ranges, often (and the motivation behind this change) read-only executable
ranges for e.g. JITters etc.
When it does this it sees if the range being changed can be 'collapsed' into a
huge page, i.e. mapped at PMD level for instance rather than PTE level to avoid
fragmentation of the direct map.
In particular the change introduces cpa_collapse_large_pages(), which frees
kernel page tables when it does this.
And this is problematic, because it did that without properly synchronising
against concurrent readers.
And I think in particular the issue here is the one fixed by Pedro in commit
1587d3394e25 ("x86/alternatives: Exclude text poking against
change_page_attr()").
>
> Since then I've spent a lot of time trying to reproduce it on a lab
> cluster, and we were able to trigger some corruption after days of
> migrating VMs back and forth. At first I suspected the Intel ice driver,
Yeah these race bugs can be VERY painful, sorry about that!
> for which I found similar reports, but we saw new problems even after
> backporting fixes (and also with Mellanox cards).
>
> So far we've hit three different kinds of problems, which may or may
> not be related:
>
> - .so library corruption right after VM migration
> - VM crashes (or process crashes inside VMs), possibly related to
> migration (those always happened during migration)
> - host crashes due to kernel structure corruption (these happened
> without any VM migration)
So there are two sides to the race: set_memory_rox() - triggered on module
load, ftrace trampoline creation and every new BPF 2M program pack.
The other side is execmem_force_rw() -> set_memory_[nx,rw]() (concurrent
module load or ftrace trampoline allocation) __text_poke() ->
vmalloc_to_page() for patching module text, kprobe slots, trampolines or
BPF.
Both are happening a lot at KVM host bringup (module autoload, per-VM
seccomp filters, perf, BPF probes, etc.
So this aligns with the theory.
>
> We first hit these problems with 6.18.31; the last crash I saw was
> with 6.18.44.
Yeah, the fact you didn't see an issue with 5.15 matches commit
41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation")
being the cause.
>
> All affected machines use AMD EPYC CPUs and act as KVM hosts; the OS
> is AlmaLinux 9.
>
> I suspect two subsystems that have seen a lot of changes:
>
> - transparent hugepages
> - NUMA balancing
>
> (but those are just my guesses)
>
> As a safety measure, we've disabled THP and NUMA balancing on all hosts.
Actually I think doing this doesn't actually save you at all, since the
collapse happens even without THP enabled, and NUMA balancing shouldn't
impact any of this.
>
> I'm aware this is still a very vague report with a lot of guessing,
> but my question is: has anybody hit similar problems with 6.18 or
> newer kernels?
Yeah, the description of a fix for this mentions something that seems
exactly like this bug:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1aac65f3e651334259ecb2a5f5ddb81c01f02599
Though note that that patch doesn't actually solve the problem, you need
fixes from 6.18.53 to resolve the bug:
Commit a1c7570cedd0 ("x86/mm/pat: Acquire init_mm write lock on collapse to avoid UAF")
Commit d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
These are prerequisites ^^^ for the actual fix for this vvv
Commit 1587d3394e25 ("x86/alternatives: Exclude text poking against change_page_attr()")
>
> I see a lot of patches in every stable release, but simply trying
> newer kernels doesn't seem efficient here. Deploying them is also
> risky, since the hosts have to be emptied by migrating VMs off them
> before reboot, and that migration itself may trigger more crashes.
> None of the released or queued fixes for 6.18 seem to be directly
> related.
>
> I tried running my migration tests on hosts with KASAN enabled, and
> also with SLUB debugging, but was never able to reproduce the problem
> with those enabled (without them, I was able to hit issues within
> days).
Ugh, unhelpful, but makes sense as it changes race windows.
>
> I'll start another round of migration tests in the lab, now with
> 6.18.54-rc1, but I still thought it would be good to report this and
> ask here.
>
Ah yeah you're already going to be testing the fixed series then :)
Obviously if the issue re-triggers there, back to the drawing board. But I
don't think it will.
> last but not least, here's kdump from last crash (this was not related
> to any VM migration, but is very similar to another few crashes
> we got):
>
> [1924553.414736] Oops: general protection fault, probably for non-canonical address 0xfffffff0c930038: 0000 [#1] SMP NOPTI
> [1924553.434800] CPU: 23 UID: 189 PID: 7538 Comm: pacemaker-contr Kdump: loaded Tainted: G E 6.18.44lb9.01 #1 PREEMPT(voluntary)
> [1924553.456934] Tainted: [E]=UNSIGNED_MODULE
> [1924553.465551] Hardware name: ASUSTeK COMPUTER INC. RS720A-E12-RS12/K14PP-D24 Series, BIOS 2305 11/21/2025
> [1924553.484152] RIP: 0010:__d_lookup+0x4a/0xc0
So __d_lookup() is where the invalid address oops happened.
> [1924553.492878] Code: ff 48 89 c5 c1 e8 07 48 8d 1c c2 e8 60 8f d1 ff 48 8b 03 48 89 c3 48 83 e3 fe 48 83 f8 01 77 0a eb 2f 48 8b 1b 48 85 db 74 27 <39> 6b 18 75 f3 4c 8d 63 78 4c 89 e7 e8
> d5 e1 7c 00 4c 39 6b 10 74
> [1924553.525191] RSP: 0018:ff7532a13699fda0 EFLAGS: 00010212
> [1924553.534986] RAX: 0fffffff0c930020 RBX: 0fffffff0c930020 RCX: 0000000000000000
> [1924553.546679] RDX: ff2e6dbe0d9b6000 RSI: ff7532a13699fe60 RDI: ff2e6d1e4e630d80
> [1924553.558367] RBP: 000000000b654440 R08: 0000000000002403 R09: 0000000000000179
> [1924553.570026] R10: 000000000000000d R11: 0000000000000000 R12: 0000000001876e5c
> [1924553.581601] R13: ff2e6d1e4e630d80 R14: ff7532a13699fe60 R15: 0000000000000000
> [1924553.593115] FS: 00007ff8743aaa80(0000) GS:ff2e6e5e94c45000(0000) knlGS:0000000000000000
> [1924553.605576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1924553.615611] CR2: 00007ffcd63a5000 CR3: 00000003ee840001 CR4: 0000000000771ef0
> [1924553.627022] PKRU: 55555554
> [1924553.633869] Call Trace:
> [1924553.640353] <TASK>
> [1924553.646369] d_lookup+0x27/0x50
> [1924553.653366] lookup_dcache+0x1f/0x80
> [1924553.660713] lookup_one_qstr_excl+0x1e/0xe0
> [1924553.668589] ? preempt_schedule_common+0x2c/0x70
> [1924553.676837] filename_create+0xc4/0x160
> [1924553.684209] do_mkdirat+0x5a/0x190
> [1924553.691050] __x64_sys_mkdir+0x42/0x60
> [1924553.698163] do_syscall_64+0x64/0xbf0
> [1924553.705145] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [1924553.713533] RIP: 0033:0x7ff8754ff08b
> [1924553.720358] Code: 8b 05 91 bd 0f 00 41 bc ff ff ff ff 64 c7 00 16 00 00 00 e9 4f ff ff ff e8 12 f7 01 00 66 90 f3 0f 1e fa b8 53 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d
> bd 0f 00 f7 d8 64 89 01 48
> [1924553.748552] RSP: 002b:00007ffc05e5c148 EFLAGS: 00000246 ORIG_RAX: 0000000000000053
> [1924553.759401] RAX: ffffffffffffffda RBX: 00005623cc0bd513 RCX: 00007ff8754ff08b
> [1924553.769760] RDX: 000000000fde421b RSI: 00000000000001c0 RDI: 00005623cc0bd4f4
> [1924553.780083] RBP: f49998db0aa753ff R08: 0000000000000004 R09: 0000000000000001
> [1924553.790348] R10: 00007ff87587d000 R11: 0000000000000246 R12: 8421084210842109
> [1924553.800604] R13: 00005623cc0bd513 R14: 00007ff8755bd740 R15: 000000000fde421b
> [1924553.810819] </TASK>
So the LLM went to town on this and it's quite interesting.
The code (6.18.44) disassembles to:
struct dentry *__d_lookup(const struct dentry *parent, const struct qstr *name)
{
...
hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
Which is:
#define hlist_bl_for_each_entry_rcu(tpos, pos, head, member) \
for (pos = hlist_bl_first_rcu(head); \
pos && \
({ tpos = hlist_bl_entry(pos, typeof(*tpos), member); 1; }); \
pos = rcu_dereference_raw(pos->next))
And:
static inline struct hlist_bl_node *hlist_bl_first_rcu(struct hlist_bl_head *h)
{
return (struct hlist_bl_node *)
((unsigned long)rcu_dereference_check(h->first, hlist_bl_is_locked(h)) & ~LIST_BL_LOCKMASK);
}
And:
static inline bool hlist_bl_is_locked(struct hlist_bl_head *b)
{
return bit_spin_is_locked(0, (unsigned long *)b);
}
mov (%rbx),%rax ; rax = h->first = 0x0fffffff0c930020
mov %rax,%rbx
and $-2,%rbx ; strip hlist_bl lock bit (no-op, bit 0 clear)
cmp $1,%rax ; hlist_bl_is_locked()
ja body ; non-empty, enter loop
...
if (dentry->d_name.hash != hash)
continue;
body:
cmp %ebp,0x18(%rbx) ; <-- FAULT: deref of dentry->d_name.hash
So dentry is corrupted.
The code and registers are consistent with this being the first iteration
of the loop, which you'd expect with corrupted dentry.
Looking further back in the code:
struct dentry *__d_lookup(const struct dentry *parent, const struct qstr *name)
{
...
struct hlist_bl_head *b = d_hash(hash);
And:
static inline struct hlist_bl_head *d_hash(unsigned long hashlen)
{
return runtime_const_ptr(dentry_hashtable) +
runtime_const_shift_right_32(hashlen, d_hash_shift);
}
Only the trailing ff is in the code output from the splat but the movabs is
there -> RDX: so:
movabs $0xff2e6dbe0d9b6000,%rdx ; runtime_const_ptr(dentry_hashtable)
mov %rax, %rbp
shr $0x7, %eax ; runtime_const_shift_right_32(hashlen, d_hash_shift);
lea (%rdx, %rax, 8), %rbx ; bucket = &dentry_hashtable[hash >> 7]
(The 8 is multiplying the size of the 8 byte pointers)
Note that rbp retains RAX's value = 0xb654440 (not clobbered elsewehre), so
the hlist_bl_head bucket pointer is
0xff2e6dbe0d9b6000 + (0xb654440 >> 7) * 8
So:
b = 0xff2e6dbe0e51b440
This matters, because in hlist_bl_first_rcu() this pointer is treated as a
valid struct hlist_bl_head pointer:
struct hlist_bl_head {
struct hlist_bl_node *first;
};
And the data at 0xff2e6dbe0e51b440 contains 0x0fffffff0c930020 (rbx), which
is assumed to be a valid struct hlist_bl_node * embedded in a dentry.
IOW, RDX contains the dentry_hashtable pointer:
static struct hlist_bl_head *dentry_hashtable __ro_after_init __used;
That is allocated in dcache_init_early() and never freed:
static void __init dcache_init_early(void)
{
...
dentry_hashtable =
alloc_large_system_hash("Dentry cache",
sizeof(struct hlist_bl_head),
dhash_entries,
13,
HASH_EARLY | HASH_ZERO,
&d_hash_shift,
NULL,
2,
0);
...
}
So that allocate has to be legit, somehow the data there got corrupted.
This is allocated early by memblock.
d_hash_shift = 7 = 32 - lg(entries), so entries = 25, and 2^25 entries of 8
bytes each = a 256 MiB table.
So it's legit data that got corrupted.
RDI contains the parent dentry at 0xff2e6d1e4e630d80 (the disassembly at
__d_lookup() confirms).
KASLR makes things tricky but this is most definitely a slab allocation in
the direct map and the stack (VMAP_STACK) is at 0xff7532a13699fda0 (rsp)
~70 TiB higher which sits at least 10 TiB padded above the direct map so
it's safe to say that this is in the direct map.
And the corrupted value (rbx) 0x0fffffff0c930020 is interesting - it's an
exactly a x86-64 swap softleaf value:
__swp_type() = val >> (64 - SWP_TYPE_BITS=5) = val >> 59 = 1
__swp_offset() = (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT) = (~val << 6 >> 14)
= 0x79b67f
I.e. it's a swap softleaf entry 0x79b67f 4 KiB pages into the swap = ~30.4 GiB.
It's also the _second_ swap in the system (Nikola - if you have a 2nd swap
on the reporting system of >=~30 GiB that kinda confirms it).
Also, interstingly, bit 5 is set, which is one of the bits allowed to be
set (possibly by hardware) in the swap entry.
__swp_entry() clears bits 0-8 and only bits 1-3 are software bits in a
non-present PTE.
This is _PAGE_ACCESSED (1 << _PAGE_BIT_ACCESSED = 5) = 0x20. So it makes
sense that hardware might have set it.
In effect - every single bit is exactly how it should be for a valid swap
entry (since commit 00839ee3b299 ("x86/mm: Move swap offset/type up in PTE
to work around erratum"))..
It seems more than a coincidence :)
This speaks to some kind of memory corruption that has resulted in a store
to an arbitrary physical address that happens to be the dentry.
Now looking to the proposed CPA cause (ultimately the thing solved by
So the speculated race here is:
pfn_exec: pfn of the module text page CPU A is changing attributes on
pfn_dh: pfn of the dentry hash table page the swap PTE ends up in
CPU A: set_memory_nx() (text_poke) CPU B: set_memory_rox() (module load)
------------------------------------ -----------------------------------------
__change_page_attr()
kpte = lookup (lockless)
<preempted>
cpa_collapse_large_pages()
set_pmd(leaf)
__free_pages(old PTE table)
some process: pte_alloc() gets
that page as a user page table
UAF!!! Writing into arbitrary memory
set_pte_atomic(kpte,
pfn_pte(pfn_exec, prot)) <- lands in that page table:
entry -> pfn_exec (module text)
process faults on it, GUP follows
it, later zap frees the data page at
pfn_exec while it is still live module
text!!!
That data page is reallocated as a PMD
table.
text_poke() keeps writing
code bytes into it -> entries
with arbitrary pfns, one of them
pfn_dh
reclaim: try_to_unmap_one()
pte_offset_map() -> __va(pfn_dh)
set_pte_at(swap PTE) <- dentry
hash table
(The reason it needs to be interpreted as a PMD page table is otherwise
reclaim wouldn't be trying to write a swap PTE entry into it).
And this is exactly what commit 1587d3394e25 ("x86/alternatives: Exclude
text poking against change_page_attr()") protects against.
Yes it's out on a limb (and I spent FAR TOO LONG going through this
analysis) but there's really no other sensible explanation as to why dentry
data got corrupted like that to that exact shape.
Which goes to show how nasty this kind of data corruption issue can be.
>
> I'll be very very gratefull for any hints here..
As above :)
>
> with best regards
>
> nikola ciprich
>
> PS: I tried to CC maintainers of suspected subsystems, but those are just my guesses,
> so I hope I won't offend anyone.
>
Don't worry about that, I think we're all happy to get legit bug
reports. We just might be too busy to reply right away :)
The LLM added on some hints for confirmation of this:
Schlopp>>
Log greps, across all affected hosts and all boots, not just the ones
that crashed:
grep -i 'Bad page map' /var/log/messages*
grep -i 'bad pmd' /var/log/messages*
grep -i 'bad pud' /var/log/messages*
grep -i 'Bad page state' /var/log/messages*
grep -i 'CPA: called for zero pte' /var/log/messages*
Any of these, particularly "bad pmd", is direct evidence that a freed
kernel PTE table was reused as a user page table. "CPA: called for zero
pte" would be the CPA walker itself tripping over a collapsed mapping.
Questions:
1. swapon --show on the host, and inside the guests. Is there a swap
device with index 1 and a size of at least roughly 30.5GiB? That
tells us whether the corrupt word is a host swap PTE or a guest one,
which distinguishes case A from case B above.
2. From the vmcore, dump the 4K page containing 0xff2e6dbe0e51b440. If
the neighbouring words are also PTE-shaped, the page was being used
as a page table and the diagnosis above is confirmed. If only the
one word is corrupt, it was a single stray store.
3. What are CONFIG_DEBUG_VM, init_on_alloc, init_on_free and
page_poison set to in the production build versus the KASAN build?
free_page_is_bad() is gated on is_check_pages_enabled(), which needs
CONFIG_DEBUG_VM, so the production kernel would not report the bad
free even if it happened.
4. Are any of the crashing guests Windows, and is hv-tlbflush set on
them? That decides whether 26505e1b5b54 matters for you.
5. Has any corruption occurred since THP was disabled? If yes, that
supports the CPA race over your THP theory.
6. When you run the 6.18.54-rc1 test, please note that 6.18.52 is only
a partial fix, so if you have any results from a 6.18.52 kernel they
should not be treated as a clean run.
<< Schlopp
But really a run against 6.18.53 being OK under heavy testing for several
days should confirm it also.
If it turns out it's not this then back to the drawing board I guess! Let
us know.
--
Cheers, Lorenzo
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hunting memory corruption bug in 6.18.x
2026-09-25 12:13 ` Lorenzo Stoakes (ARM)
@ 2026-09-26 5:58 ` Nikola Ciprich
2026-09-26 9:32 ` Lorenzo Stoakes (ARM)
0 siblings, 1 reply; 5+ messages in thread
From: Nikola Ciprich @ 2026-09-26 5:58 UTC (permalink / raw)
To: Lorenzo Stoakes (ARM)
Cc: linux-mm, linux-kernel, akpm, david, Mike Rapoport, Dave Hansen,
Pedro Falcato, Kiryl Shutsemau, Nikola Ciprich
Hello Lorenzo (and others),
thank you for your time looking into this..
(replies inline)
On Fri, Sep 25, 2026 at 01:13:52PM +0100, Lorenzo Stoakes (ARM) wrote:
> +cc various
>
> Tl;DR before I dig in, I had an AI dig into the report (as they're
> essentially superhuman at this kind of thing so always worth doing), and it
> seems the recently fixed CPA bugs are likely to be the underlying cause
> here.
>
> EDIT: OK so I spent 2+ hrs analysing this :>))) but hopefully it's useful,
> I wanted to make sure what the LLM came up with was vaguely sensible.
>
> It's speculative, but I really think the below is the best explanation for
> what you're observing.
>
> And TL;DR is that 6.18.53 should fix it.
>
> On Fri, Sep 25, 2026 at 10:48:26AM +0200, Nikola Ciprich wrote:
.. truncated ..
> >
> > The problems started after we moved from 5.15.x to 6.18.x kernels.
>
> It seems that commit 41d88484c71c ("x86/mm/pat: restore large ROX pages after
> fragmentation") is the underlying cause (landed in 6.15).
>
> It impacts CPA or 'Change Page Attributes' which is the means by which direct
> mapping page table entries are updated to reflect underlying attribute changes
> for ranges, often (and the motivation behind this change) read-only executable
> ranges for e.g. JITters etc.
>
> When it does this it sees if the range being changed can be 'collapsed' into a
> huge page, i.e. mapped at PMD level for instance rather than PTE level to avoid
> fragmentation of the direct map.
>
> In particular the change introduces cpa_collapse_large_pages(), which frees
> kernel page tables when it does this.
>
> And this is problematic, because it did that without properly synchronising
> against concurrent readers.
>
> And I think in particular the issue here is the one fixed by Pedro in commit
> 1587d3394e25 ("x86/alternatives: Exclude text poking against
> change_page_attr()").
>
> >
> > Since then I've spent a lot of time trying to reproduce it on a lab
> > cluster, and we were able to trigger some corruption after days of
> > migrating VMs back and forth. At first I suspected the Intel ice driver,
>
> Yeah these race bugs can be VERY painful, sorry about that!
>
> > for which I found similar reports, but we saw new problems even after
> > backporting fixes (and also with Mellanox cards).
> >
> > So far we've hit three different kinds of problems, which may or may
> > not be related:
> >
> > - .so library corruption right after VM migration
> > - VM crashes (or process crashes inside VMs), possibly related to
> > migration (those always happened during migration)
> > - host crashes due to kernel structure corruption (these happened
> > without any VM migration)
>
> So there are two sides to the race: set_memory_rox() - triggered on module
> load, ftrace trampoline creation and every new BPF 2M program pack.
>
> The other side is execmem_force_rw() -> set_memory_[nx,rw]() (concurrent
> module load or ftrace trampoline allocation) __text_poke() ->
> vmalloc_to_page() for patching module text, kprobe slots, trampolines or
> BPF.
>
> Both are happening a lot at KVM host bringup (module autoload, per-VM
> seccomp filters, perf, BPF probes, etc.
one note here, at least last mentioned crash (with 6.18.44) happened with
host running only windows guest, in general we're seeing those problems
mosly with windows VM hosting machines.. so maybe they're triggerng the
problem with some other, but similar mechanism?
>
> So this aligns with the theory.
>
> >
> > We first hit these problems with 6.18.31; the last crash I saw was
> > with 6.18.44.
>
> Yeah, the fact you didn't see an issue with 5.15 matches commit
> 41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation")
> being the cause.
>
> >
> > All affected machines use AMD EPYC CPUs and act as KVM hosts; the OS
> > is AlmaLinux 9.
> >
> > I suspect two subsystems that have seen a lot of changes:
> >
> > - transparent hugepages
> > - NUMA balancing
> >
> > (but those are just my guesses)
> >
> > As a safety measure, we've disabled THP and NUMA balancing on all hosts.
>
> Actually I think doing this doesn't actually save you at all, since the
> collapse happens even without THP enabled, and NUMA balancing shouldn't
> impact any of this.
>
> >
> > I'm aware this is still a very vague report with a lot of guessing,
> > but my question is: has anybody hit similar problems with 6.18 or
> > newer kernels?
>
> Yeah, the description of a fix for this mentions something that seems
> exactly like this bug:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1aac65f3e651334259ecb2a5f5ddb81c01f02599
>
> Though note that that patch doesn't actually solve the problem, you need
> fixes from 6.18.53 to resolve the bug:
>
> Commit a1c7570cedd0 ("x86/mm/pat: Acquire init_mm write lock on collapse to avoid UAF")
> Commit d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
>
> These are prerequisites ^^^ for the actual fix for this vvv
>
> Commit 1587d3394e25 ("x86/alternatives: Exclude text poking against change_page_attr()")
>
> >
> > I see a lot of patches in every stable release, but simply trying
> > newer kernels doesn't seem efficient here. Deploying them is also
> > risky, since the hosts have to be emptied by migrating VMs off them
> > before reboot, and that migration itself may trigger more crashes.
> > None of the released or queued fixes for 6.18 seem to be directly
> > related.
> >
> > I tried running my migration tests on hosts with KASAN enabled, and
> > also with SLUB debugging, but was never able to reproduce the problem
> > with those enabled (without them, I was able to hit issues within
> > days).
>
> Ugh, unhelpful, but makes sense as it changes race windows.
>
> >
> > I'll start another round of migration tests in the lab, now with
> > 6.18.54-rc1, but I still thought it would be good to report this and
> > ask here.
> >
>
> Ah yeah you're already going to be testing the fixed series then :)
>
> Obviously if the issue re-triggers there, back to the drawing board. But I
> don't think it will.
>
> > last but not least, here's kdump from last crash (this was not related
> > to any VM migration, but is very similar to another few crashes
> > we got):
> >
> > [1924553.414736] Oops: general protection fault, probably for non-canonical address 0xfffffff0c930038: 0000 [#1] SMP NOPTI
> > [1924553.434800] CPU: 23 UID: 189 PID: 7538 Comm: pacemaker-contr Kdump: loaded Tainted: G E 6.18.44lb9.01 #1 PREEMPT(voluntary)
> > [1924553.456934] Tainted: [E]=UNSIGNED_MODULE
> > [1924553.465551] Hardware name: ASUSTeK COMPUTER INC. RS720A-E12-RS12/K14PP-D24 Series, BIOS 2305 11/21/2025
> > [1924553.484152] RIP: 0010:__d_lookup+0x4a/0xc0
>
> So __d_lookup() is where the invalid address oops happened.
>
> > [1924553.492878] Code: ff 48 89 c5 c1 e8 07 48 8d 1c c2 e8 60 8f d1 ff 48 8b 03 48 89 c3 48 83 e3 fe 48 83 f8 01 77 0a eb 2f 48 8b 1b 48 85 db 74 27 <39> 6b 18 75 f3 4c 8d 63 78 4c 89 e7 e8
> > d5 e1 7c 00 4c 39 6b 10 74
> > [1924553.525191] RSP: 0018:ff7532a13699fda0 EFLAGS: 00010212
> > [1924553.534986] RAX: 0fffffff0c930020 RBX: 0fffffff0c930020 RCX: 0000000000000000
> > [1924553.546679] RDX: ff2e6dbe0d9b6000 RSI: ff7532a13699fe60 RDI: ff2e6d1e4e630d80
> > [1924553.558367] RBP: 000000000b654440 R08: 0000000000002403 R09: 0000000000000179
> > [1924553.570026] R10: 000000000000000d R11: 0000000000000000 R12: 0000000001876e5c
> > [1924553.581601] R13: ff2e6d1e4e630d80 R14: ff7532a13699fe60 R15: 0000000000000000
> > [1924553.593115] FS: 00007ff8743aaa80(0000) GS:ff2e6e5e94c45000(0000) knlGS:0000000000000000
> > [1924553.605576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [1924553.615611] CR2: 00007ffcd63a5000 CR3: 00000003ee840001 CR4: 0000000000771ef0
> > [1924553.627022] PKRU: 55555554
> > [1924553.633869] Call Trace:
> > [1924553.640353] <TASK>
> > [1924553.646369] d_lookup+0x27/0x50
> > [1924553.653366] lookup_dcache+0x1f/0x80
> > [1924553.660713] lookup_one_qstr_excl+0x1e/0xe0
> > [1924553.668589] ? preempt_schedule_common+0x2c/0x70
> > [1924553.676837] filename_create+0xc4/0x160
> > [1924553.684209] do_mkdirat+0x5a/0x190
> > [1924553.691050] __x64_sys_mkdir+0x42/0x60
> > [1924553.698163] do_syscall_64+0x64/0xbf0
> > [1924553.705145] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > [1924553.713533] RIP: 0033:0x7ff8754ff08b
> > [1924553.720358] Code: 8b 05 91 bd 0f 00 41 bc ff ff ff ff 64 c7 00 16 00 00 00 e9 4f ff ff ff e8 12 f7 01 00 66 90 f3 0f 1e fa b8 53 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d
> > bd 0f 00 f7 d8 64 89 01 48
> > [1924553.748552] RSP: 002b:00007ffc05e5c148 EFLAGS: 00000246 ORIG_RAX: 0000000000000053
> > [1924553.759401] RAX: ffffffffffffffda RBX: 00005623cc0bd513 RCX: 00007ff8754ff08b
> > [1924553.769760] RDX: 000000000fde421b RSI: 00000000000001c0 RDI: 00005623cc0bd4f4
> > [1924553.780083] RBP: f49998db0aa753ff R08: 0000000000000004 R09: 0000000000000001
> > [1924553.790348] R10: 00007ff87587d000 R11: 0000000000000246 R12: 8421084210842109
> > [1924553.800604] R13: 00005623cc0bd513 R14: 00007ff8755bd740 R15: 000000000fde421b
> > [1924553.810819] </TASK>
>
> So the LLM went to town on this and it's quite interesting.
>
> The code (6.18.44) disassembles to:
>
> struct dentry *__d_lookup(const struct dentry *parent, const struct qstr *name)
> {
> ...
>
> hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
>
> Which is:
>
> #define hlist_bl_for_each_entry_rcu(tpos, pos, head, member) \
> for (pos = hlist_bl_first_rcu(head); \
> pos && \
> ({ tpos = hlist_bl_entry(pos, typeof(*tpos), member); 1; }); \
> pos = rcu_dereference_raw(pos->next))
>
> And:
>
> static inline struct hlist_bl_node *hlist_bl_first_rcu(struct hlist_bl_head *h)
> {
> return (struct hlist_bl_node *)
> ((unsigned long)rcu_dereference_check(h->first, hlist_bl_is_locked(h)) & ~LIST_BL_LOCKMASK);
> }
>
> And:
>
> static inline bool hlist_bl_is_locked(struct hlist_bl_head *b)
> {
> return bit_spin_is_locked(0, (unsigned long *)b);
> }
>
> mov (%rbx),%rax ; rax = h->first = 0x0fffffff0c930020
>
> mov %rax,%rbx
> and $-2,%rbx ; strip hlist_bl lock bit (no-op, bit 0 clear)
> cmp $1,%rax ; hlist_bl_is_locked()
> ja body ; non-empty, enter loop
> ...
>
> if (dentry->d_name.hash != hash)
> continue;
> body:
> cmp %ebp,0x18(%rbx) ; <-- FAULT: deref of dentry->d_name.hash
>
> So dentry is corrupted.
>
> The code and registers are consistent with this being the first iteration
> of the loop, which you'd expect with corrupted dentry.
>
> Looking further back in the code:
>
> struct dentry *__d_lookup(const struct dentry *parent, const struct qstr *name)
> {
> ...
> struct hlist_bl_head *b = d_hash(hash);
>
> And:
>
> static inline struct hlist_bl_head *d_hash(unsigned long hashlen)
> {
> return runtime_const_ptr(dentry_hashtable) +
> runtime_const_shift_right_32(hashlen, d_hash_shift);
> }
>
> Only the trailing ff is in the code output from the splat but the movabs is
> there -> RDX: so:
>
> movabs $0xff2e6dbe0d9b6000,%rdx ; runtime_const_ptr(dentry_hashtable)
> mov %rax, %rbp
> shr $0x7, %eax ; runtime_const_shift_right_32(hashlen, d_hash_shift);
> lea (%rdx, %rax, 8), %rbx ; bucket = &dentry_hashtable[hash >> 7]
>
> (The 8 is multiplying the size of the 8 byte pointers)
>
> Note that rbp retains RAX's value = 0xb654440 (not clobbered elsewehre), so
> the hlist_bl_head bucket pointer is
>
> 0xff2e6dbe0d9b6000 + (0xb654440 >> 7) * 8
>
> So:
>
> b = 0xff2e6dbe0e51b440
>
> This matters, because in hlist_bl_first_rcu() this pointer is treated as a
> valid struct hlist_bl_head pointer:
>
> struct hlist_bl_head {
> struct hlist_bl_node *first;
> };
>
> And the data at 0xff2e6dbe0e51b440 contains 0x0fffffff0c930020 (rbx), which
> is assumed to be a valid struct hlist_bl_node * embedded in a dentry.
>
> IOW, RDX contains the dentry_hashtable pointer:
>
> static struct hlist_bl_head *dentry_hashtable __ro_after_init __used;
>
> That is allocated in dcache_init_early() and never freed:
>
> static void __init dcache_init_early(void)
> {
> ...
> dentry_hashtable =
> alloc_large_system_hash("Dentry cache",
> sizeof(struct hlist_bl_head),
> dhash_entries,
> 13,
> HASH_EARLY | HASH_ZERO,
> &d_hash_shift,
> NULL,
> 2,
> 0);
> ...
> }
>
> So that allocate has to be legit, somehow the data there got corrupted.
>
> This is allocated early by memblock.
>
> d_hash_shift = 7 = 32 - lg(entries), so entries = 25, and 2^25 entries of 8
> bytes each = a 256 MiB table.
>
> So it's legit data that got corrupted.
>
> RDI contains the parent dentry at 0xff2e6d1e4e630d80 (the disassembly at
> __d_lookup() confirms).
>
> KASLR makes things tricky but this is most definitely a slab allocation in
> the direct map and the stack (VMAP_STACK) is at 0xff7532a13699fda0 (rsp)
> ~70 TiB higher which sits at least 10 TiB padded above the direct map so
> it's safe to say that this is in the direct map.
>
> And the corrupted value (rbx) 0x0fffffff0c930020 is interesting - it's an
> exactly a x86-64 swap softleaf value:
>
> __swp_type() = val >> (64 - SWP_TYPE_BITS=5) = val >> 59 = 1
> __swp_offset() = (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT) = (~val << 6 >> 14)
> = 0x79b67f
>
> I.e. it's a swap softleaf entry 0x79b67f 4 KiB pages into the swap = ~30.4 GiB.
>
> It's also the _second_ swap in the system (Nikola - if you have a 2nd swap
> on the reporting system of >=~30 GiB that kinda confirms it).
I suspect this may be a bit of a red herring...
actually there is NO swap on that machine, also there were no linux guests.. so
that might just be a coincidence? not sure if it changes anything..
>
> Also, interstingly, bit 5 is set, which is one of the bits allowed to be
> set (possibly by hardware) in the swap entry.
>
> __swp_entry() clears bits 0-8 and only bits 1-3 are software bits in a
> non-present PTE.
>
> This is _PAGE_ACCESSED (1 << _PAGE_BIT_ACCESSED = 5) = 0x20. So it makes
> sense that hardware might have set it.
>
> In effect - every single bit is exactly how it should be for a valid swap
> entry (since commit 00839ee3b299 ("x86/mm: Move swap offset/type up in PTE
> to work around erratum"))..
>
> It seems more than a coincidence :)
>
> This speaks to some kind of memory corruption that has resulted in a store
> to an arbitrary physical address that happens to be the dentry.
>
> Now looking to the proposed CPA cause (ultimately the thing solved by
>
>
> So the speculated race here is:
>
> pfn_exec: pfn of the module text page CPU A is changing attributes on
> pfn_dh: pfn of the dentry hash table page the swap PTE ends up in
>
> CPU A: set_memory_nx() (text_poke) CPU B: set_memory_rox() (module load)
> ------------------------------------ -----------------------------------------
> __change_page_attr()
> kpte = lookup (lockless)
> <preempted>
> cpa_collapse_large_pages()
> set_pmd(leaf)
> __free_pages(old PTE table)
>
> some process: pte_alloc() gets
> that page as a user page table
>
> UAF!!! Writing into arbitrary memory
> set_pte_atomic(kpte,
> pfn_pte(pfn_exec, prot)) <- lands in that page table:
> entry -> pfn_exec (module text)
>
> process faults on it, GUP follows
> it, later zap frees the data page at
> pfn_exec while it is still live module
> text!!!
>
> That data page is reallocated as a PMD
> table.
>
> text_poke() keeps writing
> code bytes into it -> entries
> with arbitrary pfns, one of them
> pfn_dh
>
> reclaim: try_to_unmap_one()
> pte_offset_map() -> __va(pfn_dh)
> set_pte_at(swap PTE) <- dentry
> hash table
>
> (The reason it needs to be interpreted as a PMD page table is otherwise
> reclaim wouldn't be trying to write a swap PTE entry into it).
>
> And this is exactly what commit 1587d3394e25 ("x86/alternatives: Exclude
> text poking against change_page_attr()") protects against.
>
> Yes it's out on a limb (and I spent FAR TOO LONG going through this
> analysis) but there's really no other sensible explanation as to why dentry
> data got corrupted like that to that exact shape.
>
> Which goes to show how nasty this kind of data corruption issue can be.
>
> >
> > I'll be very very gratefull for any hints here..
>
> As above :)
>
> >
> > with best regards
> >
> > nikola ciprich
> >
> > PS: I tried to CC maintainers of suspected subsystems, but those are just my guesses,
> > so I hope I won't offend anyone.
> >
>
> Don't worry about that, I think we're all happy to get legit bug
> reports. We just might be too busy to reply right away :)
>
> The LLM added on some hints for confirmation of this:
>
> Schlopp>>
>
> Log greps, across all affected hosts and all boots, not just the ones
> that crashed:
>
> grep -i 'Bad page map' /var/log/messages*
> grep -i 'bad pmd' /var/log/messages*
> grep -i 'bad pud' /var/log/messages*
> grep -i 'Bad page state' /var/log/messages*
> grep -i 'CPA: called for zero pte' /var/log/messages*
not a single occurance (this machine uses journal, but I checked those
and no such messages.. in general i tend to check dmesg and system logs
a lot, so I'd have already reported such messages..
>
> Any of these, particularly "bad pmd", is direct evidence that a freed
> kernel PTE table was reused as a user page table. "CPA: called for zero
> pte" would be the CPA walker itself tripping over a collapsed mapping.
>
> Questions:
>
> 1. swapon --show on the host, and inside the guests. Is there a swap
> device with index 1 and a size of at least roughly 30.5GiB? That
> tells us whether the corrupt word is a host swap PTE or a guest one,
> which distinguishes case A from case B above.
>
> 2. From the vmcore, dump the 4K page containing 0xff2e6dbe0e51b440. If
> the neighbouring words are also PTE-shaped, the page was being used
> as a page table and the diagnosis above is confirmed. If only the
> one word is corrupt, it was a single stray store.
unfortunately I don't have full vmcore from that crash, as it didn't fit
to /var/crash, backtrace I posted is from vmcore-dmesg.txt so can't confirm
that..
>
> 3. What are CONFIG_DEBUG_VM, init_on_alloc, init_on_free and
> page_poison set to in the production build versus the KASAN build?
> free_page_is_bad() is gated on is_check_pages_enabled(), which needs
> CONFIG_DEBUG_VM, so the production kernel would not report the bad
> free even if it happened.
I don't have CONFIG_DEBUG_VM enabled in production..
>
> 4. Are any of the crashing guests Windows, and is hv-tlbflush set on
> them? That decides whether 26505e1b5b54 matters for you.
yes, windows, but hv-tlbflush is enabled on a sigle VM and it runs od
different node all the time.
>
> 5. Has any corruption occurred since THP was disabled? If yes, that
> supports the CPA race over your THP theory.
not yet, but it's not happening that often, so unsure here
>
> 6. When you run the 6.18.54-rc1 test, please note that 6.18.52 is only
> a partial fix, so if you have any results from a 6.18.52 kernel they
> should not be treated as a clean run.
sure, I'll start today with 6.18.54, won't consider older tests.
>
> << Schlopp
>
> But really a run against 6.18.53 being OK under heavy testing for several
> days should confirm it also.
>
> If it turns out it's not this then back to the drawing board I guess! Let
> us know.
I surely will!
cheers, nik
>
> --
> Cheers, Lorenzo
>
--
Ing. Nikola CIPRICH
technický ředitel
+420 591 166 214
+420 777 093 799
nikola.ciprich@linuxbox.cz
www.linuxbox.cz
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hunting memory corruption bug in 6.18.x
2026-09-26 5:58 ` Nikola Ciprich
@ 2026-09-26 9:32 ` Lorenzo Stoakes (ARM)
0 siblings, 0 replies; 5+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-09-26 9:32 UTC (permalink / raw)
To: Nikola Ciprich
Cc: linux-mm, linux-kernel, akpm, david, Mike Rapoport, Dave Hansen,
Pedro Falcato, Kiryl Shutsemau
[-- Attachment #1: Type: text/plain, Size: 8169 bytes --]
On Sat, Sep 26, 2026 at 07:58:22AM +0200, Nikola Ciprich wrote:
> Hello Lorenzo (and others),
>
> thank you for your time looking into this..
No worries!
> >
> > So there are two sides to the race: set_memory_rox() - triggered on module
> > load, ftrace trampoline creation and every new BPF 2M program pack.
> >
> > The other side is execmem_force_rw() -> set_memory_[nx,rw]() (concurrent
> > module load or ftrace trampoline allocation) __text_poke() ->
> > vmalloc_to_page() for patching module text, kprobe slots, trampolines or
> > BPF.
> >
> > Both are happening a lot at KVM host bringup (module autoload, per-VM
> > seccomp filters, perf, BPF probes, etc.
>
> one note here, at least last mentioned crash (with 6.18.44) happened with
> host running only windows guest, in general we're seeing those problems
> mosly with windows VM hosting machines.. so maybe they're triggerng the
> problem with some other, but similar mechanism?
Interesting! But indeed all this is host-side.
Though it came up with a possible finding around commit 26505e1b5b54 ("KVM:
SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled") which has
not been backported yet.
That's is a. AMD-specific guest memory corruption and b. only if
hv-tlbflush=on.
This is independent of the CPA stuff.
So if the CPA stuff turns out to be a red herring that's one worth looking
at? Are you able to run a modified kernel with this applied on top?
> > KASLR makes things tricky but this is most definitely a slab allocation in
> > the direct map and the stack (VMAP_STACK) is at 0xff7532a13699fda0 (rsp)
> > ~70 TiB higher which sits at least 10 TiB padded above the direct map so
> > it's safe to say that this is in the direct map.
> >
> > And the corrupted value (rbx) 0x0fffffff0c930020 is interesting - it's an
> > exactly a x86-64 swap softleaf value:
> >
> > __swp_type() = val >> (64 - SWP_TYPE_BITS=5) = val >> 59 = 1
> > __swp_offset() = (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT) = (~val << 6 >> 14)
> > = 0x79b67f
> >
> > I.e. it's a swap softleaf entry 0x79b67f 4 KiB pages into the swap = ~30.4 GiB.
> >
> > It's also the _second_ swap in the system (Nikola - if you have a 2nd swap
> > on the reporting system of >=~30 GiB that kinda confirms it).
>
> I suspect this may be a bit of a red herring...
>
> actually there is NO swap on that machine, also there were no linux guests.. so
> that might just be a coincidence? not sure if it changes anything..
>
Hmm that's really really odd. But if you had swap before or VMs before this
is a long-lasting corruption that could have been sat there for days before
you triggered it.
> > The LLM added on some hints for confirmation of this:
> >
> > Schlopp>>
> >
> > Log greps, across all affected hosts and all boots, not just the ones
> > that crashed:
> >
> > grep -i 'Bad page map' /var/log/messages*
> > grep -i 'bad pmd' /var/log/messages*
> > grep -i 'bad pud' /var/log/messages*
> > grep -i 'Bad page state' /var/log/messages*
> > grep -i 'CPA: called for zero pte' /var/log/messages*
>
> not a single occurance (this machine uses journal, but I checked those
> and no such messages.. in general i tend to check dmesg and system logs
> a lot, so I'd have already reported such messages..
Yeah I don't know why it assumed you used antiquated logging..! :)
OK that's interesting.
>
> >
> > Any of these, particularly "bad pmd", is direct evidence that a freed
> > kernel PTE table was reused as a user page table. "CPA: called for zero
> > pte" would be the CPA walker itself tripping over a collapsed mapping.
> >
> > Questions:
> >
> > 1. swapon --show on the host, and inside the guests. Is there a swap
> > device with index 1 and a size of at least roughly 30.5GiB? That
> > tells us whether the corrupt word is a host swap PTE or a guest one,
> > which distinguishes case A from case B above.
> >
> > 2. From the vmcore, dump the 4K page containing 0xff2e6dbe0e51b440. If
> > the neighbouring words are also PTE-shaped, the page was being used
> > as a page table and the diagnosis above is confirmed. If only the
> > one word is corrupt, it was a single stray store.
> unfortunately I don't have full vmcore from that crash, as it didn't fit
> to /var/crash, backtrace I posted is from vmcore-dmesg.txt so can't confirm
> that..
Ah that's a pity!
>
>
> >
> > 3. What are CONFIG_DEBUG_VM, init_on_alloc, init_on_free and
> > page_poison set to in the production build versus the KASAN build?
> > free_page_is_bad() is gated on is_check_pages_enabled(), which needs
> > CONFIG_DEBUG_VM, so the production kernel would not report the bad
> > free even if it happened.
>
> I don't have CONFIG_DEBUG_VM enabled in production..
Well that explains the lack of bad reports above. I don't know why it'd
assume you'd run kernels with that (we do not recommend that for production
:)
>
> >
> > 4. Are any of the crashing guests Windows, and is hv-tlbflush set on
> > them? That decides whether 26505e1b5b54 matters for you.
> yes, windows, but hv-tlbflush is enabled on a sigle VM and it runs od
> different node all the time.
Ah but that could be enough to cause memory corruption. The reports seem to
be about guest memory corruption though.
To be clear - are you observing it in the guest or host? I gathered host
from the splat.
>
>
> >
> > 5. Has any corruption occurred since THP was disabled? If yes, that
> > supports the CPA race over your THP theory.
> not yet, but it's not happening that often, so unsure here
Yeah it seems to be a hard one to hit. If you tried a kernel with KASAN
enabled it might be flagged earlier? But that could also kill the race
window and would slow the system down a lot.
>
> >
> > 6. When you run the 6.18.54-rc1 test, please note that 6.18.52 is only
> > a partial fix, so if you have any results from a 6.18.52 kernel they
> > should not be treated as a clean run.
> sure, I'll start today with 6.18.54, won't consider older tests.
Ack, that's the best thing to do at the moment to be honest.
If you were consistently getting corruption after X days previously, 2*X
days let's say of none can give confidence it's fixed there.
>
>
> >
> > << Schlopp
> >
> > But really a run against 6.18.53 being OK under heavy testing for several
> > days should confirm it also.
> >
> > If it turns out it's not this then back to the drawing board I guess! Let
> > us know.
> I surely will!
>
> cheers, nik
Thanks! Given the nature of the bug and the fact the LLM went a little out
on a limb.
Some more stuff from the report, which I also enclose in full here FYI.
schlopp>>
26505e1b5b54 ("KVM: SVM: make svm_flush_tlb_gva do a full asid flush if
NPT enabled") is mainline only and has not been backported to 6.18.y.
It is AMD-specific guest memory corruption via the Hyper-V PV TLB flush
path, so it only affects Windows guests running with hv-tlbflush=on. If
any of your crashing guests are Windows, this is worth backporting
separately. It is independent of the CPA race.
55ddbc2ca6d5 (upstream f7491d7c81db, "x86/mm: Fix user-space data loss
with MADV_FREE and THP") is a real bug in 6.18.44. pmd_modify() drops
_PAGE_DIRTY, which has been the case since v6.10, and it causes data
loss for MADV_FREE THP and writable file THP. It is fixed in 6.18.53.
It is data loss rather than a stray write, so it does not explain the
oops, but it is another reason to move off 6.18.44. Note that this one
needs THP, which you have now disabled.
d053eb7e09e1 (upstream 1e75a8255f11, "iommu/amd: Wait for completion
instead of returning early in iommu_completion_wait()") is already in
6.18.42 and therefore in the kernel that crashed. It is ruled out for
this oops, but it may be relevant to the earlier incidents you had on
6.18.31 through 6.18.41.
<<schlopp
Let us know how the tests get on! If you trigger a bug on 54 let us know
ASAP so we can investigate alternative theories.
Thanks!
>
>
>
>
> >
> > --
> > Cheers, Lorenzo
> >
>
> --
> Ing. Nikola CIPRICH
> technický ředitel
>
> +420 591 166 214
> +420 777 093 799
> nikola.ciprich@linuxbox.cz
>
> www.linuxbox.cz
--
Cheers, Lorenzo
[-- Attachment #2: debug-report.txt --]
[-- Type: text/plain, Size: 23666 bytes --]
Summary
=======
The corruption you are seeing is consistent with a known use-after-free
in the x86 change_page_attr (CPA) code that is present in 6.18.44 and
was only fixed in 6.18.52 and 6.18.53.
cpa_collapse_large_pages() rebuilds a leaf PMD out of its 4K PTEs and
then frees the old PTE table, with no lock held against the lockless
page table walk that __change_page_attr() performs before it stores
through the PTE pointer it cached. The stale 8-byte store of a kernel
PTE value lands in whatever the buddy allocator has since handed that
page out for.
On a KVM host the two sides of this race are both hot. set_memory_rox()
is the only caller that passes CPA_COLLAPSE, and it runs on every module
load (execmem_restore_rox()), every ftrace trampoline creation
(arch/x86/kernel/ftrace.c:423) and every new 2M BPF program pack. The
other side is any lockless walk of the same execmem tables:
set_memory_nx()/set_memory_rw() from execmem_force_rw() on module load
and trampoline allocation, and vmalloc_to_page() inside __text_poke()
for every patch of module text, kprobe slot, trampoline or BPF pack.
Module text, kprobe slots and ftrace trampolines share the same 2M ROX
cache pages, so the collapser and the victim land in the same PMD by
construction. A libvirt host does all of this constantly: module
autoload (tun, vhost_net, br_netfilter, ebtable_*, xt_*), per-VM seccomp
filters, systemd cgroup BPF, perf and BPF probes, ftrace and static call
patching on every module load.
This matches your good/bad window exactly. The collapse feature was
added in v6.15 and is not in 5.15.
It also matches the KASAN result. free_page_is_bad() is gated on
is_check_pages_enabled(), which is off unless CONFIG_DEBUG_VM is set,
and KASAN changes the allocation pattern enough that the freed page
tends not to be reused in the race window. The upstream reporter only
reproduced it by injecting a delay at the CPA page table lookup.
Recommendation: run 6.18.53 or later. 6.18.54-rc1, which you say you
are about to test, has the complete series. 6.18.52 has only the
cpa_lock patch, which does not close the window. Mainline (v7.3-rc5)
has nothing further pending for arch/x86/mm/pat/set_memory.c.
Kernel version
==============
6.18.44lb9.01 (AlmaLinux 9 build of stable 6.18.44, stable commit
1efe5d048a39). First seen on 6.18.31, last crash on 6.18.44. 5.15.x
was fine.
Machine
=======
ASUSTeK RS720A-E12-RS12 / K14PP-D24, AMD EPYC, BIOS 2305 11/21/2025.
KVM host, AlmaLinux 9, Intel ice and Mellanox NICs. Taint "G E",
unsigned module only.
Stack trace
===========
Oops: general protection fault, probably for non-canonical address
0xfffffff0c930038: 0000 [#1] SMP NOPTI
CPU: 23 UID: 189 PID: 7538 Comm: pacemaker-contr
RIP: 0010:__d_lookup+0x4a/0xc0
RAX: 0fffffff0c930020 RBX: 0fffffff0c930020 RCX: 0000000000000000
RDX: ff2e6dbe0d9b6000 RSI: ff7532a13699fe60 RDI: ff2e6d1e4e630d80
RBP: 000000000b654440
Call Trace:
<TASK>
d_lookup+0x27/0x50
lookup_dcache+0x1f/0x80
lookup_one_qstr_excl+0x1e/0xe0
filename_create+0xc4/0x160
do_mkdirat+0x5a/0x190
__x64_sys_mkdir+0x42/0x60
do_syscall_64+0x64/0xbf0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>
Other messages you reported:
Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 548:
elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) ==
R_X86_64_RELATIVE' failed!
What the oops registers say
===========================
The Code bytes decode to the hash chain walk in fs/dcache.c:__d_lookup():
mov (%rbx),%rax ; load bucket->first
mov %rax,%rbx
and $-2,%rbx ; strip the hlist_bl lock bit
cmp $1,%rax
ja body
loop:
mov (%rbx),%rbx ; node->next
test %rbx,%rbx
je out
body:
cmp %ebp,0x18(%rbx) ; <-- faulting insn, d_name.hash_len
jne loop
RAX equals RBX and RAX is only ever written by the initial bucket load,
so this is the first loop iteration. The corrupt word is the
hlist_bl_head.first field of the dentry_hashtable bucket itself, not a
dentry's d_hash.next.
RDX 0xff2e6dbe0d9b6000 is the dentry_hashtable base, and the runtime
constant d_hash_shift is patched to 7, so:
bucket = 0xff2e6dbe0d9b6000 + (0x0b654440 >> 7) * 8
= 0xff2e6dbe0e51b440
That table is a 256MB alloc_large_system_hash() allocation from
memblock. It is allocated at boot, is PG_reserved and is never freed.
So this is a stray write to a fixed physical page, not a
use-after-free of a recycled object.
The corrupt value 0x0fffffff0c930020 is an exact x86-64 non-present
swap PTE: __swp_type() is 1 and __swp_offset() is 0x79b67f, about
30.4GiB into swap device 1. The only low bit set is bit 5,
_PAGE_ACCESSED.
Every bit the swap layout constrains is as it should be: P, PSE and
the G/PROT_NONE alias are clear, the software bits 1-3 are clear, the
type is an ordinary swap type and the inverted offset gives the long
run of ones in bits 32-58. The one thing the layout does not account
for is bit 5. __swp_entry() leaves bits 0-8 clear and no helper sets
bit 5 on a non-present PTE; arch/x86/include/asm/pgtable_64.h treats
bits 5 and 6 as don't-care only because of the Intel Knights Landing
erratum (00839ee3b299, _PAGE_KNL_ERRATUM_MASK), which does not apply to
EPYC. So either something other than a Linux swap PTE happens to fit
this layout, or the word was a swap PTE that acquired a stray bit. I
cannot tell which from one word, which is why the vmcore page dump
requested below matters: 511 neighbouring PTE-shaped words would settle
it.
Suspect commit
==============
commit 41d88484c71cd4f659348da41b7b5b3dbd3be1f6
Author: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
x86/mm/pat: restore large ROX pages after fragmentation
Link: https://lore.kernel.org/r/20250126074733.1384926-4-rppt@kernel.org
This added runtime collapse of split kernel large pages, driven from
cpa_flush(), including freeing the PTE table that the collapsed PMD
replaces. It is the Fixes: target of every fix listed below. It is in
v6.15 and later, and is not in 5.15.
> diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
> --- a/arch/x86/mm/pat/set_memory.c
> +++ b/arch/x86/mm/pat/set_memory.c
> @@ -394,6 +408,40 @@ static void __cpa_flush_tlb(void *data)
> +static void cpa_collapse_large_pages(struct cpa_data *cpa)
> +{
> + unsigned long start, addr, end;
> + struct ptdesc *ptdesc, *tmp;
> + LIST_HEAD(pgtables);
> + int collapsed = 0;
> + int i;
[ ... range iteration ... ]
> + if (!collapsed)
> + return;
> +
> + flush_tlb_all();
> +
> + list_for_each_entry_safe(ptdesc, tmp, &pgtables, pt_list) {
> + list_del(&ptdesc->pt_list);
> + __free_page(ptdesc_page(ptdesc));
> + }
> +}
In 6.18.44 that last loop is pagetable_free(ptdesc), which is still an
immediate free. There is no RCU grace period and no other deferral. The
flush_tlb_all() above it only makes the hardware forget the old
translation; it does nothing about a CPU that is sitting inside
__change_page_attr() holding a pointer into that table.
> @@ -402,7 +450,7 @@ static void cpa_flush(struct cpa_data *cpa, int cache)
> if (cache && !static_cpu_has(X86_FEATURE_CLFLUSH)) {
> cpa_flush_all(cache);
> - return;
> + goto collapse_large_pages;
> }
[ ... ]
> @@ -427,6 +475,10 @@ static void cpa_flush(struct cpa_data *cpa, int cache)
> mb();
> +
> +collapse_large_pages:
> + if (cpa->flags & CPA_COLLAPSE)
> + cpa_collapse_large_pages(cpa);
> }
The collapse is hooked into cpa_flush(), which
__change_page_attr_set_clr() calls after it has already dropped
cpa_lock. So cpa_lock does not serialise the collapse against anything.
> @@ -1196,6 +1248,161 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
> +static int collapse_pmd_page(pmd_t *pmd, unsigned long addr,
> + struct list_head *pgtables)
> +{
[ ... uniformity checks over all 512 PTEs ... ]
> + old_pmd = *pmd;
> +
> + /* Success: set up a large page */
> + pgprot = pgprot_4k_2_large(pte_pgprot(first));
> + pgprot_val(pgprot) |= _PAGE_PSE;
> + _pmd = pfn_pmd(pfn, pgprot);
> + set_pmd(pmd, _pmd);
> +
> + /* Queue the page table to be freed after TLB flush */
> + list_add(&page_ptdesc(pmd_page(old_pmd))->pt_list, pgtables);
collapse_large_pages(), the caller, takes pgd_lock around this. The
lockless CPA walker never takes pgd_lock, so pgd_lock does not help
either.
The other side, in 6.18.44:
arch/x86/mm/pat/set_memory.c:__change_page_attr() {
address = __cpa_addr(cpa, cpa->curpage);
repeat:
kpte = _lookup_address_cpa(cpa, address, &level, &nx, &rw);
...
old_pte = *kpte;
...
if (level == PG_LEVEL_4K) {
...
new_pte = pfn_pte(pfn, new_prot);
...
if (pte_val(old_pte) != pte_val(new_pte)) {
set_pte_atomic(kpte, new_pte); <-- stale
cpa->flags |= CPA_FLUSHTLB;
}
_lookup_address_cpa() reaches lookup_address_in_pgd_attr(), which is a
plain lockless walk. Between the walk and the set_pte_atomic() the
caller can be preempted or take an interrupt; this runs with interrupts
on. That store is the only unsafe instruction in the function. The
large-page split branch further down is safe because __split_large_page()
revalidates under pgd_lock.
The freed table is not even a tracked page table page in 6.18.44:
arch/x86/mm/pat/set_memory.c:split_large_page() {
if (!debug_pagealloc_enabled())
spin_unlock(&cpa_lock);
base = alloc_pages(GFP_KERNEL, 0);
A bare alloc_pages(), so it goes straight back to the per-CPU free list
and can be reallocated immediately.
Race timeline
=============
CPU A (text_poke -> CPU B (module_enable_rox /
execmem_make_temp_rw -> execmem_restore_rox /
set_memory_nx/rw) bpf_jit_binary_lock_ro ->
set_memory_rox, CPA_COLLAPSE)
----- -----
__change_page_attr()
kpte = _lookup_address_cpa()
old_pte = *kpte
new_pte = pfn_pte(...)
preempted / interrupted
__change_page_attr_set_clr()
drops cpa_lock
cpa_flush()
cpa_collapse_large_pages()
collapse_pmd_page(): all 512
PTEs uniform, set_pmd() installs
a leaf, old PTE table queued
flush_tlb_all()
pagetable_free() -> immediate
__free_pages()
(any CPU) page is reallocated:
.so page cache folio, QEMU guest
RAM, a user PMD/PTE table, slab
set_pte_atomic(kpte, new_pte)
stores a PTE-shaped word into
the reallocated page
Where the swap PTE comes from (inferred continuation)
------------------------------------------------------
The race above writes a present kernel PTE, never a swap entry. To
reach the dentry hash table with a swap-PTE-shaped word the following
has to happen next. Each step is verified in the 6.18.44 code; the
sequence as a whole is inferred, not proven for this oops.
1. The freed PTE table is reallocated as a QEMU page table.
2. The stale set_pte_atomic() lands in it. The injected entry is a
translation into an execmem text page (case A/B below).
3. Case B: GUP-slow follows that entry and KVM maps the text page
into the guest; a later zap_present_folio_ptes() does an
unbalanced folio_put() and frees the still-live text page.
4. That text page is reallocated as another page table while
text_poke()/the BPF JIT keep writing instruction bytes into it
through the ROX mapping. Instruction bytes are now PMD entries
with arbitrary pfns; some pass pmd_bad().
5. Reclaim: try_to_unmap_one() -> page_vma_mapped_walk() ->
pte_offset_map_lock() reads such a PMD, computes
__va(garbage pfn) as the PTE table, and set_pte_at() stores a
swap PTE there. If that pfn is the dentry_hashtable page, one
bucket head becomes 0x0fffffff0c930020-like.
6. Days later __d_lookup() hashes into that bucket and faults.
Step 5 is the only writer of an ordinary swap type in the mm, and
the only step that can touch memory the allocator never owned.
This is not speculation about the code. The same interleaving was
reported upstream with a KASAN reproducer, in the commit that first
tried to address it:
commit 1aac65f3e651 ("x86/mm/pat: Take cpa_lock around large-page
collapse")
BUG: KASAN: use-after-free in __change_page_attr+0x7cc/0x7e0
Write of size 8 at addr ffff888181139718 by task modprobe
...
The buggy address belongs to the physical page:
pfn:0x181139 ... page_type: f2(table)
Fixes: 41d88484c71c ("x86/mm/pat: restore large ROX pages after
fragmentation")
Signed-off-by: Denis V. Lunev <den@openvz.org>
Which stable releases carry the fixes
=====================================
None of these are in 6.18.44.
The fixes that matter are the init_mm mmap lock pair, a1c7570cedd0 and
d5d8b8662e6e: the collapse runs under the init_mm write lock and the
whole attribute change, including the lockless walk and the store
through the cached pointer, runs under the read lock. That excludes
both walker-vs-collapse and collapse-vs-collapse. 1587d3394e25 covers
the third walker: __text_poke() resolves the pages it patches with
vmalloc_to_page(), a lockless walk of the same execmem tables, and now
takes the init_mm read lock around it. Without that, a collapse under
a concurrent text_poke() returns NULL (the BUG_ON at
arch/x86/kernel/alternative.c:2576 that openSUSE hit) or, if the freed
table has already been reused, a wrong page that text_poke() then
writes instruction bytes into. 9e4a3ec3411b makes the split tables
real kernel page tables so their freeing is deferred. The earlier
cpa_lock patch, 1aac65f3e651, is superseded by the write lock and adds
nothing once those are applied.
6.18.52
591b6fac9df3 x86/mm/pat: Take cpa_lock around large-page collapse
(upstream 1aac65f3e651)
6.18.53
35820cf8dd52 x86/mm/pat: Acquire init_mm write lock on collapse to
avoid UAF (upstream a1c7570cedd0)
e21a9ea81426 x86/mm/pat: Acquire init_mm read lock on attribute
changes to avoid UAF (upstream d5d8b8662e6e)
e164f4a25e23 x86/mm/pat: Convert split_large_page() to use ptdescs
5029589bb773 x86/mm/pat: Don't gate cpa_lock on
debug_pagealloc_enabled()
84e0cd79d57f x86/mm/pat: Allocate split page tables as kernel page
tables (upstream 9e4a3ec3411b)
281e6f536f2f x86/alternatives: Exclude text poking against
change_page_attr() (upstream 1587d3394e25)
74a2626044de x86/mm: Fix and document DEBUG_PAGEALLOC
(upstream 7da514d819a0)
6.18.52 does not fix this: it only takes cpa_lock around the collapse,
and the walker never holds cpa_lock across its walk-then-store window.
The init_mm mmap lock pair that closes that window is in 6.18.53, which
is the first stable release with the complete set. Use 6.18.53 or
later.
There is no clean interim mitigation on 6.18.44. CPA_COLLAPSE cannot be
disabled by a boot parameter.
How this reaches the symptoms you saw
=====================================
Symptoms 1 and 2 follow directly. The stale store writes a PTE-shaped
qword into a page that has been reallocated. If that page is a page
cache folio for a mapped .so, eight bytes of its relocation table are
replaced and ld.so trips the R_X86_64_RELATIVE assertion while the file
on disk is intact. If it is an anonymous page that QEMU has just
populated as guest RAM on a migration destination, the guest sees eight
corrupt bytes. Both of these match "right after migration": the
destination host is populating gigabytes of guest RAM and allocating
page tables at maximum rate, which is exactly when a just-freed page
gets reused inside the race window.
Symptom 3, this oops, needs one more step, because the dentry hash table
is memblock memory that is never freed and therefore cannot be the
directly reallocated page. The escalation is that the victim page is
itself a page table. Each of the following links is verified in the
6.18.44 source, but I want to be clear that the end-to-end chain for
this particular oops is plausible rather than proven from a single
vmcore.
Case A, the victim is a user PMD table. pmd_bad() is true for the
injected value, so the first user-mode touch faults and
mm/pgtable-generic.c:___pte_offset_map() heals it via pmd_clear_bad()
with a "bad pmd" pr_err. But a supervisor-side copy_to_user() is
permitted through a U=0 level. A read() or recvmsg() into guest RAM, or
vhost-net, or kvm_write_guest(), then has the hardware walker read a
qword of module text as a PTE. If that qword happens to have P and RW
set, the copied data is written to an arbitrary physical address. This
route leaves no taint.
Case B, the victim is a user PTE table. x86 has no pte_bad(). GUP-fast
rejects a U=0 entry, but GUP-slow does not:
mm/gup.c:follow_page_pte() and mm/memory.c:__vm_normal_page() check
neither _PAGE_USER nor PageReserved, so a live execmem page is returned
and KVM maps kernel module text into a guest. A later
mm/memory.c:zap_present_folio_ptes() does folio_remove_rmap_ptes() and
folio_put() on a page that was never rmapped, dropping the refcount to
zero, printing "BUG: Bad page map" and releasing live ROX text into the
buddy allocator while text_poke() and the BPF JIT keep writing
instruction bytes into it. Recycled as a user PMD table, instruction
bytes are page table entries with arbitrary pfns that pass pmd_bad(),
pte_offset_map() computes __va() of an arbitrary physical page, and
try_to_unmap_one() writes a genuine host swap PTE into it.
That last step is what the corrupt word looks like: a real host swap
PTE. The stray bit 5 is unexplained on this route too.
One caveat that argues against case B on this specific host: the taint
is "G E" with no "B". "BUG: Bad page map" had not fired on that
machine before the crash. So either the taint-free case A route or a
direct stray write carries this particular chain, or the corrupting
event happened on a different boot. This limits, but does not refute,
the cascade. The corruption can sit in a rarely used dentry bucket for
a long time before something hashes into it, which fits the 22-day
uptime on this crash and your observation that the host crashes were
not correlated with migration.
Secondary findings
==================
These came up while looking and are worth knowing about, but none of
them explains this oops.
26505e1b5b54 ("KVM: SVM: make svm_flush_tlb_gva do a full asid flush if
NPT enabled") is mainline only and has not been backported to 6.18.y.
It is AMD-specific guest memory corruption via the Hyper-V PV TLB flush
path, so it only affects Windows guests running with hv-tlbflush=on. If
any of your crashing guests are Windows, this is worth backporting
separately. It is independent of the CPA race.
55ddbc2ca6d5 (upstream f7491d7c81db, "x86/mm: Fix user-space data loss
with MADV_FREE and THP") is a real bug in 6.18.44. pmd_modify() drops
_PAGE_DIRTY, which has been the case since v6.10, and it causes data
loss for MADV_FREE THP and writable file THP. It is fixed in 6.18.53.
It is data loss rather than a stray write, so it does not explain the
oops, but it is another reason to move off 6.18.44. Note that this one
needs THP, which you have now disabled.
d053eb7e09e1 (upstream 1e75a8255f11, "iommu/amd: Wait for completion
instead of returning early in iommu_completion_wait()") is already in
6.18.42 and therefore in the kernel that crashed. It is ruled out for
this oops, but it may be relevant to the earlier incidents you had on
6.18.31 through 6.18.41.
Theories that were eliminated
=============================
KVM NPT mapping at too large a level, or with the wrong base pfn. The
mapping level comes from the host page tables and the pfn from GUP;
KVM cannot reach memblock memory on its own.
A host mm swap or migration PTE stored through a stale page table
pointer. All the store sites are bounded and the pointer provenance
checks out.
A missed MMU notifier invalidation. Notifier ordering on the recovery
path is correct, and this cannot reach never-freed memory.
NIC DMA to the wrong address. Only a teardown-time page_pool
use-after-free turned up, and the iommu/amd completion-wait fix is
already in 6.18.42.
105d04edbec8 (upstream 33192a26cddea, mm/huge_memory huge_zero_pfn
race). Real, but not present in 6.18.44, and it cannot reach memblock
memory.
0a25ee42e7d1 (upstream 3d679b7cb31f, KVM x86/mmu CMPXCHG when clearing
the Accessed bit in the TDP MMU). Not in 6.18.44, and benign for this.
TDP MMU in-place huge page recovery. Structurally excluded: notifier
zaps take mmu_lock for write, recovery takes it for read.
memblock/buddy physical aliasing. This would produce "Bad page state"
reports, which you have not seen. Worth confirming from the vmcore, see
below.
Neither THP nor NUMA balancing is involved in the CPA race. Disabling
them was a reasonable precaution but it will not stop this. If you keep
seeing corruption with THP off, that is consistent with the diagnosis
rather than against it.
What would confirm this
=======================
Log greps, across all affected hosts and all boots, not just the ones
that crashed:
grep -i 'Bad page map' /var/log/messages*
grep -i 'bad pmd' /var/log/messages*
grep -i 'bad pud' /var/log/messages*
grep -i 'Bad page state' /var/log/messages*
grep -i 'CPA: called for zero pte' /var/log/messages*
Any of these, particularly "bad pmd", is direct evidence that a freed
kernel PTE table was reused as a user page table. "CPA: called for zero
pte" would be the CPA walker itself tripping over a collapsed mapping.
Questions:
1. swapon --show on the host, and inside the guests. Is there a swap
device with index 1 and a size of at least roughly 30.5GiB? That
tells us whether the corrupt word is a host swap PTE or a guest one,
which distinguishes case A from case B above.
2. From the vmcore, dump the 4K page containing 0xff2e6dbe0e51b440. If
the neighbouring words are also PTE-shaped, the page was being used
as a page table and the diagnosis above is confirmed. If only the
one word is corrupt, it was a single stray store.
3. What are CONFIG_DEBUG_VM, init_on_alloc, init_on_free and
page_poison set to in the production build versus the KASAN build?
free_page_is_bad() is gated on is_check_pages_enabled(), which needs
CONFIG_DEBUG_VM, so the production kernel would not report the bad
free even if it happened.
4. Are any of the crashing guests Windows, and is hv-tlbflush set on
them? That decides whether 26505e1b5b54 matters for you.
5. Has any corruption occurred since THP was disabled? If yes, that
supports the CPA race over your THP theory.
6. When you run the 6.18.54-rc1 test, please note that 6.18.52 is only
a partial fix, so if you have any results from a 6.18.52 kernel they
should not be treated as a clean run.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-09-26 9:32 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-25 8:48 hunting memory corruption bug in 6.18.x Nikola Ciprich
2026-09-25 10:05 ` Lorenzo Stoakes (ARM)
2026-09-25 12:13 ` Lorenzo Stoakes (ARM)
2026-09-26 5:58 ` Nikola Ciprich
2026-09-26 9:32 ` Lorenzo Stoakes (ARM)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®