On Sat, Sep 26, 2026 at 07:58:22AM +0200, Nikola Ciprich wrote: > Hello Lorenzo (and others), > > thank you for your time looking into this.. No worries! > > > > So there are two sides to the race: set_memory_rox() - triggered on module > > load, ftrace trampoline creation and every new BPF 2M program pack. > > > > The other side is execmem_force_rw() -> set_memory_[nx,rw]() (concurrent > > module load or ftrace trampoline allocation) __text_poke() -> > > vmalloc_to_page() for patching module text, kprobe slots, trampolines or > > BPF. > > > > Both are happening a lot at KVM host bringup (module autoload, per-VM > > seccomp filters, perf, BPF probes, etc. > > one note here, at least last mentioned crash (with 6.18.44) happened with > host running only windows guest, in general we're seeing those problems > mosly with windows VM hosting machines.. so maybe they're triggerng the > problem with some other, but similar mechanism? Interesting! But indeed all this is host-side. Though it came up with a possible finding around commit 26505e1b5b54 ("KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled") which has not been backported yet. That's is a. AMD-specific guest memory corruption and b. only if hv-tlbflush=on. This is independent of the CPA stuff. So if the CPA stuff turns out to be a red herring that's one worth looking at? Are you able to run a modified kernel with this applied on top? > > KASLR makes things tricky but this is most definitely a slab allocation in > > the direct map and the stack (VMAP_STACK) is at 0xff7532a13699fda0 (rsp) > > ~70 TiB higher which sits at least 10 TiB padded above the direct map so > > it's safe to say that this is in the direct map. > > > > And the corrupted value (rbx) 0x0fffffff0c930020 is interesting - it's an > > exactly a x86-64 swap softleaf value: > > > > __swp_type() = val >> (64 - SWP_TYPE_BITS=5) = val >> 59 = 1 > > __swp_offset() = (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT) = (~val << 6 >> 14) > > = 0x79b67f > > > > I.e. it's a swap softleaf entry 0x79b67f 4 KiB pages into the swap = ~30.4 GiB. > > > > It's also the _second_ swap in the system (Nikola - if you have a 2nd swap > > on the reporting system of >=~30 GiB that kinda confirms it). > > I suspect this may be a bit of a red herring... > > actually there is NO swap on that machine, also there were no linux guests.. so > that might just be a coincidence? not sure if it changes anything.. > Hmm that's really really odd. But if you had swap before or VMs before this is a long-lasting corruption that could have been sat there for days before you triggered it. > > The LLM added on some hints for confirmation of this: > > > > Schlopp>> > > > > Log greps, across all affected hosts and all boots, not just the ones > > that crashed: > > > > grep -i 'Bad page map' /var/log/messages* > > grep -i 'bad pmd' /var/log/messages* > > grep -i 'bad pud' /var/log/messages* > > grep -i 'Bad page state' /var/log/messages* > > grep -i 'CPA: called for zero pte' /var/log/messages* > > not a single occurance (this machine uses journal, but I checked those > and no such messages.. in general i tend to check dmesg and system logs > a lot, so I'd have already reported such messages.. Yeah I don't know why it assumed you used antiquated logging..! :) OK that's interesting. > > > > > Any of these, particularly "bad pmd", is direct evidence that a freed > > kernel PTE table was reused as a user page table. "CPA: called for zero > > pte" would be the CPA walker itself tripping over a collapsed mapping. > > > > Questions: > > > > 1. swapon --show on the host, and inside the guests. Is there a swap > > device with index 1 and a size of at least roughly 30.5GiB? That > > tells us whether the corrupt word is a host swap PTE or a guest one, > > which distinguishes case A from case B above. > > > > 2. From the vmcore, dump the 4K page containing 0xff2e6dbe0e51b440. If > > the neighbouring words are also PTE-shaped, the page was being used > > as a page table and the diagnosis above is confirmed. If only the > > one word is corrupt, it was a single stray store. > unfortunately I don't have full vmcore from that crash, as it didn't fit > to /var/crash, backtrace I posted is from vmcore-dmesg.txt so can't confirm > that.. Ah that's a pity! > > > > > > 3. What are CONFIG_DEBUG_VM, init_on_alloc, init_on_free and > > page_poison set to in the production build versus the KASAN build? > > free_page_is_bad() is gated on is_check_pages_enabled(), which needs > > CONFIG_DEBUG_VM, so the production kernel would not report the bad > > free even if it happened. > > I don't have CONFIG_DEBUG_VM enabled in production.. Well that explains the lack of bad reports above. I don't know why it'd assume you'd run kernels with that (we do not recommend that for production :) > > > > > 4. Are any of the crashing guests Windows, and is hv-tlbflush set on > > them? That decides whether 26505e1b5b54 matters for you. > yes, windows, but hv-tlbflush is enabled on a sigle VM and it runs od > different node all the time. Ah but that could be enough to cause memory corruption. The reports seem to be about guest memory corruption though. To be clear - are you observing it in the guest or host? I gathered host from the splat. > > > > > > 5. Has any corruption occurred since THP was disabled? If yes, that > > supports the CPA race over your THP theory. > not yet, but it's not happening that often, so unsure here Yeah it seems to be a hard one to hit. If you tried a kernel with KASAN enabled it might be flagged earlier? But that could also kill the race window and would slow the system down a lot. > > > > > 6. When you run the 6.18.54-rc1 test, please note that 6.18.52 is only > > a partial fix, so if you have any results from a 6.18.52 kernel they > > should not be treated as a clean run. > sure, I'll start today with 6.18.54, won't consider older tests. Ack, that's the best thing to do at the moment to be honest. If you were consistently getting corruption after X days previously, 2*X days let's say of none can give confidence it's fixed there. > > > > > > << Schlopp > > > > But really a run against 6.18.53 being OK under heavy testing for several > > days should confirm it also. > > > > If it turns out it's not this then back to the drawing board I guess! Let > > us know. > I surely will! > > cheers, nik Thanks! Given the nature of the bug and the fact the LLM went a little out on a limb. Some more stuff from the report, which I also enclose in full here FYI. schlopp>> 26505e1b5b54 ("KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled") is mainline only and has not been backported to 6.18.y. It is AMD-specific guest memory corruption via the Hyper-V PV TLB flush path, so it only affects Windows guests running with hv-tlbflush=on. If any of your crashing guests are Windows, this is worth backporting separately. It is independent of the CPA race. 55ddbc2ca6d5 (upstream f7491d7c81db, "x86/mm: Fix user-space data loss with MADV_FREE and THP") is a real bug in 6.18.44. pmd_modify() drops _PAGE_DIRTY, which has been the case since v6.10, and it causes data loss for MADV_FREE THP and writable file THP. It is fixed in 6.18.53. It is data loss rather than a stray write, so it does not explain the oops, but it is another reason to move off 6.18.44. Note that this one needs THP, which you have now disabled. d053eb7e09e1 (upstream 1e75a8255f11, "iommu/amd: Wait for completion instead of returning early in iommu_completion_wait()") is already in 6.18.42 and therefore in the kernel that crashed. It is ruled out for this oops, but it may be relevant to the earlier incidents you had on 6.18.31 through 6.18.41. < > > > > > > > -- > > Cheers, Lorenzo > > > > -- > Ing. Nikola CIPRICH > technický ředitel > > +420 591 166 214 > +420 777 093 799 > nikola.ciprich@linuxbox.cz > > www.linuxbox.cz -- Cheers, Lorenzo