Re: [PATCH v2 0/1] mm/mmu_notifier: Add async OOM cleanup via call_srcu()

[shaikh kamaluddin <shaikhkamal2012@gmail.com>]

On Thu, Apr 30, 2026 at 07:46:39PM +0530, shaikh.kamal wrote:
> This series implements the after_oom_unregister callback design
> proposed by Paolo in v1 review [1].
> 
> The current OOM notifier path calls synchronize_srcu() inline from
> mmu_notifier_oom_enter(), which can deadlock on PREEMPT_RT when
> locks such as siglock are held. This series moves the cleanup to an
> asynchronous context using call_srcu(), allowing the OOM path to
> proceed without waiting for an SRCU grace period.
> 
> Subscribers opt in via a new after_oom_unregister callback in
> struct mmu_notifier_ops.
> 
> KVM is the first (and currently only) user.
> 
> Changes since v1 [1]:
> - Implement after_oom_unregister callback in struct
>   mmu_notifier_ops as proposed by Paolo
> - Add mmu_notifier_oom_enter() to detach subscriptions and
>   schedule cleanup via call_srcu()
> - Add mmu_notifier_barrier() (srcu_barrier wrapper) so consumers
>   can wait for pending callbacks during teardown
> - Move call site from __oom_kill_process() to __oom_reap_task_mm()
>   to fix KASAN vmalloc-out-of-bounds observed in v1
> - Use hlist_del_init() to keep hlist_unhashed() correct for the
>   kvm_destroy_vm() detection path, avoiding use-after-free on the
>   stack-allocated oom_list head
> - Add KVM after_oom_unregister implementation to clear
>   mn_active_invalidate_count
> - Update kvm_destroy_vm() to detect detached subscriptions via
>   hlist_unhashed() and use mmu_notifier_barrier() + mmdrop()
>   instead of mmu_notifier_unregister()
> - Remove pr_err() on GFP_ATOMIC failure per checkpatch; the
>   trade-off is documented inline
> 
> Testing
> -------
> 
> Developed and tested under virtme-ng with PREEMPT_RT, KASAN, and
> lockdep enabled.
> 
> Test setup:
> - simple_kvm.c: minimal userspace program that opens /dev/kvm,
>   creates a VM, registers memory, creates a vCPU, and sleeps
> - CONFIG_DEBUG_VM-only debugfs interface (not part of this
>   submission) at /sys/kernel/debug/oom_reap_task to invoke
>   __oom_reap_task_mm() on a target task
> 
> Test sequence:
>   $ ./simple_kvm &
>   $ echo $! | sudo tee /sys/kernel/debug/oom_reap_task
> 
> Observed with patch applied:
> - __oom_reap_task_mm() completes
> - mmu_notifier_oom_enter() detaches the KVM subscription
> - call_srcu() callback runs after (SRCU grace period)
> - KVM after_oom_unregister clears mn_active_invalidate_count
> - mmu_notifier_barrier() returns cleanly
> - No KASAN reports, no kernel BUGs, lockdep clean
> 
> Stress runs (20 iterations) showed consistent results.
> 
> Reproducing the syzbot-reported issue
> -------------------------------------
> The issue reported by syzbot is reproducible on an unpatched
> PREEMPT_RT kernel, triggering a "sleeping function called from
> invalid context" warning in kvm_mmu_notifier_invalidate_range_start().
> With this patch applied, the warning is no longer observed..
> 
> 
> Known limitations
> -----------------
> 
> Failure of GFP_ATOMIC allocation in mmu_notifier_oom_enter()
> causes the corresponding after_oom_unregister callback to be
> skipped. The OOM path cannot sleep without reintroducing the
> deadlock this series fixes, and synchronous execution would
> require waiting for SRCU readers. Cleanup still occurs later via
> the normal unregister path. A mempool-backed allocator could
> address this in the future.
>
Hi Paolo, Lorenzo, Sean,

Gentle ping on v2, which implements the after_oom_unregister
approach Paolo suggested on v1:
  https://lore.kernel.org/all/20260430141701.10859-1-shaikhkamal2012@gmail.com/

The one outstanding issue was a build failure on !CONFIG_MMU_NOTIFIER
(arc/hexagon allnoconfig) from the kernel test robot, now fixed by
adding the missing mmu_notifier_oom_enter() stub:
  https://lore.kernel.org/all/20260607152830.8775-1-shaikhkamal2012@gmail.com/

So it builds clean and is ready for review whenever you have bandwidth.

Thanks,
Kamal
> [1] https://lore.kernel.org/all/CABgObfZQM0Eq1=vzm812D+CAcjOaE1f1QAUqGo5rTzXgLnR9cQ@mail.gmail.com
> 
> Reported-by: syzbot+c3178b6b512446632bac@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=c3178b6b512446632bac
> Tested-by: Shaikh Kamaluddin <shaikhkamal2012@gmail.com>
> 
> shaikh.kamal (1):
>   mm/mmu_notifier: Add async OOM cleanup via call_srcu()
> 
>  include/linux/mmu_notifier.h |  10 +++
>  mm/mmu_notifier.c            | 123 +++++++++++++++++++++++++++++++++++
>  mm/oom_kill.c                |   3 +
>  virt/kvm/kvm_main.c          |  27 +++++++-
>  4 files changed, 162 insertions(+), 1 deletion(-)
> 
> --
> 2.43.0
>