Re: [patch V5 00/20] sched: Rewrite MM CID management

[Shrikanth Hegde <sshegde@linux.ibm.com>]

On 1/28/26 5:27 PM, Thomas Gleixner wrote:
> On Tue, Jan 27 2026 at 16:01, Ihor Solodrai wrote:
>> BPF CI caught a deadlock on current bpf-next tip (35538dba51b4).
>> Job: https://github.com/kernel-patches/bpf/actions/runs/21417415035/job/61670254640
>>
>> It appears to be related to this series. Pasting a splat below.
> 
> The deadlock splat is completely unrelated as it is a consequence of the
> panic which is triggered by the watchdog:
> 
>> [   45.009755] watchdog: CPU2: Watchdog detected hard LOCKUP on cpu 2
> 
> ...
> 
>> [   46.053170]   lock(&nmi_desc[NMI_LOCAL].lock);
>> [   46.053172]   <Interrupt>
>> [   46.053173]     lock(&nmi_desc[NMI_LOCAL].lock);
> 
> ...
> 
>> Any ideas what might be going on?
> 
> Without a full backtrace of all CPUs it's hard to tell because it's
> unclear what is holding the runqueue lock of CPU2 long enough to trigger
> the hard lockup watchdog.
> 
> I'm pretty sure the CID changes are unrelated, that new code just happen
> to show up as the messenger which gets stuck on the lock forever.
> 
>> [   46.053209] CPU: 2 UID: 0 PID: 126 Comm: test_progs Tainted: G           OE       6.19.0-rc5-g748c6d52700a-dirty #1 PREEMPT(full)
>> [   46.053214] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
>> [   46.053215] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
>> [   46.053217] Call Trace:
>> [   46.053220]  <NMI>
>> [   46.053223]  dump_stack_lvl+0x5d/0x80
>> [   46.053227]  print_usage_bug.part.0+0x22b/0x2c0
>> [   46.053231]  lock_acquire+0x272/0x2b0
>> [   46.053235]  ? __register_nmi_handler+0x83/0x350
>> [   46.053240]  _raw_spin_lock_irqsave+0x39/0x60
>> [   46.053242]  ? __register_nmi_handler+0x83/0x350
>> [   46.053246]  __register_nmi_handler+0x83/0x350
>> [   46.053250]  native_stop_other_cpus+0x31c/0x460
>> [   46.053255]  ? __pfx_native_stop_other_cpus+0x10/0x10
>> [   46.053260]  vpanic+0x1c5/0x3f0
> 
> vpanic() really should disable lockdep here before taking that lock in
> NMI context. The resulting lockdep splat is not really useful.
> 
> Thanks.
> 
>          tglx

Hi Thomas, Peter.


I remember running into this panic, once. But it wasn't consistent and i
couldn't hit it again. And it had vcpu overcommit, and fair bit of steal time.


The trace was like below from different CPUs.
------------------------

  watchdog: CPU 23 self-detected hard LOCKUP @ mm_get_cid+0xe8/0x188
  watchdog: CPU 23 TB:1434903268401795, last heartbeat TB:1434897252302837 (11750ms ago)
  NIP [c0000000001b7134] mm_get_cid+0xe8/0x188
  LR [c0000000001b7154] mm_get_cid+0x108/0x188
  Call Trace:
  [c000000004c37db0] [c000000001145d84] cpuidle_enter_state+0xf8/0x6a4 (unreliable)
  [c000000004c37e00] [c0000000001b95ac] mm_cid_switch_to+0x3c4/0x52c
  [c000000004c37e60] [c000000001147264] __schedule+0x47c/0x700
  [c000000004c37ee0] [c000000001147a70] schedule_idle+0x3c/0x64
  [c000000004c37f10] [c0000000001f6d70] do_idle+0x160/0x1b0
  [c000000004c37f60] [c0000000001f7084] cpu_startup_entry+0x48/0x50
  [c000000004c37f90] [c00000000005f570] start_secondary+0x284/0x288
  [c000000004c37fe0] [c00000000000e158] start_secondary_prolog+0x10/0x14


  watchdog: CPU 11 self-detected hard LOCKUP @ plpar_hcall_norets_notrace+0x18/0x2c
  watchdog: CPU 11 TB:1434903340004919, last heartbeat TB:1434897249749892 (11895ms ago)
  NIP [c0000000000f84fc] plpar_hcall_norets_notrace+0x18/0x2c
  LR [c000000001152588] queued_spin_lock_slowpath+0xd88/0x15d0
  Call Trace:
  [c00000056b69fb10] [c00000056b69fba0] 0xc00000056b69fba0 (unreliable)
  [c00000056b69fc30] [c000000001153ce0] _raw_spin_lock+0x80/0xa0
  [c00000056b69fc50] [c0000000001b9a34] raw_spin_rq_lock_nested+0x3c/0xf8
  [c00000056b69fc80] [c0000000001b9bb8] mm_cid_fixup_cpus_to_tasks+0xc8/0x28c
  [c00000056b69fd00] [c0000000001bff34] sched_mm_cid_exit+0x108/0x22c
  [c00000056b69fd40] [c000000000167b08] do_exit+0xf4/0x5d0
  [c00000056b69fdf0] [c00000000016800c] make_task_dead+0x0/0x178
  [c00000056b69fe10] [c0000000000316c8] system_call_exception+0x128/0x390
  [c00000056b69fe50] [c00000000000cedc] system_call_vectored_common+0x15c/0x2ec


  watchdog: CPU 65 self-detected hard LOCKUP @ queued_spin_lock_slowpath+0x10ec/0x15d0
  watchdog: CPU 65 TB:1434905824977447, last heartbeat TB:1434899309522065 (12725ms ago)
  NIP [c0000000011528ec] queued_spin_lock_slowpath+0x10ec/0x15d0
  LR [c000000001152d0c] queued_spin_lock_slowpath+0x150c/0x15d0
  Call Trace:
  [c000000777e27a60] [0000000000000009] 0x9 (unreliable)
  [c000000777e27b80] [c000000001153ce0] _raw_spin_lock+0x80/0xa0
  [c000000777e27ba0] [c0000000001b9a34] raw_spin_rq_lock_nested+0x3c/0xf8
  [c000000777e27bd0] [c0000000001babb8] ___task_rq_lock+0x64/0x140
  [c000000777e27c20] [c0000000001c8294] wake_up_new_task+0x180/0x484
  [c000000777e27ca0] [c00000000015bea4] kernel_clone+0x120/0x5bc
  [c000000777e27d30] [c00000000015c4c0] __do_sys_clone+0x88/0xc8
  [c000000777e27e10] [c0000000000316c8] system_call_exception+0x128/0x390
  [c000000777e27e50] [c00000000000cedc] system_call_vectored_common+0x15c/0x2ec




I am wondering if it this loop in mm_get_cid, which may not be getting a cid
for a long time? Is that possible?

static inline unsigned int mm_get_cid(struct mm_struct *mm)
{
         unsigned int cid = __mm_get_cid(mm, READ_ONCE(mm->mm_cid.max_cids));

         while (cid == MM_CID_UNSET) {
                 cpu_relax();
                 cid = __mm_get_cid(mm, num_possible_cpus());
         }
         return cid;
}