[BUG] sched/fair: GPF in update_blocked_averages() on stale leaf cfs_rq of freed task_group (6.1.159)

[BUG] sched/fair: GPF in update_blocked_averages() on stale leaf cfs_rq of freed task_group (6.1.159)

[Ajay Kaher]

We hit a general protection fault in update_blocked_averages() on a
6.1.159-based kernel (Photon OS 5.0, 6.1.159-9.ph5). It looks like a
use-after-free: a cfs_rq of an already-freed task_group was still on
rq->leaf_cfs_rq_list.

We have no reproducer and no vmcore.

Oops (trimmed):

  general protection fault, probably for non-canonical address
  0x54a403b28017e80: 0000 [#1] SMP NOPTI
  CPU: 7 PID: 0 Comm: swapper/7 Not tainted 6.1.159-9.ph5 #1-photon
  Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop
  Reference Platform, BIOS 6.00 11/12/2020
  RIP: 0010:update_blocked_averages+0x1b9/0x770
  Code: 48 89 de 48 89 c7 e8 ef 29 01 00 85 c0 0f 85 fc 02 00 00 48 8b
  83 50 01 00 00 48 8b 80 c8 00 00 00 4a 8b 34 30 48 85 f6 74 31 <48>
  83 be e0 00 00 00 00 48 8b 86 80 00 00 00 0f 85 68 03 00 00 48
  RSP: 0018:ffffbd3346524ef8 EFLAGS: 00010006
  RAX: ffff9e87e85c9500 RBX: ffff9e82b3d92e00 RCX: 0000000000000000
  RDX: 0000000000000001 RSI: 054a403b28017da0 RDI: 0000000000000000
  R13: ffff9e8110e17800 R14: 0000000000000038 R15: ffff9e82b3d92f40
  Call Trace:
   <IRQ>
   run_rebalance_domains+0x4b/0x77
   handle_softirqs+0x9b/0x250
   __irq_exit_rcu+0x90/0xc0
   irq_exit_rcu+0xe/0x17
   sysvec_apic_timer_interrupt+0x7c/0xa7
   </IRQ>
   <TASK>
   asm_sysvec_apic_timer_interrupt+0x1b/0x20
   RIP: 0010:native_safe_halt+0xb/0x17
   ...
   do_idle+0x21b/0x290
   cpu_startup_entry+0x2a/0x37
   start_secondary+0x174/0x197
   </TASK>
  Kernel panic - not syncing: Fatal exception in interrupt

Decoding the Code: bytes against __update_blocked_fair() (inlined):

  call update_cfs_rq_load_avg ; test eax,eax ; jne ...
  mov rax,[rbx+0x150]       ; cfs_rq->tg          (RBX = cfs_rq)
  mov rax,[rax+0xc8]        ; tg->se              (RAX = ffff9e87e85c9500)
  mov rsi,[rax+r14]         ; tg->se[cpu]         (R14 = 0x38 = cpu 7 * 8)
  test rsi,rsi ; je ...
  cmp qword [rsi+0xe0],0    ; se->avg.load_avg    <-- faults, RSI garbage

i.e.

  se = cfs_rq->tg->se[cpu];
  if (se && !skip_blocked_update(se))

The cfs_rq at ffff9e82b3d92e00 was still on CPU 7's leaf list. Its
cfs_rq->tg and tg->se still looked like valid kernel addresses, but
tg->se[7] contained 054a403b28017da0. That fits the tg->se array having
been freed by free_fair_sched_group() and reused, while the cfs_rq that
should have been unlinked by unregister_fair_sched_group() was still
walked by for_each_leaf_cfs_rq_safe().

The teardown order (sched_release_group() -> RCU ->
unregister_fair_sched_group() -> RCU -> free_fair_sched_group()) should
be safe, unless something re-adds the dying group's cfs_rq to the leaf
list after unregister_fair_sched_group() has run. The comment in
unregister_fair_sched_group() [1] says this cannot happen because only
empty groups are destroyed, so on_list can be checked speculatively
without risk of the cfs_rq being re-added. The actual removal is done
under rq->lock by list_del_leaf_cfs_rq().

We therefore suspect a path that calls list_add_leaf_cfs_rq() on a
cfs_rq of a group that is already past sched_release_group().
Candidates are enqueue_entity(), tg_unthrottle_up(),
propagate_entity_cfs_rq() (via attach/detach on a task group change or
task exit) and the enqueue/unthrottle loops. We could not tell which one
from this oops.

What we checked
 - fc661f2dcb7e ("sched/fair: Prevent dead task groups from regaining
   cfs_rq's") [2] and the 6.1.y backport of 3b4035ddbfc8 ("sched/fair:
   Fix potential memory corruption in child_cfs_rq_on_list") [3] are
   both in 6.1.159.
 - We found no later fix in mainline (as of v7.3-rc7) or in 6.1.y for
   leaf-list handling or task_group teardown. Mainline
   unregister_fair_sched_group() keeps the same on_list logic, so we
   believe mainline may still be affected. We have not reproduced it
   on mainline.

Questions
 - Is this a known issue, or has it already been resolved upstream?
 - Do you have any thoughts on what might be triggering this?

- Ajay

[1] unregister_fair_sched_group() in v6.1.159, kernel/sched/fair.c:
    https://elixir.bootlin.com/linux/v6.1.159/C/ident/unregister_fair_sched_group
[2] https://git.kernel.org/torvalds/c/fc661f2dcb7e
[3] https://git.kernel.org/torvalds/c/3b4035ddbfc8