Re: [PATCH v2] sched/debug, sys_info: Introduce SYS_INFO_CPU_RUNQUEUES

[Peter Zijlstra <peterz@infradead.org>]

On Tue, Sep 22, 2026 at 01:35:59PM -0400, Aaron Tomlin wrote:
> On Tue, Sep 22, 2026 at 04:40:27PM +0200, Peter Zijlstra wrote:
> > On Fri, Sep 11, 2026 at 09:32:40PM -0400, Aaron Tomlin wrote:
> > > When investigating kernel panics, inspectability of per-CPU runqueues
> > > and runnable task states is valuable for diagnosing CPU starvation
> > > priority inversion, etc.
> > > 
> > > While debugfs (/sys/kernel/debug/sched/debug) exposes runqueue metrics
> > > to userspace, these details are not captured during an automated kernel
> > > panic or crash dump. Capturing per-CPU runqueue state directly into
> > > log_buf fills this diagnostic gap for post-mortem crash analysis.
> > 
> > Uh, crash-dump preserves everything.
> 
> Hi Peter,
> 
> Yes, fair point. The intention here is to preserve this state directly
> within the kernel log buffer, particularly for scenarios where kdump is not
> configured, the vmcore was truncated or fails to complete.
> In such circumstances, dmesg, whether captured via pstore or serial, is
> frequently the sole surviving diagnostic artefact.

Ok. So not crash-dump then ;-)

> > > +		local_irq_save(flags);
> > > +		locked = raw_spin_rq_trylock(rq);
> > > +		if (locked) {
> > > +			nr_running = rq->nr_running;
> > > +			nr_switches = rq->nr_switches;
> > > +			curr = rcu_dereference(rq->curr);
> > > +			raw_spin_rq_unlock(rq);
> > > +		} else {
> > > +			nr_running = READ_ONCE(rq->nr_running);
> > > +			nr_switches = READ_ONCE(rq->nr_switches);
> > > +			curr = rcu_dereference(rq->curr);
> > > +		}
> > > +		local_irq_restore(flags);
> > 
> > This seems to want to avoid deadlocking on rq->lock, but then
> > print_rq()->print_cfs_stats() will unconditionally take rq->lock again.
> > 
> > So meh.
> 
> No, print_rq() does not call print_cfs_stats(). It is print_cpu() (i.e.
> used by debugfs and SysRq) that calls print_cfs_stats() and unconditionally
> takes rq->lock via print_cfs_rq().

Right you are. Clearly I got my wires crossed when looking through that.

> With sched_show_runqueues(), it specifically avoids print_cpu() and only
> calls print_rq(NULL, rq, cpu, false, true). Inside print_rq(), it merely
> iterates threads under rcu_read_lock() and calls print_task(), neither of
> which acquires rq->lock. Hence, rq->lock is never taken again, and the
> non-blocking guarantee of the trylock remains intact throughout the entire
> dump.

OK. So perhaps add a few comments / annotations that these paths should
stay clear of rq-lock. It is too easy for it to get added.

> > > +	if (si_mask & SYS_INFO_CPU_RUNQUEUES)
> > > +		sched_show_runqueues();
> > >  }
> > 
> > I really don't know if this is worth the trouble. I have *never* needed
> > this.
> 
> I understand. However, the requirement primarily arises in production
> support. In environments where kdump is not configured or the resulting
> vmcore is truncated, dmesg is frequently the sole surviving diagnostic
> artefact.
> 
> During an NMI-induced panic, capturing per-CPU runqueue depth in the log is
> invaluable for identifying CPU starvation, etc. As with the remainder of
> sys_info, this functionality is strictly opt-in and disabled by default,
> imposing no overhead on systems that do not require it while providing
> vital visibility where full memory dumps are unavailable.

What is the main thing holding 'you' back from enabling kdump
everywhere? AFAIU crash dump can be configured to only dump kernel data
structures, which is a lot more economical than a complete memory dump.

Mostly I just worry about death by a thousand cuts and all that. I don't
have a very strong objection to this, just wondering if we really need
it.