Re: [PATCH RFC 2/3] workqueue: trigger a single-CPU backtrace for stalled pools

[Petr Mladek <pmladek@suse.com>]

On Tue 2026-06-16 09:44:40, Breno Leitao wrote:
> When a CPU pool is stalled with no running worker, the task occupying the
> CPU may not be a workqueue worker at all.  Trigger a single-CPU backtrace
> for the stalled CPU to capture what it is currently executing.
> 
> The CPU is snapshotted under pool->lock and the backtrace is triggered
> after releasing the lock to avoid any potential issues with NMI delivery.
> 
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -7751,6 +7754,14 @@ static void show_cpu_pool_busy_workers(struct worker_pool *pool)
>  		show_pool_no_running_worker(pool);
>  
>  	raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
> +
> +	/*
> +	 * Trigger a backtrace on the stalled CPU to capture what it is
> +	 * currently executing. Called after releasing the lock to avoid
> +	 * any potential issues with NMI delivery.
> +	 */
> +	if (!found_running)
> +		trigger_single_cpu_backtrace(cpu);
>  }

Sashiko AI is curious whether this might be racy against CPU hotplug,
see
https://sashiko.dev/#/patchset/20260616-wq_dump_petr-v1-0-b57473ca6d18%40debian.org

<paste Sashiko comment>
Is it possible for this CPU to be offline when we trigger the backtrace?

When a CPU goes offline, its bound workqueue pools are disassociated via
unbind_workers() but retain their original pool->cpu ID. If a work item on
a disassociated pool hangs, the watchdog could detect the stall and invoke
show_cpu_pool_busy_workers().

If the target CPU is offline, it cannot process the NMI or clear its
completion bit in the backtrace mask. Does this cause
nmi_trigger_cpumask_backtrace() to busy-wait for the 10-second timeout
waiting for a response that will never arrive? Since the watchdog executes
in a softirq timer context, this could stall the CPU running the watchdog
for 10 seconds.

Should this check cpu_online(cpu) before triggering the backtrace?
</paste Sashiko comment>

It makes some sense. wq_watchdog_timer_fn() checks either
'per_cpu(wq_watchdog_touched_cpu)' or the global 'wq_watchdog_touched'
depending whether pool->cpu is set or not. And it seems to be wrong
for disassociated pools.

But this seems to be an existing problem which should be fixed
separately.

Also wq_watchdog_timer_fn() is checking the state without taking
proper locks. So it is inherently racy. But it is another story.

Summary:

This particular change looks good to me. And I believe that it
is useful.

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr