[PATCH v4] tools/workqueue/wq_dump.py: Add busy worker inspection and BH pool states

[PATCH v4] tools/workqueue/wq_dump.py: Add busy worker inspection and BH pool states

[Aaron Tomlin]

Currently, wq_dump.py displays static affinity scopes and pool topology,
offering no visibility into in-flight work items or transient pool
states. Enhance wq_dump.py to:
    1.  Distinguish between normal ("bh") and high-priority ("bh-hi") BH
        worker pools, and report transient pool states such as
        "draining" (POOL_BH_DRAINING) or "disassociated"
        (POOL_DISASSOCIATED) when an associated CPU is offlined.

    2.  Provide live busy worker inspection via a new -b|--busy
        command-line option. This iterates through pool->busy_hash to
        display in-flight workers, identifying their task PID/comm (or
        BH context), target workqueue, callback function, in-flight
        execution duration, and any custom work item description.

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
Changes since v3:

 - Decode strings with errors='replace' during busy worker inspection
   to prevent unhandled UnicodeDecodeError when encountering arbitrary
   or uninitialised bytes locklessly

 - Fall back to hex(func) when prog.symbol() raises LookupError for work
   functions without loaded debugging symbols, preventing the retry loop
   from exhausting attempts and dropping the busy worker from the output
   (Tejun Heo)

 - Link to v3: https://lore.kernel.org/lkml/20260923031344.60494-1-atomlin@atomlin.com/

Changes since v2:

 - Harden lockless busy_hash traversal against idle-list diversion,
   cycles, and iterator read faults by checking WORKER_IDLE, tracking
   visited addresses per bucket, trapping (drgn.FaultError, LookupError)
   across iterator steps, and warning if a pool dump is incomplete
   (Tejun Heo)

 - Replaced bare Exception handling during busy worker inspection with
   drgn.FaultError scoping to avoid masking unexpected errors (Breno Leitao)

 - Link to v2: https://lore.kernel.org/lkml/20260911210625.535285-1-atomlin@atomlin.com/

Changes since v1:

 - Dropped former patch 1/2 ("tools/workqueue/wq_dump.py: Support
   backward compatibility for wq->attrs rename") as it has already been
   merged upstream

 - Corrected POOL_DISASSOCIATED flag handling. Restrict inspection to
   per-CPU non-BH worker pools when their associated CPU is offline,
   preventing "disassociated" from erroneously appearing on every BH and
   unbound pool where the flag persists by design (Tejun Heo)

 - Hardened lockless busy_hash traversal against process_one_work()
   entry and exit windows (Tejun Heo)

 - Preserved backward compatibility for worker->current_start
   (Tejun Heo)

 - Fixed 32-bit jiffies wrap-around; masked (jiffies - start) to the
   target architecture's word size (jiffies_mask) so duration arithmetic
   does not yield negative values following INITIAL_JIFFIES overflow
   (Tejun Heo)

 - Link to v1: https://lore.kernel.org/lkml/20260831181554.117795-1-atomlin@atomlin.com/
---
 tools/workqueue/wq_dump.py | 87 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 84 insertions(+), 3 deletions(-)

diff --git a/tools/workqueue/wq_dump.py b/tools/workqueue/wq_dump.py
index 9313ebe0c525..ecdc434f1989 100644
--- a/tools/workqueue/wq_dump.py
+++ b/tools/workqueue/wq_dump.py
@@ -29,6 +29,10 @@ Lists all worker pools indexed by their ID. For each pool:
   workers   number of all workers
   cpu       CPU the pool is associated with (per-cpu pool)
   cpus      CPUs the workers in the pool can run on (unbound pool)
+  flags     pool flags (bh, draining, disassociated)
+
+  If -b|--busy is specified, lists all busy workers currently executing
+  work items, their task PID/comm, workqueue, callback function, and duration.
 
 Workqueue CPU -> pool
 =====================
@@ -49,12 +53,14 @@ import sys
 import argparse
 parser = argparse.ArgumentParser(description=desc,
                                  formatter_class=argparse.RawTextHelpFormatter)
+parser.add_argument('-b', '--busy', action='store_true',
+                    help='Show busy workers currently executing work items')
 args = parser.parse_args()
 
 import drgn
-from drgn.helpers.linux.list import list_for_each_entry,list_empty
+from drgn.helpers.linux.list import list_for_each_entry, list_empty, hlist_for_each_entry
 from drgn.helpers.linux.percpu import per_cpu_ptr
-from drgn.helpers.linux.cpumask import for_each_cpu,for_each_possible_cpu
+from drgn.helpers.linux.cpumask import for_each_cpu, for_each_possible_cpu
 from drgn.helpers.linux.nodemask import for_each_node
 from drgn.helpers.linux.idr import idr_for_each
 
@@ -62,6 +68,10 @@ def err(s):
     print(s, file=sys.stderr, flush=True)
     sys.exit(1)
 
+def get_hz():
+    cs = prog['clocksource_jiffies']
+    return round(1000000000 / (cs.mult.value_() >> cs.shift.value_()))
+
 def cpumask_str(cpumask):
     output = ""
     base = 0
@@ -84,6 +94,12 @@ def wq_attrs(wq):
     except AttributeError:
         return wq.unbound_attrs
 
+def worker_current_start(worker):
+    try:
+        return worker.current_start.value_()
+    except AttributeError:
+        return 0
+
 def wq_type_str(wq):
     if wq.flags & WQ_BH:
         return f'{"bh":{wq_type_len}}'
@@ -118,9 +134,15 @@ WQ_AFFN_NUMA            = prog['WQ_AFFN_NUMA']
 WQ_AFFN_SYSTEM          = prog['WQ_AFFN_SYSTEM']
 
 POOL_BH                 = prog['POOL_BH']
+POOL_BH_DRAINING        = prog['POOL_BH_DRAINING']
+POOL_DISASSOCIATED      = prog['POOL_DISASSOCIATED']
+HIGHPRI_NICE_LEVEL      = prog['HIGHPRI_NICE_LEVEL']
+WORKER_IDLE             = prog['WORKER_IDLE']
 
 WQ_NAME_LEN             = prog['WQ_NAME_LEN'].value_()
 cpumask_str_len         = len(cpumask_str(wq_unbound_cpumask))
+hz                      = get_hz()
+jiffies_mask            = (1 << (prog['jiffies'].type_.size * 8)) - 1 if 'jiffies' in prog else 0
 
 print('Affinity Scopes')
 print('===============')
@@ -168,7 +190,12 @@ for pi, pool in idr_for_each(worker_pool_idr):
     if pool.cpu >= 0:
         print(f'cpu={pool.cpu.value_():3}', end='')
         if pool.flags & POOL_BH:
-            print(' bh', end='')
+            bh_type = 'bh-hi' if pool.attrs.nice == HIGHPRI_NICE_LEVEL else 'bh'
+            print(f' {bh_type}', end='')
+            if pool.flags & POOL_BH_DRAINING:
+                print(' draining', end='')
+        elif pool.flags & POOL_DISASSOCIATED:
+            print(' disassociated', end='')
     else:
         print(f'cpus={cpumask_str(pool.attrs.cpumask)}', end='')
         print(f' pod_cpus={cpumask_str(pool.attrs.__pod_cpumask)}', end='')
@@ -176,6 +203,60 @@ for pi, pool in idr_for_each(worker_pool_idr):
             print(' strict', end='')
     print('')
 
+    if args.busy:
+        incomplete = False
+        for bkt in pool.busy_hash:
+            if incomplete:
+                break
+            seen = set()
+            try:
+                for worker in hlist_for_each_entry('struct worker', bkt.address_of_(), 'hentry'):
+                    addr = worker.value_()
+                    if addr in seen or (worker.flags & WORKER_IDLE):
+                        incomplete = True
+                        break
+                    seen.add(addr)
+
+                    for _ in range(3):
+                        try:
+                            pwq = worker.current_pwq
+                            func = worker.current_func.value_()
+                            if not pwq.value_() or not func:
+                                continue
+
+                            wq_name = pwq.wq.name.string_().decode(errors='replace')
+                            try:
+                                fn_name = prog.symbol(func).name
+                            except LookupError:
+                                fn_name = hex(func)
+
+                            dur_str = ''
+                            start = worker_current_start(worker)
+                            if 'jiffies' in prog and start:
+                                jiffies = prog['jiffies'].value_()
+                                dur_s = ((jiffies - start) & jiffies_mask) // hz
+                                dur_str = f' for {dur_s}s'
+
+                            if pool.flags & POOL_BH:
+                                w_id = 'bh' if pool.attrs.nice != HIGHPRI_NICE_LEVEL else 'bh-hi'
+                            elif worker.task.value_():
+                                w_id = f'PID {worker.task.pid.value_():<6} ({worker.task.comm.string_().decode(errors="replace")})'
+                            else:
+                                w_id = f'worker[{worker.id.value_()}]'
+
+                            desc = worker.desc.string_().decode(errors='replace')
+                            desc_str = f' desc="{desc}"' if desc and desc != wq_name else ''
+                            print(f'    busy: {w_id}: {wq_name}:{fn_name}{dur_str}{desc_str}')
+                            break
+                        except drgn.FaultError:
+                            continue
+            except (drgn.FaultError, LookupError):
+                incomplete = True
+                break
+
+        if incomplete:
+            print(f'    warning: pool[{pi:02}] busy worker dump incomplete, please retry')
+
 print('')
 print('Workqueue CPU -> pool')
 print('=====================')
-- 
2.55.0

Re: [PATCH v4] tools/workqueue/wq_dump.py: Add busy worker inspection and BH pool states

[Tejun Heo]

Hello, Aaron.

On Wed, Sep 30, 2026 at 10:39:58AM -0400, Aaron Tomlin wrote:
> +  flags     pool flags (bh, draining, disassociated)

bh-hi is missing. Also, the pool line already has a numeric flags=0x..
field, so using the same label for these is a bit confusing.

> +            bh_type = 'bh-hi' if pool.attrs.nice == HIGHPRI_NICE_LEVEL else 'bh'

The busy worker loop computes this again. Maybe compute it once per pool
and reuse it there?

> +        for bkt in pool.busy_hash:
> +            if incomplete:
> +                break

After the first bad bucket, this and the break in the except below skip
the rest of the pool. Maybe move on to the next bucket instead? The
warning still gets printed and the busy workers in the other buckets
aren't lost.

> +                            elif worker.task.value_():
...
> +                            else:
> +                                w_id = f'worker[{worker.id.value_()}]'

Only BH workers have NULL ->task and they take the POOL_BH branch, so this
can't be reached. Can we drop it?

Thanks.

--
tejun