From: Wilson Felipe Pereira <wfelipe@google.com>
To: Tejun Heo <tj@kernel.org>, Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
Randy Dunlap <rdunlap@infradead.org>,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
Wilson Felipe Pereira <wfelipe@google.com>,
Shuhao Tan <tanshuhao@google.com>
Subject: [PATCH 3/3] tools/workqueue: report queued items and retired stats in wq_monitor.py
Date: Thu, 22 Oct 2026 15:53:36 +0000 [thread overview]
Message-ID: <20261022155352.1293020-4-wfelipe@google.com> (raw)
In-Reply-To: <20261022155352.1293020-1-wfelipe@google.com>
Update wq_monitor.py to include wq->retired_stats[] when available so
cumulative counters do not drop when unbound pwqs are released. Also sum
pwq->total_in_flight (falling back to pwq->nr_in_flight[] on older
kernels) and add a 'queued' column next to 'infl' to show work items
waiting to execute.
Update Documentation/core-api/workqueue.rst to cover
/sys/kernel/debug/workqueue/stats and the new 'queued' column in
wq_monitor.py.
Signed-off-by: Wilson Felipe Pereira <wfelipe@google.com>
---
Documentation/core-api/workqueue.rst | 47 +++++++++++++++++-----------
tools/workqueue/wq_monitor.py | 24 ++++++++++++--
2 files changed, 50 insertions(+), 21 deletions(-)
diff --git a/Documentation/core-api/workqueue.rst b/Documentation/core-api/workqueue.rst
index bb770f556568..a4e08231580c 100644
--- a/Documentation/core-api/workqueue.rst
+++ b/Documentation/core-api/workqueue.rst
@@ -701,28 +701,39 @@ See the command's help message for more info.
Monitoring
==========
-Use tools/workqueue/wq_monitor.py to monitor workqueue operations: ::
+When ``CONFIG_DEBUG_FS`` is enabled, per-workqueue stats can be read
+directly from ``/sys/kernel/debug/workqueue/stats``: ::
+
+ $ cat /sys/kernel/debug/workqueue/stats
+ workqueue type queued active running started completed cpu_time_us cpu_itsv cm_wkup repat mayday rescued
+ events percpu 0 0 0 18545 18545 6100000 0 5 0 0 0
+ events_highpri percpu 0 0 0 8 8 0 0 0 0 0 0
+ events_long percpu 0 0 0 3 3 0 0 0 0 0 0
+ events_unbound unbound 0 0 0 38306 38306 100000 0 0 7 0 0
+
+Alternatively, use tools/workqueue/wq_monitor.py to continuously monitor
+workqueue operations via ``drgn``: ::
$ tools/workqueue/wq_monitor.py events
- total infl CPUtime CPUhog CMW/RPR mayday rescued
- events 18545 0 6.1 0 5 - -
- events_highpri 8 0 0.0 0 0 - -
- events_long 3 0 0.0 0 0 - -
- events_unbound 38306 0 0.1 - 7 - -
- events_freezable 0 0 0.0 0 0 - -
- events_power_efficient 29598 0 0.2 0 0 - -
- events_freezable_pwr_ef 10 0 0.0 0 0 - -
+ total queued infl CPUtime CPUitsv CMW/RPR mayday rescued
+ events 18545 0 0 6.1 0 5 - -
+ events_highpri 8 0 0 0.0 0 0 - -
+ events_long 3 0 0 0.0 0 0 - -
+ events_unbound 38306 0 0 0.1 - 7 - -
+ events_freezable 0 0 0 0.0 0 0 - -
+ events_power_efficient 29598 0 0 0.2 0 0 - -
+ events_freezable_pwr_ef 10 0 0 0.0 0 0 - -
sock_diag_events 0 0 0.0 0 0 - -
- total infl CPUtime CPUhog CMW/RPR mayday rescued
- events 18548 0 6.1 0 5 - -
- events_highpri 8 0 0.0 0 0 - -
- events_long 3 0 0.0 0 0 - -
- events_unbound 38322 0 0.1 - 7 - -
- events_freezable 0 0 0.0 0 0 - -
- events_power_efficient 29603 0 0.2 0 0 - -
- events_freezable_pwr_ef 10 0 0.0 0 0 - -
- sock_diag_events 0 0 0.0 0 0 - -
+ total queued infl CPUtime CPUitsv CMW/RPR mayday rescued
+ events 18548 0 0 6.1 0 5 - -
+ events_highpri 8 0 0 0.0 0 0 - -
+ events_long 3 0 0 0.0 0 0 - -
+ events_unbound 38322 0 0 0.1 - 7 - -
+ events_freezable 0 0 0 0.0 0 0 - -
+ events_power_efficient 29603 0 0 0.2 0 0 - -
+ events_freezable_pwr_ef 10 0 0 0.0 0 0 - -
+ sock_diag_events 0 0 0 0.0 0 0 - -
...
diff --git a/tools/workqueue/wq_monitor.py b/tools/workqueue/wq_monitor.py
index 7f47fa398e3c..2f4176df2a36 100644
--- a/tools/workqueue/wq_monitor.py
+++ b/tools/workqueue/wq_monitor.py
@@ -9,7 +9,9 @@ https://github.com/osandov/drgn.
total Total number of work items executed by the workqueue.
- infl The number of currently in-flight work items.
+ queued The number of work items currently queued and waiting to execute.
+
+ infl The number of currently executing (in-flight) work items.
CPUtime Total CPU time consumed by the workqueue in seconds. This is
sampled from scheduler ticks and only provides ballpark
@@ -55,6 +57,7 @@ workqueues = prog['workqueues']
WQ_UNBOUND = prog['WQ_UNBOUND']
WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM']
+WORK_NR_COLORS = prog['WORK_NR_COLORS']
PWQ_STAT_STARTED = prog['PWQ_STAT_STARTED'] # work items started execution
PWQ_STAT_COMPLETED = prog['PWQ_STAT_COMPLETED'] # work items completed execution
@@ -72,15 +75,29 @@ class WqStats:
self.unbound = wq.flags & WQ_UNBOUND != 0
self.mem_reclaim = wq.flags & WQ_MEM_RECLAIM != 0
self.stats = [0] * PWQ_NR_STATS
+ self.in_flight = 0
+ try:
+ for i in range(PWQ_NR_STATS):
+ self.stats[i] += int(wq.retired_stats[i])
+ except AttributeError:
+ pass
for pwq in list_for_each_entry('struct pool_workqueue', wq.pwqs.address_of_(), 'pwqs_node'):
for i in range(PWQ_NR_STATS):
self.stats[i] += int(pwq.stats[i])
+ try:
+ self.in_flight += int(pwq.total_in_flight)
+ except AttributeError:
+ for c in range(WORK_NR_COLORS):
+ self.in_flight += int(pwq.nr_in_flight[c])
+ self.running = max(self.stats[PWQ_STAT_STARTED] - self.stats[PWQ_STAT_COMPLETED], 0)
+ self.queued = max(self.in_flight - self.running, 0)
def dict(self, now):
return { 'timestamp' : now,
'name' : self.name,
'unbound' : self.unbound,
'mem_reclaim' : self.mem_reclaim,
+ 'queued' : self.queued,
'started' : self.stats[PWQ_STAT_STARTED],
'completed' : self.stats[PWQ_STAT_COMPLETED],
'cpu_time' : self.stats[PWQ_STAT_CPU_TIME],
@@ -91,7 +108,7 @@ class WqStats:
'rescued' : self.stats[PWQ_STAT_RESCUED], }
def table_header_str():
- return f'{"":>24} {"total":>8} {"infl":>5} {"CPUtime":>8} '\
+ return f'{"":>24} {"total":>8} {"queued":>6} {"infl":>5} {"CPUtime":>8} '\
f'{"CPUitsv":>7} {"CMW/RPR":>7} {"mayday":>7} {"rescued":>7}'
def table_row_str(self):
@@ -112,7 +129,8 @@ class WqStats:
out = f'{self.name[-24:]:24} ' \
f'{self.stats[PWQ_STAT_STARTED]:8} ' \
- f'{max(self.stats[PWQ_STAT_STARTED] - self.stats[PWQ_STAT_COMPLETED], 0):5} ' \
+ f'{self.queued:6} ' \
+ f'{self.running:5} ' \
f'{self.stats[PWQ_STAT_CPU_TIME] / 1000000:8.1f} ' \
f'{cpu_intensive:>7} ' \
f'{cmw_rpr:>7} ' \
--
2.56.0.218.g1ec904b173-goog
prev parent reply other threads:[~2026-10-22 15:54 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-22 15:53 [PATCH 0/3] workqueue: expose per-workqueue stats in debugfs Wilson Felipe Pereira
2026-10-22 15:53 ` [PATCH 1/3] workqueue: preserve pwq->stats[] when unbound pwqs are released Wilson Felipe Pereira
2026-10-22 15:53 ` [PATCH 2/3] workqueue: expose per-workqueue stats in debugfs Wilson Felipe Pereira
2026-10-22 15:53 ` Wilson Felipe Pereira [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261022155352.1293020-4-wfelipe@google.com \
--to=wfelipe@google.com \
--cc=corbet@lwn.net \
--cc=jiangshanlai@gmail.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rdunlap@infradead.org \
--cc=skhan@linuxfoundation.org \
--cc=tanshuhao@google.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®