From: Florian Mickler <florian@mickler.org>
To: tj@kernel.org
Cc: linux-kernel@vger.kernel.org, rdunlap@xenotime.net,
linux-doc@vger.kernel.org, Florian Mickler <florian@mickler.org>
Subject: [PATCH] workqueue: document debugging tricks
Date: Thu, 31 Mar 2011 12:37:06 +0200 [thread overview]
Message-ID: <1301567826-6116-1-git-send-email-florian@mickler.org> (raw)
In-Reply-To: <20110331065642.GA3385@htj.dyndns.org>
It is not obvious how to debug run-away workers.
These are some tips given by Tejun on lkml.
Signed-off-by: Florian Mickler <florian@mickler.org>
CC: Tejun Heo <tj@kernel.org>
---
Documentation/workqueue.txt | 38 ++++++++++++++++++++++++++++++++++++++
1 files changed, 38 insertions(+), 0 deletions(-)
diff --git a/Documentation/workqueue.txt b/Documentation/workqueue.txt
index 01c513f..cdbc3c6 100644
--- a/Documentation/workqueue.txt
+++ b/Documentation/workqueue.txt
@@ -12,6 +12,7 @@ CONTENTS
4. Application Programming Interface (API)
5. Example Execution Scenarios
6. Guidelines
+7. Debugging
1. Introduction
@@ -379,3 +380,40 @@ If q1 has WQ_CPU_INTENSIVE set,
* Unless work items are expected to consume a huge amount of CPU
cycles, using a bound wq is usually beneficial due to the increased
level of locality in wq operations and work item execution.
+
+
+7. Debugging
+
+Because the work functions are executed by generic worker threads there are a
+few tricks needed to shed some light on misbehaving workqueue users.
+
+Worker threads show up in the process list as:
+
+root 5671 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/0:1]
+root 5672 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/1:2]
+root 5673 0.0 0.0 0 0 ? S 12:12 0:00 [kworker/0:0]
+root 5674 0.0 0.0 0 0 ? S 12:13 0:00 [kworker/1:0]
+
+If kworkers are going crazy (using too much cpu), there are two types of
+possible problems:
+
+ 1. Something beeing scheduled in rapid succession
+ 2. a single work item that consumes lots of cpu cycles
+
+The first one can be tracked using tracing:
+
+ $ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
+ $ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
+ (wait a few secs)
+ ^C
+
+If something is busy looping on work queueing, it would be dominating
+the output and the offender can be determined with the work item
+function.
+
+For the second type of problem it should be possible to just check the stack
+trace of the offending worker thread.
+
+ $ cat /proc/THE_OFFENDING_KWORKER/stack
+
+The work item's function should be trivially visible in the stack trace.
--
1.7.4.1
next prev parent reply other threads:[~2011-03-31 10:37 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-03-29 4:09 vma corruption in today's -git Dave Jones
2011-03-29 4:19 ` Américo Wang
2011-03-29 4:26 ` Dave Jones
2011-03-29 4:22 ` Linus Torvalds
2011-03-31 3:09 ` excessive kworker activity when idle. (was Re: vma corruption in today's -git) Dave Jones
2011-03-31 3:34 ` Dave Jones
2011-03-31 3:44 ` Linus Torvalds
2011-03-31 4:08 ` Dave Jones
2011-03-31 15:53 ` Linus Torvalds
2011-03-31 16:21 ` Linus Torvalds
2011-03-31 21:38 ` Linus Torvalds
2011-03-31 14:59 ` Paul E. McKenney
2011-03-31 3:37 ` Linus Torvalds
2011-03-31 3:55 ` Dave Jones
2011-03-31 5:32 ` Linus Torvalds
2011-03-31 14:21 ` Arnaldo Carvalho de Melo
2011-03-31 14:58 ` Dave Jones
2011-03-31 15:03 ` Dave Jones
2011-03-31 15:09 ` Dave Jones
2011-03-31 15:45 ` Linus Torvalds
2011-03-31 15:25 ` Linus Torvalds
2011-03-31 15:49 ` Dave Jones
2011-03-31 15:58 ` Linus Torvalds
2011-03-31 16:13 ` Dave Jones
2011-03-31 6:56 ` Tejun Heo
2011-03-31 10:37 ` Florian Mickler [this message]
2011-03-31 11:41 ` [PATCH] workqueue: document debugging tricks Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1301567826-6116-1-git-send-email-florian@mickler.org \
--to=florian@mickler.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rdunlap@xenotime.net \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®