From: Petr Mladek <pmladek@suse.com>
To: Tejun Heo <tj@kernel.org>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>,
Michal Koutny <mkoutny@suse.com>,
linux-kernel@vger.kernel.org, Petr Mladek <pmladek@suse.com>
Subject: [RFC 4/5] workqueue: Warn when a rescuer could not be created
Date: Wed, 1 Feb 2023 14:45:42 +0100 [thread overview]
Message-ID: <20230201134543.13687-5-pmladek@suse.com> (raw)
In-Reply-To: <20230201134543.13687-1-pmladek@suse.com>
Rescuers are created when a workqueue with WQ_MEM_RECLAIM is allocated.
It typically happens during the system boot.
systemd switches the root filesystem from initrd to the booted system
during boot. It kills processes that block the switch for too long.
One of the process might be modprobe that tries to create a workqueue.
These problems are hard to reproduce. Also alloc_workqueue() does not
pass the error code. Make the debugging easier by printing a warning,
similar to create_worker().
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
kernel/workqueue.c | 31 +++++++++++++++++++++++++------
1 file changed, 25 insertions(+), 6 deletions(-)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 5f3327e119b4..58b835420435 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -1914,9 +1914,11 @@ static void worker_detach_from_pool(struct worker *worker)
static int create_worker_failed;
static DEFINE_SPINLOCK(create_worker_failed_lock);
-static __printf(2, 3) __cold
-void __print_create_worker_failure(long err, const char *fmt, ...)
+static __printf(3, 4) __cold
+void __print_create_worker_failure(bool is_rescuer, long err, const char *fmt, ...)
{
+ bool one_off_failure;
+
spin_lock_irq(&create_worker_failed_lock);
/*
@@ -1925,7 +1927,8 @@ void __print_create_worker_failure(long err, const char *fmt, ...)
* touch watchdogs so that more frequent messages would prevent
* reaching the watchdog thresh.
*/
- if (!create_worker_failed || err == -EINTR) {
+ one_off_failure = (err == -EINTR || is_rescuer);
+ if (!create_worker_failed || one_off_failure) {
va_list args;
va_start(args, fmt);
@@ -1933,7 +1936,7 @@ void __print_create_worker_failure(long err, const char *fmt, ...)
va_end(args);
}
- if (err != -EINTR)
+ if (!one_off_failure)
create_worker_failed++;
spin_unlock_irq(&create_worker_failed_lock);
@@ -1943,7 +1946,9 @@ void __print_create_worker_failure(long err, const char *fmt, ...)
do { \
long _err = err; \
\
- __print_create_worker_failure(_err, KERN_WARNING msg ":%pe\n", (void *)_err); \
+ __print_create_worker_failure(false, _err, \
+ KERN_WARNING msg ":%pe\n", \
+ (void *)_err); \
} while (0)
static void print_create_worker_success(void)
@@ -4323,6 +4328,15 @@ static int wq_clamp_max_active(int max_active, unsigned int flags,
return clamp_val(max_active, 1, lim);
}
+#define print_create_rescuer_failure(msg, name, err) \
+ do { \
+ long _err = err; \
+ \
+ __print_create_worker_failure(true, _err, \
+ KERN_WARNING msg ": %s :%pe\n", \
+ name, (void *)_err); \
+ } while (0)
+
/*
* Workqueues which may be used during memory reclaim should have a rescuer
* to guarantee forward progress.
@@ -4336,13 +4350,18 @@ static int init_rescuer(struct workqueue_struct *wq)
return 0;
rescuer = alloc_worker(NUMA_NO_NODE);
- if (!rescuer)
+ if (!rescuer) {
+ print_create_rescuer_failure("workqueue: Failed to allocate a rescuer",
+ wq->name, -ENOMEM);
return -ENOMEM;
+ }
rescuer->rescue_wq = wq;
rescuer->task = kthread_create(rescuer_thread, rescuer, "%s", wq->name);
if (IS_ERR(rescuer->task)) {
ret = PTR_ERR(rescuer->task);
+ print_create_rescuer_failure("workqueue: Failed to create a rescuer thread",
+ wq->name, ret);
kfree(rescuer);
return ret;
}
--
2.35.3
next prev parent reply other threads:[~2023-02-01 13:46 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-01 13:45 [RFC 0/5] workqueue: Debugging improvements Petr Mladek
2023-02-01 13:45 ` [RFC 1/5] workqueue: Fix hung time report of worker pools Petr Mladek
2023-02-01 13:45 ` [RFC 2/5] workqueue: Warn when a new worker could not be created Petr Mladek
2023-02-02 23:30 ` Tejun Heo
2023-02-03 14:10 ` Petr Mladek
2023-02-03 19:29 ` Tejun Heo
2023-02-15 18:02 ` Michal Koutný
2023-02-16 9:43 ` Petr Mladek
2023-02-01 13:45 ` [RFC 3/5] workqueue: Interrupted create_worker() is not a repeated event Petr Mladek
2023-02-01 13:45 ` Petr Mladek [this message]
2023-02-01 13:45 ` [RFC 5/5] workqueue: Print backtraces from CPUs with hung CPU bound workqueues Petr Mladek
2023-02-02 23:45 ` Tejun Heo
2023-02-03 14:26 ` Petr Mladek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230201134543.13687-5-pmladek@suse.com \
--to=pmladek@suse.com \
--cc=jiangshanlai@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mkoutny@suse.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®