From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7567E246798 for ; Wed, 11 Feb 2026 12:32:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813172; cv=none; b=KzJEO4UPLqjDTpiezqXqZ3jnKQMtDj/8hGmwJ52UBiBbPv7DdHz5sp/bQW4EUK7OI+wtxFevFdTLLuPnX041snNGRNiKbcBkaL3Rd/gncwjoUJcXcY8BjDt0ZP0ofUETnzo6Ym7LEtXe1ChrC24BcXJ1uiFkP+TiFYd8rmY9Rh8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813172; c=relaxed/simple; bh=t77vSLWgXMMeEcQf9E3d/8zqaT20FgoqfU8sZDvzcVw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=khqQe5l2GjQFySSSZ8p5uZaEbf8CMxszSwzVIiz5d8wbTNwtWcLp8NUOSr2mrC9t7H7im9v6I31gOjSe23mquXl552xabe/Hxi2XoRDDwlVKYzSUxwbbzicDI1cREJ2+t3sA818enGCLsnicNQsqi6ixdXgM77WTxQNuPF9M/cw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=jvSswlwv; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="jvSswlwv" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=vzGfRiXeh6X8RFTmty5cMTgqMHPVmYvoXIEuiBDVbgk=; b=jvSswlwvMZi429+mqVZYNGAKaQ MzlwpofH08012DkltNTEsB/WRnFZj3gEF0oeQAxPE9TevaF7QSgpGRlWCCoknQiDrEJ4W7r8Zfwnq pu+v89qBtAAGPK5IoLEcxOslxwTy1tO/FY6dQmABE4ZE15TkgEYWM6BBqdAM0bcl3pbh6LbOAeLGF rwJc0mw7lCHRicSoGN6cfUYcKVtUjApAhzKN06QWuNinR6R6e+KFftZ8+tJEfh1PfNz1ZZ+53oHNg XEkL8xidiSp7IykIK5kR2zOJoDNiBliGYuOBGVRF07YIAFfxPLF26+tAV7Hk9tBQdmSsU7xkzjzj0 UJWTlT7A==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1vq9Og-00B23B-MP; Wed, 11 Feb 2026 12:32:47 +0000 From: Breno Leitao Date: Wed, 11 Feb 2026 04:29:18 -0800 Subject: [PATCH 4/4] workqueue: Detect stalled in-flight work items with empty worklist Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260211-wqstall_start-at-v1-4-bd9499a18c19@debian.org> References: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> In-Reply-To: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, Omar Sandoval , kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.15-dev-f4305 X-Developer-Signature: v=1; a=openpgp-sha256; l=5044; i=leitao@debian.org; h=from:subject:message-id; bh=t77vSLWgXMMeEcQf9E3d/8zqaT20FgoqfU8sZDvzcVw=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpjHbcKPhWrwIYvHDyv8c7iswhohrChgddiukIr 5f/o67TAiOJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaYx23AAKCRA1o5Of/Hh3 bQRXEACSlLnvopofkkSd4WBguhrh3VvU7H3yvBwsWxj3Jx95/ias31MgH3mjjaKRLhdrqpZHsRk KbZ6/ZaF+lOOpKEKUkzRFtR8EMyhQ3XePtWVpcNmp7OwOFBYZqXKzu0LIiV0hHmXg926FNwUcrG ljVr2rZu2VpF6JCPIEaO8dzxTmHBGKrNeacqc81WDwrq5CRJDMXDbhbU/vsn/f/AdDVMfrepcbD FoMnQtw5+CimZADW3PZQ0dXMtgd7usLBtTTw+hZhWZe9VvS5SxgOxVaSbMYu6r4+UbxSbg/mNNI Cgp/A802X17lDF9B3o/RU222K/ufRnnPF9jr0ex/7Qi1wKQecvjks5dbCpCbMPCdQx/G9r+EedO /kcH6g7egdYjs5NBfNMKpHAt1pFwYjelg68p7VJjjcMVczg6+oCjyGaBabSkZ4ZBCiNshkysv1m TYWU9J9b7fhfUplGR5NyAzWd+F35Sh9zQQMHz263jtjZzuHkFYhkFcvlY79x5iAHOrLW3rVsJD7 T+n2N7mC7RmfQdxxboXvTMdxSXGEhfq7drJiWrWVxYuid5+bkYC1tYbbZKnWBk3+jZHXUUywY+D ECvLfY0UvFYl110Cem33EZSTWu+/L3Tn2ufPtgHplBC0X8flAr8lijk9+AES4f8kYN87I1UwAhX k+p9MxoWFuQWCvQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The workqueue watchdog skips pools with an empty worklist, assuming no work is pending. However, a single work item that was dequeued and is now executing on a worker will leave the worklist empty while the worker is stuck. This means a pool with one hogged worker and no pending work is invisible to the watchdog. An example is something like: static void stall_work_fn(struct work_struct *work) { for (;;) { mdelay(1000); cond_resched(); } } Fix this by scanning the pool's busy_hash for workers whose current_start timestamp exceeds the watchdog threshold, independent of worklist state. The new report_stalled_workers() function iterates all in-flight workers in a pool and reports each one that has exceeded the threshold, running as a separate detection path alongside the existing pool-level last_progress_ts check. This is an example of the report: BUG: workqueue lockup - worker 365:stall_work_fn [wq_stall] stuck in pool cpus=9 node=0 flags=0x0 nice=0 for 33s! Showing busy workqueues and worker pools: ... The feature is gated behind a new CONFIG_WQ_WATCHDOG_WORKERS option (disabled by default) under CONFIG_WQ_WATCHDOG. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- lib/Kconfig.debug | 12 ++++++++++++ 2 files changed, 62 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index e527e763162e6..719e14aa4ac56 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -7659,6 +7659,49 @@ static void wq_watchdog_reset_touched(void) per_cpu(wq_watchdog_touched_cpu, cpu) = jiffies; } +#ifdef CONFIG_WQ_WATCHDOG_WORKERS +/* + * Scan all in-flight workers in @pool for stalls. A worker is considered + * stalled if its current work item has been executing for longer than @thresh + * based on its current_start timestamp. This catches workers that are stuck + * regardless of the pool's worklist state or last_progress_ts. + */ +static bool report_stalled_workers(struct worker_pool *pool, + unsigned long now, + unsigned long thresh) +{ + struct worker *worker; + bool stall = false; + int bkt; + + /* + * Iterate busy_hash without pool->lock. This is intentionally + * lockless to avoid contention in the watchdog timer path. + * Workers that have been stalled for thresh (typically 30s) are + * unlikely to be transitioning in/out of busy_hash concurrently. + */ + hash_for_each(pool->busy_hash, bkt, worker, hentry) { + if (time_after(now, worker->current_start + thresh)) { + pr_emerg("BUG: workqueue lockup - worker "); + pr_cont_worker_id(worker); + pr_cont(":%ps stuck in pool", + worker->current_func); + pr_cont_pool_info(pool); + pr_cont(" for %us!\n", + jiffies_to_msecs(now - worker->current_start) / 1000); + stall = true; + } + } + return stall; +} +#else +static bool report_stalled_workers(struct worker_pool *pool, + unsigned long now, unsigned long thresh) +{ + return false; +} +#endif /* CONFIG_WQ_WATCHDOG_WORKERS */ + static void wq_watchdog_timer_fn(struct timer_list *unused) { unsigned long thresh = READ_ONCE(wq_watchdog_thresh) * HZ; @@ -7677,8 +7720,6 @@ static void wq_watchdog_timer_fn(struct timer_list *unused) unsigned long pool_ts, touched, ts; pool->cpu_stall = false; - if (list_empty(&pool->worklist)) - continue; /* * If a virtual machine is stopped by the host it can look to @@ -7686,6 +7727,13 @@ static void wq_watchdog_timer_fn(struct timer_list *unused) */ kvm_check_and_clear_guest_paused(); + /* Check for individual stalled workers in this pool. */ + if (report_stalled_workers(pool, now, thresh)) + lockup_detected = true; + + if (list_empty(&pool->worklist)) + continue; + /* get the latest of pool and touched timestamps */ if (pool->cpu >= 0) touched = READ_ONCE(per_cpu(wq_watchdog_touched_cpu, pool->cpu)); diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce25a8faf6e9e..dc4bb546b2033 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1320,6 +1320,18 @@ config BOOTPARAM_WQ_STALL_PANIC This setting can be overridden at runtime via the workqueue.panic_on_stall kernel parameter. +config WQ_WATCHDOG_WORKERS + bool "Detect individual stalled workqueue workers" + depends on WQ_WATCHDOG + default n + help + Say Y here to enable per-worker stall detection. When enabled, + the workqueue watchdog scans all in-flight workers in each pool + and reports any whose current work item has been executing for + longer than the watchdog threshold. This catches stalled workers + even when the pool's worklist is empty or the pool has recently + made forward progress on other work items. + config WQ_CPU_INTENSIVE_REPORT bool "Report per-cpu work items which hog CPU for too long" depends on DEBUG_KERNEL -- 2.47.3