From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C40933595D for ; Wed, 11 Feb 2026 12:32:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813161; cv=none; b=JgC5O/a+hfelo4chHUbhPq25ryffpcdKQvGoAcMiBaTHe6nGbBn1UBf9XuAgLaPrDZuwxDZvENh0z7aYfNaP+5C6vONPmfJa9nSYcxgsIRk5GhWg/8NkBN+HkBaEye/H1FCE4PTHCUoKPs6Cg3/5fbxvjpMtaj+Q5WElFgNDlcg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813161; c=relaxed/simple; bh=EgHL4WoD7RSWIi6zoGA1TwXqNYmEytPdkuvXpAvID38=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=r/fqP2tYytlF4RVbbbzY7RiZdocY4WZJog+BJSJOroHGxhVUictsfq/boFIJCIFKjqV3bsvYaDuBBhjRnMSi/2L3Behkv7ce0lJxsJyyPNRrpwNvTWTHcpkX1/RlaVvRw5gBqYZ4soEcZmavvK8mWpmeBswf8ANpMQekXidxMhg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=iO7NylRk; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="iO7NylRk" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-Id:Date:Subject:From:Reply-To:Content-ID: Content-Description:In-Reply-To:References; bh=Z3fOxBI/bYVxNtMDchGH0WeIM+x28JijSD1whZh63Ls=; b=iO7NylRkTNySs5wzGMweDs/cej ffIzqbGwZLCPA0hhTV/EOwC98S97Xun6cxE/+Ko9ua1oQxKkwo0sCuetY/y/psZqddoIV5fDdTBXm B6A202uL7HEHZ3Hz2Mb/ZYoFhkMHT7zh1EcMXFgyG4JgJcDJcAR1aeLPNuUZMstznUxFwtVWaub4O UTXlD8v2Qmio+u2Ve9XqOjuHaBJlLjM7FljOhOQEHdlVMACB3Zk8v4BxWRIadrWaD1v9aegrw9NhM 4+B21nCPi+VzIMq8Sn+XHny3zzlkDUlCWZTXtgeSe7xRKuIh2+S87C0O/IWCLHBtazw6Vowekr331 JtHEGKJw==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1vq9OS-00B22U-Vv; Wed, 11 Feb 2026 12:32:33 +0000 From: Breno Leitao Subject: [PATCH 0/4] workqueue: Detect stalled in-flight workers Date: Wed, 11 Feb 2026 04:29:14 -0800 Message-Id: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIABt2jGkC/yXMywrCMBAF0F8Z7rqBSXxhfkVEpnXUSKk2Mz6g9 N9FXZ7NmWBaixoyTaj6LFZuAzLFhtBdZDhrKEdkQuK05hQ5vEZz6fuDuVQP4kE3i7gV5pW0SzS Ee9VTef/K3f5ve7RX7fz7YJ4/XNbYcnQAAAA= X-Change-ID: 20260210-wqstall_start-at-e7319a005ab4 To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, Omar Sandoval , kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.15-dev-f4305 X-Developer-Signature: v=1; a=openpgp-sha256; l=3192; i=leitao@debian.org; h=from:subject:message-id; bh=EgHL4WoD7RSWIi6zoGA1TwXqNYmEytPdkuvXpAvID38=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpjHbc+8u5Jf00Xr4UE/TR8ozus+ZIvfEb85xN6 0cvgVqoGKWJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaYx23AAKCRA1o5Of/Hh3 bZOyEACEEYAkSLkWfxyz7jPZ5y20QjPKdwH5RotdEwTd1immQt8JfLaQHueI3JYUoBvB5cQjTb8 ile5pGLxxkN190wQ1qgZohB/CGlgJfON/gh4Hr/0AyS7xEgPMNt5Z15q7csbWJtXBK8LmvLQC8c 5k7kIILURMpSuxlTZGHtOu0SqbPjYmwFQANpIefQCAWhygz5gz98LAoEOa6e6fi4QoI2jxi3aYJ bIzEQR6l0jBW5RgnW8vWEAb2W/EEKMR3UJFenO1QiPzF7DpWo6tvQwJcmwxcsA71hfo4Kh+HSHU ggluLcABnT9GAOImXPOi009fzu5J6EB5BIRjK/PP6L1PYclG51XuaPkdiPgVrm3J8uwqCuFhbCk 2AHJ10cIPtmuHbmFeu1/ZLSdo0z/lbFTQj4hPwI9/Vtu7uBDq/lrOfMfn0rCmsmu9W8bRcuTsn6 K0cWcKrflll+0Ep03zbrPTMkhgHR17NjvFaj+VSkUh3UoUOHcwQzrmJv65moZHUUHl18yqL9Ymu BYDYdbKQhv8n4eUKK3aCcko+XmbRs7BHel9S/7AJ4IK3NX3JsX8Tm7xo07vbzbplZ9/DPTfyuNw 7GjXwxEcXDu7/4mg8t8Mxpo55u+yyp+XeEaT9Cv4r8WVW94tH6Lx5ROgTPhjvlDsiyaW2HxGtx1 xrcQwbHVTUln7+Q== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The workqueue watchdog detects pools that haven't made forward progress by checking whether pending work items on the worklist have been waiting too long. However, this approach has a blind spot: if a pool has only one work item and that item has already been dequeued and is executing on a worker, the worklist is empty and the watchdog skips the pool entirely. This means a single hogged worker with no other pending work is invisible to the stall detector. I was able to come up with the following example that shows this blind spot: static void stall_work_fn(struct work_struct *work) { for (;;) { mdelay(1000); cond_resched(); } } Additionally, when the watchdog does report stalled pools, the output doesn't show how long each in-flight work item has been running, making it harder to identify which specific worker is stuck. This series addresses both issues: Patch 1 fixes a minor semantic inconsistency where pool flags were checked against a workqueue-level constant (WQ_BH instead of POOL_BH). No behavioral change since both constants have the same value. Patch 2 renames pool->watchdog_ts to pool->last_progress_ts to better describe what the timestamp actually tracks. Patch 3 adds a current_start timestamp to struct worker, recording when a work item began executing. This is printed in show_pwq() as elapsed wall-clock time (e.g., "in-flight: 165:stall_work_fn [wq_stall] for 100s"), giving immediate visibility into how long each worker has been busy. Patch 4 introduces pool_has_stalled_worker(), which scans all workers in a pool's busy_hash for any whose current_start timestamp exceeds the watchdog threshold. This is called unconditionally for every pool, independent of worklist state, so a stuck worker is always detected. The feature is gated behind a new CONFIG_WQ_WATCHDOG_WORKERS option (disabled by default) under CONFIG_WQ_WATCHDOG. An option is to get rid of CONFIG_WQ_WATCHDOG_WORKERS completely. I've been running this change on some hosts with workloads (mainly stress-ng) and I haven't found any false positive. With this series applied, we will be able to see a stall like the one above: BUG: workqueue lockup - worker365:stall_work_fn [wq_stall] stuck in pool cpus=9 node=0 flags=0x0 nice=0 for 2570s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 38: cpus=9 node=0 flags=0x0 nice=0 active=2 refcnt=3 workqueue stall_wq: flags=0x0 --- Breno Leitao (4): workqueue: Use POOL_BH instead of WQ_BH when checking pool flags workqueue: Rename pool->watchdog_ts to pool->last_progress_ts workqueue: Show in-flight work item duration in stall diagnostics workqueue: Detect stalled in-flight work items with empty worklist kernel/workqueue.c | 71 ++++++++++++++++++++++++++++++++++++++------- kernel/workqueue_internal.h | 1 + lib/Kconfig.debug | 12 ++++++++ 3 files changed, 74 insertions(+), 10 deletions(-) --- base-commit: 9cb8b0f289560728dbb8b88158e7a957e2e90a14 change-id: 20260210-wqstall_start-at-e7319a005ab4 Best regards, -- Breno Leitao