From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 256303CE489 for ; Thu, 5 Mar 2026 16:15:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772727361; cv=none; b=oYMHvQPmSMm1XIaftpaTlBF+2FIyyDLyLDh7KDrUl6f+lflPTZCZ8vF7/akOkYY/JtwbMx1iqPOA5N2t+V3it7p909onnJshmJijyOK2abXa4y6FkPcCt/tTIQ70S1mmrP0d7y3VnOrSP58FesfiAbsXL/76ul5MHoGrMJwDE+U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772727361; c=relaxed/simple; bh=BS/RITeYj/f1b7ekg/tvxn8XU+UtVb+pBdBhbPFHQFA=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=Inb2uJXLUsqVQB6bhR+Sig5WNojSXmcz+bR5NhjX4uMLyf7NW9Wwo0pOEfnDMe0g8EsJoggzBOU72BtHak7o9yk78Ip6dm6uA+R6CYJAlGJXrZ/ixJ8B1At+XPa+9P07x/MUUjaw0Akoaph9NYKRCNf0Q64mvC08vW8umZXbKOw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=SMetq6Iq; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="SMetq6Iq" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-Id:Date:Subject:From:Reply-To:Content-ID: Content-Description:In-Reply-To:References; bh=gAbgyAsT9TtC81lH4Y/jQr/BkNpJoyjPGoY83nTcQMw=; b=SMetq6Iq9DsdJRkaQTsN+XtshO 4Gis/l075tTq4fkUMqiQN1L2uRYTPjWEmdnfpNev9o5gM59+bFCkh5fmtZUCARZx3z9IElUBemuZJ HhY34eI9Xwr2R6/hpWcmJ7Y2ej6T7iAdeWbv0Rg2rLZAEu5BKGp7FdgORxn3hgKyIstOoq/wPk3VQ /NdNVaBD07PuMoS0zx8HJ0n56+Dciv8vNexdn3XsOSr87u8TWqYl39aYvxtyujfD0I7+zY+4yylE4 OcTyS4sbL3+DM0ZoAPjFIsxZizF2GhDNqwkTQew01Ka+YY9K2yQ1uhl/FmliJ2awkS+0srys85xPo QIO4w+Lw==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1vyBMe-00Gqq5-N2; Thu, 05 Mar 2026 16:15:53 +0000 From: Breno Leitao Subject: [PATCH v2 0/5] workqueue: Detect stalled in-flight workers Date: Thu, 05 Mar 2026 08:15:36 -0800 Message-Id: <20260305-wqstall_start-at-v2-0-b60863ee0899@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIACmsqWkC/2XNyw6CMBCF4VdpZk1NW/FSVr6HIWYKI4wxoJ2KG sK7GzCuXJ7kz3dGEIpMAoUaIdLAwn0HhXKZgqrFriHNNRQKnHFb46zRz7skvF5PkjAmjUnTbm0 9GrPBkEOm4BbpzK+FPJbfLY9woSrNzly0LKmP7+VzsHP34+0/P1htdKh97j3afWX9oabA2K362 EA5TdMHq6YERMMAAAA= X-Change-ID: 20260210-wqstall_start-at-e7319a005ab4 To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, Omar Sandoval , Song Liu , Danielle Costantino , kasan-dev@googlegroups.com, Petr Mladek , kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.15-dev-363b9 X-Developer-Signature: v=1; a=openpgp-sha256; l=5488; i=leitao@debian.org; h=from:subject:message-id; bh=BS/RITeYj/f1b7ekg/tvxn8XU+UtVb+pBdBhbPFHQFA=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpqawzbK8bSaaxjKTV3jtHp1yYKwh6M88DGc2o4 PTam1dhgkeJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaamsMwAKCRA1o5Of/Hh3 bSTmD/47Zx4/113ovAT0p3wzY7nTeforO2Eoh0UFWCt9zCEUrbIWm84dgt7iqB7qJrQ8sHqM856 rhJYoDPertlwDym0+Z5HU4ywnL46cXRXWijzroC5LOETgxE12BtXsSirJp3Xjflb+iJ15ayn5Bg /29Dger8ImboNFYg07Yvo+IhrEDrHZ57tFNgT4fLi9FavDrdb/VQsvIaHPq6QS4C3Y/MHL9cNu0 aqpaQyRhIJYOhAQCsuawcm8IWG/Q11eyQGPpasgo6e4+A7A7Dpik0hQ7lb3Ziemxbe65S3yThws u99XSZ7IepQ+rz5Agi4A5gJuBkrQjn/g9pBI+Mi1mUj2Q54S2GcVDgv6GUFNDN/6urMF1n88liv TcE+0oBgChyewlCSCELdp6f4k352f/W2+NMYbxsqqTmWm147wKI/ESYttgu2fqPX51+TWhtxtZV 2QBGNMqaRFGV7qQjllmBUSsTHEn/a9bXC11SItwnqnS7XyDjfLCsGBpSor1ERBiwVyXd9QRtvEo EJFQPuwFBEqmTs/aQz+F0+Tm4M5qB2WYMy3o/3czyNigkaM0RFfAoTz7yJGOZqc9/E1+BfESXiP 9IeyStKW2gxIJLJTISuo1n+S+t3P2zTZHZ1k5nLhRxbQcNh+lQgx19SA5X3yG0PZwEHo0MHr82t IZEpb1K1550tObQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao There is a blind spot exists in the work queue stall detecetor (aka show_cpu_pool_hog()). It only prints workers whose task_is_running() is true, so a busy worker that is sleeping (e.g. wait_event_idle()) produces an empty backtrace section even though it is the cause of the stall. Additionally, when the watchdog does report stalled pools, the output doesn't show how long each in-flight work item has been running, making it harder to identify which specific worker is stuck. Example of the sample code: BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck for 132s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 18: cpus=4 node=0 flags=0x0 nice=0 active=4 refcnt=5 in-flight: 178:stall_work1_fn [wq_stall] pending: stall_work2_fn [wq_stall], free_obj_work, psi_avgs_work ... Showing backtraces of running workers in stalled CPU-bound worker pools: I see it happening on real machines, causing some stalls that doesn't have any backtrace. This is one of the code path: 1) kfence executes toggle_allocation_gate() as a delayed workqueue item (kfence_timer) on the system WQ. 2) toggle_allocation_gate() enables a static key, which IPIs every CPU to patch code: static_branch_enable(&kfence_allocation_key); 3) toggle_allocation_gate() then sleeps in TASK_IDLE waiting for a kfence allocation to occur: wait_event_idle(allocation_wait, atomic_read(&kfence_allocation_gate) > 0 || ...); This can last indefinitely if no allocation goes through the kfence path (or IPIing all the CPUs take longer, which is common on platforms that do not have NMI). The worker remains in the pool's busy_hash (in-flight) but is no longer task_is_running(). 4) The workqueue watchdog detects the stall and calls show_cpu_pool_hog(), which only prints backtraces for workers that are actively running on CPU: static void show_cpu_pool_hog(struct worker_pool *pool) { ... if (task_is_running(worker->task)) sched_show_task(worker->task); } 5) Nothing is printed because the offending worker is in TASK_IDLE state. The output shows "Showing backtraces of running workers in stalled CPU-bound worker pools:" followed by nothing, effectively hiding the actual culprit. Given I am using this detector a lot, I am also proposing additional improvements here as well. This series addresses these issues: Patch 1 fixes a minor semantic inconsistency where pool flags were checked against a workqueue-level constant (WQ_BH instead of POOL_BH). No behavioral change since both constants have the same value. Patch 2 renames pool->watchdog_ts to pool->last_progress_ts to better describe what the timestamp actually tracks. Patch 3 adds a current_start timestamp to struct worker, recording when a work item began executing. This is printed in show_pwq() as elapsed wall-clock time (e.g., "in-flight: 165:stall_work_fn [wq_stall] for 100s"), giving immediate visibility into how long each worker has been busy. Patch 4 removes the task_is_running() filter from show_cpu_pool_hog() so that every in-flight worker in the pool's busy_hash is dumped. This catches workers that are busy but sleeping or blocked, which were previously invisible in the watchdog output. With this series applied, stall output shows the backtrace for all tasks, and for how long the work is stall. Example: BUG: workqueue lockup - pool cpus=14 node=0 flags=0x0 nice=0 stuck for 42! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_shepherd pwq 58: cpus=14 node=0 flags=0x0 nice=0 active=4 refcnt=5 in-flight: 184:stall_work1_fn [wq_stall] for 39s ... Showing backtraces of busy workers in stalled CPU-bound worker pools: pool 58: task:kworker/14:1 state:I stack:0 pid:184 tgid:184 ppid:2 task_flags:0x4208040 flags:0x00080000 Call Trace: __schedule+0x1521/0x5360 schedule+0x165/0x350 stall_work1_fn+0x17f/0x250 [wq_stall] ... --- Changes in v2: - Drop the task_running() filter in show_cpu_pool_hog() instead of assuming a work item cannot stay running forever. - Add a sample code to exercise the stall detector - Link to v1: https://patch.msgid.link/20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org --- Breno Leitao (5): workqueue: Use POOL_BH instead of WQ_BH when checking pool flags workqueue: Rename pool->watchdog_ts to pool->last_progress_ts workqueue: Show in-flight work item duration in stall diagnostics workqueue: Show all busy workers in stall diagnostics workqueue: Add stall detector sample module kernel/workqueue.c | 47 +++++++------- kernel/workqueue_internal.h | 1 + samples/workqueue/stall_detector/Makefile | 1 + samples/workqueue/stall_detector/wq_stall.c | 98 +++++++++++++++++++++++++++++ 4 files changed, 124 insertions(+), 23 deletions(-) --- base-commit: c107785c7e8dbabd1c18301a1c362544b5786282 change-id: 20260210-wqstall_start-at-e7319a005ab4 Best regards, -- Breno Leitao