From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F4053E5598 for ; Wed, 18 Mar 2026 15:11:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773846723; cv=none; b=NZGoW+c5i9hL95z8kb3bPsw7ChHKjJtyRQnyderDpsnB/dpyn17t3/TGer69HTEOtO5rwt4gOVYmPBUMvQmTJCw55cY6CkpMXMOscRHw/S/i8aZpABGVZKkNFLSf2ym7bj8AWsJL7HmQrav3x/Kk1vKDhHm4iiY3paZCFSNFszw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773846723; c=relaxed/simple; bh=YtaFEmnuUp3sZPcHuOE03kZyhzmNojiEVKe1kBNWMBE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=BhHhLA2qjRJKKynqnFhTqpR4O0wnbeLYGLAkFczVz4Qh/Jt8J326ffG4FsCIZk7OB6+Sh7hhwYSL129S/dzhCImtDevqAIFZLj+rB2RukclDoE5hQQ/79d0DuoeUNqZDBqkqAZa8UojAr5kgQhrZWUITAJyTaDDn6ybIrlgK7LU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=QXj07JT4; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="QXj07JT4" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-48334ee0aeaso63079955e9.1 for ; Wed, 18 Mar 2026 08:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1773846717; x=1774451517; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=l0b+XrdafVkYRVSgoittHbFPY7qGon9A5OyHLA3D18s=; b=QXj07JT4+y4QYZQH0W/TUqwbfMYI+KWscSPtBOR7t0MdH0XT+z8BKgisLJykUlLEA2 IJcohSxKhfvd5KfC8b1tbdxTujCV3YTqgFr/HHn4x279PV9qrCvM2UISaNUZwWIPTQb+ bSYNFCn+0Cr2Cb6+kt39PhNWOP1chZVql0wNPwcbfav8Apun0OgFjGAScOQ+6H2G7mGv 6+/Mhg8x20+wDv/3NUTZwbF1UN3OzJwHosht4pBmsmJc8hJKdMABWPSHVX3vHdF5dkMT eGRxKRoq8CLRFrVeJ/3ZveJZcamcFnegVlFPQciQrJuIgqcHpjGh1orf2f4Ppg7r6/0H EmLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773846717; x=1774451517; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l0b+XrdafVkYRVSgoittHbFPY7qGon9A5OyHLA3D18s=; b=JrSYWOnSHOVRWgiXCUGjifu7CUgeBAwTyLD4AdrhRXkY5OUQz1z+aMvYSu88R5ptcE 3+XWRiZDwCqEB4ZU0GYqk0uzy3FTOhiGvXWABEsl+f4TS79l8klVNwpalvf5n1HhNnyT TVzBlPE1OqiG7qJDW4cOi7YrIHGLgMKNlhFgcoixoS52ylONfgNkY5njlR4zpT7l5HPU gZXAJM6foI09Wy6H3kS2l2VxNRRlhHDVVPne5kSPXcpI3yfm3dGLkr8Tly79OGNxsfGf 5iClDBi0CbsBcVMI5KvMr+VsGAWlz4BKncpJNvuduXc3+vzyEOwDTH7QXLVaCO2FI6YU ERTw== X-Forwarded-Encrypted: i=1; AJvYcCUXDShQ4CFTy5I3hh+FS27W7E1R/eMlXmTw+ibZh1ecoJh4W+Hmkw8oaiKrkw65ZgfITCly2NEbROOrGQ4=@vger.kernel.org X-Gm-Message-State: AOJu0Yy66liX4EO2cBKsqyEc/DrYkPDlfz+LHtj443+B/CxGuEQSCEyR TjZMKeIYA0hCKCE/mZW4mJOx/V0h1FnN/awvutFRxrPXsevq+1AxfeuEKJ+pxQeb4IDy+7/JomF zdppd X-Gm-Gg: ATEYQzyBVp7T7K2350EYIwjvoAANQ6oSD83BUCed5+Sm/wj05yhddN1NyN0LnXifFak Pn65bNrCSoESjOov5OOvIxF+KPD7vthp3a1D6Z8Jp4ycUClGc55IK7EA66M1mjLnhreAAouMt23 jSM5Yt3ivHjoMklHMCSUWH2ytRVELDwMuNxzvuydpSB9dsh2CmtQUqi1GYDvFUDSpd0X/5aVMWg JzwHY4dUyh7PIGvE9M8xiUaP+h0L9RIkY38hJV0ggB/8dB3CMeTzcPin1WzUjhtvxm4R6pQJQJD bzm5Mrs9X67P0dpQSAZlNPajLOUJ0u3eRZdrkVjpJTAeQXbI5szVs+mLwchi52ht/iGDht2Cj/G KPBP4V7YIALdOclmyPEoSENBc0tli90zdrj2YAIgF62+LK+QGTTMiQSpHyiHnDAzqFul4AG86UJ l368TRMwfIsQ6JfDDvyBmjS2X8CA== X-Received: by 2002:a05:600c:310d:b0:485:364e:934e with SMTP id 5b1f17b1804b1-486f457284bmr59962725e9.21.1773846717201; Wed, 18 Mar 2026 08:11:57 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4856ea8fad1sm267405085e9.1.2026.03.18.08.11.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Mar 2026 08:11:56 -0700 (PDT) Date: Wed, 18 Mar 2026 16:11:54 +0100 From: Petr Mladek To: Breno Leitao Cc: Tejun Heo , Lai Jiangshan , Andrew Morton , linux-kernel@vger.kernel.org, Omar Sandoval , Song Liu , Danielle Costantino , kasan-dev@googlegroups.com, kernel-team@meta.com Subject: Re: [PATCH v2 4/5] workqueue: Show all busy workers in stall diagnostics Message-ID: References: <20260305-wqstall_start-at-v2-0-b60863ee0899@debian.org> <20260305-wqstall_start-at-v2-4-b60863ee0899@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Wed 2026-03-18 04:31:08, Breno Leitao wrote: > On Fri, Mar 13, 2026 at 05:27:40PM +0100, Petr Mladek wrote: > I agree. We should probably store the last woken worker in the worker_pool > structure and print it later. > > I've spent some time verifying that the locking and lifecycle management are > correct. While I'm not completely certain, I believe it's getting closer. An > extra pair of eyes would be helpful. > > This is the new version of this patch: > > commit feccca7e696ead3272669ee4d4dc02b6946d0faf > Author: Breno Leitao > Date: Mon Mar 16 09:47:09 2026 -0700 > > workqueue: print diagnostic info when no worker is in running state > > show_cpu_pool_busy_workers() iterates over busy workers but gives no > feedback when none are found in running state, which is a key indicator > that a pool may be stuck — unable to wake an idle worker to process > pending work. > > Add a diagnostic message when no running workers are found, reporting > pool id, CPU, idle state, and worker counts. Also trigger a single-CPU > backtrace for the stalled CPU. > > To identify the task most likely responsible for the stall, add > last_woken_worker (L: pool->lock) to worker_pool and record it in > kick_pool() just before wake_up_process(). This captures the idle > worker that was kicked to take over when the last running worker went to > sleep; if the pool is now stuck with no running worker, that task is the > prime suspect and its backtrace is dumped. > > Using struct worker * rather than struct task_struct * avoids any > lifetime concern: workers are only destroyed via set_worker_dying() > which requires pool->lock, and set_worker_dying() clears > last_woken_worker when the dying worker matches. show_cpu_pool_busy_workers() > holds pool->lock while calling sched_show_task(), so last_woken_worker > is either NULL or points to a live worker with a valid task. More > precisely, set_worker_dying() clears last_woken_worker before setting > WORKER_DIE, so a non-NULL last_woken_worker means the kthread has not > yet exited and worker->task is still alive. > > The pool info message is printed inside pool->lock using > printk_deferred_enter/exit, the same pattern used by the existing > busy-worker loop, to avoid deadlocks with console drivers that queue > work while holding locks also taken in their write paths. > trigger_single_cpu_backtrace() is called after releasing the lock. > > Suggested-by: Petr Mladek > Signed-off-by: Breno Leitao > > diff --git a/kernel/workqueue.c b/kernel/workqueue.c > index b77119d71641a..38aebf4514c03 100644 > --- a/kernel/workqueue.c > +++ b/kernel/workqueue.c > @@ -7582,20 +7593,58 @@ module_param_named(panic_on_stall_time, wq_panic_on_stall_time, uint, 0644); > MODULE_PARM_DESC(panic_on_stall_time, "Panic if stall exceeds this many seconds (0=disabled)"); > > /* > - * Show workers that might prevent the processing of pending work items. > - * A busy worker that is not running on the CPU (e.g. sleeping in > - * wait_event_idle() with PF_WQ_WORKER cleared) can stall the pool just as > - * effectively as a CPU-bound one, so dump every in-flight worker. > + * Report that a pool has no worker in running state, which is a sign that the > + * pool may be stuck. Print pool info. Must be called with pool->lock held and > + * inside a printk_deferred_enter/exit region. > + */ > +static void show_pool_no_running_worker(struct worker_pool *pool) > +{ > + lockdep_assert_held(&pool->lock); > + > + printk_deferred_enter(); > + pr_info("pool %d: no worker in running state, cpu=%d is %s (nr_workers=%d nr_idle=%d)\n", > + pool->id, pool->cpu, > + idle_cpu(pool->cpu) ? "idle" : "busy", > + pool->nr_workers, pool->nr_idle); > + pr_info("The pool might have trouble waking an idle worker.\n"); > + /* > + * last_woken_worker and its task are valid here: set_worker_dying() > + * clears it under pool->lock before setting WORKER_DIE, so if > + * last_woken_worker is non-NULL the kthread has not yet exited and > + * worker->task is still alive. > + */ > + if (pool->last_woken_worker) { > + pr_info("Backtrace of last woken worker:\n"); > + sched_show_task(pool->last_woken_worker->task); > + } else { > + pr_info("Last woken worker empty\n"); This is a bit ambiguous. It sounds like that the worker is idle. I would write something like: pr_info("There is no info about the last woken worker\n"); pr_info("Missing info about the last woken worker.\n"); > + } > + printk_deferred_exit(); > +} > + Otherwise, I like this patch. I still think what might be the reason that there is no worker in the running state. Let's see if this patch brings some useful info. One more idea. It might be useful to store a timestamp when the last worker was woken. And then print either the timestamp or delta. It would help to make sure that kick_pool() was really called during the reported stall. Best Regards, Petr