From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21E142C0F69 for ; Wed, 18 Mar 2026 16:46:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773852387; cv=none; b=FWZNq4hYKwTKeI4XP2exAi5WYcA/l9tJNXjvZ6DlIySxEocPUzithCQKH5Z+03NslZmYYhQ+z+ixjh+v3IuNLxXwPv8z/zLpR0dbSHqscDFPDW49EfCjUgbv8nJLs59etTMeH8MwjNItCWxZzxxN8RIx/W+LW0/E9STNj+JHwOw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773852387; c=relaxed/simple; bh=SH35VSzi4yBuLkwZ0t7lC7EN8h93gZQBJkLj/p/RbjA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=hHvNAHHtmZ6xXhXkjH7jPU/ih1t1pb5ZTYz47HF/2+U5ofQjqRcnUEpLxRTrU/6bjdFB025L+hRrZ+tna47+XcHrtBXuCg/RX26gHqTYI8o5cBdS87Ty/QeBpeltdOYSaY8CaSThCbd9LnxyXD6GLz0bU4hM5GEqR4pj5Q36sb0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=NB53GDhM; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="NB53GDhM" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-485410a0a8aso418845e9.2 for ; Wed, 18 Mar 2026 09:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1773852384; x=1774457184; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=fFeKGHC9hypXwXSubDMW7qyXTfm4tL4LO77kLZMfvNY=; b=NB53GDhM5z4tfXCFOSMET0upOV5YJ87i1Z1ZWdlePLjbxRzEOQpPYmMuh2AidzfAfo X2Rn3/UvyNjloVtGErx0XCa0v2r35xfYcHoLtNOrjQvEMDwAhmK4S7+rWAeiVxlx1++V WEcEtvCL32YX7xnjGrt1ddz1bHbL/TfMok4ygPymQvH1Q6mbobUMLGH0QcX8M281nHN8 yukAMGgFr383otIgMURmeuvhfcXuQLYu0AHu1JAErMVVDb6XrYb80f0OotDvleaIkMkV 6TtbG5+Ky8nYHd7qdHdaeie44AD3tHLHp/Ab/U6ZHkCH9lbet3qi5T7cJjRmS1CBHN94 3xNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773852384; x=1774457184; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fFeKGHC9hypXwXSubDMW7qyXTfm4tL4LO77kLZMfvNY=; b=MS4N/tLB00KktkZNDzmU6dznrdrjWOemwFZWRPuA6T+aDf4aHO2DmoBEYfo3GyN15t kA1hKg3QJpYiGbS7HpPPEId4T8H9Qa6WjUKDpzf7+zlICUTt6mf7YX1K4IJgkckB10/N +be44zwMYK2ZYiL0lJvZW/AQRlasWD/sLs8MiiL1tJCL62PxC5O7o8JhMZ2V+Flxsdbb uX+bdHGA8rlBl0JIZo8Sk0ce0rBAknbfhOQY/p9UBHbadm02wXAfGePyta61aMLbFJAx yVj7K6ZKc+fKC8F2g12z1YEvdMpEZDRjrFlVq2r1cLgu/S7m28BkuQzOesuolasRObHO x70g== X-Forwarded-Encrypted: i=1; AJvYcCVokqQMPeKtjHmP9YtYUjC77g9TiKullKweV+JpHpXawKFNn+ufESlvnWHYut4CbQEeMwAqyv+YCdNVUvw=@vger.kernel.org X-Gm-Message-State: AOJu0YybwFm0/wDTNc3Y0k2UW1/WeM8AawHlYoeBAmsAszCv/mKR1Io2 ykuBq/cuf2wHKbQtvDlN4AfVj+XUtRycFtoYgcw48SxLB9aj5VYpZy0w4miAciLrXRY= X-Gm-Gg: ATEYQzw060xHLqQsA3Nf4Rrbd6IgprPp6jbnE6eXoNF1wqcbW9+5V2hHzMBvnDyyiyA HHm7d8bPdgUP8UngoBVeBtFHPWmJaT3+E5e/jSVGZcFsI4dC41ZZFmhmjwu7K+QauMdFAQayg0w kQLHlndZy1vnDSPri86sAWrQRQrN588gKVVN8xzlXYjl1eEF3VMDj7VUPNbOixtDQ+NdCEYh7So u9oiGaqAvq9nFNRL2dR1xC6Pv9J5DAXhW37Uutqg6x+ZOdYfYnv4j5izPrDILcAx3aJeuTSFeZ2 Yt3kQfc56B0m4hkzRqmd9PJRk6U4EDylJUyBVGoeP4LXoGufOgYAhZbu7NwwL9CD+4dEEH7pJcN LI2q4Ku8lOsbd+rn526iy8mZ9TueEIk6i42kZg6O6fYQ8ans16QYCuXSPi5kCZN5qmaXQrYE6U8 ZW9cCwSGsIPSV6aqjlf5pjFbR9Pw== X-Received: by 2002:a05:600c:58d9:b0:485:309c:2984 with SMTP id 5b1f17b1804b1-486f4429daemr50338825e9.13.1773852384144; Wed, 18 Mar 2026 09:46:24 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-486f4b87236sm39279075e9.7.2026.03.18.09.46.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Mar 2026 09:46:23 -0700 (PDT) Date: Wed, 18 Mar 2026 17:46:21 +0100 From: Petr Mladek To: Breno Leitao Cc: Tejun Heo , Lai Jiangshan , Andrew Morton , linux-kernel@vger.kernel.org, Omar Sandoval , Song Liu , Danielle Costantino , kasan-dev@googlegroups.com, kernel-team@meta.com Subject: Re: [PATCH v2 0/5] workqueue: Detect stalled in-flight workers Message-ID: References: <20260305-wqstall_start-at-v2-0-b60863ee0899@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri 2026-03-13 10:36:09, Breno Leitao wrote: > On Fri, Mar 13, 2026 at 03:38:57PM +0100, Petr Mladek wrote: > > On Fri 2026-03-13 05:24:54, Breno Leitao wrote: > > > I am currently rolling this patchset to production, and I can report once > > > I get more information. > > > > That would be great. I am really curious what is the root problem here. > > In fact, I got some instances of this issue with this new patchset, and, > still, the backtrace is empty. These are the only 3 issues I got with the new > patches applied. All of them wiht the "blk_mq_timeout_work" function. > > BUG: workqueue lockup - pool 11 cpu 2 curr 686384 (thrmon_agg) stack ffff8002bd200000 cpus=2 node=0 flags=0x0 nice=-20 stuck for 276s! > work func=blk_mq_timeout_work data=0xffff0000b88e3405 > Showing busy workqueues and worker pools: > workqueue kblockd: flags=0x18 > pwq 11: cpus=2 node=0 flags=0x0 nice=-20 active=1 refcnt=2 > pending: blk_mq_timeout_work This is report is showing the stalled "pool 11" in the list of busy worker pools. > Showing backtraces of busy workers in stalled CPU-bound worker pools: > > BUG: workqueue lockup - pool 7 cpu 1 curr 0 (swapper/1) stack ffff800084f80000 cpus=1 node=0 flags=0x0 nice=-20 stuck for 114s! > work func=blk_mq_timeout_work data=0xffff0000b88e3205 > Showing busy workqueues and worker pools: > workqueue events: flags=0x0 > pwq 510: cpus=127 node=1 flags=0x0 nice=0 active=1 refcnt=2 > pending: psi_avgs_work It is strange that "pwq 7" is not listed here. > Showing backtraces of busy workers in stalled CPU-bound worker pools: > > BUG: workqueue lockup - pool 11 cpu 2 curr 24596 (mcrcfg-fci) stack ffff8002b5a40000 cpus=2 node=0 flags=0x0 nice=-20 stuck for 282s! > work func=blk_mq_timeout_work data=0xffff0000b8706805 > Showing busy workqueues and worker pools: And the list of busy worker pools is even empty here. > Showing backtraces of busy workers in stalled CPU-bound worker pools: I would expect that the stalled pool was shown by show_one_workqueue(). show_one_workqueue() checks pwq->nr_active instead of list_empty(&pool->worklist). But my understanding is that work items added to pool->worklist should be counted by the related pwq->nr_active. In fact, pwq->nr_active seems to be decremented only when the work is proceed or removed from the queue. So that it should be counted as nr_active even when it is already in progress. As a result, show_one_workqueue() should print even pools which have the last assigned work in-flight. Maybe, I miss something. For example, the barriers are not counted as nr_active, ... Anyway, the backtrace of the last woken worker might give us some pointers. It might show that the pool is stuck on some wq_barrier or so. > > In all these cases, there is listed some pending work on the stuck > > "cpus=XXX". So, it looks more sane than the 1st report. > > > > I agree that it looks ugly that it did not print any backtraces. > > But I am not sure if the backtraces would help. > > > > If there is no running worker then wq_worker_sleeping() should wake up > > another idle worker. And if this is the last idle worker in the > > per-CPU pool than it should create another worker. > > > > Honestly, I think that there is only small chance that the backtraces > > of the sleeping workers will help to solve the problem. > > > > IMHO, the problem is that wq_worker_sleeping() was not able to > > guarantee forward progress. Note that there should always be > > at least one idle work on CPU-bound worker pools. > > > > Now, the might be more reasons why it failed: > > > > 1. It did not wake up any idle worker because it though > > it has already been done, for example because a messed > > worker->sleeping flag, worker->flags & WORKER_NOT_RUNNING flag, > > pool->nr_running count. > > > > IMHO, the chance of this bug is small. > > > > > > 2. The scheduler does not schedule the woken idle worker because: > > > > + too big load > > + soft/hardlockup on the given CPU > > + the scheduler does not schedule anything, e.g. because of > > stop_machine() > > > > It seems that this not the case on the 1st example where > > the CPU is idle. But I am not sure how exactly are the IPIs > > handled on arm64. > > I don't have information about the load of those machines when the problem > happens, but, in some case the problem happen when there is no workload > (production job) running on those machine, thus, it is hard to assume that the > load is high. > > > 3. There always must be at least one idle worker in each pool. > > But the last idle worker newer processes pending work. > > It has to create another worker instead. > > > > create_worker() might fail from more reasons: > > > > + worker pool limit (is there any?) > > + PID limit > > + memory limit > > > > I have personally seen these problems caused by PID limit. > > Note that containers might have relatively small limits by > > default !!! > > Might this justify the fact that WORK_STRUCT_PENDING bit is set for ~200 > seconds? > > > > I think that it might be interesting to print backtrace and > > state of the worker which is supposed to guarantee progress. > > Is it "pool->manager" ? > > > > Also create_worker() prints an error when it can't create worker. > > But the error is printed only once. And it might get lost on > > huge systems with extensive load and logging. > > That is definitely not the case. I've scan Meta's whole fleet for create_worker > error, and there is a single instance on a unrelated host. Good to know. I am more and more curious what would be the culprit here. Best Regards, Petr