From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17E7428BAB9 for ; Fri, 19 Jun 2026 13:43:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781876582; cv=none; b=t3bjxdAJvc4DS/KH1SZfKA33XHMeOi5KxWQZxYnwAXjZ355vEk4VVTEOHDdpKhMnnLYcTnUJOyImNyG1bSPndyQl3309IpVkfGqVOjt6rU9wXHJmA+F6uCBaCfZ68YYqmomHmE7c/CdXSmEkV/LagzgqLzzwbw9rK0TF3ny0h5M= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781876582; c=relaxed/simple; bh=iWARr9QW2nzjk79p+qRSK6p9AQQBCySNTbQH/kIur/w=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=t6uzkCGGDQqN69P5PBUnfyV0IepoonlEJzYDSTXsJ4B8XDZNRAGx45pMFbU+OAw8p/0s+0efRC8m6GIwmSh3N7Sg5zw87rbJnlM0+8QAorq7wQS4sG/OS5iBGYb4IGiFKxjx02x3B15HtVtjGygvNus8eokLmJgW1K9bp3Li8BA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=bPg6AOYo; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="bPg6AOYo" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-460662fcb4eso1548489f8f.0 for ; Fri, 19 Jun 2026 06:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1781876579; x=1782481379; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=WYhhlqXYM+lMFu5HbT9kAsMC+b0FCRERfTRr8DQ4oTc=; b=bPg6AOYooR7bY3K034BvEf7X9gKx49AZmAo/AWqelozoMgeG7BXESI2uoT+xCerqaV kgNDcXmVPDlTRyLMmMpH/rv+QigXFdNoHb0xo7DSsQIm0EAt6MaNQ/4i1kaR4689k1WJ oRL6pHuNc0IYvyF4QuS4RjJ5nUCeet+qCuYdJp7E3X/4XHmipgzP8uuuihzslJ/c8efc hrW7QEKsnjZaguQ46YCb+ZCJDuTcKS5D9guODCA03kFyzQfbCqK6izu9eZJSMnAThShx 33VZbi5YZR0YANt4+CRG5ul0u9Otk/jbq/xEvON8zPCV+hM1eDXmWjZt/kSIpt7e5OUQ sg2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781876579; x=1782481379; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WYhhlqXYM+lMFu5HbT9kAsMC+b0FCRERfTRr8DQ4oTc=; b=T6WjHENYK33OSqftAiG5M4NuXU4d+VmEzSBS34AsjvHkz2Ax2HgSQUId2DYobrmLKG Px/YNwYKnm68bWvO00i0aSfzeEVURXeu0WGMjflQLU0VVyx8f08ACe+TF7UHFVsSvQW3 6XrCo1BjNxhezsTuDVCgm1IY2LF8o8/PM8e5M3O1la7ZIV749LjZooXE0RJw7tAjAnnP JTm45i5BNBmnZzTfSHzWXgzzDHUeAApxOIwZdQCLNZwJGZjzPz5HdrzBqc1L0X5E7srQ f83wkg871S9VxYFUiXvTHnXPv65KcKvH5dRYz4O5FiarK2x7te+XHhGqNxe+3s3GBcYV JSXg== X-Forwarded-Encrypted: i=1; AFNElJ++H5Mk1Xg9t3DvhwF2qO/7L8He+/mhPaBUs3PfuvN3ygJVxP3fUFXpLQRxLX5fPJXCkMLj2cvPylrRuu0=@vger.kernel.org X-Gm-Message-State: AOJu0YzE2gH8gtOkCAp/hiJcbcKX8Lp4xqcvLVbKYrX7RqOwkJZk8s7g eb6m7O/OagFSR1z3FZ0ZVHGU3CgOgZZ3z5ItfTpZjwseLMmTC6tbxQVI19D/nk/cp3Q= X-Gm-Gg: AfdE7ck/FkNsI2UAHo3q6WQDoF57gAwgrigMhQk5xqoK5WhlyqaPlFvUSsPEDtqcoo8 vusP6DxX9vSrbXyBkptPfeWB9XyHI9CxjVDF9eXQzVvQXL3drMGzK3xajfhmGwWYTt/lqpiYjS9 Ph69a11aNQfbAwQflaCfXCogC6s0lHbDti5zAzXezMouoDmT2JFwmnltcqkYM3rZkyoGDDrduIj XZeTijhK6RzoOqX1hLyvuF2Nl1bWJEREqItT/zAO4UOESal7MybYUvoDDVmhHw/DVAa8nlV+asH HfSEcVQq8TaLwz/zO8o/QFU1IY+j072VxR7QKL5vAHvtfqfX7TUbipwbIijVibeVycobHnx+AxE x0m7ITe72qStQIQ9icPCocnsXGQeXwhbtbI8JUMLT7SXWW4DdZ6mLdHUO68m+neAeRpm+3ZCPzL lQT1UJVga2V4vbAyA= X-Received: by 2002:a05:6000:10c:b0:464:28e3:a09b with SMTP id ffacd0b85a97d-4650a2351b3mr5434724f8f.37.1781876579413; Fri, 19 Jun 2026 06:42:59 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4650b67a248sm8770494f8f.24.2026.06.19.06.42.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Jun 2026 06:42:59 -0700 (PDT) Date: Fri, 19 Jun 2026 15:42:56 +0200 From: Petr Mladek To: Breno Leitao Cc: Tejun Heo , Lai Jiangshan , Song Liu , linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: Re: [PATCH RFC 2/3] workqueue: trigger a single-CPU backtrace for stalled pools Message-ID: References: <20260616-wq_dump_petr-v1-0-b57473ca6d18@debian.org> <20260616-wq_dump_petr-v1-2-b57473ca6d18@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260616-wq_dump_petr-v1-2-b57473ca6d18@debian.org> On Tue 2026-06-16 09:44:40, Breno Leitao wrote: > When a CPU pool is stalled with no running worker, the task occupying the > CPU may not be a workqueue worker at all. Trigger a single-CPU backtrace > for the stalled CPU to capture what it is currently executing. > > The CPU is snapshotted under pool->lock and the backtrace is triggered > after releasing the lock to avoid any potential issues with NMI delivery. > > --- a/kernel/workqueue.c > +++ b/kernel/workqueue.c > @@ -7751,6 +7754,14 @@ static void show_cpu_pool_busy_workers(struct worker_pool *pool) > show_pool_no_running_worker(pool); > > raw_spin_unlock_irqrestore(&pool->lock, irq_flags); > + > + /* > + * Trigger a backtrace on the stalled CPU to capture what it is > + * currently executing. Called after releasing the lock to avoid > + * any potential issues with NMI delivery. > + */ > + if (!found_running) > + trigger_single_cpu_backtrace(cpu); > } Sashiko AI is curious whether this might be racy against CPU hotplug, see https://sashiko.dev/#/patchset/20260616-wq_dump_petr-v1-0-b57473ca6d18%40debian.org Is it possible for this CPU to be offline when we trigger the backtrace? When a CPU goes offline, its bound workqueue pools are disassociated via unbind_workers() but retain their original pool->cpu ID. If a work item on a disassociated pool hangs, the watchdog could detect the stall and invoke show_cpu_pool_busy_workers(). If the target CPU is offline, it cannot process the NMI or clear its completion bit in the backtrace mask. Does this cause nmi_trigger_cpumask_backtrace() to busy-wait for the 10-second timeout waiting for a response that will never arrive? Since the watchdog executes in a softirq timer context, this could stall the CPU running the watchdog for 10 seconds. Should this check cpu_online(cpu) before triggering the backtrace? It makes some sense. wq_watchdog_timer_fn() checks either 'per_cpu(wq_watchdog_touched_cpu)' or the global 'wq_watchdog_touched' depending whether pool->cpu is set or not. And it seems to be wrong for disassociated pools. But this seems to be an existing problem which should be fixed separately. Also wq_watchdog_timer_fn() is checking the state without taking proper locks. So it is inherently racy. But it is another story. Summary: This particular change looks good to me. And I believe that it is useful. Reviewed-by: Petr Mladek Best Regards, Petr