From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BA5E3B2FD3 for ; Thu, 12 Mar 2026 17:03:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773334993; cv=none; b=iuf18p326fEtHVt//YmRyjxNCWEPVATXyJRCaENjefo9+hLQSlMPHvp7bhbyWy6SORZQQgYn1nluEvRcYIju7iT9Tf5hr9yHEf3CeqMRvAZPqYZDgUEp7vxzcCNTGVn9hVIUEjpff+cAOe/kdbE/L9kprL7JdpIDuso8ZS96EJQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773334993; c=relaxed/simple; bh=Lu7BdKRTDZ64k7ApwIwfpZM4RCGFYmG4GxmZUZ/2eHI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=dYAgfhI79JwwhIz4hI4wILyqU5YOrsuSMbAm7drtnRCqUlStePirVgJ7QyarjomDtU6XMiyBSoMFydzgjVCfGsBBKwL9z5MUipUvEUI5RwY8qS7UzRz1ctba+Ak3l6Ap+eJTGcp0A/mh5SmObp/lDIZV32eQVc5X6KqLj9RwBKo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=PP9VcMB9; arc=none smtp.client-ip=209.85.218.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="PP9VcMB9" Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-b943b75dc42so199179466b.3 for ; Thu, 12 Mar 2026 10:03:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1773334988; x=1773939788; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=kNXlsPaTUI+d8vTQr/R9F9lyHbfTVaXsG68dynKOH10=; b=PP9VcMB9nx/QWfok4Z4gHiL6pD5nbLyQTYFJes9AvZSDi2qgM9dC9GRtDgIWgYMYCI poul0nKzFq0Clco46E7zyTPeJH36KIkP08MVWUZeIKGkUPhF4CWjsPBl5y3bAHmesf5D CW5oe0f5+TnmQ/O9XbBNBYrIpv7n2MecvBQMt7zT3F3o5JxokOaQ0c9mxUPVAP5n7Ns5 ZIrkcqPRbzMywnWLXn/usbdlmE/f7DhCupjfJHkv1IaOAzGUAut/lQ6+dP/G0NCJmy95 jwKHZ5lqrmzxVBOX6uIGLdYsVRbB98swo6XupXUb0yD4e7epTOKnPCSxvRiAjAgd23bj IVgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773334988; x=1773939788; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kNXlsPaTUI+d8vTQr/R9F9lyHbfTVaXsG68dynKOH10=; b=cmrx9CsLo2t1gVH4ItgYRVBll822gncqQO1bLElrL9OMXbiJx4UYgBsHhPFnCUQGKd +bX8HhNsrQ1mO2KNC2Rf2rV+nlFm4qxp5nBIokN0IxzheYewSy0EJ2+kGj8FAHpFGLeW Dc1mUCMvyBx2OHqO3NYg1rYDz/w48f+Kw+Mt56q2SqyUWYTfYfbsDz11JRf4YwuOFOWt 3EGSOZALCjW8wZyNTjHdeMVfUruwQDmiGGLkfMirj2RD2oWV+dXmegqn+s7x1cnciKJY KgTeTTEY37dZPOnQUp5V2YUHK//69SJMzKWeL29PykqzXRqaEcdMnSyVQ5FidFNIW3Eo XkbA== X-Forwarded-Encrypted: i=1; AJvYcCVRdgkpshuQNj7KSqe9i9i1Gqjdn5QPDi9Vg9GGm6BA+jlppLjbGEvMJ+6SklvhCEVpG6y6eVzKzMyQiec=@vger.kernel.org X-Gm-Message-State: AOJu0Yyz6YTc+zFKbvdaR5cjLtnf6ED0JWE947tu8zBJu5RAvwKI42wR DOiCzoyUIYpv/qLcDYP93wceSGNlOdH62E8srU7pGjx8QC3Z5wO1ggjGrStR9FL34xQ= X-Gm-Gg: ATEYQzwLl4iikda2g79EfMsvlOil/5EaFRoBvkS+/PScbG8qKVQtTKuGRB9FD0vN5xb 2+V5R4k5fADJFHGDkPPolz2yKsHYQ9H0lK6VeRtpOjL0NwDVa3pgakYlG24ULPbThLwxTkO6kUQ gU0f8CEMDIAphQDF1D7bz3j5/lOhetsTsTRyadXd1O/Rzomd/6JsaGo7VrfjUkjx0zxrV5F+AmP 60x1tbrKmtEguF/AUs8wlEhyltKgD2S+XJusqin00/GgY3i5CA6CTqxIjATfhAenSgtp841/er1 Gsou5zvL5aHxoghNlrOPzPYvLqEP6BxgEK2cWjR0AWTOXdrOx7EJb2awMJOlyUcXXVNurB6Q3zg gSN2Y0c7qTtks0g5coOiKi0IEWoSBsnTB2oweM+gGl7i+q5mWz8UFbtH0uj17UeSRzg4Rxdb/ZG jZVVXYaxVgt1vBawPtuGAMUSWcGQ== X-Received: by 2002:a17:906:1d4e:b0:b8f:a724:8710 with SMTP id a640c23a62f3a-b97653a8be3mr4915466b.50.1773334987845; Thu, 12 Mar 2026 10:03:07 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439fe1b287dsm9417910f8f.17.2026.03.12.10.03.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 10:03:06 -0700 (PDT) Date: Thu, 12 Mar 2026 18:03:03 +0100 From: Petr Mladek To: Breno Leitao Cc: Tejun Heo , Lai Jiangshan , Andrew Morton , linux-kernel@vger.kernel.org, Omar Sandoval , Song Liu , Danielle Costantino , kasan-dev@googlegroups.com, kernel-team@meta.com Subject: Re: [PATCH v2 4/5] workqueue: Show all busy workers in stall diagnostics Message-ID: References: <20260305-wqstall_start-at-v2-0-b60863ee0899@debian.org> <20260305-wqstall_start-at-v2-4-b60863ee0899@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20260305-wqstall_start-at-v2-4-b60863ee0899@debian.org> On Thu 2026-03-05 08:15:40, Breno Leitao wrote: > show_cpu_pool_hog() only prints workers whose task is currently running > on the CPU (task_is_running()). This misses workers that are busy > processing a work item but are sleeping or blocked — for example, a > worker that clears PF_WQ_WORKER and enters wait_event_idle(). IMHO, it is misleading. AFAIK, workers clear PF_WQ_WORKER flag only when they are going to die. They never do so when going to sleep. > Such a > worker still occupies a pool slot and prevents progress, yet produces > an empty backtrace section in the watchdog output. > > This is happening on real arm64 systems, where > toggle_allocation_gate() IPIs every single CPU in the machine (which > lacks NMI), causing workqueue stalls that show empty backtraces because > toggle_allocation_gate() is sleeping in wait_event_idle(). The wait_event_idle() called in toggle_allocation_gate() should not cause a stall. The scheduler should call wq_worker_sleeping(tsk) and wake up another idle worker. It should guarantee the progress. > Remove the task_is_running() filter so every in-flight worker in the > pool's busy_hash is dumped. The busy_hash is protected by pool->lock, > which is already held. As I explained in reply to the cover letter, sleeping workers should not block forward progress. It seems that in this case, the system was not able to wake up the other idle worker or it was the last idle worker and was not able to fork a new one. IMHO, we should warn about this when there is no running worker. It might be more useful than printing backtraces of the sleeping workers because they likely did not cause the problem. I believe that the problem, in this particular situation, is that the system can't schedule or fork new processes. It might help to warn about it and maybe show backtrace of the currently running process on the stalled CPU. Anyway, I think we could do better here. And blindly printing backtraces from all workers would do more harm then good in most situations. Best Regards, Petr