From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 783EC390210 for ; Thu, 12 Mar 2026 16:38:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773333514; cv=none; b=H+8QLiqYjKav88RuLnnrbthvRwp4hP/cKNCFSOZps974B/PKPInTI0B69/BzPGjNfKsvWPvte3B0j+xOZ6SMBXZOnz0uwZgKcFIAtLd6Ywf6o6rJFi+fVnRgRfQhJsPcodMX3ZZ996mRzClMDoiKU+XEXcQt12N1+lr39N4JBLY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773333514; c=relaxed/simple; bh=priOWCT7hQW6zppKaYvqO91hRYfijUUTF7HxQ1o8Lts=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=hQOzCHF7AiJMl+rp/VgFk/bvSt7J8Agll14ZKCi9J9Aw5op7M+v60kxrSWnSrl+hn7YayXlpRGb3GWgQDuEnloNsezrhH9JmuKzCnXeibSMZhI2ABMq+VKlFylNW9V5/SNOHcFA3KO20UaGJ9QGhbiEXdJlC0Jf2+ecMkdahjOc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=Rj6kw5VU; arc=none smtp.client-ip=209.85.221.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="Rj6kw5VU" Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-439bcec8613so1016155f8f.3 for ; Thu, 12 Mar 2026 09:38:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1773333509; x=1773938309; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Gpq2dYcsCrMCzCwlx19dQ4MTwo+ehY+ANg/IWj+5pyU=; b=Rj6kw5VULYmthAhYz+wALoSRhci1GBKtwBUxV9jdsqiN17pofVDY1uwdgRth6Xy5eZ DjhPcab+l52PBlUlESO02TJWLqn3l/hsRM9jy2qSBILQW6MPnAEE/K7UNHu4b3Gmb7ES pMb8sPjjZSRmT3cA9gHuwlK6SiNCPEuLav8kk1f5EspcoFD3AhtJFJ9NH9DRRLFBAwl/ +U+AbucfCVJoXP/wJbgJAUV/hxxt/0J/OoV40aRrQLnsOgN706DIHOnzaQVB5JGw4xwl VAy0FsWRzvTkbt3YVCBxaWogXYtr8T2HZCKWWph+GbQk4Vs9fDZfSAeGwaosa/xqdnOu LjOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773333509; x=1773938309; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Gpq2dYcsCrMCzCwlx19dQ4MTwo+ehY+ANg/IWj+5pyU=; b=YdtSJzclkM/QICRlaPD9V6m+fnHlS3rSdB8o65RH1hKss80kAAxsNfW8yXA7o9Nyfn 5NEhEtZWuQYgWZMzrDrxFJuFH+VcPSZ4QuNaJ4ofEAvzgTTdC8sR0PRX0HAyiwj15apK IJF8h23XBsUga26/bCdUw8Rjl/IckEpOhQ3UK8L56c5UixaMlgZcVaPAktWkPweAPh/M DX9wzyOf/DNJXNqxzYEs8k9o4PpfOMLG02LTY6JPpoHxosG/78nuaIBbraPjoeYerAr/ 2RngRdpggUchmXugQ/GL4pQ7FZDU6/HuKEPmqfh7DqFJvXHZCHh1/wAs4xtR6UtWhNqJ dgrw== X-Forwarded-Encrypted: i=1; AJvYcCVqdzjMFIS+Yz8I/VhbD/u2fS+9xf+1ZZAAO/k94GwkEpui/prZ9PmZYI+t8pJ3wqMnipVAxwinnui2USc=@vger.kernel.org X-Gm-Message-State: AOJu0Yw/BtdXq8kwokfL6A7+zqZd2bDfAXtw3YB2g2YMvzEpkKqoFiol 8ItJalB+p3VjEJSWjvGTVXw/u9L0PCVSabG2ywi3he9H/oNtoIZaNc5W+ZiZ0xzWIYQ= X-Gm-Gg: ATEYQzxPXZMqJJYA9OMO/6faXmbsrhqg4WmWsuXNMwBT4WPlsBNybcrAOsH+BUQyQK7 BrycRAlt9ci0ijCSlq6kf2C38gZwubmlW/dZaFkeBE1Fy7vbLd2mZgP8U1NPUIsN15SL2ffwIsF BU/DiDSRQ6RnqcPJqGAofI4buZXPM8n0VL4mlMss5RG8TQ0eg3QHuo3YHqoKA9TodyIUKY6m9cN ow1iZ/i5FREjqFLL3OSeDhzUhqLNAw2ou2Oq4C2wPA4wZ5oKbgdEAXAmU60WvfeP9jYUFY/YL8x TulIRLZKIIDZh7hbt1fDvkMcHiNENKhmCMDYqF/RXZNyffiiY8g+PQVYhzqzvTrr+xaxQsT5abA HwWzriftRtUYL2SUOj4mwa45fuXWwt1yR1GeScvhv2uF4sNk4xmdm455tQQmqJlCXDViDDud1iO wybQs/diLKGVmOvGPuRAjZIOE9zA== X-Received: by 2002:a05:6000:1848:b0:439:f55c:b556 with SMTP id ffacd0b85a97d-43a04d7cc68mr750612f8f.8.1773333508779; Thu, 12 Mar 2026 09:38:28 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439fe19aec5sm9292096f8f.4.2026.03.12.09.38.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 09:38:28 -0700 (PDT) Date: Thu, 12 Mar 2026 17:38:26 +0100 From: Petr Mladek To: Breno Leitao Cc: Tejun Heo , Lai Jiangshan , Andrew Morton , linux-kernel@vger.kernel.org, Omar Sandoval , Song Liu , Danielle Costantino , kasan-dev@googlegroups.com, kernel-team@meta.com Subject: Re: [PATCH v2 0/5] workqueue: Detect stalled in-flight workers Message-ID: References: <20260305-wqstall_start-at-v2-0-b60863ee0899@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260305-wqstall_start-at-v2-0-b60863ee0899@debian.org> On Thu 2026-03-05 08:15:36, Breno Leitao wrote: > There is a blind spot exists in the work queue stall detecetor (aka > show_cpu_pool_hog()). It only prints workers whose task_is_running() is > true, so a busy worker that is sleeping (e.g. wait_event_idle()) > produces an empty backtrace section even though it is the cause of the > stall. > > Additionally, when the watchdog does report stalled pools, the output > doesn't show how long each in-flight work item has been running, making > it harder to identify which specific worker is stuck. > > Example of the sample code: > > BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck for 132s! > Showing busy workqueues and worker pools: > workqueue events: flags=0x100 > pwq 18: cpus=4 node=0 flags=0x0 nice=0 active=4 refcnt=5 > in-flight: 178:stall_work1_fn [wq_stall] > pending: stall_work2_fn [wq_stall], free_obj_work, psi_avgs_work > ... > Showing backtraces of running workers in stalled > CPU-bound worker pools: > > > I see it happening on real machines, causing some stalls that doesn't > have any backtrace. This is one of the code path: > > 1) kfence executes toggle_allocation_gate() as a delayed workqueue > item (kfence_timer) on the system WQ. > > 2) toggle_allocation_gate() enables a static key, which IPIs every > CPU to patch code: > static_branch_enable(&kfence_allocation_key); > > 3) toggle_allocation_gate() then sleeps in TASK_IDLE waiting for a > kfence allocation to occur: > wait_event_idle(allocation_wait, > atomic_read(&kfence_allocation_gate) > 0 || ...); > > This can last indefinitely if no allocation goes through the > kfence path (or IPIing all the CPUs take longer, which is common on > platforms that do not have NMI). > > The worker remains in the pool's busy_hash > (in-flight) but is no longer task_is_running(). > > 4) The workqueue watchdog detects the stall and calls > show_cpu_pool_hog(), which only prints backtraces for workers > that are actively running on CPU: > > static void show_cpu_pool_hog(struct worker_pool *pool) { > ... > if (task_is_running(worker->task)) > sched_show_task(worker->task); > } > > 5) Nothing is printed because the offending worker is in TASK_IDLE > state. The output shows "Showing backtraces of running workers in > stalled CPU-bound worker pools:" followed by nothing, effectively > hiding the actual culprit. I am trying to better understand the situation. There was a reason why only the worker in the running state was shown. Normally, a sleeping worker should not cause a stall. The scheduler calls wq_worker_sleeping() which should wake up another idle worker. There is always at least one idle worker in the poll. It should start processing the next pending work. Or it should fork another worker when it was the last idle one. I wonder what blocked the idle worker from waking or forking a new worker. Was it caused by the IPIs? Did printing the sleeping workers helped to analyze the problem? I wonder if we could do better in this case. For example, warn that the scheduler failed to wake up another idle worker when no worker is in the running state. And maybe, print backtrace of the currently running process on the given CPU because it likely blocks waking/scheduling the idle worker. Otherwise, I like the other improvements. Best Regards, Petr