From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50F33349CDD for ; Wed, 3 Jun 2026 13:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; cv=none; b=EzrM7ZMVZ5G6aY4phoUyakNh51oaTqlzRjNbaxVlvowFP6/v3VyU6x+sMNfydWX2FnTBsq1ls/uSrAE2iqcZYeT9G9MRwfLkKHIWGIyPz5ogYaLwYvCE2mBO6+SnpRKo74yIUavxDWi83VGH8n9bNpb1x/Mk8aIyVbZJBA+j0M0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; c=relaxed/simple; bh=Rh8NcshA+JKySXxueL/MKaq6iIyPz5j3yokH2xCJHdU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ewsaMPvGZx1ZB3EzkevbnsoVazcj2+1wlSLcf1gCZYJ4nahKg3lRmB1qNcdwBFzUkLUJSLY7km0J2jSLB/x7vRo8My19DnMdEP+74vEVHsO19xMuLXzhxaNc7PzxUs9uGVitVPSMqGil6ayhrB4/XJfkEXyIZ8ywAd4AKKy4qtE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=PmcYLo6e; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="PmcYLo6e" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=hrswI0VJOBYu3MvyNJ2d8q2GqckwRefPMcCP+neOB8w=; b=PmcYLo6eVkAvHThojyqtMI+euo yF+3/pbWlUwCHRo1PGCGjR15sKwCt7RwPWju0l80uHP8UMwUUMmM3wwpcE9HFbatz0K9liOoICIqE 1BCcSK8Piyi6SCfo6W4rzteYWP37cDIxdeyngGENhrHwTsP0+czNyzqejlOHt9q4wFBYaRiVTbum/ V0nh7KPOyoTBWspbL3n5iXNjyq7Xv2Lt322d7I3BSUiMeVQL6BubqFtendZgBPZjOSD9+40kXGWlz /ytwL03mXjUVELkf41xmc17S53LUMTcB5FOAjc5pEU7Ity6PgLa9OHP7VmCTEwr+CX5/Ntg5cEqlR ugyQ8hEw==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpg-003m3x-06; Wed, 03 Jun 2026 13:40:32 +0000 From: Breno Leitao Date: Wed, 03 Jun 2026 06:40:10 -0700 Subject: [PATCH v2 3/4] workqueue: defer the worker wakeup outside pool->lock in __queue_work() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260603-fastwake-v2-3-2977512fe7fa@debian.org> References: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> In-Reply-To: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=1744; i=leitao@debian.org; h=from:subject:message-id; bh=Rh8NcshA+JKySXxueL/MKaq6iIyPz5j3yokH2xCJHdU=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+8QU697uyoPjgWM5obezKLndTtXQ61Tb+3 R7cjgJnmV2JAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bev/D/9h6jnT4YWF5x/kbz1QiR4Loc9i1HbxYqYyYIfyarik0nzu6GEYQcbSaN0llPeNYk8Nr6Z K8IavXzF4N4KCEMfqbg3RsTKFQzKSCXY8f7DhqrDTh0hrxtEYrO/YP0M4Y8UK2E8WutQ5EGsXlD g+272cYXq3eZpvPlxDtuGyz8ylGTaPOmU0cNzpry4GdNisaaoQTylFk9vPdj2XX0zc4LFFMjZvn nmK4uPzEmt7NE+HSCN6/vAF9Z94UKjWbK8PlMHTQq397Iolenq0WuXUsTYhxm4+3KlMFFlcmeqe xWJz21l/hNYK6W8UPYI4GVn2kE9X8WB8ay5V27+PN2Z2y6NFnSFEnhfZGHWsGxaTYEVRp0tCZMj VDjFUmD4Ivio9HyXilAW202esfhcNZfPfKTY2IflVfb9QMK66BjTwXoVuX5z8ZRBITTTcsssi2P RZXdyRNyXKKzymHgp+oBvBfNqqwO7aJHtiZ8/mP5/izUmQbaTWiTRMgi3DQwZebBbVIGC1K4RSg PXeW2Z2sO0vYDPzY1SJxvTmD3m/zfi0XQAnIHgis3NWfzVZaQGF5ipK8Yn/37+bWg1G4tORAP4w tD2zZHxSaf5VdxHE2EtULDFNwEjhemwF0WCBJiCbqlGi9Sn+tQ7P1U9jpYVf09e82g3EPMyB1r2 ogZGR2DwdhiwChg== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao __queue_work() is the enqueue hot path: it inserts the work item and calls kick_pool() while holding pool->lock. kick_pool() ends in a wakeup, which takes the target task's rq->lock, so rq->lock nests under pool->lock on every enqueue that wakes a worker on a contended unbound pool. Use kick_pool_pick() to select and claim the worker under pool->lock, queue it on an on-stack wake_q, and issue the wakeup with wake_up_q() right after dropping the lock via raw_spin_unlock_wake(). Worker selection, wake_cpu setup and claiming the worker off pool->idle_list still happen under the lock; only the rq->lock acquisition moves out. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 2811ada6dec6..b4246a801dd8 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -2317,6 +2317,7 @@ static void __queue_work(int cpu, struct workqueue_struct *wq, { struct pool_workqueue *pwq; struct worker_pool *last_pool, *pool; + DEFINE_WAKE_Q(wakeq); unsigned int work_flags; unsigned int req_cpu = cpu; @@ -2431,14 +2432,15 @@ static void __queue_work(int cpu, struct workqueue_struct *wq, trace_workqueue_activate_work(work); insert_work(pwq, work, &pool->worklist, work_flags); - kick_pool(pool); + kick_pool_pick(pool, &wakeq); } else { work_flags |= WORK_STRUCT_INACTIVE; insert_work(pwq, work, &pwq->inactive_works, work_flags); } out: - raw_spin_unlock(&pool->lock); + /* deferred kick_pool_pick() wakeup, issued outside pool->lock */ + raw_spin_unlock_wake(&pool->lock, &wakeq); rcu_read_unlock(); } -- 2.54.0