From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2623C333434 for ; Wed, 3 Jun 2026 13:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; cv=none; b=Tr3jJpiAfRn2AFDoJqEIF1+Cc+mYq6xbKo2l8wG9wFylwb2IorEwerJTLvDj4t6cox9uyWDFOTkt1vzrkOkcjz8g1a6aNhuFMtRxDZFyTANne1FwhTNxhR7K3jTlDEEmHiYHFfvc+pU+m7PYfjzn3VCt24NyL4HWSUSaFZGDQok= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; c=relaxed/simple; bh=PP8L3V9RvwIe7/3EQHRj1yOGzLt+tldSrZcyiaE/5Q4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rGdPvlLa2zHCr6pxU1BXFS9oxzAFe9yQhxnBiVAHlA9U/Y6FfRiN4dfTBLjxj5aZearSpsnDSe4MWD4Udwt0YVamCMaVHQ7+x4fPxKYd7oRQ1yUsGQYRPYTfjxFVvDTJY8CFk42nFH9aMUxEvV4LQrmLmBquZhdMVJDRUvH6IcU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=ULHLQ9Wy; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="ULHLQ9Wy" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=84V6l6lJO/YvuiFUAqRuwGACP7KgIG4dKLq8Inlqnk4=; b=ULHLQ9Wyga/i2FCNnBSdBxTE3N w+rd581bxqsHXE5uT9ZI2WhZ3NRmFIzIMI/qv+LhRs1cS3u9uzdKPNl7dwTPPgKSHvewH9JRZ1UJ1 SLWr3LufJm22ejmwBp8R2lPEVGi7OB8W5FGHOidwhLb1ngnEWEGBBlIRE5wIjxYE6yYm5bnz8YuSb IoMebcZf32Wdj/pgAiIJGHmGrlgff/poOyE1EtthJy1cyRYOqKR03ImiD1j+qpddWwf3sbf+l9kSi pCWUAGcFfLcExTESeBoMTKd2uP2/+sEhtRTI1J9JnhkQ1nFltOK6DmiPi99/K/sEfbZZBsQR9vbti LwLtv8BA==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpk-003m4A-0e; Wed, 03 Jun 2026 13:40:36 +0000 From: Breno Leitao Date: Wed, 03 Jun 2026 06:40:11 -0700 Subject: [PATCH v2 4/4] workqueue: defer the worker wakeup outside pool->lock in process_one_work() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260603-fastwake-v2-4-2977512fe7fa@debian.org> References: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> In-Reply-To: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=2998; i=leitao@debian.org; h=from:subject:message-id; bh=PP8L3V9RvwIe7/3EQHRj1yOGzLt+tldSrZcyiaE/5Q4=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+gn7OlcJ5+su4WhUscE3a0LyIguogbXHOD lyvDB5JaaCJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bV8QEACLUuDiz2F8pD+QBJ0o4tO1l2dF9RN4yzKLh4hT0mKArVcXfsPwSIiyGmdiaRLkzR/v9Mp 5Pm2zV0C02Mgpn/YJx0PWuJdQbmYt45Qy24HttIiZkKx3mUqjs+6Z9j9chVGJz823UTq8HKESht QnG56libGEQO8kyqrxCF2MxYeiIT9I0V828NEp2+MFvsqb1P24YdxvRx54FYHcU/VXSIA/ZcLaZ 4h+qQ7l1zqPrDPIHhJZskiCxtQ0qB1yk7fp7W26k4OWn5wElkqsloYjSN+YqcCJYf80zNiz/k62 eXJMM11bRoGO+wi/rNR3CnSGQ0pRpydGNhvZWYzqXfWeK3kISBY6lwWqeuR/SwMEqbsigjrM/PN 4ShQab4ow2XtK41CodcylKD6KRhl95aunFz3bzcnEyIjB0goTLefCqICAN4KuGgjXhPYEe+MLXp Kkwbc5L6Uq5X5TcjTJQnvd4YpKHnalOiHCKX0wdTX1yYw3adSrg5sj/DY1A56T9/2oiRLkfxYd6 HvpXnx7ymrNwPJEvoSY56HaTUYaBlMMeHZgm1UkJJXf7GWucx1dqEjxmSxz0uDC4cKmC/PELMRL T+/wmjTXd/yOkMfrbqYM58V9N19qz64JIDA/42dbTTpQWDSAw/xeupzjS/Obc8JxX/7MaZc4gao oCKFs/EtrL5DjZA== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao process_one_work() kicks the pool to chain execution of the remaining work items on WORKER_NOT_RUNNING pools (the UNBOUND and CPU_INTENSIVE ones), calling kick_pool() while holding pool->lock. As in the enqueue path, the wakeup pulls the target rq->lock in under pool->lock. Use kick_pool_pick() to select and claim the worker under pool->lock and issue the wakeup with wake_up_q() after the lock is dropped via raw_spin_unlock_irq_wake(). With both hot paths converted, measured on a CONFIG_SMP x86 VM (8 vCPUs) with the in-tree test_workqueue benchmark (lib/test_workqueue.c; each of 8 producers queues 200000 work items one at a time on a WQ_UNBOUND workqueue, waiting for each to complete), medians of five boots per scope: affinity_scope baseline patched tput p95 (items/s) (items/s) gain drop -------------- --------- --------- ------ ------ cpu 3,611,591 3,568,433 -1.2% +4.6% smt 3,601,697 3,550,632 -1.4% +6.1% cache_shard 341,913 401,213 +17.3% -36.8% cache 320,607 400,560 +24.9% -41.9% numa 324,909 389,202 +19.8% -38.0% system 314,510 392,278 +24.7% -37.5% (p95 drop is the change in the p95 enqueue latency; negative is better.) cpu/smt use per-CPU pools with no producer/consumer contention and are essentially unchanged. On the contended scopes the shorter pool->lock hold time cuts p95 enqueue latency by ~40%, and because this workload is bound by the producer<->worker round-trip that latency reduction also lifts throughput by ~20%. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index b4246a801dd8..238b02edd01d 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -3261,6 +3261,7 @@ __acquires(&pool->lock) { struct pool_workqueue *pwq = get_work_pwq(work); struct worker_pool *pool = worker->pool; + DEFINE_WAKE_Q(wakeq); unsigned long work_data; int lockdep_start_depth, rcu_start_depth; bool bh_draining = pool->flags & POOL_BH_DRAINING; @@ -3315,7 +3316,7 @@ __acquires(&pool->lock) * chain execution of the pending work items for WORKER_NOT_RUNNING * workers such as the UNBOUND and CPU_INTENSIVE ones. */ - kick_pool(pool); + kick_pool_pick(pool, &wakeq); /* * Record the last pool and clear PENDING which should be the last @@ -3326,7 +3327,8 @@ __acquires(&pool->lock) set_work_pool_and_clear_pending(work, pool->id, pool_offq_flags(pool)); pwq->stats[PWQ_STAT_STARTED]++; - raw_spin_unlock_irq(&pool->lock); + /* deferred kick_pool_pick() wakeup, issued outside pool->lock */ + raw_spin_unlock_irq_wake(&pool->lock, &wakeq); rcu_start_depth = rcu_preempt_depth(); lockdep_start_depth = lockdep_depth(current); -- 2.54.0