From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8C0430C173 for ; Wed, 3 Jun 2026 13:40:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494044; cv=none; b=S9rEifT3a3xFmnQEybRB7mqFjqubc6p9yN4SMosV5MqM+bsFG59qkXAiZGektvg8ZVRy+FnIRBkkg9J2J4DgbU7lD8Nb/TS5IkvRxkBpTAVc4GXjT73+epuh8e5Q9U+9rvGin/9+H8cum/jZOFWHuhrltTrTazSDaMz1kxSIM3I= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494044; c=relaxed/simple; bh=jWdNE/KMeZJTSiqSHYYEzG4CEGJkmcoUePz3s6zuso4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=XzVoUoh9me/vYVMRvu9GLNj8yqMf5DMSnGrXtUuYByF7bb45CmBqbEFBuTaTDqxXP6t3LRjlmZHOJkegxQm+Y8K/6O5uc/9h8erWwC5yTcPEjuBB0Bc24/2ZTxMpenSQlyzU0sQPo/cftqjbEmT51HBki0h95F6LNn1+ixYBB6I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=wG8dWCwz; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="wG8dWCwz" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=mmRjFmnHa4OtVYeUn9gDh8wWdcqmWzYGmuxNKJfY0k8=; b=wG8dWCwzuJBrKnMxVs3yMzwJGt 1LSn7r0JfPkj5CQOW2TuOABKRGP2F3s46A7Wwb+K7Inkl4plFHaj31KAAaAePfFeGtfcly0yDbAaU zgFDOEEA2JNBdrd62bTPR8OmklugooeEc5fSFDN2br2Xakci2LkH8ZVYXFBoiW8/wkIBg4E3Nb31x DqSIzWAmXVfxQzmsErv1BZMZGhvqahscll3p4StTuQ+VgGoeCXupI6ka4r7YMKfdUuFquhQzcNrcC t8wlI60zJc1ku5IbBVcVFPNPYVjIFzVpa78D8L3SByp301YTKNzSvnwZ4JgE6Eo84KiLJjiDBEc8p VYu4qg0g==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpX-003m3g-2H; Wed, 03 Jun 2026 13:40:24 +0000 From: Breno Leitao Date: Wed, 03 Jun 2026 06:40:08 -0700 Subject: [PATCH v2 1/4] workqueue: park kicked worker on pool->kicked_list Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260603-fastwake-v2-1-2977512fe7fa@debian.org> References: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> In-Reply-To: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=4239; i=leitao@debian.org; h=from:subject:message-id; bh=jWdNE/KMeZJTSiqSHYYEzG4CEGJkmcoUePz3s6zuso4=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+GH0/6Rkqt7MeHKoUzuGzTTrAHwNOgLiZj 0+E9FNyUtKJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bcDBD/9Ez8Z7D5jmDDLPNF+lq+JaRG+7fu8hHDCEJtWZW5GJHFvcWh1bOmZmrj1YUeeQfuaQuY8 AE7TPfWn1dJQpGEZ2/G3ubqYG7HmMFwg+KvWdr98COeKZyCzb52KsYTg/zuGnBvaobhosdk5gy/ VJkMadigouxT8ppTn5ev5JpK0CR+ROlM1AMHBb3uGYvhF1d0/luX35YlNlHpYKOpGzy2kdD7wQC 9RIsQ+7AhymfRrHe5SfnX1yo5KjYqBfU3fkuZUyWbO/oJp2PnhoWGeXxvGaUlgXrK/KyW1A1dVq UBOJVbWX6mzsUwQy7mep4pVvgkJM9hM/p9A+9/3QmFP2lr0WFZTHMlfYQyXyKJNR5pbLhmGbIga OYbxobZSpZ1PuzfFbiMSbr0aoFc9asWuFiKsL/yppwLbevS0AIDURMQIY2hAaH8VFTiwrpugcqk TOiOz7yKI0gghMZh1duNQPEMkQ3g9QKkYvC8Gh9jOAgbm/wKUmFYeNCu0M0sr07c7a2cl+RL803 GDMoHVxFD2VLeuTt4A4n/p4cXHMJl86Zh2dfwH1cCZgsKh/qttUcYj1TMP2FOvYu1IaTl8sgHu3 /VI4QeDn2L97pMg0r/c4EAbVYS/3dYP9kDOZRM1NkQ32GN2jlZpW3mvfiTVUBR4ikelg3hMTcKg lVJ74nkt25b7mmg== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao kick_pool() picks an idle worker and wakes it, but leaves it WORKER_IDLE on pool->idle_list until the woken kthread schedules in and runs worker_leave_idle(). idle_cull_fn() only checks WORKER_IDLE, not the task state, so a kicked-but-not-yet-scheduled worker is still a valid cull victim -- the cull can reap it before it consumes the just-enqueued work, stranding the item. The window is narrow today but later patches in this series defer the wakeup outside pool->lock, widening it. Move the picked worker from pool->idle_list to a new pool->kicked_list under pool->lock so the cull path -- which walks idle_list only -- cannot reach it. worker_leave_idle() already does list_del_init(), so it correctly removes the worker from kicked_list when it actually runs; worker_enter_idle() puts it back onto idle_list on completion. No extra list ops on the worker side. LIFO coalescing of back-to-back kicks onto the same cache-hot worker is preserved by having first_idle_worker() peek kicked_list before idle_list: the second kick lands on the already-kicked worker, the duplicate wakeup is a no-op, and the worker drains both items when it runs. Why not creating a new WORKER_KICKED flag instead, you might ask. I've tried it and the numbers decreased. Compared to tagging the worker with a new WORKER_KICKED flag, list_move() writes to worker->entry (offset 0 of struct worker), which the producer already dirties when reading the idle_list head; no new cross-CPU cacheline is introduced. Tagging worker->flags would have put a producer-side write on an otherwise worker-private cacheline, causing a coherence bounce on every kick. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 8df671066dd1..b3f8b86cb52f 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -216,12 +216,13 @@ struct worker_pool { int nr_idle; /* L: currently idle workers */ struct list_head idle_list; /* L: list of idle workers */ + struct list_head kicked_list; /* L: workers kicked but not yet running */ struct timer_list idle_timer; /* L: worker idle timeout */ struct work_struct idle_cull_work; /* L: worker idle cleanup */ struct timer_list mayday_timer; /* L: SOS timer for workers */ - /* a workers is either on busy_hash or idle_list, or the manager */ + /* a worker is either on busy_hash, idle_list, kicked_list, or the manager */ DECLARE_HASHTABLE(busy_hash, BUSY_WORKER_HASH_ORDER); /* L: hash of busy workers */ @@ -1031,6 +1032,13 @@ static inline void worker_clr_flags(struct worker *worker, unsigned int flags) /* Return the first idle worker. Called with pool->lock held. */ static struct worker *first_idle_worker(struct worker_pool *pool) { + /* + * Prefer an already-kicked worker so back-to-back kicks coalesce + * onto the same cache-hot worker (LIFO reuse). + */ + if (!list_empty(&pool->kicked_list)) + return list_first_entry(&pool->kicked_list, struct worker, entry); + if (unlikely(list_empty(&pool->idle_list))) return NULL; @@ -1310,6 +1318,16 @@ static bool kick_pool(struct worker_pool *pool) } } #endif + /* + * Move @worker to pool->kicked_list so a concurrent idle_cull_fn() + * (which only walks pool->idle_list) cannot reap it before it + * consumes the just-enqueued work. worker_leave_idle() removes the + * worker from whichever list it sits on; worker_enter_idle() puts + * it back on pool->idle_list on completion. first_idle_worker() + * peeks kicked_list first, so back-to-back kicks still coalesce + * onto the same cache-hot worker (LIFO reuse). + */ + list_move(&worker->entry, &pool->kicked_list); wake_up_process(p); return true; } @@ -4896,6 +4914,7 @@ static int init_worker_pool(struct worker_pool *pool) pool->last_progress_ts = jiffies; INIT_LIST_HEAD(&pool->worklist); INIT_LIST_HEAD(&pool->idle_list); + INIT_LIST_HEAD(&pool->kicked_list); hash_init(pool->busy_hash); timer_setup(&pool->idle_timer, idle_worker_timeout, TIMER_DEFERRABLE); -- 2.54.0