From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8C7830C17D for ; Wed, 3 Jun 2026 13:40:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494045; cv=none; b=Pb3P2NpD+gWwgPmNfHf/LGl53OtgBXsC2QKVt4FCcQk+47XVOxHG+Hpceq1Ei8D2FJ71eVwh+C9KAohONi0pOZKwbRGUCvWKU4oJOrL3/JSjcaOcv8NNOApZC/NFXGJATtwiGuEPEM8RDjhTUaufLMK0/slsU4d6bIn2l2KtKZ0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494045; c=relaxed/simple; bh=9M8x/diIeVR41NCG12Sehi61qw51pADhGydSOwi2SP8=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=FV6eUvOoRBR9xGLtDAkWDD8fnTnU4aeM6IOXADJIo02hUTBEo96lDanfEcddD/vgis2cDzz3NXxVzmYZkxGLY9K7xIbO17OHOJM9O76FcLPC+6VShbhHI5CjYPEfxFupDbAcWC3ngEvsL6yyDW5t5eOmSOlNUB2no5TY091brxI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=T4wDv+WD; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="T4wDv+WD" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-Id:Date:Subject:From:Reply-To:Content-ID: Content-Description:In-Reply-To:References; bh=63H+J32XE926g592B0jncyimb+RNJcAgaAnkeEXVmOQ=; b=T4wDv+WDrmCTKCku4YsBRFRrQ6 8v+dzT4dK6DZMc2GMO+4AkEX2cVHCjkPujyZoxQNkvh5rHv4VRHOT4Bi7dXCg0X1pZjiMjeY5YWb4 yifmHhe2Pfcwyf93xXL7F7/5bwnmitpcMgHJ6eBxPYsatp+ObP0zBe+9yT5UqgEuTXfphiPB9OEj4 PeJDHhp2G3Y4FsCH6MiulwWaugb0TzzIeyWvuAz0BL+JF1t4unw1BKFFwavnag0ZPFBA9vr0g8vLF Arfa6VhKOjFa9h6QhVmzIkW/hdxn7/1ryXxlAVRaVvUShPSGp4fdJD9bTF1m/j3JUo2UIbivLO5nP WEgvrBOA==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpT-003m3d-00; Wed, 03 Jun 2026 13:40:19 +0000 From: Breno Leitao Subject: [PATCH v2 0/4] workqueue: Shrink the lock time Date: Wed, 03 Jun 2026 06:40:07 -0700 Message-Id: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIALcuIGoC/yWNQQrCMBBFrzLMuoE4xVBzFXER24mdCq1kolZK7 l5Slx/ee39D5SSs6GHDxB9RWWb0QA1gP4b5wUYG9IBkydkzOROD5m94srF06SgOzrUnwgbwlTj KeqSut//W933iPle/EqNoXtKvfpVSdkv/+1R7AAAA X-Change-ID: 20260526-fastwake-02982fd66312 To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=3461; i=leitao@debian.org; h=from:subject:message-id; bh=9M8x/diIeVR41NCG12Sehi61qw51pADhGydSOwi2SP8=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+HSK8Hu46N3dPatPe78sN4tGNESJP99hGA 5vRR3EjLiOJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bdpTEACwu4E1qWcwInTpBWVK5aLqVc4KuyzgI/yWQyPXakTTH0QXNeRCblvJuHYUSi1PyPtdWQk jZCtehL6Py71dJ7nT5DTFHKunfv62G/zqk8ogjh41sh87V8HBer7uW38A15R3bWhbsS0zIHr/hH eqS/TCVm+3JRwD9sQu7zhC408JVylKGHa78vsX7HPMZ0MYRW9st2Zww0HvEFMf2YooMvraUeEEN S+nTuWZX2vOiksWL6p31BrSNzsNh/7KF7WwLDyfMtSGyH5J4oDGl/7l8R4D8P+4qgOZxOA1c5Q3 t3T5igupR8BZWVm6gRwRqGIdkCz0ppE8Nzv3RgAjyNkG9JDtQjULK9y7/7LUFK1qA0duUZIeNuR as+YIQH8iQJMyZ8XoBAU78SMWsHRjA0Cj4T0T7W1TcWdJQLb9LaK5ULnK9ZNyW8sL26kjWM3L60 ef7rJTJQDUik6rQ81m5wvgwbanS1pdu2Sl8IaEJy4yngGGKi7YrF+GCL5gBgxH2SPbmmJm5mlWR NX25TIHaKgXSGZuTYDLiGHm10x5v/MCCwCvZnmb2zBUTWZkdSh/T+x3otQAPzVNaZ+TPNCImWWH SewhAU/Qieff7tCvKgz25Fza+3y32JX2HHfqXuHjbGaC2yBdDymx+UVrHS8rfXElkaX1aCdLcGF I90+S7FOSJxHGyg== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The goal of this patchset is to decrease the time spent under the workqueue pool->lock. Currently the worker process is woken up inside pool->lock. The wakeup ends in wake_up_process(), which takes the target task's rq->lock, so rq->lock nests under pool->lock on the two hottest paths of a contended unbound workqueue (__queue_work() enqueue and process_one_work() chain kick). On some architectures the wakeup is even more expensive: on arm64 waking a CPU that is idle (in wfi) issues an IPI. Doing all of that while holding pool->lock lengthens the locked region and hurts throughput on contended unbound pools. This series shortens the locked region by selecting and claiming the worker to wake under pool->lock, but issuing the actual wakeup after the lock is dropped, using the wake_q machinery (wake_q_add() under the lock, wake_up_q() after). Because the win is a shorter pool->lock hold time, it shows up most clearly as lower enqueue latency under contention. Measured with the in-tree test_workqueue microbenchmark (lib/test_workqueue.c, 8 producers x 200000 items on a WQ_UNBOUND workqueue, x86 8-vCPU VM, medians of five boots), on the contended affinity scopes (cache_shard, cache, numa, system) p95 enqueue latency drops ~40% and throughput improves ~20%. The uncontended per-CPU scopes (cpu, smt) are unaffected. While reworking this, Hillf Danton pointed out -- and a closer look confirmed -- a latent race: kick_pool() wakes a worker but leaves it WORKER_IDLE on pool->idle_list until it schedules in, so a concurrent idle_cull_fn() (which only checks WORKER_IDLE, not the task state) can reap it before it consumes the work, stranding the just-enqueued item. The window is narrow today but deferring the wakeup widens it, so the first patch closes it by moving the kicked worker onto a new pool->kicked_list under pool->lock, out of reach of the cull which walks idle_list only. Patch 1 fixes the cull race and is a standalone correctness fix. Patch 2 is a pure refactor introducing kick_pool_pick(). Patch 3 defers the wakeup on the enqueue path (__queue_work()). Patch 4 defers the wakeup on the per-work chain-kick path (process_one_work()). Changes in v2: - Close the idle_cull_fn() vs kicked-worker race by parking the kicked worker on a new pool->kicked_list under pool->lock (new patch 1). Reported by Hillf Danton. - Use the wake_q machinery (wake_q_add() / wake_up_q() via raw_spin_unlock_wake()) instead of plumbing a task_struct out of the helper by hand. Suggested by Sebastian Andrzej Siewior. - Link to v1: https://lore.kernel.org/r/20260526-fastwake-v1-0-e69ad86923e6@debian.org Signed-off-by: Breno Leitao --- To: Tejun Heo To: Lai Jiangshan Cc: linux-kernel@vger.kernel.org --- Breno Leitao (4): workqueue: park kicked worker on pool->kicked_list workqueue: split kick_pool() into kick_pool_pick() + wake_up_q() workqueue: defer the worker wakeup outside pool->lock in __queue_work() workqueue: defer the worker wakeup outside pool->lock in process_one_work() kernel/workqueue.c | 65 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 55 insertions(+), 10 deletions(-) --- base-commit: c1ecb239fa3456529a32255359fc78b69eb9d847 change-id: 20260526-fastwake-02982fd66312 Best regards, -- Breno Leitao