From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CY3PR05CU001.outbound.protection.outlook.com (mail-westcentralusazon11013067.outbound.protection.outlook.com [40.93.201.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2009426B2D2 for ; Wed, 26 Aug 2026 06:34:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.201.67 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726046; cv=fail; b=AbsDmI+PQFXQb5WsICECjshufp8TiUJlaqueh9FEWOJWUxM2kIm72rSKJ3okQ70nt2tKLiF1msuMDkc0lJGnrhOK9N66OuO1uND0LG603lRhcluo9OR7KzRy591l4/cB3uXse+HCEtJ3nRVDKJqQx+x/Qr692AeRqgC9okPWtig= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726046; c=relaxed/simple; bh=YzZhWuxtveg3ZNEOx5BRhRHfdNturIs9qCnulljZzf0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FNriCf6KAf7fyumCTAckOfEa0FOV0MitQI3ybePJ9p1bk39zbBt1Rw1Em9HbYQEx9DrzCOW7BIF2jWW7+S8jwoQ3TuXhc3oFKurGB5a438RJsQ/zPlltONTaVTnzaTHsGs0+TpImmvfzyAEQwVavVHvSOxg5/ypNq7tGscokCHc= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QeQ8tp0C; arc=fail smtp.client-ip=40.93.201.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QeQ8tp0C" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lPedsPMFcxCMWFwF7MVHBcDXlE5GMDQVev3AFrfMXbyDbK3TGOBWs53rANOaj3i78E110zNVHdUlOx6ITZFR8R6YZsX35OTpVja6uVzcX9yBvq7NOkfUZuns9/daoqebZ2MG1DV7XbcHs5ob2EPd6DTVNQ6H8Tm5fFcdVlG1OKYYranEBdT+foOKjTFvLGIob+VC9RHkdJY8r+pkZZOLlUIjOdojmEJLxUSl+TbyA1d1MzRYYXvQiNWuIOgmwNcMhrGUzOkwmYGwDgiN2CfW4gnrX/WIaGIkuG6PT+1srxc47rwNmQwOPcf84n6/Jrr1ScwrgvuMcytLxE9BB1WRhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zQV08FaGagoR7EY/zZz9C9jlCjK4pXuei/Fxt1upO+M=; b=G/RErbQ4BwzvZbpM1fqKK7j4QzDE+qFX2aDvS4w/eI0cAXQGY2UatvseRmA+QJAjLNYHRq7VCkx0O2Ve1EGU1tYFriFAGqjbo5dn3FG1kyWWjsYfbxfCRVbq2o/w3gvtXkmpflFAEVGlOlk/PE6O6+cBy0GGZoIsp/cmo527Zrcbi0aEr+mLMqmE45d0Urv/TTDUaVquCx2kKaFKOjNz3MjblVrTaNfYICbf4L2SrvrnwwBJE4HfS/3X+957Q0AgiCtUN6dXOy4s+3I9iuDkFUvprQvVgl7NDU+ShfY8b8R1fewqDaGXU8u2CW99Y6SrfWqA9ABVhyQDds56NPrFyw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zQV08FaGagoR7EY/zZz9C9jlCjK4pXuei/Fxt1upO+M=; b=QeQ8tp0CXUv6+inE0DU2TOYiXnqohgjJJH0w5X/c2ucbYfJvJ66luqIcr4GD0KqKWTuz3vvwAfN8omLUJ3ytCJXN1gZIA1l5uPexJuuKSWq4TDAoTNSprYBVvz8JcdouL9yG2f+Su8lWTPCGQCktLlcOnGusqCGySXNH4mPPqYs= Received: from DM6PR05CA0052.namprd05.prod.outlook.com (2603:10b6:5:335::21) by BY5PR12MB4052.namprd12.prod.outlook.com (2603:10b6:a03:209::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:33:57 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:5:335:cafe::16) by DM6PR05CA0052.outlook.office365.com (2603:10b6:5:335::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.7 via Frontend Transport; Wed, 26 Aug 2026 06:33:56 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:33:56 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:33:51 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 13/16] sched/core: Use p->is_linked state to unlink from sleeping owner early Date: Wed, 26 Aug 2026 06:28:57 +0000 Message-ID: <20260826062901.2137-14-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|BY5PR12MB4052:EE_ X-MS-Office365-Filtering-Correlation-Id: bbc2172a-a63a-4142-56be-08df033c014b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700016|1800799024|23010399003|921020|10067099003|11063799006|56012099006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: 1uJMeSyrpNazLKrZAgF4BbxAvEfv3j4rpVtUkv3vdegHRjXKXApn8JlCE363e/oTOGLG+baqpSP3kQ+1W5mjz6H+qUhTE4GQmblEb6IDyIZ8qwmHfm1YBZ7Sc+4s89QPvAlrjBdFaUs90RlJf/ibZx9iI7HfTg9rGqCrhM8C4Sm7nQmTfdqgnCxOUfbVbOTjPHZMzfPIoMBiapR4n02HcoPhJSafiWJ8JpX0lTU6qvhDKAtgsE6fKcL4G9gZG89r+1gam3PwHR88agxS1EjrEr/rNjPnRS2IimvhLXhx4jbX1pRUvfxIIq8CNtIT54AL1QPDy+uSzHcrSCi61Z8QYh4mbI5Or7TwoJF4+BjTk28hDSmVmEJ2ZDxGd8OjxzUdfNmwb0Jg6kSJPbJildoqQuJ05ilKjyEFQVUvl/a4wYEodGIa2pkq3l/JvGEj8L/2pc7RYKzsXlBmLzqUh9AluN+H3EaO+4DgqYiEkD/gt3URIB0ACEiuB1P/xUGOIFNwdUriL5ajl6gAiNM50JkfJbsHD/w0cInvT9GC7WizLNESex25DVKfiRNo0Mu7x3Sy4zzlv9++yQFnNY4P5z5JSX34Inaazw207ZCUtMC47dGbizPA8EeQiMduzZC9L+OlLMFeQJi8ceBR5WIWUvYAyg+PBNLHjG269+Aw4IpTsONfa12ehtQ28BAjKl/ry5qwPABTgH+YQ4c9FUQecxRi+j9JsHszZgiUV7eXBTmaFWlAeiMjHM9tm2dZRTjbGZIK X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700016)(1800799024)(23010399003)(921020)(10067099003)(11063799006)(56012099006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sowFeQTswjtzEmL7N9DC2FkkY44RyRtixQGYt+XWSVHNgVzD+TWH+vr8LHTn3wnjm3GUdsHo3mQU+62xGJaWZfOzNhsRJWLc9B9azDcQkfUsvZUn/Va7JCmVgkyGfyoxY5yoo3einiKtyVOQd96DNqSJp7iDnOxy7JFzIy5uQ8lOfVZ2g/Tfe6eFLqH/goJGkkKYChNnILSzQbcyLggbtWdyQDdGMGmu1gFZY7y+WCOzES9MMexKAdGWE8wn9SNRgKO1ip2CfEM+vbhGrbkApca0nTHIp+pXBbfKwQojt10BwR5IxR6cgYG62vN08Ptx1VvNJ7zfOxP7sbnEnmC0lJlh19JhkIyqr/bdt/ms/dAA60G1bu2Q/w/1HITrvjgLu6DP5bHhT7b6Jcj5vbd+LH40J2huQnAlu6juyho0jkBg6MfC1QiNvDRNvt9d0I3v X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:33:56.3638 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bbc2172a-a63a-4142-56be-08df033c014b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4052 Use p->is_linked indicator to unlink the task from a sleeping owner's blocked node early and continue to proxy_needs_return() on the wakeup path. proxy_needs_return() is now the single point of proxy donor handling on the wakeup path. The cases that handle enqueue of owner is as follows: if (!owner->on_rq) { guard(raw_spinlock)(&owner->blocked_lock) activate_task(owner) enqueue_on_owner() owner->on_rq = QUEUED / MIGRATING smp_mb() smp_mb() if (owner->on_rq) /* Raced */ if (!list_empty(owner->blocked_head)) dequeue_from_owner() guard(raw_spinlock)(&owner->blocked_lock) activate_blocked_donor() Case that handles concurrent activation of blocked_donor is as follows: try_to_wake_up(p) guard(raw_spinlock)(&p->pi_lock) /* State matches */ if (p->on_rq || p->is_linked) activate_blocked_donor(owner) ttwu_runnable() if (!list_empty(owner->blocked_head) task_rq_lock(p) rq_lock(cpu_rq(owner->blocked_cpu)) /* Same as task_rq(p) */ Either activate_blocked_donor() wins and activates task or ttwu_runnable() wins and completes the wakeup after dequeuing from sleeping owner. activate_blocked_donor() changes are added in the subsequent patch. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 176 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 174 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ff499d4532ad..e6abcea97683 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2252,6 +2252,96 @@ static void __proxy_dequeue_from_owner(struct task_struct *p) list_del_init(&p->blocked_node); WRITE_ONCE(p->sleeping_owner, NULL); WRITE_ONCE(p->is_linked, 0); + /* + * p->blocked_cpu is intentionally left un-altered + * since the wakeup path requires it to grab the + * correct rq for chain wakeup if we are here from + * the activation path. + * + * "p->blocked_cpu" only has significance for blocked + * task on "blocked_head" and a stale value in the + * field is harmless. + */ +} + +static void proxy_dequeue_from_owner(struct task_struct *p) +{ + struct task_struct *owner = p->sleeping_owner; + + scoped_guard(raw_spinlock, &owner->blocked_lock) { + WARN_ON_ONCE(p->sleeping_owner != owner); + __proxy_dequeue_from_owner(p); + } +} + +static bool proxy_try_dequeue_from_owner(struct task_struct *p) +{ + struct task_struct *owner = READ_ONCE(p->sleeping_owner); + + /* + * Task is outside __task_rq_lock(). These fields + * can race with a chain-wakeup. + */ + if (!owner || READ_ONCE(p->on_rq)) + return false; + + /* + * The chain-wakeup slowpath grabs both the rq_lock() and the + * owner->blocked_lock while removing the tasks on + * owner->blocked_head. + * + * If there is no concurrent chain-wakeup for this task, it is + * safe to just hold the owner->blocked_lock which serializes + * the removal like: + * + * proxy_try_dequeue_from_owner(donor) proxy_activate_blocked_donor(owner) + * LOCK rq_lock(p->blocked_cpu) + * LOCK &owner->blocked_lock LOCK &owner->blocked_lock + * + * # Defensive checks STORE donor->on_rq = MIGRATING + * __proxy_dequeue_from_owner() smp_mb() + * + * UNLOCK &owner->blocked_lock UNLOCK &owner->blocked_lock + * + * # If proxy_try_dequeue fails + * __task_rq_lock(donor) __proxy_dequeue_from_owner(donor) + * if (donor->on_rq == MIGRATING) __activate_task(donor) + * cpu_relax() STORE p->on_rq = 1 + * + * # Sees consistent state. + * + * Either proxy_activate_blocked_donor() manages to dequeue the donor + * and stall the __task_rq_lock() until the task is actually stable + * or proxy_try_dequeue_from_owner() wins and manages to remove the + * task from "owner->blocked_head" first. + */ + scoped_guard(raw_spinlock, &owner->blocked_lock) { + /* + * Chain-wakeup has raced to queue the task. + * Try again with __task_rq_lock() held. + */ + if (READ_ONCE(p->on_rq)) + return false; + + /* + * Something changes in the link. Try again with + * __task_rq_lock() held via proxy_needs_return() + * where both p->on_rq and p->is_linked is stable. + */ + if (!READ_ONCE(p->is_linked) || owner != READ_ONCE(p->sleeping_owner)) + return false; + + __proxy_dequeue_from_owner(p); + } + + return true; +} + +#else /* !CONFIG_SCHED_PROXY_EXEC */ + +static bool proxy_try_dequeue_from_owner(struct task_struct *p) +{ + return false; } #endif /* CONFIG_SCHED_PROXY_EXEC */ @@ -3799,6 +3889,40 @@ void __proxy_block_task(struct task_struct *p) if (!sched_proxy_exec()) return; + if (unlikely(p->is_linked && p->blocked_cpu != task_cpu(p))) { + WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); + + /* Move iowait signal to new CPU. */ + if (p->in_iowait) { + atomic_dec(&task_rq(p)->nr_iowait); + atomic_inc(&cpu_rq(p->blocked_cpu)->nr_iowait); + } + + /* + * Moved the linked task to p->blocked_cpu which is now + * resposnible for chain wakeup. TASK_ON_RQ_MIGRATING + * above stalls task_rq_lock() until p->on_rq is + * transitioned to 0 (See ___task_rq_lock()). + */ + proxy_set_task_cpu(p, p->blocked_cpu); + + /* + * Preserve p->blocked_cpu for any potential future + * additions to p->blocked_head. The chain should be + * re-directed to the same "blocked_cpu". + * + * XXX: Technically, it is possible to keep the + * "blocked_cpu" for this sub-chain different + * compared to our ancestors but the chain-wakeup + * would need to juggle multiple locks then to + * ensure it doesn't race with concurrent wakeups. + * Matching "blocked_cpu" requires only a single + * rq_lock which is simpler. + */ + return; + } + /* * This is the CPU where the __active_blocked_donor() slowpath will * resume activation of blocked donors queued on this task. @@ -3819,6 +3943,25 @@ void __proxy_block_task(struct task_struct *p) */ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p) { + if (!sched_proxy_exec()) + return false; + + /* See try_to_wake_up() for ordering guarantees. */ + if (unlikely(p->is_linked)) { + /* + * Task is queued on an owner's blocked_node. + * + * Dequeue from owner's blocked_node before going + * further. Do not take the fast-path since this task is + * fully blocked and it is necessary to clear + * p->blocked_on indicator before waking up. + */ + WARN_ON_ONCE(p->on_rq); + proxy_dequeue_from_owner(p); + clear_task_blocked_on(p, NULL); + return true; + } + /* * Typically per __set_task_cpu(), task_cpu(p) == p->wake_cpu. * @@ -3922,10 +4065,34 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, */ static int ttwu_runnable(struct task_struct *p, int wake_flags) { + struct rq *rq; + + if (sched_proxy_exec() && READ_ONCE(p->is_linked)) { + /* + * If p->is_linked put us on the ttwu_runnable() path, + * try to de-link the blocked donor outside the + * __task_rq_lock() first. + * + * If the task state changed, a wakeup raced with + * ttwu_runnable() (see proxy_try_dequeue_from_owner() + * for interleaving). Try again with rq_lock held where + * state is guaranteed to be stable. + */ + if (proxy_try_dequeue_from_owner(p)) { + WARN_ON_ONCE(READ_ONCE(p->on_rq)); + clear_task_blocked_on(p, NULL); + return 0; + } + } + ACQUIRE(__task_rq_lock, guard)(p); - struct rq *rq = guard.rq; + rq = guard.rq; - if (!task_on_rq_queued(p)) + /* + * schedule() managed to block the task and there is + * no unlinking needed. Continue with wakeup. + */ + if (!p->needs_rq_sync) return 0; update_rq_clock(rq); @@ -6972,6 +7139,11 @@ static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owner, WARN_ON(!p->on_rq); WARN_ON(p->sleeping_owner); + /* + * See __proxy_block_task() on how p->blocked_cpu is used + * to re-direct the task to a handler CPU when + */ + WRITE_ONCE(p->blocked_cpu, READ_ONCE(owner->blocked_cpu)); WRITE_ONCE(p->sleeping_owner, owner); list_add(&p->blocked_node, &owner->blocked_head); WRITE_ONCE(p->is_linked, 1); -- 2.34.1