From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012017.outbound.protection.outlook.com [40.93.195.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 642773B0AC4 for ; Wed, 26 Aug 2026 06:34:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.17 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726062; cv=fail; b=Pfl/Yd4wz6ew6TiPeP47QEhIyMVMPBz5NIWKkfRL3suZHJmduhhd038FBych2ykuIUBkmPYYPZO6dMu146gojcgbd6HBxV7JgtSB8nWLlGe9xgitKfG4qg5RU20+da7gsyPhi+mtLmRrBrXEprSlLgJiovdfCy595+wf1j+FnoA= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726062; c=relaxed/simple; bh=Ur/5m/baFfECEGg6Zrpl0SQ1HdH6G0mLG8RH93eqHRg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=t+OUSfqnULipJX2Iyz/GZhXFaO95A1FWxrRmvQgZExoCDS0LFYxwRbFzme/6EFfv5JH75tpLwtcjB212iF1Xe4tS6ZH2BsTgGo6afCNrTh7dgIcuK0kfNWidFhKiDEgk9PSBcY1XoVT6pxxbLhfM3dwL2RBFudE1pA7KVY2m+pk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=unV0q6Bt; arc=fail smtp.client-ip=40.93.195.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="unV0q6Bt" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cf8K3TWrXbspYz+keRcjiTDdgq9ySqRch3qsKc+CUxkYAzbp+wS/xDmCpmra428TlBEXCZ6p2HXjNo9Ov8a2dDvv23kAcWCJNTERoBzYofQJ8tL+DmzUfN4VOmsMP1LNV1NfFj2DHcrKUbFEIBAy3PEX8ya5CQtxOji6O3l/mDjsdKwxj+X56eFu0rDMmK0JN4Qcez1jK2tSqYGkYu9lorQoMHaX1ZDyFjXnnoNOeKx/fSgCqZEoKOUXPfSfnEE+ekbOHso2VzJXNIvbm2mmxmBkbVilspg0isfu90Csu3oflWYCllQO2og7JFRM702KUIIRa1QLDunUCovhiFXPxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6KYDFc/+Y7XqWlVIILwkLGo87HLzsHRcJP8pS8Zz2EM=; b=egXkHfzroEqp9GJ6mPtnBGB1NpcZ52f4W8OwECpg9jgBAIqNt+TJMH3A0CjikAtRXRo5jJ/3f4WayRBKsGE2n9UZJl3CFVlamBGd4L+x1AoBIZA1Hh410GBgdpniLtAR1DljPBt6Va5w+AVie9u48Yw/VkzaTcQ4f0x9qeO6ANTaFocCk6GcsTnsREf3zfjUFVOoEGMGRWftS0niQ2PbszdaTEd25OPVyGZUs07QrL8l7vYghS8FA4vE4F0UwToimJ0yM5v00LiUhJYCaTrmyLwXfLuA30KgL7ZtdlFf66dWwEqGtSddab8lNZvkkRonxRh30U40n7VZtJXlZBNEag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6KYDFc/+Y7XqWlVIILwkLGo87HLzsHRcJP8pS8Zz2EM=; b=unV0q6Bt2CXHO6y26UsP8hgDMwQ2c0mSYKdeXPL9vY3jRpP94L3scv7Q2Dfy0e2jb5+UvcnwHOneghEKwxSLDX4DC2TIqV56gKf7Zr+Hc0DbhBzToqhaiy9imxN1sUJkMUkVgQQq1Y8a1idRoNkawn5Hp7S6RNg0bPfe+699ceU= Received: from CY8PR10CA0022.namprd10.prod.outlook.com (2603:10b6:930:4f::23) by DS7PR12MB6358.namprd12.prod.outlook.com (2603:10b6:8:95::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:34:14 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2603:10b6:930:4f:cafe::62) by CY8PR10CA0022.outlook.office365.com (2603:10b6:930:4f::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.10 via Frontend Transport; Wed, 26 Aug 2026 06:34:14 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:34:14 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:34:09 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 14/16] sched/core: Introduce chain-wakeup to activate blocked donors Date: Wed, 26 Aug 2026 06:28:58 +0000 Message-ID: <20260826062901.2137-15-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|DS7PR12MB6358:EE_ X-MS-Office365-Filtering-Correlation-Id: 24088f05-6d7e-4c93-a34f-08df033c0c1e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|23010399003|82310400026|1800799024|36860700016|7416014|10067099003|56012099006|22082099003|18002099003|11063799006|921020; X-Microsoft-Antispam-Message-Info: rR0j1mQJDU6I1TxxGa0Cn/aIPfiR+XpHS9N1e1thIOKPZSWO26leMFTYLnarvisC/NI06eOMlRxS/XnBp0d0JAmx/0KFjKKvlIIM+xMLKYTqMo3g6g+9fZ8xcBZ94uEEcHwSnPmvTcMNnPjU0nhOd+qXEf2I1zkrPviL9DcHflB70ciFjXwPLt/wq0ATKv/N/upjIW6pydKOYgRXinEm8USLQgKED59s4CEiKShV98MZiTwx2kwDNsu3WkgiL0jKItMrwIsORs4dMs3tSz8+wfZjPVgmcH0KmpWKWDZIzZ3sLzRi+Sc0HLaBtMWD1s8LtNCK7lhtIsT0/w7wZCiGKEZCrVjPIfxKpsEnze/DL+pOW/rsufzkDTOMRlkZoFDjWeAV3FNzE/GyWSbsHL1foYe5fOtBdLrR3Vxr8q3c6S8uR7/xWo6AgthSeLbgY1OBO7MSLme6wL9xmhK11YktJ++HtKdeEFfyMYQPFeFQamnLYv8z78Ifh1+toMgrn9bPErMTVag8EMkX1lnTz0daB+TRMnKbJtSSJMXmQ5ZOZbFyzGfSEttCte2i1SYWfKwwVX+DkBeEMgiwNPG+YCK/GuO2gDe6hd+U6Sk7wgtum+jAelUnHyTb4uKnzUury18E4N/AVi6rrKFe2rsVWZjbmPgO9n+WUmquj5ymYApFMtRRGK8QDpLS17CHW3GpwbvtRCZ33ZI1XH9qMGqYfx0wT0rk+MnSvvVEDfx78/XrAsTBOqIlLfB7yhY6jw+jq3GG X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(23010399003)(82310400026)(1800799024)(36860700016)(7416014)(10067099003)(56012099006)(22082099003)(18002099003)(11063799006)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: W+sew0AFRlPBKTmGXQgHU753rf86fakJ/0cpQHVJmiAuR0pKfIC+AFwwG74IkWnb6lxjdoWCJRg012HhWPWDzwMogg/8kH5gvv3GUKCWgbI79gEYHHLJs3Nj5p2PrjrFqiRfHw8IuT7J898ISX1iqSVIAEJIoucJqYmvfQX33C+qPHHPb/OE0GTUgyuehpguPxohgJzKITlJOcLOtYvSAE9tDVlou/8lxEj8AzM++FoClLEbL7TWLFT9mD6KRvdEpGrmKXXd7JlrpQ0zRrYWvlQ986GZXvO3LRXoVIlpOh9BUxtOU//+RLLX/OL0gMfMBJpZoJQrxo+EGOzOUNrdomnEDphxAbg6JIkABlN2wo5Uh7D4NJ1u+HQmzf7gk70F4zwmM5+l42FRGQXSpkrD7C1RBn5mGV7hlFxjptf2mlulkrz/Pelvtwf87PUzggJ+ X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:34:14.5266 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 24088f05-6d7e-4c93-a34f-08df033c0c1e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6358 Activating blocked task with sched_proxy_exec() needed grabbing p->blocked_lock for all wakeups to prevent missing additions to blocked_head during wakeup. With proxy_enqueue_on_owner() converted to plug the race between enqueue on owner and owner's wakeup, simplify the wakeup path with the following three step process: activate_blocked_task(owner) /* Part 1: Enqueue race */ enqueue_on_owner(owner, p) owner->on_rq = TASK_ON_RQ_MIGRATING; list_add(&p->blocked_node, &owner->blocked_head) smp_mb(); smp_mb(); if (list_empty(&owner->blocked_head)) if (owner->on_rq) /* Raced */ /* list_del_init(&p->blocked_node) * Fast-path: Future additions will return; * see owner->on_rq != 0 and dequeue /* * themself. * Owner has to observe !list_empty() after */ * this point so it is safe to block. __activate_task(p) */ block_task(p); /* END: Part 1 */ enqueue_on_owner() holds mutex->wait_lock and "owner" needs to grab the same wait_lock during unlock so "owner" cannot disappear during enqueue_on_owner(). /* Part 2: Prepare migration chain */ rq_unlock(rq); LIST_HEAD(migration_list) LIST_HEAD(wakeup_list) try_to_wake_up() /* Sees proxy_task_is_linked(p) */ ttwu_runnable(donor) rq_lock(rq_of(owner->blocked_cpu)) task_rq_lock(donor) /* Same rq */ do { list_for_each_entry(donor, &owner->blocked_head) if (task_is_linked(donor) donor->on_rq = TASK_ON_RQ_MIGRATING /* Part 1 */ list_del_init(&p->blocked_node) if (list_empty(&owner->blocked_head) activate_blocked_task(donor) list_add(&doner, &migration_list) /* Same as left */ else list_add(&donor, &wakeup_list) list_add(&owner, &wakeup_list) owner = list_next_entry(&migration_list) } while (!list_empty(&migration_list)) rq_unlock(rq_of(owner->blocked_cpu)) /* END: Part 2 */ Since owner->blocked_cpu matches with task_cpu() of the entire chain (including tasks queued on the blocked donors, and task queued on them, and so on), grabbing "rq_of(owner->blocked_cpu)" is enough to stall any concurrent wakeups for any of the linked tasks. If try_to_wake_up() wins, donor will remove itself from the chain and continue its wakeup as if it is a blocked task waking up. If activate_blocked_task() wins, it will set all the donors to TASK_ON_RQ_MIGRATING, repeating Part 1, to prevent other tasks from queueing on it form then on and iterates its "blocked_head". This is done in a breadth-first fashion until full chain is set to TASK_ON_RQ_MIGRATING. "owner->blocked_head" traversal and manipulations are still guarded by "owner->blocked_lock" which has not been depicted here. /* Part 3: Wakeup chain */ rq_lock(rq); __activate_task(p); /* Activate top owner first */ /* Activate the entire chain next */ list_for_each_entry(donor, &owner->blocked_head) list_del_init(&donor->blocked_node) __activate_task(donor) /* END: Part 3 */ The wakeup steps skipped for donors is performormed during the chain wakeup. Advantages of this method: o Does not involve juggling p->pi_lock, rq_lock, owner->blocked_lock in addition to checking task state. Task on chain is fully blocked and only one rq_lock + onwer_blocked lock can hold off ttwu_runnable() from waking them. o No need to borrow reference to "owner" before queuing. The mechanism ensures p->blocked_owner remains valid as long as is task is on chain. Disadvantages: o Since chain wakeups funnel through a single CPU, it is possible for increased lock contention however large chain wakeups and concurrent wakeups of donors for reasons other than an unlock is rare. Co-developed-by: John Stultz Signed-off-by: John Stultz Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 183 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 181 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e6abcea97683..3153ad007447 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2247,6 +2247,8 @@ void activate_task(struct rq *rq, struct task_struct *p, int en_flags) #ifdef CONFIG_SCHED_PROXY_EXEC +static inline void proxy_set_task_cpu(struct task_struct *p, int cpu); + static void __proxy_dequeue_from_owner(struct task_struct *p) { list_del_init(&p->blocked_node); @@ -2337,6 +2339,183 @@ static bool proxy_try_dequeue_from_owner(struct task_struct *p) return true; } +static void +proxy_activate_blocked_task(struct rq *rq, struct task_struct *p, int en_flags) +{ + int iowait_count = 0, load_contrib_count = 0; + struct task_struct *donor, *owner, *tmp; + int blocked_cpu, this_cpu = cpu_of(rq); + LIST_HEAD(migration_head); + LIST_HEAD(wakeup_head); + struct rq *blocked_rq; + bool needs_migration; + + WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); + + /* + * Pairs against smp_mb() in proxy_enqueue_on_owner() which + * orders ownwer->on_rq state against the blocked_head addition. + */ + smp_mb(); + + /* + * Fast-path: Tasks blocking on us will see p->on_rq updated and + * will bail out natturally in proxy_enqueue_on_owner(). + * + * proxy_enqueue_on_owner() holds the wait_lock to prevent owner + * from running and disappearing before the transient task can + * observe p->on_rq change and dequeue itself. + */ + if (list_empty(&p->blocked_head)) { + __activate_task(rq, p, en_flags); + return; + } + + /* + * Slow-path: Since we need to wakeup tasks (and task queued on + * them, and task queued on them, and ... you get the gist) we + * need to grab some locks. + * + * Lucky for us, we have simplified this via p->blocked_cpu + * which is the task_cpu() for the entire chain. Wakeup is + * broken into two parts. + * + * Part1: Under the rq_lock + blocked lock, mark all tasks on + * the chain as TASK_ON_RQ_MIGRATING in breadth-first manner. + * + * Part2: Active all the tasks in bulk. + */ + + /* Part 1: Prepare the blocked donor chain. */ + blocked_cpu = p->blocked_cpu; + blocked_rq = cpu_rq(blocked_cpu); + + /* + * Blocked chain is linked to a different rq. Drop the rq_lock + * and migrate the chain over before wkaing it up here. + */ + needs_migration = blocked_rq != rq; + if (needs_migration) { + raw_spin_rq_unlock(rq); + raw_spin_rq_lock(blocked_rq); + } + + owner = p; + + do { + guard(raw_spinlock)(&owner->blocked_lock); + + /* Iterate the tasks blocked on this owner. */ + list_for_each_entry_safe(donor, tmp, &owner->blocked_head, blocked_node) { + /* Everything hinges on this assumption. */ + WARN_ON_ONCE(task_cpu(donor) != blocked_cpu); + WARN_ON_ONCE(donor->se.sched_delayed); + + WRITE_ONCE(donor->on_rq, TASK_ON_RQ_MIGRATING); + ASSERT_EXCLUSIVE_WRITER(donor->on_rq); + + if (donor->in_iowait) { + delayacct_blkio_end(donor); + iowait_count++; + } + + if (donor->sched_contributes_to_load) + load_contrib_count++; + + /* + * Pairs against the smp_mb() in proxy_enqueue_on_owner(). + * See the comment at the beginning of the function. + */ + smp_mb(); + + /* + * If the donor has more task queued on it, + * add it to the migration list else directly + * queue it onto the wakeup list. + */ + if (!list_empty(&donor->blocked_head)) { + list_move_tail(&donor->blocked_node, &migration_head); + } else { + list_move_tail(&donor->blocked_node, &wakeup_head); + } + } + + owner = list_first_entry_or_null(&migration_head, + typeof(*owner), + blocked_node); + + /* + * Move the owner to the wakeup list before preparing + * the sub-chain queued on it. + */ + if (owner) + list_move_tail(&owner->blocked_node, &wakeup_head); + + } while (owner); + + /* Part 2: Bulk wakeup */ + + atomic_sub(iowait_count, &blocked_rq->nr_iowait); + + if (needs_migration) { + raw_spin_rq_unlock(blocked_rq); + raw_spin_rq_lock(rq); + + update_rq_clock(rq); + en_flags |= ENQUEUE_NOCLOCK; + } + + rq->nr_uninterruptible -= load_contrib_count; + + if (!(en_flags & ENQUEUE_NOCLOCK)) { + update_rq_clock(rq); + en_flags |= ENQUEUE_NOCLOCK; + } + + /* Activate the top owner first. */ + __activate_task(rq, p, en_flags); + + /* + * Add ENQUEUE_MIGRATED for all tasks on the chain + * since they are all sched_migrated_on_blocking + * and have skipped ->migrate_task_rq() callback. + */ + en_flags |= ENQUEUE_MIGRATING; + + list_for_each_entry_safe(donor, tmp, &wakeup_head, blocked_node) { + __proxy_dequeue_from_owner(donor); + + /* + * Correct the task_cpu() before activating. + * + * Since chain preparation already adds enough barrier + * after storing p->on_rq = MIGRATING, no additional + * barriers are required after modifying p->is_linked + * via __proxy_dequeue_from_owner() above. + */ + if (needs_migration) + proxy_set_task_cpu(donor, this_cpu); + + __activate_task(rq, donor, en_flags); + wakeup_preempt(rq, donor, en_flags); + /* + * Blocked tasks do not add push callbacks and it is + * safe to skip calling ->task_woken() here. + */ + } +} + +static void activate_blocked_task(struct rq *rq, struct task_struct *p, int en_flags) +{ + if (!sched_proxy_exec()) { + __activate_task(rq, p, en_flags); + return; + } + + proxy_activate_blocked_task(rq, p, en_flags); +} + #else /* !CONFIG_SCHED_PROXY_EXEC */ static bool proxy_try_dequeue_from_owner(struct task_struct *p) @@ -2344,13 +2523,13 @@ static bool proxy_try_dequeue_from_owner(struct task_struct *p) return false; } -#endif /* CONFIG_SCHED_PROXY_EXEC */ - static void activate_blocked_task(struct rq *rq, struct task_struct *p, int en_flags) { __activate_task(rq, p, en_flags); } +#endif /* CONFIG_SCHED_PROXY_EXEC */ + void deactivate_task(struct rq *rq, struct task_struct *p, int flags) { WARN_ON_ONCE(flags & DEQUEUE_SLEEP); -- 2.34.1