From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012004.outbound.protection.outlook.com [52.101.48.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 965003AB26B for ; Fri, 17 Jul 2026 11:17:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.4 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287051; cv=fail; b=eROoffsO0XvEg/mUBdSMz+WnSgqLlV2jexWk5rxY7yBoOmFhNwc3CLugRgfCH/L0PbomgCbQimVnJbeNxssoyhsSh1iwz0S9HgcRayM96IGj2JWOzK0/8r+cSKH7D68E/V01pRKdBNbX9bn1WIvZ0a750X8dJ/x+hQdF72iP7BM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287051; c=relaxed/simple; bh=hSFS3eBjoTASMIPuZcq6mS9DXvgZMQIGpZtCV0G16BI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pdeSoZOeEuByrvoCmCef/YiF8fWHHAvWcEgNvQm6Xr7nM2htPiG8xv0MjT75dYw22OKKIFS8laepEYBTv4crm+8lfFA3KOGsstQLdS3w3eZbC6+v02Q2iF6UJF44pkVyTUtDYzaIMHpYd5E42hgx5mDORvpYfSZcPbNLscoaNkw= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=vgxt7rV0; arc=fail smtp.client-ip=52.101.48.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="vgxt7rV0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ie+VdbzCabXykwC4pTjMFdZ+SJaq7voI//PRsD83zZNl5pn9430Mm2+Pca0A0ixT3Cb2k1ZMqcm9GkoabejUZKfVzvEFOu+sXDvM1KPR626CEGgPkht1lUnGoK4gHLcCdXR3RW4Wy4OEQJg/Jg70Ko1ieONwldoDprrrH5ivfetxAAc22eqYYRZtQJQBq00l/u6vLcGfglzoGaYe73l8djv+OLMbyuRUjGzvZA0Jt5Cwtkq46UPGcobJy+nsV1A3YzJTNxXjC9/++O1KV939CtslrzR8oL9f5iDrJ+gI3/iZZj3x3nyPo0V0kQer2Q9JWeJGOnqesyrtVTURGDgvtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sGUydlTqwMgDD3KWMmE0rls83fUeJnK1mGwwzo5Zj7Q=; b=KekiqcM1/uApc9HHtKxkk6dAteK+kC+p4CEzFdlkqJDVCOBJbRPl3nCFoZJbPUpWqI1KRV6w3kEhQpwmCj3LwFUeEEH/fWd1NBvoOINHjRfOZp2oGYFncslKN8Iyod4XErkN+eIWOPj5IVG71xIVU2JfQuKWbGhiXsg/X1mhD798KOLigH8+CqtrUxyu5Y3DL+sFw33zNXYYcKeQmDjoIfjFByNsLplrswz8a9iBKyKByVV2CPm6wOt6g7jKB8JBbBj2hJq1EEQMO4gphoAwlQyUcWoFqOyD+IS/WvfFrDO1VUojizuPuSyeJl5RdxRYNI3l/whp8YRi1Z9UfGDJVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sGUydlTqwMgDD3KWMmE0rls83fUeJnK1mGwwzo5Zj7Q=; b=vgxt7rV0NXZvuaKGTWj1d0CKxPoRgHALa6igpEjg4rI+v/+6KAZEgOB6iX9h3x3GuG3g/6JCTC3tv/C1PI7TllSL67W4ukJjl0mwZSDSZmgGMXu1Tx7UQKebwwKeGOUBduFvsPhION+/u0ApO92go7cXkXwc2qC1GSNMecfO/1k= Received: from BN0PR10CA0009.namprd10.prod.outlook.com (2603:10b6:408:143::6) by PH7PR12MB6634.namprd12.prod.outlook.com (2603:10b6:510:211::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.13; Fri, 17 Jul 2026 11:17:25 +0000 Received: from BN2PEPF00004FBE.namprd04.prod.outlook.com (2603:10b6:408:143:cafe::d) by BN0PR10CA0009.outlook.office365.com (2603:10b6:408:143::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.223.13 via Frontend Transport; Fri, 17 Jul 2026 11:17:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN2PEPF00004FBE.mail.protection.outlook.com (10.167.243.184) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.5 via Frontend Transport; Fri, 17 Jul 2026 11:17:23 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Fri, 17 Jul 2026 06:17:16 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , John Stultz , CC: Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Joel Fernandes , Qais Yousef , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , "Daniel Lezcano" , Suleiman Souhlal , kuyo chang , hupu , Vasily Gorbik , Subject: [RFC PATCH 3/3] sched/core: Swap to lock owner for core-wide pick on core-cookie mismatch Date: Fri, 17 Jul 2026 11:16:01 +0000 Message-ID: <20260717111601.6017-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717111601.6017-1-kprateek.nayak@amd.com> References: <20260717111601.6017-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBE:EE_|PH7PR12MB6634:EE_ X-MS-Office365-Filtering-Correlation-Id: 533a5bea-2c91-43da-e808-08dee3f4f9fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|7416014|376014|23010399003|1800799024|82310400026|10067099003|56012099006|4143699003|11063799006|5023799004|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: 2ec4IuBjndIPSJ/tTtnb9765dYCAy2Oh6JWkE2LJ7cipqu12yKJY0xiNNTzf/xNNydkREkH/9FwqnhAYNQuF3JOQQiCsW6yUgYlF8CK1gdkNxakHC6Vx2eU1MfGPr3l4Iqitz15rh169qmA3cYNjj0Mpbrc3qZSuETgpJJeyVoAFqLNW7Yp4rNIpzbpjpNko27cKVudLLmID4sJWQQatgcXSPYb586oSvclJLfOTboglXsR+2sJdwcERte8f85ChlPXuwSDGTucmO/jixUX7KdsbRKL+bfnEhOjI4WSp70CEMSMq8DOsVTKZaEy5k9CMCK3QasiUmkXXlxd6lggAv25Gqe+6CqFRATEUhiAD4xcxTfRx9TRyBw5AOYnDAtUUunCv++eZJ9IOOviNlkHcpGtcBmbrySwhjcIpDjwuFpSamJh+NEM6Bt/3urHO3/rg7QOa5rBVpAyuJO8lkkKAtw5b+YARRKtzNkl1rCqmEcEOGyS2fFR/1fxiSEl/bNVnk91vf394nYxJpXfPzVHvqJQE3vg7pJHARRQWUxlsGimssjJPJW5WamaybiCp64XvDtEvbeRVc609+UHpVtDbqcYCA2GhsRLncHEbL0v1yS51dLdiYBagJMgRM7cc7gISeODhi9dcMWa/cEEnc6xiNlCRRH5nAZ/CbPjdNKIb8n2j/IEkpHqeWIKowqH2a6C4NVpRSzeTBtl68D2dy5xadg== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(7416014)(376014)(23010399003)(1800799024)(82310400026)(10067099003)(56012099006)(4143699003)(11063799006)(5023799004)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 3w+wX2ZFQ3+BvLyTua482NGor42faLJlB/XDJ7e9xu2VINOeULYYY7epM3TZxouU7DfzgAdXM0fLoWaf1z2yyXVRqHZqkxu8x7Ea0UU/kJdzXc0mFzioyYfd0Kf/+xF7/KvAipREklYV+JsJl/u98JVVFFI0j3wE0SEtK593PjZPDZTp8j9XLi6CWxAG6aaVM94R6o75Wd2ECmnImplWamwxhoxg+mVjfiu0nMIHIQ3eQWz2ZHGbOQLj3YNvKE2JagceWHT6J6DHEJPSACkRX5UhgWevqjg/Z+1N6w+20POMKTq9zSPkUzW9twJAaDAZOrcTALOyYecsjQB1d1w7SkaUT9urL1PGOrX7TNoaMZ2KmiBbaeQjp/foKkO4m7fVcV2Wt5yPz8V0Q4Af/kslw2rH9oZ4HrFRL6myJzYj+MuvB+hnQUYbNtIwvXdM6Hc7 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jul 2026 11:17:23.8552 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 533a5bea-2c91-43da-e808-08dee3f4f9fc X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBE.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6634 Proxy donor can have a different core cookie compared to the lock owner. Simply running a lock owner with a mismatched core-cookie in not an option as this breaks the core scheduling guarantees. To prevent a cookie mismatch, use the "rq->core_pick_leader" indicator to know if the CPU with the blocked donor is the one with the highest priority task. If the rq contains the highest priority task, it is that rq's pick which dictates the core-cookie. Once the find_proxy_task() chain stabilizes on the rq (find_proxy_task() returns the valid lock owner queued on the same rq), set a second "rq->core_pick_blocked_donor" and retry the pick. The flattened flow looks as follows: donor = pick_next_task() rq_set_donor(rq, donor) if (donor->is_blocked) next = find_proxy_task() /* * next has a different cookie compared to doner and * rq->core_pick_leader is set. */ sched_core_swap_task() rq->core_pick_blocked_donor = 1; /* rq->donor was the last pick */ return RETRY_TASK; pick_next_task() donor = rq->donor; owner = NULL; if (rq->core_pick_blocked_donor) owner = find_proxy_task(donor); goto restart_multi; /* * Finish the core-pick and validate that this rq * is still the pick leaader. */ if (rq->core_pick_blocked_donor && rq->core_pick_leader && rq->core_pick == donor) rq->core_pick = owner; /* Swapped donor with owner. */ rq->core->coore_cookie = rq->core_pick->cookie; /* owner cookie */ /* Continue with the rest */ return rq->core_pick; /* Owner */ if (sched_core_retain_donor(rq)) rq->sched_core_proxy_pick = 0 /* Consume */ Since the owner is not blocked, it continues to run and rq->donor is preserved while the "rq->core_pick_blocked_donor" gets consumed. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 81 +++++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 2 files changed, 74 insertions(+), 8 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ed11410f1416..ab3b3104cd86 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6598,7 +6598,8 @@ __pick_next_task(struct rq *rq, struct rq_flags *rf) BUG(); /* The idle class should always have a runnable task. */ } -static void proxy_deactivate(struct rq *rq, struct task_struct *donor); +static struct task_struct * +find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf); #ifdef CONFIG_SCHED_CORE static inline bool is_task_rq_idle(struct task_struct *t) @@ -6643,10 +6644,16 @@ extern void task_vruntime_update(struct rq *rq, struct task_struct *p, bool in_f static void queue_core_balance(struct rq *rq); +static __always_inline bool sched_core_proxy_pick(struct rq *rq) +{ + return sched_proxy_exec() && unlikely(rq->core_pick_blocked_donor); +} + static struct task_struct * pick_next_task(struct rq *rq, struct rq_flags *rf) __must_hold(__rq_lockp(rq)) { + struct task_struct *owner = NULL, *donor = rq->donor; struct task_struct *next, *p, *max; const struct cpumask *smt_mask; bool fi_before = false; @@ -6728,6 +6735,24 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) */ rq->core->core_task_seq++; + /* Last core_pick resolved to a blocked_donor! */ + if (sched_core_proxy_pick(rq)) { + WARN_ON_ONCE(!donor->is_blocked); + + donor->blocked_donor = NULL; + owner = find_proxy_task(rq, donor, rf); + if (owner && owner != rq->idle) + goto restart_multi; + /* + * Something changed in the proxy chain! + * Retry pick like normal. Increment core_task_seq + * since the core-wide lock might have been dropped + * during proxy-migration in find_proxy_task(). + */ + rq->core_pick_blocked_donor = false; + rq->core->core_task_seq++; + } + /* * Optimize for common case where this CPU has no cookies * and there are no cookied tasks running on siblings. @@ -6786,6 +6811,21 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) } rq_max->core_pick_leader = true; + + if (sched_core_proxy_pick(rq)) { + WARN_ON_ONCE(!owner); + + /* + * Core-wide pick resolved to the same state as last time. + * Swap the donor with the lock owner and continue the + * rest of the pick sequence. + */ + if (rq->core_pick_leader && rq->core_pick == donor) { + rq_max->core_pick = owner; + max = owner; + } + } + cookie = rq->core->core_cookie = max->core_cookie; /* @@ -6883,9 +6923,16 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) } out_set_next: - put_prev_set_next_task(rq, rq->donor, next); - if (rq->core->core_forceidle_count && next == rq->idle) - queue_core_balance(rq); + /* + * If this is a redo for setting the owner's core-cookie, + * put_prev_set_next_task() was already done during the + * last pick and there is nothing to do now. + */ + if (!sched_core_proxy_pick(rq)) { + put_prev_set_next_task(rq, rq->donor, next); + if (rq->core->core_forceidle_count && next == rq->idle) + queue_core_balance(rq); + } return next; } @@ -7114,11 +7161,21 @@ sched_core_swap_pick(struct rq *rq, struct task_struct *next) return rq->idle; } - clear_task_blocked_on(rq->donor, NULL); - proxy_deactivate(rq, rq->donor); + rq->core_pick_blocked_donor = true; return RETRY_TASK; } +static bool sched_core_retain_donor(struct rq *rq) +{ + bool retain = rq->core_pick_blocked_donor; + + if (!sched_core_enabled(rq)) + return false; + + rq->core_pick_blocked_donor = false; + return retain; +} + #else /* !CONFIG_SCHED_CORE: */ static inline void sched_core_cpu_starting(unsigned int cpu) {} @@ -7144,6 +7201,11 @@ sched_core_swap_pick(struct rq *rq, struct task_struct *next) return next; } +static bool sched_core_retain_donor(struct rq *rq) +{ + return false; +} + #endif /* !CONFIG_SCHED_CORE */ /* @@ -7694,8 +7756,11 @@ static void __sched notrace __schedule(int sched_mode) if (sched_proxy_exec()) { struct task_struct *prev_donor = rq->donor; - rq_set_donor(rq, next); - next->blocked_donor = NULL; + if (!sched_core_retain_donor(rq)) { + rq_set_donor(rq, next); + next->blocked_donor = NULL; + } + if (unlikely(next->is_blocked)) { next = find_proxy_task(rq, next, &rf); if (!next) { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b80a5eb384f0..541b2ae673d8 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1352,6 +1352,7 @@ struct rq { /* private state for sched_proxy_exec() */ bool core_pick_leader; + bool core_pick_blocked_donor; /* shared state -- careful with sched_core_cpu_deactivate() */ unsigned int core_task_seq; -- 2.43.0