From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CH1PR05CU001.outbound.protection.outlook.com (mail-northcentralusazon11010014.outbound.protection.outlook.com [52.101.193.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B0B52E736F for ; Wed, 26 Aug 2026 06:33:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.193.14 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726006; cv=fail; b=M0FDgif48xxnqRXxBfISSwqv3zedJ/JyZTM0mji1fzBmIYgX+mjg9gaITZPjrsPHP7PAwSlJtWuGTvjQ+28Dltsb1Sp3A+D8gYxbgu6OJu1i+sTEBReSKC62rJg3sTNKZxXZzq8GDDvU93wlsXjN+ZcPN1T3JiFwXeSwtohtJBM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726006; c=relaxed/simple; bh=ic3u5AlAYbG6zDX6bQb3Ud6gvHT2cLi+mM/1nqNTmRk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fXZaC9FbwBKKkHiUTl/Y9ITDVzuMjbcb1ZiEz1YQRnjQsow61c8AcAAGcqX4lxME6oTzhmz+fROdHiPGT/KZIboxlrgAr2ibYNj9RmRuxlzetS1bk05sElSCOt635J3sWX2fxf7Fv3lo63e1z59jwn3yvjC+kPl2Ynk78ZzjQ5M= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RoQ5rlsV; arc=fail smtp.client-ip=52.101.193.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RoQ5rlsV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rMG1JkaImVS7I503260HTQlPgy+pUPFXctM+AcTUw0jFASMNmWNEAaA1CpMT6xe+vm4JTn19xhh316NBjDFAhAQDkbukgI66s9V5w+89/q4D8VueZTwdIKPLwKmO7+ZXz1JuvmbGx1La74C8uBtK1T6kEG/w0Eua877v0Q7ac+JwhgZNWx6gdwMdnC9T+Q2MDkaUtbAEgRF1X9oz6dgglr1r7yyjtvR6KDWx8OGmpZhYF3Q2BL8t2ZeHMLq13rTXwYKmAhLZ2ZJeZIDIpUMsUZ0dw35v6bxFs1XYAa3k8CvWrqU89RjJt4Of2xtz+bZGxZMZN1/fGn/Y88Swo2O4hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9KiYYUj56CZcaGJ34n2xTLli85bT5S9KrcPISUv9tZo=; b=cjlmflauZgjPot2xCAwnXJPNBTZfUzROLtCMnGjg7OPSLOVyHsOymWjCAE12pQk1XsLEiwMERwiHdhByRkXsElqso+a5v5eE5iRDrBd3ShWIUkayeuHxa7IzARLK9k9V6r30PHOIuG9eOmVi7nvc/KMVmVLsAs9Jg5nc66F8ncuNNYyqmrBRsTM0DHYMNuyiVPZei6hnI3Zu/QoK1SCEbMNtrpFO+7J72DmhSahHmyz4AxIdRbzzrkS8yz1KzfG562nYX8xcMENPt4CBMnF5rBONDmyCAh7ggUPLoonpkKVG7xZ1zAryk59PHg282iJpXbytx/FxVvMUSV3+GDbplQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9KiYYUj56CZcaGJ34n2xTLli85bT5S9KrcPISUv9tZo=; b=RoQ5rlsVb5ymcRj5SVTLpAVs4oDe9D3GT32Yt7YD6pqslPgF5P8qs6poKwURKYjMQR54jGAabCJt29BWZcpI0Erg2XpQA82VwZT6tKNLqwTFWmEwVE8DB5y58VGs33SMX9/EplXXblFL6KjWklmjOxIQE4uk8cT/cvWChm2drvM= Received: from DM6PR05CA0058.namprd05.prod.outlook.com (2603:10b6:5:335::27) by BN7PPF9C6E5285F.namprd12.prod.outlook.com (2603:10b6:40f:fc02::6db) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.6; Wed, 26 Aug 2026 06:33:20 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:5:335:cafe::af) by DM6PR05CA0058.outlook.office365.com (2603:10b6:5:335::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.6 via Frontend Transport; Wed, 26 Aug 2026 06:33:20 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:33:20 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:33:15 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 11/16] sched/core: Prepare to inspect ->is_linked alongside ->on_rq during wakeup Date: Wed, 26 Aug 2026 06:28:55 +0000 Message-ID: <20260826062901.2137-12-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|BN7PPF9C6E5285F:EE_ X-MS-Office365-Filtering-Correlation-Id: a9b5c33c-8da4-447e-ca6f-08df033bebc2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|23010399003|82310400026|36860700016|376014|7416014|56012099006|10067099003|22082099003|18002099003|11063799006|921020; X-Microsoft-Antispam-Message-Info: z7zJH0uAbG4RjMfRvuDKmltAHHj9zYC0/Om97/I2wS0BEdoySYzgp14bRPhVzfCES+V1lbX9M+acJ+FvUFrLZ/yQ/inQNKeWrmu7sqZy525ByOOoImpS3Apffg/3FMgRuVhTeTUU7+7g2JemjCz45b7cHF3zxTYzeMJpJfu98TGoXaWvCgwRMs6ZeCuXpRHF9bsDPoznA19USdFmB6HAg7bAR5VPbilEn00IsBTLlv37dz1pyGcWVz88p52roxoK/xA13TLktdmmPwyf67skIDjACLqof33ErIi93rmKjAp0bKMlSJSPEOVDSvykyyO/3tGTcDwMfGDOen3fjR5hy+TzmSgbdC7JtDtphv3lh4iohTtP9INLE9icftXKm10GfgW0bv0BtV6dr64w+pGou2opAaZn5uTO0cBQLI6Oc0wX6UxsD6VhsnFMjAAOrM0IyH4lI1nnN9YzzKPWpFdUuJSjbchSSyjM9YkgQXaeezoVKDdtSzzkR9Gg+eysZ6xhhSAtZyb2ZdD6ZFnYr6gYnR1D33yQOXT2nV7ULG84OkPPlEsod12YlH1pWsKF//aqu1qdYKopbVZLkBBVVJbyxo0uUf8NlljvkITBrDM6YKa+ON8Th0sYF6G65PIzSJW3pJkyLzxp3Dcs/i1nt41XdMtPjCmH991DGpHBmuWL36MRmqryYLOiZxnXNaIRNspD4gdi0XuHQRqkRLOOuch/BSEBB+hkAwGHWC2Tjyv0Cx7/5vYfmiK6BScm3gs6ZA9b X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(1800799024)(23010399003)(82310400026)(36860700016)(376014)(7416014)(56012099006)(10067099003)(22082099003)(18002099003)(11063799006)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fbhNYbP27XQTEc1nyeb6xjA11h5kfbax6Nsi9izvVAvZZYJAQuJaiAMa3Gpu4lJ0ztOOGAm47udYd8mqlbQXSeACu+b6sv9e5xwjuexgdlirOCw8OywQ52MDd5fKTS849RhhHLZl9LvpWe4y3jokC8YEB2Lw382q+3gYcfUPVGkROapvRCXhNq2dfyroaxjolT1vaBeYf6bUifuChEq35nlmFkST8vljRHzN5GwDkJZf7bdMSAAIGCbpIhmyBQuyc01ZLfhpRBUg1QkvSuoTzMPqpK82dUqAXW+8TQhNHcC2I6n14y7t4EH0gx+hjuw77WnslqEJFUTFP+Vn8M8l42d06EGwoZoGnX8dRifcl3XF5beyGEqlY8YdtMXlcYlcsDqSbb1LYTVSBiH5fKkGWBINOIiI+eJ5CawX8llAchWFpIoKApQ/5qJKSYlsErJa X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:33:20.2375 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a9b5c33c-8da4-447e-ca6f-08df033bebc2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PPF9C6E5285F To handle wakeup of blocked donors queued on owner's blocked_head via ttwu_runnable(), prepare to inspect p->is_blocked in addition to p->on_rq on paths that demand grabbing rq_lock to prevent "->on_rq" transitions. Unionize p->on_rq with p->is_linked to derive a 16-bit p->needs_rq_sync state that can be atomically inspected on all architectures. p->needs_rq_sync check follows the same ordering requirements as p->on_rq check. Subsequent tasks will guarntee ordering of p->is_linked against p->on_rq transitions ensuring that loading p->needs_rq_sync state is enough to cover both and the readers don't falsely observe them to be cleared if read individually during their transition. get_wchan() has been reflowed using guards as a pert of the change to prevent awkward indentation. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 18 +++++++++++++-- kernel/sched/core.c | 53 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 64 insertions(+), 7 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index ec65abf090a4..76de348fb88b 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -852,9 +852,23 @@ struct task_struct { struct alloc_tag *alloc_tag; #endif + union { + struct { + u8 on_rq; + u8 is_linked; + }; + /* + * Allows inspecting on_rq and is_linked + * atomically with a single read outside + * the task_rq_lock(). + * + * See the comment above the load in + * try_to_wake_up() for ordering + * guarantees. + */ + u16 needs_rq_sync; + }; u8 on_cpu; - u8 on_rq; - u8 is_linked; u8 is_blocked; struct __call_single_node wake_entry; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d3cdfcb400d1..957f30632b82 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2171,12 +2171,16 @@ unsigned long get_wchan(struct task_struct *p) return 0; /* Only get wchan if task is blocked and we can keep it that way. */ - raw_spin_lock_irq(&p->pi_lock); + guard(raw_spinlock_irq)(&p->pi_lock); + state = READ_ONCE(p->__state); + if (state == TASK_RUNNING || state == TASK_WAKING) + return 0; + smp_rmb(); /* see try_to_wake_up() */ - if (state != TASK_RUNNING && state != TASK_WAKING && !p->on_rq) + + if (!READ_ONCE(p->needs_rq_sync)) ip = __get_wchan(p); - raw_spin_unlock_irq(&p->pi_lock); return ip; } @@ -4365,10 +4369,49 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) * Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in * __schedule(). See the comment for smp_mb__after_spinlock(). * + * Additionally, this also guards against activation of blocked + * donors queued on a sleeping owner when a wakeup races for + * the same task with sched_proxy_exec(). + * + * proxy_enqueue_on_owner() + * STORE p->is_linked = 1 + * + * block_task() try_to_wake_up() + * smp_mb() + * STRORE p->on_rq = 0 LOAD p->needs_rq_sync + * + * The read below will atomically observe either p->on_rq or + * p->is_linked being set and put the task on ttwu_runnable() + * path. + * + * On the chain-wakeup path, p->on_rq is first transitioned to + * TASK_ON_RQ_MIGRATING before p->is_linked is cleared and the + * task is transitioned to TASK_ON_RQ_QUEUED + * + * activate_blocked_task() + * task_rq_lock() ttwu_runnable() + * STORE donor->on_rq = MIGRATING task_rq_lock() + * # STALL + * smp_wmb(); + * + * proxy_dequeue_from_owner() + * STORE donor->is_linked = 0 + * + * activate_task() + * STORE donor->on_rq = QUEUED + * + * rq_unlock() # ACQUIRED + * LOAD donor->needs_rq_sync + * + * In this case too try_to_wake_up() will correctly observe + * either p->on_rq != 0 or p->is_linked != 0 by atomically + * reading p->needs_rq_sync when the task is being activated as + * a part of chain wakeup. + * * A similar smp_rmb() lives in __task_needs_rq_lock(). */ smp_rmb(); - if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) + if (READ_ONCE(p->needs_rq_sync) && ttwu_runnable(p, wake_flags)) break; /* @@ -4488,7 +4531,7 @@ static bool __task_needs_rq_lock(struct task_struct *p) * See try_to_wake_up() for a longer comment. */ smp_rmb(); - if (p->on_rq) + if (READ_ONCE(p->needs_rq_sync)) return true; /* -- 2.34.1