From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010041.outbound.protection.outlook.com [52.101.85.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCCC03B6C18 for ; Wed, 26 Aug 2026 06:31:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.85.41 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725915; cv=fail; b=mSrNHG6QYI1MdSLzEceubpY/gXGvOXrXYaLAI4hXyYNujZCQ6NlnSsLKM7W4tDWX7mm0JqXcUU3qPSHvlU5YQDNbjRbplYZYz167plzF9n+NyrUXfDfrXP4EoXL9uE2ODflglUht10IxrQbt5IPkVoO8oWsFZ8Kr7PHIGEDvHiQ= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725915; c=relaxed/simple; bh=I5B8CRJUX0Z9y/JJPNubHnTU6633zFjaPDasLepyk00=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ShJGnWYB70tkSnh/iD0wG340AHHd/Yqf2P41wPKvJ4HjgmAF141ZFUH9SaNApQBfk55+FUHOQwj8J+kwzbEaLjeRJgnFVdnZ5Eo+W4OMoBqk78G+igbQ3L+sH9pZDRjR2VrJhNFUOVKzwUEyByjg5php5YkRebazZqjpmLrJtyo= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=LBqfT50F; arc=fail smtp.client-ip=52.101.85.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="LBqfT50F" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AXX5y0ncF7it6wZdAcb8AXCoHfH91kesLuctnTU0pY+/iQF3EzVqrpoMX9dyS8hQP2tlImwDWOoehcxxGhAE9UNbzGse9rUw1KrGU9gpmmmLrK2QDje3xQ9Sifdqb9+CUeGJ4OP0IbJAdU9Vi9focMjptEW0oLCpffZqH9I1NOxG/FmglYU3yGfWHr0TJf7ECPRgxsyuWqAE68bgKqMMQFpxOTU2eTV8gNBYj1co6n+R/jMvjJIWCv76jHojO9NS4nHMwC9PkF0+ID4xasKzZZcCBgoTNuxMbEns2+2cM0nFymq8AbNz7J1br3+FWfboj0rGgB/KTXpzgWRTxGQfXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UrIhgTeRXP9uV+VqG3fSVK8czaRuZZfhJlF9VmJbTyc=; b=bvJQeHu21V7aHw2LLUYf0F8qt+MZI5idqpPgAe6B+12gwaItoUfTvbBcNpyXUH5erYWqhk/HT318MNeQ9qHGOgM5DLY5vZ2lDFmoNqg/qhwTh7jyo0zn3mksDMkzCq3P9TNrNnab3mULcu1L3/jipeVDoJUU/7vppB0BW/UBRBcGTUCbzhNmZj5QueXoEFw3TasBXmdeApeeBZnGBXHBf4gXSxLhINPuRN39oINkMcRVxJHIAR0P2dRfk9U7cCWznraMS2260zkINoeb3QNsQHDC7V2yG0wsaUrExFzZKHO6QHDU1HUEegt4UCwB/lHYYf/stkP/zlkgubnlD6Pzhg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UrIhgTeRXP9uV+VqG3fSVK8czaRuZZfhJlF9VmJbTyc=; b=LBqfT50F6jlOu5gUP1gBSATXtwhC5P4127ld3OjEbpApeXZUgb2xoc/vfXzn+HLpU4lM3UKN6rLLO1BzyvtstwUmZYSavJJZZSGALsbkvxJwwWQh2GJXQh942n2clIUrcooA0BqVPkjAZdPd37a8iFCVXGdVMLxS43QmI32EY0o= Received: from CH0PR03CA0105.namprd03.prod.outlook.com (2603:10b6:610:cd::20) by SA3PR12MB176418.namprd12.prod.outlook.com (2603:10b6:806:513::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:31:49 +0000 Received: from CH2PEPF00000146.namprd02.prod.outlook.com (2603:10b6:610:cd:cafe::17) by CH0PR03CA0105.outlook.office365.com (2603:10b6:610:cd::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.7 via Frontend Transport; Wed, 26 Aug 2026 06:31:49 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000146.mail.protection.outlook.com (10.167.244.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:31:49 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:31:14 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 06/16] sched/core: Queue blocked donor onto sleeping owner for chain-wakeup Date: Wed, 26 Aug 2026 06:28:50 +0000 Message-ID: <20260826062901.2137-7-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000146:EE_|SA3PR12MB176418:EE_ X-MS-Office365-Filtering-Correlation-Id: 9116e675-5a80-4d12-bc4a-08df033bb5b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|1800799024|23010399003|36860700016|7416014|56012099006|11063799006|18002099003|22082099003|7136999003|10067099003|921020; X-Microsoft-Antispam-Message-Info: sHCbWFImd90Qkjwfbna7uoavOJudNTE8wtajjnThvXOO1NsY1IQSUST7kaFyYh70PB/+J3NmPgf3v2JhexEpM4ORjvRP8OP+qrhC5RytdEgpqRzMfjYW7LRQLHRze4daUom6nkgch1uC1cAjZ9Z9rxtpx26RixMKYldaU8OCD2advYFLV48RWZE8WR3drL7oWliUruSu36wQYUSmueRL0RGIoTvW4KE3fb56V6HgGJsGzGr7ontdiIoK4S8DcZVyswFvs3SxynAOUAcHqz0ZpMQ8qHOnbZoSSLGxR9blOkeRd24GahKqMx7LsrE6xwrsMxipwoS3vjKq+7gH++1WDAbjraT6DNHUUH4UaAIbXrxdtoZPVuARy9qHo08A6TpHReXECrAJM+zrFS95i6s0/u5rckEWgba7G9MwKtxvZGKGokNL5shpRjNMBnksS5A6Dlj2U8rtpc1FmmwDIyYcZHXR0d41UOGGC6sjcqOIp4xZ+zJOCDxUbT1YtmDDm/b1rQ22nQXeNfVmLMlyW5IilJUANsyxLlpcpTVOl//F5FEEMEUzYIiQOdep+sD7kDfSe5+bB+/59xogBuq9eg233F93XF8K1kQ6XuC9R7CcSOGoqmMN95kKR9CGTwSK3B2FAbdSmqd6uqR1E5ljv7E3QIP2VyAHjzasOhAyvp8G6EeRJE2NMxxluS9Xpzs3guKY9KFhIRWCtRhKJ2lra7/fSn78K/E8UVKUqj2effJEFl0DUbWsk52krP4AQxAZzVue X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(82310400026)(1800799024)(23010399003)(36860700016)(7416014)(56012099006)(11063799006)(18002099003)(22082099003)(7136999003)(10067099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iaounACrVVVfIImg03KKcY6t8ZAskGBmZ8aotZff3WTb9YeD1Hxcn8k4V2Ly2SsuVEGr7LSbAURsSy5KP7Qxs81trEwQ0sfNwcMHWc2F/5D1nLtEcBltRktDjKSs4rVouReo/9a8NYxG57jQoRFhN0j5+cQ+4KaqdUwAZgxs3iRIwaaS1nyBB0fbXpAXfLBDY/JdMRKmavUY+JknAailNXqGUKbm/QrbEIMZLYPdQkcWzb/+nCi9vhcCOutsz1Fbyt5vymNjeRuIoEchGnyM+2PtvJGOI75/Pe9UVO6H8zTJKDhP9n2zV+nHNMORLkQ9u5c3pe99qhz5BvmR+4CoD+H6TXENAybsUsGcjhzFY3iDcgb1htKmqtvkrFxGvfkZxM2pYQ1/QQke9NH4u/Zk5W2bwes0DLkt26/KRuIiC0O9JqDJlA1HDqX6RSwLUUeW X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:31:49.5341 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9116e675-5a80-4d12-bc4a-08df033bb5b5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000146.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB176418 Add a blocked_donor to sleeping owner's "blocked_head" when find_proxy_task() resolves to a blocked task. Once added to the list, double check if the owner has woken up by checking "owner->on_rq". If the owner has woken up, remove the task from "blocked_head" and continue try find_proxy_task() agiain to re-evaluate the state of owner and take the correct steps. Since find_proxy_task() is called with wait_lock held, which the owner needs during unlock, it is guaranteed that owner cannot disappear under us in the process. The added data memebers in task_struct serve the following pusrpose: - blocked_head: Contains the blocked donors queued on us - blocked_node: The list head used to queue onto blocked_head of a sleeping owner - sleeping_owner: Sleeping owner on which the task is queued. Co-developed-by: John Stultz Signed-off-by: John Stultz Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 5 ++++ init/init_task.c | 5 ++++ kernel/fork.c | 5 ++++ kernel/sched/core.c | 68 +++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 81 insertions(+), 2 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..bf0f4b6be7c4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1251,6 +1251,11 @@ struct task_struct { struct mutex *blocked_on; /* lock we're blocked on */ raw_spinlock_t blocked_lock; +#ifdef CONFIG_SCHED_PROXY_EXEC + struct list_head blocked_head; /* tasks blocked on this task */ + struct list_head blocked_node; /* our entry on someone elses blocked_head */ + struct task_struct *sleeping_owner; /* task our blocked_node is enqueued on */ +#endif /* * The task that is boosting this task; a back link for the current diff --git a/init/init_task.c b/init/init_task.c index b67ef6040a65..a097c0def4c4 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -211,6 +211,11 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { &init_task.alloc_lock), #endif .blocked_donor = NULL, +#ifdef CONFIG_SCHED_PROXY_EXEC + .blocked_head = LIST_HEAD_INIT(init_task.blocked_head), + .blocked_node = LIST_HEAD_INIT(init_task.blocked_node), + .sleeping_owner = NULL, +#endif #ifdef CONFIG_RT_MUTEXES .pi_waiters = RB_ROOT_CACHED, .pi_top_task = NULL, diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5..88f2b6e08c46 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2247,6 +2247,11 @@ __latent_entropy struct task_struct *copy_process( p->blocked_on = NULL; /* not blocked yet */ p->blocked_donor = NULL; /* nobody is boosting p yet */ +#ifdef CONFIG_SCHED_PROXY_EXEC + INIT_LIST_HEAD(&p->blocked_head); + INIT_LIST_HEAD(&p->blocked_node); + p->sleeping_owner = NULL; +#endif #ifdef CONFIG_BCACHE p->sequential_io = 0; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b4a2771eb290..e53967a126a9 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2229,6 +2229,16 @@ void activate_task(struct rq *rq, struct task_struct *p, int en_flags) __activate_task(rq, p, en_flags | ENQUEUE_MIGRATING); } +#ifdef CONFIG_SCHED_PROXY_EXEC + +static void __proxy_dequeue_from_owner(struct task_struct *p) +{ + list_del_init(&p->blocked_node); + WRITE_ONCE(p->sleeping_owner, NULL); +} + +#endif /* CONFIG_SCHED_PROXY_EXEC */ + static void activate_blocked_task(struct rq *rq, struct task_struct *p, int en_flags) { __activate_task(rq, p, en_flags); @@ -6872,6 +6882,43 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf, proxy_reacquire_rq_lock(rq, rf); } +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owner, + struct task_struct *p) +{ + lockdep_assert_rq_held(rq); + lockdep_assert_held(&owner->blocked_lock); + + WARN_ON(!p->on_rq); + WARN_ON(p->sleeping_owner); + + WRITE_ONCE(p->sleeping_owner, owner); + list_add(&p->blocked_node, &owner->blocked_head); + proxy_resched_idle(rq); + + /* + * Order against __activate_blocked_task_slowpath() checking + * owner->blocked_list after setting owner->on_rq. + */ + smp_mb(); + + if (READ_ONCE(owner->on_rq)) { + /* + * owner has woken up and may miss activating us. + * Remove ourself from "owner->blocked_head" and try + * find_proxy_task() again considering the owner's + * new state. + */ + __proxy_dequeue_from_owner(p); + return; + } + + /* + * Owner is fully blocked. __activate_blocked_task_slowpath() + * will see us on the list during wakeup and DTRT. + */ + block_task(rq, p, READ_ONCE(p->__state)); +} + /* * Find runnable lock owner to proxy for mutex blocked donor * @@ -6962,8 +7009,25 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) /* XXX Don't handle blocked owners yet */ if (curr_in_chain) return proxy_resched_idle(rq); - __clear_task_blocked_on(p, NULL); - goto deactivate; + /* + * If !@owner->on_rq, holding @rq->lock will not pin the task, + * so we cannot drop @mutex->wait_lock until we're sure its a blocked + * task on this rq. + * + * We use @owner->blocked_lock to serialize against ttwu_activate(). + * Either we see its new owner->on_rq or it will see our list_add(). + */ + WARN_ON(owner == p); + + raw_spin_unlock(&p->blocked_lock); + raw_spin_lock(&owner->blocked_lock); + + proxy_enqueue_on_owner(rq, owner, p); + + raw_spin_unlock(&owner->blocked_lock); + raw_spin_lock(&p->blocked_lock); + + return NULL; /* retry task selection */ } owner_cpu = task_cpu(owner); -- 2.34.1