From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CY3PR05CU001.outbound.protection.outlook.com (mail-westcentralusazon11013057.outbound.protection.outlook.com [40.93.201.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E36F3A1E8C for ; Wed, 26 Aug 2026 06:29:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.201.57 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725772; cv=fail; b=fJBfY1UdPJweGa7lhCYWrKayLvZHMN6Ami2plNHhQTRah3prGYu6AyRmsWCS4k5OR2Jf7M8mhfOsUjWPM1oszm+WUSqwmiZkM8VOpgV0flYZo3ERAvFTnUb3UVoEVjHqMfLao2GpEoNOpiTmKwXuPiwVlCHRktxgRc+9g2KWFf8= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725772; c=relaxed/simple; bh=q3aCaiUz5Zj41HLsp8NrtN6hPWciud93DvIY39Hr4f4=; h=From:To:CC:Subject:Date:Message-ID:MIME-Version:Content-Type; b=m82Q4V0vb8gLKbvcYMdsJ3FP3gLFue0XynX78QYcJ1G5iTt14eINPCUg8e4BaPxCPgNAd5cwWvbm3SeGEf6VZraMMSvToF7j+Rl7q1LfRjp/59RjBOFrcmAKLrmkWYwfL+gX1bW391WOWVHewEfFeaOlq7G4Zrj7JlRbSGzvlc4= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=jNXwd6oz; arc=fail smtp.client-ip=40.93.201.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="jNXwd6oz" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xda6fgBnyQJ42AvuhOMHKIv29ZAojYFZ2DPvGw/Z9QMuyaiC9an4QpgGAZN3X9S3WJU+1u0jNIRp3PURKfQfjNwBHW0uaYvC7QXTstOqShc6DLQFTowK72SlGu8CXJPciVk2feENQJW4bnEs95i3qqG686N948qtRp73EaQfo+l3tW0D+pGNnY97vndtZV9TQjq02Jf6n8arkzqpltD8qVcZykG+5xMDeHMijbckKRz22GEgOoh+k6YgyzhvKd3RFPEiByCeKGuRQT6ILh2OEHPxGr14ZHZTTx9A1Y3QmurSlL8XEv50uDUqhCeEi9ivhpQGdltFzlbB5Diac0FWCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YS35aJYrKOOZQROWjgdcwG+BnauDDQ8E/Fcm9+I9Yiw=; b=tNsFZhnp0PmLhyyceI4dwOTqdVSKav0NYTMRk57GsqhHaV9wBSvU9jRgjp7gUHuz4t/kQa+h3YRetyUlcqwVAHLWaMnbyxujOgfwERPQqTyIZiP2xCQZ45El3l8M9R1i/NsEMKKIMZOKXxbQpOpgKOw5etrBQIHfWjafrBvQW9QlCqo+l72HSLw1vAcPy0rxuMdNKcv+JKsFBPYd7L3rh55+qpUbmEdr6UgDKPqQOz13AXmALVDO+AshFUrNzCKjm48K5Cs3h91hZZExiWwf1YzOcPtITwStH2Ey4qYbriKCO+NaSKpumLoO7wKx8BCf/+uHJ3gmzfNbBtj7FAiHDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YS35aJYrKOOZQROWjgdcwG+BnauDDQ8E/Fcm9+I9Yiw=; b=jNXwd6ozDOhN8dkNTLdXXVKVmy2iIs3WzeAx+rUDnkyykwC8pD6zu3VufjVPJRFxsBPws1ssnGo/eJoS7LyseHIOvjxIhlmvbkKuk8QguHmRwCfA/851hqEQA59ysEZ01aOj9CZwQkPmKwESSt8+dXTwdVZPUS46RvGu4BosZTo= Received: from CH5P222CA0017.NAMP222.PROD.OUTLOOK.COM (2603:10b6:610:1ee::21) by MN0PR12MB6149.namprd12.prod.outlook.com (2603:10b6:208:3c7::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:29:22 +0000 Received: from CH2PEPF00000148.namprd02.prod.outlook.com (2603:10b6:610:1ee:cafe::58) by CH5P222CA0017.outlook.office365.com (2603:10b6:610:1ee::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.8 via Frontend Transport; Wed, 26 Aug 2026 06:29:22 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000148.mail.protection.outlook.com (10.167.244.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:29:22 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:29:16 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 00/16][PoC] sched/core: Alternate approach to sleeping-owner handling in PROXY_EXEC Date: Wed, 26 Aug 2026 06:28:44 +0000 Message-ID: <20260826062901.2137-1-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000148:EE_|MN0PR12MB6149:EE_ X-MS-Office365-Filtering-Correlation-Id: 4263b833-8ee9-4bd0-0b80-08df033b5e02 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|36860700016|82310400026|7416014|23010399003|18002099003|11063799006|56012099006|13003099007|10067099003|921020; X-Microsoft-Antispam-Message-Info: z6yIvqdagtIx506WSdKiewl5VoOTWXCRtqyISFvybMHC8C9oBHqk4nk7S56PhuzJruaSztdZk4DyBJmMZMERKVS+pEOX94Xz2or9n58qSDJZXwdQqBTgUQ5hcMr+ApdCKWf0O6oiK87iNcYvTHKxH1Ucwe35zkTuN6E0yWSAlkaTJxg6vTtwbnA8XFJUXKEiVM5bd6PqKAwkeICfnbtFejFmxQMIRswd3Um+GZp7OzjhVS6usTwmsZb4qtEYVhALhxIGv6zsex9GDhSeVoDtG6Z3In29TomdzF3+u03+xMfbqtGdwOZTWP3JOra3Xwj4u+he+kvWO19obpBMvZIa4/j/xJfB8ApIHrD4Lsz09dtDyiZERpQWF6pzEI71dbPWJ02pjEpkOk9911B2kAmz3qHSthMVBzIt5LRoKqns5tKFbPhX7T4N/JxWSSDXWGhw/0Ztc6yZ9S43WDJkz8NECwoyzk9SAVoBOE2DxqWLPeh+i80kJV7Bi+wGBTUrYqVLe/1G5TWq393PrQ9r+v8vYTB234KSb4/WQYUQJoWKwmsGT2bNbiHNFZbZTEXZgJkSXv0VLdQOuPCNh2PZgLvxwQJ50eYQrEqVdm6HLxyAxma2EhrzcLhs5J3PZZqxXoi7I+ss4kyR+A9XPQP6y+/rLA4sZ6OyXR+Vl3PFNl3oD5OS6Kfka3t0iyQGZOxEF/jmGdZbllrQk6umlF0oYORjUxJAXEXYOiXzJQgx3Uw51Nzgv7nwbF9tDiz/v+/cJ0lA X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(36860700016)(82310400026)(7416014)(23010399003)(18002099003)(11063799006)(56012099006)(13003099007)(10067099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: MTcewo4lZl3wqSxDyWjex/JSn23lUGmlIgx0+cyWW55RRA89EOKIN6Pbyt87c6On7+ryGWHejWCOErljT7p7X7QhdfIUhBG/fhcAKr02TDXwHFTtNZ6YlqUxxll4XYOUQ8pbF8h4GiTSRHq5UPQ75zl/kjNyKMuKN4hV8Gq6nJxNZABEzpU5VCXvPjLKLuajCZrS4La8aFt3cxDjoZlfbVd8fBmDaaZy+Q8y6VxdV2WJ3LuRIsPWlg1qR5r2xOeyoaCQlduu7S2ZXXY+IEfM5Y9Fwr2GvULcYerWLigw5pBNOJcKaKa9MsdUqaGT7fVlE9uUv4sKbq3oGAl30igUoEQW0PDJ0uMeYeYzzIXjokqCt9RQxMc4scTCJOAHwBxsbkmyFgjJLXVZAfGQ3ltgJTxVUQTmRFa1XHxSe6WNOUtymIbPabEpCxPFghkVMX2J X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:29:22.3615 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4263b833-8ee9-4bd0-0b80-08df033b5e02 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000148.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6149 Hello folks, I had promised John I would share an insane idea if I got it working(ish) so this RFC presents an alternate approach to handle enqueued donor wakeup vs owner's chain-wakeup race via ttwu_runnable() path and make the activate path as light as possible for tasks that don't need to do a chain wakeup. The series essentially breaks down John's large patch at https://lore.kernel.org/lkml/20260807035232.1881495-9-jstultz@google.com/ into smaller chunks while introducing the new approach and fixing a few snags encountered along the way (Patch 1 - Patch 4 are fixes that can be discussed without getting into the rest of the RFC). Disclaimer: Series in not bisectible in any way at the moment - related bits are introduced together at and intermediate builds may fail. Introduction ============ Handling sleeping owner requires proxy-chains to queue on the said owner and perform a chain activation when the blocked owner wakes up. Since a blocked donor can be woken up by another concurrent wake event, the activation path becomes complicated and wakeup has to take an extra lock (p->blocked_lock) to prevent any modifications to "p->blocked_head" and miss activating any blocked donors. The nesting rules for the locks are as follows: p->pi_lock __task_rq_lock() lock->wait_lock p->blocked_lock Current chain-wakeup scheme juggles p->pi_lock, and p->blocked_lock to prevent concurrent wake events from modifying the chain. This RFC introduces a __task_rq_lock() based scheme that requires just the __task_rq_lock() (always held) + p->blocked_lock (juggled for each owner in chain) when queued tasks are detected to minimize the lock bouncing and introduces a no-extra-locking fast-path for activation. Trade offs are discussed at the end. Approach ======== To implement this, we try a bunch of bad ideas: o Waking up blocked donors linked to sleeping owner via ttwu_runnable() To prevent chain-wakeup transitioning "p->on_rq" to QUEUED while a concurrent wakeup is in progress, "p->on_rq" is unionized with a new state "p->is_linked" as: union { struct { u8 on_rq; /* Task on CPU */ u8 is_linked; /* Task on sleeping owner */ }; u16 needs_rq_sync; } Instead of checking READ_ONCE(p->on_rq), to send task down the ttwu_runnable() route, the new "needs_rq_sync" indicator is used which can be atomically inspected outside rq_lock. In ttwu_runnable() a fast-path first tries to remove the blocked donor from the sleeping owner by just grabbing the "owner->blocked_lock" and forces a full __task_rq_lock() if the blocked donor's state has changed while trying to grab the blocked lock. The chain-wakeup path takes a single __task_rq_lock() to prevent any race from concurrent wakeups (described later). o Proactively removing the blocked donor from owner if it wakes up Proactively remove the blocked donor from sleeping owner in proxy_enqueue_on_owner() if owner->on_rq is transitioned to != 0. This ensures owner need not care about any additional donors that come in after it transitions p->on_rq. proxy_enqueue_on_owner() holds lock->wait_lock + owner->blocked_lock. Since owner must grab the wait_lock again to wake up the blocked donor, it is guaranteed that owner cannot disappear and get_task_struct() / put_task_struct() dance is avoided. o Migrating enqueued donors while blocking (SLEEP | MIGRATING) To ensure a single __task_rq_lock() can handle the entire chain, a blocked donor is migrated to a deterministic CPU before it is transitioned to p->on_rq = 0. This ensures the whole chain is owned by a single CPU. o No delayed tasks on chain Delayed tasks pose a challenge that they are still on_rq and require special handling to ensure the above condition of chain resolving to a single CPU holds true. For simplicity, all delayed tasks on the chain are fully blocked before they are queued. This can be handled differently but I got a bit busy lately so that will be handled in RFC v2 (if folks think this approach holds water). o Single shot chain-wakeup (ab)using TASK_ON_RQ_MIGRATING Tasks on chain are transitioned from being blocked to TASK_ON_RQ_MIGRATING under __task_rq_lock(). Tasks on ttwu_runnable() path that are trying to grab __task_rq_lock() is stalled until the task is woken up on destination rq and the rq_lock() there is dropped. o Tracking task that can potentially resolve to __mutex_owner() This indicator is used to selectively set the lock holding tasks to the slow-path. Interleaving ============ How are different races handled (individual patches have these highlighted too, either in comment message, or via code comments): o Owner wakeup vs donor enqueue activate_task(owner) owner->on_rq = MIGRATING proxy_enqueue_on_owner() guard(raw_spinlock)(&owner->blocked_lock) list_add(donor, owner->blocked_head) smp_mb() /* MIGRATING is visible */ smp_mb() /* List addition is visible */ if (list_empty(&owner->blocked_head)) if (!READ_ONCE(owner->on_rq)) return __activate_task(owner) block_task(donor) ... /* Do chain-activation */ __proxy_dequeue_from_owner(donor) o Chain-wakeup vs donor wakeup proxy_activate_donor_task() try_to_wake_up(donor) rq_lock(rq_of(onwner->blocked_cpu)) proxy_try_dequeue_from_owner(p) guard(raw_spinlock)(&owner->blocked_lock) guard(raw_spinlock)(&owner->blocked_lock) for_each(donor, owner->blocked_head) donor->on_rq = MIGRATING; smp_mb(); if (donor->on_rq) goto slowpath; ... /* More stuff */ rq_unlock(rq_of(onwner->blocked_cpu)); __task_rq_lock(donor) if (donor->on_rq == MIGRATING) cpu_relax(); __proxy_dequeue_from_owner(donor) __activate_task(donor) p->on_rq = QUEUED; /* Acquired */ if (p->is_linked) proxy_dequeue_from_owner(donor) /* Do wakeup. */ Chain wakeup is covered extensively in Patch 14. Series breakdown ================ Patch 01-04: Fixes to existing bits (Patch 4 needs more thinking) Patch 05-07: Donor queuing with some convenience added to skip delayed handling later on. Patch 08-13: Putting tasks on ttwu_running() path and swapping task_cpu() while blocking Patch 14 : Chain wake-up Patch 15-16: Optimizations to limit slowpath. Trade-off ========= Advantages: o Only need to juggle blocked_lock(s) under a single rq_lock(). o Re-use ttwu_runnable bit to handle removal of proxy donor. o No need to do get_task_stuct() / put_task_struct() juggling. Disadvantages: o Possible increased rq_lock contention on the chain-wakeup path but those events are generally rare. o Lack of delayed task handling since the src_rq need to be locked separately to migrate it. For the delayed handling, it is possible to use proxy_migrate_task() to move the task after blocking. Series does not implement this yet to limit the number of bad ideas. How to apply ============ This series is based on John's tree at https://github.com/johnstultz-work/linux-dev.git proxy-exec-v31-7.2-rc4 at commit 18bea15ce9fa ("sched: Migrate whole chain in proxy_migrate_task()"). For convenience, a tree is present at: https://github.com/kudureranganath/linux.git sched/proxy/sleeping_owner_rfc_v1 Sorry in advance to folks whose VPNs hate dealing with Github but I'm told pulling it locally and inspecting the patches on tree works better. Testing ======= Tested with perf bench sched messaging, test-ww_mutex, and a kernel module that implements a duty-cycle with sleep_ms() + busy-loop in kernel. I made sure to build and test !SCHED_PROXY_EXEC version this time around to save John some trouble :-) Acknowledgement =============== Most of the good work is by John and all the bad ideas are by me. Sorry in advance. --- K Prateek Nayak (16): sched/core: Break activation of blocked task into a separate helper sched/core: Use enqueue/dequeue flags instead of task_on_rq_migrating() sched/fair: Use enqueue flags for DO_ATTACH in update_load_avg() sched/core: Activate blocked donor when no owner is found sched/core: Do not queue blocked donor on a delayed owner sched/core: Queue blocked donor onto sleeping owner for chain-wakeup sched/core: Avoid delaying blocked donors queued on sleeping owner sched/deadline: Prepare for blocking and proxy activation with MIGRATING flag sched/core: Track CPU where the task was blocked on sched/core: Introduce p->is_linked to track if task is queued on sleeping owner sched/core: Prepare to inspect ->is_linked alongside ->on_rq during wakeup sched:core: Add MIGRATING flags when blocking and activating linked donors sched/core: Use p->is_linked state to unlink from sleeping owner early sched/core: Introduce chain-wakeup to activate blocked donors locking/mutex: Track locks owned by a task in a per-task counter sched/core: Set activation of non lock-holders to fast-path include/linux/sched.h | 31 ++- init/init_task.c | 5 + kernel/fork.c | 5 + kernel/locking/mutex.c | 57 +++- kernel/sched/core.c | 580 ++++++++++++++++++++++++++++++++++++++-- kernel/sched/deadline.c | 15 +- kernel/sched/fair.c | 16 +- kernel/sched/sched.h | 6 + kernel/sched/stats.h | 2 +- 9 files changed, 666 insertions(+), 51 deletions(-) base-commit: 18bea15ce9faaa0c1bd827012e6c60dbf35578c7 -- 2.34.1