From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012023.outbound.protection.outlook.com [40.107.209.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C28732E7162 for ; Wed, 26 Aug 2026 06:34:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.23 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726079; cv=fail; b=NcB1Y3SR+15hxoazwqZnvjfFYSdUJoiTsYxJBigR2bYizXcvZbNUC2GIEvFpdpNepUhlhLUqVdZtyvHuJsIQIQ5t4kkM3Zy8WlI4W26QedRzJ21NPOqIUbVoCgy9iWhrOzsdM8tjnPfph5P2LEeOP1ScQMlS89iElBsIZlt4kio= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726079; c=relaxed/simple; bh=CqXUgQZFz4PTFo2oD7ogkUbezg2VWT1Na9MewITSpMo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=lpXfJeNkIFfMUL80ZcmrhTMv+ut98Y84cO0Y2+J1UhKQ7xSxtAEuLa2AnqKVG4um6FDnsn51J+d9IUITJm71tE7tWjFyfwQyuknpdCu0LZBEu/wMEOXj+I98AeBnuOlUSVVzExrGVKufTXNXeAc8bQM0VAVcCjTw4O1a4f+WnkU= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=5pgbC/Tw; arc=fail smtp.client-ip=40.107.209.23 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="5pgbC/Tw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ikmfwnlyrNx+76iYS5jBh51VIhVtEeJs+/+5XyyXK06zymsphhW5haLT9poi7Z2Gg269cKAiLyq7FTu0hfOS0Ohx1vHy5RcNM3+HrN2a8xqWE+VZeU5S6KFryBehS9AYcjHFeJx0r8E4GLOCLwQTU3joxcFylW03jvymsa54FejXEgcbigXJCkNjMDp7gCLveE7RwOpJMZkI3x872VIldjX4bziP0Th+Pydro8bYyc9grxTlUo4oFn09J3ZKYU3r1WzwvkqpHtIc4mt5fevtR8DtPDPZzvrqJuAmh7eNgrDD0CH7BH0e0MsWSgmcf3PXHLSYE0zVYUZ96Seo3MGkOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f6kVmQL16rORhVmLfjDlY+924TDnTsWH99rTIi8TASA=; b=MNdSAb+MOSba9gRsxUN5qJoFFNEgEn7eN3DHpRgRGbDkSz8wRuRmzPkxPPYbnuAD4Z4Po3mlgrLFmEhEZMxqXh+IRB96jp/tiMrv6LycYsTakoOBjjdjuJnQmExlinF2vOJBDRPuxtnWOhdm0tOdomFiK6G2VaEckdnbGrVOskReApiu+oo3yU8UuiBn4LdDIAkPmF2NOE5HrESS6KgXXB1lDZ4Z4qtgMeK/UocF4AF0R9rnWrSyLSr2xZb6OWFFcGl8o8ABZVsom0ZArqUam0HaT3EkEOENNOyCbnVh0uVabaZklqVsUHycg3eauqzhhavVUiZujK7DFDXkKRUsZw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f6kVmQL16rORhVmLfjDlY+924TDnTsWH99rTIi8TASA=; b=5pgbC/TwLMCcQMBhMjRq+6phoBLLdR//zq1lURYGcwKUQGKidXszMT0pvq38xaMEFE350XaRF11uY6d7MHi1bvsC2+zdF6Tir+zrMrHZ7j1x3LjFv5LRrc+hf2kEJ59EC6lN/TC5utH23gwzApnM2pE0NWCyd4NwvqqwKqX/HJg= Received: from CYZPR14CA0003.namprd14.prod.outlook.com (2603:10b6:930:8f::16) by MN0PR12MB6342.namprd12.prod.outlook.com (2603:10b6:208:3c1::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.6; Wed, 26 Aug 2026 06:34:33 +0000 Received: from CY4PEPF0000E9DA.namprd05.prod.outlook.com (2603:10b6:930:8f:cafe::3f) by CYZPR14CA0003.outlook.office365.com (2603:10b6:930:8f::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.8 via Frontend Transport; Wed, 26 Aug 2026 06:34:32 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9DA.mail.protection.outlook.com (10.167.241.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:34:32 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:34:27 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 15/16] locking/mutex: Track locks owned by a task in a per-task counter Date: Wed, 26 Aug 2026 06:28:59 +0000 Message-ID: <20260826062901.2137-16-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DA:EE_|MN0PR12MB6342:EE_ X-MS-Office365-Filtering-Correlation-Id: 74406ff6-785c-48a2-d8b3-08df033c16e6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|82310400026|36860700016|1800799024|7416014|376014|921020|10067099003|11063799006|56012099006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: f/bZ2RHswvnCHI3/zpcotoyqwya/SlRzC7KHVCTBQe+bYJagnIBJ/WYfgnPdAtbsYNuBFeitOBmABgXZ6t8tmi0gOS5aWY5Dehh8TtdyKQvbje3BuW4A36wi81C6KH2pgqVTGr1Iz+hK8iDByWNBllC8Z4pMRV5+rVRawo0TVL6YUep3sOROJkl5si2yInploQD/PKFmQ/q5pZAW6Jmp0cobIKBjKCUt7OtAeQjr3MJ7RVMgLyKT4JLcoTLvptJlMRy3tg/HfrrNTlwtbBVjo6Xm8WZLWnbYlnmrQbpBnzc+wGUnvHV/FL+TIxaOMXvi96rH+5cQmLXIpDodMtCMwFGnK6gT03d7EjhanRFQ8yna3pG56VLF0Tp/veUkAH6rYmgoxaixt931pQBYkQepe8I+DDL14P/eESlvasLrMbDSLc1yfiZikO6bhqzCCSO9C7h8lyglr1gD1GjcPBeU0E2rawzs0T9Q1Z+7UhKZY/fCyKILXypeiaJ6WXWLm42db2B4Hayj/SZ9QZO/I1dKk6DUOumAQ6zGnpH0OPgs3dwT+gQ/6s90u0BelM2sNdGp02X7Ku5OM6KS7Ow7Y7vONyzdrH3seYEgXtB/GM+UgWjmxn69X8xziTyklXFTcGnW34n0s+2ejuks6hELdGCojSLb3k1Tk/jRKiuYu2icmzAEgL1vzQHH2Xt/kWIMORBMqzLBRhp2qx3XFh7MIcZFzE25q2kGLcE5JOh/MmX8gN2/JKxvlkT85ZBxDrA8F4lO X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(23010399003)(82310400026)(36860700016)(1800799024)(7416014)(376014)(921020)(10067099003)(11063799006)(56012099006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mPIyEr+R6ob7pQwFfN7nT3TvUN1Cb5fh9hILmnwl12sHQ2bl5KhfLLjX4bYh6+baXPGThL6HW8IBbuVh2z/3dgCGbuI0gHYZz/J6ITEZeRjIMz20BccKQ2Q1Rebs2XgWGmZhagYD8gJUh7nWQAnjcK4sNO9Lu4n4mm/5mI1zovMZl38zUF5JwukSd0P9e6/6SXbY3+GEW+kt5Ql/y1Ew/1ch9kneWnKASy6PjRYyZGJtBTBickSg1+R/J2+OeXUb8ldNVo3fjgNXgAShvsat87D8DQwGZVRHfSsRdD6idT+uNUDVI0vJMpdArvEW4PXIccBONaKaf0gaXNOExp7+UY7dJHmjma7GgWs2hQ3Og9ubrvWcMbpZ/wxTV+bYauUokGxU8Al8RaHna0ZUS/Hz+gyzmRXV4tesq8vWEcfKnc1NKL6rsRXupCT/unEp+IAP X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:34:32.6166 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 74406ff6-785c-48a2-d8b3-08df033c16e6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DA.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6342 With sleeping owner handling, the task activation path will need to be extra careful when waking up a task that can resolve to a mutex_owner(). Track the lock acquisitions in a per-task counter to selectively put mutex owner activations onto a slowpath. The per-task accounting is done locklessly on the current task. This is only relevant for a blocking owner and barrie in __block_task() ensures the observer sees the correct "lock_nesting" before p->on_rq is transitioned to 0. The counter will be used in subsequent patches to optimize activate_blocked_task() and __proxy_block_task(). One exception to the lock tracking rule is the increment before schedule_preempt_disabled() and decrement after to fix the former mis-count. This is necessary for mutex since a handoff can set fully blocked task as mutex_owner and, at that moment, other tasks can start queuing themselves onto the blocked owner which requires the wakeup path to go via slow-path and ensure blocked waiters (if any) are activated. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 6 +++++ kernel/locking/mutex.c | 57 ++++++++++++++++++++++++++++++++---------- 2 files changed, 50 insertions(+), 13 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index a05240b52806..cb58992814fb 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1272,6 +1272,12 @@ struct task_struct { struct task_struct *sleeping_owner; /* task our blocked_node is enqueued on */ int blocked_cpu; /* CPU where task was blocked. */ #endif + /* + * HACK: These bits should technically live in CONFIG_SCHED_PROXY_EXEC + * block and only account when sched_proxy_exec() is true but for the + * PoC keep it outside and assume proxy is always enabled ;-) + */ + unsigned int lock_nesting; /* * The task that is boosting this task; a back link for the current diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 8a85912d7ee6..23a63e117c3e 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -84,7 +84,8 @@ unsigned long mutex_get_owner(struct mutex *lock) */ static inline struct task_struct *__mutex_trylock_common(struct mutex *lock, bool handoff) { - unsigned long owner, curr = (unsigned long)current; + struct task_struct *cur_task = current; + unsigned long owner, curr = (unsigned long)cur_task; owner = atomic_long_read(&lock->owner); for (;;) { /* must loop, can race against a flag */ @@ -109,8 +110,10 @@ static inline struct task_struct *__mutex_trylock_common(struct mutex *lock, boo } if (atomic_long_try_cmpxchg_acquire(&lock->owner, &owner, task | flags)) { - if (task == curr) + if (task == curr) { + cur_task->lock_nesting++; return NULL; + } break; } } @@ -153,13 +156,16 @@ EXPORT_SYMBOL(mutex_init_generic); static __always_inline bool __mutex_trylock_fast(struct mutex *lock) __cond_acquires(true, lock) { - unsigned long curr = (unsigned long)current; + struct task_struct *cur_task = current; + unsigned long curr = (unsigned long)cur_task; unsigned long zero = 0UL; MUTEX_WARN_ON(lock->magic != lock); - if (atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr)) + if (atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr)) { + cur_task->lock_nesting++; return true; + } return false; } @@ -167,9 +173,15 @@ static __always_inline bool __mutex_trylock_fast(struct mutex *lock) static __always_inline bool __mutex_unlock_fast(struct mutex *lock) __cond_releases(true, lock) { - unsigned long curr = (unsigned long)current; + struct task_struct *cur_task = current; + unsigned long curr = (unsigned long)cur_task; + + if (atomic_long_try_cmpxchg_release(&lock->owner, &curr, 0UL)) { + cur_task->lock_nesting--; + return true; + } - return atomic_long_try_cmpxchg_release(&lock->owner, &curr, 0UL); + return false; } #else /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -723,8 +735,23 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q); + /* + * Mutex handoff can set the current as the lock owner while + * it is blocked. Increment "nesting_count" on the way to + * schedule() to notify the wakeup path that __mutex_owner() + * can resolve to this task on the way back. + */ + current->lock_nesting++; + schedule_preempt_disabled(); + /* + * Fix the guard against handoff. If the task got the lock, + * __mutex_trylock*() will increment the lock_nesting when the + * acquisition is finalized before exiting. + */ + current->lock_nesting--; + first = lock->first_waiter == &waiter; raw_spin_lock_irqsave(&lock->wait_lock, flags); @@ -981,7 +1008,7 @@ EXPORT_SYMBOL_GPL(ww_mutex_lock_interruptible); static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigned long ip) __releases(lock) { - struct task_struct *donor, *next = NULL; + struct task_struct *cur_task, *donor, *next = NULL; struct mutex_waiter *waiter; unsigned long owner; unsigned long flags; @@ -997,6 +1024,10 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne * missed. */ guard(preempt)(); + + cur_task = current; + cur_task->lock_nesting--; + /* * Release the lock before (potentially) taking the spinlock such that * other contenders can get on with things ASAP. @@ -1006,10 +1037,10 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne */ owner = atomic_long_read(&lock->owner); for (;;) { - MUTEX_WARN_ON(__owner_task(owner) != current); + MUTEX_WARN_ON(__owner_task(owner) != cur_task); MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP); - if (sched_proxy_exec() && current->blocked_donor) { + if (sched_proxy_exec() && cur_task->blocked_donor) { /* force handoff if we have a blocked_donor */ owner = MUTEX_FLAG_HANDOFF; break; @@ -1027,7 +1058,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne } raw_spin_lock_irqsave(&lock->wait_lock, flags); - raw_spin_lock(¤t->blocked_lock); + raw_spin_lock(&cur_task->blocked_lock); debug_mutex_unlock(lock); if (sched_proxy_exec()) { @@ -1036,7 +1067,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne * current through this lock, hand the lock to that task, as that * is the highest waiter, as selected by the scheduling function. */ - donor = current->blocked_donor; + donor = cur_task->blocked_donor; if (donor) { struct mutex *next_lock; @@ -1045,7 +1076,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne if (next_lock == lock) { next = get_task_struct(donor); __clear_task_blocked_on(next, lock); - current->blocked_donor = NULL; + cur_task->blocked_donor = NULL; } raw_spin_unlock(&donor->blocked_lock); } @@ -1071,7 +1102,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne if (owner & MUTEX_FLAG_HANDOFF) __mutex_handoff(lock, next); - raw_spin_unlock(¤t->blocked_lock); + raw_spin_unlock(&cur_task->blocked_lock); raw_spin_unlock_irqrestore(&lock->wait_lock, flags); if (next) { wake_up_process(next); -- 2.34.1