From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SJ2PR03CU001.outbound.protection.outlook.com (mail-westusazon11012048.outbound.protection.outlook.com [52.101.43.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2D1238E124 for ; Thu, 28 May 2026 09:50:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.43.48 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779961804; cv=fail; b=UEn8MVkgc5+v8i0Tx1xmQ/GZMP+d4DrWP+aC8J0xoFhCfdPo49DtKfKU1imlY0++VtMS36zVjuAH13RbSPIdigkqBitdrDprQK73JI2OSwLZ5+3y36OsQP3v1cOaGTBTXx47BXmVXdpUNEgzmHQvz0XeXXpgUkb1Ycrf2N20zIg= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779961804; c=relaxed/simple; bh=7u5sx4hZqNnhAs4vDRO1xMdUXgAYl/LWBlxXGW/VnKE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=a5GhB2f4IIPt6YaXWeW1mkg79KW5AOC5qSxX5EoqPW5ywXmeUrkgNrw9xCeHxObUw53FNzsqs7gQiDdN/W8uUvaoTpkkwzu5OMN2rH1gZmZcQDY0Be8B6+alB+qh30pKjfY3sV1ukT7b6+Ixs3XNaKbSgg48Epe1dgnO04STg3A= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=t5ZrHXQj; arc=fail smtp.client-ip=52.101.43.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="t5ZrHXQj" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BbjLjJ9TBrakaQBBTQzaBe+O6lsYIaPyZghjeTnBMgzwybVR9WX22r0QvvkvRSjRZ92r9L3ar3H/KvST9HItSKGgwtFGqCNW+czSWmzhgyPODgNMCJED1BSqydqGhnA10iLjG+aPic8xbDmNez/5zts3KsxLXUFyOLSEh8P2gnNY7W1JYPQLCq7HWsXJPQTtPkOfr9113yWr6txo9ihsvVU8qnbkw8+G/JHs2aZG5mmsquzQZywhHeuSk11hH3Bl11AVdLYKCPfsZnMFuAmLxEK9HP3YxBsrNybFoS4XAX+jBur4UGwUaWXsAD0mtyInWfAT5+ZGO0SF40bmWCjYhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rsk/2Anf9RbeNEZNIebMMCLH9kg1DDL1TyB8I4XVdcQ=; b=m8A4ni5tTJltiU7/J5GkSr6vOO/TNwCn1QeoHZNA3FYxSEQKJfCVYuBiAtEUHchbYS9mQsEBmUm34pM/CcfLDaXSs2Ca5B8ZnwlXa6E89jmPbSmcoqnIXCnDpLpiOtF0E7WFX3gDNZE9FxMKIpyPnwdMmF0we/8MMZqVklNHFJM09NAAYYXNO3jk7RMCWc2fIP4wIigOzw28EwrKN3dt34XLJ8wPXZx3cP+t+IjrraHSGEwsR2Sr6R/gnIrPvV7VLGphIwQUM3URzpjrQ+eQhOyPf3ErpNQ3E/YHFg2iKYGyLJEnOEaP4fvkjK55+AUIIwcY61GK8IrQkFrVqxOAmw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rsk/2Anf9RbeNEZNIebMMCLH9kg1DDL1TyB8I4XVdcQ=; b=t5ZrHXQjPTiifTgA+IqmqD5pwyGX3vfEGK2B3TBZNwuJgr/lJmcEVscFOOGGFZmVY5d19dJvQSFg8Tr3xWB5OTltX+QL5mrOQ2XvL9J130eF+pFkntUKJjs6THBVq+owldf9Zi3fcP2OL6xYVkXDbiIU6FfxEW1zxsTotnA4Odk= Received: from MN2PR08CA0029.namprd08.prod.outlook.com (2603:10b6:208:239::34) by SA1PR12MB6970.namprd12.prod.outlook.com (2603:10b6:806:24d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.13; Thu, 28 May 2026 09:49:55 +0000 Received: from BN2PEPF00004FBD.namprd04.prod.outlook.com (2603:10b6:208:239:cafe::69) by MN2PR08CA0029.outlook.office365.com (2603:10b6:208:239::34) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.71.13 via Frontend Transport; Thu, 28 May 2026 09:49:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN2PEPF00004FBD.mail.protection.outlook.com (10.167.243.183) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.7 via Frontend Transport; Thu, 28 May 2026 09:49:55 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Thu, 28 May 2026 04:49:51 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Aaron Lu , Josh Don , K Prateek Nayak , Subject: [PATCH 4/5] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() Date: Thu, 28 May 2026 09:48:29 +0000 Message-ID: <20260528094830.13291-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528094830.13291-1-kprateek.nayak@amd.com> References: <20260528094830.13291-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBD:EE_|SA1PR12MB6970:EE_ X-MS-Office365-Filtering-Correlation-Id: 92824945-942f-48ed-e905-08debc9e78f6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|7416014|376014|82310400026|1800799024|11063799006|56012099006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: Uyz12MQgGvyVINoVH1UeFhI7VVlaMfhSG9bErYe8twpSXYCiUrHLeU8ISkIN5lWIdV1txR/x8FeoOYyYtFtNu0vcqpC5OCNx5ZatQkVYvFIGHOXflWfLYUEy1ECkM3jU+tN2tRPKnbhekhrBos1nVIkdeHO0LZfSh1zFV7c7BFvRl62c7dBOykgpvEzTiXiZvXXSHrpiqEyVhBDcRwcnoxdD+OLRVo82IGYj+CqcU28OS/XH84Q7EAmc+Ys+k2MBcLmNa+WXLUBI3dEpIQKnZ3njS8EnAa1fplZYXBNTlV8jqAhcSKfIdTwsN8rgF/9lcinpYUA1cyjA5Up668xUTJDlaL/4saFRhT+FLPjallcjbyzpn3zcbJjaop0l82OLAyjtPM85chD/J/lu0NiJmYjmIKdvGuyVvkD09psxRTik1oObjztCov9ruNXG0+0JtotCA8Yt9kEPyGLptRtYyB2dy/Yy+Cp3dB9IzPH9pSbiDGDgp7VuAFa3GjyYrCLTkfNAT0VTKsysSMQMywG2kyp1hvCCSGTaqZ1TTmkU21LpnFoogaX6ObWht4+vx7b0DLdc5q0IE4FhMZ6KVbD3HC9kN5JlfmVw4G5fz7pKsrGMyn3/7UNFwIZ/IUAC1DGCa1jW4lY2InJzrSaTlp+ovqIoWHetYjd5pftcRRI1Hw0Vkxof5KrCX7eDHKTNJaO7rhNmxVuijnoiiA/hiQDBn3Y5dsxkmzSTiGe0Kvztl4Y= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(7416014)(376014)(82310400026)(1800799024)(11063799006)(56012099006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: D8l9G6qbMTeBuAR3DOKgrI7sFVQjw+0FaOVJTud2cW1KBDiiyDnmi9ONH/YqHHq31V7oi4TDtxmNB8OkD+C4191A7Era/5vefKn3ajK/YhGI+5i/5fZzk/1NfP9Orc1Dx3LZndzH1BS77EudUl3mADTkLXj+p7Ef4bWeJXp0Z921Tm7Qi5rF0Ww7U2REd3yTkWSDdECHb1t5iiTgnWpK16v4fySD1oxdpPbJUP81Wj/Ob4iqJLo9NlYjX07d1xww5ZiEsQdlLoImOznT3jijA5QN+epGscXMcLvAYGsIvs1DoMbukxUoQ8ukpYawGhcJALJg8E04dHnwzuk+/MnNfeXICd0WNlbZYzH3bAaVySraJPE0zgvwEehOUAKeKQFimbsX9bKDc/gjsN6vvotEvCO87hfH7jjHwGhGhtr9KrdYV5LdTtzHdFIAgSs/V7Ba X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 09:49:55.3055 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 92824945-942f-48ed-e905-08debc9e78f6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBD.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6970 An update_curr() during the enqueue of throttled task will start throttling the hierarchy from subsequent commit. This can lead to tg_throttle_down() seeing non-empty throttled_limbo_list for the cfs_rq attaching the task from throttled_limbo_list one by one. For example: R | A / \ *B C | rq->curr *B is throttled with tasks on hte limbo list. When the tasks are unthrottled via tg_unthrottle_up() and entity of group B is placed onto A, update_curr() is called to catch up the vruntime and it may throttle group A causing the subsequent tg_throttle_down() to see the pending task's on B's limbo list. tg_unthrottle_up() /* --cfs_rq->throttle_count == 0 */ list_for_each_entry_safe(p, cfs_rq->throttled_limbo_list) enqueue_task_fair() enqueue_entity(se /* B->se */) update_curr(cfs_rq /* A->gcfs_rq */) account_cfs_rq_runtime(cfs_rq) throttle_cfs_rq(cfs_rq /* A->gcfs_rq */ ) tg_throttle_down() /* Reaches B->cfs_rq with throttle_count == 0 */ !!! !list_empty(&cfs_rq->throttled_limbo_list)) !!! Move the tasks from throttled_limbo_list onto a local list before starting the unthrottle to prevent the splat described above. If the hierarchy is throttled again in middle of an unthrottle, put the pending tasks back onto the limbo list to prevent running them unnecessarily. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b3b3172702a9..c48eaf2d7919 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6710,6 +6710,7 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) struct rq *rq = data; struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; struct task_struct *p, *tmp; + LIST_HEAD(throttled_tasks); /* * If cfs_rq->curr is set, the cfs_rq might not have caught up @@ -6740,13 +6741,31 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) cfs_rq->throttled_clock_self_time += delta; } + /* + * Move the tasks to a local list since an update_curr() during + * enqueue_task_fair() can throttle a higher cfs_rq, and it can + * see the "throttled_limbo_list" being non-empty in + * tg_throttle_down() if throttle_count turned 0 above. + */ + list_splice_init(&cfs_rq->throttled_limbo_list, &throttled_tasks); + /* Re-enqueue the tasks that have been throttled at this level. */ - list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) { + list_for_each_entry_safe(p, tmp, &throttled_tasks, throttle_node) { + /* + * Back to being throttled! Break out and put the remaining + * tasks back onto the limbo_list to prevent running them + * unnecessarily. + */ + if (cfs_rq->throttle_count) + break; + list_del_init(&p->throttle_node); p->throttled = false; - enqueue_task_fair(rq_of(cfs_rq), p, ENQUEUE_WAKEUP); + enqueue_task_fair(rq, p, ENQUEUE_WAKEUP); } + list_splice(&throttled_tasks, &cfs_rq->throttled_limbo_list); + /* Add cfs_rq with load or one or more already running entities to the list */ if (!cfs_rq_is_decayed(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); -- 2.43.0