From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CH4PR04CU002.outbound.protection.outlook.com (mail-northcentralusazon11013048.outbound.protection.outlook.com [40.107.201.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AB102AE7A for ; Tue, 2 Jun 2026 05:26:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.201.48 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780377973; cv=fail; b=n/1u10qU7lWWifcOfhY4ukDmmKbLBaR19soxDwKWnbNF22SuzgVoUrGHoEuqUx/2F8G4ksErfYJ45TB61ZTtnYMJFrdsr3KWziErta8BnPtxDMOECAi9RSUIRGt5Mrjj5rryz68VBmD9zaWRi6PK2rLChXaWwpMTkJcFPqgGyuA= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780377973; c=relaxed/simple; bh=3qyJID67jStDgUb4mCRnJY43iAfB1h0g35WJKxQ+lic=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fSG3JMIz89MWRBP41cg3Tmh2fTNcYO5r7HUAp4I1Gzj799lfTCxDtEqiop32OOtVBPQ4J8ymKxunBznElzZgdF6q/GJESIeln7AD5VcwFxBPV2VIEDr3L/CwzljiEJo5ISiaMRolD2t+OwXjC0INKLew7bqvQhCpsn695SMNO3U= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=fvv6Guf3; arc=fail smtp.client-ip=40.107.201.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="fvv6Guf3" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Rz5lA8E0K16Iq0grSoSUWYqVHDKcB7zb9evtAQj9OnznH8I4QiDLxL7v/z+9gOm4ky9rB2mHI6+RgqSm7hp3opVAZsJrpBo5cIVHnON4Tpz7ue7btFzs9AHEtzUWxUyZLC1XVA0DM7qgFpcWuIJ1YAgLVmSLNXUxx/9KRy/0Ko4Q8UaIAeFg4d/LoV7m4sB1IPR5fx1RXBU50ZXC+HFMQ/3w45QBABlZ/2nc7MvlHA46FGpd7zWS89yii0bidpGalA9RNu1Dv+thrpfiJxyU8FrL3qc0Xnng10LCzNsKrliT0lve9P0P2ZRvQkO+METlaAdhftnwM9ffYqJxdwxjjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ga8MFoHOm5Ieugtnd5nMkAUAHZ+gKwabJ7cOut6lboQ=; b=xAfqRQnAVHay3SPidFb8pgnFzlxPmMbCtJm98Q/yCPNXORNzhrCNsQ9KUMh5zcXHAqVel5zorWWIR6STBpeH9BynN3uI3HWTIrVTw70RF1+iAg7JH6qWsdFGg72x23x7CLbV48lbJpbDbpx2HgXrMfFfkdaDzveNjZ1F9zegvLA10pBzqhBX4Gp8n1b4zwxb8/2SLdwx1w1YwusobVCFDPXZAO3uBh1QbqaJsItWD+PYAElnX0g2B9ta/DGqwzeyCjCnDT8FlOGk9EVUj6lXZc5hqZ5pYmyh1nzVxWqc8J6gtZWEcfTIFVVCJIVPwxlQ0xeJXXMp2xSLidrMYLaMUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ga8MFoHOm5Ieugtnd5nMkAUAHZ+gKwabJ7cOut6lboQ=; b=fvv6Guf3B+H+NgrxWugUJCp/+QJhi8LxvNg/b/zsZL+aM8r/G9+KP4zHhpFKuVXEwYEcPpuxh4JUKEgamF3UD68jtJig2EXaCW8+jTkxnRcgRZamNykIHYvs967sykHaPSMjibMoR7vIzU+0MhQmlhdzgRO3RytkU9Xxay65YwQ= Received: from SJ0PR03CA0017.namprd03.prod.outlook.com (2603:10b6:a03:33a::22) by DS0PR12MB8814.namprd12.prod.outlook.com (2603:10b6:8:14e::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.48.17; Tue, 2 Jun 2026 05:26:07 +0000 Received: from CO1PEPF00012E84.namprd03.prod.outlook.com (2603:10b6:a03:33a:cafe::49) by SJ0PR03CA0017.outlook.office365.com (2603:10b6:a03:33a::22) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.71.16 via Frontend Transport; Tue, 2 Jun 2026 05:26:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by CO1PEPF00012E84.mail.protection.outlook.com (10.167.249.59) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Tue, 2 Jun 2026 05:26:07 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Tue, 2 Jun 2026 00:26:01 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Aaron Lu , Josh Don , K Prateek Nayak , Subject: [PATCH v2 4/5] sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() Date: Tue, 2 Jun 2026 05:25:30 +0000 Message-ID: <20260602052531.11450-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260602050005.11160-1-kprateek.nayak@amd.com> References: <20260602050005.11160-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF00012E84:EE_|DS0PR12MB8814:EE_ X-MS-Office365-Filtering-Correlation-Id: 220c924b-e6bb-4782-b2ce-08dec06772b7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|1800799024|36860700016|7416014|11063799006|56012099006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: jSXpNPR34CF9WgwdO3TbcoHm91kf/67xoq/s4cJnI8lMqIjKvDEGndM+Q2AUgUuQyS87uoZqT1DHVmUP5nrA6lSpIiq9hFPg4JToJJE2q/zabJ6cT2+fr/RyxuVF5Z0bv+i+JMKR2EZQ/l0YH6eH6ZI87kyWZZ9rY0Zx2CbuqaHFSDCWPEsvOToP43Lsgl2bT7CLCfhdKS9g1ZWiuJZU3kAIWVNeHyeG65LKr6Rhl3983mjAWWWhklp5d72h8foM8d8wWst/6WTJvzudpmgO4xJXNlFp8m9SaKv9wqag/JuXKEXofvZPh1ROMWYyZoi2NnWCdzRJ9MEF7O+FbDc+ZBromWNygPkS2YRPEKZgdiXOpYKzqYwnmaoPRpqWtWt8jZ1TrNBh8Cd9SwMoySPKVF3oBe6jlmiWE9aIiXSCMphSDhpSC9yDYjQGKoh5v8W1Q9WcEgjVzjH9p394i3xEdCYVFSq2ozUJbnnwtryHasCcJ1E+ZFtjSCu4QAAUf79fohMqJKrxO+P4Nx8Ls9z1K2zffY1LSzJIIEwEgKrh93OTRwBfW3tIcpEcMOewIu154QHXbj4jeUXX0xAw9J8Qxnq3DasQtnhmehA2wKPhlG6qphWlTh8HWbq+Gy498g1YkjAXLsujAWNxkdMcWwtOH3m/LYtPaOvzEKK8Tdoqcz/smzBS1s9TuXIfTKU5gz1tUPVflx71LG3RnKy7XFYeAfy4OreBxnn/cXOgpX0XtkU= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(1800799024)(36860700016)(7416014)(11063799006)(56012099006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: K13yZvaIjzIOJhHAHikIDmwGPsIs+hUPixWEPI3/ZWaGyjdNvwy3X/w6tirhW0+lGNWJmiJnQNynvzD0ITGOQ++YZoBAqgWlSHgUnUUQcqWfqvtbUWAod4D96w73ANoUSMF2gXE+XhynmFOiw22Jnnp6uPPFw9Mkk7Tbbg81CUFVknogj39GK/9WXTuAPJvRrbqNZVf8XMNzayiqwRr1LMUXyUEWWKyjYeWR5m0iuJBxVX2tPcOfmbqHu5LxEDCDt7J3aRSao5BpxWGUpdSukSnEg3TjMfTIC0nJOAv/1ZB2vhObqshQclEol43zkqLK+lJzTDfI7nWeohZGvpMEbI3SKFFZWQxrI2Ifd4AiJGHehn/mqrQxgzCf7oFKIW7mltu97lcDQq+GIC+OgRda5A2U/uFpt2oj3Egq8pYX0DruYykkLGfzpZ//6OqHYTRk X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2026 05:26:07.0654 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 220c924b-e6bb-4782-b2ce-08dec06772b7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF00012E84.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8814 An update_curr() during the enqueue of throttled task will start throttling the hierarchy from subsequent commit. This can lead to tg_throttle_down() seeing non-empty throttled_limbo_list for the cfs_rq attaching the task from throttled_limbo_list one by one. For example: R | A / \ *B C | rq->curr *B is throttled with tasks on hte limbo list. When the tasks are unthrottled via tg_unthrottle_up() and entity of group B is placed onto A, update_curr() is called to catch up the vruntime and it may throttle group A causing the subsequent tg_throttle_down() to see the pending task's on B's limbo list. tg_unthrottle_up() /* --cfs_rq->throttle_count == 0 */ list_for_each_entry_safe(p, cfs_rq->throttled_limbo_list) enqueue_task_fair() enqueue_entity(se /* B->se */) update_curr(cfs_rq /* A->gcfs_rq */) account_cfs_rq_runtime(cfs_rq) throttle_cfs_rq(cfs_rq /* A->gcfs_rq */ ) tg_throttle_down() /* Reaches B->cfs_rq with throttle_count == 0 */ !!! !list_empty(&cfs_rq->throttled_limbo_list)) !!! Move the tasks from throttled_limbo_list onto a local list before starting the unthrottle to prevent the splat described above. If the hierarchy is throttled again in middle of an unthrottle, put the pending tasks back onto the limbo list to prevent running them unnecessarily. Reviewed-by: Benjamin Segall Signed-off-by: K Prateek Nayak Tested-by: Aaron Lu --- changelog v1..v2: o Collected tags from Ben, and Aaron (Thanks a ton!) --- kernel/sched/fair.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 16e0ccaffe6e..8169bcedcfc8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6710,6 +6710,7 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) struct rq *rq = data; struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq)); struct task_struct *p, *tmp; + LIST_HEAD(throttled_tasks); /* * If cfs_rq->curr is set, the cfs_rq might not have caught up @@ -6740,13 +6741,31 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) cfs_rq->throttled_clock_self_time += delta; } + /* + * Move the tasks to a local list since an update_curr() during + * enqueue_task_fair() can throttle a higher cfs_rq, and it can + * see the "throttled_limbo_list" being non-empty in + * tg_throttle_down() if throttle_count turned 0 above. + */ + list_splice_init(&cfs_rq->throttled_limbo_list, &throttled_tasks); + /* Re-enqueue the tasks that have been throttled at this level. */ - list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) { + list_for_each_entry_safe(p, tmp, &throttled_tasks, throttle_node) { + /* + * Back to being throttled! Break out and put the remaining + * tasks back onto the limbo_list to prevent running them + * unnecessarily. + */ + if (cfs_rq->throttle_count) + break; + list_del_init(&p->throttle_node); p->throttled = false; - enqueue_task_fair(rq_of(cfs_rq), p, ENQUEUE_WAKEUP); + enqueue_task_fair(rq, p, ENQUEUE_WAKEUP); } + list_splice(&throttled_tasks, &cfs_rq->throttled_limbo_list); + /* Add cfs_rq with load or one or more already running entities to the list */ if (!cfs_rq_is_decayed(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); -- 2.34.1