From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azon11010006.outbound.protection.outlook.com [52.101.46.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 878931DC1AB for ; Tue, 2 Jun 2026 05:00:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.46.6 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780376435; cv=fail; b=FBwBFzIMBvgNTHDaSdP6gKsRfSOm7lwg4X85Tk+VsSANYY9AU+zovY77GF5jIyMpAhg9bSHnYLGtZi2kEYGvUjsfw+meuZG9WQVYYAiwSyzxv7GyRgO1nFjQb/hmItA3MAIjF7SzurLVjXSSIzDl2BYfppdkZ9nO5bHxHOGQuJA= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780376435; c=relaxed/simple; bh=g8dJiXGzP/tuuV2mxfwb78d2AKGeh/UV4wjfIfh1ia8=; h=From:To:CC:Subject:Date:Message-ID:MIME-Version:Content-Type; b=ph+Gc4+rUdMXvU2Kr/zYIdHMyiKyDW8INJzh3ui/+rtLaTW7g9SlSxgbmYOFLcx/v51FWBwkKLqI6vg7yci3tGFqxFR9OGiE/2LcNFJ8Z8uS0gk+xJTVgc9JFhegoqzNqNkD71EIV9VQHj8iKstg9fdzBrFkg13LzXZbylMq7aU= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=XnZlME2F; arc=fail smtp.client-ip=52.101.46.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="XnZlME2F" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bmnGm2oeJ1Q/diKfRomK6iB8n7SmN5YcoiyJxA8I/VXx687hRX6DZU6lP559rWl4YlhU40CaRvP5REG0OuUGtw7mPfKzAiuCH48odA6Lt5kDfCU6iDJ48cM35PQMeM4tQ2WMqumdQRB7AwqWMkWo+pLL/I/8HHmcndU1Z0tSAJNKwN8Cm3sY3qwT8hXpMNc6Bkn7P/ZOCRypjR4eOfpxdHeUmUoMUNi9nkOuHe1hMISA0+awSFHwAYSzSJyb/hJWx/YNLEXMYI4gOJGGQ2D2jYY12Uv0K+7NHV3bS9walPIA/XSpBUfQT3rgFr66aoDl7/zZnHy5Jh1htdLFgkVbZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0qTeq6lt0mJVsRh0PQxrvIziExvbsCvNxmje/zwftq4=; b=WEWqeaXbT0iHk19ossNoA9GtF0bokBI1djIN6W/GvjmY+vUKN7Vl425GUn3TGushsARUQEUFQSXZIeud+FcU6refpYkkVzGfuazc2sSYGBZmXYBRZTUwXy7GXYou65chazqVMKkab0XhB52ecCj94bIyxf04SN0Njns4cfHgtLNyLp1hT4ERFWf8Mwr7owDO8A2ys+bAnbMF/kX1KD+Dgq0FMDIVLPDVAgIfJO3f4tRowXESE9BdaZvdSEhjodqQqs7QlT/IS+BL9gTt5WY3MRZ9DnJj6yLErD4+OsRexKrmqRqbDQzKebY5OMwRKFggH2B4Hz0XEmtF9A6vKgw30Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0qTeq6lt0mJVsRh0PQxrvIziExvbsCvNxmje/zwftq4=; b=XnZlME2FNdwryIk4t7p5Gs46hOdyidEkQwDuvPSz3qbcSq+ISEVsyFM2Rc2kzio4cvej1TD0wCNa3ehg+Pyp4okwBDCQaRPIvYZGdfr/1IOUFPnUGo1cT3j04txxeRUpvFm7sn53a+GUgRzR2UUm/z2e8Hd7Rf9xtF1h22EXpZM= Received: from MW4PR04CA0359.namprd04.prod.outlook.com (2603:10b6:303:8a::34) by CH3PR12MB9194.namprd12.prod.outlook.com (2603:10b6:610:19f::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.15; Tue, 2 Jun 2026 05:00:28 +0000 Received: from SJ5PEPF000001D5.namprd05.prod.outlook.com (2603:10b6:303:8a:cafe::93) by MW4PR04CA0359.outlook.office365.com (2603:10b6:303:8a::34) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.71.16 via Frontend Transport; Tue, 2 Jun 2026 05:00:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by SJ5PEPF000001D5.mail.protection.outlook.com (10.167.242.57) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Tue, 2 Jun 2026 05:00:27 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Tue, 2 Jun 2026 00:00:22 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Aaron Lu , Josh Don , K Prateek Nayak , Subject: [PATCH v2 0/5] sched/fair: Allow account_cfs_rq_runtime() to throttle current hierarchy Date: Tue, 2 Jun 2026 05:00:00 +0000 Message-ID: <20260602050005.11160-1-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001D5:EE_|CH3PR12MB9194:EE_ X-MS-Office365-Filtering-Correlation-Id: 13483922-fd84-4e82-bdfa-08dec063dd12 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|36860700016|82310400026|5023799004|11063799006|56012099006|18002099003|13003099007; X-Microsoft-Antispam-Message-Info: gx6se/WWCXBMhvLtIZPtDqeu6SHVxgnamb6VFjG62mPyH1uvbGA95YK1oJPHH7fQ0xRw4QCZTZW32fK/T9qo5VRFNpS757tB5AoMxmaR9UKkYTKh7BLpMnE4bCulxwvN9RIGY+u6LvEjDCGFw6lKx9todw/RykgmZgN6ZJkZfnFAjq42Lcjp0VQvUi37r4JeWymNb6E6xnrOy1P+E24cDSz+SjjvDYreolfV/rcd+CFI8UGNTTj67m/RXgxRWwZe6WFAviFMg6XaAb2sbpWII1QqALM617/s/fXUVmnEF5FPMkRiKpRId8sqOc+xyEd3YAJ/5Q5Md3w0HVRBmgd4A2KFIyHc3yzcookZqpOfczDM4ptc3RuEyUFz8XMT6a47C+62jWut8M3qr7wIinTjgy8P0bGXtmn3Ex8uRuJolysSHhfDNK49UAIYpC4XDDp/Pp5F1ZlPYZqUcLC3eYfbLwdiqM7KtfunU1rgSkaMR3Bpf+7GNvNBGh5fXo77pC0WdLsnZVVT/mlha0Etq9Q10nPciQ2/XpHehK9UmXrpwRYFHNySrBYL9BWv6IEu0jlkzIM4J8g2E9PDld+OWVPF055p2kGbDDe+Gy0AC+iiCdAmNIK2OjqNDdEK+ilPmnXNFXaT81UOO2wxt/JNbipfYXRjliWyzog4DYIfwA9y5slXWKEbqgKokkg3XTq10XHF X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(36860700016)(82310400026)(5023799004)(11063799006)(56012099006)(18002099003)(13003099007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: XUwbx2Fn99a57UhkVVXFNB5xPCRI8dMR1/e5PJHfGQ/TS40wMOjjsmEPXgDpZj4s/O8FI4SyojDzk9tvZ3CFu/81HRunphol7n0ykYeIIA8s+eg2EWOtOoqcaOGnFG0eaf8KbPdoT0C+10RRXijDeyIdtKtIIpIGuXaebeonzI0gMOsMcoCKWjHH7iykj///155wKQY1NV8QOcGZEXizUoIIa3e3ywjFK1HIB0x5Lw0f6je59Btty4zyDucMe234fhFMArM1XXedKXHpbjBBGdKgEXA84mdXUR62hvk2fNKPdC8PLOzVq7kr8r2Atc5iXlRHU8hAOPJjNQiGbdJeTPyMZDKKAVbYacIh+1x3GDiLb5iQC9zYYMaUELIiJ8yeMNzWnscm0IJYozvvBAnkQp2wHkkqy7vUQcL7wUBr1ODYV/LoPjXTvqGKvGcE1S8O X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2026 05:00:27.5124 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 13483922-fd84-4e82-bdfa-08dec063dd12 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001D5.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9194 v2 addresses comments from Ben, Aaron, and Peter with a copule of small optimizations on top. Individual patches contain the changelog. Introduction ============ The current hierarchy is always throttled in __schedule() during the pick when update_curr() detects a cfs_rq running out of the bandwidth and issues a resched. This was necessary prior to per-task throttling where the entire throttled hierarchy was dequeued at the point of first throttle during the pick but with per-task throttling, tasks continue to run as usual until they exit to userspace and dequeue themselves one-by-one until the hierarchy is deemed fully throttled and the PELT is frozen. throttle_cfs_rq() is now simply a propagator of throttle indicators and nothing more. Implementation ============== Unify the throttling for current hierarchy under account_cfs_rq_runtime() which is responsible for the time accounting. If the bandwidth runs out, account_cfs_rq_runtime() will request for sched_cfs_bandwidth_slice() and mark the hierarchy as throttled if it fails to grab bandwidth. throttle_cfs_rq() will do a task_throttle_setup_work() if it finds the current task to be on a throttled hierarchy and the task will naturally dequeue itself when it exits to the userspace without needing an explicit resched. First four patches are cleanups and preparation for the final bit that switches over to using account_cfs_rq_runtime() for throttling which was provided by Peter in [1]. Benchmarking ============ Following are the results of running hackbench running 3 levels deep with the setup from "Testing" section on [2] when compared to tip:sched/core: kernel : tip tip + series Min : 207.33 202.20 Max : 210.20 222.47 Median : 207.83 218.33 AMean : 208.29 215.36 GMean : 208.29 215.25 HMean : 208.29 215.13 AMean Stddev : 1.02 7.37 AMean CoefVar : 0.49 pct 3.42 pct All numbers are in seconds. There is a slight boot to boot variation for this benchmark but the utilization numbers in top is more or less similar between the two. Additional testing and feedback is always appreciated as usual :-) Patches on top of queue:sched/core at commit ce348f2b1998 ("sched/fair: Allocate cfs_tg_state with percpu allocator"). All testing was done on a dual socket 4th Generation EPYC system (2 x 128C/256T). CONFIG_CFS_BANDWIDTH=n was only build tested. Changelog ========= v1..v2: o Addressed comments from Ben in Patch1 to keep the call to distribute_cfs_runtime() in do_sched_cfs_slack_timer and used scoped_guard(). o Instead of adding back "else { throttled = true; }" in distribute_cfs_runtime(), invert the condition to "cfs_rq->runtime_remaining <= 0" and break early after setting "throttled = true". (Ben) o Added missing update_rq_clock() in tg_set_cfs_bandwidth(). (Aaron, Peter, Intel Test Robot) o Added an update_curr() in distribute_cfs_runtime() which was found to be beneficial to prevent calling unthrottle_cfs_rq() unnecessarily. o Collected tags from Ben and Aaron (Thanks a ton!). o Rebased patches on top of: git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/core at commit ce348f2b1998 ("sched/fair: Allocate cfs_tg_state with percpu allocator"). References ========== [1] https://lore.kernel.org/lkml/20260512110932.GB1889694@noisy.programming.kicks-ass.net/ [2] https://lore.kernel.org/lkml/20250220093257.9380-1-kprateek.nayak@amd.com/ [3] https://lore.kernel.org/lkml/20260522141623.600235-4-zli94@ncsu.edu/ --- K Prateek Nayak (4): sched/fair: Convert cfs bandwidth throttling to use guards sched/fair: Use throttled_csd_list for local unthrottle sched/fair: Call update_curr() before unthrottling the hierarchy sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() Peter Zijlstra (1): sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() kernel/sched/core.c | 5 +- kernel/sched/fair.c | 360 +++++++++++++++++++++++--------------------- 2 files changed, 193 insertions(+), 172 deletions(-) base-commit: ce348f2b1998b4c90053a3be407c32102b132800 -- 2.34.1