From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SA9PR02CU001.outbound.protection.outlook.com (mail-southcentralusazon11013003.outbound.protection.outlook.com [40.93.196.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E8E23A83A8 for ; Tue, 2 Jun 2026 07:10:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.196.3 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780384230; cv=fail; b=B8KgmZjUp+Ay9RGswj4rciWAv0MuyAL1hS4vYCstQUxG4siQ4AAVLeicmDSwhCr8cOON2H7yUkezC1NFQtDt/EBKPGslFtnXXlAsnC2K1noZBe/H+Yo1sHzEUTq0MmyHsI+HWCxjhdrcUCC8Eqbv6mnBCnNdxaYSWQ5ElT3aq74= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780384230; c=relaxed/simple; bh=ng0/aFG2g+AXLfrXtceqgmnizcS9JiKRdmDeI7K9cFI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=UAPea1jUXERhqiCynoovBlt3IYXoDrMFHmlpSKvKaFsjrHfSLcZ2E7IFYTEP5mCluFm9VqKAUEfNZTnGMfEKyn54UDiPQb2XFKJb1BsM7ZzWuBG8zhw+5b/i7LUpspvEQnfvdcdv1X1YiuP+JDX1JyR9sRZtyJK/VwxsXHNSXR0= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=vpi0HWRL; arc=fail smtp.client-ip=40.93.196.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="vpi0HWRL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HmcPLCF575tCFggiuMLAb0I3TlRFkIO8iK9e3/rfOvi53089XS1hClBiIEYo7eS4W7eypP1ViuQZkGeHl/K/OkAAbmRT9Z7AcoaZxw5dW+9Innq6PrLS6ZjBvRzkrxUiCrgtbyjs4eIDFf8uQubmZeyEU0xkOTk5dfP4JZR4AI8FtJLrdX7nhAvuqwprL6dEFNy0PRjND1s1bwzzw1WBKS/nzzP0BWEwZpFlz+RNavER/Xg7MCCd8MhblLJ/lY0AwUjpxYR+Z/mrryjovNPkiDmlaUABm/tYeyATyj9t2rmUpwvJxbYC+r2fLGCvNVxW0GcuxQS5he3d4zB4Hb5Smw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SVSR5MzDajgcpuGpigE0OodkptlWHUJb1MS+JHDjQ7I=; b=Mf2WUP3qA35Ho7ekaSAavctf6EVbN8Pt8ctIWRb3a/HhwBdpstwfO/ngdhBQVcE2YupfpclIGierorioftqjlImRLbWtKKBrHl4mX/KC48A5hzf6MxYE23zEw+OKHr+GymF2jdKyfbbWXeTgSjPwqlFQP8OchFXTW0IeGKA7kYodWMFEzSGoYgjEXJM67pNGp3jIONXETWcWTLIBHZ5SG7sZ3ipAyaG6PHwLTfhtaOFHpEwUq3HT5S8lBKXa+KSVP4FV7LAGxtBYwTJDgLqCh2kBn9UMr1zX2Jgo84+oxVYOJ4h2fDv3r7ImWCgxKPf0J8PV4cDBqcOCgUDjXMWlRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SVSR5MzDajgcpuGpigE0OodkptlWHUJb1MS+JHDjQ7I=; b=vpi0HWRLytd6tFY6R2lMDHcFyVujEr0+fip9L+xP0dW+XQ9+f6Hxev60ZyjGe1nX5Dmv4Hy3jiHXwQUXgk++Y3pyPdby1O+UG3MsxZb4L8POuwO8vyF+9E8Nlb8N0IBYjO1vbjYSmxXEYTPn3Qd2jMUAynjmSt+zu1As1vwX100= Received: from MN2PR16CA0061.namprd16.prod.outlook.com (2603:10b6:208:234::30) by IA4PR12MB9809.namprd12.prod.outlook.com (2603:10b6:208:54f::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.17; Tue, 2 Jun 2026 07:10:23 +0000 Received: from BL02EPF0001A0FC.namprd03.prod.outlook.com (2603:10b6:208:234:cafe::97) by MN2PR16CA0061.outlook.office365.com (2603:10b6:208:234::30) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.71.16 via Frontend Transport; Tue, 2 Jun 2026 07:10:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A0FC.mail.protection.outlook.com (10.167.242.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Tue, 2 Jun 2026 07:10:23 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Tue, 2 Jun 2026 02:10:19 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Aaron Lu , Josh Don , K Prateek Nayak , Subject: [PATCH v2.1 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() Date: Tue, 2 Jun 2026 07:10:05 +0000 Message-ID: <20260602071005.11942-1-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260602052531.11450-3-kprateek.nayak@amd.com> References: <20260602052531.11450-3-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FC:EE_|IA4PR12MB9809:EE_ X-MS-Office365-Filtering-Correlation-Id: 4e2f8863-2ad8-412a-ceb0-08dec07603bf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700016|82310400026|376014|7416014|13003099007|18002099003|22082099003|56012099006|11063799006; X-Microsoft-Antispam-Message-Info: 0M5s4sKWW5RlVAR1Y6bOZVf5yFnSSfbdw6EHztEmc+hPqjghZop9eSrGoVioguMD/QEsNjXTsKhvy6F3esjuiphRwz3sIAO7Qy9B7xP6TxoqyU86Kf4sUbCMUgOrkHUzDBEyx7yBvUomyrBgQxriznaWdMrh5THSnKk6O6mAFgwGplM3P2NJbNOXQY68zbhRT18kMEDOiUEi+B3qmRzweYRDbYmFHAiKM1eVVn/wrBTX+cNG26vpJ4Gl2Caw3KzpIZtuWaYO0le10x91qaH/vR4NI0dUpTTVaiEW+nMdRDu6jtL3N/uSXNE6qHwgBfWzvXbl765DnFpOAuhvbeU/QeHq4CUdBgUAKQm1j5P6h/6+HCtRYmMcuyhCfkhsHOdNyOqRm+f2L5q5dSNiaoc0fk1V2fGzowLmcPe7zpM9363Qd9jbzjYEXYvTuGQu1cQn2Xrdm71W708ZBTZcDDII5UdFZgA9pma66Q4kyfDB1b9OMuTkS+L0e28MTBK3gSW45w/0UwLrSK8WIeGxSCR6/w86VPpG7IMWyewoZSByxx72kXPRKBoPQyCtaDmTCrzw+es5+Ce+P8c5lUusabGoNH/iZElyjBUEHG0VHsjDOuJYR/kU+CGL7cQAF1tSENTwW8hLjjfBOXpqL5OzCNvCZEa88kqccTwE/Ql5c+SBFkiWxTKHP8AOtQzef1Q2b7CjyD6vMboNnmjkU5dsiK5UmD4gLxUJm/dgnZK8axFFPJ8= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700016)(82310400026)(376014)(7416014)(13003099007)(18002099003)(22082099003)(56012099006)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PcTClYtOgI9teBgC0vnHrGFtmtWUFa7lflt6u1TEfUDFrJYS7JLPov4kBEsdeSR+6mn8x7uA/SXdvvaBEUjo2XtZTgjvcKCtKg0BDEM5rP5jgOUttG4TTOnkz70cab72KIWfOmHs7LAfaPVGLOzE7JCZzZMx3jJYWw4o0QDUdd3fYjjbHiihGPFM1XqAKnFC+u2D8NEzxftELrVp399a4DnIv3X5nZjIenYvo++rNG08V5NTKe1eYezRse60o1wwusngLYauh6t6BiepaE95QifufRlu6q+keTurU6wZOok7r02AkAuumdBhp7yRddloxi0Nt4+/ENYVeCHMFZuh5iwHWBR1hXSETzPizQ/lwrnjeHhxTyl08VRly4JTbvmceUUSW+BdFivbsbJ10NSo3m5MWuJkBktJZ+WXamw2VcgUUFfCNC/gKsToKJpUfhqb X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2026 07:10:23.4420 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4e2f8863-2ad8-412a-ceb0-08dec07603bf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FC.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA4PR12MB9809 From: Peter Zijlstra assign_cfs_rq_runtime() during update_curr() sets the resched indicator and relies on check_cfs_rq_runtime() during pick_next_task() / put_prev_entity() to throttle the hierarchy once current task is preempted / blocks. Per-task throttle, on the other hand, uses throttle_cfs_rq() to simply propagate the throttle signals, and then relies on task work to individually throttle the runnable tasks on their way out to the userspace. Remove check_cfs_rq_runtime() and unify throttling into account_cfs_rq_runtime() which only sets the cfs_rq->throttled, cfs_rq->throttle_count indicators via throttle_cfs_rq() and optionally adds the task work to the current task (donor) it is on the throttled hierarchy. throttle_cfs_rq() requests for sched_cfs_bandwidth_slice() worth of bandwidth for the current hierarchy that enable it to continue running uninterrupted when selected. For the rest, it requests a bare minimum of "1" to ensure some bandwidth is available and pass the "runtime_remaining > 0" checks once selected. For SCHED_PROXY_EXEC, a mutex holder cannot exit to userspace without dropping it first and the mutex_unlock() ensures proxy is stopped before the mutex handoff which preserves the current semantics for running a throttled task until it exits to the userspace even if it acts as a donor. [ prateek: rebased on tip, comments, commit message. ] Reviewed-By: Benjamin Segall Signed-off-by: Peter Zijlstra Signed-off-by: K Prateek Nayak Tested-by: Aaron Lu --- changelog v2..v2.1: o Folded in a the optimization to add task work only in set_next_task() as pointed out by Peter in https://lore.kernel.org/lkml/20260601134842.GP343181@noisy.programming.kicks-ass.net/ (Took me a while to wrap my head around it; Diff on top of v2 can be found on the same thread in https://lore.kernel.org/lkml/20cc9820-35b2-4d97-bbc2-84af56802dfd@amd.com/) --- kernel/sched/fair.c | 101 ++++++++++++++++++++------------------------ 1 file changed, 46 insertions(+), 55 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8169bcedcfc8..fa8c0b1a1cf1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -525,7 +525,7 @@ static int se_is_idle(struct sched_entity *se) #endif /* !CONFIG_FAIR_GROUP_SCHED */ static __always_inline -void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec); +bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec); /************************************************************** * Scheduling class tree data structure manipulation methods: @@ -6359,8 +6359,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect) return se; } -static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq); - static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) { /* @@ -6370,9 +6368,6 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) if (prev->on_rq) update_curr(cfs_rq); - /* throttle cfs_rqs exceeding runtime */ - check_cfs_rq_runtime(cfs_rq); - if (prev->on_rq) { update_stats_wait_start_fair(cfs_rq, prev); /* Put 'current' back into the tree. */ @@ -6507,41 +6502,32 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b, return cfs_rq->runtime_remaining > 0; } -/* returns 0 on failure to allocate runtime */ -static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq) -{ - struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); - - guard(raw_spinlock)(&cfs_b->lock); +static bool throttle_cfs_rq(struct cfs_rq *cfs_rq); - return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice()); -} - -static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) +static bool __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { /* dock delta_exec before expiring quota (as it could span periods) */ cfs_rq->runtime_remaining -= delta_exec; if (likely(cfs_rq->runtime_remaining > 0)) - return; + return false; if (cfs_rq->throttled) - return; + return true; /* - * if we're unable to extend our runtime we resched so that the active - * hierarchy can be throttled + * throttle_cfs_rq() will try to extend the runtime first + * before throttling the hierarchy. */ - if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr)) - resched_curr(rq_of(cfs_rq)); + return throttle_cfs_rq(cfs_rq); } static __always_inline -void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) +bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { if (!cfs_bandwidth_used() || !cfs_rq->runtime_enabled) - return; + return false; - __account_cfs_rq_runtime(cfs_rq, delta_exec); + return __account_cfs_rq_runtime(cfs_rq, delta_exec); } static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq) @@ -6829,10 +6815,24 @@ static int tg_throttle_down(struct task_group *tg, void *data) static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { - struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); + struct sched_entity *curr = cfs_rq->curr; + struct rq *rq = rq_of(cfs_rq); scoped_guard(raw_spinlock, &cfs_b->lock) { + u64 target_runtime = 1; + + /* + * If cfs_rq->curr is still runnable, we are here from an + * update_curr(). Request sysctl_sched_cfs_bandwidth_slice + * worth of bandwidth to continue running. + * + * If the curr is not runnable, just request enough bandwidth + * to be runnable next time the pick selects this cfs_rq. + */ + if (curr && curr->on_rq) + target_runtime = sched_cfs_bandwidth_slice(); + /* * Check if We have raced with bandwidth becoming available. If * we actually throttled the timer might not unthrottle us for @@ -6843,7 +6843,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) * * This will start the period timer if necessary. */ - if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) + if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, target_runtime)) return false; /* @@ -6864,6 +6864,17 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) */ cfs_rq->throttled = 1; WARN_ON_ONCE(cfs_rq->throttled_clock); + + /* + * If current hierarchy was throttled, add throttle work to the + * current donor. In case of proxy-execution, the execution + * context cannot exit to the userspace while holding a mutex + * and the rule of throttle deferral to only throttle the + * throttled context at exit to userspace is still preserved. + */ + if (curr && curr->on_rq) + task_throttle_setup_work(rq->donor); + return true; } @@ -7254,7 +7265,7 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq) if (!cfs_bandwidth_used()) return; - /* an active group must be handled by the update_curr()->put() path */ + /* an active group must be handled by the update_curr() path */ if (!cfs_rq->runtime_enabled || cfs_rq->curr) return; @@ -7264,8 +7275,6 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq) /* update runtime allocation */ account_cfs_rq_runtime(cfs_rq, 0); - if (cfs_rq->runtime_remaining <= 0) - throttle_cfs_rq(cfs_rq); } static void sync_throttle(struct task_group *tg, int cpu) @@ -7295,25 +7304,6 @@ static void sync_throttle(struct task_group *tg, int cpu) cfs_rq->pelt_clock_throttled = 1; } -/* conditionally throttle active cfs_rq's from put_prev_entity() */ -static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) -{ - if (!cfs_bandwidth_used()) - return false; - - if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0)) - return false; - - /* - * it's possible for a throttled entity to be forced into a running - * state (e.g. set_curr_task), in this case we're finished. - */ - if (cfs_rq_throttled(cfs_rq)) - return true; - - return throttle_cfs_rq(cfs_rq); -} - static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer) { struct cfs_bandwidth *cfs_b = @@ -7567,8 +7557,7 @@ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p) #else /* !CONFIG_CFS_BANDWIDTH: */ -static void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {} -static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) { return false; } +static bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { return false; } static void check_enqueue_throttle(struct cfs_rq *cfs_rq) {} static inline void sync_throttle(struct task_group *tg, int cpu) {} static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {} @@ -9904,8 +9893,6 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) if (cfs_rq->curr && cfs_rq->curr->on_rq) update_curr(cfs_rq); - throttled |= check_cfs_rq_runtime(cfs_rq); - se = pick_next_entity(rq, cfs_rq, true); if (!se) goto again; @@ -14823,8 +14810,8 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {} */ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued) { - struct cfs_rq *cfs_rq; struct sched_entity *se = &curr->se; + struct cfs_rq *cfs_rq; for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); @@ -15006,6 +14993,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p) static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) { struct sched_entity *se = &p->se; + bool throttled = false; for_each_sched_entity(se) { struct cfs_rq *cfs_rq = cfs_rq_of(se); @@ -15016,9 +15004,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) set_next_entity(cfs_rq, se, first); /* ensure bandwidth has been allocated on our new cfs_rq */ - account_cfs_rq_runtime(cfs_rq, 0); + throttled |= account_cfs_rq_runtime(cfs_rq, 0); } + if (throttled) + task_throttle_setup_work(p); + se = &p->se; if (task_on_rq_queued(p)) { -- 2.34.1