From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010036.outbound.protection.outlook.com [52.101.56.36]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5695C2AE7A for ; Tue, 2 Jun 2026 05:26:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.36 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780377994; cv=fail; b=C1Az2+9ZdqKRydlDOGF6pteT8Jk1NXuLw78R0Zy1+xF1CywozWqz9yIBqbqPZSHUKpwSrGFEp4UDY1G1VRL3/iJobwxASTpaFiOJMzKPNoWanVFBWK9XcD04iAn0ORshYtlD2C06MA2ADO2Y/eTFX1XL/WLGf/KnpQThHGnByj8= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780377994; c=relaxed/simple; bh=wHajlFojkKJrtMWosKpRFbUkLkoQrb12v67+H4W7ucQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FZ24cI/mCCJyhltbuO83ENxabUIOhUg7C70gJ6mTDpSM7Wp79jk7pXy5k7uDTwVD3myn9YMU9e4DvOz05yNT0WM+yRfBG8k0DNVF3+ZXtN0LarZI2nih4AlRN3JyOI5FFwc7aQ10VPxMScaWbIpu5Oo+GechhEHVJRSklHJObxY= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=j6W5P+rT; arc=fail smtp.client-ip=52.101.56.36 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="j6W5P+rT" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=X1/RcUXhrx9N8vEFELiWcbECHgAi5KcLXRFDfUK4MzAIW4KSlhbg2E23V3adQorM8ev5folDZFsfkByHBiVNTcqzvZv+4SQhTEI7sZGfw+bqkKcJ5I9Uyac6q1N8CBjirIwld5zNP6ntCgOmHiPyzKF6NFCq6oaw27NHIJnrF3c/c40w+HeLMaQGSBNpWiDuatGVsnuLhGn//Ez+XBKizp2+zuonfzGjU1BdHdhkud0X3zB1GGQRNZdpzMwHTOw5AsLC4qlEd6sUYcR6e/bZ4XYS2fG8niySour5nCa4HSZoPlmUh696+C6XuuvtvwcpojSbVlEwictI+bpOjcAe1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QMhkIoyvVPigGjZ9jafZqbwOyR5cSUyQz69+fpeLwWc=; b=izbHu8rcnPI4ZOjshmbI8B3xcQ11mhTz3Du7wHx/P60NxEb1Lcrpy9bx0uW6KUqxs2Sifn75cGEMi/3KPFgEsMchyFyNTPU7klgaR7qdRfJRX0/YhyizoCJAjDlybNnTJwXNSIWP7BzrBFf6ISyNxb/GFij25swvGE0PMPCEGA0PODV5V6h1toLLjfDdzKJVhT79qv1RNvmbMJBvvjX0skSgyZe8IzINWSuRCltqXE4ccF8RrYhT5cih3LG/2KTQSKG7n2BJudSwDDueUlpNtwtjg0OYdlcDRoaHXGfH80Zx5ajZtUiamObQPtQX++GjBf+lIHP+DPczwrRx1QAFGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QMhkIoyvVPigGjZ9jafZqbwOyR5cSUyQz69+fpeLwWc=; b=j6W5P+rTZICBO0Yl+vzZfd5Uvl+L5UgHxVyHoJFMgFHjMdATUBqMahmuMYhusZCZIJn+kEPW2otbrm+UE8UAxCHi2szcrhJ2EWzuHN5hwEpguDRDXMg2EW09xQzwTZquOwGuf/ybZHLOFnkKLxvVtDBQV87ikqg0TIHIXRiGA60= Received: from SJ0PR03CA0019.namprd03.prod.outlook.com (2603:10b6:a03:33a::24) by CYYPR12MB8938.namprd12.prod.outlook.com (2603:10b6:930:c7::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.16; Tue, 2 Jun 2026 05:26:24 +0000 Received: from CO1PEPF00012E84.namprd03.prod.outlook.com (2603:10b6:a03:33a:cafe::4e) by SJ0PR03CA0019.outlook.office365.com (2603:10b6:a03:33a::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.71.17 via Frontend Transport; Tue, 2 Jun 2026 05:26:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by CO1PEPF00012E84.mail.protection.outlook.com (10.167.249.59) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Tue, 2 Jun 2026 05:26:24 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Tue, 2 Jun 2026 00:26:19 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Aaron Lu , Josh Don , K Prateek Nayak , Subject: [PATCH v2 5/5] sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() Date: Tue, 2 Jun 2026 05:25:31 +0000 Message-ID: <20260602052531.11450-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260602050005.11160-1-kprateek.nayak@amd.com> References: <20260602050005.11160-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF00012E84:EE_|CYYPR12MB8938:EE_ X-MS-Office365-Filtering-Correlation-Id: 701e699a-47d9-4132-de52-08dec0677d11 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700016|1800799024|82310400026|376014|22082099003|18002099003|56012099006|11063799006; X-Microsoft-Antispam-Message-Info: DkGilhtMyBpEiJseZDToqdRz+U3uADFfsLewPHsvZDnPEB71EJob/nT2GYGB3KJ2MbUEQFIlzlFwSAQpcCoRRz7DlyUwLpD75ny1CQCKxujX/UjCZij/Ws0u3xB57sfYYQRTStxXNOFHODXY87V1EmrpHDr1X2vHE6wYxR4xPDHfKloyhrhP1vjolpW3ptuVOIHvf94FLXkYA95JNKdDRrgoUCAvefsrvviVUefhGLj99JSh+ZXHA08bj0aIULqav+0r/JlQe/yuRyfrxnb1ZweSXw4E7Y3bdPvNrRdkZXeNlyRJafi8zqH1JCcBZ8lJb73ep0sYr2/3m2NATjVk3SV3oJh4wNWfHuT2VSuQhY2QBXsvMkeMLMJsjev9qscIEAqamTiZL4CafifVEgrs4wCqfKKbyNMxlQtcly7iD4BrZ1VJrUdrDLMmGimCuKjjhny6TemFzwCglRxBsRD3gEKN1qbihrw+NkrEBqnkdMeCtBMoWDbIK57wzr46tIqczSE0YlSVY3dqW2z4Okc/abiJAUapgvd437Xohi9VUregPgt1TL+X/rHEPHsFMdspFijXk1dfcj42cvK239JEWm1z3KcaA2KV/2BcPQJh31hgSosWuhvqjRnirqqAVYUba6Wn3YsZ4XSROrXlBGqqvD/Oew1iRlMpiyIu5two4fGIrbYd5zTLkykVulrYQiZ+qMQI52IJ37Oi13rYfkbRZoaoBY3sg3n0fW85vCtbeKU= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(36860700016)(1800799024)(82310400026)(376014)(22082099003)(18002099003)(56012099006)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 6VO53X8ZDJqgkggQKPMmmVBlLYGeGCbvB/YLCTvB2K9nAM6qNioXQ5EYGxE9ovKWM93pMKgopTWrksQeWq33sUkK/m/eRkgGJwhvRYwexq21/iVrE7FWDtlRkavMLd8gUh4pmG1x+XrckgdDMxmxTsMM1ZCZMlGrKLkmS2OL9Hi2/ZS/MFKz4rFmPCNMJifyTPGZWd4Nnh6UdgElENfvYtRQ6nBaP75L3Zm35EDFIAWNAb5pMEEWN7CayFQIaIIGKL68/ajWg9LE3QxaqcLf+ET0qzDQzttSXQRFZw7ChmdZwZt0XUUSB2gmVLbOWvlPh96sCOVdHYWOnLaV59zDGgNZoPxNxsI7OBvFoEHbzZrK8DPcaOHHmPEm86nht9t+OyYa+yOKFuNqMuDr08/FYq/N4i0+0CtDASGSEOoZMta7oTGtbolht7d4A0BflvFd X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2026 05:26:24.4344 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 701e699a-47d9-4132-de52-08dec0677d11 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF00012E84.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8938 From: Peter Zijlstra assign_cfs_rq_runtime() during update_curr() sets the resched indicator and relies on check_cfs_rq_runtime() during pick_next_task() / put_prev_entity() to throttle the hierarchy once current task is preempted / blocks. Per-task throttle, on the other hand, uses throttle_cfs_rq() to simply propagate the throttle signals, and then relies on task work to individually throttle the runnable tasks on their way out to the userspace. Remove check_cfs_rq_runtime() and unify throttling into account_cfs_rq_runtime() which only sets the cfs_rq->throttled, cfs_rq->throttle_count indicators via throttle_cfs_rq() and optionally adds the task work to the current task (donor) it is on the throttled hierarchy. throttle_cfs_rq() requests for sched_cfs_bandwidth_slice() worth of bandwidth for the current hierarchy that enable it to continue running uninterrupted when selected. For the rest, it requests a bare minimum of "1" to ensure some bandwidth is available and pass the "runtime_remaining > 0" checks once selected. For SCHED_PROXY_EXEC, a mutex holder cannot exit to userspace without dropping it first and the mutex_unlock() ensures proxy is stopped before the mutex handoff which preserves the current semantics for running a throttled task until it exits to the userspace even if it acts as a donor. [ prateek: rebased on tip, comments, commit message. ] Reviewed-by: Benjamin Segall Signed-off-by: Peter Zijlstra Signed-off-by: K Prateek Nayak Tested-by: Aaron Lu --- changelog v1..v2: o Added S-o-b from Peter matching queue:sched/core. o Do account_cfs_rq_runtime() selectively in set_next_task_fair() on "!first" since pick would have ensured sufficient bandwidth is available if set_next_task_fair() is called after pick which is more common after the recent pick_next_task removal. (Peter) o Collected tags from Ben, and Aaron (Thanks a ton!) --- kernel/sched/fair.c | 116 +++++++++++++++++++++++--------------------- 1 file changed, 60 insertions(+), 56 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8169bcedcfc8..ce5cf494b934 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -525,7 +525,7 @@ static int se_is_idle(struct sched_entity *se) #endif /* !CONFIG_FAIR_GROUP_SCHED */ static __always_inline -void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec); +bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec); /************************************************************** * Scheduling class tree data structure manipulation methods: @@ -6359,8 +6359,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect) return se; } -static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq); - static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) { /* @@ -6370,9 +6368,6 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) if (prev->on_rq) update_curr(cfs_rq); - /* throttle cfs_rqs exceeding runtime */ - check_cfs_rq_runtime(cfs_rq); - if (prev->on_rq) { update_stats_wait_start_fair(cfs_rq, prev); /* Put 'current' back into the tree. */ @@ -6507,41 +6502,32 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b, return cfs_rq->runtime_remaining > 0; } -/* returns 0 on failure to allocate runtime */ -static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq) -{ - struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); - - guard(raw_spinlock)(&cfs_b->lock); - - return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice()); -} +static bool throttle_cfs_rq(struct cfs_rq *cfs_rq); -static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) +static bool __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { /* dock delta_exec before expiring quota (as it could span periods) */ cfs_rq->runtime_remaining -= delta_exec; if (likely(cfs_rq->runtime_remaining > 0)) - return; + return false; if (cfs_rq->throttled) - return; + return true; /* - * if we're unable to extend our runtime we resched so that the active - * hierarchy can be throttled + * throttle_cfs_rq() will try to extend the runtime first + * before throttling the hierarchy. */ - if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr)) - resched_curr(rq_of(cfs_rq)); + return throttle_cfs_rq(cfs_rq); } static __always_inline -void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) +bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { if (!cfs_bandwidth_used() || !cfs_rq->runtime_enabled) - return; + return false; - __account_cfs_rq_runtime(cfs_rq, delta_exec); + return __account_cfs_rq_runtime(cfs_rq, delta_exec); } static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq) @@ -6829,10 +6815,24 @@ static int tg_throttle_down(struct task_group *tg, void *data) static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { - struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); + struct sched_entity *curr = cfs_rq->curr; + struct rq *rq = rq_of(cfs_rq); scoped_guard(raw_spinlock, &cfs_b->lock) { + u64 target_runtime = 1; + + /* + * If cfs_rq->curr is still runnable, we are here from an + * update_curr(). Request sysctl_sched_cfs_bandwidth_slice + * worth of bandwidth to continue running. + * + * If the curr is not runnable, just request enough bandwidth + * to be runnable next time the pick selects this cfs_rq. + */ + if (curr && curr->on_rq) + target_runtime = sched_cfs_bandwidth_slice(); + /* * Check if We have raced with bandwidth becoming available. If * we actually throttled the timer might not unthrottle us for @@ -6843,7 +6843,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) * * This will start the period timer if necessary. */ - if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) + if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, target_runtime)) return false; /* @@ -6864,6 +6864,17 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) */ cfs_rq->throttled = 1; WARN_ON_ONCE(cfs_rq->throttled_clock); + + /* + * If current hierarchy was throttled, add throttle work to the + * current donor. In case of proxy-execution, the execution + * context cannot exit to the userspace while holding a mutex + * and the rule of throttle deferral to only throttle the + * throttled context at exit to userspace is still preserved. + */ + if (curr && curr->on_rq) + task_throttle_setup_work(rq->donor); + return true; } @@ -7254,7 +7265,7 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq) if (!cfs_bandwidth_used()) return; - /* an active group must be handled by the update_curr()->put() path */ + /* an active group must be handled by the update_curr() path */ if (!cfs_rq->runtime_enabled || cfs_rq->curr) return; @@ -7264,8 +7275,6 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq) /* update runtime allocation */ account_cfs_rq_runtime(cfs_rq, 0); - if (cfs_rq->runtime_remaining <= 0) - throttle_cfs_rq(cfs_rq); } static void sync_throttle(struct task_group *tg, int cpu) @@ -7295,25 +7304,6 @@ static void sync_throttle(struct task_group *tg, int cpu) cfs_rq->pelt_clock_throttled = 1; } -/* conditionally throttle active cfs_rq's from put_prev_entity() */ -static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) -{ - if (!cfs_bandwidth_used()) - return false; - - if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0)) - return false; - - /* - * it's possible for a throttled entity to be forced into a running - * state (e.g. set_curr_task), in this case we're finished. - */ - if (cfs_rq_throttled(cfs_rq)) - return true; - - return throttle_cfs_rq(cfs_rq); -} - static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer) { struct cfs_bandwidth *cfs_b = @@ -7567,8 +7557,7 @@ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p) #else /* !CONFIG_CFS_BANDWIDTH: */ -static void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {} -static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) { return false; } +static bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { return false; } static void check_enqueue_throttle(struct cfs_rq *cfs_rq) {} static inline void sync_throttle(struct task_group *tg, int cpu) {} static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {} @@ -9903,8 +9892,15 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) /* Might not have done put_prev_entity() */ if (cfs_rq->curr && cfs_rq->curr->on_rq) update_curr(cfs_rq); - - throttled |= check_cfs_rq_runtime(cfs_rq); + /* + * For the current hierarchy, update_curr() above would + * have set the throttle indicators if the cfs_rq has + * run out of bandwidth. For others, enqueue / last + * update_curr() for the cfs_rq would have ensured the + * throttle indicators are set if bandwidth was not + * available. + */ + throttled |= cfs_rq_throttled(cfs_rq); se = pick_next_entity(rq, cfs_rq, true); if (!se) @@ -14823,8 +14819,8 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {} */ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued) { - struct cfs_rq *cfs_rq; struct sched_entity *se = &curr->se; + struct cfs_rq *cfs_rq; for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); @@ -15006,6 +15002,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p) static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) { struct sched_entity *se = &p->se; + bool throttled = false; for_each_sched_entity(se) { struct cfs_rq *cfs_rq = cfs_rq_of(se); @@ -15015,10 +15012,17 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) break; set_next_entity(cfs_rq, se, first); - /* ensure bandwidth has been allocated on our new cfs_rq */ - account_cfs_rq_runtime(cfs_rq, 0); + /* + * Ensure bandwidth has been allocated on our new cfs_rq + * if we've reached here for reasons other than pick. + */ + if (!first) + throttled |= account_cfs_rq_runtime(cfs_rq, 0); } + if (throttled) + task_throttle_setup_work(p); + se = &p->se; if (task_on_rq_queued(p)) { -- 2.34.1