From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BL2PR02CU003.outbound.protection.outlook.com (mail-eastusazon11011007.outbound.protection.outlook.com [52.101.52.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C13A3B2FEB for ; Thu, 28 May 2026 09:49:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.52.7 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779961756; cv=fail; b=mQjupOHGy8xbsf35ijM1CCOuWkcQ0iuAr8kEJ5SKFqtseHJvTxFTKxxh4ie5km3J1NzZf87vNLjVQ0bwWUCF9/KVM1TAY4S/04S54kA0MA/hKPNc3V5YKhet70HZsh3MRzl0LXNwrnf9druUCCpXB41Az+9FVD9+/OL/cgqLtMo= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779961756; c=relaxed/simple; bh=kzGKE95Qc3mMA4tSyPV8kF0KvPYBNFNLKk8aLyIi0pQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kiDiJJFlrn8XymFoCu7j43CA5HRM6Tcb8b9Utz8Wy0qcCGDkEZMd2JYJhUEpenTNbXB2NbK1Rl2ZfDu7fxlvCFUOitDMtly3dVdmt8ar0N05hWgzT/7K10VUW0sr4i0H1teecOO2uTNYJJz8ARnDQIXMoZgK9Tm/GW8cLlqEMxE= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=onfYTM5T; arc=fail smtp.client-ip=52.101.52.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="onfYTM5T" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=J9SNDPZN9nveRfJCYGv1GtMTzo01SgHE7iRD10Lf3SRgzhZiy4HAmcUkuho6WTpBZfkc1SCi37ajc1PJezJ4zASJZpQsevy0pjuM5SzVkd7EjHiP6bflHDje+4qCxQMeK5ojqcINh1u6XZu1MQJB2JqCCNsgBT5qX68GMvtJdOdDe1Ar/7SGAYkUH1drbteikIjPFw3AvdUDmP3I4N4RLTki5QWNMBZPRwO5W/gqxfM1dVlH++PyAeIesYJHlr/+nNMi11y8VLf9a+3/NmL1ocCY0ot1VHq9KqHds8P+cWe8muD+U40GBGrvK0r8ffHgBRHKxiFSr3s/8lZUL7PyeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZfPCn8VdRb+p4peHhvxW4m6aHFYrHuShF1AGpVBzplQ=; b=BC4dApjHeDBmGq2SwVKTqh+veMy1dWZnk3buwpXySWLutogQAGHDVYUwtJhBsA6Nqd2z9JECldzwCRYS0s0ifGmNi/ax/SLllEBSBTe1HhGYFUGt7AcHy6c7XeltmshN7dRCzaBHnv1DmcCmZmQFVZmmjg5xngbFibtPnkIe3w2eBFmYexGtexcvAq88Z23ja8DDtmtJdl7K5FekGUb7xFBknu7/gv+OOE1B6oix7nDjiNR6bHEQaqpJEArZcPeNzctd3P5yU+NVLC22Uw6AYB3hkejcfsn70pxNJoUFbY0muyT7M3XxlGT370sHNGGa5AnkEZaS4+PFexFyaR9/Ew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZfPCn8VdRb+p4peHhvxW4m6aHFYrHuShF1AGpVBzplQ=; b=onfYTM5T7hwo6TKrBoX3xy6G7pxlXelQF0gXP+PoBwy89eQgkyneICmqELRCyMuv7MALcfQBEgJJJt7LLkezak1NNCmnG21FL3LJH1SnA3Ck0PkyzSMYQpUiUskTcDayvtKqU7Ut/yrHcn5QTfJ/kQRXf4xXu7ivUf5Xe0j+wao= Received: from BN9P221CA0002.NAMP221.PROD.OUTLOOK.COM (2603:10b6:408:10a::9) by SA3PR12MB9225.namprd12.prod.outlook.com (2603:10b6:806:39e::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.13; Thu, 28 May 2026 09:49:06 +0000 Received: from BN2PEPF00004FBB.namprd04.prod.outlook.com (2603:10b6:408:10a:cafe::6) by BN9P221CA0002.outlook.office365.com (2603:10b6:408:10a::9) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.71.14 via Frontend Transport; Thu, 28 May 2026 09:49:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN2PEPF00004FBB.mail.protection.outlook.com (10.167.243.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.7 via Frontend Transport; Thu, 28 May 2026 09:49:06 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Thu, 28 May 2026 04:49:00 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Aaron Lu , Josh Don , K Prateek Nayak , Subject: [PATCH 1/5] sched/fair: Convert cfs bandwidth throttling to use guards Date: Thu, 28 May 2026 09:48:26 +0000 Message-ID: <20260528094830.13291-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528094830.13291-1-kprateek.nayak@amd.com> References: <20260528094830.13291-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBB:EE_|SA3PR12MB9225:EE_ X-MS-Office365-Filtering-Correlation-Id: 1474209f-3c25-46e6-99e7-08debc9e5be8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700016|376014|7416014|22082099003|18002099003|11063799006|56012099006|6133799003; X-Microsoft-Antispam-Message-Info: XTntGZh/caIIvQie6tFro3+o8YgG7ffFODf8sLJhuREsSQW1h/1RtyG0PuV3BpnDw0tgX7wAcMbk0M0POPRyfG+lhMHYFtta2finOMC6/hPEodPvpFDBbPBPX0ev2EmkwGIvx/fvmUsNYU9zLmVftWjWmjkwjb7OxN1XkSYnhkqt2bQsRzJ1+J+HaR3B1j5QQXdHdcI4lHg0kjrl+o3x1QSw8Q6MJZ3idunEhOwtYwvWHFd+gKtTECJRlDBQWFpEYuUlnCsjbLJQZmXxC1D6TJ18sYy5TMB2rtRvOtEbpRIWcow8hnuAPFtOsyliW5OWzTmcFgSYt6yE6gPgW2cMntFIgatXSZ+EhtJ5kmXNiZB7C+RTKyOck2czZh0fbu/baYsmTFOLKxkQY5gCwDgmDamp82mBKFewKY61bDqpgoIXwL+GKXnWlEds6ubUavS8YirQezzWq0bvQqLw55ivhlR0O0/S78SonidsAhEZkPrEVAQKoTrcXEgV+Ldom0NKLWrZmno9yh7HyEy/U0Hy+tieONWGdV/RfdjgH3bBL0fBFy6RmS9c0X54usyX0gspYEe8cRwesqB6LtolqRwVOCmroqao3ydk3ehLNLW4sSqceQlUCKkNUOVK9AvlWAOfxPQuPcSbsMpqedU6OMTyMfxbyko8yhi0XuZbqUHlXhS13yFKzl1EbyDkqfc6Y2/+fZCtOJz+L9XJTLYX/FjHR3eXTFtVlVq8Uf9ogPCwQS8= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700016)(376014)(7416014)(22082099003)(18002099003)(11063799006)(56012099006)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: YCYAczTEKDyFwtjeUAsHg/5HXN8jRY3QxVFzxZ3rTAEjhF4AfaUO3SqpAydhRiBiQ4cA+QcJVZcrnr5h4d+pQSIRn1enkFcRysQc8SYkxdyHv6OGI9/KKsz71Rq9Hwf97Alrl/xdWoLKK5TDTK4UNaZe4p9h+Av4DAKLBogbkiWoozKvHMC/RBt3SuPvo3hou+gJhn3WQlFPyCzePkJeEpGvucW5zSpMQJ3hjkApoQaOZcbHPzCr3gu6d9LHgbJNNfafSOruLpAZ7mkYvoAxCJy3jKSUBFLbvRFCFzTfr0xqez8UiQwB6E7qaL0eSULnC3+rn2g0Cu6V+0PzJz5/cySCP+hZd3TZTllGWe0Bpum2IcmLsutuLrIqwMSM9Yb3C0vH/oIC+8DOPRR46jIXjhl76o/wVxtXS7H2W6UYYpO0kphQcw9UXSVde0JUPZnl X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 09:49:06.5618 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1474209f-3c25-46e6-99e7-08debc9e5be8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBB.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB9225 Routine conversion of rcu_read_lock(), spin_lock*, and rq_lock usage within the cfs bandwidth controller to use class guards. Only notable changes are: - Conversion of do_sched_cfs_slack_timer() to return a bool and moving the call to distribute_cfs_runtime() into the caller for a cleaner guard usage. - Reordering of list_del_rcu() against throttled_clock indicator update in unthrottle_cfs_rq(). Both are done with "cfs_b->lock" held after the "cfs_rq->throttled" is cleared which make the reordering safe against concurrent list modifications. No functional changes intended. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 174 ++++++++++++++++++++------------------------ 1 file changed, 79 insertions(+), 95 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8e858ca6bcd0..8c2a5a2f046d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6511,13 +6511,10 @@ static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b, static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq) { struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); - int ret; - raw_spin_lock(&cfs_b->lock); - ret = __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice()); - raw_spin_unlock(&cfs_b->lock); + guard(raw_spinlock)(&cfs_b->lock); - return ret; + return __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice()); } static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) @@ -6806,33 +6803,32 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); - int dequeue = 1; - raw_spin_lock(&cfs_b->lock); - /* This will start the period timer if necessary */ - if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) { + scoped_guard(raw_spinlock, &cfs_b->lock) { /* - * We have raced with bandwidth becoming available, and if we - * actually throttled the timer might not unthrottle us for an - * entire period. We additionally needed to make sure that any - * subsequent check_cfs_rq_runtime calls agree not to throttle - * us, as we may commit to do cfs put_prev+pick_next, so we ask - * for 1ns of runtime rather than just check cfs_b. + * Check if We have raced with bandwidth becoming available. If + * we actually throttled the timer might not unthrottle us for + * an entire period. We additionally needed to make sure that + * any subsequent check_cfs_rq_runtime calls agree not to + * throttle us, as we may commit to do cfs put_prev+pick_next, + * so we ask for 1ns of runtime rather than just check cfs_b. + * + * This will start the period timer if necessary. + */ + if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) + return false; + + /* + * No bandwidth available; Add ourselves on the list to be + * unthrottled later. */ - dequeue = 0; - } else { list_add_tail_rcu(&cfs_rq->throttled_list, &cfs_b->throttled_cfs_rq); } - raw_spin_unlock(&cfs_b->lock); - - if (!dequeue) - return false; /* Throttle no longer required. */ /* freeze hierarchy runnable averages while throttled */ - rcu_read_lock(); - walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq); - rcu_read_unlock(); + scoped_guard(rcu) + walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq); /* * Note: distribution will already see us throttled via the @@ -6865,13 +6861,15 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) update_rq_clock(rq); - raw_spin_lock(&cfs_b->lock); - if (cfs_rq->throttled_clock) { + scoped_guard(raw_spinlock, &cfs_b->lock) { + list_del_rcu(&cfs_rq->throttled_list); + + if (!cfs_rq->throttled_clock) + break; + cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock; cfs_rq->throttled_clock = 0; } - list_del_rcu(&cfs_rq->throttled_list); - raw_spin_unlock(&cfs_b->lock); /* update hierarchical throttle state */ walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_up, (void *)rq); @@ -6900,9 +6898,8 @@ static void __cfsb_csd_unthrottle(void *arg) { struct cfs_rq *cursor, *tmp; struct rq *rq = arg; - struct rq_flags rf; - rq_lock(rq, &rf); + guard(rq_lock)(rq); /* * Iterating over the list can trigger several call to @@ -6919,7 +6916,7 @@ static void __cfsb_csd_unthrottle(void *arg) * race with group being freed in the window between removing it * from the list and advancing to the next entry in the list. */ - rcu_read_lock(); + guard(rcu)(); list_for_each_entry_safe(cursor, tmp, &rq->cfsb_csd_list, throttled_csd_list) { @@ -6929,10 +6926,7 @@ static void __cfsb_csd_unthrottle(void *arg) unthrottle_cfs_rq(cursor); } - rcu_read_unlock(); - rq_clock_stop_loop_update(rq); - rq_unlock(rq, &rf); } static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq) @@ -6972,11 +6966,11 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b) u64 runtime, remaining = 1; bool throttled = false; struct cfs_rq *cfs_rq, *tmp; - struct rq_flags rf; struct rq *rq; LIST_HEAD(local_unthrottle); - rcu_read_lock(); + guard(rcu)(); + list_for_each_entry_rcu(cfs_rq, &cfs_b->throttled_cfs_rq, throttled_list) { rq = rq_of(cfs_rq); @@ -6986,24 +6980,25 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b) break; } - rq_lock_irqsave(rq, &rf); + guard(rq_lock_irqsave)(rq); + if (!cfs_rq_throttled(cfs_rq)) - goto next; + continue; /* Already queued for async unthrottle */ if (!list_empty(&cfs_rq->throttled_csd_list)) - goto next; + continue; /* By the above checks, this should never be true */ WARN_ON_ONCE(cfs_rq->runtime_remaining > 0); - raw_spin_lock(&cfs_b->lock); - runtime = -cfs_rq->runtime_remaining + 1; - if (runtime > cfs_b->runtime) - runtime = cfs_b->runtime; - cfs_b->runtime -= runtime; - remaining = cfs_b->runtime; - raw_spin_unlock(&cfs_b->lock); + scoped_guard(raw_spinlock, &cfs_b->lock) { + runtime = -cfs_rq->runtime_remaining + 1; + if (runtime > cfs_b->runtime) + runtime = cfs_b->runtime; + cfs_b->runtime -= runtime; + remaining = cfs_b->runtime; + } cfs_rq->runtime_remaining += runtime; @@ -7011,40 +7006,33 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b) if (cfs_rq->runtime_remaining > 0) { if (cpu_of(rq) != this_cpu) { unthrottle_cfs_rq_async(cfs_rq); - } else { - /* - * We currently only expect to be unthrottling - * a single cfs_rq locally. - */ - WARN_ON_ONCE(!list_empty(&local_unthrottle)); - list_add_tail(&cfs_rq->throttled_csd_list, - &local_unthrottle); + continue; } - } else { - throttled = true; + + /* + * We currently only expect to be unthrottling + * a single cfs_rq locally. + */ + WARN_ON_ONCE(!list_empty(&local_unthrottle)); + list_add_tail(&cfs_rq->throttled_csd_list, &local_unthrottle); + continue; } -next: - rq_unlock_irqrestore(rq, &rf); + throttled = true; } list_for_each_entry_safe(cfs_rq, tmp, &local_unthrottle, throttled_csd_list) { struct rq *rq = rq_of(cfs_rq); - rq_lock_irqsave(rq, &rf); + guard(rq_lock_irqsave)(rq); list_del_init(&cfs_rq->throttled_csd_list); - if (cfs_rq_throttled(cfs_rq)) unthrottle_cfs_rq(cfs_rq); - - rq_unlock_irqrestore(rq, &rf); } WARN_ON_ONCE(!list_empty(&local_unthrottle)); - rcu_read_unlock(); - return throttled; } @@ -7167,7 +7155,8 @@ static void __return_cfs_rq_runtime(struct cfs_rq *cfs_rq) if (slack_runtime <= 0) return; - raw_spin_lock(&cfs_b->lock); + guard(raw_spinlock)(&cfs_b->lock); + if (cfs_b->quota != RUNTIME_INF) { cfs_b->runtime += slack_runtime; @@ -7176,7 +7165,6 @@ static void __return_cfs_rq_runtime(struct cfs_rq *cfs_rq) !list_empty(&cfs_b->throttled_cfs_rq)) start_cfs_slack_bandwidth(cfs_b); } - raw_spin_unlock(&cfs_b->lock); /* even if it's not valid for return we don't want to try again */ cfs_rq->runtime_remaining -= slack_runtime; @@ -7197,29 +7185,25 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) * This is done with a timer (instead of inline with bandwidth return) since * it's necessary to juggle rq->locks to unthrottle their respective cfs_rqs. */ -static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b) +static bool do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b) { u64 runtime = 0, slice = sched_cfs_bandwidth_slice(); - unsigned long flags; /* confirm we're still not at a refresh boundary */ - raw_spin_lock_irqsave(&cfs_b->lock, flags); + guard(raw_spinlock_irqsave)(&cfs_b->lock); + cfs_b->slack_started = false; - if (runtime_refresh_within(cfs_b, min_bandwidth_expiration)) { - raw_spin_unlock_irqrestore(&cfs_b->lock, flags); - return; - } + if (runtime_refresh_within(cfs_b, min_bandwidth_expiration)) + return false; if (cfs_b->quota != RUNTIME_INF && cfs_b->runtime > slice) runtime = cfs_b->runtime; - raw_spin_unlock_irqrestore(&cfs_b->lock, flags); - if (!runtime) - return; + return false; - distribute_cfs_runtime(cfs_b); + return true; } /* @@ -7297,7 +7281,8 @@ static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer) struct cfs_bandwidth *cfs_b = container_of(timer, struct cfs_bandwidth, slack_timer); - do_sched_cfs_slack_timer(cfs_b); + if (do_sched_cfs_slack_timer(cfs_b)) + distribute_cfs_runtime(cfs_b); return HRTIMER_NORESTART; } @@ -7306,18 +7291,18 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer) { struct cfs_bandwidth *cfs_b = container_of(timer, struct cfs_bandwidth, period_timer); - unsigned long flags; int overrun; int idle = 0; int count = 0; - raw_spin_lock_irqsave(&cfs_b->lock, flags); + CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock); + for (;;) { overrun = hrtimer_forward_now(timer, cfs_b->period); if (!overrun) break; - idle = do_sched_cfs_period_timer(cfs_b, overrun, flags); + idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags); if (++count > 3) { u64 new, old = ktime_to_ns(cfs_b->period); @@ -7350,11 +7335,13 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer) count = 0; } } - if (idle) + + if (idle) { cfs_b->period_active = 0; - raw_spin_unlock_irqrestore(&cfs_b->lock, flags); + return HRTIMER_NORESTART; + } - return idle ? HRTIMER_NORESTART : HRTIMER_RESTART; + return HRTIMER_RESTART; } void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth *parent) @@ -7421,14 +7408,12 @@ static void destroy_cfs_bandwidth(struct cfs_bandwidth *cfs_b) */ for_each_possible_cpu(i) { struct rq *rq = cpu_rq(i); - unsigned long flags; if (list_empty(&rq->cfsb_csd_list)) continue; - local_irq_save(flags); - __cfsb_csd_unthrottle(rq); - local_irq_restore(flags); + scoped_guard(irqsave) + __cfsb_csd_unthrottle(rq); } } @@ -7446,16 +7431,15 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq) lockdep_assert_rq_held(rq); - rcu_read_lock(); + guard(rcu)(); + list_for_each_entry_rcu(tg, &task_groups, list) { struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth; struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; - raw_spin_lock(&cfs_b->lock); - cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF; - raw_spin_unlock(&cfs_b->lock); + scoped_guard(raw_spinlock, &cfs_b->lock) + cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF; } - rcu_read_unlock(); } /* cpu offline callback */ @@ -7476,7 +7460,8 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq) */ rq_clock_start_loop_update(rq); - rcu_read_lock(); + guard(rcu)(); + list_for_each_entry_rcu(tg, &task_groups, list) { struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; @@ -7499,7 +7484,6 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq) cfs_rq->runtime_remaining = 1; unthrottle_cfs_rq(cfs_rq); } - rcu_read_unlock(); rq_clock_stop_loop_update(rq); } -- 2.43.0