From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010005.outbound.protection.outlook.com [52.101.201.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E3802DF6E6 for ; Tue, 28 Apr 2026 05:17:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.201.5 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777353463; cv=fail; b=MfPtuCIHuM2NQifCn0ybs761aCPHHcMTna7ohdZWrfwG7GkMpxiHU5ZHYq38WadPlH99mTBhVKhJFcke18iV9L3xRWgrRcCaYCjWPcfBn1nwZwJfsavfjyHZtTcLFEtZDyEOwJNYj214wOSDW/5k+B+aUHpBkYKVZWDaW1bO7IU= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777353463; c=relaxed/simple; bh=MSwRwguxkND07ICqSjGDMXutbnYW/7PhM33e9gkmOZw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=O5+QrFrjzfPHRc5FM79skfyG6ODz1VL/WW02pmPBmA48V8hfOHovUefHvdPUsHDFs7Qlqgtp67z+K4jHkkgybDkP2TggEFEISi/JlobhwsFAyROVHxBq2qO8cP9OlKLRtPyCEsdhpGr+8OiOuYJEVeqmM2CH+B5eEaO1S48I0jQ= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=dbsLnbp/; arc=fail smtp.client-ip=52.101.201.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="dbsLnbp/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KKXH6XrSmd7TaClzTVyHunI6JWHZWKgpS2aFyYgIJN76xCn2LtcBZKCFAE5eeYt89M2fUObA4t1wQjm0BlhCF9iz30j5IUrVAwrvaNIvs15hXp5i5mpw3fJNxY2/Vd6SXAGvuBdd5INmdKd/cvqaLqGxjxdJeXD/JPdjnJ/oKmNwht6ph9Y2qdkCN4uYzov8gH7MdTEucLYslK38VWEa10rp5moO1yDxTbnGuD15UGmZAu1FY3eIIMGcohWl5DUfnnvmJ+UQ7N7EdgGtDtozI6+qpEgTqHEOe5au9SoidaWjAIMhjToAqRvJpxoicbuqHwEFjXA1FrbKG8p19wHS/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7lCTdUxybG7NjDIKGbTvsdHl2/xUmIomPiC0KHT12fk=; b=a4/3ZxulmcSGjnRj97Yti/FSGZOcx55Ngv/tlZPGry2x8PE3s+1myJSaaz3CRSkccyIt6c0huY8DSzkQGLc9Jx7Uu+68b/8mGHM+DDaHv0lIhHXZLltecS7r1H3gWRIZIdL2zIMjQRUfJcE2K7SySTk4o2qQ+/Tc2Os2rPKf0+lkR4WuRG8wHxoZcz5Ygj0VcsX5NPlNO59mATPiGVTrvi/E9LY9VVPTQhqHhFYka00583Rgjn2kZRDkPpWqPQnrow1cIUgpuIMwTeAtN/44xRsFMKAQ33+sePTILppEl8w3tIPISDa9Cw2nUlx4xKP2q/he4dy5z+8We/i1cvPz0w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7lCTdUxybG7NjDIKGbTvsdHl2/xUmIomPiC0KHT12fk=; b=dbsLnbp/IOiBvvw/gfm70lnYbaG9CKz/nHPuQwrkPqzcQ+rlfrmdeCOvz9cjU80+WV3v5LLIqQFWXVvf2wanttIJhUPk6dZAwYR1flSPJ5tMUaOJJjvNM4hjLziYHZNoGJhoq111BDx5EjGUQ2LFNoM0r3/49t3AxgSPpee0hXbwGtaDBUdeF0q2D8aAw9q00yzsHEsgDKfFqhjT8wc2jVkJhoAdwX1J96y5i/Z9YWRMYueoO9iRB71tydLKIiBKUD5TtbDhodVEiSbayuqHuet1D/dcw5OS2XmUOgrGL+pxVg49dzREPrc2evCiNc2JTf1zJUhUgtAL2PuFgn56Mg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by CY8PR12MB8337.namprd12.prod.outlook.com (2603:10b6:930:7d::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.16; Tue, 28 Apr 2026 05:17:37 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9870.013; Tue, 28 Apr 2026 05:17:37 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 1/6] sched/fair: Use guard(rcu) for sched_domain RCU sections Date: Tue, 28 Apr 2026 07:16:35 +0200 Message-ID: <20260428051720.3180182-2-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260428051720.3180182-1-arighi@nvidia.com> References: <20260428051720.3180182-1-arighi@nvidia.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: SJ0PR05CA0086.namprd05.prod.outlook.com (2603:10b6:a03:332::31) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|CY8PR12MB8337:EE_ X-MS-Office365-Filtering-Correlation-Id: a5d0dd7f-fc37-4742-6239-08dea4e57637 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: zMKLPFbSmuM6Xos4P4E10HoIx4SZsf3BD7oIEiJ8ruXxBI3yxKc/vLyPlBVLvXXrU4XUZu59a6f5fEs08dI0M8DLkIdkUJU7NMcbPmUSLU2XjbW16FOa49SlNG6vUUSHNkt44riIUPnvR7vPjYNk4dETS1TXyEjyevx5qyMV9ekBssjUVzPYf0EdK8D0POHODOot5UpLFm1thvJ0JpZiaebORclhyLG3WxC7fAnjDg88nDUctQEFGyFN8Px3WYoHH7CWw6CyjJq3ZHfBFleOkANuFTxm8kSVEmqwH95B7j1CMjm/hnQQea8IbXKcpMIXBbMXmf/TrLGFbTOokaxlwbRsDICyIpzby8QKXA0piVaOTFzhWTx2/FpI1d6dJtY1cfAMEC34IGFB2nraA2v1lGnIn1JghFdwWeEzjFX4K0OakKNjUkiDl+7yz55lz5tVo41wp8s7Ilj3TlAqEwgPb0rRO+BrAxBnydll9dihNWE1ATv4ti66AMEsj66X1p5ag5rqUeKS3KmbgUvy3evxSAxHs23gtQWLeX/W3UsiTzfrBC6n70PR+vlk/2zipcUHR+y8T6ZuORKPW0d9/7SGGacrnrQdF9I3ygpEagM6JfECsmhArsU46sbmdt9Ng/9g4eB9KblFfF6/qX0icKumWYPMgxHdy5tfib7KrJ0fUWxkHlGLbmWLiNOTf7hlrMvj2VGI0ZJQFZ8pCOnWjnmFOOqt7OUYl+sZRxsLEQEmNRU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(1800799024)(376014)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9BCw88Fh4/iR2c6Hi5ECcJfaLXVwGIoSL6VhgvGXdciWLl73PUv7MjiUfo2G?= =?us-ascii?Q?Kg8i5R+WsKy+X0PPjSQ/N9q6YpE5qY3OTMjFycRWlrnxD5wC+QUpzErfUekb?= =?us-ascii?Q?Jna1Yr6uD1ws3faLLFhzL/4Rtrp+0SYbhkieXIteaRFCK0LupprwDCIsCFwu?= =?us-ascii?Q?KhADCWIwzjmckvBc3s8hpqZ57jDnwcI5hesfihkNb263t1iJXm6vnIg3S9Jy?= =?us-ascii?Q?+FpOKVmzXdotrxxUb7ifQ23yywJKYEM5svNockS7aPg98efJhfz2o7pomeKC?= =?us-ascii?Q?2PcIkB4DIDkAXgxl9U3gBXsAn1qKL3Br0iLcPdzOwbBvuaGTDJ1RwVDjkjtm?= =?us-ascii?Q?W2sTFG8YEhOTvbp9IUS8zT18qFIAHM5yUXcwnkuS4px11CpKv2404qluM4La?= =?us-ascii?Q?zC1gUgJTrbJ5XasQbnxDxOClqXOKDDeOj6P298s1m/HHUkiYVVJXiq0egMif?= =?us-ascii?Q?G9KQto4+mlX8kGZUIUI2yugfQFrTkFwgoGEM9hYqeeejErnTzlpmMNdMEqtF?= =?us-ascii?Q?w3c3RsCrbTNSSJlDcO6YV7mVVWRaMYhPoNDIwtbxFOmdERr9eHkT6yUrrZPG?= =?us-ascii?Q?9nvh6cHjznwdunz2YQikO8M0GAL/nmxYYVH6nQXv+CYkWrlZtO66ej9qkpt1?= =?us-ascii?Q?M2ohWPH/H31zlGMPtWSyN923B45T+167M5476bStKMrSV1TPrTWDdcCBU2ZK?= =?us-ascii?Q?BiWc2Z4KLUxDJUetuKXvnOnWhx2PoZ2qf4CXYY5k+m+V0r44tYOic/5X1esN?= =?us-ascii?Q?aC9Mf4aN9Zg3sU29vc882wdJUiI9JSpuNQBVJdRnzX/btQW2bRwiB2RS1SmL?= =?us-ascii?Q?e9IFQBapsuZ/uTtbo7io8K061XfXujheyzQYggsRLktvv7yRXlTsznskB5wY?= =?us-ascii?Q?etbYxg3EgcYNUCNbrVmqO2w/un/59I1UElgq2LC7HD/EtAd+tATqivLwJSvo?= =?us-ascii?Q?yzVkwAR+LdJALqVtnaAs9msie7wpdgXpyJW3KXKZG/Nqhl0runHebMF1NGKp?= =?us-ascii?Q?KCkEG2RAjzCIaZ5ETl9l/h6QAYgRDVwIpzbdrwN7HNvsdP4NRzJPT4rjUzLG?= =?us-ascii?Q?Ilvllyl+IE5ouDAYJo3v8gYc6tsZtgd5azxRokVDBNfsS5XJpxyUxQnt4Vkh?= =?us-ascii?Q?an+2CCm1MByA+GvRkeZNcJ09bJ6dCkBKXHtEC1DQmS1UbYv21jT8t/GXN5it?= =?us-ascii?Q?L/blUjcAuLZVFkJv2iiephPcdWKXEzGxfdl9YSADUM//jw/pfe/B8Kg9MNem?= =?us-ascii?Q?3t/n7PzWZ2ie29h6HDU0aLSstnDF3n8CDAEwDsLbzZf6r8FITXgvj5Xm6ERY?= =?us-ascii?Q?XHoXZ6ILPQRYW7Xef1M3K8HrGur1he/oGgV4yEb4KnHouA187nSiYpbzWpWh?= =?us-ascii?Q?96lv4Pe+y2DZxbUxWO3eKEc1Lug6TmWk38993ziSMlz2pjV59eWrnoZkGuIU?= =?us-ascii?Q?Rr8UzbuaQEoUUyAQFV7KqZ4V1Yxk+mO66KAwrE35tbiCJ15s/K0FlsU85AkO?= =?us-ascii?Q?oaYjmzDdTv7XEXur8jeq2Ue+P2WQJ/O/FYhE8j08jmrH89mgJejpfzAErNvI?= =?us-ascii?Q?Ma2TLoh/yau7rs54iH12kJBQtVHzmBUlkgLlEp4jEXwmTedak+govKbUFC1I?= =?us-ascii?Q?EG1sLOErbTcLa9+ekVcRlc1vU/3syEAhovfUNJ2TsyjOqi7iMI5TKpW3K15c?= =?us-ascii?Q?7Bh4nf4CE11/uGkIg8NuYa5+qNrzDtEanbRKisPcCKhOdOXhJiOtUj0EOLXJ?= =?us-ascii?Q?0VPXe1pVzg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a5d0dd7f-fc37-4742-6239-08dea4e57637 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2026 05:17:37.3240 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: s7GqHYHKMgyZV5eA41NdYW8jvIKaI8kmuFFeZMgQjLiP3JeBulRhr5U+7smbFdwxDJoSGxAMiOdpBna3hjUZ6Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8337 Use the scoped guard(rcu)() helper to safely access sched_domain pointers. No functional change intended, this is preparation for topology work where sched_domain lifetimes are easier to reason about with explicit, scope-bounded RCU critical sections. Suggested-by: K Prateek Nayak Signed-off-by: Andrea Righi --- kernel/sched/fair.c | 141 ++++++++++++++++++++++---------------------- 1 file changed, 71 insertions(+), 70 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 69361c63353ad..fc0828150c780 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8083,6 +8083,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) */ lockdep_assert_irqs_disabled(); + guard(rcu)(); + if (choose_idle_cpu(target, p) && asym_fits_cpu(task_util, util_min, util_max, target)) return target; @@ -12701,55 +12703,16 @@ static void kick_ilb(unsigned int flags) } /* - * Current decision point for kicking the idle load balancer in the presence - * of idle CPUs in the system. + * Decide whether the ILB needs a stats and/or balance kick based on + * sched_domain state. */ -static void nohz_balancer_kick(struct rq *rq) +static bool nohz_balancer_needs_kick(struct rq *rq) { - unsigned long now = jiffies; struct sched_domain_shared *sds; struct sched_domain *sd; int nr_busy, i, cpu = rq->cpu; - unsigned int flags = 0; - - if (unlikely(rq->idle_balance)) - return; - - /* - * We may be recently in ticked or tickless idle mode. At the first - * busy tick after returning from idle, we will update the busy stats. - */ - nohz_balance_exit_idle(rq); - - if (READ_ONCE(nohz.has_blocked_load) && - time_after(now, READ_ONCE(nohz.next_blocked))) - flags = NOHZ_STATS_KICK; - - /* - * Most of the time system is not 100% busy. i.e nohz.nr_cpus > 0 - * Skip the read if time is not due. - * - * If none are in tickless mode, there maybe a narrow window - * (28 jiffies, HZ=1000) where flags maybe set and kick_ilb called. - * But idle load balancing is not done as find_new_ilb fails. - * That's very rare. So read nohz.nr_cpus only if time is due. - */ - if (time_before(now, nohz.next_balance)) - goto out; - /* - * None are in tickless mode and hence no need for NOHZ idle load - * balancing - */ - if (unlikely(cpumask_empty(nohz.idle_cpus_mask))) - return; - - if (rq->nr_running >= 2) { - flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto out; - } - - rcu_read_lock(); + guard(rcu)(); sd = rcu_dereference_all(rq->sd); if (sd) { @@ -12757,10 +12720,8 @@ static void nohz_balancer_kick(struct rq *rq) * If there's a runnable CFS task and the current CPU has reduced * capacity, kick the ILB to see if there's a better CPU to run on: */ - if (rq->cfs.h_nr_runnable >= 1 && check_cpu_capacity(rq, sd)) { - flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; - } + if (rq->cfs.h_nr_runnable >= 1 && check_cpu_capacity(rq, sd)) + return true; } sd = rcu_dereference_all(per_cpu(sd_asym_packing, cpu)); @@ -12774,10 +12735,8 @@ static void nohz_balancer_kick(struct rq *rq) * preferred CPU must be idle. */ for_each_cpu_and(i, sched_domain_span(sd), nohz.idle_cpus_mask) { - if (sched_asym(sd, i, cpu)) { - flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; - } + if (sched_asym(sd, i, cpu)) + return true; } } @@ -12787,10 +12746,8 @@ static void nohz_balancer_kick(struct rq *rq) * When ASYM_CPUCAPACITY; see if there's a higher capacity CPU * to run the misfit task on. */ - if (check_misfit_status(rq)) { - flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; - } + if (check_misfit_status(rq)) + return true; /* * For asymmetric systems, we do not want to nicely balance @@ -12799,7 +12756,7 @@ static void nohz_balancer_kick(struct rq *rq) * * Skip the LLC logic because it's not relevant in that case. */ - goto unlock; + return false; } sds = rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); @@ -12814,13 +12771,61 @@ static void nohz_balancer_kick(struct rq *rq) * like this LLC domain has tasks we could move. */ nr_busy = atomic_read(&sds->nr_busy_cpus); - if (nr_busy > 1) { - flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; - } + if (nr_busy > 1) + return true; } -unlock: - rcu_read_unlock(); + + return false; +} + +/* + * Current decision point for kicking the idle load balancer in the presence + * of idle CPUs in the system. + */ +static void nohz_balancer_kick(struct rq *rq) +{ + unsigned long now = jiffies; + unsigned int flags = 0; + + if (unlikely(rq->idle_balance)) + return; + + /* + * We may be recently in ticked or tickless idle mode. At the first + * busy tick after returning from idle, we will update the busy stats. + */ + nohz_balance_exit_idle(rq); + + if (READ_ONCE(nohz.has_blocked_load) && + time_after(now, READ_ONCE(nohz.next_blocked))) + flags = NOHZ_STATS_KICK; + + /* + * Most of the time system is not 100% busy. i.e nohz.nr_cpus > 0 + * Skip the read if time is not due. + * + * If none are in tickless mode, there maybe a narrow window + * (28 jiffies, HZ=1000) where flags maybe set and kick_ilb called. + * But idle load balancing is not done as find_new_ilb fails. + * That's very rare. So read nohz.nr_cpus only if time is due. + */ + if (time_before(now, nohz.next_balance)) + goto out; + + /* + * None are in tickless mode and hence no need for NOHZ idle load + * balancing + */ + if (unlikely(cpumask_empty(nohz.idle_cpus_mask))) + return; + + if (rq->nr_running >= 2) { + flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; + goto out; + } + + if (nohz_balancer_needs_kick(rq)) + flags |= NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; out: if (READ_ONCE(nohz.needs_update)) flags |= NOHZ_NEXT_KICK; @@ -12833,16 +12838,14 @@ static void set_cpu_sd_state_busy(int cpu) { struct sched_domain *sd; - rcu_read_lock(); + guard(rcu)(); sd = rcu_dereference_all(per_cpu(sd_llc, cpu)); if (!sd || !sd->nohz_idle) - goto unlock; + return; sd->nohz_idle = 0; atomic_inc(&sd->shared->nr_busy_cpus); -unlock: - rcu_read_unlock(); } void nohz_balance_exit_idle(struct rq *rq) @@ -12862,16 +12865,14 @@ static void set_cpu_sd_state_idle(int cpu) { struct sched_domain *sd; - rcu_read_lock(); + guard(rcu)(); sd = rcu_dereference_all(per_cpu(sd_llc, cpu)); if (!sd || sd->nohz_idle) - goto unlock; + return; sd->nohz_idle = 1; atomic_dec(&sd->shared->nr_busy_cpus); -unlock: - rcu_read_unlock(); } /* -- 2.54.0