From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CWXP265CU008.outbound.protection.outlook.com (mail-ukwestazon11020115.outbound.protection.outlook.com [52.101.195.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A601F3AD531 for ; Tue, 25 Aug 2026 18:46:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.195.115 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787683609; cv=fail; b=tXiSPehjIg4J89h2GadMOE4uoCEsIkss4Bd6FOvA1wfFRTjK8k0A24Hf7iDGQMBfPSQcSF0mrE/O+yg280G+jGwosBsgNuAdngzLQbjnWawouxFNGnVAsuBbjC4uyaL6MVgeo67cYKwAjrutf+CmfocYBm6gFmcWfj5n2XJh+nk= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787683609; c=relaxed/simple; bh=guLRGRpcYPMXw+wCnbNxeBgH85MFtkvd0KIWUXR5jPM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=T8IVGvaDWbo2It/WP0KNyVakLKWFT7aH2pxiAvJ+axsI2niaUCp7sIbPGqx3/xEtB6Umm6WSchCYfjrFurKQUxewFlDRwJmFk+nsupfxvvLq9jeU1+ChuxI+OwqwPeXm6+64pBF85xVMXx+m1xdtlJk+wYbHfQJJrL0dpImfy5U= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com; spf=pass smtp.mailfrom=atomlin.com; arc=fail smtp.client-ip=52.101.195.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=atomlin.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iQb8mPnTFLqb00+lD2OsGJHu8yV4MV3pjTqCQiuxvBu7bfSex4P20D3JrJ+JmYqDQ6KJFKheFt3Eyy/yIDsgVVMsoY3Ab6E8VimTUK+GM8t+jEvdE8ds54b2p/YymU6vPhqsh3xLKxxrYhjbhN9yS+AgxcPH8LH48uCE+VwWTQeUYYKZStbWfaq/Z/pT2XCZ6vBqY9gG7xmPOhJPIahR1q8UOpy7/uy8F8pHWPoyTHn46hMDqKRjPkM45sTkeIoddSofNdFMzkg3bAE2+wJ0ZrjTtHqMopv7O+4RoHCx+zkKm+aBHO8s981RW9oe1Ts4fNoMXz7Tkn4hLXx9Ez52Rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:MIME-Version; bh=5w5uf12V33B4BT629MQ03nvD+WvztsB8y3Q+6+jREAM=; b=UX7q3SYdvve/GzTeYiEmOt5y3VCuG9P82GQNnc4EeDZXi5v7YblLujSOV7BtON3D20JatoEyC6ru1D5oHGVimQP0XKNRuhc4ra7J9G07S294uEQ7xfpebga7i81JqHswM/yWjT0pMrTFxm2wo2UX/4Qc2UL4RTf3+j59k8/l1G5seu/P/qCRC5vi3yoItVOqBEVRuLUJPOVo0M9kJbVnQohdNcKfETDBtcDZwAYK3POPnA53HYmCYTZEI2YCN6CQAACYtZGNmKroPY+Na5GHxjxDflO14MLSYeWY3EWSvyBwI6ML9d54ddlBfg+z8YSDYEwXjBsMBlX2dsj4ejJz7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=atomlin.com; dmarc=pass action=none header.from=atomlin.com; dkim=pass header.d=atomlin.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=atomlin.com; Received: from LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:2b2::11) by LO7P123MB7501.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:40c::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.8; Tue, 25 Aug 2026 18:46:41 +0000 Received: from LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM ([fe80::3975:c5e7:bc1a:f383]) by LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM ([fe80::3975:c5e7:bc1a:f383%4]) with mapi id 15.21.0360.005; Tue, 25 Aug 2026 18:46:41 +0000 From: Aaron Tomlin To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org Cc: dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com, zhanxusheng1024@gmail.com, neelx@suse.com, atomlin@atomlin.com, chjohnst@mail.com, mproche@mail.com, sean@ashe.io, steve@abita.co, rishil1999@outlook.com, linux-kernel@vger.kernel.org Subject: [PATCH v6 1/6] sched: Annotate rq->rd with __rcu and update lockless readers Date: Tue, 25 Aug 2026 14:46:32 -0400 Message-ID: <20260825184637.888364-2-atomlin@atomlin.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260825184637.888364-1-atomlin@atomlin.com> References: <20260825184637.888364-1-atomlin@atomlin.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: LO4P123CA0091.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:191::6) To LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:2b2::11) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LO6P123MB6616:EE_|LO7P123MB7501:EE_ X-MS-Office365-Filtering-Correlation-Id: 52d5f4c2-cab2-4a4f-4d85-08df02d93402 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|7416014|1800799024|366016|10067099003|56012099006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: VCKYWovPEomLldAfq+zGQFCuomy4IgseiYR975thVZvjgSccEeJVwmrNuYAjBda7eBBPEQ3pKErs3lRpwIZUCKoUXTOoWQSHtjKIA8fN0Y+jS+kNsyfYwJive/mmDgrIpOq/TPfv2zOAKyj2B28tahJkjgAV71iFAJ6c8EpOEbcL55u9zTBIG6j3Bm34g+E2pHHSbaipo/i73AIWVhzB6jFa2/XeCBQORs5rMgdtPMRmMbl3sxItcCaLd0KgvsBKgU3spmKJDdHo140yxT62eATbuDpYV4YcIiRlkaBk2qQt0VGGVNM2ZcFCpqBNpjJPRcZMD7GhTnW5zj4Md5DEEp3kFGy0qDG93AhTOUOC+I7IehD8i7MLX/AG4Odev/cAnyu4ts2dDCylOOJQxUOO6OxoqEFC3crUK0jd2xhnuAm+85ZTqHKZN2zH70vvg+AOBLshUA2M+0JfP3BGT7ZBPyDbfoTpS6l8v2a5/IICJlac5bNW45THDlUzD6ctYzkq49CniZL9CW/9PTnuN+ANN1H4tb2ruwablktHZMWUkTatb8nSzzbAAz42cGRoORnP57NWRZtriqWkUqRaj8WFms4iv+pSSUEgOk7AMQN9AKjYPMKsKIxH9IF8cs2Anfs2zocu4W5jYmCoFIASTmi/E1x3QJ4FeELQkWqnae8tl7Y= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(7416014)(1800799024)(366016)(10067099003)(56012099006)(18002099003)(22082099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nbRv4NAvR9d9UEIiY25JG/HkaZFCsRerjPUUlpZrurpiZnTC3gbtj2y6qvBi?= =?us-ascii?Q?6t/ZbpqJcv6g1PfHtKx9QLIzcWM/scy7DcaDGRan9A/6LpmX/EXHykQ8OSB1?= =?us-ascii?Q?4ocw+WL0ZUfFTqJOqr420k7TVOzoiLQHmBxYrVmHBnpWQpK2xT9Ajt0n1TNS?= =?us-ascii?Q?U9/X6OITl5Ef9G+A4VnYIzBRaN5MtRTKZaM4JqMfv+/iuSe5vDZsNN/0gKvZ?= =?us-ascii?Q?BL6fo5Tt0cQ0J+Q7MjFrjaFBZ1GITMrif0WMj7ubXIING5aiuXvkwJkjum7I?= =?us-ascii?Q?PSb/kKbgL7godx5gZdPPpo/lLDdcJiX6RsbMHUU9R4MoF31SuX3J6Aib7U7V?= =?us-ascii?Q?pHCUaKcoflfNZgYbypUVg2kE8kLmXOW1l9scjjSxbCrpCdH4EK4qx3obqkiY?= =?us-ascii?Q?1y8RRYbYSYgmNw0SEqp1CgBKz6tKEBtn5xO1+lctoXK6z93wI7W++qRmHn9y?= =?us-ascii?Q?D82zOhu9Ha11EfkOFnvzIZyvsY6awbEfFGxVh7eC9LCAkF4+aPPemHv6EHQs?= =?us-ascii?Q?sT+cfQ3rOL3cmuCT9cBGMmRS/GzyN2yvgNpjLiSbNzMY4dhgYiT5+uWx+Imk?= =?us-ascii?Q?O4Gc9HAnXH4EsMJ89cu7mzNTgXP0aWR2yFF6b3WpitDMGFjC6zcFNC7RnDOD?= =?us-ascii?Q?ySUBQbcVFsUdijytNb6ZRDxt+ZRQTSZ6vid7248i1OQbhC1QaZRHuteNH7Qe?= =?us-ascii?Q?EC4zSUrYYV+PCwlnP9/KHevjgbCNQF1y09Ewio0G4e0RpWI9J0BxIEGlKI1J?= =?us-ascii?Q?kvezsw99fw0fgvaxzMQBsRrUmqgZaRfhdclfp6Q5qMQlRF0g+mP7RqSyE+mR?= =?us-ascii?Q?vj+D7/uV95WoIs0AaNO1qb/N8XWsVLmzp4VqkldsJCHnfzwSW386Olxz7h5b?= =?us-ascii?Q?sywPFUMFc211A8CAl16Q2mJkY31gSb/3qqe5r0Et0g3u8Nue5k7dbfpsufvd?= =?us-ascii?Q?3Tz61P9sCiZ2ySIglwrNk1sU6mgv1ivEwNMvM0rb2K73Z3lzs+5Ac83Qnv3H?= =?us-ascii?Q?gAtvviREv/S1LvRSvNpI/kSd9df3Q8ZeMrH+sDLBGYaAYHf4N0+YjNuwkBcO?= =?us-ascii?Q?znREWFTn2HAdmjCsIfBtuQ/NExjrxB4YmXUoDE2SwvvZTl2zXl1KK+oxIwuG?= =?us-ascii?Q?+9u8sxbRWfwrZ4r4VKUhapKlTv8A6P5Ia9heh4/ZIAqPgSzmRtTI2OaPXbk1?= =?us-ascii?Q?sRm8UzsIxajdTR8QNzIdxMfJfrnAxzhovxX9s3LLtall0BoGnbpEhWU+c94D?= =?us-ascii?Q?P7cMmtdphhc4sXSlyV3lMnEwYClR6y5EAyVERuHYab4nuAfYgGk+IWHnHWlS?= =?us-ascii?Q?8juWdJHYHBRMowm2/BJIC6slTRDnOXivH5lqs7eKKuvM4aAh4BiC7G14kIip?= =?us-ascii?Q?hD6lkhf2uXcj/NJtLVqHiyyiSC7SQLAU5OVH383CuZ8/8yv5Oy/ij6FlPK2c?= =?us-ascii?Q?Uk97s6JIjJaSclUM4hQS54hOSfuU5TJz4DIj72pHdf/tKeQEKg2++g0BfT+E?= =?us-ascii?Q?cEOBJjtQ0MhQUfbVMAKhNUVI8BsBY/tM0gZgxun/nhn6gkhvZ9CSLCcAniwR?= =?us-ascii?Q?q7nKUn5eUBnRiFOPD3PMTMVe9vnHjwCCbsxQObjse+lOF7iNkhBtDq9SzZyb?= =?us-ascii?Q?dmsZ2x7mQWPtLVFoVApeJgihgSNXBjE/dDUbJEG1ibrg74FPiVNnKptrUWsn?= =?us-ascii?Q?/KQCZjS6QemGuACi/nLFc445Qy3NFtE/Gbl673JSQPTfGHhPFX0vlW+rEg9R?= =?us-ascii?Q?0UoKx7Kn2A=3D=3D?= X-OriginatorOrg: atomlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: 52d5f4c2-cab2-4a4f-4d85-08df02d93402 X-MS-Exchange-CrossTenant-AuthSource: LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2026 18:46:41.4558 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e6a32402-7d7b-4830-9a2b-76945bbbcb57 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EIU1VKPwS0qoEcOVbu3K/lPYWDG50aCG8En3SJwkPWv7G2VOa65oXgpRun4gCNbAAfKTL7HtaHkn5WVHPSb/nw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO7P123MB7501 The root_domain pointer rd field in struct rq is updated dynamically using RCU, and its memory reclamation is deferred via call_rcu() in rq_attach_root(). However, struct rq's rd field was missing the __rcu compiler annotation, and several lockless readers across the scheduler subsystem accessed rq->rd directly without using RCU dereference primitives. Add the __rcu annotation to struct rq's rd field in kernel/sched/sched.h. Update lockless readers across kernel/sched/ to use rcu_dereference(), rcu_dereference_sched() or rcu_access_pointer() appropriately. This ensures proper data-dependency barriers on all architectures, enables Sparse static analysis validation, and documents RCU read-side ownership contracts. Signed-off-by: Aaron Tomlin --- kernel/sched/core.c | 16 ++++++++++------ kernel/sched/deadline.c | 8 ++++---- kernel/sched/fair.c | 29 +++++++++++++++-------------- kernel/sched/sched.h | 2 +- 4 files changed, 30 insertions(+), 25 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2e7cde033a31..8c81f7c0cea0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8577,10 +8577,12 @@ void set_rq_offline(struct rq *rq) static inline void sched_set_rq_online(struct rq *rq, int cpu) { struct rq_flags rf; + struct root_domain *rd; rq_lock_irqsave(rq, &rf); - if (rq->rd) { - BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span)); + rd = rcu_dereference_protected(rq->rd, lockdep_is_held(&rq->__lock)); + if (rd) { + BUG_ON(!cpumask_test_cpu(cpu, rd->span)); set_rq_online(rq); } rq_unlock_irqrestore(rq, &rf); @@ -8589,10 +8591,12 @@ static inline void sched_set_rq_online(struct rq *rq, int cpu) static inline void sched_set_rq_offline(struct rq *rq, int cpu) { struct rq_flags rf; + struct root_domain *rd; rq_lock_irqsave(rq, &rf); - if (rq->rd) { - BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span)); + rd = rcu_dereference_protected(rq->rd, lockdep_is_held(&rq->__lock)); + if (rd) { + BUG_ON(!cpumask_test_cpu(cpu, rd->span)); set_rq_offline(rq); } rq_unlock_irqrestore(rq, &rf); @@ -9009,8 +9013,8 @@ void __init sched_init(void) #endif rq->next_class = &idle_sched_class; - rq->sd = NULL; - rq->rd = NULL; + RCU_INIT_POINTER(rq->sd, NULL); + RCU_INIT_POINTER(rq->rd, NULL); rq->cpu_capacity = SCHED_CAPACITY_SCALE; rq->balance_callback = &balance_push_callback; rq->active_balance = 0; diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 857dbe3519a8..4137f8bbcef5 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -122,12 +122,12 @@ static inline struct dl_bw *dl_bw_of(int i) { RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), "sched RCU must be held"); - return &cpu_rq(i)->rd->dl_bw; + return &rcu_dereference_sched(cpu_rq(i)->rd)->dl_bw; } static inline int dl_bw_cpus(int i) { - struct root_domain *rd = cpu_rq(i)->rd; + struct root_domain *rd = rcu_dereference_sched(cpu_rq(i)->rd); RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), "sched RCU must be held"); @@ -159,13 +159,13 @@ static inline unsigned long dl_bw_capacity(int i) RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), "sched RCU must be held"); - return __dl_bw_capacity(cpu_rq(i)->rd->span); + return __dl_bw_capacity(rcu_dereference_sched(cpu_rq(i)->rd)->span); } } bool dl_bw_visited(int cpu, u64 cookie) { - struct root_domain *rd = cpu_rq(cpu)->rd; + struct root_domain *rd = rcu_dereference_sched(cpu_rq(cpu)->rd); if (rd->visit_cookie == cookie) return true; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f79fcba4afec..d55c21811306 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7865,13 +7865,10 @@ static inline void set_rd_overutilized(struct root_domain *rd, bool flag) static inline void check_update_overutilized_status(struct rq *rq) { - /* - * overutilized field is used for load balancing decisions only - * if energy aware scheduler is being used - */ + struct root_domain *rd = rcu_dereference(rq->rd); - if (!is_rd_overutilized(rq->rd) && cpu_overutilized(rq->cpu)) - set_rd_overutilized(rq->rd, 1); + if (rd && !is_rd_overutilized(rd) && cpu_overutilized(rq->cpu)) + set_rd_overutilized(rd, 1); } /* Runqueue only has SCHED_IDLE tasks enqueued */ @@ -9500,7 +9497,7 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu) unsigned long prev_delta = ULONG_MAX, best_delta = ULONG_MAX; unsigned long p_util_min = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MIN) : 0; unsigned long p_util_max = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MAX) : 1024; - struct root_domain *rd = this_rq()->rd; + struct root_domain *rd = rcu_dereference(this_rq()->rd); int cpu, best_energy_cpu, target = -1; int prev_fits = -1, best_fits = -1; unsigned long best_actual_cap = 0; @@ -9704,7 +9701,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) cpumask_test_cpu(cpu, p->cpus_ptr)) return cpu; - if (!is_rd_overutilized(this_rq()->rd)) { + if (!is_rd_overutilized(rcu_dereference(this_rq()->rd))) { new_cpu = find_energy_efficient_cpu(p, prev_cpu); if (new_cpu >= 0) return new_cpu; @@ -12690,13 +12687,15 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd env->fbq_type = fbq_classify_group(&sds->busiest_stat); if (!env->sd->parent) { + struct root_domain *rd = rcu_dereference(env->dst_rq->rd); + /* update overload indicator if we are at root domain */ - set_rd_overloaded(env->dst_rq->rd, sg_overloaded); + set_rd_overloaded(rd, sg_overloaded); /* Update over-utilization (tipping point, U >= 0) indicator */ - set_rd_overutilized(env->dst_rq->rd, sg_overutilized); + set_rd_overutilized(rd, sg_overutilized); } else if (sg_overutilized) { - set_rd_overutilized(env->dst_rq->rd, sg_overutilized); + set_rd_overutilized(rcu_dereference(env->dst_rq->rd), sg_overutilized); } update_idle_cpu_scan(env, sum_util); @@ -12942,8 +12941,10 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env) if (busiest->group_type == group_misfit_task) goto force_balance; - if (!is_rd_overutilized(env->dst_rq->rd) && - rcu_dereference_all(env->dst_rq->rd->pd)) + struct root_domain *rd = rcu_dereference(env->dst_rq->rd); + + if (rd && !is_rd_overutilized(rd) && + rcu_dereference_all(rd->pd)) goto out_balanced; /* ASYM feature bypasses nice load balance check */ @@ -14573,7 +14574,7 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf) if (!sd) goto out; - if (!get_rd_overloaded(this_rq->rd) || + if (!get_rd_overloaded(rcu_dereference(this_rq->rd)) || this_rq->avg_idle < sd->max_newidle_lb_cost) { update_next_balance(sd, &next_balance); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 26ae13c86b69..13a437032855 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1256,7 +1256,7 @@ struct rq { int membarrier_state; #endif - struct root_domain *rd; + struct root_domain __rcu *rd; struct sched_domain __rcu *sd; struct balance_callback *balance_callback; -- 2.55.0