From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl2-f40.google.com (mail-dl2-f40.google.com [74.125.229.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 798294A64E5 for ; Fri, 2 Oct 2026 13:11:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.229.168 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790946688; cv=none; b=pwq8XJ991TWZANbANVUk1RlGHZOWFAhWrLguujRhpM51uTQn3nk9URPLodOUmEL1qDLbSunBI36+tVsNJvaz0lYqub62yCzkU+4fUvkzh9d62lD00f7WfaIfvkcQ6z1Zs2Pi54TQ1keD+E4njUz6TBufUwfii+NXmlD91lGgSU0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790946688; c=relaxed/simple; bh=+SqvbLaUu9l7Y+KxTii2lnGODDCoc7bkuP468WrdWtw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ZU1kBONSxz1MaSIPWzuIIQgQ3RRPi5A4DLf/ldU9OJfN9J6C2Ff0T5hHMV0svoASnGZMYTe2MMTosMKYx8e8JeZVaaQYqoo4ymBJebJmxb2X5by/k9OHWqxRhdRR3X+t+QVnB9ADv4lTYTVRWOxdDI4yJMBzCMTEH0++P+sPE2U= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=L3jttJA+; arc=none smtp.client-ip=74.125.229.168 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="L3jttJA+" Received: by mail-dl2-f40.google.com with SMTP id a92af1059eb24-14f381f0424so540755c88.0 for ; Fri, 02 Oct 2026 06:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790946682; x=1791551482; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Ve2PMleKVRfVJjBVnkNE6jeqRclzRFzcrDXbN4vG6Q8=; b=L3jttJA+cHCsnglHef5Bm2QtwpHPwD95DXbSZJegqD+6NUm89HEpBR5TaGxfcnNN+b IEpGufVq1c/RVbfWeyG0UHVViid6Nu7NpEV9Kcvo868vUDaKUrnVurcLEdIJGcSwVBfs le3Oc0C4qZ5uNuNBJVF2GA95GUl69+puOmdThPNGXDeTif4nkSjgiwQLCMxUfsSyeDq+ Nhs+E4CVCvOBDOQucGnaf0mqqXpZX68rmdRVxj7KVvtMT8snnGswPp65QL93A4lnk7W+ kXiglkM/SMsO4LMW/fJPSHYE99o/rUWB9gxNtQ/L/hp64rExWKktIy2p3UKvg6bXb9qt qhCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790946682; x=1791551482; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Ve2PMleKVRfVJjBVnkNE6jeqRclzRFzcrDXbN4vG6Q8=; b=vmuBPC4YN7UhQ7fDxToJuJf8Up3rsLlKsDD9FsLG1HUySEWEPiC8Iino9G5tHXGCxT SVR8i9caOKFgiaF/qGVfH+xXAoNttUc7l79S9HVaw9kHyAXj+I03EaDg3KXtFKsIkA/S hQqnRZGEaZcqxvQ5WoyCd6xcTsb/yCpR3RFBMbeh4pxlKxTEOGmuSgdJVLsBVFZHoH/S 41KlI0KbdpUrBPcZfSFKsMDBXsud1JUC1eLAPulv65P7OtXUoQ+Idl4MgrV7w6koxB7t exLD1H0aRHwMBFPRfuaNSuYheGWjXOGzP1fF8mAx1E62qB1jIZ6MKwNYx7Z0h4SHbmi6 HPPg== X-Forwarded-Encrypted: i=1; AKwUvByxhoYK4sDJ4qqi/pHNlUrj+yERFHPDlSwgWjvpl6uVZMGa9RshUZEllhm/XvOUyVyffW1r6LRqQPOjvZ4=@vger.kernel.org X-Gm-Message-State: AFuF++mJusD6O4+w6vKI+l6STV3X9LGxWDzWyFY8xBOcc4BI1Y1llPqm wvu/tySnqaau3fJyVjavLKsjT3N/OkBn+bBKa34bKwRJnQr+XO5WB1E3 X-Gm-Gg: AYBFou0wuz/bKUYLq+a2xMysfQ6ut6lVl5gKnkx52A36M7EeEy4z6LywUHlH8jKo9qL sn3JrUiKDvPujlWvhBV3+je2CCSB+AMIKBzmSe1kLsXCe3M5RFmRhsn3Q1sTUp8Kqdf2joJ0AVp 5WJgCvy9lNaO9zILdDDWTUlt1TdPKNfesr/VVZsDPxPQVKTYmP9vJ8KJw3+HoXYMSXNE8hIY6nt vrynV/44VLiINaTObvOVIU/auLpkgvNAB3Jx+myNqqDFiSDAyJw+7U9jgfzq8UcJErBl5necAz6 JVks38cjfbHZi3FMRgrAtnV2Xem7PSF7eaOCYR+hxidvBDu8XgEZsVACIUvd7+zmAwAJ7xZWYXY OQRH6dnt9PU0oyduD3SK2fG/Its3B1PB1IXlfa969Uh/hJXZiYC/KWPcYHz4aP0SaOUSHvapDmT l12G3M91JOQH4FWx4iLxV4u7NZAKOtcukbM2Aas5rd2+p4++2JMVEgGhyUxjOfWc6SrA== X-Received: by 2002:a05:7022:f90a:b0:143:33de:978f with SMTP id a92af1059eb24-14f5d5b569emr2488888c88.43.1790946681636; Fri, 02 Oct 2026 06:11:21 -0700 (PDT) Received: from [127.0.1.1] ([23.254.208.9]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-34f0672c739sm7309347eec.3.2026.10.02.06.11.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Oct 2026 06:11:21 -0700 (PDT) From: Qiliang Yuan Date: Fri, 02 Oct 2026 21:10:26 +0800 Subject: [PATCH v5 06/12] rcu/nocb: Support lazy init for runtime CPU isolation Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261002-wujing-dhm-v5-6-78a6996d87ad@gmail.com> References: <20261002-wujing-dhm-v5-0-78a6996d87ad@gmail.com> In-Reply-To: <20261002-wujing-dhm-v5-0-78a6996d87ad@gmail.com> To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Anna-Maria Behnsen , Tejun Heo , Jonathan Corbet , Shuah Khan , Shuah Khan , Thomas Gleixner Cc: Waiman Long , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Qiliang Yuan X-Mailer: b4 0.13.0 When a cpuset isolated partition first requests kernel-noise isolation, it needs to offload RCU callbacks from the affected CPUs. The existing rcu_nocb_cpu_offload() requires rcu_nocbs= or nohz_full= at boot so that rcu_organize_nocb_kthreads() has already set rdp->nocb_gp_rdp for each CPU. Without a boot parameter, nocb_gp_rdp is NULL and offload fails immediately. Introduce lazy nocb initialization so that the first call into the isolation path triggers the one-time setup automatically: rcu_nocb_lazy_init() - allocates rcu_nocb_mask and calls rcu_organize_nocb_kthreads() (now without __init) to set nocb_gp_rdp for every possible CPU. Uses a dedicated mutex for serialization with a fast-path read of nocb_is_setup. Registers the CONFIG_RCU_LAZY shrinker on this first-ever setup too: rcu_init_nohz() only does so when nocb_is_setup was already true at boot, so the lazy-callback reclaim path would otherwise never exist on a system with no rcu_nocbs=/nohz_full= boot parameter. Factor the registration into rcu_nocb_register_lazy_shrinker(), shared with rcu_init_nohz(). rcu_nocb_cpu_isolate() - exported entry point called per-CPU while the CPU is offline. Calls rcu_nocb_lazy_init() for the one-time setup, spawns the GP and CB kthreads via the existing rcu_spawn_cpu_nocb_kthread(), then finalizes offload through rcu_nocb_cpu_offload(). Adding the CPU to the GP kthread's nocb_head_rdp list is handled by nocb_gp_toggle_rdp() in the GP kthread, so rcu_organize_nocb_kthreads() can run with an empty mask. Remove __init from rcu_organize_nocb_kthreads() to allow this runtime call path; the function itself has no __initdata dependencies. Signed-off-by: Qiliang Yuan --- include/linux/rcupdate.h | 2 ++ kernel/rcu/tree.h | 2 +- kernel/rcu/tree_nocb.h | 81 ++++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 71 insertions(+), 14 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 44c07a66edfff..1425f04ceb4e9 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -158,6 +158,7 @@ static __always_inline void rcu_irq_work_resched(void) { } void rcu_init_nohz(void); int rcu_nocb_cpu_offload(int cpu); int rcu_nocb_cpu_deoffload(int cpu); +int rcu_nocb_cpu_isolate(int cpu); void rcu_nocb_flush_deferred_wakeup(void); #define RCU_NOCB_LOCKDEP_WARN(c, s) RCU_LOCKDEP_WARN(c, s) @@ -167,6 +168,7 @@ void rcu_nocb_flush_deferred_wakeup(void); static inline void rcu_init_nohz(void) { } static inline int rcu_nocb_cpu_offload(int cpu) { return -EINVAL; } static inline int rcu_nocb_cpu_deoffload(int cpu) { return 0; } +static inline int rcu_nocb_cpu_isolate(int cpu) { return -EINVAL; } static inline void rcu_nocb_flush_deferred_wakeup(void) { } #define RCU_NOCB_LOCKDEP_WARN(c, s) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index eedfa43059e80..60bcf52817f3e 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -522,7 +522,7 @@ static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp, unsigned long flags); static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp); #ifdef CONFIG_RCU_NOCB_CPU -static void __init rcu_organize_nocb_kthreads(void); +static void rcu_organize_nocb_kthreads(void); /* * Disable IRQs before checking offloaded state so that local diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 19bb42672baf8..99f3e4cec34e3 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1344,12 +1344,30 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) } #endif // #ifdef CONFIG_RCU_LAZY +#ifdef CONFIG_RCU_LAZY +static void rcu_nocb_register_lazy_shrinker(void) +{ + struct shrinker *lazy_rcu_shrinker = shrinker_alloc(0, "rcu-lazy"); + + if (!lazy_rcu_shrinker) { + pr_err("Failed to allocate lazy_rcu shrinker!\n"); + return; + } + + lazy_rcu_shrinker->count_objects = lazy_rcu_shrink_count; + lazy_rcu_shrinker->scan_objects = lazy_rcu_shrink_scan; + + shrinker_register(lazy_rcu_shrinker); +} +#else +static void rcu_nocb_register_lazy_shrinker(void) { } +#endif // #ifdef CONFIG_RCU_LAZY + void __init rcu_init_nohz(void) { int cpu; struct rcu_data *rdp; const struct cpumask *cpumask = NULL; - struct shrinker * __maybe_unused lazy_rcu_shrinker; #if defined(CONFIG_NO_HZ_FULL) if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask)) @@ -1375,17 +1393,7 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; -#ifdef CONFIG_RCU_LAZY - lazy_rcu_shrinker = shrinker_alloc(0, "rcu-lazy"); - if (!lazy_rcu_shrinker) { - pr_err("Failed to allocate lazy_rcu shrinker!\n"); - } else { - lazy_rcu_shrinker->count_objects = lazy_rcu_shrink_count; - lazy_rcu_shrinker->scan_objects = lazy_rcu_shrink_scan; - - shrinker_register(lazy_rcu_shrinker); - } -#endif // #ifdef CONFIG_RCU_LAZY + rcu_nocb_register_lazy_shrinker(); if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); @@ -1409,6 +1417,53 @@ void __init rcu_init_nohz(void) rcu_organize_nocb_kthreads(); } +static DEFINE_MUTEX(rcu_nocb_lazy_mutex); + +/* + * Lazily initialize nocb infrastructure on the first call. Allocates + * rcu_nocb_mask and sets nocb_gp_rdp for every possible CPU so that + * rcu_nocb_cpu_isolate() can offload callbacks without rcu_nocbs= at boot. + */ +static noinline int rcu_nocb_lazy_init(void) +{ + if (rcu_state.nocb_is_setup) + return 0; + + mutex_lock(&rcu_nocb_lazy_mutex); + if (!rcu_state.nocb_is_setup) { + if (!zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL)) { + mutex_unlock(&rcu_nocb_lazy_mutex); + return -ENOMEM; + } + rcu_organize_nocb_kthreads(); + /* + * Boot-time rcu_init_nohz() only registers the lazy-callback + * shrinker when nocb_is_setup was already true at boot; this + * is the first time it is set, so register it here instead. + */ + rcu_nocb_register_lazy_shrinker(); + rcu_state.nocb_is_setup = true; + } + mutex_unlock(&rcu_nocb_lazy_mutex); + return 0; +} + +/* + * Offload RCU callbacks for a CPU entering a kernel-noise isolated partition. + * @cpu must be offline. Lazily initializes nocb infrastructure on first use. + */ +int rcu_nocb_cpu_isolate(int cpu) +{ + int ret; + + ret = rcu_nocb_lazy_init(); + if (ret) + return ret; + rcu_spawn_cpu_nocb_kthread(cpu); + return rcu_nocb_cpu_offload(cpu); +} +EXPORT_SYMBOL_GPL(rcu_nocb_cpu_isolate); + /* Initialize per-rcu_data variables for no-CBs CPUs. */ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) { @@ -1501,7 +1556,7 @@ module_param(rcu_nocb_gp_stride, int, 0444); /* * Initialize GP-CB relationships for all no-CBs CPU. */ -static void __init rcu_organize_nocb_kthreads(void) +static void rcu_organize_nocb_kthreads(void) { int cpu; bool firsttime = true; -- 2.43.0