From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63A0638D3E0 for ; Thu, 2 Jul 2026 19:28:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=158.69.130.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783020529; cv=none; b=TcbGmb//0ASWjXJIwVrJygBULG0OJ4Jd89XQ5hXlTT5K01P0a2KrvurS5Tyt8quSY0c9tXvGFFmIOLLvt5kjj8WHvZUtC8XpBk1+Dy4M1wZtELfNTyQWD6IIxbNuMZzhJMl2V/R5kLTSDyplDKVQdZlG44pdigD+BbXnIZ8M/yo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783020529; c=relaxed/simple; bh=4fz76FfEPVgS1YycMMJl1WjEiJPFhMbyhdGbXtzMdJY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BuA/plh0crVEDVuZCJftuCVvnelX1knPyfU8i5IbCkSRuqTBBAa29rbdCJMnEJULc03IMEViI658Ihp6Z09hhuxtAHOEIzbWG4SNxuxvNYNWNaeVK0EMXlqol3t0cjg2n3R03y3AWS51U4JDueJkmL+vWuO2TGf5v8js9WncoGM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=s+ZSOJ3V; arc=none smtp.client-ip=158.69.130.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="s+ZSOJ3V" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1783020143; bh=k29Fcw9auekiYo5/Hy+pXaWJ+Eenn65/fBLI33kysuo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s+ZSOJ3VURN+Oauib5pvVhcmzUqT4m4St2eMT8+Ua98L0l5MLyHDgGN5xJtDGjl5v izBM/OzK9JgvsphjUshqoKvzXxTZ6kKeTNsGG3RiZauXTicAKAARbf/di/ya0rtW/U 3dSn1nE61k+Obudk9V5He5dUYzf43WuQ5FoQ1+ydvCjIEW15cLzxg2AP3+A/Dum81k PW7GDzCxEgHDiwbUIU5eAblfYBPof5gaFep4ubUMU5NZ1Qob8yarzAqtxUxBOLGPGh iAabeeAbb3lSNpZhBWxY+RoyQz30cNffav6bx/lzDU6qCK4odgNnTGvdmaBc2yFwPm MIHX/Tr9VnJkQ== Received: from compudjdev.. (mtl.efficios.com [216.120.195.104]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4grmwv0VwpzS4R; Thu, 02 Jul 2026 15:22:23 -0400 (EDT) From: Mathieu Desnoyers To: Andrew Morton Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers Subject: [PATCH v19 4/6] mm: reorder mm_struct flexible array to place mm_cpumask first Date: Thu, 2 Jul 2026 15:21:57 -0400 Message-ID: <20260702192214.41467-5-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260702192214.41467-1-mathieu.desnoyers@efficios.com> References: <20260702192214.41467-1-mathieu.desnoyers@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Reorder the mm_struct flexible array layout from: [HPCC tree items][mm_cpumask][mm_cid] to: [mm_cpumask][mm_cid][alignment padding][HPCC tree items] The previous layout placed the HPCC tree items first, which required mm_cpumask() to skip past them using percpu_counter_tree_items_size(). This created a boot-time initialization ordering dependency: any use of mm_cpumask() before percpu_counter_tree_subsystem_init() would compute the wrong pointer offset, reading from or writing to the HPCC items region instead of the actual cpumask. On powerpc, switch_mm_irqs_off() accesses mm_cpumask() early in boot via VM_WARN_ON_ONCE(!cpumask_test_cpu(cpu, mm_cpumask(prev))), which could fire spuriously or cause silent memory corruption if the HPCC subsystem was not yet initialized. By placing mm_cpumask first, mm_cpumask() remains a simple &mm->flexible_array with no runtime dependency on HPCC initialization. The HPCC tree items are accessed via get_rss_stat_items_offset(), which skips past the cpumask and mm_cid with appropriate cacheline alignment padding. This accessor is only used during mm_struct initialization (percpu_counter_tree_init_many), not on context switch or other hot paths. Introduce the PERCPU_COUNTER_TREE_ITEMS_ALIGN() macro to handle the SMP cacheline alignment vs !SMP no-op in a single place, used by both the static init_mm flexible array initializer and the runtime offset computation. The cost is at most one cacheline (typically 64 bytes) of padding per mm_struct between the mm_cid data and the HPCC tree items. Signed-off-by: Mathieu Desnoyers --- include/linux/mm.h | 1 + include/linux/mm_types.h | 47 +++++++++++++++++------------ include/linux/percpu_counter_tree.h | 3 ++ kernel/fork.c | 4 ++- 4 files changed, 34 insertions(+), 21 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 45b49f15bb5e..b1f6703cace8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3344,6 +3344,7 @@ static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct m unsigned long ptr = (unsigned long)mm; ptr += offsetof(struct mm_struct, flexible_array); + ptr += get_rss_stat_items_offset(); return (struct percpu_counter_tree_level_item *)ptr; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 414618ffe329..4e51b666d72b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1420,11 +1420,12 @@ struct mm_struct { } __randomize_layout; /* - * The rss hierarchical counter items, mm_cpumask, and mm_cid - * masks need to be at the end of mm_struct, because they are + * The mm_cpumask, mm_cid masks, and rss hierarchical counter + * items need to be at the end of mm_struct, because they are * dynamically sized based on nr_cpu_ids. - * The content of the flexible array needs to be placed in - * decreasing alignment requirement order. + * The HPCC counter tree items are placed last and aligned + * with PERCPU_COUNTER_TREE_ITEMS_ALIGN to satisfy their + * cacheline alignment requirement. */ char flexible_array[] __mm_struct_flexible_array_aligned; }; @@ -1463,25 +1464,18 @@ static inline void __mm_flags_set_mask_bits_word(struct mm_struct *mm, MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; -#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \ -{ \ - [0 ... (PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE * NR_MM_COUNTERS) + sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0 \ -} - -static inline size_t get_rss_stat_items_size(void) -{ - return percpu_counter_tree_items_size() * NR_MM_COUNTERS; +#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \ +{ \ + [0 ... PERCPU_COUNTER_TREE_ITEMS_ALIGN( \ + sizeof(cpumask_t) + MM_CID_STATIC_SIZE) \ + + (PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE * NR_MM_COUNTERS) \ + - 1] = 0 \ } /* Future-safe accessor for struct mm_struct's cpu_vm_mask. */ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) { - unsigned long ptr = (unsigned long)mm; - - ptr += offsetof(struct mm_struct, flexible_array); - /* Skip RSS stats counters. */ - ptr += get_rss_stat_items_size(); - return (struct cpumask *)ptr; + return (struct cpumask *)&mm->flexible_array; } static inline void mm_init_cpumask(struct mm_struct *mm) @@ -1578,8 +1572,6 @@ static inline cpumask_t *mm_cpus_allowed(struct mm_struct *mm) unsigned long bitmap = (unsigned long)mm; bitmap += offsetof(struct mm_struct, flexible_array); - /* Skip RSS stats counters. */ - bitmap += get_rss_stat_items_size(); /* Skip cpu_bitmap */ bitmap += cpumask_size(); return (struct cpumask *)bitmap; @@ -1662,6 +1654,21 @@ static inline void mm_destroy_sched(struct mm_struct *mm) { } #endif /* CONFIG_SCHED_CACHE */ +static inline size_t get_rss_stat_items_size(void) +{ + return percpu_counter_tree_items_size() * NR_MM_COUNTERS; +} + +/* + * Return the offset of the RSS stat HPCC items within the mm_struct + * flexible array. The items are placed after the cpumask and mm_cid, + * aligned to the cacheline boundary required by the tree level items. + */ +static inline size_t get_rss_stat_items_offset(void) +{ + return PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size()); +} + struct mmu_gather; extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm); extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm); diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h index 828c763edd4a..3e8a820e2d1d 100644 --- a/include/linux/percpu_counter_tree.h +++ b/include/linux/percpu_counter_tree.h @@ -66,6 +66,8 @@ struct percpu_counter_tree_level_item { #define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE \ (PERCPU_COUNTER_TREE_STATIC_NR_ITEMS * sizeof(struct percpu_counter_tree_level_item)) +#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) \ + ALIGN((offset), __alignof__(struct percpu_counter_tree_level_item)) struct percpu_counter_tree { /* Fast-path fields. */ @@ -167,6 +169,7 @@ void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree * #else /* !CONFIG_SMP */ #define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE 0 +#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) (offset) struct percpu_counter_tree_level_item; diff --git a/kernel/fork.c b/kernel/fork.c index cf48409a7737..faca9760efb8 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -3106,7 +3106,9 @@ void __init mm_cache_init(void) * dynamically sized based on the maximum CPU number this system * can have, taking hotplug into account (nr_cpu_ids). */ - mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size() + get_rss_stat_items_size(); + mm_size = sizeof(struct mm_struct) + + PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size()) + + get_rss_stat_items_size(); mm_cachep = kmem_cache_create_usercopy("mm_struct", mm_size, ARCH_MIN_MMSTRUCT_ALIGN, -- 2.43.0