From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA6FF414A07 for ; Tue, 7 Jul 2026 13:16:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=158.69.130.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783430178; cv=none; b=l0j/T2Ek1eYF7eLxm2Q9EY4urqlrqtndwdWFibD7OYy+P0z79d6xKc+2xX1seCoU+EopM0s2jqYbWN+OuY6VHrk/b6Z/MONkjB7SOKRBS4cIEucEdnBJnQDaGa/5UaYneSYu8IwC7/ZDa8BZPVGtegcttTR/TDBThtK1ZPA1iN8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783430178; c=relaxed/simple; bh=jQPvaFVqN59DJi1ytGyR77toplHD2afaBxn3fnMRqRY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AG6yWs/tO6s0SqfW18/i2BW840qCYDGAHgFTn4PwRGF6RLr3le0roVjvkhJJtvdS3r5IaSc81ZwfZmssgl7JXzyBN7SXea94NWKo5JHj1j+BcJNLaKE06CuMjtusBniLOqnjVce2hti3l6zDRkiVzFiSQAnWhW0c7jfL95AuWKA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=Gsl7R0dN; arc=none smtp.client-ip=158.69.130.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="Gsl7R0dN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1783430170; bh=NLVsTA79D3eBsqKyFzMDEDGTJjQykSEp3X7YWYLl/cM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Gsl7R0dNtjip3vuNk6STS1X+NWp4auQfjQLDIWYZsWo8nT7MbcoJMDdvnoX8yiZ+3 ZC4uR9k6NWfpIpjvMu+jiuOOuR9VTtLxsHTA/uuXbji8im/INA23hv1GNwosew9B8i W7Lsool62l8ugM+3Hd8/U271zkpocE2BFfWO6tbkio3xNwA7aibiVnQbwBa+fLEAOn CXnDFxqIb0oeFNoAuZvG4LrHow3oqtg2JYIo6z7UY4Nh5Yi/NrRg+eK3SPymoInhR6 /z4OnEUs1i5HWp56t2aqB1N5MRptpI1fXYLa9kh/jjCjBPG6TUhA9Eu6C9hFyWUzwL uSn5irr9ugEqg== Received: from compudjdev.. (mtl.efficios.com [216.120.195.104]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4gvhZ24BXYzTlr; Tue, 07 Jul 2026 09:16:10 -0400 (EDT) From: Mathieu Desnoyers To: Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers Subject: [PATCH v20 4/6] mm: reorder mm_struct flexible array to place mm_cpumask first Date: Tue, 7 Jul 2026 09:15:34 -0400 Message-ID: <20260707131544.75906-5-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260707131544.75906-1-mathieu.desnoyers@efficios.com> References: <20260707131544.75906-1-mathieu.desnoyers@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Reorder the mm_struct flexible array layout from: [HPCC tree items][mm_cpumask][mm_cid] to: [mm_cpumask][mm_cid][alignment padding][HPCC tree items] The previous layout placed the HPCC tree items first, which required mm_cpumask() to skip past them using percpu_counter_tree_items_size(). This created a boot-time initialization ordering dependency: any use of mm_cpumask() before percpu_counter_tree_subsystem_init() would compute the wrong pointer offset, reading from or writing to the HPCC items region instead of the actual cpumask. On powerpc, switch_mm_irqs_off() accesses mm_cpumask() early in boot via VM_WARN_ON_ONCE(!cpumask_test_cpu(cpu, mm_cpumask(prev))), which could fire spuriously or cause silent memory corruption if the HPCC subsystem was not yet initialized. By placing mm_cpumask first, mm_cpumask() remains a simple &mm->flexible_array with no runtime dependency on HPCC initialization. The HPCC tree items are accessed via get_rss_stat_items_offset(), which skips past the cpumask and mm_cid with appropriate cacheline alignment padding. This accessor is only used during mm_struct initialization (percpu_counter_tree_init_many), not on context switch or other hot paths. Introduce the PERCPU_COUNTER_TREE_ITEMS_ALIGN() macro to handle the SMP cacheline alignment vs !SMP no-op in a single place, used by both the static init_mm flexible array initializer and the runtime offset computation. The cost is at most one cacheline (typically 64 bytes) of padding per mm_struct between the mm_cid data and the HPCC tree items. Signed-off-by: Mathieu Desnoyers --- include/linux/mm.h | 1 + include/linux/mm_types.h | 47 +++++++++++++++++------------ include/linux/percpu_counter_tree.h | 3 ++ kernel/fork.c | 4 ++- 4 files changed, 34 insertions(+), 21 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 45b49f15bb5e..b1f6703cace8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3344,6 +3344,7 @@ static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct m unsigned long ptr = (unsigned long)mm; ptr += offsetof(struct mm_struct, flexible_array); + ptr += get_rss_stat_items_offset(); return (struct percpu_counter_tree_level_item *)ptr; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 414618ffe329..4e51b666d72b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1420,11 +1420,12 @@ struct mm_struct { } __randomize_layout; /* - * The rss hierarchical counter items, mm_cpumask, and mm_cid - * masks need to be at the end of mm_struct, because they are + * The mm_cpumask, mm_cid masks, and rss hierarchical counter + * items need to be at the end of mm_struct, because they are * dynamically sized based on nr_cpu_ids. - * The content of the flexible array needs to be placed in - * decreasing alignment requirement order. + * The HPCC counter tree items are placed last and aligned + * with PERCPU_COUNTER_TREE_ITEMS_ALIGN to satisfy their + * cacheline alignment requirement. */ char flexible_array[] __mm_struct_flexible_array_aligned; }; @@ -1463,25 +1464,18 @@ static inline void __mm_flags_set_mask_bits_word(struct mm_struct *mm, MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; -#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \ -{ \ - [0 ... (PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE * NR_MM_COUNTERS) + sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0 \ -} - -static inline size_t get_rss_stat_items_size(void) -{ - return percpu_counter_tree_items_size() * NR_MM_COUNTERS; +#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \ +{ \ + [0 ... PERCPU_COUNTER_TREE_ITEMS_ALIGN( \ + sizeof(cpumask_t) + MM_CID_STATIC_SIZE) \ + + (PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE * NR_MM_COUNTERS) \ + - 1] = 0 \ } /* Future-safe accessor for struct mm_struct's cpu_vm_mask. */ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) { - unsigned long ptr = (unsigned long)mm; - - ptr += offsetof(struct mm_struct, flexible_array); - /* Skip RSS stats counters. */ - ptr += get_rss_stat_items_size(); - return (struct cpumask *)ptr; + return (struct cpumask *)&mm->flexible_array; } static inline void mm_init_cpumask(struct mm_struct *mm) @@ -1578,8 +1572,6 @@ static inline cpumask_t *mm_cpus_allowed(struct mm_struct *mm) unsigned long bitmap = (unsigned long)mm; bitmap += offsetof(struct mm_struct, flexible_array); - /* Skip RSS stats counters. */ - bitmap += get_rss_stat_items_size(); /* Skip cpu_bitmap */ bitmap += cpumask_size(); return (struct cpumask *)bitmap; @@ -1662,6 +1654,21 @@ static inline void mm_destroy_sched(struct mm_struct *mm) { } #endif /* CONFIG_SCHED_CACHE */ +static inline size_t get_rss_stat_items_size(void) +{ + return percpu_counter_tree_items_size() * NR_MM_COUNTERS; +} + +/* + * Return the offset of the RSS stat HPCC items within the mm_struct + * flexible array. The items are placed after the cpumask and mm_cid, + * aligned to the cacheline boundary required by the tree level items. + */ +static inline size_t get_rss_stat_items_offset(void) +{ + return PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size()); +} + struct mmu_gather; extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm); extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm); diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h index 828c763edd4a..3e8a820e2d1d 100644 --- a/include/linux/percpu_counter_tree.h +++ b/include/linux/percpu_counter_tree.h @@ -66,6 +66,8 @@ struct percpu_counter_tree_level_item { #define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE \ (PERCPU_COUNTER_TREE_STATIC_NR_ITEMS * sizeof(struct percpu_counter_tree_level_item)) +#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) \ + ALIGN((offset), __alignof__(struct percpu_counter_tree_level_item)) struct percpu_counter_tree { /* Fast-path fields. */ @@ -167,6 +169,7 @@ void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree * #else /* !CONFIG_SMP */ #define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE 0 +#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) (offset) struct percpu_counter_tree_level_item; diff --git a/kernel/fork.c b/kernel/fork.c index d66f5ccd5414..9ff85bd86bb9 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -3107,7 +3107,9 @@ void __init mm_cache_init(void) * dynamically sized based on the maximum CPU number this system * can have, taking hotplug into account (nr_cpu_ids). */ - mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size() + get_rss_stat_items_size(); + mm_size = sizeof(struct mm_struct) + + PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size()) + + get_rss_stat_items_size(); mm_cachep = kmem_cache_create_usercopy("mm_struct", mm_size, ARCH_MIN_MMSTRUCT_ALIGN, -- 2.43.0