From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from va-1-112.ptr.blmpb.com (va-1-112.ptr.blmpb.com [209.127.230.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B74728643A for ; Fri, 20 Mar 2026 14:33:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.112 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774017235; cv=none; b=TS4CwrJr/bNanIpbSP6zCc6KDh2s3qYcIEq03ZMubtm+Yu7e/KGJ4wE67CJZhos7ua96ibd4nyDNKJXYl0LP+Ms/9bGVCzEfqf3UifLrpFMG0kd5d3gfvcVgcU1yPTqG7kot5YsGsoDAEL+qcDcLEb1RilxYFJRlT41bGR1zsVk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774017235; c=relaxed/simple; bh=kjSSbDnl9Ve/ofZTP0YdEJe3bVHalmbBwOjVMVF4ifU=; h=Cc:Content-Type:To:References:From:Subject:Date:Message-Id: Mime-Version:In-Reply-To; b=GXvyncFuDuXYsvEuGdqr3eMmF9ghl/L19dmIWz61hrSQMdezXKn4MxonIQogMob4UN0lWQWzK8Cqi3c27ll3I5VI+pAAfBjgrg0xWeMjzXwCciRnc3fISHFHfZ2qnnZRXcWi7Fhu7mWJWvZmWp/iBpqRIMmkq644rRO05cAiUNk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LHnWDbfp; arc=none smtp.client-ip=209.127.230.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LHnWDbfp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1774017216; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=Pn9Tlsdkb3FBPgp5/+SpX9iC3zPY9ysQRHBy8hyljso=; b=LHnWDbfpimTh+aLvr7Cv7bB4Z/BA4LGewliasWq3i7AYhp5MMGjYGN5wd49hBc0nLk1Ynk EE+uKrZtVcvvDFh4g7NURG4nLmuxlfHjSxgewISDKEtFw89CahwBTDIqjz8p6HWjXHy+Et 8yiIKmD/dmKweY3QroXiH13b9RBPgrKMPZ6mvZRU61BNeE8gMvO/nWslqJrXT/HDjC88lA rNKzpTmZlahSfTreFRhKPc1EGxQxf4cH1nYqGRXC7ZMiYOCZtWjOdBeSSap+y668mCgsKT B9JazSxp1ZpOx4lu09qTBnSCYnqOmyrJzjOdixZCoGbOVTgJLh3eciDkHHxcQA== Cc: , , , , , , , , , , , Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 To: "Sebastian Andrzej Siewior" , "Nadav Amit" References: <20260318045638.1572777-1-zhouchuyi@bytedance.com> <20260318045638.1572777-11-zhouchuyi@bytedance.com> <20260318172143.ICooJ3-U@linutronix.de> X-Original-From: Chuyi Zhou From: "Chuyi Zhou" Subject: Re: [PATCH v3 10/12] x86/mm: Move flush_tlb_info back to the stack Date: Fri, 20 Mar 2026 22:33:17 +0800 X-Lms-Return-Path: Message-Id: <1a5230b2-ccd9-40c0-ab26-da7a96413c80@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 In-Reply-To: <20260318172143.ICooJ3-U@linutronix.de> User-Agent: Mozilla Thunderbird Hello, On 2026-03-19 1:21 a.m., Sebastian Andrzej Siewior wrote: > +Nadav, org post https://lore.kernel.org/all/20260318045638.1572777-11-zh= ouchuyi@bytedance.com/ >=20 > On 2026-03-18 12:56:36 [+0800], Chuyi Zhou wrote: >> Commit 3db6d5a5ecaf ("x86/mm/tlb: Remove 'struct flush_tlb_info' from th= e >> stack") converted flush_tlb_info from stack variable to per-CPU variable= . >> This brought about a performance improvement of around 3% in extreme tes= t. >> However, it also required that all flush_tlb* operations keep preemption >> disabled entirely to prevent concurrent modifications of flush_tlb_info. >> flush_tlb* needs to send IPIs to remote CPUs and synchronously wait for >> all remote CPUs to complete their local TLB flushes. The process could >> take tens of milliseconds when interrupts are disabled or with a large >> number of remote CPUs. > =E2=80=A6 > PeterZ wasn't too happy to reverse this. > The snippet below results in the following assembly: >=20 > | 0000000000001ab0 : > =E2=80=A6 > | 1ac9: 48 89 e5 mov %rsp,%rbp > | 1acc: 48 83 e4 c0 and $0xffffffffffffffc0,%rsp > | 1ad0: 48 83 ec 40 sub $0x40,%rsp >=20 > so it would align it properly which should result in the same cache-line > movement. I'm not sure about the virtual-to-physical translation of the > variables as in TLB misses since here we have a virtual mapped stack and > there we have virtual mapped per-CPU memory. >=20 > Here the below is my quick hack. Does this work, or still a now? I have > no numbers so=E2=80=A6 >=20 I applied this patch on tip sched/core: fe7171d0d5df ("sched/fair:=20 Simplify SIS_UTIL handling in select_idle_cpu()") and retest the=20 microbenchmark in [PATCH v3 10/12], pinning the tasks to CPUs using=20 cpuset, and disabling pti and mitigations. base on-stack-aligned on-stack ---- --------- ----------- avg (usec/op) 2.5278 2.5261 2.5508 stddev 0.0007 0.0027 0.0023 Based on the data above, there is no performance regression. Dose this make sense? Do you have other testing suggestions? Thanks > diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbfl= ush.h > index 5a3cdc439e38d..4a7f40c7f939a 100644 > --- a/arch/x86/include/asm/tlbflush.h > +++ b/arch/x86/include/asm/tlbflush.h > @@ -227,7 +227,7 @@ struct flush_tlb_info { > u8 stride_shift; > u8 freed_tables; > u8 trim_cpumask; > -}; > +} __aligned(SMP_CACHE_BYTES); > =20 > void flush_tlb_local(void); > void flush_tlb_one_user(unsigned long addr); > diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c > index 621e09d049cb9..99b70e94ec281 100644 > --- a/arch/x86/mm/tlb.c > +++ b/arch/x86/mm/tlb.c > @@ -1394,28 +1394,12 @@ void flush_tlb_multi(const struct cpumask *cpumas= k, > */ > unsigned long tlb_single_page_flush_ceiling __read_mostly =3D 33; > =20 > -static DEFINE_PER_CPU_SHARED_ALIGNED(struct flush_tlb_info, flush_tlb_in= fo); > - > -#ifdef CONFIG_DEBUG_VM > -static DEFINE_PER_CPU(unsigned int, flush_tlb_info_idx); > -#endif > - > -static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm, > - unsigned long start, unsigned long end, > - unsigned int stride_shift, bool freed_tables, > - u64 new_tlb_gen) > +static void get_flush_tlb_info(struct flush_tlb_info *info, > + struct mm_struct *mm, > + unsigned long start, unsigned long end, > + unsigned int stride_shift, bool freed_tables, > + u64 new_tlb_gen) > { > - struct flush_tlb_info *info =3D this_cpu_ptr(&flush_tlb_info); > - > -#ifdef CONFIG_DEBUG_VM > - /* > - * Ensure that the following code is non-reentrant and flush_tlb_info > - * is not overwritten. This means no TLB flushing is initiated by > - * interrupt handlers and machine-check exception handlers. > - */ > - BUG_ON(this_cpu_inc_return(flush_tlb_info_idx) !=3D 1); > -#endif > - > /* > * If the number of flushes is so large that a full flush > * would be faster, do a full flush. > @@ -1433,8 +1417,6 @@ static struct flush_tlb_info *get_flush_tlb_info(st= ruct mm_struct *mm, > info->new_tlb_gen =3D new_tlb_gen; > info->initiating_cpu =3D smp_processor_id(); > info->trim_cpumask =3D 0; > - > - return info; > } > =20 > static void put_flush_tlb_info(void) > @@ -1450,15 +1432,16 @@ void flush_tlb_mm_range(struct mm_struct *mm, uns= igned long start, > unsigned long end, unsigned int stride_shift, > bool freed_tables) > { > - struct flush_tlb_info *info; > + struct flush_tlb_info _info; > + struct flush_tlb_info *info =3D &_info; > int cpu =3D get_cpu(); > u64 new_tlb_gen; > =20 > /* This is also a barrier that synchronizes with switch_mm(). */ > new_tlb_gen =3D inc_mm_tlb_gen(mm); > =20 > - info =3D get_flush_tlb_info(mm, start, end, stride_shift, freed_tables, > - new_tlb_gen); > + get_flush_tlb_info(&_info, mm, start, end, stride_shift, freed_tables, > + new_tlb_gen); > =20 > /* > * flush_tlb_multi() is not optimized for the common case in which onl= y > @@ -1548,17 +1531,15 @@ static void kernel_tlb_flush_range(struct flush_t= lb_info *info) > =20 > void flush_tlb_kernel_range(unsigned long start, unsigned long end) > { > - struct flush_tlb_info *info; > + struct flush_tlb_info info; > =20 > - guard(preempt)(); > + get_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false, > + TLB_GENERATION_INVALID); > =20 > - info =3D get_flush_tlb_info(NULL, start, end, PAGE_SHIFT, false, > - TLB_GENERATION_INVALID); > - > - if (info->end =3D=3D TLB_FLUSH_ALL) > - kernel_tlb_flush_all(info); > + if (info.end =3D=3D TLB_FLUSH_ALL) > + kernel_tlb_flush_all(&info); > else > - kernel_tlb_flush_range(info); > + kernel_tlb_flush_range(&info); > =20 > put_flush_tlb_info(); > } > @@ -1728,12 +1709,11 @@ EXPORT_SYMBOL_FOR_KVM(__flush_tlb_all); > =20 > void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) > { > - struct flush_tlb_info *info; > + struct flush_tlb_info info; > =20 > int cpu =3D get_cpu(); > - > - info =3D get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false, > - TLB_GENERATION_INVALID); > + get_flush_tlb_info(&info, NULL, 0, TLB_FLUSH_ALL, 0, false, > + TLB_GENERATION_INVALID); > /* > * flush_tlb_multi() is not optimized for the common case in which onl= y > * a local TLB flush is needed. Optimize this use-case by calling > @@ -1743,11 +1723,11 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unm= ap_batch *batch) > invlpgb_flush_all_nonglobals(); > batch->unmapped_pages =3D false; > } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { > - flush_tlb_multi(&batch->cpumask, info); > + flush_tlb_multi(&batch->cpumask, &info); > } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { > lockdep_assert_irqs_enabled(); > local_irq_disable(); > - flush_tlb_func(info); > + flush_tlb_func(&info); > local_irq_enable(); > } >