From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF743EAD0 for ; Mon, 30 Dec 2024 19:24:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735586673; cv=none; b=c8PKtFE6+Tqvw2GEtkfl7WiDwzJ4RFbDt6y06q53be3Oo0pcqKKEGG2yCBAlnrlIzSj4XsX/CUX2TXweZQvnPYV2NwBMAJTdet25FfaLen0h2Lz3peTXDMh7lNTJWe6cGyStaPgDcvsM7xHNI6U7k0UGfVIjrWFw6jFYVnEFqIo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735586673; c=relaxed/simple; bh=M1km7HjDeVph6cIAJtr/4N3NmMCjuJ17Pc5/q6JAsbw=; h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc: Message-Id:References:To; b=lPdd46nUZ89kzxjwQfrSbjLRWvAeVBBZCjr9Q9VLsd9xfEeev6Y9KImgOS6QvlADy0TV3QTq/NAxI6xD0JIfGoxRK4Pk3MtIg8J7uNIbxZHvM6RatJ4EIMppbBsFqFh4UaW2jP9wBA7lM7uuu5hSaeMIaAW6S6ExWK6YZ7rjIyQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LSL7T/C2; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LSL7T/C2" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-aabfb33aff8so1647336566b.0 for ; Mon, 30 Dec 2024 11:24:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735586669; x=1736191469; darn=vger.kernel.org; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=MvZhetW4yctH43dLLBYpm6faFlepdUMw/rl8YLjOU9c=; b=LSL7T/C2sgHJU12CxhcAg1kVAJ62fN+9fhWLPq2qcNHW7ETLTHdMZFUvxoCepyoTre q3Ko+kePhZRAoj8gqefGU+ZY+fIPdWjRG4QaLOkVgIqJPOt3KrVi4Q7WSjXSM376RYR1 X4hwNljJPiDev39v5hRk6KbLpWVK76uPzL8CrA1cCW4Isda98J1XEklhI4D6Zhq7eTXj Oz/As7/M3TyH60cXnx28IJfKXPouprSKgcAGuWgiyPG47L0cQX6k3bQ5gK8AsgQwlLxV 2Y87dcKBUDUqRlfia/MgyhQ91EEFxsaXm5IgvwNlpdfH4w2LBJ/CN3bP3NcGSq1NZtqr qp8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735586669; x=1736191469; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MvZhetW4yctH43dLLBYpm6faFlepdUMw/rl8YLjOU9c=; b=pi8Wom9mAzcFlOfsj+Wsj4+dBCCf4C6UiB1wfQ5yueW1mCphICBT3EVWhJAxDHZBt7 Di90NtKBE1BOPhT3MbiReJL6GGgiLB8u04DwkXLsgO9nZtnPVodwlzc0x/40GHPjjCmA wO8nlEo39lEAlz071moRAKzW+Azst+u1n4rjzQ+jCA1gtvL1OZsLzH8bySSPe6IhlDvf abGtRAk28B1CysHNrBtg+oG1YJ+OjmS8Fi4LNJPhwJWxLhUL6caMzYSeRp4lSwCV/M6Q 0Gha+njq4jFDhyq1+BnQh6a0mOqNXuel76zw37UTJvvycjSZZN7ZBGROBoB7Ys5HrHmx +VDQ== X-Forwarded-Encrypted: i=1; AJvYcCWe0IjgYr5AE9h3f3LU+bIOtuTG3CnlZMmQVDVw1HE0X/mumdAMyE7+IHYGp7snbXxkyAU4f6a+5v4k9sw=@vger.kernel.org X-Gm-Message-State: AOJu0YwBYTCmUqGmHJqhJdFU2t10NbCBsg/t8JkLc4lIeAs/hJIWoZub hYyArL9jXLZNdPeyb2mMOkT0yGQOOz8aNKaVyr5p2e0dIa8IRVya X-Gm-Gg: ASbGncsri8bwwcaxh9cYloAlHzWQ+6nZXaSpEzFcEYVSKud5IZetIXOEJGQtsSUcsfC wHqdiITiyKJQb1pm+op8dm6+9013+IK4VekWNyYYviDt6wovhNrmKlwvgNsO6Dp+2YGYFlORUpw H5E/freA1CUFulYH7nicBReSIT25kRPVDn0janlPSMvNINIJA+HU8yA/ATjjta+Pf6LSLTmi6Lz ittbjlbMDzomQbf4QiVcbMdXX7kV8hbbIAqI5+QzaMRVzwmhlF5RPlCSSTE697g2mWx3hvq/w== X-Google-Smtp-Source: AGHT+IGRWuNlvPcQdEx1bm6cGGFNa6MWn7Lw5KH6sGaNgmNSN8jx426e1ViadlzPJdWL7CIkY48bOQ== X-Received: by 2002:a17:906:dc8f:b0:aab:a1a8:92be with SMTP id a640c23a62f3a-aac080feebdmr3538958466b.5.1735586668551; Mon, 30 Dec 2024 11:24:28 -0800 (PST) Received: from smtpclient.apple ([132.69.243.32]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aac0e895194sm1495899366b.70.2024.12.30.11.24.25 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 30 Dec 2024 11:24:27 -0800 (PST) Content-Type: text/plain; charset=utf-8 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.300.87.4.3\)) Subject: Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes From: Nadav Amit In-Reply-To: <20241230175550.4046587-10-riel@surriel.com> Date: Mon, 30 Dec 2024 21:24:13 +0200 Cc: the arch/x86 maintainers , Linux Kernel Mailing List , kernel-team@meta.com, Dave Hansen , luto@kernel.org, peterz@infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Andrew Morton , zhengqi.arch@bytedance.com, "open list:MEMORY MANAGEMENT" Content-Transfer-Encoding: quoted-printable Message-Id: References: <20241230175550.4046587-1-riel@surriel.com> <20241230175550.4046587-10-riel@surriel.com> To: Rik van Riel X-Mailer: Apple Mail (2.3826.300.87.4.3) > On 30 Dec 2024, at 19:53, Rik van Riel wrote: >=20 > Use broadcast TLB invalidation, using the INVPLGB instruction, on AMD = EPYC 3 > and newer CPUs. >=20 > In order to not exhaust PCID space, and keep TLB flushes local for = single > threaded processes, we only hand out broadcast ASIDs to processes = active on > 3 or more CPUs, and gradually increase the threshold as broadcast ASID = space > is depleted. >=20 > Signed-off-by: Rik van Riel > --- >=20 [snip] > --- a/arch/x86/include/asm/mmu_context.h > +++ b/arch/x86/include/asm/mmu_context.h > @@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct = mm_struct *mm) > #define enter_lazy_tlb enter_lazy_tlb > extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct = *tsk); >=20 > +extern void destroy_context_free_broadcast_asid(struct mm_struct = *mm); > + > /* > * Init a new mm. Used on mm copies, like at fork() > * and on mm's that are brand-new, like at execve(). > @@ -161,6 +163,13 @@ static inline int init_new_context(struct = task_struct *tsk, > mm->context.execute_only_pkey =3D -1; > } > #endif > + > +#ifdef CONFIG_CPU_SUP_AMD > + INIT_LIST_HEAD(&mm->context.broadcast_asid_list); > + mm->context.broadcast_asid =3D 0; > + mm->context.asid_transition =3D false; > +#endif > + > mm_reset_untag_mask(mm); > init_new_context_ldt(mm); > return 0; > @@ -170,6 +179,9 @@ static inline int init_new_context(struct = task_struct *tsk, > static inline void destroy_context(struct mm_struct *mm) > { > destroy_context_ldt(mm); > +#ifdef CONFIG_CPU_SUP_AMD > + destroy_context_free_broadcast_asid(mm); > +#endif This ifdef=E2=80=99ry is not great. I think it=E2=80=99s better to have = entire functions in ifdef than put ifdef=E2=80=99s within the code.=20 > } >=20 > extern void switch_mm(struct mm_struct *prev, struct mm_struct *next, > diff --git a/arch/x86/include/asm/tlbflush.h = b/arch/x86/include/asm/tlbflush.h > index 20074f17fbcd..5e9956af98d1 100644 > --- a/arch/x86/include/asm/tlbflush.h > +++ b/arch/x86/include/asm/tlbflush.h > @@ -65,6 +65,23 @@ static inline void cr4_clear_bits(unsigned long = mask) > */ > #define TLB_NR_DYN_ASIDS 6 >=20 > +#ifdef CONFIG_CPU_SUP_AMD > +#define is_dyn_asid(asid) (asid) < TLB_NR_DYN_ASIDS > +#define is_broadcast_asid(asid) (asid) >=3D TLB_NR_DYN_ASIDS > +#define in_asid_transition(info) (info->mm && = info->mm->context.asid_transition) > +#define mm_broadcast_asid(mm) (mm->context.broadcast_asid) > +#else > +#define is_dyn_asid(asid) true > +#define is_broadcast_asid(asid) false > +#define in_asid_transition(info) false > +#define mm_broadcast_asid(mm) 0 I don=E2=80=99t see a reason why those should be #define instead of = inline functions. Arguably, those are better due to type-checking, etc. For instance = is_dyn_asid() is missing brackets to be safe. > + > +inline bool needs_broadcast_asid_reload(struct mm_struct *next, u16 = prev_asid) > +{ > + return false; > +} > +#endif > + > struct tlb_context { > u64 ctx_id; > u64 tlb_gen; > diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c > index 64f1679c37e1..eb83391385ce 100644 > --- a/arch/x86/mm/tlb.c > +++ b/arch/x86/mm/tlb.c > @@ -74,13 +74,15 @@ > * use different names for each of them: > * > * ASID - [0, TLB_NR_DYN_ASIDS-1] > - * the canonical identifier for an mm > + * the canonical identifier for an mm, dynamically allocated = on each CPU > + * [TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1] > + * the canonical, global identifier for an mm, identical = across all CPUs > * > - * kPCID - [1, TLB_NR_DYN_ASIDS] > + * kPCID - [1, MAX_ASID_AVAILABLE] > * the value we write into the PCID part of CR3; corresponds = to the > * ASID+1, because PCID 0 is special. > * > - * uPCID - [2048 + 1, 2048 + TLB_NR_DYN_ASIDS] > + * uPCID - [2048 + 1, 2048 + MAX_ASID_AVAILABLE] > * for KPTI each mm has two address spaces and thus needs two > * PCID values, but we can still do with a single ASID = denomination > * for each mm. Corresponds to kPCID + 2048. > @@ -225,6 +227,18 @@ static void choose_new_asid(struct mm_struct = *next, u64 next_tlb_gen, > return; > } >=20 > + /* > + * TLB consistency for this ASID is maintained with INVLPGB; > + * TLB flushes happen even while the process isn't running. > + */ > +#ifdef CONFIG_CPU_SUP_AMD I=E2=80=99m pretty sure IS_ENABLED() can be used here. > + if (static_cpu_has(X86_FEATURE_INVLPGB) && = mm_broadcast_asid(next)) { > + *new_asid =3D mm_broadcast_asid(next); Isn=E2=80=99t there a risk of a race changing broadcast_asid between the = two reads? Maybe use READ_ONCE() also since the value is modified asynchronously?=20= > + *need_flush =3D false; > + return; > + } > +#endif > + > if (this_cpu_read(cpu_tlbstate.invalidate_other)) > clear_asid_other(); >=20 > @@ -251,6 +265,245 @@ static void choose_new_asid(struct mm_struct = *next, u64 next_tlb_gen, > *need_flush =3D true; > } >=20 > +#ifdef CONFIG_CPU_SUP_AMD > +/* > + * Logic for AMD INVLPGB support. > + */ > +static DEFINE_RAW_SPINLOCK(broadcast_asid_lock); > +static u16 last_broadcast_asid =3D TLB_NR_DYN_ASIDS; > +static DECLARE_BITMAP(broadcast_asid_used, MAX_ASID_AVAILABLE) =3D { = 0 }; > +static LIST_HEAD(broadcast_asid_list); > +static int broadcast_asid_available =3D MAX_ASID_AVAILABLE - = TLB_NR_DYN_ASIDS - 1; Presumably some of these data structures are shared, and some are = accessed frequently together. Wouldn=E2=80=99t it make more sense to put them = inside a struct(s) and make it cacheline aligned? > + > +static void reset_broadcast_asid_space(void) > +{ > + mm_context_t *context; > + > + lockdep_assert_held(&broadcast_asid_lock); > + > + /* > + * Flush once when we wrap around the ASID space, so we won't = need > + * to flush every time we allocate an ASID for boradcast = flushing. > + */ > + invlpgb_flush_all_nonglobals(); > + tlbsync(); > + > + /* > + * Leave the currently used broadcast ASIDs set in the bitmap, = since > + * those cannot be reused before the next wraparound and flush.. > + */ > + bitmap_clear(broadcast_asid_used, 0, MAX_ASID_AVAILABLE); > + list_for_each_entry(context, &broadcast_asid_list, = broadcast_asid_list) > + __set_bit(context->broadcast_asid, broadcast_asid_used); > + > + last_broadcast_asid =3D TLB_NR_DYN_ASIDS; > +} > + > +static u16 get_broadcast_asid(void) > +{ > + lockdep_assert_held(&broadcast_asid_lock); > + > + do { > + u16 start =3D last_broadcast_asid; > + u16 asid =3D find_next_zero_bit(broadcast_asid_used, = MAX_ASID_AVAILABLE, start); > + > + if (asid >=3D MAX_ASID_AVAILABLE) { > + reset_broadcast_asid_space(); > + continue; > + } > + > + /* Try claiming this broadcast ASID. */ > + if (!test_and_set_bit(asid, broadcast_asid_used)) { IIUC, broadcast_asid_used is always protected with broadcast_asid_lock. So why test_and_set_bit ? > + last_broadcast_asid =3D asid; > + return asid; > + } > + } while (1); > +} > + > +/* > + * Returns true if the mm is transitioning from a CPU-local ASID to a = broadcast > + * (INVLPGB) ASID, or the other way around. > + */ > +static bool needs_broadcast_asid_reload(struct mm_struct *next, u16 = prev_asid) > +{ > + u16 broadcast_asid =3D mm_broadcast_asid(next); > + > + if (broadcast_asid && prev_asid !=3D broadcast_asid) > + return true; > + > + if (!broadcast_asid && is_broadcast_asid(prev_asid)) > + return true; > + > + return false; > +} > + > +void destroy_context_free_broadcast_asid(struct mm_struct *mm) > +{ > + if (!mm->context.broadcast_asid) mm_broadcast_asid()? > + return; > + > + guard(raw_spinlock_irqsave)(&broadcast_asid_lock); > + mm->context.broadcast_asid =3D 0; > + list_del(&mm->context.broadcast_asid_list); > + broadcast_asid_available++; > +} > + > +static bool mm_active_cpus_exceeds(struct mm_struct *mm, int = threshold) > +{ > + int count =3D 0; > + int cpu; > + > + if (cpumask_weight(mm_cpumask(mm)) <=3D threshold) > + return false; > + > + for_each_cpu(cpu, mm_cpumask(mm)) { > + /* Skip the CPUs that aren't really running this = process. */ > + if (per_cpu(cpu_tlbstate.loaded_mm, cpu) !=3D mm) > + continue; > + > + if (per_cpu(cpu_tlbstate_shared.is_lazy, cpu)) > + continue; > + > + if (++count > threshold) > + return true; > + } > + return false; > +} > + > +/* > + * Assign a broadcast ASID to the current process, protecting against > + * races between multiple threads in the process. > + */ > +static void use_broadcast_asid(struct mm_struct *mm) > +{ > + guard(raw_spinlock_irqsave)(&broadcast_asid_lock); > + > + /* This process is already using broadcast TLB invalidation. */ > + if (mm->context.broadcast_asid) > + return; > + > + mm->context.broadcast_asid =3D get_broadcast_asid(); This is read without the lock, so do you want WRITE_ONCE() here?=20 > + mm->context.asid_transition =3D true; And what about asid_transition? Presumably also need WRITE_ONCE(). But = more importantly than this theoretical compiler optimization, is there some = assumed ordering with setting broadcast_asid? > + list_add(&mm->context.broadcast_asid_list, = &broadcast_asid_list); > + broadcast_asid_available--; > +} > + > +/* > + * Figure out whether to assign a broadcast (global) ASID to a = process. > + * We vary the threshold by how empty or full broadcast ASID space = is. > + * 1/4 full: >=3D 4 active threads > + * 1/2 full: >=3D 8 active threads > + * 3/4 full: >=3D 16 active threads > + * 7/8 full: >=3D 32 active threads > + * etc > + * > + * This way we should never exhaust the broadcast ASID space, even on = very > + * large systems, and the processes with the largest number of active > + * threads should be able to use broadcast TLB invalidation. > + */ > +#define HALFFULL_THRESHOLD 8 > +static bool meets_broadcast_asid_threshold(struct mm_struct *mm) > +{ > + int avail =3D broadcast_asid_available; > + int threshold =3D HALFFULL_THRESHOLD; > + > + if (!avail) > + return false; > + > + if (avail > MAX_ASID_AVAILABLE * 3 / 4) { > + threshold =3D HALFFULL_THRESHOLD / 4; > + } else if (avail > MAX_ASID_AVAILABLE / 2) { > + threshold =3D HALFFULL_THRESHOLD / 2; > + } else if (avail < MAX_ASID_AVAILABLE / 3) { > + do { > + avail *=3D 2; > + threshold *=3D 2; > + } while ((avail + threshold) < MAX_ASID_AVAILABLE / 2); > + } > + > + return mm_active_cpus_exceeds(mm, threshold); > +} > + > +static void count_tlb_flush(struct mm_struct *mm) > +{ > + if (!static_cpu_has(X86_FEATURE_INVLPGB)) > + return; > + > + /* Check every once in a while. */ > + if ((current->pid & 0x1f) !=3D (jiffies & 0x1f)) > + return; > + > + if (meets_broadcast_asid_threshold(mm)) > + use_broadcast_asid(mm); > +} I don=E2=80=99t think count_tlb_flush() is a name that reflects what = this function does. > + > +static void finish_asid_transition(struct flush_tlb_info *info) > +{ > + struct mm_struct *mm =3D info->mm; > + int bc_asid =3D mm_broadcast_asid(mm); > + int cpu; > + > + if (!mm->context.asid_transition) is_asid_transition()? > + return; > + > + for_each_cpu(cpu, mm_cpumask(mm)) { > + if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) !=3D = mm) > + continue; > + > + /* > + * If at least one CPU is not using the broadcast ASID = yet, > + * send a TLB flush IPI. The IPI should cause stragglers > + * to transition soon. > + */ > + if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) !=3D = bc_asid) { > + flush_tlb_multi(mm_cpumask(info->mm), info); > + return; > + } > + } > + > + /* All the CPUs running this process are using the broadcast = ASID. */ > + mm->context.asid_transition =3D 0; > +} > + > +static void broadcast_tlb_flush(struct flush_tlb_info *info) > +{ > + bool pmd =3D info->stride_shift =3D=3D PMD_SHIFT; > + unsigned long maxnr =3D invlpgb_count_max; > + unsigned long asid =3D info->mm->context.broadcast_asid; > + unsigned long addr =3D info->start; > + unsigned long nr; > + > + /* Flushing multiple pages at once is not supported with 1GB = pages. */ > + if (info->stride_shift > PMD_SHIFT) > + maxnr =3D 1; > + > + if (info->end =3D=3D TLB_FLUSH_ALL) { > + invlpgb_flush_single_pcid(kern_pcid(asid)); > + /* Do any CPUs supporting INVLPGB need PTI? */ > + if (static_cpu_has(X86_FEATURE_PTI)) > + invlpgb_flush_single_pcid(user_pcid(asid)); > + } else do { I couldn=E2=80=99t find any use of =E2=80=9Celse do=E2=80=9D in the = kernel. Might it be confusing? > + /* > + * Calculate how many pages can be flushed at once; if = the > + * remainder of the range is less than one page, flush = one. > + */ > + nr =3D min(maxnr, (info->end - addr) >> = info->stride_shift); > + nr =3D max(nr, 1); > + > + invlpgb_flush_user_nr(kern_pcid(asid), addr, nr, pmd); > + /* Do any CPUs supporting INVLPGB need PTI? */ > + if (static_cpu_has(X86_FEATURE_PTI)) > + invlpgb_flush_user_nr(user_pcid(asid), addr, nr, = pmd); > + addr +=3D nr << info->stride_shift; > + } while (addr < info->end); > + > + finish_asid_transition(info); > + > + /* Wait for the INVLPGBs kicked off above to finish. */ > + tlbsync(); > +} > +#endif /* CONFIG_CPU_SUP_AMD */ > + > /* > * Given an ASID, flush the corresponding user ASID. We can delay = this > * until the next time we switch to it. > @@ -556,8 +809,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, = struct mm_struct *next, > */ > if (prev =3D=3D next) { > /* Not actually switching mm's */ > - = VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=3D > - next->context.ctx_id); > + if (is_dyn_asid(prev_asid)) > + = VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=3D > + next->context.ctx_id); Why not to add the condition into the VM_WARN_ON and avoid the nesting? >=20 > /* > * If this races with another thread that enables lam, = 'new_lam' > @@ -573,6 +827,23 @@ void switch_mm_irqs_off(struct mm_struct *unused, = struct mm_struct *next, > !cpumask_test_cpu(cpu, = mm_cpumask(next)))) > cpumask_set_cpu(cpu, mm_cpumask(next)); >=20 > + /* > + * Check if the current mm is transitioning to a new = ASID. > + */ > + if (needs_broadcast_asid_reload(next, prev_asid)) { > + next_tlb_gen =3D = atomic64_read(&next->context.tlb_gen); > + > + choose_new_asid(next, next_tlb_gen, &new_asid, = &need_flush); > + goto reload_tlb; > + } > + > + /* > + * Broadcast TLB invalidation keeps this PCID up to date > + * all the time. > + */ > + if (is_broadcast_asid(prev_asid)) > + return; > + > /* > * If the CPU is not in lazy TLB mode, we are just = switching > * from one thread in a process to another thread in the = same > @@ -626,8 +897,10 @@ void switch_mm_irqs_off(struct mm_struct *unused, = struct mm_struct *next, > barrier(); > } >=20 > +reload_tlb: > new_lam =3D mm_lam_cr3_mask(next); > if (need_flush) { > + VM_BUG_ON(is_broadcast_asid(new_asid)); > this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, = next->context.ctx_id); > this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, = next_tlb_gen); > load_new_mm_cr3(next->pgd, new_asid, new_lam, true); > @@ -746,7 +1019,7 @@ static void flush_tlb_func(void *info) > const struct flush_tlb_info *f =3D info; > struct mm_struct *loaded_mm =3D = this_cpu_read(cpu_tlbstate.loaded_mm); > u32 loaded_mm_asid =3D = this_cpu_read(cpu_tlbstate.loaded_mm_asid); > - u64 local_tlb_gen =3D = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); > + u64 local_tlb_gen; > bool local =3D smp_processor_id() =3D=3D f->initiating_cpu; > unsigned long nr_invalidate =3D 0; > u64 mm_tlb_gen; > @@ -769,6 +1042,16 @@ static void flush_tlb_func(void *info) > if (unlikely(loaded_mm =3D=3D &init_mm)) > return; >=20 > + /* Reload the ASID if transitioning into or out of a broadcast = ASID */ > + if (needs_broadcast_asid_reload(loaded_mm, loaded_mm_asid)) { > + switch_mm_irqs_off(NULL, loaded_mm, NULL); > + loaded_mm_asid =3D = this_cpu_read(cpu_tlbstate.loaded_mm_asid); > + } > + > + /* Broadcast ASIDs are always kept up to date with INVLPGB. */ > + if (is_broadcast_asid(loaded_mm_asid)) > + return; > + > = VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].ctx_id) !=3D > loaded_mm->context.ctx_id); >=20 > @@ -786,6 +1069,8 @@ static void flush_tlb_func(void *info) > return; > } >=20 > + local_tlb_gen =3D = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); > + > if (unlikely(f->new_tlb_gen !=3D TLB_GENERATION_INVALID && > f->new_tlb_gen <=3D local_tlb_gen)) { > /* > @@ -953,7 +1238,7 @@ STATIC_NOPV void native_flush_tlb_multi(const = struct cpumask *cpumask, > * up on the new contents of what used to be page tables, while > * doing a speculative memory access. > */ > - if (info->freed_tables) > + if (info->freed_tables || in_asid_transition(info)) > on_each_cpu_mask(cpumask, flush_tlb_func, (void *)info, = true); > else > on_each_cpu_cond_mask(should_flush_tlb, flush_tlb_func, > @@ -1026,14 +1311,18 @@ void flush_tlb_mm_range(struct mm_struct *mm, = unsigned long start, > bool freed_tables) > { > struct flush_tlb_info *info; > + unsigned long threshold =3D tlb_single_page_flush_ceiling; > u64 new_tlb_gen; > int cpu; >=20 > + if (static_cpu_has(X86_FEATURE_INVLPGB)) > + threshold *=3D invlpgb_count_max; I know it=E2=80=99s not really impacting performance, but it is hard for = me to see such calculations happening unnecessarily every time...