From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 451BC14EC7E for ; Mon, 20 Jan 2025 20:05:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737403502; cv=none; b=usFt5OAdAxncv72Q6IJVDalMMJH4adPr+Dp8m35Z1qR5hrpKCgmcnka1uHq9gr5Iw9g8IB/+N++wZpGQ0Hpj6K8b+KH/udEhbE78GywzkJU2ARdKRiX7nWZpmrzIEEso1mMaGOGwb2Pi2WdSpM1rRTbcMO9qvXdo41oYaHRxcRg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737403502; c=relaxed/simple; bh=rv/jLcaYpvtcaasRUY+8jOyyZjuMCKLqP29zCWZvJy0=; h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc: Message-Id:References:To; b=D61EoiXY8G0ry+MQb4UoCpARXglbBgbX4kTwtg36D5i39vOB8vISZXhnaVPSEtp/0JLhguTNHEd19jagmAhN1Pfc+rvqn1Kf4BPeBxQB6wFjRKR9A2k/yllBxSrBKeFU62VC5Apkea/RxBK/ZpdcQrKMmzdDdgRu3cMOEv54Sqo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kvggiNG2; arc=none smtp.client-ip=209.85.208.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kvggiNG2" Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-5d647d5df90so8134404a12.2 for ; Mon, 20 Jan 2025 12:05:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737403499; x=1738008299; darn=vger.kernel.org; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/3zA6/rhJygiXhlmQObZDKbmJPf5h9J80NCk24OJ8UQ=; b=kvggiNG2OSoDLsQ9NtYUTh0mMWri+mzJum08Jf5bBqxIJM01iXc0wHCzG7cI4lSNT6 1tcrTmCabGMQqsRLLUio+2cSTVNRFaFnPpKv5E2zvTH2DDjFm/VRcaKCEfQoAZVQyLpV u5pvbmvF2WxHVDMufdfYrwGroIGULhkPkI3pLWvhkFRexQgz6QCk2wOcTG3TjIOeaI/o o1jocVGf3CQc2byh0cPiBfaEShsXHY2+jcIzsyJCKvhGtkcKdopoPeEwaFV3r66fwrPI zAoEMYGlfNFIaLiXX2qWN/RvTpkaXwRPyeiTvmNIIxMa4a036xWT8HbbeCTEZYvrub+h XL6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737403499; x=1738008299; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/3zA6/rhJygiXhlmQObZDKbmJPf5h9J80NCk24OJ8UQ=; b=oX1OtsKvIvDMraSAF/9Q6fxegP5/5/ApyPMELriC2VgLm3CCDpCFSAxWMx30PFTaWf NAjq9cqCTvhXhYf0q3/CB01Yfng0w79x9MyamwfSu4DrIOGpUz82CN4vIKpsujDkX7It M/1rO6w2M1EqatOsql9/xkaEo5zNsAPhBUQdxEFJvSrNeXl+OznVjSnbsunxJKb9j8aG 3oNX8yAvuGTOMuIRrRYXQCHwACpSWUPiuTxWKIHaWSDVurzzPZa6TQzBma3SIkGjupYb fOJVS51Frl63um7DOYh+n2eyYzDTp6or7dXx4hBSTL7b+stg6Xml/ufH3aBrNZjAK51R gy3A== X-Forwarded-Encrypted: i=1; AJvYcCUpviLSYuWdSb8c1bnDgZjKKpYEImHj5NnkzXa7lWvPsN5Cb+n9mMV2j0xCScy+c/9vjg8NCvJK6LzN3pQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxHR4yEUKeZmAFEdG7aVFxJWzCXDy6jNPQfpLvLn9mWyxlVZuih O9Z3Un3zqsiCiX9deTvQW919gwnyoBbixThvjnik7IRizsjHV/Xk X-Gm-Gg: ASbGncsriTqyXFdNG+Bq+HNfoGx6NTaDvW3EWnHIRfxB/wBcZhv7UdPjN+QG5sgnKk9 Fl8cLgqsq3kr12lsLzJHZ6O6iP3Ohq0OGZqv3Vgv/mXxg0pzsOOQ4jL4MobSJ8RtzzWHsouCWs3 G642UYKZI8Q8ZhnDBnXB2He3Po0FMK2Or2FzCr9J4o27VAGFqK4gZp7P2t+jhAkhubbE1i9X9TS cAWzom7s9iHCT7Jgk/T/u40tei6sd3V7n/XNNFl5rwGry+iLesZ6VSiaurYZwJfhHcmjOvUfJ83 0vQ1ooNEtWAdJvhAM/P5 X-Google-Smtp-Source: AGHT+IFBsaEesnJ759o+iAnExQGWf582lTLCGHFFJ6J8OJa0wx62JV85W/0tWwiEUMGiJrjCFrQlbA== X-Received: by 2002:a17:907:7fa8:b0:aaf:123a:e4f0 with SMTP id a640c23a62f3a-ab38b0b6886mr1505818266b.6.1737403498258; Mon, 20 Jan 2025 12:04:58 -0800 (PST) Received: from smtpclient.apple ([132.68.46.60]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ab384ce11eesm660135766b.38.2025.01.20.12.04.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Jan 2025 12:04:57 -0800 (PST) Content-Type: text/plain; charset=utf-8 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.300.87.4.3\)) Subject: Re: [PATCH v6 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes From: Nadav Amit In-Reply-To: Date: Mon, 20 Jan 2025 22:04:45 +0200 Cc: the arch/x86 maintainers , Linux Kernel Mailing List , Borislav Petkov , peterz@infradead.org, Dave Hansen , zhengqi.arch@bytedance.com, thomas.lendacky@amd.com, kernel-team@meta.com, "open list:MEMORY MANAGEMENT" , Andrew Morton , jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com Content-Transfer-Encoding: quoted-printable Message-Id: References: <20250120024104.1924753-1-riel@surriel.com> <20250120024104.1924753-10-riel@surriel.com> <84ba1c3e-d975-458f-89f5-a6f5d04a3d22@gmail.com> To: Rik van Riel X-Mailer: Apple Mail (2.3826.300.87.4.3) > On 20 Jan 2025, at 18:09, Rik van Riel wrote: >=20 > On Mon, 2025-01-20 at 16:02 +0200, Nadav Amit wrote: >>=20 >>=20 >> On 20/01/2025 4:40, Rik van Riel wrote: >>>=20 >>> +static inline void broadcast_tlb_flush(struct flush_tlb_info >>> *info) >>> +{ >>> + VM_WARN_ON_ONCE(1); >>=20 >> Not sure why not the use VM_WARN_ONCE() instead with some more=20 >> informative message (anyhow, a string is allocated for it). >>=20 > VM_WARN_ON_ONCE only has a condition, not a message. Right, my bad. >=20 >>> + /* Slower check to make sure. */ >>> + for_each_cpu(cpu, mm_cpumask(mm)) { >>> + /* Skip the CPUs that aren't really running this >>> process. */ >>> + if (per_cpu(cpu_tlbstate.loaded_mm, cpu) !=3D mm) >>> + continue; >>=20 >> Then perhaps at least add a comment next to loaded_mm, that it's not=20= >> private per-se, but rarely accessed by other cores? >>=20 > I don't see any comment in struct tlb_state that > suggests it was ever private to begin with. >=20 > Which comment are you referring to that should > be edited? You can see there is a tlb_state_shared, so one assumes tlb_state is private... (at least that was my intention separating them). >=20 >>>=20 >>> + WRITE_ONCE(mm->context.asid_transition, true); >>> + WRITE_ONCE(mm->context.global_asid, get_global_asid()); >>=20 >> I know it is likely correct in practice (due to TSO memory model), >> but=20 >> it is not clear, at least for me, how those write order affects the >> rest=20 >> of the code. I managed to figure out how it relates to the reads in=20= >> flush_tlb_mm_range() and native_flush_tlb_multi(), but I wouldn't say >> it=20 >> is trivial and doesn't worth a comment (or smp_wmb/smp_rmb). >>=20 >=20 > What kind of wording should we add here to make it > easier to understand? >=20 > "The TLB invalidation code reads these variables in > the opposite order in which they are written" ? Usually in such cases, you make a reference to wherever there are = readers that rely on the ordering. This is how documenting smp_wmb()/smp_rmb() ordering is usually done. >=20 >=20 >>>=20 >>> + /* >>> + * If at least one CPU is not using the global ASID yet, >>> + * send a TLB flush IPI. The IPI should cause stragglers >>> + * to transition soon. >>> + * >>> + * This can race with the CPU switching to another task; >>> + * that results in a (harmless) extra IPI. >>> + */ >>> + if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm_asid, cpu)) = !=3D bc_asid) { >>> + flush_tlb_multi(mm_cpumask(info->mm), info); >>> + return; >>> + } >>=20 >> I am trying to figure out why we return here. The transition might >> not=20 >> be over? Why is it "soon"? Wouldn't flush_tlb_func() reload it=20 >> unconditionally? >=20 > The transition _should_ be over, but what if another > CPU got an NMI while in the middle of switch_mm_irqs_off, > and set its own bit in the mm_cpumask after we send this > IPI? >=20 > On the other hand, if it sets its mm_cpumask bit after > this point, it will also load the mm->context.global_asid > after this point, and should definitely get the new ASID. >=20 > I think we are probably fine to set asid_transition to > false here, but I've had to tweak this code so much over > the past months that I don't feel super confident any more :) I fully relate, but I am not sure it is that great. The problem is that nobody would have the guts to change that code later... >>=20 >> And I guess this is now a bug, if info->stride_shift > PMD_SHIFT... >>=20 > We set maxnr to 1 for larger stride shifts at the top of the function: >=20 You=E2=80=99re right, all safe. >=20 >>> + goto reload_tlb; >>=20 >> Not a fan of the goto's when they are not really needed, and I don't=20= >> think it is really needed here. Especially that the name of the tag=20= >> "reload_tlb" does not really convey that the page-tables are reloaded >> at=20 >> that point. >=20 > In this particular case, the CPU continues running with > the same page tables, but with a different PCID. I understand it is =E2=80=9Creload_tlb=E2=80=9D from your point of view, = or from the point of view of the code that does the =E2=80=9Cgoto=E2=80=9D, but if I = showed you the code that follows the =E2=80=9Creload_tlb=E2=80=9D, I=E2=80=99m not = sure you=E2=80=99d know it is so. [ snip, taking your valid points ] >>=20 >>> + loaded_mm_asid =3D >>> this_cpu_read(cpu_tlbstate.loaded_mm_asid); >>> + } >>> + >>> + /* Broadcast ASIDs are always kept up to date with >>> INVLPGB. */ >>> + if (is_global_asid(loaded_mm_asid)) >>> + return; >>=20 >> The comment does not clarify to me, and I don't manage to clearly=20 >> explain to myself, why it is guaranteed that all the IPI TLB flushes, >> which were potentially issued before the transition, are not needed. >>=20 > IPI TLB flushes that were issued before the transition went > to the CPUs when they were using dynamic ASIDs (numbers 1-5). >=20 > Reloading the TLB with a different PCID, even pointed at the > same page tables, means that the TLB should load the > translations fresh from the page tables, and not re-use any > that it had previously loaded under a different PCID. >=20 What about this scenario for instance? CPU0 CPU1 CPU2 ---- ---- ---- (1) use_global_asid(mm): =20 mm->context.asid_trans =3D T; mm->context.global_asid =3D G; (2) switch_mm(..., next=3Dmm): *Observes global_asid =3D G =3D> loads CR3 with PCID=3DG =3D> fills TLB under G. TLB caches PTE[G, V] =3D P (for some reason) (3) flush_tlb_mm_range(mm): *Sees global_asid =3D=3D = 0 (stale/old value) =3D> flush_tlb_multi() =3D> IPI flush for dyn. (4) IPI arrives on CPU1: flush_tlb_func(...):=20 is_global_asid(G)? yes, skip invalidate; broadcast flush assumed to cover it. (5) IPI completes on CPU2: Dyn. ASIDs are flushed,=20= but CPU1=E2=80=99s = global ASID was never invalidated! (6) CPU1 uses stale TLB entries under ASID G. TLB continues to use PTE[G, V] =3D P, as it was not invalidated.