From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 822E01EE02F for ; Tue, 21 Jan 2025 10:45:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737456329; cv=none; b=HmqCLpz6aov5cbvN/gVh17g7jsTChhD5wajhNryCARNqTVcjpfsqYt/ZvinxgzndQMestrC7KQiF1Ds3mjIsvMNl+Yl7X1nFNufwXVjKrLtlQZTcXooLPgPP3wjK5yEIE71Z+QBHwPka2q2fLcgakE8toN06V3J1Rco8ynWvETs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737456329; c=relaxed/simple; bh=odrTs+Y5d56oPAj1zsyu/uuyWbIUNObGGxUweX8/yPI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=iylhLysCrzIMwVIrqavrAxJ9fuxYB09KmVfN3Jb4rIyt3apA96LhwDdgkTSNuE9vklnDCZkyUC1n7UysHmpxNXePNUv19dDRC42qcKcaHphpK+IMRKQNpnOs0BxAA76Szw6ldT0fCBnglHXJDamRz3Th78/OTX1CXON6rNPDt1s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Ix8NA+lk; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Ix8NA+lk" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=r5vpCvDvnd04BK2QN6O+XOM0rF9uo+eThCy305D/h4g=; b=Ix8NA+lk0iihGFXlWiDcV9WNaQ Ze2y2xSgj+/qDyuVOMPc4LAx/y/ZVvtNkvWuteFysuY8tYiVHiZmf9n8fsEteh+9m1bO41uZ8V+LL zCSMAtDNyfVdlHEYPvtzAhuabwuVxWju4Vdx0sjDuBjR1GaTMPj9axVsavEqTvFtal9ryrKB3+e5Y dg7euTE2OiFtElwlVO9gJT9QKGHsBiFrTn/+DG01ccaPez9+IvXq2rgETmqYh1LeeMmMaV5q/gPYq peni83KOJD6Du2SVyWXKdpEWsNHN7z0GdkGwsR5Kj2Lk7kaVsU9TPRtvJ8JfejBQVVb+j4LRICL3t sN3s5VMA==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1taBkv-00000007g1E-344r; Tue, 21 Jan 2025 10:45:13 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 4BBD7300619; Tue, 21 Jan 2025 11:45:13 +0100 (CET) Date: Tue, 21 Jan 2025 11:45:13 +0100 From: Peter Zijlstra To: Andrew Cooper Cc: Michael Kelley , "riel@surriel.com" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" , "bp@alien8.de" , "dave.hansen@linux.intel.com" , "zhengqi.arch@bytedance.com" , "nadav.amit@gmail.com" , "thomas.lendacky@amd.com" , "kernel-team@meta.com" , "linux-mm@kvack.org" , "akpm@linux-foundation.org" , "jannh@google.com" Subject: Re: [PATCH v5 00/12] AMD broadcast TLB invalidation Message-ID: <20250121104513.GB7145@noisy.programming.kicks-ass.net> References: <20250116023127.1531583-1-riel@surriel.com> <20250116223748.GA33629@noisy.programming.kicks-ass.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri, Jan 17, 2025 at 12:00:33AM +0000, Andrew Cooper wrote: > On 16/01/2025 10:37 pm, Peter Zijlstra wrote: > > On Thu, Jan 16, 2025 at 06:14:00PM +0000, Michael Kelley wrote: > >> So CoCo > >> VMs may still use the paravirtualization that makes hypercalls to do > >> TLB flushes. It's future work to *always* use INVLPGB (if available) > >> in a CoCo VM. > > That would place a limit on the number of CPUs, to be no larger than the > > number of available ASIDs. > > Can you please be specific between PCID (the x86 architectural thing > commonly called ASID) or ASID (the thing named by the AMD architecture). > > INVLPGB instruction under virt can use PCIDs to its hearts content, but > ASIDs are rewritten behind the scenes because VM does not usually know > the ASID the VMM assigned to it. Sorry, I had to re-read the documents again to find out what you were talking about since I habitually skip all virt bits when reading. Urgh @ AMD for adding ASID, although I think I understand what they're doing. Anyway, PCID, the thing commonly called ASID by pretty much all other architectures using broadcast TLBI. Since ASIDs be global and independent execution of CPUs means you then need at least one ASID per CPU (when they're all running a different process), you must not have more CPUs than available ASIDs, or you're into trouble. Since PCID space is 12 bits, we cannot have more than 4k CPUs (minus a few for that other PCID crap we do), or half that when PTI. And while that sounds like a lot -- its getting easier to hit every passing year, we're having how many CPUs on a single Epyc now? 192*2 or somesuch nonsense IIRC. So if someone wants to mandate Broadcast TLBI, someone needs to figure out what to do when we hit this limit.