From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 209FD3596C for ; Tue, 7 Jan 2025 03:26:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736220369; cv=none; b=trPO0VESVoFCYnYu/aK3z87Ot8bwNHu36ffuBM19YL2GWap+6SxmPXAzjhX4OOXJaJgJ5hKe+N7DIlk2smFYg9yin5SPyCWyMfGcbt3JmVtdn6j0pJnibnQqSCjjZOS/iS+FnFsrUabyB0HPWsofnoCi4ZcEDnDgdjhvBHsMwVw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736220369; c=relaxed/simple; bh=8zPfBj62QNhZU7OTTJpBg4FM9n0kiBXK6Rgl+/aoGaA=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=HGwLbQEJm9AE0ZlIVFSRpjXPmz/zkPlbIxXAeGacL/xSw2CNV5QomPNdqc9kH0bA12V8ekFee1+fGYRWDHw1KaT8XV0vCejCnXqcP+3WjPc31yg8a16w+g9RsT/1q2FWoKFUT7Vzo5GeDDgXJJRuEpNjYeCsnm5pFeUO/YNH1Hk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=shelob.surriel.com; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shelob.surriel.com Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tV0Dr-0000000085m-2QqM; Mon, 06 Jan 2025 22:25:39 -0500 Message-ID: <95a7349f887e538b5e63f77da6b2a1d7efc9a43f.camel@surriel.com> Subject: Re: [PATCH v3 00/12] AMD broadcast TLB invalidation From: Rik van Riel To: Yosry Ahmed Cc: x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, akpm@linux-foundation.org, nadav.amit@gmail.com, zhengqi.arch@bytedance.com, linux-mm@kvack.org, Reiji Watanabe , Brendan Jackman Date: Mon, 06 Jan 2025 22:25:39 -0500 In-Reply-To: References: <20241230175550.4046587-1-riel@surriel.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.1 (3.54.1-1.fc41) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Sender: riel@surriel.com On Mon, 2025-01-06 at 14:49 -0800, Yosry Ahmed wrote: >=20 > We briefly looked at using INVLPGB/TLBSYNC as part of the ASI work to > optimize away the async freeing logic which sends TLB flush IPIs. >=20 > I have a high-level question about INVLPGB/TLBSYNC that I could not > immediately find the answer to in the AMD manual. Sorry if I missed > the answer or if I missed something obvious. >=20 > Do we know what the underlying mechanism for delivering the TLB > flushes is? If a CPU has interrupts disabled, does it still receive > the broadcast TLB flush request and handle it? I assume TLB invalidation is probably handled similarly to how cache coherency is handled between CPUs. However, it probably does not need to be quite as fast, since cache coherency traffic is probably 2-6 orders of magnitude more common than TLB invalidation traffic. >=20 > My main concern is that TLBSYNC is a single instruction that seems > like it will wait for an arbitrary amount of time, and IIUC > interrupts > (and NMIs) will not be delivered to the running CPU until after the > instruction completes execution (only at an instruction boundary). >=20 > Are there any guarantees about other CPUs handling the broadcast TLB > flush in a timely manner, or an explanation of how CPUs handle the > incoming requests in general? The performance numbers I got with the tlb_flush2_threads microbenchmark strongly suggest that INVLPGB flushes are handled by the receiving CPUs even while interrupts are disabled. CPU time spent in flush_tlb_mm_range goes down with INVLPGB, compared with IPI based TLB flushing, even when the IPIs only go to a subset of CPUs. I have no idea whether the invalidation is handled by something like microcode in the CPU, by the (more external?) logic that handles cache coherency, or something else entirely. I suspect AMD wouldn't tell us exactly ;) --=20 All Rights Reversed.