From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1310627456 for ; Sun, 14 Jun 2026 18:32:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.136 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781461949; cv=none; b=nQjiz/dSOa/KMymf6RmaDIVjutxvTjhdkUFE5ZxpETSjvXh7fd0Nsa9fOcHMhWGMAWYd6EJim4uf0yEfYsTRJU5P4VQ2akxGMppmWKRdIi3Bap1pHc1WfRmvz0CLNiaLyxH9bIGw5N3n0drE0CcCGObgldR9gAg2fRAV1nzXuQM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781461949; c=relaxed/simple; bh=975crYMmRHrOPRJE2fmRKGbi1lKHGvYjWiRAVKCN++c=; h=Date:From:To:CC:Subject:In-Reply-To:References:Message-ID: MIME-Version:Content-Type; b=Xs7ZZ3ll6k+sdoJKPZOR8kX7aklk0g8SEex3M1bL21yI9aV15RSuImPHAFScwtYnjO92yVkcLhTrQw9vZWk4bbo4glM8ANZZa+g0CD5GXxdMQZ5+WNl5r/F42j1KfJlEjYPO54O9uMZVob5qOo7KmHVIwgH7UXWaR7FC+HpwXF8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com; spf=pass smtp.mailfrom=zytor.com; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b=XCS41cv5; arc=none smtp.client-ip=198.137.202.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zytor.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b="XCS41cv5" Received: from ehlo.thunderbird.net (c-76-133-66-138.hsd1.ca.comcast.net [76.133.66.138]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 65EIW0Dj1937730 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sun, 14 Jun 2026 11:32:00 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 65EIW0Dj1937730 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2026052701; t=1781461921; bh=Q/rcE8ZvzO5A7p/rI/9LeQeV3kdPtAPNtR92wsKAlJk=; h=Date:From:To:CC:Subject:In-Reply-To:References:From; b=XCS41cv5TSxZbqeaxnVSJ41uCdC8oRzljyGeDcOYbcl8nZqBgPsGpNDwii8EjnwSX qWx3nvxnKn8OoV1LePU+Lorwqb8U7V4VuwtHqqMaWs91bk9C/yhPLlwjL18XJJF6yO ezWgYYbGYiNhmPrAaM/FHdfdxwUVa+8FAf+HIimVZ5jYS+3OFiblVqeOegkDIrYr8J y4SqIVmoGb63zm5v92i1Wy4XcH0v1RB4fMRG4kc0XoienKMv6flYjpwayAmXYuYaco 2eP0TWP10UsDlrDaRhRxyFOhtdkVmEXPLuEI/2+ueBkj1XDF/q3JYM72FPpj0zMob2 4i/w4JZaX1oHw== Date: Sun, 14 Jun 2026 11:31:54 -0700 From: "H. Peter Anvin" To: Xin Li CC: Peter Zijlstra , tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML Subject: =?US-ASCII?Q?Re=3A_8aeb879baf12_-_significant_syst?= =?US-ASCII?Q?em_call_latency_regression=2C_bisected?= User-Agent: K-9 Mail for Android In-Reply-To: <53FAD47A-FAE6-4F9B-A7A8-F8A151EBA33D@zytor.com> References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> <53FAD47A-FAE6-4F9B-A7A8-F8A151EBA33D@zytor.com> Message-ID: <15AE39F1-5D45-4CEF-A3EE-2D1F25EE2E26@zytor.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On June 14, 2026 11:08:59 AM PDT, Xin Li wrote: > >> On Jun 13, 2026, at 6:50=E2=80=AFPM, H=2E Peter Anvin = wrote: >>=20 >> On 2026-06-13 16:52, H=2E Peter Anvin wrote: >>> On 2026-06-13 13:34, H=2E Peter Anvin wrote: >>>> On 2026-06-13 01:59, Peter Zijlstra wrote: >>>>> On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H=2E Peter Anvin" (Intel)= wrote: >>>>>> So I was trying to figure out a significant -- about 13% -- increas= e >>>>>> in system call latency between v7=2E0 and the current master, and i= t >>>>>> bisects down to: >>>>>>=20 >>>>>> 8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build >>>>>>=20 >>>>>> This is on Panther Lake (Core Ultra X7 358H) with FRED enabled=2E T= his >>>>>> is a bare metal boot, no KVM=2E >>>>>>=20 >>>>>> I'm personally extremely puzzled how this could possibly be related= , >>>>>> and I will be investigating the possibility that this is a false >>>>>> bisect, but it is not a Heisenbug in any way; it has been extremely >>>>>> reproducible, and the difference is statistically valid by close to= 10 >>>>>> sigma=2E Futhermore, the bisection at least gave the appearance of >>>>>> stability=2E >>>>>>=20 >>>>>> Given how late in the cycle this is I wanted to send an alert soone= r >>>>>> rather than later; I will update as I get more data=2E >>>>>=20 >>>>> Uhm, massive WTF indeed=2E I don't immediately see how this could po= ssibly >>>>> affect a FRED host either, except perhaps in code layout=2E >>>>>=20 >>>>> I don't actually have a FRED capable machine, but have you tried run= ning >>>>> one of those top-down perf things on it, to see where its hurting? >>>>=20 >>>> Not yet, but I'm investigating right now (I have some family obligati= ons this weekend, so my duty cycle is somewhat limited=2E) >>>>=20 >>>> I reverted the patch on top of rc7, and it did, in fact, fix the regr= ession, >>> > but I'm doing a clean from-scratch rebuild of both trees to make sur= e >>> > there isn't anything in my test setup that could introduce any kind = of >>> > "memory" between builds=2E=2E=2E> >>> Nope, even with the clean rebuild it is 100% reproducible=2E It is in = fact worse than I originally stated: the average with 7=2E1rc7 is 478=C2=B1= 6 cycles (with the top and bottom octiles removed as outlier protection); w= ith 7=2E1rc7 with the above patch reverted it is 397=2E5=C2=B10=2E4=2E - th= is is in fact a 20% increase in latency, not 13%=2E=2E=2E >>=20 >> OK, I have, I believe root-caused this=2E >>=20 >> It is a padding issue; removing the code changes __pfx_x64_sys_call to = be 32-byte aligned, with the result that x64_sys_call gets *mis*aligned=2E >>=20 >> Reverting the patch but adding an alignment statement to x64_sys_call r= e-introduces the performance regression=2E > > >The problem doesn=E2=80=99t happen to IDT? > > >>=20 >> I am concerned because this could mean that the __pfx stubs add substan= tial overhead elsewhere, unless this just happens to be a particularly sens= itive case=2E=2E=2E > > >Good point, alignment check should be applied to all such entries=2E > >Thanks > Xin The problem is that if you put an alignment directive on a function, it al= igns the __pfx stub, which is exactly The Wrong Thing=E2=84=A2=2E Otherwise this would be easy to fix, permanently=2E=20 I haven't had time to test IDT yet=2E I assume it is similar=2E