From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A393464 for ; Mon, 15 Jun 2026 00:36:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.136 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781483804; cv=none; b=Dv3jGMktIYl5P0vRytMmkioMN8mO7JjFbrrPopGcx/nBsiAwdzRgUwAcePuYhGIx09XuWnLHgEYpHK7JRKPzwWJurA71f14hmQgcKdyPQIzNl4smjPtAqa9KTuSLl3BGsB6/zA87h99Fr/4qGHpfExmTX0CZqgZ0qJuMiRk27hY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781483804; c=relaxed/simple; bh=k+rH1/d/yGNmbU2w5GBN2uzaGrthW9CvY4CzYJdRiLE=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=ljwpEqg7Xw0Z86QwV2GTcxkZIb1eLRVkcsztPiIuzUD5+M1ZUrnzA7atoWycm3ePP0cgzml0HGjxCF8zOfQMp7B4zMH+XnCgzvaDnfO7xmSKIZJX46XUxX36jK8OIxbeThHdWOBqA3RK6s1oRzfI8AHEvyApYqnuL0ykNdi7dks= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com; spf=pass smtp.mailfrom=zytor.com; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b=Se8jB9qY; arc=none smtp.client-ip=198.137.202.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zytor.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b="Se8jB9qY" Received: from [10.124.221.82] ([192.55.55.44]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 65F0KA5L2529558 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sun, 14 Jun 2026 17:20:13 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 65F0KA5L2529558 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2026052701; t=1781482815; bh=SYQxac0BIEqiDtInJqBXKwbR9D3I29LdKzQVI0UmYDQ=; h=Date:Subject:From:To:Cc:References:In-Reply-To:From; b=Se8jB9qYE7opdIHl5dzSX3QVh+HVOkQQ3xGOcp4hqj2uSYsZVBVXVjZFEnmucklKJ GMjFYhbWLsv3MLgs4Th6TTdNIbe20UbyKIfLDzUefuK1pRHgrXsrF4HkXEyWlg58tB bzDkmkvGYD0N+taoG/hyXZt4hkhmLtPpaG97iC+HJazDFBanKfkVsRxsPZPXWxGhFr 0hKx4YUGu0tG/FTNumGiTKhPVds0KPI0umgB76yzHMzqx2JPdNVDRwqN+H1VYAmmFt xJ71cJDwhvchZf0AukcSmI/iuZxxUK3EUncR/+ByOrkxXFBWOX54vY++k70Ukf+ICB EXpkvx6lQJzbQ== Message-ID: <338ead9a-91f4-4579-9954-e18911fa3f68@zytor.com> Date: Sun, 14 Jun 2026 17:19:59 -0700 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: 8aeb879baf12 - significant system call latency regression, bisected From: "H. Peter Anvin" To: Peter Zijlstra Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> Content-Language: en-US, sv-SE In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2026-06-13 18:50, H. Peter Anvin wrote: > On 2026-06-13 16:52, H. Peter Anvin wrote: >> On 2026-06-13 13:34, H. Peter Anvin wrote: >>> On 2026-06-13 01:59, Peter Zijlstra wrote: >>>> On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H. Peter Anvin" (Intel) >>>> wrote: >>>>> So I was trying to figure out a significant -- about 13% -- increase >>>>> in system call latency between v7.0 and the current master, and it >>>>> bisects down to: >>>>> >>>>>     8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build >>>>> >>>>> This is on Panther Lake (Core Ultra X7 358H) with FRED enabled. This >>>>> is a bare metal boot, no KVM. >>>>> >>>>> I'm personally extremely puzzled how this could possibly be related, >>>>> and I will be investigating the possibility that this is a false >>>>> bisect, but it is not a Heisenbug in any way; it has been extremely >>>>> reproducible, and the difference is statistically valid by close to 10 >>>>> sigma. Futhermore, the bisection at least gave the appearance of >>>>> stability. >>>>> >>>>> Given how late in the cycle this is I wanted to send an alert sooner >>>>> rather than later; I will update as I get more data. >>>> >>>> Uhm, massive WTF indeed. I don't immediately see how this could >>>> possibly >>>> affect a FRED host either, except perhaps in code layout. >>>> >>>> I don't actually have a FRED capable machine, but have you tried >>>> running >>>> one of those top-down perf things on it, to see where its hurting? >>> >>> Not yet, but I'm investigating right now (I have some family >>> obligations this weekend, so my duty cycle is somewhat limited.) >>> >>> I reverted the patch on top of rc7, and it did, in fact, fix the >>> regression, >>  > but I'm doing a clean from-scratch rebuild of both trees to make sure >>  > there isn't anything in my test setup that could introduce any kind of >>  > "memory" between builds...> >> Nope, even with the clean rebuild it is 100% reproducible. It is in >> fact worse than I originally stated: the average with 7.1rc7 is 478±6 >> cycles (with the top and bottom octiles removed as outlier >> protection); with 7.1rc7 with the above patch reverted it is >> 397.5±0.4. - this is in fact a 20% increase in latency, not 13%... >> > > OK, I have, I believe root-caused this. > > It is a padding issue; removing the code changes __pfx_x64_sys_call to > be 32-byte aligned, with the result that x64_sys_call gets *mis*aligned. > > Reverting the patch but adding an alignment statement to x64_sys_call > re-introduces the performance regression. > > I am concerned because this could mean that the __pfx stubs add > substantial overhead elsewhere, unless this just happens to be a > particularly sensitive case... > OK, so v7.1 was released with this sizable performance regression. That begs the question how to deal with it. One option that might be reasonable for -stable is to simply add back 16 bytes of NOPs into the assembly file. However, that is obviously not a long term fix. Any thoughts? -hpa