From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 234C51DD0D4 for ; Sun, 14 Jun 2026 02:08:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.136 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781402940; cv=none; b=Lsgzf4Pzgr9R7TySLIxtx6ON0pGtEgmjGRIR9tYMLKG2vUDolCENahmCKMBN3mb01hykBSR0HHKcx6Wvj2myUhp8BRO96VAuGYXj79c4EuVTYDtnF+7t09AKFw8aUDTBicTf1gY39sR3+qzGFhnjKi0/DpjQqM3vWCT/NzrDZzY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781402940; c=relaxed/simple; bh=7WaGTpGuDBsF1bDN6zal6X3Rg3ohCxFHKJH7SgFlDZA=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=AjoePqstcoBPh6oZEVlvzVDC8/durlkrAsJXKOaq2C7pUQKQ04N3PDlJqeJXrniIQPWZBjWQ5oiabAJThZzScpEcvX7LdcPa36TLf7BWqZ8FKnKmi014N0mQnelhR5xuF1BqZ4RPpOnYUUgyPqxUSCQSUNU2OwNhZGXPmYQHK4I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com; spf=pass smtp.mailfrom=zytor.com; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b=qeLVDHmt; arc=none smtp.client-ip=198.137.202.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zytor.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b="qeLVDHmt" Received: from [10.124.221.60] ([192.55.54.45]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 65E1oTBl262544 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 13 Jun 2026 18:50:37 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 65E1oTBl262544 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2026052701; t=1781401838; bh=KyyjqnwFDTbq2TjgMO5GfWABwlpsYiTeh0dMTbUyPJY=; h=Date:Subject:From:To:Cc:References:In-Reply-To:From; b=qeLVDHmthI3scH6CgHzG1ZzUi1NrW/+C/95/e2UfPuRjkeX5NoFo3ifDXJ17BXAbn XB3z2SBZJcWpn7FH9RSbuStEZiqhLwyzDbBdhlZrl44OPm4q+sYSHDp5XNYtXc3slz 9CgdPM0VFx4Ndkpm9z+7EYLYEGaWm0/eyJW3VDDXc0gLgqfbao+bbZySxjmEDV9ilc ZYyrL26XrcnitvcgjH0GpXYY2yjf2iWgpUilbRHmlvVLjoA4FmfhcxoaJyEflGlwDz iZ6eZVdSHGYgVplb3q+KusOqFyxY0iBHtDvlZt8ljshTIC9wyVSNcqPatGYtETF4Wq bx99dqbNBV50w== Message-ID: Date: Sat, 13 Jun 2026 18:50:24 -0700 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: 8aeb879baf12 - significant system call latency regression, bisected From: "H. Peter Anvin" To: Peter Zijlstra Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> Content-Language: en-US, sv-SE In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2026-06-13 16:52, H. Peter Anvin wrote: > On 2026-06-13 13:34, H. Peter Anvin wrote: >> On 2026-06-13 01:59, Peter Zijlstra wrote: >>> On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H. Peter Anvin" (Intel) >>> wrote: >>>> So I was trying to figure out a significant -- about 13% -- increase >>>> in system call latency between v7.0 and the current master, and it >>>> bisects down to: >>>> >>>>     8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build >>>> >>>> This is on Panther Lake (Core Ultra X7 358H) with FRED enabled. This >>>> is a bare metal boot, no KVM. >>>> >>>> I'm personally extremely puzzled how this could possibly be related, >>>> and I will be investigating the possibility that this is a false >>>> bisect, but it is not a Heisenbug in any way; it has been extremely >>>> reproducible, and the difference is statistically valid by close to 10 >>>> sigma. Futhermore, the bisection at least gave the appearance of >>>> stability. >>>> >>>> Given how late in the cycle this is I wanted to send an alert sooner >>>> rather than later; I will update as I get more data. >>> >>> Uhm, massive WTF indeed. I don't immediately see how this could possibly >>> affect a FRED host either, except perhaps in code layout. >>> >>> I don't actually have a FRED capable machine, but have you tried running >>> one of those top-down perf things on it, to see where its hurting? >> >> Not yet, but I'm investigating right now (I have some family >> obligations this weekend, so my duty cycle is somewhat limited.) >> >> I reverted the patch on top of rc7, and it did, in fact, fix the >> regression, > > but I'm doing a clean from-scratch rebuild of both trees to make sure > > there isn't anything in my test setup that could introduce any kind of > > "memory" between builds...> > Nope, even with the clean rebuild it is 100% reproducible. It is in fact > worse than I originally stated: the average with 7.1rc7 is 478±6 cycles > (with the top and bottom octiles removed as outlier protection); with > 7.1rc7 with the above patch reverted it is 397.5±0.4. - this is in fact > a 20% increase in latency, not 13%... > OK, I have, I believe root-caused this. It is a padding issue; removing the code changes __pfx_x64_sys_call to be 32-byte aligned, with the result that x64_sys_call gets *mis*aligned. Reverting the patch but adding an alignment statement to x64_sys_call re-introduces the performance regression. I am concerned because this could mean that the __pfx stubs add substantial overhead elsewhere, unless this just happens to be a particularly sensitive case... -hpa