From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F5B026D4C3 for ; Sat, 13 Jun 2026 23:53:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.136 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781394826; cv=none; b=pPh0VCYfNmz2Vg5QlN5DdKtosw6NNjTNcWEMtQiaW6m3HtHsptC3xCjMoEpURHuNcurHOdSgUsHB9wN+0BvT9K37uvQOak5EZxm0DxUJs3YCUNBVhK/m6l8iL6vhcqn4EjRKR/+1YYg9rwWfys5FpdTVDZXCc6YIY6gAxVQaXX4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781394826; c=relaxed/simple; bh=75ohB5bEfygMXIzTklNYNs5vxa96o01pElDmdf6Pw2I=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=k2bFkAaHhe7hejY90sWxeOGPpyLl8hJCGQDKrTx/AiT75k5Iqw5KylYAmEX3USJMCeu4HhfrlnrfgjKwIuzeWZ29iOC6ipTeAG6k7cei0GhpXEtOgdc4skkLQqHIxe0o9mmpcjYUT4p0ts49Z2o3wVuITR+CKEbtfrf9HDK227o= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com; spf=pass smtp.mailfrom=zytor.com; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b=CV9hhhHE; arc=none smtp.client-ip=198.137.202.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zytor.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b="CV9hhhHE" Received: from [10.124.221.55] ([192.55.55.44]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 65DNqCQj068054 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 13 Jun 2026 16:53:10 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 65DNqCQj068054 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2026052701; t=1781394791; bh=Wbl9a4wbS18rQr+Zgp4FDRGnwugqMOrbJXz33mdfXIE=; h=Date:Subject:From:To:Cc:References:In-Reply-To:From; b=CV9hhhHEw7S/ouVsoqnoeQCr5zm82Ke+q1zY2XqdVMwLWZslyCOyoS0CjSdUt8A1f 4WE5dykXj/tEotZ3z+ZP2ZBvq7PCHdvgTv0df1JScgdDsVOdta1UdO1monUPND+sBe oAhf7HnH1ByhQqNweyYKfLjVSpEOKJDBMEcSLecLPXf/NNeOw1Qt8ezZMLmwls0aHu 1C9ej1nw0DETI2r2G3y6nrN1eWWbOO3kJ3gNyOP2QRJ+H8J8KH0z1JyaJPhw0BBMQn YUSs8O14PoZkm76AiLV0p07EQiKgLtMPXeORj1wExIXC9OxOgBH9ZltstKYFi411c3 w78ycXCRAQCnA== Message-ID: Date: Sat, 13 Jun 2026 16:52:06 -0700 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: 8aeb879baf12 - significant system call latency regression, bisected From: "H. Peter Anvin" To: Peter Zijlstra Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> Content-Language: en-US, sv-SE In-Reply-To: <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2026-06-13 13:34, H. Peter Anvin wrote: > On 2026-06-13 01:59, Peter Zijlstra wrote: >> On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H. Peter Anvin" (Intel) wrote: >>> So I was trying to figure out a significant -- about 13% -- increase >>> in system call latency between v7.0 and the current master, and it >>> bisects down to: >>> >>> 8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build >>> >>> This is on Panther Lake (Core Ultra X7 358H) with FRED enabled. This >>> is a bare metal boot, no KVM. >>> >>> I'm personally extremely puzzled how this could possibly be related, >>> and I will be investigating the possibility that this is a false >>> bisect, but it is not a Heisenbug in any way; it has been extremely >>> reproducible, and the difference is statistically valid by close to 10 >>> sigma. Futhermore, the bisection at least gave the appearance of >>> stability. >>> >>> Given how late in the cycle this is I wanted to send an alert sooner >>> rather than later; I will update as I get more data. >> >> Uhm, massive WTF indeed. I don't immediately see how this could possibly >> affect a FRED host either, except perhaps in code layout. >> >> I don't actually have a FRED capable machine, but have you tried running >> one of those top-down perf things on it, to see where its hurting? > > Not yet, but I'm investigating right now (I have some family obligations this weekend, so my duty cycle is somewhat limited.) > > I reverted the patch on top of rc7, and it did, in fact, fix the regression, > but I'm doing a clean from-scratch rebuild of both trees to make sure > there isn't anything in my test setup that could introduce any kind of > "memory" between builds...> Nope, even with the clean rebuild it is 100% reproducible. It is in fact worse than I originally stated: the average with 7.1rc7 is 478±6 cycles (with the top and bottom octiles removed as outlier protection); with 7.1rc7 with the above patch reverted it is 397.5±0.4. - this is in fact a 20% increase in latency, not 13%... -hpa