From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7545938B14C for ; Sat, 13 Jun 2026 20:35:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.136 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781382922; cv=none; b=UQ2PEw+LoQPlJtaSyCnf1n/YUl5uMmvKstCLwx7LBeBem7KUy8VoIYO0OZAsPk+JaiiMkFajgA6h0uNkjOu12rUhZNDvhekIiNHK6FYNEDADVLCcRmmSq/WI0ajgHtYoe5NGKqf4rjOJj8Af7eT7R5hKIRjUyriuDjgxOHyOGlE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781382922; c=relaxed/simple; bh=WK9Uy37Yi5iPJmM4aMhMTluAWijdwrdbGAqHc8sisx8=; h=Date:From:To:CC:Subject:In-Reply-To:References:Message-ID: MIME-Version:Content-Type; b=poGSN802u6OhGyw8DUBHBvyeTfnJvS6o+5FDRJV2whiUT4SEvvRomi5RjUeGgjiJ74djIsE6/Lk8IY2PN9wljxqcAVR937JJrh3ysRGkSs5Yz2/27wO+r4C1NaqYH8Lx0039tKZafOkfhvtBigrEyV2OtG2NGbqX0BocESRzc6A= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com; spf=pass smtp.mailfrom=zytor.com; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b=IEhGpywr; arc=none smtp.client-ip=198.137.202.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zytor.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b="IEhGpywr" Received: from ehlo.thunderbird.net ([172.59.161.16]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 65DKYjL73933840 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 13 Jun 2026 13:34:46 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 65DKYjL73933840 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2026052701; t=1781382887; bh=UWGvGXgQwAnNetyPfSC+IuDNE8xWJAnFJoa0aYZCiKU=; h=Date:From:To:CC:Subject:In-Reply-To:References:From; b=IEhGpywrR8R/4OMGQ8MNKeA4umBWLXuBGMnHk6RlYdt4tr0MkjEOZVw8QXsg0LYHa ECjZSGrXuvL0x4myhalbQv+FmZ2XbOE5vLalDMleZLh7mF3cc+UqCWYaVN+NfUif05 qeOwy434G0uOPc2TI6EfuN8Z609Fj8hLC3dKfg8A1BqnvQsiAIpCocdS+qrO9+rjxX oOyWaABIzCFc8GPGnlyfY9dXEDevZMtXF45Z/wPGnuGczydzhCzFprD5LlSYrWOYoS 1H+EqlSW5t4d7suR68qefm/JhXibAZetfwTwReG9MAmEuH4fqNaq5fVeNpVryVj6BD RKuY/lEUHUl4g== Date: Sat, 13 Jun 2026 13:34:38 -0700 From: "H. Peter Anvin" To: Peter Zijlstra CC: tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML Subject: =?US-ASCII?Q?Re=3A_8aeb879baf12_-_significant_syst?= =?US-ASCII?Q?em_call_latency_regression=2C_bisected?= User-Agent: K-9 Mail for Android In-Reply-To: <20260613085919.GF42921@noisy.programming.kicks-ass.net> References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> Message-ID: <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 2026-06-13 01:59, Peter Zijlstra wrote: > On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H=2E Peter Anvin" (Intel) wro= te: >> So I was trying to figure out a significant -- about 13% -- increase >> in system call latency between v7=2E0 and the current master, and it >> bisects down to: >>=20 >> 8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build >>=20 >> This is on Panther Lake (Core Ultra X7 358H) with FRED enabled=2E This >> is a bare metal boot, no KVM=2E >>=20 >> I'm personally extremely puzzled how this could possibly be related, >> and I will be investigating the possibility that this is a false >> bisect, but it is not a Heisenbug in any way; it has been extremely >> reproducible, and the difference is statistically valid by close to 10 >> sigma=2E Futhermore, the bisection at least gave the appearance of >> stability=2E >>=20 >> Given how late in the cycle this is I wanted to send an alert sooner >> rather than later; I will update as I get more data=2E >=20 > Uhm, massive WTF indeed=2E I don't immediately see how this could possib= ly > affect a FRED host either, except perhaps in code layout=2E >=20 > I don't actually have a FRED capable machine, but have you tried running > one of those top-down perf things on it, to see where its hurting? Not yet, but I'm investigating right now (I have some family obligations t= his weekend, so my duty cycle is somewhat limited=2E) I reverted the patch on top of rc7, and it did, in fact, fix the regressio= n, but I'm doing a clean from-scratch rebuild of both trees to make sure th= ere isn't anything in my test setup that could introduce any kind of "memor= y" between builds=2E=2E=2E