From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EE761DDC35 for ; Sun, 14 Jun 2026 18:09:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.136 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781460594; cv=none; b=I9YJASeGIekTZndeRw/UORmnOwkie6ClkI52orc6yxLOtNU1dx0OsFoEpvzAH0NZfVWnUpcMaSYcF5o4msTxpQ13BLRDg7vwcNBKXDRaBWga6Pu5iAoVkcgH3MuigEP5HznKjkHq7XLiNZXxIJn+Q79jCp+ZeZ+fuaerdrHYj84= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781460594; c=relaxed/simple; bh=4SLse0cAYLdP3L7xpdWS+XLgURR1+gKVMmodE16zU98=; h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc: Message-Id:References:To; b=T3XYUdNT8IEN70fnG6KB9jwwQqoSMP0MraMeYkjN8IljQPr6S5sOeNEw+QEZD3L6oLUH8fN3wNRj2H9LIkHrWp3qP2d/1qwz1SByrsXN/eJGJhzDKK2qGhQbyUVdvW7XXsohEaOxgFNIrD/tX2DBAzxrHLk5lpv4fIY4Vm9mYIo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com; spf=pass smtp.mailfrom=zytor.com; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b=f3ymS3XT; arc=none smtp.client-ip=198.137.202.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zytor.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b="f3ymS3XT" Received: from smtpclient.apple (c-24-130-165-117.hsd1.ca.comcast.net [24.130.165.117]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 65EI9A3n1899180 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 14 Jun 2026 11:09:10 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 65EI9A3n1899180 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2026052701; t=1781460551; bh=gy9YGTi7850CzO1QK2YvwcEkuQXMpTsmFue1jk+4eys=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From; b=f3ymS3XT8TMf1GzAjHZg86jooTo/sjbM9ZqzZMZcr8bpLZJekgWO6votbu5XZHZc0 LhyQLkhHSxfCnKOlYUDtzVo2OCgfF76lNmF1KsOdTK5Dzm9jbbqKocEb5rifft+bkt 41HnDC7eqJBl617EIunUbrZfvMrm9nR42IPbDMATafEd59KNVMm5EU33D0COaa6xrk I4X6PB9xqhh+V8t+l4LVrzJeeCUMYAGgchate/G+6elTJROBNnVUJHDRz4wBjxbHjt Gvb5XEBatb3RCb76WZWIqetP1eNxJ+1DMxwJXNGOtyzjyMta6HOZ5brHLtHAjKG+rm 8D+xGZ7ymWvyw== Content-Type: text/plain; charset=utf-8 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.600.51.1.1\)) Subject: Re: 8aeb879baf12 - significant system call latency regression, bisected From: Xin Li In-Reply-To: Date: Sun, 14 Jun 2026 11:08:59 -0700 Cc: Peter Zijlstra , tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML Content-Transfer-Encoding: quoted-printable Message-Id: <53FAD47A-FAE6-4F9B-A7A8-F8A151EBA33D@zytor.com> References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> To: "H. Peter Anvin" X-Mailer: Apple Mail (2.3864.600.51.1.1) > On Jun 13, 2026, at 6:50=E2=80=AFPM, H. Peter Anvin = wrote: >=20 > On 2026-06-13 16:52, H. Peter Anvin wrote: >> On 2026-06-13 13:34, H. Peter Anvin wrote: >>> On 2026-06-13 01:59, Peter Zijlstra wrote: >>>> On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H. Peter Anvin" (Intel) = wrote: >>>>> So I was trying to figure out a significant -- about 13% -- = increase >>>>> in system call latency between v7.0 and the current master, and it >>>>> bisects down to: >>>>>=20 >>>>> 8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build >>>>>=20 >>>>> This is on Panther Lake (Core Ultra X7 358H) with FRED enabled. = This >>>>> is a bare metal boot, no KVM. >>>>>=20 >>>>> I'm personally extremely puzzled how this could possibly be = related, >>>>> and I will be investigating the possibility that this is a false >>>>> bisect, but it is not a Heisenbug in any way; it has been = extremely >>>>> reproducible, and the difference is statistically valid by close = to 10 >>>>> sigma. Futhermore, the bisection at least gave the appearance of >>>>> stability. >>>>>=20 >>>>> Given how late in the cycle this is I wanted to send an alert = sooner >>>>> rather than later; I will update as I get more data. >>>>=20 >>>> Uhm, massive WTF indeed. I don't immediately see how this could = possibly >>>> affect a FRED host either, except perhaps in code layout. >>>>=20 >>>> I don't actually have a FRED capable machine, but have you tried = running >>>> one of those top-down perf things on it, to see where its hurting? >>>=20 >>> Not yet, but I'm investigating right now (I have some family = obligations this weekend, so my duty cycle is somewhat limited.) >>>=20 >>> I reverted the patch on top of rc7, and it did, in fact, fix the = regression, >> > but I'm doing a clean from-scratch rebuild of both trees to make = sure >> > there isn't anything in my test setup that could introduce any kind = of >> > "memory" between builds...> >> Nope, even with the clean rebuild it is 100% reproducible. It is in = fact worse than I originally stated: the average with 7.1rc7 is 478=C2=B16= cycles (with the top and bottom octiles removed as outlier protection); = with 7.1rc7 with the above patch reverted it is 397.5=C2=B10.4. - this = is in fact a 20% increase in latency, not 13%... >=20 > OK, I have, I believe root-caused this. >=20 > It is a padding issue; removing the code changes __pfx_x64_sys_call to = be 32-byte aligned, with the result that x64_sys_call gets *mis*aligned. >=20 > Reverting the patch but adding an alignment statement to x64_sys_call = re-introduces the performance regression. The problem doesn=E2=80=99t happen to IDT? >=20 > I am concerned because this could mean that the __pfx stubs add = substantial overhead elsewhere, unless this just happens to be a = particularly sensitive case... Good point, alignment check should be applied to all such entries. Thanks Xin=