From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f173.google.com (mail-dy1-f173.google.com [74.125.82.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C5E31A6800 for ; Sun, 14 Jun 2026 02:14:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781403249; cv=none; b=jekU0w6+CVhCeuJMSU8BQ7KpPxflgkFBqWiu5ZX2oXNd1B8d4lJAShFcY6nmuLbnWVx/pAomTZe2xOQpcgylc7yz6N6g8EVCDg0cm7LJo72mAU/rOmqYqj+36C3uKwW56pOfTWVNnIpcpJS2AaXEt6aIBfvdf6Ebj6jpkhCUpHw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781403249; c=relaxed/simple; bh=GNUh+O+2K/CyMckfd0dD8aLH2bCiOxDBXnwOocFwOPE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=EIiJ6swMFvOEG1YeNpD262qYiWyHJKhlgpnFqGTiq5FGiWiwS+yxGOHqKGLkuFdoBBAO5CPFfzrfXwM6dAie98v1vlMTNQ1PmUotzFn1tJOyJP3IQHJLq9EvJ5ylcgFpFGEtqSb/rwyOseDG2It8SQ+wioFaLKf/VUPkVcev1xo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=wbinvd.org; spf=pass smtp.mailfrom=wbinvd.org; dkim=pass (2048-bit key) header.d=wbinvd.org header.i=@wbinvd.org header.b=OOC+jWIw; arc=none smtp.client-ip=74.125.82.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=wbinvd.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=wbinvd.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=wbinvd.org header.i=@wbinvd.org header.b="OOC+jWIw" Received: by mail-dy1-f173.google.com with SMTP id 5a478bee46e88-30807ba471bso6396450eec.1 for ; Sat, 13 Jun 2026 19:14:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wbinvd.org; s=wbinvd; t=1781403248; x=1782008048; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=KBmjNE7LSe2dXoJyaK0CkOplqbYUAITzqX2CZLVWVzo=; b=OOC+jWIwt1DnVQUs7GTlAepCEJbSDSSW0MaKbpKq0+FjuYTkKSaMdSug4odweNpESG Z7EqRvK1mbDAsTtFt9uuZEx4anTarvUGVDMfGi+GdNkPRvMQE0Z1RHqqZtFfZV6T4sNx DhDi67Z6ffaB5aDQZpJGk6euFrlsJ9i6oN9lNIFwbrSjwIsjG/Y+EwV2VyD0E52n9Ra2 13EN+6kFPMMvbQa6uhW1XkScGHHirwM6elo/wKiSVKrNpvKT4EUBSmOSJ17+22fiZyRV fR/yVxt8p+Nx5OfWLbYQenUy+tmRiA7kCpUXj6n2OsVHnjPKKTMIrkszlbRyeLA8mp58 oXoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781403248; x=1782008048; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KBmjNE7LSe2dXoJyaK0CkOplqbYUAITzqX2CZLVWVzo=; b=mtRTXZmTPO/RidZ5STdh9kvsLYrHYnVE0MY82u+s4Girva+WXFzvEDbXSwStwKOQxP 23BqWY+oBFEkMhML6b6Yz28YdOZosm0hWnZuJurC0wyx1AAsliYKw8ifF7F7guCbTTgv aduMCK1JvHvSrAsd+n+mY6GNA+ufDAK7K3DQ57NH9e7Dii6liYbD5Wnyz8s+NSQP3NSc uNlCsZSkIrgZ6c0Dt24K7bBKj2jKC1ZhIw/efz+k9DxdXbQ6C8nUtSZ7kK+w6yT2LyWX omJK/1iLbQ8eaq1yJWgvpP75rzp9tPLTS8lf3aF2TdoNrMx78vWyWQvflLjknX3m3Kt0 C1bA== X-Forwarded-Encrypted: i=1; AFNElJ8UpEWdR7ZHHkscxV4Y92VNi9Q6oovJK65rV8/hpSPp2wxLAt+dHXSy8Cdm/Ibx/j2c9O2/sSRm98/9G+o=@vger.kernel.org X-Gm-Message-State: AOJu0Yyq2vrUUP0/Z7hG91pE1GKB8AbDWzBhoC0Y13A1vvwaIFA7xlRu GNvR50d/sYJc9vX6sooxLUt6M/nGcXGl9VLovp/W/i5yUjjyDDOf+ExT0wNZ9C4KAQM= X-Gm-Gg: Acq92OFELn++ywzlUuXxmUdbZ9SLNRsx4Hr4qTj3GanHMucljFUQ1+3yDObhkzUHoGu BCCaFh3aR+bPKk9WTOaQxd8uCODUX8gdBGGfSa2N1PZ90wiXPzZEI1JMX7O1xqYp2dsiE3Fx3lx 7Ch/n+2yF/CbL2Iy2KT9lL+MqwBNtxpWNj/cnFguaGZok1MjBHESVIBJnQrqUyPWzi2hwrallGJ xSs/O5oa4l5E49/pPop+snsPdeIl5L7QilK8v7rKMAkIBIi2FJThsMSpnaxmrz83P754QEaATP5 VcjXWcsQwSwg2mlV5PEAk5jlVP0jvcto3DrecaMr7KNH/wOZBomhmq5B15x/fk58P1mZW6nxuvO XXx0MDOJWZR2aUwdjDMFaKyqxf+kXR6twtCtzL4ILOTTblC2rHSfsOub+je0qogiXlpk78mXDsr +Py+4dAvD287ys8peKru7EwXYeOnPb8AJS9Stq X-Received: by 2002:a05:7300:7309:b0:2c0:c5e4:605f with SMTP id 5a478bee46e88-308200a1963mr5017778eec.24.1781403247617; Sat, 13 Jun 2026 19:14:07 -0700 (PDT) Received: from mozart.vkv.me ([2001:5a8:468b:d015:82c8:4bda:6d8f:26af]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3081e920243sm9568231eec.18.2026.06.13.19.14.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 13 Jun 2026 19:14:07 -0700 (PDT) Date: Sat, 13 Jun 2026 19:14:04 -0700 From: Calvin Owens To: "H. Peter Anvin" Cc: Peter Zijlstra , tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML Subject: Re: 8aeb879baf12 - significant system call latency regression, bisected Message-ID: References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Saturday 06/13 at 19:11 -0700, Calvin Owens wrote: > On Saturday 06/13 at 16:52 -0700, H. Peter Anvin wrote: > > On 2026-06-13 13:34, H. Peter Anvin wrote: > > > On 2026-06-13 01:59, Peter Zijlstra wrote: > > > > On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H. Peter Anvin" (Intel) wrote: > > > > > So I was trying to figure out a significant -- about 13% -- increase > > > > > in system call latency between v7.0 and the current master, and it > > > > > bisects down to: > > > > > > > > > > 8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build > > > > > > > > > > This is on Panther Lake (Core Ultra X7 358H) with FRED enabled. This > > > > > is a bare metal boot, no KVM. > > > > > > > > > > I'm personally extremely puzzled how this could possibly be related, > > > > > and I will be investigating the possibility that this is a false > > > > > bisect, but it is not a Heisenbug in any way; it has been extremely > > > > > reproducible, and the difference is statistically valid by close to 10 > > > > > sigma. Futhermore, the bisection at least gave the appearance of > > > > > stability. > > > > > > > > > > Given how late in the cycle this is I wanted to send an alert sooner > > > > > rather than later; I will update as I get more data. > > > > > > > > Uhm, massive WTF indeed. I don't immediately see how this could possibly > > > > affect a FRED host either, except perhaps in code layout. > > > > > > > > I don't actually have a FRED capable machine, but have you tried running > > > > one of those top-down perf things on it, to see where its hurting? > > > > > > Not yet, but I'm investigating right now (I have some family obligations this weekend, so my duty cycle is somewhat limited.) > > > > > > I reverted the patch on top of rc7, and it did, in fact, fix the regression, > > > but I'm doing a clean from-scratch rebuild of both trees to make sure > > > there isn't anything in my test setup that could introduce any kind of > > > "memory" between builds...> > > Nope, even with the clean rebuild it is 100% reproducible. It is in fact > > worse than I originally stated: the average with 7.1rc7 is 478±6 cycles > > (with the top and bottom octiles removed as outlier protection); with 7.1rc7 > > with the above patch reverted it is 397.5±0.4. - this is in fact a 20% > > increase in latency, not 13%... > > It has to be the .text layout, doesn't it? > > I notice we're splitting a cache line here now with the prefix symbol, > 7.0-rc7 has: Whoops, I meant 7.1-rc7. But seeing your other mail, sounds like this is it :) > ffffffff812175f0 <__pfx_x64_sys_call>: > ffffffff81217600 : > > If I revert 8aeb879baf12, I get: > > ffffffff812175c0 <__pfx_x64_sys_call>: > ffffffff812175d0 : > > Could that be it? > > Unfortunately I don't have any hardware new enough to poke at it myself. > > Cheers, > Calvin