From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F055D3CC33F for ; Tue, 16 Jun 2026 17:45:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.136 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781631916; cv=none; b=GF1A8BNMjciuzoptsV0KpY1jwojyEVUHqaHntW+jZc+2U3wsoelzujRUXTyyWRn2J4UBoVxXAlCU5JQN8dAlmO3g1fAEWci6s0Vq3kXXv41wj5kDa8Ffc89q4qkZMjMRHHI6nYaJSzv7+gtWSvxUMmRaVQe75fva4mn9DXBXR2k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781631916; c=relaxed/simple; bh=zso4MBkY+qGdxM3OxEN1eHJN+XSwwYHd3NIms+KufU8=; h=Date:From:To:CC:Subject:In-Reply-To:References:Message-ID: MIME-Version:Content-Type; b=eNP7HQNXigOIKN9h/bGAAtyUcrtKJrZV1zPF9khXyGV163fPAfgDpYOXrOA4Odnxb6E5DansTPnj8+FUDEuxjxYaV0L+XwS1tL7RcAi7v6qqd9JK70jwLmsLiT3Wf80P3dqSYF2hoj0/syBMyOo5zlzBMRvjgWd1v/gxo3qtjhk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com; spf=pass smtp.mailfrom=zytor.com; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b=qtW94faZ; arc=none smtp.client-ip=198.137.202.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zytor.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zytor.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=zytor.com header.i=@zytor.com header.b="qtW94faZ" Received: from ehlo.thunderbird.net (c-76-133-66-138.hsd1.ca.comcast.net [76.133.66.138]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 65GHiQDL2878896 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Tue, 16 Jun 2026 10:44:26 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 65GHiQDL2878896 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2026052701; t=1781631867; bh=lSI0WwlkmCDl8x+A14tz478Fa5oA0p3/GTWEn0wZfz8=; h=Date:From:To:CC:Subject:In-Reply-To:References:From; b=qtW94faZ4hVdj1tZkH4cwDsBlds+o7R1QEarHLkaaTwajt5Qb4+iidLagtzoT6a6J +iGKI+LuV8Pvjk6dQndj3JWRul6p43kt1b3qhoYwgYlklJxjrooKD8FyX6f3blyFxO +T9aiic9xCow2pmrbfdtTb1rsP62eVCQozh+D+9ENAVfySqrTS/pVschfGosh/CFW7 jOvyykilD2fwtUQbFK6/3/9KSSyh0gPM0n4k+NYNh7N5jKKAZ81ez1lpS5/YdtYdyG xmMeRUYbLKtfjpCUlWbGO6i8UmoEawb4gG+nd+FaAY/16CpEIJfQSLLSBBoGFk3U1E Q5SSDOCR0cztg== Date: Tue, 16 Jun 2026 10:44:19 -0700 From: "H. Peter Anvin" To: Ingo Molnar , Linus Torvalds CC: Peter Zijlstra , tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , x86-ML , LKML Subject: =?US-ASCII?Q?Re=3A_8aeb879baf12_-_significant_syst?= =?US-ASCII?Q?em_call_latency_regression=2C_bisected?= User-Agent: K-9 Mail for Android In-Reply-To: References: <20260613085919.GF42921@noisy.programming.kicks-ass.net> <203E61B7-290F-4F87-860F-B352D0072703@zytor.com> <20260616082814.GQ48970@noisy.programming.kicks-ass.net> Message-ID: <9D5BDF61-3C14-4F21-9D8C-4AB7F28B7EEC@zytor.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On June 16, 2026 2:51:12 AM PDT, Ingo Molnar wrote: > >* Linus Torvalds wrote: > >> On Tue, 16 Jun 2026 at 13:58, Peter Zijlstra w= rote: >> > >> > So ISTR the Intel I-fetch window was 16 bytes, so the above things wo= uld >> > make sense=2E However, Gemini, or whatever AI sits in google search, = is >> > trying to tell me Intel moved to 32 byte I-fetch with Alderlake=2E >>=20 >> Even with 16-byte fetch, the cacheline size is 64 bytes, so it hurts >> to not be 64-byte aligned - simply because you may need to fetch more >> cachelines (assuming fairly linear code)=2E >>=20 >> And afaik, some of the newer ones aren't 32-byte wide, but can do 48 >> bytes as three 16-byte fetches=2E >>=20 >> But I don't know if they can do the old "split line access" that older >> cores could do, where a Pentium would do two 8-byte accesses at the >> same time, and they didn't have to be in the same cache line=2E >>=20 >> So 64-byte alignment would always be the best option if you only look >> at a *particular* piece of code=2E >>=20 >> But it obviously is very wasteful and hurts when there is code around >> it that could be loaded into the cache at the same time=2E >>=20 >> So almost certainly not a good idea in general=2E >>=20 >> But 64-byte alignment is probably what things like interrupt and >> system call entrypoints should use, because those things would make >> sense to look at as isolated things, not part of a bigger load"=2E And >> they are quite likely to start from a fairly cold-cache situation=2E >>=20 >> So *not* some general compiler option in a config file, but maybe a >> special "entry point alignment" macro? > >Yeah, agreed on that approach - but before/while we fix it, >I'm also still somewhat baffled by the numbers hpa reported: > >>>> Nope, even with the clean rebuild it is 100% reproducible=2E It is in= fact >>>> worse than I originally stated: the average with 7=2E1rc7 is 478=C2= =B16 cycles >>>> (with the top and bottom octiles removed as outlier protection); with= 7=2E1rc7 >>>> with the above patch reverted it is 397=2E5=C2=B10=2E4=2E - this is i= n fact a 20% >>>> increase in latency, not 13%=2E=2E=2E > >Now that we know that this regression is caused by entry function >alignment changes, do we know *why* it causes a 80 cycles >shift in system call entry performance? > >What does the benchmark measure, cache-cold or cache-hot >execution? > >1) Cache-cold performance: > >If it is cold-cache performance, does the misaligned case fetch >one more cold cacheline? > >>From which cache does it miss? Fetching from the 2-4MB Panther Lake >L2 shouldn't be 80 cycles, it should be ~17 cycles=2E > >If it's fetching from the 18MB L3 (which I'd say is the norm for >most workloads), then the L3->L1I latency is around ~55 cycles on >Panther Lake, with everything included=2E > >It cannot really be DRAM latency, ie=2E true cache-cold latency, >as that would be much more severe, in the 400 cycles range even >with premium DRAM modules - and more like 500 cycles with >mainstream DRAM modules and layouts=2E (Unless we are *lucky* with >alignment and sizing and the alignment regression doesn't trigger >full DRAM latency=2E) The on-die DRAM MSC cache's latency should >be around 300 cycles - that too is too high=2E > >2) Cache-hot performance: > >While cache-hot performance is less relevance for system calls >(which tend to be cache-cold in practice), if the benchmark >measures cache-hot performance, why is there a 80 cycles shift >from just a single misaligned symbol? > >Ie=2E the specific and rather stable figure of 80 cycles overhead >does not seem to match any of the Panther Lake latencies that >ought to be relevant to this regression, if we use the simplest >mental model of what's going on when alignment changes=2E > >So it is either some other uarch pathology, triggered by bad >alignment, or something doesn't add up in my mental model >of the root cause of this problem=2E :-) > >Side notes: > > - The 6 cycles noise in the 478=C2=B16 cycles measurement > does suggest that we might have missed out to a > deeper cache hierarchy level, versus the rather > stable 397=2E5=C2=B10=2E4 pre-regression figure=2E > > - I'm also assuming that 'cycles' here is a frequency-invariant > standardized constant 5=2E1 GHz TSC value or so? > >Thanks, > > Ingo It's cache hot, calling getppid() in a tight loop=2E The units are renorma= lized to from TSC cycles to core cycles using fixed counter 1 to determine = the actual ratio=2E