From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34F562882CC for ; Mon, 25 Aug 2025 07:56:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756108621; cv=none; b=pNagsJEJ3En33Uy0Lr4mpVVfH9g7q60A03r7tInTAcKtmk0/scDKaMe7cJSRt8MRfoHPBRRA2onXZWp/4XcIk3IYUxtdtKZkWHtjy6EYkSFQsVCtdynJtpt1cYOLus2OpNeA3RNdK252uwKN4vuIEX0h9Lbfqguh/BHLC5d9isw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756108621; c=relaxed/simple; bh=zrR7XMABDhMTJECD45nVsjYnWUp0Y6PS7dMmGjtm0uI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=L5vQOICBQXph69BW1rH0wrzSdb3ixZ9WAMclRGu0mS792G6jRubx6iGYB5HyFeE9CzKUPjQnB3KV4XCNWjfQCkA69E/coQk+ufHyjjHt3Tkrty6gf4tBHMWv0rls2MoBzpEoGIEfePq1iVt6/9xGPdILtSMkNFivjHYES3LlDf0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=dOJZKwj4; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="dOJZKwj4" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Ougs+rHCkaIhdPeCZhoUIh51Oe7lb1hKIxpeB732pMs=; b=dOJZKwj46s4J/aP7YGV4XRtUOU 3M2VV0S72VkYk4ykDAlZXTDOcbBor9m6mb2fWfia8v2/gl04AJA9sAfbbesPC3uToYoo5BNf+2i/Q kGcUseYn6cIOncmE11VGEYIjv8jiN7CWqIPGiL+bdW0EBurGX5Lro0HDG8R17QGeOm/Y8BaSQDldO N26YUYyZKBRIdU8fsHCWyKhMLnlKMN3dN57vznlqh07SFE8ubXk7IjXIkF9mEpFSOZnh/gPO10utW WAx+X4uQzpmtuP30WESyQ07yMUZhWXtGyHmdLEkPMDwW03Y2TlwlA2n+CugPH6PLpheVdzfAJcg8j Y2KLi49g==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uqS4I-000000082Ft-1tKI; Mon, 25 Aug 2025 07:56:43 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 349CC3002ED; Mon, 25 Aug 2025 09:56:42 +0200 (CEST) Date: Mon, 25 Aug 2025 09:56:42 +0200 From: Peter Zijlstra To: "Chen, Yu C" Cc: Tim Chen , Ingo Molnar , Juri Lelli , Dietmar Eggemann , Ben Segall , Mel Gorman , Valentin Schneider , Tim Chen , Vincent Guittot , Libo Chen , Abel Wu , Len Brown , linux-kernel@vger.kernel.org, K Prateek Nayak , "Gautham R . Shenoy" , Zhao Liu , Vinicius Costa Gomes , Chen Yu Subject: Re: [PATCH 2/2] sched: Fix sched domain build error for GNR-X, CWF-X in SNC-3 mode Message-ID: <20250825075642.GQ3245006@noisy.programming.kicks-ass.net> References: <86ddfe75510497829a84e696b29bfdd7a4940009.1755893468.git.tim.c.chen@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Mon, Aug 25, 2025 at 01:08:39PM +0800, Chen, Yu C wrote: > On 8/23/2025 4:14 AM, Tim Chen wrote: > > It is possible for Granite Rapids X (GNR) and Clearwater Forest X > > (CWF) to have up to 3 dies per package. When sub-numa cluster (SNC-3) > > is enabled, each die will become a separate NUMA node in the package > > with different distances between dies within the same package. > > > > For example, on GNR-X, we see the following numa distances for a 2 socket > > system with 3 dies per socket: > > > > package 1 package2 > > ---------------- > > | | > > --------- --------- > > | 0 | | 3 | > > --------- --------- > > | | > > --------- --------- > > | 1 | | 4 | > > --------- --------- > > | | > > --------- --------- > > | 2 | | 5 | > > --------- --------- > > | | > > ---------------- > > > > node distances: > > node 0 1 2 3 4 5 > > 0: 10 15 17 21 28 26 > > 1: 15 10 15 23 26 23 > > 2: 17 15 10 26 23 21 > > 3: 21 28 26 10 15 17 > > 4: 23 26 23 15 10 15 > > 5: 26 23 21 17 15 10 > > > > diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c > > index 33e166f6ab12..c425e84c88b5 100644 > > --- a/arch/x86/kernel/smpboot.c > > +++ b/arch/x86/kernel/smpboot.c > > @@ -515,6 +515,34 @@ static void __init build_sched_topology(void) > > set_sched_topology(topology); > > } > > +int sched_node_distance(int from, int to) > > +{ > > + int d = node_distance(from, to); > > + > > + if (!x86_has_numa_in_package) > > + return d; > > + > > + switch (boot_cpu_data.x86_vfm) { > > + case INTEL_GRANITERAPIDS_X: > > + case INTEL_ATOM_DARKMONT_X: > > + if (d < REMOTE_DISTANCE) > > + return d; > > + > > + /* > > + * Trim finer distance tuning for nodes in remote package > > + * for the purpose of building sched domains. > > + * Put NUMA nodes in each remote package in a single sched group. > > + * Simplify NUMA domains and avoid extra NUMA levels including different > > + * NUMA nodes in remote packages. > > + * > > + * GNR-x and CWF-X has GLUELESS-MESH topology with SNC > > + * turned on. > > + */ > > + d = (d / 10) * 10; > > Does the '10' here mean that, the distance of the hierarchy socket > is 10 from SLIT table? For example, from a socket0 point of view, > the distance of socket1 to socket0 is within [20, 29), the distance > of socket2 to socket0 is [30,39), and so on. If this is the case, > maybe add a comment above for future reference. This is all because of the ACPI SLIT distance definitions I suppose, 10 for local and 20 for remote (which IMO is actively wrong, since it mandates distances that are not relative performance). Additionally, the table above magically has all the remote distances in the range of [20,29] and so the strip 1s thing works. The problem of course is that the SLIT table is fully under control of the BIOS and random BIOS monkey could cause this to not be so making the above code not work as intended. Eg. if the remote distances ends up being in the range of [20,35] or whatever, then it all goes sideways. ( There is a history of manupulating the SLIT table to influence scheduler behaviour of OS of choice :-/ ) Similarly, when doing a 4 node system, it is possible a 2 hop distances doesn't align nicely with the 10s and we're up a creek again. This is all very fragile. A much better way would be to allocate a new SLIT table, identify the (local) clusters and replace all remote instances with an average. Eg. since (21+28+26+23+26+23+26+23+21)/9 ~ 24, you end up with: node 0 1 2 3 4 5 0: 10 15 17 24 24 24 1: 15 10 15 24 24 24 2: 17 15 10 24 24 24 3: 24 24 24 10 15 17 4: 24 24 24 15 10 15 5: 24 24 24 17 15 10