From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF96028D8D1 for ; Tue, 17 Feb 2026 23:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771369968; cv=none; b=W0KmC5cPHmsHtyZeLU76BSXzZ8N1+dSTCuR1h7dF6VpCvrn5pkrLa2gLu6WCXclvC90deJ8trxdXWzw6a3ICerBkTDangaTQDer3CUQ08o5tKQaIFQRhECNolO/iO9Aq7XP69tzN3lYNCY/Ucv+BarrQ/yamg9ixO9CjoC/4/Hg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771369968; c=relaxed/simple; bh=v2miejPuzhkSrWcHeA/Tz5eJ49iIHt17uDHi+cZyFIo=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=U5StYNRfS0n+DyMY3zwvP8fRoxwrxN4y+DCtYKXqyuUeOF6X6ArnpU+5os9vkJRXol6p48oj1Sb3UXJqIMcDnT1qhz+lyZRq7JfeTS632x7tjlD8RUJgfqyU/wh5IphnSE5ehroxZJJyz+kJwvbl8AZPwWCMpSymRDMdawrdevY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=catbDgqo; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="catbDgqo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1771369967; x=1802905967; h=message-id:subject:from:to:cc:date:in-reply-to: references:content-transfer-encoding:mime-version; bh=v2miejPuzhkSrWcHeA/Tz5eJ49iIHt17uDHi+cZyFIo=; b=catbDgqo9AaYoNBuDO2HynfpoD61ah/BkDDIeCz8W7gezLmMZ8UEQkBH O/jS5cgGaaqPG+TJCRxF39lpfol2TqAJwe/pxBNnf/aPXnYyUegiPANaJ CurqOyy5qqi9GbcltFuE/GCewju3h9A2B/sLZkJ0B5BXX/UTV+Ei22QUv hcUnFGRTzK2+Ow8XGsS9vxqYVbEasRGp1ARU6cPb1j6KtU12/ZIGI1RQ0 SfsU655nZN6sfTlAhX4Cff2Rti2UT19aBEGW/ahWgVJxRy7mHpPDt8pkJ gQyKXdn9hN7aIokN9/yxCmpjYqtLue64yMialI5ulS+3/8cscvMBnVVb4 w==; X-CSE-ConnectionGUID: 9MfQLjoSTuCLrIMNTWORrA== X-CSE-MsgGUID: nVt29UhRSQCUg9vQ3ObB2A== X-IronPort-AV: E=McAfee;i="6800,10657,11704"; a="72360854" X-IronPort-AV: E=Sophos;i="6.21,297,1763452800"; d="scan'208";a="72360854" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2026 15:12:46 -0800 X-CSE-ConnectionGUID: wd9hLdwDTuKFVV2ffVEFxQ== X-CSE-MsgGUID: K90/7BfeTTuG8YTtyb0gwA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,297,1763452800"; d="scan'208";a="213875605" Received: from unknown (HELO [10.241.243.83]) ([10.241.243.83]) by orviesa009-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2026 15:12:46 -0800 Message-ID: <8df218ecf39015830856c7b1053a0f777cc1ea42.camel@linux.intel.com> Subject: Re: [PATCH v3 04/21] sched/cache: Make LLC id continuous From: Tim Chen To: K Prateek Nayak , "Chen, Yu C" Cc: Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Qais Yousef , Libo Chen , linux-kernel@vger.kernel.org, Peter Zijlstra , "Gautham R . Shenoy" , Vincent Guittot , Ingo Molnar Date: Tue, 17 Feb 2026 15:12:45 -0800 In-Reply-To: <6a73ac0d-1fc9-49a7-87a2-65951ab2c844@amd.com> References: <60a05a3f50d14a7bf3b968f62cca87893c5c552c.1770760558.git.tim.c.chen@linux.intel.com> <79755f7d-cc68-4189-b6d8-850378e54017@amd.com> <95923e36-4117-4209-97aa-a92b60f2bd49@intel.com> <6a73ac0d-1fc9-49a7-87a2-65951ab2c844@amd.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.58.1 (3.58.1-1.fc43) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 On Tue, 2026-02-17 at 13:39 +0530, K Prateek Nayak wrote: > Hello Chenyu, >=20 >=20 [...snip...] > > > > =C2=A0=C2=A0 */ > > > > =C2=A0 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); > > > > =C2=A0 DEFINE_PER_CPU(int, sd_llc_size); > > > > -DEFINE_PER_CPU(int, sd_llc_id); > > > > +DEFINE_PER_CPU(int, sd_llc_id) =3D -1; > > > > =C2=A0 DEFINE_PER_CPU(int, sd_share_id); > > > > =C2=A0 DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_sh= ared); > > > > =C2=A0 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); > > > > @@ -684,7 +685,6 @@ static void update_top_cache_domain(int cpu) > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rcu_assign_pointer(per_cpu(sd= _llc, cpu), sd); > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 per_cpu(sd_llc_size, cpu) =3D size; > > > > -=C2=A0=C2=A0=C2=A0 per_cpu(sd_llc_id, cpu) =3D id; > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rcu_assign_pointer(per_cpu(sd_llc_sh= ared, cpu), sds); > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sd =3D lowest_flag_domain(cpu= , SD_CLUSTER); > > > > @@ -2567,10 +2567,18 @@ build_sched_domains(const struct cpumask *c= pu_map, struct sched_domain_attr *att > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* Set up domains for CPUs sp= ecified by the cpu_map: */ > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for_each_cpu(i, cpu_map) { > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct sched_domain_top= ology_level *tl; > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct sched_domain_top= ology_level *tl, *tl_llc =3D NULL; > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 int lid; > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sd = =3D NULL; > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for_each_sd_= topology(tl) { > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= int flags =3D 0; > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= if (tl->sd_flags) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 flags =3D (*tl->sd_flags)(); > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= if (flags & SD_SHARE_LLC) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 tl_llc =3D tl; > > >=20 > > > nit. This loop breaks out when sched_domain_span(sd) covers the entir= e > > > cpu_map and it might have not reached the topmost SD_SHARE_LLC domain > > > yet. Is that cause for any concern? > > >=20 > >=20 > > Could you please elaborate a little more on this? If it covers the > > entire cpu_map shouldn't it stop going up to its parent domain? > > Do you mean, sd_llc_1 and its parent sd_llc_2 could cover the same cpu_= map, > > and we should let tl_llc to assigned to sd_llc_2 (sd_llc_1 be degenerat= ed? ) >=20 > I'm not sure if this is technically possible but assume following > topology: >=20 > [ LLC: 8-15 ] > [ SMT: 8,9 ][ SMT: 10,11 ] ... [ SMT: 14,15 ] >=20 > and the following series of events: >=20 > o All CPUs in LLC are offline to begin with (maxcpus =3D 1 like scenario)= . >=20 > o CPUs 10-15 are onlined first. >=20 > o CPU8 is put in a separate root partition and brought online. > (XXX: I'm not 100% sure if this is possible in this order) >=20 > o build_sched_domains() will bail out at SMT domain since the cpumap > is covered by tl->mask() and tl_llc =3D tl_smt. >=20 > o llc_id calculation uses the tl_smt->mask() which will not contain > CPUs 10-15 and CPU8 will get a unique LLC id even though there are > other online CPUs in the LLC with a different llc_id (!!!) >=20 >=20 > Instead, if we traversed to tl_mc, we would have seen all the online > CPUs in the MC and reused the llc_id from them. Might not be an issue on > its own but if this root partition is removed later, CPU8 will continue > to have the unique llc_id even after merging into the same MC domain. There is really no reason to reuse the llc_id as far as cache aware schedul= ing goes in its v3 revision (see my reply to Madadi on this patch). =C2=A0 =20 I am thinking that if we just simply rebuild LLC id across sched domain rebuilds, that is probably the cleanest solution. There could be some races in cpus_share_cache() as llc_id gets reassigned for some CPUs when they come online/offline. But we also having similar races in current mainline = code. Worst it can do is some temporary sub-optimal scheduling task placement.= =20 Thoughts? Tim=20 >=20 > [..snip..] >=20 > > >=20 > > > It doesn't compact tl_max_llcs, but it should promote reuse of llc_id= if > > > all CPUs of a LLC go offline. I know it is a ridiculous scenario but = it > > > is possible nonetheless. > > >=20 > > > I'll let Peter and Valentin be the judge of additional space and > > > complexity needed for these bits :-) > > >=20 > >=20 > > Smart approach! Dynamically reallocating the llc_id should be feasible, > > as it releases the llc_id when the last CPU of that LLC is offlined. My > > only concern is data synchronization issues arising from the reuse of > > llc_id during load balancing - I=E2=80=99ll audit the logic to check fo= r any race > > conditions. Alternatively, what if we introduce a tl->static_mask? It w= ould > > be similar to tl->mask, but would not remove CPUs from static_mask when= they > > are offlined. This way, we can always find and reuse the llc_id of CPUs= in > > that LLC (even if all CPUs in the LLC have been offlined at some point, > > provided they were once online), and we would thus maintain a static ll= c_id. >=20 > That is possible but it would require a larger arch/ wide audit to add > support for. Might be less complex to handle in the generic layer but > again I'll let Peter and Valentin comment on this part :-) >=20 > >=20 > > Anyway, let do some testings on your proposal as well as static_mask th= ings, > > and I'll reply to this thread later. Thanks for the insights! >=20 > Thanks a ton! Much appreciated.