From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 679DB32F759 for ; Wed, 18 Feb 2026 18:45:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771440336; cv=none; b=V96mpEHoAriKrwbvrVjMzdX4U8HiGyfyF8X8wZDgN/Z2UZLjCRKPv/fRthbrXI3Avo5NCUo1lvDf6RL05Te9ZLsBmdUs5/K8Z5I/en1EFrI75QYcvIRdcre3DFuhc+YN+DzFXWZJKNnsM9nH09tifbQvrv2038yyYm7vHF3xdE0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771440336; c=relaxed/simple; bh=WgDe9Qdy6dAtGOrFjDwEX0uB0s8/xSnlP6On4HIDMGk=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=H8zlSwwZ7Svgi40MZnjnXAhziC9L/pRoF4M0q7lOGMhfbUeT7XD5qJAXBbnFYuaelXer5StfxxRNFYl0UuX+or6ot3ruco6Ftv/SUun6SZj0VyvrQLRiy5/jt4ZQIcmqM5zFM2WCyqLg7O+F5ukkwck2axI6/pTi6K/Pqcbn1CY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=i6JoP0GA; arc=none smtp.client-ip=198.175.65.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="i6JoP0GA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1771440334; x=1802976334; h=message-id:subject:from:to:cc:date:in-reply-to: references:content-transfer-encoding:mime-version; bh=WgDe9Qdy6dAtGOrFjDwEX0uB0s8/xSnlP6On4HIDMGk=; b=i6JoP0GAuDkpt/+VkcSBSdmYURHumDbaoXKQGgs3tX0CwqDqtUC74hTQ Cv5ZFiWSdzhcznoYpZDmDpdoD/nQ2v4gotdu/5C/YW0YeqXmQGUz/SSMl M1G5qSGLhhhUhPi1N77Z5OExr+Zj+Z2bndgDxUeuEJjQcr/Vwdhp3ZuGr d8+M0FHEWxL5OvZFcSM0PnPCGq/6y3b0ir4LhAGi4C6tQZtCipByL1LED widNinrGu6s19bziyN5SkPWswrkeETCr2aLtKdKYNp1MxKUjykb19JclV f3sL5LFb+lOv7XJtQ+sDsok7jBj2xHAl77rPHgWvqH0+P1PwAJDRlBZcs w==; X-CSE-ConnectionGUID: uveqQHK8QPWGFZq0Ov8GRg== X-CSE-MsgGUID: 92akrinRQB+zfEG+noiiSw== X-IronPort-AV: E=McAfee;i="6800,10657,11705"; a="72567326" X-IronPort-AV: E=Sophos;i="6.21,298,1763452800"; d="scan'208";a="72567326" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Feb 2026 10:45:34 -0800 X-CSE-ConnectionGUID: 24PgiD9dTAaXSwsCgXaxqg== X-CSE-MsgGUID: Du5M9g99SFmOJBGEPV5Qcg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,298,1763452800"; d="scan'208";a="214277823" Received: from unknown (HELO [10.241.243.83]) ([10.241.243.83]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Feb 2026 10:45:33 -0800 Message-ID: Subject: Re: [PATCH v3 04/21] sched/cache: Make LLC id continuous From: Tim Chen To: "Chen, Yu C" , K Prateek Nayak Cc: Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Qais Yousef , Libo Chen , linux-kernel@vger.kernel.org, Peter Zijlstra , "Gautham R . Shenoy" , Vincent Guittot , Ingo Molnar Date: Wed, 18 Feb 2026 10:45:33 -0800 In-Reply-To: References: <60a05a3f50d14a7bf3b968f62cca87893c5c552c.1770760558.git.tim.c.chen@linux.intel.com> <79755f7d-cc68-4189-b6d8-850378e54017@amd.com> <95923e36-4117-4209-97aa-a92b60f2bd49@intel.com> <6a73ac0d-1fc9-49a7-87a2-65951ab2c844@amd.com> <8df218ecf39015830856c7b1053a0f777cc1ea42.camel@linux.intel.com> <579c9039-cd66-41c3-8a72-e73d6f836ee8@amd.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.58.1 (3.58.1-1.fc43) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 On Wed, 2026-02-18 at 23:22 +0800, Chen, Yu C wrote: > On 2/18/2026 11:28 AM, K Prateek Nayak wrote: > > Hello Tim, > >=20 > > On 2/18/2026 4:42 AM, Tim Chen wrote: > > > On Tue, 2026-02-17 at 13:39 +0530, K Prateek Nayak wrote: > > > > Hello Chenyu, > > > >=20 > > > >=20 > > >=20 > > > [...snip...] > > >=20 > > >=20 > > > > > > > =C2=A0=C2=A0 */ > > > > > > > =C2=A0 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); > > > > > > > =C2=A0 DEFINE_PER_CPU(int, sd_llc_size); > > > > > > > -DEFINE_PER_CPU(int, sd_llc_id); > > > > > > > +DEFINE_PER_CPU(int, sd_llc_id) =3D -1; > > > > > > > =C2=A0 DEFINE_PER_CPU(int, sd_share_id); > > > > > > > =C2=A0 DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd= _llc_shared); > > > > > > > =C2=A0 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); > > > > > > > @@ -684,7 +685,6 @@ static void update_top_cache_domain(int c= pu) > > > > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rcu_assign_pointer(per= _cpu(sd_llc, cpu), sd); > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 per_cpu(sd_llc_size, cpu) =3D= size; > > > > > > > -=C2=A0=C2=A0=C2=A0 per_cpu(sd_llc_id, cpu) =3D id; > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rcu_assign_pointer(per_cpu(sd= _llc_shared, cpu), sds); > > > > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sd =3D lowest_flag_dom= ain(cpu, SD_CLUSTER); > > > > > > > @@ -2567,10 +2567,18 @@ build_sched_domains(const struct cpum= ask *cpu_map, struct sched_domain_attr *att > > > > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* Set up domains for = CPUs specified by the cpu_map: */ > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for_each_cpu(i, cpu_map) { > > > > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct sched_doma= in_topology_level *tl; > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct sched_doma= in_topology_level *tl, *tl_llc =3D NULL; > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 int lid; > > > > > > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 sd =3D NULL; > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for_e= ach_sd_topology(tl) { > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 int flags =3D 0; > > > > > > > + > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 if (tl->sd_flags) > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 flags =3D (*tl->sd_flags)(); > > > > > > > + > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 if (flags & SD_SHARE_LLC) > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 tl_llc =3D tl; > > > > > >=20 > > > > > > nit. This loop breaks out when sched_domain_span(sd) covers the= entire > > > > > > cpu_map and it might have not reached the topmost SD_SHARE_LLC = domain > > > > > > yet. Is that cause for any concern? > > > > > >=20 > > > > >=20 > > > > > Could you please elaborate a little more on this? If it covers th= e > > > > > entire cpu_map shouldn't it stop going up to its parent domain? > > > > > Do you mean, sd_llc_1 and its parent sd_llc_2 could cover the sam= e cpu_map, > > > > > and we should let tl_llc to assigned to sd_llc_2 (sd_llc_1 be deg= enerated? ) > > > >=20 > > > > I'm not sure if this is technically possible but assume following > > > > topology: > > > >=20 > > > > [ LLC: 8-15 ] > > > > [ SMT: 8,9 ][ SMT: 10,11 ] ... [ SMT: 14,15 ] > > > >=20 > > > > and the following series of events: > > > >=20 > > > > o All CPUs in LLC are offline to begin with (maxcpus =3D 1 like sce= nario). > > > >=20 > > > > o CPUs 10-15 are onlined first. > > > >=20 > > > > o CPU8 is put in a separate root partition and brought online. > > > > (XXX: I'm not 100% sure if this is possible in this order) > > > >=20 > > > > o build_sched_domains() will bail out at SMT domain since the cpuma= p > > > > is covered by tl->mask() and tl_llc =3D tl_smt. > > > >=20 > > > > o llc_id calculation uses the tl_smt->mask() which will not contain > > > > CPUs 10-15 and CPU8 will get a unique LLC id even though there a= re > > > > other online CPUs in the LLC with a different llc_id (!!!) > > > >=20 > > > >=20 > > > > Instead, if we traversed to tl_mc, we would have seen all the onlin= e > > > > CPUs in the MC and reused the llc_id from them. Might not be an iss= ue on > > > > its own but if this root partition is removed later, CPU8 will cont= inue > > > > to have the unique llc_id even after merging into the same MC domai= n. > > >=20 > > > There is really no reason to reuse the llc_id as far as cache aware s= cheduling > > > goes in its v3 revision (see my reply to Madadi on this patch). > >=20 > > Even I don't mind having some holes in the llc_id space when CPUs are > > offlined but my major concern would be seeing an inconsistent state > > where CPUs in same MC domains end up with different llc_id when after > > a bunch of hotplug activity. > >=20 > > > =20 > > > I am thinking that if we just simply rebuild LLC id across sched doma= in > > > rebuilds, that is probably the cleanest solution.=20 >=20 > Tim, do you mean reset all CPUs' LLC id to -1 whenever there is hotplug > event in partition_sched_domains_locked(), and rebuild them from scratch > in build_sched_domains(), so we already refresh the LLC id for every > CPU(I discussed with Vineeth here: > https://lore.kernel.org/all/54e60704-b0f3-44df-9b83-070806b5a00c@intel.co= m/) Yes, that's what I was thinking. However, there could be some races in cpus_share_cache() with this approach. Tim >=20 >=20 > > > There could be some races > > > in cpus_share_cache() as llc_id gets reassigned for some CPUs when th= ey > > > come online/offline. But we also having similar races in current mai= nline code. > > > Worst it can do is some temporary sub-optimal scheduling task placeme= nt. > > >=20 > > > Thoughts? > >=20 > > If you are suggesting populating the sd_llc_id for all the CPUs on > > topology rebuild, I'm not entirely against the idea. > >=20 > > On a separate note, if we add a dependency on SCHED_MC for SCHED_CACHE, > > we can simply look at cpu_coregroup_mask() and either allocate a new > > llc_id / borrow llc id in sched_cpu_activate() when CPU is onlined or > > reassign them in sched_cpu_deactivate() if an entire LLC is offlined. > >=20 >=20 > Prateek, may I know if you are thinking of updating every CPU's LLC id > during its hotplug and not update all percpu LLC id in=20 > build_sched_domains()? >=20 > thanks, > Chenyu