From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA86642A9D for ; Sat, 14 Feb 2026 17:54:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771091699; cv=none; b=EtbDmNZqThFxp6oz0nfRoT0bg252j08W3wI4iI6DuXadJq1Bg2gs/WUogZaMbz3niSbZfrdEuALHQH9bRXbw3GXpH0I3TrinesZKkZqDHbXKrCptAhoZ3822VSvbBS8UswY8sprVQdeb6WQRH4mP6sFdNOOk0x9oXlOJkOyZ4vs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771091699; c=relaxed/simple; bh=kcJZCfRMRZrSGLBoczz8eKy8WK5kfIzsx9U5EKkO8j8=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=CQ/LdvOzZ04b1lULtglWbzh6fEkFoGS+EnruhEbZRv1kOHjNkvnxpFn+HKLSgibeVro3Bc+s5RgMFceCb0R1HYYBbqDbgFOZPixPfRhGqPi+84r20BAtNPA8uzb/+T/FAh/y44hpsp9erUGq1+JcPjqB64tpTxag1M8Q5JmPFBs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=jhv87Ian; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="jhv87Ian" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 61EFjPKA515161; Sat, 14 Feb 2026 17:54:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=lhbneN +pJ5ng+F7Y3r9n+kfvkbpc8kCX3NNdeXfUhko=; b=jhv87IanbNRSUBt2HfzKiv 1SUcHYTX6T+iXfwziibWrogLAKFxeLnz7Xc6WXR7m924E6DnP4511miwdpQb23WQ 8jhONOXnweOoQ+N3WrdwfYx6aq6YIioMUHC2efQHpiEyDvfk4y1VEUcoW1RRumW7 9hjdYbKwsV7frRI3s4L9ksqjYU2FKLE7CXo5HR8ZzCfL01LacHr2aQOUoqMK6hwN 6ni3POwndJzaHr6KehquGWqjbyRH6wQ/sC8bLUVsrGFInNvge9W4MomA6L9zFqMS /vtVgiVLvWo9WolvRno6ZeiB9oIGRTM4iPCA9ZNkA797HLLoOdctQTtHlNj4ef2Q == Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4cajcqhf1g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 14 Feb 2026 17:54:02 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 61EH162F008378; Sat, 14 Feb 2026 17:54:00 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([172.16.1.7]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4c6g3yufbt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 14 Feb 2026 17:54:00 +0000 Received: from smtpav04.wdc07v.mail.ibm.com (smtpav04.wdc07v.mail.ibm.com [10.39.53.231]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 61EHrxvF30802686 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 14 Feb 2026 17:53:59 GMT Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 41E0758056; Sat, 14 Feb 2026 17:53:59 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8AA1158045; Sat, 14 Feb 2026 17:53:38 +0000 (GMT) Received: from [9.67.107.13] (unknown [9.67.107.13]) by smtpav04.wdc07v.mail.ibm.com (Postfix) with ESMTP; Sat, 14 Feb 2026 17:53:38 +0000 (GMT) Message-ID: <437fef08-cabe-461f-a2d2-4bc385e9d513@linux.ibm.com> Date: Sat, 14 Feb 2026 23:23:35 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 04/21] sched/cache: Make LLC id continuous To: Tim Chen Cc: Peter Zijlstra , Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot , Chen Yu , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Qais Yousef , Libo Chen , linux-kernel@vger.kernel.org, Madadi Vineeth Reddy References: <60a05a3f50d14a7bf3b968f62cca87893c5c552c.1770760558.git.tim.c.chen@linux.intel.com> Content-Language: en-US From: Madadi Vineeth Reddy In-Reply-To: <60a05a3f50d14a7bf3b968f62cca87893c5c552c.1770760558.git.tim.c.chen@linux.intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Proofpoint-GUID: Mz-vCsMXfD_iLtSppQeoOutOXBhLwkPk X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwMjE0MDE0NiBTYWx0ZWRfX5Bn6DMEwisrk 9u6bJVGBW/YWoLN9ghX2XWsi1b9f6jEFs4LgprARgbStikj10pivzz7SM3LKo3q+iJj8D7r0Wo/ y+mDzeqEUHjnPcnOk6a0kMgDB/6nUla6FZIUGZML6kfLLtDRDDTRwVAX1/dBzVT1/zOYs1+t7Az tAiB3Cq5Jo8G2gs5vkycRErTZrfWYdjNokniBj9wklDJTrdLXvanUcDdnZUJMp1tJBPlIgZP2q8 Sqep8KMqJvnTzMa1U7H2e7hW3cOhOWTCH+gsX1UrLB8g//7mYNXy2U86lj9ynyJr9c7Iu3LzFrT zQdre9LmhkSMChwE/OqvqwTflDd+vSxjcJuottcfD+KcRKmUR42kXp6ZuVSBCpKh6Bs8/XRSViM 5UxZ03BFRVQID6tYTiYtxDKHMNZ3pHs5mXEEf6HsRiKeMFD0LjwWz7gZ/qDvGVQAbQZLXAlRO4P UY9gvJoFKlERhw8l/VQ== X-Authority-Analysis: v=2.4 cv=UPXQ3Sfy c=1 sm=1 tr=0 ts=6990b6ba cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=IkcTkHD0fZMA:10 a=HzLeVaNsDn8A:10 a=VkNPw1HP01LnGYTKEx00:22 a=Mpw57Om8IfrbqaoTuvik:22 a=GgsMoib0sEa3-_RKJdDe:22 a=QyXUC8HyAAAA:8 a=zd2uoN0lAAAA:8 a=bPzKVmalH0duPtMDrccA:9 a=QEXdDO2ut3YA:10 X-Proofpoint-ORIG-GUID: DmhwVuQKDaGBeSNWqX1919RoRy7nxEWi X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1121,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-02-14_02,2026-02-13_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 suspectscore=0 phishscore=0 impostorscore=0 adultscore=0 bulkscore=0 clxscore=1015 lowpriorityscore=0 priorityscore=1501 spamscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2601150000 definitions=main-2602140146 On 11/02/26 03:48, Tim Chen wrote: > From: Chen Yu > > Introduce an index mapping between CPUs and their LLCs. This provides > a continuous per LLC index needed for cache-aware load balancing in > later patches. > > The existing per_cpu llc_id usually points to the first CPU of the > LLC domain, which is sparse and unsuitable as an array index. Using > llc_id directly would waste memory. > > With the new mapping, CPUs in the same LLC share a continuous id: > > per_cpu(llc_id, CPU=0...15) = 0 > per_cpu(llc_id, CPU=16...31) = 1 > per_cpu(llc_id, CPU=32...47) = 2 > ... > > Once a CPU has been assigned an llc_id, this ID persists even when > the CPU is taken offline and brought back online, which can facilitate > the management of the ID. tl_max_llcs is never reset across multiple invocations of build_sched_domains(). While this preserves LLC IDs across normal CPU hotplug events, I'm wondering about scenarios where hardware topology changes, such as physically removing/replacing CPU sockets. Example scenario: Boot with 3 LLCs: IDs {0,1,2}, tl_max_llcs=3 Physical hardware change removes LLC 1 New hardware added at a different position gets ID=3 After multiple such events: System has 4 LLCs but IDs {0,2,5,7}, tl_max_llcs=8 This creates gaps in the ID space. However, I understand this trade-off might be intentional since physical topology changes are rare, and resetting tl_max_llcs and all sd_llc_id values would rebuild IDs on every invocation of build_sched_domains(). Would like to know your thoughts on overhead of resetting tl_max_llcs and sd_llc_id so that IDs are rebuilt on each invocation of build_sched_domains() to always maintain a dense mapping. Thanks, Vineeth > > Co-developed-by: Tim Chen > Signed-off-by: Tim Chen > Co-developed-by: K Prateek Nayak > Signed-off-by: K Prateek Nayak > Signed-off-by: Chen Yu > --- > > Notes: > v2->v3: > Allocate the LLC id according to the topology level data directly, rather > than calculating from the sched domain. This simplifies the code. > (Peter Zijlstra, K Prateek Nayak) > > kernel/sched/topology.c | 47 ++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 44 insertions(+), 3 deletions(-) > > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c > index cf643a5ddedd..ca46b5cf7f78 100644 > --- a/kernel/sched/topology.c > +++ b/kernel/sched/topology.c > @@ -20,6 +20,7 @@ void sched_domains_mutex_unlock(void) > /* Protected by sched_domains_mutex: */ > static cpumask_var_t sched_domains_tmpmask; > static cpumask_var_t sched_domains_tmpmask2; > +static int tl_max_llcs; > > static int __init sched_debug_setup(char *str) > { > @@ -658,7 +659,7 @@ static void destroy_sched_domains(struct sched_domain *sd) > */ > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); > DEFINE_PER_CPU(int, sd_llc_size); > -DEFINE_PER_CPU(int, sd_llc_id); > +DEFINE_PER_CPU(int, sd_llc_id) = -1; > DEFINE_PER_CPU(int, sd_share_id); > DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); > @@ -684,7 +685,6 @@ static void update_top_cache_domain(int cpu) > > rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); > per_cpu(sd_llc_size, cpu) = size; > - per_cpu(sd_llc_id, cpu) = id; > rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); > > sd = lowest_flag_domain(cpu, SD_CLUSTER); > @@ -2567,10 +2567,18 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att > > /* Set up domains for CPUs specified by the cpu_map: */ > for_each_cpu(i, cpu_map) { > - struct sched_domain_topology_level *tl; > + struct sched_domain_topology_level *tl, *tl_llc = NULL; > + int lid; > > sd = NULL; > for_each_sd_topology(tl) { > + int flags = 0; > + > + if (tl->sd_flags) > + flags = (*tl->sd_flags)(); > + > + if (flags & SD_SHARE_LLC) > + tl_llc = tl; > > sd = build_sched_domain(tl, cpu_map, attr, sd, i); > > @@ -2581,6 +2589,39 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att > if (cpumask_equal(cpu_map, sched_domain_span(sd))) > break; > } > + > + lid = per_cpu(sd_llc_id, i); > + if (lid == -1) { > + int j; > + > + /* > + * Assign the llc_id to the CPUs that do not > + * have an LLC. > + */ > + if (!tl_llc) { > + per_cpu(sd_llc_id, i) = tl_max_llcs++; > + > + continue; > + } > + > + /* try to reuse the llc_id of its siblings */ > + for_each_cpu(j, tl_llc->mask(tl_llc, i)) { > + if (i == j) > + continue; > + > + lid = per_cpu(sd_llc_id, j); > + > + if (lid != -1) { > + per_cpu(sd_llc_id, i) = lid; > + > + break; > + } > + } > + > + /* a new LLC is detected */ > + if (lid == -1) > + per_cpu(sd_llc_id, i) = tl_max_llcs++; > + } > } > > if (WARN_ON(!topology_span_sane(cpu_map)))