From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7303118C03E for ; Thu, 19 Feb 2026 15:40:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771515644; cv=none; b=pXnpgK69Y//NPwDJlvm27/Yub5BOAwVWPDHYLbfX3N9fPWTN/scZGSIkel/FMRfoRJqMrQ6Tdm1S/ZVcLnEAZWzRdCFL3MCAM3Ohzch8yP0KGHUuRNLo+MVH1GhFcupj2XoN3qXrRdEGqVoml3k3sWOdGDg33ruyyhF7wOvAGlM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771515644; c=relaxed/simple; bh=NIYLQMXrxUNGIizt+vnZLk27um/OYl9zdG76sdSw+Co=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=rxZ7KaXHhSQSk2LXKbwV/0VFCsR/bv2Vda1hzBZNqd/8n2vJYlVE6JQSwgWoK1UnsXNk6QiE+bTOo1NyEzjNp+1EZn9FggwWooCINAAGwQGkP3HIeihmnvuStMvjMblfJjNC8IWRyltPxdebL8UqDe1zsJ8M1rw92Bh2Y0D0rlk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=MVXI9zvc; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="MVXI9zvc" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=vqyzEwGoCHs/jvl9XYc1yCMG9kW1KTyDUU7sEmTy00c=; b=MVXI9zvco2w3lYe3LUYuEuyhCd kmgFjLUwbRPblvQqawovtD234aOekxXZv9L54zP7AG9RPV/Y+/Cxjud/3w6URdV11r+AgoLU+U3XN SAqptEHCtTja9orMwmH+gMf8zWEy9Lv5t64J078iKyWY09Dd7NOMQsuy/1abslXI23IuBIQMbulJu o1KvPqGOsbqVvaZhPsuZ3+VKhvwY7vG87KNS2bR7WMI5/4gqz5b4E4li81NhtPtxrE/bWFJUN44aB sed5qbVY8xZVqIzTOtzg9AB11cn+vbX4++/YEOYJKT5Oa5QAGQ3yXZE6t+j1LinEXtTGZUS8kide4 x7P77qwg==; Received: from 2001-1c00-8d85-5700-266e-96ff-fe07-7dcc.cable.dynamic.v6.ziggo.nl ([2001:1c00:8d85:5700:266e:96ff:fe07:7dcc] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1vt68O-00000001TzN-1Bu4; Thu, 19 Feb 2026 15:40:08 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id D0796300B40; Thu, 19 Feb 2026 16:40:07 +0100 (CET) Date: Thu, 19 Feb 2026 16:40:07 +0100 From: Peter Zijlstra To: K Prateek Nayak Cc: Tim Chen , Ingo Molnar , "Gautham R . Shenoy" , Vincent Guittot , Chen Yu , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Qais Yousef , Libo Chen , linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 04/21] sched/cache: Make LLC id continuous Message-ID: <20260219154007.GI1395266@noisy.programming.kicks-ass.net> References: <60a05a3f50d14a7bf3b968f62cca87893c5c552c.1770760558.git.tim.c.chen@linux.intel.com> <79755f7d-cc68-4189-b6d8-850378e54017@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <79755f7d-cc68-4189-b6d8-850378e54017@amd.com> On Mon, Feb 16, 2026 at 01:14:20PM +0530, K Prateek Nayak wrote: > > @@ -2581,6 +2589,39 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att > > if (cpumask_equal(cpu_map, sched_domain_span(sd))) > > break; > > } > > + > > + lid = per_cpu(sd_llc_id, i); > > + if (lid == -1) { > > + int j; > > + > > + /* > > + * Assign the llc_id to the CPUs that do not > > + * have an LLC. > > + */ > > + if (!tl_llc) { > > + per_cpu(sd_llc_id, i) = tl_max_llcs++; > > + > > + continue; > > + } > > + > > + /* try to reuse the llc_id of its siblings */ > > + for_each_cpu(j, tl_llc->mask(tl_llc, i)) { > > > My only large concern that remains is the fact that offline CPUs are > taken out the the tl->mask() which can lead to interesting cases where > CPUs on same LLC can have different llc_id: > > o Boot with maxcpus=1 > > o Run: > > for i in {1..$NRCPUS}; do > echo 1 > /sys/devices/system/cpu/cpu$i/online; > echo 0 > /sys/devices/system/cpu/cpu$i/online; > done Lol, cute ;-) > diff --git a/kernel/sched/core.c b/kernel/sched/core.c > index c6efa71cf500..aee1be89ab4c 100644 > --- a/kernel/sched/core.c > +++ b/kernel/sched/core.c > @@ -8268,6 +8268,8 @@ static void cpuset_cpu_active(void) > static void cpuset_cpu_inactive(unsigned int cpu) > { > if (!cpuhp_tasks_frozen) { > + /* XXX: Is this the right spot? */ > + sched_domains_free_llc_id(cpu); > cpuset_update_active_cpus(); > } else { > num_cpus_frozen++; > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > index de5b701c3950..31a8910297c7 100644 > --- a/kernel/sched/sched.h > +++ b/kernel/sched/sched.h > @@ -3903,6 +3903,7 @@ static inline bool sched_cache_enabled(void) > } > #endif > extern void init_sched_mm(struct task_struct *p); > +void sched_domains_free_llc_id(int cpu); > > extern u64 avg_vruntime(struct cfs_rq *cfs_rq); > extern int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se); > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c > index ca46b5cf7f78..04c1ab489ee2 100644 > --- a/kernel/sched/topology.c > +++ b/kernel/sched/topology.c > @@ -18,6 +18,7 @@ void sched_domains_mutex_unlock(void) > } > > /* Protected by sched_domains_mutex: */ > +static cpumask_var_t sched_domains_llc_id_allocmask; > static cpumask_var_t sched_domains_tmpmask; > static cpumask_var_t sched_domains_tmpmask2; > static int tl_max_llcs; > @@ -2543,6 +2544,53 @@ static bool topology_span_sane(const struct cpumask *cpu_map) > return true; > } > > +static int __sched_domains_alloc_llc_id(void) > +{ > + int lid; > + > + lockdep_assert_held(&sched_domains_mutex); > + > + lid = cpumask_first_zero(sched_domains_llc_id_allocmask); > + if (lid >= tl_max_llcs) > + tl_max_llcs++; Urgh,. should we not rather track the max lid? Also, we allocate max_llc sized data structures, if this thing is 'variable' we must also always store a copy of the 'lid' size of the time of allocation. > + > + /* > + * llc_id space should never grow larger than the > + * possible number of CPUs in the system. > + */ > + if (!unlikely(WARN_ON_ONCE(lid >= nr_cpumask_bits))) > + cpumask_set_cpu(lid, sched_domains_llc_id_allocmask); __cpumask_set_cpu() Since you're serializing everything with that sched_domains_mutex, this need not be an atomic op. > + return lid; > +} > + > +static void __sched_domains_free_llc_id(int cpu) > +{ > + int i, lid; > + > + lockdep_assert_held(&sched_domains_mutex); > + > + lid = per_cpu(sd_llc_id, cpu); > + if (lid == -1) > + return; > + > + per_cpu(sd_llc_id, cpu) = -1; > + > + for_each_online_cpu(i) { > + /* An online CPU owns the llc_id. */ > + if (per_cpu(sd_llc_id, i) == lid) > + return; > + } > + > + cpumask_clear_cpu(lid, sched_domains_llc_id_allocmask); __cpumask_clear_cpu() > +} So this deals with Madadi's issue I suppose. > +void sched_domains_free_llc_id(int cpu) > +{ > + sched_domains_mutex_lock(); > + __sched_domains_free_llc_id(cpu); > + sched_domains_mutex_unlock(); > +} > + > /* > * Build sched domains for a given set of CPUs and attach the sched domains > * to the individual CPUs > @@ -2599,7 +2647,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att > * have an LLC. > */ > if (!tl_llc) { > - per_cpu(sd_llc_id, i) = tl_max_llcs++; > + per_cpu(sd_llc_id, i) = __sched_domains_alloc_llc_id(); > > continue; > } > @@ -2620,7 +2668,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att > > /* a new LLC is detected */ > if (lid == -1) > - per_cpu(sd_llc_id, i) = tl_max_llcs++; > + per_cpu(sd_llc_id, i) = __sched_domains_alloc_llc_id(); > } > } > > @@ -2798,6 +2846,7 @@ int __init sched_init_domains(const struct cpumask *cpu_map) > { > int err; > > + zalloc_cpumask_var(&sched_domains_llc_id_allocmask, GFP_KERNEL); > zalloc_cpumask_var(&sched_domains_tmpmask, GFP_KERNEL); > zalloc_cpumask_var(&sched_domains_tmpmask2, GFP_KERNEL); > zalloc_cpumask_var(&fallback_doms, GFP_KERNEL); > --- > > It doesn't compact tl_max_llcs, but it should promote reuse of llc_id if > all CPUs of a LLC go offline. I know it is a ridiculous scenario but it > is possible nonetheless. > > I'll let Peter and Valentin be the judge of additional space and > complexity needed for these bits :-) It appears straight forward enough I suppose.