From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04BFA326D43 for ; Mon, 15 Dec 2025 20:49:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.10 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765831746; cv=none; b=P/WN1nCarRNo2m3/q3iNEiUu0uP2pRnnEfRxzNtHpYQ2NmZdXvM7M1ILK4V+GQojJMSPfd3YahLZajLSNW2jl8hq+Loaual2Tbq/0AFs19JDyZM57Tf2E75fP52iOsWtNryhRRwh0eba4Sja4vvLvWtZeR0EuoaedJ4UzpVSCzs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765831746; c=relaxed/simple; bh=vVZ//h7AzmjTmUeyXAnGXWE8ZIxx/em1xggydtmEASE=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=ptqEMxVCOGkF7lL/udhiF5hZprA5fI1X4+epEkQ1kXJDNlt5/kAZTn3NEjlcGsm85ZPCOidD2l3HRQD5BivtlxYJ0TzDYIujG6wW5Be5emKaf1c2m8t6wcs3rGpUQoWxIGH4A2YaxxIsO6lsHC/Wm7ox3I5YbWxV13n4mZYQXXQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=JgWk2jgQ; arc=none smtp.client-ip=198.175.65.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="JgWk2jgQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1765831745; x=1797367745; h=message-id:subject:from:to:cc:date:in-reply-to: references:content-transfer-encoding:mime-version; bh=vVZ//h7AzmjTmUeyXAnGXWE8ZIxx/em1xggydtmEASE=; b=JgWk2jgQxsoX6h403Llyzntq1S+N+bFAwsYqrvDWjHFZ7dYnn68XtGim Ow60eBLxWfZltUh8xZWTOXgd9JRoZNpwkTzlki2GMN4+Bz3KZuRGamw6Y 5h51myKfAve7tOs11YDiBbqX1n3w+LxXMWRu4wIs9EZWcjTwgQXJWtTRu P640cAiKbB06YLCSGTAebVnWkqyzF9rmIk299MgQisaIdGKwnjYz94n+d +pBGjUorHs+uciyYXORBL3bJ/6t/GN2F2EsJQ4LuzRkXqzWMf5soFC/Ih /9eV7Wtu1dhG5bsLqtHj2vSe9oNiATy4ywxJRLWqVyTCZS1Klr4hZ9so5 w==; X-CSE-ConnectionGUID: r9gNoxrfTUK7gf1BcuI+lA== X-CSE-MsgGUID: cm4cA7xoTKS7OSiWRllP/A== X-IronPort-AV: E=McAfee;i="6800,10657,11643"; a="85156438" X-IronPort-AV: E=Sophos;i="6.21,151,1763452800"; d="scan'208";a="85156438" Received: from orviesa003.jf.intel.com ([10.64.159.143]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 12:49:04 -0800 X-CSE-ConnectionGUID: 7SjbM4gjT6ehTzQmF3X2tw== X-CSE-MsgGUID: z4GwhSjmRyawUA2V5zPytA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,151,1763452800"; d="scan'208";a="202004094" Received: from schen9-mobl4.amr.corp.intel.com (HELO [10.125.110.163]) ([10.125.110.163]) by ORVIESA003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 12:49:02 -0800 Message-ID: <3bb9bf31222af552ee4b0db30bb1624bd84d3cba.camel@linux.intel.com> Subject: Re: [PATCH v2 04/23] sched/cache: Make LLC id continuous From: Tim Chen To: Peter Zijlstra Cc: Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , linux-kernel@vger.kernel.org Date: Mon, 15 Dec 2025 12:49:01 -0800 In-Reply-To: <20251209115846.GL3707891@noisy.programming.kicks-ass.net> References: <20251209115846.GL3707891@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.58.1 (3.58.1-1.fc43) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 On Tue, 2025-12-09 at 12:58 +0100, Peter Zijlstra wrote: > On Wed, Dec 03, 2025 at 03:07:23PM -0800, Tim Chen wrote: >=20 > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > > index 710ed9943d27..0a3918269906 100644 > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -1210,10 +1210,17 @@ __read_mostly unsigned int llc_imb_pct = =3D 20; > > =20 > > static int llc_id(int cpu) > > { > > + int llc; > > + > > if (cpu < 0) > > return -1; > > =20 > > + llc =3D per_cpu(sd_llc_id, cpu); > > + /* avoid race with cpu hotplug */ > > + if (unlikely(llc >=3D max_llcs)) > > + return -1; > > + > > + return llc; > > } > > =20 > > void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *_pc= pu_sched) >=20 > > @@ -668,6 +670,55 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asy= m_cpucapacity); > > DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity); > > DEFINE_STATIC_KEY_FALSE(sched_cluster_active); > > =20 > > +/* > > + * Assign continuous llc id for the CPU, and return > > + * the assigned llc id. > > + */ > > +static int update_llc_id(struct sched_domain *sd, > > + int cpu) > > +{ > > + int id =3D per_cpu(sd_llc_id, cpu), i; > > + > > + if (id >=3D 0) > > + return id; > > + > > + if (sd) { > > + /* Look for any assigned id and reuse it.*/ > > + for_each_cpu(i, sched_domain_span(sd)) { > > + id =3D per_cpu(sd_llc_id, i); > > + > > + if (id >=3D 0) { > > + per_cpu(sd_llc_id, cpu) =3D id; > > + return id; > > + } > > + } > > + } > > + > > + /* > > + * When 1. there is no id assigned to this LLC domain, > > + * or 2. the sd is NULL, we reach here. > > + * Consider the following scenario, > > + * CPU0~CPU95 are in the node0, CPU96~CPU191 are > > + * in the node1. During bootup, maxcpus=3D96 is > > + * appended. > > + * case 1: When running cpu_attach_domain(CPU24) > > + * during boot up, CPU24 is the first CPU in its > > + * non-NULL LLC domain. However, > > + * its corresponding llc id has not been assigned yet. > > + * > > + * case 2: After boot up, the CPU100 is brought up > > + * via sysfs manually. As a result, CPU100 has only a > > + * Numa domain attached, because CPU100 is the only CPU > > + * of a sched domain, all its bottom domains are degenerated. > > + * The LLC domain pointer sd is NULL for CPU100. > > + * > > + * For both cases, we want to increase the number of LLCs. > > + */ > > + per_cpu(sd_llc_id, cpu) =3D max_llcs++; > > + > > + return per_cpu(sd_llc_id, cpu); > > +} >=20 > I'm not sure I follow. So partition_sched_domains() first calls > detach_destroy_domains() on the old set, and then build_sched_domains() > on the new set. >=20 > Do detach_destroy_domain() will do: >=20 > cpu_attach_domain(NULL,..); >=20 > That is, it will explicitly attach the NULL sched_domain to a CPU. At > which point I feel update_llc_id() should be returning -1, no? >=20 > Then later, build_sched_domains() will set a !NULL sched_domain, at > which point update_llc_id() can set a real value. >=20 > This should then also get rid of that weird max_llcs check in llc_id(), > right? Thanks for pointing this out. Yes, we should take care of the attachment of NULL sd. Will update the code accordingly. Tim