From: K Prateek Nayak <kprateek.nayak@amd.com>
To: Leon Romanovsky <leon@kernel.org>, Steve Wahl <steve.wahl@hpe.com>
Cc: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
linux-kernel@vger.kernel.org,
Vishal Chourasia <vishalc@linux.ibm.com>,
samir <samir@linux.ibm.com>,
Naman Jain <namjain@linux.microsoft.com>,
Saurabh Singh Sengar <ssengar@linux.microsoft.com>,
srivatsa@csail.mit.edu, Michael Kelley <mhklinux@outlook.com>,
Russ Anderson <rja@hpe.com>, Dimitri Sivanich <sivanich@hpe.com>
Subject: Re: [PATCH v4 1/2] sched/topology: improve topology_span_sane speed
Date: Tue, 17 Jun 2025 14:52:19 +0530 [thread overview]
Message-ID: <dcc9a2de-95ee-466e-b6d4-64658e315781@amd.com> (raw)
In-Reply-To: <20250617073442.GG750234@unreal>
Hello Leon,
On 6/17/2025 1:04 PM, Leon Romanovsky wrote:
> On Mon, Jun 16, 2025 at 09:18:41AM -0500, Steve Wahl wrote:
>> On Sun, Jun 15, 2025 at 09:42:07AM +0300, Leon Romanovsky wrote:
>>> On Thu, Jun 12, 2025 at 04:11:52PM +0530, K Prateek Nayak wrote:
>>>> On 6/12/2025 3:00 PM, K Prateek Nayak wrote:
>>>>> Ah! Since this happens so early topology isn't created yet for
>>>>> the debug prints to hit! Is it possible to get a dmesg with
>>>>> "ignore_loglevel" and "sched_verbose" on an older kernel that
>>>>> did not throw this error on the same host?
>>>
>>> This is dmesg with reverted two commits "ched/topology: Refinement to
>>> topology_span_sane speedup" and "sched/topology: improve
>>> topology_span_sane speed"
>
> <...>
>
>>>>
>>>> One better would be running with the following diff on top of v6.16-rc1
>>>> is possible:
>>>
>>> We are working to get this one too.
Thank you for all the data! Using the NUMA topology from the other
thread:
On 6/17/2025 1:25 PM, Leon Romanovsky wrote:
> [leonro@vm ~]$ sudo numactl -H
> available: 5 nodes (0-4)
> node 0 cpus: 0 1
> node 0 size: 2927 MB
> node 0 free: 1603 MB
> node 1 cpus: 2 3
> node 1 size: 3023 MB
> node 1 free: 3008 MB
> node 2 cpus: 4 5
> node 2 size: 3023 MB
> node 2 free: 3007 MB
> node 3 cpus: 6 7
> node 3 size: 3023 MB
> node 3 free: 3002 MB
> node 4 cpus: 8 9
> node 4 size: 3022 MB
> node 4 free: 2718 MB
> node distances:
> node 0 1 2 3 4
> 0: 10 39 38 37 36
> 1: 39 10 38 37 36
> 2: 38 38 10 37 36
> 3: 37 37 37 10 36
> 4: 36 36 36 36 10
I could reproduce the warning using:
sudo ~/dev/qemu/build/qemu-system-x86_64 -enable-kvm \
-cpu host \
-m 20G -smp cpus=10,sockets=10 -machine q35 \
-object memory-backend-ram,size=4G,id=m0 \
-object memory-backend-ram,size=4G,id=m1 \
-object memory-backend-ram,size=4G,id=m2 \
-object memory-backend-ram,size=4G,id=m3 \
-object memory-backend-ram,size=4G,id=m4 \
-numa node,cpus=0-1,memdev=m0,nodeid=0 \
-numa node,cpus=2-3,memdev=m1,nodeid=1 \
-numa node,cpus=4-5,memdev=m2,nodeid=2 \
-numa node,cpus=6-7,memdev=m3,nodeid=3 \
-numa node,cpus=8-9,memdev=m4,nodeid=4 \
-numa dist,src=0,dst=1,val=39 \
-numa dist,src=0,dst=2,val=38 \
-numa dist,src=0,dst=3,val=37 \
-numa dist,src=0,dst=4,val=36 \
-numa dist,src=1,dst=0,val=39 \
-numa dist,src=1,dst=2,val=38 \
-numa dist,src=1,dst=3,val=37 \
-numa dist,src=1,dst=4,val=36 \
-numa dist,src=2,dst=0,val=38 \
-numa dist,src=2,dst=1,val=38 \
-numa dist,src=2,dst=3,val=37 \
-numa dist,src=2,dst=4,val=36 \
-numa dist,src=3,dst=0,val=37 \
-numa dist,src=3,dst=1,val=37 \
-numa dist,src=3,dst=2,val=37 \
-numa dist,src=3,dst=4,val=36 \
-numa dist,src=4,dst=0,val=36 \
-numa dist,src=4,dst=1,val=36 \
-numa dist,src=4,dst=2,val=36 \
-numa dist,src=4,dst=3,val=36 \
...
>
> [ 0.435961] smp: Bringing up secondary CPUs ...
> [ 0.437573] smpboot: x86: Booting SMP configuration:
> [ 0.438611] .... node #0, CPUs: #1
> [ 0.440449] .... node #1, CPUs: #2 #3
> [ 0.442906] .... node #2, CPUs: #4 #5
> [ 0.445298] .... node #3, CPUs: #6 #7
> [ 0.447715] .... node #4, CPUs: #8 #9
> [ 0.481482] smp: Brought up 5 nodes, 10 CPUs
> [ 0.483160] smpboot: Total of 10 processors activated (45892.16 BogoMIPS)
> [ 0.486872] tl(SMT) CPU(0) ID(0) CPU_TL_SPAN(0) ID_TL_SPAN(0)
> [ 0.488029] tl(SMT) CPU(1) ID(1) CPU_TL_SPAN(1) ID_TL_SPAN(1)
> [ 0.489151] tl(SMT) CPU(2) ID(2) CPU_TL_SPAN(2) ID_TL_SPAN(2)
> [ 0.489761] tl(SMT) CPU(3) ID(3) CPU_TL_SPAN(3) ID_TL_SPAN(3)
> [ 0.490876] tl(SMT) CPU(4) ID(4) CPU_TL_SPAN(4) ID_TL_SPAN(4)
> [ 0.491996] tl(SMT) CPU(5) ID(5) CPU_TL_SPAN(5) ID_TL_SPAN(5)
> [ 0.493115] tl(SMT) CPU(6) ID(6) CPU_TL_SPAN(6) ID_TL_SPAN(6)
> [ 0.493754] tl(SMT) CPU(7) ID(7) CPU_TL_SPAN(7) ID_TL_SPAN(7)
> [ 0.494875] tl(SMT) CPU(8) ID(8) CPU_TL_SPAN(8) ID_TL_SPAN(8)
> [ 0.496008] tl(SMT) CPU(9) ID(9) CPU_TL_SPAN(9) ID_TL_SPAN(9)
> [ 0.497129] tl(PKG) CPU(0) ID(0) CPU_TL_SPAN(0-1) ID_TL_SPAN(0-1)
> [ 0.497763] tl(PKG) CPU(1) ID(0) CPU_TL_SPAN(0-1) ID_TL_SPAN(0-1)
> [ 0.498954] tl(PKG) CPU(2) ID(2) CPU_TL_SPAN(2-3) ID_TL_SPAN(2-3)
> [ 0.500167] tl(PKG) CPU(3) ID(2) CPU_TL_SPAN(2-3) ID_TL_SPAN(2-3)
> [ 0.501371] tl(PKG) CPU(4) ID(4) CPU_TL_SPAN(4-5) ID_TL_SPAN(4-5)
> [ 0.501792] tl(PKG) CPU(5) ID(4) CPU_TL_SPAN(4-5) ID_TL_SPAN(4-5)
> [ 0.503001] tl(PKG) CPU(6) ID(6) CPU_TL_SPAN(6-7) ID_TL_SPAN(6-7)
> [ 0.504202] tl(PKG) CPU(7) ID(6) CPU_TL_SPAN(6-7) ID_TL_SPAN(6-7)
> [ 0.505419] tl(PKG) CPU(8) ID(8) CPU_TL_SPAN(8-9) ID_TL_SPAN(8-9)
> [ 0.506637] tl(PKG) CPU(9) ID(8) CPU_TL_SPAN(8-9) ID_TL_SPAN(8-9)
> [ 0.507843] tl(NODE) CPU(0) ID(0) CPU_TL_SPAN(0-1,8-9) ID_TL_SPAN(0-1,8-9)
> [ 0.509199] tl(NODE) CPU(1) ID(0) CPU_TL_SPAN(0-1,8-9) ID_TL_SPAN(0-1,8-9)
> [ 0.509792] tl(NODE) CPU(2) ID(2) CPU_TL_SPAN(2-3,8-9) ID_TL_SPAN(2-3,8-9)
Looking at this, NODE should be a SD_OVERLAP domain here since the spans
across the nodes overlap. The following solves the warning for me:
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 8e06b1d22e91..759f7b8e24e6 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -2010,6 +2010,7 @@ void sched_init_numa(int offline_node)
*/
tl[i++] = (struct sched_domain_topology_level){
.mask = sd_numa_mask,
+ .flags = SDTL_OVERLAP,
.numa_level = 0,
SD_INIT_NAME(NODE)
};
--
NODE domain gets degenerated eventually via the default return in
sd_parent_degenerate() based on my tracing since "~cflags & pflags"
between PKG and NODE is 0 (node always has 1 group) but I'm not
sure if this requires more fundamental modification to
"sd_numa_mask".
Valentin, Peter, what is the right solution here?
> [ 0.511143] Failed tl: NODE
> [ 0.511789] Failed for CPU: 2
> [ 0.512466] ID CPU at tl: 2
> [ 0.513115] Failed CPU span at tl: 2-3,8-9
> [ 0.513701] ID CPU span: 2-3,8-9
> [ 0.514419] ID CPUs seen: 0
> [ 0.515055] CPUs covered: 0-1,8-9
--
Thanks and Regards,
Prateek
next prev parent reply other threads:[~2025-06-17 9:22 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-03-04 16:08 [PATCH v4 0/2] Improving topology_span_sane Steve Wahl
2025-03-04 16:08 ` [PATCH v4 1/2] sched/topology: improve topology_span_sane speed Steve Wahl
2025-04-08 19:05 ` [tip: sched/core] " tip-bot2 for Steve Wahl
2025-06-10 11:07 ` [PATCH v4 1/2] " Leon Romanovsky
2025-06-10 11:33 ` K Prateek Nayak
2025-06-10 12:36 ` Leon Romanovsky
2025-06-10 13:09 ` Leon Romanovsky
2025-06-10 19:39 ` Steve Wahl
2025-06-11 6:06 ` Leon Romanovsky
2025-06-11 6:56 ` K Prateek Nayak
2025-06-12 7:41 ` Leon Romanovsky
2025-06-12 9:30 ` K Prateek Nayak
2025-06-12 10:41 ` K Prateek Nayak
2025-06-15 6:42 ` Leon Romanovsky
2025-06-16 14:18 ` Steve Wahl
2025-06-17 3:04 ` K Prateek Nayak
2025-06-17 7:55 ` Leon Romanovsky
2025-06-17 7:34 ` Leon Romanovsky
2025-06-17 9:22 ` K Prateek Nayak [this message]
2025-06-23 6:06 ` K Prateek Nayak
2025-03-04 16:08 ` [PATCH v4 2/2] sched/topology: Refinement to topology_span_sane speedup Steve Wahl
2025-04-08 19:05 ` [tip: sched/core] " tip-bot2 for Steve Wahl
2025-03-06 6:46 ` [PATCH v4 0/2] Improving topology_span_sane K Prateek Nayak
2025-03-06 14:33 ` Valentin Schneider
2025-03-07 10:06 ` Madadi Vineeth Reddy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dcc9a2de-95ee-466e-b6d4-64658e315781@amd.com \
--to=kprateek.nayak@amd.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mhklinux@outlook.com \
--cc=mingo@redhat.com \
--cc=namjain@linux.microsoft.com \
--cc=peterz@infradead.org \
--cc=rja@hpe.com \
--cc=rostedt@goodmis.org \
--cc=samir@linux.ibm.com \
--cc=sivanich@hpe.com \
--cc=srivatsa@csail.mit.edu \
--cc=ssengar@linux.microsoft.com \
--cc=steve.wahl@hpe.com \
--cc=vincent.guittot@linaro.org \
--cc=vishalc@linux.ibm.com \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®