From: Dietmar Eggemann <dietmar.eggemann@arm.com>
To: K Prateek Nayak <kprateek.nayak@amd.com>,
Breno Leitao <leitao@debian.org>
Cc: Andrea Righi <arighi@nvidia.com>, Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
Christian Loehle <christian.loehle@arm.com>,
Phil Auld <pauld@redhat.com>, Koba Ko <kobak@nvidia.com>,
Felix Abecassis <fabecassis@nvidia.com>,
Balbir Singh <balbirs@nvidia.com>,
Joel Fernandes <joelagnelf@nvidia.com>,
Shrikanth Hegde <sshegde@linux.ibm.com>,
linux-kernel@vger.kernel.org
Subject: Re: kmemleak: sched_domain_shared leaked on asymmetric-capacity + SCHED_CACHE
Date: Mon, 6 Jul 2026 16:38:36 +0200 [thread overview]
Message-ID: <9476c607-92bd-4f98-b9a0-c9ca559c9d91@arm.com> (raw)
In-Reply-To: <73697ae3-f826-4b0e-be05-446976223822@amd.com>
Hi Breno & Prateek,
On 04.07.26 08:16, K Prateek Nayak wrote:
> Hello Breno,
>
> On 7/3/2026 9:49 PM, Breno Leitao wrote:
>> Hello Prateek,
>>
>> On Fri, Jul 03, 2026 at 04:05:59PM +0530, K Prateek Nayak wrote:
>>> On 7/3/2026 3:52 PM, Breno Leitao wrote:
>>>> On current linux-next (CONFIG_SCHED_CACHE=y) we hit a kmemleak on an
>>>> arm64 box with asymmetric CPU capacity, triggered by a cpuset-driven
>>>> sched-domain rebuild:
>>>>
>>>> kmemleak: unreferenced object 0xffff000100c95e80 (size 32):
>>>> comm "kworker/22:1", pid 407, jiffies 4294669077
>>>> hex dump (first 32 bytes):
>>>> 48 00 00 00 48 00 00 00 00 00 00 00 20 00 00 00 H...H....... ...
>>>> backtrace (crc ec5d7053):
>>>> __kmalloc_cache_node_noprof
>>>> build_sched_domains
>>>> partition_sched_domains
>>>> rebuild_sched_domains_locked
>>>> rebuild_sched_domains
>>>> process_scheduled_works
>>>> kthread
>>>> ret_from_fork
>>>> kmemleak: 1 new suspected memory leaks
>>>>
>>>> The leaked object is a struct sched_domain_shared (32 bytes with
>>>> CONFIG_SCHED_CACHE) allocated in __sds_alloc(), inlined into
>>>> build_sched_domains()
>>>>
>>>> Is this a known issue?
>>>
>>> It is theoretically possible but there is a defensive WARN_ON()
>>> in topology code that you should have hit first. Do you see any
>>> other warning?
>>>
>>> If it is not too much trouble, could you add "sched_verbose"
>>> to your kernel cmdline (or do
>>> echo Y > /sys/kernel/debug/sched/verbose) and redo this cpuset
>>> that leaks the data and share the dmesg. It should give some clue
>>> what the topology looks like that causes this.
>>
>> Sure, does this one help:
>
> Unfortunately no! This doesn't give the span and capacity
> information so I cannot really tell what kind of topology
> actually triggers this.
I'm trying to recreate this on Arm64 qemu with dtb file to get the
asymmetric CPU capacities (emulating Pixel6):
$ cat /sys/devices/system/cpu/cpu*/cpu_capacity
160
160
160
160
498
498
1024
1024
I'm on linux_next master:
eb8711ece2c - (HEAD -> linux-next, tag: next-20260702, linux-next/master)
so it has:
9e005ed21152 - sched/topology: Allow multiple domains to claim
sched_domain_shared (2026-05-19 K Prateek Nayak)
$ cat /sys/kernel/debug/sched/domains/cpu0/domain0/{name,flags}
MC
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE
SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING
8 CPUs flat:
$ cat /proc/schedstat | grep "^c\|^d" | awk '{ print $1 " " $2 " " $3}'
cpu0 0 0
domain0 MC ff
cpu1 0 0
domain0 MC ff
cpu2 0 0
domain0 MC ff
cpu3 0 0
domain0 MC ff
cpu4 0 0
domain0 MC ff
cpu5 0 0
domain0 MC ff
cpu6 0 0
domain0 MC ff
cpu7 0 0
domain0 MC ff
> Ideally, you enable verbose logging, do the cpuset operation
> that leaks the object, and then save the demesg after which
> will have a complete topology for all CPUs there including
> their capacities and we can piece together a qemu environment
> to debug where it all went wrong.
+1
$ cat /proc/cmdline
... sched_verbose=1
What exactly is this cpuset operation forcing a sched domain rebuild?
I'm trying a split into 2 exclusive cpusets {0,2,4,6} and {1,3,5,7} to
get 2 asymmetric CPU capacity islands.
...
[ 95.747359] root domain span: 1,3,5,7
[ 95.747760] root domain span: 0,2,4,6
[ 95.747771] _sched_cache_active_set: enabling cache aware scheduling
[ 95.747788] rd 1,3,5,7: Checking EAS: frequency-invariant load
tracking not yet supported
...
$ cat /sys/kernel/debug/sched/domains/cpu0/domain0/{name,flags}
MC
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE
SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING
$ cat /proc/schedstat | grep "^c\|^d" | awk '{ print $1 " " $2 " " $3}'
cpu0 0 0
domain0 MC 55
cpu1 0 0
domain0 MC aa
cpu2 0 0
domain0 MC 55
cpu3 0 0
domain0 MC aa
cpu4 0 0
domain0 MC 55
cpu5 0 0
domain0 MC aa
cpu6 0 0
domain0 MC 55
cpu7 0 0
domain0 MC aa
But this hasn't triggered kmemleak so far.
That said, it triggered once (~600s uptime) but w/o any cpuset
operation? Never happened again since then :-(
[...]
> Is this the defualt topology before the cpuset? Because
> min_interval of 8 means there are 8 cPUs in the span and
> you have 8 CPUs in your machine so there seems to be a
> single root partition when you captured this.
+1
> Also does your dmesg have any warnings from topology.c?
>
> The only way I can see this triggering is there is a single
> cpu SD_SHARE_LLC at top which claims all the sd->shared and
> then you get a SD_ASYM_CPUCAPACITY_FULL on top which too
> tries to claim a sd->shared but all are already gone.
?
> Afaict, all CPUs in your system are on single LLC and you
> can't SD_ASYM_CPUCAPACITY_FULL without having more than one
> capacity in the span so I'm at a loss currently as to how
> this reference gets dropped without being freed.
$ cat /sys/devices/system/cpu/cpu*/cpu_capacity should tell.
Another thing: does your system enable 'cache aware scheduling', i.e. do
you not see:
pr_info("%s: cache aware scheduling not supported on this platform\n",
__func__);
next prev parent reply other threads:[~2026-07-06 14:38 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-09 18:07 [PATCH v6 0/5 RESEND] sched/fair: SMT-aware asymmetric CPU capacity Andrea Righi
2026-05-09 18:07 ` [PATCH 1/5] sched/fair: Drop redundant RCU read lock in NOHZ kick path Andrea Righi
2026-05-11 13:04 ` Vincent Guittot
2026-05-15 6:49 ` Shrikanth Hegde
2026-05-16 5:45 ` Andrea Righi
2026-05-16 17:15 ` Shrikanth Hegde
2026-05-20 8:34 ` [tip: sched/core] " tip-bot2 for Andrea Righi
[not found] ` <CGME20260521194704eucas1p19ffbed79a4ae514fb1136218c567add6@eucas1p1.samsung.com>
2026-05-21 19:47 ` [PATCH 1/5] " Marek Szyprowski
2026-05-21 20:13 ` Andrea Righi
2026-05-09 18:07 ` [PATCH 2/5] sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity Andrea Righi
2026-05-11 13:04 ` Vincent Guittot
2026-05-15 10:05 ` Shrikanth Hegde
2026-05-16 5:58 ` [PATCH v2 " Andrea Righi
2026-05-16 17:19 ` Shrikanth Hegde
2026-05-18 20:58 ` Peter Zijlstra
2026-05-18 21:31 ` Andrea Righi
2026-05-19 5:52 ` K Prateek Nayak
2026-05-19 6:43 ` Andrea Righi
2026-05-19 7:47 ` K Prateek Nayak
2026-05-19 7:54 ` Andrea Righi
2026-05-19 8:46 ` Peter Zijlstra
2026-05-19 11:27 ` K Prateek Nayak
2026-05-19 11:47 ` Peter Zijlstra
2026-05-25 8:30 ` Chen, Yu C
2026-05-20 8:34 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-07-03 10:22 ` kmemleak: sched_domain_shared leaked on asymmetric-capacity + SCHED_CACHE Breno Leitao
2026-07-03 10:35 ` K Prateek Nayak
2026-07-03 16:19 ` Breno Leitao
2026-07-04 6:16 ` K Prateek Nayak
2026-07-06 14:38 ` Dietmar Eggemann [this message]
2026-07-07 4:11 ` K Prateek Nayak
2026-07-07 13:31 ` Breno Leitao
2026-07-07 13:59 ` K Prateek Nayak
2026-07-07 15:31 ` Dietmar Eggemann
2026-05-09 18:07 ` [PATCH 3/5] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection Andrea Righi
2026-05-11 13:07 ` Vincent Guittot
2026-05-11 13:45 ` Andrea Righi
2026-05-11 14:25 ` [PATCH v2 " Andrea Righi
2026-05-20 8:34 ` [tip: sched/core] " tip-bot2 for Andrea Righi
2026-05-09 18:07 ` [PATCH 4/5] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity Andrea Righi
2026-05-11 13:07 ` Vincent Guittot
2026-05-15 10:09 ` Shrikanth Hegde
2026-05-16 9:04 ` Andrea Righi
2026-05-20 8:34 ` [tip: sched/core] " tip-bot2 for Andrea Righi
2026-05-09 18:07 ` [PATCH 5/5] sched/fair: Add SIS_UTIL support to select_idle_capacity() Andrea Righi
2026-05-11 13:08 ` Vincent Guittot
2026-05-20 8:34 ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9476c607-92bd-4f98-b9a0-c9ca559c9d91@arm.com \
--to=dietmar.eggemann@arm.com \
--cc=arighi@nvidia.com \
--cc=balbirs@nvidia.com \
--cc=bsegall@google.com \
--cc=christian.loehle@arm.com \
--cc=fabecassis@nvidia.com \
--cc=joelagnelf@nvidia.com \
--cc=juri.lelli@redhat.com \
--cc=kobak@nvidia.com \
--cc=kprateek.nayak@amd.com \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sshegde@linux.ibm.com \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®