mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Dietmar Eggemann <dietmar.eggemann@arm.com>
To: K Prateek Nayak <kprateek.nayak@amd.com>,
	Breno Leitao <leitao@debian.org>
Cc: Andrea Righi <arighi@nvidia.com>, Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Juri Lelli <juri.lelli@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Valentin Schneider <vschneid@redhat.com>,
	Christian Loehle <christian.loehle@arm.com>,
	Phil Auld <pauld@redhat.com>, Koba Ko <kobak@nvidia.com>,
	Felix Abecassis <fabecassis@nvidia.com>,
	Balbir Singh <balbirs@nvidia.com>,
	Joel Fernandes <joelagnelf@nvidia.com>,
	Shrikanth Hegde <sshegde@linux.ibm.com>,
	linux-kernel@vger.kernel.org
Subject: Re: kmemleak: sched_domain_shared leaked on asymmetric-capacity + SCHED_CACHE
Date: Mon, 6 Jul 2026 16:38:36 +0200	[thread overview]
Message-ID: <9476c607-92bd-4f98-b9a0-c9ca559c9d91@arm.com> (raw)
In-Reply-To: <73697ae3-f826-4b0e-be05-446976223822@amd.com>

Hi Breno & Prateek,

On 04.07.26 08:16, K Prateek Nayak wrote:
> Hello Breno,
> 
> On 7/3/2026 9:49 PM, Breno Leitao wrote:
>> Hello Prateek,
>>
>> On Fri, Jul 03, 2026 at 04:05:59PM +0530, K Prateek Nayak wrote:
>>> On 7/3/2026 3:52 PM, Breno Leitao wrote:
>>>> On current linux-next (CONFIG_SCHED_CACHE=y) we hit a kmemleak on an
>>>> arm64 box with asymmetric CPU capacity, triggered by a cpuset-driven
>>>> sched-domain rebuild:
>>>>
>>>> kmemleak: unreferenced object 0xffff000100c95e80 (size 32):
>>>>   comm "kworker/22:1", pid 407, jiffies 4294669077
>>>>   hex dump (first 32 bytes):
>>>>     48 00 00 00 48 00 00 00 00 00 00 00 20 00 00 00  H...H....... ...
>>>>   backtrace (crc ec5d7053):
>>>>     __kmalloc_cache_node_noprof
>>>>     build_sched_domains
>>>>     partition_sched_domains
>>>>     rebuild_sched_domains_locked
>>>>     rebuild_sched_domains
>>>>     process_scheduled_works
>>>>     kthread
>>>>     ret_from_fork
>>>>   kmemleak: 1 new suspected memory leaks
>>>>
>>>> The leaked object is a struct sched_domain_shared (32 bytes with
>>>> CONFIG_SCHED_CACHE) allocated in __sds_alloc(), inlined into
>>>> build_sched_domains()
>>>>
>>>> Is this a known issue?
>>>
>>> It is theoretically possible but there is a defensive WARN_ON()
>>> in topology code that you should have hit first. Do you see any
>>> other warning?
>>>
>>> If it is not too much trouble, could you add "sched_verbose"
>>> to your kernel cmdline (or do
>>> echo Y > /sys/kernel/debug/sched/verbose) and redo this cpuset
>>> that leaks the data and share the dmesg. It should give some clue
>>> what the topology looks like that causes this.
>>
>> Sure, does this one help:
> 
> Unfortunately no! This doesn't give the span and capacity
> information so I cannot really tell what kind of topology
> actually triggers this.

I'm trying to recreate this on Arm64 qemu with dtb file to get the
asymmetric CPU capacities (emulating Pixel6):

$ cat /sys/devices/system/cpu/cpu*/cpu_capacity
160
160
160
160
498
498
1024
1024

I'm on linux_next master:

eb8711ece2c - (HEAD -> linux-next, tag: next-20260702, linux-next/master)

so it has:

9e005ed21152 - sched/topology: Allow multiple domains to claim
sched_domain_shared (2026-05-19 K Prateek Nayak)

$ cat /sys/kernel/debug/sched/domains/cpu0/domain0/{name,flags}
MC
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE
SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING

8 CPUs flat:

$ cat /proc/schedstat | grep "^c\|^d" | awk '{ print $1 " " $2 " " $3}'
cpu0 0 0
domain0 MC ff
cpu1 0 0
domain0 MC ff
cpu2 0 0
domain0 MC ff
cpu3 0 0
domain0 MC ff
cpu4 0 0
domain0 MC ff
cpu5 0 0
domain0 MC ff
cpu6 0 0
domain0 MC ff
cpu7 0 0
domain0 MC ff

> Ideally, you enable verbose logging, do the cpuset operation
> that leaks the object, and then save the demesg after which
> will have a complete topology for all CPUs there including
> their capacities and we can piece together a qemu environment
> to debug where it all went wrong.

+1

$ cat /proc/cmdline
... sched_verbose=1

What exactly is this cpuset operation forcing a sched domain rebuild?

I'm trying a split into 2 exclusive cpusets {0,2,4,6} and {1,3,5,7} to
get 2 asymmetric CPU capacity islands.

...

[   95.747359] root domain span: 1,3,5,7
[   95.747760] root domain span: 0,2,4,6
[   95.747771] _sched_cache_active_set: enabling cache aware scheduling
[   95.747788] rd 1,3,5,7: Checking EAS: frequency-invariant load
tracking not yet supported
...

$ cat /sys/kernel/debug/sched/domains/cpu0/domain0/{name,flags}
MC
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE
SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING

$ cat /proc/schedstat | grep "^c\|^d" | awk '{ print $1 " " $2 " " $3}'
cpu0 0 0
domain0 MC 55
cpu1 0 0
domain0 MC aa
cpu2 0 0
domain0 MC 55
cpu3 0 0
domain0 MC aa
cpu4 0 0
domain0 MC 55
cpu5 0 0
domain0 MC aa
cpu6 0 0
domain0 MC 55
cpu7 0 0
domain0 MC aa

But this hasn't triggered kmemleak so far.

That said, it triggered once (~600s uptime) but w/o any cpuset
operation? Never happened again since then :-(

[...]

> Is this the defualt topology before the cpuset? Because
> min_interval of 8 means there are 8 cPUs in the span and
> you have 8 CPUs in your machine so there seems to be a
> single root partition when you captured this.

+1

> Also does your dmesg have any warnings from topology.c?
> 
> The only way I can see this triggering is there is a single
> cpu SD_SHARE_LLC at top which claims all the sd->shared and
> then you get a SD_ASYM_CPUCAPACITY_FULL on top which too
> tries to claim a sd->shared but all are already gone.

?

> Afaict, all CPUs in your system are on single LLC and you
> can't SD_ASYM_CPUCAPACITY_FULL without having more than one
> capacity in the span so I'm at a loss currently as to how
> this reference gets dropped without being freed.

$ cat /sys/devices/system/cpu/cpu*/cpu_capacity  should tell.


Another thing: does your system enable 'cache aware scheduling', i.e. do
you not see:

pr_info("%s: cache aware scheduling not supported on this platform\n",
__func__);




  reply	other threads:[~2026-07-06 14:38 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-05-09 18:07 [PATCH v6 0/5 RESEND] sched/fair: SMT-aware asymmetric CPU capacity Andrea Righi
2026-05-09 18:07 ` [PATCH 1/5] sched/fair: Drop redundant RCU read lock in NOHZ kick path Andrea Righi
2026-05-11 13:04   ` Vincent Guittot
2026-05-15  6:49   ` Shrikanth Hegde
2026-05-16  5:45     ` Andrea Righi
2026-05-16 17:15       ` Shrikanth Hegde
2026-05-20  8:34   ` [tip: sched/core] " tip-bot2 for Andrea Righi
     [not found]   ` <CGME20260521194704eucas1p19ffbed79a4ae514fb1136218c567add6@eucas1p1.samsung.com>
2026-05-21 19:47     ` [PATCH 1/5] " Marek Szyprowski
2026-05-21 20:13       ` Andrea Righi
2026-05-09 18:07 ` [PATCH 2/5] sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity Andrea Righi
2026-05-11 13:04   ` Vincent Guittot
2026-05-15 10:05   ` Shrikanth Hegde
2026-05-16  5:58     ` [PATCH v2 " Andrea Righi
2026-05-16 17:19       ` Shrikanth Hegde
2026-05-18 20:58       ` Peter Zijlstra
2026-05-18 21:31         ` Andrea Righi
2026-05-19  5:52         ` K Prateek Nayak
2026-05-19  6:43           ` Andrea Righi
2026-05-19  7:47             ` K Prateek Nayak
2026-05-19  7:54               ` Andrea Righi
2026-05-19  8:46           ` Peter Zijlstra
2026-05-19 11:27             ` K Prateek Nayak
2026-05-19 11:47               ` Peter Zijlstra
2026-05-25  8:30                 ` Chen, Yu C
2026-05-20  8:34       ` [tip: sched/core] " tip-bot2 for K Prateek Nayak
2026-07-03 10:22   ` kmemleak: sched_domain_shared leaked on asymmetric-capacity + SCHED_CACHE Breno Leitao
2026-07-03 10:35     ` K Prateek Nayak
2026-07-03 16:19       ` Breno Leitao
2026-07-04  6:16         ` K Prateek Nayak
2026-07-06 14:38           ` Dietmar Eggemann [this message]
2026-07-07  4:11             ` K Prateek Nayak
2026-07-07 13:31               ` Breno Leitao
2026-07-07 13:59                 ` K Prateek Nayak
2026-07-07 15:31                   ` Dietmar Eggemann
2026-05-09 18:07 ` [PATCH 3/5] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection Andrea Righi
2026-05-11 13:07   ` Vincent Guittot
2026-05-11 13:45     ` Andrea Righi
2026-05-11 14:25     ` [PATCH v2 " Andrea Righi
2026-05-20  8:34       ` [tip: sched/core] " tip-bot2 for Andrea Righi
2026-05-09 18:07 ` [PATCH 4/5] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity Andrea Righi
2026-05-11 13:07   ` Vincent Guittot
2026-05-15 10:09   ` Shrikanth Hegde
2026-05-16  9:04     ` Andrea Righi
2026-05-20  8:34   ` [tip: sched/core] " tip-bot2 for Andrea Righi
2026-05-09 18:07 ` [PATCH 5/5] sched/fair: Add SIS_UTIL support to select_idle_capacity() Andrea Righi
2026-05-11 13:08   ` Vincent Guittot
2026-05-20  8:34   ` [tip: sched/core] " tip-bot2 for K Prateek Nayak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9476c607-92bd-4f98-b9a0-c9ca559c9d91@arm.com \
    --to=dietmar.eggemann@arm.com \
    --cc=arighi@nvidia.com \
    --cc=balbirs@nvidia.com \
    --cc=bsegall@google.com \
    --cc=christian.loehle@arm.com \
    --cc=fabecassis@nvidia.com \
    --cc=joelagnelf@nvidia.com \
    --cc=juri.lelli@redhat.com \
    --cc=kobak@nvidia.com \
    --cc=kprateek.nayak@amd.com \
    --cc=leitao@debian.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=pauld@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=sshegde@linux.ibm.com \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®