From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azon11010015.outbound.protection.outlook.com [52.101.46.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2575224AF1 for ; Sat, 4 Jul 2026 06:16:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.46.15 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783145780; cv=fail; b=aR51tnMTTqHY89RsYSZSxJvjpSoqTfUSThWYNfDQ5GMvzDFnwxvOkxlg1C7IsREWUi0hWLLAJlfrjwdRpxadfTZNTyId8UFIS9gdtqagKb/tgri1F5xwn5wTDc8+MkB/wkV2S0vAhbHtx4NNwT0L8gY+X31lox6JPyc2NHQeE6A= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783145780; c=relaxed/simple; bh=FdZ/44Z9PrE91I07bYOj1878zm834SCG7B3nbgteHug=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=NXxAW7krShNm32QvrR1vyJIiEkMTGiK/zkd1ZFuaemtEe3U8I+0ftkD4l0RtYsb1HEOZYH9WJvpzpU+rqLROUPxhjyRVYigiaGpeW3ljAJBhIgQbXy9OKyJ+lfKaxpelWW1vmxs3/Ar+B06X76eCQUuCpH3g6jl9JdP9DJrcFsc= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=MBbtfeBm; arc=fail smtp.client-ip=52.101.46.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="MBbtfeBm" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Z31G7+TjMcXqBpH034Puoo8sWBnIevTzxNh/ekKN8V0HAqopq8K5BR1eIpviJbZph2H7t+JryyoO7hjqQycAeXzMfzoGJ0dVL5YaOesf0F58bwqz8aBdsoiUn/LgLFubPAlMSqyX/9gRitWNpvhX9zvHRhfnLqVdHlS12ABtDaRECPSa1hp1Ypch6fUBlns0o7Ay74SNnb0/VJDHUbjtgaWnbZrim8Ijm59B4rSzedgoRA8DgRDk4vlOP6tK6rT1SOOfECIiXUw1WH7eNL2qel7z8AsrHAklVW7rEtYGsoGP+jwY5WU14QMDgZMVb/PT5OLbXE12OB0/81z9KbbeHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lwaxgO10IeWh96gLXAsfDMREgLHOMStxbZ8MRzpSwZQ=; b=nNCNe9DS4jnumFJYtt50sJRbSpENk/JYnaDSMw3glNg57Y+7eyP6kH4NeOekHfbF6FRCEfX0S9E+/9AdW/U877GFwCfWCCD0Q4Aov6JiPe0VJtj1CA7eqIAJjKS5qzA2CYGqJbuYkQTLbmEEoP/FYpn1wZqs4DUgarIp/Ml9CzUUDuEtpk8ncXNDEmCmECj8+iwac6ZwaQQ1sImxy09JUeSQin/rRG/W9iHPE+nK4WsiRPvl426p/aDRl6uPQ3/aN7Rphlo4WK+M9dhUuSec2aBDSHa842UjY3o6OZ/6kbNxl2gX/Z/VfS92NXnRecW31vWq6p30LLwDNneFpV6UCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=debian.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lwaxgO10IeWh96gLXAsfDMREgLHOMStxbZ8MRzpSwZQ=; b=MBbtfeBmP/WSD4RRoWP07OfTYoQ4/TwIz5QvayB6jeLv0+9lvt6KsppyZ2bfP1IADdbTVpQqeeLVlyyFqne3wsC9VxRSPZ/840kc/C1iaoF+ip5O3/9NaqjQ3M5ErcmAwu3HNJbA4b5T7ILHRIRsPC/poATbYn48bgE8AbyLIGo= Received: from SA0PR11CA0148.namprd11.prod.outlook.com (2603:10b6:806:131::33) by IA0PPF80FB91A80.namprd12.prod.outlook.com (2603:10b6:20f:fc04::bd5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.12; Sat, 4 Jul 2026 06:16:13 +0000 Received: from SN1PEPF000397B1.namprd05.prod.outlook.com (2603:10b6:806:131:cafe::a8) by SA0PR11CA0148.outlook.office365.com (2603:10b6:806:131::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.181.11 via Frontend Transport; Sat, 4 Jul 2026 06:16:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by SN1PEPF000397B1.mail.protection.outlook.com (10.167.248.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.6 via Frontend Transport; Sat, 4 Jul 2026 06:16:12 +0000 Received: from satlexmb10.amd.com (10.181.42.219) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sat, 4 Jul 2026 01:16:12 -0500 Received: from satlexmb07.amd.com (10.181.42.216) by satlexmb10.amd.com (10.181.42.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sat, 4 Jul 2026 01:16:11 -0500 Received: from [172.31.178.191] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.41 via Frontend Transport; Sat, 4 Jul 2026 01:16:05 -0500 Message-ID: <73697ae3-f826-4b0e-be05-446976223822@amd.com> Date: Sat, 4 Jul 2026 11:46:04 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: kmemleak: sched_domain_shared leaked on asymmetric-capacity + SCHED_CACHE To: Breno Leitao CC: Andrea Righi , Ingo Molnar , "Peter Zijlstra" , Juri Lelli , "Vincent Guittot" , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Phil Auld , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , References: <20260509180955.1840064-1-arighi@nvidia.com> <20260509180955.1840064-3-arighi@nvidia.com> <73685ae1-6d90-4799-a1d7-08a1fcf24b8b@amd.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000397B1:EE_|IA0PPF80FB91A80:EE_ X-MS-Office365-Filtering-Correlation-Id: b1971aa1-6320-4da7-085b-08ded993bf5d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|1800799024|82310400026|23010399003|376014|7416014|56012099006|11063799006|4143699003|18002099003|22082099003|6133799003|10086099003; X-Microsoft-Antispam-Message-Info: +6Tb8nfpFkdGKcE7qdi2GzQrBt7qA8XD3HYNqPZoyjhOXEBcKihuD5l8i8JS6jGJjluhkzJI5w/Zvnm5JC6p3BSD7wPBjluvzrJYfU0ne5kFRYKm606hrcKs2y3Irn937JH7SYpN11pukKo8dmFEeWBxBUv/++T7mP2laRhf19NwgXlpy4F0SCcUYNCKINHGxK41T326R9jiIrH+B5oKYuy6nig2dxyOuyHirfRQsh6csojLh1jbXfIk21J91cC6LgqiKVRjF2RVgcodb2PAQblv85dwcEAf8pKj1kM/YKjVGq3sP0+qna6ZXj7ok2avINlSO7QC8Ofc+Qi1CcSnY2OhpRksKW0Q6Njy5+xaDuZCS0KNJn4ebmAhCFrvyoRYJ9AktBACH6YgnCoxJvLQEDmQxb3vnGfNoLYUf7+Y/Yfa9zuGCv5ys8MpdoUZHq0uPh555aux6J7+TyZyC7yShcjb7H4IPfXO8jKcdOoZXhNIhft/eRsO4K56vXxElW+VYjrNFo4aq/btF9M50yZ2CzKN7kSSKrRC1x56Wp5Tp2mddOmzVgJNzcGIU6mApA4rMS9TGuFg1xmjJw0ZCVS3Mw605825hfzxJcHYFsz/HziIaOX7YA6YKWE7lEGqJJgXnOovrlhk4wEB2TEajxXTx4N/nrFYyBJ3nBueDVXA7MCL9DOAi2w0y8qrXEkR3c3qNnSxnBWG+ITy7mZ7aGTPhw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(1800799024)(82310400026)(23010399003)(376014)(7416014)(56012099006)(11063799006)(4143699003)(18002099003)(22082099003)(6133799003)(10086099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: dLpRaLxvk3+hZU7t+EpEY45reB/sLt5k/f9SKp3gfPASdu90iodWiZUc1n2I1nJmzJGelcIO5XGzpDr1PD9FI5NGul/5TOt6CdjFRJCj97mM61wnOdjs/49PF/IFc2+0yPZdUFJTZrT8XsT+UtPBsg8itJvowL5VWM90OmISTNeNA1coB02ak7mPYjS7A5GVUYwG6oSjKGk62lNHzG85rUQ5yQ6IauhyO7QavkSluq6ZLPbDqPMtYdi+oOvN/BXaMaBeLo8Hhb0Swhlt1hqI2S/b+NUOhEPXXnAGxLpDntDZ6zgg3LerCzFCO73BL+AHHradkhCBy/Qr7seQY8LOKaSiFjlgJhnjxkH3zZ3pnynSfInOGJcAixfwHxWoCJ2nughDqiuaNC6HCLmYTMlqXDi41ebTJm+OcHNCm3JHfcshki4pVRxIceeCH6Z5K6+r X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2026 06:16:12.6486 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b1971aa1-6320-4da7-085b-08ded993bf5d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000397B1.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PPF80FB91A80 Hello Breno, On 7/3/2026 9:49 PM, Breno Leitao wrote: > Hello Prateek, > > On Fri, Jul 03, 2026 at 04:05:59PM +0530, K Prateek Nayak wrote: >> On 7/3/2026 3:52 PM, Breno Leitao wrote: >>> On current linux-next (CONFIG_SCHED_CACHE=y) we hit a kmemleak on an >>> arm64 box with asymmetric CPU capacity, triggered by a cpuset-driven >>> sched-domain rebuild: >>> >>> kmemleak: unreferenced object 0xffff000100c95e80 (size 32): >>> comm "kworker/22:1", pid 407, jiffies 4294669077 >>> hex dump (first 32 bytes): >>> 48 00 00 00 48 00 00 00 00 00 00 00 20 00 00 00 H...H....... ... >>> backtrace (crc ec5d7053): >>> __kmalloc_cache_node_noprof >>> build_sched_domains >>> partition_sched_domains >>> rebuild_sched_domains_locked >>> rebuild_sched_domains >>> process_scheduled_works >>> kthread >>> ret_from_fork >>> kmemleak: 1 new suspected memory leaks >>> >>> The leaked object is a struct sched_domain_shared (32 bytes with >>> CONFIG_SCHED_CACHE) allocated in __sds_alloc(), inlined into >>> build_sched_domains() >>> >>> Is this a known issue? >> >> It is theoretically possible but there is a defensive WARN_ON() >> in topology code that you should have hit first. Do you see any >> other warning? >> >> If it is not too much trouble, could you add "sched_verbose" >> to your kernel cmdline (or do >> echo Y > /sys/kernel/debug/sched/verbose) and redo this cpuset >> that leaks the data and share the dmesg. It should give some clue >> what the topology looks like that causes this. > > Sure, does this one help: Unfortunately no! This doesn't give the span and capacity information so I cannot really tell what kind of topology actually triggers this. Ideally, you enable verbose logging, do the cpuset operation that leaks the object, and then save the demesg after which will have a complete topology for all CPUs there including their capacities and we can piece together a qemu environment to debug where it all went wrong. > > > > bash-5.1# cat /sys/kernel/debug/kmemleak > unreferenced object 0xffff0000c1180820 (size 32): > comm "swapper/0", pid 1, jiffies 4294667323 > hex dump (first 32 bytes): > 08 00 00 00 08 00 00 00 00 00 00 00 20 00 00 00 ............ ... > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > backtrace (crc f4478cb7): > kmemleak_alloc+0x44/0xd8 > __kmalloc_cache_node_noprof+0x344/0x5d8 > build_sched_domains+0x2f8/0x2110 > sched_init_domains+0xec/0x160 > sched_init_smp+0x48/0x108 > kernel_init_freeable+0x140/0x200 > kernel_init+0x30/0x170 > ret_from_fork+0x10/0x20 > > bash-5.1# cd /sys/kernel/debug/sched/domains/ > > bash-5.1# grep . -r * > cpu0/domain0/level:1 > cpu0/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu0/domain0/name:MC > cpu0/domain0/cache_nice_tries:1 > cpu0/domain0/imbalance_pct:117 > cpu0/domain0/busy_factor:16 > cpu0/domain0/max_newidle_lb_cost:2127 > cpu0/domain0/max_interval:16 > cpu0/domain0/min_interval:8 > cpu1/domain0/level:1 > cpu1/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu1/domain0/name:MC > cpu1/domain0/cache_nice_tries:1 > cpu1/domain0/imbalance_pct:117 > cpu1/domain0/busy_factor:16 > cpu1/domain0/max_newidle_lb_cost:18112 > cpu1/domain0/max_interval:16 > cpu1/domain0/min_interval:8 > cpu2/domain0/level:1 > cpu2/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu2/domain0/name:MC > cpu2/domain0/cache_nice_tries:1 > cpu2/domain0/imbalance_pct:117 > cpu2/domain0/busy_factor:16 > cpu2/domain0/max_newidle_lb_cost:3147 > cpu2/domain0/max_interval:16 > cpu2/domain0/min_interval:8 > cpu3/domain0/level:1 > cpu3/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu3/domain0/name:MC > cpu3/domain0/cache_nice_tries:1 > cpu3/domain0/imbalance_pct:117 > cpu3/domain0/busy_factor:16 > cpu3/domain0/max_newidle_lb_cost:16399 > cpu3/domain0/max_interval:16 > cpu3/domain0/min_interval:8 > cpu4/domain0/level:1 > cpu4/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu4/domain0/name:MC > cpu4/domain0/cache_nice_tries:1 > cpu4/domain0/imbalance_pct:117 > cpu4/domain0/busy_factor:16 > cpu4/domain0/max_newidle_lb_cost:27180 > cpu4/domain0/max_interval:16 > cpu4/domain0/min_interval:8 > cpu5/domain0/level:1 > cpu5/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu5/domain0/name:MC > cpu5/domain0/cache_nice_tries:1 > cpu5/domain0/imbalance_pct:117 > cpu5/domain0/busy_factor:16 > cpu5/domain0/max_newidle_lb_cost:18384 > cpu5/domain0/max_interval:16 > cpu5/domain0/min_interval:8 > cpu6/domain0/level:1 > cpu6/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu6/domain0/name:MC > cpu6/domain0/cache_nice_tries:1 > cpu6/domain0/imbalance_pct:117 > cpu6/domain0/busy_factor:16 > cpu6/domain0/max_newidle_lb_cost:16261 > cpu6/domain0/max_interval:16 > cpu6/domain0/min_interval:8 > cpu7/domain0/level:1 > cpu7/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING > cpu7/domain0/name:MC > cpu7/domain0/cache_nice_tries:1 > cpu7/domain0/imbalance_pct:117 > cpu7/domain0/busy_factor:16 > cpu7/domain0/max_newidle_lb_cost:7780 > cpu7/domain0/max_interval:16 > cpu7/domain0/min_interval:8 Is this the defualt topology before the cpuset? Because min_interval of 8 means there are 8 cPUs in the span and you have 8 CPUs in your machine so there seems to be a single root partition when you captured this. Also does your dmesg have any warnings from topology.c? The only way I can see this triggering is there is a single cpu SD_SHARE_LLC at top which claims all the sd->shared and then you get a SD_ASYM_CPUCAPACITY_FULL on top which too tries to claim a sd->shared but all are already gone. Afaict, all CPUs in your system are on single LLC and you can't SD_ASYM_CPUCAPACITY_FULL without having more than one capacity in the span so I'm at a loss currently as to how this reference gets dropped without being freed. -- Thanks and Regards, Prateek