From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90F623644C6 for ; Fri, 27 Mar 2026 16:31:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774629090; cv=none; b=alAetiD1bZ++uitIDh4GAoejdRL+HlT1N4raGY6FsUkcG+9HNr6Kjt5RehRSbOaKSYpH9w0CQuLNXUuIAXgju5zz/zt2YEbH2HX7YUjLhgj6QuP522x1El18R3yRiNyFmUQfFD2DbhB+gHz49c3JbDu8rNTIDKA0HEVxLdQ2KMQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774629090; c=relaxed/simple; bh=eHHE+nWBaVz4UZxomVmPMG4om+Om67TXU9L6xgu+n14=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=RguLtxp7Nx5JJxWqL7MYgv3E/ozGpKEWhcRLLT+RbDDBa8Z6YM9PtREEQY2FykjFZXyCTPvgRjT+ry67tD/DnYnfibI9RR70yXA6V3aKa2M7qPhfsGE/VyNHahxTpH5lu0QHTCIn4OJWBRKYRUnYNdauLKOD+txSac7t8p9QoOk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=rM6uRWgx; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="rM6uRWgx" Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 62RAmBeO2477909; Fri, 27 Mar 2026 16:31:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=9YxLaf KEJk+aDLnnc/B9k4RTqckOxjq1SbTS6tMiWfo=; b=rM6uRWgx1aHimd1FPtxXBf QugOqXBopuhiB7k2abebemkcGJ3Uj3wJ1atiPxp5pE4TJCiLk8v3trJc+xt+joCf 772/y7IwPL1HnPY2HOsvwObay1hF7K62nTLSaN7gLzJAvG9OGI4oKojPI8vByadF P8+/A5/T8bVnXDgAtSKoLXvb8LUHN6kZZ0KhSVCcDrHJYZuIpkTz7HjZw2rN2Czs HhPRAuyT5tfb+9x0EddZSYPC/0HPX0o7JLgvjqYcHo842BRNl8DG9YX8m9dMeezn T8bUQ4w2HBw7SkUKGNVKDHFTNro5q6ZTUtyx/332KqqzdR2ZqVViduGwHgzESDAQ == Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4d1kun20fg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 27 Mar 2026 16:31:09 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 62RGEVlB012185; Fri, 27 Mar 2026 16:31:08 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4d27vkg2j8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 27 Mar 2026 16:31:08 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 62RGV7Xk15466912 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Mar 2026 16:31:07 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0AD592004B; Fri, 27 Mar 2026 16:31:07 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4314920040; Fri, 27 Mar 2026 16:31:04 +0000 (GMT) Received: from [9.39.30.147] (unknown [9.39.30.147]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 27 Mar 2026 16:31:04 +0000 (GMT) Message-ID: Date: Fri, 27 Mar 2026 22:01:03 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/4] sched/fair: SMT-aware asymmetric CPU capacity To: Andrea Righi , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , linux-kernel@vger.kernel.org References: <20260326151211.1862600-1-arighi@nvidia.com> From: Shrikanth Hegde Content-Language: en-US In-Reply-To: <20260326151211.1862600-1-arighi@nvidia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Proofpoint-GUID: 5U_XxIrrp1fwOnoIz3x8X5fxHIObpzM9 X-Proofpoint-ORIG-GUID: nxbBZ2nj1Onsg_9gortoWlDamUBkD-RG X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwMzI3MDExMyBTYWx0ZWRfX1Cqfyi7AsK/0 t8D+7RY7JhMnV7ctGgkQtd2yHX4CT2yLlKFRY2LXTnMrcAeCoD0xNW4jziXVDOv4+qMhrNlkKe7 euyQ+yAQCfE/Ec5hLuF3t8vcb3IRmQjCwY4Ci3+Vz2y3fmlfS5/9NPItokzQpSpzbFhAZ49/FOc bP4KLlhfvZqGxh/vQd6RXA288Pu7y4OtOwvW3VPyT223d022E3CzEUXExxqJqjulcuKQMFIpOz6 GSpdAJ+QsVTdVnEHDxgXuJjeF0uJ9Wg0XGEbbPKgWvNAodXcMEqPs0glGzInPgOWEpl3KvGoWQ4 TGkqns5RjKUh9YvR2BQgafAC5V8LuM2pZ68eih0pEpePQNyxW4AxSINOHWPdLucAiRaU30Qqe5M uc6JEgQHDmJvoMqEiOKQjkd5JmzespfVbwrFebsN/3u2oQ+xjepk4/bT4tlwBj8loJ06HboTHqV TrrbeREGDz7b3rt5lwQ== X-Authority-Analysis: v=2.4 cv=KbXfcAYD c=1 sm=1 tr=0 ts=69c6b0ce cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=IkcTkHD0fZMA:10 a=Yq5XynenixoA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=RzCfie-kr_QcCd8fBx8p:22 a=VwQbUJbxAAAA:8 a=Ikd4Dj_1AAAA:8 a=7CQSdrXTAAAA:8 a=-r8SXpYiyxsPhI3ECzgA:9 a=QEXdDO2ut3YA:10 a=a-qgeE7W1pNrGK8U0ZQC:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-03-27_01,2026-03-26_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 impostorscore=0 malwarescore=0 adultscore=0 clxscore=1015 priorityscore=1501 bulkscore=0 lowpriorityscore=0 phishscore=0 spamscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2603050001 definitions=main-2603270113 Hi Andrea. On 3/26/26 8:32 PM, Andrea Righi wrote: > This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by > introducing SMT awareness. > > = Problem = > > Nominal per-logical-CPU capacity can overstate usable compute when an SMT > sibling is busy, because the physical core doesn't deliver its full nominal > capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs > that are not actually good destinations. > How does energy model define the opp for SMT? SMT systems have multiple of different functional blocks, a few ALU(arithmetic), LSU(load store unit) etc. If same/similar workload runs on sibling, it would affect the performance, but sibling is using different functional blocks, then it would not. So underlying actual CPU Capacity of each thread depends on what each sibling is running. I don't understand how does the firmware/energy models define this. > = Proposed Solution = > > This patch set aligns those paths with a simple rule already used > elsewhere: when SMT is active, prefer fully idle cores and avoid treating > partially idle SMT siblings as full-capacity targets where that would > mislead load balance. > > Patch set summary: > > - [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection > > Prefer fully-idle SMT cores in asym-capacity idle selection. In the > wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so > idle selection can prefer CPUs on fully idle cores, with a safe fallback. > > - [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity > > Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. > Provided for consistency with PATCH 1/4. > > - [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems > > Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for > consistency with PATCH 1/4. I've also tested with/without > /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't > noticed any regression. > > - [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer > > When choosing the housekeeping CPU that runs the idle load balancer, > prefer an idle CPU on a fully idle core so migrated work lands where > effective capacity is available. > > The change is still consistent with the same "avoid CPUs with busy > sibling" logic and it shows some benefits on Vera, but could have > negative impact on other systems, I'm including it for completeness > (feedback is appreciated). > > This patch set has been tested on the new NVIDIA Vera Rubin platform, where > SMT is enabled and the firmware exposes small frequency variations (+/-~5%) > as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set. > I assume the CPU_CAPACITY values fixed? first sibling has max, while other has less? > Without these patches, performance can drop up to ~2x with CPU-intensive > workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not > account for busy SMT siblings. > How is the performance measured here? Which benchmark? By any chance you are running number_running_task <= (nr_cpus / smt_threads_per_core), so it is all fitting nicely? If you increase those numbers, how does the performance numbers compare? Also, whats the system is like? SMT level? > Alternative approaches have been evaluated, such as equalizing CPU > capacities, either by exposing uniform values via firmware (ACPI/CPPC) or > normalizing them in the kernel by grouping CPUs within a small capacity > window (+-5%) [1][2], or enabling asympacking [3]. > > However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better > results so far. Improving this policy also seems worthwhile in general, as > other platforms in the future may enable SMT with asymmetric CPU > topologies. > > [1] https://lore.kernel.org/lkml/20260324005509.1134981-1-arighi@nvidia.com > [2] https://lore.kernel.org/lkml/20260318092214.130908-1-arighi@nvidia.com > [3] https://lore.kernel.org/all/20260325181314.3875909-1-christian.loehle@arm.com/ > > Andrea Righi (4): > sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection > sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity > sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems > sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer > > kernel/sched/fair.c | 163 +++++++++++++++++++++++++++++++++++++++++++----- > kernel/sched/topology.c | 9 --- > 2 files changed, 147 insertions(+), 25 deletions(-)