From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EB37D1DF72C for ; Mon, 30 Mar 2026 22:31:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774909869; cv=none; b=O2/UvsWqfDtCVhqbPk4FjWe70J7JnL92cvenGwfJEZKHVYCSMF8JL0SAoBeO0ePiGJns8wmhBC4rWrqsbOZtf46lwcnGmh2b++A9o45639aiS+9IzwAeQjW2maq3e5Iq8Qm6ht7E1GZxb59gWPByIrzWsC4vP6enu7xT1UAIWmU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774909869; c=relaxed/simple; bh=P+isKeXtWZY11yihQmlbrQLctx82EKnyFw4aT1EImeQ=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=BbFotxoWOfZ/DprdR0Dwk2xkC1khY54neZlEtCmgznn+dUKg+gNi1i6GqujsQZCRlHagVd0Lzy1Q7SQHa6yoYIWpFlYLuJLFd7e68KDOQoCqj8ZYgqAbRyqomY1dpqFJnvOV04baIFtTuJjm2aMGhyRlKO+j/hVykNP0RC/6DI0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=mmy6/w5t; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="mmy6/w5t" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CD02F2403; Mon, 30 Mar 2026 15:30:53 -0700 (PDT) Received: from [10.57.18.88] (unknown [10.57.18.88]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A6BD73F7D8; Mon, 30 Mar 2026 15:30:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1774909859; bh=P+isKeXtWZY11yihQmlbrQLctx82EKnyFw4aT1EImeQ=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=mmy6/w5ttvFQOWO6UjkygTN6HnHi/7P1GoqEhKviGd/4yEeUm78FOZVQdLCWE+UVj bsyjoqMXmT+MXYzvBFKt3FM2c05orMtzfcGeSS4Y7VrqJZrQOjqIUiuoNYgwz2qsIT z8JrcjMG57eHNYnO3txdPNJ2jJhLbQCSDjyyIhYY= Message-ID: Date: Tue, 31 Mar 2026 00:30:55 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/4] sched/fair: SMT-aware asymmetric CPU capacity To: Andrea Righi , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , linux-kernel@vger.kernel.org References: <20260326151211.1862600-1-arighi@nvidia.com> Content-Language: en-GB From: Dietmar Eggemann In-Reply-To: <20260326151211.1862600-1-arighi@nvidia.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Hi Andrea, On 26.03.26 16:02, Andrea Righi wrote: [...] > This patch set has been tested on the new NVIDIA Vera Rubin platform, where > SMT is enabled and the firmware exposes small frequency variations (+/-~5%) > as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set. > > Without these patches, performance can drop up to ~2x with CPU-intensive > workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not > account for busy SMT siblings. > > Alternative approaches have been evaluated, such as equalizing CPU > capacities, either by exposing uniform values via firmware (ACPI/CPPC) or > normalizing them in the kernel by grouping CPUs within a small capacity > window (+-5%) [1][2], or enabling asympacking [3]. > > However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better > results so far. Improving this policy also seems worthwhile in general, as > other platforms in the future may enable SMT with asymmetric CPU > topologies. I still wonder whether we really need select_idle_capacity() (plus the smt part) for asymmetric CPU capacity systems where the CPU capacity differences are < 5% of SCHED_CAPACITY_SCALE. The known example would be the NVIDIA Grace (!smt) server with its slightly different perf_caps.highest_perf values. We did run DCPerf Mediawiki on this thing with: (1) ASYM_CPUCAPACITY (default) (2) NO ASYM_CPUCAPACITY We also ran on a comparable ARM64 server (!smt) for comparison: (1) ASYM_CPUCAPACITY (2) NO ASYM_CPUCAPACITY (default) Both systems have 72 CPUs, run v6.8 and have a single MC sched domain with LLC spanning over all 72 CPUs. During the tests there were ~750 tasks among them the workload related: #hhvmworker 147 #mariadbd 204 #memcached 11 #nginx 8 #wrk 144 #ProxygenWorker 1 load_balance: not_idle 3x more on (2) idle 2x more on (2) newly_idle 2-10x more on (2) wakeup: move_affine 2-3x more on (1) ttwu_local 1.5-2 more on (2) We also instrumented all the bailout conditions in select_task_sibling() (sis())->select_idle_cpu() and select_idle_capacity() (sic()). In (1) almost all wakeups end up in select_idle_cpu() returning -1 due to the fact that 'sd->shared->nr_idle_scan' under SIS_UTIL is 0. So sis() in (1) almost always returns target (this_cpu or prev_cpu). sic() doesn't do this. What I haven't done is to try (1) with SIS_UTIL or (2) with NO_SIS_UTIL. I wonder whether this is the underlying reason for the benefit of (1) over (2) we see here with smt now? So IMHO before adding smt support to (1) for these small CPPC based CPU capacity differences we should make sure that the same can't be achieved by disabling SIS_UTIL or to soften it a bit. So does (2) with NO_SIS_UTIL performs worse than (1) with your smt related add-ons in sic()?