From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E69F23E92BB for ; Mon, 6 Jul 2026 14:50:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783349412; cv=none; b=q1x5KxRvE2Kp6uHAHuFvQe259RerHgVq+Wjnh0LDDzZISraHEnYYcUuv+I4VJtqHALM2HBrA1Aoy/0s/nki7+1b8qT/MhRaOdigOPThgbraWgK/bhK26jVouTIq5Ot4Q6m4KjDs/8TY0eCWoTmYTz9Qpc6HJiwPC3aoU8CFQlCM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783349412; c=relaxed/simple; bh=CgZwNfk4tGCLfaz5XkD5DiRLSHzHRBRwoY526CjZuIY=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=QoBb20lI11SRmTNQRoofMp5qMcSamHbDJ/7e/QHRQeotG/PGVvrbuMWo8JsRuRE/KAXCqLRKwZ3Prpqi/Smcj8Voahx/g6jsIIxskSVyv/9ZYpc7rZOEg3E4zBUq0il4wBdfD8kd+CeQZjIXck78zG54k33mdqkseunShqIrang= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=IoNgbMxm; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="IoNgbMxm" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0F6AD1BB2; Mon, 6 Jul 2026 07:50:05 -0700 (PDT) Received: from [10.1.27.96] (e127648.arm.com [10.1.27.96]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 697FA3F905; Mon, 6 Jul 2026 07:50:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1783349409; bh=CgZwNfk4tGCLfaz5XkD5DiRLSHzHRBRwoY526CjZuIY=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=IoNgbMxmK0VA5rBmnH9vG/Fvr4x3v/hZQr1gwo9L4DXuRkQSI6kBAFe4S3roqfjQv iH1rhZYEyNRmeoSBvG+yL9CCAgZlBoGyUxv/CVZCrKQjOFnbZBCLdP88EnjGPjKSDA eS2a2TN4d9W/cIc3KGkOYh0dWffCC0mkZRr60PuQ= Message-ID: Date: Mon, 6 Jul 2026 15:50:06 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFT][DONOTMERGE][PATCH 0/6] timers/migration: Prioritize lower capacity CPUs as migrators To: Frederic Weisbecker Cc: LKML , Sehee Jeong , Anna-Maria Behnsen , Thomas Gleixner , Peter Zijlstra References: <20260625164114.51454-1-frederic@kernel.org> <7bc624c7-141f-4b14-9422-5621e07c3fcf@arm.com> Content-Language: en-US From: Christian Loehle In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 7/3/26 13:53, Frederic Weisbecker wrote: > Le Fri, Jul 03, 2026 at 11:13:13AM +0100, Christian Loehle a écrit : >> On 6/25/26 17:41, Frederic Weisbecker wrote: >>> Hi, >>> >>> It has been reported that the timer migration tree wasn't too helpful >>> on asymetric systems, with timers migrating too often to big capacity >>> CPUs, waking them more frequently from deep idle state. >>> >>> Splitting the tree into per CPU capacity hierarchies doesn't seem to >>> improve the situation. >>> >>> So here is another proposal: gather CPUs belonging to the same capacity >>> under the same groups and finally join those capacity groups under a >>> common root. Exactly in the same way it is done for NUMA systems. >> >> This seems to work for my orion o6, the resulting hierarchy is: >> >> root lvl1 >> family=-1, want_low_migrator=1 >> ________________|___________|___________|____________ >> / / / / / >> mask 0x1 mask 0x2 mask 0x4 mask 0x8 mask 0x10 >> grp279 grp866 grp905 grp984 grp1024 >> CPUs 2,3,4,5 CPUs 8,9 CPUs 6,7 CPUs 10,11 CPUs 0,1 >> >>> >>> And then always pick up the lowest capacity non-idle CPU group as the >>> migrator with the hope that handling timers will keep lower capacity >>> CPUs busy enough to increase their chances to become global idle >>> migrators. >>> >>> Please give it a try but make sure you have fewer than 9 capacities >>> on your system. Handling more will require a bit more throughts. Let's >>> see first if this is the right way to go. >> >> Honestly I think buckets should be fine here? One for each 128-capacity >> step (or even 256), which would also make these awkward Grace capacities all >> fall into one, which makes a lot of sense IMO? > > It may be hard to find the right threshold. Also migrating from capX to capX-50 > can still make a visible difference. So I'd rather keep all capacities separate > (for now) and create more levels above in the hierarchy to group capacities if > necessary. It should only create one more level if nr_cap > 8 and two levels if > nr_cap > 64, but reaching that latter doesn't sound reasonable on the hardware, > right? (famous last words). So I'm only aware of systems with <=3 different uArch CPUs, there's a few with many more capacity classes, but these will be the same uArch with slightly different available frequencies, like in my example there's: Quad Big Cortex®-A720 Up to 2.6GHz Quad Medium Cortex®-A720 Up to 2.4GHz Quad Little Cortex®-A520 1.8GHz i.e. A720 and A520. I think as long as the capacity classes don't mix uArchs we should be fine... We're far off from nr_cap > 64 FWIW but technically this can change even for existing silicon e.g. if FW notices the different cores age differently it could start reporting different frequencies from one boot to the next. That being said I agree nr_cap < 64 is okay for now... I would've grouped them just because we do fundamentally want bigger capacity classes, right? > >> Thanks for giving this a shot, here are the results, I tested this against >> tmigr-off (which I've added ontop of series and capacity-unaware dd50a6e260ea "timers/migration: Revert per CPU capacity hierarchy")) >> In short: tmigr-off-600, capacity-unaware-600, series-600 >> >> >> idle-600s >> ---------- >> >> Overall idle states: >> >> +------------------------+---------+---------+---------+--------------+ >> | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| >> +------------------------+---------+---------+---------+--------------+ >> | tmigr-off-600 | 3771.0s | 650.9s | 2212.0s | 2862.9s | >> | capacity-unaware-600 | 2461.7s | 1245.3s | 2708.9s | 3954.3s | >> | series-600 | 2429.2s | 1260.7s | 2726.5s | 3987.2s | >> +------------------------+---------+---------+---------+--------------+ >> >> Delta vs tmigr-off-600: >> >> +------------------------+----------+---------+---------+--------------+ >> | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| >> +------------------------+----------+---------+---------+--------------+ >> | capacity-unaware-600 | -1309.2s | 594.4s | 497.0s | 1091.4s | >> | series-600 | -1341.7s | 609.8s | 514.5s | 1124.3s | >> +------------------------+----------+---------+---------+--------------+ > > Ok here it looks good. FWIW this should be the most important case. > >> >> LPI-2 by capacity group: >> >> +-------+----------------+------------------------+------------+ >> | Group | tmigr-off-600 | capacity-unaware-600 | series-600 | >> +-------+----------------+------------------------+------------+ >> | 279 | 805.3s | 1007.3s | 1003.5s | >> | 866 | 391.7s | 483.5s | 477.3s | >> | 905 | 411.6s | 466.1s | 501.8s | >> | 984 | 361.3s | 468.3s | 458.9s | >> | 1024 | 242.2s | 283.8s | 285.0s | >> +-------+----------------+------------------------+------------+ >> >> >> schbench-600s >> ------------- >> >> Overall idle states: >> >> +------------------------+--------+--------+--------+--------------+ >> | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| >> +------------------------+--------+--------+--------+--------------+ >> | tmigr-off-600 | 167.7s | 34.1s | 22.3s | 56.4s | >> | capacity-unaware-600 | 158.8s | 37.2s | 28.1s | 65.3s | >> | series-600 | 161.5s | 37.3s | 26.3s | 63.6s | >> +------------------------+--------+--------+--------+--------------+ > > But IIUC here it performs worse than before on LPI-2 ? > >> >> Delta vs tmigr-off-600: >> >> +------------------------+--------+-------+-------+--------------+ >> | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| >> +------------------------+--------+-------+-------+--------------+ >> | capacity-unaware-600 | -8.9s | 3.1s | 5.8s | 8.9s | >> | series-600 | -6.2s | 3.2s | 4.0s | 7.3s | >> +------------------------+--------+-------+-------+--------------+ > > And overall worse in fact? > >> >> LPI-2 by capacity group: >> >> +-------+----------------+------------------------+------------+ >> | Group | tmigr-off-600 | capacity-unaware-600 | series-600 | >> +-------+----------------+------------------------+------------+ >> | 279 | 8.9s | 10.6s | 10.3s | >> | 866 | 3.3s | 4.8s | 4.3s | >> | 905 | 3.8s | 4.6s | 4.3s | >> | 984 | 3.3s | 4.8s | 4.7s | >> | 1024 | 3.0s | 3.2s | 2.7s | >> +-------+----------------+------------------------+------------+ >> >> >> tbench-4clients-600s >> -------------------- >> >> Overall idle states: >> >> +------------------------+---------+---------+--------+--------------+ >> | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| >> +------------------------+---------+---------+--------+--------------+ >> | tmigr-off-600 | 3125.7s | 1395.7s | 425.9s | 1821.6s | >> | capacity-unaware-600 | 3009.4s | 1437.7s | 456.3s | 1894.0s | >> | series-600 | 2998.3s | 1436.3s | 453.6s | 1889.9s | >> +------------------------+---------+---------+--------+--------------+ > > Same here? > >> >> Delta vs tmigr-off-600: >> >> +------------------------+---------+--------+--------+--------------+ >> | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| >> +------------------------+---------+--------+--------+--------------+ >> | capacity-unaware-600 | -116.2s | 41.9s | 30.5s | 72.4s | >> | series-600 | -127.3s | 40.6s | 27.7s | 68.3s | >> +------------------------+---------+--------+--------+--------------+ >> >> LPI-2 by capacity group: >> >> +-------+----------------+------------------------+------------+ >> | Group | tmigr-off-600 | capacity-unaware-600 | series-600 | >> +-------+----------------+------------------------+------------+ >> | 279 | 162.9s | 179.3s | 180.5s | >> | 866 | 68.6s | 73.4s | 71.9s | >> | 905 | 67.7s | 69.5s | 71.8s | >> | 984 | 72.9s | 75.5s | 71.5s | >> | 1024 | 53.7s | 58.6s | 57.9s | >> +-------+----------------+------------------------+------------+ >> >> Seems the series does what it claims > > Does it really? In the sense that it seems to restore the benefit of tmigr in the first place, the rest would be interesting to hear from Sehee. I tried a little but found nothing to produce something that would explain such a big power regression. (Because the 'bad' 'little->big' migrations would have to absolutely dominate over the 'good' little->big (or good little->little and big->big) migrations). Your interpretation on schbench and tbench capacity-unaware-600 vs series-600 is correct though, I'll go look some more into why that is... It might just be the fragmentation of the bigs... (i.e. a migr grp984->grp1024 now disallowed but most likely it would still be 'good' and the fact we have many more big cores than little cores).