From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 94EAD358388 for ; Fri, 3 Jul 2026 10:13:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783073598; cv=none; b=SEfwIju9et+RvOlOQ+k16lElPxK+zxeCN68qHU3U0d1THgoL+JmWuZgfBYOwzd9M82IHiYK+WOkw7EZAKPXMShc3ywbzUNR6DetsymwnM7ndGS6EXnenRB9urQwCpz41eF9qgX0cyK+HqsbiE6WitvgAqAXXD02dD4hGDGvcvz0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783073598; c=relaxed/simple; bh=A+3nVQ6nDEUpdp5BC6foltfA9K8GCycY4CRNwT03E1A=; h=Message-ID:Date:MIME-Version:From:Subject:To:Cc:References: In-Reply-To:Content-Type; b=EIOzGqbM8gQFA+zZuOT+6aIBEbK7uiiGh3YiN3/tnsrAezVLMwD+1/gI36akEd8gqGpI/rEdS3fDE3ea/Qphf6fOzflmSDcP9BJoRoaW7kTig2gdHFtj9fMiV2T/8ubvgWSSpmOaNdRtXy8J/AQlwgICytR0YKbc/oy8xsLoL+Q= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=Xlx3yoWR; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="Xlx3yoWR" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9E8E31F91; Fri, 3 Jul 2026 03:13:11 -0700 (PDT) Received: from [10.57.31.140] (unknown [10.57.31.140]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 05F0A3F905; Fri, 3 Jul 2026 03:13:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1783073596; bh=A+3nVQ6nDEUpdp5BC6foltfA9K8GCycY4CRNwT03E1A=; h=Date:From:Subject:To:Cc:References:In-Reply-To:From; b=Xlx3yoWRvMaHGWnvumU5ZNH5oF1bLdpzyJSBsj2GHrhPPdlhwxs9gLz1X9rvpK6xk kfTpORU9w7//mYhDKUdeFGQLP6MrIua9HHFzJmBJBof3RGfBRh74scRqsgCMmQwJyN e10VJ3WmiE2MJmgzP69mC8tW0yIKCYFqx9judIFY= Message-ID: <7bc624c7-141f-4b14-9422-5621e07c3fcf@arm.com> Date: Fri, 3 Jul 2026 11:13:13 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Christian Loehle Subject: Re: [RFT][DONOTMERGE][PATCH 0/6] timers/migration: Prioritize lower capacity CPUs as migrators To: Frederic Weisbecker Cc: LKML , Sehee Jeong , Anna-Maria Behnsen , Thomas Gleixner , Peter Zijlstra References: <20260625164114.51454-1-frederic@kernel.org> Content-Language: en-US In-Reply-To: <20260625164114.51454-1-frederic@kernel.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 6/25/26 17:41, Frederic Weisbecker wrote: > Hi, > > It has been reported that the timer migration tree wasn't too helpful > on asymetric systems, with timers migrating too often to big capacity > CPUs, waking them more frequently from deep idle state. > > Splitting the tree into per CPU capacity hierarchies doesn't seem to > improve the situation. > > So here is another proposal: gather CPUs belonging to the same capacity > under the same groups and finally join those capacity groups under a > common root. Exactly in the same way it is done for NUMA systems. This seems to work for my orion o6, the resulting hierarchy is: root lvl1 family=-1, want_low_migrator=1 ________________|___________|___________|____________ / / / / / mask 0x1 mask 0x2 mask 0x4 mask 0x8 mask 0x10 grp279 grp866 grp905 grp984 grp1024 CPUs 2,3,4,5 CPUs 8,9 CPUs 6,7 CPUs 10,11 CPUs 0,1 > > And then always pick up the lowest capacity non-idle CPU group as the > migrator with the hope that handling timers will keep lower capacity > CPUs busy enough to increase their chances to become global idle > migrators. > > Please give it a try but make sure you have fewer than 9 capacities > on your system. Handling more will require a bit more throughts. Let's > see first if this is the right way to go. Honestly I think buckets should be fine here? One for each 128-capacity step (or even 256), which would also make these awkward Grace capacities all fall into one, which makes a lot of sense IMO? > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git > timers/migration-capacity > > HEAD: 28f02a6747ac4343f3596aaa3b749dea95577478 > Thanks, > Frederic > --- > > Frederic Weisbecker (6): > timers/migration: Revert per CPU capacity hierarchy > timers/migration: Defer initialization after capacity topology is setup > sched/topology: Account asym capacities number > timers/migration: Group CPUs per capacity > timers/migration: Prefer lower capacity groups as migrators > scripts/timer_migration_tree.py: Dump mask of each group > > include/linux/sched/topology.h | 3 + > include/trace/events/timer_migration.h | 48 ++-- > kernel/sched/topology.c | 44 ++++ > kernel/time/timer_migration.c | 393 +++++++++++++++------------------ > kernel/time/timer_migration.h | 33 +-- > scripts/timer_migration_tree.py | 65 +++--- > 6 files changed, 288 insertions(+), 298 deletions(-) Thanks for giving this a shot, here are the results, I tested this against tmigr-off (which I've added ontop of series and capacity-unaware dd50a6e260ea "timers/migration: Revert per CPU capacity hierarchy")) In short: tmigr-off-600, capacity-unaware-600, series-600 idle-600s ---------- Overall idle states: +------------------------+---------+---------+---------+--------------+ | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| +------------------------+---------+---------+---------+--------------+ | tmigr-off-600 | 3771.0s | 650.9s | 2212.0s | 2862.9s | | capacity-unaware-600 | 2461.7s | 1245.3s | 2708.9s | 3954.3s | | series-600 | 2429.2s | 1260.7s | 2726.5s | 3987.2s | +------------------------+---------+---------+---------+--------------+ Delta vs tmigr-off-600: +------------------------+----------+---------+---------+--------------+ | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| +------------------------+----------+---------+---------+--------------+ | capacity-unaware-600 | -1309.2s | 594.4s | 497.0s | 1091.4s | | series-600 | -1341.7s | 609.8s | 514.5s | 1124.3s | +------------------------+----------+---------+---------+--------------+ LPI-2 by capacity group: +-------+----------------+------------------------+------------+ | Group | tmigr-off-600 | capacity-unaware-600 | series-600 | +-------+----------------+------------------------+------------+ | 279 | 805.3s | 1007.3s | 1003.5s | | 866 | 391.7s | 483.5s | 477.3s | | 905 | 411.6s | 466.1s | 501.8s | | 984 | 361.3s | 468.3s | 458.9s | | 1024 | 242.2s | 283.8s | 285.0s | +-------+----------------+------------------------+------------+ schbench-600s ------------- Overall idle states: +------------------------+--------+--------+--------+--------------+ | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| +------------------------+--------+--------+--------+--------------+ | tmigr-off-600 | 167.7s | 34.1s | 22.3s | 56.4s | | capacity-unaware-600 | 158.8s | 37.2s | 28.1s | 65.3s | | series-600 | 161.5s | 37.3s | 26.3s | 63.6s | +------------------------+--------+--------+--------+--------------+ Delta vs tmigr-off-600: +------------------------+--------+-------+-------+--------------+ | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| +------------------------+--------+-------+-------+--------------+ | capacity-unaware-600 | -8.9s | 3.1s | 5.8s | 8.9s | | series-600 | -6.2s | 3.2s | 4.0s | 7.3s | +------------------------+--------+-------+-------+--------------+ LPI-2 by capacity group: +-------+----------------+------------------------+------------+ | Group | tmigr-off-600 | capacity-unaware-600 | series-600 | +-------+----------------+------------------------+------------+ | 279 | 8.9s | 10.6s | 10.3s | | 866 | 3.3s | 4.8s | 4.3s | | 905 | 3.8s | 4.6s | 4.3s | | 984 | 3.3s | 4.8s | 4.7s | | 1024 | 3.0s | 3.2s | 2.7s | +-------+----------------+------------------------+------------+ tbench-4clients-600s -------------------- Overall idle states: +------------------------+---------+---------+--------+--------------+ | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| +------------------------+---------+---------+--------+--------------+ | tmigr-off-600 | 3125.7s | 1395.7s | 425.9s | 1821.6s | | capacity-unaware-600 | 3009.4s | 1437.7s | 456.3s | 1894.0s | | series-600 | 2998.3s | 1436.3s | 453.6s | 1889.9s | +------------------------+---------+---------+--------+--------------+ Delta vs tmigr-off-600: +------------------------+---------+--------+--------+--------------+ | Variant | LPI-0 | LPI-1 | LPI-2 | LPI-1 + LPI-2| +------------------------+---------+--------+--------+--------------+ | capacity-unaware-600 | -116.2s | 41.9s | 30.5s | 72.4s | | series-600 | -127.3s | 40.6s | 27.7s | 68.3s | +------------------------+---------+--------+--------+--------------+ LPI-2 by capacity group: +-------+----------------+------------------------+------------+ | Group | tmigr-off-600 | capacity-unaware-600 | series-600 | +-------+----------------+------------------------+------------+ | 279 | 162.9s | 179.3s | 180.5s | | 866 | 68.6s | 73.4s | 71.9s | | 905 | 67.7s | 69.5s | 71.8s | | 984 | 72.9s | 75.5s | 71.5s | | 1024 | 53.7s | 58.6s | 57.9s | +-------+----------------+------------------------+------------+ Seems the series does what it claims, I'd love to also hear about Sehee's results to the original regression (as honestly, capacity-unaware doesn't seem so bad for me in the first case, so maybe my testing has a blind spot here?)