Re: [RFT][DONOTMERGE][PATCH 0/6] timers/migration: Prioritize lower capacity CPUs as migrators

[Frederic Weisbecker <frederic@kernel.org>]

Le Fri, Jul 03, 2026 at 11:13:13AM +0100, Christian Loehle a écrit :
> On 6/25/26 17:41, Frederic Weisbecker wrote:
> > Hi,
> > 
> > It has been reported that the timer migration tree wasn't too helpful
> > on asymetric systems, with timers migrating too often to big capacity
> > CPUs, waking them more frequently from deep idle state.
> > 
> > Splitting the tree into per CPU capacity hierarchies doesn't seem to
> > improve the situation.
> > 
> > So here is another proposal: gather CPUs belonging to the same capacity
> > under the same groups and finally join those capacity groups under a
> > common root. Exactly in the same way it is done for NUMA systems.
> 
> This seems to work for my orion o6, the resulting hierarchy is:
> 
>                            root lvl1
>                       family=-1, want_low_migrator=1
>           ________________|___________|___________|____________
>          /                /           /           /            /
>     mask 0x1         mask 0x2    mask 0x4    mask 0x8    mask 0x10
>     grp279           grp866      grp905      grp984      grp1024
>     CPUs 2,3,4,5     CPUs 8,9    CPUs 6,7    CPUs 10,11  CPUs 0,1
> 
> > 
> > And then always pick up the lowest capacity non-idle CPU group as the
> > migrator with the hope that handling timers will keep lower capacity
> > CPUs busy enough to increase their chances to become global idle
> > migrators.
> > 
> > Please give it a try but make sure you have fewer than 9 capacities
> > on your system. Handling more will require a bit more throughts. Let's
> > see first if this is the right way to go.
> 
> Honestly I think buckets should be fine here? One for each 128-capacity
> step (or even 256), which would also make these awkward Grace capacities all
> fall into one, which makes a lot of sense IMO?

It may be hard to find the right threshold. Also migrating from capX to capX-50
can still make a visible difference. So I'd rather keep all capacities separate
(for now) and create more levels above in the hierarchy to group capacities if
necessary. It should only create one more level if nr_cap > 8 and two levels if
nr_cap > 64, but reaching that latter doesn't sound reasonable on the hardware,
right? (famous last words).

> Thanks for giving this a shot, here are the results, I tested this against
> tmigr-off (which I've added ontop of series and capacity-unaware dd50a6e260ea "timers/migration: Revert per CPU capacity hierarchy"))
> In short: tmigr-off-600, capacity-unaware-600, series-600
> 
> 
> idle-600s
> ----------
> 
> Overall idle states:
> 
> +------------------------+---------+---------+---------+--------------+
> | Variant                | LPI-0   | LPI-1   | LPI-2   | LPI-1 + LPI-2|
> +------------------------+---------+---------+---------+--------------+
> | tmigr-off-600          | 3771.0s | 650.9s  | 2212.0s | 2862.9s      |
> | capacity-unaware-600   | 2461.7s | 1245.3s | 2708.9s | 3954.3s      |
> | series-600             | 2429.2s | 1260.7s | 2726.5s | 3987.2s      |
> +------------------------+---------+---------+---------+--------------+
> 
> Delta vs tmigr-off-600:
> 
> +------------------------+----------+---------+---------+--------------+
> | Variant                | LPI-0    | LPI-1   | LPI-2   | LPI-1 + LPI-2|
> +------------------------+----------+---------+---------+--------------+
> | capacity-unaware-600   | -1309.2s | 594.4s  | 497.0s  | 1091.4s      |
> | series-600             | -1341.7s | 609.8s  | 514.5s  | 1124.3s      |
> +------------------------+----------+---------+---------+--------------+

Ok here it looks good.

> 
> LPI-2 by capacity group:
> 
> +-------+----------------+------------------------+------------+
> | Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
> +-------+----------------+------------------------+------------+
> | 279   | 805.3s         | 1007.3s                | 1003.5s    |
> | 866   | 391.7s         | 483.5s                 | 477.3s     |
> | 905   | 411.6s         | 466.1s                 | 501.8s     |
> | 984   | 361.3s         | 468.3s                 | 458.9s     |
> | 1024  | 242.2s         | 283.8s                 | 285.0s     |
> +-------+----------------+------------------------+------------+
> 
> 
> schbench-600s
> -------------
> 
> Overall idle states:
> 
> +------------------------+--------+--------+--------+--------------+
> | Variant                | LPI-0  | LPI-1  | LPI-2  | LPI-1 + LPI-2|
> +------------------------+--------+--------+--------+--------------+
> | tmigr-off-600          | 167.7s | 34.1s  | 22.3s  | 56.4s        |
> | capacity-unaware-600   | 158.8s | 37.2s  | 28.1s  | 65.3s        |
> | series-600             | 161.5s | 37.3s  | 26.3s  | 63.6s        |
> +------------------------+--------+--------+--------+--------------+

But IIUC here it performs worse than before on LPI-2 ?

> 
> Delta vs tmigr-off-600:
> 
> +------------------------+--------+-------+-------+--------------+
> | Variant                | LPI-0  | LPI-1 | LPI-2 | LPI-1 + LPI-2|
> +------------------------+--------+-------+-------+--------------+
> | capacity-unaware-600   | -8.9s  | 3.1s  | 5.8s  | 8.9s         |
> | series-600             | -6.2s  | 3.2s  | 4.0s  | 7.3s         |
> +------------------------+--------+-------+-------+--------------+

And overall worse in fact?

> 
> LPI-2 by capacity group:
> 
> +-------+----------------+------------------------+------------+
> | Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
> +-------+----------------+------------------------+------------+
> | 279   | 8.9s           | 10.6s                  | 10.3s      |
> | 866   | 3.3s           | 4.8s                   | 4.3s       |
> | 905   | 3.8s           | 4.6s                   | 4.3s       |
> | 984   | 3.3s           | 4.8s                   | 4.7s       |
> | 1024  | 3.0s           | 3.2s                   | 2.7s       |
> +-------+----------------+------------------------+------------+
> 
> 
> tbench-4clients-600s
> --------------------
> 
> Overall idle states:
> 
> +------------------------+---------+---------+--------+--------------+
> | Variant                | LPI-0   | LPI-1   | LPI-2  | LPI-1 + LPI-2|
> +------------------------+---------+---------+--------+--------------+
> | tmigr-off-600          | 3125.7s | 1395.7s | 425.9s | 1821.6s      |
> | capacity-unaware-600   | 3009.4s | 1437.7s | 456.3s | 1894.0s      |
> | series-600             | 2998.3s | 1436.3s | 453.6s | 1889.9s      |
> +------------------------+---------+---------+--------+--------------+

Same here?

> 
> Delta vs tmigr-off-600:
> 
> +------------------------+---------+--------+--------+--------------+
> | Variant                | LPI-0   | LPI-1  | LPI-2  | LPI-1 + LPI-2|
> +------------------------+---------+--------+--------+--------------+
> | capacity-unaware-600   | -116.2s | 41.9s  | 30.5s  | 72.4s        |
> | series-600             | -127.3s | 40.6s  | 27.7s  | 68.3s        |
> +------------------------+---------+--------+--------+--------------+
> 
> LPI-2 by capacity group:
> 
> +-------+----------------+------------------------+------------+
> | Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
> +-------+----------------+------------------------+------------+
> | 279   | 162.9s         | 179.3s                 | 180.5s     |
> | 866   | 68.6s          | 73.4s                  | 71.9s      |
> | 905   | 67.7s          | 69.5s                  | 71.8s      |
> | 984   | 72.9s          | 75.5s                  | 71.5s      |
> | 1024  | 53.7s          | 58.6s                  | 57.9s      |
> +-------+----------------+------------------------+------------+
> 
> Seems the series does what it claims

Does it really?

> I'd love to also hear about Sehee's
> results to the original regression (as honestly, capacity-unaware doesn't
> seem so bad for me in the first case, so maybe my testing has a blind spot
> here?)

Me too yes.

Thanks.

-- 
Frederic Weisbecker
SUSE Labs