[PATCH RFC 0/2] sched: Introduce idle SMT priority for asymmetric capacity systems

[Mete Durlu <meted@linux.ibm.com>]

Summary
===========================================================================
On systems with asymmetric CPU capacities, the scheduler prefers fully idle
cores over idle SMT siblings of busy cores. This works generally well but
virtualized platforms where low capacity cores should be avoided are not
considered. Introduce a new config option and arch hook to prioritize
SMT utilization.

Background
===========================================================================
The scheduler has been moving toward better utilization of fully idle cores
and avoiding SMT penalties. This trend (notably after commit 25a32e400a14
("sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection")
broke the behavior s390 is relying on to concentrate workloads on its
high-capacity cores. On s390 CPU capacities represent the CPUs runtime
entitlement assigned by the hypervisor. The idle SMT siblings of busy
high-capacity cores start to perform better than fully idle low-capacity
cores as the whole machine(containing the logical partitions) starts
approaching to a {fully,over}loaded state. Grouping load on high-capacity
cores keeps shared low-capacity cores(which are shared more aggressively)
idle longer, reducing noise to neighboring partitions and improving
overall performance.

Approach
===========================================================================
This series introduces SCHED_IDLE_SMT_PRIO config option and the
sched_idle_smt_prio static branch, allowing architectures to treat idle
SMT siblings of busy cores as equal candidates during asymmetric capacity
load balancing.
Static branch checks are placed at paths considering fully idle cores
over idle SMT threads in presence of asymmetric CPU capacities within
scheduling groups. Inserted checks mostly override hints for idle core
selection or cause early exits favoring idle SMT siblings.
One more static branch check is added to new task path in order to favor
high capacity SMT siblings during initial task placement, therefore the
tasks are immediately placed in high capacity SMT siblings instead of
considering most idle but low capacity cores.

Architectures opt in by selecting ARCH_SUPPORTS_SCHED_IDLE_SMT_PRIO and
implementing arch_needs_idle_smt_prio(), which is evaluated during each
asym_cpu_capacity_scan() to track the state as runtime topology changes.

The second patch enables the feature for s390 when running on
hardware-backed topology in an LPAR with vertical polarization active
and system is approaching to a target state.
Otherwise the branch stays disabled and the scheduler falls back to the
standard idle-core preference.

No functional change on architectures that do not select
ARCH_SUPPORTS_SCHED_IDLE_SMT_PRIO.

Performance Results
===========================================================================
Since this change effects the {fully,over}loaded state it is difficult
to test at the moment. Therefore there are no concrete numbers or
metrics for now for s390.
For architectures which do not opt in to this feature, no performance
change is expected.

Considerations and Open Questions
===========================================================================
The main goal for this series is adding a simple mechanism for
architectures to switch between idle core and idle SMT priority while
keeping the introduced footprint as small as possible. But there are
some ideas and questions to consider;

1. Should arch_needs_idle_smt_prio() hook be removed?
   The architecture hook is there to allow for any sort of logic to
   dynamically decide when to flip the mechanism, but it can also be
   removed if everyone agrees that this behaviour is not something that
   should be dynamically flipped. It can be simply tied to detection of
   asymmetric capacities and selection of Kconfig option.

2. Is there a simpler way to implement this mechanism?
   The proposed approach is chosen as the other features effecting the
   scheduler's behavior, use the same method. If there is a more
   efficient way to implement the same mechanism I'd be glad to use
   that instead.

3. There are no performance measurements *yet*.
   On s390 the ideal conditions for this mechanism to be beneficial
   usually surface when the whole system is fully loaded and resource
   sharing between the logical partitions starts to get expensive.
   Creating such an environment requires time, therefore no benchmark
   results are available yet.

---
base-commit: 587858367581b9c55c3690f4e63382ad622719d4

Mete Durlu (2):
      kernel/sched: Introduce idle SMT priority
      s390/topology: Enable SCHED_IDLE_SMT_PRIO

 arch/Kconfig                     | 14 +++++++++
 arch/s390/Kconfig                |  1 +
 arch/s390/include/asm/topology.h |  5 +++
 arch/s390/kernel/topology.c      |  9 ++++++
 include/linux/sched/topology.h   |  9 ++++++
 kernel/sched/fair.c              | 67 +++++++++++++++++++++++++++++-----------
 kernel/sched/sched.h             | 10 ++++++
 kernel/sched/topology.c          | 17 ++++++++++
 8 files changed, 114 insertions(+), 18 deletions(-)