Cache-aware scheduling does not work well with amd big/little cores

[Klaus Kusche <klaus.kusche@computerix.info>]

Hello,

I'm running linux on an AMD Ryzen HX 370.
I use Gentoo and compile a lot of things (including the kernel)
with Clang using full LTO, which results in a single LTO process
running for minutes on an otherwise idle processor.

Before Cache-aware scheduling was introduced,
AMD's pstate driver informed the scheduler about big and little cores,
and the scheduler always moved those LTO processes to a big/fast core.

Now, the cache-aware scheduling takes precedence over big/little
scheduling: All big cores form one L3 cache domain,
and all little cores form the other L3 cache domain.

When a process is started on a little core (and LTO is in most cases),
it will no longer be moved to a big core,
even if it runs for minutes and all big cores are idle.

In case of LTO this hurts twice and badly
(slows down LTO not just noticeably, but dramatically:
In total, any compilation with LTO-linking takes much longer
compared to earlier kernels without cache-aware scheduling):

* The little cores run slower (3.3 GHz instead of 5.1 GHz).

* LTO linking is very cache intensive,
and the L3 cache of the little cores is only half the size (8 MB)
of the L3 cache of the big cores (16 MB).


So if a big core is idling, and a process on a little core
is long-running, big/little scheduling should overrule
cache-aware scheduling and move that process to a big core,
even if this moves the process to another cache domain.

Greetings

-- 
Prof. Dr. Klaus Kusche
Privat: Söllmnitz 32 d, D-07554 Gera/Söllmnitz
036695/859909 klaus.kusche@computerix.info https://www.computerix.info
Dienstlich: DHGE Gera, Weg der Freundschaft 4, D-07546 Gera
klaus.kusche@dhge.de https://www.dhge.de