Re: 5.11-rc4+git: Shortest NUMA path spans too many nodes

[Dietmar Eggemann <dietmar.eggemann@arm.com>]

On 21/01/2021 22:17, Song Bao Hua (Barry Song) wrote:
> 
> 
>> -----Original Message-----
>> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com]
>> Sent: Friday, January 22, 2021 7:54 AM
>> To: Valentin Schneider <valentin.schneider@arm.com>; Meelis Roos
>> <mroos@linux.ee>; LKML <linux-kernel@vger.kernel.org>
>> Cc: Peter Zijlstra <peterz@infradead.org>; Vincent Guittot
>> <vincent.guittot@linaro.org>; Song Bao Hua (Barry Song)
>> <song.bao.hua@hisilicon.com>; Mel Gorman <mgorman@suse.de>
>> Subject: Re: 5.11-rc4+git: Shortest NUMA path spans too many nodes
>>
>> On 21/01/2021 19:21, Valentin Schneider wrote:
>>> On 21/01/21 19:39, Meelis Roos wrote:

[...]

>> # cat /sys/devices/system/node/node*/distance
>> 10 12 12 14 14 14 14 16
>> 12 10 14 12 14 14 12 14
>> 12 14 10 14 12 12 14 14
>> 14 12 14 10 12 12 14 14
>> 14 14 12 12 10 14 12 14
>> 14 14 12 12 14 10 14 12
>> 14 12 14 14 12 14 10 12
>> 16 14 14 14 14 12 12 10
>>
>> The '16' seems to be the culprit. How does such a topo look like?

Maybe like this:

      _________
      |       |
    .-6   0   4-.
    |  \ / \ /  |
    |   1   2   |
    |   \    \  |
    --7  3----5 |
      |  |____|_|
      |_______|

> 
> Once we get a topology like this:
> 
> 
>          +------+         +------+        +-------+       +------+
>          | node |         |node  |        | node  |       |node  |
>          |      +---------+      +--------+       +-------+      |
>          +------+         +------+        +-------+       +------+
> 
> We can reproduce this issue. 
> For example, every cpu with the below numa_distance can have 
> "groups don't span domain->span":
> node   0   1   2   3
>   0:  10  12  20  22
>   1:  12  10  22  24
>   2:  20  22  10  12
>   3:  22  24  12  10
                             2     20     2
So this should look like: 1 --- 0 ---- 2 --- 3