Perf regression from scheduler load_balance rework in 5.5?

[David Chen <david.chen@nutanix.com>]

Hi,

I'm working on upgrading our kernel from 4.14 to 5.10
However, I'm seeing performance regression when doing rand read from windows client through smbd
with a well cached file.

One thing I noticed is that on the new kernel, the smbd thread doing socket I/O tends to stay on
the same cpu core as the net_rx softirq, where as in the old kernel it tends to be moved around
more randomly. And when they are on the same cpu, it tends to saturate the cpu more and causes
performance to drop.

For example, here's the duration (ns) the thread spend on each cpu I captured using bpftrace
On 4.14:
@cputime[7]: 20741458382
@cputime[0]: 25219285005
@cputime[6]: 30892418441
@cputime[5]: 31032404613
@cputime[3]: 33511324691
@cputime[1]: 35564174562
@cputime[4]: 39313421965
@cputime[2]: 55779811909 (net_rx cpu)

On 5.10:
@cputime[3]: 2150554823
@cputime[5]: 3294276626
@cputime[7]: 4277890448
@cputime[4]: 5094586003
@cputime[1]: 6058168291
@cputime[0]: 14688093441
@cputime[6]: 17578229533
@cputime[2]: 223473400411 (net_rx cpu)

I also tried setting the cpu affinity of the smbd thread away from the net_rx cpu and indeed that
seems to bring the perf on par with old kernel.

I noticed that there's scheduler load_balance rework in 5.5, so I did the test on 5.4 and 5.5 and
it did show the behavior changed between 5.4 and 5.5.

Anyone know how to work around this?

Thanks,
David