Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[tmenninger@purestorage.com]

> The lock profile did not move. free_slowpath fell by four orders
> of magnitude and __slab_free took list_lock just as often. So the
> list_lock contention is not the remote frees, and it never was!

Aha! Okay, thank you for all of your time and help with this.

If I am understanding correctly, the SLUB/barn/sheaf behavior we are
looking at here is pre-existing rather than introduced by this series.
The SMT affinity change is what changes the placement enough to expose
the remote-free cost on this workload.

> - Total and average wait for the list_lock address in the pinned
>   run. The unpinned run showed 5.05 min and 90 us average; if the
>   pinned run is similar, the lock absorbs the same time whichever
>   way the frees go.

For the node0-pinned run, filtering the dominant lock gave:

    2,907,948   3.37 min   69.62 us   __slab_free
      269,174  23.00 sec   85.46 us   get_partial_node_bulk
      175,965  15.17 sec   86.22 us   __refill_objects_node

So in total that is about 3.353M contentions, 4.01 minutes of
aggregate wait, and roughly 71.8 us average wait.

Compared with the unpinned run:

                        unpinned       node0 pinned
    contentions         3.346M         3.353M
    aggregate wait      5.05 min       ~4.01 min
    average wait        90.48 us       ~71.8 us

So the contention count is essentially unchanged even though
free_slowpath drops from ~70M to ~9K. The pinned case does have
somewhat lower wait time per contention.

> - mpstat idle for node 0 in both runs, to tell whether that wait
>   is idle time being spent or throughput being lost.

Averaging the node0 CPUs (0-23,48-71) in that sample gives about 66% idle.
In the earlier unpinned/all-local bad-state samples, those CPUs were nearly
saturated, generally around 0-2% idle.

> - Throughput for the node 0 / node 0 run that produced the counter
>   table above. It is the one configuration missing from your
>   throughput table.

Oh yeah, it was the same trend as the workload table below: unpinned was
43.4 GB/s and node 0 pinned was 46.5 GB/s.

At this point I think we have a reasonably clear separation between the
pre-existing allocator behavior and the regression introduced by the
SMT affinity change. I don't have any additional data I think would
materially change that conclusion, but I'm happy to run any targeted
follow-up you think would still be useful.