Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[tmenninger@purestorage.com]

> To separate the two, keep the all-local CQ placement and pin the
> workload to node 0:
>
> numactl --cpunodebind=0 --membind=0 elbencho <same args>
>
> Sample the same nfs_page counters and perf lock -l over the same
> 10-second window. free_slowpath should drop toward zero.

I waited for an all-node-0 CQ placement and ran elbencho with both CPU
and memory binding to node 0. mpstat showed the NFS/kernel work
concentrated on node 0.

The 10-second nfs_page deltas were:

                        unpinned       node0 pinned
    free_fastpath:     +49,161,373    +119,680,629
    free_slowpath:     +70,391,872          +8,999
    sheaf_flush:        +1,582,440     +13,436,760
    barn_put:             +673,877      +1,596,175
    barn_put_fail:         +26,374        +223,950
    barn_get:             +673,860      +1,596,219
    barn_get_fail:      +1,199,576        +224,090
    alloc_slowpath:              0               0

So free_slowpath does in fact drop essentially to zero when allocation
and completion/freeing are kept on the same NUMA node. That seems to
confirm the alloc-on-one-node/free-on-the-other explanation for the
large free_slowpath count.

The dominant slab lock was still heavily contended in that run.
Filtering it by caller gave:

    2,907,948   __slab_free+0x126
      269,174   get_partial_node_bulk+0x9c
      175,965   __refill_objects_node+0x4d0

So roughly 87% of the contention was still coming from __slab_free,
with the remaining ~13% split between the two refill paths.

I then tried to answer the remaining question of whether the remote-free
behavior actually bounds throughput or just consumes CPU.

I ran several trials with the workload pinned to node 0, node 1, or
left unpinned. In this set of runs the unbalanced CQ placement happened
to land on node 1:

    workload      CQ placement      throughput
    unpinned      balanced          46-47 GB/s
    unpinned      NUMA 1            43-44 GB/s
    NUMA 0        balanced          45-45 GB/s
    NUMA 0        NUMA 1            42-44 GB/s
    NUMA 1        balanced          46-47 GB/s
    NUMA 1        NUMA 1            46-47 GB/s

I did briefly get a 46 GB/s NUMA-0-workload / NUMA-1-CQ run after the
machine had been up for a while, which made me question whether the slab
contention was actually on the throughput-critical path. However, after
rebooting and repeating the experiment, I could no longer reproduce that
result and it returned to 42-44 GB/s.

For NUMA-1 workload / NUMA-1 CQs at about 46 GB/s:

    free_fastpath:   +119,574,437
    free_slowpath:        +12,127

For the cross-node NUMA-0 workload + NUMA-1 CQs, free_slowpath becomes
large again.

Taken together, I think your explanation is correct, that the workload
allocates nfs_page objects on one NUMA node and rpciod completes and
frees them on the other, those frees bypass the sheaves, hit
__slab_free, and create the slab list-lock contention. Keeping the
allocation and completion/freeing sides on the same node makes
free_slowpath essentially disappear and restores the full throughput.

The one remaining oddity is that the severity appears somewhat
uptime-dependent. The cross-node placement was briefly able to sustain
46 GB/s after the system had been running for a while, but that behavior
disappeared after reboot and the original 42-44 GB/s result became
reproducible again.