Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[tmenninger@purestorage.com]

> cache_shard is the system default, and that is what the workqueue
> used before this series

Right, bummer, I was thinking default was WQ_AFFN_CACHE.

> 1. Toggle affinity_strict under smt:
> 
>        echo 1 > /sys/bus/workqueue/devices/rpciod/affinity_strict
> 
>    Strict pins each pool's kworkers to its SMT pair. If strict
>    recovers throughput, the loss comes from non-strict workers
>    being wake-affined or migrated onto the saturated node. If
>    strict makes it worse, node 0's pools are starved and the fix
>    is to let work spill to node 1. Either result cuts the
>    hypothesis space in half, so if you have time for only one of
>    these, this is the one.

Strict makes it worse. Across five runs, the three all-local CQ
placements ran at 38, 36, and 39 GB/s, while the two split placements
both ran at 46 GB/s.

> 2. Walk the scope ladder: cpu, smt, cache, cache_shard, numa, and
>    report throughput for each. If cpu is as bad as smt, pool
>    granularity itself is the problem. If cache already recovers,
>    the threshold sits between 2-thread and 16-thread pods.

    cpu:         37 GBps
    smt:         43 GBps
    cache:       47 GBps
    cache_shard: 47 GBps
    numa:        47 GBps

> 3. Profile the smt and cache_shard windows of one run, node 0 CPUs
>    only, so the two captures differ in nothing but the scope:
> 
>        perf record -a -g -C 0-23,48-71 -- sleep 10
>        perf lock contention -a -C 0-23,48-71 -- sleep 10
>        perf stat -a -C 0-23,48-71 \
>            -e context-switches,cpu-migrations,sched:sched_wakeup \
>            -- sleep 10

The lock profile is dominated by __slab_free in both cases.

For smt, perf report shows:

    native_queued_spin_lock_slowpath     87.07% self

and its callchain is almost entirely:

    rpc_async_release
      -> rpc_free_task
      -> ff_layout_read_release
      -> pnfs_generic_rw_release
      -> nfs_pgio_release
      -> nfs_direct_read_completion
      -> nfs_release_request
      -> nfs_free_request
      -> kmem_cache_free
      -> __slab_free
      -> _raw_spin_lock_irqsave
      -> native_queued_spin_lock_slowpath

cache_shard looks surprisingly similar:

    native_queued_spin_lock_slowpath     85.84% self

with the same nfs_release_request -> nfs_free_request ->
kmem_cache_free -> __slab_free path dominating the profile.

perf lock reports the same general picture. For smt:

    __slab_free:
        2.07M contentions
        3.83 minutes aggregate wait
        111 us average wait

and for cache_shard:

    __slab_free:
        2.27M contentions
        3.65 minutes aggregate wait
        96 us average wait

process_one_work itself is much smaller:

                        smt             cache_shard
    contentions         34              2087
    total wait          91 us           7.13 ms

Scheduler counters:

                        smt             cache_shard
    context switches    2,184,584       3,107,991
    CPU migrations        129,706         499,400
    sched_wakeup        1,176,787       1,776,031

>    The two candidates I have in mind are a downstream lock, such as
>    the transport's queue_lock or recv_lock, contended by 24 small
>    pools running completions in parallel; or scheduler overhead
>    from each pool waking its own kworkers. The %sys, %irq, and
>    %soft columns from the same mpstat runs would help too, since
>    idle alone does not say what the busy CPUs are doing.

For the all-local case, essentially all of the busy time on that node
is %sys. %irq and %soft are both approximately zero. A few CPUs have
low-single-digit %usr.

For the 17/15 split, CPUs 0-2 have low-single-digit %soft, but aggregate
%soft is only about 0.10%. Otherwise it looks the same: the busy time
is overwhelmingly %sys, with a few CPUs showing low-single-digit %usr.

> 4. With the even 17/15 CQ split and rpciod at cache_shard, a perf
>    profile shows whether the pool-lock slowpath the series targets
>    appears on your box at all. If it does not, cache_shard is
>    correct for your system, and the series needs a way to express
>    that rather than one hardcoded scope.

It is present, but it does not appear significant compared with the
slab contention.

With the 17/15 split and cache_shard:

    process_one_work:
        5,598 contentions
        29.33 ms aggregate wait
        5.24 us average wait

In the same 5-second capture, __slab_free has about 1.61M contentions
and 2.66 minutes aggregate wait. get_partial_node_bulk and
__refill_objects_node are both around 19K contentions and ~200 ms total
wait.