From: "Chuck Lever" <cel@kernel.org>
To: tmenninger@purestorage.com
Cc: "Trond Myklebust" <trondmy@kernel.org>,
"Anna Schumaker" <anna@kernel.org>, "Tejun Heo" <tj@kernel.org>,
"Lai Jiangshan" <jiangshanlai@gmail.com>,
linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org,
"Eric Badger" <ebadger@everpuredata.com>,
"Jon Curley" <jcurley@everpuredata.com>
Subject: Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention
Date: Tue, 15 Sep 2026 10:42:07 -0400 [thread overview]
Message-ID: <0e2f9688-097c-4bb9-a7ec-82b41bdb3653@slotpi15m67> (raw)
In-Reply-To: <20260914173958.2045683-1-tmenninger@purestorage.com>
On Mon, 14 Sep 2026, Tim Menninger wrote:
> If I am understanding correctly, the SLUB/barn/sheaf behavior we are
> looking at here is pre-existing rather than introduced by this series.
> The SMT affinity change is what changes the placement enough to expose
> the remote-free cost on this workload.
That is the basic idea, but there are nuances. Split it into
three pieces with different owners:
1. The list_lock contention is pre-existing and does not bound
throughput. Your numbers show it directly:
unpinned node0 pinned
contentions 3.346M 3.353M
aggregate wait 5.05 min 4.01 min
node 0 idle 0-2% 66%
throughput 43.4 GB/s 46.5 GB/s
The lock is taken just as often in the run that has full
throughput and two thirds of node 0 idle.
I have since reproduced the acquisition rate here, on a single-node
EPYC 8224P (24c/48t) reading direct 1M blocks over NFS/RDMA at
12 GB/s, with the barn depth made a runtime knob and
CONFIG_SLUB_STATS on. At depth 1, 28% of barn puts fail and
free_add_partial takes list_lock 5.0M times per 10 seconds, the
same order as your 3.35M contended acquisitions. Total wait on the
lock in that window was 36 ms, and throughput did not change at any
depth from 1 to 64, nor with the work spread across all 48 CPUs,
nor with 16x the in-flight population at iodepth 256.
So the acquisition rate is ordinary. What is not ordinary is your
72 us average wait, against about 7 ns here. That is an allocator
question, and I'd like to take it to linux-mm with your numbers.
More on that below.
2. The throughput loss in the all-local placement comes from the
70M remote frees per 10 seconds, not from the lock. Pinning the
workload to node 0 removed the remote frees, returned 3 GB/s, and
left node 0 two thirds idle with the lock profile unchanged. That
cost is a function of where the CQs land relative to the
application threads, which the RDMA core decides at connect time,
and it exists with or without this series.
3. The smt scope costs a further 4 GB/s in that same placement
(43 vs 47 on your scope ladder), and nothing in the slab data
explains it: the frees stay on node 0 under both smt and
cache_shard, so the scope is not what moves them across nodes. I
don't have an explanation.
Since you have a robust reproducer, can you take this to the
linux-mm list (cc me) with the following three items (no new runs
needed for the first two):
- The debug config. Any of these lengthens every list_lock hold
without disabling sheaves:
zcat /proc/config.gz | grep -E \
'DEBUG_SPINLOCK|DEBUG_LIST|PROVE_LOCKING|LOCK_STAT|KASAN'
- The CPU model and socket count.
- A cmpxchg_double_fail delta from /sys/kernel/slab/nfs_page over
the same 10-second window as your other counters. __slab_free
drops list_lock and retries when the freelist cmpxchg fails, so
a large number there means each logical free costs several lock
acquisitions. Mine was under a hundred.
One more thing I can't reconcile: 4.01 minutes of wait inside a
10-second window is 240 CPU-seconds, half of node 0's 48 threads
for the whole window, which does not fit 66% idle. Were the perf
lock capture and the mpstat sample the same 10 seconds, and did
perf lock cover all 96 CPUs or only node 0? If the wait is real,
node 0 spends half its time spinning even in the good run, and
the lock is where the headroom goes once the remote frees load
the node. If it is an artifact of the window, the lock costs less
than the profile says, and the report to linux-mm is a simpler
one.
--
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
next prev parent reply other threads:[~2026-09-15 14:42 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-31 18:21 [PATCH RFC 0/8] Reduce lock contention in the NFS client Chuck Lever
2026-08-31 18:21 ` [PATCH RFC 1/8] SUNRPC: Use atomic_t for XID allocation Chuck Lever
2026-08-31 18:21 ` [PATCH RFC 2/8] SUNRPC: Execute initial async RPC states in caller's context Chuck Lever
2026-08-31 18:21 ` [PATCH RFC 3/8] SUNRPC: Split recv_lock out of xprt->queue_lock Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 4/8] Set WQ_SYSFS on key NFS-related workqueues Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 5/8] workqueue: Export the functions needed for WQ attribute modification Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention Chuck Lever
2026-09-02 20:40 ` Tim Menninger
2026-09-03 13:33 ` Chuck Lever
2026-09-03 23:50 ` Tim Menninger
2026-09-04 14:13 ` Chuck Lever
2026-09-04 23:23 ` Tim Menninger
2026-09-06 16:21 ` Chuck Lever
2026-09-08 18:19 ` tmenninger
2026-09-08 21:11 ` Chuck Lever
2026-09-08 23:39 ` tmenninger
2026-09-09 16:14 ` Chuck Lever
2026-09-12 14:59 ` tmenninger
2026-09-12 15:49 ` Chuck Lever
2026-09-13 13:13 ` tmenninger
2026-09-13 16:40 ` Chuck Lever
2026-09-14 17:39 ` tmenninger
2026-09-15 14:42 ` Chuck Lever [this message]
2026-09-16 23:25 ` Tim Menninger
2026-08-31 18:22 ` [PATCH RFC 7/8] NFS: Reduce nfsiod " Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 8/8] SUNRPC: Reduce xprtiod " Chuck Lever
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0e2f9688-097c-4bb9-a7ec-82b41bdb3653@slotpi15m67 \
--to=cel@kernel.org \
--cc=anna@kernel.org \
--cc=ebadger@everpuredata.com \
--cc=jcurley@everpuredata.com \
--cc=jiangshanlai@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=tj@kernel.org \
--cc=tmenninger@purestorage.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®