mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Chuck Lever" <cel@kernel.org>
To: "Tim Menninger" <tmenninger@everpuredata.com>
Cc: "Trond Myklebust" <trondmy@kernel.org>,
	"Anna Schumaker" <anna@kernel.org>, "Tejun Heo" <tj@kernel.org>,
	"Lai Jiangshan" <jiangshanlai@gmail.com>,
	linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org,
	"Eric Badger" <ebadger@everpuredata.com>,
	"Jon Curley" <jcurley@everpuredata.com>
Subject: Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention
Date: Fri, 04 Sep 2026 10:13:28 -0400	[thread overview]
Message-ID: <336623df-ead4-4802-91c5-aa3e785f2971@slotpi15m67> (raw)
In-Reply-To: <20260903235032.960219-1-tmenninger@everpuredata.com>

On Thu, 3 Sep 2026, Tim Menninger wrote:
> The v2 series behaves similarly insofar as some runs see ~15 GB/s and
> other runs see ~45 GB/s, but the low-throughput state no longer appears
> to be directly controlled by the rpciod affinity scope.

Based on your follow-up, I have a theory about the bimodal performance
behavior:

> The substantial completion traffic in these captures is on
> 0000:2a:00.0. Its mlx5 completion IRQs are individually affinitized to
> CPUs. For example:
>
>     mlx5_comp0   -> CPU 0
>     mlx5_comp1   -> CPU 1
>     ...
>     mlx5_comp23  -> CPU 23
>     mlx5_comp24  -> CPU 48
>     ...
>     mlx5_comp47  -> CPU 71
>     mlx5_comp48  -> CPU 24
>     ...
>     mlx5_comp62  -> CPU 38

That is the mlx5 default spread. It fills the NIC's local NUMA node
first, both SMT threads, and overflows the remaining vectors onto the
other socket. If I read the numbering right, CPUs 0-23 and 48-71 are
the two threads of the node that 0000:2a:00.0 hangs off, and comp48
through comp62 land on the remote socket. Please confirm with:

    cat /sys/bus/pci/devices/0000:2a:00.0/numa_node
    lscpu | grep 'NUMA node'

rpcrdma.ko allocates a send and a receive completion queue for each
transport, and the RDMA core assigns the completion vector from a
host-global round-robin counter. The completion vector is selected at
transport connect time and is fixed for the life of the connection.

Completions on a remote vector run the CQ handler, the RPC reply
processing, and the rpciod wakeup all on the wrong socket. My
suggestion is that you should pin the NIC's interrupt vectors to
CPUs on the NIC's local node so that the round-robin cannot select
a remote one. Something like the following, adjusted once you have
confirmed the local CPU list:

    for irq in $(awk '/mlx5_comp(4[89]|5[0-9]|6[0-2])@pci:0000:2a:00.0/ \
                      { sub(":", "", $1); print $1 }' /proc/interrupts); do
        echo 0-23,48-71 > /proc/irq/$irq/smp_affinity_list
    done

Keep irqbalance disabled, since it will undo the interrupt steering.


-- 
Chuck Lever

  reply	other threads:[~2026-09-04 14:13 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-08-31 18:21 [PATCH RFC 0/8] Reduce lock contention in the NFS client Chuck Lever
2026-08-31 18:21 ` [PATCH RFC 1/8] SUNRPC: Use atomic_t for XID allocation Chuck Lever
2026-08-31 18:21 ` [PATCH RFC 2/8] SUNRPC: Execute initial async RPC states in caller's context Chuck Lever
2026-08-31 18:21 ` [PATCH RFC 3/8] SUNRPC: Split recv_lock out of xprt->queue_lock Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 4/8] Set WQ_SYSFS on key NFS-related workqueues Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 5/8] workqueue: Export the functions needed for WQ attribute modification Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention Chuck Lever
2026-09-02 20:40   ` Tim Menninger
2026-09-03 13:33     ` Chuck Lever
2026-09-03 23:50       ` Tim Menninger
2026-09-04 14:13         ` Chuck Lever [this message]
2026-09-04 23:23           ` Tim Menninger
2026-09-06 16:21             ` Chuck Lever
2026-09-08 18:19               ` tmenninger
2026-09-08 21:11                 ` Chuck Lever
2026-09-08 23:39                   ` tmenninger
2026-09-09 16:14                     ` Chuck Lever
2026-09-12 14:59                       ` tmenninger
2026-09-12 15:49                         ` Chuck Lever
2026-09-13 13:13                           ` tmenninger
2026-09-13 16:40                             ` Chuck Lever
2026-09-14 17:39                               ` tmenninger
2026-09-15 14:42                                 ` Chuck Lever
2026-09-16 23:25                                   ` Tim Menninger
2026-08-31 18:22 ` [PATCH RFC 7/8] NFS: Reduce nfsiod " Chuck Lever
2026-08-31 18:22 ` [PATCH RFC 8/8] SUNRPC: Reduce xprtiod " Chuck Lever

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=336623df-ead4-4802-91c5-aa3e785f2971@slotpi15m67 \
    --to=cel@kernel.org \
    --cc=anna@kernel.org \
    --cc=ebadger@everpuredata.com \
    --cc=jcurley@everpuredata.com \
    --cc=jiangshanlai@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=tj@kernel.org \
    --cc=tmenninger@everpuredata.com \
    --cc=trondmy@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®