From: Jesper Dangaard Brouer <jbrouer@redhat.com>
To: Felix Fietkau <nbd@nbd.name>, Jakub Kicinski <kuba@kernel.org>
Cc: brouer@redhat.com, netdev@vger.kernel.org,
Jonathan Corbet <corbet@lwn.net>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Paolo Abeni <pabeni@redhat.com>,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
Dave Taht <dave.taht@gmail.com>
Subject: Re: [PATCH net-next] net/core: add optional threading for backlog processing
Date: Wed, 29 Mar 2023 18:14:32 +0200 [thread overview]
Message-ID: <193501d0-094a-cc5a-c3ae-4553a56e3a3a@redhat.com> (raw)
In-Reply-To: <2d251879-1cf4-237d-8e62-c42bb4feb047@nbd.name>
On 24/03/2023 18.35, Felix Fietkau wrote:
> On 24.03.23 18:20, Jakub Kicinski wrote:
>> On Fri, 24 Mar 2023 18:13:14 +0100 Felix Fietkau wrote:
>>> When dealing with few flows or an imbalance on CPU utilization, static RPS
>>> CPU assignment can be too inflexible. Add support for enabling threaded NAPI
>>> for backlog processing in order to allow the scheduler to better balance
>>> processing. This helps better spread the load across idle CPUs.
>>
>> Can you explain the use case a little bit more?
>
> I'm primarily testing this on routers with 2 or 4 CPUs and limited
> processing power, handling routing/NAT. RPS is typically needed to
> properly distribute the load across all available CPUs. When there is
> only a small number of flows that are pushing a lot of traffic, a static
> RPS assignment often leaves some CPUs idle, whereas others become a
> bottleneck by being fully loaded. Threaded NAPI reduces this a bit, but
> CPUs can become bottlenecked and fully loaded by a NAPI thread alone.
>
> Making backlog processing threaded helps split up the processing work
> even more and distribute it onto remaining idle CPUs.
>
> It can basically be used to make RPS a bit more dynamic and
> configurable, because you can assign multiple backlog threads to a set
> of CPUs and selectively steer packets from specific devices / rx queues
> to them and allow the scheduler to take care of the rest.
>
My experience with RPS was that it was too slow on the RX-CPU. Meaning
that it doesn't really scale, because the RX-CPU becomes the scaling
bottleneck. (My data is old and it might scale differently on your ARM
boards).
This is why I/we created the XDP "cpumap". It also creates a kernel
threaded model via a kthread on "map-configured" CPUs. It scales
significantly better than RPS, but it doesn't handle flows and packet
Out-of-Order (OoO) situations automatically like RPS. That is left up
to the BPF-programmer. The kernel samples/bpf xdp_redirect_cpu[0] have
code that shows strategies of load-balancing flows.
The project xdp-cpumap-tc[1] runs in production (3 ISPs using this) and
works in concert with netstack Traffic Control (TC) for scaling
bandwidth shaping at the ISPs. OoO is solved by redirecting all
customers IPs to the same TX/egress CPU. As the README[1] describes it
is recommended to reduce the number of RX-CPUs processing packets, and
have more TX-CPUs that basically runs netstack/TC. One ISP with
2x25Gbit/s is using 2 CPUs for RX and 6 CPUs for TX.
--Jesper
[0]
https://github.com/torvalds/linux/blob/master/samples/bpf/xdp_redirect_cpu.bpf.c
[1] https://github.com/xdp-project/xdp-cpumap-tc
prev parent reply other threads:[~2023-03-29 16:16 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-24 17:13 Felix Fietkau
2023-03-24 17:20 ` Jakub Kicinski
2023-03-24 17:35 ` Felix Fietkau
2023-03-24 17:47 ` Jakub Kicinski
2023-03-24 17:57 ` Felix Fietkau
2023-03-25 3:19 ` Jakub Kicinski
2023-03-25 5:42 ` Felix Fietkau
2023-03-28 2:06 ` Jakub Kicinski
2023-03-28 9:46 ` Felix Fietkau
2023-03-28 9:29 ` Paolo Abeni
2023-03-28 9:45 ` Felix Fietkau
2023-03-28 15:13 ` Paolo Abeni
2023-03-28 15:21 ` Felix Fietkau
2023-03-29 16:14 ` Jesper Dangaard Brouer [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=193501d0-094a-cc5a-c3ae-4553a56e3a3a@redhat.com \
--to=jbrouer@redhat.com \
--cc=brouer@redhat.com \
--cc=corbet@lwn.net \
--cc=dave.taht@gmail.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=kuba@kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=nbd@nbd.name \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®