From: Haifeng Xu <haifeng.xu@shopee.com>
To: Eric Dumazet <edumazet@google.com>
Cc: Tony Nguyen <anthony.l.nguyen@intel.com>,
Przemek Kitszel <przemyslaw.kitszel@intel.com>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
intel-wired-lan@lists.osuosl.org
Subject: Re: [Question] ixgbe：Mechanism of RSS
Date: Fri, 3 Jan 2025 10:36:50 +0800 [thread overview]
Message-ID: <03b02880-4aae-4a4a-9533-7a756cc84001@shopee.com> (raw)
In-Reply-To: <CANn89iLbC3qkeptG9xv1nZyWHUTdtXBf4w3LGaisRGc7xj4pMw@mail.gmail.com>
On 2025/1/2 19:46, Eric Dumazet wrote:
> On Thu, Jan 2, 2025 at 12:23 PM Haifeng Xu <haifeng.xu@shopee.com> wrote:
>>
>>
>>
>> On 2025/1/2 18:34, Eric Dumazet wrote:
>>> On Thu, Jan 2, 2025 at 9:43 AM Haifeng Xu <haifeng.xu@shopee.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2025/1/2 16:13, Eric Dumazet wrote:
>>>>> On Thu, Jan 2, 2025 at 4:53 AM Haifeng Xu <haifeng.xu@shopee.com> wrote:
>>>>>>
>>>>>> Hi masters,
>>>>>>
>>>>>> We use the Intel Corporation 82599ES NIC in our production environment. And it has 63 rx queues, every rx queue interrupt is processed by a single cpu.
>>>>>> The RSS configuration can be seen as follow:
>>>>>>
>>>>>> RX flow hash indirection table for eno5 with 63 RX ring(s):
>>>>>> 0: 0 1 2 3 4 5 6 7
>>>>>> 8: 8 9 10 11 12 13 14 15
>>>>>> 16: 0 1 2 3 4 5 6 7
>>>>>> 24: 8 9 10 11 12 13 14 15
>>>>>> 32: 0 1 2 3 4 5 6 7
>>>>>> 40: 8 9 10 11 12 13 14 15
>>>>>> 48: 0 1 2 3 4 5 6 7
>>>>>> 56: 8 9 10 11 12 13 14 15
>>>>>> 64: 0 1 2 3 4 5 6 7
>>>>>> 72: 8 9 10 11 12 13 14 15
>>>>>> 80: 0 1 2 3 4 5 6 7
>>>>>> 88: 8 9 10 11 12 13 14 15
>>>>>> 96: 0 1 2 3 4 5 6 7
>>>>>> 104: 8 9 10 11 12 13 14 15
>>>>>> 112: 0 1 2 3 4 5 6 7
>>>>>> 120: 8 9 10 11 12 13 14 15
>>>>>>
>>>>>> The maximum number of RSS queues is 16. So I have some questions about this. Will other cpus except 0~15 receive the rx interrupts?
>>>>>>
>>>>>> In our production environment, cpu 16~62 also receive the rx interrupts. Was our RSS misconfigured?
>>>>>
>>>>> It really depends on which cpus are assigned to each IRQ.
>>>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> Each irq was assigned to a single cpu, for exapmle:
>>>>
>>>> irq cpu
>>>>
>>>> 117 0
>>>> 118 1
>>>>
>>>> ......
>>>>
>>>> 179 62
>>>>
>>>> All cpus trigger interrupts not only cpus 0~15.
>>>> It seems that the result is inconsistent with the RSS hash value.
>>>>
>>>>
>>>
>>> I misread your report, I thought you had 16 receive queues.
>>>
>>> Why don't you change "ethtool -L eno5 rx 16", instead of trying to
>>> configure RSS manually ?
>>
>> Hi Eric,
>>
>> We want to make full use of cpu resources to receive packets. So
>> we enable 63 rx queues. But we found the rate of interrupt growth
>> on cpu 0~15 is faster than other cpus(almost twice). I don't know
>> whether it is related to RSS configuration. We didn't make any changes
>> on the RSS configration after the server is up.
>>
>>
>>
>> FYI, on another server, we use Mellanox Technologies MT27800 NIC.
>> The rate of interrupt growth on cpu 0~63 seems have little gap.
>>
>> It's RSS configration can be seen as follow:
>>
>> RX flow hash indirection table for ens2f0np0 with 63 RX ring(s):
>> 0: 0 1 2 3 4 5 6 7
>> 8: 8 9 10 11 12 13 14 15
>> 16: 16 17 18 19 20 21 22 23
>> 24: 24 25 26 27 28 29 30 31
>> 32: 32 33 34 35 36 37 38 39
>> 40: 40 41 42 43 44 45 46 47
>> 48: 48 49 50 51 52 53 54 55
>> 56: 56 57 58 59 60 61 62 0
>> 64: 1 2 3 4 5 6 7 8
>> 72: 9 10 11 12 13 14 15 16
>> 80: 17 18 19 20 21 22 23 24
>> 88: 25 26 27 28 29 30 31 32
>> 96: 33 34 35 36 37 38 39 40
>> 104: 41 42 43 44 45 46 47 48
>> 112: 49 50 51 52 53 54 55 56
>> 120: 57 58 59 60 61 62 0 1
>> 128: 2 3 4 5 6 7 8 9
>> 136: 10 11 12 13 14 15 16 17
>> 144: 18 19 20 21 22 23 24 25
>> 152: 26 27 28 29 30 31 32 33
>> 160: 34 35 36 37 38 39 40 41
>> 168: 42 43 44 45 46 47 48 49
>> 176: 50 51 52 53 54 55 56 57
>> 184: 58 59 60 61 62 0 1 2
>> 192: 3 4 5 6 7 8 9 10
>> 200: 11 12 13 14 15 16 17 18
>> 208: 19 20 21 22 23 24 25 26
>> 216: 27 28 29 30 31 32 33 34
>> 224: 35 36 37 38 39 40 41 42
>> 232: 43 44 45 46 47 48 49 50
>> 240: 51 52 53 54 55 56 57 58
>> 248: 59 60 61 62 0 1 2 3
>>
>>
>> I am confused that why ixgbe NIC can dispatch the packets
>> to the rx queues that not specified in RSS configuration.
>
> Perhaps make sure to change RX flow hash indirection table on the
> Intel NIC then...
>
> Maybe you changed the default configuration.
>
> ethtool -X eno5 equal 64
The maximum number of RSS queues supported by Intel Corporation 82599ES NIC
is 16. When I specify the number which is larger than 16, it shows the below message.
"Cannot set RX flow hash configuration: Invalid argument."
> Or
> ethtool -X eno5 default
This command can run sucessfully and as I saied above, it only has 16 queues.
RX flow hash indirection table for eno5 with 63 RX ring(s):
0: 0 1 2 3 4 5 6 7
8: 8 9 10 11 12 13 14 15
16: 0 1 2 3 4 5 6 7
24: 8 9 10 11 12 13 14 15
32: 0 1 2 3 4 5 6 7
40: 8 9 10 11 12 13 14 15
48: 0 1 2 3 4 5 6 7
56: 8 9 10 11 12 13 14 15
64: 0 1 2 3 4 5 6 7
72: 8 9 10 11 12 13 14 15
80: 0 1 2 3 4 5 6 7
88: 8 9 10 11 12 13 14 15
96: 0 1 2 3 4 5 6 7
104: 8 9 10 11 12 13 14 15
112: 0 1 2 3 4 5 6 7
120: 8 9 10 11 12 13 14 15
Thanks!
next prev parent reply other threads:[~2025-01-03 2:36 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-01-02 3:53 Haifeng Xu
2025-01-02 8:13 ` Eric Dumazet
2025-01-02 8:43 ` Haifeng Xu
2025-01-02 10:34 ` Eric Dumazet
2025-01-02 11:23 ` Haifeng Xu
2025-01-02 11:46 ` Eric Dumazet
2025-01-03 2:36 ` Haifeng Xu [this message]
2025-01-02 16:01 ` Edward Cree
2025-01-02 16:39 ` Jakub Kicinski
2025-01-03 2:37 ` Haifeng Xu
2025-01-03 3:05 ` Haifeng Xu
2025-01-07 17:16 ` Tony Nguyen
2025-01-08 3:36 ` Haifeng Xu
2025-01-08 21:06 ` Tony Nguyen
2025-01-09 3:26 ` Haifeng Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=03b02880-4aae-4a4a-9533-7a756cc84001@shopee.com \
--to=haifeng.xu@shopee.com \
--cc=anthony.l.nguyen@intel.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=intel-wired-lan@lists.osuosl.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=przemyslaw.kitszel@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®