[BUG] ice: Temporary packet processing overload causes permanent RX drops

[BUG] ice: Temporary packet processing overload causes permanent RX drops

[Marcus Wichelmann]

Hi there, I broke some network cards again. This time I noticed continuous RX packet drops with an Intel E810-XXV.

When such a card temporarily (just for a few seconds) receives a large flood of packets and the kernel cannot keep
up with processing them, the following appears in the Kernel log:

kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
kernel: workqueue: ice_rx_dim_work [ice] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
kernel: workqueue: ice_rx_dim_work [ice] hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
kernel: workqueue: psi_avgs_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 7 times, consider switching to WQ_UNBOUND
kernel: workqueue: ice_rx_dim_work [ice] hogged CPU for >10000us 7 times, consider switching to WQ_UNBOUND
kernel: workqueue: psi_avgs_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
...

After that, the NIC seems to be in a permanently broken state and continues to drop a few percent of the received
packets, even at low data rates. When reducing the incoming packet rate to just 10.000 pps, I can see over 500 pps
of that being dropped. After reinitializing the NIC (e.g. by changing the channel count using ethtool), the error
goes away and it's rock solid again. Until the next packet flood.

We have reproduced this with:
  Linux 6.8.0-88-generic (Ubuntu 24.04)
  Linux 6.14.0-36-generic (Ubuntu 24.04 HWE)
  Linux 6.18.0-061800-generic (Ubuntu Mainline PPA)

CPU: AMD EPYC 9825 144-Core Processor (288 threads)

lspci | grep Ethernet
  c7:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)
  c7:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)

ethtool -i eth0
  driver: ice
  version: 6.18.0-061800-generic
  firmware-version: 4.90 0x80020ef6 1.3863.0
  expansion-rom-version: 
  bus-info: 0000:c7:00.0
  supports-statistics: yes
  supports-test: yes
  supports-eeprom-access: yes
  supports-register-dump: yes
  supports-priv-flags: yes

ethtool -l eth0
  Channel parameters for eth0:
  Pre-set maximums:
  RX:		288
  TX:		288
  Other:		1
  Combined:	288
  Current hardware settings:
  RX:		0
  TX:		32
  Other:		1
  Combined:	256
These are the defaults after boot.

ethtool -S eth0 | grep rx_dropped
  rx_dropped: 7206525
  rx_dropped.nic: 0
ethtool -S eth1 | grep rx_dropped
  rx_dropped: 6889634
  rx_dropped.nic: 0

How to reproduce:

1. Use another host to flood the host with the E810 NIC with 64 byte large UDP packets. I used trafgen for that and
made sure, that the source ports are randomized to make RSS spread the load over all channels. The packet rate must
be high enough to overload the packet processing on the receiving host.
In my case, 4 Mpps was already enough to make the errors show up in the kernel log and trigger the permanent packet
loss, but the needed packet rate may depend on how CPU intensive the processing of each packet is. Dropping packets
early (e.g. using iptables) makes reproducing harder.

2. Monitor the rx_dropped counter and the kernel log. After a few seconds, above warnings/errors should show up in
the kernel log.

3. Stop the traffic generator and re-run it with a way lower packet rate, e.g. 10.000 pps. Now it can be seen that
a good part of these packets is being dropped, even though the kernel could easily keep up with this small packet rate.

In my case the two ports of the E810 NIC were part of a bonding, but I don't think this is required to reproduce the
issue.

Please let me know, if there is more information I could provide.

Thanks,
Marcus

Re: [Intel-wired-lan] [BUG] ice: Temporary packet processing overload causes permanent RX drops

[Jacob Keller]

[-- Attachment #1.1: Type: text/plain, Size: 5373 bytes --]



On 12/5/2025 6:01 AM, Marcus Wichelmann wrote:
> Hi there, I broke some network cards again. This time I noticed continuous RX packet drops with an Intel E810-XXV.
> 
> When such a card temporarily (just for a few seconds) receives a large flood of packets and the kernel cannot keep
> up with processing them, the following appears in the Kernel log:
> 
> kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
> kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
> kernel: workqueue: ice_rx_dim_work [ice] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
> kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
> kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
> kernel: workqueue: ice_rx_dim_work [ice] hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
> kernel: workqueue: psi_avgs_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
> kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
> kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 7 times, consider switching to WQ_UNBOUND
> kernel: workqueue: ice_rx_dim_work [ice] hogged CPU for >10000us 7 times, consider switching to WQ_UNBOUND

I am a bit curious why the ice_rx_dim_work hogs so much CPU here..

> kernel: workqueue: psi_avgs_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
> kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
> ...
> 
> After that, the NIC seems to be in a permanently broken state and continues to drop a few percent of the received
> packets, even at low data rates. When reducing the incoming packet rate to just 10.000 pps, I can see over 500 pps
> of that being dropped. After reinitializing the NIC (e.g. by changing the channel count using ethtool), the error
> goes away and it's rock solid again. Until the next packet flood.
> 

A reset likely causes a bunch of stuff to get flushed and reconfigured.

> We have reproduced this with:
>   Linux 6.8.0-88-generic (Ubuntu 24.04)
>   Linux 6.14.0-36-generic (Ubuntu 24.04 HWE)
>   Linux 6.18.0-061800-generic (Ubuntu Mainline PPA)
> 

I think we recently merged a bunch of work on the Rx path as part of our
conversion to page pool. It would be interesting to see if those changes
impact this. Clearly the issue goes back some time since v6.8 at least..

> CPU: AMD EPYC 9825 144-Core Processor (288 threads)
> 
> lspci | grep Ethernet
>   c7:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)
>   c7:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)
> 
> ethtool -i eth0
>   driver: ice
>   version: 6.18.0-061800-generic
>   firmware-version: 4.90 0x80020ef6 1.3863.0
>   expansion-rom-version: 
>   bus-info: 0000:c7:00.0
>   supports-statistics: yes
>   supports-test: yes
>   supports-eeprom-access: yes
>   supports-register-dump: yes
>   supports-priv-flags: yes
> 
> ethtool -l eth0
>   Channel parameters for eth0:
>   Pre-set maximums:
>   RX:		288
>   TX:		288
>   Other:		1
>   Combined:	288
>   Current hardware settings:
>   RX:		0
>   TX:		32
>   Other:		1
>   Combined:	256
> These are the defaults after boot.
> 
> ethtool -S eth0 | grep rx_dropped
>   rx_dropped: 7206525
>   rx_dropped.nic: 0
> ethtool -S eth1 | grep rx_dropped
>   rx_dropped: 6889634
>   rx_dropped.nic: 0
> 

Interesting. From reviewing the code, the rx_dropped counter appears to
be the hardware Rx discard counter which comes from GLV_RDPC, which
means its definitely hardware that is dropping the packets. Possibly
because the rings are full and somehow don't get cleared even after the
traffic stops...

> How to reproduce:
> 
> 1. Use another host to flood the host with the E810 NIC with 64 byte large UDP packets. I used trafgen for that and
> made sure, that the source ports are randomized to make RSS spread the load over all channels. The packet rate must
> be high enough to overload the packet processing on the receiving host.
> In my case, 4 Mpps was already enough to make the errors show up in the kernel log and trigger the permanent packet
> loss, but the needed packet rate may depend on how CPU intensive the processing of each packet is. Dropping packets
> early (e.g. using iptables) makes reproducing harder.
> 
> 2. Monitor the rx_dropped counter and the kernel log. After a few seconds, above warnings/errors should show up in
> the kernel log.
> 
> 3. Stop the traffic generator and re-run it with a way lower packet rate, e.g. 10.000 pps. Now it can be seen that
> a good part of these packets is being dropped, even though the kernel could easily keep up with this small packet rate.
> 

I assume the rx_dropped counter still incrementing here?

> In my case the two ports of the E810 NIC were part of a bonding, but I don't think this is required to reproduce the
> issue.
> 
> Please let me know, if there is more information I could provide.
> 
> Thanks,
> Marcus


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [Intel-wired-lan] [BUG] ice: Temporary packet processing overload causes permanent RX drops

[Marcus Wichelmann]

Am 09.12.25 um 01:05 schrieb Jacob Keller:
> On 12/5/2025 6:01 AM, Marcus Wichelmann wrote:
>> Hi there, I broke some network cards again. This time I noticed continuous RX packet drops with an Intel E810-XXV.
>>>> We have reproduced this with:
>>   Linux 6.8.0-88-generic (Ubuntu 24.04)
>>   Linux 6.14.0-36-generic (Ubuntu 24.04 HWE)
>>   Linux 6.18.0-061800-generic (Ubuntu Mainline PPA)
> 
> I think we recently merged a bunch of work on the Rx path as part of our
> conversion to page pool. It would be interesting to see if those changes
> impact this. Clearly the issue goes back some time since v6.8 at least..
Hi Jacob,

I guess you mean 93f53db9f9dc ("ice: switch to Page Pool")?

I have now repeated all tests with a kernel built from latest net-next
branch and can still reproduce it, even though I needed way higher packet
rates (15 instead of 4 Mpps when using 256 channels). Something about the
packet processing on our test system seems to have gotten way more
efficient with this kernel update.

The symptoms are the same. The following IO_PAGE_FAULTs appear in the
kernel log and after that, there is a permanent packet loss of 1-10%
even at very low packet rates.

  kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
  kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
  kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
  kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
  kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
  kernel: ice 0000:c7:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x4000180000 flags=0x0020]
  [...]
  kernel: ice 0000:c7:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x4000180000 flags=0x0020]
  kernel: amd_iommu_report_page_fault: 10 callbacks suppressed
  [...]

I experimented with a few different channel counts and noticed that
the issue only occurs with a combined channel count >128. So on
systems with less many CPU cores, this bug probably never occurs.

  256: reproduced.
  254: reproduced.
  200: reproduced.
  129: reproduced.
  128: stable.
   64: stable.

Tested using "ethtool -L eth{0,1} combined XXX".

With <=128 channels, only the "... hogged CPU ..." warnings appear
but no IO_PAGE_FAULTs. There is also no permanent packet loss after
stopping the traffic generator.

>> [...]
>>
>> 3. Stop the traffic generator and re-run it with a way lower packet rate, e.g. 10.000 pps. Now it can be seen that
>> a good part of these packets is being dropped, even though the kernel could easily keep up with this small packet rate.
> 
> I assume the rx_dropped counter still incrementing here?

Yes. After the NIC is in this broken state, a few percent of all
packets is being dropped and the rx_dropped counter increases
with each of them.

>> [...]

I also looked into why the packet processing load on this system
is so high and `perf top` shows that it almost completely
originates from native_queued_spin_lock_slowpath.

When digging deeper using `perf lock contention -Y spinlock`:

 contended   total wait     max wait     avg wait         type   caller
   1724043      4.36 m     198.66 us    151.66 us     spinlock   __netif_receive_skb_core.constprop.0+0x832
     35960      2.51 s     112.57 ms     69.51 us     spinlock   __netif_receive_skb_core.constprop.0+0x832
       620    103.79 ms    189.87 us    167.40 us     spinlock   do_sys_poll+0x26f

I'm not yet sure what is causing this.
I don't think it's related to this issue, but maybe that's part of
what brings this bug to daylight, so probably still worth a mention.

I hope you can make some sense of all that.

Thanks,
Marcus

Re: [Intel-wired-lan] [BUG] ice: Temporary packet processing overload causes permanent RX drops

[Jacob Keller]

[-- Attachment #1.1: Type: text/plain, Size: 4465 bytes --]



On 12/11/2025 10:00 AM, Marcus Wichelmann wrote:
> Am 09.12.25 um 01:05 schrieb Jacob Keller:
>> On 12/5/2025 6:01 AM, Marcus Wichelmann wrote:
>>> Hi there, I broke some network cards again. This time I noticed continuous RX packet drops with an Intel E810-XXV.
>>>>> We have reproduced this with:
>>>   Linux 6.8.0-88-generic (Ubuntu 24.04)
>>>   Linux 6.14.0-36-generic (Ubuntu 24.04 HWE)
>>>   Linux 6.18.0-061800-generic (Ubuntu Mainline PPA)
>>
>> I think we recently merged a bunch of work on the Rx path as part of our
>> conversion to page pool. It would be interesting to see if those changes
>> impact this. Clearly the issue goes back some time since v6.8 at least..
> Hi Jacob,
> 
> I guess you mean 93f53db9f9dc ("ice: switch to Page Pool")?
> 
> I have now repeated all tests with a kernel built from latest net-next
> branch and can still reproduce it, even though I needed way higher packet
> rates (15 instead of 4 Mpps when using 256 channels). Something about the
> packet processing on our test system seems to have gotten way more
> efficient with this kernel update.
> 

Good info. I'm not certain if the refactors to switch to page pool would
fully explain this, but we did have some big improvements for certain
workloads with the switch.

> The symptoms are the same. The following IO_PAGE_FAULTs appear in the
> kernel log and after that, there is a permanent packet loss of 1-10%
> even at very low packet rates.
> 
>   kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
>   kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
>   kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
>   kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
>   kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
>   kernel: ice 0000:c7:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x4000180000 flags=0x0020]
>   [...]
>   kernel: ice 0000:c7:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x4000180000 flags=0x0020]
>   kernel: amd_iommu_report_page_fault: 10 callbacks suppressed
>   [...]
> 
> I experimented with a few different channel counts and noticed that
> the issue only occurs with a combined channel count >128. So on
> systems with less many CPU cores, this bug probably never occurs.
> 

Interesting that it only triggers if you get IO_PAGE_FAULT.

>   256: reproduced.
>   254: reproduced.
>   200: reproduced.
>   129: reproduced.
>   128: stable.
>    64: stable.
> 
> Tested using "ethtool -L eth{0,1} combined XXX".
> 
> With <=128 channels, only the "... hogged CPU ..." warnings appear
> but no IO_PAGE_FAULTs. There is also no permanent packet loss after
> stopping the traffic generator.
> 
>>> [...]
>>>
>>> 3. Stop the traffic generator and re-run it with a way lower packet rate, e.g. 10.000 pps. Now it can be seen that
>>> a good part of these packets is being dropped, even though the kernel could easily keep up with this small packet rate.
>>
>> I assume the rx_dropped counter still incrementing here?
> 
> Yes. After the NIC is in this broken state, a few percent of all
> packets is being dropped and the rx_dropped counter increases
> with each of them.
> 

Right. That's quite strange.

>>> [...]
> 
> I also looked into why the packet processing load on this system
> is so high and `perf top` shows that it almost completely
> originates from native_queued_spin_lock_slowpath.
> 
> When digging deeper using `perf lock contention -Y spinlock`:
> 
>  contended   total wait     max wait     avg wait         type   caller
>    1724043      4.36 m     198.66 us    151.66 us     spinlock   __netif_receive_skb_core.constprop.0+0x832
>      35960      2.51 s     112.57 ms     69.51 us     spinlock   __netif_receive_skb_core.constprop.0+0x832
>        620    103.79 ms    189.87 us    167.40 us     spinlock   do_sys_poll+0x26f
> 
> I'm not yet sure what is causing this.
> I don't think it's related to this issue, but maybe that's part of
> what brings this bug to daylight, so probably still worth a mention.
> 
> I hope you can make some sense of all that.
> 
> Thanks,
> Marcus


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]