Re: Latency spikes on V6.15.1 Preempt RT and maybe related to intel? IGB

["Marc Strämke" <marc.straemke@eltropuls.de>]

Hi Sebastian,

On 14.06.25 10:52, Marc Strämke wrote:
>
> I do not really understand where the IRQ/Preemption disabling is 
> happening. What would the next thing be to do? Function (graph?) 
> tracing on all the functions visible in the backtrace?
>
>
> I tried to look at the event race output starting with the call to 
> igb_set_rx_mode. I have attached the trace with all events and a 
> function filter on igb on only the cpu executing ip.  I cannot 
> understand what is happening between timestasmp 700.149995 and the IRQ 
> disable event on 700.150795....
>
>
> Thanks for your help,
>
> Marc
>
>
I am still trying to figure that puzzle out: Please see the following 
function trace snippet:

ip-690530  [000] ..... 178636.460435: rtnl_is_locked <-__dev_change_flags
ip-690530  [000] ..... 178636.460435: __local_bh_disable_ip 
<-__dev_change_flags
ip-690530  [000] ..... 178636.460435: migrate_disable 
<-__local_bh_disable_ip
ip-690530  [000] ...1. 178636.460435: preempt_disable: 
caller=__local_bh_disable_ip+0x76/0xe0 
parent=__local_bh_disable_ip+0x76/0xe0
ip-690530  [000] ...11 178636.460435: preempt_enable: 
caller=__local_bh_disable_ip+0x76/0xe0 
parent=__local_bh_disable_ip+0x76/0xe0
ip-690530  [000] ....1 178636.460435: rt_spin_lock <-__local_bh_disable_ip
ip-690530  [000] ....1 178636.460436: __rcu_read_lock <-rt_spin_lock
ip-690530  [000] ....1 178636.460436: migrate_disable 
<-__local_bh_disable_ip
ip-690530  [000] ....2 178636.460436: __rcu_read_lock 
<-__local_bh_disable_ip
ip-690530  [000] b...2 178636.460436: rt_spin_lock <-__dev_change_flags
ip-690530  [000] b...2 178636.460436: __rcu_read_lock <-rt_spin_lock
ip-690530  [000] b...2 178636.460436: migrate_disable <-__dev_change_flags
ip-690530  [000] b...3 178636.460436: __dev_set_rx_mode <-__dev_change_flags
ip-690530  [000] b...3 178636.460437: igb_set_rx_mode <-__dev_change_flags

It is my understanding that __local_bh_disable_ip called from 
netif_addr_lock_bh seems to be the origin of my latency.

What i do not understand is, even if the bottom halves are disabled. 
Shouldn't I see the interrupt arriving in the trace?

If i understood it correctly, your talk "Unblocking the softirq lock for 
PREEMPT_RT" during the plumbers conference 2023 is exactly about that 
case, right?

Probably fixing this issue is out of my abilities for now ;-) The wast 
variety of locking concepts inside the kernel is quite intimidating for 
a newcomer in kernel land...


Regards,


Marc