Re: [PATCH v4 1/3] softirq: Allow raising SCHED_SOFTIRQ from SMP-call-function on RT kernel

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2024-10-30 07:15:55 [+0000], K Prateek Nayak wrote:
…
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -280,17 +280,24 @@ static inline void invoke_softirq(void)
>  		wakeup_softirqd();
>  }
>  
> +#define SCHED_SOFTIRQ_MASK	BIT(SCHED_SOFTIRQ)
> +
>  /*
>   * flush_smp_call_function_queue() can raise a soft interrupt in a function
> - * call. On RT kernels this is undesired and the only known functionality
> - * in the block layer which does this is disabled on RT. If soft interrupts
> - * get raised which haven't been raised before the flush, warn so it can be
> + * call. On RT kernels this is undesired and the only known functionalities
> + * are in the block layer which is disabled on RT, and in the scheduler for
> + * idle load balancing. If soft interrupts get raised which haven't been
> + * raised before the flush, warn if it is not a SCHED_SOFTIRQ so it can be
>   * investigated.
>   */
>  void do_softirq_post_smp_call_flush(unsigned int was_pending)
>  {
> -	if (WARN_ON_ONCE(was_pending != local_softirq_pending()))
> +	unsigned int is_pending = local_softirq_pending();
> +
> +	if (unlikely(was_pending != is_pending)) {
> +		WARN_ON_ONCE(was_pending != (is_pending & ~SCHED_SOFTIRQ_MASK));
>  		invoke_softirq();

This behaviour also happens with threadirqs on !PREEMPT_RT but without
the warning. I haven't checked it but I expect invoke_softirq() to wake
ksoftirqd here, too.
This only happens because of 2/3 in the series as far as I can tell.

Now I am curious to hear from the sched/ NOHZ folks if it makes sense to
invoke SCHED_SOFTIRQ from within ksoftirqd because unlike on an idle CPU
the CPU is now not seen as idle due to ksoftirqd running on the CPU.
There is code that checks rq->nr_running and/ or idle_cpu().

> +	}
>  }

Sebastian