[PATCH + QUESTION] bpf: use cond_resched_tasks_rcu_qs in bpf_fd_array_map_clear() loop

[PATCH + QUESTION] bpf: use cond_resched_tasks_rcu_qs in bpf_fd_array_map_clear() loop

[Rik van Riel]

syzkaller creates a PROG_ARRAY with huge max_entries and triggers perf
tracepoint open close in parallel. The hung task detector reports
"INFO: task hung in perf_tp_event_init" with event_mutex held waiting
for synchronize_rcu_tasks().

bpf_fd_array_map_clear walks max_entries under RCU read lock on a
workqueue kworker. Commit 4406942e65ca added cond_resched for classic
RCU, but under PREEMPT it is a no-op and a preempted kworker never
reports RCU-tasks QS. The loop then blocks the grace period while
event_mutex is held in ftrace_shutdown, stalling perf_event_open().

Use cond_resched_tasks_rcu_qs to report RCU-tasks QS independent of
preemption model. The grace period completes and event_mutex is
released normally.

Fixes: da765a2f5993 ("bpf: Add poke dependency tracking for prog array maps")
Fixes: 4406942e65ca ("bpf: Fix RCU stall in bpf_fd_array_map_clear()")
Cc: stable@vger.kernel.org
Signed-off-by: Rik van Riel <riel@surriel.com>
Assisted-by: Claude:claude-opus-4-8
---
QUESTION: should we change cond_resched() instead, so it unblocks RCU
in any preempt configuration where cond_resched() is a noop?

That would take care of every bug of this shape with one change,
and rcu_tasks_qs() looks cheap enough?

 kernel/bpf/arraymap.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 248b4818178c..55fd97375130 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -1015,7 +1015,8 @@ static void bpf_fd_array_map_clear(struct bpf_map *map, bool need_defer)
 
 	for (i = 0; i < array->map.max_entries; i++) {
 		__fd_array_map_delete_elem(map, &i, need_defer);
-		cond_resched();
+		/* cond_resched() is a noop with preempt; unblock RCU */
+		cond_resched_tasks_rcu_qs();
 	}
 }
 
-- 
2.53.0-Meta

Re: [PATCH + QUESTION] bpf: use cond_resched_tasks_rcu_qs in bpf_fd_array_map_clear() loop

[Paul E. McKenney]

On Wed, Jul 15, 2026 at 09:53:13PM -0400, Rik van Riel wrote:
> syzkaller creates a PROG_ARRAY with huge max_entries and triggers perf
> tracepoint open close in parallel. The hung task detector reports
> "INFO: task hung in perf_tp_event_init" with event_mutex held waiting
> for synchronize_rcu_tasks().
> 
> bpf_fd_array_map_clear walks max_entries under RCU read lock on a
> workqueue kworker. Commit 4406942e65ca added cond_resched for classic
> RCU, but under PREEMPT it is a no-op and a preempted kworker never
> reports RCU-tasks QS. The loop then blocks the grace period while
> event_mutex is held in ftrace_shutdown, stalling perf_event_open().
> 
> Use cond_resched_tasks_rcu_qs to report RCU-tasks QS independent of
> preemption model. The grace period completes and event_mutex is
> released normally.
> 
> Fixes: da765a2f5993 ("bpf: Add poke dependency tracking for prog array maps")
> Fixes: 4406942e65ca ("bpf: Fix RCU stall in bpf_fd_array_map_clear()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Rik van Riel <riel@surriel.com>
> Assisted-by: Claude:claude-opus-4-8
> ---
> QUESTION: should we change cond_resched() instead, so it unblocks RCU
> in any preempt configuration where cond_resched() is a noop?
> 
> That would take care of every bug of this shape with one change,
> and rcu_tasks_qs() looks cheap enough?

Just confirming that rcu_tasks_qs() is quite cheap:  Check a constant,
check a flag in the task_struct structure, and, if set, set another flag
in this same structure.  If there is no RCU Tasks grace period in effect,
the first flag will always be zero.

							Thanx, Paul

>  kernel/bpf/arraymap.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
> index 248b4818178c..55fd97375130 100644
> --- a/kernel/bpf/arraymap.c
> +++ b/kernel/bpf/arraymap.c
> @@ -1015,7 +1015,8 @@ static void bpf_fd_array_map_clear(struct bpf_map *map, bool need_defer)
>  
>  	for (i = 0; i < array->map.max_entries; i++) {
>  		__fd_array_map_delete_elem(map, &i, need_defer);
> -		cond_resched();
> +		/* cond_resched() is a noop with preempt; unblock RCU */
> +		cond_resched_tasks_rcu_qs();
>  	}
>  }
>  
> -- 
> 2.53.0-Meta
> 
> 

Re: [PATCH + QUESTION] bpf: use cond_resched_tasks_rcu_qs in bpf_fd_array_map_clear() loop

[Rik van Riel]

On Wed, 2026-07-15 at 21:53 -0400, Rik van Riel wrote:
> syzkaller creates a PROG_ARRAY with huge max_entries and triggers
> perf
> tracepoint open close in parallel. The hung task detector reports
> "INFO: task hung in perf_tp_event_init" with event_mutex held waiting
> for synchronize_rcu_tasks().
> 
OK, this is more interesting than it looked at first,
and this patch does not look like the right approach,
but I'm also not sure what other approach would work :(

The actual error being thrown is a hung task:



The syzkaller kernel failure is a tragedy in 3 parts.

First, we have perf_event_open() waiting on the event_mutex:

   syz.7.35551 (pid 31354, state D): perf_event_open 
  -> perf_tp_event_init
   -> perf_trace_init
    -> __mutex_lock
     -> blocked on event_mutex.


Second, we have perf_trace_destroy() indirectly
waiting in synchronize_rcu_tasks(), with the
event_mutex held.

    syz.5.35491 (state:I)  do_exit 
  -> task_work_run 
   -> __fput 
    -> perf_release
     -> perf_event_release_kernel
      -> __free_event
       -> perf_trace_destroy
        -> perf_trace_event_close
         -> reg(TRACE_REG_PERF_CLOSE)
          -> perf_ftrace_event_register
           -> perf_ftrace_function_unregister
            -> unregister_ftrace_function 
             -> ftrace_shutdown
              -> synchronize_rcu_tasks()   <-- BLOCKS HERE, forever,
holding event_mutex

Tasks RCU waits for every task to have gone through
a voluntary reschedule, before the grace period can
be advanced. A preemption does not count.


Third, we have a kworker looping for a very long time:
 
  kworker/0:0 (pid 9, TASK_RUNNING, on CPU0): process_one_work
  -> prog_array_map_clear_deferred
   -> bpf_fd_array_map_clear
    -> __fd_array_map_delete_elem. Looping over a huge PROG_ARRAY
max_entries.

With CONFIG_PREEMPT_LAZY=y this gets preempted,
but since involuntary preemptions do not count
for tasks RCU, that does nothnig to help
ftrace_shutdown() get unstuck.


How do we solve this?

Do we actually need to have cond_resched() points
quiesce the task RCU state when running with lazy
preempt, since those are points where we could
preempt voluntarily?

Do we need to prohibit calling synchronize_rcu_tasks()
while holding a mutex?

Do we need to do something else?

What is the best way forward here?

-- 
All Rights Reversed.

Re: [PATCH + QUESTION] bpf: use cond_resched_tasks_rcu_qs in bpf_fd_array_map_clear() loop

[Paul E. McKenney]

On Thu, Jul 16, 2026 at 10:04:47PM -0400, Rik van Riel wrote:
> On Wed, 2026-07-15 at 21:53 -0400, Rik van Riel wrote:
> > syzkaller creates a PROG_ARRAY with huge max_entries and triggers
> > perf
> > tracepoint open close in parallel. The hung task detector reports
> > "INFO: task hung in perf_tp_event_init" with event_mutex held waiting
> > for synchronize_rcu_tasks().
> > 
> OK, this is more interesting than it looked at first,
> and this patch does not look like the right approach,
> but I'm also not sure what other approach would work :(
> 
> The actual error being thrown is a hung task:
> 
> 
> 
> The syzkaller kernel failure is a tragedy in 3 parts.
> 
> First, we have perf_event_open() waiting on the event_mutex:
> 
>    syz.7.35551 (pid 31354, state D): perf_event_open 
>   -> perf_tp_event_init
>    -> perf_trace_init
>     -> __mutex_lock
>      -> blocked on event_mutex.
> 
> 
> Second, we have perf_trace_destroy() indirectly
> waiting in synchronize_rcu_tasks(), with the
> event_mutex held.
> 
>     syz.5.35491 (state:I)  do_exit 
>   -> task_work_run 
>    -> __fput 
>     -> perf_release
>      -> perf_event_release_kernel
>       -> __free_event
>        -> perf_trace_destroy
>         -> perf_trace_event_close
>          -> reg(TRACE_REG_PERF_CLOSE)
>           -> perf_ftrace_event_register
>            -> perf_ftrace_function_unregister
>             -> unregister_ftrace_function 
>              -> ftrace_shutdown
>               -> synchronize_rcu_tasks()   <-- BLOCKS HERE, forever,
> holding event_mutex
> 
> Tasks RCU waits for every task to have gone through
> a voluntary reschedule, before the grace period can
> be advanced. A preemption does not count.
> 
> 
> Third, we have a kworker looping for a very long time:
>  
>   kworker/0:0 (pid 9, TASK_RUNNING, on CPU0): process_one_work
>   -> prog_array_map_clear_deferred
>    -> bpf_fd_array_map_clear
>     -> __fd_array_map_delete_elem. Looping over a huge PROG_ARRAY
> max_entries.
> 
> With CONFIG_PREEMPT_LAZY=y this gets preempted,
> but since involuntary preemptions do not count
> for tasks RCU, that does nothnig to help
> ftrace_shutdown() get unstuck.
> 
> 
> How do we solve this?
> 
> Do we actually need to have cond_resched() points
> quiesce the task RCU state when running with lazy
> preempt, since those are points where we could
> preempt voluntarily?
> 
> Do we need to prohibit calling synchronize_rcu_tasks()
> while holding a mutex?
> 
> Do we need to do something else?

The traditional approach is cond_resched_tasks_rcu_qs(), which is to
synchronize_rcu_tasks() as cond_resched() is to synchronize_rcu().

But if we can avoid holding mutexes across synchronize_rcu_tasks(), that
would be a very good thing.  The synchronize_rcu_tasks() latency can be
many minutes, which isn't usually what you want when blocking on a mutex.

Or maybe we want a way of sleeping on a mutex at TASK_IDLE.

> What is the best way forward here?

Excellent question!!!

							Thanx, Paul