[PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu

[PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu

[Steven Rostedt]

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

ftracetest triggered:

 INFO: rcu_tasks detected stalls on tasks:
 00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
 task:osnoise/7       state:R  running task     stack:    0 pid: 2133 ppid:     2 flags:0x00004000
 Call Trace:
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_hardirqs_on+0x2b/0xe0
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_clock_local+0xc/0x20
  ? osnoise_main+0x10e/0x450
  ? trace_softirq_entry_callback+0x50/0x50
  ? kthread+0x153/0x170
  ? __kthread_bind_mask+0x60/0x60
  ? ret_from_fork+0x22/0x30

While running osnoise tracer with other tracers that rely on
synchronize_rcu_tasks(), where that just hung.

The reason is that osnoise_main() never schedules out if the interval
is less than 1, and this will cause synchronize_rcu_tasks() to never
return.

Fixes: bce29ac9ce0bb ("trace: Add osnoise tracer")
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 kernel/trace/trace_osnoise.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
index 38aa5e208ffd..556d530af805 100644
--- a/kernel/trace/trace_osnoise.c
+++ b/kernel/trace/trace_osnoise.c
@@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
 		 * differently from hwlat_detector, the osnoise tracer can run
 		 * without a pause because preemption is on.
 		 */
-		if (interval < 1)
+		if (interval < 1) {
+			/* Let synchronize_rcu_tasks() make progress */
+			cond_resched_tasks_rcu_qs();
 			continue;
+		}
 
 		if (msleep_interruptible(interval))
 			break;
-- 
2.29.2

Re: [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu

[Paul E. McKenney]

On Mon, Jun 28, 2021 at 11:49:53AM -0400, Steven Rostedt wrote:
> From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
> 
> ftracetest triggered:
> 
>  INFO: rcu_tasks detected stalls on tasks:
>  00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
>  task:osnoise/7       state:R  running task     stack:    0 pid: 2133 ppid:     2 flags:0x00004000
>  Call Trace:
>   ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>   ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>   ? trace_hardirqs_on+0x2b/0xe0
>   ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>   ? trace_clock_local+0xc/0x20
>   ? osnoise_main+0x10e/0x450
>   ? trace_softirq_entry_callback+0x50/0x50
>   ? kthread+0x153/0x170
>   ? __kthread_bind_mask+0x60/0x60
>   ? ret_from_fork+0x22/0x30
> 
> While running osnoise tracer with other tracers that rely on
> synchronize_rcu_tasks(), where that just hung.
> 
> The reason is that osnoise_main() never schedules out if the interval
> is less than 1, and this will cause synchronize_rcu_tasks() to never
> return.
> 
> Fixes: bce29ac9ce0bb ("trace: Add osnoise tracer")
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

That looks like something I would do!

Acked-by: Paul E. McKenney <paulmck@kernel.org>

> ---
>  kernel/trace/trace_osnoise.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
> index 38aa5e208ffd..556d530af805 100644
> --- a/kernel/trace/trace_osnoise.c
> +++ b/kernel/trace/trace_osnoise.c
> @@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
>  		 * differently from hwlat_detector, the osnoise tracer can run
>  		 * without a pause because preemption is on.
>  		 */
> -		if (interval < 1)
> +		if (interval < 1) {
> +			/* Let synchronize_rcu_tasks() make progress */
> +			cond_resched_tasks_rcu_qs();
>  			continue;
> +		}
>  
>  		if (msleep_interruptible(interval))
>  			break;
> -- 
> 2.29.2
> 

Re: [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu

[Daniel Bristot de Oliveira]

On 6/28/21 5:49 PM, Steven Rostedt wrote:
> From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
> 
> ftracetest triggered:
> 
>  INFO: rcu_tasks detected stalls on tasks:
>  00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
>  task:osnoise/7       state:R  running task     stack:    0 pid: 2133 ppid:     2 flags:0x00004000
>  Call Trace:
>   ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>   ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>   ? trace_hardirqs_on+0x2b/0xe0
>   ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>   ? trace_clock_local+0xc/0x20
>   ? osnoise_main+0x10e/0x450
>   ? trace_softirq_entry_callback+0x50/0x50
>   ? kthread+0x153/0x170
>   ? __kthread_bind_mask+0x60/0x60
>   ? ret_from_fork+0x22/0x30
> 
> While running osnoise tracer with other tracers that rely on
> synchronize_rcu_tasks(), where that just hung.
> 
> The reason is that osnoise_main() never schedules out if the interval
> is less than 1,

and there is no other task setting need resched (it has a regular cond_resched()
on the run_osnoise() loop).

and this will cause synchronize_rcu_tasks() to never
> return.
> 
> Fixes: bce29ac9ce0bb ("trace: Add osnoise tracer")
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

Reviewed-by: Daniel Bristot de Oliveira <bristot@redhat.com>

Thanks!
-- Daniel

> ---
>  kernel/trace/trace_osnoise.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
> index 38aa5e208ffd..556d530af805 100644
> --- a/kernel/trace/trace_osnoise.c
> +++ b/kernel/trace/trace_osnoise.c
> @@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
>  		 * differently from hwlat_detector, the osnoise tracer can run
>  		 * without a pause because preemption is on.
>  		 */
> -		if (interval < 1)
> +		if (interval < 1) {
> +			/* Let synchronize_rcu_tasks() make progress */
> +			cond_resched_tasks_rcu_qs();
>  			continue;
> +		}
>  
>  		if (msleep_interruptible(interval))
>  			break;
>