[PATCHv2 0/2] trace/hwlat: Prevent startup of multiple per-cpu threads

[PATCHv2 0/2] trace/hwlat: Prevent startup of multiple per-cpu threads

[Tero Kristo]

Hi,

v2 of this series just splits the patch into two, and adds the
reproducer script into the commit of patch #2. Separately, neither of
the patches fix the issue completely, both are needed.

-Tero

[PATCHv2 1/2] trace/hwlat: Do not wipe the contents of per-cpu thread data

[Tero Kristo]

Do not wipe the contents of the per-cpu kthread data when starting the
tracer, as this will completely forget about already running instances
and can later start new additional per-cpu threads.

Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
v2:
  * split into separate patch

 kernel/trace/trace_hwlat.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
index d440ddd5fd8b..edc26dc22c3f 100644
--- a/kernel/trace/trace_hwlat.c
+++ b/kernel/trace/trace_hwlat.c
@@ -584,9 +584,6 @@ static int start_per_cpu_kthreads(struct trace_array *tr)
 	 */
 	cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
 
-	for_each_online_cpu(cpu)
-		per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
-
 	for_each_cpu(cpu, current_mask) {
 		retval = start_cpu_kthread(cpu);
 		if (retval)
-- 
2.25.1

Re: [PATCHv2 1/2] trace/hwlat: Do not wipe the contents of per-cpu thread data

[Daniel Bristot de Oliveira]

On 3/10/23 11:04, Tero Kristo wrote:
> Do not wipe the contents of the per-cpu kthread data when starting the
> tracer, as this will completely forget about already running instances
> and can later start new additional per-cpu threads.
> 
> Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>

Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org>
-- Daniel

[PATCHv2 2/2] trace/hwlat: Do not start per-cpu thread if it is already running

[Tero Kristo]

The hwlatd tracer will end up starting multiple per-cpu threads with
the following script:

    #!/bin/sh
    cd /sys/kernel/debug/tracing
    echo 0 > tracing_on
    echo hwlat > current_tracer
    echo per-cpu > hwlat_detector/mode
    echo 100000 > hwlat_detector/width
    echo 200000 > hwlat_detector/window
    echo 1 > tracing_on

To fix the issue, check if the hwlatd thread for the cpu is already
running, before starting a new one. Along with the previous patch, this
avoids running multiple instances of the same CPU thread on the system.

Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
v2:
  * split into separate patch
  * added reproducer script into commit log

 kernel/trace/trace_hwlat.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
index edc26dc22c3f..c4945f8adc11 100644
--- a/kernel/trace/trace_hwlat.c
+++ b/kernel/trace/trace_hwlat.c
@@ -492,6 +492,10 @@ static int start_cpu_kthread(unsigned int cpu)
 {
 	struct task_struct *kthread;
 
+	/* Do not start a new hwlatd thread if it is already running */
+	if (per_cpu(hwlat_per_cpu_data, cpu).kthread)
+		return 0;
+
 	kthread = kthread_run_on_cpu(kthread_fn, NULL, cpu, "hwlatd/%u");
 	if (IS_ERR(kthread)) {
 		pr_err(BANNER "could not start sampling thread\n");
-- 
2.25.1

Re: [PATCHv2 2/2] trace/hwlat: Do not start per-cpu thread if it is already running

[Daniel Bristot de Oliveira]

On 3/10/23 11:04, Tero Kristo wrote:
> The hwlatd tracer will end up starting multiple per-cpu threads with
> the following script:
> 
>     #!/bin/sh
>     cd /sys/kernel/debug/tracing
>     echo 0 > tracing_on
>     echo hwlat > current_tracer
>     echo per-cpu > hwlat_detector/mode
>     echo 100000 > hwlat_detector/width
>     echo 200000 > hwlat_detector/window
>     echo 1 > tracing_on
> 
> To fix the issue, check if the hwlatd thread for the cpu is already
> running, before starting a new one. Along with the previous patch, this
> avoids running multiple instances of the same CPU thread on the system.
> 
> Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>

Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org>

-- Daniel