[PATCH 1/3] perf test: Fix errors when deleting data in perf record tests

[PATCH 1/3] perf test: Fix errors when deleting data in perf record tests

[Namhyung Kim]

perf record with multithreads would create a data directory instead of a
file.  But the cleanup and retry logic expect a file and try to delete
them with 'rm -f' which would fail.  Add '-r' flag to remove directories.

  rm: cannot remove '/tmp/__perf_test.perf.data.ZVHTw.old': Is a directory

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/shell/lib/perf_record.sh | 2 +-
 tools/perf/tests/shell/record.sh          | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/perf/tests/shell/lib/perf_record.sh b/tools/perf/tests/shell/lib/perf_record.sh
index 2b9e11b66dc7a801..a95fafb8b6fa8b40 100644
--- a/tools/perf/tests/shell/lib/perf_record.sh
+++ b/tools/perf/tests/shell/lib/perf_record.sh
@@ -30,7 +30,7 @@ perf_record_with_retry() {
   fi
 
   for duration in 0.01 0.1 0.3 1.0 2.0; do
-    rm -f "${perfdata}".old
+    rm -rf "${perfdata}".old
     ${cmd_prefix} "$@" -o "${perfdata}" ${testprog_base} ${duration} > "$logfile" 2>&1
     local record_exit=$?
 
diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index 4a4931cb5ba91849..cf8ba3d3411b0bcc 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -39,8 +39,8 @@ default_fd_limit=$(ulimit -Sn)
 min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
 
 cleanup() {
-  rm -f "${perfdata}"
-  rm -f "${perfdata}".old
+  rm -rf "${perfdata}"
+  rm -rf "${perfdata}".old
   rm -f "${script_output}"
   perf_record_cleanup
 
-- 
2.56.0.rc1.310.g51773c2048-goog

[PATCH 2/3] perf test: Fix record tests on Intel Broadwell

[Namhyung Kim]

I noticed it failed for leader sampling tests on Broadwell.  It's
because the period of groups members are a little bit different.  Maybe
there's an hardware issue but anyway it's close enough to treat them as
same.  I printed the value like below.

  $ perf test record
  ...
  prev: 462507, curr: 462506
  prev: 494783, curr: 494782
  prev: 454350, curr: 454349
  prev: 502782, curr: 502781
  prev: 566693, curr: 566692
  prev: 606558, curr: 606557
  prev: 559770, curr: 559769
  prev: 595737, curr: 595736
  prev: 643386, curr: 643385
  prev: 663672, curr: 663671
  prev: 620622, curr: 620621
  prev: 650419, curr: 650418
  Leader sampling [Failed inconsistent cycles count]

Let's update the logic to compare values to allow 1% of errors.

Also I think it should compare only if sees the non-leader samples so
split the check of (($index % 2)).

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/shell/record.sh | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index cf8ba3d3411b0bcc..04df35b30b0c15fd 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -550,11 +550,14 @@ test_leader_sampling() {
   while IFS= read -r line
   do
     cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}')
-    if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ]
+    if [ $(($index%2)) -ne 0 ]
     then
-      invalid_counts=$(($invalid_counts+1))
-    else
-      valid_counts=$(($valid_counts+1))
+      if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") ))
+      then
+        valid_counts=$(($valid_counts+1))
+      else
+        invalid_counts=$(($invalid_counts+1))
+      fi
     fi
     index=$(($index+1))
     prev_cycles=$cycles
-- 
2.56.0.rc1.310.g51773c2048-goog

[PATCH 3/3] perf test: Fix record tests on hybrid machines

[Namhyung Kim]

Currently the leader sampling test looks for "cycles:" string to match
the event and retrieve the value before the event name.  But it won't
work on hybrid systems as the event name is different like below.

  $ perf script | grep -m 10 brstack
      perf 3322152 1873980.978431:     625379 cpu_atom/cycles/:      558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.978431:     625379 cpu_atom/cycles/:      558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.978598:     612177 cpu_atom/cycles/:      558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.978598:     612177 cpu_atom/cycles/:      558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.978762:     591859 cpu_atom/cycles/:      558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.978762:     591859 cpu_atom/cycles/:      558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.978948:     628505 cpu_atom/cycles/:      558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.978948:     628505 cpu_atom/cycles/:      558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.979146:     669816 cpu_atom/cycles/:      558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf)
      perf 3322152 1873980.979146:     669816 cpu_atom/cycles/:      558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf)

But we can control the output format of perf script using -F option.  So
we knows where the period is in the output.  To filter by symbol name,
"ip" and "sym" fields are required as well.

  $ perf script -F period,ip,sym | grep -m 10 brstack
      625379      558200cc40cf brstack_bench
      625379      558200cc40cf brstack_bench
      612177      558200cc4082 brstack_foo
      612177      558200cc4082 brstack_foo
      591859      558200cc403d brstack_bar
      591859      558200cc403d brstack_bar
      628505      558200cc403e brstack_bar
      628505      558200cc403e brstack_bar
      669816      558200cc40e5 brstack_bench
      669816      558200cc40e5 brstack_bench

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/shell/record.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index 04df35b30b0c15fd..fa8d17e389a0116d 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -537,7 +537,7 @@ test_leader_sampling() {
     err=1
     return
   fi
-  perf script -i "${perfdata}" | grep brstack > $script_output
+  perf script -i "${perfdata}" -F period,ip,sym | grep brstack > $script_output
   # Check if the two instruction counts are equal in each record.
   # However, the throttling code doesn't consider event grouping. During throttling, only the
   # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this,
@@ -549,7 +549,7 @@ test_leader_sampling() {
   tolerance_rate=0.8
   while IFS= read -r line
   do
-    cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}')
+    cycles=$(echo $line | awk '{ print $1 }')
     if [ $(($index%2)) -ne 0 ]
     then
       if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") ))
-- 
2.56.0.rc1.310.g51773c2048-goog

Re: [PATCH 1/3] perf test: Fix errors when deleting data in perf record tests

[Ian Rogers]

On Wed, Sep 23, 2026 at 2:52 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> perf record with multithreads would create a data directory instead of a
> file.  But the cleanup and retry logic expect a file and try to delete
> them with 'rm -f' which would fail.  Add '-r' flag to remove directories.
>
>   rm: cannot remove '/tmp/__perf_test.perf.data.ZVHTw.old': Is a directory
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

Tested-by: Ian Rogers <irogers@google.com>

Thanks,
Ian

> ---
>  tools/perf/tests/shell/lib/perf_record.sh | 2 +-
>  tools/perf/tests/shell/record.sh          | 4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/tools/perf/tests/shell/lib/perf_record.sh b/tools/perf/tests/shell/lib/perf_record.sh
> index 2b9e11b66dc7a801..a95fafb8b6fa8b40 100644
> --- a/tools/perf/tests/shell/lib/perf_record.sh
> +++ b/tools/perf/tests/shell/lib/perf_record.sh
> @@ -30,7 +30,7 @@ perf_record_with_retry() {
>    fi
>
>    for duration in 0.01 0.1 0.3 1.0 2.0; do
> -    rm -f "${perfdata}".old
> +    rm -rf "${perfdata}".old
>      ${cmd_prefix} "$@" -o "${perfdata}" ${testprog_base} ${duration} > "$logfile" 2>&1
>      local record_exit=$?
>
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index 4a4931cb5ba91849..cf8ba3d3411b0bcc 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -39,8 +39,8 @@ default_fd_limit=$(ulimit -Sn)
>  min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
>
>  cleanup() {
> -  rm -f "${perfdata}"
> -  rm -f "${perfdata}".old
> +  rm -rf "${perfdata}"
> +  rm -rf "${perfdata}".old
>    rm -f "${script_output}"
>    perf_record_cleanup
>
> --
> 2.56.0.rc1.310.g51773c2048-goog
>

Re: [PATCH 3/3] perf test: Fix record tests on hybrid machines

[Ian Rogers]

On Wed, Sep 23, 2026 at 2:52 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Currently the leader sampling test looks for "cycles:" string to match
> the event and retrieve the value before the event name.  But it won't
> work on hybrid systems as the event name is different like below.
>
>   $ perf script | grep -m 10 brstack
>       perf 3322152 1873980.978431:     625379 cpu_atom/cycles/:      558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.978431:     625379 cpu_atom/cycles/:      558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.978598:     612177 cpu_atom/cycles/:      558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.978598:     612177 cpu_atom/cycles/:      558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.978762:     591859 cpu_atom/cycles/:      558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.978762:     591859 cpu_atom/cycles/:      558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.978948:     628505 cpu_atom/cycles/:      558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.978948:     628505 cpu_atom/cycles/:      558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.979146:     669816 cpu_atom/cycles/:      558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf)
>       perf 3322152 1873980.979146:     669816 cpu_atom/cycles/:      558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf)
>
> But we can control the output format of perf script using -F option.  So
> we knows where the period is in the output.  To filter by symbol name,
> "ip" and "sym" fields are required as well.
>
>   $ perf script -F period,ip,sym | grep -m 10 brstack
>       625379      558200cc40cf brstack_bench
>       625379      558200cc40cf brstack_bench
>       612177      558200cc4082 brstack_foo
>       612177      558200cc4082 brstack_foo
>       591859      558200cc403d brstack_bar
>       591859      558200cc403d brstack_bar
>       628505      558200cc403e brstack_bar
>       628505      558200cc403e brstack_bar
>       669816      558200cc40e5 brstack_bench
>       669816      558200cc40e5 brstack_bench
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

Tested-by: Ian Rogers <irogers@google.com>

Thanks,
Ian

> ---
>  tools/perf/tests/shell/record.sh | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index 04df35b30b0c15fd..fa8d17e389a0116d 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -537,7 +537,7 @@ test_leader_sampling() {
>      err=1
>      return
>    fi
> -  perf script -i "${perfdata}" | grep brstack > $script_output
> +  perf script -i "${perfdata}" -F period,ip,sym | grep brstack > $script_output
>    # Check if the two instruction counts are equal in each record.
>    # However, the throttling code doesn't consider event grouping. During throttling, only the
>    # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this,
> @@ -549,7 +549,7 @@ test_leader_sampling() {
>    tolerance_rate=0.8
>    while IFS= read -r line
>    do
> -    cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}')
> +    cycles=$(echo $line | awk '{ print $1 }')
>      if [ $(($index%2)) -ne 0 ]
>      then
>        if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") ))
> --
> 2.56.0.rc1.310.g51773c2048-goog
>

Re: [PATCH 2/3] perf test: Fix record tests on Intel Broadwell

[Ian Rogers]

On Wed, Sep 23, 2026 at 2:53 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> I noticed it failed for leader sampling tests on Broadwell.  It's
> because the period of groups members are a little bit different.  Maybe
> there's an hardware issue but anyway it's close enough to treat them as
> same.  I printed the value like below.
>
>   $ perf test record
>   ...
>   prev: 462507, curr: 462506
>   prev: 494783, curr: 494782
>   prev: 454350, curr: 454349
>   prev: 502782, curr: 502781
>   prev: 566693, curr: 566692
>   prev: 606558, curr: 606557
>   prev: 559770, curr: 559769
>   prev: 595737, curr: 595736
>   prev: 643386, curr: 643385
>   prev: 663672, curr: 663671
>   prev: 620622, curr: 620621
>   prev: 650419, curr: 650418
>   Leader sampling [Failed inconsistent cycles count]
>
> Let's update the logic to compare values to allow 1% of errors.
>
> Also I think it should compare only if sees the non-leader samples so
> split the check of (($index % 2)).
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

+Dapeng Mi

Tested-by: Ian Rogers <irogers@google.com>

Thanks,
Ian

> ---
>  tools/perf/tests/shell/record.sh | 11 +++++++----
>  1 file changed, 7 insertions(+), 4 deletions(-)
>
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index cf8ba3d3411b0bcc..04df35b30b0c15fd 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -550,11 +550,14 @@ test_leader_sampling() {
>    while IFS= read -r line
>    do
>      cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}')
> -    if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ]
> +    if [ $(($index%2)) -ne 0 ]
>      then
> -      invalid_counts=$(($invalid_counts+1))
> -    else
> -      valid_counts=$(($valid_counts+1))
> +      if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") ))
> +      then
> +        valid_counts=$(($valid_counts+1))
> +      else
> +        invalid_counts=$(($invalid_counts+1))
> +      fi
>      fi
>      index=$(($index+1))
>      prev_cycles=$cycles
> --
> 2.56.0.rc1.310.g51773c2048-goog
>

Re: [PATCH 2/3] perf test: Fix record tests on Intel Broadwell

[Mi, Dapeng]

On 9/24/2026 11:45 AM, Ian Rogers wrote:
> On Wed, Sep 23, 2026 at 2:53 PM Namhyung Kim <namhyung@kernel.org> wrote:
>> I noticed it failed for leader sampling tests on Broadwell.  It's
>> because the period of groups members are a little bit different.  Maybe
>> there's an hardware issue but anyway it's close enough to treat them as
>> same.  I printed the value like below.
>>
>>   $ perf test record
>>   ...
>>   prev: 462507, curr: 462506
>>   prev: 494783, curr: 494782
>>   prev: 454350, curr: 454349
>>   prev: 502782, curr: 502781
>>   prev: 566693, curr: 566692
>>   prev: 606558, curr: 606557
>>   prev: 559770, curr: 559769
>>   prev: 595737, curr: 595736
>>   prev: 643386, curr: 643385
>>   prev: 663672, curr: 663671
>>   prev: 620622, curr: 620621
>>   prev: 650419, curr: 650418
>>   Leader sampling [Failed inconsistent cycles count]

Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>

It looks a hardware overcount issue.  I don't see this issue on recent
platforms, like SPR+, so I suppose the issue got fixed. 

BTW, recently I found the SAMPLE_READ may report invalid counts in some
PEBS based corner sampling cases, like ACR events + SAMPLE_READ, or
multiple PEBS events simultaneously run SAMPLE_READ, I'm working on a patch
series to fix the issues.

Thanks.


>> Let's update the logic to compare values to allow 1% of errors.
>>
>> Also I think it should compare only if sees the non-leader samples so
>> split the check of (($index % 2)).
>>
>> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> +Dapeng Mi
>
> Tested-by: Ian Rogers <irogers@google.com>
>
> Thanks,
> Ian
>
>> ---
>>  tools/perf/tests/shell/record.sh | 11 +++++++----
>>  1 file changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
>> index cf8ba3d3411b0bcc..04df35b30b0c15fd 100755
>> --- a/tools/perf/tests/shell/record.sh
>> +++ b/tools/perf/tests/shell/record.sh
>> @@ -550,11 +550,14 @@ test_leader_sampling() {
>>    while IFS= read -r line
>>    do
>>      cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}')
>> -    if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ]
>> +    if [ $(($index%2)) -ne 0 ]
>>      then
>> -      invalid_counts=$(($invalid_counts+1))
>> -    else
>> -      valid_counts=$(($valid_counts+1))
>> +      if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") ))
>> +      then
>> +        valid_counts=$(($valid_counts+1))
>> +      else
>> +        invalid_counts=$(($invalid_counts+1))
>> +      fi
>>      fi
>>      index=$(($index+1))
>>      prev_cycles=$cycles
>> --
>> 2.56.0.rc1.310.g51773c2048-goog
>>