mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: James Clark <james.clark@linaro.org>
To: Ian Rogers <irogers@google.com>
Cc: adrian.hunter@intel.com, gmx@google.com, jolsa@kernel.org,
	linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org,
	mingo@redhat.com, peterz@infradead.org, acme@kernel.org,
	namhyung@kernel.org
Subject: Re: [PATCH v19 5/5] perf test: Add inject ASLR test
Date: Thu, 11 Jun 2026 09:32:53 +0100	[thread overview]
Message-ID: <c92e86bc-e596-487e-918d-bc9d8377910b@linaro.org> (raw)
In-Reply-To: <CAP-5=fXvqFVRMZvHcHFO04TUf-o9Z4i1jHa0Tgw_qXf4Av+dmw@mail.gmail.com>



On 10/06/2026 5:15 pm, Ian Rogers wrote:
> On Wed, Jun 10, 2026 at 6:26 AM James Clark <james.clark@linaro.org> wrote:
>>
>>
>>
>> On 08/06/2026 6:48 am, Ian Rogers wrote:
>>> Add a new shell test  to verify the feature. The test covers:
>>> - Basic address remapping for user space samples.
>>> - Pipe mode coverage for piped into.
>>> - Callchain address remapping.
>>> - Consistency of output before and after injection.
>>> - Pipe mode report consistency.
>>> - Dropping of samples that leak ASLR info (physical addresses).
>>> - Kernel address remapping (utilizing a dedicated kernel-intensive VFS
>>>     dd workload to guarantee continuous timer interrupts sampling flow
>>>     inside kernel privilege states).
>>> - Kernel report consistency with address normalization.
>>>
>>> The test suite is hardened with global 'set -o pipefail' assertions
>>> to catch pipeline failures, stream-consuming awk processors to handle
>>> SIGPIPE signals, and a dedicated pipe output scenario validating raw
>>> 'perf inject -o -' stdout streams.
>>>
>>> Note on kernel DSO normalization in the test script:
>>> The test script deliberately normalizes all kernel DSOs to a generic
>>> [kernel] tag before diffing, as obfuscating physical kernel addresses
>>> forces perf report to occasionally shift samples between individual
>>> modules and [kernel.kallsyms] due to the lack of valid host module
>>> boundary maps.
>>>
>>> Signed-off-by: Ian Rogers <irogers@google.com>
>>> Assisted-by: Antigravity:gemini-3.1-pro
>>
>> Hi Ian,
>>
>> All the tests pass for me now on x86, and the previous Arm failures are
>> fixed. But I still have the slowness issue  on Arm (takes almost an hour
>> to complete) and one new test failure.
>>
>> I narrowed the slowness issue down to check_invariants(). If I compile
>> without DEBUG=1 then it's the same speed as x86. Here's the stack where
>> it spends all of its time:
>>
>>     maps__split_kallsyms() at util/symbol.c:1105
>>     __dso__load_kallsyms() at util/symbol.c:1649
>>     dso__load_kallsyms() at util/symbol.c:1655
>>     dso__load_kernel_sym() at util/symbol.c:2244
>>     dso__load() at util/symbol.c:1840
>>     map__load() at util/map.c:351
>>     thread__find_map() at util/event.c:744
>>     machine__resolve()at util/event.c:818
>>     process_sample_event() at builtin-script.c:2695
>>     evlist__deliver_sample() at util/session.c:1807
>>     machines__deliver_event() at util/session.c:1995
>>     perf_session__deliver_event() at util/session.c:2226
>>     ordered_events__deliver_event() at util/session.c:134
>>     do_flush() at util/ordered-events.c:245
>>     __ordered_events__flush()at util/ordered-events.c:324
>>     ordered_events__flush() at util/ordered-events.c:342
>>     __perf_session__process_events() at util/session.c:3578
>>     perf_session__process_events() at util/session.c:3745
>>     __cmd_script() at builtin-script.c:3255
>>     cmd_script() at builtin-script.c:4603
>>     run_builtin() at perf.c:348
>>     handle_internal_command() at perf.c:398
>>     run_argv() at perf.c:442
>>     main() at perf.c:549
>>
>> The other issue is that test_kernel_report_aslr() fails.
>> report_kernel1.clean looks like:
>>
>>         66.72%  dd       [kernel.kallsyms]  [k] chacha_permute
>>        7.75%  dd       [kernel.kallsyms]  [k] chacha_block_generic
>>        6.14%  dd       [kernel.kallsyms]  [k] _copy_to_iter
>>        3.88%  dd       [kernel.kallsyms]  [k] lock_acquire
>>        2.91%  dd       [kernel.kallsyms]  [k] lock_release
>>        2.75%  dd       [kernel.kallsyms]  [k] rcu_is_watching
>>        2.58%  dd       [kernel.kallsyms]  [k] __arch_copy_to_user
>>        2.26%  dd       [kernel.kallsyms]  [k] __might_resched
>>        1.94%  dd       [kernel.kallsyms]  [k] get_random_bytes_user
>>        1.29%  dd       [kernel.kallsyms]  [k] __might_fault
>>        0.81%  dd       [kernel.kallsyms]  [k] debug_lockdep_rcu_enabled
>>        0.48%  dd       [kernel.kallsyms]  [k] __might_sleep
>>        0.16%  dd       [kernel.kallsyms]  [k] kmem_cache_free
>>        0.16%  dd       [kernel.kallsyms]  [k] seqcount_lockdep_reader_access
>>        0.16%  dd       [kernel.kallsyms]  [k] set_pte_range
>>
>> But report_kernel2.clean is maybe 10x longer and doesn't have any symbols:
>>
>>        8.08%  dd       [unknown]          [k] 0xffff8000108416ec
>>        2.75%  dd       [unknown]          [k] 0xffff80001084170c
>>        1.94%  dd       [unknown]          [k] 0xffff800010176360
>>        1.94%  dd       [unknown]          [k] 0xffff80001084156c
>>        1.45%  dd       [unknown]          [k] 0xffff8000101be4bc
>>        ...
>>
>> I think this could be related to why it gets stuck in
>> maps__split_kallsyms() doing check_invariants(). I'm not sure if I can
>> do anything to help debug, or if it's working for you on Arm so we can
>> compare our setups?
> 
> So my focus has been making this solid on x86 and I hadn't been
> testing on ARM. Kernel symbolization is always a pain. Perhaps for now
> we should just skip the ASLR testing on ARM, and fix it in follow-up
> patches.
> 
> Thanks,
> Ian
> 

Makes sense, you'll probably have to skip both test_kernel_report_aslr() 
and test_kernel_aslr() even though the second one passes eventually 
because it takes so long.


>>> ---
>>>    tools/perf/tests/shell/inject_aslr.sh | 525 ++++++++++++++++++++++++++
>>>    1 file changed, 525 insertions(+)
>>>    create mode 100755 tools/perf/tests/shell/inject_aslr.sh
>>>
>>> diff --git a/tools/perf/tests/shell/inject_aslr.sh b/tools/perf/tests/shell/inject_aslr.sh
>>> new file mode 100755
>>> index 000000000000..4af21ed2ce9a
>>> --- /dev/null
>>> +++ b/tools/perf/tests/shell/inject_aslr.sh
>>> @@ -0,0 +1,525 @@
>>> +#!/bin/bash
>>> +# SPDX-License-Identifier: GPL-2.0
>>> +# perf inject --aslr test
>>> +
>>> +set -e
>>> +set -o pipefail
>>> +
>>> +shelldir=$(dirname "$0")
>>> +# shellcheck source=lib/perf_has_symbol.sh
>>> +. "${shelldir}"/lib/perf_has_symbol.sh
>>> +
>>> +sym="noploop"
>>> +
>>> +skip_test_missing_symbol ${sym}
>>> +
>>> +# Create global temp directory
>>> +temp_dir=$(mktemp -d /tmp/perf-test-aslr.XXXXXXXXXX)
>>> +
>>> +prog="perf test -w noploop"
>>> +[ "$(uname -m)" = "s390x" ] && prog="$prog 3"
>>> +err=0
>>> +kprog="dd if=/dev/urandom of=/dev/null bs=1M count=50"
>>> +
>>> +cleanup() {
>>> +  local exit_code=${1:-$?}
>>> +  trap - EXIT TERM INT
>>> +  if [ "${exit_code}" -ne 0 ] || [ "${err}" -ne 0 ]; then
>>> +    echo "Test failed! Preserving temp directory: ${temp_dir}"
>>> +    return
>>> +  fi
>>> +  # Check if temp_dir is set and looks sane before removing
>>> +  if [[ "${temp_dir}" =~ ^/tmp/perf-test-aslr\. ]]; then
>>> +    rm -rf "${temp_dir}"
>>> +  fi
>>> +}
>>> +
>>> +trap_cleanup() {
>>> +  local exit_code=$?
>>> +  echo "Unexpected signal in ${FUNCNAME[1]}"
>>> +  cleanup ${exit_code}
>>> +  exit ${exit_code}
>>> +}
>>> +trap trap_cleanup EXIT TERM INT
>>> +
>>> +get_noploop_addr() {
>>> +  local file=$1
>>> +  perf script -i "$file" | awk '
>>> +    BEGIN { found=0 }
>>> +    {
>>> +      for (i=1; i<=NF; i++) {
>>> +        if ($i ~ /noploop\+/) {
>>> +          if (!found) {
>>> +            print $(i-1)
>>> +            found=1
>>> +          }
>>> +        }
>>> +      }
>>> +    }'
>>> +}
>>> +
>>> +test_basic_aslr() {
>>> +  echo "Test basic ASLR remapping"
>>> +  local data
>>> +  data=$(mktemp "${temp_dir}/perf.data.basic.XXXXXX")
>>> +  local data2
>>> +  data2=$(mktemp "${temp_dir}/perf.data2.basic.XXXXXX")
>>> +
>>> +  perf record -e task-clock:u -o "${data}" ${prog}
>>> +  perf inject -v --aslr -i "${data}" -o "${data2}"
>>> +
>>> +  orig_addr=$(get_noploop_addr "${data}")
>>> +  new_addr=$(get_noploop_addr "${data2}")
>>> +
>>> +  echo "Basic ASLR: orig_addr=$orig_addr, new_addr=$new_addr"
>>> +
>>> +  if [ -z "$orig_addr" ]; then
>>> +    echo "Basic ASLR test [Failed - no noploop samples in original file]"
>>> +    err=1
>>> +  elif [ -z "$new_addr" ]; then
>>> +    echo "Basic ASLR test [Failed - could not find remapped address]"
>>> +    err=1
>>> +  elif [ "$orig_addr" = "$new_addr" ]; then
>>> +    echo "Basic ASLR test [Failed - addresses are not remapped]"
>>> +    err=1
>>> +  else
>>> +    echo "Basic ASLR test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_pipe_aslr() {
>>> +  echo "Test pipe mode ASLR remapping"
>>> +  local data
>>> +  data=$(mktemp "${temp_dir}/perf.data.pipe.XXXXXX")
>>> +  local data2
>>> +  data2=$(mktemp "${temp_dir}/perf.data2.pipe.XXXXXX")
>>> +
>>> +  # Use tee to save the original pipe data for comparison
>>> +  perf record -e task-clock:u -o - ${prog} | tee "${data}" | perf inject --aslr -o "${data2}"
>>> +
>>> +  orig_addr=$(get_noploop_addr "${data}")
>>> +  new_addr=$(get_noploop_addr "${data2}")
>>> +
>>> +  echo "Pipe ASLR: orig_addr=$orig_addr, new_addr=$new_addr"
>>> +
>>> +  if [ -z "$orig_addr" ]; then
>>> +    echo "Pipe ASLR test [Failed - no noploop samples in original file]"
>>> +    err=1
>>> +  elif [ -z "$new_addr" ]; then
>>> +    echo "Pipe ASLR test [Failed - could not find remapped address]"
>>> +    err=1
>>> +  elif [ "$orig_addr" = "$new_addr" ]; then
>>> +    echo "Pipe ASLR test [Failed - addresses are not remapped]"
>>> +    err=1
>>> +  else
>>> +    echo "Pipe ASLR test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_callchain_aslr() {
>>> +  echo "Test Callchain ASLR remapping"
>>> +  local data
>>> +  data=$(mktemp "${temp_dir}/perf.data.callchain.XXXXXX")
>>> +  local data2
>>> +  data2=$(mktemp "${temp_dir}/perf.data2.callchain.XXXXXX")
>>> +
>>> +  perf record -g -e task-clock:u -o "${data}" ${prog}
>>> +  perf inject --aslr -i "${data}" -o "${data2}"
>>> +
>>> +  orig_addr=$(get_noploop_addr "${data}")
>>> +  new_addr=$(get_noploop_addr "${data2}")
>>> +
>>> +  echo "Callchain ASLR: orig_addr=$orig_addr, new_addr=$new_addr"
>>> +
>>> +  if [ -z "$orig_addr" ]; then
>>> +    echo "Callchain ASLR test [Failed - no noploop samples in original file]"
>>> +    err=1
>>> +  elif [ -z "$new_addr" ]; then
>>> +    echo "Callchain ASLR test [Failed - could not find remapped address]"
>>> +    err=1
>>> +  elif [ "$orig_addr" = "$new_addr" ]; then
>>> +    echo "Callchain ASLR test [Failed - addresses are not remapped]"
>>> +    err=1
>>> +  else
>>> +    # Extract callchain addresses (indented lines starting with hex addresses)
>>> +    orig_callchain=$(perf script -i "${data}" | awk '/^[[:space:]]+[0-9a-f]+/ {print $1}')
>>> +    new_callchain=$(perf script -i "${data2}" | awk '/^[[:space:]]+[0-9a-f]+/ {print $1}')
>>> +
>>> +    if [ -z "$orig_callchain" ]; then
>>> +      echo "Callchain ASLR test [Failed - no callchain samples in original file]"
>>> +      err=1
>>> +    elif [ -z "$new_callchain" ]; then
>>> +      echo "Callchain ASLR test [Failed - callchain data was dropped]"
>>> +      err=1
>>> +    elif [ "$orig_callchain" = "$new_callchain" ]; then
>>> +      echo "Callchain ASLR test [Failed - callchain addresses were not remapped]"
>>> +      err=1
>>> +    else
>>> +      echo "Callchain ASLR test [Success]"
>>> +    fi
>>> +  fi
>>> +}
>>> +
>>> +test_report_aslr() {
>>> +  echo "Test perf report consistency"
>>> +  local data
>>> +  data=$(mktemp "${temp_dir}/perf.data.report.XXXXXX")
>>> +  local data2
>>> +  data2=$(mktemp "${temp_dir}/perf.data2.report.XXXXXX")
>>> +  local data_clean
>>> +  data_clean=$(mktemp "${temp_dir}/perf.data.clean.XXXXXX")
>>> +
>>> +  perf record -e task-clock:u -o "${data}" ${prog}
>>> +  # Use -b to inject build-ids and force ordered events processing in both
>>> +  perf inject -b -i "${data}" -o "${data_clean}"
>>> +  perf inject -v -b --aslr -i "${data}" -o "${data2}"
>>> +
>>> +  local report1="${temp_dir}/report1_basic"
>>> +  local report2="${temp_dir}/report2_basic"
>>> +  local report1_clean="${temp_dir}/report1_basic.clean"
>>> +  local report2_clean="${temp_dir}/report2_basic.clean"
>>> +  local diff_file="${temp_dir}/diff_basic"
>>> +
>>> +  perf report -i "${data_clean}" --stdio > "${report1}"
>>> +  perf report -i "${data2}" --stdio > "${report2}"
>>> +
>>> +  # Strip headers and compare lines with percentages
>>> +  grep '%' "${report1}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | sort > "${report1_clean}" || true
>>> +  grep '%' "${report2}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | sort > "${report2_clean}" || true
>>> +
>>> +  diff -u -w "${report1_clean}" "${report2_clean}" > "${diff_file}" || true
>>> +
>>> +  if [ ! -s "${report1_clean}" ]; then
>>> +    echo "Report ASLR test [Failed - no samples captured]"
>>> +    err=1
>>> +  elif [ -s "${diff_file}" ]; then
>>> +    echo "Report ASLR test [Failed - reports differ]"
>>> +    echo "Showing first 20 lines of diff:"
>>> +    head -n 20 "${diff_file}"
>>> +    err=1
>>> +  else
>>> +    echo "Report ASLR test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_pipe_report_aslr() {
>>> +  echo "Test pipe mode perf report consistency"
>>> +  local data
>>> +  data=$(mktemp "${temp_dir}/perf.data.pipe_report.XXXXXX")
>>> +  local data2
>>> +  data2=$(mktemp "${temp_dir}/perf.data2.pipe_report.XXXXXX")
>>> +  local data_clean
>>> +  data_clean=$(mktemp "${temp_dir}/perf.data.clean.XXXXXX")
>>> +
>>> +  # Use tee to save the original pipe data, then process it with inject -b
>>> +  perf record -e task-clock:u -o - ${prog} | \
>>> +    tee "${data}" | \
>>> +    perf inject -b --aslr -o "${data2}"
>>> +  perf inject -b -i "${data}" -o "${data_clean}"
>>> +
>>> +  local report1="${temp_dir}/report1_pipe"
>>> +  local report2="${temp_dir}/report2_pipe"
>>> +  local report1_clean="${temp_dir}/report1_pipe.clean"
>>> +  local report2_clean="${temp_dir}/report2_pipe.clean"
>>> +  local diff_file="${temp_dir}/diff_pipe"
>>> +
>>> +  perf report -i "${data_clean}" --stdio > "${report1}"
>>> +  perf report -i "${data2}" --stdio > "${report2}"
>>> +
>>> +  # Strip headers and compare lines with percentages
>>> +  grep '%' "${report1}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | sort > "${report1_clean}" || true
>>> +  grep '%' "${report2}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | sort > "${report2_clean}" || true
>>> +
>>> +  diff -u -w "${report1_clean}" "${report2_clean}" > "${diff_file}" || true
>>> +
>>> +  if [ ! -s "${report1_clean}" ]; then
>>> +    echo "Pipe Report ASLR test [Failed - no samples captured]"
>>> +    err=1
>>> +  elif [ -s "${diff_file}" ]; then
>>> +    echo "Pipe Report ASLR test [Failed - reports differ]"
>>> +    echo "Showing first 20 lines of diff:"
>>> +    head -n 20 "${diff_file}"
>>> +    err=1
>>> +  else
>>> +    echo "Pipe Report ASLR test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_pipe_out_report_aslr() {
>>> +  echo "Test pipe output mode perf report consistency"
>>> +  local data
>>> +  data=$(mktemp "${temp_dir}/perf.data.pipe_out_report.XXXXXX")
>>> +  local data_clean
>>> +  data_clean=$(mktemp "${temp_dir}/perf.data.clean.XXXXXX")
>>> +
>>> +  perf record -e task-clock:u -o "${data}" ${prog}
>>> +  perf inject -b -i "${data}" -o "${data_clean}"
>>> +
>>> +  local report1="${temp_dir}/report1_pipe_out"
>>> +  local report2="${temp_dir}/report2_pipe_out"
>>> +  local report1_clean="${temp_dir}/report1_pipe_out.clean"
>>> +  local report2_clean="${temp_dir}/report2_pipe_out.clean"
>>> +  local diff_file="${temp_dir}/diff_pipe_out"
>>> +
>>> +  perf report -i "${data_clean}" --stdio > "${report1}"
>>> +  perf inject -b --aslr -i "${data}" -o - | perf report -i - --stdio > "${report2}"
>>> +
>>> +  # Strip headers and compare lines with percentages
>>> +  grep '%' "${report1}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | sort > "${report1_clean}" || true
>>> +  grep '%' "${report2}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | sort > "${report2_clean}" || true
>>> +
>>> +  diff -u -w "${report1_clean}" "${report2_clean}" > "${diff_file}" || true
>>> +
>>> +  if [ ! -s "${report1_clean}" ]; then
>>> +    echo "Pipe Output Report ASLR test [Failed - no samples captured]"
>>> +    err=1
>>> +  elif [ -s "${diff_file}" ]; then
>>> +    echo "Pipe Output Report ASLR test [Failed - reports differ]"
>>> +    echo "Showing first 20 lines of diff:"
>>> +    head -n 20 "${diff_file}"
>>> +    err=1
>>> +  else
>>> +    echo "Pipe Output Report ASLR test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_dropped_samples() {
>>> +  echo "Test dropped samples (phys-data)"
>>> +  local data
>>> +  data=$(mktemp "${temp_dir}/perf.data.dropped.XXXXXX")
>>> +  local data2
>>> +  data2=$(mktemp "${temp_dir}/perf.data2.dropped.XXXXXX")
>>> +
>>> +  # Check if --phys-data is supported by recording a short run
>>> +  if ! perf record -e task-clock:u --phys-data -o "${data}" -- sleep 0.1 > /dev/null 2>&1; then
>>> +    echo "Skipping dropped samples test as --phys-data is not supported"
>>> +    return
>>> +  fi
>>> +
>>> +  perf record -e task-clock:u --phys-data -o "${data}" ${prog}
>>> +  perf inject --aslr -i "${data}" -o "${data2}"
>>> +
>>> +  # Verify that the original file actually contained samples!
>>> +  orig_samples=$(perf script -i "${data}" | wc -l)
>>> +  if [ "$orig_samples" -eq 0 ]; then
>>> +    echo "Dropped samples test [Failed - no samples in original file]"
>>> +    err=1
>>> +  else
>>> +    # Verify that samples are dropped.
>>> +    samples_count=$(perf script -i "${data2}" | wc -l)
>>> +
>>> +    if [ "$samples_count" -gt 0 ]; then
>>> +      echo "Dropped samples test [Failed - samples were not dropped]"
>>> +      err=1
>>> +    else
>>> +      echo "Dropped samples test [Success]"
>>> +    fi
>>> +  fi
>>> +}
>>> +
>>> +test_kernel_aslr() {
>>> +  echo "Test kernel ASLR remapping"
>>> +  local kdata
>>> +  kdata=$(mktemp "${temp_dir}/perf.data.kernel.XXXXXX")
>>> +  local kdata2
>>> +  kdata2=$(mktemp "${temp_dir}/perf.data2.kernel.XXXXXX")
>>> +  local log_file
>>> +  log_file=$(mktemp "${temp_dir}/kernel_record.log.XXXXXX")
>>> +
>>> +  # Try to record kernel samples
>>> +  if ! perf record -e task-clock:k -o "${kdata}" ${kprog} > "${log_file}" 2>&1; then
>>> +    echo "Skipping kernel ASLR test as recording failed (maybe no permissions)"
>>> +    return
>>> +  fi
>>> +
>>> +  # Check for warning about kernel map restriction
>>> +  if grep -q "Couldn't record kernel reference relocation symbol" "${log_file}"; then
>>> +    echo "Skipping kernel ASLR test as kernel map could not be recorded (permissions restricted)"
>>> +    return
>>> +  fi
>>> +
>>> +  perf inject -v --aslr -i "${kdata}" -o "${kdata2}"
>>> +
>>> +  # Check if kernel addresses are remapped.
>>> +  # Find the field that ends with :k: (the event name) and take the next field!
>>> +  orig_addr=$(perf script -i "${kdata}" | awk '
>>> +    BEGIN { found=0 }
>>> +    {
>>> +      for (i=1; i<NF; i++) {
>>> +        if ($i ~ /:[k]+:?$/) {
>>> +          if (!found) {
>>> +            print $(i+1)
>>> +            found=1
>>> +          }
>>> +        }
>>> +      }
>>> +    }')
>>> +  new_addr=$(perf script -i "${kdata2}" | awk '
>>> +    BEGIN { found=0 }
>>> +    {
>>> +      for (i=1; i<NF; i++) {
>>> +        if ($i ~ /:[k]+:?$/) {
>>> +          if (!found) {
>>> +            print $(i+1)
>>> +            found=1
>>> +          }
>>> +        }
>>> +      }
>>> +    }')
>>> +
>>> +  echo "Kernel ASLR: orig_addr=$orig_addr, new_addr=$new_addr"
>>> +
>>> +  if [ -z "$orig_addr" ]; then
>>> +    echo "Kernel ASLR test [Failed - no kernel samples in original file]"
>>> +    err=1
>>> +  elif [ -z "$new_addr" ]; then
>>> +    echo "Kernel ASLR test [Failed - could not find remapped address]"
>>> +    err=1
>>> +  elif [ "$orig_addr" = "$new_addr" ]; then
>>> +    echo "Kernel ASLR test [Failed - addresses are not remapped]"
>>> +    err=1
>>> +  else
>>> +    echo "Kernel ASLR test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_kernel_report_aslr() {
>>> +  echo "Test kernel perf report consistency"
>>> +  local kdata
>>> +  kdata=$(mktemp "${temp_dir}/perf.data.kernel_report.XXXXXX")
>>> +  local kdata2
>>> +  kdata2=$(mktemp "${temp_dir}/perf.data2.kernel_report.XXXXXX")
>>> +  local data_clean
>>> +  data_clean=$(mktemp "${temp_dir}/perf.data.clean.XXXXXX")
>>> +  local log_file
>>> +  log_file=$(mktemp "${temp_dir}/kernel_report_record.log.XXXXXX")
>>> +
>>> +  # Try to record kernel samples
>>> +  if ! perf record -e task-clock:k -o "${kdata}" ${kprog} > "${log_file}" 2>&1; then
>>> +    echo "Skipping kernel report test as recording failed (maybe no permissions)"
>>> +    return
>>> +  fi
>>> +
>>> +  # Check for warning about kernel map restriction
>>> +  if grep -q "Couldn't record kernel reference relocation symbol" "${log_file}"; then
>>> +    echo "Skipping kernel report test as kernel map could not be recorded (permissions restricted)"
>>> +    return
>>> +  fi
>>> +
>>> +  # Use -b to inject build-ids and force ordered events processing in both
>>> +  perf inject -b -i "${kdata}" -o "${data_clean}"
>>> +  perf inject -v -b --aslr -i "${kdata}" -o "${kdata2}"
>>> +
>>> +  local report1="${temp_dir}/report_kernel1"
>>> +  local report2="${temp_dir}/report_kernel2"
>>> +  local report1_clean="${temp_dir}/report_kernel1.clean"
>>> +  local report2_clean="${temp_dir}/report_kernel2.clean"
>>> +
>>> +  perf report -i "${data_clean}" --stdio > "${report1}"
>>> +  perf report -i "${kdata2}" --stdio > "${report2}"
>>> +
>>> +  # Strip headers and compare lines with percentages
>>> +  grep '%' "${report1}" | grep -v '^#' > "${report1_clean}" || true
>>> +  grep '%' "${report2}" | grep -v '^#' > "${report2_clean}" || true
>>> +
>>> +  # Normalize kernel DSOs and addresses in clean reports
>>> +  # This allows kernel modules to be either a module or kernel.kallsyms
>>> +  local report1_norm="${temp_dir}/report_kernel1.norm"
>>> +  local report2_norm="${temp_dir}/report_kernel2.norm"
>>> +  local diff_file="${temp_dir}/diff_kernel"
>>> +
>>> +  grep -v -E '0x[0-9a-f]{8,}|0000000000000000' "${report1_clean}" | \
>>> +    awk '{gsub(/\[[a-zA-Z0-9_.-]{2,}\](\.[a-zA-Z0-9_]+)?/, "[kernel]", $0); print}' | \
>>> +    sort > "${report1_norm}" || true
>>> +  grep -v -E '0x[0-9a-f]{8,}|0000000000000000' "${report2_clean}" | \
>>> +    awk '{gsub(/\[[a-zA-Z0-9_.-]{2,}\](\.[a-zA-Z0-9_]+)?/, "[kernel]", $0); print}' | \
>>> +    sort > "${report2_norm}" || true
>>> +
>>> +  diff -u -w "${report1_norm}" "${report2_norm}" > "${diff_file}" || true
>>> +
>>> +  if [ ! -s "${report1_norm}" ]; then
>>> +    echo "Kernel Report ASLR test [Failed - no samples captured]"
>>> +    err=1
>>> +  elif [ -s "${diff_file}" ]; then
>>> +    echo "Kernel Report ASLR test [Failed - reports differ]"
>>> +    echo "Showing first 20 lines of diff:"
>>> +    head -n 20 "${diff_file}"
>>> +    err=1
>>> +  else
>>> +    echo "Kernel Report ASLR test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_regs_stripping() {
>>> +  echo "Test user register stripping"
>>> +  local rdata="${temp_dir}/perf.data.regs"
>>> +  local rdata2="${temp_dir}/perf.data.regs.injected"
>>> +  local rdata_clean="${temp_dir}/perf.data.regs.clean"
>>> +
>>> +  if ! perf record -e cycles:u --user-regs -o "${rdata}" ${prog} > /dev/null 2>&1; then
>>> +    echo "Skipping user registers test as recording failed (unsupported flag/platform)"
>>> +    return
>>> +  fi
>>> +
>>> +  perf inject -b -i "${rdata}" -o "${rdata_clean}"
>>> +  perf inject -v -b --aslr -i "${rdata}" -o "${rdata2}"
>>> +
>>> +  local report1="${temp_dir}/report_regs1"
>>> +  local report2="${temp_dir}/report_regs2"
>>> +  local report1_clean="${temp_dir}/report_regs1.clean"
>>> +  local report2_clean="${temp_dir}/report_regs2.clean"
>>> +  local diff_file="${temp_dir}/diff_regs"
>>> +
>>> +  perf report -i "${rdata_clean}" --stdio > "${report1}" 2>/dev/null || true
>>> +  perf report -i "${rdata2}" --stdio > "${report2}" 2>/dev/null || true
>>> +
>>> +  grep '%' "${report1}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | \
>>> +    sort > "${report1_clean}" || true
>>> +  grep '%' "${report2}" | grep -v '^#' | \
>>> +    grep -v -E '0x[0-9a-f]{8,}|0000000000000000' | \
>>> +    sort > "${report2_clean}" || true
>>> +
>>> +  diff -u -w "${report1_clean}" "${report2_clean}" > "${diff_file}" || true
>>> +
>>> +  if [ ! -s "${report1_clean}" ]; then
>>> +    echo "User registers stripping test [Failed - profile trace starved/empty]"
>>> +    err=1
>>> +    return
>>> +  elif [ -s "${diff_file}" ]; then
>>> +    echo "User registers stripping test [Failed - report parsing differs]"
>>> +    echo "Showing first 20 lines of diff:"
>>> +    head -n 20 "${diff_file}"
>>> +    err=1
>>> +    return
>>> +  fi
>>> +
>>> +  local script_dump="${temp_dir}/script_regs_dump"
>>> +  perf script -D -i "${rdata2}" > "${script_dump}" 2>/dev/null || true
>>> +  if grep -q "user regs:" "${script_dump}"; then
>>> +    echo "User registers stripping test [Failed - register dumps still present]"
>>> +    err=1
>>> +  else
>>> +    echo "User registers stripping test [Success]"
>>> +  fi
>>> +}
>>> +
>>> +test_basic_aslr
>>> +test_pipe_aslr
>>> +test_callchain_aslr
>>> +test_report_aslr
>>> +test_pipe_report_aslr
>>> +test_pipe_out_report_aslr
>>> +test_dropped_samples
>>> +test_kernel_aslr
>>> +test_kernel_report_aslr
>>> +test_regs_stripping
>>> +
>>> +cleanup ${err}
>>> +exit $err
>>


  reply	other threads:[~2026-06-11  8:32 UTC|newest]

Thread overview: 137+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-04-24 22:05 [PATCH v1 1/2] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-04-24 22:05 ` [PATCH v1 2/2] perf test: Add inject ASLR test Ian Rogers
2026-04-25  2:05 ` [PATCH v2 1/2] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-04-25  2:05   ` [PATCH v2 2/2] perf test: Add inject ASLR test Ian Rogers
2026-05-04  3:51   ` [PATCH v3 0/4] perf tools: Add inject --aslr feature and prerequisite robustness fixes Ian Rogers
2026-05-04  3:51     ` [PATCH v3 1/4] perf sched: Add missing mmap2 handler in timehist Ian Rogers
2026-05-04  3:51     ` [PATCH v3 2/4] perf tool: Fix missing schedstat delegates and dont_split_sample_group in delegate_tool Ian Rogers
2026-05-04  3:51     ` [PATCH v3 3/4] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-05-04  3:51     ` [PATCH v3 4/4] perf test: Add inject ASLR test Ian Rogers
2026-05-04  7:29     ` [PATCH v4 0/4] perf tools: Add inject --aslr feature and prerequisite robustness fixes Ian Rogers
2026-05-04  7:29       ` [PATCH v4 1/4] perf sched: Add missing mmap2 handler in timehist Ian Rogers
2026-05-04  7:29       ` [PATCH v4 2/4] perf tool: Fix missing schedstat delegates and dont_split_sample_group in delegate_tool Ian Rogers
2026-05-04  7:29       ` [PATCH v4 3/4] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-05-04  7:29       ` [PATCH v4 4/4] perf test: Add inject ASLR test Ian Rogers
2026-05-04  8:23       ` [PATCH v4 0/4] perf tools: Add inject --aslr feature and prerequisite robustness fixes Ian Rogers
2026-05-06  0:45       ` [PATCH v5 0/5] " Ian Rogers
2026-05-06  0:45         ` [PATCH v5 1/5] perf sched: Add missing mmap2 handler in timehist Ian Rogers
2026-05-06 13:22           ` Arnaldo Carvalho de Melo
2026-05-06 16:16             ` Ian Rogers
2026-05-06  0:45         ` [PATCH v5 2/5] perf tool: Fix missing schedstat delegates and dont_split_sample_group in delegate_tool Ian Rogers
2026-05-06  0:45         ` [PATCH v5 3/5] perf symbols: Fix map removal sequence inside dso__process_kernel_symbol() Ian Rogers
2026-05-06  0:45         ` [PATCH v5 4/5] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-05-06 18:52           ` Namhyung Kim
2026-05-06 20:01             ` Ian Rogers
2026-05-06  0:45         ` [PATCH v5 5/5] perf test: Add inject ASLR test Ian Rogers
2026-05-07 15:58           ` James Clark
2026-05-07 16:17             ` Ian Rogers
2026-05-08 10:42               ` James Clark
2026-05-08 10:49                 ` James Clark
2026-05-08  8:27         ` [PATCH v6 0/6] perf tools: Add inject --aslr feature and prerequisite robustness fixes Ian Rogers
2026-05-08  8:27           ` [PATCH v6 1/6] perf sched: Add missing mmap2 handler in timehist Ian Rogers
2026-05-08  8:27           ` [PATCH v6 2/6] perf tool: Missing delegate_tool schedstat delegates and dont_split_sample_group Ian Rogers
2026-05-08  8:27           ` [PATCH v6 3/6] perf maps: Add maps__mutate_mapping Ian Rogers
2026-05-08 10:57             ` James Clark
2026-05-11  7:07             ` Namhyung Kim
2026-06-10 10:40               ` James Clark
2026-05-08  8:27           ` [PATCH v6 4/6] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-05-11  7:32             ` Namhyung Kim
2026-05-08  8:27           ` [PATCH v6 5/6] perf test: Add inject ASLR test Ian Rogers
2026-05-08 13:29             ` James Clark
2026-05-08 14:29               ` James Clark
2026-05-11  7:34             ` Namhyung Kim
2026-05-08  8:27           ` [PATCH v6 6/6] perf aslr: Strip sample registers Ian Rogers
2026-05-19  8:08           ` [PATCH v7 0/4] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-05-19  8:08             ` [PATCH v7 1/4] perf maps: Add maps__mutate_mapping Ian Rogers
2026-05-19  8:08             ` [PATCH v7 2/4] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-05-19  8:08             ` [PATCH v7 3/4] perf test: Add inject ASLR test Ian Rogers
2026-05-19  8:08             ` [PATCH v7 4/4] perf aslr: Strip sample registers Ian Rogers
2026-05-20  6:30             ` [PATCH v8 0/4] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-05-20  6:30               ` [PATCH v8 1/4] perf maps: Add maps__mutate_mapping Ian Rogers
2026-05-20  6:30               ` [PATCH v8 2/4] perf inject/aslr: Add aslr tool to remap/obfuscate virtual addresses Ian Rogers
2026-05-23 14:44                 ` kernel test robot
2026-05-20  6:30               ` [PATCH v8 3/4] perf test: Add inject ASLR test Ian Rogers
2026-05-20  6:30               ` [PATCH v8 4/4] perf aslr: Strip sample registers Ian Rogers
2026-06-04 17:28               ` [PATCH v9 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-04 17:28                 ` [PATCH v9 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-04 17:28                 ` [PATCH v9 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-04 17:28                 ` [PATCH v9 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-04 17:28                 ` [PATCH v9 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-04 17:28                 ` [PATCH v9 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-05  6:06                 ` [PATCH v10 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-05  6:06                   ` [PATCH v10 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-05  6:06                   ` [PATCH v10 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-05  6:06                   ` [PATCH v10 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-05  6:06                   ` [PATCH v10 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-05  6:06                   ` [PATCH v10 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-05 18:52                   ` [PATCH v11 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-05 18:52                     ` [PATCH v11 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-05 18:52                     ` [PATCH v11 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-05 18:52                     ` [PATCH v11 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-05 18:52                     ` [PATCH v11 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-05 18:52                     ` [PATCH v11 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-05 19:24                     ` [PATCH v12 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-05 19:24                       ` [PATCH v12 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-05 19:24                       ` [PATCH v12 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-05 19:24                       ` [PATCH v12 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-05 19:24                       ` [PATCH v12 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-05 19:24                       ` [PATCH v12 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-05 19:48                       ` [PATCH v13 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-05 19:48                         ` [PATCH v13 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-05 19:48                         ` [PATCH v13 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-05 19:48                         ` [PATCH v13 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-05 19:48                         ` [PATCH v13 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-05 19:48                         ` [PATCH v13 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-05 20:56                         ` [PATCH v14 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-05 20:56                           ` [PATCH v14 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-05 20:56                           ` [PATCH v14 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-05 20:56                           ` [PATCH v14 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-05 20:56                           ` [PATCH v14 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-05 20:56                           ` [PATCH v14 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-06  7:21                           ` [PATCH v15 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-06  7:21                             ` [PATCH v15 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-06  7:21                             ` [PATCH v15 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-06  7:21                             ` [PATCH v15 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-06  7:21                             ` [PATCH v15 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-06  7:21                             ` [PATCH v15 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-06 15:14                             ` [PATCH v16 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-06 15:14                               ` [PATCH v16 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-06 15:14                               ` [PATCH v16 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-06 15:14                               ` [PATCH v16 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-06 15:14                               ` [PATCH v16 4/5] perf test: Add inject ASLR test Ian Rogers
2026-06-06 15:14                               ` [PATCH v16 5/5] perf aslr: Strip sample registers Ian Rogers
2026-06-07  6:09                               ` [PATCH v17 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-07  6:09                                 ` [PATCH v17 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-07  6:09                                 ` [PATCH v17 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-07  6:09                                 ` [PATCH v17 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-07  6:09                                 ` [PATCH v17 4/5] perf aslr: Strip sample registers Ian Rogers
2026-06-07  6:09                                 ` [PATCH v17 5/5] perf test: Add inject ASLR test Ian Rogers
2026-06-07 21:36                                 ` [PATCH v18 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-07 21:36                                   ` [PATCH v18 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-07 21:36                                   ` [PATCH v18 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-07 21:36                                   ` [PATCH v18 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-07 21:36                                   ` [PATCH v18 4/5] perf aslr: Strip sample registers Ian Rogers
2026-06-07 21:37                                   ` [PATCH v18 5/5] perf test: Add inject ASLR test Ian Rogers
2026-06-08  5:48                                   ` [PATCH v19 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-08  5:48                                     ` [PATCH v19 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-08  5:48                                     ` [PATCH v19 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-08  5:48                                     ` [PATCH v19 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-08  5:48                                     ` [PATCH v19 4/5] perf aslr: Strip sample registers Ian Rogers
2026-06-08  5:48                                     ` [PATCH v19 5/5] perf test: Add inject ASLR test Ian Rogers
2026-06-10 13:26                                       ` James Clark
2026-06-10 16:15                                         ` Ian Rogers
2026-06-11  8:32                                           ` James Clark [this message]
2026-06-08 15:08                                     ` [PATCH v19 0/5] perf tools: Add inject --aslr feature, early maps loading, and decoupling fixes Ian Rogers
2026-06-11 16:41                                     ` [PATCH v20 0/5] perf tools: Add inject --aslr feature Ian Rogers
2026-06-11 16:41                                       ` [PATCH v20 1/5] perf maps: Add maps__mutate_mapping Ian Rogers
2026-06-11 16:41                                       ` [PATCH v20 2/5] perf inject/aslr: Add ASLR tool infrastructure and MMAP tracking Ian Rogers
2026-06-11 16:41                                       ` [PATCH v20 3/5] perf inject/aslr: Implement sample address remapping Ian Rogers
2026-06-15 12:13                                         ` Arnaldo Carvalho de Melo
2026-06-15 15:38                                           ` Ian Rogers
2026-06-15 17:44                                           ` Arnaldo Carvalho de Melo
2026-06-11 16:41                                       ` [PATCH v20 4/5] perf aslr: Strip sample registers Ian Rogers
2026-06-11 16:41                                       ` [PATCH v20 5/5] perf test: Add inject ASLR test Ian Rogers
2026-06-11 18:29                                       ` [PATCH v20 0/5] perf tools: Add inject --aslr feature Ian Rogers
2026-06-13  0:26                                         ` Arnaldo Carvalho de Melo
2026-06-13  2:55                                           ` Ian Rogers
2026-06-15  9:10                                       ` James Clark

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c92e86bc-e596-487e-918d-bc9d8377910b@linaro.org \
    --to=james.clark@linaro.org \
    --cc=acme@kernel.org \
    --cc=adrian.hunter@intel.com \
    --cc=gmx@google.com \
    --cc=irogers@google.com \
    --cc=jolsa@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-perf-users@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=namhyung@kernel.org \
    --cc=peterz@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox