From: Namhyung Kim <namhyung@kernel.org>
To: Chun-Tse Shao <ctshao@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Mark Rutland <mark.rutland@arm.com>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Jiri Olsa <jolsa@kernel.org>, Ian Rogers <irogers@google.com>,
Adrian Hunter <adrian.hunter@intel.com>,
James Clark <james.clark@linaro.org>,
Hang Zhengyang <hang.zhengyang1010@gmail.com>,
linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v8] perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output
Date: Wed, 18 Nov 2026 23:27:33 -0800 [thread overview]
Message-ID: <av6k5QSkHPLjCyp7@google.com> (raw)
In-Reply-To: <20261111183209.1892421-1-ctshao@google.com>
Hi CT,
On Wed, Nov 11, 2026 at 10:32:09AM -0800, Chun-Tse Shao wrote:
> When running `perf stat` with `-A` (--no-aggr) and `--metric-only` in
> JSON output mode (`-j`), `perf stat` evaluates metric expressions
> across all matching PMUs (including uncore PMUs like `uncore_iio_0`,
> `uncore_iio_1`, etc.).
What about other outputs like stdio and csv?
Thanks,
Namhyung
>
> However, `perf stat` previously formatted JSON output by printing only
> "cpu" : "<id>" and grouping all metric values on a single line per CPU
> without identifying which PMU instance evaluated each metric. As a
> result, when an uncore event spans multiple PMU boxes, `perf stat`
> printed repeated, ambiguous metric keys without PMU names.
>
> Fix this by:
> 1. Including "pmu" : "<pmu_name>" in print_aggr_id_json in metric-only
> AGGR_NONE mode (-A) when evsel->pmu is a non-core hardware or hybrid
> PMU. Software and tool events don't belong to a hardware PMU instance
> and so are not labeled.
> 2. Starting a new JSON metric line in AGGR_NONE mode (-A) whenever the
> PMU name changes, so that a line only has metrics from the PMU it is
> labeled with.
> 3. Opening a metric line only for events that have a metric, as other
> events print nothing in metric-only mode. This avoids spurious lines
> like {"cpu" : "0", "pmu" : "tool"} for events like user_time, while
> metrics on tool events, like tma_info_system_time (duration_time),
> are still printed with the line prefix.
> 4. Updating perf_json_output_lint.py to recognize the new "pmu" key in
> the JSON test suite.
>
> Before the fix:
> $ perf stat -M iio_bandwidth_read -a -A --metric-only -j -I 1000
> {"interval" : 1.001017947, "cpu" : "0", "MB/s iio_bandwidth_read" : "0.0", "MB/s iio_bandwidth_read" : "0.0", "MB/s iio_bandwidth_read" : "22.5", "MB/s iio_bandwidth_read" : "0.0", "MB/s iio_bandwidth_read" : "0.2", "MB/s iio_bandwidth_read" : "0.0", "MB/s iio_bandwidth_read" : "0.0", "MB/s iio_bandwidth_read" : "0.0", "MB/s iio_bandwidth_read" : "0.0", "MB/s iio_bandwidth_read" : "0.0"}
>
> There is no way to determine which uncore device generated the metrics.
>
> After:
> $ perf stat -M iio_bandwidth_read -a -A --metric-only -j -I 1000
> {"interval" : 1.000314908, "cpu" : "0", "pmu" : "uncore_iio_0", "MB/s iio_bandwidth_read" : "0.0"}
> {"interval" : 1.000314908, "cpu" : "0", "pmu" : "uncore_iio_1", "MB/s iio_bandwidth_read" : "0.1"}
> {"interval" : 1.000314908, "cpu" : "0", "pmu" : "uncore_iio_11", "MB/s iio_bandwidth_read" : "0.0"}
> ...
> {"interval" : 1.000314908, "cpu" : "56", "pmu" : "uncore_iio_0", "MB/s iio_bandwidth_read" : "0.0"}
> {"interval" : 1.000314908, "cpu" : "56", "pmu" : "uncore_iio_1", "MB/s iio_bandwidth_read" : "0.0"}
> {"interval" : 1.000314908, "cpu" : "56", "pmu" : "uncore_iio_11", "MB/s iio_bandwidth_read" : "0.0"}
>
> Assisted-by: Gemini:gemini-3.1-pro-preview
> Signed-off-by: Chun-Tse Shao <ctshao@google.com>
> ---
> v8:
> - Gate "pmu" output in print_aggr_id_json() on metric-only mode and
> skip software PMUs via perf_pmu__is_software(), keeping plain
> `perf stat -a -A -j` unchanged and avoiding splitting default
> metrics onto a separate "software" line. (Hang Zhengyang)
> - Rebase onto the latest perf-tools-next.
> v7: lore.kernel.org/20261030192537.3079368-1-ctshao@google.com
> - Pass the evsel to print_metric_begin() explicitly from all callers
> instead of falling back to evlist__first(). (Ian)
> v6: lore.kernel.org/20261027203115.2647914-1-ctshao@google.com
> - Open metric lines for events that have a metric instead of skipping
> tool events, so metrics on tool events (e.g. tma_info_system_time)
> keep the line prefix and are terminated properly. (Sashiko)
> - Split JSON lines only when the shown PMU name changes. Tool event
> metrics are no longer mislabeled with an uncore PMU name, and the
> STD and CSV output are no longer split.
> - Rebase onto the latest perf-tools-next.
> v5: lore.kernel.org/20260811230518.3360-2-ctshao@google.com
> - Fix perf test "perf stat JSON output linter" failure.
> v4: lore.kernel.org/20260804210659.112192-1-ctshao@google.com
> - Fix the line splitting issue, causing spurious empty {"pmu": "tool"}
> objects and order-dependent PMU mislabeling.
> v3: lore.kernel.org/20260729170416.41904-1-ctshao@google.com
> - Add before the fix output.
> v2: lore.kernel.org/20260715203055.2981363-1-ctshao@google.com
> - Fix print_metric_begin() initialization in print_no_aggr_metric()
> to ensure tool events always open lines cleanly.
>
> .../tests/shell/lib/perf_json_output_lint.py | 1 +
> tools/perf/util/stat-display.c | 79 ++++++++++++++++---
> 2 files changed, 68 insertions(+), 12 deletions(-)
>
> diff --git a/tools/perf/tests/shell/lib/perf_json_output_lint.py b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> index 8cde22ac9d97..0a16cae92766 100644
> --- a/tools/perf/tests/shell/lib/perf_json_output_lint.py
> +++ b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> @@ -65,6 +65,7 @@ def check_json_output(expected_count):
> 'metricgroup': lambda x: True,
> 'node': lambda x: True,
> 'pcnt-running': isfloat,
> + 'pmu': lambda x: True,
> 'socket': lambda x: True,
> 'thread': lambda x: True,
> 'unit': lambda x: True,
> diff --git a/tools/perf/util/stat-display.c b/tools/perf/util/stat-display.c
> index b337cc23f413..6c9358b64b38 100644
> --- a/tools/perf/util/stat-display.c
> +++ b/tools/perf/util/stat-display.c
> @@ -1,6 +1,7 @@
> #include <stdlib.h>
> #include <stdio.h>
> #include <inttypes.h>
> +#include <string.h>
> #include <linux/string.h>
> #include <linux/time64.h>
> #include <math.h>
> @@ -23,6 +24,7 @@
> #include "pmu.h"
> #include "pmus.h"
> #include "tool_pmu.h"
> +#include "metricgroup.h"
>
> #define CNTR_NOT_SUPPORTED "<not supported>"
> #define CNTR_NOT_COUNTED "<not counted>"
> @@ -361,9 +363,34 @@ static void print_aggr_id_csv(struct perf_stat_config *config,
> }
> }
>
> +/*
> + * In no-aggregation metric-only mode, JSON output includes the PMU name of
> + * non-core hardware and hybrid events so that metrics from different PMU
> + * instances, like uncore_iio_0 and uncore_iio_1, can be told apart. Software
> + * and tool events don't belong to a hardware PMU instance and so are not
> + * labeled.
> + */
> +static const char *json_pmu_name(const struct evsel *evsel)
> +{
> + if (!evsel || !evsel->pmu || perf_pmu__is_software(evsel->pmu))
> + return NULL;
> + if (evsel->pmu->is_core && !evsel__is_hybrid(evsel))
> + return NULL;
> + return evsel->pmu->name;
> +}
> +
> +static bool json_pmu_name_changed(const char *name, const char *prev)
> +{
> + if (!name || !prev)
> + return name != prev;
> + return strcmp(name, prev) != 0;
> +}
> +
> static void print_aggr_id_json(struct perf_stat_config *config, struct outstate *os,
> struct evsel *evsel, struct aggr_cpu_id id, int aggr_nr)
> {
> + const char *pmu_name;
> +
> switch (config->aggr_mode) {
> case AGGR_CORE:
> json_out(os, "\"core\" : \"S%d-D%d-C%d\", \"counters\" : %d",
> @@ -397,6 +424,11 @@ static void print_aggr_id_json(struct perf_stat_config *config, struct outstate
> json_out(os, "\"cpu\" : \"%d\"",
> id.cpu.cpu);
> }
> + if (config->metric_only) {
> + pmu_name = json_pmu_name(evsel);
> + if (pmu_name)
> + json_out(os, "\"pmu\" : \"%s\"", pmu_name);
> + }
> break;
> case AGGR_THREAD:
> json_out(os, "\"thread\" : \"%s-%d\"",
> @@ -1010,13 +1042,16 @@ static void print_counter_aggrdata(struct perf_stat_config *config,
> fputc('\n', output);
> }
>
> +/*
> + * Start a metric-only line. @evsel only provides the line prefix: the
> + * aggregation id, the PMU name in JSON (see json_pmu_name()) and the cgroup.
> + */
> static void print_metric_begin(struct perf_stat_config *config,
> - struct evlist *evlist,
> + struct evsel *evsel,
> struct outstate *os, int aggr_idx)
> {
> struct perf_stat_aggr *aggr;
> struct aggr_cpu_id id;
> - struct evsel *evsel;
>
> os->first = true;
> if (!config->metric_only)
> @@ -1031,7 +1066,6 @@ static void print_metric_begin(struct perf_stat_config *config,
> else
> fprintf(config->output, "%s", os->timestamp);
> }
> - evsel = evlist__first(evlist);
> id = config->aggr_map->map[aggr_idx];
> aggr = &evsel->stats->aggr[aggr_idx];
> aggr_printout(config, os, evsel, id, aggr->nr);
> @@ -1069,7 +1103,7 @@ static void print_aggr(struct perf_stat_config *config,
> * Without each counter has its own line.
> */
> cpu_aggr_map__for_each_idx(aggr_idx, config->aggr_map) {
> - print_metric_begin(config, evlist, os, aggr_idx);
> + print_metric_begin(config, evlist__first(evlist), os, aggr_idx);
>
> evlist__for_each_entry(evlist, counter) {
> print_counter_aggrdata(config, counter, aggr_idx, os);
> @@ -1095,7 +1129,7 @@ static void print_aggr_cgroup(struct perf_stat_config *config,
> os->cgrp = evsel->cgrp;
>
> cpu_aggr_map__for_each_idx(aggr_idx, config->aggr_map) {
> - print_metric_begin(config, evlist, os, aggr_idx);
> + print_metric_begin(config, evsel, os, aggr_idx);
>
> evlist__for_each_entry(evlist, counter) {
> if (counter->cgrp != os->cgrp)
> @@ -1131,7 +1165,8 @@ static void print_no_aggr_metric(struct perf_stat_config *config,
>
> perf_cpu_map__for_each_cpu(cpu, all_idx, evlist__core(evlist)->user_requested_cpus) {
> struct evsel *counter;
> - bool first = true;
> + const char *line_pmu = NULL;
> + bool line_open = false;
>
> evlist__for_each_entry(evlist, counter) {
> u64 ena, run, val;
> @@ -1146,12 +1181,32 @@ static void print_no_aggr_metric(struct perf_stat_config *config,
> if (config->aggr_map->map[aggr_idx].cpu.cpu == cpu.cpu)
> break;
> }
> + if (aggr_idx >= config->aggr_map->nr)
> + continue;
>
> os->evsel = counter;
> os->id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);
> - if (first) {
> - print_metric_begin(config, evlist, os, aggr_idx);
> - first = false;
> +
> + /*
> + * Only events with a metric print something in metric-only
> + * mode, so let the first of them open the line. For JSON,
> + * start a new line when the PMU name changes so that all
> + * metrics on a line share the same "cpu" and "pmu" prefix.
> + */
> + if (config->iostat_run ||
> + metricgroup__lookup(evlist__metric_events(evlist), counter, false)) {
> + const char *pmu_name = json_pmu_name(counter);
> +
> + if (line_open && config->json_output &&
> + json_pmu_name_changed(pmu_name, line_pmu)) {
> + print_metric_end(config, os);
> + line_open = false;
> + }
> + if (!line_open) {
> + print_metric_begin(config, counter, os, aggr_idx);
> + line_pmu = pmu_name;
> + line_open = true;
> + }
> }
> val = ps->aggr[aggr_idx].counts.val;
> ena = ps->aggr[aggr_idx].counts.ena;
> @@ -1160,7 +1215,7 @@ static void print_no_aggr_metric(struct perf_stat_config *config,
> uval = val * counter->scale;
> printout(config, os, uval, run, ena, 1.0, aggr_idx);
> }
> - if (!first)
> + if (line_open)
> print_metric_end(config, os);
> }
> }
> @@ -1523,7 +1578,7 @@ static void print_cgroup_counter(struct perf_stat_config *config, struct evlist
> print_metric_end(config, os);
>
> os->cgrp = counter->cgrp;
> - print_metric_begin(config, evlist, os, /*aggr_idx=*/0);
> + print_metric_begin(config, counter, os, /*aggr_idx=*/0);
> }
>
> print_counter(config, counter, os);
> @@ -1573,7 +1628,7 @@ void evlist__print_counters(struct evlist *evlist, struct perf_stat_config *conf
> } else if (config->cgroup_list) {
> print_cgroup_counter(config, evlist, &os);
> } else {
> - print_metric_begin(config, evlist, &os, /*aggr_idx=*/0);
> + print_metric_begin(config, evlist__first(evlist), &os, /*aggr_idx=*/0);
> evlist__for_each_entry(evlist, counter) {
> print_counter(config, counter, &os);
> }
> --
> 2.56.0.542.g3ef89b7020-goog
>
prev parent reply other threads:[~2026-11-19 7:27 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 18:32 Chun-Tse Shao
2026-11-11 18:53 ` Hang Zhengyang
2026-11-19 7:27 ` Namhyung Kim [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av6k5QSkHPLjCyp7@google.com \
--to=namhyung@kernel.org \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=ctshao@google.com \
--cc=hang.zhengyang1010@gmail.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®