From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f69.google.com (mail-dl1-f69.google.com [74.125.82.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50701329C48 for ; Wed, 23 Sep 2026 18:12:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.69 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790187158; cv=none; b=TELlT97fprpim7pqZtY/6kxA/2ojTvEDh/a1aL4eaMmnRc1+trFbIgQ0p4U54Vo4EQzSA4kYQcSKHLSZSggjUDdhqXaZ/m0apsPjXLxiozm/iHsAYDAs5abhP8KU8s9TXY1gtN3GKlSqNfS3grkW+j36P6e7ZEqsqetjldgckAU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790187158; c=relaxed/simple; bh=F++0gZ4CGBsW3dISIfVcqKeKBBNMwGurLHRaRLuMiXQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aq4WsGJABbKw8CKxpH+ZGJfeinN6vWjnU6tFJmjAjnn3Y9ZEkGU8N7OBbePPgYJSM0jF9SQG4m+eptXZU54tGYgfZZgPhBHbeJUlt0rFMmO5wM4zGxjHkz6+Ao6KRskIeDfNKnTGE5IyjxHq+ODue14YBw901hOKxoU/bX2W0fU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sie0Fnoc; arc=none smtp.client-ip=74.125.82.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sie0Fnoc" Received: by mail-dl1-f69.google.com with SMTP id a92af1059eb24-14383177746so2645158c88.1 for ; Wed, 23 Sep 2026 11:12:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1790187154; x=1790791954; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=bEI7Eum47JZ1rVrjRmdaIUAF1BiiH65WcxpIXKkjfIQ=; b=sie0FnocFBFQBmUA2hRqT3BCCfILw3jqawmFdQMze4CFRxexfZXe/b7nX2ivneBuqz w26UockhBkXJSSnPeXuh5S3+KnHFBW5xLb2STuGyfc6qvcIwzpwo1YzD2E3OYRDGX2O1 M6dv87cu1sOMXUSMfudvxab4dzhWqDeBWqkXErkWlf/NaVLBX/D4xLxVFB7uTyIEZ9fi 3FCi9DIjzvIDOgxgK1LXjSER4TsSlq4v9dIUDq5RoBerMnMr9hW71ofzx87dWTC87w2W J2RnurkNYdc1AN79aywo2Lki716QBOkHNtaDit3MCIDkPUyjKPimqg7bdPuV/1edxMz4 2g7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790187154; x=1790791954; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bEI7Eum47JZ1rVrjRmdaIUAF1BiiH65WcxpIXKkjfIQ=; b=PKJtW4/2FB6mBw6FSU+zTaKwFgBrzVxnERrOpnRhNopZ7j+Wb48pw6QGxPzuUIinYg Rp7SQ9WO6wAu0W5x2zALZTB+IAocZ+F3DmXeWmbdSxtoyTC2Iy6c2v4abR1RbhJwGU/e 29daSHNfaozef9HMNQYPa/VDRQ+5Oryo4epECjCuAphBi5csUVJgBPhrr+9Y1H7Crudo XgQHlO0yQ9Agwrcc+RrRRhttsFaEcLqnuZRRIM2oD9m5HNqZXXprXSSVtmcfSq6FAvCc kWlw0tfcyryrohUUdkdXPjVjeO+laGF1jIKe3edIuGmgUVzJwEUfPV0B5HEVKf+9cxMX PwQg== X-Forwarded-Encrypted: i=1; AKwUvBwQqDavKLHKbXk/LvYKCsdIAe34ZYtsZ/gKg5vZJDOwcYuTNDhcH3YbHCIXrng6+9Pnn9gcgImNW1LIpfc=@vger.kernel.org X-Gm-Message-State: AFuF++nJ5jODPFofUCKIEGWXSlR8TtGbOixZoOEA7TvCiT1XvWe+1RUu TKmLsT/Aj2m5kA8nFhCAvH/t/yjeMoWPmeMtoMtsTvbGqCF+8sdPlWrwti2CZfETTcUtOOV8ZLT hCACD0ugXdQ== X-Received: from dlbpt9.prod.google.com ([2002:a05:7022:e809:b0:144:bfb6:b5b2]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a05:701b:2305:b0:143:3617:c5ae with SMTP id a92af1059eb24-144f90d92b3mr3010379c88.2.1790187153670; Wed, 23 Sep 2026 11:12:33 -0700 (PDT) Date: Wed, 23 Sep 2026 11:11:23 -0700 In-Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.56.0.rc1.310.g51773c2048-goog Message-ID: <20260923181213.3032038-1-irogers@google.com> Subject: [PATCH v3 00/49] perf: Complete transition to standalone Python scripts From: Ian Rogers To: irogers@google.com, acme@kernel.org, alice.mei.rogers@gmail.com, namhyung@kernel.org Cc: adrian.hunter@intel.com, dapeng1.mi@linux.intel.com, james.clark@linaro.org, leo.yan@linux.dev, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, tmricht@linux.ibm.com Content-Type: text/plain; charset="UTF-8" The perf script command has historically supported running Python and Perl scripts by embedding libpython and libperl. That approach has several drawbacks: - high overhead from constructing Python dictionaries for every event (whether used by the script or not), - complex build dependencies on specific Python/Perl libraries, - threading and interpreter lifecycle complications from embedding the interpreter inside perf, - no unified model for standalone scripts such as ilist.py, treport.py, tracepoint.py, and twatch.py. Parts of this series were previously posted as part of the larger "perf: Reorganize scripting support" series: https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ with earlier RFC and mailing list discussion at: https://lore.kernel.org/linux-perf-users/20251029053413.355154-1-irogers@google.com/ https://lore.kernel.org/lkml/CAP-5=fWDqE8SYfOLZkg_0=4Ayx6E7O+h7uUp4NDeCFkiN4b7-w@mail.gmail.com/ >From that earlier series, the prerequisite patches (header dependency cleanups, evlist/evsel reference counting, initial perf.data and perf.session C extension wrappers, callchain/brstack/stat/syscall/config accessors, perf.pyi type stubs, LiveSession, and removal of embedded libperl support) have now been merged. We restarted the version number at v1 for the remaining patches, along with additional C extension capabilities, build-time static analysis (mypy and pylint), and review fixes. Running scripts as standalone Python applications using the perf Python C extension module provides a dramatic speedup. For example, with mem-phys-addr.py: Before (using embedded libpython in perf script): ``` $ perf mem record -a sleep 1 $ time perf script tools/perf/scripts/python/mem-phys-addr.py Event: cpu_core/mem-loads-aux/ Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m3.754s user 0m0.023s sys 0m0.018s ``` After (using standalone Python script with perf module): ``` $ PYTHONPATH=/tmp/perf/python time python3 tools/perf/python/mem-phys-addr.py Event: evsel(cpu_core/mem-loads-aux/) Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m0.106s user 0m0.021s sys 0m0.020s ``` The 49 patches in this series are organized into four groups: 1. Perf Python C Extension & Existing Script Updates (Patches 01-10): - Refine syscall_name()/syscall_id() argument parsing, expose perf.arch_strerrno() from the auto-generated trace beauty errno tables, and update callchain stubs and session.find_thread() lookup in tools/perf/util/python.c and tools/perf/python/perf.pyi. - Clean up pylint warnings and improve `perf script -l` descriptions in existing standalone scripts (ilist.py, treport.py, tracepoint.py, twatch.py). - Expose destination address location attributes (addr_dso, addr_symbol, addr_sym_offset), branch/transaction fields, sym_offset, context_switch callbacks, and Intel PT call_return / itrace synthesis directly in the perf Python extension (many parts previously posted in https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/). - Allow KeyboardInterrupt to propagate cleanly in LiveSession. 2. Build-Time Python Static Analysis (Patches 11-14): - Clean up mypy and pylint warnings in tools/perf/pmu-events/ and tools/perf/tests/shell/lib/. - Make MYPY and PYLINT build checks enabled by default when mypy and pylint (>= 2.16.0) are installed, with NO_MYPY=1 and NO_PYLINT=1 opt-out controls (matching NO_SHELLCHECK=1), so all Python scripts in tools/perf are continuously type-checked and linted at build time. 3. Porting Legacy Python & Perl Scripts to tools/perf/python/ (Patches 15-45): - Previously sent as part of https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ and updated with static type annotations, mypy/pylint cleanups, and accompanying shell tests under tools/perf/tests/shell/. Sashiko-driven review and improvement was undertaken while porting so that the ported scripts are in the best shape possible: * Temporarily install tools/perf/python/ scripts during the transition (Patch 15). * General & profiling scripts: stat-cpi.py, mem-phys-addr.py, stackcollapse.py, flamegraph.py, gecko.py, event_analyzing_sample.py (Patches 16-21). * Syscall & I/O scripts (ported from legacy Python and Perl): syscall-counts.py, syscall-counts-by-pid.py, failed-syscalls-by-pid.py, failed-syscalls.py, sctop.py, rw-by-file.py, rw-by-pid.py, rwtop.py (Patches 22-29). * Scheduling, locking, MM & networking scripts: futex-contention.py, task-analyzer.py, sched-migration.py (with SchedGui.py), wakeup-latency.py, compaction-times.py, net_dropmonitor.py, netdev-times.py, check-perf-trace.py (Patches 30-37). * Hardware trace & architecture-specific scripts: arm-cs-trace-disasm.py, powerpc-hcalls.py, intel-pt-events.py (with libxed.py), and migrating the Intel PT virtual LBR test to the Python API (Patches 38-41). * Database export & viewer utilities: export-to-sqlite.py, export-to-postgresql.py, exported-sql-viewer.py, and parallel-perf.py (Patches 42-45). 4. Removing Embedded libpython & Updating Build, CLI, and Docs (Patches 46-49): - Previously sent as part of https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ : * Remove embedded libpython support (scripting-engines/trace-event-python.c) and delete the legacy tools/perf/scripts/python/ directory (Patch 46). * Replace the libpython feature check with test-python-module.c and update Python script installation paths (Patch 47). * Update `perf script` (builtin-script.c) to execute standalone scripts directly and remove embedded scripting engine support (Patch 48). * Update perf man pages (perf-script.txt, perf-script-python.txt, perf-check.txt) for standalone Python scripts (Patch 49). Changes in v3: - Expose `perf_env__arch_strerrno()` as `perf.arch_strerrno()` in `tools/perf/util/python.c` (and tighten `arch_errno_names.sh` macro matching to `E[A-Z0-9]+` with `EM_SPARC32PLUS` support) so `failed-syscalls-by-pid.py` uses the auto-generated C beauty errno tables instead of maintaining Python errno dictionaries (Patches 01, 24). - Change `static char * const kwlist[]` to `static char *kwlist[]` in `pyrf_session__new()` in `tools/perf/util/python.c` to support Python < 3.13 where `PyArg_ParseTupleAndKeywords()` expects `char **` rather than `char *const *`, and register `psession->tool.event_update` so pipe-mode `PERF_RECORD_EVENT_UPDATE` metadata is processed (Patches 08, 09). - Explicitly annotate `d_hcoal: Union[Event, int]` and `d_mcoal: Union[Event, int]` in `AmdDtlb()` in `tools/perf/pmu-events/amd_metrics.py` so older `mypy` versions do not infer `join(Event, int) = object` (Patch 11). - Pass `--ignore-missing-imports` to `MYPY` and `--ignored-modules=textual,setproctitle` to `PYLINT` in `tools/perf/Makefile.perf` so default static analysis builds succeed on systems without optional third-party Python packages (`python3-textual`, `python3-setproctitle`) installed (Patches 13, 14). - Chain `$(INSTALL)` commands with `&&` in `Makefile.perf` for the `python-scripts` installation rules and invoke `"perf script "` without `-s` from `script_browse()` in `ui/browsers/scripts.c` (Patches 15, 47). - Add Subresource Integrity (`integrity="sha256-..."`) attributes to the CDN `` and `