From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4E25495AE7; Thu, 1 Oct 2026 07:24:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790839492; cv=none; b=nqwzppVk6om1ikFAPsZQ3oJ0mvZCo2Y+N38k2G3KStAbVIztZQ/AeAaK+3jZr70YN9syHTmW2Va7nj7pOhXIB3P8kzb7J8c3Z8HAFV53oWUZWpgZaHg7bnBbxRXQrVRwUJa/48e0r35MaRC0JAPi7bCufKiqhk1ohA6XNUYxVUM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790839492; c=relaxed/simple; bh=suq++U8maVc1MZD8XduDV/eZA8m9s6ShB7xjuAhDj4U=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ORldFiruNsiTuUxiT+YZcqVznXoIxGTcR1QLI1wsTv4wSDKJqHKiuOMn2cuZx0vfpEkKGBZaX8jn4u5JbBToTrf1HNUqbcfnAGeaZpKwh0NOwwz7sJtLhYlwDFHl1ZgM8iP2HekjUlS+Zr+w1fMvcgbQtEj9IJt33czYwh2u/ts= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iM8qYeD1; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iM8qYeD1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 104351F000FF; Thu, 1 Oct 2026 07:24:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1790839485; bh=cemCbxcoXGCmsGnSgqml/bkSsdAI28pvjgdmhqUo4EM=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=iM8qYeD1t5j+0zRaBOrbv56K1UqBpD56ZrIEpGSC61vIvSDRaTb1H7BAqMfH4cnVg x3Z/7MxIOy2BQUdoqo926+vYdxlpJtFo0U3Ktm/wGawqKT5xF8AxFLbrkgONel75lA rhTe8xCLPWflzsSheeeOA95G8ZDzyV1VCAsMpUiWgheCFr7vx9stYXW2JvsNDlJtXi raGpudKbm+kis9z6hlQF5q6KFtYS4gJ9Wc2GVmOCyf6wH0nZSpy2jdNOlHCm6HBGLh EH5CYby92iyL9dwJ3JtYcjCxlYb7pIpb5wLf97M2lbVV1LbUnsErnn7AnfPZK7fxX/ Zo2Cc0rFjEUAA== Date: Thu, 1 Oct 2026 00:24:43 -0700 From: Namhyung Kim To: Arnaldo Carvalho de Melo Cc: Ingo Molnar , Thomas Gleixner , James Clark , Jiri Olsa , Ian Rogers , Adrian Hunter , Clark Williams , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Arnaldo Carvalho de Melo Subject: Re: [PATCH v6 4/5] perf scripts: Add perf-stuck, to tell where a running perf is stuck Message-ID: References: <20260930213716.2633750-1-acme@kernel.org> <20260930213716.2633750-5-acme@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20260930213716.2633750-5-acme@kernel.org> On Wed, Sep 30, 2026 at 11:37:15PM +0200, Arnaldo Carvalho de Melo wrote: > From: Arnaldo Carvalho de Melo > > A perf that takes forever is hard to tell apart from one stuck in a > loop, and there is no way to see where without attaching gdb. > perf-stuck.sh samples a running process' /proc entries and its progress > line at a fixed interval, and with -g runs gdb (perf-stuck.gdb, adding > the perf-die-chain command) when no progress is made across two > samples, printing the DIE chain a DWARF type chase is stuck in. > > It is a prototype: the plan is to turn it into a first class 'perf > stuck' command. The process name is resolved with pgrep among the > caller's own processes only, as root an unscoped one would attach gdb > to the first process of any user with a matching name. > > -i and -n are now validated instead of failing later inside the > sampling loop, -x now requires -g and gets the same readability check > as the default gdb command file, and gdb_done is cleared whenever > progress resumes, so -g can fire again on a later stall instead of at > most once per run. > > Assisted-by: LLM > Signed-off-by: Arnaldo Carvalho de Melo > --- > tools/perf/scripts/perf-stuck.gdb | 129 ++++++++++++++++++++ > tools/perf/scripts/perf-stuck.sh | 193 ++++++++++++++++++++++++++++++ > 2 files changed, 322 insertions(+) > create mode 100644 tools/perf/scripts/perf-stuck.gdb > create mode 100755 tools/perf/scripts/perf-stuck.sh > > diff --git a/tools/perf/scripts/perf-stuck.gdb b/tools/perf/scripts/perf-stuck.gdb > new file mode 100644 > index 0000000000000000..be7b9e3fcb5d6d98 > --- /dev/null > +++ b/tools/perf/scripts/perf-stuck.gdb > @@ -0,0 +1,129 @@ > +# SPDX-License-Identifier: GPL-2.0 > +# > +# gdb commands for a stuck perf, used by perf-stuck.sh -g and usable directly: > +# > +# gdb -p $(pgrep -x perf) -batch -x perf-stuck.gdb -ex bt > +# > +# PROTOTYPE: part of the perf-stuck.sh stopgap, wants to become a first > +# class 'perf stuck' command printing these DIE chains without gdb. > +# > +# The commands are for the DWARF type chasers in util/dwarf-aux.c: > +# > +# perf-die-chain [iterations] > +# perf-die-chain-all [iterations] > +# perf-dso > +# > +# For each iteration of the chasing loop they print the DIE address, the > +# CU, its file offset, tag and name: a cycle shows as the same (addr, cu) > +# pairs repeating, and a CU changing between iterations means the chase > +# hops between a debug file and its dwz common file. > + > +set pagination off > +set confirm off > +set debuginfod enabled off > +set print pretty on > +set height 0 > +set width 0 > + > +define perf-die-chain > + if $argc < 2 > + printf "usage: perf-die-chain [iterations]\n" > + else > + frame function $arg0 > + if $argc == 3 > + set $perf_die_chain_n = $arg2 > + else > + set $perf_die_chain_n = 10 > + end > + set $perf_die_chain_head = $pc > + set $perf_die_chain_i = 0 > + while $perf_die_chain_i < $perf_die_chain_n > + # Pointer type DIEs have no DW_AT_name, so dwarf_diename() can > + # return NULL: printf %s of it would error out and abort this > + # batch script, handle it. > + set $perf_die_chain_name = (char *) dwarf_diename($arg1) > + printf "chain[%d] die=%p addr=%p cu=%p off=0x%lx tag=%d name=", $perf_die_chain_i, $arg1, $arg1->addr, $arg1->cu, ((Dwarf_Off) dwarf_dieoffset($arg1)), ((int) dwarf_tag($arg1)) Nit: can you please break this line? Thanks, Namhyung > + if $perf_die_chain_name == 0 > + printf "(null)\n" > + else > + printf "%s\n", $perf_die_chain_name > + end > + until *$perf_die_chain_head > + set $perf_die_chain_i = $perf_die_chain_i + 1 > + end > + end > +end > + > +document perf-die-chain > +Print the DIE chain being walked by a DWARF type chasing loop. > +usage: perf-die-chain [iterations] > + perf-die-chain die_get_pointer_type type_die > + perf-die-chain __die_get_real_type vr_die > + perf-die-chain die_get_real_type vr_die > +end > + > +define perf-die-chain-all > + if $argc == 0 > + set $perf_die_chain_n = 10 > + else > + set $perf_die_chain_n = $arg0 > + end > + if $_any_caller_is("die_get_pointer_type", 20) > + printf "stuck in die_get_pointer_type():\n" > + perf-die-chain die_get_pointer_type type_die $perf_die_chain_n > + else > + if $_any_caller_is("__die_get_real_type", 20) > + printf "stuck in __die_get_real_type():\n" > + perf-die-chain __die_get_real_type vr_die $perf_die_chain_n > + else > + if $_any_caller_is("die_get_real_type", 20) > + printf "stuck in die_get_real_type():\n" > + perf-die-chain die_get_real_type vr_die $perf_die_chain_n > + else > + printf "not in a DWARF type chaser, try: bt\n" > + end > + end > + end > +end > + > +document perf-die-chain-all > +Find which DWARF type chaser the process is in and print the DIE chain. > +usage: perf-die-chain-all [iterations] > +end > + > +define perf-dso > + if $_any_caller_is("find_data_type", 20) > + frame function find_data_type > + # REFCNT_CHECKING, implied by an ASan/LSan build, wraps struct map and > + # struct dso in a proxy keeping the real object in ->orig, the dso being > + # map->orig->dso->orig there; struct symbol is not wrapped, so sym->name > + # needs no ->orig. There is no way to ask gdb which layout it is looking > + # at, so walk each candidate until one evaluates. Requiring gdb's Python > + # here adds nothing: $_any_caller_is() is one of its functions already. > + python > +import gdb > + > +dso = None > +for expr in ("dloc->ms->map->dso->name", > + "dloc->ms->map->orig->dso->orig->name"): > + try: > + gdb.parse_and_eval(expr) > + dso = expr > + break > + except gdb.error: > + pass > + > +if dso is None: > + print("dso=(no such member in struct map)") > +else: > + gdb.execute('printf "dso=%s ip=0x%lx sym=%s\\n", ' + dso + > + ', dloc->ip, dloc->ms->sym ? dloc->ms->sym->name : "(no symbol)"') > + end > + else > + printf "not in find_data_type()\n" > + end > +end > + > +document perf-dso > +Print the dso, ip and symbol of the data location being resolved. > +end > diff --git a/tools/perf/scripts/perf-stuck.sh b/tools/perf/scripts/perf-stuck.sh > new file mode 100755 > index 0000000000000000..f9b545c107905a4e > --- /dev/null > +++ b/tools/perf/scripts/perf-stuck.sh > @@ -0,0 +1,193 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# > +# perf-stuck - tell a spinning perf apart from a blocked or recursing one > +# > +# Arnaldo Carvalho de Melo > +# > +# PROTOTYPE: wants to become a first class 'perf stuck' command, sampling > +# a process from inside perf, with the knowledge of perf's phases and of > +# the DWARF type chasing loops built in, instead of poking /proc and > +# shelling out to gdb. > +# > +# Samples /proc/ at a fixed interval and prints the CPU time used > +# since the previous sample, the [stack] mapping start and size, and the > +# last line of a progress log when one is given, e.g. the stderr of > +# 'perf report --progress': burning a full interval with a constant > +# stack is a loop, a [stack] start moving down is runaway recursion. > +# > +# With -g it runs gdb (perf-stuck.gdb) when no progress is made for two > +# consecutive samples, printing the DIE chain a DWARF type chase is > +# walking. > +# > +# usage: perf-stuck.sh [options] > + > +set -u > + > +usage() { > + cat <<-EOF > + usage: perf-stuck.sh [options] > + > + -i sampling interval (default: 10) > + -n stop after this many samples (default: watch till it exits) > + -l progress log, its last line is printed with every sample > + -g run gdb with perf-stuck.gdb when no progress is made for > + two consecutive samples, writing the output to a temp file > + -x use this gdb command file instead of perf-stuck.gdb > + -h this help > + EOF > + exit "${1:-0}" > +} > + > +interval=10 > +count=0 > +progress_log= > +use_gdb= > +gdb_cmds= > + > +while getopts "i:n:l:gx:h" opt; do > + case "$opt" in > + i) interval=$OPTARG ;; > + n) count=$OPTARG ;; > + l) progress_log=$OPTARG ;; > + g) use_gdb=1 ;; > + x) gdb_cmds=$OPTARG ;; > + h) usage 0 ;; > + *) usage 1 ;; > + esac > +done > +shift $((OPTIND - 1)) > + > +[[ "$interval" =~ ^[0-9]+$ ]] && [ "$interval" -gt 0 ] || > + { echo "-i wants a positive integer, got '$interval'"; usage 1; } > +[[ "$count" =~ ^[0-9]+$ ]] || > + { echo "-n wants a non-negative integer, got '$count'"; usage 1; } > +[ -n "$gdb_cmds" ] && [ -z "$use_gdb" ] && { echo "-x needs -g"; usage 1; } > + > +[ $# -eq 1 ] || usage 1 > + > +if [[ "$1" =~ ^[0-9]+$ ]]; then > + pid=$1 > +else > + # Resolve the name against the caller's own processes: as root, > + # unscoped pgrep picks the first match of any user, e.g. one planted > + # to get gdb attached to it, use an explicit pid to look at a perf of > + # another user. > + pid=$(pgrep -x -u "$(id -u)" -- "$1" | head -1) > + [ -n "$pid" ] || { echo "no process named '$1' owned by $(id -un)"; exit 1; } > +fi > + > +[ -d /proc/"$pid" ] || { echo "no process $pid"; exit 1; } > + > +if [ -n "$use_gdb" ]; then > + # Fail before watching, not two samples in when -g would fire. > + command -v gdb > /dev/null || { echo "gdb not found, -g needs it"; exit 1; } > + [ -z "$gdb_cmds" ] && gdb_cmds=$(dirname "$0")/perf-stuck.gdb > + [ -r "$gdb_cmds" ] || { echo "cannot read $gdb_cmds"; exit 1; } > +fi > + > +hz=$(getconf CLK_TCK) > +psz=$(getconf PAGESIZE) > +prev_cpu= > +prev_stack= > +prev_progress= > +stuck=0 > +gdb_done= > +nsample=0 > + > +# The command line is whatever the process was started with, so drop the > +# control characters from it: a process started with escape sequences in > +# its arguments, e.g. one replaying a log line, would otherwise get them > +# replayed on the terminal of whoever runs this. > +cmdline=$(tr '\0' ' ' < /proc/"$pid"/cmdline | tr -d '[:cntrl:]') > + > +echo "watching $pid ($cmdline) every ${interval}s" > + > +while :; do > + if [ ! -d /proc/"$pid" ]; then > + echo "$(date +%T) process gone" > + break > + fi > + > + # Field 2, the command name, is in parentheses and can contain > + # spaces, so drop it together with the pid before splitting so the > + # fields line up. %d keeps the CPU time out of scientific > + # notation, that bash arithmetic can't parse past six digits. > + if ! stat_line=$(awk '{ sub(/^[^ ]+ \(.*\) /, ""); > + printf "%s %d %d\n", $1, $12 + $13, $22 }' \ > + /proc/"$pid"/stat 2>/dev/null); then > + echo "$(date +%T) process gone" > + break > + fi > + > + # The process can be gone between the check above and this read, in > + # which case there is nothing to report: 'set -u' would otherwise > + # turn the unbound fields into an aborted script. > + if [ -z "$stat_line" ]; then > + echo "$(date +%T) process gone" > + break > + fi > + > + stat=($stat_line) > + state=${stat[0]} > + cpu=${stat[1]} > + # field 24 of /proc//stat, the resident set size in pages > + rss=$(( stat[2] * psz / 1024 )) > + > + stack=$(awk '/\[stack\]/{print $1; exit}' /proc/"$pid"/maps 2>/dev/null) > + if [ -n "$stack" ]; then > + stack_start=0x${stack%-*} > + stack_size=$(( 0x${stack#*-} - stack_start )) > + stack_txt="$stack size=$((stack_size / 1024))kB" > + else > + stack_start= > + stack_txt="-" > + fi > + > + progress= > + [ -n "$progress_log" ] && [ -s "$progress_log" ] && progress=$(tail -1 -- "$progress_log") > + > + if [ -n "$prev_cpu" ]; then > + cpu_delta=$(( cpu - prev_cpu )) > + # No progress log, or one with nothing written to it yet, leaves > + # no progress to look at, so count the samples that show none: > + # -g then looks at where the process is after two intervals. > + # Otherwise count the repeats of the same last line. > + if [ -z "$progress" ] || [ "$progress" = "$prev_progress" ]; then > + stuck=$((stuck + 1)) > + else > + stuck=0 > + gdb_done= > + fi > + stuck_txt="stuck=${stuck}" > + [ "$stack_start" != "$prev_stack" ] && stuck_txt="$stuck_txt STACK" > + else > + cpu_delta=0 > + stuck_txt="" > + fi > + > + printf '%s state=%s cpu=+%d (%d.%02ds) rss=%dkB stack=%s %s %s\n' \ > + "$(date +%T)" "$state" "$cpu_delta" \ > + $(( cpu_delta / hz )) $(( (cpu_delta % hz) * 100 / hz )) \ > + "$rss" "$stack_txt" "$stuck_txt" "${progress:-(no progress log)}" > + > + if [ -n "$use_gdb" ] && [ -z "$gdb_done" ] && [ "$stuck" -ge 2 ]; then > + gdb_log=$(mktemp /tmp/perf-stuck-gdb.XXXXXX) > + # The gdb script makes inferior calls, and a call into a perf > + # wedged in a loop never returns, so bound the run; SIGINT > + # releases the inferior instead of leaving perf stopped. > + timeout --signal=INT 30 gdb -p "$pid" -batch -x "$gdb_cmds" -ex bt \ > + -ex 'perf-die-chain-all' -ex perf-dso -ex detach > "$gdb_log" 2>&1 > + gdb_done=1 > + echo "... gdb output of $pid in $gdb_log" > + fi > + > + prev_cpu=$cpu > + prev_stack=$stack_start > + prev_progress=$progress > + > + nsample=$((nsample + 1)) > + [ "$count" -gt 0 ] && [ "$nsample" -ge "$count" ] && break > + > + sleep "$interval" > +done > -- > 2.55.0 >