From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.214.176 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794030760; b=hqrcrYtuP5+3jtDClAAdTYcf2cYtEFogeupZOJF8qPIH2FJAg9/gDnNGy+eTjte0BGpA Lqb6aVEKSxKgTj6/WFprJMs7frpY+rvzMTVukFZSBrCJnyJrcbutn9WQs8p/hWc/ZX+V1 0HpO/b4r4FeAo40rQrd5gYvO8fDVWwRsWrztabSFPJDpjxZRr/6Xf3OyWWi9ov3wByiMU efBtnZDsFTD+l8Vcnkgeze0tp8pyGUBFb71a482HLFeYgC2DRcV3flIC1HXUuNUORTkZq xh7C35jJUUxVZC36FZYUN+eTYPO2f5SqmE9aw0mzEk9BNw56yGgtIG9uJSq8UXV4/cA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794030760; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:X-Google-Original-From:To:Cc:Subject:Date:Message-Id:X-Mailer: In-Reply-To:References:MIME-Version:Content-Transfer-Encoding; bh=u4nTXiGvoLBPJ6m6vBAshKxDWZY7SBPxeVPpU162HBU=; b=x9rWIDLSvSKa9uVjnNQzPRwFcVMPJEu+iimTTP7jGcZTWrvtyitPr815WSviuGGOPGSN MgobwnSYh4NeF5Llqu1GI075FL3aduUUvq/HjyKWQhIlliQbc9SCKzbWDJxrcx54BQgvm z57lXdErC/EGJ1lxJAdwTMUl9q4aXMDOkI0gTkTc+rISTdlG4AYQBCevaTNOt2kTOrn6G ZzRKQDJo3B5P6HdTt5KraNoB6cBXwbnerm+VwYX/z4sBzK/WPnYeqh/QX0mKKWqL7/bWX ZwzWlSc9di2SmVBf1NQmkVuo6uTIErxI7PbQMZbfA9plerWvchX5IjQarV/z9/raFfA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=sTqb2z+O; arc=none smtp.remote-ip=209.85.214.176 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D619606A32C for ; Sat, 7 Nov 2026 05:52:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 8D619606A32C Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=sTqb2z+O Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2ed1b9e8f43so5575675ad.1 for ; Fri, 06 Nov 2026 21:52:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794030759; x=1794635559; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=u4nTXiGvoLBPJ6m6vBAshKxDWZY7SBPxeVPpU162HBU=; b=sTqb2z+Ot/B3iVi+LnWhL32Lbw/KBWJT8Nl/gLMpAslqtMx1sopqaITkjS/bC9tz2a oMmWG2xiahslalTLRYjLCx7AHb/oUXDC/7gnoI79oOqpVZF3sjaeRgvGdkERzNMWcrGg MZHWbGpaOJyeqiMP471einPsifL6/nuAzOdE7gBMFvq/oeQybdZ0BkjUiXk3joRvqGiT WGsROdgKOIoNFsdscRwWsP/CF1LJVluuJjTx2W2iIb62Cm5hQ5ZBOEp0T0Sa3dcUjlb7 uv8eu6CoaXeF+rjFA5TEqXyzKN2I550LLOcG9BMHrss6JdP5FADjqvtZ1W0TE0YGvCbw XPGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794030759; x=1794635559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=u4nTXiGvoLBPJ6m6vBAshKxDWZY7SBPxeVPpU162HBU=; b=sV4ko37hnBDZCYY9PEg28cCmxtgVm38R20jphhZ6b5bUodmsoYRph93TWw5IEricXX Xk3dmQZmEWoui+DVUC4wzbT2/5YAsjQQf1I9AFgA4SEuhRBweBG+j0dMznxaGgGC7RAx gfmelSo21QBfsnodQcSj/jVAJr4g4yvANJMyvXqiVw6OZvMcZ9AmCvMA5XlImdnpEtBZ tqhua048C6zZ3RDChlnEO5ZgX0hVOK5GYfPJ6DUgl3vomK5KtvCmkcQrn3cvkxhVip3r Nd2d4NVMhZpLm4DmttsO+4tBBQ0AFolMFqjZd+uZYY7atx9cQfcs1jSjHe/wbRGY6n5M rIQQ== X-Forwarded-Encrypted: i=1; AL+hErH4dJfcr+CyHLlk34jbIcqX3i5SGYFumefBSTFKoSel6kRgpda6X+pPvvN0gJgHYKtq5QuUF44nQGO8bdk=@vger.kernel.org X-Gm-Message-State: APuU4Z/JUiu0oJWJNgNKF1qnuWb+1UC5poo26HdY9N+6ECi0xOFptcNe tyaMwpFdUhpFT0dwvxjsijrrpTjISf22NP8S4i5h1GrGSv6TxVVwtOnu X-Gm-Gg: AYBFou3QpLk0RGWHbChAROzGas9ODATaEuCWTfhiJGynRJfUFBmIn+1Pn0VYItjiniQ qkI/eLc1YyVNiTq5a7neNY9CmYPyVCTI2+r0/3QPLrA41FqqQo84BBu6+BW4QtoiI23MWMB6c6u RoSRPgDwdEnE8AZMtq8jRCeDEivyFxjsf+IhC9EOJIzr2hEBtnJGSybZiKD8K+bnW0CRQFP3oo/ xE/sKtBEMy2+6pechoEsawPAcA5/HDfh3BXldEoxgrUbUoIl1mi9silixcK3vAl1cB37Y1FZBYC yRqf3hSx4Rmh3W7jSJqeG2WpElMW7NCYsOt8Z5pgAv6HZv+U0wG/fFpfEMUBJ8HlZMQ4waqqps/ 3x6UiT9UI5dDU27QNEVKLAkN+6fDtibKrQxm2s98nI4GBroERnt3e0C56BNINsU+0oZBS1UzSA0 UPCBP/ubFVpSxsjKpTMvB70RChtkLt/cdfvIFaWOk/D5aQyozwc2DXHJg+mB7Lt8nUPHdxOnmiA CKQ1nVWNh/SIdV/TOFgKtj1di86YUVuEEKHrQXrgZNlLPVKM+PA1ij55K+Ff8ye48JO+qUuGzI1 Doa8+D6YFyGLDLoM2ImHXwSTgJymnIon62EUegwYTpJ6p8WjOzd8kXBnok2ohYYFiPBK6A== X-Received: by 2002:a17:903:1a30:b0:2ed:18cc:d3b3 with SMTP id d9443c01a7336-2ed405ee120mr57893445ad.6.1794030758978; Fri, 06 Nov 2026 21:52:38 -0800 (PST) Received: from lipengfei28-ThinkStation-P368.mioffice.cn ([2408:8607:1b00:8:a1a:7bc2:20ba:a479]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2ed47d8949fsm11189465ad.62.2026.11.06.21.52.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2026 21:52:37 -0800 (PST) From: Li Pengfei X-Google-Original-From: Li Pengfei To: rostedt@goodmis.org, mhiramat@kernel.org Cc: mathieu.desnoyers@efficios.com, mark.rutland@arm.com, corbet@lwn.net, skhan@linuxfoundation.org, linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, zhangbo56@xiaomi.com, lipengfei28@xiaomi.com, lkp@intel.com Subject: [PATCH v8 4/5] Documentation: tracing: Document the ftrace stackmap Date: Sat, 7 Nov 2026 13:51:38 +0800 Message-Id: <20261107055139.3138633-5-lipengfei28@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20261107055139.3138633-1-lipengfei28@xiaomi.com> References: <20261107055139.3138633-1-lipengfei28@xiaomi.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Pengfei Li Describe how to use the stackmap, how reset and free behave, and how to resolve a trace that spans resets. List the option in ftrace.rst. Assisted-by: LLM Signed-off-by: Pengfei Li --- Documentation/trace/ftrace-stackmap.rst | 192 ++++++++++++++++++++++++ Documentation/trace/ftrace.rst | 6 + Documentation/trace/index.rst | 1 + 3 files changed, 199 insertions(+) create mode 100644 Documentation/trace/ftrace-stackmap.rst diff --git a/Documentation/trace/ftrace-stackmap.rst b/Documentation/trace/ftrace-stackmap.rst new file mode 100644 index 000000000000..58af7c564b10 --- /dev/null +++ b/Documentation/trace/ftrace-stackmap.rst @@ -0,0 +1,192 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================== +Ftrace Stack Map +====================== + +:Author: Pengfei Li + +Overview +======== + +The ftrace stack map provides stack trace deduplication for the ftrace +ring buffer. When enabled, instead of a full kernel stack trace event +(typically 80-160 bytes) for each stack the map can store, ftrace writes a +fixed-size event of 20-24 bytes, depending on the architecture, that holds +a 64-bit cookie: the map generation in the high 32 bits and the slot index +in the low 32. The full stacks are kept once in a separate hash table and +exported via tracefs for userspace to resolve. + +It is similar to eBPF's ``BPF_MAP_TYPE_STACK_TRACE``. + +Configuration +============= + +Enable ``CONFIG_FTRACE_STACKMAP=y`` in the kernel config. + +The map holds up to 16384 distinct stacks. It is allocated when the +``stackmap`` option is first enabled; if the allocation fails, enabling +the option fails with ``-ENOMEM``. The map stays allocated while the +option is turned off, so a trace can still be resolved, until ``free`` +is written to ``stack_map``. On a 64-bit kernel the map takes about +8.6 MiB. + +Usage +===== + +Enable stack deduplication for event stacks:: + + echo 1 > /sys/kernel/tracing/options/stackmap + echo 1 > /sys/kernel/tracing/options/stacktrace + echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable + +The function tracer records stacks through its own option instead of +``stacktrace``:: + + echo vfs_read > /sys/kernel/tracing/set_ftrace_filter + echo function > /sys/kernel/tracing/current_tracer + echo 1 > /sys/kernel/tracing/options/func_stack_trace + +The trace output will show ```` instead of full stack +traces, where N is the slot and G the map generation:: + + sh-1234 [006] d.h.. 123.456789: + +Stacks are not always replaced. A stack deeper than 64 frames, a stack +recorded before the map is allocated or during a reset, a stack that +another context is inserting at the same moment, and a stack the map +cannot store (element pool exhausted, probe limit, or ring buffer +reservation failure) are recorded in full. A trace can therefore contain +both ```` records and full ```` records. +With the option enabled, a stack recorded where RCU is not watching is +discarded. + +To view the actual stacks:: + + cat /sys/kernel/tracing/stack_map + +Output format:: + + stack_id 42 gen 1 ref 1337 depth 8 + => schedule+0x48/0xc0 + => schedule_timeout+0x1c/0x30 + ... + +``ref`` is the number of times the stack was recorded as a cookie since +the last reset. It is not decremented when events leave the ring buffer, +and it saturates at ``INT_MAX``. + +To view statistics:: + + cat /sys/kernel/tracing/stack_map_stat + +Output:: + + generation: 1 + entries: 2500 + capacity: 16384 + successes: 148923 + fallbacks: 0 + +``entries`` is the number of map elements reserved since the last reset; +an insertion in progress can make it briefly exceed the number of records +in ``stack_map``. ``successes`` is the number of stacks recorded as a +cookie. ``fallbacks`` is the number of stacks the map could not store +because the element pool was exhausted or the probe limit was reached; +those are recorded in full instead. The other cases above are not +counted. A growing +``fallbacks`` count means the map is too small or too crowded; a reset +makes room again. The file is empty while no map is allocated. + +To reset the stack map:: + + echo 0 > /sys/kernel/tracing/stack_map + +To release the map once the option is off:: + + echo 0 > /sys/kernel/tracing/options/stackmap + echo free > /sys/kernel/tracing/stack_map + +``free`` fails with ``-EBUSY`` while the option is enabled. A ``stack_map`` +file opened for reading before ``free`` keeps the old map. ``free`` blocks +until all such files are closed, then releases the memory, so a process +that has one open must close it first. If ``free`` is written through a +file that is also open for reading, that file is not waited for: ``free`` +returns and the memory is released when that file is closed. If the writer +is killed while it waits, the map has already been removed, and its memory +is released when the last such file is closed. + +Reset clears the map without touching the trace buffer, and tracing does +not have to be stopped. Reset and ``free`` both leave cookies in the trace +buffer; the generation changes, so those cookies no longer match the map. +A cookie only resolves against ``stack_map`` records of the same +generation. The generation in a cookie is 32 bits. It only has to tell +resets apart, and wraps after 2^32 resets and frees. + +To resolve a trace that spans resets, save ``stack_map`` before each +reset. ``stack_map`` is not a snapshot of concurrent insertions: a read +can include stacks inserted while it is in progress and can miss stacks +inserted into slots it has already passed. A reader that spans a reset +can get records of two generations across separate ``read()`` calls; the +``gen`` field shows this. Stop tracing across the copy and the reset if +every record must resolve. + +Boot-time activation +==================== + +The stackmap option can be enabled from the kernel command line:: + + trace_options=stackmap,stacktrace + +The map is then allocated when tracefs initializes. Stacks recorded before +that are recorded in full. If the allocation fails, the option is turned +off and a warning is printed. + +The stack_map files +=================== + +``stack_map`` (mode 0640) and ``stack_map_stat`` (mode 0440) exist when +``CONFIG_FTRACE_STACKMAP`` is enabled, also before the map is allocated. +While no map is allocated, both are empty. Writing ``0`` to ``stack_map`` +resets the map; writing ``free`` releases it. + +Design +====== + +The stack map is modeled after ``tracing_map.c`` (used by hist triggers): + +- **Lookup/Insert**: Lock-free via ``cmpxchg``, safe in NMI and IRQ context + where RCU is watching +- **Memory**: Pre-allocated element pool, no allocation on the hot path +- **Collision**: Linear probing with a 2x over-provisioned table; probe + length is bounded so worst-case insert/lookup is O(1) +- **Scope**: Top level trace array only +- **Hash**: 32-bit jhash with a per-map random seed; full ``memcmp`` + confirms matches + +The map holds at most one copy of a stack per generation. When a lookup +reaches a slot with the same hash whose element is not yet published, +another context is inserting that stack, or another stack with the same +hash. The lookup does not wait, because in NMI context it may have +interrupted that insert, and it does not add a second copy; that one stack +is recorded in full instead. A context that loses the ``cmpxchg`` for a +free slot re-reads the slot for the same reason. + +Differences from ``tracing_map`` +-------------------------------- + +The table, the element pool and the probing follow ``tracing_map``. The +stack map differs where its callers differ: + +- ``tracing_map_clear()`` requires its caller to keep inserts out. The + stack map does this itself: new callers are kept out and admitted ones + are drained with RCU before the map is cleared, so a reset can be + written to ``stack_map`` while tracing is active. +- An insert that meets an unpublished element of the same hash records + the stack in full, which loses nothing. ``tracing_map`` spins instead, + since a lost insert there is a lost histogram update. +- Probing stops after 64 slots, so a slot claimed without an element can + stay claimed. ``tracing_map`` has no probe limit and releases such a + slot, so that its loop always reaches a free slot. +- The cookie is the slot index, so it stays valid while the map is not + reset; ``tracing_map`` hands out element pointers. diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst index 84f06bf0da9b..f90cf7575458 100644 --- a/Documentation/trace/ftrace.rst +++ b/Documentation/trace/ftrace.rst @@ -1396,6 +1396,12 @@ Here are the available options: When set, a stack trace is recorded after any trace event is recorded. + stackmap + When set, a recorded stack trace is replaced by a cookie that + refers to a single copy of the stack in the stack_map file. + Only for the top level trace array. See + Documentation/trace/ftrace-stackmap.rst. + branch Enable branch tracing with the tracer. This enables branch tracer along with the currently set tracer. Enabling this diff --git a/Documentation/trace/index.rst b/Documentation/trace/index.rst index 5d9bf4694d5d..ac8b1141c23a 100644 --- a/Documentation/trace/index.rst +++ b/Documentation/trace/index.rst @@ -33,6 +33,7 @@ the Linux kernel. ftrace ftrace-design ftrace-uses + ftrace-stackmap kprobes kprobetrace fprobetrace -- 2.34.1