[PATCH v8 0/5] tracing: Add stack trace deduplication for the ring buffer

[PATCH v8 0/5] tracing: Add stack trace deduplication for the ring buffer

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

Hi Steven, Masami,

With options/stackmap set, ftrace records a 64-bit cookie (map
generation << 32 | slot) instead of a full stack. A stack stored in the
map is kept once per generation and exported as text in stack_map, with
counters in stack_map_stat. A stack the map cannot store is recorded in
full as before; a stack recorded while RCU is not watching is discarded.
A cookie only resolves against stack_map records of the same generation.

Use case: long tracing on phones, e.g. slab allocations with stacks from
early boot on, where the ring buffer and the storage are small and the
buffer fills with the same stacks over and over. With
trace_options=stackmap, stacks are deduplicated from tracefs
initialization on.

Measured on QEMU arm64 (4 CPUs, buffer_size_kb=512 per CPU),
kmem_cache_alloc with options/stacktrace, the same fixed workload in
both runs (100 rounds of 500 open()/close() of /proc/self/stat):

                  events retained   retained bytes/event
  stackmap off           4,867             107.4
  stackmap on           10,552              49.8

Bytes/event is the ring buffer data kept divided by the events kept, not
the size of one event. The gain depends on how often stacks repeat. This
measures retention only, not the CPU cost of the hash lookup; I only have
QEMU here, where timing numbers would not mean much.

This series is the core, the recording path, the statistics, the
documentation and a selftest. The binary export from v7 will carry the
generation in its header, so I would like the core and the cookie format
to settle first; the binary export, its parser and the capacity boot
parameter will follow as a separate series.

I used an LLM for a lot of this, the code and tests as well as the
comments and changelogs. The design is mine, and I went over every
change and the test results myself. After your v7 review I went through
the comments and changelogs again and cut most of them.

Design notes
============

- The hash table is modeled on tracing_map, the map behind hist
  triggers. The documentation lists where it differs: the slot index is
  the cookie, an insert that meets an unpublished element records the
  stack in full instead of spinning, and the map can be reset while
  tracing runs.
- Only the top level trace array is supported. A map per instance would
  multiply the memory, and boot-mapped instances could not resolve
  cookies against the running kernel's map.
- The capacity is fixed at 16384 stacks; stack_map_stat shows when that
  is too small. A boot parameter for the size comes with the follow-up.
- Turning the option off keeps the map, so the trace already in the
  buffer can still be resolved; writing "free" releases it. free waits
  until the stack_map files opened on the old map are closed, so a reader
  that started before it reads a complete map.
- The trace output prints the cookie, not the stack, including in
  ftrace_dump_on_oops. Printing the stack from the map when the trace is
  read could be added later, since the map is kept after the option is
  turned off.
- Elements have a fixed size, so an insert only takes one atomic index
  increment; a variable-size pool would need an allocator usable from
  NMI.

Changes since v7
================

- Record a u64 cookie (generation, slot); the trace prints
  "<stack_id N gen G>" and stack_map records print the generation.
- Drop rcu_tasks_rude: a stack recorded while RCU is not watching is
  discarded, and reset and free only use synchronize_rcu() (Steven).
- Use a mutex instead of an rwsem for the readers (Steven).
- Move the hot path into record_stackmap() and the tracefs setup into a
  helper, with stubs instead of #ifdefs; name the init state enum (Steven).
- Trim the comments and changelogs (Steven).
- Capitalize the subjects (Steven), and use the "tracing:" prefix.
- Reset is "echo 0 > stack_map" only, and stack_map uses the " => "
  frame format of <stack trace>.
- stack_map_stat has one field per line, adds generation and capacity,
  drops table_size and success_rate, and renames drops to fallbacks,
  since those stacks are recorded in full, not lost.
- The map is allocated when the option is enabled at runtime; with
  trace_options=stackmap it is allocated during tracefs initialization.
  It stays allocated when the option is disabled, until "free" is written
  to stack_map.
- The map holds at most one copy of a stack per generation. If another
  context is inserting the same stack, that stack is recorded in full
  instead of waiting.
- Binary export, parser and capacity boot parameter split out, see above.
- RFC dropped.

v7: https://lore.kernel.org/linux-trace-kernel/20260912083753.3426176-1-lipengfei28@xiaomi.com/

Known limitations
=================

- Top level trace array only.
- Fixed capacity of 16384 stacks, about 8.6 MiB on 64-bit while allocated.
- Kernel stacks only.

Testing
=======

- Each patch builds on x86_64, and on arm32 with and without
  CONFIG_FTRACE_STACKMAP.
- QEMU arm64, with and without KASAN and lockdep: the selftest in patch
  5, boot with trace_options=stackmap,stacktrace, and a 60 s stress run
  of tracing with resets, stack_map and stack_map_stat readers, and free
  with and without an open reader. No warnings.
- Function tracer with func_stack_trace: every cookie in the trace
  resolves against stack_map. Filling all 16384 elements: fallbacks
  grow, the remaining stacks are recorded in full, every cookie still
  resolves, and a reset empties the map.

Based on 4539944e5151 ("Merge tag 'liveupdate-fixes-2026-07-22' ...").

Pengfei Li (5):
  tracing: Add lock-free stackmap for stack trace deduplication
  tracing: Use the stackmap from the ftrace stack recording path
  tracing: Add stackmap statistics interface
  Documentation: tracing: Document the ftrace stackmap
  selftests/ftrace: Add a stackmap basic functionality test

 Documentation/trace/ftrace-stackmap.rst       | 192 +++++++
 Documentation/trace/ftrace.rst                |   6 +
 Documentation/trace/index.rst                 |   1 +
 kernel/trace/Kconfig                          |  13 +
 kernel/trace/Makefile                         |   1 +
 kernel/trace/trace.c                          | 132 ++++-
 kernel/trace/trace.h                          |  15 +
 kernel/trace/trace_entries.h                  |  16 +
 kernel/trace/trace_functions_graph.c          |   1 +
 kernel/trace/trace_output.c                   |  24 +
 kernel/trace/trace_selftest.c                 |   1 +
 kernel/trace/trace_stackmap.c                 | 495 ++++++++++++++++++
 kernel/trace/trace_stackmap.h                 |  23 +
 .../ftrace/test.d/ftrace/stackmap-basic.tc    | 222 ++++++++
 14 files changed, 1139 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/trace/ftrace-stackmap.rst
 create mode 100644 kernel/trace/trace_stackmap.c
 create mode 100644 kernel/trace/trace_stackmap.h
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/stackmap-basic.tc


base-commit: 4539944e515183668109bdf4d0c3d7d228383d88
-- 
2.34.1

[PATCH v8 1/5] tracing: Add lock-free stackmap for stack trace deduplication

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

A kernel stack costs 80-160 bytes per ftrace event, and the same stacks
recur. Add a preallocated hash map that stores each distinct stack once
and returns a 64-bit cookie, made of the map generation and the table
slot, that the stack recording path can record instead of the stack.

Lookup and insert are lock-free and may run in NMI context, so the map
depends on ARCH_HAVE_NMI_SAFE_CMPXCHG like tracing_map. Reset clears the
map in place and bumps the generation so that old cookies stop matching.
The stack_map file that resolves cookies is created by the next patch.

Assisted-by: LLM
Signed-off-by: Pengfei Li <lipengfei28@xiaomi.com>
---
 kernel/trace/Kconfig          |  13 ++
 kernel/trace/Makefile         |   1 +
 kernel/trace/trace.h          |   3 +
 kernel/trace/trace_stackmap.c | 382 ++++++++++++++++++++++++++++++++++
 kernel/trace/trace_stackmap.h |  22 ++
 5 files changed, 421 insertions(+)
 create mode 100644 kernel/trace/trace_stackmap.c
 create mode 100644 kernel/trace/trace_stackmap.h

diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index 084f34dc6c9f..a83c53c02eea 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -412,6 +412,19 @@ config STACK_TRACER
 
 	  Say N if unsure.
 
+config FTRACE_STACKMAP
+	bool "Ftrace stack map deduplication"
+	depends on TRACING
+	depends on STACKTRACE
+	depends on ARCH_HAVE_NMI_SAFE_CMPXCHG
+	help
+	  Store each distinct kernel stack recorded by ftrace once, in a
+	  table allocated when options/stackmap is enabled, and record a
+	  64-bit cookie in the ring buffer instead of the full stack.
+	  See Documentation/trace/ftrace-stackmap.rst.
+
+	  If unsure, say N.
+
 config TRACE_PREEMPT_TOGGLE
 	bool
 	help
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index f934ff586bd4..7710ec2659e9 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -86,6 +86,7 @@ obj-$(CONFIG_HWLAT_TRACER) += trace_hwlat.o
 obj-$(CONFIG_OSNOISE_TRACER) += trace_osnoise.o
 obj-$(CONFIG_NOP_TRACER) += trace_nop.o
 obj-$(CONFIG_STACK_TRACER) += trace_stack.o
+obj-$(CONFIG_FTRACE_STACKMAP) += trace_stackmap.o
 obj-$(CONFIG_MMIOTRACE) += trace_mmiotrace.o
 obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += trace_functions_graph.o
 obj-$(CONFIG_TRACE_BRANCH_PROFILING) += trace_branch.o
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 80fe152af1dd..7b2cdc0dc253 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -453,6 +453,9 @@ struct trace_array {
 	struct cond_snapshot	*cond_snapshot;
 #endif
 	struct trace_func_repeats	__percpu *last_func_repeats;
+#ifdef CONFIG_FTRACE_STACKMAP
+	struct ftrace_stackmap __rcu	*stackmap;
+#endif
 	/*
 	 * On boot up, the ring buffer is set to the minimum size, so that
 	 * we do not waste memory on systems that are not using tracing.
diff --git a/kernel/trace/trace_stackmap.c b/kernel/trace/trace_stackmap.c
new file mode 100644
index 000000000000..e5ffd1850ebf
--- /dev/null
+++ b/kernel/trace/trace_stackmap.c
@@ -0,0 +1,382 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Stack trace deduplication for ftrace: each distinct stack is stored once,
+ * and the ring buffer records a cookie (map generation << 32 | slot).
+ */
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/jhash.h>
+#include <linux/seq_file.h>
+#include <linux/vmalloc.h>
+#include <linux/atomic.h>
+#include <linux/random.h>
+#include <linux/rcupdate.h>
+#include <linux/refcount.h>
+
+#include "trace.h"
+#include "trace_stackmap.h"
+
+#define FTRACE_STACKMAP_BITS		14
+#define FTRACE_STACKMAP_MAX_PROBE	64
+
+struct stackmap_elt {
+	u32		nr;
+	atomic_t	ref_count;
+	unsigned long	ips[FTRACE_STACKMAP_MAX_DEPTH];
+};
+
+struct stackmap_entry {
+	u32			key;
+	struct stackmap_elt	*val;
+};
+
+struct ftrace_stackmap {
+	unsigned int		map_bits;
+	unsigned int		map_size;
+	unsigned int		max_elts;
+	u32			hash_seed;
+	atomic_t		next_elt;
+	struct stackmap_entry	*entries;
+	struct stackmap_elt	*elts;
+	atomic_t		resetting;
+	u32			generation;
+	refcount_t		ref;
+};
+
+/* Serializes tracefs readers against reset. */
+static DEFINE_MUTEX(stackmap_lock);
+
+static u32 stackmap_next_gen = 1;
+
+static struct ftrace_stackmap *stackmap_locked(struct trace_array *tr)
+{
+	return rcu_dereference_protected(tr->stackmap,
+					 lockdep_is_held(&trace_types_lock));
+}
+
+static struct stackmap_elt *stackmap_load_elt(struct stackmap_entry *entry)
+{
+	/* Pairs with smp_store_release() in ftrace_stackmap_get_cookie(). */
+	return smp_load_acquire(&entry->val);
+}
+
+static struct stackmap_elt *stackmap_get_elt(struct ftrace_stackmap *smap)
+{
+	int idx;
+
+	idx = atomic_fetch_add_unless(&smap->next_elt, 1, smap->max_elts);
+	return idx < smap->max_elts ? &smap->elts[idx] : NULL;
+}
+
+static void ftrace_stackmap_destroy(struct ftrace_stackmap *smap)
+{
+	vfree(smap->elts);
+	vfree(smap->entries);
+	kfree(smap);
+}
+
+static struct ftrace_stackmap *ftrace_stackmap_create(void)
+{
+	struct ftrace_stackmap *smap;
+
+	smap = kzalloc_obj(*smap, GFP_KERNEL);
+	if (!smap)
+		return ERR_PTR(-ENOMEM);
+
+	smap->map_bits = FTRACE_STACKMAP_BITS;
+	smap->max_elts = 1U << FTRACE_STACKMAP_BITS;
+	smap->map_size = 1U << (FTRACE_STACKMAP_BITS + 1);
+
+	smap->entries = __vcalloc(smap->map_size, sizeof(*smap->entries),
+				  GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
+	if (!smap->entries)
+		goto fail;
+
+	smap->elts = __vcalloc(smap->max_elts, sizeof(*smap->elts),
+			       GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
+	if (!smap->elts)
+		goto fail;
+
+	smap->hash_seed = get_random_u32();
+	smap->generation = stackmap_next_gen;
+	refcount_set(&smap->ref, 1);
+	return smap;
+
+fail:
+	ftrace_stackmap_destroy(smap);
+	return ERR_PTR(-ENOMEM);
+}
+
+int ftrace_stackmap_alloc(struct trace_array *tr)
+{
+	struct ftrace_stackmap *smap;
+
+	if (stackmap_locked(tr))
+		return 0;
+
+	smap = ftrace_stackmap_create();
+	if (IS_ERR(smap))
+		return PTR_ERR(smap);
+
+	rcu_assign_pointer(tr->stackmap, smap);
+	return 0;
+}
+
+static struct ftrace_stackmap *stackmap_get(struct trace_array *tr)
+{
+	struct ftrace_stackmap *smap;
+
+	guard(mutex)(&trace_types_lock);
+	smap = stackmap_locked(tr);
+	if (smap)
+		refcount_inc(&smap->ref);
+	return smap;
+}
+
+static void stackmap_put(struct ftrace_stackmap *smap)
+{
+	if (!smap)
+		return;
+	if (refcount_dec_and_test(&smap->ref))
+		ftrace_stackmap_destroy(smap);
+}
+
+static void stackmap_reset(struct trace_array *tr)
+{
+	struct ftrace_stackmap *smap = stackmap_locked(tr);
+
+	if (!smap)
+		return;
+
+	/* Keep new callers out, then wait for the ones already inside. */
+	atomic_set(&smap->resetting, 1);
+	synchronize_rcu();
+
+	scoped_guard(mutex, &stackmap_lock) {
+		memset(smap->entries, 0, sizeof(*smap->entries) * smap->map_size);
+		memset(smap->elts, 0, sizeof(*smap->elts) * smap->max_elts);
+		atomic_set(&smap->next_elt, 0);
+		smap->generation++;
+	}
+
+	/* Pairs with atomic_read_acquire() in ftrace_stackmap_get_cookie(). */
+	atomic_set_release(&smap->resetting, 0);
+}
+
+bool ftrace_stackmap_get_cookie(struct ftrace_stackmap *smap,
+				unsigned long *ips, unsigned int nr_entries,
+				u64 *cookie)
+{
+	u32 key_hash, idx, test_key, trace_len;
+	struct stackmap_entry *entry;
+	struct stackmap_elt *val;
+	int probes = 0;
+
+	/* Pairs with atomic_set_release() in stackmap_reset(). */
+	if (atomic_read_acquire(&smap->resetting))
+		return false;
+
+	trace_len = nr_entries * sizeof(unsigned long);
+	key_hash = jhash2((const u32 *)ips, trace_len / sizeof(u32),
+			  smap->hash_seed);
+	/* Zero marks a free slot. */
+	if (key_hash == 0)
+		key_hash = 1;
+
+	idx = key_hash >> (32 - (smap->map_bits + 1));
+
+	while (probes < FTRACE_STACKMAP_MAX_PROBE) {
+		idx &= smap->map_size - 1;
+		entry = &smap->entries[idx];
+		test_key = READ_ONCE(entry->key);
+
+		if (test_key == key_hash) {
+			val = stackmap_load_elt(entry);
+			/* Insert in progress; from NMI it may be the interrupted one. */
+			if (!val)
+				return false;
+			if (READ_ONCE(val->nr) == nr_entries &&
+			    memcmp(val->ips, ips, trace_len) == 0) {
+				atomic_add_unless(&val->ref_count, 1, INT_MAX);
+				goto found;
+			}
+		} else if (!test_key) {
+			if (atomic_read(&smap->next_elt) >= smap->max_elts)
+				break;
+			/* Lost the race; the slot may now hold this stack. */
+			if (cmpxchg(&entry->key, 0, key_hash) != 0)
+				continue;
+
+			val = stackmap_get_elt(smap);
+			if (!val)
+				break;
+
+			val->nr = nr_entries;
+			atomic_set(&val->ref_count, 1);
+			memcpy(val->ips, ips, trace_len);
+			/* Pairs with stackmap_load_elt(). */
+			smp_store_release(&entry->val, val);
+			goto found;
+		}
+
+		idx++;
+		probes++;
+	}
+
+	return false;
+found:
+	*cookie = ((u64)smap->generation << FTRACE_STACKMAP_COOKIE_GEN_SHIFT) | idx;
+	return true;
+}
+
+struct stackmap_seq_private {
+	struct trace_array	*tr;
+	struct ftrace_stackmap	*smap;
+};
+
+static struct stackmap_entry *stackmap_seq_seek(struct ftrace_stackmap *smap,
+						loff_t *pos)
+{
+	loff_t i;
+
+	for (i = *pos; i < smap->map_size; i++) {
+		if (READ_ONCE(smap->entries[i].key) &&
+		    stackmap_load_elt(&smap->entries[i])) {
+			*pos = i;
+			return &smap->entries[i];
+		}
+	}
+	/* Advance past the end or seq_read() repeats the last entry. */
+	*pos = smap->map_size;
+	return NULL;
+}
+
+static void *stackmap_seq_start(struct seq_file *m, loff_t *pos)
+{
+	struct stackmap_seq_private *priv = m->private;
+
+	mutex_lock(&stackmap_lock);
+	if (!priv->smap)
+		return NULL;
+	return stackmap_seq_seek(priv->smap, pos);
+}
+
+static void *stackmap_seq_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	struct stackmap_seq_private *priv = m->private;
+
+	(*pos)++;
+	return stackmap_seq_seek(priv->smap, pos);
+}
+
+static void stackmap_seq_stop(struct seq_file *m, void *v)
+{
+	/* seq_file calls stop() even when start() returned NULL. */
+	mutex_unlock(&stackmap_lock);
+}
+
+static int stackmap_seq_show(struct seq_file *m, void *v)
+{
+	struct stackmap_entry *entry = v;
+	struct stackmap_seq_private *priv = m->private;
+	struct stackmap_elt *elt;
+	u32 idx = entry - priv->smap->entries;
+	u32 i, nr;
+
+	elt = stackmap_load_elt(entry);
+	if (!elt)
+		return 0;
+
+	nr = READ_ONCE(elt->nr);
+	if (nr > FTRACE_STACKMAP_MAX_DEPTH)
+		nr = FTRACE_STACKMAP_MAX_DEPTH;
+
+	seq_printf(m, "stack_id %u gen %u ref %u depth %u\n",
+		   idx, priv->smap->generation, atomic_read(&elt->ref_count), nr);
+	for (i = 0; i < nr; i++) {
+		unsigned long ip = elt->ips[i];
+
+		if (ip == FTRACE_TRAMPOLINE_MARKER)
+			seq_puts(m, " => [FTRACE TRAMPOLINE]\n");
+		else
+			seq_printf(m, " => %pS\n", (void *)ip);
+	}
+	return 0;
+}
+
+static const struct seq_operations stackmap_seq_ops = {
+	.start	= stackmap_seq_start,
+	.next	= stackmap_seq_next,
+	.stop	= stackmap_seq_stop,
+	.show	= stackmap_seq_show,
+};
+
+static int stackmap_open(struct inode *inode, struct file *file)
+{
+	struct trace_array *tr = inode->i_private;
+	struct stackmap_seq_private *priv;
+	struct seq_file *m;
+	int ret;
+
+	ret = tracing_check_open_get_tr(tr);
+	if (ret)
+		return ret;
+
+	ret = seq_open_private(file, &stackmap_seq_ops,
+			       sizeof(struct stackmap_seq_private));
+	if (ret) {
+		trace_array_put(tr);
+		return ret;
+	}
+
+	m = file->private_data;
+	priv = m->private;
+	priv->tr = tr;
+	if (file->f_mode & FMODE_READ)
+		priv->smap = stackmap_get(tr);
+	return 0;
+}
+
+static int stackmap_release(struct inode *inode, struct file *file)
+{
+	struct seq_file *m = file->private_data;
+	struct stackmap_seq_private *priv = m->private;
+	struct ftrace_stackmap *smap = priv->smap;
+	struct trace_array *tr = priv->tr;
+	int ret;
+
+	ret = seq_release_private(inode, file);
+	stackmap_put(smap);
+	trace_array_put(tr);
+	return ret;
+}
+
+static ssize_t stackmap_write(struct file *file, const char __user *ubuf,
+			      size_t count, loff_t *ppos)
+{
+	struct seq_file *m = file->private_data;
+	struct stackmap_seq_private *priv = m->private;
+	char buf[8];
+
+	if (count >= sizeof(buf))
+		return -EINVAL;
+	if (copy_from_user(buf, ubuf, count))
+		return -EFAULT;
+	buf[count] = '\0';
+
+	if (sysfs_streq(buf, "0")) {
+		guard(mutex)(&trace_types_lock);
+		stackmap_reset(priv->tr);
+		return count;
+	}
+	return -EINVAL;
+}
+
+const struct file_operations ftrace_stackmap_fops = {
+	.open		= stackmap_open,
+	.read		= seq_read,
+	.write		= stackmap_write,
+	.llseek		= seq_lseek,
+	.release	= stackmap_release,
+};
diff --git a/kernel/trace/trace_stackmap.h b/kernel/trace/trace_stackmap.h
new file mode 100644
index 000000000000..6699988da390
--- /dev/null
+++ b/kernel/trace/trace_stackmap.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _TRACE_STACKMAP_H
+#define _TRACE_STACKMAP_H
+
+#include <linux/types.h>
+
+#define FTRACE_STACKMAP_MAX_DEPTH	64
+
+/* A cookie holds the generation in its high half and the slot below. */
+#define FTRACE_STACKMAP_COOKIE_GEN_SHIFT	32
+
+struct trace_array;
+struct ftrace_stackmap;
+
+int ftrace_stackmap_alloc(struct trace_array *tr);
+bool ftrace_stackmap_get_cookie(struct ftrace_stackmap *smap,
+				unsigned long *ips, unsigned int nr_entries,
+				u64 *cookie);
+
+extern const struct file_operations ftrace_stackmap_fops;
+
+#endif /* _TRACE_STACKMAP_H */
-- 
2.34.1

[PATCH v8 2/5] tracing: Use the stackmap from the ftrace stack recording path

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

With the new stackmap option set, __ftrace_trace_stack() records a
TRACE_STACK_ID event holding the cookie instead of the full stack, and
records the full stack when the map cannot store it. A stack recorded
while RCU is not watching is discarded, as reset and free rely on
synchronize_rcu(). Only the top level trace array supports the option.

The map is allocated when the option is first enabled and kept when it
is disabled, so the trace stays resolvable. Writing "free" to stack_map
releases it once the files that opened it are closed.

Assisted-by: LLM
Signed-off-by: Pengfei Li <lipengfei28@xiaomi.com>
---
 kernel/trace/trace.c                 | 128 ++++++++++++++++++++++++++-
 kernel/trace/trace.h                 |  12 +++
 kernel/trace/trace_entries.h         |  16 ++++
 kernel/trace/trace_functions_graph.c |   1 +
 kernel/trace/trace_output.c          |  24 +++++
 kernel/trace/trace_selftest.c        |   1 +
 kernel/trace/trace_stackmap.c        |  35 ++++++++
 7 files changed, 214 insertions(+), 3 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 18710c190c92..cdae2976d224 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -57,6 +57,7 @@
 
 #include "trace.h"
 #include "trace_output.h"
+#include "trace_stackmap.h"
 
 #ifdef CONFIG_FTRACE_STARTUP_TEST
 /*
@@ -509,12 +510,13 @@ EXPORT_SYMBOL_GPL(unregister_ftrace_export);
 /* trace_options that are only supported by global_trace */
 #define TOP_LEVEL_TRACE_FLAGS (TRACE_ITER(PRINTK) |			\
 	       TRACE_ITER(PRINTK_MSGONLY) | TRACE_ITER(RECORD_CMD) |	\
-	       TRACE_ITER(PROF_TEXT_OFFSET) | FPROFILE_DEFAULT_FLAGS)
+	       TRACE_ITER(PROF_TEXT_OFFSET) | TRACE_ITER(STACKMAP) |	\
+	       FPROFILE_DEFAULT_FLAGS)
 
 /* trace_flags that are default zero for instances */
 #define ZEROED_TRACE_FLAGS \
 	(TRACE_ITER(EVENT_FORK) | TRACE_ITER(FUNC_FORK) | TRACE_ITER(TRACE_PRINTK) | \
-	 TRACE_ITER(COPY_MARKER))
+	 TRACE_ITER(COPY_MARKER) | TRACE_ITER(STACKMAP))
 
 /*
  * The global_trace is the descriptor that holds the top-level tracing
@@ -2118,6 +2120,106 @@ struct ftrace_stacks {
 static DEFINE_PER_CPU(struct ftrace_stacks, ftrace_stacks);
 static DEFINE_PER_CPU(int, ftrace_stack_reserve);
 
+#ifdef CONFIG_FTRACE_STACKMAP
+enum stackmap_init_state {
+	STACKMAP_INIT_PENDING,	/* tracer_init_tracefs() not run yet */
+	STACKMAP_INIT_READY,	/* stack_map file created; map allocated on enable */
+	STACKMAP_INIT_FAILED,	/* permanent failure, never available */
+};
+
+/* Written and read under trace_types_lock. */
+static enum stackmap_init_state stackmap_state = STACKMAP_INIT_PENDING;
+
+/* Returns true if the stack was handled through the stack map. */
+static inline bool
+record_stackmap(struct trace_array *tr, struct trace_buffer *buffer,
+		struct ftrace_stack *fstack, unsigned int nr_entries,
+		unsigned int trace_ctx)
+{
+	struct ring_buffer_event *event;
+	struct ftrace_stackmap *smap;
+	struct stack_id_entry *entry;
+	u64 cookie;
+
+	if (!(tr->trace_flags & TRACE_ITER(STACKMAP)))
+		return false;
+
+	/* Reset and free use synchronize_rcu(), which does not wait for this. */
+	if (unlikely(!rcu_is_watching()))
+		return true;
+
+	if (!nr_entries || nr_entries > FTRACE_STACKMAP_MAX_DEPTH)
+		return false;
+
+	smap = rcu_dereference_sched(tr->stackmap);
+	if (!smap)
+		return false;
+
+	event = __trace_buffer_lock_reserve(buffer, TRACE_STACK_ID,
+					    sizeof(*entry), trace_ctx);
+	if (!event)
+		return false;
+
+	if (!ftrace_stackmap_get_cookie(smap, fstack->calls, nr_entries, &cookie)) {
+		__trace_event_discard_commit(buffer, event);
+		return false;
+	}
+
+	entry = ring_buffer_event_data(event);
+	entry->stack_cookie = cookie;
+	__buffer_unlock_commit(buffer, event);
+	return true;
+}
+
+static void __init stackmap_mark_init_failed(void)
+{
+	guard(mutex)(&trace_types_lock);
+	stackmap_state = STACKMAP_INIT_FAILED;
+	global_trace.trace_flags &= ~TRACE_ITER(STACKMAP);
+}
+
+static int stackmap_set_flag(struct trace_array *tr)
+{
+	/* The stack map is only for the top level trace array. */
+	if (tr != &global_trace || stackmap_state == STACKMAP_INIT_FAILED)
+		return -EINVAL;
+	/* stackmap_init_tracefs() allocates the map for trace_options=stackmap. */
+	if (stackmap_state != STACKMAP_INIT_READY)
+		return 0;
+	return ftrace_stackmap_alloc(tr);
+}
+
+static __init void stackmap_init_tracefs(void)
+{
+	if (!trace_create_file("stack_map", TRACE_MODE_WRITE, NULL,
+			       &global_trace, &ftrace_stackmap_fops)) {
+		stackmap_mark_init_failed();
+		return;
+	}
+
+	guard(mutex)(&trace_types_lock);
+	stackmap_state = STACKMAP_INIT_READY;
+	/* trace_options=stackmap on the command line */
+	if ((global_trace.trace_flags & TRACE_ITER(STACKMAP)) &&
+	    ftrace_stackmap_alloc(&global_trace)) {
+		pr_warn("ftrace stackmap: allocation failed, stackmap disabled\n");
+		global_trace.trace_flags &= ~TRACE_ITER(STACKMAP);
+	}
+}
+#else
+static inline bool
+record_stackmap(struct trace_array *tr, struct trace_buffer *buffer,
+		struct ftrace_stack *fstack, unsigned int nr_entries,
+		unsigned int trace_ctx)
+{
+	return false;
+}
+
+static inline void stackmap_mark_init_failed(void) { }
+static inline int stackmap_set_flag(struct trace_array *tr) { return 0; }
+static inline void stackmap_init_tracefs(void) { }
+#endif /* CONFIG_FTRACE_STACKMAP */
+
 void __ftrace_trace_stack(struct trace_array *tr,
 			  struct trace_buffer *buffer,
 			  unsigned int trace_ctx,
@@ -2184,6 +2286,9 @@ void __ftrace_trace_stack(struct trace_array *tr,
 	}
 #endif
 
+	if (record_stackmap(tr, buffer, fstack, nr_entries, trace_ctx))
+		goto out;
+
 	event = __trace_buffer_lock_reserve(buffer, TRACE_STACK,
 				    struct_size(entry, caller, nr_entries),
 				    trace_ctx);
@@ -3996,6 +4101,15 @@ int set_tracer_flag(struct trace_array *tr, u64 mask, int enabled)
 			return -EINVAL;
 
 	switch (mask) {
+	case TRACE_ITER(STACKMAP):
+		if (enabled) {
+			int ret = stackmap_set_flag(tr);
+
+			if (ret)
+				return ret;
+		}
+		break;
+
 	case TRACE_ITER(TRACE_PRINTK):
 		if (enabled) {
 			update_printk_trace(tr);
@@ -4282,6 +4396,10 @@ static const char readme_msg[] =
 	"\t\t\t  traces\n"
 #endif
 #endif /* CONFIG_STACK_TRACER */
+#ifdef CONFIG_FTRACE_STACKMAP
+	"  stack_map\t\t- Stacks recorded as cookies with options/stackmap\n"
+	"\t\t\t  Write '0' to reset it, 'free' to release it\n"
+#endif
 #ifdef CONFIG_DYNAMIC_EVENTS
 	"  dynamic_events\t\t- Create/append/remove/show the generic dynamic events\n"
 	"\t\t\t  Write into this file to define/undefine new trace events.\n"
@@ -9222,6 +9340,8 @@ static __init void tracer_init_tracefs_work_func(struct work_struct *work)
 			NULL, &tracing_dyn_info_fops);
 #endif
 
+	stackmap_init_tracefs();
+
 	create_trace_instances(NULL);
 
 	update_tracer_options();
@@ -9234,8 +9354,10 @@ static __init int tracer_init_tracefs(void)
 	trace_access_lock_init();
 
 	ret = tracing_init_dentry();
-	if (ret)
+	if (ret) {
+		stackmap_mark_init_failed();
 		return 0;
+	}
 
 	if (trace_init_wq) {
 		INIT_WORK(&tracerfs_init_work, tracer_init_tracefs_work_func);
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 7b2cdc0dc253..70b36d45b855 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -57,6 +57,7 @@ enum trace_type {
 	TRACE_TIMERLAT,
 	TRACE_RAW_DATA,
 	TRACE_FUNC_REPEATS,
+	TRACE_STACK_ID,
 
 	__TRACE_LAST_TYPE,
 };
@@ -582,6 +583,8 @@ extern void __ftrace_bad_type(void);
 			  TRACE_GRAPH_RET);		\
 		IF_ASSIGN(var, ent, struct func_repeats_entry,		\
 			  TRACE_FUNC_REPEATS);				\
+		IF_ASSIGN(var, ent, struct stack_id_entry,		\
+			  TRACE_STACK_ID);				\
 		__ftrace_bad_type();					\
 	} while (0)
 
@@ -1452,6 +1455,14 @@ extern int trace_get_user(struct trace_parser *parser, const char __user *ubuf,
 # define STACK_FLAGS
 #endif
 
+#ifdef CONFIG_FTRACE_STACKMAP
+# define STACKMAP_FLAGS				\
+		C(STACKMAP,		"stackmap"),
+#else
+# define STACKMAP_FLAGS
+# define TRACE_ITER_STACKMAP_BIT	-1
+#endif
+
 #ifdef CONFIG_FUNCTION_PROFILER
 # define PROFILER_FLAGS					\
 		C(PROF_TEXT_OFFSET,	"prof-text-offset"),
@@ -1509,6 +1520,7 @@ extern int trace_get_user(struct trace_parser *parser, const char __user *ubuf,
 		FUNCTION_FLAGS					\
 		FGRAPH_FLAGS					\
 		STACK_FLAGS					\
+		STACKMAP_FLAGS					\
 		BRANCH_FLAGS					\
 		PROFILER_FLAGS					\
 		FPROFILE_FLAGS
diff --git a/kernel/trace/trace_entries.h b/kernel/trace/trace_entries.h
index 54417468fdeb..2b3dd23ef5a9 100644
--- a/kernel/trace/trace_entries.h
+++ b/kernel/trace/trace_entries.h
@@ -250,6 +250,22 @@ FTRACE_ENTRY(user_stack, userstack_entry,
 		 (void *)__entry->caller[6], (void *)__entry->caller[7])
 );
 
+#define STACK_COOKIE_GET_GEN(entry)	((u32)((entry)->stack_cookie >> 32))
+#define STACK_COOKIE_GET_ID(entry)	((u32)((entry)->stack_cookie))
+
+FTRACE_ENTRY(stack_id, stack_id_entry,
+
+	TRACE_STACK_ID,
+
+	F_STRUCT(
+		__field(	u64,		stack_cookie	)
+	),
+
+	F_printk("<stack_id %u gen %u>",
+		 STACK_COOKIE_GET_ID(__entry),
+		 STACK_COOKIE_GET_GEN(__entry))
+);
+
 /*
  * trace_printk entry:
  */
diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
index 0d2d3a2ea7dd..0e1a390a6130 100644
--- a/kernel/trace/trace_functions_graph.c
+++ b/kernel/trace/trace_functions_graph.c
@@ -1492,6 +1492,7 @@ print_graph_function_flags(struct trace_iterator *iter, u32 flags)
 		return print_graph_return(field, s, entry, iter, flags);
 	}
 	case TRACE_STACK:
+	case TRACE_STACK_ID:
 	case TRACE_FN:
 		/* dont trace stack and functions as comments */
 		return TRACE_TYPE_UNHANDLED;
diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c
index a5ad76175d10..0a45bb5e00ec 100644
--- a/kernel/trace/trace_output.c
+++ b/kernel/trace/trace_output.c
@@ -1517,6 +1517,29 @@ static struct trace_event trace_user_stack_event = {
 	.funcs		= &trace_user_stack_funcs,
 };
 
+static enum print_line_t trace_stack_id_print(struct trace_iterator *iter,
+					      int flags, struct trace_event *event)
+{
+	struct stack_id_entry *field;
+	struct trace_seq *s = &iter->seq;
+
+	trace_assign_type(field, iter->ent);
+	trace_seq_printf(s, "<stack_id %u gen %u>\n",
+			 STACK_COOKIE_GET_ID(field),
+			 STACK_COOKIE_GET_GEN(field));
+
+	return trace_handle_return(s);
+}
+
+static struct trace_event_functions trace_stack_id_funcs = {
+	.trace		= trace_stack_id_print,
+};
+
+static struct trace_event trace_stack_id_event = {
+	.type		= TRACE_STACK_ID,
+	.funcs		= &trace_stack_id_funcs,
+};
+
 /* TRACE_HWLAT */
 static enum print_line_t
 trace_hwlat_print(struct trace_iterator *iter, int flags,
@@ -1908,6 +1931,7 @@ static struct trace_event *events[] __initdata = {
 	&trace_wake_event,
 	&trace_stack_event,
 	&trace_user_stack_event,
+	&trace_stack_id_event,
 	&trace_bputs_event,
 	&trace_bprint_event,
 	&trace_print_event,
diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c
index 929c84075315..0c97065b0d68 100644
--- a/kernel/trace/trace_selftest.c
+++ b/kernel/trace/trace_selftest.c
@@ -14,6 +14,7 @@ static inline int trace_valid_entry(struct trace_entry *entry)
 	case TRACE_CTX:
 	case TRACE_WAKE:
 	case TRACE_STACK:
+	case TRACE_STACK_ID:
 	case TRACE_PRINT:
 	case TRACE_BRANCH:
 	case TRACE_GRAPH_ENT:
diff --git a/kernel/trace/trace_stackmap.c b/kernel/trace/trace_stackmap.c
index e5ffd1850ebf..b02eb22c76a9 100644
--- a/kernel/trace/trace_stackmap.c
+++ b/kernel/trace/trace_stackmap.c
@@ -47,6 +47,8 @@ struct ftrace_stackmap {
 /* Serializes tracefs readers against reset. */
 static DEFINE_MUTEX(stackmap_lock);
 
+static DECLARE_WAIT_QUEUE_HEAD(stackmap_free_wq);
+
 static u32 stackmap_next_gen = 1;
 
 static struct ftrace_stackmap *stackmap_locked(struct trace_array *tr)
@@ -140,6 +142,8 @@ static void stackmap_put(struct ftrace_stackmap *smap)
 		return;
 	if (refcount_dec_and_test(&smap->ref))
 		ftrace_stackmap_destroy(smap);
+	else
+		wake_up(&stackmap_free_wq);
 }
 
 static void stackmap_reset(struct trace_array *tr)
@@ -164,6 +168,32 @@ static void stackmap_reset(struct trace_array *tr)
 	atomic_set_release(&smap->resetting, 0);
 }
 
+/* @self is the map pinned by the writing file, which is not waited for. */
+static int stackmap_free(struct trace_array *tr, struct ftrace_stackmap *self)
+{
+	struct ftrace_stackmap *smap;
+	int ret;
+
+	scoped_guard(mutex, &trace_types_lock) {
+		if (tr->trace_flags & TRACE_ITER(STACKMAP))
+			return -EBUSY;
+		smap = stackmap_locked(tr);
+		if (!smap)
+			return 0;
+
+		RCU_INIT_POINTER(tr->stackmap, NULL);
+		synchronize_rcu();
+		/* Old cookies must not match the next map. */
+		stackmap_next_gen = smap->generation + 1;
+	}
+
+	/* Closing a file takes trace_types_lock, so wait without it. */
+	ret = wait_event_killable(stackmap_free_wq,
+				  refcount_read(&smap->ref) == 1 + (self == smap));
+	stackmap_put(smap);
+	return ret;
+}
+
 bool ftrace_stackmap_get_cookie(struct ftrace_stackmap *smap,
 				unsigned long *ips, unsigned int nr_entries,
 				u64 *cookie)
@@ -358,6 +388,7 @@ static ssize_t stackmap_write(struct file *file, const char __user *ubuf,
 	struct seq_file *m = file->private_data;
 	struct stackmap_seq_private *priv = m->private;
 	char buf[8];
+	int ret;
 
 	if (count >= sizeof(buf))
 		return -EINVAL;
@@ -370,6 +401,10 @@ static ssize_t stackmap_write(struct file *file, const char __user *ubuf,
 		stackmap_reset(priv->tr);
 		return count;
 	}
+	if (sysfs_streq(buf, "free")) {
+		ret = stackmap_free(priv->tr, priv->smap);
+		return ret ? ret : count;
+	}
 	return -EINVAL;
 }
 
-- 
2.34.1

[PATCH v8 3/5] tracing: Add stackmap statistics interface

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

Add stack_map_stat to report the map generation, the elements used, the
stacks recorded as a cookie, and the stacks the map could not store
because the element pool or the probe limit was exhausted, so that users
can tell when the map is too small.

Assisted-by: LLM
Signed-off-by: Pengfei Li <lipengfei28@xiaomi.com>
---
 kernel/trace/trace.c          |  4 ++
 kernel/trace/trace_stackmap.c | 78 +++++++++++++++++++++++++++++++++++
 kernel/trace/trace_stackmap.h |  1 +
 3 files changed, 83 insertions(+)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index cdae2976d224..e58e7936c867 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -2197,6 +2197,9 @@ static __init void stackmap_init_tracefs(void)
 		return;
 	}
 
+	trace_create_file("stack_map_stat", TRACE_MODE_READ, NULL,
+			  &global_trace, &ftrace_stackmap_stat_fops);
+
 	guard(mutex)(&trace_types_lock);
 	stackmap_state = STACKMAP_INIT_READY;
 	/* trace_options=stackmap on the command line */
@@ -4399,6 +4402,7 @@ static const char readme_msg[] =
 #ifdef CONFIG_FTRACE_STACKMAP
 	"  stack_map\t\t- Stacks recorded as cookies with options/stackmap\n"
 	"\t\t\t  Write '0' to reset it, 'free' to release it\n"
+	"  stack_map_stat\t- Usage of the stack map\n"
 #endif
 #ifdef CONFIG_DYNAMIC_EVENTS
 	"  dynamic_events\t\t- Create/append/remove/show the generic dynamic events\n"
diff --git a/kernel/trace/trace_stackmap.c b/kernel/trace/trace_stackmap.c
index b02eb22c76a9..5d2a406f39b9 100644
--- a/kernel/trace/trace_stackmap.c
+++ b/kernel/trace/trace_stackmap.c
@@ -42,6 +42,8 @@ struct ftrace_stackmap {
 	atomic_t		resetting;
 	u32			generation;
 	refcount_t		ref;
+	atomic_long_t __percpu	*successes;
+	atomic_long_t __percpu	*fallbacks;
 };
 
 /* Serializes tracefs readers against reset. */
@@ -73,6 +75,8 @@ static struct stackmap_elt *stackmap_get_elt(struct ftrace_stackmap *smap)
 
 static void ftrace_stackmap_destroy(struct ftrace_stackmap *smap)
 {
+	free_percpu(smap->fallbacks);
+	free_percpu(smap->successes);
 	vfree(smap->elts);
 	vfree(smap->entries);
 	kfree(smap);
@@ -100,6 +104,14 @@ static struct ftrace_stackmap *ftrace_stackmap_create(void)
 	if (!smap->elts)
 		goto fail;
 
+	smap->successes = alloc_percpu(atomic_long_t);
+	if (!smap->successes)
+		goto fail;
+
+	smap->fallbacks = alloc_percpu(atomic_long_t);
+	if (!smap->fallbacks)
+		goto fail;
+
 	smap->hash_seed = get_random_u32();
 	smap->generation = stackmap_next_gen;
 	refcount_set(&smap->ref, 1);
@@ -149,6 +161,7 @@ static void stackmap_put(struct ftrace_stackmap *smap)
 static void stackmap_reset(struct trace_array *tr)
 {
 	struct ftrace_stackmap *smap = stackmap_locked(tr);
+	int cpu;
 
 	if (!smap)
 		return;
@@ -161,6 +174,10 @@ static void stackmap_reset(struct trace_array *tr)
 		memset(smap->entries, 0, sizeof(*smap->entries) * smap->map_size);
 		memset(smap->elts, 0, sizeof(*smap->elts) * smap->max_elts);
 		atomic_set(&smap->next_elt, 0);
+		for_each_possible_cpu(cpu) {
+			atomic_long_set(per_cpu_ptr(smap->successes, cpu), 0);
+			atomic_long_set(per_cpu_ptr(smap->fallbacks, cpu), 0);
+		}
 		smap->generation++;
 	}
 
@@ -194,6 +211,11 @@ static int stackmap_free(struct trace_array *tr, struct ftrace_stackmap *self)
 	return ret;
 }
 
+static void stackmap_count(atomic_long_t __percpu *counter)
+{
+	atomic_long_add_unless(this_cpu_ptr(counter), 1, LONG_MAX);
+}
+
 bool ftrace_stackmap_get_cookie(struct ftrace_stackmap *smap,
 				unsigned long *ips, unsigned int nr_entries,
 				u64 *cookie)
@@ -254,8 +276,10 @@ bool ftrace_stackmap_get_cookie(struct ftrace_stackmap *smap,
 		probes++;
 	}
 
+	stackmap_count(smap->fallbacks);
 	return false;
 found:
+	stackmap_count(smap->successes);
 	*cookie = ((u64)smap->generation << FTRACE_STACKMAP_COOKIE_GEN_SHIFT) | idx;
 	return true;
 }
@@ -415,3 +439,57 @@ const struct file_operations ftrace_stackmap_fops = {
 	.llseek		= seq_lseek,
 	.release	= stackmap_release,
 };
+
+static int stackmap_stat_show(struct seq_file *m, void *v)
+{
+	struct trace_array *tr = m->private;
+	struct ftrace_stackmap *smap;
+	u64 successes = 0, fallbacks = 0;
+	int cpu;
+
+	/* Reset and free also take trace_types_lock. */
+	guard(mutex)(&trace_types_lock);
+	smap = stackmap_locked(tr);
+	if (!smap)
+		return 0;
+
+	for_each_possible_cpu(cpu) {
+		successes += atomic_long_read(per_cpu_ptr(smap->successes, cpu));
+		fallbacks += atomic_long_read(per_cpu_ptr(smap->fallbacks, cpu));
+	}
+
+	seq_printf(m, "generation: %u\n", smap->generation);
+	seq_printf(m, "entries: %d\n", atomic_read(&smap->next_elt));
+	seq_printf(m, "capacity: %u\n", smap->max_elts);
+	seq_printf(m, "successes: %llu\n", successes);
+	seq_printf(m, "fallbacks: %llu\n", fallbacks);
+	return 0;
+}
+
+static int stackmap_stat_open(struct inode *inode, struct file *file)
+{
+	struct trace_array *tr = inode->i_private;
+	int ret;
+
+	ret = tracing_check_open_get_tr(tr);
+	if (ret)
+		return ret;
+
+	ret = single_open(file, stackmap_stat_show, tr);
+	if (ret)
+		trace_array_put(tr);
+	return ret;
+}
+
+static int stackmap_stat_release(struct inode *inode, struct file *file)
+{
+	trace_array_put(inode->i_private);
+	return single_release(inode, file);
+}
+
+const struct file_operations ftrace_stackmap_stat_fops = {
+	.open		= stackmap_stat_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= stackmap_stat_release,
+};
diff --git a/kernel/trace/trace_stackmap.h b/kernel/trace/trace_stackmap.h
index 6699988da390..a18e41c08494 100644
--- a/kernel/trace/trace_stackmap.h
+++ b/kernel/trace/trace_stackmap.h
@@ -18,5 +18,6 @@ bool ftrace_stackmap_get_cookie(struct ftrace_stackmap *smap,
 				u64 *cookie);
 
 extern const struct file_operations ftrace_stackmap_fops;
+extern const struct file_operations ftrace_stackmap_stat_fops;
 
 #endif /* _TRACE_STACKMAP_H */
-- 
2.34.1

[PATCH v8 4/5] Documentation: tracing: Document the ftrace stackmap

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

Describe how to use the stackmap, how reset and free behave, and how to
resolve a trace that spans resets. List the option in ftrace.rst.

Assisted-by: LLM
Signed-off-by: Pengfei Li <lipengfei28@xiaomi.com>
---
 Documentation/trace/ftrace-stackmap.rst | 192 ++++++++++++++++++++++++
 Documentation/trace/ftrace.rst          |   6 +
 Documentation/trace/index.rst           |   1 +
 3 files changed, 199 insertions(+)
 create mode 100644 Documentation/trace/ftrace-stackmap.rst

diff --git a/Documentation/trace/ftrace-stackmap.rst b/Documentation/trace/ftrace-stackmap.rst
new file mode 100644
index 000000000000..58af7c564b10
--- /dev/null
+++ b/Documentation/trace/ftrace-stackmap.rst
@@ -0,0 +1,192 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================
+Ftrace Stack Map
+======================
+
+:Author: Pengfei Li <lipengfei28@xiaomi.com>
+
+Overview
+========
+
+The ftrace stack map provides stack trace deduplication for the ftrace
+ring buffer. When enabled, instead of a full kernel stack trace event
+(typically 80-160 bytes) for each stack the map can store, ftrace writes a
+fixed-size event of 20-24 bytes, depending on the architecture, that holds
+a 64-bit cookie: the map generation in the high 32 bits and the slot index
+in the low 32. The full stacks are kept once in a separate hash table and
+exported via tracefs for userspace to resolve.
+
+It is similar to eBPF's ``BPF_MAP_TYPE_STACK_TRACE``.
+
+Configuration
+=============
+
+Enable ``CONFIG_FTRACE_STACKMAP=y`` in the kernel config.
+
+The map holds up to 16384 distinct stacks. It is allocated when the
+``stackmap`` option is first enabled; if the allocation fails, enabling
+the option fails with ``-ENOMEM``. The map stays allocated while the
+option is turned off, so a trace can still be resolved, until ``free``
+is written to ``stack_map``. On a 64-bit kernel the map takes about
+8.6 MiB.
+
+Usage
+=====
+
+Enable stack deduplication for event stacks::
+
+    echo 1 > /sys/kernel/tracing/options/stackmap
+    echo 1 > /sys/kernel/tracing/options/stacktrace
+    echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable
+
+The function tracer records stacks through its own option instead of
+``stacktrace``::
+
+    echo vfs_read > /sys/kernel/tracing/set_ftrace_filter
+    echo function > /sys/kernel/tracing/current_tracer
+    echo 1 > /sys/kernel/tracing/options/func_stack_trace
+
+The trace output will show ``<stack_id N gen G>`` instead of full stack
+traces, where N is the slot and G the map generation::
+
+    sh-1234 [006] d.h.. 123.456789: <stack_id 42 gen 1>
+
+Stacks are not always replaced. A stack deeper than 64 frames, a stack
+recorded before the map is allocated or during a reset, a stack that
+another context is inserting at the same moment, and a stack the map
+cannot store (element pool exhausted, probe limit, or ring buffer
+reservation failure) are recorded in full. A trace can therefore contain
+both ``<stack_id N gen G>`` records and full ``<stack trace>`` records.
+With the option enabled, a stack recorded where RCU is not watching is
+discarded.
+
+To view the actual stacks::
+
+    cat /sys/kernel/tracing/stack_map
+
+Output format::
+
+    stack_id 42 gen 1 ref 1337 depth 8
+     => schedule+0x48/0xc0
+     => schedule_timeout+0x1c/0x30
+     ...
+
+``ref`` is the number of times the stack was recorded as a cookie since
+the last reset. It is not decremented when events leave the ring buffer,
+and it saturates at ``INT_MAX``.
+
+To view statistics::
+
+    cat /sys/kernel/tracing/stack_map_stat
+
+Output::
+
+    generation: 1
+    entries: 2500
+    capacity: 16384
+    successes: 148923
+    fallbacks: 0
+
+``entries`` is the number of map elements reserved since the last reset;
+an insertion in progress can make it briefly exceed the number of records
+in ``stack_map``. ``successes`` is the number of stacks recorded as a
+cookie. ``fallbacks`` is the number of stacks the map could not store
+because the element pool was exhausted or the probe limit was reached;
+those are recorded in full instead. The other cases above are not
+counted. A growing
+``fallbacks`` count means the map is too small or too crowded; a reset
+makes room again. The file is empty while no map is allocated.
+
+To reset the stack map::
+
+    echo 0 > /sys/kernel/tracing/stack_map
+
+To release the map once the option is off::
+
+    echo 0 > /sys/kernel/tracing/options/stackmap
+    echo free > /sys/kernel/tracing/stack_map
+
+``free`` fails with ``-EBUSY`` while the option is enabled. A ``stack_map``
+file opened for reading before ``free`` keeps the old map. ``free`` blocks
+until all such files are closed, then releases the memory, so a process
+that has one open must close it first. If ``free`` is written through a
+file that is also open for reading, that file is not waited for: ``free``
+returns and the memory is released when that file is closed. If the writer
+is killed while it waits, the map has already been removed, and its memory
+is released when the last such file is closed.
+
+Reset clears the map without touching the trace buffer, and tracing does
+not have to be stopped. Reset and ``free`` both leave cookies in the trace
+buffer; the generation changes, so those cookies no longer match the map.
+A cookie only resolves against ``stack_map`` records of the same
+generation. The generation in a cookie is 32 bits. It only has to tell
+resets apart, and wraps after 2^32 resets and frees.
+
+To resolve a trace that spans resets, save ``stack_map`` before each
+reset. ``stack_map`` is not a snapshot of concurrent insertions: a read
+can include stacks inserted while it is in progress and can miss stacks
+inserted into slots it has already passed. A reader that spans a reset
+can get records of two generations across separate ``read()`` calls; the
+``gen`` field shows this. Stop tracing across the copy and the reset if
+every record must resolve.
+
+Boot-time activation
+====================
+
+The stackmap option can be enabled from the kernel command line::
+
+    trace_options=stackmap,stacktrace
+
+The map is then allocated when tracefs initializes. Stacks recorded before
+that are recorded in full. If the allocation fails, the option is turned
+off and a warning is printed.
+
+The stack_map files
+===================
+
+``stack_map`` (mode 0640) and ``stack_map_stat`` (mode 0440) exist when
+``CONFIG_FTRACE_STACKMAP`` is enabled, also before the map is allocated.
+While no map is allocated, both are empty. Writing ``0`` to ``stack_map``
+resets the map; writing ``free`` releases it.
+
+Design
+======
+
+The stack map is modeled after ``tracing_map.c`` (used by hist triggers):
+
+- **Lookup/Insert**: Lock-free via ``cmpxchg``, safe in NMI and IRQ context
+  where RCU is watching
+- **Memory**: Pre-allocated element pool, no allocation on the hot path
+- **Collision**: Linear probing with a 2x over-provisioned table; probe
+  length is bounded so worst-case insert/lookup is O(1)
+- **Scope**: Top level trace array only
+- **Hash**: 32-bit jhash with a per-map random seed; full ``memcmp``
+  confirms matches
+
+The map holds at most one copy of a stack per generation. When a lookup
+reaches a slot with the same hash whose element is not yet published,
+another context is inserting that stack, or another stack with the same
+hash. The lookup does not wait, because in NMI context it may have
+interrupted that insert, and it does not add a second copy; that one stack
+is recorded in full instead. A context that loses the ``cmpxchg`` for a
+free slot re-reads the slot for the same reason.
+
+Differences from ``tracing_map``
+--------------------------------
+
+The table, the element pool and the probing follow ``tracing_map``. The
+stack map differs where its callers differ:
+
+- ``tracing_map_clear()`` requires its caller to keep inserts out. The
+  stack map does this itself: new callers are kept out and admitted ones
+  are drained with RCU before the map is cleared, so a reset can be
+  written to ``stack_map`` while tracing is active.
+- An insert that meets an unpublished element of the same hash records
+  the stack in full, which loses nothing. ``tracing_map`` spins instead,
+  since a lost insert there is a lost histogram update.
+- Probing stops after 64 slots, so a slot claimed without an element can
+  stay claimed. ``tracing_map`` has no probe limit and releases such a
+  slot, so that its loop always reaches a free slot.
+- The cookie is the slot index, so it stays valid while the map is not
+  reset; ``tracing_map`` hands out element pointers.
diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst
index 84f06bf0da9b..f90cf7575458 100644
--- a/Documentation/trace/ftrace.rst
+++ b/Documentation/trace/ftrace.rst
@@ -1396,6 +1396,12 @@ Here are the available options:
 	When set, a stack trace is recorded after any trace event
 	is recorded.
 
+  stackmap
+	When set, a recorded stack trace is replaced by a cookie that
+	refers to a single copy of the stack in the stack_map file.
+	Only for the top level trace array. See
+	Documentation/trace/ftrace-stackmap.rst.
+
   branch
 	Enable branch tracing with the tracer. This enables branch
 	tracer along with the currently set tracer. Enabling this
diff --git a/Documentation/trace/index.rst b/Documentation/trace/index.rst
index 5d9bf4694d5d..ac8b1141c23a 100644
--- a/Documentation/trace/index.rst
+++ b/Documentation/trace/index.rst
@@ -33,6 +33,7 @@ the Linux kernel.
    ftrace
    ftrace-design
    ftrace-uses
+   ftrace-stackmap
    kprobes
    kprobetrace
    fprobetrace
-- 
2.34.1

[PATCH v8 5/5] selftests/ftrace: Add a stackmap basic functionality test

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

Check that stacks are deduplicated, that a reset succeeds while tracing
runs and clears the map but not the ring buffer, that the statistics
match the map, and that free waits for open readers and the next map
continues the generation.

Assisted-by: LLM
Signed-off-by: Pengfei Li <lipengfei28@xiaomi.com>
---
 .../ftrace/test.d/ftrace/stackmap-basic.tc    | 222 ++++++++++++++++++
 1 file changed, 222 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/stackmap-basic.tc

diff --git a/tools/testing/selftests/ftrace/test.d/ftrace/stackmap-basic.tc b/tools/testing/selftests/ftrace/test.d/ftrace/stackmap-basic.tc
new file mode 100644
index 000000000000..f5d1ebd8728e
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/ftrace/stackmap-basic.tc
@@ -0,0 +1,222 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: ftrace - stackmap basic functionality
+# requires: stack_map stack_map_stat options/stackmap options/stacktrace events/sched/sched_switch/enable
+
+# Check deduplication, that reset clears the map but not the ring buffer,
+# that the statistics match the map, and the free and generation rules.
+
+fail() {
+    echo "FAIL: $1"
+    exit_fail
+}
+
+records() { awk '/^stack_id / { n++ } END { print n + 0 }' stack_map; }
+hits() { awk '/^stack_id / { s += $6 } END { print s + 0 }' stack_map; }
+max_ref() { awk '/^stack_id / { if ($6 > m) m = $6 } END { print m + 0 }' stack_map; }
+first_gen() { awk '/^stack_id / { print $4; exit }' stack_map; }
+stat_val() { awk -v k="$1:" '$1 == k { print $2 }' stack_map_stat; }
+
+wait_for_worker_stopped() {
+    tries=0
+    while [ "$tries" -lt 100 ]; do
+        state=
+        if state=$(awk '/^State:/ { print $2 }' "/proc/$worker/status" \
+                2>/dev/null); then
+            case "$state" in
+            T|t)
+                return 0
+                ;;
+            esac
+        else
+            return 1
+        fi
+        sleep 0.01
+        tries=$((tries + 1))
+    done
+    return 1
+}
+
+worker=
+free_pid=
+
+cleanup() {
+    exec 3<&- 2>/dev/null || :
+    [ -z "$free_pid" ] || wait "$free_pid" 2>/dev/null || :
+    disable_tracing 2>/dev/null || :
+    if [ -n "$worker" ]; then
+        kill -CONT "$worker" 2>/dev/null || :
+        kill "$worker" 2>/dev/null || :
+        wait "$worker" 2>/dev/null || :
+    fi
+    echo 0 > events/sched/sched_switch/filter 2>/dev/null || :
+    echo 0 > events/sched/sched_switch/enable 2>/dev/null || :
+    echo 0 > options/stackmap 2>/dev/null || :
+    echo 0 > options/stacktrace 2>/dev/null || :
+    echo free > stack_map 2>/dev/null || :
+}
+trap cleanup EXIT
+
+disable_tracing
+clear_trace
+echo 0 > stack_map || fail "initial stackmap reset failed"
+
+echo 1 > options/stackmap
+echo 1 > options/stacktrace
+echo 1 > events/sched/sched_switch/enable
+
+# A sleeping loop: a busy loop on an idle CPU is rarely switched out.
+(while :; do sleep 0.01; done) &
+worker=$!
+echo "prev_pid == $worker || next_pid == $worker" > \
+    events/sched/sched_switch/filter || fail "could not filter sched_switch workload"
+
+enable_tracing
+sleep 1
+
+# While the writer runs, reset only has to succeed. With it stopped and
+# tracing off, reset must leave the map empty.
+i=0
+while [ "$i" -lt 5 ]; do
+    tries=0
+    pre_hits=0
+    while [ "$tries" -lt 1000 ]; do
+        pre_hits=$(hits)
+        [ "$pre_hits" -ge 8 ] && break
+        sleep 0.01
+        tries=$((tries + 1))
+    done
+    [ "$pre_hits" -ge 8 ] ||
+        fail "active writer did not reach 8 hits before reset (got $pre_hits)"
+
+    echo 0 > stack_map || fail "stackmap reset failed with active writer"
+
+    kill -STOP "$worker" || fail "could not stop reset workload"
+    wait_for_worker_stopped ||
+        fail "reset workload did not enter the stopped state"
+    disable_tracing
+
+    echo 0 > stack_map || fail "stopped-writer verification reset failed"
+    n=$(records)
+    [ "$n" -eq 0 ] || fail "stopped-writer reset left $n records"
+    n=$(stat_val entries)
+    [ "$n" -eq 0 ] || fail "stopped-writer reset left stat entries $n"
+
+    kill -CONT "$worker" || fail "could not resume reset workload"
+    enable_tracing
+    i=$((i + 1))
+done
+
+# Reset leaves old cookies in the buffer; drop them.
+clear_trace
+
+# Wait until some record is hit twice, i.e. reused.
+i=0
+while [ "$i" -lt 100 ]; do
+    [ "$(max_ref)" -ge 2 ] && break
+    sleep 0.1
+    i=$((i + 1))
+done
+
+kill -STOP "$worker" || fail "could not stop final reset workload"
+wait_for_worker_stopped ||
+    fail "final reset workload did not enter the stopped state"
+disable_tracing
+
+entries=$(records)
+[ "$entries" -gt 0 ] || fail "stackmap did not refill after active-writer reset"
+ref=$(max_ref)
+[ "$ref" -ge 2 ] || fail "no record reused after reset ($entries records, max ref $ref)"
+
+# Nothing records now, so every stored stack and every hit is counted.
+n=$(stat_val entries)
+[ "$n" -eq "$entries" ] || fail "stat entries $n, stack_map has $entries records"
+n=$(stat_val successes)
+[ "$n" -eq "$(hits)" ] || fail "stat successes $n, stack_map has $(hits) hits"
+n=$(stat_val generation)
+[ "$n" -eq "$(first_gen)" ] || fail "stat generation $n, stack_map has $(first_gen)"
+
+kill -CONT "$worker" 2>/dev/null || :
+kill "$worker" 2>/dev/null || :
+wait "$worker" 2>/dev/null || :
+worker=
+# Without the filter, this shell's own sleeps produce the switches below.
+echo 0 > events/sched/sched_switch/filter
+
+# Stop the event before inspecting the fixed buffer and map contents.
+echo 0 > events/sched/sched_switch/enable
+trace_pairs=$(sed -n \
+    's/.*<stack_id \([0-9][0-9]*\) gen \([0-9][0-9]*\)>.*/\1:\2/p' \
+    trace | sort -u)
+[ -n "$trace_pairs" ] ||
+    fail "trace has no <stack_id N gen G> events after active-writer reset"
+
+# Every cookie left in the trace must match a current stack_map record.
+for pair in $trace_pairs; do
+    id=${pair%%:*}
+    gen=${pair##*:}
+    grep -q "^stack_id $id gen $gen " stack_map ||
+        fail "trace cookie id $id gen $gen has no current stack_map record"
+done
+
+# Reset clears the map only; the cookies in the ring buffer stay.
+gen=$(first_gen)
+before_records=$(grep "<stack_id" trace || true)
+echo 0 > stack_map || fail "stopped reset failed"
+[ "$(records)" -eq 0 ] || fail "reset did not clear the map"
+after_records=$(grep "<stack_id" trace || true)
+[ "$after_records" = "$before_records" ] || fail "reset changed the trace buffer records"
+gen=$((gen + 1))
+
+# Trace until the map holds a record, for up to 10 s.
+refill() {
+    echo 1 > events/sched/sched_switch/enable
+    enable_tracing
+    i=0
+    while [ "$(records)" -eq 0 ] && [ "$i" -lt 100 ]; do
+        sleep 0.1
+        i=$((i + 1))
+    done
+    disable_tracing
+    echo 0 > events/sched/sched_switch/enable
+}
+
+# free is refused while the option is enabled.
+if echo free > stack_map 2>/dev/null; then
+    fail "free accepted while options/stackmap is enabled"
+fi
+
+# Disabling keeps the map, so the trace can still be resolved.
+refill
+echo 0 > options/stackmap
+[ "$(records)" -gt 0 ] || fail "map released by disabling the option"
+[ "$(first_gen)" -eq "$gen" ] || fail "generation is $(first_gen) after reset (expected $gen)"
+
+# free waits for a reader that opened the map, which still reads it.
+exec 3< stack_map
+(exec 3<&-; echo free > stack_map) &
+free_pid=$!
+# The waiting writer sleeps killable (D); an exited one is a zombie (Z).
+i=0
+st=
+while [ "$i" -lt 100 ]; do
+    st=$(awk '/^State:/ { print $2 }' "/proc/$free_pid/status" 2>/dev/null)
+    [ "$st" = D ] && break
+    sleep 0.1
+    i=$((i + 1))
+done
+[ "$st" = D ] || fail "free is not blocked on the open reader (state '$st')"
+[ "$(records)" -eq 0 ] || fail "a new reader still sees the map during free"
+grep -q '^stack_id ' <&3 || fail "the open reader lost the map"
+exec 3<&-
+wait "$free_pid" || fail "free failed after the reader closed"
+free_pid=
+[ "$(records)" -eq 0 ] || fail "stack_map still has records after free"
+[ -z "$(cat stack_map_stat)" ] || fail "stack_map_stat not empty after free"
+
+# Re-enabling allocates a map that continues the generation.
+echo 1 > options/stackmap
+refill
+[ "$(records)" -gt 0 ] || fail "new map has no records"
+[ "$(first_gen)" -eq $((gen + 1)) ] ||
+    fail "generation after free is $(first_gen) (expected $((gen + 1)))"
-- 
2.34.1

Re: [PATCH v8 3/5] tracing: Add stackmap statistics interface

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

On Sat, 07 Nov 2026 06:04:59 +0000, sashiko-bot@kernel.org wrote:

> Can ftrace_stackmap_get_cookie() undercount smap->fallbacks when the
> element pool runs out during a concurrent insert?

Yes. The slot left without an element makes later lookups of that stack
take the "insert in progress" exit, so they are recorded in full but not
counted. It only happens once the pool is exhausted, when fallbacks is
already non-zero, so the counter still shows that the map is full; it
just reads low. The next version counts that case as a fallback.

Thanks,
Pengfei Li

Re: [PATCH v8 5/5] selftests/ftrace: Add a stackmap basic functionality test

[Li Pengfei]

From: Pengfei Li <lipengfei28@xiaomi.com>

On Sat, 07 Nov 2026 06:07:50 +0000, sashiko-bot@kernel.org wrote:

> Can clear_trace race with the resumed worker and leave the trace ring
> buffer empty when max_ref reaches 2?

Yes. Hits recorded between enable_tracing and clear_trace raise ref but
their cookies are cleared. The worker switches every 10 ms, so in
practice more events follow before tracing is disabled, but the test
should not depend on that. The next version clears the trace while the
worker is stopped and tracing is off, before resuming it on the last
iteration.

Thanks,
Pengfei Li

Re: [PATCH v8 2/5] tracing: Use the stackmap from the ftrace stack recording path

[kernel test robot]

Hi Li,

kernel test robot noticed the following build errors:

[auto build test ERROR on 4539944e515183668109bdf4d0c3d7d228383d88]

url:    https://github.com/intel-lab-lkp/linux/commits/Li-Pengfei/tracing-Add-lock-free-stackmap-for-stack-trace-deduplication/20261107-135135
base:   4539944e515183668109bdf4d0c3d7d228383d88
patch link:    https://lore.kernel.org/r/20261107055139.3138633-3-lipengfei28%40xiaomi.com
patch subject: [PATCH v8 2/5] tracing: Use the stackmap from the ftrace stack recording path
config: powerpc-randconfig-1001-20261108 (https://download.01.org/0day-ci/archive/20261108/202611080654.sqFxIlwl-lkp@intel.com/config)
compiler: powerpc-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261108/202611080654.sqFxIlwl-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202611080654.sqFxIlwl-lkp@intel.com/

All errors (new ones prefixed by >>):

   In file included from include/linux/rbtree.h:24,
                    from include/linux/mm_types.h:11,
                    from include/linux/mmzone.h:22,
                    from include/linux/gfp.h:7,
                    from include/linux/mm.h:8,
                    from include/linux/ring_buffer.h:5,
                    from kernel/trace/trace.c:15:
   kernel/trace/trace.c: In function 'record_stackmap':
>> include/linux/rcupdate.h:502:9: error: dereferencing pointer to incomplete type 'struct ftrace_stackmap'
     typeof(*p) *local = (typeof(*p) *__force)READ_ONCE(p); \
            ^
   include/linux/rcupdate.h:684:2: note: in expansion of macro '__rcu_dereference_check'
     __rcu_dereference_check((p), __UNIQUE_ID(rcu), \
     ^~~~~~~~~~~~~~~~~~~~~~~
   include/linux/rcupdate.h:758:34: note: in expansion of macro 'rcu_dereference_sched_check'
    #define rcu_dereference_sched(p) rcu_dereference_sched_check(p, 0)
                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
   kernel/trace/trace.c:2154:9: note: in expansion of macro 'rcu_dereference_sched'
     smap = rcu_dereference_sched(tr->stackmap);
            ^~~~~~~~~~~~~~~~~~~~~


vim +502 include/linux/rcupdate.h

76c8eaafe4f061f Paul E. McKenney        2021-04-21  492  
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  493  #define __rcu_access_pointer(p, local, space) \
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  494  ({ \
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  495  	typeof(*p) *local = (typeof(*p) *__force)READ_ONCE(p); \
423a86a610cad12 Joel Fernandes (Google  2018-12-12  496) 	rcu_check_sparse(p, space); \
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  497  	((typeof(*p) __force __kernel *)(local)); \
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  498  })
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  499  #define __rcu_dereference_check(p, local, c, space) \
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  500  ({ \
ac59853c06993a4 Pranith Kumar           2014-11-13  501  	/* Dependency order vs. p above. */ \
24ba53017e188e0 Chun-Hung Tseng         2021-09-15 @502  	typeof(*p) *local = (typeof(*p) *__force)READ_ONCE(p); \
f78f5b90c4ffa55 Paul E. McKenney        2015-06-18  503  	RCU_LOCKDEP_WARN(!(c), "suspicious rcu_dereference_check() usage"); \
423a86a610cad12 Joel Fernandes (Google  2018-12-12  504) 	rcu_check_sparse(p, space); \
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  505  	((typeof(*p) __force __kernel *)(local)); \
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  506  })
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  507  #define __rcu_dereference_protected(p, local, c, space) \
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  508  ({ \
f78f5b90c4ffa55 Paul E. McKenney        2015-06-18  509  	RCU_LOCKDEP_WARN(!(c), "suspicious rcu_dereference_protected() usage"); \
423a86a610cad12 Joel Fernandes (Google  2018-12-12  510) 	rcu_check_sparse(p, space); \
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  511  	((typeof(*p) __force __kernel *)(p)); \
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  512  })
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  513  #define __rcu_dereference_raw(p, local) \
995f1405610bd84 Paul E. McKenney        2016-07-01  514  ({ \
995f1405610bd84 Paul E. McKenney        2016-07-01  515  	/* Dependency order vs. p above. */ \
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  516  	typeof(p) local = READ_ONCE(p); \
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  517  	((typeof(*p) __force __kernel *)(local)); \
995f1405610bd84 Paul E. McKenney        2016-07-01  518  })
24ba53017e188e0 Chun-Hung Tseng         2021-09-15  519  #define rcu_dereference_raw(p) __rcu_dereference_raw(p, __UNIQUE_ID(rcu))
ca5ecddfa8fcbd9 Paul E. McKenney        2010-04-28  520  

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki