* [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
@ 2026-10-17 5:45 Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (5 more replies)
0 siblings, 6 replies; 59+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers,
Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also extends both
`lock_held()` and `perf lock contention --use-bpf` to support RCU and
SRCU.
For example:
# perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (with or without leading `&`, e.g.
`lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, etc., which are enabled on many
`!CONFIG_PREEMPT_RT` / `!CONFIG_PROVE_LOCKING` builds unless
`CONFIG_UNINLINE_SPIN_UNLOCK` or lockdep/preemption forces them out of
line), `_raw_spin_lock*` / `_raw_spin_unlock*` and
`_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and
have no BTF function entry points for `fentry`/`fexit` to attach to.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address), spinlocks, rwlocks,
semaphores, SRCU, and reader-locked `rw_semaphore`s (where
`__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
do not maintain an authoritative per-task owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
Patch overview:
1. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
2. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
updated by selectively attached `fentry`/`fexit` hooks.
3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (5):
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 51 +-
tools/perf/builtin-lock.c | 27 -
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 119 ++
tools/perf/util/bpf-filter.c | 1070 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_lock_contention.c | 167 ++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 410 +++----
tools/perf/util/bpf_skel/lock_data.h | 360 +++++-
tools/perf/util/bpf_skel/sample-filter.h | 71 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1051 ++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 22 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/parse-events.c | 3 +-
19 files changed, 3273 insertions(+), 332 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 59+ messages in thread* [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers ` (4 subsequent siblings) 5 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF sample filtering. Also add rw_semaphore.count and struct rt_mutex_base to the minimal vmlinux.h and extend get_lock_owner() to support rt_mutex and check writer-ownership and handoff flags. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +-------------- tools/perf/util/bpf_skel/lock_data.h | 316 ++++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + tools/perf/util/lock-contention.c | 18 + tools/perf/util/lock-contention.h | 13 + 6 files changed, 370 insertions(+), 274 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..feaf1cfd08cf 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1477,33 +1477,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..0f2118006005 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -96,13 +82,6 @@ struct { __uint(max_entries, MAX_ENTRIES); } task_data SEC(".maps"); -struct { - __uint(type, BPF_MAP_TYPE_HASH); - __uint(key_size, sizeof(__u64)); - __uint(value_size, sizeof(__u32)); - __uint(max_entries, MAX_ENTRIES); -} lock_syms SEC(".maps"); - struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(__u32)); @@ -138,6 +117,13 @@ struct { __uint(max_entries, 1); } cgroup_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, MAX_ENTRIES); +} lock_syms SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(long)); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -202,6 +170,11 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + int enabled; int perf_subsys_id = -1; @@ -222,8 +195,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,145 +800,24 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - -const volatile __u64 contig_page_data_addr; -const volatile __u64 node_data_addr; -const volatile int nr_nodes; -const volatile int sizeof_zone; - -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") -int BPF_PROG(collect_lock_syms) +int BPF_PROG(end_timestamp) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - + end_ts = bpf_ktime_get_ns(); return 0; } SEC("raw_tp/bpf_test_finish") -int BPF_PROG(end_timestamp) +int BPF_PROG(collect_lock_syms) { - end_ts = bpf_ktime_get_ns(); - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..fed445c9d8b4 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,312 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t count; + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex *mutex = (void *)lock; + + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + struct rw_semaphore___new *rwsem = (void *)lock; + + if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) { + __u64 count = BPF_CORE_READ(rwsem, count.counter); + + /* Must have RWSEM_WRITER_LOCKED (bit 0) set */ + if (!(count & 1UL)) + return NULL; + } + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *old = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(old, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + owner = BPF_CORE_READ(rwsem, owner.counter); + } +#else + /* assume new struct */ + owner = BPF_CORE_READ(rwsem, owner.counter); +#endif + /* RWSEM_READER_OWNED is bit 0 of owner */ + if (flags == LCB_F_WRITE && (owner & 1UL)) + return NULL; + } else if (flags == LCB_F_RT) { + struct rt_mutex_base *rtm = (void *)lock; + + if (bpf_core_type_exists(struct rt_mutex_base)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..a1bb78d6e29c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -22,6 +22,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers ` (3 subsequent siblings) 5 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported <lock> specifications match perf lock contention: - Global/static kernel lock symbols (with or without '&', e.g. lock_held(&cgroup_mutex) or lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. To minimize tracing overhead, only the BPF hooks for the requested lock family and mode are autoloaded and attached. Example usage: # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 41 +- tools/perf/util/bpf-filter.c | 934 ++++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_skel/sample-filter.h | 59 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 920 ++++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2021 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..246ae085744d 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,46 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)'). + <lock> can be a kernel lock symbol name (with or without a leading + '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), + a hexadecimal lock address, a slab cache name prefixed with '&' + (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'semaphore'). Both read/shared and write/exclusive tracking are + supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), + 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') + support write/exclusive tracking only. + + Note that 'lock_held(<lock>)' introduces overhead outside of sample + overflow handling: in addition to checking the lock's owner field on + each sample where possible (such as for mutexes, rt-mutexes, and + write-locked rwsems), it attaches BPF fentry/fexit programs to kernel + lock acquire and release functions to track held locks (such as + spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a + BPF map. These BPF hooks run on every kernel lock acquisition and + release for the attached lock family, even when no perf sample is + being taken. To minimize this overhead, only the hooks for the + relevant lock family and mode are attached when known (for example, + 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, + 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, + '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + spinlock functions), whereas slab filters ('&<slab>'), raw hex + addresses, or symbols without a recognizable lock-type suffix attach + hooks across all lock families (including high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..35d6f7c63868 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> +#include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -155,9 +171,713 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) { - int i = 0; + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) == 0) + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, bool needs_lock_syms, + bool needs_slab_iter) +{ + unsigned int enabled_progs = 0; + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWSEM_WRITE; + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups)) + return; + + if (!has_amp && parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (has_amp) + *needs_slab_iter = true; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { + *prog_groups |= LOCK_PROG_RTMUTEX; + } else if (strtailcmp(base, "mutex") == 0) { + *prog_groups |= LOCK_PROG_MUTEX; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWSEM_WRITE; + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWLOCK_WRITE; + } else { + *prog_groups |= LOCK_PROG_ALL; + } + + if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0) + *needs_slab_iter = true; +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + needs_lock_syms, needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, needs_lock_syms, + needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->part` + * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate + * the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock` (with or without leading '&'). + * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `semaphore`, etc. + * 3. Hexadecimal lock address (when no leading '&'). + * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&', + * e.g. `&cgroup_mutex` or `cgroup_mutex`). + * 6. Slab cache name without leading '&' fallback. + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) +{ + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->target = 0; + if (skel) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->target = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->target = encoded_pair; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (!has_amp && parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + if (has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + if (!has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) +{ + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -171,6 +891,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + entry[i].target = 0; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -184,6 +910,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + entry[i].target = 0; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -423,10 +1155,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct bpf_program *prog; struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0; #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -438,15 +1173,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &needs_lock_syms, &needs_slab_iter); - if (needs_idx_hash && geteuid() != 0) { + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + if (has_lock_filter) { + pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -484,13 +1228,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms, + needs_slab_iter); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, false, false); + } + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -513,11 +1313,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -532,6 +1334,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -539,7 +1356,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -591,7 +1408,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -602,6 +1419,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -622,14 +1461,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -641,6 +1477,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -654,6 +1537,8 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, false, false); + /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); @@ -674,7 +1559,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -683,7 +1568,7 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); + unpin_filter_object(skel, path); ret = dir_fd; goto err; } @@ -727,6 +1612,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -740,13 +1627,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + configure_lock_progs(skel, NULL, 0, false, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -754,7 +1648,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..6ee6edef13f7 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -11,6 +11,7 @@ #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..cd711d1d04b6 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,14 +53,73 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the + * specified kernel lock is currently held by the running task/CPU, or + * 0 otherwise. The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`) and the resolved target address, + * class ID, or type flag encoding is stored in `entry.target`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.target` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.target` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.target` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; __u32 part; /* sub-sample type info when it has multiple values */ enum perf_bpf_filter_term term; __u64 value; + __u64 target; /* lock address, kmem_cache *, class, or type flags */ }; struct idx_hash_key { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..fbd2e51815d7 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,198 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr) +{ + if (!curr) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr; + __u32 pid; + __u32 cpu_id; + __u32 kind; + __u32 mode; + __u64 lock_addr; + + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->target. + */ + if (use_idx_hash || !lock_enabled) + return 0; + + curr = bpf_get_current_task(); + pid = (__u32)bpf_get_current_pid_tgid(); + cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + kind = entry->part & PBF_LOCK_KIND_MASK; + mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->target; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return 0; + if ((mode & PBF_LOCK_MODE_WRITE) && + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return 0; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id); + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id); + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + + default: + return 0; + } +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +402,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -295,4 +524,695 @@ int perf_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, flags); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) + __sync_fetch_and_add(&existing->read_count, 1); + else + __sync_fetch_and_add(&existing->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count <= 1 && existing->write_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else if (existing->read_count > 0) + __sync_fetch_and_add(&existing->read_count, -1); + } else { + if (existing->write_count == 0 && existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v1 3/5] perf test: Add tests for lock_held() BPF sample filter 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers ` (2 subsequent siblings) 5 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 ++++++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 99 ++++++++++++++++++ 2 files changed, 207 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..7ec75d969bd6 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,101 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed &${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rq_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +293,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (2 preceding siblings ...) 2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 5 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 42 +++-- tools/perf/tests/parse-events.c | 13 ++ tools/perf/tests/shell/record_bpf_filter.sh | 20 +++ tools/perf/util/bpf-filter.c | 158 +++++++++++++++++-- tools/perf/util/bpf_skel/lock_data.h | 44 +++++- tools/perf/util/bpf_skel/sample-filter.h | 12 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 402 insertions(+), 32 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 246ae085744d..c8ccfb0e0ce6 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -219,28 +219,38 @@ OPTIONS '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known - lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock - type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', - 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', - 'semaphore'). Both read/shared and write/exclusive tracking are - supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu', + 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', + 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex', + 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', + 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking + are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu', + and 'srcu' (where read/shared mode matches 'rcu_read_lock()' / + 'srcu_read_lock()' critical sections and write/exclusive mode matches + grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()', + 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace' + ('rcu_read_lock_trace()') supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'lock_held(<lock>)' introduces overhead outside of sample overflow handling: in addition to checking the lock's owner field on - each sample where possible (such as for mutexes, rt-mutexes, and - write-locked rwsems), it attaches BPF fentry/fexit programs to kernel - lock acquire and release functions to track held locks (such as - spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a - BPF map. These BPF hooks run on every kernel lock acquisition and - release for the attached lock family, even when no perf sample is - being taken. To minimize this overhead, only the hooks for the - relevant lock family and mode are attached when known (for example, - 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, - 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, - '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + each sample where possible (such as for mutexes, rt-mutexes, + write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting' + directly with no extra tracing hooks), it attaches BPF fentry/fexit + programs to kernel lock acquire and release functions to track held + locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks, + RCU synchronize waits, and slab/type filters) in a BPF map. These + BPF hooks run on every kernel lock acquisition and release for the + attached lock family, even when no perf sample is being taken. To + minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(rcu)' + and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)' + only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only + hooks rwsem write functions, '&cgroup_mutex' only hooks mutex + functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 7ec75d969bd6..39ba628022db 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -235,6 +235,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 35d6f7c63868..d4d7f7a50f18 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -190,11 +190,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -308,6 +316,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -616,6 +650,17 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RCU_WRITE; + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups)) return; @@ -646,6 +691,11 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *prog_groups |= LOCK_PROG_RWLOCK_READ; if (mode & PBF_LOCK_MODE_WRITE) *prog_groups |= LOCK_PROG_RWLOCK_WRITE; + } else if (strtailcmp(base, "srcu") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_SRCU_WRITE; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -729,16 +779,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, return false; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + s32 id; + u16 i, vlen; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + t = btf__type_by_id(btf, id); + if (!t) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && !strcmp(name, field_name)) + return true; + } + return false; +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->part` * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate * the corresponding BPF filter maps and BSS control flags. * * Resolution order: - * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock` (with or without leading '&'). + * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&'). * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, - * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`, * `semaphore`, etc. * 3. Hexadecimal lock address (when no leading '&'). * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). @@ -746,8 +834,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, * e.g. `&cgroup_mutex` or `cgroup_mutex`). * 6. Slab cache name without leading '&' fallback. */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -811,6 +899,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->target = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->target = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->target = 0; + return 0; + } + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->target = encoded_pair; @@ -875,7 +1005,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, } static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, - struct perf_bpf_filter_entry *entry) + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -893,7 +1023,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = expr->val; entry[i].target = 0; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -912,7 +1042,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = group->val; entry[i].target = 0; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1185,7 +1315,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - ret = get_filter_entries(evsel, NULL, entry); + ret = get_filter_entries(evsel, NULL, NULL, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; @@ -1247,6 +1377,12 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + find_kernel_lock_sym("rcu_state", &rcu_addr); + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms, needs_slab_iter); if (ret < 0) @@ -1264,7 +1400,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (has_lock_filter && needs_slab_iter) run_slab_cache_iter(skel); - ret = get_filter_entries(evsel, skel, entry); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index fed445c9d8b4..e32ba230d5f5 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -233,6 +236,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > 0; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -265,6 +302,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index cd711d1d04b6..37c108608bbc 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.target`, + * where `entry.target` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > 0`. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index fbd2e51815d7..22d9b01e5851 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -123,6 +123,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -136,6 +137,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; __u8 lock_type_filter[LCB_F_TYPE_MAX]; void *bpf_cast_to_kern_ctx(void *) __ksym; @@ -269,6 +271,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->target)) + return 1; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return 1; + return 0; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr)) + return 1; + return 0; + default: return 0; } @@ -600,11 +616,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, flags); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1202,6 +1219,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index a1bb78d6e29c..b45b075f3346 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (3 preceding siblings ...) 2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 5 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 167 ++++++++++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 183 ++++++++++++++---- 4 files changed, 322 insertions(+), 49 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..c6d37c910d4b 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..fa7df9187b08 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,7 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start); + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +311,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +325,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +375,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +399,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +435,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +452,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +635,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +743,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0f2118006005..257e3402e815 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -172,6 +185,7 @@ const volatile int aggr_mode; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -588,8 +602,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -597,7 +610,7 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; @@ -613,14 +626,19 @@ int contention_end(u64 *ctx) */ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) + if (pelem->lock != lock) return 0; } else { pid = bpf_get_current_pid_tgid(); pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) - return 0; - need_delete = true; + if (pelem && pelem->lock == lock) { + ts_map = &tstamp; + } else { + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp_rcu; + } } timestamp = bpf_ktime_get_ns(); @@ -648,7 +666,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -713,7 +730,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -795,11 +812,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(end_timestamp) { -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (4 preceding siblings ...) 2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers ` (6 more replies) 5 siblings, 7 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu This series adds a new BPF sample filter predicate, `lock_held(<lock>)` (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` mode suffixes), to `perf record --filter`, allowing perf event samples to be recorded only when a specified kernel lock is held (or not held with `!`) by the running kernel thread or CPU. It also extends both `lock_held()` and `perf lock contention --use-bpf` to support RCU and SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock contention`. For example: # perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... # perf lock contention -a -b -Y rcu,srcu -- sleep 1 Supported `<lock>` specifications match `perf lock contention`: - Global/static kernel lock symbols (with or without leading `&`, e.g. `lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`) - Hexadecimal kernel lock addresses - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, `lock_held_read(&dentry)`) - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, `rcu:W`, `semaphore` Limitations and weaknesses in determining whether a lock is held: - Inlined lock/unlock operations: - When the kernel is configured to inline spinlock or rwlock operations (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and have no BTF function entry points for `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that both acquire and release entry points exist in BTF before enabling a lock family. - Fast-path `percpu_down_read()` / `percpu_up_read()` and `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined per-CPU counter increments without function calls, so only `percpu_down_write()` / `percpu_up_write()` and out-of-line `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to `preempt_disable()` and does not maintain `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` read-side inspection requires `CONFIG_PREEMPT_RCU`. - Lack of an inspectable owner field on non-exclusive / spinning locks: - While `struct mutex`, `struct rt_mutex`, and write-locked `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` (allowing `perf_sample_filter` to inspect kernel memory directly on sample overflow when filtering by lock address), spinlocks, rwlocks, semaphores, SRCU, and reader-locked `rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task owner pointer. - Tracking these locks (as well as slab-cache and lock-type filters) relies on `fexit` (acquire) and `fentry` (release) BPF hooks maintaining the `held_locks` BPF hash map: - Any such lock already held before `perf record` attaches its BPF programs will not be present in `held_locks` until it is released and re-acquired. - The attached `fentry`/`fexit` hooks execute on every kernel lock acquire and release for the hooked lock family across the system, adding tracing overhead outside of perf sample handling. Patch overview: 1. "perf lock contention: Share common BPF and lock type definitions in lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`, `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`, `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`. 2. "perf bpf-filter: Add lock_held() sample filter": Implements `lock_held()`, `lock_held_read()`/`lock_held_shared()`, `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode suffixes using direct lock owner inspection where possible (`mutex`, `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map updated by selectively attached `fentry`/`fexit` hooks. 3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit tests in `tests/parse-events.c` and shell tests in `tests/shell/record_bpf_filter.sh`. 4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU": Supports `rcu` (`rcu:R` via direct `current->rcu_read_lock_nesting` inspection with zero tracing hooks; `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` via `synchronize_srcu*()` / `srcu_barrier()`). 5. "perf lock contention: Extend BPF lock contention to RCU and SRCU": Extends `perf lock contention --use-bpf` to track RCU and SRCU grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). 6. "perf lock contention: Filter events before creating lock_stat": Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to apply `-Y`, `-L`, and `-S` filters before calling `lock_stat_findnew()`. Changes in v2: - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to `sample_filter.bpf.c` and `lock_progs[]`. - Verify in `supported_lock_prog_groups()` that both acquire and all unlock BTF entry points exist for a lock family before enabling its tracing hooks, preventing acquire-only attachment when spinlock/rwlock unlock functions are inlined. - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` on `track_rwsem_read` and verify non-zero read/write counts in `record_lock_released_addr()` before decrementing aggregate slab/type counts. - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in `lock_contention.bpf.c` and recognize `synchronize_rcu*`, `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in `machine__is_lock_function()`. - Add patch 6 fixing `report_lock_contention_begin_event()` to check type, address, and callstack filters before creating `lock_stat` entries. Assisted-by: Antigravity:gemini-3.1-pro Ian Rogers (6): perf lock contention: Share common BPF and lock type definitions in lock_data.h perf bpf-filter: Add lock_held() sample filter perf test: Add tests for lock_held() BPF sample filter perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU perf lock contention: Extend BPF lock contention to RCU and SRCU perf lock contention: Filter events before creating lock_stat tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/Documentation/perf-record.txt | 51 +- tools/perf/builtin-lock.c | 88 +- tools/perf/tests/parse-events.c | 121 ++ tools/perf/tests/shell/lock_contention.sh | 14 + tools/perf/tests/shell/record_bpf_filter.sh | 123 ++ tools/perf/util/bpf-filter.c | 1188 ++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_lock_contention.c | 167 ++- .../perf/util/bpf_skel/lock_contention.bpf.c | 427 +++--- tools/perf/util/bpf_skel/lock_data.h | 368 ++++- tools/perf/util/bpf_skel/sample-filter.h | 71 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 1096 +++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + tools/perf/util/lock-contention.c | 22 + tools/perf/util/lock-contention.h | 15 + tools/perf/util/machine.c | 11 + tools/perf/util/parse-events.c | 3 +- 20 files changed, 3507 insertions(+), 362 deletions(-) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers ` (5 subsequent siblings) 6 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF sample filtering. Also add rw_semaphore.count and struct rt_mutex_base to the minimal vmlinux.h and extend get_lock_owner() to support rt_mutex and check writer-ownership and handoff flags. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +------------- tools/perf/util/bpf_skel/lock_data.h | 324 ++++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + tools/perf/util/lock-contention.c | 18 + tools/perf/util/lock-contention.h | 13 + 6 files changed, 378 insertions(+), 274 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..feaf1cfd08cf 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1477,33 +1477,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..0f2118006005 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -96,13 +82,6 @@ struct { __uint(max_entries, MAX_ENTRIES); } task_data SEC(".maps"); -struct { - __uint(type, BPF_MAP_TYPE_HASH); - __uint(key_size, sizeof(__u64)); - __uint(value_size, sizeof(__u32)); - __uint(max_entries, MAX_ENTRIES); -} lock_syms SEC(".maps"); - struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(__u32)); @@ -138,6 +117,13 @@ struct { __uint(max_entries, 1); } cgroup_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, MAX_ENTRIES); +} lock_syms SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(long)); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -202,6 +170,11 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + int enabled; int perf_subsys_id = -1; @@ -222,8 +195,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,145 +800,24 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - -const volatile __u64 contig_page_data_addr; -const volatile __u64 node_data_addr; -const volatile int nr_nodes; -const volatile int sizeof_zone; - -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") -int BPF_PROG(collect_lock_syms) +int BPF_PROG(end_timestamp) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - + end_ts = bpf_ktime_get_ns(); return 0; } SEC("raw_tp/bpf_test_finish") -int BPF_PROG(end_timestamp) +int BPF_PROG(collect_lock_syms) { - end_ts = bpf_ktime_get_ns(); - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..bf39128e0be1 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,320 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t count; + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex *mutex = (void *)lock; + + if (bpf_core_field_exists(mutex->owner.counter)) { + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (bpf_core_type_exists(struct rt_mutex_base)) { + struct rt_mutex_base *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + struct rw_semaphore___new *rwsem = (void *)lock; + + if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) { + __u64 count = BPF_CORE_READ(rwsem, count.counter); + + /* Must have RWSEM_WRITER_LOCKED (bit 0) set */ + if (!(count & 1UL)) + return NULL; + } + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *old = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(old, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + owner = BPF_CORE_READ(rwsem, owner.counter); + } +#else + /* assume new struct */ + if (bpf_core_field_exists(rwsem->owner.counter)) + owner = BPF_CORE_READ(rwsem, owner.counter); +#endif + /* RWSEM_READER_OWNED is bit 0 of owner */ + if (flags == LCB_F_WRITE && (owner & 1UL)) + return NULL; + } else if (flags == LCB_F_RT) { + struct rt_mutex_base *rtm = (void *)lock; + + if (bpf_core_type_exists(struct rt_mutex_base)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..a1bb78d6e29c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -22,6 +22,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers ` (4 subsequent siblings) 6 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported <lock> specifications match perf lock contention: - Global/static kernel lock symbols (with or without '&', e.g. lock_held(&cgroup_mutex) or lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. To minimize tracing overhead, only the BPF hooks for the requested lock family and mode are autoloaded and attached. Example usage: # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 41 +- tools/perf/util/bpf-filter.c | 1036 +++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_skel/sample-filter.h | 59 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 965 ++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2168 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..246ae085744d 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,46 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)'). + <lock> can be a kernel lock symbol name (with or without a leading + '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), + a hexadecimal lock address, a slab cache name prefixed with '&' + (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'semaphore'). Both read/shared and write/exclusive tracking are + supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), + 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') + support write/exclusive tracking only. + + Note that 'lock_held(<lock>)' introduces overhead outside of sample + overflow handling: in addition to checking the lock's owner field on + each sample where possible (such as for mutexes, rt-mutexes, and + write-locked rwsems), it attaches BPF fentry/fexit programs to kernel + lock acquire and release functions to track held locks (such as + spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a + BPF map. These BPF hooks run on every kernel lock acquisition and + release for the attached lock family, even when no perf sample is + being taken. To minimize this overhead, only the hooks for the + relevant lock family and mode are attached when known (for example, + 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, + 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, + '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + spinlock functions), whereas slab filters ('&<slab>'), raw hex + addresses, or symbols without a recognizable lock-type suffix attach + hooks across all lock families (including high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..2f4c6bd85f89 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> +#include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -155,9 +171,814 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) +{ + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) == 0) + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +static bool btf_has_func(struct btf *btf, const char *name) +{ + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; +} + +static unsigned int supported_lock_prog_groups(struct btf *btf) +{ + unsigned int supported = 0; + + if (!btf) + return 0; + + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) && + btf_has_func(btf, "mutex_unlock")) + supported |= LOCK_PROG_MUTEX; + + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) && + btf_has_func(btf, "up_read")) + supported |= LOCK_PROG_RWSEM_READ; + + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) && + btf_has_func(btf, "up_write")) + supported |= LOCK_PROG_RWSEM_WRITE; + + /* + * Require both acquire and all unlock entry points in BTF so we never + * attach spinlock/rwlock acquire hooks when unlock functions are + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK). + */ + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) && + btf_has_func(btf, "_raw_spin_unlock") && + btf_has_func(btf, "_raw_spin_unlock_irq") && + btf_has_func(btf, "_raw_spin_unlock_irqrestore") && + btf_has_func(btf, "_raw_spin_unlock_bh")) + supported |= LOCK_PROG_SPIN; + + if (btf_has_func(btf, "_raw_read_lock") && + btf_has_func(btf, "_raw_read_unlock") && + btf_has_func(btf, "_raw_read_unlock_irq") && + btf_has_func(btf, "_raw_read_unlock_irqrestore") && + btf_has_func(btf, "_raw_read_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_READ; + + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) && + btf_has_func(btf, "_raw_write_unlock") && + btf_has_func(btf, "_raw_write_unlock_irq") && + btf_has_func(btf, "_raw_write_unlock_irqrestore") && + btf_has_func(btf, "_raw_write_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_WRITE; + + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) && + btf_has_func(btf, "rt_mutex_unlock")) + supported |= LOCK_PROG_RTMUTEX; + + if (btf_has_func(btf, "down") && btf_has_func(btf, "up")) + supported |= LOCK_PROG_SEM; + + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) + supported |= LOCK_PROG_PERCPU_RWSEM; + + return supported; +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, + unsigned int required_groups, + bool needs_lock_syms, bool needs_slab_iter) +{ + unsigned int supported_groups = supported_lock_prog_groups(btf); + unsigned int enabled_progs = 0; + + if ((required_groups & supported_groups) != required_groups) { + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n"); + return -ENOTSUP; + } + + needed_groups &= supported_groups; + if (skel) + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ); + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode, groups = 0; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWSEM_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) { + *prog_groups |= groups; + *required_groups |= groups; + return; + } + + if (!has_amp && parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (has_amp) + *needs_slab_iter = true; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { + *prog_groups |= LOCK_PROG_RTMUTEX; + *required_groups |= LOCK_PROG_RTMUTEX; + } else if (strtailcmp(base, "mutex") == 0) { + *prog_groups |= LOCK_PROG_MUTEX; + *required_groups |= LOCK_PROG_MUTEX; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_RWSEM_WRITE; + } + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWLOCK_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + } else { + *prog_groups |= LOCK_PROG_ALL; + } + + if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0) + *needs_slab_iter = true; +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *required_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + required_groups, needs_lock_syms, + needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, required_groups, + needs_lock_syms, needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->part` + * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate + * the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock` (with or without leading '&'). + * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `semaphore`, etc. + * 3. Hexadecimal lock address (when no leading '&'). + * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&', + * e.g. `&cgroup_mutex` or `cgroup_mutex`). + * 6. Slab cache name without leading '&' fallback. + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) +{ + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->target = 0; + if (skel) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->target = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->target = encoded_pair; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (!has_amp && parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + if (has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + if (!has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) { - int i = 0; + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -171,6 +992,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + entry[i].target = 0; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -184,6 +1011,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + entry[i].target = 0; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -423,10 +1256,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct bpf_program *prog; struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0, required_groups = 0; #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -438,15 +1274,25 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &required_groups, &needs_lock_syms, + &needs_slab_iter); - if (needs_idx_hash && geteuid() != 0) { + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + if (has_lock_filter) { + pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -484,13 +1330,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, required_groups, + needs_lock_syms, needs_slab_iter); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, 0, false, false); + } + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -513,11 +1415,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -532,6 +1436,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -539,7 +1458,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -591,7 +1510,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -602,6 +1521,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -622,14 +1563,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -641,6 +1579,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -654,6 +1639,8 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false); + /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); @@ -674,7 +1661,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -683,7 +1670,7 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); + unpin_filter_object(skel, path); ret = dir_fd; goto err; } @@ -727,6 +1714,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -740,13 +1729,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -754,7 +1750,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..6ee6edef13f7 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -11,6 +11,7 @@ #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..cd711d1d04b6 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,14 +53,73 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the + * specified kernel lock is currently held by the running task/CPU, or + * 0 otherwise. The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`) and the resolved target address, + * class ID, or type flag encoding is stored in `entry.target`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.target` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.target` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.target` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; __u32 part; /* sub-sample type info when it has multiple values */ enum perf_bpf_filter_term term; __u64 value; + __u64 target; /* lock address, kmem_cache *, class, or type flags */ }; struct idx_hash_key { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..480ba4b49793 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,199 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +int track_rwsem_read; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr) +{ + if (!curr) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr; + __u32 pid; + __u32 cpu_id; + __u32 kind; + __u32 mode; + __u64 lock_addr; + + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->target. + */ + if (use_idx_hash || !lock_enabled) + return 0; + + curr = bpf_get_current_task(); + pid = (__u32)bpf_get_current_pid_tgid(); + cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + kind = entry->part & PBF_LOCK_KIND_MASK; + mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->target; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return 0; + if ((mode & PBF_LOCK_MODE_WRITE) && + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return 0; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id); + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id); + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + + default: + return 0; + } +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +403,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -295,4 +525,739 @@ int perf_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, flags); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) + __sync_fetch_and_add(&existing->read_count, 1); + else + __sync_fetch_and_add(&existing->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count == 0) + return 0; + if (existing->read_count == 1 && existing->write_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + __sync_fetch_and_add(&existing->read_count, -1); + } else { + if (existing->write_count == 0) + return 0; + if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/_mutex_lock_nest_lock") +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_lock_killable") +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass, + void *nest_lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_trylock_nest_lock") +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/_down_write_nest_lock") +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + if (track_rwsem_read) + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nest_lock") +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/_rt_mutex_lock_nest_lock") +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v2 3/6] perf test: Add tests for lock_held() BPF sample filter 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers ` (3 subsequent siblings) 6 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 ++++++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 103 +++++++++++++++++++ 2 files changed, 211 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..adaa5363d0b7 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,105 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed &${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required" + then + echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]" + err=1 + return + fi + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +297,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (2 preceding siblings ...) 2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers ` (2 subsequent siblings) 6 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> # Conflicts: # tools/perf/util/bpf-filter.c # tools/perf/util/bpf_skel/sample_filter.bpf.c --- tools/perf/Documentation/perf-record.txt | 42 +++-- tools/perf/tests/parse-events.c | 13 ++ tools/perf/tests/shell/record_bpf_filter.sh | 20 +++ tools/perf/util/bpf-filter.c | 174 +++++++++++++++++-- tools/perf/util/bpf_skel/lock_data.h | 44 ++++- tools/perf/util/bpf_skel/sample-filter.h | 12 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 418 insertions(+), 32 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 246ae085744d..c8ccfb0e0ce6 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -219,28 +219,38 @@ OPTIONS '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known - lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock - type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', - 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', - 'semaphore'). Both read/shared and write/exclusive tracking are - supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu', + 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', + 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex', + 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', + 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking + are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu', + and 'srcu' (where read/shared mode matches 'rcu_read_lock()' / + 'srcu_read_lock()' critical sections and write/exclusive mode matches + grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()', + 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace' + ('rcu_read_lock_trace()') supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'lock_held(<lock>)' introduces overhead outside of sample overflow handling: in addition to checking the lock's owner field on - each sample where possible (such as for mutexes, rt-mutexes, and - write-locked rwsems), it attaches BPF fentry/fexit programs to kernel - lock acquire and release functions to track held locks (such as - spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a - BPF map. These BPF hooks run on every kernel lock acquisition and - release for the attached lock family, even when no perf sample is - being taken. To minimize this overhead, only the hooks for the - relevant lock family and mode are attached when known (for example, - 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, - 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, - '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + each sample where possible (such as for mutexes, rt-mutexes, + write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting' + directly with no extra tracing hooks), it attaches BPF fentry/fexit + programs to kernel lock acquire and release functions to track held + locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks, + RCU synchronize waits, and slab/type filters) in a BPF map. These + BPF hooks run on every kernel lock acquisition and release for the + attached lock family, even when no perf sample is being taken. To + minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(rcu)' + and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)' + only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only + hooks rwsem write functions, '&cgroup_mutex' only hooks mutex + functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index adaa5363d0b7..dc1c1a02eaad 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -239,6 +239,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 2f4c6bd85f89..6261a79b1455 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -190,11 +190,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -314,6 +322,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -597,6 +631,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) supported |= LOCK_PROG_PERCPU_RWSEM; + if ((btf_has_func(btf, "__srcu_read_lock") || + btf_has_func(btf, "__srcu_read_lock_nmisafe")) && + (btf_has_func(btf, "__srcu_read_unlock") || + btf_has_func(btf, "__srcu_read_unlock_nmisafe"))) + supported |= LOCK_PROG_SRCU_READ; + + if (btf_has_func(btf, "synchronize_srcu")) + supported |= LOCK_PROG_SRCU_WRITE; + + if (btf_has_func(btf, "synchronize_rcu")) + supported |= LOCK_PROG_RCU_WRITE; + return supported; } @@ -701,6 +747,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RCU_WRITE; + *required_groups |= LOCK_PROG_RCU_WRITE; + } + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) { *prog_groups |= groups; *required_groups |= groups; @@ -744,6 +803,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, groups |= LOCK_PROG_RWLOCK_WRITE; *prog_groups |= groups; *required_groups |= groups; + } else if (strtailcmp(base, "srcu") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_SRCU_WRITE; + *prog_groups |= groups; + *required_groups |= groups; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -830,16 +896,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, return false; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + s32 id; + u16 i, vlen; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + t = btf__type_by_id(btf, id); + if (!t) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && !strcmp(name, field_name)) + return true; + } + return false; +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->part` * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate * the corresponding BPF filter maps and BSS control flags. * * Resolution order: - * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock` (with or without leading '&'). + * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&'). * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, - * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`, * `semaphore`, etc. * 3. Hexadecimal lock address (when no leading '&'). * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). @@ -847,8 +951,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, * e.g. `&cgroup_mutex` or `cgroup_mutex`). * 6. Slab cache name without leading '&' fallback. */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -912,6 +1016,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->target = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->target = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->target = 0; + return 0; + } + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->target = encoded_pair; @@ -976,7 +1122,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, } static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, - struct perf_bpf_filter_entry *entry) + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -994,7 +1140,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = expr->val; entry[i].target = 0; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -1013,7 +1159,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = group->val; entry[i].target = 0; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1287,7 +1433,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - ret = get_filter_entries(evsel, NULL, entry); + ret = get_filter_entries(evsel, NULL, NULL, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; @@ -1349,6 +1495,12 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + find_kernel_lock_sym("rcu_state", &rcu_addr); + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, needs_lock_syms, needs_slab_iter); if (ret < 0) @@ -1366,7 +1518,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (has_lock_filter && needs_slab_iter) run_slab_cache_iter(skel); - ret = get_filter_entries(evsel, skel, entry); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index bf39128e0be1..85570798baae 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -241,6 +244,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > 0; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -273,6 +310,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index cd711d1d04b6..37c108608bbc 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.target`, + * where `entry.target` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > 0`. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index 480ba4b49793..6f3adbbe0289 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -123,6 +123,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -136,6 +137,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; int track_rwsem_read; __u8 lock_type_filter[LCB_F_TYPE_MAX]; @@ -270,6 +272,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->target)) + return 1; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return 1; + return 0; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr)) + return 1; + return 0; + default: return 0; } @@ -601,11 +617,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, flags); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1247,6 +1264,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index a1bb78d6e29c..b45b075f3346 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (3 preceding siblings ...) 2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 6 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 167 +++++++++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 200 ++++++++++++++---- tools/perf/util/machine.c | 11 + 5 files changed, 348 insertions(+), 51 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..c6d37c910d4b 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..fa7df9187b08 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,7 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start); + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +311,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +325,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +375,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +399,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +435,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +452,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +635,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +743,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0f2118006005..58d45c43150b 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -172,6 +185,7 @@ const volatile int aggr_mode; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -559,8 +573,21 @@ int contention_begin(u64 *ctx) } } skip_owner: - pelem->stack_id = bpf_get_stackid(ctx, &stacks, - BPF_F_FAST_STACK_CMP | stack_skip); + { + int skip = stack_skip; + + /* + * Unlike tp_btf/contention_begin (which fires inside a + * lock slowpath via trace_contention_begin), fentry + * RCU/SRCU programs attach directly to the top-level + * wait API, which is 2 frames shallower. + */ + if (flags & (LCB_F_RCU | LCB_F_SRCU)) + skip = stack_skip > 2 ? stack_skip - 2 : 0; + + pelem->stack_id = bpf_get_stackid(ctx, &stacks, + BPF_F_FAST_STACK_CMP | skip); + } if (pelem->stack_id < 0) __sync_fetch_and_add(&stack_fail, 1); } else if (aggr_mode == LOCK_AGGR_TASK) { @@ -588,8 +615,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -597,7 +623,7 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; @@ -613,14 +639,19 @@ int contention_end(u64 *ctx) */ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) + if (pelem->lock != lock) return 0; } else { pid = bpf_get_current_pid_tgid(); pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) - return 0; - need_delete = true; + if (pelem && pelem->lock == lock) { + ts_map = &tstamp; + } else { + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp_rcu; + } } timestamp = bpf_ktime_get_ns(); @@ -648,7 +679,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -713,7 +743,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -795,11 +825,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(end_timestamp) { diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index 1d9d3bf57720..0a0f223b3335 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr) return true; } + { + struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL); + + if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) || + !strncmp(sym->name, "synchronize_srcu", 16) || + !strcmp(sym->name, "__synchronize_srcu") || + !strcmp(sym->name, "rcu_barrier") || + !strcmp(sym->name, "srcu_barrier"))) + return true; + } + return false; } -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (4 preceding siblings ...) 2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 6 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu In report_lock_contention_begin_event(), lock_stat_findnew() was called before checking filters.nr_types (-Y), filters.nr_addrs (-L), and match_callstack_filter() (-S). When a filtered-out lock event (such as a mutex contention when running with -Y spinlock) shared the same aggregation key as a subsequent matching event (for example, contention on the mutex's internal wait_lock spinlock from the same caller), the first event created the lock_stat entry with the filtered-out lock's flags and the second event reused that entry, causing non-spinlock entries to appear in -Y spinlock output. Check the type, address, and callstack filters before looking up or creating the lock_stat entry. Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") Assisted-by: Antigravity:gemini-3.1-pro Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index feaf1cfd08cf..a031709a2cc2 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) struct machine *machine = &session->machines.host; struct map *kmap; struct symbol *sym; + u64 *callstack = NULL; ret = get_key_by_aggr_mode(&key, addr, sample); if (ret < 0) @@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } } - ls = lock_stat_find(key); - if (!ls) { - char buf[128]; - const char *name = ""; - - switch (aggr_mode) { - case LOCK_AGGR_ADDR: - sym = machine__find_kernel_symbol(machine, key, &kmap); - if (sym) - name = sym->name; - break; - case LOCK_AGGR_CALLER: - name = buf; - if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) - name = "Unknown"; - break; - case LOCK_AGGR_CGROUP: - case LOCK_AGGR_TASK: - default: - break; - } - - ls = lock_stat_findnew(key, name, flags); - if (!ls) - return -ENOMEM; - } - if (filters.nr_types) { bool found = false; @@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } if (needs_callstack()) { - u64 *callstack = get_callstack(sample, max_stack_depth); + callstack = get_callstack(sample, max_stack_depth); if (callstack == NULL) return 0; @@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) free(callstack); return 0; } + } + + ls = lock_stat_find(key); + if (!ls) { + char buf[128]; + const char *name = ""; + + switch (aggr_mode) { + case LOCK_AGGR_ADDR: + sym = machine__find_kernel_symbol(machine, key, &kmap); + if (sym) + name = sym->name; + break; + case LOCK_AGGR_CALLER: + name = buf; + if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) + name = "Unknown"; + break; + case LOCK_AGGR_CGROUP: + case LOCK_AGGR_TASK: + default: + break; + } + + ls = lock_stat_findnew(key, name, flags); + if (!ls) { + free(callstack); + return -ENOMEM; + } + } + if (callstack) { if (ls->callstack == NULL) ls->callstack = callstack; else -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (5 preceding siblings ...) 2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers @ 2026-10-18 5:37 ` Ian Rogers 2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers ` (7 more replies) 6 siblings, 8 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu This series adds a new BPF sample filter predicate, `lock_held(<lock>)` (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` mode suffixes), to `perf record --filter`, allowing perf event samples to be recorded only when a specified kernel lock is held (or not held with `!`) by the running kernel thread or CPU. It also extends both `lock_held()` and `perf lock contention --use-bpf` to support RCU and SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock contention`. For example: # perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... # perf lock contention -a -b -Y rcu,srcu -- sleep 1 An important use-case is detecting memory allocations or direct memory reclaim occurring while kernel locks are held, which can stall lock waiters for extended periods under memory pressure: # Direct memory reclaim triggered while holding any mutex or rwsem: # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM) # while holding a mutex or rwsem: # perf record -a -g -e kmem:mm_page_alloc \ --filter 'gfp_flags & 0x400' \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Major page faults occurring while holding a mutex or rwsem: # perf record -a -g -e major-faults \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... Supported `<lock>` specifications match `perf lock contention`: - Global/static kernel lock symbols (with or without leading `&`, e.g. `lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`) - Hexadecimal kernel lock addresses - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, `lock_held_read(&dentry)`) - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, `rcu:W`, `semaphore` Limitations and weaknesses in determining whether a lock is held: - Inlined lock/unlock operations: - When the kernel is configured to inline spinlock or rwlock operations (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and have no BTF function entry points for `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that both acquire and release entry points exist in BTF before enabling a lock family. - Fast-path `percpu_down_read()` / `percpu_up_read()` and `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined per-CPU counter increments without function calls, so only `percpu_down_write()` / `percpu_up_write()` and out-of-line `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to `preempt_disable()` and does not maintain `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` read-side inspection requires `CONFIG_PREEMPT_RCU`. - Lack of an inspectable owner field on non-exclusive / spinning locks: - While `struct mutex`, `struct rt_mutex`, and write-locked `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` (allowing `perf_sample_filter` to inspect kernel memory directly on sample overflow when filtering by lock address), spinlocks, rwlocks, semaphores, SRCU, and reader-locked `rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task owner pointer. - Tracking these locks (as well as slab-cache and lock-type filters) relies on `fexit` (acquire) and `fentry` (release) BPF hooks maintaining the `held_locks` BPF hash map: - Any such lock already held before `perf record` attaches its BPF programs will not be present in `held_locks` until it is released and re-acquired. - The attached `fentry`/`fexit` hooks execute on every kernel lock acquire and release for the hooked lock family across the system, adding tracing overhead outside of perf sample handling. Patch overview: 1. "perf lock contention: Share common BPF and lock type definitions in lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`, `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`, `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`. 2. "perf bpf-filter: Add lock_held() sample filter": Implements `lock_held()`, `lock_held_read()`/`lock_held_shared()`, `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode suffixes using direct lock owner inspection where possible (`mutex`, `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map updated by selectively attached `fentry`/`fexit` hooks, supporting both PMU/software events (`SEC("perf_event")`) and tracepoint events (`SEC("tracepoint")`). 3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit tests in `tests/parse-events.c` and shell tests in `tests/shell/record_bpf_filter.sh`. 4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU": Supports `rcu` (`rcu:R` via direct `current->rcu_read_lock_nesting` inspection with zero tracing hooks; `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` via `synchronize_srcu*()` / `srcu_barrier()`). 5. "perf lock contention: Extend BPF lock contention to RCU and SRCU": Extends `perf lock contention --use-bpf` to track RCU and SRCU grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). 6. "perf lock contention: Filter events before creating lock_stat": Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to apply `-Y`, `-L`, and `-S` filters before calling `lock_stat_findnew()`. Changes in v3: - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c` so `perf lock contention -b -o` still reports reader owners when a writer contends on a reader-owned `rw_semaphore`, and scope `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`, `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`). - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()` when attaching, and document memory allocation / direct reclaim filtering with `lock_held()`. - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data` cannot be resolved and check `node_data_addr != 0` in `do_collect_zone_lock()`. - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call `_raw_read_lock_irqsave()`. - Offset `rcu_state_addr` by 1 byte so it is distinct from `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match timestamp maps. Changes in v2: - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to `sample_filter.bpf.c` and `lock_progs[]`. - Verify in `supported_lock_prog_groups()` that both acquire and all unlock BTF entry points exist for a lock family before enabling its tracing hooks, preventing acquire-only attachment when spinlock/rwlock unlock functions are inlined. - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` on `track_rwsem_read` and verify non-zero read/write counts in `record_lock_released_addr()` before decrementing aggregate slab/type counts. - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in `lock_contention.bpf.c` and recognize `synchronize_rcu*`, `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in `machine__is_lock_function()`. - Add patch 6 fixing `report_lock_contention_begin_event()` to check type, address, and callstack filters before creating `lock_stat` entries. Assisted-by: Antigravity:gemini-3.1-pro Ian Rogers (6): perf lock contention: Share common BPF and lock type definitions in lock_data.h perf bpf-filter: Add lock_held() sample filter perf test: Add tests for lock_held() BPF sample filter perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU perf lock contention: Extend BPF lock contention to RCU and SRCU perf lock contention: Filter events before creating lock_stat tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/Documentation/perf-record.txt | 64 +- tools/perf/builtin-lock.c | 88 +- tools/perf/tests/parse-events.c | 121 ++ tools/perf/tests/shell/lock_contention.sh | 14 + tools/perf/tests/shell/record_bpf_filter.sh | 134 ++ tools/perf/util/bpf-filter.c | 1271 ++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_lock_contention.c | 171 ++- .../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++--- tools/perf/util/bpf_skel/lock_data.h | 360 ++++- tools/perf/util/bpf_skel/sample-filter.h | 77 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 1225 ++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + tools/perf/util/lock-contention.c | 22 + tools/perf/util/lock-contention.h | 15 + tools/perf/util/machine.c | 11 + tools/perf/util/parse-events.c | 3 +- 20 files changed, 3741 insertions(+), 387 deletions(-) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers @ 2026-10-18 5:37 ` Ian Rogers 2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers ` (6 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF sample filtering. Also add rw_semaphore.count and struct rt_mutex_base to the minimal vmlinux.h and extend get_lock_owner() to support rt_mutex and check the mutex handoff flag. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +-------------- tools/perf/util/bpf_skel/lock_data.h | 316 ++++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + tools/perf/util/lock-contention.c | 18 + tools/perf/util/lock-contention.h | 13 + 6 files changed, 370 insertions(+), 274 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..feaf1cfd08cf 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1477,33 +1477,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..0f2118006005 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -96,13 +82,6 @@ struct { __uint(max_entries, MAX_ENTRIES); } task_data SEC(".maps"); -struct { - __uint(type, BPF_MAP_TYPE_HASH); - __uint(key_size, sizeof(__u64)); - __uint(value_size, sizeof(__u32)); - __uint(max_entries, MAX_ENTRIES); -} lock_syms SEC(".maps"); - struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(__u32)); @@ -138,6 +117,13 @@ struct { __uint(max_entries, 1); } cgroup_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, MAX_ENTRIES); +} lock_syms SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(long)); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -202,6 +170,11 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + int enabled; int perf_subsys_id = -1; @@ -222,8 +195,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,145 +800,24 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - -const volatile __u64 contig_page_data_addr; -const volatile __u64 node_data_addr; -const volatile int nr_nodes; -const volatile int sizeof_zone; - -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") -int BPF_PROG(collect_lock_syms) +int BPF_PROG(end_timestamp) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - + end_ts = bpf_ktime_get_ns(); return 0; } SEC("raw_tp/bpf_test_finish") -int BPF_PROG(end_timestamp) +int BPF_PROG(collect_lock_syms) { - end_ts = bpf_ktime_get_ns(); - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..8cf24f07204c 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,312 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t count; + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex *mutex = (void *)lock; + + if (bpf_core_field_exists(mutex->owner.counter)) { + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (bpf_core_type_exists(struct rt_mutex_base)) { + struct rt_mutex_base *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *old = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(old, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + struct rw_semaphore___new *new = (void *)lock; + + owner = BPF_CORE_READ(new, owner.counter); + } +#else + /* assume new struct */ + struct rw_semaphore___new *new = (void *)lock; + + if (bpf_core_field_exists(new->owner.counter)) + owner = BPF_CORE_READ(new, owner.counter); +#endif + } else if (flags == LCB_F_RT) { + struct rt_mutex_base *rtm = (void *)lock; + + if (bpf_core_type_exists(struct rt_mutex_base)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (node_data_addr && nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..a1bb78d6e29c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -22,6 +22,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-18 5:37 ` Ian Rogers 2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers ` (5 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported <lock> specifications match perf lock contention: - Global/static kernel lock symbols (with or without '&', e.g. lock_held(&cgroup_mutex) or lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. To minimize tracing overhead, only the BPF hooks for the requested lock family and mode are autoloaded and attached. Example usage: # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 54 +- tools/perf/util/bpf-filter.c | 1114 +++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_skel/sample-filter.h | 65 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 1094 +++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2380 insertions(+), 40 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..39dfe63fe3a8 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,59 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)'). + <lock> can be a kernel lock symbol name (with or without a leading + '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), + a hexadecimal lock address, a slab cache name prefixed with '&' + (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'semaphore'). Both read/shared and write/exclusive tracking are + supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), + 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') + support write/exclusive tracking only. + + For example, 'lock_held()' can be combined with memory allocation or + reclaim events to detect allocations or direct memory reclaim occurring + while a kernel lock is held (which can stall lock waiters for extended + periods): + + # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e kmem:mm_page_alloc \ + --filter 'gfp_flags & 0x400' \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e major-faults \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + + Note that 'lock_held(<lock>)' introduces overhead outside of sample + overflow handling: in addition to checking the lock's owner field on + each sample where possible (such as for mutexes, rt-mutexes, and + write-locked rwsems), it attaches BPF fentry/fexit programs to kernel + lock acquire and release functions to track held locks (such as + spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a + BPF map. These BPF hooks run on every kernel lock acquisition and + release for the attached lock family, even when no perf sample is + being taken. To minimize this overhead, only the hooks for the + relevant lock family and mode are attached when known (for example, + 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, + 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, + '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + spinlock functions), whereas slab filters ('&<slab>'), raw hex + addresses, or symbols without a recognizable lock-type suffix attach + hooks across all lock families (including high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..0c133b31b5d5 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> +#include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -123,15 +138,27 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * { const struct perf_sample_info *info; - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) - return 0; - - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU || + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP) + return 0; + if (expr->op != PBF_OP_GROUP_BEGIN) { + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n", + evsel__name(evsel), expr->term); + return -1; + } + } + + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) + return 0; + if (expr->op == PBF_OP_GROUP_BEGIN) { struct perf_bpf_filter_expr *group; @@ -155,9 +182,831 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) +{ + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) != 0) + return; + + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +static bool btf_has_func(struct btf *btf, const char *name) +{ + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; +} + +static unsigned int supported_lock_prog_groups(struct btf *btf) +{ + unsigned int supported = 0; + + if (!btf) + return 0; + + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) && + btf_has_func(btf, "mutex_unlock")) + supported |= LOCK_PROG_MUTEX; + + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) && + btf_has_func(btf, "up_read")) + supported |= LOCK_PROG_RWSEM_READ; + + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) && + btf_has_func(btf, "up_write")) + supported |= LOCK_PROG_RWSEM_WRITE; + + /* + * Require both acquire and all unlock entry points in BTF so we never + * attach spinlock/rwlock acquire hooks when unlock functions are + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK). + */ + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) && + btf_has_func(btf, "_raw_spin_unlock") && + btf_has_func(btf, "_raw_spin_unlock_irq") && + btf_has_func(btf, "_raw_spin_unlock_irqrestore") && + btf_has_func(btf, "_raw_spin_unlock_bh")) + supported |= LOCK_PROG_SPIN; + + if (btf_has_func(btf, "_raw_read_lock") && + btf_has_func(btf, "_raw_read_unlock") && + btf_has_func(btf, "_raw_read_unlock_irq") && + btf_has_func(btf, "_raw_read_unlock_irqrestore") && + btf_has_func(btf, "_raw_read_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_READ; + + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) && + btf_has_func(btf, "_raw_write_unlock") && + btf_has_func(btf, "_raw_write_unlock_irq") && + btf_has_func(btf, "_raw_write_unlock_irqrestore") && + btf_has_func(btf, "_raw_write_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_WRITE; + + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) && + btf_has_func(btf, "rt_mutex_unlock")) + supported |= LOCK_PROG_RTMUTEX; + + if (btf_has_func(btf, "down") && btf_has_func(btf, "up")) + supported |= LOCK_PROG_SEM; + + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) + supported |= LOCK_PROG_PERCPU_RWSEM; + + return supported; +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, + unsigned int required_groups, + bool needs_lock_syms, bool needs_slab_iter, + bool is_tracepoint) +{ + unsigned int supported_groups = supported_lock_prog_groups(btf); + unsigned int enabled_progs = 0; + + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + + if ((required_groups & supported_groups) != required_groups) { + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n"); + return -ENOTSUP; + } + + needed_groups &= supported_groups; + if (skel) + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ); + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode, groups = 0; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWSEM_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) { + *prog_groups |= groups; + *required_groups |= groups; + return; + } + + if (!has_amp && parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (has_amp) + *needs_slab_iter = true; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { + *prog_groups |= LOCK_PROG_RTMUTEX; + *required_groups |= LOCK_PROG_RTMUTEX; + } else if (strtailcmp(base, "mutex") == 0) { + *prog_groups |= LOCK_PROG_MUTEX; + *required_groups |= LOCK_PROG_MUTEX; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_RWSEM_WRITE; + } + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWLOCK_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + } else { + *prog_groups |= LOCK_PROG_ALL; + } + + if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0) + *needs_slab_iter = true; +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *required_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + required_groups, needs_lock_syms, + needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, required_groups, + needs_lock_syms, needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->part` + * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate + * the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock` (with or without leading '&'). + * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `semaphore`, etc. + * 3. Hexadecimal lock address (when no leading '&'). + * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&', + * e.g. `&cgroup_mutex` or `cgroup_mutex`). + * 6. Slab cache name without leading '&' fallback. + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) { - int i = 0; + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->target = 0; + if (skel) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->target = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->target = encoded_pair; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (!has_amp && parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + if (has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + unsigned int direct_flag = 0; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) + direct_flag = PBF_LOCK_DIRECT_RTMUTEX; + else if (strtailcmp(base, "mutex") == 0) + direct_flag = PBF_LOCK_DIRECT_MUTEX; + else if (strtailcmp(base, "rwsem") == 0 && + strtailcmp(base, "percpu_rwsem") != 0 && + strtailcmp(base, "pcpu_rwsem") != 0) + direct_flag = PBF_LOCK_DIRECT_RWSEM; + + entry->part = PBF_LOCK_ADDR | mode | direct_flag; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + if (!has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) +{ + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -171,6 +1020,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + entry[i].target = 0; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -184,6 +1039,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + entry[i].target = 0; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -423,10 +1284,14 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct bpf_program *prog; struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0, required_groups = 0; #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -438,15 +1303,25 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &required_groups, &needs_lock_syms, + &needs_slab_iter); - if (needs_idx_hash && geteuid() != 0) { + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + if (has_lock_filter || is_tracepoint) { + pr_err("Error: BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -484,13 +1359,71 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, required_groups, + needs_lock_syms, needs_slab_iter, + is_tracepoint); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint); + } + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -501,23 +1434,44 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - prog = skel->progs.perf_sample_filter; - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), - &pe_opts); - if (IS_ERR(link)) { - pr_err("Failed to attach perf sample-filter program\n"); - ret = PTR_ERR(link); - goto err; + if (is_tracepoint) { + /* + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the + * BPF program to event->tp_event->prog_array on the shared + * trace_event_call, which perf_trace_run_bpf_submit() executes + * for all CPUs and tasks before iterating the per-CPU perf + * events. Attach once on the first FD so tp_event->prog_array + * does not get duplicate copies for each per-CPU FD. + */ + prog = skel->progs.perf_tp_sample_filter; + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0), + &pe_opts); + ret = libbpf_get_error(link); + if (ret) { + pr_err("Failed to attach perf tracepoint sample-filter program\n"); + goto err; + } + } else { + prog = skel->progs.perf_sample_filter; + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), + &pe_opts); + ret = libbpf_get_error(link); + if (ret) { + pr_err("Failed to attach perf sample-filter program\n"); + goto err; + } } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -532,6 +1486,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -539,7 +1508,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -591,7 +1560,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -602,6 +1571,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -622,14 +1613,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -641,6 +1629,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -654,6 +1689,8 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false, false); + /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); @@ -674,7 +1711,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -683,8 +1720,8 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); - ret = dir_fd; + ret = -errno; + unpin_filter_object(skel, path); goto err; } @@ -727,6 +1764,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -740,13 +1779,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -754,7 +1800,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..6ee6edef13f7 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -11,6 +11,7 @@ #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..215998897243 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,14 +53,79 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the + * specified kernel lock is currently held by the running task/CPU, or + * 0 otherwise. The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`) and the resolved target address, + * class ID, or type flag encoding is stored in `entry.target`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.target` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.target` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.target` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner) + * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner) + * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base owner) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY +#define PBF_LOCK_DIRECT_MUTEX (1U << 6) +#define PBF_LOCK_DIRECT_RWSEM (1U << 7) +#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8) + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; __u32 part; /* sub-sample type info when it has multiple values */ enum perf_bpf_filter_term term; __u64 value; + __u64 target; /* lock address, kmem_cache *, class, or type flags */ }; struct idx_hash_key { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..583a5b98b929 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,220 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +int track_rwsem_read; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr) +{ + struct rw_semaphore___new *rwsem = (void *)lock_addr; + + if (bpf_core_field_exists(rwsem->count.counter) && + !(BPF_CORE_READ(rwsem, count.counter) & 1UL)) + return false; + if (bpf_core_field_exists(rwsem->owner.counter) && + (BPF_CORE_READ(rwsem, owner.counter) & 1UL)) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr; +} + +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part) +{ + if (!curr) + return false; + + if ((part & PBF_LOCK_DIRECT_MUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr) + return true; + if ((part & PBF_LOCK_DIRECT_RWSEM) && + is_rwsem_writer_owner(lock_addr, curr)) + return true; + if ((part & PBF_LOCK_DIRECT_RTMUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr) + return true; + return false; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr; + __u32 pid; + __u32 cpu_id; + __u32 kind; + __u32 mode; + __u64 lock_addr; + + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->target. + */ + if (use_idx_hash || !lock_enabled) + return 0; + + curr = bpf_get_current_task(); + pid = (__u32)bpf_get_current_pid_tgid(); + cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + kind = entry->part & PBF_LOCK_KIND_MASK; + mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->target; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr, entry->part)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return 0; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_rwsem_writer_owner(lock_addr, curr)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return 0; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id); + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id); + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + + default: + return 0; + } +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +424,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -192,6 +443,30 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return 0; } +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry) +{ + switch (entry->term) { + case PBF_TERM_TID: + if (entry->part) + return bpf_get_current_pid_tgid() >> 32; + return (__u32)bpf_get_current_pid_tgid(); + case PBF_TERM_CPU: + return bpf_get_smp_processor_id(); + case PBF_TERM_TIME: + return bpf_ktime_get_ns(); + case PBF_TERM_CGROUP: + return bpf_get_current_cgroup_id(); + case PBF_TERM_UID: + return bpf_get_current_uid_gid() & 0xFFFFFFFF; + case PBF_TERM_GID: + return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); + default: + return 0; + } +} + #define CHECK_RESULT(data, op, val) \ if (!(data op val)) { \ if (!in_group) \ @@ -295,4 +570,823 @@ int perf_sample_filter(void *ctx) return 0; } +/* BPF program to be called for tracepoint perf events */ +SEC("tracepoint") +int perf_tp_sample_filter(void *ctx) +{ + struct perf_bpf_filter_entry *entry; + __u64 sample_data; + int in_group = 0; + int group_result = 0; + int i, k = 0; + int *losts; + + entry = bpf_map_lookup_elem(&filters, &k); + if (entry == NULL) + goto drop; + + for (i = 0; i < MAX_FILTERS; i++) { + sample_data = perf_get_common_sample(&entry[i]); + + switch (entry[i].op) { + case PBF_OP_EQ: + CHECK_RESULT(sample_data, ==, entry[i].value) + break; + case PBF_OP_NEQ: + CHECK_RESULT(sample_data, !=, entry[i].value) + break; + case PBF_OP_GT: + CHECK_RESULT(sample_data, >, entry[i].value) + break; + case PBF_OP_GE: + CHECK_RESULT(sample_data, >=, entry[i].value) + break; + case PBF_OP_LT: + CHECK_RESULT(sample_data, <, entry[i].value) + break; + case PBF_OP_LE: + CHECK_RESULT(sample_data, <=, entry[i].value) + break; + case PBF_OP_AND: + CHECK_RESULT(sample_data, &, entry[i].value) + break; + case PBF_OP_GROUP_BEGIN: + in_group = 1; + group_result = 0; + break; + case PBF_OP_GROUP_END: + if (group_result == 0) + goto drop; + in_group = 0; + break; + case PBF_OP_DONE: + return 1; + } + } + return 1; + +drop: + losts = bpf_map_lookup_elem(&dropped, &k); + if (losts != NULL) + __sync_fetch_and_add(losts, 1); + + return 0; +} + +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +#define LCB_F_READ_IRQSAVE (1U << 30) +#define LCB_F_READ_IRQ_BH (1U << 31) + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, type_flag); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && type_flag == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* + * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC + * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call + * the out-of-line _raw_read_lock_irqsave(). Clear the marker + * and skip the duplicate outer fexit increment. + */ + if ((flags & LCB_F_READ_IRQ_BH) && + (existing->flags & LCB_F_READ_IRQSAVE)) { + existing->flags &= ~LCB_F_READ_IRQSAVE; + return 0; + } + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) { + __sync_fetch_and_add(&existing->read_count, 1); + if (flags & LCB_F_READ_IRQSAVE) + existing->flags |= LCB_F_READ_IRQSAVE; + else + existing->flags &= ~LCB_F_READ_IRQSAVE; + } else { + __sync_fetch_and_add(&existing->write_count, 1); + } + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE), + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, type_flag, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count == 0) + return 0; + if (existing->read_count == 1 && existing->write_count == 0) { + bpf_map_delete_elem(&held_locks, &addr_key); + } else { + existing->flags &= ~LCB_F_READ_IRQSAVE; + __sync_fetch_and_add(&existing->read_count, -1); + } + } else { + if (existing->write_count == 0) + return 0; + if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/_mutex_lock_nest_lock") +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_lock_killable") +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass, + void *nest_lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_trylock_nest_lock") +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/_down_write_nest_lock") +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + if (track_rwsem_read) + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nest_lock") +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/_rt_mutex_lock_nest_lock") +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v3 3/6] perf test: Add tests for lock_held() BPF sample filter 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-18 5:37 ` Ian Rogers 2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers ` (4 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 +++++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 114 ++++++++++++++++++++ 2 files changed, 222 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..9334b8b9b1c2 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,116 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed &${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required" + then + echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]" + err=1 + return + fi + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]" + err=1 + return + fi + + if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc + then + if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]" + err=1 + return + fi + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +308,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (2 preceding siblings ...) 2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-18 5:37 ` Ian Rogers 2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers ` (3 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> # Conflicts: # tools/perf/util/bpf-filter.c # tools/perf/util/bpf_skel/sample_filter.bpf.c # Conflicts: # tools/perf/util/bpf_skel/sample_filter.bpf.c --- tools/perf/Documentation/perf-record.txt | 42 +++-- tools/perf/tests/parse-events.c | 13 ++ tools/perf/tests/shell/record_bpf_filter.sh | 20 +++ tools/perf/util/bpf-filter.c | 179 +++++++++++++++++-- tools/perf/util/bpf_skel/lock_data.h | 44 ++++- tools/perf/util/bpf_skel/sample-filter.h | 12 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 423 insertions(+), 32 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 39dfe63fe3a8..f45d2eec901c 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -219,11 +219,17 @@ OPTIONS '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known - lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock - type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', - 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', - 'semaphore'). Both read/shared and write/exclusive tracking are - supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu', + 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', + 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex', + 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', + 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking + are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu', + and 'srcu' (where read/shared mode matches 'rcu_read_lock()' / + 'srcu_read_lock()' critical sections and write/exclusive mode matches + grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()', + 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace' + ('rcu_read_lock_trace()') supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. @@ -243,17 +249,21 @@ OPTIONS Note that 'lock_held(<lock>)' introduces overhead outside of sample overflow handling: in addition to checking the lock's owner field on - each sample where possible (such as for mutexes, rt-mutexes, and - write-locked rwsems), it attaches BPF fentry/fexit programs to kernel - lock acquire and release functions to track held locks (such as - spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a - BPF map. These BPF hooks run on every kernel lock acquisition and - release for the attached lock family, even when no perf sample is - being taken. To minimize this overhead, only the hooks for the - relevant lock family and mode are attached when known (for example, - 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, - 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, - '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + each sample where possible (such as for mutexes, rt-mutexes, + write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting' + directly with no extra tracing hooks), it attaches BPF fentry/fexit + programs to kernel lock acquire and release functions to track held + locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks, + RCU synchronize waits, and slab/type filters) in a BPF map. These + BPF hooks run on every kernel lock acquisition and release for the + attached lock family, even when no perf sample is being taken. To + minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(rcu)' + and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)' + only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only + hooks rwsem write functions, '&cgroup_mutex' only hooks mutex + functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 9334b8b9b1c2..6b58f1556c15 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -239,6 +239,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 0c133b31b5d5..4a5ef78e2690 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -201,11 +201,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -325,6 +333,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -610,6 +644,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) supported |= LOCK_PROG_PERCPU_RWSEM; + if ((btf_has_func(btf, "__srcu_read_lock") || + btf_has_func(btf, "__srcu_read_lock_nmisafe")) && + (btf_has_func(btf, "__srcu_read_unlock") || + btf_has_func(btf, "__srcu_read_unlock_nmisafe"))) + supported |= LOCK_PROG_SRCU_READ; + + if (btf_has_func(btf, "synchronize_srcu")) + supported |= LOCK_PROG_SRCU_WRITE; + + if (btf_has_func(btf, "synchronize_rcu")) + supported |= LOCK_PROG_RCU_WRITE; + return supported; } @@ -718,6 +764,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RCU_WRITE; + *required_groups |= LOCK_PROG_RCU_WRITE; + } + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) { *prog_groups |= groups; *required_groups |= groups; @@ -761,6 +820,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, groups |= LOCK_PROG_RWLOCK_WRITE; *prog_groups |= groups; *required_groups |= groups; + } else if (strtailcmp(base, "srcu") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_SRCU_WRITE; + *prog_groups |= groups; + *required_groups |= groups; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -847,16 +913,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, return false; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + s32 id; + u16 i, vlen; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + t = btf__type_by_id(btf, id); + if (!t) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && !strcmp(name, field_name)) + return true; + } + return false; +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->part` * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate * the corresponding BPF filter maps and BSS control flags. * * Resolution order: - * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock` (with or without leading '&'). + * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&'). * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, - * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`, * `semaphore`, etc. * 3. Hexadecimal lock address (when no leading '&'). * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). @@ -864,8 +968,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, * e.g. `&cgroup_mutex` or `cgroup_mutex`). * 6. Slab cache name without leading '&' fallback. */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -929,6 +1033,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->target = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->target = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->target = 0; + return 0; + } + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->target = encoded_pair; @@ -1004,7 +1150,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, } static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, - struct perf_bpf_filter_entry *entry) + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -1022,7 +1168,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = expr->val; entry[i].target = 0; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -1041,7 +1187,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = group->val; entry[i].target = 0; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1316,7 +1462,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - ret = get_filter_entries(evsel, NULL, entry); + ret = get_filter_entries(evsel, NULL, NULL, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; @@ -1378,6 +1524,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0) + rcu_addr += 1; + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, needs_lock_syms, needs_slab_iter, is_tracepoint); @@ -1396,7 +1553,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (has_lock_filter && needs_slab_iter) run_slab_cache_iter(skel); - ret = get_filter_entries(evsel, skel, entry); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 8cf24f07204c..d11ad1746d09 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -233,6 +236,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > 0; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -265,6 +302,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 215998897243..f6a030abfb0d 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.target`, + * where `entry.target` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > 0`. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index 583a5b98b929..776425c27a4a 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -123,6 +123,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -136,6 +137,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; int track_rwsem_read; __u8 lock_type_filter[LCB_F_TYPE_MAX]; @@ -291,6 +293,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->target)) + return 1; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return 1; + return 0; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr)) + return 1; + return 0; + default: return 0; } @@ -712,11 +728,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, type_flag); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1376,6 +1393,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index a1bb78d6e29c..b45b075f3346 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (3 preceding siblings ...) 2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-18 5:37 ` Ian Rogers 2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers ` (2 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 171 ++++++++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 221 ++++++++++++++---- tools/perf/util/machine.c | 11 + 5 files changed, 362 insertions(+), 62 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..c6d37c910d4b 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..dfb384f1c5e6 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,7 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +181,133 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1; + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +315,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +329,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +379,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +403,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +439,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +456,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +639,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +747,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0f2118006005..26936f329dba 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -172,6 +185,7 @@ const volatile int aggr_mode; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -559,8 +573,21 @@ int contention_begin(u64 *ctx) } } skip_owner: - pelem->stack_id = bpf_get_stackid(ctx, &stacks, - BPF_F_FAST_STACK_CMP | stack_skip); + { + int skip = stack_skip; + + /* + * Unlike tp_btf/contention_begin (which fires inside a + * lock slowpath via trace_contention_begin), fentry + * RCU/SRCU programs attach directly to the top-level + * wait API, which is 2 frames shallower. + */ + if (flags & (LCB_F_RCU | LCB_F_SRCU)) + skip = stack_skip > 2 ? stack_skip - 2 : 0; + + pelem->stack_id = bpf_get_stackid(ctx, &stacks, + BPF_F_FAST_STACK_CMP | skip); + } if (pelem->stack_id < 0) __sync_fetch_and_add(&stack_fail, 1); } else if (aggr_mode == LOCK_AGGR_TASK) { @@ -588,8 +615,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -597,30 +623,34 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; /* - * For spinlock and rwlock, it needs to get the timestamp for the - * per-cpu map. However, contention_end does not have the flags - * so it cannot know whether it reads percpu or hash map. - * - * Try per-cpu map first and check if there's active contention. - * If it is, do not read hash map because it cannot go to sleeping - * locks before releasing the spinning locks. + * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu + * map. For tp_btf/contention_end, try the per-cpu map first (used by + * spinlocks and rwlocks) and fall back to the per-task tstamp map. */ - pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); - if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) - return 0; - } else { + if (is_rcu) { pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) return 0; - need_delete = true; + ts_map = &tstamp_rcu; + } else { + pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); + if (pelem && pelem->lock) { + if (pelem->lock != lock) + return 0; + } else { + pid = bpf_get_current_pid_tgid(); + pelem = bpf_map_lookup_elem(&tstamp, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp; + } } timestamp = bpf_ktime_get_ns(); @@ -648,7 +678,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -713,7 +742,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -795,11 +824,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(end_timestamp) { diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index 1d9d3bf57720..0a0f223b3335 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr) return true; } + { + struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL); + + if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) || + !strncmp(sym->name, "synchronize_srcu", 16) || + !strcmp(sym->name, "__synchronize_srcu") || + !strcmp(sym->name, "rcu_barrier") || + !strcmp(sym->name, "srcu_barrier"))) + return true; + } + return false; } -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (4 preceding siblings ...) 2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-18 5:37 ` Ian Rogers 2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu In report_lock_contention_begin_event(), lock_stat_findnew() was called before checking filters.nr_types (-Y), filters.nr_addrs (-L), and match_callstack_filter() (-S). When a filtered-out lock event (such as a mutex contention when running with -Y spinlock) shared the same aggregation key as a subsequent matching event (for example, contention on the mutex's internal wait_lock spinlock from the same caller), the first event created the lock_stat entry with the filtered-out lock's flags and the second event reused that entry, causing non-spinlock entries to appear in -Y spinlock output. Check the type, address, and callstack filters before looking up or creating the lock_stat entry. Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index feaf1cfd08cf..a031709a2cc2 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) struct machine *machine = &session->machines.host; struct map *kmap; struct symbol *sym; + u64 *callstack = NULL; ret = get_key_by_aggr_mode(&key, addr, sample); if (ret < 0) @@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } } - ls = lock_stat_find(key); - if (!ls) { - char buf[128]; - const char *name = ""; - - switch (aggr_mode) { - case LOCK_AGGR_ADDR: - sym = machine__find_kernel_symbol(machine, key, &kmap); - if (sym) - name = sym->name; - break; - case LOCK_AGGR_CALLER: - name = buf; - if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) - name = "Unknown"; - break; - case LOCK_AGGR_CGROUP: - case LOCK_AGGR_TASK: - default: - break; - } - - ls = lock_stat_findnew(key, name, flags); - if (!ls) - return -ENOMEM; - } - if (filters.nr_types) { bool found = false; @@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } if (needs_callstack()) { - u64 *callstack = get_callstack(sample, max_stack_depth); + callstack = get_callstack(sample, max_stack_depth); if (callstack == NULL) return 0; @@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) free(callstack); return 0; } + } + + ls = lock_stat_find(key); + if (!ls) { + char buf[128]; + const char *name = ""; + + switch (aggr_mode) { + case LOCK_AGGR_ADDR: + sym = machine__find_kernel_symbol(machine, key, &kmap); + if (sym) + name = sym->name; + break; + case LOCK_AGGR_CALLER: + name = buf; + if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) + name = "Unknown"; + break; + case LOCK_AGGR_CGROUP: + case LOCK_AGGR_TASK: + default: + break; + } + + ls = lock_stat_findnew(key, name, flags); + if (!ls) { + free(callstack); + return -ENOMEM; + } + } + if (callstack) { if (ls->callstack == NULL) ls->callstack = callstack; else -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (5 preceding siblings ...) 2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers @ 2026-10-19 7:00 ` Namhyung Kim 2026-10-19 19:08 ` Ian Rogers 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers 7 siblings, 1 reply; 59+ messages in thread From: Namhyung Kim @ 2026-10-19 7:00 UTC (permalink / raw) To: Ian Rogers Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Hi Ian, On Sat, Oct 17, 2026 at 10:37:04PM -0700, Ian Rogers wrote: > This series adds a new BPF sample filter predicate, `lock_held(<lock>)` > (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, > `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` > mode suffixes), to `perf record --filter`, allowing perf event samples to > be recorded only when a specified kernel lock is held (or not held with > `!`) by the running kernel thread or CPU. It also extends both > `lock_held()` and `perf lock contention --use-bpf` to support RCU and > SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock > contention`. Awesome! Looks really useful! > > For example: > > # perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ... > # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ... > # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ... > # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... > # perf lock contention -a -b -Y rcu,srcu -- sleep 1 > > An important use-case is detecting memory allocations or direct memory > reclaim occurring while kernel locks are held, which can stall lock > waiters for extended periods under memory pressure: > > # Direct memory reclaim triggered while holding any mutex or rwsem: > # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... > > # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM) > # while holding a mutex or rwsem: > # perf record -a -g -e kmem:mm_page_alloc \ > --filter 'gfp_flags & 0x400' \ > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... It seems the first filter is processed by the kernel tracepoint filtering logic and the second is passed to the BPF filter, right? Does the ordering matter? > > # Major page faults occurring while holding a mutex or rwsem: > # perf record -a -g -e major-faults \ > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... > > Supported `<lock>` specifications match `perf lock contention`: > - Global/static kernel lock symbols (with or without leading `&`, e.g. > `lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`) It used to have no '&' for global locks. While having it is more natural, I think it's better to leave it for slab locks so that we can distinguish them easily. > - Hexadecimal kernel lock addresses > - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, > `lock_held_read(&dentry)`) > - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, > `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` > - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, > `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, > `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, > `rcu:W`, `semaphore` > > Limitations and weaknesses in determining whether a lock is held: > - Inlined lock/unlock operations: > - When the kernel is configured to inline spinlock or rwlock operations > (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, > `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, > `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / > `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are > inlined into callers and have no BTF function entry points for > `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that > both acquire and release entry points exist in BTF before enabling a > lock family. > - Fast-path `percpu_down_read()` / `percpu_up_read()` and > `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined > per-CPU counter increments without function calls, so only > `percpu_down_write()` / `percpu_up_write()` and out-of-line > `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. > - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to > `preempt_disable()` and does not maintain > `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` > read-side inspection requires `CONFIG_PREEMPT_RCU`. Looks ok. What happens if user specified unsupported locks - e.g. rcu:R on !CONFIG_PREEMPT_RCU. > - Lack of an inspectable owner field on non-exclusive / spinning locks: > - While `struct mutex`, `struct rt_mutex`, and write-locked > `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` > (allowing `perf_sample_filter` to inspect kernel memory directly on > sample overflow when filtering by lock address), spinlocks, rwlocks, > semaphores, SRCU, and reader-locked `rw_semaphore`s (where > `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) > do not maintain an authoritative per-task owner pointer. So do you use the owner field for the supported locks? Then maybe it can skip tracing mutex_lock() and mutex_unlock(). Does it already do? > - Tracking these locks (as well as slab-cache and lock-type filters) > relies on `fexit` (acquire) and `fentry` (release) BPF hooks > maintaining the `held_locks` BPF hash map: > - Any such lock already held before `perf record` attaches its BPF > programs will not be present in `held_locks` until it is released > and re-acquired. Right, but this can be mitigated when it runs for long. > - The attached `fentry`/`fexit` hooks execute on every kernel lock > acquire and release for the hooked lock family across the system, > adding tracing overhead outside of perf sample handling. This is a real problem. As it touches the fastpath it can cause a lot of overhead if locking is frequent. But having this capability would be very useful anyway! Thanks, Namhyung > > Patch overview: > 1. "perf lock contention: Share common BPF and lock type definitions in > lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and > `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers > (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`, > `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`, > `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`. > 2. "perf bpf-filter: Add lock_held() sample filter": Implements > `lock_held()`, `lock_held_read()`/`lock_held_shared()`, > `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode > suffixes using direct lock owner inspection where possible (`mutex`, > `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map > updated by selectively attached `fentry`/`fexit` hooks, supporting > both PMU/software events (`SEC("perf_event")`) and tracepoint events > (`SEC("tracepoint")`). > 3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit > tests in `tests/parse-events.c` and shell tests in > `tests/shell/record_bpf_filter.sh`. > 4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, > and SRCU": Supports `rcu` (`rcu:R` via direct > `current->rcu_read_lock_nesting` inspection with zero tracing hooks; > `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), > `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` > (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` > via `synchronize_srcu*()` / `srcu_barrier()`). > 5. "perf lock contention: Extend BPF lock contention to RCU and SRCU": > Extends `perf lock contention --use-bpf` to track RCU and SRCU > grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). > 6. "perf lock contention: Filter events before creating lock_stat": > Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to > apply `-Y`, `-L`, and `-S` filters before calling > `lock_stat_findnew()`. > > Changes in v3: > - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in > `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c` > so `perf lock contention -b -o` still reports reader owners when a > writer contends on a reader-owned `rw_semaphore`, and scope > `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`, > `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`). > - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF > sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to > `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and > `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()` > when attaching, and document memory allocation / direct reclaim > filtering with `lock_held()`. > - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data` > cannot be resolved and check `node_data_addr != 0` in > `do_collect_zone_lock()`. > - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in > `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y && > !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call > `_raw_read_lock_irqsave()`. > - Offset `rcu_state_addr` by 1 byte so it is distinct from > `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool > is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so > `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match > timestamp maps. > > Changes in v2: > - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` > and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE > relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. > - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points > (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, > `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, > `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to > `sample_filter.bpf.c` and `lock_progs[]`. > - Verify in `supported_lock_prog_groups()` that both acquire and all > unlock BTF entry points exist for a lock family before enabling its > tracing hooks, preventing acquire-only attachment when spinlock/rwlock > unlock functions are inlined. > - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when > a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). > - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` > on `track_rwsem_read` and verify non-zero read/write counts in > `record_lock_released_addr()` before decrementing aggregate slab/type > counts. > - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in > `lock_contention.bpf.c` and recognize `synchronize_rcu*`, > `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in > `machine__is_lock_function()`. > - Add patch 6 fixing `report_lock_contention_begin_event()` to check > type, address, and callstack filters before creating `lock_stat` > entries. > > Assisted-by: Antigravity:gemini-3.1-pro > > Ian Rogers (6): > perf lock contention: Share common BPF and lock type definitions in > lock_data.h > perf bpf-filter: Add lock_held() sample filter > perf test: Add tests for lock_held() BPF sample filter > perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, > and SRCU > perf lock contention: Extend BPF lock contention to RCU and SRCU > perf lock contention: Filter events before creating lock_stat > > tools/perf/Documentation/perf-lock.txt | 7 +- > tools/perf/Documentation/perf-record.txt | 64 +- > tools/perf/builtin-lock.c | 88 +- > tools/perf/tests/parse-events.c | 121 ++ > tools/perf/tests/shell/lock_contention.sh | 14 + > tools/perf/tests/shell/record_bpf_filter.sh | 134 ++ > tools/perf/util/bpf-filter.c | 1271 ++++++++++++++++- > tools/perf/util/bpf-filter.h | 13 + > tools/perf/util/bpf-filter.l | 31 +- > tools/perf/util/bpf-filter.y | 46 +- > tools/perf/util/bpf_lock_contention.c | 171 ++- > .../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++--- > tools/perf/util/bpf_skel/lock_data.h | 360 ++++- > tools/perf/util/bpf_skel/sample-filter.h | 77 + > tools/perf/util/bpf_skel/sample_filter.bpf.c | 1225 ++++++++++++++++ > tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + > tools/perf/util/lock-contention.c | 22 + > tools/perf/util/lock-contention.h | 15 + > tools/perf/util/machine.c | 11 + > tools/perf/util/parse-events.c | 3 +- > 20 files changed, 3741 insertions(+), 387 deletions(-) > > -- > 2.56.0.212.g129ede20d9-goog > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim @ 2026-10-19 19:08 ` Ian Rogers 0 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-19 19:08 UTC (permalink / raw) To: Namhyung Kim Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu On Mon, Oct 19, 2026 at 12:00 AM Namhyung Kim <namhyung@kernel.org> wrote: > > Hi Ian, > > On Sat, Oct 17, 2026 at 10:37:04PM -0700, Ian Rogers wrote: > > This series adds a new BPF sample filter predicate, `lock_held(<lock>)` > > (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, > > `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` > > mode suffixes), to `perf record --filter`, allowing perf event samples to > > be recorded only when a specified kernel lock is held (or not held with > > `!`) by the running kernel thread or CPU. It also extends both > > `lock_held()` and `perf lock contention --use-bpf` to support RCU and > > SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock > > contention`. > > Awesome! Looks really useful! > > > > > For example: > > > > # perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ... > > # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ... > > # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ... > > # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... > > # perf lock contention -a -b -Y rcu,srcu -- sleep 1 > > > > An important use-case is detecting memory allocations or direct memory > > reclaim occurring while kernel locks are held, which can stall lock > > waiters for extended periods under memory pressure: > > > > # Direct memory reclaim triggered while holding any mutex or rwsem: > > # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ > > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... > > > > # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM) > > # while holding a mutex or rwsem: > > # perf record -a -g -e kmem:mm_page_alloc \ > > --filter 'gfp_flags & 0x400' \ > > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... > > It seems the first filter is processed by the kernel tracepoint > filtering logic and the second is passed to the BPF filter, right? > Does the ordering matter? Right — set_filter() in util/parse-events.c inspects each --filter string independently with is_possible_tp_filter(): expressions without BPF-only terms (uid, gid, lock_held) are appended to evsel->filter (the kernel tracepoint filter), while expressions with lock_held() are added to evsel->bpf_filters. The command-line order of the two --filter options does not matter: in evlist__apply_filters() both evsel->filter (PERF_EVENT_IOC_SET_FILTER) and evsel->bpf_filters (PERF_EVENT_IOC_SET_BPF) are applied to the tracepoint event, and in the kernel (perf_trace_run_bpf_submit() -> perf_tp_event()) both must match (logical AND) for the sample to be recorded. > > > > # Major page faults occurring while holding a mutex or rwsem: > > # perf record -a -g -e major-faults \ > > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... > > > > Supported `<lock>` specifications match `perf lock contention`: > > - Global/static kernel lock symbols (with or without leading `&`, e.g. > > `lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`) > > It used to have no '&' for global locks. While having it is more > natural, I think it's better to leave it for slab locks so that we can > distinguish them easily. Good point — in v4 I updated this so '&' is reserved strictly for slab cache locks (e.g. lock_held(&dentry)), and global/static lock symbols and well-known locks do not use '&' (e.g. lock_held(cgroup_mutex), lock_held_read(mmap_lock)). This also avoids running the kmem_cache BPF iterator unless a '&<slab>' filter is specified. > > - Hexadecimal kernel lock addresses > > - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, > > `lock_held_read(&dentry)`) > > - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, > > `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` > > - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, > > `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, > > `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, > > `rcu:W`, `semaphore` > > > > Limitations and weaknesses in determining whether a lock is held: > > - Inlined lock/unlock operations: > > - When the kernel is configured to inline spinlock or rwlock operations > > (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, > > `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, > > `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / > > `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are > > inlined into callers and have no BTF function entry points for > > `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that > > both acquire and release entry points exist in BTF before enabling a > > lock family. > > - Fast-path `percpu_down_read()` / `percpu_up_read()` and > > `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined > > per-CPU counter increments without function calls, so only > > `percpu_down_write()` / `percpu_up_write()` and out-of-line > > `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. > > - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to > > `preempt_disable()` and does not maintain > > `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` > > read-side inspection requires `CONFIG_PREEMPT_RCU`. > > Looks ok. What happens if user specified unsupported locks - e.g. rcu:R > on !CONFIG_PREEMPT_RCU. resolve_lock_expr() checks vmlinux BTF for task_struct's rcu_read_lock_nesting field (and trc_reader_nesting for rcu-trace) and configure_lock_progs() checks that both the acquire and unlock BTF entry points exist for any required lock family. If the kernel lacks support, perf record prints an error message (e.g. "Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU" or "Error: Kernel BTF lacks required lock/unlock tracing entry points") and fails immediately with -EOPNOTSUPP / -ENOTSUP. > > - Lack of an inspectable owner field on non-exclusive / spinning locks: > > - While `struct mutex`, `struct rt_mutex`, and write-locked > > `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` > > (allowing `perf_sample_filter` to inspect kernel memory directly on > > sample overflow when filtering by lock address), spinlocks, rwlocks, > > semaphores, SRCU, and reader-locked `rw_semaphore`s (where > > `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) > > do not maintain an authoritative per-task owner pointer. > > So do you use the owner field for the supported locks? Then maybe it > can skip tracing mutex_lock() and mutex_unlock(). Does it already do? Great catch — in v3 it was checking lock->owner in perf_sample_filter, but prescan_one_lock_expr() was still enabling the fentry/fexit hooks for *_mutex, *_rt_mutex, and mmap_lock:W. In v4 I will update prescan_one_lock_expr() to skip attaching fentry/fexit hooks altogether for specific *_mutex symbols, *_rt_mutex symbols, and write-locked mmap_lock (mmap_lock:W / lock_held_write(mmap_lock)), so those filters only inspect lock->owner on sample overflow with zero lock fastpath tracing overhead. > > - Tracking these locks (as well as slab-cache and lock-type filters) > > relies on `fexit` (acquire) and `fentry` (release) BPF hooks > > maintaining the `held_locks` BPF hash map: > > - Any such lock already held before `perf record` attaches its BPF > > programs will not be present in `held_locks` until it is released > > and re-acquired. > > Right, but this can be mitigated when it runs for long. > > > > - The attached `fentry`/`fexit` hooks execute on every kernel lock > > acquire and release for the hooked lock family across the system, > > adding tracing overhead outside of perf sample handling. > > This is a real problem. As it touches the fastpath it can cause a lot > of overhead if locking is frequent. But having this capability would be > very useful anyway! Agreed. We may also be able to optimize or compromise between lock performance and observability once this gets adopted. Thanks, Ian > Thanks, > Namhyung > > > > > Patch overview: > > 1. "perf lock contention: Share common BPF and lock type definitions in > > lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and > > `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers > > (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`, > > `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`, > > `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`. > > 2. "perf bpf-filter: Add lock_held() sample filter": Implements > > `lock_held()`, `lock_held_read()`/`lock_held_shared()`, > > `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode > > suffixes using direct lock owner inspection where possible (`mutex`, > > `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map > > updated by selectively attached `fentry`/`fexit` hooks, supporting > > both PMU/software events (`SEC("perf_event")`) and tracepoint events > > (`SEC("tracepoint")`). > > 3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit > > tests in `tests/parse-events.c` and shell tests in > > `tests/shell/record_bpf_filter.sh`. > > 4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, > > and SRCU": Supports `rcu` (`rcu:R` via direct > > `current->rcu_read_lock_nesting` inspection with zero tracing hooks; > > `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), > > `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` > > (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` > > via `synchronize_srcu*()` / `srcu_barrier()`). > > 5. "perf lock contention: Extend BPF lock contention to RCU and SRCU": > > Extends `perf lock contention --use-bpf` to track RCU and SRCU > > grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). > > 6. "perf lock contention: Filter events before creating lock_stat": > > Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to > > apply `-Y`, `-L`, and `-S` filters before calling > > `lock_stat_findnew()`. > > > > Changes in v3: > > - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in > > `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c` > > so `perf lock contention -b -o` still reports reader owners when a > > writer contends on a reader-owned `rw_semaphore`, and scope > > `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`, > > `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`). > > - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF > > sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to > > `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and > > `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()` > > when attaching, and document memory allocation / direct reclaim > > filtering with `lock_held()`. > > - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data` > > cannot be resolved and check `node_data_addr != 0` in > > `do_collect_zone_lock()`. > > - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in > > `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y && > > !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call > > `_raw_read_lock_irqsave()`. > > - Offset `rcu_state_addr` by 1 byte so it is distinct from > > `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool > > is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so > > `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match > > timestamp maps. > > > > Changes in v2: > > - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` > > and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE > > relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. > > - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points > > (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, > > `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, > > `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to > > `sample_filter.bpf.c` and `lock_progs[]`. > > - Verify in `supported_lock_prog_groups()` that both acquire and all > > unlock BTF entry points exist for a lock family before enabling its > > tracing hooks, preventing acquire-only attachment when spinlock/rwlock > > unlock functions are inlined. > > - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when > > a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). > > - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` > > on `track_rwsem_read` and verify non-zero read/write counts in > > `record_lock_released_addr()` before decrementing aggregate slab/type > > counts. > > - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in > > `lock_contention.bpf.c` and recognize `synchronize_rcu*`, > > `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in > > `machine__is_lock_function()`. > > - Add patch 6 fixing `report_lock_contention_begin_event()` to check > > type, address, and callstack filters before creating `lock_stat` > > entries. > > > > Assisted-by: Antigravity:gemini-3.1-pro > > > > Ian Rogers (6): > > perf lock contention: Share common BPF and lock type definitions in > > lock_data.h > > perf bpf-filter: Add lock_held() sample filter > > perf test: Add tests for lock_held() BPF sample filter > > perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, > > and SRCU > > perf lock contention: Extend BPF lock contention to RCU and SRCU > > perf lock contention: Filter events before creating lock_stat > > > > tools/perf/Documentation/perf-lock.txt | 7 +- > > tools/perf/Documentation/perf-record.txt | 64 +- > > tools/perf/builtin-lock.c | 88 +- > > tools/perf/tests/parse-events.c | 121 ++ > > tools/perf/tests/shell/lock_contention.sh | 14 + > > tools/perf/tests/shell/record_bpf_filter.sh | 134 ++ > > tools/perf/util/bpf-filter.c | 1271 ++++++++++++++++- > > tools/perf/util/bpf-filter.h | 13 + > > tools/perf/util/bpf-filter.l | 31 +- > > tools/perf/util/bpf-filter.y | 46 +- > > tools/perf/util/bpf_lock_contention.c | 171 ++- > > .../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++--- > > tools/perf/util/bpf_skel/lock_data.h | 360 ++++- > > tools/perf/util/bpf_skel/sample-filter.h | 77 + > > tools/perf/util/bpf_skel/sample_filter.bpf.c | 1225 ++++++++++++++++ > > tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + > > tools/perf/util/lock-contention.c | 22 + > > tools/perf/util/lock-contention.h | 15 + > > tools/perf/util/machine.c | 11 + > > tools/perf/util/parse-events.c | 3 +- > > 20 files changed, 3741 insertions(+), 387 deletions(-) > > > > -- > > 2.56.0.212.g129ede20d9-goog > > ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 0/7] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (6 preceding siblings ...) 2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim @ 2026-10-19 22:09 ` Ian Rogers 2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers ` (7 more replies) 7 siblings, 8 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu This series adds a new BPF sample filter predicate, `lock_held(<lock>)` (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` mode suffixes), to `perf record --filter`, allowing perf event samples to be recorded only when a specified kernel lock is held (or not held with `!`) by the running kernel thread or CPU. It also fixes a pre-existing `lock_stat` filtering bug in `perf lock contention`, enables BPF sample filters on tracepoint events (`PERF_TYPE_TRACEPOINT`), and extends both `lock_held()` and `perf lock contention --use-bpf` to support RCU and SRCU. For example: # perf record -a -g -e cycles --filter 'lock_held(cgroup_mutex)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' -- ... # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... # perf lock contention -a -b -Y rcu,srcu -- sleep 1 An important use-case is detecting memory allocations or direct memory reclaim occurring while kernel locks are held, which can stall lock waiters for extended periods under memory pressure: # Direct memory reclaim triggered while holding any mutex or rwsem: # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM) # while holding a mutex or rwsem: # perf record -a -g -e kmem:mm_page_alloc \ --filter 'gfp_flags & 0x400' \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Major page faults occurring while holding a mutex or rwsem: # perf record -a -g -e major-faults \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... Supported `<lock>` specifications match `perf lock contention`: - Global/static kernel lock symbols (e.g. `lock_held(cgroup_mutex)`) - Hexadecimal kernel lock addresses - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, `lock_held_read(&dentry)`) - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, `rcu:W`, `semaphore` Limitations and weaknesses in determining whether a lock is held: - Inlined lock/unlock operations: - When the kernel is configured to inline spinlock or rwlock operations (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and have no BTF function entry points for `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that both acquire and release entry points exist in BTF before enabling a lock family. - Fast-path `percpu_down_read()` / `percpu_up_read()` and `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined per-CPU counter increments without function calls, so only `percpu_down_write()` / `percpu_up_write()` and out-of-line `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to `preempt_disable()` and does not maintain `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` read-side inspection requires `CONFIG_PREEMPT_RCU`. - Lack of an inspectable owner field on non-exclusive / spinning locks: - While `struct mutex`, `struct rt_mutex`, and write-locked `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` (allowing `perf_sample_filter` to inspect kernel memory directly on sample overflow when filtering by lock address without attaching any `fentry`/`fexit` hooks for `*_mutex`, `*_rt_mutex`, and `mmap_lock:W`), spinlocks, rwlocks, semaphores, SRCU, and reader-locked `rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task owner pointer. - Tracking these locks (as well as slab-cache and lock-type filters) relies on `fexit` (acquire) and `fentry` (release) BPF hooks maintaining the `held_locks` BPF LRU hash map: - Any such lock already held before `perf record` attaches its BPF programs will not be present in `held_locks` until it is released and re-acquired. - The attached `fentry`/`fexit` hooks execute on every kernel lock acquire and release for the hooked lock family across the system, adding tracing overhead outside of perf sample handling. - Cross-task semaphore signaling (`down*()` / `up()`): - Counting `struct semaphore`s do not enforce same-task ownership in the kernel: when a semaphore is used as a completion/signaling mechanism (`down()` in one task woken by `up()` in another task or interrupt handler, which proposed kernel dependency trackers like `CONFIG_DEPT` model as wait/event dependencies bounded by execution context rather than lock ownership), `fexit/down` and `fentry/up` do not pair on the same task ID. Semaphore hooks are therefore only enabled when `semaphore` or a `*_sem` symbol (such as `console_sem`) is explicitly specified, and `held_locks` uses `BPF_MAP_TYPE_LRU_HASH` so any unpaired cross-task entries are automatically evicted. Patch overview (patches 1 and 2 are standalone fixes independent of the rest of the series): 1. "perf lock contention: Filter events before creating lock_stat": Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to apply `-Y`, `-L`, and `-S` filters before calling `lock_stat_findnew()`. 2. "perf bpf-filter: Support BPF sample filters on tracepoint events": Adds `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF sample filters can attach to `PERF_TYPE_TRACEPOINT` events (where `__perf_event_set_bpf_prog()` in `kernel/events/core.c` requires `BPF_PROG_TYPE_TRACEPOINT` rather than `BPF_PROG_TYPE_PERF_EVENT`). 3. "perf lock contention: Share common BPF and lock type definitions in lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`, `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`, `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`. 4. "perf bpf-filter: Add lock_held() sample filter": Implements `lock_held()`, `lock_held_read()`/`lock_held_shared()`, `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode suffixes using direct lock owner inspection where possible (`mutex`, `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF LRU hash map updated by selectively attached `fentry`/`fexit` hooks. 5. "perf test: Add tests for lock_held() BPF sample filter": Adds unit tests in `tests/parse-events.c` and shell tests in `tests/shell/record_bpf_filter.sh`. 6. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU": Supports `rcu` (`rcu:R` via direct `current->rcu_read_lock_nesting` inspection with zero tracing hooks; `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` via `synchronize_srcu*()` / `srcu_barrier()`). 7. "perf lock contention: Extend BPF lock contention to RCU and SRCU": Extends `perf lock contention --use-bpf` to track RCU and SRCU grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). Changes in v4: - Move the two bugfix commits (`Filter events before creating lock_stat` and `Support BPF sample filters on tracepoint events`) to patches 1 and 2 at the start of the series for easier cherry-picking. - Reserve leading `&` strictly for slab cache locks (e.g. `lock_held(&dentry)`) so global/static lock symbols and well-known locks do not use `&` (e.g. `lock_held(cgroup_mutex)`, `lock_held_read(mmap_lock)`), matching `perf lock contention` (Namhyung). - Skip attaching `fentry`/`fexit` hooks when filtering on a specific `mutex` symbol (`*_mutex`), `rt_mutex` symbol (`*_rt_mutex`), or write-locked `mmap_lock` (`mmap_lock:W` / `lock_held_write(mmap_lock)`), evaluating them purely via direct `owner` inspection (Namhyung). - Use a CO-RE flavor struct `struct mutex___non_rt` in `get_lock_owner()` in `lock_data.h` so `GEN_VMLINUX_H=1` builds succeed on `CONFIG_PREEMPT_RT=y` kernels. - Switch `held_locks` to `BPF_MAP_TYPE_LRU_HASH`, exclude `LOCK_PROG_SEM` from `LOCK_PROG_ALL` unless `semaphore` or a `*_sem` symbol is explicitly requested, and document cross-task semaphore signaling limitations (and comparison with `CONFIG_DEPT`). - Skip `LCB_F_MUTEX | LCB_F_SPIN` in `parse_lock_type_name()`, reject read-only mode on exclusive-only lock symbol suffixes in `resolve_lock_expr()`, check `needed_groups & supported_groups` before masking `needed_groups` in `configure_lock_progs()`, and store `skel->links.perf_tp_sample_filter` so `sample_filter_bpf__destroy()` cleans up the tracepoint BPF link. - Recursively traverse anonymous structs/unions in `btf_has_task_field()` so `lock_held(rcu)` and `lock_held(rcu-trace)` work on `CONFIG_RANDSTRUCT` kernels. Changes in v3: - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c` so `perf lock contention -b -o` still reports reader owners when a writer contends on a reader-owned `rw_semaphore`, and scope `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`, `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`). - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()` when attaching, and document memory allocation / direct reclaim filtering with `lock_held()`. - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data` cannot be resolved and check `node_data_addr != 0` in `do_collect_zone_lock()`. - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call `_raw_read_lock_irqsave()`. - Offset `rcu_state_addr` by 1 byte so it is distinct from `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match timestamp maps. Changes in v2: - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to `sample_filter.bpf.c` and `lock_progs[]`. - Verify in `supported_lock_prog_groups()` that both acquire and all unlock BTF entry points exist for a lock family before enabling its tracing hooks, preventing acquire-only attachment when spinlock/rwlock unlock functions are inlined. - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` on `track_rwsem_read` and verify non-zero read/write counts in `record_lock_released_addr()` before decrementing aggregate slab/type counts. - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in `lock_contention.bpf.c` and recognize `synchronize_rcu*`, `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in `machine__is_lock_function()`. - Add patch fixing `report_lock_contention_begin_event()` to check type, address, and callstack filters before creating `lock_stat` entries. Assisted-by: Antigravity:gemini-3.1-pro Ian Rogers (7): perf lock contention: Filter events before creating lock_stat perf bpf-filter: Support BPF sample filters on tracepoint events perf lock contention: Share common BPF and lock type definitions in lock_data.h perf bpf-filter: Add lock_held() sample filter perf test: Add tests for lock_held() BPF sample filter perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU perf lock contention: Extend BPF lock contention to RCU and SRCU tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/Documentation/perf-record.txt | 66 +- tools/perf/builtin-lock.c | 88 +- tools/perf/tests/parse-events.c | 121 ++ tools/perf/tests/shell/lock_contention.sh | 14 + tools/perf/tests/shell/record_bpf_filter.sh | 142 ++ tools/perf/util/bpf-filter.c | 1300 ++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_lock_contention.c | 171 ++- .../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++--- tools/perf/util/bpf_skel/lock_data.h | 364 ++++- tools/perf/util/bpf_skel/sample-filter.h | 77 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 1231 ++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + tools/perf/util/lock-contention.c | 22 + tools/perf/util/lock-contention.h | 15 + tools/perf/util/machine.c | 11 + tools/perf/util/parse-events.c | 3 +- 20 files changed, 3790 insertions(+), 387 deletions(-) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers @ 2026-10-19 22:09 ` Ian Rogers 2026-10-20 4:34 ` Namhyung Kim 2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers ` (6 subsequent siblings) 7 siblings, 1 reply; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu In report_lock_contention_begin_event(), lock_stat_findnew() was called before checking filters.nr_types (-Y), filters.nr_addrs (-L), and match_callstack_filter() (-S). When a filtered-out lock event (such as a mutex contention when running with -Y spinlock) shared the same aggregation key as a subsequent matching event (for example, contention on the mutex's internal wait_lock spinlock from the same caller), the first event created the lock_stat entry with the filtered-out lock's flags and the second event reused that entry, causing non-spinlock entries to appear in -Y spinlock output. Check the type, address, and callstack filters before looking up or creating the lock_stat entry. Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..778923c92f42 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) struct machine *machine = &session->machines.host; struct map *kmap; struct symbol *sym; + u64 *callstack = NULL; ret = get_key_by_aggr_mode(&key, addr, sample); if (ret < 0) @@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } } - ls = lock_stat_find(key); - if (!ls) { - char buf[128]; - const char *name = ""; - - switch (aggr_mode) { - case LOCK_AGGR_ADDR: - sym = machine__find_kernel_symbol(machine, key, &kmap); - if (sym) - name = sym->name; - break; - case LOCK_AGGR_CALLER: - name = buf; - if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) - name = "Unknown"; - break; - case LOCK_AGGR_CGROUP: - case LOCK_AGGR_TASK: - default: - break; - } - - ls = lock_stat_findnew(key, name, flags); - if (!ls) - return -ENOMEM; - } - if (filters.nr_types) { bool found = false; @@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } if (needs_callstack()) { - u64 *callstack = get_callstack(sample, max_stack_depth); + callstack = get_callstack(sample, max_stack_depth); if (callstack == NULL) return 0; @@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) free(callstack); return 0; } + } + + ls = lock_stat_find(key); + if (!ls) { + char buf[128]; + const char *name = ""; + + switch (aggr_mode) { + case LOCK_AGGR_ADDR: + sym = machine__find_kernel_symbol(machine, key, &kmap); + if (sym) + name = sym->name; + break; + case LOCK_AGGR_CALLER: + name = buf; + if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) + name = "Unknown"; + break; + case LOCK_AGGR_CGROUP: + case LOCK_AGGR_TASK: + default: + break; + } + + ls = lock_stat_findnew(key, name, flags); + if (!ls) { + free(callstack); + return -ENOMEM; + } + } + if (callstack) { if (ls->callstack == NULL) ls->callstack = callstack; else -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat 2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers @ 2026-10-20 4:34 ` Namhyung Kim 2026-10-21 21:39 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 59+ messages in thread From: Namhyung Kim @ 2026-10-20 4:34 UTC (permalink / raw) To: Ian Rogers Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu On Mon, Oct 19, 2026 at 03:09:09PM -0700, Ian Rogers wrote: > In report_lock_contention_begin_event(), lock_stat_findnew() was > called before checking filters.nr_types (-Y), filters.nr_addrs (-L), > and match_callstack_filter() (-S). > > When a filtered-out lock event (such as a mutex contention when > running with -Y spinlock) shared the same aggregation key as a > subsequent matching event (for example, contention on the mutex's > internal wait_lock spinlock from the same caller), the first event > created the lock_stat entry with the filtered-out lock's flags and the > second event reused that entry, causing non-spinlock entries to appear > in -Y spinlock output. > > Check the type, address, and callstack filters before looking up or > creating the lock_stat entry. > > Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") > Signed-off-by: Ian Rogers <irogers@google.com> Reviewed-by: Namhyung Kim <namhyung@kernel.org> Thanks, Namhyung > --- > tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------ > 1 file changed, 33 insertions(+), 28 deletions(-) > > diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c > index d5c0d55cb82d..778923c92f42 100644 > --- a/tools/perf/builtin-lock.c > +++ b/tools/perf/builtin-lock.c > @@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) > struct machine *machine = &session->machines.host; > struct map *kmap; > struct symbol *sym; > + u64 *callstack = NULL; > > ret = get_key_by_aggr_mode(&key, addr, sample); > if (ret < 0) > @@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) > } > } > > - ls = lock_stat_find(key); > - if (!ls) { > - char buf[128]; > - const char *name = ""; > - > - switch (aggr_mode) { > - case LOCK_AGGR_ADDR: > - sym = machine__find_kernel_symbol(machine, key, &kmap); > - if (sym) > - name = sym->name; > - break; > - case LOCK_AGGR_CALLER: > - name = buf; > - if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) > - name = "Unknown"; > - break; > - case LOCK_AGGR_CGROUP: > - case LOCK_AGGR_TASK: > - default: > - break; > - } > - > - ls = lock_stat_findnew(key, name, flags); > - if (!ls) > - return -ENOMEM; > - } > - > if (filters.nr_types) { > bool found = false; > > @@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) > } > > if (needs_callstack()) { > - u64 *callstack = get_callstack(sample, max_stack_depth); > + callstack = get_callstack(sample, max_stack_depth); > if (callstack == NULL) > return 0; > > @@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) > free(callstack); > return 0; > } > + } > + > + ls = lock_stat_find(key); > + if (!ls) { > + char buf[128]; > + const char *name = ""; > + > + switch (aggr_mode) { > + case LOCK_AGGR_ADDR: > + sym = machine__find_kernel_symbol(machine, key, &kmap); > + if (sym) > + name = sym->name; > + break; > + case LOCK_AGGR_CALLER: > + name = buf; > + if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) > + name = "Unknown"; > + break; > + case LOCK_AGGR_CGROUP: > + case LOCK_AGGR_TASK: > + default: > + break; > + } > + > + ls = lock_stat_findnew(key, name, flags); > + if (!ls) { > + free(callstack); > + return -ENOMEM; > + } > + } > > + if (callstack) { > if (ls->callstack == NULL) > ls->callstack = callstack; > else > -- > 2.56.0.212.g129ede20d9-goog > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat 2026-10-20 4:34 ` Namhyung Kim @ 2026-10-21 21:39 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 59+ messages in thread From: Arnaldo Carvalho de Melo @ 2026-10-21 21:39 UTC (permalink / raw) To: Namhyung Kim Cc: Ian Rogers, adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu On Mon, Oct 19, 2026 at 09:34:04PM -0700, Namhyung Kim wrote: > On Mon, Oct 19, 2026 at 03:09:09PM -0700, Ian Rogers wrote: > > In report_lock_contention_begin_event(), lock_stat_findnew() was > > called before checking filters.nr_types (-Y), filters.nr_addrs (-L), > > and match_callstack_filter() (-S). > > > > When a filtered-out lock event (such as a mutex contention when > > running with -Y spinlock) shared the same aggregation key as a > > subsequent matching event (for example, contention on the mutex's > > internal wait_lock spinlock from the same caller), the first event > > created the lock_stat entry with the filtered-out lock's flags and the > > second event reused that entry, causing non-spinlock entries to appear > > in -Y spinlock output. > > > > Check the type, address, and callstack filters before looking up or > > creating the lock_stat entry. > > > > Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") > > Signed-off-by: Ian Rogers <irogers@google.com> > > Reviewed-by: Namhyung Kim <namhyung@kernel.org> Cherry picked this one as its the first in the series. - Arnaldo ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers 2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers @ 2026-10-19 22:09 ` Ian Rogers 2026-10-20 5:41 ` Namhyung Kim 2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers ` (5 subsequent siblings) 7 siblings, 1 reply; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper") updated set_filter() to route non-tracepoint filter terms (such as uid and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead of the ftrace tracepoint predicate filter. However, in the kernel's __perf_event_set_bpf_prog() (kernel/events/core.c), attaching a BPF program to a perf_event fd branches on perf_event_is_tracing(event): - Non-tracing events (hardware PMU, software events, breakpoints) call perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT (SEC("perf_event")). - Tracepoint events (event->pmu == &perf_tracepoint) reject BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to event->tp_event->prog_array and executed in perf_trace_run_bpf_submit() before perf_tp_event(), dropping the sample if the BPF program returns 0. Furthermore, the bpf_cast_to_kern_ctx() kfunc is only permitted in BPF_PROG_TYPE_PERF_EVENT programs. Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in sample_filter.bpf.c to evaluate context-independent filter terms (uid, gid, tid, cpu, time, cgroup) using BPF helpers without calling bpf_cast_to_kern_ctx(). When preparing a BPF filter on a PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of perf_sample_filter and attach it once to the shared trace_event_call. Before: # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument Failed to attach perf sample-filter program failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument) After: # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ] Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/util/bpf-filter.c | 136 ++++++++++++++++--- tools/perf/util/bpf_skel/sample_filter.bpf.c | 85 ++++++++++++ 2 files changed, 201 insertions(+), 20 deletions(-) diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..4e99af4764f1 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * { const struct perf_sample_info *info; - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) - return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU || + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP) + return 0; + if (expr->op != PBF_OP_GROUP_BEGIN) { + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n", + evsel__name(evsel), expr->term); + return -1; + } + } + + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) + return 0; + if (expr->op == PBF_OP_GROUP_BEGIN) { struct perf_bpf_filter_expr *group; @@ -427,6 +438,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -444,9 +456,15 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - if (needs_idx_hash && geteuid() != 0) { + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + if (is_tracepoint) { + pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -484,8 +502,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; @@ -501,15 +528,35 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - prog = skel->progs.perf_sample_filter; - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), - &pe_opts); - if (IS_ERR(link)) { - pr_err("Failed to attach perf sample-filter program\n"); - ret = PTR_ERR(link); - goto err; + if (is_tracepoint) { + /* + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the + * BPF program to event->tp_event->prog_array on the shared + * trace_event_call, which perf_trace_run_bpf_submit() executes + * for all CPUs and tasks before iterating the per-CPU perf + * events. Attach once on the first FD so tp_event->prog_array + * does not get duplicate copies for each per-CPU FD. + */ + prog = skel->progs.perf_tp_sample_filter; + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0), + &pe_opts); + ret = libbpf_get_error(link); + if (ret) { + pr_err("Failed to attach perf tracepoint sample-filter program\n"); + goto err; + } + skel->links.perf_tp_sample_filter = link; + } else { + prog = skel->progs.perf_sample_filter; + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), + &pe_opts); + ret = libbpf_get_error(link); + if (ret) { + pr_err("Failed to attach perf sample-filter program\n"); + goto err; + } } } } @@ -641,6 +688,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -655,6 +741,7 @@ int perf_bpf_filter__pin(void) } /* pinned program will use pid-hash */ + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); @@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -683,8 +770,8 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); - ret = dir_fd; + ret = -errno; + unpin_filter_object(skel, path); goto err; } @@ -727,6 +814,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -740,13 +829,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -754,7 +850,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..576fbed270e5 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return 0; } +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry) +{ + switch (entry->term) { + case PBF_TERM_TID: + if (entry->part) + return bpf_get_current_pid_tgid() >> 32; + return (__u32)bpf_get_current_pid_tgid(); + case PBF_TERM_CPU: + return bpf_get_smp_processor_id(); + case PBF_TERM_TIME: + return bpf_ktime_get_ns(); + case PBF_TERM_CGROUP: + return bpf_get_current_cgroup_id(); + case PBF_TERM_UID: + return bpf_get_current_uid_gid() & 0xFFFFFFFF; + case PBF_TERM_GID: + return bpf_get_current_uid_gid() >> 32; + default: + return 0; + } +} + #define CHECK_RESULT(data, op, val) \ if (!(data op val)) { \ if (!in_group) \ @@ -295,4 +317,67 @@ int perf_sample_filter(void *ctx) return 0; } +/* BPF program to be called for tracepoint perf events */ +SEC("tracepoint") +int perf_tp_sample_filter(void *ctx) +{ + struct perf_bpf_filter_entry *entry; + __u64 sample_data; + int in_group = 0; + int group_result = 0; + int i, k = 0; + int *losts; + + entry = bpf_map_lookup_elem(&filters, &k); + if (entry == NULL) + goto drop; + + for (i = 0; i < MAX_FILTERS; i++) { + sample_data = perf_get_common_sample(&entry[i]); + + switch (entry[i].op) { + case PBF_OP_EQ: + CHECK_RESULT(sample_data, ==, entry[i].value) + break; + case PBF_OP_NEQ: + CHECK_RESULT(sample_data, !=, entry[i].value) + break; + case PBF_OP_GT: + CHECK_RESULT(sample_data, >, entry[i].value) + break; + case PBF_OP_GE: + CHECK_RESULT(sample_data, >=, entry[i].value) + break; + case PBF_OP_LT: + CHECK_RESULT(sample_data, <, entry[i].value) + break; + case PBF_OP_LE: + CHECK_RESULT(sample_data, <=, entry[i].value) + break; + case PBF_OP_AND: + CHECK_RESULT(sample_data, &, entry[i].value) + break; + case PBF_OP_GROUP_BEGIN: + in_group = 1; + group_result = 0; + break; + case PBF_OP_GROUP_END: + if (group_result == 0) + goto drop; + in_group = 0; + break; + case PBF_OP_DONE: + return 1; + } + } + return 1; + +drop: + losts = bpf_map_lookup_elem(&dropped, &k); + if (losts != NULL) + __sync_fetch_and_add(losts, 1); + + return 0; +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events 2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers @ 2026-10-20 5:41 ` Namhyung Kim 2026-10-20 15:30 ` Ian Rogers 0 siblings, 1 reply; 59+ messages in thread From: Namhyung Kim @ 2026-10-20 5:41 UTC (permalink / raw) To: Ian Rogers Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu On Mon, Oct 19, 2026 at 03:09:10PM -0700, Ian Rogers wrote: > Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper") > updated set_filter() to route non-tracepoint filter terms (such as uid > and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead > of the ftrace tracepoint predicate filter. > > However, in the kernel's __perf_event_set_bpf_prog() > (kernel/events/core.c), attaching a BPF program to a perf_event fd > branches on perf_event_is_tracing(event): > - Non-tracing events (hardware PMU, software events, breakpoints) call > perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT > (SEC("perf_event")). > - Tracepoint events (event->pmu == &perf_tracepoint) reject > BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require > BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to > event->tp_event->prog_array and executed in > perf_trace_run_bpf_submit() before perf_tp_event(), dropping the > sample if the BPF program returns 0. Furthermore, the > bpf_cast_to_kern_ctx() kfunc is only permitted in > BPF_PROG_TYPE_PERF_EVENT programs. > > Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in > sample_filter.bpf.c to evaluate context-independent filter terms (uid, > gid, tid, cpu, time, cgroup) using BPF helpers without calling > bpf_cast_to_kern_ctx(). When preparing a BPF filter on a > PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of > perf_sample_filter and attach it once to the shared trace_event_call. > > Before: > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true > libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument > Failed to attach perf sample-filter program > failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument) > > After: > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true > [ perf record: Woken up 1 times to write data ] > [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ] > > Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper") > Signed-off-by: Ian Rogers <irogers@google.com> > --- > tools/perf/util/bpf-filter.c | 136 ++++++++++++++++--- > tools/perf/util/bpf_skel/sample_filter.bpf.c | 85 ++++++++++++ > 2 files changed, 201 insertions(+), 20 deletions(-) > > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c > index 72e83ac1d6a4..4e99af4764f1 100644 > --- a/tools/perf/util/bpf-filter.c > +++ b/tools/perf/util/bpf-filter.c > @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * > { > const struct perf_sample_info *info; > > - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && > - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) > - return 0; > - > if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { > /* Not dependent on the sample_type as computed from a BPF helper. */ > return 0; > } > > + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { > + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU || > + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP) > + return 0; > + if (expr->op != PBF_OP_GROUP_BEGIN) { > + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n", > + evsel__name(evsel), expr->term); > + return -1; > + } > + } > + > + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && > + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) > + return 0; > + > if (expr->op == PBF_OP_GROUP_BEGIN) { > struct perf_bpf_filter_expr *group; > > @@ -427,6 +438,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > struct bpf_link *link; > struct perf_bpf_filter_entry *entry; > bool needs_idx_hash = !target__has_cpu(target); > + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); > #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) > DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, > .dont_enable = true); > @@ -444,9 +456,15 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > goto err; > } > > - if (needs_idx_hash && geteuid() != 0) { > + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { > int zero = 0; > > + if (is_tracepoint) { > + pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); > + ret = -EPERM; > + goto err; > + } > + > /* The filters map is shared among other processes */ > ret = create_idx_hash(evsel, entry); > if (ret < 0) > @@ -484,8 +502,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > return 0; > } > > - skel = sample_filter_bpf__open_and_load(); > + skel = sample_filter_bpf__open(); > if (!skel) { > + ret = -errno; > + pr_err("Failed to open perf sample-filter BPF skeleton\n"); > + goto err; > + } > + > + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); > + > + if (sample_filter_bpf__load(skel) < 0) { > ret = -errno; > pr_err("Failed to load perf sample-filter BPF skeleton\n"); > goto err; > @@ -501,15 +528,35 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > goto err; > } > > - prog = skel->progs.perf_sample_filter; > - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { > - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { > - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), > - &pe_opts); > - if (IS_ERR(link)) { > - pr_err("Failed to attach perf sample-filter program\n"); > - ret = PTR_ERR(link); > - goto err; > + if (is_tracepoint) { > + /* > + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the > + * BPF program to event->tp_event->prog_array on the shared > + * trace_event_call, which perf_trace_run_bpf_submit() executes > + * for all CPUs and tasks before iterating the per-CPU perf > + * events. Attach once on the first FD so tp_event->prog_array > + * does not get duplicate copies for each per-CPU FD. > + */ > + prog = skel->progs.perf_tp_sample_filter; > + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0), > + &pe_opts); > + ret = libbpf_get_error(link); > + if (ret) { > + pr_err("Failed to attach perf tracepoint sample-filter program\n"); > + goto err; > + } > + skel->links.perf_tp_sample_filter = link; > + } else { > + prog = skel->progs.perf_sample_filter; > + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { > + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { > + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), > + &pe_opts); This link should be saves somewhere. I've sent a fix for it. > + ret = libbpf_get_error(link); > + if (ret) { > + pr_err("Failed to attach perf sample-filter program\n"); > + goto err; > + } > } > } > } > @@ -641,6 +688,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) > return ret; > } > > +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) > +{ > + char *prog_path = NULL; > + int ret; > + > + ret = bpf_object__pin_maps(skel->obj, path); > + if (ret < 0) > + return ret; > + > + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { > + bpf_object__unpin_maps(skel->obj, path); > + return -ENOMEM; > + } > + > + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); > + if (ret < 0) > + bpf_object__unpin_maps(skel->obj, path); > + > + free(prog_path); > + return ret; > +} > + > +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) > +{ > + char *prog_path = NULL; > + int ret, map_ret; > + > + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { > + bpf_object__unpin_maps(skel->obj, path); > + return -ENOMEM; > + } > + > + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); > + free(prog_path); > + > + map_ret = bpf_object__unpin_maps(skel->obj, path); > + return ret < 0 ? ret : map_ret; > +} > + > int perf_bpf_filter__pin(void) > { > struct sample_filter_bpf *skel; > @@ -655,6 +741,7 @@ int perf_bpf_filter__pin(void) > } > > /* pinned program will use pid-hash */ > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); > bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); > bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); > bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); > @@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void) > goto err; > } > > - ret = bpf_object__pin(skel->obj, path); > + ret = pin_filter_object(skel, path); Why is this change needed? Thanks, Namhyung > if (ret < 0) { > pr_err("Failed to pin BPF filter objects\n"); > goto err; > @@ -683,8 +770,8 @@ int perf_bpf_filter__pin(void) > /* setup access permissions for the pinned objects */ > dir_fd = open(path, O_PATH); > if (dir_fd < 0) { > - bpf_object__unpin(skel->obj, path); > - ret = dir_fd; > + ret = -errno; > + unpin_filter_object(skel, path); > goto err; > } > > @@ -727,6 +814,8 @@ int perf_bpf_filter__pin(void) > > err_close: > close(dir_fd); > + if (ret < 0) > + unpin_filter_object(skel, path); > > err: > free(path); > @@ -740,13 +829,20 @@ int perf_bpf_filter__unpin(void) > char *path = NULL; > int ret = -1; > > - skel = sample_filter_bpf__open_and_load(); > + skel = sample_filter_bpf__open(); > if (!skel) { > ret = -errno; > pr_err("Failed to open perf sample-filter BPF skeleton\n"); > goto err; > } > > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); > + if (sample_filter_bpf__load(skel) < 0) { > + ret = -errno; > + pr_err("Failed to load perf sample-filter BPF skeleton\n"); > + goto err; > + } > + > if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), > PERF_BPF_FILTER_PIN_PATH) < 0) { > ret = -errno; > @@ -754,7 +850,7 @@ int perf_bpf_filter__unpin(void) > goto err; > } > > - ret = bpf_object__unpin(skel->obj, path); > + ret = unpin_filter_object(skel, path); > > err: > free(path); > diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c > index e5666d4c1722..576fbed270e5 100644 > --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c > +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c > @@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, > return 0; > } > > +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry) > +{ > + switch (entry->term) { > + case PBF_TERM_TID: > + if (entry->part) > + return bpf_get_current_pid_tgid() >> 32; > + return (__u32)bpf_get_current_pid_tgid(); > + case PBF_TERM_CPU: > + return bpf_get_smp_processor_id(); > + case PBF_TERM_TIME: > + return bpf_ktime_get_ns(); > + case PBF_TERM_CGROUP: > + return bpf_get_current_cgroup_id(); > + case PBF_TERM_UID: > + return bpf_get_current_uid_gid() & 0xFFFFFFFF; > + case PBF_TERM_GID: > + return bpf_get_current_uid_gid() >> 32; > + default: > + return 0; > + } > +} > + > #define CHECK_RESULT(data, op, val) \ > if (!(data op val)) { \ > if (!in_group) \ > @@ -295,4 +317,67 @@ int perf_sample_filter(void *ctx) > return 0; > } > > +/* BPF program to be called for tracepoint perf events */ > +SEC("tracepoint") > +int perf_tp_sample_filter(void *ctx) > +{ > + struct perf_bpf_filter_entry *entry; > + __u64 sample_data; > + int in_group = 0; > + int group_result = 0; > + int i, k = 0; > + int *losts; > + > + entry = bpf_map_lookup_elem(&filters, &k); > + if (entry == NULL) > + goto drop; > + > + for (i = 0; i < MAX_FILTERS; i++) { > + sample_data = perf_get_common_sample(&entry[i]); > + > + switch (entry[i].op) { > + case PBF_OP_EQ: > + CHECK_RESULT(sample_data, ==, entry[i].value) > + break; > + case PBF_OP_NEQ: > + CHECK_RESULT(sample_data, !=, entry[i].value) > + break; > + case PBF_OP_GT: > + CHECK_RESULT(sample_data, >, entry[i].value) > + break; > + case PBF_OP_GE: > + CHECK_RESULT(sample_data, >=, entry[i].value) > + break; > + case PBF_OP_LT: > + CHECK_RESULT(sample_data, <, entry[i].value) > + break; > + case PBF_OP_LE: > + CHECK_RESULT(sample_data, <=, entry[i].value) > + break; > + case PBF_OP_AND: > + CHECK_RESULT(sample_data, &, entry[i].value) > + break; > + case PBF_OP_GROUP_BEGIN: > + in_group = 1; > + group_result = 0; > + break; > + case PBF_OP_GROUP_END: > + if (group_result == 0) > + goto drop; > + in_group = 0; > + break; > + case PBF_OP_DONE: > + return 1; > + } > + } > + return 1; > + > +drop: > + losts = bpf_map_lookup_elem(&dropped, &k); > + if (losts != NULL) > + __sync_fetch_and_add(losts, 1); > + > + return 0; > +} > + > char LICENSE[] SEC("license") = "Dual BSD/GPL"; > -- > 2.56.0.212.g129ede20d9-goog > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events 2026-10-20 5:41 ` Namhyung Kim @ 2026-10-20 15:30 ` Ian Rogers 2026-10-20 22:29 ` Namhyung Kim 0 siblings, 1 reply; 59+ messages in thread From: Ian Rogers @ 2026-10-20 15:30 UTC (permalink / raw) To: Namhyung Kim Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu On Mon, Oct 19, 2026 at 10:41 PM Namhyung Kim <namhyung@kernel.org> wrote: > > On Mon, Oct 19, 2026 at 03:09:10PM -0700, Ian Rogers wrote: > > Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper") > > updated set_filter() to route non-tracepoint filter terms (such as uid > > and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead > > of the ftrace tracepoint predicate filter. > > > > However, in the kernel's __perf_event_set_bpf_prog() > > (kernel/events/core.c), attaching a BPF program to a perf_event fd > > branches on perf_event_is_tracing(event): > > - Non-tracing events (hardware PMU, software events, breakpoints) call > > perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT > > (SEC("perf_event")). > > - Tracepoint events (event->pmu == &perf_tracepoint) reject > > BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require > > BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to > > event->tp_event->prog_array and executed in > > perf_trace_run_bpf_submit() before perf_tp_event(), dropping the > > sample if the BPF program returns 0. Furthermore, the > > bpf_cast_to_kern_ctx() kfunc is only permitted in > > BPF_PROG_TYPE_PERF_EVENT programs. > > > > Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in > > sample_filter.bpf.c to evaluate context-independent filter terms (uid, > > gid, tid, cpu, time, cgroup) using BPF helpers without calling > > bpf_cast_to_kern_ctx(). When preparing a BPF filter on a > > PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of > > perf_sample_filter and attach it once to the shared trace_event_call. > > > > Before: > > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true > > libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument > > Failed to attach perf sample-filter program > > failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument) > > > > After: > > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true > > [ perf record: Woken up 1 times to write data ] > > [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ] > > > > Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper") > > Signed-off-by: Ian Rogers <irogers@google.com> > > --- > > tools/perf/util/bpf-filter.c | 136 ++++++++++++++++--- > > tools/perf/util/bpf_skel/sample_filter.bpf.c | 85 ++++++++++++ > > 2 files changed, 201 insertions(+), 20 deletions(-) > > > > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c > > index 72e83ac1d6a4..4e99af4764f1 100644 > > --- a/tools/perf/util/bpf-filter.c > > +++ b/tools/perf/util/bpf-filter.c > > @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * > > { > > const struct perf_sample_info *info; > > > > - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && > > - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) > > - return 0; > > - > > if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { > > /* Not dependent on the sample_type as computed from a BPF helper. */ > > return 0; > > } > > > > + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { > > + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU || > > + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP) > > + return 0; > > + if (expr->op != PBF_OP_GROUP_BEGIN) { > > + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n", > > + evsel__name(evsel), expr->term); > > + return -1; > > + } > > + } > > + > > + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && > > + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) > > + return 0; > > + > > if (expr->op == PBF_OP_GROUP_BEGIN) { > > struct perf_bpf_filter_expr *group; > > > > @@ -427,6 +438,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > > struct bpf_link *link; > > struct perf_bpf_filter_entry *entry; > > bool needs_idx_hash = !target__has_cpu(target); > > + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); > > #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) > > DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, > > .dont_enable = true); > > @@ -444,9 +456,15 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > > goto err; > > } > > > > - if (needs_idx_hash && geteuid() != 0) { > > + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { > > int zero = 0; > > > > + if (is_tracepoint) { > > + pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); > > + ret = -EPERM; > > + goto err; > > + } > > + > > /* The filters map is shared among other processes */ > > ret = create_idx_hash(evsel, entry); > > if (ret < 0) > > @@ -484,8 +502,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > > return 0; > > } > > > > - skel = sample_filter_bpf__open_and_load(); > > + skel = sample_filter_bpf__open(); > > if (!skel) { > > + ret = -errno; > > + pr_err("Failed to open perf sample-filter BPF skeleton\n"); > > + goto err; > > + } > > + > > + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); > > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); > > + > > + if (sample_filter_bpf__load(skel) < 0) { > > ret = -errno; > > pr_err("Failed to load perf sample-filter BPF skeleton\n"); > > goto err; > > @@ -501,15 +528,35 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > > goto err; > > } > > > > - prog = skel->progs.perf_sample_filter; > > - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { > > - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { > > - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), > > - &pe_opts); > > - if (IS_ERR(link)) { > > - pr_err("Failed to attach perf sample-filter program\n"); > > - ret = PTR_ERR(link); > > - goto err; > > + if (is_tracepoint) { > > + /* > > + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the > > + * BPF program to event->tp_event->prog_array on the shared > > + * trace_event_call, which perf_trace_run_bpf_submit() executes > > + * for all CPUs and tasks before iterating the per-CPU perf > > + * events. Attach once on the first FD so tp_event->prog_array > > + * does not get duplicate copies for each per-CPU FD. > > + */ > > + prog = skel->progs.perf_tp_sample_filter; > > + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0), > > + &pe_opts); > > + ret = libbpf_get_error(link); > > + if (ret) { > > + pr_err("Failed to attach perf tracepoint sample-filter program\n"); > > + goto err; > > + } > > + skel->links.perf_tp_sample_filter = link; > > + } else { > > + prog = skel->progs.perf_sample_filter; > > + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { > > + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { > > + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), > > + &pe_opts); > > This link should be saves somewhere. I've sent a fix for it. Good catch! I'll follow up on that change. > > + ret = libbpf_get_error(link); > > + if (ret) { > > + pr_err("Failed to attach perf sample-filter program\n"); > > + goto err; > > + } > > } > > } > > } > > @@ -641,6 +688,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) > > return ret; > > } > > > > +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) > > +{ > > + char *prog_path = NULL; > > + int ret; > > + > > + ret = bpf_object__pin_maps(skel->obj, path); > > + if (ret < 0) > > + return ret; > > + > > + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { > > + bpf_object__unpin_maps(skel->obj, path); > > + return -ENOMEM; > > + } > > + > > + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); > > + if (ret < 0) > > + bpf_object__unpin_maps(skel->obj, path); > > + > > + free(prog_path); > > + return ret; > > +} > > + > > +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) > > +{ > > + char *prog_path = NULL; > > + int ret, map_ret; > > + > > + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { > > + bpf_object__unpin_maps(skel->obj, path); > > + return -ENOMEM; > > + } > > + > > + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); > > + free(prog_path); > > + > > + map_ret = bpf_object__unpin_maps(skel->obj, path); > > + return ret < 0 ? ret : map_ret; > > +} > > + > > int perf_bpf_filter__pin(void) > > { > > struct sample_filter_bpf *skel; > > @@ -655,6 +741,7 @@ int perf_bpf_filter__pin(void) > > } > > > > /* pinned program will use pid-hash */ > > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); > > bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); > > bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); > > bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); > > @@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void) > > goto err; > > } > > > > - ret = bpf_object__pin(skel->obj, path); > > + ret = pin_filter_object(skel, path); > > Why is this change needed? Before this patch, sample_filter.bpf.c only contained a single BPF program (perf_sample_filter), which was always loaded. This patch adds a second program (perf_tp_sample_filter) to the same skeleton (and patch 4 adds the lock-tracking programs to it as well), and uses bpf_program__set_autoload() before loading the skeleton so we can select only the program(s) needed for a given event. In perf_bpf_filter__pin() and perf_bpf_filter__unpin() (for 'perf record --setup-filter pin|unpin'), only perf_sample_filter is used, because the pinned filter mechanism shares a single program across privileged sessions by looking up kctx->event->id via bpf_cast_to_kern_ctx(), which is only available to SEC("perf_event") programs. So autoload is disabled for perf_tp_sample_filter (and in patch 4, for the lock-tracking programs) when pinning and unpinning. A consequence of disabling autoload on any program in the skeleton is that libbpf's bpf_object__pin() and bpf_object__unpin() fail: - bpf_object__pin() calls bpf_object__pin_maps() and bpf_object__pin_programs(). - While bpf_object__pin_maps() skips maps that have autocreate disabled, bpf_object__pin_programs() and bpf_object__unpin_programs() iterate over every program in skel->obj without checking whether autoload was enabled. - When bpf_program__pin() or bpf_program__unpin() is called on a program that wasn't loaded (prog->fd < 0), libbpf returns -EINVAL ("prog 'perf_tp_sample_filter': can't pin/unpin program that wasn't loaded"). pin_filter_object() and unpin_filter_object() avoid this by calling bpf_object__pin_maps() / bpf_object__unpin_maps() for the maps and explicitly pinning/unpinning only the loaded perf_sample_filter program. Thanks, Ian > Thanks, > Namhyung > > > > if (ret < 0) { > > pr_err("Failed to pin BPF filter objects\n"); > > goto err; > > @@ -683,8 +770,8 @@ int perf_bpf_filter__pin(void) > > /* setup access permissions for the pinned objects */ > > dir_fd = open(path, O_PATH); > > if (dir_fd < 0) { > > - bpf_object__unpin(skel->obj, path); > > - ret = dir_fd; > > + ret = -errno; > > + unpin_filter_object(skel, path); > > goto err; > > } > > > > @@ -727,6 +814,8 @@ int perf_bpf_filter__pin(void) > > > > err_close: > > close(dir_fd); > > + if (ret < 0) > > + unpin_filter_object(skel, path); > > > > err: > > free(path); > > @@ -740,13 +829,20 @@ int perf_bpf_filter__unpin(void) > > char *path = NULL; > > int ret = -1; > > > > - skel = sample_filter_bpf__open_and_load(); > > + skel = sample_filter_bpf__open(); > > if (!skel) { > > ret = -errno; > > pr_err("Failed to open perf sample-filter BPF skeleton\n"); > > goto err; > > } > > > > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); > > + if (sample_filter_bpf__load(skel) < 0) { > > + ret = -errno; > > + pr_err("Failed to load perf sample-filter BPF skeleton\n"); > > + goto err; > > + } > > + > > if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), > > PERF_BPF_FILTER_PIN_PATH) < 0) { > > ret = -errno; > > @@ -754,7 +850,7 @@ int perf_bpf_filter__unpin(void) > > goto err; > > } > > > > - ret = bpf_object__unpin(skel->obj, path); > > + ret = unpin_filter_object(skel, path); > > > > err: > > free(path); > > diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c > > index e5666d4c1722..576fbed270e5 100644 > > --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c > > +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c > > @@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, > > return 0; > > } > > > > +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry) > > +{ > > + switch (entry->term) { > > + case PBF_TERM_TID: > > + if (entry->part) > > + return bpf_get_current_pid_tgid() >> 32; > > + return (__u32)bpf_get_current_pid_tgid(); > > + case PBF_TERM_CPU: > > + return bpf_get_smp_processor_id(); > > + case PBF_TERM_TIME: > > + return bpf_ktime_get_ns(); > > + case PBF_TERM_CGROUP: > > + return bpf_get_current_cgroup_id(); > > + case PBF_TERM_UID: > > + return bpf_get_current_uid_gid() & 0xFFFFFFFF; > > + case PBF_TERM_GID: > > + return bpf_get_current_uid_gid() >> 32; > > + default: > > + return 0; > > + } > > +} > > + > > #define CHECK_RESULT(data, op, val) \ > > if (!(data op val)) { \ > > if (!in_group) \ > > @@ -295,4 +317,67 @@ int perf_sample_filter(void *ctx) > > return 0; > > } > > > > +/* BPF program to be called for tracepoint perf events */ > > +SEC("tracepoint") > > +int perf_tp_sample_filter(void *ctx) > > +{ > > + struct perf_bpf_filter_entry *entry; > > + __u64 sample_data; > > + int in_group = 0; > > + int group_result = 0; > > + int i, k = 0; > > + int *losts; > > + > > + entry = bpf_map_lookup_elem(&filters, &k); > > + if (entry == NULL) > > + goto drop; > > + > > + for (i = 0; i < MAX_FILTERS; i++) { > > + sample_data = perf_get_common_sample(&entry[i]); > > + > > + switch (entry[i].op) { > > + case PBF_OP_EQ: > > + CHECK_RESULT(sample_data, ==, entry[i].value) > > + break; > > + case PBF_OP_NEQ: > > + CHECK_RESULT(sample_data, !=, entry[i].value) > > + break; > > + case PBF_OP_GT: > > + CHECK_RESULT(sample_data, >, entry[i].value) > > + break; > > + case PBF_OP_GE: > > + CHECK_RESULT(sample_data, >=, entry[i].value) > > + break; > > + case PBF_OP_LT: > > + CHECK_RESULT(sample_data, <, entry[i].value) > > + break; > > + case PBF_OP_LE: > > + CHECK_RESULT(sample_data, <=, entry[i].value) > > + break; > > + case PBF_OP_AND: > > + CHECK_RESULT(sample_data, &, entry[i].value) > > + break; > > + case PBF_OP_GROUP_BEGIN: > > + in_group = 1; > > + group_result = 0; > > + break; > > + case PBF_OP_GROUP_END: > > + if (group_result == 0) > > + goto drop; > > + in_group = 0; > > + break; > > + case PBF_OP_DONE: > > + return 1; > > + } > > + } > > + return 1; > > + > > +drop: > > + losts = bpf_map_lookup_elem(&dropped, &k); > > + if (losts != NULL) > > + __sync_fetch_and_add(losts, 1); > > + > > + return 0; > > +} > > + > > char LICENSE[] SEC("license") = "Dual BSD/GPL"; > > -- > > 2.56.0.212.g129ede20d9-goog > > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events 2026-10-20 15:30 ` Ian Rogers @ 2026-10-20 22:29 ` Namhyung Kim 0 siblings, 0 replies; 59+ messages in thread From: Namhyung Kim @ 2026-10-20 22:29 UTC (permalink / raw) To: Ian Rogers Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu On Tue, Oct 20, 2026 at 08:30:26AM -0700, Ian Rogers wrote: > On Mon, Oct 19, 2026 at 10:41 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > On Mon, Oct 19, 2026 at 03:09:10PM -0700, Ian Rogers wrote: > > > Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper") > > > updated set_filter() to route non-tracepoint filter terms (such as uid > > > and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead > > > of the ftrace tracepoint predicate filter. > > > > > > However, in the kernel's __perf_event_set_bpf_prog() > > > (kernel/events/core.c), attaching a BPF program to a perf_event fd > > > branches on perf_event_is_tracing(event): > > > - Non-tracing events (hardware PMU, software events, breakpoints) call > > > perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT > > > (SEC("perf_event")). > > > - Tracepoint events (event->pmu == &perf_tracepoint) reject > > > BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require > > > BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to > > > event->tp_event->prog_array and executed in > > > perf_trace_run_bpf_submit() before perf_tp_event(), dropping the > > > sample if the BPF program returns 0. Furthermore, the > > > bpf_cast_to_kern_ctx() kfunc is only permitted in > > > BPF_PROG_TYPE_PERF_EVENT programs. > > > > > > Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in > > > sample_filter.bpf.c to evaluate context-independent filter terms (uid, > > > gid, tid, cpu, time, cgroup) using BPF helpers without calling > > > bpf_cast_to_kern_ctx(). When preparing a BPF filter on a > > > PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of > > > perf_sample_filter and attach it once to the shared trace_event_call. > > > > > > Before: > > > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true > > > libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument > > > Failed to attach perf sample-filter program > > > failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument) > > > > > > After: > > > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true > > > [ perf record: Woken up 1 times to write data ] > > > [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ] > > > > > > Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper") > > > Signed-off-by: Ian Rogers <irogers@google.com> > > > --- [SNIP] > > > @@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void) > > > goto err; > > > } > > > > > > - ret = bpf_object__pin(skel->obj, path); > > > + ret = pin_filter_object(skel, path); > > > > Why is this change needed? > > Before this patch, sample_filter.bpf.c only contained a single BPF > program (perf_sample_filter), which was always loaded. This patch adds > a second program (perf_tp_sample_filter) to the same skeleton (and > patch 4 adds the lock-tracking programs to it as well), and uses > bpf_program__set_autoload() before loading the skeleton so we can > select only the program(s) needed for a given event. > > In perf_bpf_filter__pin() and perf_bpf_filter__unpin() (for 'perf > record --setup-filter pin|unpin'), only perf_sample_filter is used, > because the pinned filter mechanism shares a single program across > privileged sessions by looking up kctx->event->id via > bpf_cast_to_kern_ctx(), which is only available to SEC("perf_event") > programs. So autoload is disabled for perf_tp_sample_filter (and in > patch 4, for the lock-tracking programs) when pinning and unpinning. > > A consequence of disabling autoload on any program in the skeleton is > that libbpf's bpf_object__pin() and bpf_object__unpin() fail: > - bpf_object__pin() calls bpf_object__pin_maps() and > bpf_object__pin_programs(). > - While bpf_object__pin_maps() skips maps that have autocreate > disabled, bpf_object__pin_programs() and bpf_object__unpin_programs() > iterate over every program in skel->obj without checking whether > autoload was enabled. Looks like it needs a fix in the libbpf. > - When bpf_program__pin() or bpf_program__unpin() is called on a > program that wasn't loaded (prog->fd < 0), libbpf returns -EINVAL > ("prog 'perf_tp_sample_filter': can't pin/unpin program that wasn't > loaded"). > > pin_filter_object() and unpin_filter_object() avoid this by calling > bpf_object__pin_maps() / bpf_object__unpin_maps() for the maps and > explicitly pinning/unpinning only the loaded perf_sample_filter > program. Thanks for the explanation. It'd be nice if we can have it in a comment. Thanks, Namhyung ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers 2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers @ 2026-10-19 22:09 ` Ian Rogers 2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers ` (4 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF sample filtering. Also add rw_semaphore.count and struct rt_mutex_base to the minimal vmlinux.h and extend get_lock_owner() to support rt_mutex and check the mutex handoff flag. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +-------------- tools/perf/util/bpf_skel/lock_data.h | 320 ++++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + tools/perf/util/lock-contention.c | 18 + tools/perf/util/lock-contention.h | 13 + 6 files changed, 374 insertions(+), 274 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 778923c92f42..a031709a2cc2 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1482,33 +1482,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..0f2118006005 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -96,13 +82,6 @@ struct { __uint(max_entries, MAX_ENTRIES); } task_data SEC(".maps"); -struct { - __uint(type, BPF_MAP_TYPE_HASH); - __uint(key_size, sizeof(__u64)); - __uint(value_size, sizeof(__u32)); - __uint(max_entries, MAX_ENTRIES); -} lock_syms SEC(".maps"); - struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(__u32)); @@ -138,6 +117,13 @@ struct { __uint(max_entries, 1); } cgroup_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, MAX_ENTRIES); +} lock_syms SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(long)); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -202,6 +170,11 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + int enabled; int perf_subsys_id = -1; @@ -222,8 +195,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,145 +800,24 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - -const volatile __u64 contig_page_data_addr; -const volatile __u64 node_data_addr; -const volatile int nr_nodes; -const volatile int sizeof_zone; - -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") -int BPF_PROG(collect_lock_syms) +int BPF_PROG(end_timestamp) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - + end_ts = bpf_ktime_get_ns(); return 0; } SEC("raw_tp/bpf_test_finish") -int BPF_PROG(end_timestamp) +int BPF_PROG(collect_lock_syms) { - end_ts = bpf_ktime_get_ns(); - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..43043d62ea9f 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,316 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct mutex___non_rt { + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t count; + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex___non_rt *mutex = (void *)lock; + + if (bpf_core_field_exists(mutex->owner.counter)) { + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (bpf_core_type_exists(struct rt_mutex_base)) { + struct rt_mutex_base *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *old = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(old, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + struct rw_semaphore___new *new = (void *)lock; + + owner = BPF_CORE_READ(new, owner.counter); + } +#else + /* assume new struct */ + struct rw_semaphore___new *new = (void *)lock; + + if (bpf_core_field_exists(new->owner.counter)) + owner = BPF_CORE_READ(new, owner.counter); +#endif + } else if (flags == LCB_F_RT) { + struct rt_mutex_base *rtm = (void *)lock; + + if (bpf_core_type_exists(struct rt_mutex_base)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (node_data_addr && nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..a1bb78d6e29c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -22,6 +22,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers ` (2 preceding siblings ...) 2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-19 22:09 ` Ian Rogers 2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers ` (3 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported <lock> specifications match perf lock contention: - Global/static kernel lock symbols (e.g. lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF LRU hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. When filtering on a specific mutex symbol (*_mutex), rt-mutex symbol (*_rt_mutex), or write-locked mmap_lock (mmap_lock:W / lock_held_write(mmap_lock)), no fentry/fexit hooks are attached at all. Note that counting semaphores (down*() / up()) do not enforce same-task ownership in the kernel: when a semaphore is used as a cross-task completion/signaling mechanism (down() in one task woken by up() in another task or interrupt handler, which proposed kernel trackers like CONFIG_DEPT model as wait/event dependencies bounded by execution context rather than lock ownership), fexit/down and fentry/up will not pair on the same task ID. Semaphore hooks are therefore only enabled when 'semaphore' or a '*_sem' symbol (such as console_sem) is explicitly specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any unpaired cross-task entries are automatically evicted. Example usage: # perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> # Conflicts: # tools/perf/util/bpf-filter.c # tools/perf/util/bpf_skel/sample_filter.bpf.c --- tools/perf/Documentation/perf-record.txt | 58 +- tools/perf/util/bpf-filter.c | 1004 ++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_skel/sample-filter.h | 65 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 1015 ++++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2209 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..8496b20987a0 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,63 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)'). + <lock> can be a kernel lock symbol name (e.g. + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache + name prefixed with '&' (e.g. 'lock_held(&dentry)', + 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock', + 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex', + 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', + 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). + Both read/shared and write/exclusive tracking are supported for + 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock' + (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', + 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support + write/exclusive tracking only. Note that 'semaphore' tracking + ('down*()' / 'up()') assumes binary mutex-like usage where the + acquiring task also releases the semaphore (such as 'console_sem'), + rather than cross-task completion signaling. + + For example, 'lock_held()' can be combined with memory allocation or + reclaim events to detect allocations or direct memory reclaim occurring + while a kernel lock is held (which can stall lock waiters for extended + periods): + + # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e kmem:mm_page_alloc \ + --filter 'gfp_flags & 0x400' \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e major-faults \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + + When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol + ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' + or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's + 'owner' field directly in kernel memory on each sample without attaching + any BPF fentry/fexit hooks. For locks that do not maintain an + authoritative per-task owner pointer (such as spinlocks, rwlocks, + semaphores, reader-locked rwsems, and slab/type filters), it attaches + BPF fentry/fexit programs to kernel lock acquire and release functions + to track held locks in a BPF map. These BPF hooks run on every kernel + lock acquisition and release for the attached lock family, even when no + perf sample is being taken. To minimize this overhead, only the hooks + for the relevant lock family and mode are attached when known (for + example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions + and 'rq_lock' only hooks spinlock functions), whereas slab filters + ('&<slab>'), raw hex addresses, or symbols without a recognizable + lock-type suffix attach hooks across all lock families (including + high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 4e99af4764f1..ab9b58f6cb71 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> +#include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -123,7 +138,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * { const struct perf_sample_info *info; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -166,9 +182,844 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN)) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) +{ + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) != 0) + return; + + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +static bool btf_has_func(struct btf *btf, const char *name) +{ + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; +} + +static unsigned int supported_lock_prog_groups(struct btf *btf) +{ + unsigned int supported = 0; + + if (!btf) + return 0; + + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) && + btf_has_func(btf, "mutex_unlock")) + supported |= LOCK_PROG_MUTEX; + + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) && + btf_has_func(btf, "up_read")) + supported |= LOCK_PROG_RWSEM_READ; + + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) && + btf_has_func(btf, "up_write")) + supported |= LOCK_PROG_RWSEM_WRITE; + + /* + * Require both acquire and all unlock entry points in BTF so we never + * attach spinlock/rwlock acquire hooks when unlock functions are + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK). + */ + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) && + btf_has_func(btf, "_raw_spin_unlock") && + btf_has_func(btf, "_raw_spin_unlock_irq") && + btf_has_func(btf, "_raw_spin_unlock_irqrestore") && + btf_has_func(btf, "_raw_spin_unlock_bh")) + supported |= LOCK_PROG_SPIN; + + if (btf_has_func(btf, "_raw_read_lock") && + btf_has_func(btf, "_raw_read_unlock") && + btf_has_func(btf, "_raw_read_unlock_irq") && + btf_has_func(btf, "_raw_read_unlock_irqrestore") && + btf_has_func(btf, "_raw_read_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_READ; + + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) && + btf_has_func(btf, "_raw_write_unlock") && + btf_has_func(btf, "_raw_write_unlock_irq") && + btf_has_func(btf, "_raw_write_unlock_irqrestore") && + btf_has_func(btf, "_raw_write_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_WRITE; + + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) && + btf_has_func(btf, "rt_mutex_unlock")) + supported |= LOCK_PROG_RTMUTEX; + + if (btf_has_func(btf, "down") && btf_has_func(btf, "up")) + supported |= LOCK_PROG_SEM; + + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) + supported |= LOCK_PROG_PERCPU_RWSEM; + + return supported; +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, + unsigned int required_groups, + bool needs_lock_syms, bool needs_slab_iter, + bool is_tracepoint) +{ + unsigned int supported_groups = supported_lock_prog_groups(btf); + unsigned int enabled_progs = 0; + + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + + if ((required_groups & supported_groups) != required_groups || + (needed_groups && !(needed_groups & supported_groups))) { + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n"); + return -ENOTSUP; + } + + needed_groups &= supported_groups; + if (skel) + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ); + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode, groups = 0; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */ + if (has_amp) { + *needs_slab_iter = true; + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + /* + * Write ownership of mm->mmap_lock is inspected directly via + * is_rwsem_writer_owner() without tracing hooks; only reader + * tracking requires LOCK_PROG_RWSEM_READ. + */ + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { + *prog_groups |= groups; + *required_groups |= groups; + return; + } + + if (parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 || + strtailcmp(base, "mutex") == 0) { + /* Inspected directly via lock->owner; no tracing hooks needed. */ + return; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_RWSEM_WRITE; + } + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "sem") == 0) { + *prog_groups |= LOCK_PROG_SEM; + *required_groups |= LOCK_PROG_SEM; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWLOCK_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + } else { + *prog_groups |= LOCK_PROG_ALL; + } +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *required_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + required_groups, needs_lock_syms, + needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, required_groups, + needs_lock_syms, needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->part` + * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate + * the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`. + * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, + * `rwlock`, `percpu-rwsem`, `semaphore`, etc. + * 4. Hexadecimal lock address. + * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) { - int i = 0; + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (has_amp) { + if (find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + pr_err("Error: Cannot resolve slab cache '%s'\n", name); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->target = 0; + if (skel && (mode & PBF_LOCK_MODE_READ)) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->target = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->target = encoded_pair; + return 0; + } + + if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + unsigned int direct_flag = 0; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) + direct_flag = PBF_LOCK_DIRECT_RTMUTEX; + else if (strtailcmp(base, "mutex") == 0) + direct_flag = PBF_LOCK_DIRECT_MUTEX; + else if (strtailcmp(base, "rwsem") == 0 && + strtailcmp(base, "percpu_rwsem") != 0 && + strtailcmp(base, "pcpu_rwsem") != 0) + direct_flag = PBF_LOCK_DIRECT_RWSEM; + + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX || + direct_flag == PBF_LOCK_DIRECT_RTMUTEX || + strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0 || + strtailcmp(base, "spinlock") == 0 || + (strtailcmp(base, "sem") == 0 && + strtailcmp(base, "rwsem") != 0)) && + !(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + + entry->part = PBF_LOCK_ADDR | mode | direct_flag; + entry->target = addr; + if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX && + direct_flag != PBF_LOCK_DIRECT_RTMUTEX) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) +{ + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -182,6 +1033,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + entry[i].target = 0; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -195,6 +1052,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + entry[i].target = 0; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -434,11 +1297,14 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct bpf_program *prog; struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0, required_groups = 0; #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -450,21 +1316,25 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &required_groups, &needs_lock_syms, + &needs_slab_iter); if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; - if (is_tracepoint) { - pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); + if (has_lock_filter || is_tracepoint) { + pr_err("Error: BPF filter requires CAP_BPF capability\n"); ret = -EPERM; goto err; } + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -509,8 +1379,26 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, required_groups, + needs_lock_syms, needs_slab_iter, + is_tracepoint); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint); + } if (sample_filter_bpf__load(skel) < 0) { ret = -errno; @@ -518,6 +1406,37 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -560,11 +1479,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -579,6 +1500,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -586,7 +1522,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -638,7 +1574,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -649,6 +1585,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -669,14 +1627,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -688,6 +1643,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) { char *prog_path = NULL; @@ -740,8 +1703,9 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false, false); + /* pinned program will use pid-hash */ - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); @@ -836,7 +1800,7 @@ int perf_bpf_filter__unpin(void) goto err; } - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); + configure_lock_progs(skel, NULL, 0, 0, false, false, false); if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..6ee6edef13f7 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -11,6 +11,7 @@ #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..215998897243 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,14 +53,79 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the + * specified kernel lock is currently held by the running task/CPU, or + * 0 otherwise. The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`) and the resolved target address, + * class ID, or type flag encoding is stored in `entry.target`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.target` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.target` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.target` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner) + * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner) + * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base owner) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY +#define PBF_LOCK_DIRECT_MUTEX (1U << 6) +#define PBF_LOCK_DIRECT_RWSEM (1U << 7) +#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8) + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; __u32 part; /* sub-sample type info when it has multiple values */ enum perf_bpf_filter_term term; __u64 value; + __u64 target; /* lock address, kmem_cache *, class, or type flags */ }; struct idx_hash_key { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index 576fbed270e5..31bce5e55b62 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,226 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* + * Currently held kernel locks tracked by fentry/fexit hooks. + * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or + * non-owner rwsem is acquired in one task and released in another task or + * interrupt handler (cross-context signaling), any orphaned entry is + * automatically evicted rather than exhausting the map. + */ +struct { + __uint(type, BPF_MAP_TYPE_LRU_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +int track_rwsem_read; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr) +{ + struct rw_semaphore___new *rwsem = (void *)lock_addr; + + if (bpf_core_field_exists(rwsem->count.counter) && + !(BPF_CORE_READ(rwsem, count.counter) & 1UL)) + return false; + if (bpf_core_field_exists(rwsem->owner.counter) && + (BPF_CORE_READ(rwsem, owner.counter) & 1UL)) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr; +} + +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part) +{ + if (!curr) + return false; + + if ((part & PBF_LOCK_DIRECT_MUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr) + return true; + if ((part & PBF_LOCK_DIRECT_RWSEM) && + is_rwsem_writer_owner(lock_addr, curr)) + return true; + if ((part & PBF_LOCK_DIRECT_RTMUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr) + return true; + return false; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr; + __u32 pid; + __u32 cpu_id; + __u32 kind; + __u32 mode; + __u64 lock_addr; + + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->target. + */ + if (use_idx_hash || !lock_enabled) + return 0; + + curr = bpf_get_current_task(); + pid = (__u32)bpf_get_current_pid_tgid(); + cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + kind = entry->part & PBF_LOCK_KIND_MASK; + mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->target; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr, entry->part)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return 0; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_rwsem_writer_owner(lock_addr, curr)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return 0; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id); + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id); + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + + default: + return 0; + } +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +430,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -209,6 +466,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); default: return 0; } @@ -380,4 +639,760 @@ int perf_tp_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +#define LCB_F_READ_IRQSAVE (1U << 30) +#define LCB_F_READ_IRQ_BH (1U << 31) + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, type_flag); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && type_flag == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* + * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC + * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call + * the out-of-line _raw_read_lock_irqsave(). Clear the marker + * and skip the duplicate outer fexit increment. + */ + if ((flags & LCB_F_READ_IRQ_BH) && + (existing->flags & LCB_F_READ_IRQSAVE)) { + existing->flags &= ~LCB_F_READ_IRQSAVE; + return 0; + } + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) { + __sync_fetch_and_add(&existing->read_count, 1); + if (flags & LCB_F_READ_IRQSAVE) + existing->flags |= LCB_F_READ_IRQSAVE; + else + existing->flags &= ~LCB_F_READ_IRQSAVE; + } else { + __sync_fetch_and_add(&existing->write_count, 1); + } + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE), + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, type_flag, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count == 0) + return 0; + if (existing->read_count == 1 && existing->write_count == 0) { + bpf_map_delete_elem(&held_locks, &addr_key); + } else { + existing->flags &= ~LCB_F_READ_IRQSAVE; + __sync_fetch_and_add(&existing->read_count, -1); + } + } else { + if (existing->write_count == 0) + return 0; + if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/_mutex_lock_nest_lock") +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_lock_killable") +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass, + void *nest_lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_trylock_nest_lock") +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/_down_write_nest_lock") +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + if (track_rwsem_read) + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nest_lock") +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/_rt_mutex_lock_nest_lock") +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 5/7] perf test: Add tests for lock_held() BPF sample filter 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers ` (3 preceding siblings ...) 2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-19 22:09 ` Ian Rogers 2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers ` (2 subsequent siblings) 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 +++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 122 ++++++++++++++++++++ 2 files changed, 230 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..ebaf8ee41622 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,124 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed ${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&dentry)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &dentry]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required" + then + echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]" + err=1 + return + fi + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock:R) || lock_held(mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock:R/:W]" + err=1 + return + fi + + if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc + then + if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]" + err=1 + return + fi + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +316,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers ` (4 preceding siblings ...) 2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-19 22:09 ` Ian Rogers 2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> # Conflicts: # tools/perf/util/bpf-filter.c # tools/perf/util/bpf_skel/sample_filter.bpf.c # Conflicts: # tools/perf/util/bpf_skel/sample_filter.bpf.c # Conflicts: # tools/perf/util/bpf-filter.c # Conflicts: # tools/perf/Documentation/perf-record.txt # tools/perf/util/bpf-filter.c --- tools/perf/Documentation/perf-record.txt | 44 +++-- tools/perf/tests/parse-events.c | 13 ++ tools/perf/tests/shell/record_bpf_filter.sh | 20 ++ tools/perf/util/bpf-filter.c | 194 +++++++++++++++++-- tools/perf/util/bpf_skel/lock_data.h | 44 ++++- tools/perf/util/bpf_skel/sample-filter.h | 12 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 ++++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 438 insertions(+), 34 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 8496b20987a0..3eaebdc9efc2 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -219,11 +219,17 @@ OPTIONS 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock', - 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex', - 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', - 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). - Both read/shared and write/exclusive tracking are supported for - 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock' + 'siglock', 'rq_lock', 'zone_lock', 'rcu', 'rcu-trace'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', 'rcu:W', 'semaphore'). Both + read/shared and write/exclusive tracking are supported for 'rwsem' + (including 'mmap_lock'), 'rwlock', 'rcu', and 'srcu' (where + read/shared mode matches 'rcu_read_lock()' / 'srcu_read_lock()' + critical sections and write/exclusive mode matches grace-period + waits 'synchronize_rcu*()', 'synchronize_srcu*()', 'rcu_barrier()', + and 'srcu_barrier()'); 'rcu-trace' ('rcu_read_lock_trace()') + supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'semaphore' tracking @@ -245,19 +251,21 @@ OPTIONS --filter 'lock_held(mutex) || lock_held(rwsem)' When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol - ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' - or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's - 'owner' field directly in kernel memory on each sample without attaching - any BPF fentry/fexit hooks. For locks that do not maintain an - authoritative per-task owner pointer (such as spinlocks, rwlocks, - semaphores, reader-locked rwsems, and slab/type filters), it attaches - BPF fentry/fexit programs to kernel lock acquire and release functions - to track held locks in a BPF map. These BPF hooks run on every kernel - lock acquisition and release for the attached lock family, even when no - perf sample is being taken. To minimize this overhead, only the hooks - for the relevant lock family and mode are attached when known (for - example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions - and 'rq_lock' only hooks spinlock functions), whereas slab filters + ('*_rt_mutex'), write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' + or 'lock_held(mmap_lock:W)'), or 'rcu:R' / 'rcu-trace' (which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'), + 'perf_sample_filter' inspects the kernel fields directly on each sample + without attaching any BPF fentry/fexit hooks. For locks that do not + maintain an authoritative per-task owner pointer (such as spinlocks, + rwlocks, semaphores, reader-locked rwsems, SRCU locks, RCU synchronize + waits, and slab/type filters), it attaches BPF fentry/fexit programs to + kernel lock acquire and release functions to track held locks in a BPF + map. These BPF hooks run on every kernel lock acquisition and release + for the attached lock family, even when no perf sample is being taken. + To minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(mmap_lock)' + only hooks rwsem read functions, 'srcu' only hooks SRCU functions, and + 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index ebaf8ee41622..47f697f53a85 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -247,6 +247,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index ab9b58f6cb71..887b5c406429 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -201,11 +201,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU & ~LOCK_PROG_SEM, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -325,6 +333,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -612,6 +646,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) supported |= LOCK_PROG_PERCPU_RWSEM; + if ((btf_has_func(btf, "__srcu_read_lock") || + btf_has_func(btf, "__srcu_read_lock_nmisafe")) && + (btf_has_func(btf, "__srcu_read_unlock") || + btf_has_func(btf, "__srcu_read_unlock_nmisafe"))) + supported |= LOCK_PROG_SRCU_READ; + + if (btf_has_func(btf, "synchronize_srcu")) + supported |= LOCK_PROG_SRCU_WRITE; + + if (btf_has_func(btf, "synchronize_rcu")) + supported |= LOCK_PROG_RCU_WRITE; + return supported; } @@ -731,6 +777,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RCU_WRITE; + *required_groups |= LOCK_PROG_RCU_WRITE; + } + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { *prog_groups |= groups; *required_groups |= groups; @@ -772,6 +831,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, groups |= LOCK_PROG_RWLOCK_WRITE; *prog_groups |= groups; *required_groups |= groups; + } else if (strtailcmp(base, "srcu") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_SRCU_WRITE; + *prog_groups |= groups; + *required_groups |= groups; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -855,6 +921,59 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, return false; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static bool btf_struct_has_field(const struct btf *btf, s32 type_id, + const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + u16 i, vlen; + + type_id = btf__resolve_type(btf, type_id); + if (type_id < 0) + return false; + + t = btf__type_by_id(btf, type_id); + if (!t || (!btf_is_struct(t) && !btf_is_union(t))) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && *name) { + if (!strcmp(name, field_name)) + return true; + } else if (btf_struct_has_field(btf, m->type, field_name)) { + return true; + } + } + return false; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + s32 id; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + return btf_struct_has_field(btf, id, field_name); +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->part` * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate @@ -862,15 +981,15 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, * * Resolution order: * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`). - * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock`. + * 2. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`. * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, - * `rwlock`, `percpu-rwsem`, `semaphore`, etc. + * `rwlock`, `percpu-rwsem`, `srcu`, `semaphore`, etc. * 4. Hexadecimal lock address. * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -948,6 +1067,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->target = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->target = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->target = 0; + return 0; + } + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->target = encoded_pair; @@ -1017,7 +1178,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, } static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, - struct perf_bpf_filter_entry *entry) + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -1035,7 +1196,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = expr->val; entry[i].target = 0; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -1054,7 +1215,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = group->val; entry[i].target = 0; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1329,7 +1490,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - ret = get_filter_entries(evsel, NULL, entry); + ret = get_filter_entries(evsel, NULL, NULL, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; @@ -1391,6 +1552,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0) + rcu_addr += 1; + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, needs_lock_syms, needs_slab_iter, is_tracepoint); @@ -1409,7 +1581,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (has_lock_filter && needs_slab_iter) run_slab_cache_iter(skel); - ret = get_filter_entries(evsel, skel, entry); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 43043d62ea9f..136ffbffbfff 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -237,6 +240,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > 0; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -269,6 +306,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 215998897243..f6a030abfb0d 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.target`, + * where `entry.target` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > 0`. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index 31bce5e55b62..a94b493ee6b2 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -129,6 +129,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -142,6 +143,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; int track_rwsem_read; __u8 lock_type_filter[LCB_F_TYPE_MAX]; @@ -297,6 +299,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->target)) + return 1; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return 1; + return 0; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr)) + return 1; + return 0; + default: return 0; } @@ -718,11 +734,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, type_flag); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1382,6 +1399,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index a1bb78d6e29c..b45b075f3346 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers ` (5 preceding siblings ...) 2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-19 22:09 ` Ian Rogers 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 7 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 171 ++++++++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 221 ++++++++++++++---- tools/perf/util/machine.c | 11 + 5 files changed, 362 insertions(+), 62 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..c6d37c910d4b 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..dfb384f1c5e6 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,7 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +181,133 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1; + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +315,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +329,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +379,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +403,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +439,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +456,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +639,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +747,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0f2118006005..26936f329dba 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -172,6 +185,7 @@ const volatile int aggr_mode; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -559,8 +573,21 @@ int contention_begin(u64 *ctx) } } skip_owner: - pelem->stack_id = bpf_get_stackid(ctx, &stacks, - BPF_F_FAST_STACK_CMP | stack_skip); + { + int skip = stack_skip; + + /* + * Unlike tp_btf/contention_begin (which fires inside a + * lock slowpath via trace_contention_begin), fentry + * RCU/SRCU programs attach directly to the top-level + * wait API, which is 2 frames shallower. + */ + if (flags & (LCB_F_RCU | LCB_F_SRCU)) + skip = stack_skip > 2 ? stack_skip - 2 : 0; + + pelem->stack_id = bpf_get_stackid(ctx, &stacks, + BPF_F_FAST_STACK_CMP | skip); + } if (pelem->stack_id < 0) __sync_fetch_and_add(&stack_fail, 1); } else if (aggr_mode == LOCK_AGGR_TASK) { @@ -588,8 +615,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -597,30 +623,34 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; /* - * For spinlock and rwlock, it needs to get the timestamp for the - * per-cpu map. However, contention_end does not have the flags - * so it cannot know whether it reads percpu or hash map. - * - * Try per-cpu map first and check if there's active contention. - * If it is, do not read hash map because it cannot go to sleeping - * locks before releasing the spinning locks. + * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu + * map. For tp_btf/contention_end, try the per-cpu map first (used by + * spinlocks and rwlocks) and fall back to the per-task tstamp map. */ - pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); - if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) - return 0; - } else { + if (is_rcu) { pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) return 0; - need_delete = true; + ts_map = &tstamp_rcu; + } else { + pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); + if (pelem && pelem->lock) { + if (pelem->lock != lock) + return 0; + } else { + pid = bpf_get_current_pid_tgid(); + pelem = bpf_map_lookup_elem(&tstamp, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp; + } } timestamp = bpf_ktime_get_ns(); @@ -648,7 +678,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -713,7 +742,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -795,11 +824,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(end_timestamp) { diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index 1d9d3bf57720..0a0f223b3335 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr) return true; } + { + struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL); + + if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) || + !strncmp(sym->name, "synchronize_srcu", 16) || + !strcmp(sym->name, "__synchronize_srcu") || + !strcmp(sym->name, "rcu_barrier") || + !strcmp(sym->name, "srcu_barrier"))) + return true; + } + return false; } -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers ` (6 preceding siblings ...) 2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers ` (8 more replies) 7 siblings, 9 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu This series adds a new BPF sample filter predicate, `lock_held(<lock>)` (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` mode suffixes), to `perf record --filter`, allowing perf event samples to be recorded only when a specified kernel lock is held (or not held with `!`) by the running kernel thread or CPU. It also fixes a pre-existing `lock_stat` filtering bug in `perf lock contention`, fixes a `bpf_link` memory leak in `perf_bpf_filter__prepare()`, enables BPF sample filters on tracepoint events (`PERF_TYPE_TRACEPOINT`), and extends both `lock_held()` and `perf lock contention --use-bpf` to support RCU and SRCU. For example: # perf record -a -g -e cycles --filter 'lock_held(cgroup_mutex)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' -- ... # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... # perf lock contention -a -b -Y rcu,srcu -- sleep 1 An important use-case is detecting memory allocations or direct memory reclaim occurring while kernel locks are held, which can stall lock waiters for extended periods under memory pressure: # Direct memory reclaim triggered while holding any mutex or rwsem: # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM) # while holding a mutex or rwsem: # perf record -a -g -e kmem:mm_page_alloc \ --filter 'gfp_flags & 0x400' \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Major page faults occurring while holding a mutex or rwsem: # perf record -a -g -e major-faults \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... Supported `<lock>` specifications match `perf lock contention`: - Global/static kernel lock symbols (e.g. `lock_held(cgroup_mutex)`) - Hexadecimal kernel lock addresses - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, `lock_held_read(&dentry)`) - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, `rcu:W`, `semaphore` Limitations and weaknesses in determining whether a lock is held: - Inlined lock/unlock operations: - When the kernel is configured to inline spinlock or rwlock operations (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and have no BTF function entry points for `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that both acquire and release entry points exist in BTF before enabling a lock family. - Fast-path `percpu_down_read()` / `percpu_up_read()` and `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined per-CPU counter increments without function calls, so only `percpu_down_write()` / `percpu_up_write()` and out-of-line `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to `preempt_disable()` and does not maintain `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` read-side inspection requires `CONFIG_PREEMPT_RCU`. - Lack of an inspectable owner field on non-exclusive / spinning locks: - While `struct mutex`, `struct rt_mutex`, and write-locked `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` (allowing `perf_sample_filter` to inspect kernel memory directly on sample overflow when filtering by lock address without attaching any `fentry`/`fexit` hooks for `*_mutex`, `*_rt_mutex`, and `mmap_lock:W`), spinlocks, rwlocks, semaphores, SRCU, and reader-locked `rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task owner pointer. - Tracking these locks (as well as slab-cache and lock-type filters) relies on `fexit` (acquire) and `fentry` (release) BPF hooks maintaining the `held_locks` BPF LRU hash map: - Any such lock already held before `perf record` attaches its BPF programs will not be present in `held_locks` until it is released and re-acquired. - The attached `fentry`/`fexit` hooks execute on every kernel lock acquire and release for the hooked lock family across the system, adding tracing overhead outside of perf sample handling. - Cross-task semaphore signaling (`down*()` / `up()`): - Counting `struct semaphore`s do not enforce same-task ownership in the kernel: when a semaphore is used as a completion/signaling mechanism (`down()` in one task woken by `up()` in another task or interrupt handler, which proposed kernel dependency trackers like `CONFIG_DEPT` model as wait/event dependencies bounded by execution context rather than lock ownership), `fexit/down` and `fentry/up` do not pair on the same task ID. Semaphore hooks are therefore only enabled when `semaphore` or a `*_sem` symbol (such as `console_sem`) is explicitly specified, and `held_locks` uses `BPF_MAP_TYPE_LRU_HASH` so any unpaired cross-task entries are automatically evicted. Patch overview (patches 1-3 are standalone fixes independent of the rest of the series): 1. "perf lock contention: Filter events before creating lock_stat": Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to apply `-Y`, `-L`, and `-S` filters before calling `lock_stat_findnew()`. 2. "perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF": Attaches `perf_sample_filter` directly via `PERF_EVENT_IOC_SET_BPF` in `perf_bpf_filter__prepare()`, matching the pinned filter path and avoiding leaking `struct bpf_link_perf` or double-closing `FD(evsel, x, y)`. 3. "perf bpf-filter: Support BPF sample filters on tracepoint events": Adds `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF sample filters can attach to `PERF_TYPE_TRACEPOINT` events (where `__perf_event_set_bpf_prog()` in `kernel/events/core.c` requires `BPF_PROG_TYPE_TRACEPOINT` rather than `BPF_PROG_TYPE_PERF_EVENT`). 4. "perf lock contention: Share common BPF and lock type definitions in lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`, `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`, `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`. 5. "perf bpf-filter: Add lock_held() sample filter": Implements `lock_held()`, `lock_held_read()`/`lock_held_shared()`, `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode suffixes using direct lock owner inspection where possible (`mutex`, `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF LRU hash map updated by selectively attached `fentry`/`fexit` hooks. 6. "perf test: Add tests for lock_held() BPF sample filter": Adds unit tests in `tests/parse-events.c` and shell tests in `tests/shell/record_bpf_filter.sh`. 7. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU": Supports `rcu` (`rcu:R` via direct `current->rcu_read_lock_nesting` inspection with zero tracing hooks; `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` via `synchronize_srcu*()` / `srcu_barrier()`). 8. "perf lock contention: Extend BPF lock contention to RCU and SRCU": Extends `perf lock contention --use-bpf` to track RCU and SRCU grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). Changes in v5: - Add patch 2 ("perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF") attaching BPF sample filters via `ioctl(..., PERF_EVENT_IOC_SET_BPF, fd)` instead of `bpf_program__attach_perf_event_opts()`, fixing the `bpf_link` memory leak without double-closing `FD(evsel, x, y)`, and use the same attachment in patch 3 for tracepoint sample filters (Namhyung). - Expand patch 3's commit message to detail why `pin_filter_object()` and `unpin_filter_object()` replace `bpf_object__pin()` and `bpf_object__unpin()` when `autoload` is disabled on `perf_tp_sample_filter` (Namhyung). - Store the resolved lock target in `perf_bpf_filter_entry.value` (with `op` normalized to `PBF_OP_EQ` or `PBF_OP_NEQ`), keeping `struct perf_bpf_filter_entry` unchanged at 24 bytes for backwards compatibility with pinned `filters` maps. - Add a CO-RE `struct rt_mutex___old` fallback in `get_lock_owner()` in `lock_data.h` for kernels prior to v5.15, and pass baseline nesting depth to `is_rcu_trace_lock_held()` for syscall tracepoints. Changes in v4: - Move the two bugfix commits (`Filter events before creating lock_stat` and `Support BPF sample filters on tracepoint events`) to patches 1 and 2 at the start of the series for easier cherry-picking. - Reserve leading `&` strictly for slab cache locks (e.g. `lock_held(&dentry)`) so global/static lock symbols and well-known locks do not use `&` (e.g. `lock_held(cgroup_mutex)`, `lock_held_read(mmap_lock)`), matching `perf lock contention` (Namhyung). - Skip attaching `fentry`/`fexit` hooks when filtering on a specific `mutex` symbol (`*_mutex`), `rt_mutex` symbol (`*_rt_mutex`), or write-locked `mmap_lock` (`mmap_lock:W` / `lock_held_write(mmap_lock)`), evaluating them purely via direct `owner` inspection (Namhyung). - Use a CO-RE flavor struct `struct mutex___non_rt` in `get_lock_owner()` in `lock_data.h` so `GEN_VMLINUX_H=1` builds succeed on `CONFIG_PREEMPT_RT=y` kernels. - Switch `held_locks` to `BPF_MAP_TYPE_LRU_HASH`, exclude `LOCK_PROG_SEM` from `LOCK_PROG_ALL` unless `semaphore` or a `*_sem` symbol is explicitly requested, and document cross-task semaphore signaling limitations (and comparison with `CONFIG_DEPT`). - Skip `LCB_F_MUTEX | LCB_F_SPIN` in `parse_lock_type_name()`, reject read-only mode on exclusive-only lock symbol suffixes in `resolve_lock_expr()`, and check `needed_groups & supported_groups` before masking `needed_groups` in `configure_lock_progs()`. - Recursively traverse anonymous structs/unions in `btf_has_task_field()` so `lock_held(rcu)` and `lock_held(rcu-trace)` work on `CONFIG_RANDSTRUCT` kernels. Changes in v3: - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c` so `perf lock contention -b -o` still reports reader owners when a writer contends on a reader-owned `rw_semaphore`, and scope `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`, `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`). - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()` when attaching, and document memory allocation / direct reclaim filtering with `lock_held()`. - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data` cannot be resolved and check `node_data_addr != 0` in `do_collect_zone_lock()`. - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call `_raw_read_lock_irqsave()`. - Offset `rcu_state_addr` by 1 byte so it is distinct from `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match timestamp maps. Changes in v2: - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to `sample_filter.bpf.c` and `lock_progs[]`. - Verify in `supported_lock_prog_groups()` that both acquire and all unlock BTF entry points exist for a lock family before enabling its tracing hooks, preventing acquire-only attachment when spinlock/rwlock unlock functions are inlined. - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` on `track_rwsem_read` and verify non-zero read/write counts in `record_lock_released_addr()` before decrementing aggregate slab/type counts. - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in `lock_contention.bpf.c` and recognize `synchronize_rcu*`, `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in `machine__is_lock_function()`. - Add patch fixing `report_lock_contention_begin_event()` to check type, address, and callstack filters before creating `lock_stat` entries. Assisted-by: Antigravity:gemini-3.1-pro Ian Rogers (8): perf lock contention: Filter events before creating lock_stat perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF perf bpf-filter: Support BPF sample filters on tracepoint events perf lock contention: Share common BPF and lock type definitions in lock_data.h perf bpf-filter: Add lock_held() sample filter perf test: Add tests for lock_held() BPF sample filter perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU perf lock contention: Extend BPF lock contention to RCU and SRCU tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/Documentation/perf-record.txt | 66 +- tools/perf/builtin-lock.c | 88 +- tools/perf/tests/parse-events.c | 121 ++ tools/perf/tests/shell/lock_contention.sh | 14 + tools/perf/tests/shell/record_bpf_filter.sh | 142 ++ tools/perf/util/bpf-filter.c | 1417 ++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 47 +- tools/perf/util/bpf_lock_contention.c | 171 +- .../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++--- tools/perf/util/bpf_skel/lock_data.h | 385 ++++- tools/perf/util/bpf_skel/sample-filter.h | 80 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 1241 +++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + tools/perf/util/lock-contention.c | 23 + tools/perf/util/lock-contention.h | 15 + tools/perf/util/machine.c | 11 + tools/perf/util/parse-events.c | 3 +- 20 files changed, 3937 insertions(+), 393 deletions(-) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers ` (7 subsequent siblings) 8 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu In report_lock_contention_begin_event(), lock_stat_findnew() was called before checking filters.nr_types (-Y), filters.nr_addrs (-L), and match_callstack_filter() (-S). When a filtered-out lock event (such as a mutex contention when running with -Y spinlock) shared the same aggregation key as a subsequent matching event (for example, contention on the mutex's internal wait_lock spinlock from the same caller), the first event created the lock_stat entry with the filtered-out lock's flags and the second event reused that entry, causing non-spinlock entries to appear in -Y spinlock output. Check the type, address, and callstack filters before looking up or creating the lock_stat entry. Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..778923c92f42 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) struct machine *machine = &session->machines.host; struct map *kmap; struct symbol *sym; + u64 *callstack = NULL; ret = get_key_by_aggr_mode(&key, addr, sample); if (ret < 0) @@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } } - ls = lock_stat_find(key); - if (!ls) { - char buf[128]; - const char *name = ""; - - switch (aggr_mode) { - case LOCK_AGGR_ADDR: - sym = machine__find_kernel_symbol(machine, key, &kmap); - if (sym) - name = sym->name; - break; - case LOCK_AGGR_CALLER: - name = buf; - if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) - name = "Unknown"; - break; - case LOCK_AGGR_CGROUP: - case LOCK_AGGR_TASK: - default: - break; - } - - ls = lock_stat_findnew(key, name, flags); - if (!ls) - return -ENOMEM; - } - if (filters.nr_types) { bool found = false; @@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } if (needs_callstack()) { - u64 *callstack = get_callstack(sample, max_stack_depth); + callstack = get_callstack(sample, max_stack_depth); if (callstack == NULL) return 0; @@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) free(callstack); return 0; } + } + + ls = lock_stat_find(key); + if (!ls) { + char buf[128]; + const char *name = ""; + + switch (aggr_mode) { + case LOCK_AGGR_ADDR: + sym = machine__find_kernel_symbol(machine, key, &kmap); + if (sym) + name = sym->name; + break; + case LOCK_AGGR_CALLER: + name = buf; + if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) + name = "Unknown"; + break; + case LOCK_AGGR_CGROUP: + case LOCK_AGGR_TASK: + default: + break; + } + + ls = lock_stat_findnew(key, name, flags); + if (!ls) { + free(callstack); + return -ENOMEM; + } + } + if (callstack) { if (ls->callstack == NULL) ls->callstack = callstack; else -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers ` (6 subsequent siblings) 8 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu In perf_bpf_filter__prepare(), the unpinned filter path attaches skel->progs.perf_sample_filter to each FD(evsel, x, y) using bpf_program__attach_perf_event_opts() and ignores the returned struct bpf_link *, leaking the allocated struct bpf_link_perf and its BPF_PERF_EVENT link fd. Furthermore, bpf_program__attach_perf_event_opts() is designed for callers that transfer ownership of the perf_event fd to the returned bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach() unconditionally disables and closes perf_link->perf_event_fd. Because evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(), destroying the returned bpf_link would double-close the perf_event file descriptors. Fix this by attaching the program directly with ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd), matching the pinned filter path right above it in perf_bpf_filter__prepare(). In the kernel, PERF_EVENT_IOC_SET_BPF takes a reference on the BPF program and _free_event() -> perf_event_free_bpf_prog() automatically detaches and releases it when the perf_event fd is closed. This also avoids enabling the event prematurely, allowing pe_opts (.dont_enable = true) and the LIBBPF_CURRENT_VERSION_GEQ(1, 7) guard to be dropped. Before (built with EXTRA_CFLAGS="-fsanitize=address"): # perf record -e cycles --filter 'period > 1000' -- true [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.024 MB perf.data ] ================================================================= ==51328==ERROR: LeakSanitizer: detected memory leaks Direct leak of 1568 byte(s) in 28 object(s) allocated from: #0 0x7ffbe76ef24f in calloc (/lib64/libasan.so.8+0xef24f) #1 0x625df7 in bpf_program__attach_perf_event_opts (tools/perf/perf+0x625df7) #2 0xbeddc0 in perf_bpf_filter__prepare util/bpf-filter.c:507 #3 0x8f3def in evlist__apply_filters util/evlist.c:1421 #4 0x44bbcf in record__open tools/perf/builtin-record.c:1469 #5 0x453f9e in __cmd_record tools/perf/builtin-record.c:2594 #6 0x45dadb in cmd_record tools/perf/builtin-record.c:4427 #7 0x5df56b in run_builtin tools/perf/perf.c:383 #8 0x5dfa0e in handle_internal_command tools/perf/perf.c:436 #9 0x5dfddc in run_argv tools/perf/perf.c:480 #10 0x5e0333 in main tools/perf/perf.c:581 SUMMARY: AddressSanitizer: 1568 byte(s) leaked in 28 allocation(s). After (built with EXTRA_CFLAGS="-fsanitize=address"): # perf record -e cycles --filter 'period > 1000' -- true [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.024 MB perf.data ] Reported-by: Namhyung Kim <namhyung@kernel.org> Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/ Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/util/bpf-filter.c | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..1596b08fe4e6 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -423,16 +423,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; - struct bpf_program *prog; - struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); -#if LIBBPF_CURRENT_VERSION_GEQ(1, 7) - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, - .dont_enable = true); -#else - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); -#endif entry = calloc(MAX_FILTERS, sizeof(*entry)); if (entry == NULL) @@ -501,14 +493,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - prog = skel->progs.perf_sample_filter; + fd = bpf_program__fd(skel->progs.perf_sample_filter); for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), - &pe_opts); - if (IS_ERR(link)) { + ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd); + if (ret < 0) { + ret = -errno; pr_err("Failed to attach perf sample-filter program\n"); - ret = PTR_ERR(link); goto err; } } -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers ` (5 subsequent siblings) 8 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper") updated set_filter() to route non-tracepoint filter terms (such as uid and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead of the ftrace tracepoint predicate filter. However, in the kernel's __perf_event_set_bpf_prog() (kernel/events/core.c), attaching a BPF program to a perf_event fd branches on perf_event_is_tracing(event): - Non-tracing events (hardware PMU, software events, breakpoints) call perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT (SEC("perf_event")). - Tracepoint events (event->pmu == &perf_tracepoint) reject BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to event->tp_event->prog_array and executed in perf_trace_run_bpf_submit() before perf_tp_event(), dropping the sample if the BPF program returns 0. Furthermore, the bpf_cast_to_kern_ctx() kfunc is only permitted in BPF_PROG_TYPE_PERF_EVENT programs. Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in sample_filter.bpf.c to evaluate context-independent filter terms (uid, gid, tid, cpu, time, cgroup) using BPF helpers without calling bpf_cast_to_kern_ctx(). When preparing a BPF filter on a PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of perf_sample_filter and attach it once to the shared trace_event_call. Because returning 0 from a BPF_PROG_TYPE_TRACEPOINT program filters at the shared trace_event_call->prog_array prior to perf_tp_event(), tracepoint BPF filters are never pinned for unprivileged use and require CAP_BPF or root privileges. Because perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load perf_sample_filter and disable autoload on perf_tp_sample_filter, libbpf's bpf_object__pin() and bpf_object__unpin() can no longer be used directly (as bpf_object__pin_programs() and bpf_object__unpin_programs() iterate over all programs in the object without checking autoload and fail with -EINVAL when a program has not been loaded). Introduce pin_filter_object() and unpin_filter_object() to pin/unpin the maps and only the loaded perf_sample_filter program. Before: # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument Failed to attach perf sample-filter program failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument) After: # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ] Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/util/bpf-filter.c | 132 ++++++++++++++++--- tools/perf/util/bpf_skel/sample_filter.bpf.c | 95 +++++++++++++ 2 files changed, 208 insertions(+), 19 deletions(-) diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 1596b08fe4e6..ac0e7cf40f3a 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * { const struct perf_sample_info *info; - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) - return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU || + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP) + return 0; + if (expr->op != PBF_OP_GROUP_BEGIN) { + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n", + evsel__name(evsel), expr->term); + return -1; + } + } + + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) + return 0; + if (expr->op == PBF_OP_GROUP_BEGIN) { struct perf_bpf_filter_expr *group; @@ -425,6 +436,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) struct sample_filter_bpf *skel = NULL; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); entry = calloc(MAX_FILTERS, sizeof(*entry)); if (entry == NULL) @@ -436,7 +448,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - if (needs_idx_hash && geteuid() != 0) { + if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { + pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; /* The filters map is shared among other processes */ @@ -476,8 +494,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; @@ -493,14 +520,32 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - fd = bpf_program__fd(skel->progs.perf_sample_filter); - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { - ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd); - if (ret < 0) { - ret = -errno; - pr_err("Failed to attach perf sample-filter program\n"); - goto err; + if (is_tracepoint) { + /* + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the + * BPF program to event->tp_event->prog_array on the shared + * trace_event_call, which perf_trace_run_bpf_submit() executes + * for all CPUs and tasks before iterating the per-CPU perf + * events. Attach once on the first FD so tp_event->prog_array + * does not get duplicate copies for each per-CPU FD. + */ + fd = bpf_program__fd(skel->progs.perf_tp_sample_filter); + ret = ioctl(FD(evsel, 0, 0), PERF_EVENT_IOC_SET_BPF, fd); + if (ret < 0) { + ret = -errno; + pr_err("Failed to attach perf tracepoint sample-filter program\n"); + goto err; + } + } else { + fd = bpf_program__fd(skel->progs.perf_sample_filter); + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { + ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd); + if (ret < 0) { + ret = -errno; + pr_err("Failed to attach perf sample-filter program\n"); + goto err; + } } } } @@ -632,6 +677,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -646,6 +730,7 @@ int perf_bpf_filter__pin(void) } /* pinned program will use pid-hash */ + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); @@ -665,7 +750,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -674,8 +759,8 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); - ret = dir_fd; + ret = -errno; + unpin_filter_object(skel, path); goto err; } @@ -718,6 +803,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -731,13 +818,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -745,7 +839,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..a190374ed87a 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return 0; } +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry) +{ + switch (entry->term) { + case PBF_TERM_TID: + if (entry->part) + return bpf_get_current_pid_tgid() >> 32; + return (__u32)bpf_get_current_pid_tgid(); + case PBF_TERM_CPU: + return bpf_get_smp_processor_id(); + case PBF_TERM_TIME: + return bpf_ktime_get_ns(); + case PBF_TERM_CGROUP: + return bpf_get_current_cgroup_id(); + case PBF_TERM_UID: + return bpf_get_current_uid_gid() & 0xFFFFFFFF; + case PBF_TERM_GID: + return bpf_get_current_uid_gid() >> 32; + default: + return 0; + } +} + #define CHECK_RESULT(data, op, val) \ if (!(data op val)) { \ if (!in_group) \ @@ -295,4 +317,77 @@ int perf_sample_filter(void *ctx) return 0; } +/* + * BPF program to be called for tracepoint perf events. + * + * For PERF_TYPE_TRACEPOINT events, the kernel's __perf_event_set_bpf_prog() + * rejects BPF_PROG_TYPE_PERF_EVENT and only accepts BPF_PROG_TYPE_TRACEPOINT, + * attaching the program to the shared trace_event_call->prog_array executed + * by perf_trace_run_bpf_submit() prior to perf_tp_event(). Because returning 0 + * filters the event at trace_event_call->prog_array (affecting the + * trace_event_call globally while attached), tracepoint BPF filters are never + * pinned for unprivileged use and require CAP_BPF / root privileges. + */ +SEC("tracepoint") +int perf_tp_sample_filter(void *ctx) +{ + struct perf_bpf_filter_entry *entry; + __u64 sample_data; + int in_group = 0; + int group_result = 0; + int i, k = 0; + int *losts; + + entry = bpf_map_lookup_elem(&filters, &k); + if (entry == NULL) + goto drop; + + for (i = 0; i < MAX_FILTERS; i++) { + sample_data = perf_get_common_sample(&entry[i]); + + switch (entry[i].op) { + case PBF_OP_EQ: + CHECK_RESULT(sample_data, ==, entry[i].value) + break; + case PBF_OP_NEQ: + CHECK_RESULT(sample_data, !=, entry[i].value) + break; + case PBF_OP_GT: + CHECK_RESULT(sample_data, >, entry[i].value) + break; + case PBF_OP_GE: + CHECK_RESULT(sample_data, >=, entry[i].value) + break; + case PBF_OP_LT: + CHECK_RESULT(sample_data, <, entry[i].value) + break; + case PBF_OP_LE: + CHECK_RESULT(sample_data, <=, entry[i].value) + break; + case PBF_OP_AND: + CHECK_RESULT(sample_data, &, entry[i].value) + break; + case PBF_OP_GROUP_BEGIN: + in_group = 1; + group_result = 0; + break; + case PBF_OP_GROUP_END: + if (group_result == 0) + goto drop; + in_group = 0; + break; + case PBF_OP_DONE: + return 1; + } + } + return 1; + +drop: + losts = bpf_map_lookup_elem(&dropped, &k); + if (losts != NULL) + __sync_fetch_and_add(losts, 1); + + return 0; +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (2 preceding siblings ...) 2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 22:42 ` Namhyung Kim 2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers ` (4 subsequent siblings) 8 siblings, 1 reply; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF sample filtering. Also add rw_semaphore.count and struct rt_mutex_base to the minimal vmlinux.h and extend get_lock_owner() to support rt_mutex and check the mutex handoff flag. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +------------- tools/perf/util/bpf_skel/lock_data.h | 338 ++++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + tools/perf/util/lock-contention.c | 19 + tools/perf/util/lock-contention.h | 13 + 6 files changed, 393 insertions(+), 274 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 778923c92f42..a031709a2cc2 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1482,33 +1482,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..0f2118006005 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -96,13 +82,6 @@ struct { __uint(max_entries, MAX_ENTRIES); } task_data SEC(".maps"); -struct { - __uint(type, BPF_MAP_TYPE_HASH); - __uint(key_size, sizeof(__u64)); - __uint(value_size, sizeof(__u32)); - __uint(max_entries, MAX_ENTRIES); -} lock_syms SEC(".maps"); - struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(__u32)); @@ -138,6 +117,13 @@ struct { __uint(max_entries, 1); } cgroup_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, MAX_ENTRIES); +} lock_syms SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(long)); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -202,6 +170,11 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + int enabled; int perf_subsys_id = -1; @@ -222,8 +195,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,145 +800,24 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - -const volatile __u64 contig_page_data_addr; -const volatile __u64 node_data_addr; -const volatile int nr_nodes; -const volatile int sizeof_zone; - -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") -int BPF_PROG(collect_lock_syms) +int BPF_PROG(end_timestamp) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - + end_ts = bpf_ktime_get_ns(); return 0; } SEC("raw_tp/bpf_test_finish") -int BPF_PROG(end_timestamp) +int BPF_PROG(collect_lock_syms) { - end_ts = bpf_ktime_get_ns(); - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..317d8ccae63e 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,334 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct mutex___non_rt { + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct rt_mutex___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t count; + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex___non_rt *mutex = (void *)lock; + + if (bpf_core_field_exists(mutex->owner.counter)) { + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (bpf_core_type_exists(struct rt_mutex_base)) { + struct rt_mutex_base *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } else { + struct rt_mutex___old *rtm = (void *)lock; + + if (bpf_core_field_exists(rtm->owner)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *old = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(old, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + struct rw_semaphore___new *new = (void *)lock; + + owner = BPF_CORE_READ(new, owner.counter); + } +#else + /* assume new struct */ + struct rw_semaphore___new *new = (void *)lock; + + if (bpf_core_field_exists(new->owner.counter)) + owner = BPF_CORE_READ(new, owner.counter); +#endif + } else if (flags == LCB_F_RT) { + if (bpf_core_type_exists(struct rt_mutex_base)) { + struct rt_mutex_base *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } else { + struct rt_mutex___old *rtm = (void *)lock; + + if (bpf_core_field_exists(rtm->owner)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (node_data_addr && nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..f6e68d740860 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -6,6 +6,7 @@ #include "symbol.h" #include <limits.h> +#include <stdlib.h> #include <string.h> #include <linux/hash.h> @@ -22,6 +23,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-20 22:42 ` Namhyung Kim 0 siblings, 0 replies; 59+ messages in thread From: Namhyung Kim @ 2026-10-20 22:42 UTC (permalink / raw) To: Ian Rogers Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu On Tue, Oct 20, 2026 at 01:58:54PM -0700, Ian Rogers wrote: > Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers > (get_lock_owner, get_current_mmap_lock, get_current_siglock, > get_lock_addr, check_lock_type, do_collect_zone_lock, > do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into > tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF > sample filtering. This is fine. > > Also add rw_semaphore.count and struct rt_mutex_base to the minimal > vmlinux.h and extend get_lock_owner() to support rt_mutex and check > the mutex handoff flag. Please do this in a separate commit. I'd like to see this kind of commit only moves the existing code. > > Signed-off-by: Ian Rogers <irogers@google.com> > --- > tools/perf/builtin-lock.c | 27 -- > .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +------------- > tools/perf/util/bpf_skel/lock_data.h | 338 ++++++++++++++++++ > tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + > tools/perf/util/lock-contention.c | 19 + > tools/perf/util/lock-contention.h | 13 + > 6 files changed, 393 insertions(+), 274 deletions(-) > > diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c > index 778923c92f42..a031709a2cc2 100644 > --- a/tools/perf/builtin-lock.c > +++ b/tools/perf/builtin-lock.c > @@ -1482,33 +1482,6 @@ static void sort_result(void) > } > } > > -static const struct { > - unsigned int flags; > - /* > - * Name of the lock flags (access), with delimeter ':'. > - * For example, rwsem:R of rwsem:W. > - */ > - const char *flags_name; > - /* Name of the lock (type), for example, rwlock or rwsem. */ > - const char *lock_name; > -} lock_type_table[] = { > - { 0, "semaphore", "semaphore" }, > - { LCB_F_SPIN, "spinlock", "spinlock" }, > - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, > - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, > - { LCB_F_READ, "rwsem:R", "rwsem" }, > - { LCB_F_WRITE, "rwsem:W", "rwsem" }, > - { LCB_F_RT, "rt-mutex", "rt-mutex" }, > - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, > - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, > - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, > - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, > - { LCB_F_MUTEX, "mutex", "mutex" }, > - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, > - /* alias for optimistic spinning only */ > - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, > -}; > - > static const char *get_type_flags_name(unsigned int flags) > { > flags &= LCB_F_TYPE_MASK; > diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c > index 0d9c6f55050e..0f2118006005 100644 > --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c > +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c > @@ -8,23 +8,9 @@ > > #include "lock_data.h" > > -/* for collect_lock_syms(). 4096 was rejected by the verifier */ > -#define MAX_CPUS 1024 > - > -/* for collect_zone_lock(). It should be more than the actual zones. */ > -#define MAX_ZONES 10 > - > /* for do_lock_delay(). Arbitrarily set to 1 million. */ > #define MAX_LOOP (1U << 20) > > -/* lock contention flags from include/trace/events/lock.h */ > -#define LCB_F_SPIN (1U << 0) > -#define LCB_F_READ (1U << 1) > -#define LCB_F_WRITE (1U << 2) > -#define LCB_F_RT (1U << 3) > -#define LCB_F_PERCPU (1U << 4) > -#define LCB_F_MUTEX (1U << 5) > - > /* callstack storage */ > struct { > __uint(type, BPF_MAP_TYPE_STACK_TRACE); > @@ -96,13 +82,6 @@ struct { > __uint(max_entries, MAX_ENTRIES); > } task_data SEC(".maps"); > > -struct { > - __uint(type, BPF_MAP_TYPE_HASH); > - __uint(key_size, sizeof(__u64)); > - __uint(value_size, sizeof(__u32)); > - __uint(max_entries, MAX_ENTRIES); > -} lock_syms SEC(".maps"); > - > struct { > __uint(type, BPF_MAP_TYPE_HASH); > __uint(key_size, sizeof(__u32)); > @@ -138,6 +117,13 @@ struct { > __uint(max_entries, 1); > } cgroup_filter SEC(".maps"); > > +struct { > + __uint(type, BPF_MAP_TYPE_HASH); > + __uint(key_size, sizeof(__u64)); > + __uint(value_size, sizeof(__u32)); > + __uint(max_entries, MAX_ENTRIES); > +} lock_syms SEC(".maps"); > + Looks like un uncessasry change. > struct { > __uint(type, BPF_MAP_TYPE_HASH); > __uint(key_size, sizeof(long)); > @@ -159,22 +145,6 @@ struct { > __uint(max_entries, 1); > } lock_delays SEC(".maps"); > > -struct rw_semaphore___old { > - struct task_struct *owner; > -} __attribute__((preserve_access_index)); > - > -struct rw_semaphore___new { > - atomic_long_t owner; > -} __attribute__((preserve_access_index)); > - > -struct mm_struct___old { > - struct rw_semaphore mmap_sem; > -} __attribute__((preserve_access_index)); > - > -struct mm_struct___new { > - struct rw_semaphore mmap_lock; > -} __attribute__((preserve_access_index)); > - > struct cas_ctx { > struct contention_data *data; > u64 duration; > @@ -182,8 +152,6 @@ struct cas_ctx { [SNIP] > diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h > index 652e114e6b87..317d8ccae63e 100644 > --- a/tools/perf/util/bpf_skel/lock_data.h > +++ b/tools/perf/util/bpf_skel/lock_data.h > @@ -45,6 +45,14 @@ struct contention_task_data { > #define LCB_F_SLAB_ID_END (1U << 26) > #define LCB_F_SLAB_ID_MASK 0x03FF0000U > > +/* lock contention flags from include/trace/events/lock.h */ > +#define LCB_F_SPIN (1U << 0) > +#define LCB_F_READ (1U << 1) > +#define LCB_F_WRITE (1U << 2) > +#define LCB_F_RT (1U << 3) > +#define LCB_F_PERCPU (1U << 4) > +#define LCB_F_MUTEX (1U << 5) > + > #define LCB_F_TYPE_MAX (1U << 7) > #define LCB_F_TYPE_MASK 0x0000007FU > > @@ -76,4 +84,334 @@ struct slab_cache_data { > char name[SLAB_NAME_MAX]; > }; > > +#ifdef __BPF__ Why is this needed? Thanks, Namhyung ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (3 preceding siblings ...) 2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 22:49 ` Namhyung Kim 2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers ` (3 subsequent siblings) 8 siblings, 1 reply; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported <lock> specifications match perf lock contention: - Global/static kernel lock symbols (e.g. lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF LRU hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. When filtering on a specific mutex symbol (*_mutex), rt-mutex symbol (*_rt_mutex), or write-locked mmap_lock (mmap_lock:W / lock_held_write(mmap_lock)), no fentry/fexit hooks are attached at all. Note that counting semaphores (down*() / up()) do not enforce same-task ownership in the kernel: when a semaphore is used as a cross-task completion/signaling mechanism (down() in one task woken by up() in another task or interrupt handler, which proposed kernel trackers like CONFIG_DEPT model as wait/event dependencies bounded by execution context rather than lock ownership), fexit/down and fentry/up will not pair on the same task ID. Semaphore hooks are therefore only enabled when 'semaphore' or a '*_sem' symbol (such as console_sem) is explicitly specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any unpaired cross-task entries are automatically evicted. Example usage: # perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 58 +- tools/perf/util/bpf-filter.c | 1071 +++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 47 +- tools/perf/util/bpf_skel/sample-filter.h | 66 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 1014 +++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2277 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..8496b20987a0 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,63 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)'). + <lock> can be a kernel lock symbol name (e.g. + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache + name prefixed with '&' (e.g. 'lock_held(&dentry)', + 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock', + 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex', + 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', + 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). + Both read/shared and write/exclusive tracking are supported for + 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock' + (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', + 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support + write/exclusive tracking only. Note that 'semaphore' tracking + ('down*()' / 'up()') assumes binary mutex-like usage where the + acquiring task also releases the semaphore (such as 'console_sem'), + rather than cross-task completion signaling. + + For example, 'lock_held()' can be combined with memory allocation or + reclaim events to detect allocations or direct memory reclaim occurring + while a kernel lock is held (which can stall lock waiters for extended + periods): + + # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e kmem:mm_page_alloc \ + --filter 'gfp_flags & 0x400' \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e major-faults \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + + When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol + ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' + or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's + 'owner' field directly in kernel memory on each sample without attaching + any BPF fentry/fexit hooks. For locks that do not maintain an + authoritative per-task owner pointer (such as spinlocks, rwlocks, + semaphores, reader-locked rwsems, and slab/type filters), it attaches + BPF fentry/fexit programs to kernel lock acquire and release functions + to track held locks in a BPF map. These BPF hooks run on every kernel + lock acquisition and release for the attached lock family, even when no + perf sample is being taken. To minimize this overhead, only the hooks + for the relevant lock family and mode are attached when known (for + example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions + and 'rq_lock' only hooks spinlock functions), whereas slab filters + ('&<slab>'), raw hex addresses, or symbols without a recognizable + lock-type suffix attach hooks across all lock families (including + high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index ac0e7cf40f3a..d5a85ee98d5f 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> +#include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -123,7 +138,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * { const struct perf_sample_info *info; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -166,9 +182,913 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN)) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) +{ + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) != 0) + return; + + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +static bool btf_has_func(struct btf *btf, const char *name) +{ + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; +} + +static unsigned int supported_lock_prog_groups(struct btf *btf) +{ + unsigned int supported = 0; + + if (!btf) + return 0; + + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) && + btf_has_func(btf, "mutex_unlock")) + supported |= LOCK_PROG_MUTEX; + + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) && + btf_has_func(btf, "up_read")) + supported |= LOCK_PROG_RWSEM_READ; + + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) && + btf_has_func(btf, "up_write")) + supported |= LOCK_PROG_RWSEM_WRITE; + + /* + * Require both acquire and all unlock entry points in BTF so we never + * attach spinlock/rwlock acquire hooks when unlock functions are + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK). + */ + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) && + btf_has_func(btf, "_raw_spin_unlock") && + btf_has_func(btf, "_raw_spin_unlock_irq") && + btf_has_func(btf, "_raw_spin_unlock_irqrestore") && + btf_has_func(btf, "_raw_spin_unlock_bh")) + supported |= LOCK_PROG_SPIN; + + if (btf_has_func(btf, "_raw_read_lock") && + btf_has_func(btf, "_raw_read_unlock") && + btf_has_func(btf, "_raw_read_unlock_irq") && + btf_has_func(btf, "_raw_read_unlock_irqrestore") && + btf_has_func(btf, "_raw_read_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_READ; + + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) && + btf_has_func(btf, "_raw_write_unlock") && + btf_has_func(btf, "_raw_write_unlock_irq") && + btf_has_func(btf, "_raw_write_unlock_irqrestore") && + btf_has_func(btf, "_raw_write_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_WRITE; + + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) && + btf_has_func(btf, "rt_mutex_unlock")) + supported |= LOCK_PROG_RTMUTEX; + + if (btf_has_func(btf, "down") && btf_has_func(btf, "up")) + supported |= LOCK_PROG_SEM; + + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) + supported |= LOCK_PROG_PERCPU_RWSEM; + + return supported; +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, + unsigned int required_groups, + bool needs_lock_syms, bool needs_slab_iter, + bool is_tracepoint) +{ + unsigned int supported_groups = supported_lock_prog_groups(btf); + unsigned int enabled_progs = 0; + + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + + if ((required_groups & supported_groups) != required_groups || + (needed_groups && !(needed_groups & supported_groups))) { + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n"); + return -ENOTSUP; + } + + needed_groups &= supported_groups; + if (skel) + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ); + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode, groups = 0; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */ + if (has_amp) { + *needs_slab_iter = true; + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + /* + * Write ownership of mm->mmap_lock is inspected directly via + * is_rwsem_writer_owner() without tracing hooks; only reader + * tracking requires LOCK_PROG_RWSEM_READ. + */ + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { + *prog_groups |= groups; + *required_groups |= groups; + return; + } + + if (parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 || + strtailcmp(base, "mutex") == 0) { + /* Inspected directly via lock->owner; no tracing hooks needed. */ + return; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_RWSEM_WRITE; + } + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "sem") == 0) { + *prog_groups |= LOCK_PROG_SEM; + *required_groups |= LOCK_PROG_SEM; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWLOCK_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + } else { + *prog_groups |= LOCK_PROG_ALL; + } +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *required_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + required_groups, needs_lock_syms, + needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, required_groups, + needs_lock_syms, needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr, + enum perf_bpf_filter_op *op_out) +{ + switch (expr->op) { + case PBF_OP_EQ: + if (expr->val == 1) { + *op_out = PBF_OP_EQ; + return 0; + } + if (expr->val == 0) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_NEQ: + if (expr->val == 0) { + *op_out = PBF_OP_EQ; + return 0; + } + if (expr->val == 1) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_GT: + if (expr->val == 0) { + *op_out = PBF_OP_EQ; + return 0; + } + break; + case PBF_OP_GE: + if (expr->val == 1) { + *op_out = PBF_OP_EQ; + return 0; + } + break; + case PBF_OP_LT: + if (expr->val == 1) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_LE: + if (expr->val == 0) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_AND: + if (expr->val & 1) { + *op_out = PBF_OP_EQ; + return 0; + } + break; + case PBF_OP_GROUP_BEGIN: + case PBF_OP_GROUP_END: + case PBF_OP_DONE: + default: + break; + } + pr_err("Error: Invalid comparison for boolean lock_held(%s) predicate\n", + expr->lock_name ?: ""); + return -EINVAL; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->op`, + * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`, + * and populate the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`. + * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, + * `rwlock`, `percpu-rwsem`, `semaphore`, etc. + * 4. Hexadecimal lock address. + * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) +{ + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + ret = normalize_lock_expr_op(expr, &entry->op); + if (ret < 0) + return ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (has_amp) { + if (find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->value = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + pr_err("Error: Cannot resolve slab cache '%s'\n", name); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->value = 0; + if (skel && (mode & PBF_LOCK_MODE_READ)) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->value = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->value = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->value = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->value = encoded_pair; + return 0; + } + + if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->value = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + unsigned int direct_flag = 0; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) + direct_flag = PBF_LOCK_DIRECT_RTMUTEX; + else if (strtailcmp(base, "mutex") == 0) + direct_flag = PBF_LOCK_DIRECT_MUTEX; + else if (strtailcmp(base, "rwsem") == 0 && + strtailcmp(base, "percpu_rwsem") != 0 && + strtailcmp(base, "pcpu_rwsem") != 0) + direct_flag = PBF_LOCK_DIRECT_RWSEM; + + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX || + direct_flag == PBF_LOCK_DIRECT_RTMUTEX || + strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0 || + strtailcmp(base, "spinlock") == 0 || + (strtailcmp(base, "sem") == 0 && + strtailcmp(base, "rwsem") != 0)) && + !(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + + entry->part = PBF_LOCK_ADDR | mode | direct_flag; + entry->value = addr; + if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX && + direct_flag != PBF_LOCK_DIRECT_RTMUTEX) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) { - int i = 0; + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -182,6 +1102,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -195,6 +1120,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -434,22 +1364,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0, required_groups = 0; entry = calloc(MAX_FILTERS, sizeof(*entry)); if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &required_groups, &needs_lock_syms, + &needs_slab_iter); - if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { - pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); + if ((has_lock_filter || is_tracepoint) && geteuid() != 0 && + !perf_cap__capable(CAP_BPF)) { + pr_err("Error: BPF filter requires CAP_BPF capability\n"); ret = -EPERM; goto err; } @@ -457,6 +1389,11 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -501,8 +1438,26 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, required_groups, + needs_lock_syms, needs_slab_iter, + is_tracepoint); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint); + } if (sample_filter_bpf__load(skel) < 0) { ret = -errno; @@ -510,6 +1465,37 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -549,11 +1535,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -568,6 +1556,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -575,7 +1578,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -627,7 +1630,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -638,6 +1641,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -658,14 +1683,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -677,6 +1699,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) { char *prog_path = NULL; @@ -729,8 +1759,9 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false, false); + /* pinned program will use pid-hash */ - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); @@ -825,7 +1856,7 @@ int perf_bpf_filter__unpin(void) goto err; } - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); + configure_lock_progs(skel, NULL, 0, 0, false, false, false); if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..5ae05d4608c6 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -8,9 +8,11 @@ #endif #include <stdio.h> +#include <stdlib.h> #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +33,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +47,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +117,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..a65ee780d64d 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,8 +53,74 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates whether the + * specified kernel lock is currently held by the running task/CPU. + * The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to + * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held), + * and the resolved target address, class ID, or type flag encoding is + * stored in `entry.value`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.value` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.value` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.value` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.value` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner) + * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner) + * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base / rt_mutex owner) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY +#define PBF_LOCK_DIRECT_MUTEX (1U << 6) +#define PBF_LOCK_DIRECT_RWSEM (1U << 7) +#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8) + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index a190374ed87a..d1192a93179e 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,225 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* + * Currently held kernel locks tracked by fentry/fexit hooks. + * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or + * non-owner rwsem is acquired in one task and released in another task or + * interrupt handler (cross-context signaling), any orphaned entry is + * automatically evicted rather than exhausting the map. + */ +struct { + __uint(type, BPF_MAP_TYPE_LRU_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +int track_rwsem_read; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr) +{ + struct rw_semaphore___new *rwsem = (void *)lock_addr; + + if (bpf_core_field_exists(rwsem->count.counter) && + !(BPF_CORE_READ(rwsem, count.counter) & 1UL)) + return false; + if (bpf_core_field_exists(rwsem->owner.counter) && + (BPF_CORE_READ(rwsem, owner.counter) & 1UL)) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr; +} + +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part) +{ + if (!curr) + return false; + + if ((part & PBF_LOCK_DIRECT_MUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr) + return true; + if ((part & PBF_LOCK_DIRECT_RWSEM) && + is_rwsem_writer_owner(lock_addr, curr)) + return true; + if ((part & PBF_LOCK_DIRECT_RTMUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr) + return true; + return false; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr = bpf_get_current_task(); + __u32 pid = (__u32)bpf_get_current_pid_tgid(); + __u32 cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + __u32 kind = entry->part & PBF_LOCK_KIND_MASK; + __u32 mode = entry->part & PBF_LOCK_MODE_MASK; + __u64 lock_addr; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->value; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr, entry->part)) + return true; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return false; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_rwsem_writer_owner(lock_addr, curr)) + return true; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return false; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0; + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->value, PBF_LOCK_SLAB, mode, pid, cpu_id) != 0; + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0; + + default: + return false; + } +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->value. + */ + if (use_idx_hash || !lock_enabled) + return entry->op == PBF_OP_EQ ? ~entry->value : entry->value; + + return is_lock_held(entry) ? entry->value : ~entry->value; +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +429,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -209,6 +465,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); default: return 0; } @@ -390,4 +648,760 @@ int perf_tp_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +#define LCB_F_READ_IRQSAVE (1U << 30) +#define LCB_F_READ_IRQ_BH (1U << 31) + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, type_flag); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && type_flag == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* + * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC + * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call + * the out-of-line _raw_read_lock_irqsave(). Clear the marker + * and skip the duplicate outer fexit increment. + */ + if ((flags & LCB_F_READ_IRQ_BH) && + (existing->flags & LCB_F_READ_IRQSAVE)) { + existing->flags &= ~LCB_F_READ_IRQSAVE; + return 0; + } + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) { + __sync_fetch_and_add(&existing->read_count, 1); + if (flags & LCB_F_READ_IRQSAVE) + existing->flags |= LCB_F_READ_IRQSAVE; + else + existing->flags &= ~LCB_F_READ_IRQSAVE; + } else { + __sync_fetch_and_add(&existing->write_count, 1); + } + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE), + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, type_flag, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count == 0) + return 0; + if (existing->read_count == 1 && existing->write_count == 0) { + bpf_map_delete_elem(&held_locks, &addr_key); + } else { + existing->flags &= ~LCB_F_READ_IRQSAVE; + __sync_fetch_and_add(&existing->read_count, -1); + } + } else { + if (existing->write_count == 0) + return 0; + if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/_mutex_lock_nest_lock") +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_lock_killable") +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass, + void *nest_lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_trylock_nest_lock") +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/_down_write_nest_lock") +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + if (track_rwsem_read) + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nest_lock") +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/_rt_mutex_lock_nest_lock") +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter 2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-20 22:49 ` Namhyung Kim 0 siblings, 0 replies; 59+ messages in thread From: Namhyung Kim @ 2026-10-20 22:49 UTC (permalink / raw) To: Ian Rogers Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu On Tue, Oct 20, 2026 at 01:58:55PM -0700, Ian Rogers wrote: > Add BPF sample filter predicates lock_held(<lock>), > lock_held_read(<lock>) / lock_held_shared(<lock>), and > lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' > and ':W' suffixes on <lock>) to allow recording perf event samples only > when a specified kernel lock is held (or not held with '!'). > > Supported <lock> specifications match perf lock contention: > - Global/static kernel lock symbols (e.g. lock_held(cgroup_mutex)) > - Hexadecimal kernel lock addresses > - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), > lock_held_read(&dentry)) > - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, > zone_lock > - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, > rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, > semaphore Can we do this step by step? I think you can add the core logic with, for example, global mutex first. Then add other lock types, semi- global, slab cache support and so on in the later patches. I believe that would help reviewers. Thanks, Namhyung > > Supported lock hold modes by lock family: > - Both read/shared and write/exclusive: rwsem (including mmap_lock) > and rwlock. > - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, > zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() > / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() > are inlined per-CPU counter operations without function entry points). > > When evaluating the filter on sample overflow, perf_sample_filter first > inspects the lock's owner field directly in kernel memory where > possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls > back to a held_locks BPF LRU hash map (tracking separate read_count and > write_count) populated by fentry/fexit hooks on kernel lock/unlock > functions. When filtering on a specific mutex symbol (*_mutex), > rt-mutex symbol (*_rt_mutex), or write-locked mmap_lock (mmap_lock:W / > lock_held_write(mmap_lock)), no fentry/fexit hooks are attached at all. > > Note that counting semaphores (down*() / up()) do not enforce same-task > ownership in the kernel: when a semaphore is used as a cross-task > completion/signaling mechanism (down() in one task woken by up() in > another task or interrupt handler, which proposed kernel trackers like > CONFIG_DEPT model as wait/event dependencies bounded by execution > context rather than lock ownership), fexit/down and fentry/up will not > pair on the same task ID. Semaphore hooks are therefore only enabled > when 'semaphore' or a '*_sem' symbol (such as console_sem) is > explicitly specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any > unpaired cross-task entries are automatically evicted. > > Example usage: > > # perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \ > -- perf bench mem mmap -t 2 -l 100 > [ perf record: Woken up 1 times to write data ] > [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] > > # perf report --stdio --no-children > # Overhead Command Shared Object Symbol > # ........ ........ ................. ........................ > # > 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms > | > ---clear_page_erms > get_page_from_freelist > __alloc_pages_noprof > folio_alloc_mpol_noprof > vma_alloc_folio_noprof > do_anonymous_page > __handle_mm_fault > handle_mm_fault > do_user_addr_fault > exc_page_fault > asm_exc_page_fault > worker_thread > > Read/shared vs write/exclusive holding can also be filtered separately: > > # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ... > # perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ... > # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ... > > Signed-off-by: Ian Rogers <irogers@google.com> > --- > tools/perf/Documentation/perf-record.txt | 58 +- > tools/perf/util/bpf-filter.c | 1071 +++++++++++++++++- > tools/perf/util/bpf-filter.h | 13 + > tools/perf/util/bpf-filter.l | 31 +- > tools/perf/util/bpf-filter.y | 47 +- > tools/perf/util/bpf_skel/sample-filter.h | 66 ++ > tools/perf/util/bpf_skel/sample_filter.bpf.c | 1014 +++++++++++++++++ > tools/perf/util/parse-events.c | 3 +- > 8 files changed, 2277 insertions(+), 26 deletions(-) > > diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt > index 2351f7b4f5fc..8496b20987a0 100644 > --- a/tools/perf/Documentation/perf-record.txt > +++ b/tools/perf/Documentation/perf-record.txt > @@ -204,7 +204,63 @@ OPTIONS > ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, > code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, > p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, > - mem_dtlb, mem_blk, mem_hops, uid, gid > + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), > + lock_held_read(<lock>), lock_held_shared(<lock>), > + lock_held_write(<lock>), lock_held_exclusive(<lock>) > + > + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters > + samples based on whether the specified kernel lock is currently held > + by the kernel thread or CPU. To differentiate between shared/read > + holding and exclusive/write holding, use 'lock_held_read(<lock>)' > + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' > + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to > + <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)'). > + <lock> can be a kernel lock symbol name (e.g. > + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache > + name prefixed with '&' (e.g. 'lock_held(&dentry)', > + 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock', > + 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex', > + 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', > + 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). > + Both read/shared and write/exclusive tracking are supported for > + 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock' > + (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', > + 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support > + write/exclusive tracking only. Note that 'semaphore' tracking > + ('down*()' / 'up()') assumes binary mutex-like usage where the > + acquiring task also releases the semaphore (such as 'console_sem'), > + rather than cross-task completion signaling. > + > + For example, 'lock_held()' can be combined with memory allocation or > + reclaim events to detect allocations or direct memory reclaim occurring > + while a kernel lock is held (which can stall lock waiters for extended > + periods): > + > + # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ > + --filter 'lock_held(mutex) || lock_held(rwsem)' > + # perf record -a -g -e kmem:mm_page_alloc \ > + --filter 'gfp_flags & 0x400' \ > + --filter 'lock_held(mutex) || lock_held(rwsem)' > + # perf record -a -g -e major-faults \ > + --filter 'lock_held(mutex) || lock_held(rwsem)' > + > + When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol > + ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' > + or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's > + 'owner' field directly in kernel memory on each sample without attaching > + any BPF fentry/fexit hooks. For locks that do not maintain an > + authoritative per-task owner pointer (such as spinlocks, rwlocks, > + semaphores, reader-locked rwsems, and slab/type filters), it attaches > + BPF fentry/fexit programs to kernel lock acquire and release functions > + to track held locks in a BPF map. These BPF hooks run on every kernel > + lock acquisition and release for the attached lock family, even when no > + perf sample is being taken. To minimize this overhead, only the hooks > + for the relevant lock family and mode are attached when known (for > + example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions > + and 'rq_lock' only hooks spinlock functions), whereas slab filters > + ('&<slab>'), raw hex addresses, or symbols without a recognizable > + lock-type suffix attach hooks across all lock families (including > + high-frequency spinlocks). > > The <operator> can be one of: > ==, !=, >, >=, <, <=, & > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c > index ac0e7cf40f3a..d5a85ee98d5f 100644 > --- a/tools/perf/util/bpf-filter.c > +++ b/tools/perf/util/bpf-filter.c > @@ -40,21 +40,35 @@ > * The 'dropped' map is to keep how many samples it dropped by the filter and > * it will be reported as lost samples. > */ > +#include <errno.h> > +#include <stdio.h> > #include <stdlib.h> > +#include <stdbool.h> > +#include <stddef.h> > +#include <string.h> > #include <fcntl.h> > +#include <unistd.h> > #include <sys/ioctl.h> > #include <sys/stat.h> > > #include <bpf/bpf.h> > +#include <bpf/btf.h> > #include <linux/err.h> > +#include <linux/kernel.h> > #include <linux/list.h> > +#include <linux/string.h> > +#include <linux/zalloc.h> > #include <api/fs/fs.h> > #include <internal/xyarray.h> > #include <perf/threadmap.h> > +#include <symbol/kallsyms.h> > > #include "util/cap.h" > #include "util/debug.h" > #include "util/evsel.h" > +#include "util/lock-contention.h" > +#include "util/string2.h" > +#include "util/symbol_conf.h" > #include "util/target.h" > #include "util/thread_map.h" > #include "util/bpf-utils.h" > @@ -63,6 +77,7 @@ > #include <util/bpf-filter-flex.h> > #include <util/bpf-filter-bison.h> > > +#include "bpf_skel/lock_data.h" > #include "bpf_skel/sample-filter.h" > #include "bpf_skel/sample_filter.skel.h" > > @@ -123,7 +138,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * > { > const struct perf_sample_info *info; > > - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { > + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || > + expr->term == PBF_TERM_LOCK_HELD) { > /* Not dependent on the sample_type as computed from a BPF helper. */ > return 0; > } > @@ -166,9 +182,913 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * > return -1; > } > > -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) > +/* > + * Program groups for `lock_held(...)` BPF tracing hooks. > + * > + * When a `lock_held(...)` filter is specified, only the BPF tracing program > + * groups required for the requested lock(s) are autoloaded and attached. > + * If no `lock_held(...)` filter is present, none of these programs are loaded. > + */ > +enum lock_prog_group { > + LOCK_PROG_MUTEX = 1U << 0, > + LOCK_PROG_RWSEM_READ = 1U << 1, > + LOCK_PROG_RWSEM_WRITE = 1U << 2, > + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, > + LOCK_PROG_SPIN = 1U << 3, > + LOCK_PROG_RWLOCK_READ = 1U << 4, > + LOCK_PROG_RWLOCK_WRITE = 1U << 5, > + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, > + LOCK_PROG_RTMUTEX = 1U << 6, > + LOCK_PROG_SEM = 1U << 7, > + LOCK_PROG_PERCPU_RWSEM = 1U << 8, > + LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM, > +}; > + > +static unsigned int lock_flags_to_prog_group(unsigned int flags) > +{ > + if (flags & LCB_F_MUTEX) > + return LOCK_PROG_MUTEX; > + if (flags & LCB_F_RT) > + return LOCK_PROG_RTMUTEX; > + if (flags & LCB_F_PERCPU) > + return LOCK_PROG_PERCPU_RWSEM; > + if (flags == LCB_F_SPIN) > + return LOCK_PROG_SPIN; > + if (flags & LCB_F_SPIN) > + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; > + if (flags & LCB_F_READ) > + return LOCK_PROG_RWSEM_READ; > + if (flags & LCB_F_WRITE) > + return LOCK_PROG_RWSEM_WRITE; > + return LOCK_PROG_SEM; > +} > + > +struct lock_prog_desc { > + size_t prog_offset; > + const char *btf_func; > + unsigned int group; > +}; > + > +#define LOCK_FPROG(field, func, grp) \ > + { offsetof(struct sample_filter_bpf, progs.field), func, grp } > + > +static const struct lock_prog_desc lock_progs[] = { > + /* Mutex */ > + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", > + LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), > + /* RW semaphore */ > + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), > + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), > + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), > + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), > + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", > + LOCK_PROG_RWSEM_READ), > + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), > + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), > + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), > + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE), > + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), > + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", > + LOCK_PROG_RWSEM_WRITE), > + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), > + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), > + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), > + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), > + /* Spinlock */ > + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", > + LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), > + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), > + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), > + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), > + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", > + LOCK_PROG_SPIN), > + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", > + LOCK_PROG_SPIN), > + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), > + /* RWlock */ > + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", > + LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), > + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", > + LOCK_PROG_RWLOCK_WRITE), > + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), > + /* RT-Mutex */ > + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), > + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), > + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX), > + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", > + LOCK_PROG_RTMUTEX), > + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), > + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), > + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), > + /* Semaphore */ > + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), > + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), > + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), > + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), > + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), > + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), > + /* Percpu-RWsem */ > + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), > + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), > +}; > + > +#undef LOCK_FPROG > + > +static int split_lock_name_mode(const char *name, int expr_part, > + char *base_buf, size_t buf_sz, > + bool *has_amp, unsigned int *mode_out) > +{ > + const char *bare; > + size_t len; > + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; > + unsigned int suffix_mode = 0; > + > + if (!name || !*name) > + return -EINVAL; > + > + *has_amp = (name[0] == '&'); > + bare = *has_amp ? name + 1 : name; > + len = strlen(bare); > + if (len == 0) > + return -EINVAL; > + > + if (len > 2 && !strcmp(bare + len - 2, ":R")) { > + suffix_mode = PBF_LOCK_MODE_READ; > + len -= 2; > + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { > + suffix_mode = PBF_LOCK_MODE_WRITE; > + len -= 2; > + } > + > + if (len == 0 || len >= buf_sz) > + return -EINVAL; > + > + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && > + suffix_mode && expr_mode != suffix_mode) > + return -EINVAL; > + > + memcpy(base_buf, bare, len); > + base_buf[len] = '\0'; > + > + if (suffix_mode) > + *mode_out = suffix_mode; > + else if (expr_mode) > + *mode_out = expr_mode; > + else > + *mode_out = PBF_LOCK_MODE_ANY; > + > + return 0; > +} > + > +/* > + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", > + * "spinlock", "percpu-rwsem"). > + * > + * A lock type name may correspond to up to two flag combinations (e.g. > + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed > + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is > + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. > + */ > +static bool parse_lock_type_name(const char *name, unsigned int mode, > + struct sample_filter_bpf *skel, > + u64 *encoded_pair, unsigned int *prog_groups) > +{ > + bool found = false; > + u32 flag1 = 0xff, flag2 = 0xff; > + unsigned int groups = 0; > + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; > + > + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { > + u32 f = lock_type_table[i].flags; > + bool is_read = (f & LCB_F_READ) != 0; > + > + if (strcmp(lock_type_table[i].lock_name, lookup) && > + strcmp(lock_type_table[i].flags_name, name)) > + continue; > + > + /* > + * percpu_down_read() / percpu_up_read() fast paths are inlined > + * per-CPU counter increments without function entry points, so > + * only writer holding (percpu_down_write / percpu_up_write) can > + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. > + */ > + if ((f & LCB_F_PERCPU) && is_read) > + continue; > + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) > + continue; > + if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN)) > + continue; > + > + if (is_read && !(mode & PBF_LOCK_MODE_READ)) > + continue; > + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) > + continue; > + > + if (!found) > + flag1 = f; > + else if (f != flag1) > + flag2 = f; > + if (skel && f < LCB_F_TYPE_MAX) { > + skel->bss->has_lock_type = 1; > + skel->bss->lock_type_filter[f] = 1; > + } > + groups |= lock_flags_to_prog_group(f); > + found = true; > + } > + > + if (!found) > + return false; > + > + if (encoded_pair) > + *encoded_pair = flag1 | ((u64)flag2 << 8); > + if (prog_groups) > + *prog_groups |= groups; > + return true; > +} > + > +struct lock_sym_cb_args { > + const char *name; > + u64 addr; > + bool found; > +}; > + > +static int find_lock_sym_cb(void *arg, const char *name, > + char type __maybe_unused, u64 start) > +{ > + struct lock_sym_cb_args *args = arg; > + > + if (!strcmp(name, args->name)) { > + args->found = true; > + args->addr = start; > + if (start != 0) > + return 1; > + } > + return 0; > +} > + > +static int find_kernel_lock_sym(const char *name, u64 *addr) > +{ > + struct lock_sym_cb_args args = { > + .name = name, > + }; > + const char *kallsyms_filename = "/proc/kallsyms"; > + > + if (symbol_conf.kallsyms_name != NULL) > + kallsyms_filename = symbol_conf.kallsyms_name; > + > + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) > + return -ENOENT; > + > + if (!args.found) > + return -ENOENT; > + > + if (args.addr == 0) { > + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", > + name); > + return -EACCES; > + } > + > + *addr = args.addr; > + return 0; > +} > + > +static bool parse_lock_hex_addr(const char *str, u64 *addr) > +{ > + char *end; > + unsigned long long val; > + > + if (!str || !*str) > + return false; > + > + errno = 0; > + val = strtoull(str, &end, 16); > + if (*end != '\0' || errno != 0 || val == 0) > + return false; > + > + *addr = val; > + return true; > +} > + > +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) > +{ > + u64 addr = 0; > + char *buf = NULL, *p; > + size_t len; > + long last = -1; > + int ret; > + > + if (!btf) > + return; > + > + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); > + if (ret < 0) > + return; > + > + ret = btf__resolve_size(btf, ret); > + if (ret < 0) > + return; > + > + skel->rodata->sizeof_zone = ret; > + > + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { > + skel->rodata->contig_page_data_addr = addr; > + return; > + } > + > + if (find_kernel_lock_sym("node_data", &addr) != 0) > + return; > + > + skel->rodata->node_data_addr = addr; > + > + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { > + p = buf; > + while (p && *p) { > + char *end; > + > + last = strtol(p, &end, 0); > + if (end == p) > + break; > + p = end; > + if (*p == ',' || *p == '-' || *p == '\n') > + p++; > + } > + skel->rodata->nr_nodes = last + 1; > + free(buf); > + } > +} > + > +static bool btf_has_func(struct btf *btf, const char *name) > +{ > + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; > +} > + > +static unsigned int supported_lock_prog_groups(struct btf *btf) > +{ > + unsigned int supported = 0; > + > + if (!btf) > + return 0; > + > + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) && > + btf_has_func(btf, "mutex_unlock")) > + supported |= LOCK_PROG_MUTEX; > + > + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) && > + btf_has_func(btf, "up_read")) > + supported |= LOCK_PROG_RWSEM_READ; > + > + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) && > + btf_has_func(btf, "up_write")) > + supported |= LOCK_PROG_RWSEM_WRITE; > + > + /* > + * Require both acquire and all unlock entry points in BTF so we never > + * attach spinlock/rwlock acquire hooks when unlock functions are > + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK). > + */ > + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) && > + btf_has_func(btf, "_raw_spin_unlock") && > + btf_has_func(btf, "_raw_spin_unlock_irq") && > + btf_has_func(btf, "_raw_spin_unlock_irqrestore") && > + btf_has_func(btf, "_raw_spin_unlock_bh")) > + supported |= LOCK_PROG_SPIN; > + > + if (btf_has_func(btf, "_raw_read_lock") && > + btf_has_func(btf, "_raw_read_unlock") && > + btf_has_func(btf, "_raw_read_unlock_irq") && > + btf_has_func(btf, "_raw_read_unlock_irqrestore") && > + btf_has_func(btf, "_raw_read_unlock_bh")) > + supported |= LOCK_PROG_RWLOCK_READ; > + > + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) && > + btf_has_func(btf, "_raw_write_unlock") && > + btf_has_func(btf, "_raw_write_unlock_irq") && > + btf_has_func(btf, "_raw_write_unlock_irqrestore") && > + btf_has_func(btf, "_raw_write_unlock_bh")) > + supported |= LOCK_PROG_RWLOCK_WRITE; > + > + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) && > + btf_has_func(btf, "rt_mutex_unlock")) > + supported |= LOCK_PROG_RTMUTEX; > + > + if (btf_has_func(btf, "down") && btf_has_func(btf, "up")) > + supported |= LOCK_PROG_SEM; > + > + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) > + supported |= LOCK_PROG_PERCPU_RWSEM; > + > + return supported; > +} > + > +/* > + * Enable or disable autoload for each lock-tracking BPF program in the > + * skeleton before `sample_filter_bpf__load()` is called. > + * > + * Each program is enabled only if its group is in `needed_groups` and its > + * target kernel function or tracepoint exists in the running kernel's BTF. > + */ > +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, > + unsigned int needed_groups, > + unsigned int required_groups, > + bool needs_lock_syms, bool needs_slab_iter, > + bool is_tracepoint) > +{ > + unsigned int supported_groups = supported_lock_prog_groups(btf); > + unsigned int enabled_progs = 0; > + > + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); > + > + if ((required_groups & supported_groups) != required_groups || > + (needed_groups && !(needed_groups & supported_groups))) { > + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n"); > + return -ENOTSUP; > + } > + > + needed_groups &= supported_groups; > + if (skel) > + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ); > + > + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { > + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; > + struct bpf_program *prog = *progp; > + bool enable = false; > + > + if ((needed_groups & lock_progs[i].group) && btf && > + lock_progs[i].btf_func) { > + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, > + BTF_KIND_FUNC) > 0) > + enable = true; > + } > + > + if (enable) > + enabled_progs++; > + bpf_program__set_autoload(prog, enable); > + } > + > + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); > + if (needs_slab_iter && btf && > + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { > + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); > + } else { > + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); > + } > + > + if (needed_groups && !enabled_progs) { > + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); > + return -ENOTSUP; > + } > + return 0; > +} > + > +/* > + * Pre-scan a single filter expression prior to loading the BPF skeleton in > + * order to determine which BPF maps need to be sized and which lock tracing > + * program groups need to be autoloaded. > + */ > +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, > + bool *has_lock_filter, > + unsigned int *prog_groups, > + unsigned int *required_groups, > + bool *needs_lock_syms, > + bool *needs_slab_iter) > +{ > + char base[128]; > + bool has_amp; > + unsigned int mode, groups = 0; > + u64 dummy; > + > + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) > + return; > + > + *has_lock_filter = true; > + if (split_lock_name_mode(expr->lock_name, expr->part, base, > + sizeof(base), &has_amp, &mode) < 0) > + return; > + > + /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */ > + if (has_amp) { > + *needs_slab_iter = true; > + *prog_groups |= LOCK_PROG_ALL; > + return; > + } > + > + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { > + /* > + * Write ownership of mm->mmap_lock is inspected directly via > + * is_rwsem_writer_owner() without tracing hooks; only reader > + * tracking requires LOCK_PROG_RWSEM_READ. > + */ > + if (mode & PBF_LOCK_MODE_READ) { > + *prog_groups |= LOCK_PROG_RWSEM_READ; > + *required_groups |= LOCK_PROG_RWSEM_READ; > + } > + return; > + } > + if (!strcmp(base, "siglock")) { > + *prog_groups |= LOCK_PROG_SPIN; > + *required_groups |= LOCK_PROG_SPIN; > + return; > + } > + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { > + *prog_groups |= LOCK_PROG_SPIN; > + *required_groups |= LOCK_PROG_SPIN; > + *needs_lock_syms = true; > + return; > + } > + if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { > + *prog_groups |= groups; > + *required_groups |= groups; > + return; > + } > + > + if (parse_lock_hex_addr(base, &dummy)) { > + *prog_groups |= LOCK_PROG_ALL; > + return; > + } > + > + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 || > + strtailcmp(base, "mutex") == 0) { > + /* Inspected directly via lock->owner; no tracing hooks needed. */ > + return; > + } else if (strtailcmp(base, "percpu_rwsem") == 0 || > + strtailcmp(base, "pcpu_rwsem") == 0) { > + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; > + *required_groups |= LOCK_PROG_PERCPU_RWSEM; > + } else if (strtailcmp(base, "rwsem") == 0) { > + if (mode & PBF_LOCK_MODE_READ) { > + *prog_groups |= LOCK_PROG_RWSEM_READ; > + *required_groups |= LOCK_PROG_RWSEM_READ; > + } > + if (mode & PBF_LOCK_MODE_WRITE) { > + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM; > + *required_groups |= LOCK_PROG_RWSEM_WRITE; > + } > + } else if (strtailcmp(base, "spinlock") == 0) { > + *prog_groups |= LOCK_PROG_SPIN; > + *required_groups |= LOCK_PROG_SPIN; > + } else if (strtailcmp(base, "sem") == 0) { > + *prog_groups |= LOCK_PROG_SEM; > + *required_groups |= LOCK_PROG_SEM; > + } else if (strtailcmp(base, "rwlock") == 0) { > + if (mode & PBF_LOCK_MODE_READ) > + groups |= LOCK_PROG_RWLOCK_READ; > + if (mode & PBF_LOCK_MODE_WRITE) > + groups |= LOCK_PROG_RWLOCK_WRITE; > + *prog_groups |= groups; > + *required_groups |= groups; > + } else { > + *prog_groups |= LOCK_PROG_ALL; > + } > +} > + > +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, > + unsigned int *prog_groups, > + unsigned int *required_groups, > + bool *needs_lock_syms, > + bool *needs_slab_iter) > +{ > + struct perf_bpf_filter_expr *expr, *group; > + > + *has_lock_filter = false; > + *prog_groups = 0; > + *required_groups = 0; > + *needs_lock_syms = false; > + *needs_slab_iter = false; > + > + list_for_each_entry(expr, &evsel->bpf_filters, list) { > + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, > + required_groups, needs_lock_syms, > + needs_slab_iter); > + if (expr->op == PBF_OP_GROUP_BEGIN) { > + list_for_each_entry(group, &expr->groups, list) { > + prescan_one_lock_expr(group, has_lock_filter, > + prog_groups, required_groups, > + needs_lock_syms, needs_slab_iter); > + } > + } > + } > +} > + > +static void run_slab_cache_iter(struct sample_filter_bpf *skel) > +{ > + struct bpf_link *link; > + int fd; > + char buf[256]; > + > + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) > + return; > + > + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); > + if (!link || IS_ERR(link)) { > + pr_debug("cannot attach slab cache iter\n"); > + return; > + } > + > + fd = bpf_iter_create(bpf_link__fd(link)); > + if (fd >= 0) { > + while (read(fd, buf, sizeof(buf)) > 0) > + continue; > + close(fd); > + } > + > + bpf_link__destroy(link); > +} > + > +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, > + u64 *kmem_cache_addr) > +{ > + int cache_fd; > + u64 key, *prev_key = NULL; > + > + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) > + return false; > + > + cache_fd = bpf_map__fd(skel->maps.slab_caches); > + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { > + struct slab_cache_data data; > + > + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) > + break; > + > + if (!strcmp(data.name, name)) { > + *kmem_cache_addr = key; > + return true; > + } > + prev_key = &key; > + } > + return false; > +} > + > +static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr, > + enum perf_bpf_filter_op *op_out) > +{ > + switch (expr->op) { > + case PBF_OP_EQ: > + if (expr->val == 1) { > + *op_out = PBF_OP_EQ; > + return 0; > + } > + if (expr->val == 0) { > + *op_out = PBF_OP_NEQ; > + return 0; > + } > + break; > + case PBF_OP_NEQ: > + if (expr->val == 0) { > + *op_out = PBF_OP_EQ; > + return 0; > + } > + if (expr->val == 1) { > + *op_out = PBF_OP_NEQ; > + return 0; > + } > + break; > + case PBF_OP_GT: > + if (expr->val == 0) { > + *op_out = PBF_OP_EQ; > + return 0; > + } > + break; > + case PBF_OP_GE: > + if (expr->val == 1) { > + *op_out = PBF_OP_EQ; > + return 0; > + } > + break; > + case PBF_OP_LT: > + if (expr->val == 1) { > + *op_out = PBF_OP_NEQ; > + return 0; > + } > + break; > + case PBF_OP_LE: > + if (expr->val == 0) { > + *op_out = PBF_OP_NEQ; > + return 0; > + } > + break; > + case PBF_OP_AND: > + if (expr->val & 1) { > + *op_out = PBF_OP_EQ; > + return 0; > + } > + break; > + case PBF_OP_GROUP_BEGIN: > + case PBF_OP_GROUP_END: > + case PBF_OP_DONE: > + default: > + break; > + } > + pr_err("Error: Invalid comparison for boolean lock_held(%s) predicate\n", > + expr->lock_name ?: ""); > + return -EINVAL; > +} > + > +/* > + * Resolve a `lock_held(<name>)` filter expression into `entry->op`, > + * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`, > + * and populate the corresponding BPF filter maps and BSS control flags. > + * > + * Resolution order: > + * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`). > + * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, > + * `rq_lock`, `zone_lock`. > + * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, > + * `rwlock`, `percpu-rwsem`, `semaphore`, etc. > + * 4. Hexadecimal lock address. > + * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). > + */ > +static int resolve_lock_expr(struct sample_filter_bpf *skel, > + struct perf_bpf_filter_expr *expr, > + struct perf_bpf_filter_entry *entry) > +{ > + const char *name = expr->lock_name; > + char base[128]; > + bool has_amp; > + unsigned int mode; > + u64 addr = 0, encoded_pair = 0; > + u8 one = 1; > + int ret; > + > + ret = normalize_lock_expr_op(expr, &entry->op); > + if (ret < 0) > + return ret; > + > + if (split_lock_name_mode(name, expr->part, base, sizeof(base), > + &has_amp, &mode) < 0) { > + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); > + return -EINVAL; > + } > + > + if (has_amp) { > + if (find_slab_cache(skel, base, &addr)) { > + int fd = bpf_map__fd(skel->maps.slab_filter); > + > + entry->part = PBF_LOCK_SLAB | mode; > + entry->value = addr; > + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); > + skel->bss->has_lock_slab = 1; > + return 0; > + } > + pr_err("Error: Cannot resolve slab cache '%s'\n", name); > + return -EINVAL; > + } > + > + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { > + entry->part = PBF_LOCK_MMAP | mode; > + entry->value = 0; > + if (skel && (mode & PBF_LOCK_MODE_READ)) > + skel->bss->has_mmap_lock = 1; > + return 0; > + } > + > + if (!strcmp(base, "siglock")) { > + if (!(mode & PBF_LOCK_MODE_WRITE)) { > + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", > + name); > + return -EINVAL; > + } > + entry->part = PBF_LOCK_SIGHAND | mode; > + entry->value = 0; > + if (skel) > + skel->bss->has_siglock = 1; > + return 0; > + } > + > + if (!strcmp(base, "rq_lock")) { > + if (!(mode & PBF_LOCK_MODE_WRITE)) { > + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", > + name); > + return -EINVAL; > + } > + entry->part = PBF_LOCK_CLASS | mode; > + entry->value = LOCK_CLASS_RQLOCK; > + if (skel) > + skel->bss->has_lock_class = 1; > + return 0; > + } > + > + if (!strcmp(base, "zone_lock")) { > + if (!(mode & PBF_LOCK_MODE_WRITE)) { > + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", > + name); > + return -EINVAL; > + } > + entry->part = PBF_LOCK_CLASS | mode; > + entry->value = LOCK_CLASS_ZONE_LOCK; > + if (skel) > + skel->bss->has_lock_class = 1; > + return 0; > + } > + > + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { > + entry->part = PBF_LOCK_TYPE | mode; > + entry->value = encoded_pair; > + return 0; > + } > + > + if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { > + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", > + name); > + return -EINVAL; > + } > + > + if (parse_lock_hex_addr(base, &addr)) { > + entry->part = PBF_LOCK_ADDR | mode; > + entry->value = addr; > + if (skel) { > + int fd = bpf_map__fd(skel->maps.lock_addr_filter); > + > + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); > + skel->bss->has_lock_addr = 1; > + } > + return 0; > + } > + > + ret = find_kernel_lock_sym(base, &addr); > + if (ret == 0) { > + unsigned int direct_flag = 0; > + > + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) > + direct_flag = PBF_LOCK_DIRECT_RTMUTEX; > + else if (strtailcmp(base, "mutex") == 0) > + direct_flag = PBF_LOCK_DIRECT_MUTEX; > + else if (strtailcmp(base, "rwsem") == 0 && > + strtailcmp(base, "percpu_rwsem") != 0 && > + strtailcmp(base, "pcpu_rwsem") != 0) > + direct_flag = PBF_LOCK_DIRECT_RWSEM; > + > + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX || > + direct_flag == PBF_LOCK_DIRECT_RTMUTEX || > + strtailcmp(base, "percpu_rwsem") == 0 || > + strtailcmp(base, "pcpu_rwsem") == 0 || > + strtailcmp(base, "spinlock") == 0 || > + (strtailcmp(base, "sem") == 0 && > + strtailcmp(base, "rwsem") != 0)) && > + !(mode & PBF_LOCK_MODE_WRITE)) { > + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", > + name); > + return -EINVAL; > + } > + > + entry->part = PBF_LOCK_ADDR | mode | direct_flag; > + entry->value = addr; > + if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX && > + direct_flag != PBF_LOCK_DIRECT_RTMUTEX) { > + int fd = bpf_map__fd(skel->maps.lock_addr_filter); > + > + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); > + skel->bss->has_lock_addr = 1; > + } > + return 0; > + } > + if (ret == -EACCES) > + return ret; > + > + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); > + return -EINVAL; > +} > + > +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, > + struct perf_bpf_filter_entry *entry) > { > - int i = 0; > + int i = 0, ret; > struct perf_bpf_filter_expr *expr; > > list_for_each_entry(expr, &evsel->bpf_filters, list) { > @@ -182,6 +1102,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry > entry[i].part = expr->part; > entry[i].term = expr->term; > entry[i].value = expr->val; > + if (expr->term == PBF_TERM_LOCK_HELD) { > + ret = resolve_lock_expr(skel, expr, &entry[i]); > + if (ret < 0) > + return ret; > + } > i++; > > if (expr->op == PBF_OP_GROUP_BEGIN) { > @@ -195,6 +1120,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry > entry[i].part = group->part; > entry[i].term = group->term; > entry[i].value = group->val; > + if (group->term == PBF_TERM_LOCK_HELD) { > + ret = resolve_lock_expr(skel, group, &entry[i]); > + if (ret < 0) > + return ret; > + } > i++; > } > > @@ -434,22 +1364,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > { > int i, x, y, fd, ret; > struct sample_filter_bpf *skel = NULL; > + struct btf *btf = NULL; > struct perf_bpf_filter_entry *entry; > bool needs_idx_hash = !target__has_cpu(target); > bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); > + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; > + unsigned int prog_groups = 0, required_groups = 0; > > entry = calloc(MAX_FILTERS, sizeof(*entry)); > if (entry == NULL) > return -1; > > - ret = get_filter_entries(evsel, entry); > - if (ret < 0) { > - pr_err("Failed to process filter entries\n"); > - goto err; > - } > + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, > + &required_groups, &needs_lock_syms, > + &needs_slab_iter); > > - if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { > - pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); > + if ((has_lock_filter || is_tracepoint) && geteuid() != 0 && > + !perf_cap__capable(CAP_BPF)) { > + pr_err("Error: BPF filter requires CAP_BPF capability\n"); > ret = -EPERM; > goto err; > } > @@ -457,6 +1389,11 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { > int zero = 0; > > + ret = get_filter_entries(evsel, NULL, entry); > + if (ret < 0) { > + pr_err("Failed to process filter entries\n"); > + goto err; > + } > /* The filters map is shared among other processes */ > ret = create_idx_hash(evsel, entry); > if (ret < 0) > @@ -501,8 +1438,26 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > goto err; > } > > - bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); > - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); > + if (has_lock_filter) { > + btf = btf__load_vmlinux_btf(); > + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); > + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); > + if (needs_slab_iter) { > + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); > + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); > + } > + if (needs_lock_syms) { > + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); > + init_numa_data(skel, btf); > + } > + ret = configure_lock_progs(skel, btf, prog_groups, required_groups, > + needs_lock_syms, needs_slab_iter, > + is_tracepoint); > + if (ret < 0) > + goto err; > + } else { > + configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint); > + } > > if (sample_filter_bpf__load(skel) < 0) { > ret = -errno; > @@ -510,6 +1465,37 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > goto err; > } > > + if (has_lock_filter && needs_slab_iter) > + run_slab_cache_iter(skel); > + > + ret = get_filter_entries(evsel, skel, entry); > + if (ret < 0) { > + pr_err("Failed to process filter entries\n"); > + goto err; > + } > + > + if (has_lock_filter) { > + if (needs_lock_syms && skel->bss->has_lock_class) { > + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, > + .flags = BPF_F_TEST_RUN_ON_CPU, > + ); > + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); > + > + bpf_prog_test_run_opts(prog_fd, &opts); > + } > + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); > + bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false); > + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); > + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); > + > + ret = sample_filter_bpf__attach(skel); > + if (ret < 0) { > + pr_err("Failed to attach lock sample-filter BPF programs\n"); > + goto err; > + } > + skel->bss->lock_enabled = 1; > + } > + > i = 0; > fd = bpf_map__fd(skel->maps.filters); > > @@ -549,11 +1535,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > } > } > } > + btf__free(btf); > free(entry); > evsel->bpf_skel = skel; > return 0; > > err: > + btf__free(btf); > free(entry); > if (!list_empty(&pinned_filters)) { > struct pinned_filter_idx *pfi, *tmp; > @@ -568,6 +1556,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > return ret; > } > > +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) > +{ > + struct perf_bpf_filter_expr *group, *tmp; > + > + if (!expr) > + return; > + > + list_for_each_entry_safe(group, tmp, &expr->groups, list) { > + list_del(&group->list); > + perf_bpf_filter_expr__delete(group); > + } > + zfree(&expr->lock_name); > + free(expr); > +} > + > int perf_bpf_filter__destroy(struct evsel *evsel) > { > struct perf_bpf_filter_expr *expr, *tmp; > @@ -575,7 +1578,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) > > list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { > list_del(&expr->list); > - free(expr); > + perf_bpf_filter_expr__delete(expr); > } > sample_filter_bpf__destroy(evsel->bpf_skel); > > @@ -627,7 +1630,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term > { > struct perf_bpf_filter_expr *expr; > > - expr = malloc(sizeof(*expr)); > + expr = zalloc(sizeof(*expr)); > if (expr != NULL) { > expr->term = term; > expr->part = part; > @@ -638,6 +1641,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term > return expr; > } > > +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, > + int mode, > + enum perf_bpf_filter_op op, > + unsigned long val) > +{ > + struct perf_bpf_filter_expr *expr; > + > + if (!lock_name || !*lock_name) > + return NULL; > + > + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); > + if (!expr) > + return NULL; > + > + expr->lock_name = strdup(lock_name); > + if (!expr->lock_name) { > + free(expr); > + return NULL; > + } > + return expr; > +} > + > static bool check_bpf_filter_capable(void) > { > int fd; > @@ -658,14 +1683,11 @@ static bool check_bpf_filter_capable(void) > return false; > } > > -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) > +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) > { > YY_BUFFER_STATE buffer; > int ret; > > - if (!check_bpf_filter_capable()) > - return -EPERM; > - > buffer = perf_bpf_filter__scan_string(str); > > ret = perf_bpf_filter_parse(expr_head); > @@ -677,6 +1699,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) > return ret; > } > > +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) > +{ > + if (!check_bpf_filter_capable()) > + return -EPERM; > + > + return __perf_bpf_filter__parse(expr_head, str); > +} > + > static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) > { > char *prog_path = NULL; > @@ -729,8 +1759,9 @@ int perf_bpf_filter__pin(void) > goto err; > } > > + configure_lock_progs(skel, NULL, 0, 0, false, false, false); > + > /* pinned program will use pid-hash */ > - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); > bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); > bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); > bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); > @@ -825,7 +1856,7 @@ int perf_bpf_filter__unpin(void) > goto err; > } > > - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); > + configure_lock_progs(skel, NULL, 0, 0, false, false, false); > if (sample_filter_bpf__load(skel) < 0) { > ret = -errno; > pr_err("Failed to load perf sample-filter BPF skeleton\n"); > diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h > index 818c554b91b2..fda95ff226b3 100644 > --- a/tools/perf/util/bpf-filter.h > +++ b/tools/perf/util/bpf-filter.h > @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { > int part; > enum perf_bpf_filter_term term; > unsigned long val; > + /* > + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions > + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). > + * Resolved into `entry.part` and `entry.target` during > + * perf_bpf_filter__prepare(). > + */ > + char *lock_name; > }; > > struct evsel; > @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term > int part, > enum perf_bpf_filter_op op, > unsigned long val); > +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, > + int mode, > + enum perf_bpf_filter_op op, > + unsigned long val); > +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); > +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); > int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); > int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); > int perf_bpf_filter__destroy(struct evsel *evsel); > diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l > index 1be9df6550fc..b4022110f1e5 100644 > --- a/tools/perf/util/bpf-filter.l > +++ b/tools/perf/util/bpf-filter.l > @@ -1,9 +1,12 @@ > %option prefix="perf_bpf_filter_" > %option noyywrap > > +%x LOCK_ARG > + > %{ > #include <stdio.h> > #include <stdlib.h> > +#include <string.h> > #include <linux/perf_event.h> > > #include "bpf-filter.h" > @@ -76,7 +79,7 @@ static int path_or_error(void) > num_dec [0-9]+ > num_hex 0[Xx][0-9a-fA-F]+ > space [ \t]+ > -path [^ \t\n,]+ > +path [^ \t\n,()!=>&|<]+ > ident [_a-zA-Z][_a-zA-Z0-9]+ > > %% > @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } > uid { return sample(PBF_TERM_UID); } > gid { return sample(PBF_TERM_GID); } > cgroup { return sample_path(PBF_TERM_CGROUP); } > - > +lock_held[ \t]*"(" { > + BEGIN(LOCK_ARG); > + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; > + return BFT_LOCK_HELD; > + } > +(lock_held_read|lock_held_shared)[ \t]*"(" { > + BEGIN(LOCK_ARG); > + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; > + return BFT_LOCK_HELD; > + } > +(lock_held_write|lock_held_exclusive)[ \t]*"(" { > + BEGIN(LOCK_ARG); > + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; > + return BFT_LOCK_HELD; > + } > + > +<LOCK_ARG>{space} { } > +<LOCK_ARG>[^ \t\n()]+ { > + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); > + return BFT_LOCK_NAME; > + } > +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } > +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } > + > +"!" { return '!'; } > "==" { return operator(PBF_OP_EQ); } > "!=" { return operator(PBF_OP_NEQ); } > ">" { return operator(PBF_OP_GT); } > diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y > index 5a79a8e7a45b..5ae05d4608c6 100644 > --- a/tools/perf/util/bpf-filter.y > +++ b/tools/perf/util/bpf-filter.y > @@ -8,9 +8,11 @@ > #endif > > #include <stdio.h> > +#include <stdlib.h> > #include <string.h> > #include <linux/compiler.h> > #include <linux/list.h> > +#include <linux/zalloc.h> > #include "bpf-filter.h" > #include "cgroup.h" > > @@ -31,6 +33,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, > { > unsigned long num; > char *path; > + char *lock_name; > + struct { > + char *name; > + int mode; > + } lock; > struct { > enum perf_bpf_filter_term term; > int part; > @@ -40,12 +47,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, > } > > %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH > +%token BFT_LOCK_HELD BFT_LOCK_NAME > %type <expr> filter_term filter_expr > -%destructor { free ($$); } <expr> > +%destructor { perf_bpf_filter_expr__delete($$); } <expr> > +%destructor { free($$); } <lock_name> > +%destructor { free($$.name); } <lock> > %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH > %type <op> BFT_OP > -%type <num> BFT_NUM > +%type <num> BFT_NUM BFT_LOCK_HELD > %type <path> BFT_PATH > +%type <lock_name> BFT_LOCK_NAME > +%type <lock> lock_term > > %% > > @@ -105,5 +117,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH > $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); > cgroup__put(cgrp); > } > +| > +lock_term > +{ > + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); > + zfree(&$1.name); > + if (!$$) > + YYERROR; > +} > +| > +'!' lock_term > +{ > + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); > + zfree(&$2.name); > + if (!$$) > + YYERROR; > +} > +| > +lock_term BFT_OP BFT_NUM > +{ > + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); > + zfree(&$1.name); > + if (!$$) > + YYERROR; > +} > + > +lock_term: > +BFT_LOCK_HELD BFT_LOCK_NAME ')' > +{ > + $$.name = $2; > + $$.mode = $1; > +} > > %% > diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h > index 683fec85e71e..a65ee780d64d 100644 > --- a/tools/perf/util/bpf_skel/sample-filter.h > +++ b/tools/perf/util/bpf_skel/sample-filter.h > @@ -53,8 +53,74 @@ enum perf_bpf_filter_term { > /* Terms computed from BPF helpers. */ > PBF_TERM_UID, > PBF_TERM_GID, > + /* > + * Kernel lock held predicate (`lock_held(...)`). Evaluates whether the > + * specified kernel lock is currently held by the running task/CPU. > + * The lock specification kind is stored in `entry.part` > + * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to > + * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held), > + * and the resolved target address, class ID, or type flag encoding is > + * stored in `entry.value`. > + */ > + PBF_TERM_LOCK_HELD, > }; > > +/* > + * Lock target categories for PBF_TERM_LOCK_HELD filters. > + * Stored in `perf_bpf_filter_entry.part`. > + */ > +enum perf_bpf_lock_target_kind { > + PBF_LOCK_NONE = 0, > + /* > + * Specific global or static lock address (resolved from kallsyms or > + * given as a hex address). `entry.value` holds the kernel address. > + */ > + PBF_LOCK_ADDR, > + /* > + * Dynamic lock allocated inside a slab cache (`&slab_name`). > + * `entry.value` holds the kernel `struct kmem_cache *` address. > + */ > + PBF_LOCK_SLAB, > + /* > + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). > + */ > + PBF_LOCK_MMAP, > + /* > + * Per-process `current->sighand->siglock`. > + */ > + PBF_LOCK_SIGHAND, > + /* > + * Semi-global lock class (`rq_lock` or `zone_lock`). > + * `entry.value` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or > + * LOCK_CLASS_ZONE_LOCK). > + */ > + PBF_LOCK_CLASS, > + /* > + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). > + * `entry.value` packs up to two `LCB_F_*` flag combinations in the > + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). > + */ > + PBF_LOCK_TYPE, > +}; > + > +/* > + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: > + * Bits 0..3: `enum perf_bpf_lock_target_kind` > + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) > + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) > + * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner) > + * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner) > + * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base / rt_mutex owner) > + */ > +#define PBF_LOCK_KIND_MASK 0x0fU > +#define PBF_LOCK_MODE_READ (1U << 4) > +#define PBF_LOCK_MODE_WRITE (1U << 5) > +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) > +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY > +#define PBF_LOCK_DIRECT_MUTEX (1U << 6) > +#define PBF_LOCK_DIRECT_RWSEM (1U << 7) > +#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8) > + > /* BPF map entry for filtering */ > struct perf_bpf_filter_entry { > enum perf_bpf_filter_op op; > diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c > index a190374ed87a..d1192a93179e 100644 > --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c > +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c > @@ -5,8 +5,47 @@ > #include <bpf/bpf_tracing.h> > #include <bpf/bpf_core_read.h> > > +#include "lock_data.h" > #include "sample-filter.h" > > +/* > + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. > + * > + * - `addr_or_id`: > + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance > + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) > + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address > + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags > + * - `owner_id`: > + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), > + * since the holder may sleep and migrate across CPUs while holding the lock. > + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks > + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption > + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be > + * acquired by one task and released by the next task across `context_switch()`. > + * - `kind`: `enum perf_bpf_lock_target_kind`. > + */ > +struct held_lock_key { > + __u64 addr_or_id; > + __u32 owner_id; > + __u32 kind; > +}; > + > +/* > + * Value for `held_locks` entries. > + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) > + * and exclusive/write acquisitions (including nested class/slab/type > + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of > + * the lock so release tracepoints that do not carry flags (such as > + * `lock:contended_release`) can decrement matching class/slab/type entries. > + */ > +struct held_lock_val { > + __u32 read_count; > + __u32 write_count; > + __u32 flags; > + __u32 pad; > +}; > + > /* BPF map that will be filled by user space */ > struct filters { > __uint(type, BPF_MAP_TYPE_HASH); > @@ -43,10 +82,225 @@ struct lost_count { > __uint(max_entries, 1); > } dropped SEC(".maps"); > > +/* > + * Currently held kernel locks tracked by fentry/fexit hooks. > + * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or > + * non-owner rwsem is acquired in one task and released in another task or > + * interrupt handler (cross-context signaling), any orphaned entry is > + * automatically evicted rather than exhausting the map. > + */ > +struct { > + __uint(type, BPF_MAP_TYPE_LRU_HASH); > + __type(key, struct held_lock_key); > + __type(value, struct held_lock_val); > + __uint(max_entries, 1); > +} held_locks SEC(".maps"); > + > +/* Target lock addresses for PBF_LOCK_ADDR filters. */ > +struct { > + __uint(type, BPF_MAP_TYPE_HASH); > + __type(key, __u64); > + __type(value, __u8); > + __uint(max_entries, 1); > +} lock_addr_filter SEC(".maps"); > + > +struct { > + __uint(type, BPF_MAP_TYPE_HASH); > + __uint(key_size, sizeof(__u64)); > + __uint(value_size, sizeof(__u32)); > + __uint(max_entries, 1); > +} lock_syms SEC(".maps"); > + > +struct { > + __uint(type, BPF_MAP_TYPE_HASH); > + __uint(key_size, sizeof(__u64)); > + __uint(value_size, sizeof(__u8)); > + __uint(max_entries, 1); > +} slab_filter SEC(".maps"); > + > +struct { > + __uint(type, BPF_MAP_TYPE_HASH); > + __uint(key_size, sizeof(__u64)); > + __uint(value_size, sizeof(struct slab_cache_data)); > + __uint(max_entries, 1); > +} slab_caches SEC(".maps"); > + > volatile const int use_idx_hash; > > +const volatile __u64 contig_page_data_addr; > +const volatile __u64 node_data_addr; > +const volatile int nr_nodes; > +const volatile int sizeof_zone; > + > +__u32 slab_cache_id; > + > +/* Control flags set by userspace when lock_held(...) filters are active. */ > +int lock_enabled; > +int has_lock_addr; > +int has_lock_slab; > +int has_lock_class; > +int has_lock_type; > +int has_mmap_lock; > +int has_siglock; > +int track_rwsem_read; > +__u8 lock_type_filter[LCB_F_TYPE_MAX]; > + > void *bpf_cast_to_kern_ctx(void *) __ksym; > > +/* > + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. > + * > + * For locks that maintain a task_struct pointer in their `owner` field > + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), > + * `perf_sample_filter` can inspect the lock structure directly in kernel memory > + * and compare the owner against `current` (`bpf_get_current_task()`). > + * > + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear > + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and > + * spinlocks are checked via the `held_locks` BPF map instead. > + */ > +static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr) > +{ > + struct rw_semaphore___new *rwsem = (void *)lock_addr; > + > + if (bpf_core_field_exists(rwsem->count.counter) && > + !(BPF_CORE_READ(rwsem, count.counter) & 1UL)) > + return false; > + if (bpf_core_field_exists(rwsem->owner.counter) && > + (BPF_CORE_READ(rwsem, owner.counter) & 1UL)) > + return false; > + > + return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr; > +} > + > +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part) > +{ > + if (!curr) > + return false; > + > + if ((part & PBF_LOCK_DIRECT_MUTEX) && > + (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr) > + return true; > + if ((part & PBF_LOCK_DIRECT_RWSEM) && > + is_rwsem_writer_owner(lock_addr, curr)) > + return true; > + if ((part & PBF_LOCK_DIRECT_RTMUTEX) && > + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr) > + return true; > + return false; > +} > + > +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) > +{ > + if (!val) > + return false; > + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) > + return true; > + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) > + return true; > + return false; > +} > + > +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, > + __u32 mode, __u32 pid, __u32 cpu_id) > +{ > + struct held_lock_key key = { > + .addr_or_id = addr_or_id, > + .owner_id = pid, > + .kind = kind, > + }; > + struct held_lock_val *val; > + > + if (pid) { > + val = bpf_map_lookup_elem(&held_locks, &key); > + if (match_held_val(val, mode)) > + return 1; > + } > + > + key.owner_id = cpu_id; > + val = bpf_map_lookup_elem(&held_locks, &key); > + if (match_held_val(val, mode)) > + return 1; > + > + return 0; > +} > + > +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, > + __u32 pid, __u32 cpu_id) > +{ > + __u32 flag1 = target & 0xff; > + __u32 flag2 = (target >> 8) & 0xff; > + > + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) > + return 1; > + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) > + return 1; > + return 0; > +} > + > +static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) > +{ > + __u64 curr = bpf_get_current_task(); > + __u32 pid = (__u32)bpf_get_current_pid_tgid(); > + __u32 cpu_id = 0x80000000U | bpf_get_smp_processor_id(); > + __u32 kind = entry->part & PBF_LOCK_KIND_MASK; > + __u32 mode = entry->part & PBF_LOCK_MODE_MASK; > + __u64 lock_addr; > + > + if (!mode) > + mode = PBF_LOCK_MODE_ANY; > + > + switch (kind) { > + case PBF_LOCK_ADDR: > + lock_addr = entry->value; > + if ((mode & PBF_LOCK_MODE_WRITE) && > + is_direct_lock_owner(lock_addr, curr, entry->part)) > + return true; > + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; > + > + case PBF_LOCK_MMAP: > + lock_addr = get_current_mmap_lock(curr); > + if (!lock_addr) > + return false; > + if ((mode & PBF_LOCK_MODE_WRITE) && > + is_rwsem_writer_owner(lock_addr, curr)) > + return true; > + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; > + > + case PBF_LOCK_SIGHAND: > + lock_addr = get_current_siglock(curr); > + if (!lock_addr) > + return false; > + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; > + > + case PBF_LOCK_CLASS: > + return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0; > + > + case PBF_LOCK_SLAB: > + return is_lock_in_map(entry->value, PBF_LOCK_SLAB, mode, pid, cpu_id) != 0; > + > + case PBF_LOCK_TYPE: > + return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0; > + > + default: > + return false; > + } > +} > + > +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) > +{ > + /* > + * Pinned filters (use_idx_hash != 0) share a world-writable filters > + * map and do not support lock_held(). Reject evaluation when pinned > + * or when lock_enabled is not set so unprivileged users cannot probe > + * arbitrary kernel addresses via entry->value. > + */ > + if (use_idx_hash || !lock_enabled) > + return entry->op == PBF_OP_EQ ? ~entry->value : entry->value; > + > + return is_lock_held(entry) ? entry->value : ~entry->value; > +} > + > /* new kernel perf_sample_data definition */ > struct perf_sample_data___new { > __u64 sample_flags; > @@ -175,6 +429,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, > return bpf_get_current_uid_gid() & 0xFFFFFFFF; > case PBF_TERM_GID: > return bpf_get_current_uid_gid() >> 32; > + case PBF_TERM_LOCK_HELD: > + return check_lock_held(entry); > case PBF_TERM_NONE: > case __PBF_UNUSED_TERM4: > case __PBF_UNUSED_TERM5: > @@ -209,6 +465,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry > return bpf_get_current_uid_gid() & 0xFFFFFFFF; > case PBF_TERM_GID: > return bpf_get_current_uid_gid() >> 32; > + case PBF_TERM_LOCK_HELD: > + return check_lock_held(entry); > default: > return 0; > } > @@ -390,4 +648,760 @@ int perf_tp_sample_filter(void *ctx) > return 0; > } > > +static __always_inline __u32 get_lock_owner_id(__u32 flags) > +{ > + __u32 owner_id; > + > + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) > + return 0x80000000U | bpf_get_smp_processor_id(); > + > + owner_id = (__u32)bpf_get_current_pid_tgid(); > + if (!owner_id) > + owner_id = 0x80000000U | bpf_get_smp_processor_id(); > + return owner_id; > +} > + > +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, > + __u32 kind, __u32 flags, bool acquired) > +{ > + struct held_lock_key key = { > + .addr_or_id = addr_or_id, > + .owner_id = owner_id, > + .kind = kind, > + }; > + struct held_lock_val *val; > + bool is_read = (flags & LCB_F_READ) != 0; > + > + val = bpf_map_lookup_elem(&held_locks, &key); > + if (acquired) { > + if (val) { > + if (is_read) > + __sync_fetch_and_add(&val->read_count, 1); > + else > + __sync_fetch_and_add(&val->write_count, 1); > + } else { > + struct held_lock_val first = { > + .read_count = is_read ? 1 : 0, > + .write_count = is_read ? 0 : 1, > + .flags = flags, > + }; > + > + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); > + } > + } else if (val) { > + if (is_read) { > + if (val->read_count <= 1 && val->write_count == 0) > + bpf_map_delete_elem(&held_locks, &key); > + else if (val->read_count > 0) > + __sync_fetch_and_add(&val->read_count, -1); > + } else { > + if (val->write_count <= 1 && val->read_count == 0) > + bpf_map_delete_elem(&held_locks, &key); > + else if (val->write_count > 0) > + __sync_fetch_and_add(&val->write_count, -1); > + } > + } > +} > + > +#define LCB_F_READ_IRQSAVE (1U << 30) > +#define LCB_F_READ_IRQ_BH (1U << 31) > + > +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) > +{ > + struct held_lock_key addr_key = { > + .addr_or_id = lock_addr, > + .owner_id = owner_id, > + .kind = PBF_LOCK_ADDR, > + }; > + struct held_lock_val *existing; > + bool matched = false; > + bool is_read = (flags & LCB_F_READ) != 0; > + __u32 class_id = 0; > + __u64 kmem_cache_addr = 0; > + __u32 type_flag = flags & LCB_F_TYPE_MASK; > + bool match_type = false; > + > + if (!lock_enabled || !lock_addr) > + return 0; > + > + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) > + matched = true; > + > + if (has_mmap_lock || has_siglock) { > + __u32 ltype = check_lock_type(lock_addr, type_flag); > + > + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || > + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) > + matched = true; > + } > + > + if (has_lock_class && type_flag == LCB_F_SPIN) { > + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); > + > + if (cid) { > + class_id = *cid; > + matched = true; > + } > + } > + > + if (has_lock_slab && bpf_get_kmem_cache) { > + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); > + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) > + matched = true; > + else > + kmem_cache_addr = 0; > + } > + > + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { > + match_type = true; > + matched = true; > + } > + > + if (!matched) > + return 0; > + > + existing = bpf_map_lookup_elem(&held_locks, &addr_key); > + if (existing) { > + /* > + * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC > + * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call > + * the out-of-line _raw_read_lock_irqsave(). Clear the marker > + * and skip the duplicate outer fexit increment. > + */ > + if ((flags & LCB_F_READ_IRQ_BH) && > + (existing->flags & LCB_F_READ_IRQSAVE)) { > + existing->flags &= ~LCB_F_READ_IRQSAVE; > + return 0; > + } > + /* Non-reader locks are not recursive; avoid double-counting */ > + if (!is_read && existing->write_count > 0) > + return 0; > + if (is_read) { > + __sync_fetch_and_add(&existing->read_count, 1); > + if (flags & LCB_F_READ_IRQSAVE) > + existing->flags |= LCB_F_READ_IRQSAVE; > + else > + existing->flags &= ~LCB_F_READ_IRQSAVE; > + } else { > + __sync_fetch_and_add(&existing->write_count, 1); > + } > + } else { > + struct held_lock_val first = { > + .read_count = is_read ? 1 : 0, > + .write_count = is_read ? 0 : 1, > + .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE), > + }; > + > + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) > + return 0; > + } > + > + if (class_id) > + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true); > + if (kmem_cache_addr) { > + update_held_lock_count((__u64)kmem_cache_addr, owner_id, > + PBF_LOCK_SLAB, type_flag, true); > + } > + if (match_type) > + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true); > + > + return 0; > +} > + > +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) > +{ > + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); > +} > + > +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) > +{ > + __u32 owner_id = get_lock_owner_id(flags); > + struct held_lock_key addr_key = { > + .addr_or_id = lock_addr, > + .owner_id = owner_id, > + .kind = PBF_LOCK_ADDR, > + }; > + struct held_lock_val *existing; > + __u32 type_flag = flags & LCB_F_TYPE_MASK; > + bool is_read = (flags & LCB_F_READ) != 0; > + > + if (!lock_enabled || !lock_addr) > + return 0; > + > + existing = bpf_map_lookup_elem(&held_locks, &addr_key); > + if (!existing) > + return 0; > + > + if (is_read) { > + if (existing->read_count == 0) > + return 0; > + if (existing->read_count == 1 && existing->write_count == 0) { > + bpf_map_delete_elem(&held_locks, &addr_key); > + } else { > + existing->flags &= ~LCB_F_READ_IRQSAVE; > + __sync_fetch_and_add(&existing->read_count, -1); > + } > + } else { > + if (existing->write_count == 0) > + return 0; > + if (existing->read_count == 0) > + bpf_map_delete_elem(&held_locks, &addr_key); > + else > + existing->write_count = 0; > + } > + > + if (has_lock_class && flags == LCB_F_SPIN) { > + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); > + > + if (cid) > + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); > + } > + > + if (has_lock_slab && bpf_get_kmem_cache) { > + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); > + > + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { > + update_held_lock_count(kmem_cache_addr, owner_id, > + PBF_LOCK_SLAB, flags, false); > + } > + } > + > + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) > + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); > + > + return 0; > +} > + > +static __always_inline int record_lock_released(void *ctx, __u32 flags) > +{ > + return record_lock_released_addr(get_lock_addr(ctx), flags); > +} > + > +/* Mutex acquire/release hooks */ > +SEC("fexit/mutex_lock") > +int BPF_PROG(fexit_mutex_lock, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_MUTEX); > +} > + > +SEC("fexit/mutex_lock_nested") > +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) > +{ > + return record_lock_acquired(ctx, LCB_F_MUTEX); > +} > + > +SEC("fexit/_mutex_lock_nest_lock") > +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock) > +{ > + return record_lock_acquired(ctx, LCB_F_MUTEX); > +} > + > +SEC("fexit/mutex_lock_io") > +int BPF_PROG(fexit_mutex_lock_io, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_MUTEX); > +} > + > +SEC("fexit/mutex_lock_io_nested") > +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) > +{ > + return record_lock_acquired(ctx, LCB_F_MUTEX); > +} > + > +SEC("fexit/mutex_lock_interruptible") > +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_MUTEX); > + return 0; > +} > + > +SEC("fexit/mutex_lock_interruptible_nested") > +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_MUTEX); > + return 0; > +} > + > +SEC("fexit/mutex_lock_killable") > +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_MUTEX); > + return 0; > +} > + > +SEC("fexit/_mutex_lock_killable") > +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass, > + void *nest_lock, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_MUTEX); > + return 0; > +} > + > +SEC("fexit/mutex_trylock") > +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_MUTEX); > + return 0; > +} > + > +SEC("fexit/_mutex_trylock_nest_lock") > +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_MUTEX); > + return 0; > +} > + > +SEC("fentry/mutex_unlock") > +int BPF_PROG(fentry_mutex_unlock, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_MUTEX); > +} > + > +/* RW semaphore acquire/release hooks */ > +SEC("fexit/down_read") > +int BPF_PROG(fexit_down_read, void *sem) > +{ > + return record_lock_acquired(ctx, LCB_F_READ); > +} > + > +SEC("fexit/down_read_nested") > +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) > +{ > + return record_lock_acquired(ctx, LCB_F_READ); > +} > + > +SEC("fexit/down_read_interruptible") > +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_READ); > + return 0; > +} > + > +SEC("fexit/down_read_killable") > +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_READ); > + return 0; > +} > + > +SEC("fexit/down_read_killable_nested") > +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_READ); > + return 0; > +} > + > +SEC("fexit/down_read_trylock") > +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_READ); > + return 0; > +} > + > +SEC("fexit/down_write") > +int BPF_PROG(fexit_down_write, void *sem) > +{ > + return record_lock_acquired(ctx, LCB_F_WRITE); > +} > + > +SEC("fexit/down_write_nested") > +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) > +{ > + return record_lock_acquired(ctx, LCB_F_WRITE); > +} > + > +SEC("fexit/_down_write_nest_lock") > +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock) > +{ > + return record_lock_acquired(ctx, LCB_F_WRITE); > +} > + > +SEC("fexit/down_write_killable") > +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_WRITE); > + return 0; > +} > + > +SEC("fexit/down_write_killable_nested") > +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_WRITE); > + return 0; > +} > + > +SEC("fexit/down_write_trylock") > +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_WRITE); > + return 0; > +} > + > +SEC("fexit/downgrade_write") > +int BPF_PROG(fexit_downgrade_write, void *sem) > +{ > + if (track_rwsem_read) > + record_lock_acquired(ctx, LCB_F_READ); > + return record_lock_released(ctx, LCB_F_WRITE); > +} > + > +SEC("fentry/up_read") > +int BPF_PROG(fentry_up_read, void *sem) > +{ > + return record_lock_released(ctx, LCB_F_READ); > +} > + > +SEC("fentry/up_write") > +int BPF_PROG(fentry_up_write, void *sem) > +{ > + return record_lock_released(ctx, LCB_F_WRITE); > +} > + > +/* Spinlock acquire/release hooks */ > +SEC("fexit/_raw_spin_lock") > +int BPF_PROG(fexit_raw_spin_lock, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_lock_nested") > +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_lock_nest_lock") > +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_lock_irq") > +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_lock_irq_disable") > +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_lock_irqsave") > +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_lock_irqsave_nested") > +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_lock_bh") > +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN); > +} > + > +SEC("fexit/_raw_spin_trylock") > +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_SPIN); > + return 0; > +} > + > +SEC("fexit/_raw_spin_trylock_bh") > +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_SPIN); > + return 0; > +} > + > +SEC("fentry/_raw_spin_unlock") > +int BPF_PROG(fentry_raw_spin_unlock, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN); > +} > + > +SEC("fentry/_raw_spin_unlock_irq") > +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN); > +} > + > +SEC("fentry/_raw_spin_unlock_irq_enable") > +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN); > +} > + > +SEC("fentry/_raw_spin_unlock_irqrestore") > +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN); > +} > + > +SEC("fentry/_raw_spin_unlock_bh") > +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN); > +} > + > +/* RWlock acquire/release hooks */ > +SEC("fexit/_raw_read_lock") > +int BPF_PROG(fexit_raw_read_lock, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fexit/_raw_read_lock_irq") > +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); > +} > + > +SEC("fexit/_raw_read_lock_irqsave") > +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE); > +} > + > +SEC("fexit/_raw_read_lock_bh") > +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); > +} > + > +SEC("fexit/_raw_read_trylock") > +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); > + return 0; > +} > + > +SEC("fentry/_raw_read_unlock") > +int BPF_PROG(fentry_raw_read_unlock, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fentry/_raw_read_unlock_irq") > +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fentry/_raw_read_unlock_irqrestore") > +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fentry/_raw_read_unlock_bh") > +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fexit/_raw_write_lock") > +int BPF_PROG(fexit_raw_write_lock, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fexit/_raw_write_lock_nested") > +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fexit/_raw_write_lock_irq") > +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fexit/_raw_write_lock_irqsave") > +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fexit/_raw_write_lock_bh") > +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fexit/_raw_write_trylock") > +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); > + return 0; > +} > + > +SEC("fentry/_raw_write_unlock") > +int BPF_PROG(fentry_raw_write_unlock, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fentry/_raw_write_unlock_irq") > +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fentry/_raw_write_unlock_irqrestore") > +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +SEC("fentry/_raw_write_unlock_bh") > +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); > +} > + > +/* RT-Mutex acquire/release hooks */ > +SEC("fexit/rt_mutex_lock") > +int BPF_PROG(fexit_rt_mutex_lock, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_RT); > +} > + > +SEC("fexit/rt_mutex_lock_nested") > +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) > +{ > + return record_lock_acquired(ctx, LCB_F_RT); > +} > + > +SEC("fexit/_rt_mutex_lock_nest_lock") > +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock) > +{ > + return record_lock_acquired(ctx, LCB_F_RT); > +} > + > +SEC("fexit/rt_mutex_lock_interruptible") > +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_RT); > + return 0; > +} > + > +SEC("fexit/rt_mutex_lock_killable") > +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, LCB_F_RT); > + return 0; > +} > + > +SEC("fexit/rt_mutex_trylock") > +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) > +{ > + if (ret) > + record_lock_acquired(ctx, LCB_F_RT); > + return 0; > +} > + > +SEC("fentry/rt_mutex_unlock") > +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) > +{ > + return record_lock_released(ctx, LCB_F_RT); > +} > + > +/* Semaphore acquire/release hooks */ > +SEC("fexit/down") > +int BPF_PROG(fexit_down, void *sem) > +{ > + return record_lock_acquired(ctx, 0); > +} > + > +SEC("fexit/down_interruptible") > +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, 0); > + return 0; > +} > + > +SEC("fexit/down_killable") > +int BPF_PROG(fexit_down_killable, void *sem, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, 0); > + return 0; > +} > + > +SEC("fexit/down_timeout") > +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, 0); > + return 0; > +} > + > +SEC("fexit/down_trylock") > +int BPF_PROG(fexit_down_trylock, void *sem, int ret) > +{ > + if (ret == 0) > + record_lock_acquired(ctx, 0); > + return 0; > +} > + > +SEC("fentry/up") > +int BPF_PROG(fentry_up, void *sem) > +{ > + return record_lock_released(ctx, 0); > +} > + > +/* Percpu-rwsem write acquire/release hooks */ > +SEC("fexit/percpu_down_write") > +int BPF_PROG(fexit_percpu_down_write, void *sem) > +{ > + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); > +} > + > +SEC("fentry/percpu_up_write") > +int BPF_PROG(fentry_percpu_up_write, void *sem) > +{ > + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); > +} > + > +SEC("raw_tp/bpf_test_finish") > +int BPF_PROG(collect_lock_syms) > +{ > + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, > + node_data_addr, nr_nodes, sizeof_zone); > +} > + > +SEC("iter/kmem_cache") > +int slab_cache_iter(void *ctx) > +{ > + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); > +} > + > char LICENSE[] SEC("license") = "Dual BSD/GPL"; > diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c > index 2f3317ce17f0..1fcb424b75c0 100644 > --- a/tools/perf/util/parse-events.c > +++ b/tools/perf/util/parse-events.c > @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, > /* Will a tracepoint filter work for str or should a BPF filter be used? */ > static bool is_possible_tp_filter(const char *str) > { > - return strstr(str, "uid") == NULL; > + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && > + strstr(str, "lock_held") == NULL; > } > > static int set_filter(struct evsel *evsel, const void *arg) > -- > 2.56.0.212.g129ede20d9-goog > ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 6/8] perf test: Add tests for lock_held() BPF sample filter 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (4 preceding siblings ...) 2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers ` (2 subsequent siblings) 8 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 +++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 122 ++++++++++++++++++++ 2 files changed, 230 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..ebaf8ee41622 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,124 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed ${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&dentry)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &dentry]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required" + then + echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]" + err=1 + return + fi + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock:R) || lock_held(mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock:R/:W]" + err=1 + return + fi + + if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc + then + if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]" + err=1 + return + fi + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +316,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (5 preceding siblings ...) 2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 8 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 44 ++-- tools/perf/tests/parse-events.c | 13 + tools/perf/tests/shell/record_bpf_filter.sh | 20 ++ tools/perf/util/bpf-filter.c | 243 ++++++++++++++++++- tools/perf/util/bpf_skel/lock_data.h | 47 +++- tools/perf/util/bpf_skel/sample-filter.h | 14 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 136 ++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 493 insertions(+), 34 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 8496b20987a0..3eaebdc9efc2 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -219,11 +219,17 @@ OPTIONS 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock', - 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex', - 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', - 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). - Both read/shared and write/exclusive tracking are supported for - 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock' + 'siglock', 'rq_lock', 'zone_lock', 'rcu', 'rcu-trace'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', 'rcu:W', 'semaphore'). Both + read/shared and write/exclusive tracking are supported for 'rwsem' + (including 'mmap_lock'), 'rwlock', 'rcu', and 'srcu' (where + read/shared mode matches 'rcu_read_lock()' / 'srcu_read_lock()' + critical sections and write/exclusive mode matches grace-period + waits 'synchronize_rcu*()', 'synchronize_srcu*()', 'rcu_barrier()', + and 'srcu_barrier()'); 'rcu-trace' ('rcu_read_lock_trace()') + supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'semaphore' tracking @@ -245,19 +251,21 @@ OPTIONS --filter 'lock_held(mutex) || lock_held(rwsem)' When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol - ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' - or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's - 'owner' field directly in kernel memory on each sample without attaching - any BPF fentry/fexit hooks. For locks that do not maintain an - authoritative per-task owner pointer (such as spinlocks, rwlocks, - semaphores, reader-locked rwsems, and slab/type filters), it attaches - BPF fentry/fexit programs to kernel lock acquire and release functions - to track held locks in a BPF map. These BPF hooks run on every kernel - lock acquisition and release for the attached lock family, even when no - perf sample is being taken. To minimize this overhead, only the hooks - for the relevant lock family and mode are attached when known (for - example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions - and 'rq_lock' only hooks spinlock functions), whereas slab filters + ('*_rt_mutex'), write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' + or 'lock_held(mmap_lock:W)'), or 'rcu:R' / 'rcu-trace' (which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'), + 'perf_sample_filter' inspects the kernel fields directly on each sample + without attaching any BPF fentry/fexit hooks. For locks that do not + maintain an authoritative per-task owner pointer (such as spinlocks, + rwlocks, semaphores, reader-locked rwsems, SRCU locks, RCU synchronize + waits, and slab/type filters), it attaches BPF fentry/fexit programs to + kernel lock acquire and release functions to track held locks in a BPF + map. These BPF hooks run on every kernel lock acquisition and release + for the attached lock family, even when no perf sample is being taken. + To minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(mmap_lock)' + only hooks rwsem read functions, 'srcu' only hooks SRCU functions, and + 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index ebaf8ee41622..47f697f53a85 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -247,6 +247,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index d5a85ee98d5f..08a6e49ca16e 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -201,11 +201,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU & ~LOCK_PROG_SEM, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -325,6 +333,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -612,6 +646,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) supported |= LOCK_PROG_PERCPU_RWSEM; + if ((btf_has_func(btf, "__srcu_read_lock") || + btf_has_func(btf, "__srcu_read_lock_nmisafe")) && + (btf_has_func(btf, "__srcu_read_unlock") || + btf_has_func(btf, "__srcu_read_unlock_nmisafe"))) + supported |= LOCK_PROG_SRCU_READ; + + if (btf_has_func(btf, "synchronize_srcu")) + supported |= LOCK_PROG_SRCU_WRITE; + + if (btf_has_func(btf, "synchronize_rcu")) + supported |= LOCK_PROG_RCU_WRITE; + return supported; } @@ -731,6 +777,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RCU_WRITE; + *required_groups |= LOCK_PROG_RCU_WRITE; + } + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { *prog_groups |= groups; *required_groups |= groups; @@ -772,6 +831,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, groups |= LOCK_PROG_RWLOCK_WRITE; *prog_groups |= groups; *required_groups |= groups; + } else if (strtailcmp(base, "srcu") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_SRCU_WRITE; + *prog_groups |= groups; + *required_groups |= groups; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -920,6 +986,108 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr, return -EINVAL; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + /* + * Non-sleepable BPF sample filter programs always run under at least + * one kernel rcu_read_lock(): + * - PERF_TYPE_PERF_EVENT: bpf_overflow_handler() takes rcu_read_lock() + * (and perf_swevent_event() takes an outer rcu_read_lock() for + * non-timer PERF_TYPE_SOFTWARE events, giving depth 2). + * - Standard tracepoints: trace_call_bpf() takes rcu_read_lock(). + * - Faultable syscall tracepoints: trace_call_bpf_faultable() invokes + * bpf_prog_run_array_sleepable(), which takes guard(rcu)() for + * non-sleepable programs (!prog->sleepable) such as + * perf_sample_filter_tp. + */ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static u64 rcu_trace_min_depth(struct evsel *evsel, struct btf *btf) +{ + const char *ev_name; + + /* + * Only PERF_TYPE_PERF_EVENT and PERF_TYPE_TRACEPOINT events can attach + * sample_filter.bpf.c (kprobes and uprobes require BPF_PROG_TYPE_KPROBE + * in __perf_event_set_bpf_prog()). + * + * For syscall tracepoints: + * - raw_syscalls:sys_enter and raw_syscalls:sys_exit are defined with + * TRACE_EVENT_SYSCALL(), whose __do_trace_sys_enter/exit wrapper + * takes guard(rcu_tasks_trace)() (depth 1) before perf_trace_buf + * submission calls trace_call_bpf(). + * - syscalls:sys_enter_* and syscalls:sys_exit_* register + * perf_syscall_enter/exit on the raw sys_enter/sys_exit tracepoints + * (1st guard(rcu_tasks_trace)()) and, on kernels with + * trace_call_bpf_faultable(), acquire a 2nd guard(rcu_tasks_trace)() + * in trace_call_bpf_faultable() (depth 2). + */ + if (evsel && evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { + ev_name = evsel__name(evsel); + if (!ev_name) + return 0; + if (strstarts(ev_name, "raw_syscalls:")) + return 1; + if (strstarts(ev_name, "syscalls:")) { + if (!btf || + btf__find_by_name_kind(btf, "trace_call_bpf_faultable", + BTF_KIND_FUNC) > 0) + return 2; + return 1; + } + } + return 0; +} + +static bool btf_struct_has_field(const struct btf *btf, s32 type_id, + const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + u16 i, vlen; + + type_id = btf__resolve_type(btf, type_id); + if (type_id < 0) + return false; + + t = btf__type_by_id(btf, type_id); + if (!t || (!btf_is_struct(t) && !btf_is_union(t))) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && *name) { + if (!strcmp(name, field_name)) + return true; + } else if (btf_struct_has_field(btf, m->type, field_name)) { + return true; + } + } + return false; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + s32 id; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + return btf_struct_has_field(btf, id, field_name); +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->op`, * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`, @@ -927,15 +1095,15 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr, * * Resolution order: * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`). - * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock`. + * 2. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`. * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, - * `rwlock`, `percpu-rwsem`, `semaphore`, etc. + * `rwlock`, `percpu-rwsem`, `srcu`, `semaphore`, etc. * 4. Hexadecimal lock address. * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -1017,6 +1185,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->value = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->value = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->value = rcu_trace_min_depth(evsel, btf); + return 0; + } + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->value = encoded_pair; @@ -1086,7 +1296,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, } static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, - struct perf_bpf_filter_entry *entry) + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -1103,7 +1313,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].term = expr->term; entry[i].value = expr->val; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -1121,7 +1331,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].term = group->term; entry[i].value = group->val; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1389,7 +1599,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; - ret = get_filter_entries(evsel, NULL, entry); + ret = get_filter_entries(evsel, NULL, NULL, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; @@ -1450,6 +1660,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0) + rcu_addr += 1; + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, needs_lock_syms, needs_slab_iter, is_tracepoint); @@ -1468,7 +1689,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (has_lock_filter && needs_slab_iter) run_slab_cache_iter(skel); - ret = get_filter_entries(evsel, skel, entry); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 317d8ccae63e..12bed223f9a5 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -255,6 +258,43 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). `min_depth` accounts for any baseline nesting + * contributed by the kernel tracing wrapper itself (2 on faultable + * `syscalls:*` tracepoints via `__DECLARE_TRACE_SYSCALL` and + * `trace_call_bpf_faultable()`, 1 on `raw_syscalls:*`, 0 otherwise). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > min_depth; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -287,6 +327,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index a65ee780d64d..7abc530ed50d 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -101,6 +101,20 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.value`, + * where `entry.value` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > entry.value`, + * where `entry.value` is the baseline nesting depth added by the + * kernel event tracing path. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index d1192a93179e..60fe4c3a99a2 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -129,6 +129,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -142,6 +143,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; int track_rwsem_read; __u8 lock_type_filter[LCB_F_TYPE_MAX]; @@ -282,6 +284,21 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0; + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->value)) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return true; + return false; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_trace_lock_held(curr, (int)entry->value)) + return true; + return false; + default: return false; } @@ -727,11 +744,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, type_flag); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1391,6 +1409,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index f6e68d740860..37415f7b7f2c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -39,6 +39,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (6 preceding siblings ...) 2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-20 20:58 ` Ian Rogers 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 8 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 171 ++++++++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 221 ++++++++++++++---- tools/perf/util/machine.c | 11 + 5 files changed, 362 insertions(+), 62 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..999dcfd1ef26 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..dfb384f1c5e6 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,7 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +181,133 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1; + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +315,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +329,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +379,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +403,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +439,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +456,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +639,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +747,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0f2118006005..26936f329dba 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -172,6 +185,7 @@ const volatile int aggr_mode; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -559,8 +573,21 @@ int contention_begin(u64 *ctx) } } skip_owner: - pelem->stack_id = bpf_get_stackid(ctx, &stacks, - BPF_F_FAST_STACK_CMP | stack_skip); + { + int skip = stack_skip; + + /* + * Unlike tp_btf/contention_begin (which fires inside a + * lock slowpath via trace_contention_begin), fentry + * RCU/SRCU programs attach directly to the top-level + * wait API, which is 2 frames shallower. + */ + if (flags & (LCB_F_RCU | LCB_F_SRCU)) + skip = stack_skip > 2 ? stack_skip - 2 : 0; + + pelem->stack_id = bpf_get_stackid(ctx, &stacks, + BPF_F_FAST_STACK_CMP | skip); + } if (pelem->stack_id < 0) __sync_fetch_and_add(&stack_fail, 1); } else if (aggr_mode == LOCK_AGGR_TASK) { @@ -588,8 +615,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -597,30 +623,34 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; /* - * For spinlock and rwlock, it needs to get the timestamp for the - * per-cpu map. However, contention_end does not have the flags - * so it cannot know whether it reads percpu or hash map. - * - * Try per-cpu map first and check if there's active contention. - * If it is, do not read hash map because it cannot go to sleeping - * locks before releasing the spinning locks. + * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu + * map. For tp_btf/contention_end, try the per-cpu map first (used by + * spinlocks and rwlocks) and fall back to the per-task tstamp map. */ - pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); - if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) - return 0; - } else { + if (is_rcu) { pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) return 0; - need_delete = true; + ts_map = &tstamp_rcu; + } else { + pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); + if (pelem && pelem->lock) { + if (pelem->lock != lock) + return 0; + } else { + pid = bpf_get_current_pid_tgid(); + pelem = bpf_map_lookup_elem(&tstamp, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp; + } } timestamp = bpf_ktime_get_ns(); @@ -648,7 +678,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -713,7 +742,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -795,11 +824,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(end_timestamp) { diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index 1d9d3bf57720..0a0f223b3335 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr) return true; } + { + struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL); + + if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) || + !strncmp(sym->name, "synchronize_srcu", 16) || + !strcmp(sym->name, "__synchronize_srcu") || + !strcmp(sym->name, "rcu_barrier") || + !strcmp(sym->name, "srcu_barrier"))) + return true; + } + return false; } -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (7 preceding siblings ...) 2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers ` (10 more replies) 8 siblings, 11 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu This series adds a new BPF sample filter predicate, `lock_held(<lock>)` (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` mode suffixes), to `perf record --filter`, allowing perf event samples to be recorded only when a specified kernel lock is held (or not held with `!`) by the running kernel thread or CPU. It also fixes a `bpf_link` memory leak in `perf_bpf_filter__prepare()`, enables BPF sample filters on tracepoint events (`PERF_TYPE_TRACEPOINT`), and extends both `lock_held()` and `perf lock contention --use-bpf` to support RCU and SRCU. For example: # perf record -a -g -e cycles --filter 'lock_held(cgroup_mutex)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' -- ... # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... # perf lock contention -a -b -Y rcu,srcu -- sleep 1 An important use-case is detecting memory allocations or direct memory reclaim occurring while kernel locks are held, which can stall lock waiters for extended periods under memory pressure: # Direct memory reclaim triggered while holding any mutex or rwsem: # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM) # while holding a mutex or rwsem: # perf record -a -g -e kmem:mm_page_alloc \ --filter 'gfp_flags & 0x400' \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... # Major page faults occurring while holding a mutex or rwsem: # perf record -a -g -e major-faults \ --filter 'lock_held(mutex) || lock_held(rwsem)' -- ... Supported `<lock>` specifications match `perf lock contention`: - Global/static kernel lock symbols (e.g. `lock_held(cgroup_mutex)`) - Hexadecimal kernel lock addresses - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, `lock_held_read(&dentry)`) - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, `rcu:W`, `semaphore` Limitations and weaknesses in determining whether a lock is held: - Inlined lock/unlock operations: - When the kernel is configured to inline spinlock or rwlock operations (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and have no BTF function entry points for `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that both acquire and release entry points exist in BTF before enabling a lock family. - Fast-path `percpu_down_read()` / `percpu_up_read()` and `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined per-CPU counter increments without function calls, so only `percpu_down_write()` / `percpu_up_write()` and out-of-line `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to `preempt_disable()` and does not maintain `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` read-side inspection requires `CONFIG_PREEMPT_RCU`. - Lack of an inspectable owner field on non-exclusive / spinning locks: - While `struct mutex`, `struct rt_mutex`, and write-locked `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` (allowing `perf_sample_filter` to inspect kernel memory directly on sample overflow when filtering by lock address without attaching any `fentry`/`fexit` hooks for `*_mutex`, `*_rt_mutex`, and `mmap_lock:W`), spinlocks, rwlocks, semaphores, SRCU, and reader-locked `rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task owner pointer. - Tracking these locks (as well as slab-cache and lock-type filters) relies on `fexit` (acquire) and `fentry` (release) BPF hooks maintaining the `held_locks` BPF LRU hash map: - Any such lock already held before `perf record` attaches its BPF programs will not be present in `held_locks` until it is released and re-acquired. - The attached `fentry`/`fexit` hooks execute on every kernel lock acquire and release for the hooked lock family across the system, adding tracing overhead outside of perf sample handling. - Cross-task semaphore signaling (`down*()` / `up()`): - Counting `struct semaphore`s do not enforce same-task ownership in the kernel: when a semaphore is used as a completion/signaling mechanism (`down()` in one task woken by `up()` in another task or interrupt handler, which proposed kernel dependency trackers like `CONFIG_DEPT` model as wait/event dependencies bounded by execution context rather than lock ownership), `fexit/down` and `fentry/up` do not pair on the same task ID. Semaphore hooks are therefore only enabled when `semaphore` or a `*_sem` symbol (such as `console_sem`) is explicitly specified, and `held_locks` uses `BPF_MAP_TYPE_LRU_HASH` so any unpaired cross-task entries are automatically evicted. - Shared `trace_event_call->prog_array` on `PERF_TYPE_TRACEPOINT`: - In `kernel/events/core.c`, `__perf_event_set_bpf_prog()` requires `BPF_PROG_TYPE_TRACEPOINT` for `PERF_TYPE_TRACEPOINT` events and attaches the program to `event->tp_event->prog_array` on the shared `trace_event_call`. Because `perf_trace_run_bpf_submit()` executes `trace_call_bpf()` before `perf_tp_event()`, returning 0 from `perf_tp_sample_filter` filters that tracepoint invocation before `perf_tp_event()` iterates `call->perf_events` (affecting concurrent `perf_event` listeners on the same tracepoint while the privileged `perf record` session is active). Patch overview (patches 1-2 are standalone fixes independent of the rest of the series): 1. "perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF": Attaches `perf_sample_filter` directly via `PERF_EVENT_IOC_SET_BPF` in `perf_bpf_filter__prepare()`, matching the pinned filter path and avoiding leaking `struct bpf_link_perf` or double-closing `FD(evsel, x, y)`. 2. "perf bpf-filter: Support BPF sample filters on tracepoint events": Adds `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF sample filters can attach to `PERF_TYPE_TRACEPOINT` events (where `__perf_event_set_bpf_prog()` in `kernel/events/core.c` requires `BPF_PROG_TYPE_TRACEPOINT` rather than `BPF_PROG_TYPE_PERF_EVENT`). 3. "perf lock contention: Share common BPF and lock type definitions in lock_data.h": Pure refactoring moving `lock_type_table` to `util/lock-contention.c` and `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers into `tools/perf/util/bpf_skel/lock_data.h`. 4. "perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff": Extends `get_lock_owner()` in `lock_data.h` to support `rt_mutex` (`LCB_F_RT`), `CONFIG_PREEMPT_RT` mutexes, and `MUTEX_FLAG_HANDOFF`. 5. "perf bpf-filter: Add lock_held() sample filter for global mutexes": Adds the `lock_held()`, `lock_held_read()`/`lock_held_shared()`, `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` filter syntax and evaluates global/static kernel mutexes via direct `lock->owner` inspection. 6. "perf bpf-filter: Extend lock_held() to other kernel lock types": Extends `lock_held()` to spinlocks, rwsems, rwlocks, rt-mutexes, percpu-rwsems, and semaphores (as well as hex addresses and lock type filters) using a `held_locks` BPF LRU hash map updated by selectively attached `fentry`/`fexit` hooks. 7. "perf bpf-filter: Support per-process and semi-global locks in lock_held()": Adds support for `mmap_lock`, `siglock`, `rq_lock`, and `zone_lock`. 8. "perf bpf-filter: Support slab cache locks (&<name>) in lock_held()": Adds support for slab cache locks prefixed with `&` (e.g. `lock_held(&dentry)`) using the `kmem_cache` BPF iterator and `slab_caches` map. 9. "perf test: Add tests for lock_held() BPF sample filter": Adds unit tests in `tests/parse-events.c` and shell tests in `tests/shell/record_bpf_filter.sh`. 10. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU": Supports `rcu` (`rcu:R` via direct `current->rcu_read_lock_nesting` inspection with zero tracing hooks; `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` via `synchronize_srcu*()` / `srcu_barrier()`). 11. "perf lock contention: Extend BPF lock contention to RCU and SRCU": Extends `perf lock contention --use-bpf` to track RCU and SRCU grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). Changes in v6: - Drop "perf lock contention: Filter events before creating lock_stat" as Arnaldo cherry-picked it. - Add an inline comment above `pin_filter_object()` in `util/bpf-filter.c` in patch 2 explaining why `bpf_object__pin()` / `bpf_object__unpin()` cannot be used when programs in `sample_filter.bpf.c` have `autoload` disabled (Namhyung). - Split the `lock_data.h` refactoring into a pure code-movement commit (patch 3) and a separate commit extending `get_lock_owner()` for `rt_mutex` and `MUTEX_FLAG_HANDOFF` (patch 4), keep `lock_syms` in its original location in `lock_contention.bpf.c`, and add a comment above `#ifdef __BPF__` in `lock_data.h` explaining that the header is shared between userspace and BPF programs (Namhyung). - Split the `lock_held()` implementation into 4 incremental commits (patches 5-8): (5) core parser, expression evaluation, and global mutex support; (6) other kernel lock types (`spinlock`, `rwsem`, `rwlock`, `rt-mutex`, `percpu-rwsem`, `semaphore`), hex addresses, and lock type filters; (7) per-process (`mmap_lock`, `siglock`) and semi-global (`rq_lock`, `zone_lock`) locks; and (8) slab cache locks (`&<name>`) (Namhyung). - Update `is_possible_tp_filter()` in `util/parse-events.c` in patch 2 to check all tracepoint BPF sample filter terms enabled by that patch (`uid`, `gid`, `tid`, `pid`, `cpu`, `time`, `cgroup`) using word-boundary matching (`has_filter_word()`) and `!evsel__field()` so they are routed to `evsel->bpf_filters` when not matching a tracepoint format field, and extend `bpf_tp_terms[]` with `lock_held*` in patch 5. - Replace `strtailcmp(base, ...)` with `str_ends_with(base, ...)` in `prescan_one_lock_expr()` and `resolve_lock_expr()` in `util/bpf-filter.c` (patches 5, 6, and 10) so lock symbol names shorter than a tested suffix (such as `lock` vs `spinlock`, or `u_rwsem` vs `percpu_rwsem`) are not misclassified. - Use a CO-RE flavor struct `struct rt_mutex_base___new` in `lock_data.h` (patch 4) and add explicit standard C header includes (`<errno.h>`, `<stdio.h>`, `<unistd.h>`, `<string.h>`, `<stddef.h>`, `<stdlib.h>`) in patches 1, 2, and 11 for musl libc compatibility. - Reject kernel text functions, module text functions, and absolute or undefined symbols when resolving lock addresses from `/proc/kallsyms`, and check for `__percpu_rwsem_rc_<name>` in BTF so `percpu_rw_semaphore` symbols ending in `_rwsem` (such as `cgroup_threadgroup_rwsem` or `file_rwsem`) do not set `PBF_LOCK_DIRECT_RWSEM` or allow read-only mode. - Link to v5: https://lore.kernel.org/linux-perf-users/20261020205859.1080743-1-irogers@google.com/ Changes in v5: - Add patch 2 ("perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF") attaching BPF sample filters via `ioctl(..., PERF_EVENT_IOC_SET_BPF, fd)` instead of `bpf_program__attach_perf_event_opts()`, fixing the `bpf_link` memory leak without double-closing `FD(evsel, x, y)`, and use the same attachment in patch 3 for tracepoint sample filters (Namhyung). - Expand patch 3's commit message to detail why `pin_filter_object()` and `unpin_filter_object()` replace `bpf_object__pin()` and `bpf_object__unpin()` when `autoload` is disabled on `perf_tp_sample_filter` (Namhyung). - Store the resolved lock target in `perf_bpf_filter_entry.value` (with `op` normalized to `PBF_OP_EQ` or `PBF_OP_NEQ`), keeping `struct perf_bpf_filter_entry` unchanged at 24 bytes for backwards compatibility with pinned `filters` maps. - Add a CO-RE `struct rt_mutex___old` fallback in `get_lock_owner()` in `lock_data.h` for kernels prior to v5.15, and pass baseline nesting depth to `is_rcu_trace_lock_held()` for syscall tracepoints. Changes in v4: - Move the two bugfix commits (`Filter events before creating lock_stat` and `Support BPF sample filters on tracepoint events`) to patches 1 and 2 at the start of the series for easier cherry-picking. - Reserve leading `&` strictly for slab cache locks (e.g. `lock_held(&dentry)`) so global/static lock symbols and well-known locks do not use `&` (e.g. `lock_held(cgroup_mutex)`, `lock_held_read(mmap_lock)`), matching `perf lock contention` (Namhyung). - Skip attaching `fentry`/`fexit` hooks when filtering on a specific `mutex` symbol (`*_mutex`), `rt_mutex` symbol (`*_rt_mutex`), or write-locked `mmap_lock` (`mmap_lock:W` / `lock_held_write(mmap_lock)`), evaluating them purely via direct `owner` inspection (Namhyung). - Use a CO-RE flavor struct `struct mutex___non_rt` in `get_lock_owner()` in `lock_data.h` so `GEN_VMLINUX_H=1` builds succeed on `CONFIG_PREEMPT_RT=y` kernels. - Switch `held_locks` to `BPF_MAP_TYPE_LRU_HASH`, exclude `LOCK_PROG_SEM` from `LOCK_PROG_ALL` unless `semaphore` or a `*_sem` symbol is explicitly requested, and document cross-task semaphore signaling limitations (and comparison with `CONFIG_DEPT`). - Skip `LCB_F_MUTEX | LCB_F_SPIN` in `parse_lock_type_name()`, reject read-only mode on exclusive-only lock symbol suffixes in `resolve_lock_expr()`, and check `needed_groups & supported_groups` before masking `needed_groups` in `configure_lock_progs()`. - Recursively traverse anonymous structs/unions in `btf_has_task_field()` so `lock_held(rcu)` and `lock_held(rcu-trace)` work on `CONFIG_RANDSTRUCT` kernels. Changes in v3: - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c` so `perf lock contention -b -o` still reports reader owners when a writer contends on a reader-owned `rw_semaphore`, and scope `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`, `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`). - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()` when attaching, and document memory allocation / direct reclaim filtering with `lock_held()`. - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data` cannot be resolved and check `node_data_addr != 0` in `do_collect_zone_lock()`. - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call `_raw_read_lock_irqsave()`. - Offset `rcu_state_addr` by 1 byte so it is distinct from `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match timestamp maps. Changes in v2: - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to `sample_filter.bpf.c` and `lock_progs[]`. - Verify in `supported_lock_prog_groups()` that both acquire and all unlock BTF entry points exist for a lock family before enabling its tracing hooks, preventing acquire-only attachment when spinlock/rwlock unlock functions are inlined. - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` on `track_rwsem_read` and verify non-zero read/write counts in `record_lock_released_addr()` before decrementing aggregate slab/type counts. - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in `lock_contention.bpf.c` and recognize `synchronize_rcu*`, `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in `machine__is_lock_function()`. - Add patch fixing `report_lock_contention_begin_event()` to check type, address, and callstack filters before creating `lock_stat` entries. Assisted-by: Antigravity:gemini-3.1-pro Ian Rogers (11): perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF perf bpf-filter: Support BPF sample filters on tracepoint events perf lock contention: Share common BPF and lock type definitions in lock_data.h perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff perf bpf-filter: Add lock_held() sample filter for global mutexes perf bpf-filter: Extend lock_held() to other kernel lock types perf bpf-filter: Support per-process and semi-global locks in lock_held() perf bpf-filter: Support slab cache locks (&<name>) in lock_held() perf test: Add tests for lock_held() BPF sample filter perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU perf lock contention: Extend BPF lock contention to RCU and SRCU tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/Documentation/perf-record.txt | 74 +- tools/perf/builtin-lock.c | 27 - tools/perf/tests/parse-events.c | 121 ++ tools/perf/tests/shell/lock_contention.sh | 14 + tools/perf/tests/shell/record_bpf_filter.sh | 142 ++ tools/perf/util/bpf-filter.c | 1475 ++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 47 +- tools/perf/util/bpf_lock_contention.c | 174 +- .../perf/util/bpf_skel/lock_contention.bpf.c | 431 ++--- tools/perf/util/bpf_skel/lock_data.h | 394 ++++- tools/perf/util/bpf_skel/sample-filter.h | 80 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 1241 ++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + tools/perf/util/lock-contention.c | 23 + tools/perf/util/lock-contention.h | 15 + tools/perf/util/machine.c | 11 + tools/perf/util/parse-events.c | 40 +- 20 files changed, 4008 insertions(+), 359 deletions(-) -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 22:06 ` Namhyung Kim 2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers ` (9 subsequent siblings) 10 siblings, 1 reply; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu In perf_bpf_filter__prepare(), the unpinned filter path attaches skel->progs.perf_sample_filter to each FD(evsel, x, y) using bpf_program__attach_perf_event_opts() and ignores the returned struct bpf_link *, leaking the allocated struct bpf_link_perf and its BPF_PERF_EVENT link fd. Furthermore, bpf_program__attach_perf_event_opts() is designed for callers that transfer ownership of the perf_event fd to the returned bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach() unconditionally disables and closes perf_link->perf_event_fd. Because evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(), destroying the returned bpf_link would double-close the perf_event file descriptors. Fix this by attaching the program directly with ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd), matching the pinned filter path right above it in perf_bpf_filter__prepare(). In the kernel, PERF_EVENT_IOC_SET_BPF takes a reference on the BPF program and _free_event() -> perf_event_free_bpf_prog() automatically detaches and releases it when the perf_event fd is closed. This also avoids enabling the event prematurely, allowing pe_opts (.dont_enable = true) and the LIBBPF_CURRENT_VERSION_GEQ(1, 7) guard to be dropped. Before (built with EXTRA_CFLAGS="-fsanitize=address"): # perf record -e cycles --filter 'period > 1000' -- true [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.024 MB perf.data ] ================================================================= ==51328==ERROR: LeakSanitizer: detected memory leaks Direct leak of 1568 byte(s) in 28 object(s) allocated from: #0 0x7ffbe76ef24f in calloc (/lib64/libasan.so.8+0xef24f) #1 0x625df7 in bpf_program__attach_perf_event_opts (tools/perf/perf+0x625df7) #2 0xbeddc0 in perf_bpf_filter__prepare util/bpf-filter.c:507 #3 0x8f3def in evlist__apply_filters util/evlist.c:1421 #4 0x44bbcf in record__open tools/perf/builtin-record.c:1469 #5 0x453f9e in __cmd_record tools/perf/builtin-record.c:2594 #6 0x45dadb in cmd_record tools/perf/builtin-record.c:4427 #7 0x5df56b in run_builtin tools/perf/perf.c:383 #8 0x5dfa0e in handle_internal_command tools/perf/perf.c:436 #9 0x5dfddc in run_argv tools/perf/perf.c:480 #10 0x5e0333 in main tools/perf/perf.c:581 SUMMARY: AddressSanitizer: 1568 byte(s) leaked in 28 allocation(s). After (built with EXTRA_CFLAGS="-fsanitize=address"): # perf record -e cycles --filter 'period > 1000' -- true [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.024 MB perf.data ] Reported-by: Namhyung Kim <namhyung@kernel.org> Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/ Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/util/bpf-filter.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..506414e5c407 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,6 +40,7 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> #include <stdlib.h> #include <fcntl.h> #include <sys/ioctl.h> @@ -423,16 +424,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; - struct bpf_program *prog; - struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); -#if LIBBPF_CURRENT_VERSION_GEQ(1, 7) - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, - .dont_enable = true); -#else - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); -#endif entry = calloc(MAX_FILTERS, sizeof(*entry)); if (entry == NULL) @@ -501,14 +494,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - prog = skel->progs.perf_sample_filter; + fd = bpf_program__fd(skel->progs.perf_sample_filter); for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), - &pe_opts); - if (IS_ERR(link)) { + ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd); + if (ret < 0) { + ret = -errno; pr_err("Failed to attach perf sample-filter program\n"); - ret = PTR_ERR(link); goto err; } } -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF 2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers @ 2026-10-22 22:06 ` Namhyung Kim 0 siblings, 0 replies; 59+ messages in thread From: Namhyung Kim @ 2026-10-22 22:06 UTC (permalink / raw) To: Ian Rogers Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu On Thu, Oct 22, 2026 at 01:29:53PM -0700, Ian Rogers wrote: > In perf_bpf_filter__prepare(), the unpinned filter path attaches > skel->progs.perf_sample_filter to each FD(evsel, x, y) using > bpf_program__attach_perf_event_opts() and ignores the returned > struct bpf_link *, leaking the allocated struct bpf_link_perf and its > BPF_PERF_EVENT link fd. > > Furthermore, bpf_program__attach_perf_event_opts() is designed for > callers that transfer ownership of the perf_event fd to the returned > bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach() > unconditionally disables and closes perf_link->perf_event_fd. Because > evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(), > destroying the returned bpf_link would double-close the perf_event file > descriptors. > > Fix this by attaching the program directly with > ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd), matching the pinned > filter path right above it in perf_bpf_filter__prepare(). In the > kernel, PERF_EVENT_IOC_SET_BPF takes a reference on the BPF program and > _free_event() -> perf_event_free_bpf_prog() automatically detaches and > releases it when the perf_event fd is closed. This also avoids enabling > the event prematurely, allowing pe_opts (.dont_enable = true) and the > LIBBPF_CURRENT_VERSION_GEQ(1, 7) guard to be dropped. > > Before (built with EXTRA_CFLAGS="-fsanitize=address"): > # perf record -e cycles --filter 'period > 1000' -- true > [ perf record: Woken up 1 times to write data ] > [ perf record: Captured and wrote 0.024 MB perf.data ] > > ================================================================= > ==51328==ERROR: LeakSanitizer: detected memory leaks > > Direct leak of 1568 byte(s) in 28 object(s) allocated from: > #0 0x7ffbe76ef24f in calloc (/lib64/libasan.so.8+0xef24f) > #1 0x625df7 in bpf_program__attach_perf_event_opts (tools/perf/perf+0x625df7) > #2 0xbeddc0 in perf_bpf_filter__prepare util/bpf-filter.c:507 > #3 0x8f3def in evlist__apply_filters util/evlist.c:1421 > #4 0x44bbcf in record__open tools/perf/builtin-record.c:1469 > #5 0x453f9e in __cmd_record tools/perf/builtin-record.c:2594 > #6 0x45dadb in cmd_record tools/perf/builtin-record.c:4427 > #7 0x5df56b in run_builtin tools/perf/perf.c:383 > #8 0x5dfa0e in handle_internal_command tools/perf/perf.c:436 > #9 0x5dfddc in run_argv tools/perf/perf.c:480 > #10 0x5e0333 in main tools/perf/perf.c:581 > > SUMMARY: AddressSanitizer: 1568 byte(s) leaked in 28 allocation(s). > > After (built with EXTRA_CFLAGS="-fsanitize=address"): > # perf record -e cycles --filter 'period > 1000' -- true > [ perf record: Woken up 1 times to write data ] > [ perf record: Captured and wrote 0.024 MB perf.data ] > > Reported-by: Namhyung Kim <namhyung@kernel.org> > Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/ > Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering") > Signed-off-by: Ian Rogers <irogers@google.com> Tested-by: Namhyung Kim <namhyung@kernel.org> Thanks, Namhyung > --- > tools/perf/util/bpf-filter.c | 18 +++++------------- > 1 file changed, 5 insertions(+), 13 deletions(-) > > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c > index 72e83ac1d6a4..506414e5c407 100644 > --- a/tools/perf/util/bpf-filter.c > +++ b/tools/perf/util/bpf-filter.c > @@ -40,6 +40,7 @@ > * The 'dropped' map is to keep how many samples it dropped by the filter and > * it will be reported as lost samples. > */ > +#include <errno.h> > #include <stdlib.h> > #include <fcntl.h> > #include <sys/ioctl.h> > @@ -423,16 +424,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > { > int i, x, y, fd, ret; > struct sample_filter_bpf *skel = NULL; > - struct bpf_program *prog; > - struct bpf_link *link; > struct perf_bpf_filter_entry *entry; > bool needs_idx_hash = !target__has_cpu(target); > -#if LIBBPF_CURRENT_VERSION_GEQ(1, 7) > - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, > - .dont_enable = true); > -#else > - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); > -#endif > > entry = calloc(MAX_FILTERS, sizeof(*entry)); > if (entry == NULL) > @@ -501,14 +494,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) > goto err; > } > > - prog = skel->progs.perf_sample_filter; > + fd = bpf_program__fd(skel->progs.perf_sample_filter); > for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { > for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { > - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y), > - &pe_opts); > - if (IS_ERR(link)) { > + ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd); > + if (ret < 0) { > + ret = -errno; > pr_err("Failed to attach perf sample-filter program\n"); > - ret = PTR_ERR(link); > goto err; > } > } > -- > 2.56.0.307.g5fe7bd680c-goog > ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers ` (8 subsequent siblings) 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper") updated set_filter() to route non-tracepoint filter terms (such as uid and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead of the ftrace tracepoint predicate filter. However, in the kernel's __perf_event_set_bpf_prog() (kernel/events/core.c), attaching a BPF program to a perf_event fd branches on perf_event_is_tracing(event): - Non-tracing events (hardware PMU, software events, breakpoints) call perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT (SEC("perf_event")). - Tracepoint events (event->pmu == &perf_tracepoint) reject BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to event->tp_event->prog_array and executed in perf_trace_run_bpf_submit() before perf_tp_event(), dropping the sample if the BPF program returns 0. Furthermore, the bpf_cast_to_kern_ctx() kfunc is only permitted in BPF_PROG_TYPE_PERF_EVENT programs. Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in sample_filter.bpf.c to evaluate context-independent filter terms (uid, gid, tid, cpu, time, cgroup) using BPF helpers without calling bpf_cast_to_kern_ctx(). When preparing a BPF filter on a PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of perf_sample_filter and attach it once to the shared trace_event_call. Because returning 0 from a BPF_PROG_TYPE_TRACEPOINT program filters at the shared trace_event_call->prog_array prior to perf_tp_event(), tracepoint BPF filters are never pinned for unprivileged use and require CAP_BPF or root privileges. Because perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load perf_sample_filter and disable autoload on perf_tp_sample_filter, libbpf's bpf_object__pin() and bpf_object__unpin() can no longer be used directly (as bpf_object__pin_programs() and bpf_object__unpin_programs() iterate over all programs in the object without checking autoload and fail with -EINVAL when a program has not been loaded). Introduce pin_filter_object() and unpin_filter_object() to pin/unpin the maps and only the loaded perf_sample_filter program. Before: # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument Failed to attach perf sample-filter program failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument) After: # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ] Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper") Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 8 +- tools/perf/util/bpf-filter.c | 144 ++++++++++++++++--- tools/perf/util/bpf_skel/sample_filter.bpf.c | 95 ++++++++++++ tools/perf/util/parse-events.c | 38 ++++- 4 files changed, 262 insertions(+), 23 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..fcf1ea62bf1c 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -173,7 +173,13 @@ OPTIONS A BPF filter can access the sample data and make a decision based on the data. Users need to set an appropriate sample type to use the BPF - filter. BPF filters need root privilege. + filter. BPF filters need root privilege (or CAP_BPF, or a pinned BPF + filter for non-tracepoint events). On tracepoint events, BPF filters + support context-independent terms ('uid', 'gid', 'tid', 'pid', 'cpu', + 'time', 'cgroup') by attaching a 'BPF_PROG_TYPE_TRACEPOINT' program to + the tracepoint's shared 'trace_event_call->prog_array', which filters + the tracepoint before 'perf_tp_event()' runs across concurrent + perf_event listeners on that tracepoint. The sample data field can be specified in lower case letter. Multiple filters can be separated with comma. For example, diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 506414e5c407..0d3c700d3dcd 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -41,8 +41,10 @@ * it will be reported as lost samples. */ #include <errno.h> +#include <stdio.h> #include <stdlib.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> @@ -124,15 +126,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * { const struct perf_sample_info *info; - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) - return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU || + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP) + return 0; + if (expr->op != PBF_OP_GROUP_BEGIN) { + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n", + evsel__name(evsel), expr->term); + return -1; + } + } + + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END && + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) + return 0; + if (expr->op == PBF_OP_GROUP_BEGIN) { struct perf_bpf_filter_expr *group; @@ -426,6 +439,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) struct sample_filter_bpf *skel = NULL; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); entry = calloc(MAX_FILTERS, sizeof(*entry)); if (entry == NULL) @@ -437,7 +451,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - if (needs_idx_hash && geteuid() != 0) { + if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { + pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; /* The filters map is shared among other processes */ @@ -477,8 +497,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; @@ -494,14 +523,32 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - fd = bpf_program__fd(skel->progs.perf_sample_filter); - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { - ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd); - if (ret < 0) { - ret = -errno; - pr_err("Failed to attach perf sample-filter program\n"); - goto err; + if (is_tracepoint) { + /* + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the + * BPF program to event->tp_event->prog_array on the shared + * trace_event_call, which perf_trace_run_bpf_submit() executes + * for all CPUs and tasks before iterating the per-CPU perf + * events. Attach once on the first FD so tp_event->prog_array + * does not get duplicate copies for each per-CPU FD. + */ + fd = bpf_program__fd(skel->progs.perf_tp_sample_filter); + ret = ioctl(FD(evsel, 0, 0), PERF_EVENT_IOC_SET_BPF, fd); + if (ret < 0) { + ret = -errno; + pr_err("Failed to attach perf tracepoint sample-filter program\n"); + goto err; + } + } else { + fd = bpf_program__fd(skel->progs.perf_sample_filter); + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) { + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) { + ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd); + if (ret < 0) { + ret = -errno; + pr_err("Failed to attach perf sample-filter program\n"); + goto err; + } } } } @@ -633,6 +680,55 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +/* + * perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load + * perf_sample_filter and disable autoload on other programs in the skeleton + * (such as perf_tp_sample_filter). While libbpf's bpf_object__pin_maps() skips + * maps that were not created, bpf_object__pin_programs() and + * bpf_object__unpin_programs() iterate over every program in skel->obj without + * checking whether autoload was enabled and fail with -EINVAL when a program + * was not loaded. Pin and unpin the maps and the loaded perf_sample_filter + * program explicitly instead of calling bpf_object__pin() / bpf_object__unpin(). + */ +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -647,6 +743,7 @@ int perf_bpf_filter__pin(void) } /* pinned program will use pid-hash */ + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); @@ -666,7 +763,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -675,8 +772,8 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); - ret = dir_fd; + ret = -errno; + unpin_filter_object(skel, path); goto err; } @@ -719,6 +816,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -732,13 +831,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -746,7 +852,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..a190374ed87a 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return 0; } +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry) +{ + switch (entry->term) { + case PBF_TERM_TID: + if (entry->part) + return bpf_get_current_pid_tgid() >> 32; + return (__u32)bpf_get_current_pid_tgid(); + case PBF_TERM_CPU: + return bpf_get_smp_processor_id(); + case PBF_TERM_TIME: + return bpf_ktime_get_ns(); + case PBF_TERM_CGROUP: + return bpf_get_current_cgroup_id(); + case PBF_TERM_UID: + return bpf_get_current_uid_gid() & 0xFFFFFFFF; + case PBF_TERM_GID: + return bpf_get_current_uid_gid() >> 32; + default: + return 0; + } +} + #define CHECK_RESULT(data, op, val) \ if (!(data op val)) { \ if (!in_group) \ @@ -295,4 +317,77 @@ int perf_sample_filter(void *ctx) return 0; } +/* + * BPF program to be called for tracepoint perf events. + * + * For PERF_TYPE_TRACEPOINT events, the kernel's __perf_event_set_bpf_prog() + * rejects BPF_PROG_TYPE_PERF_EVENT and only accepts BPF_PROG_TYPE_TRACEPOINT, + * attaching the program to the shared trace_event_call->prog_array executed + * by perf_trace_run_bpf_submit() prior to perf_tp_event(). Because returning 0 + * filters the event at trace_event_call->prog_array (affecting the + * trace_event_call globally while attached), tracepoint BPF filters are never + * pinned for unprivileged use and require CAP_BPF / root privileges. + */ +SEC("tracepoint") +int perf_tp_sample_filter(void *ctx) +{ + struct perf_bpf_filter_entry *entry; + __u64 sample_data; + int in_group = 0; + int group_result = 0; + int i, k = 0; + int *losts; + + entry = bpf_map_lookup_elem(&filters, &k); + if (entry == NULL) + goto drop; + + for (i = 0; i < MAX_FILTERS; i++) { + sample_data = perf_get_common_sample(&entry[i]); + + switch (entry[i].op) { + case PBF_OP_EQ: + CHECK_RESULT(sample_data, ==, entry[i].value) + break; + case PBF_OP_NEQ: + CHECK_RESULT(sample_data, !=, entry[i].value) + break; + case PBF_OP_GT: + CHECK_RESULT(sample_data, >, entry[i].value) + break; + case PBF_OP_GE: + CHECK_RESULT(sample_data, >=, entry[i].value) + break; + case PBF_OP_LT: + CHECK_RESULT(sample_data, <, entry[i].value) + break; + case PBF_OP_LE: + CHECK_RESULT(sample_data, <=, entry[i].value) + break; + case PBF_OP_AND: + CHECK_RESULT(sample_data, &, entry[i].value) + break; + case PBF_OP_GROUP_BEGIN: + in_group = 1; + group_result = 0; + break; + case PBF_OP_GROUP_END: + if (group_result == 0) + goto drop; + in_group = 0; + break; + case PBF_OP_DONE: + return 1; + } + } + return 1; + +drop: + losts = bpf_map_lookup_elem(&dropped, &k); + if (losts != NULL) + __sync_fetch_and_add(losts, 1); + + return 0; +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..d6bab89ade5d 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -3,8 +3,10 @@ #include <linux/err.h> #include <linux/list_sort.h> #include <linux/zalloc.h> +#include <ctype.h> #include <dirent.h> #include <errno.h> +#include <string.h> #include <sys/ioctl.h> #include <sys/param.h> #include "cpumap.h" @@ -2593,10 +2595,40 @@ foreach_evsel_in_last_glob(struct evlist *evlist, return 0; } +static bool is_filter_word_char(char c) +{ + return isalnum((unsigned char)c) || c == '_' || c == '/' || c == '.'; +} + +static bool has_filter_word(const char *str, const char *word) +{ + size_t len = strlen(word); + const char *p = str; + + while ((p = strstr(p, word)) != NULL) { + bool start_ok = (p == str) || !is_filter_word_char(p[-1]); + bool end_ok = !is_filter_word_char(p[len]); + + if (start_ok && end_ok) + return true; + p += len; + } + return false; +} + /* Will a tracepoint filter work for str or should a BPF filter be used? */ -static bool is_possible_tp_filter(const char *str) +static bool is_possible_tp_filter(struct evsel *evsel, const char *str) { - return strstr(str, "uid") == NULL; + static const char *const bpf_tp_terms[] = { + "uid", "gid", "tid", "pid", "cpu", "time", "cgroup", + }; + + for (size_t i = 0; i < ARRAY_SIZE(bpf_tp_terms); i++) { + if (has_filter_word(str, bpf_tp_terms[i]) && + !evsel__field(evsel, bpf_tp_terms[i])) + return false; + } + return true; } static int set_filter(struct evsel *evsel, const void *arg) @@ -2611,7 +2643,7 @@ static int set_filter(struct evsel *evsel, const void *arg) return -1; } - if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(str)) { + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(evsel, str)) { if (evsel__append_tp_filter(evsel, str) < 0) { fprintf(stderr, "not enough memory to hold filter string\n"); -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers 2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers ` (7 subsequent siblings) 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h and tools/perf/util/lock-contention.[ch] so they can be reused by BPF sample filtering. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 235 +------------- tools/perf/util/bpf_skel/lock_data.h | 303 ++++++++++++++++++ tools/perf/util/lock-contention.c | 19 ++ tools/perf/util/lock-contention.h | 13 + 5 files changed, 338 insertions(+), 259 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..feaf1cfd08cf 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1477,33 +1477,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..c70c3955e559 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -222,8 +190,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +313,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,99 +795,16 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; const volatile int nr_nodes; const volatile int sizeof_zone; -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } SEC("raw_tp/bpf_test_finish") @@ -1010,43 +814,10 @@ int BPF_PROG(end_timestamp) return 0; } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..91d464ac18bd 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,299 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +/* + * Common BPF CO-RE structures and helper functions shared between + * lock_contention.bpf.c and sample_filter.bpf.c. Guarded by __BPF__ + * because lock_data.h is also included by userspace code. + */ +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex *mutex = (void *)lock; + + owner = BPF_CORE_READ(mutex, owner.counter); + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *rwsem = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(rwsem, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + struct rw_semaphore___new *rwsem = (void *)lock; + + owner = BPF_CORE_READ(rwsem, owner.counter); + } +#else + /* assume new struct */ + struct rw_semaphore *rwsem = (void *)lock; + + owner = BPF_CORE_READ(rwsem, owner.counter); +#endif + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (node_data_addr && nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..f6e68d740860 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -6,6 +6,7 @@ #include "symbol.h" #include <limits.h> +#include <stdlib.h> #include <string.h> #include <linux/hash.h> @@ -22,6 +23,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (2 preceding siblings ...) 2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers ` (6 subsequent siblings) 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend get_lock_owner() in tools/perf/util/bpf_skel/lock_data.h to: - Support rt_mutex locks (LCB_F_RT) and CONFIG_PREEMPT_RT mutexes via struct rt_mutex_base and struct rt_mutex___old, masking out the low waiter bit (0x1UL). - Return NULL when a mutex has MUTEX_FLAG_HANDOFF (0x4UL) set on owner.counter and is in the middle of being handed off rather than actively held. Also add rw_semaphore.count and struct rt_mutex_base to the fallback tools/perf/util/bpf_skel/vmlinux/vmlinux.h. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/util/bpf_skel/lock_data.h | 62 ++++++++++++++++++---- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 ++ 2 files changed, 58 insertions(+), 9 deletions(-) diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 91d464ac18bd..43d8f5030bc1 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -104,11 +104,24 @@ extern struct rq runqueues __ksym __weak; # define __has_builtin(x) 0 #endif +struct mutex___non_rt { + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct rt_mutex_base___new { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rt_mutex___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct rw_semaphore___old { struct task_struct *owner; } __attribute__((preserve_access_index)); struct rw_semaphore___new { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -146,9 +159,25 @@ static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flag __u64 owner = 0; if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - - owner = BPF_CORE_READ(mutex, owner.counter); + struct mutex___non_rt *mutex = (void *)lock; + + if (bpf_core_field_exists(mutex->owner.counter)) { + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (bpf_core_type_exists(struct rt_mutex_base___new)) { + struct rt_mutex_base___new *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } else { + struct rt_mutex___old *rtm = (void *)lock; + + if (bpf_core_field_exists(rtm->owner)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { /* * Support for the BPF_TYPE_MATCHES argument to the @@ -158,20 +187,35 @@ static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flag */ #if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; + struct rw_semaphore___old *old = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); + owner = (unsigned long)BPF_CORE_READ(old, owner); } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; + struct rw_semaphore___new *new = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); + owner = BPF_CORE_READ(new, owner.counter); } #else /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; + struct rw_semaphore___new *new = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); + if (bpf_core_field_exists(new->owner.counter)) + owner = BPF_CORE_READ(new, owner.counter); #endif + } else if (flags == LCB_F_RT) { + if (bpf_core_type_exists(struct rt_mutex_base___new)) { + struct rt_mutex_base___new *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } else { + struct rt_mutex___old *rtm = (void *)lock; + + if (bpf_core_field_exists(rtm->owner)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } } if (!owner) diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (3 preceding siblings ...) 2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers ` (5 subsequent siblings) 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). In this initial step, support global/static kernel mutex symbols (e.g. lock_held(cgroup_mutex)) resolved via /proc/kallsyms. On sample overflow, perf_sample_filter and perf_tp_sample_filter inspect the mutex's owner field directly in kernel memory via get_lock_owner() without attaching any BPF fentry/fexit hooks. Subsequent commits extend lock_held() to other kernel lock types, per-process and semi-global locks, slab cache locks, and RCU/SRCU. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 16 +- tools/perf/util/bpf-filter.c | 350 ++++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 47 ++- tools/perf/util/bpf_skel/sample-filter.h | 36 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 54 +++ tools/perf/util/parse-events.c | 2 + 8 files changed, 530 insertions(+), 19 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index fcf1ea62bf1c..445cc98a6c33 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -210,7 +210,21 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread. To differentiate between shared/read holding + and exclusive/write holding, use 'lock_held_read(<lock>)' (alias + 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias + 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock>. + <lock> can be a kernel mutex symbol name (e.g. + 'lock_held(cgroup_mutex)'). When filtering on a mutex symbol + ('*_mutex'), 'perf_sample_filter' inspects the lock's 'owner' field + directly in kernel memory on each sample without attaching any BPF + fentry/fexit hooks. The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 0d3c700d3dcd..639db6c03137 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -43,6 +43,9 @@ #include <errno.h> #include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> #include <unistd.h> #include <sys/ioctl.h> @@ -50,14 +53,19 @@ #include <bpf/bpf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -126,7 +134,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * { const struct perf_sample_info *info; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -169,9 +178,263 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool in_text; + bool in_data; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, char type, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, "_text") || !strcmp(name, "_stext") || + !strcmp(name, "_sinittext")) { + args->in_text = true; + return 0; + } + if (!strcmp(name, "_etext") || !strcmp(name, "_einittext")) { + args->in_text = false; + return 0; + } + if (!strcmp(name, "_sdata") || !strcmp(name, "__bss_start")) { + args->in_text = false; + args->in_data = true; + return 0; + } + if (!strcmp(name, "__bss_stop")) { + args->in_data = false; + return 0; + } + + /* + * Ignore kernel text functions, module text functions, and + * absolute/undefined symbols. On CONFIG_LTO_CLANG kernels, vmlinux + * .data symbols between _sdata and _edata may have kallsyms type + * 'T'/'t', so allow 'T'/'t' within vmlinux data/bss (args->in_data). + */ + if (args->in_text || type == 'A' || type == 'a' || + type == 'U' || type == 'u' || + (!args->in_data && kallsyms__is_function(type))) + return 0; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) +{ + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter) +{ + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) + prescan_one_lock_expr(group, has_lock_filter); + } + } +} + +static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr, + enum perf_bpf_filter_op *op_out) +{ + switch (expr->op) { + case PBF_OP_EQ: + if (expr->val == 1) { + *op_out = PBF_OP_EQ; + return 0; + } + if (expr->val == 0) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_NEQ: + if (expr->val == 0) { + *op_out = PBF_OP_EQ; + return 0; + } + if (expr->val == 1) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_GT: + if (expr->val == 0) { + *op_out = PBF_OP_EQ; + return 0; + } + break; + case PBF_OP_GE: + if (expr->val == 1) { + *op_out = PBF_OP_EQ; + return 0; + } + break; + case PBF_OP_LT: + if (expr->val == 1) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_LE: + if (expr->val == 0) { + *op_out = PBF_OP_NEQ; + return 0; + } + break; + case PBF_OP_AND: + if (expr->val & 1) { + *op_out = PBF_OP_EQ; + return 0; + } + break; + case PBF_OP_GROUP_BEGIN: + case PBF_OP_GROUP_END: + case PBF_OP_DONE: + default: + break; + } + pr_err("Error: Invalid comparison for boolean lock_held(%s) predicate\n", + expr->lock_name ?: ""); + return -EINVAL; +} + +static int resolve_lock_expr(struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) +{ + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0; + int ret; + + ret = normalize_lock_expr_op(expr, &entry->op); + if (ret < 0) + return ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0 || has_amp) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (str_ends_with(base, "mutex") && + !str_ends_with(base, "rt_mutex") && + !str_ends_with(base, "rtmutex")) { + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_ADDR | mode | PBF_LOCK_DIRECT_MUTEX; + entry->value = addr; + return 0; + } + if (ret == -EACCES) + return ret; + } + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) { - int i = 0; + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -185,6 +448,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -198,6 +466,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -440,20 +713,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); + bool has_lock_filter = false; entry = calloc(MAX_FILTERS, sizeof(*entry)); if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); + prescan_lock_filters(evsel, &has_lock_filter); + + if ((has_lock_filter || is_tracepoint) && geteuid() != 0 && + !perf_cap__capable(CAP_BPF)) { + pr_err("Error: BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; goto err; } - if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { - pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n"); - ret = -EPERM; + ret = get_filter_entries(evsel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); goto err; } @@ -513,6 +790,9 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } + if (has_lock_filter) + skel->bss->lock_enabled = 1; + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -571,6 +851,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -578,7 +873,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -630,7 +925,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -641,6 +936,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -661,14 +978,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -680,6 +994,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + /* * perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load * perf_sample_filter and disable autoload on other programs in the skeleton diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..5ae05d4608c6 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -8,9 +8,11 @@ #endif #include <stdio.h> +#include <stdlib.h> #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +33,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +47,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +117,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..ca1cf586ed6e 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,8 +53,44 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates whether the + * specified kernel lock is currently held by the running task/CPU. + * The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to + * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held), + * and the resolved target address is stored in `entry.value`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address resolved from kallsyms. + * `entry.value` holds the kernel address. + */ + PBF_LOCK_ADDR, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY +#define PBF_LOCK_DIRECT_MUTEX (1U << 6) + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index a190374ed87a..42556ab0e2d5 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,6 +5,7 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" /* BPF map that will be filled by user space */ @@ -45,8 +46,57 @@ struct lost_count { volatile const int use_idx_hash; +/* Control flag set by userspace when lock_held(...) filters are active. */ +int lock_enabled; + void *bpf_cast_to_kern_ctx(void *) __ksym; +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part) +{ + if (!curr) + return false; + + if ((part & PBF_LOCK_DIRECT_MUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr) + return true; + return false; +} + +static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr = bpf_get_current_task(); + __u32 kind = entry->part & PBF_LOCK_KIND_MASK; + __u32 mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(entry->value, curr, entry->part)) + return true; + return false; + + default: + return false; + } +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->value. + */ + if (use_idx_hash || !lock_enabled) + return entry->op == PBF_OP_EQ ? ~entry->value : entry->value; + + return is_lock_held(entry) ? entry->value : ~entry->value; +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +225,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -209,6 +261,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); default: return 0; } diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index d6bab89ade5d..da687e9d1a67 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2621,6 +2621,8 @@ static bool is_possible_tp_filter(struct evsel *evsel, const char *str) { static const char *const bpf_tp_terms[] = { "uid", "gid", "tid", "pid", "cpu", "time", "cgroup", + "lock_held", "lock_held_read", "lock_held_shared", + "lock_held_write", "lock_held_exclusive", }; for (size_t i = 0; i < ARRAY_SIZE(bpf_tp_terms); i++) { -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (4 preceding siblings ...) 2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers ` (4 subsequent siblings) 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held() BPF sample filter beyond global mutexes to support: - Other global/static kernel lock symbols (rwsem, spinlock, rwlock, rt-mutex, percpu-rwsem, semaphore) and hexadecimal kernel lock addresses. - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore. For rt-mutexes and write-locked rwsems, perf_sample_filter inspects the lock's owner field directly in kernel memory. For locks and modes without an authoritative per-task owner pointer (spinlocks, rwlocks, semaphores, reader-locked rwsems, percpu-rwsems, and lock type filters), attach BPF fentry/fexit hooks on kernel lock acquire and release functions to track held locks in the held_locks BPF LRU hash map. Note that counting semaphores (down*() / up()) do not enforce same-task ownership in the kernel: when a semaphore is used as a cross-task completion/signaling mechanism, fexit/down and fentry/up will not pair on the same task ID. Semaphore hooks are therefore only enabled when 'semaphore' or a '*_sem' symbol (such as console_sem) is explicitly specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any unpaired cross-task entries are automatically evicted. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 53 +- tools/perf/util/bpf-filter.c | 596 ++++++++++++- tools/perf/util/bpf_skel/sample-filter.h | 17 +- tools/perf/util/bpf_skel/sample_filter.bpf.c | 851 ++++++++++++++++++- 4 files changed, 1465 insertions(+), 52 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 445cc98a6c33..9fb5d15de135 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -216,15 +216,50 @@ OPTIONS The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters samples based on whether the specified kernel lock is currently held - by the kernel thread. To differentiate between shared/read holding - and exclusive/write holding, use 'lock_held_read(<lock>)' (alias - 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias - 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock>. - <lock> can be a kernel mutex symbol name (e.g. - 'lock_held(cgroup_mutex)'). When filtering on a mutex symbol - ('*_mutex'), 'perf_sample_filter' inspects the lock's 'owner' field - directly in kernel memory on each sample without attaching any BPF - fentry/fexit hooks. + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(rwsem:R)' or 'lock_held(rwsem:W)'). + <lock> can be a kernel lock symbol name (e.g. + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, or a lock type + ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'semaphore'). Both read/shared and write/exclusive tracking are + supported for 'rwsem' and 'rwlock'; 'mutex', 'spinlock', 'rt-mutex', + 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support + write/exclusive tracking only. Note that 'semaphore' tracking + ('down*()' / 'up()') assumes binary mutex-like usage where the + acquiring task also releases the semaphore (such as 'console_sem'), + rather than cross-task completion signaling. + + For example, 'lock_held()' can be combined with memory allocation or + reclaim events to detect allocations or direct memory reclaim occurring + while a kernel lock is held (which can stall lock waiters for extended + periods): + + # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e kmem:mm_page_alloc \ + --filter 'gfp_flags & 0x400' \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + # perf record -a -g -e major-faults \ + --filter 'lock_held(mutex) || lock_held(rwsem)' + + When filtering on a specific mutex symbol ('*_mutex') or rt-mutex + symbol ('*_rt_mutex'), 'perf_sample_filter' inspects the lock's + 'owner' field directly in kernel memory on each sample without attaching + any BPF fentry/fexit hooks. For locks that do not maintain an + authoritative per-task owner pointer (such as spinlocks, rwlocks, + semaphores, reader-locked rwsems, and lock type filters), it attaches + BPF fentry/fexit programs to kernel lock acquire and release functions + to track held locks in a BPF map. These BPF hooks run on every kernel + lock acquisition and release for the attached lock family, even when no + perf sample is being taken. To minimize this overhead, only the hooks + for the relevant lock family and mode are attached when known, whereas + raw hex addresses or symbols without a recognizable lock-type suffix + attach hooks across all lock families (including high-frequency + spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 639db6c03137..8797f67698f7 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -52,6 +52,7 @@ #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> #include <linux/kernel.h> #include <linux/list.h> @@ -65,6 +66,7 @@ #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" #include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" @@ -74,6 +76,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -178,6 +181,153 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + static int split_lock_name_mode(const char *name, int expr_part, char *base_buf, size_t buf_sz, bool *has_amp, unsigned int *mode_out) @@ -224,6 +374,72 @@ static int split_lock_name_mode(const char *name, int expr_part, return 0; } +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN)) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + struct lock_sym_cb_args { const char *name; u64 addr; @@ -301,26 +517,225 @@ static int find_kernel_lock_sym(const char *name, u64 *addr) return 0; } +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static bool btf_has_func(struct btf *btf, const char *name) +{ + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; +} + +static unsigned int supported_lock_prog_groups(struct btf *btf) +{ + unsigned int supported = 0; + + if (!btf) + return 0; + + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) && + btf_has_func(btf, "mutex_unlock")) + supported |= LOCK_PROG_MUTEX; + + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) && + btf_has_func(btf, "up_read")) + supported |= LOCK_PROG_RWSEM_READ; + + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) && + btf_has_func(btf, "up_write")) + supported |= LOCK_PROG_RWSEM_WRITE; + + /* + * Require both acquire and all unlock entry points in BTF so we never + * attach spinlock/rwlock acquire hooks when unlock functions are + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK). + */ + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) && + btf_has_func(btf, "_raw_spin_unlock") && + btf_has_func(btf, "_raw_spin_unlock_irq") && + btf_has_func(btf, "_raw_spin_unlock_irqrestore") && + btf_has_func(btf, "_raw_spin_unlock_bh")) + supported |= LOCK_PROG_SPIN; + + if (btf_has_func(btf, "_raw_read_lock") && + btf_has_func(btf, "_raw_read_unlock") && + btf_has_func(btf, "_raw_read_unlock_irq") && + btf_has_func(btf, "_raw_read_unlock_irqrestore") && + btf_has_func(btf, "_raw_read_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_READ; + + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) && + btf_has_func(btf, "_raw_write_unlock") && + btf_has_func(btf, "_raw_write_unlock_irq") && + btf_has_func(btf, "_raw_write_unlock_irqrestore") && + btf_has_func(btf, "_raw_write_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_WRITE; + + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) && + btf_has_func(btf, "rt_mutex_unlock")) + supported |= LOCK_PROG_RTMUTEX; + + if (btf_has_func(btf, "down") && btf_has_func(btf, "up")) + supported |= LOCK_PROG_SEM; + + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) + supported |= LOCK_PROG_PERCPU_RWSEM; + + return supported; +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, + unsigned int required_groups, + bool is_tracepoint) +{ + unsigned int supported_groups = supported_lock_prog_groups(btf); + unsigned int enabled_progs = 0; + + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + + if ((required_groups & supported_groups) != required_groups || + (needed_groups && !(needed_groups & supported_groups))) { + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n"); + return -ENOTSUP; + } + + needed_groups &= supported_groups; + if (skel) + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ); + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, - bool *has_lock_filter) + bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups) { + char base[128]; + bool has_amp; + unsigned int mode, groups = 0; + u64 dummy; + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) return; *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0 || has_amp) + return; + + if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { + *prog_groups |= groups; + *required_groups |= groups; + return; + } + + if (parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex") || + str_ends_with(base, "mutex")) { + /* Inspected directly via lock->owner; no tracing hooks needed. */ + return; + } else if (str_ends_with(base, "percpu_rwsem") || + str_ends_with(base, "pcpu_rwsem")) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (str_ends_with(base, "rwsem")) { + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_RWSEM_WRITE; + } + } else if (str_ends_with(base, "spinlock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + } else if (str_ends_with(base, "sem")) { + *prog_groups |= LOCK_PROG_SEM; + *required_groups |= LOCK_PROG_SEM; + } else if (str_ends_with(base, "rwlock")) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWLOCK_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + } else { + *prog_groups |= LOCK_PROG_ALL; + } } -static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter) +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups) { struct perf_bpf_filter_expr *expr, *group; *has_lock_filter = false; + *prog_groups = 0; + *required_groups = 0; list_for_each_entry(expr, &evsel->bpf_filters, list) { - prescan_one_lock_expr(expr, has_lock_filter); + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + required_groups); if (expr->op == PBF_OP_GROUP_BEGIN) { - list_for_each_entry(group, &expr->groups, list) - prescan_one_lock_expr(group, has_lock_filter); + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, required_groups); + } } } } @@ -390,14 +805,42 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr, return -EINVAL; } -static int resolve_lock_expr(struct perf_bpf_filter_expr *expr, +static bool is_percpu_rwsem_sym(struct btf *btf, const char *name) +{ + char rc_name[160]; + + if (str_ends_with(name, "percpu_rwsem") || + str_ends_with(name, "pcpu_rwsem")) + return true; + + if (!btf) + return false; + + snprintf(rc_name, sizeof(rc_name), "__percpu_rwsem_rc_%s", name); + return btf__find_by_name_kind(btf, rc_name, BTF_KIND_VAR) > 0; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->op`, + * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`, + * and populate the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, + * `rwlock`, `percpu-rwsem`, `semaphore`, etc. + * 2. Hexadecimal lock address. + * 3. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf, + struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; char base[128]; bool has_amp; unsigned int mode; - u64 addr = 0; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; int ret; ret = normalize_lock_expr_op(expr, &entry->op); @@ -410,29 +853,79 @@ static int resolve_lock_expr(struct perf_bpf_filter_expr *expr, return -EINVAL; } - if (str_ends_with(base, "mutex") && - !str_ends_with(base, "rt_mutex") && - !str_ends_with(base, "rtmutex")) { - ret = find_kernel_lock_sym(base, &addr); - if (ret == 0) { - if (!(mode & PBF_LOCK_MODE_WRITE)) { - pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", - name); - return -EINVAL; - } - entry->part = PBF_LOCK_ADDR | mode | PBF_LOCK_DIRECT_MUTEX; - entry->value = addr; - return 0; + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->value = encoded_pair; + return 0; + } + + if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->value = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; } - if (ret == -EACCES) - return ret; + return 0; } + if (btf_has_func(btf, base)) { + pr_err("Error: '%s' is a kernel function, not a lock\n", name); + return -EINVAL; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + unsigned int direct_flag = 0; + bool is_pcpu_rwsem = is_percpu_rwsem_sym(btf, base); + + if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex")) + direct_flag = PBF_LOCK_DIRECT_RTMUTEX; + else if (str_ends_with(base, "mutex")) + direct_flag = PBF_LOCK_DIRECT_MUTEX; + else if (str_ends_with(base, "rwsem") && !is_pcpu_rwsem) + direct_flag = PBF_LOCK_DIRECT_RWSEM; + + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX || + direct_flag == PBF_LOCK_DIRECT_RTMUTEX || + is_pcpu_rwsem || + str_ends_with(base, "spinlock") || + (str_ends_with(base, "sem") && + !str_ends_with(base, "rwsem"))) && + !(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + + entry->part = PBF_LOCK_ADDR | mode | direct_flag; + entry->value = addr; + if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX && + direct_flag != PBF_LOCK_DIRECT_RTMUTEX) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); return -EINVAL; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -449,7 +942,7 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].term = expr->term; entry[i].value = expr->val; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(expr, &entry[i]); + ret = resolve_lock_expr(skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -467,7 +960,7 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].term = group->term; entry[i].value = group->val; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(group, &entry[i]); + ret = resolve_lock_expr(skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -710,16 +1203,19 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); bool has_lock_filter = false; + unsigned int prog_groups = 0, required_groups = 0; entry = calloc(MAX_FILTERS, sizeof(*entry)); if (entry == NULL) return -1; - prescan_lock_filters(evsel, &has_lock_filter); + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &required_groups); if ((has_lock_filter || is_tracepoint) && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { @@ -728,15 +1224,14 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } - if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + ret = get_filter_entries(evsel, NULL, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -781,8 +1276,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + ret = configure_lock_progs(skel, btf, prog_groups, required_groups, + is_tracepoint); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, 0, is_tracepoint); + } if (sample_filter_bpf__load(skel) < 0) { ret = -errno; @@ -790,8 +1294,23 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - if (has_lock_filter) + ret = get_filter_entries(evsel, skel, btf, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } skel->bss->lock_enabled = 1; + } i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -832,11 +1351,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -1064,8 +1585,9 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, 0, false); + /* pinned program will use pid-hash */ - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH); @@ -1160,7 +1682,7 @@ int perf_bpf_filter__unpin(void) goto err; } - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false); + configure_lock_progs(skel, NULL, 0, 0, false); if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index ca1cf586ed6e..c240eee782f8 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -59,7 +59,8 @@ enum perf_bpf_filter_term { * The lock specification kind is stored in `entry.part` * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held), - * and the resolved target address is stored in `entry.value`. + * and the resolved target address or type flag encoding is stored in + * `entry.value`. */ PBF_TERM_LOCK_HELD, }; @@ -71,10 +72,16 @@ enum perf_bpf_filter_term { enum perf_bpf_lock_target_kind { PBF_LOCK_NONE = 0, /* - * Specific global or static lock address resolved from kallsyms. - * `entry.value` holds the kernel address. + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.value` holds the kernel address. */ PBF_LOCK_ADDR, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.value` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, }; /* @@ -83,6 +90,8 @@ enum perf_bpf_lock_target_kind { * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner) + * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner) + * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base / rt_mutex owner) */ #define PBF_LOCK_KIND_MASK 0x0fU #define PBF_LOCK_MODE_READ (1U << 4) @@ -90,6 +99,8 @@ enum perf_bpf_lock_target_kind { #define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) #define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY #define PBF_LOCK_DIRECT_MUTEX (1U << 6) +#define PBF_LOCK_DIRECT_RWSEM (1U << 7) +#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8) /* BPF map entry for filtering */ struct perf_bpf_filter_entry { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index 42556ab0e2d5..e527fbaf8a46 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -8,6 +8,44 @@ #include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -44,13 +82,65 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* + * Currently held kernel locks tracked by fentry/fexit hooks. + * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or + * non-owner rwsem is acquired in one task and released in another task or + * interrupt handler (cross-context signaling), any orphaned entry is + * automatically evicted rather than exhausting the map. + */ +struct { + __uint(type, BPF_MAP_TYPE_LRU_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + volatile const int use_idx_hash; -/* Control flag set by userspace when lock_held(...) filters are active. */ +/* Control flags set by userspace when lock_held(...) filters are active. */ int lock_enabled; +int has_lock_addr; +int has_lock_type; +int track_rwsem_read; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr) +{ + struct rw_semaphore___new *rwsem = (void *)lock_addr; + + if (bpf_core_field_exists(rwsem->count.counter) && + !(BPF_CORE_READ(rwsem, count.counter) & 1UL)) + return false; + if (bpf_core_field_exists(rwsem->owner.counter) && + (BPF_CORE_READ(rwsem, owner.counter) & 1UL)) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr; +} + static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part) { if (!curr) @@ -59,24 +149,85 @@ static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __ if ((part & PBF_LOCK_DIRECT_MUTEX) && (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr) return true; + if ((part & PBF_LOCK_DIRECT_RWSEM) && + is_rwsem_writer_owner(lock_addr, curr)) + return true; + if ((part & PBF_LOCK_DIRECT_RTMUTEX) && + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr) + return true; return false; } +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) { __u64 curr = bpf_get_current_task(); + __u32 pid = (__u32)bpf_get_current_pid_tgid(); + __u32 cpu_id = 0x80000000U | bpf_get_smp_processor_id(); __u32 kind = entry->part & PBF_LOCK_KIND_MASK; __u32 mode = entry->part & PBF_LOCK_MODE_MASK; + __u64 lock_addr; if (!mode) mode = PBF_LOCK_MODE_ANY; switch (kind) { case PBF_LOCK_ADDR: + lock_addr = entry->value; if ((mode & PBF_LOCK_MODE_WRITE) && - is_direct_lock_owner(entry->value, curr, entry->part)) + is_direct_lock_owner(lock_addr, curr, entry->part)) return true; - return false; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0; default: return false; @@ -444,4 +595,698 @@ int perf_tp_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +#define LCB_F_READ_IRQSAVE (1U << 30) +#define LCB_F_READ_IRQ_BH (1U << 31) + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* + * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC + * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call + * the out-of-line _raw_read_lock_irqsave(). Clear the marker + * and skip the duplicate outer fexit increment. + */ + if ((flags & LCB_F_READ_IRQ_BH) && + (existing->flags & LCB_F_READ_IRQSAVE)) { + existing->flags &= ~LCB_F_READ_IRQSAVE; + return 0; + } + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) { + __sync_fetch_and_add(&existing->read_count, 1); + if (flags & LCB_F_READ_IRQSAVE) + existing->flags |= LCB_F_READ_IRQSAVE; + else + existing->flags &= ~LCB_F_READ_IRQSAVE; + } else { + __sync_fetch_and_add(&existing->write_count, 1); + } + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE), + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count == 0) + return 0; + if (existing->read_count == 1 && existing->write_count == 0) { + bpf_map_delete_elem(&held_locks, &addr_key); + } else { + existing->flags &= ~LCB_F_READ_IRQSAVE; + __sync_fetch_and_add(&existing->read_count, -1); + } + } else { + if (existing->write_count == 0) + return 0; + if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/_mutex_lock_nest_lock") +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_lock_killable") +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass, + void *nest_lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_trylock_nest_lock") +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/_down_write_nest_lock") +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + if (track_rwsem_read) + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nest_lock") +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/_rt_mutex_lock_nest_lock") +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (5 preceding siblings ...) 2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers @ 2026-10-22 20:29 ` Ian Rogers 2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers ` (3 subsequent siblings) 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held() BPF sample filter to support well-known per-process and semi-global kernel locks: - mmap_lock (or mmap_sem): current->mm->mmap_lock, supporting both read/shared (lock_held_read(mmap_lock) / lock_held(mmap_lock:R)) and write/exclusive (lock_held_write(mmap_lock) / lock_held(mmap_lock:W)) modes. Write ownership is inspected directly via is_rwsem_writer_owner() without attaching fentry/fexit hooks. - siglock: current->sighand->siglock. - rq_lock: per-CPU scheduler runqueue locks. - zone_lock: memory management NUMA zone locks. For rq_lock and zone_lock, reuse do_collect_lock_syms() from lock_data.h via a raw_tp/bpf_test_finish program run at filter setup time to populate the lock_syms map with each CPU's rq_lock and each NUMA node's zone_lock addresses. Example usage: # perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 24 +-- tools/perf/util/bpf-filter.c | 169 +++++++++++++++++-- tools/perf/util/bpf_skel/sample-filter.h | 18 +- tools/perf/util/bpf_skel/sample_filter.bpf.c | 67 ++++++++ 4 files changed, 252 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 9fb5d15de135..2b2aaaead76b 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -220,13 +220,15 @@ OPTIONS holding and exclusive/write holding, use 'lock_held_read(<lock>)' (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to - <lock> (e.g. 'lock_held(rwsem:R)' or 'lock_held(rwsem:W)'). + <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)'). <lock> can be a kernel lock symbol name (e.g. - 'lock_held(cgroup_mutex)'), a hexadecimal lock address, or a lock type - ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a well-known + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). Both read/shared and write/exclusive tracking are - supported for 'rwsem' and 'rwlock'; 'mutex', 'spinlock', 'rt-mutex', + supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', + 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'semaphore' tracking ('down*()' / 'up()') assumes binary mutex-like usage where the @@ -246,8 +248,9 @@ OPTIONS # perf record -a -g -e major-faults \ --filter 'lock_held(mutex) || lock_held(rwsem)' - When filtering on a specific mutex symbol ('*_mutex') or rt-mutex - symbol ('*_rt_mutex'), 'perf_sample_filter' inspects the lock's + When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol + ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' + or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's 'owner' field directly in kernel memory on each sample without attaching any BPF fentry/fexit hooks. For locks that do not maintain an authoritative per-task owner pointer (such as spinlocks, rwlocks, @@ -256,10 +259,11 @@ OPTIONS to track held locks in a BPF map. These BPF hooks run on every kernel lock acquisition and release for the attached lock family, even when no perf sample is being taken. To minimize this overhead, only the hooks - for the relevant lock family and mode are attached when known, whereas - raw hex addresses or symbols without a recognizable lock-type suffix - attach hooks across all lock families (including high-frequency - spinlocks). + for the relevant lock family and mode are attached when known (for + example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions + and 'rq_lock' only hooks spinlock functions), whereas raw hex addresses + or symbols without a recognizable lock-type suffix attach hooks across + all lock families (including high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 8797f67698f7..f542920f914f 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -534,6 +534,54 @@ static bool parse_lock_hex_addr(const char *str, u64 *addr) return true; } +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) != 0) + return; + + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + static bool btf_has_func(struct btf *btf, const char *name) { return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; @@ -607,7 +655,7 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, unsigned int needed_groups, unsigned int required_groups, - bool is_tracepoint) + bool needs_lock_syms, bool is_tracepoint) { unsigned int supported_groups = supported_lock_prog_groups(btf); unsigned int enabled_progs = 0; @@ -642,6 +690,8 @@ static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, bpf_program__set_autoload(prog, enable); } + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needed_groups && !enabled_progs) { pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); return -ENOTSUP; @@ -657,7 +707,8 @@ static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, bool *has_lock_filter, unsigned int *prog_groups, - unsigned int *required_groups) + unsigned int *required_groups, + bool *needs_lock_syms) { char base[128]; bool has_amp; @@ -672,6 +723,29 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, sizeof(base), &has_amp, &mode) < 0 || has_amp) return; + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + /* + * Write ownership of mm->mmap_lock is inspected directly via + * is_rwsem_writer_owner() without tracing hooks; only reader + * tracking requires LOCK_PROG_RWSEM_READ. + */ + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { *prog_groups |= groups; *required_groups |= groups; @@ -720,21 +794,24 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, unsigned int *prog_groups, - unsigned int *required_groups) + unsigned int *required_groups, + bool *needs_lock_syms) { struct perf_bpf_filter_expr *expr, *group; *has_lock_filter = false; *prog_groups = 0; *required_groups = 0; + *needs_lock_syms = false; list_for_each_entry(expr, &evsel->bpf_filters, list) { prescan_one_lock_expr(expr, has_lock_filter, prog_groups, - required_groups); + required_groups, needs_lock_syms); if (expr->op == PBF_OP_GROUP_BEGIN) { list_for_each_entry(group, &expr->groups, list) { prescan_one_lock_expr(group, has_lock_filter, - prog_groups, required_groups); + prog_groups, required_groups, + needs_lock_syms); } } } @@ -826,10 +903,12 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name) * and populate the corresponding BPF filter maps and BSS control flags. * * Resolution order: - * 1. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, + * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`. + * 2. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, * `rwlock`, `percpu-rwsem`, `semaphore`, etc. - * 2. Hexadecimal lock address. - * 3. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). + * 3. Hexadecimal lock address. + * 4. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). */ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf, struct perf_bpf_filter_expr *expr, @@ -853,6 +932,53 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf, return -EINVAL; } + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->value = 0; + if (skel && (mode & PBF_LOCK_MODE_READ)) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->value = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->value = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->value = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->value = encoded_pair; @@ -1207,7 +1333,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); - bool has_lock_filter = false; + bool has_lock_filter = false, needs_lock_syms = false; unsigned int prog_groups = 0, required_groups = 0; entry = calloc(MAX_FILTERS, sizeof(*entry)); @@ -1215,7 +1341,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return -1; prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, - &required_groups); + &required_groups, &needs_lock_syms); if ((has_lock_filter || is_tracepoint) && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { @@ -1280,12 +1406,16 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) btf = btf__load_vmlinux_btf(); bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, - is_tracepoint); + needs_lock_syms, is_tracepoint); if (ret < 0) goto err; } else { - configure_lock_progs(skel, NULL, 0, 0, is_tracepoint); + configure_lock_progs(skel, NULL, 0, 0, false, is_tracepoint); } if (sample_filter_bpf__load(skel) < 0) { @@ -1301,8 +1431,19 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + ret = bpf_prog_test_run_opts(prog_fd, &opts); + if (ret < 0) { + pr_err("Failed to collect kernel lock symbols\n"); + goto err; + } + } bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); ret = sample_filter_bpf__attach(skel); if (ret < 0) { @@ -1585,7 +1726,7 @@ int perf_bpf_filter__pin(void) goto err; } - configure_lock_progs(skel, NULL, 0, 0, false); + configure_lock_progs(skel, NULL, 0, 0, false, false); /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); @@ -1682,7 +1823,7 @@ int perf_bpf_filter__unpin(void) goto err; } - configure_lock_progs(skel, NULL, 0, 0, false); + configure_lock_progs(skel, NULL, 0, 0, false, false); if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index c240eee782f8..836b398dc62b 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -59,8 +59,8 @@ enum perf_bpf_filter_term { * The lock specification kind is stored in `entry.part` * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held), - * and the resolved target address or type flag encoding is stored in - * `entry.value`. + * and the resolved target address, class ID, or type flag encoding is + * stored in `entry.value`. */ PBF_TERM_LOCK_HELD, }; @@ -76,6 +76,20 @@ enum perf_bpf_lock_target_kind { * given as a hex address). `entry.value` holds the kernel address. */ PBF_LOCK_ADDR, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.value` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, /* * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). * `entry.value` packs up to two `LCB_F_*` flag combinations in the diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e527fbaf8a46..2087fb05e4c7 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -104,12 +104,27 @@ struct { __uint(max_entries, 1); } lock_addr_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + /* Control flags set by userspace when lock_held(...) filters are active. */ int lock_enabled; int has_lock_addr; +int has_lock_class; int has_lock_type; +int has_mmap_lock; +int has_siglock; int track_rwsem_read; __u8 lock_type_filter[LCB_F_TYPE_MAX]; @@ -226,6 +241,24 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) return true; return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return false; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_rwsem_writer_owner(lock_addr, curr)) + return true; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return false; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0; + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0; + case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0; @@ -663,6 +696,7 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla struct held_lock_val *existing; bool matched = false; bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; __u32 type_flag = flags & LCB_F_TYPE_MASK; bool match_type = false; @@ -672,6 +706,23 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, type_flag); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && type_flag == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { match_type = true; matched = true; @@ -716,6 +767,8 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla return 0; } + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true); if (match_type) update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true); @@ -764,6 +817,13 @@ static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flag existing->write_count = 0; } + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); @@ -1289,4 +1349,11 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) in lock_held() 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (6 preceding siblings ...) 2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers @ 2026-10-22 20:30 ` Ian Rogers 2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers ` (2 subsequent siblings) 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held() BPF sample filter to support dynamically allocated kernel locks inside slab caches prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry), lock_held_write(&dentry)). Reuse do_slab_cache_iter() from lock_data.h via a SEC("iter/kmem_cache") program to resolve the requested slab cache name to its kernel struct kmem_cache * address, and use the bpf_get_kmem_cache() kfunc in record_lock_acquired_owner() and record_lock_released_addr() to match held locks allocated from that slab cache. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 25 ++-- tools/perf/util/bpf-filter.c | 125 ++++++++++++++++--- tools/perf/util/bpf_skel/sample-filter.h | 5 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 48 +++++++ 4 files changed, 175 insertions(+), 28 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2b2aaaead76b..8fc3196ed93d 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -222,13 +222,15 @@ OPTIONS (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)'). <lock> can be a kernel lock symbol name (e.g. - 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a well-known - lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock - type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', - 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', - 'semaphore'). Both read/shared and write/exclusive tracking are - supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', - 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache + name prefixed with '&' (e.g. 'lock_held(&dentry)', + 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock', + 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex', + 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', + 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). + Both read/shared and write/exclusive tracking are supported for + 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock' + (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'semaphore' tracking ('down*()' / 'up()') assumes binary mutex-like usage where the @@ -254,16 +256,17 @@ OPTIONS 'owner' field directly in kernel memory on each sample without attaching any BPF fentry/fexit hooks. For locks that do not maintain an authoritative per-task owner pointer (such as spinlocks, rwlocks, - semaphores, reader-locked rwsems, and lock type filters), it attaches + semaphores, reader-locked rwsems, and slab/type filters), it attaches BPF fentry/fexit programs to kernel lock acquire and release functions to track held locks in a BPF map. These BPF hooks run on every kernel lock acquisition and release for the attached lock family, even when no perf sample is being taken. To minimize this overhead, only the hooks for the relevant lock family and mode are attached when known (for example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions - and 'rq_lock' only hooks spinlock functions), whereas raw hex addresses - or symbols without a recognizable lock-type suffix attach hooks across - all lock families (including high-frequency spinlocks). + and 'rq_lock' only hooks spinlock functions), whereas slab filters + ('&<slab>'), raw hex addresses, or symbols without a recognizable + lock-type suffix attach hooks across all lock families (including + high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index f542920f914f..1a75d779e891 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -655,10 +655,14 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, unsigned int needed_groups, unsigned int required_groups, - bool needs_lock_syms, bool is_tracepoint) + bool needs_lock_syms, bool needs_slab_iter, + bool is_tracepoint) { unsigned int supported_groups = supported_lock_prog_groups(btf); unsigned int enabled_progs = 0; + bool has_slab_iter = needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", + BTF_KIND_STRUCT) > 0; bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint); bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint); @@ -691,6 +695,7 @@ static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, } bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + bpf_program__set_autoload(skel->progs.slab_cache_iter, has_slab_iter); if (needed_groups && !enabled_progs) { pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); @@ -708,7 +713,8 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, bool *has_lock_filter, unsigned int *prog_groups, unsigned int *required_groups, - bool *needs_lock_syms) + bool *needs_lock_syms, + bool *needs_slab_iter) { char base[128]; bool has_amp; @@ -720,9 +726,16 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *has_lock_filter = true; if (split_lock_name_mode(expr->lock_name, expr->part, base, - sizeof(base), &has_amp, &mode) < 0 || has_amp) + sizeof(base), &has_amp, &mode) < 0) return; + /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */ + if (has_amp) { + *needs_slab_iter = true; + *prog_groups |= LOCK_PROG_ALL; + return; + } + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { /* * Write ownership of mm->mmap_lock is inspected directly via @@ -795,7 +808,8 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, unsigned int *prog_groups, unsigned int *required_groups, - bool *needs_lock_syms) + bool *needs_lock_syms, + bool *needs_slab_iter) { struct perf_bpf_filter_expr *expr, *group; @@ -803,20 +817,72 @@ static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, *prog_groups = 0; *required_groups = 0; *needs_lock_syms = false; + *needs_slab_iter = false; list_for_each_entry(expr, &evsel->bpf_filters, list) { prescan_one_lock_expr(expr, has_lock_filter, prog_groups, - required_groups, needs_lock_syms); + required_groups, needs_lock_syms, + needs_slab_iter); if (expr->op == PBF_OP_GROUP_BEGIN) { list_for_each_entry(group, &expr->groups, list) { prescan_one_lock_expr(group, has_lock_filter, prog_groups, required_groups, - needs_lock_syms); + needs_lock_syms, needs_slab_iter); } } } } +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr, enum perf_bpf_filter_op *op_out) { @@ -903,12 +969,13 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name) * and populate the corresponding BPF filter maps and BSS control flags. * * Resolution order: - * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, * `rq_lock`, `zone_lock`. - * 2. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, + * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, * `rwlock`, `percpu-rwsem`, `semaphore`, etc. - * 3. Hexadecimal lock address. - * 4. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). + * 4. Hexadecimal lock address. + * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). */ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf, struct perf_bpf_filter_expr *expr, @@ -927,11 +994,25 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf, return ret; if (split_lock_name_mode(name, expr->part, base, sizeof(base), - &has_amp, &mode) < 0 || has_amp) { + &has_amp, &mode) < 0) { pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); return -EINVAL; } + if (has_amp) { + if (find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->value = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + pr_err("Error: Cannot resolve slab cache '%s'\n", name); + return -EINVAL; + } + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { entry->part = PBF_LOCK_MMAP | mode; entry->value = 0; @@ -1333,7 +1414,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT); - bool has_lock_filter = false, needs_lock_syms = false; + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; unsigned int prog_groups = 0, required_groups = 0; entry = calloc(MAX_FILTERS, sizeof(*entry)); @@ -1341,7 +1422,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return -1; prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, - &required_groups, &needs_lock_syms); + &required_groups, &needs_lock_syms, + &needs_slab_iter); if ((has_lock_filter || is_tracepoint) && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { @@ -1406,16 +1488,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) btf = btf__load_vmlinux_btf(); bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } if (needs_lock_syms) { bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, - needs_lock_syms, is_tracepoint); + needs_lock_syms, needs_slab_iter, + is_tracepoint); if (ret < 0) goto err; } else { - configure_lock_progs(skel, NULL, 0, 0, false, is_tracepoint); + configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint); } if (sample_filter_bpf__load(skel) < 0) { @@ -1424,6 +1511,9 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); @@ -1444,6 +1534,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false); bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); ret = sample_filter_bpf__attach(skel); if (ret < 0) { @@ -1726,7 +1817,7 @@ int perf_bpf_filter__pin(void) goto err; } - configure_lock_progs(skel, NULL, 0, 0, false, false); + configure_lock_progs(skel, NULL, 0, 0, false, false, false); /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); @@ -1823,7 +1914,7 @@ int perf_bpf_filter__unpin(void) goto err; } - configure_lock_progs(skel, NULL, 0, 0, false, false); + configure_lock_progs(skel, NULL, 0, 0, false, false, false); if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 836b398dc62b..a65ee780d64d 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -76,6 +76,11 @@ enum perf_bpf_lock_target_kind { * given as a hex address). `entry.value` holds the kernel address. */ PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.value` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, /* * Per-process `current->mm->mmap_lock` (or `mmap_sem`). */ diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index 2087fb05e4c7..d1192a93179e 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -111,6 +111,20 @@ struct { __uint(max_entries, 1); } lock_syms SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; @@ -118,9 +132,12 @@ const volatile __u64 node_data_addr; const volatile int nr_nodes; const volatile int sizeof_zone; +__u32 slab_cache_id; + /* Control flags set by userspace when lock_held(...) filters are active. */ int lock_enabled; int has_lock_addr; +int has_lock_slab; int has_lock_class; int has_lock_type; int has_mmap_lock; @@ -259,6 +276,9 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) case PBF_LOCK_CLASS: return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0; + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->value, PBF_LOCK_SLAB, mode, pid, cpu_id) != 0; + case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0; @@ -697,6 +717,7 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla bool matched = false; bool is_read = (flags & LCB_F_READ) != 0; __u32 class_id = 0; + __u64 kmem_cache_addr = 0; __u32 type_flag = flags & LCB_F_TYPE_MASK; bool match_type = false; @@ -723,6 +744,14 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla } } + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { match_type = true; matched = true; @@ -769,6 +798,10 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (class_id) update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, type_flag, true); + } if (match_type) update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true); @@ -824,6 +857,15 @@ static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flag update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); } + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); @@ -1356,4 +1398,10 @@ int BPF_PROG(collect_lock_syms) node_data_addr, nr_nodes, sizeof_zone); } +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (7 preceding siblings ...) 2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers @ 2026-10-22 20:30 ` Ian Rogers 2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 +++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 122 ++++++++++++++++++++ 2 files changed, 230 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..ebaf8ee41622 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,124 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed ${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&dentry)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &dentry]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required" + then + echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]" + err=1 + return + fi + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock:R) || lock_held(mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock:R/:W]" + err=1 + return + fi + + if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc + then + if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]" + err=1 + return + fi + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +316,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (8 preceding siblings ...) 2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers @ 2026-10-22 20:30 ` Ian Rogers 2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 44 ++-- tools/perf/tests/parse-events.c | 13 + tools/perf/tests/shell/record_bpf_filter.sh | 20 ++ tools/perf/util/bpf-filter.c | 237 ++++++++++++++++++- tools/perf/util/bpf_skel/lock_data.h | 47 +++- tools/perf/util/bpf_skel/sample-filter.h | 14 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 136 ++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 490 insertions(+), 31 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 8fc3196ed93d..6869a465d8e3 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -225,11 +225,17 @@ OPTIONS 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock', - 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex', - 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R', - 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore'). - Both read/shared and write/exclusive tracking are supported for - 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock' + 'siglock', 'rq_lock', 'zone_lock', 'rcu', 'rcu-trace'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', 'rcu:W', 'semaphore'). Both + read/shared and write/exclusive tracking are supported for 'rwsem' + (including 'mmap_lock'), 'rwlock', 'rcu', and 'srcu' (where + read/shared mode matches 'rcu_read_lock()' / 'srcu_read_lock()' + critical sections and write/exclusive mode matches grace-period + waits 'synchronize_rcu*()', 'synchronize_srcu*()', 'rcu_barrier()', + and 'srcu_barrier()'); 'rcu-trace' ('rcu_read_lock_trace()') + supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'semaphore' tracking @@ -251,19 +257,21 @@ OPTIONS --filter 'lock_held(mutex) || lock_held(rwsem)' When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol - ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' - or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's - 'owner' field directly in kernel memory on each sample without attaching - any BPF fentry/fexit hooks. For locks that do not maintain an - authoritative per-task owner pointer (such as spinlocks, rwlocks, - semaphores, reader-locked rwsems, and slab/type filters), it attaches - BPF fentry/fexit programs to kernel lock acquire and release functions - to track held locks in a BPF map. These BPF hooks run on every kernel - lock acquisition and release for the attached lock family, even when no - perf sample is being taken. To minimize this overhead, only the hooks - for the relevant lock family and mode are attached when known (for - example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions - and 'rq_lock' only hooks spinlock functions), whereas slab filters + ('*_rt_mutex'), write-locked 'mmap_lock' ('lock_held_write(mmap_lock)' + or 'lock_held(mmap_lock:W)'), or 'rcu:R' / 'rcu-trace' (which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'), + 'perf_sample_filter' inspects the kernel fields directly on each sample + without attaching any BPF fentry/fexit hooks. For locks that do not + maintain an authoritative per-task owner pointer (such as spinlocks, + rwlocks, semaphores, reader-locked rwsems, SRCU locks, RCU synchronize + waits, and slab/type filters), it attaches BPF fentry/fexit programs to + kernel lock acquire and release functions to track held locks in a BPF + map. These BPF hooks run on every kernel lock acquisition and release + for the attached lock family, even when no perf sample is being taken. + To minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(mmap_lock)' + only hooks rwsem read functions, 'srcu' only hooks SRCU functions, and + 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index ebaf8ee41622..47f697f53a85 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -247,6 +247,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 1a75d779e891..0161ba1f4769 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -200,11 +200,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU & ~LOCK_PROG_SEM, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -324,6 +332,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -642,6 +676,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) supported |= LOCK_PROG_PERCPU_RWSEM; + if ((btf_has_func(btf, "__srcu_read_lock") || + btf_has_func(btf, "__srcu_read_lock_nmisafe")) && + (btf_has_func(btf, "__srcu_read_unlock") || + btf_has_func(btf, "__srcu_read_unlock_nmisafe"))) + supported |= LOCK_PROG_SRCU_READ; + + if (btf_has_func(btf, "synchronize_srcu")) + supported |= LOCK_PROG_SRCU_WRITE; + + if (btf_has_func(btf, "synchronize_rcu")) + supported |= LOCK_PROG_RCU_WRITE; + return supported; } @@ -759,6 +805,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RCU_WRITE; + *required_groups |= LOCK_PROG_RCU_WRITE; + } + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) { *prog_groups |= groups; *required_groups |= groups; @@ -800,6 +859,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, groups |= LOCK_PROG_RWLOCK_WRITE; *prog_groups |= groups; *required_groups |= groups; + } else if (str_ends_with(base, "srcu")) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_SRCU_WRITE; + *prog_groups |= groups; + *required_groups |= groups; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -963,6 +1029,108 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name) return btf__find_by_name_kind(btf, rc_name, BTF_KIND_VAR) > 0; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + /* + * Non-sleepable BPF sample filter programs always run under at least + * one kernel rcu_read_lock(): + * - PERF_TYPE_PERF_EVENT: bpf_overflow_handler() takes rcu_read_lock() + * (and perf_swevent_event() takes an outer rcu_read_lock() for + * non-timer PERF_TYPE_SOFTWARE events, giving depth 2). + * - Standard tracepoints: trace_call_bpf() takes rcu_read_lock(). + * - Faultable syscall tracepoints: trace_call_bpf_faultable() invokes + * bpf_prog_run_array_sleepable(), which takes guard(rcu)() for + * non-sleepable programs (!prog->sleepable) such as + * perf_sample_filter_tp. + */ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static u64 rcu_trace_min_depth(struct evsel *evsel, struct btf *btf) +{ + const char *ev_name; + + /* + * Only PERF_TYPE_PERF_EVENT and PERF_TYPE_TRACEPOINT events can attach + * sample_filter.bpf.c (kprobes and uprobes require BPF_PROG_TYPE_KPROBE + * in __perf_event_set_bpf_prog()). + * + * For syscall tracepoints: + * - raw_syscalls:sys_enter and raw_syscalls:sys_exit are defined with + * TRACE_EVENT_SYSCALL(), whose __do_trace_sys_enter/exit wrapper + * takes guard(rcu_tasks_trace)() (depth 1) before perf_trace_buf + * submission calls trace_call_bpf(). + * - syscalls:sys_enter_* and syscalls:sys_exit_* register + * perf_syscall_enter/exit on the raw sys_enter/sys_exit tracepoints + * (1st guard(rcu_tasks_trace)()) and, on kernels with + * trace_call_bpf_faultable(), acquire a 2nd guard(rcu_tasks_trace)() + * in trace_call_bpf_faultable() (depth 2). + */ + if (evsel && evsel->core.attr.type == PERF_TYPE_TRACEPOINT) { + ev_name = evsel__name(evsel); + if (!ev_name) + return 0; + if (strstarts(ev_name, "raw_syscalls:")) + return 1; + if (strstarts(ev_name, "syscalls:")) { + if (!btf || + btf__find_by_name_kind(btf, "trace_call_bpf_faultable", + BTF_KIND_FUNC) > 0) + return 2; + return 1; + } + } + return 0; +} + +static bool btf_struct_has_field(const struct btf *btf, s32 type_id, + const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + u16 i, vlen; + + type_id = btf__resolve_type(btf, type_id); + if (type_id < 0) + return false; + + t = btf__type_by_id(btf, type_id); + if (!t || (!btf_is_struct(t) && !btf_is_union(t))) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && *name) { + if (!strcmp(name, field_name)) + return true; + } else if (btf_struct_has_field(btf, m->type, field_name)) { + return true; + } + } + return false; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + s32 id; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + return btf_struct_has_field(btf, id, field_name); +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->op`, * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`, @@ -970,15 +1138,15 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name) * * Resolution order: * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`). - * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock`. + * 2. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`. * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, - * `rwlock`, `percpu-rwsem`, `semaphore`, etc. + * `rwlock`, `percpu-rwsem`, `srcu`, `semaphore`, etc. * 4. Hexadecimal lock address. * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`). */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -1060,6 +1228,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->value = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->value = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->value = rcu_trace_min_depth(evsel, btf); + return 0; + } + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->value = encoded_pair; @@ -1149,7 +1359,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].term = expr->term; entry[i].value = expr->val; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, btf, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -1167,7 +1377,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].term = group->term; entry[i].value = group->val; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, btf, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1496,6 +1706,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0) + rcu_addr += 1; + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, needs_lock_syms, needs_slab_iter, is_tracepoint); diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 43d8f5030bc1..f3f8076fdd0b 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -264,6 +267,43 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). `min_depth` accounts for any baseline nesting + * contributed by the kernel tracing wrapper itself (2 on faultable + * `syscalls:*` tracepoints via `__DECLARE_TRACE_SYSCALL` and + * `trace_call_bpf_faultable()`, 1 on `raw_syscalls:*`, 0 otherwise). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > min_depth; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -296,6 +336,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index a65ee780d64d..7abc530ed50d 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -101,6 +101,20 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.value`, + * where `entry.value` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > entry.value`, + * where `entry.value` is the baseline nesting depth added by the + * kernel event tracing path. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index d1192a93179e..60fe4c3a99a2 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -129,6 +129,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -142,6 +143,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; int track_rwsem_read; __u8 lock_type_filter[LCB_F_TYPE_MAX]; @@ -282,6 +284,21 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry) case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0; + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->value)) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return true; + return false; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_trace_lock_held(curr, (int)entry->value)) + return true; + return false; + default: return false; } @@ -727,11 +744,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, type_flag); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1391,6 +1409,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index f6e68d740860..37415f7b7f2c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -39,6 +39,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (9 preceding siblings ...) 2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-22 20:30 ` Ian Rogers 10 siblings, 0 replies; 59+ messages in thread From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 174 +++++++++++++- .../perf/util/bpf_skel/lock_contention.bpf.c | 222 ++++++++++++++---- tools/perf/util/machine.c | 11 + 5 files changed, 366 insertions(+), 62 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..999dcfd1ef26 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..081a1ecc10a6 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,10 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <stddef.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +184,133 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + /* + * Offset by 1 byte so rcu_state_addr is distinct from + * &rcu_state.node[0].lock at offset 0 of rcu_state. + */ + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1; + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +318,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +332,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +382,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +406,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +442,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +459,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +642,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +750,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index c70c3955e559..8ae3fc889528 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -170,6 +183,8 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 rcu_state_addr; + int enabled; int perf_subsys_id = -1; @@ -213,7 +228,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -237,7 +252,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -246,10 +260,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -267,26 +280,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -346,6 +360,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -358,8 +373,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -367,12 +383,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -469,21 +485,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -554,8 +569,21 @@ int contention_begin(u64 *ctx) } } skip_owner: - pelem->stack_id = bpf_get_stackid(ctx, &stacks, - BPF_F_FAST_STACK_CMP | stack_skip); + { + int skip = stack_skip; + + /* + * Unlike tp_btf/contention_begin (which fires inside a + * lock slowpath via trace_contention_begin), fentry + * RCU/SRCU programs attach directly to the top-level + * wait API, which is 2 frames shallower. + */ + if (flags & (LCB_F_RCU | LCB_F_SRCU)) + skip = stack_skip > 2 ? stack_skip - 2 : 0; + + pelem->stack_id = bpf_get_stackid(ctx, &stacks, + BPF_F_FAST_STACK_CMP | skip); + } if (pelem->stack_id < 0) __sync_fetch_and_add(&stack_fail, 1); } else if (aggr_mode == LOCK_AGGR_TASK) { @@ -583,8 +611,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -592,30 +619,34 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; /* - * For spinlock and rwlock, it needs to get the timestamp for the - * per-cpu map. However, contention_end does not have the flags - * so it cannot know whether it reads percpu or hash map. - * - * Try per-cpu map first and check if there's active contention. - * If it is, do not read hash map because it cannot go to sleeping - * locks before releasing the spinning locks. + * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu + * map. For tp_btf/contention_end, try the per-cpu map first (used by + * spinlocks and rwlocks) and fall back to the per-task tstamp map. */ - pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); - if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) - return 0; - } else { + if (is_rcu) { pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) return 0; - need_delete = true; + ts_map = &tstamp_rcu; + } else { + pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); + if (pelem && pelem->lock) { + if (pelem->lock != lock) + return 0; + } else { + pid = bpf_get_current_pid_tgid(); + pelem = bpf_map_lookup_elem(&tstamp, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp; + } } timestamp = bpf_ktime_get_ns(); @@ -643,7 +674,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -708,7 +738,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -790,11 +820,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0, true); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0, true); +} + const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; const volatile int nr_nodes; diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index 1d9d3bf57720..0a0f223b3335 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr) return true; } + { + struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL); + + if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) || + !strncmp(sym->name, "synchronize_srcu", 16) || + !strcmp(sym->name, "__synchronize_srcu") || + !strcmp(sym->name, "rcu_barrier") || + !strcmp(sym->name, "srcu_barrier"))) + return true; + } + return false; } -- 2.56.0.307.g5fe7bd680c-goog ^ permalink raw reply [flat|nested] 59+ messages in thread
end of thread, other threads:[~2026-10-22 22:06 UTC | newest] Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers 2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers 2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers 2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers 2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers 2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers 2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim 2026-10-19 19:08 ` Ian Rogers 2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers 2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-20 4:34 ` Namhyung Kim 2026-10-21 21:39 ` Arnaldo Carvalho de Melo 2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers 2026-10-20 5:41 ` Namhyung Kim 2026-10-20 15:30 ` Ian Rogers 2026-10-20 22:29 ` Namhyung Kim 2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers 2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers 2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers 2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers 2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-20 22:42 ` Namhyung Kim 2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers 2026-10-20 22:49 ` Namhyung Kim 2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers 2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers 2026-10-22 22:06 ` Namhyung Kim 2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers 2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers 2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers 2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers 2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers 2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers 2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers 2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®