From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f198.google.com (mail-dy1-f198.google.com [74.125.82.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3099A3E49FD for ; Sat, 17 Oct 2026 05:46:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.198 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792215996; cv=none; b=EOjE+BJbqrIn3SNyH7PaPXJSr5JORfNcscB6rexPQY1BFoiQ+f052irXQtOlMXBqFbtgFzDDNO6D8B6rgDXnx4a5ZiWVDVev/GbS5dT4KuRv4SS7JIw03GN4XW+zfsg4xUeAvh1ThaB8V9/Tw4hFsiGyHd4Z4bRCOy/z9JUgpJs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792215996; c=relaxed/simple; bh=dEZ6JOZsV0uRahUeVcEpxrTduQptuPKacIhAc412PGU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=l1xw2CzRtAbYGIhJCzYMqRK0CEELxPfT0g7VXsA3Dm2fIjdGAKqL8Ri1i3C3yyLU7fxUtB/4Y01cmIfGZYIhW4AHizclZDKtdjlonU5nexC1mHKaOJ7A2/B4+yNg2qscZnVDfEfldpWY0sMNUqmvZ89TjZydT6JQsIw2fCzwi/w= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Hc1dlwec; arc=none smtp.client-ip=74.125.82.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Hc1dlwec" Received: by mail-dy1-f198.google.com with SMTP id 5a478bee46e88-35c5aef2a74so3410777eec.0 for ; Fri, 16 Oct 2026 22:46:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792215990; x=1792820790; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=vtJl92KrpgGBhMWCQBrE1Wd++/gIWcercwbCHavk80s=; b=Hc1dlwecOnejFiXpXI3uOW40V4hfEGiDxDlSpu+yRllpr/uz38jjlLQA9hLqG0zBM5 i4eIOwBEBdwUnzmf64xw7+jqMo1NnkOT8gskuFvQiPRrT4z18e/86Viz8J8u2E/+6GmX FV95k4LxlgMUbo9rdGZkbdcUTxmJQc0KQmkWITGFSFHtjTfzHZa/KQq6H1eiamtcI1H8 XOQu6psajnF5IjZKyK/I4wWduoCPvbO8uWGBlpmE6kEGJ0pmXDDI5W1VMc6bAadOF1R4 FiO594KiLzgTMFJIHduEFEVnw5Csw+t/ZPcP8C1N9n5SSd+DgHTh3LZr2lhJQPh0Q5g5 FdwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792215990; x=1792820790; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vtJl92KrpgGBhMWCQBrE1Wd++/gIWcercwbCHavk80s=; b=SGRkAczupsAwScB7ZRX3vQThA8Rf98QhxIwhsi4WnLqaKXuFE8OT1d2sjUog1O5G+V Tsq6j2zcwPijCYKdKTUyUwu5DsADz+AECKalby39dmN2D1Mw+GTA4iV8PrY7pJs7TmaB GvZqJQCtN6Eiuv5pjKObpE16/lo/omXjc3RdCCiy104Pe89HVZ6Q8A1Va5KE8/lVLXGR z8QU2CJ8+mAisuCffzVC2BNS/sONEQv6xzSUoemcwhzBEmaiKzAR3MtuReEJ6evnYrhZ TzsdvYRHInCHw7F+auOnWuLEx3OufTBv38iEYflFiG0zzj84zYVzr5W2oQvlXCz/rkRr DPyw== X-Forwarded-Encrypted: i=1; AKwUvBy/S4rV5zif2SRAE73VzMgsp/D0zPn/O1zFYgWHcdO/ondE/2e8AqUURkqUP2n2OT6JU8ETDpH/yMzbYVg=@vger.kernel.org X-Gm-Message-State: AFq9FYKt/HFhRrf6rVDCWzALwKxybZHuafvRDeYcNM9eMaAaYqxVpFxz YDZddwt0jhDx3YAcDNdlsEQYitjlTRMKoZX+gAHPyoji2Tnap8l2dMVqVFSSWQnlYJNLljcvT/H goPmsPT8PVQ== X-Received: from dyuw26-n1.prod.google.com ([2002:a05:693c:881a:10b0:351:7270:d79a]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7301:a02:b0:360:8596:657d with SMTP id 5a478bee46e88-361a3128fdamr2226710eec.32.1792215988097; Fri, 16 Oct 2026 22:46:28 -0700 (PDT) Date: Fri, 16 Oct 2026 22:45:52 -0700 In-Reply-To: <20261017054555.1655407-1-irogers@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261017054555.1655407-1-irogers@google.com> X-Mailer: git-send-email 2.56.0.212.g129ede20d9-goog Message-ID: <20261017054555.1655407-3-irogers@google.com> Subject: [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter From: Ian Rogers To: Arnaldo Carvalho de Melo , Namhyung Kim Cc: Peter Zijlstra , Ingo Molnar , Jiri Olsa , Ian Rogers , Adrian Hunter , James Clark , linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Add BPF sample filter predicates lock_held(), lock_held_read() / lock_held_shared(), and lock_held_write() / lock_held_exclusive() (as well as ':R' and ':W' suffixes on ) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported specifications match perf lock contention: - Global/static kernel lock symbols (with or without '&', e.g. lock_held(&cgroup_mutex) or lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. To minimize tracing overhead, only the BPF hooks for the requested lock family and mode are autoloaded and attached. Example usage: # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ... Signed-off-by: Ian Rogers --- tools/perf/Documentation/perf-record.txt | 41 +- tools/perf/util/bpf-filter.c | 934 ++++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_skel/sample-filter.h | 59 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 920 ++++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2021 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..246ae085744d 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,46 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(), + lock_held_read(), lock_held_shared(), + lock_held_write(), lock_held_exclusive() + + The 'lock_held()' predicate (or '!lock_held()') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read()' + (alias 'lock_held_shared()') or 'lock_held_write()' + (alias 'lock_held_exclusive()'), or append ':R' or ':W' to + (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)'). + can be a kernel lock symbol name (with or without a leading + '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), + a hexadecimal lock address, a slab cache name prefixed with '&' + (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'semaphore'). Both read/shared and write/exclusive tracking are + supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), + 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') + support write/exclusive tracking only. + + Note that 'lock_held()' introduces overhead outside of sample + overflow handling: in addition to checking the lock's owner field on + each sample where possible (such as for mutexes, rt-mutexes, and + write-locked rwsems), it attaches BPF fentry/fexit programs to kernel + lock acquire and release functions to track held locks (such as + spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a + BPF map. These BPF hooks run on every kernel lock acquisition and + release for the attached lock family, even when no perf sample is + being taken. To minimize this overhead, only the hooks for the + relevant lock family and mode are attached when known (for example, + 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, + 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, + '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + spinlock functions), whereas slab filters ('&'), raw hex + addresses, or symbols without a recognizable lock-type suffix attach + hooks across all lock families (including high-frequency spinlocks). The can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..35d6f7c63868 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include +#include #include +#include +#include +#include #include +#include #include #include #include +#include #include +#include #include +#include +#include #include #include #include +#include #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include #include +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -155,9 +171,713 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) { - int i = 0; + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) == 0) + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, bool needs_lock_syms, + bool needs_slab_iter) +{ + unsigned int enabled_progs = 0; + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWSEM_WRITE; + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups)) + return; + + if (!has_amp && parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (has_amp) + *needs_slab_iter = true; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { + *prog_groups |= LOCK_PROG_RTMUTEX; + } else if (strtailcmp(base, "mutex") == 0) { + *prog_groups |= LOCK_PROG_MUTEX; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWSEM_WRITE; + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWLOCK_WRITE; + } else { + *prog_groups |= LOCK_PROG_ALL; + } + + if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0) + *needs_slab_iter = true; +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + needs_lock_syms, needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, needs_lock_syms, + needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +/* + * Resolve a `lock_held()` filter expression into `entry->part` + * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate + * the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock` (with or without leading '&'). + * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `semaphore`, etc. + * 3. Hexadecimal lock address (when no leading '&'). + * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&', + * e.g. `&cgroup_mutex` or `cgroup_mutex`). + * 6. Slab cache name without leading '&' fallback. + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) +{ + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->target = 0; + if (skel) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->target = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->target = encoded_pair; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (!has_amp && parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + if (has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + if (!has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) +{ + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -171,6 +891,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + entry[i].target = 0; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -184,6 +910,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + entry[i].target = 0; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -423,10 +1155,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct bpf_program *prog; struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0; #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -438,15 +1173,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &needs_lock_syms, &needs_slab_iter); - if (needs_idx_hash && geteuid() != 0) { + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + if (has_lock_filter) { + pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -484,13 +1228,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms, + needs_slab_iter); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, false, false); + } + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -513,11 +1313,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -532,6 +1334,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -539,7 +1356,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -591,7 +1408,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -602,6 +1419,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -622,14 +1461,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -641,6 +1477,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -654,6 +1537,8 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, false, false); + /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); @@ -674,7 +1559,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -683,7 +1568,7 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); + unpin_filter_object(skel, path); ret = dir_fd; goto err; } @@ -727,6 +1612,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -740,13 +1627,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + configure_lock_progs(skel, NULL, 0, false, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -754,7 +1648,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include #include +#include #include #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +{space} { } +[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +")" { BEGIN(INITIAL); return ')'; } +.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..6ee6edef13f7 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -11,6 +11,7 @@ #include #include #include +#include #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type filter_term filter_expr -%destructor { free ($$); } +%destructor { perf_bpf_filter_expr__delete($$); } +%destructor { free($$); } +%destructor { free($$.name); } %type BFT_SAMPLE BFT_SAMPLE_PATH %type BFT_OP -%type BFT_NUM +%type BFT_NUM BFT_LOCK_HELD %type BFT_PATH +%type BFT_LOCK_NAME +%type lock_term %% @@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..cd711d1d04b6 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,14 +53,73 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the + * specified kernel lock is currently held by the running task/CPU, or + * 0 otherwise. The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`) and the resolved target address, + * class ID, or type flag encoding is stored in `entry.target`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.target` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.target` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.target` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; __u32 part; /* sub-sample type info when it has multiple values */ enum perf_bpf_filter_term term; __u64 value; + __u64 target; /* lock address, kmem_cache *, class, or type flags */ }; struct idx_hash_key { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..fbd2e51815d7 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include #include +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,198 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr) +{ + if (!curr) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr; + __u32 pid; + __u32 cpu_id; + __u32 kind; + __u32 mode; + __u64 lock_addr; + + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->target. + */ + if (use_idx_hash || !lock_enabled) + return 0; + + curr = bpf_get_current_task(); + pid = (__u32)bpf_get_current_pid_tgid(); + cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + kind = entry->part & PBF_LOCK_KIND_MASK; + mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->target; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return 0; + if ((mode & PBF_LOCK_MODE_WRITE) && + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return 0; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id); + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id); + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + + default: + return 0; + } +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +402,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -295,4 +524,695 @@ int perf_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, flags); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) + __sync_fetch_and_add(&existing->read_count, 1); + else + __sync_fetch_and_add(&existing->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count <= 1 && existing->write_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else if (existing->read_count > 0) + __sync_fetch_and_add(&existing->read_count, -1); + } else { + if (existing->write_count == 0 && existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog