From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C6EB2063D5 for ; Fri, 10 Jan 2025 05:13:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736486039; cv=none; b=rK5ks0GEI/wYbk2vaZSZpkhyh9JlspOLBIXQahmtzMJqgMGpZJzRuMDdEjHBuvEmIBpd9wmedH0wZ03exL8GBFbT5dKLnwkrsE72JaiNkX0ck/qN8CR3SVtHv5+4dWaKZ/mSd8JNj6ff1S8pkTnYwqWYwD3fflTXD6d2rHgIKck= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736486039; c=relaxed/simple; bh=slXJnd44zykQM6Ny+pdk7SNyYIz26u28jByG4PgsBrk=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=iJGKOnKko1mP3Cr2QdP/yBGx6wuu0A/sJBHAcFj/RNwiNko9SEAGEcoYE0rq1aB3PTd5/A8glFMRVqFx1mgTVB87CerB5GXS63EYOpJxvwpr3XXa8g9c4Dajq5stpm6nf/KXpqVhTXjQA+Dyk/IuYOCW5AU3Y632o96fldDM0U8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WYlmF1u7; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WYlmF1u7" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2166a1a5cc4so28573865ad.3 for ; Thu, 09 Jan 2025 21:13:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736486037; x=1737090837; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=blP3qUW1zAZ81VuDJCt7lJik7KORrQPlPvK969LS39g=; b=WYlmF1u7z/FlqNZs/U3CP4Q8w0mmfUR1Ul3eYY5J2YZpGVZmZF7x7aS4zGobEqq0PF KidxIKJ+HVZuX9HlMWcY1QedyON7+LTgHbL61x3xzM8SJHP5NGfrQeRvOmES8Dk4v8+p Kql9uvzn8C5SU2Qz9MNt/Rty4j0jH2sQCXLDL3KnFcJMpYsHE/5N1ja1fYDIKX7qP94q OrOTSwdD6AN8eOJFbpxTzFtf9mViKbE7mRigVch6MOmlukTGqyU0Pt2gQIuz5/CwERZ3 iA9Xo3Oaofd1xbOonE07omdXBWbkvsBldXVoAYr3Cm8vMoH/um6rV4b0snZnYUmTtfaJ qwzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736486037; x=1737090837; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=blP3qUW1zAZ81VuDJCt7lJik7KORrQPlPvK969LS39g=; b=GQDIHWL6EcIOCZCrhUoSB0XsGgAcaAyet8vjBgVURs7dFz1GXIP2KhNc3bKHiF2JnE 9Aa0w1ugqlRWoojlsNHGF7n2w11HA/eTaX/mo9PY6zlhMGGZJCLJ4qg584Edy7rqIMPd FJFJRMQ5RMFsQE8pe5q4a1NLQQs37gN5S+vovPUen7oq9ntRHBnNu/C+0i6hNpAlnS3J seZh7xjLwjCCIM7/nIEZFI45h92dDwGTIqVvAQy+lQrW8DsNzMuuYMRo9UXdaErcX8FG ijzlAgRcanv6bIpPNAcnSVtm/P/Xn4+dJwlK3F271vRG/I18p6aTeT54zp5czqONMADy fI6A== X-Gm-Message-State: AOJu0Yy3pgT8YyVaxL0ldt5pqpy3a945A7+KXrmGVzXFsVQh+K3cxj5h QIfA2dUJYGrb8p1FjCSBXe68oMJaYj65uom5zOI5HzxtRh41dVYiGpZF+ZAYrHSyyxk1L5MMtwV HwVc1mg6N+5aq76BdHMOEAX/GiOxlbXfd3x4HArhti+kv0J9pYpE06YepChpZLte0BVCYUIvs8Y +T7mCqeXcilC1M9ll9u5bl8dWUF9FAZXhk5NetOovR X-Google-Smtp-Source: AGHT+IH0Ei5gmc9/UKfOPVRdw1TevO8tp0MWg7nqnILImIxo9F7p5b2S63f7075CvI1N+QUliiIkh+qyPHA= X-Received: from pfbcz5.prod.google.com ([2002:aa7:9305:0:b0:72a:bc6b:89ad]) (user=ctshao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:244d:b0:1d9:fbc:457c with SMTP id adf61e73a8af0-1e88d0a4770mr16871431637.36.1736486036508; Thu, 09 Jan 2025 21:13:56 -0800 (PST) Date: Thu, 9 Jan 2025 21:11:40 -0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.47.1.688.g23fc6f90ad-goog Message-ID: <20250110051346.1507178-1-ctshao@google.com> Subject: [PATCH 0/4] perf lock: Tracing contention lock owner call stack From: Chun-Tse Shao To: linux-kernel@vger.kernel.org Cc: Chun-Tse Shao , peterz@infradead.org, mingo@redhat.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org Content-Type: text/plain; charset="UTF-8" For perf lock contention, the current owner tracking (-o option) only works with per-thread mode (-t option). Enabling call stack mode for owner can be useful for diagnosing why a system running slow in lock contention. Example output: $ sudo ~/linux/tools/perf/perf lock con -abvo -Y mutex-spin -E16 perf bench sched pipe ... contended total wait max wait avg wait type caller 171 1.55 ms 20.26 us 9.06 us mutex pipe_read+0x57 0xffffffffac6318e7 pipe_read+0x57 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 36 193.71 us 15.27 us 5.38 us mutex pipe_write+0x50 0xffffffffac631ee0 pipe_write+0x50 0xffffffffac6241db vfs_write+0x3bb 0xffffffffac6244ab ksys_write+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 4 51.22 us 16.47 us 12.80 us mutex do_epoll_wait+0x24d 0xffffffffac691f0d do_epoll_wait+0x24d 0xffffffffac69249b do_epoll_pwait.part.0+0xb 0xffffffffac693ba5 __x64_sys_epoll_pwait+0x95 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 2 20.88 us 11.95 us 10.44 us mutex do_epoll_wait+0x24d 0xffffffffac691f0d do_epoll_wait+0x24d 0xffffffffac693943 __x64_sys_epoll_wait+0x73 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 1 7.33 us 7.33 us 7.33 us mutex do_epoll_ctl+0x6c1 0xffffffffac692e01 do_epoll_ctl+0x6c1 0xffffffffac6937e0 __x64_sys_epoll_ctl+0x70 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 1 6.64 us 6.64 us 6.64 us mutex do_epoll_ctl+0x3d4 0xffffffffac692b14 do_epoll_ctl+0x3d4 0xffffffffac6937e0 __x64_sys_epoll_ctl+0x70 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 === owner stack trace === 3 31.24 us 15.27 us 10.41 us mutex pipe_read+0x348 0xffffffffac631bd8 pipe_read+0x348 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 ... Chun-Tse Shao (4): perf lock: Add bpf maps for owner stack tracing perf lock: Retrieve owner callstack in bpf program perf lock: Make rb_tree helper functions generic perf lock: Report owner stack in usermode tools/perf/builtin-lock.c | 55 +++-- tools/perf/util/bpf_lock_contention.c | 58 +++++- .../perf/util/bpf_skel/lock_contention.bpf.c | 189 +++++++++++++++++- tools/perf/util/bpf_skel/lock_data.h | 6 + tools/perf/util/lock-contention.h | 2 + 5 files changed, 290 insertions(+), 20 deletions(-) -- 2.47.1.688.g23fc6f90ad-goog