From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6FE23B8D78 for ; Sun, 30 Aug 2026 14:36:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788100610; cv=none; b=B5Xx0w2C2wAxT3DUqKVu4rcJTBrq9ytaabcTXLy6AKl1KxBX/XqIHbQHJ9NoyJchVO4GjdjWt1N1gKhMudBPHcrnYJTPbICntumdN1JIfHPrE5fwaPSP1TgDwtA8Ut5M3/MXIM6ZkuR0woOKCoglLPk8ne1+mOIlbJFc4AE7/8g= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788100610; c=relaxed/simple; bh=EKGu/YNz/Y7Su/Py1AF/c0JjS1K1+FhCsnHRcD6212k=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=l+CQD8bTYx97n7GHbUVBGXy8jkPbu17OuPFn6O7IZp92wU2kNyPMpqjc4gxEinZ8W529xEMq38ewMbeDtueGItWpcEm5ssBW6G49WtD1S/OGqn59h+P/0Q+79Jerlsswu4wPWlc+hSl1cMBw416oNQb/m8r40TmOu4iO8NXIpn4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M6AMajcv; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M6AMajcv" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-482e5733a5aso1539916f8f.0 for ; Sun, 30 Aug 2026 07:36:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788100607; x=1788705407; darn=vger.kernel.org; h=cc:to:message-id:content-transfer-encoding:content-type :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=K5qhWC1RezqgUxSYoHlPtRL/UlEjuJcpyuDmW9yQF/c=; b=M6AMajcv9d3bV/V/kHgyF0q07U/GB6zk1WjBYchXaG90SUXQsdxuLEbWlg0w7JxGVk 54UW7lpLUDqMgPGEO4W96uLKdPHpQufeI1V2crndm0gg8ezVSkAnoc+/JX2I7S2BwN83 p5v8dirUIg4oMLK8xecr59msl+C1XqWCN1XFtR9UHOuxKvTVOKzXnPbH1oTvWyq8GU// J0CdbMtloGrtC37rS+ZTceUd6vBcpOUa1EAIHDSD+6Yvov2TmytI6+jP7YsLQDyoGspK yy3JHVxuskPZco14uinURQU2IIb8avZpssa8b2J8pWbP9pPVQVy6tY9sbNCGgq17AR2H TYYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788100607; x=1788705407; h=cc:to:message-id:content-transfer-encoding:content-type :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=K5qhWC1RezqgUxSYoHlPtRL/UlEjuJcpyuDmW9yQF/c=; b=AUfQkQ5cwqos/XLuHb74v8t+czyJceOfH+uOznx+ZAWeDvHd2OddTU1xhcHQ60Dudl ivenwVJpz6Sqm+UJ4nkha4Rifn7hFa3TVWGeV49boBnsuiRJnrlrgn61ypnVRhEMBQ9Q VYzjM4NOEyewmTpxgeung4k272JUU43pceNTora64hpq2QYjcSMAfc8zjzwh35oz7vAj h5csa7EyS7FUdkqWbLMuiVj69YhSOHDF4vIl0t6t7RLSUAVT7c7K4xgRJP0nLw/8wweC QQjcOMHlzC40B+Wlrh+9fjNT6x4WBxxSzcpeN/GON679ebX/HNJw1f6Yo13goXnvplJA J00g== X-Forwarded-Encrypted: i=1; AHgh+RpUEXmt0Gdw+zxbykLc7xaiBZChU4qCSArez5GapQj8+Jjc4eRqdEVFFfkaRVuNgynxWoHZg41cK4N0Uf4=@vger.kernel.org X-Gm-Message-State: AFuF++mKwJArTYyLfu+zOcjtTLycxv54F0HuuTqq6ASJQCB/C6uzBv2K rrIgZTimUd3ivF+toM0hrfd3zq8H2F75xUjIVRvR0mv/62SxLB7x6YtA X-Gm-Gg: AR+sD1308uPuA7FSZb0k+RTBOFAwmjYuQgkcierMpI5JgOA1QntEAHToAi+HV5nxVFx LIT1fnGkzMjePT5g4BYUDUcNoIqODPSjvZf8tkc7Pa8FnZn3LQDvyVx8cz5thf6cLEzmwk9R8vd tfRf9nBe2A4JqwQOZDF4nPVIhwomwNmfjsahSN7mC0vPGZfFKhZYA6aRbtLpB/dN7vP5DMcj+at g5DmiYvC4YpG5XHVhC+WSvsaHlbTLhEsmLqmlH5h8rJwDe1z6sS6UckGeNCJi5Vn3DyAj84sywk DRHJfH2x1g9iGcV9IskSxD108w8B1opdma12uXO9a6CUDojj9d9mPmaLE5j+uk3/zleH/0/zufi dwYbCKKVYyUsGDko/I1SAS5iLWTxVdN9UQjtn6dvuEWjkxjHtzoHJ01UPmDhOIPd6WB8xYIpYAK VL9b6nnOs3T4oJYZrMuiHscv007ZK7vX8oG/QVaIdAoxt7zFc8k7qUGM00Ydyj9lb74uSpq7H09 t11PK8GdML9HcdVi/Wa X-Received: by 2002:a05:600c:8885:b0:49b:909e:922e with SMTP id 5b1f17b1804b1-49b91c4884amr257020755e9.10.1788100606661; Sun, 30 Aug 2026 07:36:46 -0700 (PDT) Received: from [127.0.1.1] (89-65-152-218.dynamic.play.pl. [89.65.152.218]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49b9500c80asm245179715e9.9.2026.08.30.07.36.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Aug 2026 07:36:46 -0700 (PDT) From: Roman 'Hedin' Storozhenko Date: Sun, 30 Aug 2026 16:36:37 +0200 Subject: [PATCH v2] riscv: mm: Trace TLB flush path selection Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260830-tlb_tracepoint-v2-1-e3c88c24df5b@gmail.com> X-B4-Tracking: v=1; b=H4sIAPQ/lGoC/3XM0QqDIBTG8VeJcz2Hiq3a1d5jxDA91YHKUJGN8 N3nut/l/4Pvd0BATxjgXh3gMVEgt5WQlwrMrLcJGdnSILm88VZ2LC7DK3ptcHe0RdYqJXith5p 3Espp9zjS+wSffemZQnT+c/pJ/Na/VBJMMDtajRYbbJR4TKum5WrcCn3O+QuOmNa2rAAAAA== To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Roman 'Hedin' Storozhenko X-Mailer: b4 0.13.0 Make RISC-V TLB flush path selection observable. Record whether Linux handles an invalidation locally, delegates it to SBI RFENCE, or executes it through a cross-CPU call, so MM activity can be correlated with the RISC-V, firmware, or Linux cross-CPU path carrying the request. The generic tlb:tlb_flush event describes TLB flush activity using architecture-independent reason and page-count information. The RISC-V implementation subsequently selects between local invalidation, SBI RFENCE, and Linux cross-CPU coordination, with additional architecture-specific request context available at that point. Making this selection observable is useful when debugging RISC-V TLB shootdowns. When a remote invalidation is observed to be slow, the selected path determines whether to investigate SBI firmware and platform handling or Linux cross-CPU and IPI handling. An unexpectedly broad target mask can reveal an unintended address-space CPU footprint, while the range and stride distinguish invalidation requests with different mapping granularities. Place the event in the RISC-V implementation because the local, SBI RFENCE, or cross-CPU choice is made there, and SBI RFENCE and the invalidation stride are RISC-V-specific semantics rather than properties of the generic MM flush request. Add riscv_tlb:riscv_tlb_flush_path in flush_tlb_all() and __flush_tlb_range(). Record start, size, stride, the hardware-visible ASID, whether a specific mm is associated with the request, the target CPU mask and its weight, the requested scope, and the selected path. Record the complete target mask in addition to its weight because CPU identity cannot be reconstructed from a count and is needed to correlate the request with per-CPU scheduler, IPI, and firmware activity. The event records the invalidation request and the path selected by Linux before the operation is dispatched. In particular, selecting the SBI RFENCE path means that Linux delegated the request to firmware; the event does not describe the implementation or outcome of that delegated operation. Tested on QEMU virt with OpenSBI using local and shared-mm mprotect()/munmap() workloads. Local requests reported path=local, while remote requests reported path=sbi-rfence and were followed by the existing riscv:sbi_call RFENCE event. The cross-CPU-call path was tested with QEMU virt using APLIC+IMSIC. A MADV_PAGEOUT reclaim workload was used to exercise mm-independent global flushes. All reported path values (local, sbi-rfence and cross-cpu-call) and scope values (single, range, address-space and all) were observed. Signed-off-by: Roman 'Hedin' Storozhenko --- Add a RISC-V tracepoint for observing the path selected by Linux for TLB invalidation requests: local invalidation, SBI RFENCE, or Linux cross-CPU coordination. The tracepoint is intended to make RISC-V TLB shootdown behavior easier to correlate with MM activity, CPU targeting, SBI calls, and IPI handling. The patch records the invalidation request context and the Linux path-selection decision before the operation is dispatched. The patch was tested on QEMU virt with both the SBI RFENCE path and an APLIC+IMSIC configuration. Local, SBI RFENCE, and cross-CPU-call paths were exercised. All reported scope values -- single, range, address-space, and all -- were also observed. --- Changes in v2: - Use trace_call__riscv_tlb_flush_path() after the explicit trace_riscv_tlb_flush_path_enabled() check to avoid a second tracepoint static-key test, as suggested by Steven Rostedt. - Link to v1: https://lore.kernel.org/r/20260829-tlb_tracepoint-v1-1-dfdaede7e741@gmail.com --- arch/riscv/mm/tlbflush.c | 60 +++++++++++++++++++-- include/trace/events/riscv_tlb.h | 113 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+), 4 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 962db300a166..cefce9364bd2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -9,6 +9,9 @@ #include #include +#define CREATE_TRACE_POINTS +#include + #define has_svinval() riscv_has_extension_unlikely(RISCV_ISA_EXT_SVINVAL) /* @@ -63,6 +66,33 @@ void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) local_flush_tlb_range_asid(start, end - start, PAGE_SIZE, FLUSH_TLB_NO_ASID); } +static enum riscv_tlb_flush_scope +riscv_tlb_get_flush_scope(unsigned long size, unsigned long stride, bool has_mm) +{ + if (size == FLUSH_TLB_MAX_SIZE) + return has_mm ? RISCV_TLB_FLUSH_SCOPE_ADDRESS_SPACE : + RISCV_TLB_FLUSH_SCOPE_ALL; + + return size <= stride ? RISCV_TLB_FLUSH_SCOPE_SINGLE : + RISCV_TLB_FLUSH_SCOPE_RANGE; +} + +static __always_inline void +riscv_tlb_trace_flush_path(const struct cpumask *cmask, unsigned long start, + unsigned long size, unsigned long stride, + unsigned long asid, bool has_mm, + enum riscv_tlb_flush_path path) +{ + enum riscv_tlb_flush_scope scope; + + if (!trace_riscv_tlb_flush_path_enabled()) + return; + + scope = riscv_tlb_get_flush_scope(size, stride, has_mm); + trace_call__riscv_tlb_flush_path(start, size, stride, asid, has_mm, + cmask, scope, path); +} + static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); @@ -70,12 +100,26 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (num_online_cpus() < 2) + if (num_online_cpus() < 2) { + riscv_tlb_trace_flush_path(cpu_online_mask, 0, + FLUSH_TLB_MAX_SIZE, 0, + FLUSH_TLB_NO_ASID, false, + RISCV_TLB_FLUSH_PATH_LOCAL); local_flush_tlb_all(); - else if (riscv_use_sbi_for_rfence()) - sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); - else + } else if (riscv_use_sbi_for_rfence()) { + riscv_tlb_trace_flush_path(cpu_online_mask, 0, + FLUSH_TLB_MAX_SIZE, 0, + FLUSH_TLB_NO_ASID, false, + RISCV_TLB_FLUSH_PATH_SBI_RFENCE); + sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, + FLUSH_TLB_NO_ASID); + } else { + riscv_tlb_trace_flush_path(cpu_online_mask, 0, + FLUSH_TLB_MAX_SIZE, 0, + FLUSH_TLB_NO_ASID, false, + RISCV_TLB_FLUSH_PATH_CROSS_CPU_CALL); on_each_cpu(__ipi_flush_tlb_all, NULL, 1); + } } struct flush_tlb_range_data { @@ -107,12 +151,20 @@ static void __flush_tlb_range(struct mm_struct *mm, /* Check if the TLB flush needs to be sent to other CPUs. */ if (cpumask_any_but(cmask, cpu) >= nr_cpu_ids) { + riscv_tlb_trace_flush_path(cmask, start, size, stride, asid, + !!mm, RISCV_TLB_FLUSH_PATH_LOCAL); local_flush_tlb_range_asid(start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { + riscv_tlb_trace_flush_path(cmask, start, size, stride, asid, + !!mm, RISCV_TLB_FLUSH_PATH_SBI_RFENCE); sbi_remote_sfence_vma_asid(cmask, start, size, asid); } else { struct flush_tlb_range_data ftd; + riscv_tlb_trace_flush_path(cmask, start, size, stride, asid, + !!mm, + RISCV_TLB_FLUSH_PATH_CROSS_CPU_CALL); + ftd.asid = asid; ftd.start = start; ftd.size = size; diff --git a/include/trace/events/riscv_tlb.h b/include/trace/events/riscv_tlb.h new file mode 100644 index 000000000000..3eff171ec54f --- /dev/null +++ b/include/trace/events/riscv_tlb.h @@ -0,0 +1,113 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM riscv_tlb + +#if !defined(_TRACE_RISCV_TLB_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_RISCV_TLB_H + +#include +#include + +#ifndef _TRACE_RISCV_TLB_ENUMS +#define _TRACE_RISCV_TLB_ENUMS + +enum riscv_tlb_flush_scope { + RISCV_TLB_FLUSH_SCOPE_SINGLE, + RISCV_TLB_FLUSH_SCOPE_RANGE, + RISCV_TLB_FLUSH_SCOPE_ADDRESS_SPACE, + RISCV_TLB_FLUSH_SCOPE_ALL, +}; + +enum riscv_tlb_flush_path { + RISCV_TLB_FLUSH_PATH_LOCAL, + RISCV_TLB_FLUSH_PATH_SBI_RFENCE, + RISCV_TLB_FLUSH_PATH_CROSS_CPU_CALL, +}; + +#endif /* _TRACE_RISCV_TLB_ENUMS */ + +TRACE_DEFINE_ENUM(RISCV_TLB_FLUSH_SCOPE_SINGLE); +TRACE_DEFINE_ENUM(RISCV_TLB_FLUSH_SCOPE_RANGE); +TRACE_DEFINE_ENUM(RISCV_TLB_FLUSH_SCOPE_ADDRESS_SPACE); +TRACE_DEFINE_ENUM(RISCV_TLB_FLUSH_SCOPE_ALL); + +TRACE_DEFINE_ENUM(RISCV_TLB_FLUSH_PATH_LOCAL); +TRACE_DEFINE_ENUM(RISCV_TLB_FLUSH_PATH_SBI_RFENCE); +TRACE_DEFINE_ENUM(RISCV_TLB_FLUSH_PATH_CROSS_CPU_CALL); + +#define show_riscv_tlb_flush_scope(scope) \ + __print_symbolic(scope, \ + { RISCV_TLB_FLUSH_SCOPE_SINGLE, "single" }, \ + { RISCV_TLB_FLUSH_SCOPE_RANGE, "range" }, \ + { RISCV_TLB_FLUSH_SCOPE_ADDRESS_SPACE, "address-space" }, \ + { RISCV_TLB_FLUSH_SCOPE_ALL, "all" }) + +#define show_riscv_tlb_flush_path(path) \ + __print_symbolic(path, \ + { RISCV_TLB_FLUSH_PATH_LOCAL, "local" }, \ + { RISCV_TLB_FLUSH_PATH_SBI_RFENCE, "sbi-rfence" }, \ + { RISCV_TLB_FLUSH_PATH_CROSS_CPU_CALL, "cross-cpu-call" }) + +/* + * Record the invalidation request received by the RISC-V architecture code + * and the path selected by Linux. + * + * The target CPU mask represents the CPUs Linux intends to cover for the + * request. It can be correlated with per-CPU activity, but does not describe + * which harts ultimately performed an invalidation. + * + * The ASID is hardware-visible and may be reused. It must not be treated as a + * persistent identifier for an mm. + * + * The stride describes the invalidation granularity supplied to the RISC-V + * implementation. SBI RFENCE receives start, size and ASID, but not stride. + * + * The event is emitted at path selection time. For SBI RFENCE, it records + * delegation of the request to firmware; firmware processing after that + * point is outside the event's scope. + */ +TRACE_EVENT(riscv_tlb_flush_path, + TP_PROTO(unsigned long start, unsigned long size, + unsigned long stride, unsigned long asid, bool has_mm, + const struct cpumask *cmask, + enum riscv_tlb_flush_scope scope, + enum riscv_tlb_flush_path path), + + TP_ARGS(start, size, stride, asid, has_mm, cmask, scope, path), + + TP_STRUCT__entry( + __field(unsigned long, start) + __field(unsigned long, size) + __field(unsigned long, stride) + __field(unsigned long, asid) + __field(bool, has_mm) + __field(unsigned int, target_mask_weight) + __cpumask(target_cpus) + __field(u8, scope) + __field(u8, path) + ), + + TP_fast_assign( + __entry->start = start; + __entry->size = size; + __entry->stride = stride; + __entry->asid = asid; + __entry->has_mm = has_mm; + __entry->target_mask_weight = cpumask_weight(cmask); + __assign_cpumask(target_cpus, cpumask_bits(cmask)); + __entry->scope = scope; + __entry->path = path; + ), + + TP_printk("start=%#lx size=%#lx stride=%#lx asid=%#lx has_mm=%d target_mask_weight=%u target_cpus=%s scope=%s path=%s", + __entry->start, __entry->size, __entry->stride, + __entry->asid, __entry->has_mm, + __entry->target_mask_weight, __get_cpumask(target_cpus), + show_riscv_tlb_flush_scope(__entry->scope), + show_riscv_tlb_flush_path(__entry->path)) +); + +#endif /* _TRACE_RISCV_TLB_H */ + +/* This part must be outside protection. */ +#include --- base-commit: 77ae27fd98f3b548797c9f22c10ab5cf1c4ada53 change-id: 20260829-tlb_tracepoint-844105ab5092 Best regards, -- Roman 'Hedin' Storozhenko