From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A50B1314D04 for ; Mon, 17 Nov 2025 03:49:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.65 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763351363; cv=none; b=Q1x/iU73oJttt1G4KIu0+XgEmUDIbJjopKYki8Vvw3WqyBkS7do7/jTFo5IJCnKl4a9dTZXefUYD05bHHW2ack+1ncjFSQjYQsUe0aBxN+mrhwn8tJbiuwYp0gtYlX2fezgYFxz1rmAXDFUA43MbNj/qDGsHKqZkaHBAaKaWXVU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763351363; c=relaxed/simple; bh=eN8XLnT7oDDbVfBQj9WK/lxzOCXsGmTMCnuXBGXGsuw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HbIeVmV/XZbeQHwh+Egc1URMy/ROOk4TTr6Q0BPQC/0E4B7+YBwCTzAE4yR23z42dhT4Pu3/w6OKgOLx1570r3h222gHf6QSxZcv/ibObGX9Nvm3l6DfUu3q2HKrJ0n2ZyaiJhvYMv1aNikZp10HjY/EPhTXu5Dp0S4A+xYde1o= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YB9odUOS; arc=none smtp.client-ip=209.85.216.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YB9odUOS" Received: by mail-pj1-f65.google.com with SMTP id 98e67ed59e1d1-343dfb673a8so3610062a91.0 for ; Sun, 16 Nov 2025 19:49:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763351361; x=1763956161; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tB6Rux0HrLKcgyll4hZbEBMFFfGOp9PtogHaw5ekHkk=; b=YB9odUOSt3QaSuY1CE6bSAlUnwP8jWfvN2KmY9jm/a2GAPgWKQc/NZvt5yRzxkcpF3 33chlIBSEYM/hNRhZPRcoh2BHsO0AMeRwdZ7C+gzOjTCUtzbTXwcv3eLKkjb7ZeA7vmj MksRaAfr2RamGsipWi8fHeHY4CSbk8bZu//xIpMm7PiPtCKQeGZ8mkAr6Od+VqutpjZK uNH2QJpyqnxNQzl/FgEKLng+OPshMc3WXogmijXcIA3g/gObLe2YkPI43ywmUr1kFXPk FywprulMk+GeZuf9DRJy2SUcqf8xtarXKTn1auQiNan+Cq0twTJtlaBLd+ImR4S7GAVd zigw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763351361; x=1763956161; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=tB6Rux0HrLKcgyll4hZbEBMFFfGOp9PtogHaw5ekHkk=; b=EmOhgIj78lJfwHkbmnC16pMyli230lZXndsZk3xg494OByU8mEvRXzBl8FBoKBP8Kv CVkaDPl4Xb2hvdK5VHDWWsvJAvSKic7xrktcI6+JyNEE3gL7pUammNn9Mxgb+EwWhS2w S9E++VTtwJwZWnXav1LktaVywbbUzsRevRd3gCoyR+RcvbEd3XYq3zP/E2jffwMHFfyW NFw8uOGfQoN/IvQ+IPOzegTAGOeQM+UaTUrPrq3j6l4/33z+63guVsYTrxmozMJF58A6 qtU6+Sy5RnaC3LdURVwUEwRtxXdH4eGoN2f6AY14POzysW6ffObOI2SYB7DpoZ1+/vy4 JzhQ== X-Forwarded-Encrypted: i=1; AJvYcCXy7IAccJQd3rnUMny6cQvmtBnA2SVXrBP5juj1Ruv2dOMKdPdYGXDY6zCPQzHpands7sSjtiR4Qy4Rbb8=@vger.kernel.org X-Gm-Message-State: AOJu0YwFMaFbJSmX0toHX8nsDYFJ006kfK24dUZssQZBUJnPneWYYMtI koEZibRQE6KiAug859A6kXULxWtnTJVFAj9a/QRUJBhiBd0aXJ9+WDuV X-Gm-Gg: ASbGncuLOBnXGmTviOS2yEjKh73oSUJmSGNkl7Q7M7B3C3j14U47y7EDCPZknuXjrt4 pUuH1XKhwxgmk3dMCoyXyOfuBbcV/h5W8yc2+AESV+2WgOmpNnTWCxSdKpM0Xdcn7dye7W+W9lk mlILlfYkbiPOH7gl/8JDYbFlf72r2yNkxSTgr3c+Lq4rdS+oeIqx9tHhCx6MarQxAP0mZ1/kgOA geiHc4A8ouXjqTPITMHFdB2VtUByGgNUbv9FMzew3veFwW06aE1KzD5DoJuRMdxWjp2+KiR2Ig8 hYXEqa1zP+cZc14CV3sWfJIClyW9XPbzOcS/+vKSOq6pZPIr9kP2CQf8LVGoeXrgfxIkf5xSyKh KJggkPR8O8mrTpMRPdxXFXw0s0t20Pd86PHGt/iStrxoR8G3yOYIO0pL5tjSuDYdSa8tHNorTRU 4Q X-Google-Smtp-Source: AGHT+IHd1el4WKJ8buLYRG3q/1pExUSAcCsDzHZs0s3Ph2PwZFUd1QCWFuHC2Xz4Nk7dDgEi1CtzGw== X-Received: by 2002:a17:90a:da8d:b0:341:a9e7:e5f9 with SMTP id 98e67ed59e1d1-343f99c41admr11436935a91.0.1763351360865; Sun, 16 Nov 2025 19:49:20 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-bc37703a0d9sm10348179a12.31.2025.11.16.19.49.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Nov 2025 19:49:20 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v2 1/6] ftrace: introduce FTRACE_OPS_FL_JMP Date: Mon, 17 Nov 2025 11:49:01 +0800 Message-ID: <20251117034906.32036-2-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251117034906.32036-1-dongml2@chinatelecom.cn> References: <20251117034906.32036-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit For now, the "nop" will be replaced with a "call" instruction when a function is hooked by the ftrace. However, sometimes the "call" can break the RSB and introduce extra overhead. Therefore, introduce the flag FTRACE_OPS_FL_JMP, which indicate that the ftrace_ops should be called with a "jmp" instead of "call". For now, it is only used by the direct call case. When a direct ftrace_ops is marked with FTRACE_OPS_FL_JMP, the last bit of the ops->direct_call will be set to 1. Therefore, we can tell if we should use "jmp" for the callback in ftrace_call_replace(). Signed-off-by: Menglong Dong --- include/linux/ftrace.h | 33 +++++++++++++++++++++++++++++++++ kernel/trace/Kconfig | 12 ++++++++++++ kernel/trace/ftrace.c | 9 ++++++++- 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 07f8c309e432..015dd1049bea 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -359,6 +359,7 @@ enum { FTRACE_OPS_FL_DIRECT = BIT(17), FTRACE_OPS_FL_SUBOP = BIT(18), FTRACE_OPS_FL_GRAPH = BIT(19), + FTRACE_OPS_FL_JMP = BIT(20), }; #ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS @@ -577,6 +578,38 @@ static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, unsigned long addr) { } #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return addr & 1; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr | 1UL; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr & ~1UL; +} +#else +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return false; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr; +} +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_JMP */ + #ifdef CONFIG_STACK_TRACER int stack_trace_sysctl(const struct ctl_table *table, int write, void *buffer, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index d2c79da81e4f..4661b9e606e0 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -80,6 +80,12 @@ config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE If the architecture generates __patchable_function_entries sections but does not want them included in the ftrace locations. +config HAVE_DYNAMIC_FTRACE_WITH_JMP + bool + help + If the architecture supports to replace the __fentry__ with a + "jmp" instruction. + config HAVE_SYSCALL_TRACEPOINTS bool help @@ -330,6 +336,12 @@ config DYNAMIC_FTRACE_WITH_ARGS depends on DYNAMIC_FTRACE depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS +config DYNAMIC_FTRACE_WITH_JMP + def_bool y + depends on DYNAMIC_FTRACE + depends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS + depends on HAVE_DYNAMIC_FTRACE_WITH_JMP + config FPROBE bool "Kernel Function Probe (fprobe)" depends on HAVE_FUNCTION_GRAPH_FREGS && HAVE_FTRACE_GRAPH_FUNC diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 59cfacb8a5bb..a6c060a4f50b 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5951,7 +5951,8 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long for (i = 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { del = __ftrace_lookup_ip(direct_functions, entry->ip); - if (del && del->direct == addr) { + if (del && ftrace_jmp_get(del->direct) == + ftrace_jmp_get(addr)) { remove_hash_entry(direct_functions, del); kfree(del); } @@ -6018,6 +6019,9 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) mutex_lock(&direct_mutex); + if (ops->flags & FTRACE_OPS_FL_JMP) + addr = ftrace_jmp_set(addr); + /* Make sure requested entries are not already registered.. */ size = 1 << hash->size_bits; for (i = 0; i < size; i++) { @@ -6138,6 +6142,9 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) lockdep_assert_held_once(&direct_mutex); + if (ops->flags & FTRACE_OPS_FL_JMP) + addr = ftrace_jmp_set(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash = ops->func_hash; -- 2.51.2