From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E17F4C33CA2 for ; Thu, 9 Jan 2020 14:12:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AD4AC2067D for ; Thu, 9 Jan 2020 14:12:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731548AbgAIOM6 (ORCPT ); Thu, 9 Jan 2020 09:12:58 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:8682 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1731527AbgAIOM6 (ORCPT ); Thu, 9 Jan 2020 09:12:58 -0500 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 1697D9ABB84B413FC16B; Thu, 9 Jan 2020 22:12:49 +0800 (CST) Received: from huawei.com (10.175.104.193) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.439.0; Thu, 9 Jan 2020 22:12:41 +0800 From: Cheng Jian To: , CC: , , , , , , Subject: [RFC PATCH] arm64/ftrace: support dynamically allocated trampolines Date: Thu, 9 Jan 2020 14:27:36 +0000 Message-ID: <20200109142736.1122-1-cj.chengjian@huawei.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.175.104.193] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When we tracing multiple functions, it has to use a list function and cause all the other functions being traced to check the hash of the ftrace_ops. But this is very inefficient. we can call a dynamically allocated trampoline which calls the callback directly to solve this problem. This patch introduce dynamically alloced trampolines for ARM64. If a callback is registered to a function and there's no other callback registered to that function, the ftrace_ops will get its own trampoline allocated for it that will call the function directly. We merge two functions (ftrace_caller/ftrace_regs_caller and ftrace_common) into one function, so we no longer need a jump to ftrace_common and fix it to NOP. similar to X86_64, save the local ftrace_ops at the end. the ftrace trampoline layout : low ftrace_(regs_)caller => +---------------+ | ftrace_caller | ftrace_common => +---------------+ | ftrace_common | function_trace_op_ptr => | ... | ldr x2, | | b ftrace_stub | | | nop | fgraph call | +---------------+ +------------>| ftrace_ops | +---------------+ | PLT entrys | (TODO) +---------------+ high Known issues : If kaslr is enabled, the address of tramp and ftrace call may be far away. Therefore, long jump support is required. Here I intend to use the same solution as module relocating, Reserve enough space for PLT at the end when allocating, can use PLT to complete these long jumps. Signed-off-by: Cheng Jian --- arch/arm64/kernel/entry-ftrace.S | 4 + arch/arm64/kernel/ftrace.c | 310 +++++++++++++++++++++++++++++++ 2 files changed, 314 insertions(+) diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index 7d02f9966d34..f5ee797804ac 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -77,17 +77,20 @@ ENTRY(ftrace_regs_caller) ftrace_regs_entry 1 +GLOBAL(ftrace_regs_caller_end) b ftrace_common ENDPROC(ftrace_regs_caller) ENTRY(ftrace_caller) ftrace_regs_entry 0 +GLOBAL(ftrace_caller_end) b ftrace_common ENDPROC(ftrace_caller) ENTRY(ftrace_common) sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) mov x1, x9 // parent_ip (callsite's LR) +GLOBAL(function_trace_op_ptr) ldr_l x2, function_trace_op // op mov x3, sp // regs @@ -121,6 +124,7 @@ ftrace_common_return: /* Restore the callsite's SP */ add sp, sp, #S_FRAME_SIZE + 16 +GLOBAL(ftrace_common_end) ret x9 ENDPROC(ftrace_common) diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 8618faa82e6d..95ea68ef6228 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -10,11 +10,13 @@ #include #include #include +#include #include #include #include #include +#include #ifdef CONFIG_DYNAMIC_FTRACE /* @@ -47,6 +49,314 @@ static int ftrace_modify_code(unsigned long pc, u32 old, u32 new, return 0; } +/* ftrace dynamic trampolines */ +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +#ifdef CONFIG_MODULES +#include + +static inline void *alloc_tramp(unsigned long size) +{ + return module_alloc(size); +} + +static inline void tramp_free(void *tramp) +{ + module_memfree(tramp); +} +#else +static inline void *alloc_tramp(unsigned long size) +{ + return NULL; +} + +static inline void tramp_free(void *tramp) {} +#endif + +extern void ftrace_regs_caller_end(void); +extern void ftrace_caller_end(void); +extern void ftrace_common(void); +extern void ftrace_common_end(void); + +extern void function_trace_op_ptr(void); + +extern struct ftrace_ops *function_trace_op; + +/* + * ftrace_caller() or ftrace_regs_caller() trampoline + * +-----------------------+ + * ftrace_(regs_)caller => | ...... | + * ftrace_(regs_)caller_end => | b ftrace_common | => nop + * +-----------------------+ + * ftrace_common => | ...... | + * function_trace_op_ptr => | adrp x2, sym | => nop + * | ldr x2,[x2,:lo12:sym]| => ldr x2 + * | ...... | + * ftrace_common_end => | retq | + * +-----------------------+ + * ftrace_opt => | ftrace_opt | + * +-----------------------+ + */ +static unsigned long create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) +{ + unsigned long start_offset_caller, end_offset_caller, caller_size; + unsigned long start_offset_common, end_offset_common, common_size; + unsigned long op_offset, offset, size, ip, npages; + void *trampoline; + unsigned long *ptr; + /* ldr x2,