From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754343AbdKAErz (ORCPT ); Wed, 1 Nov 2017 00:47:55 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:45820 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751185AbdKAErx (ORCPT ); Wed, 1 Nov 2017 00:47:53 -0400 Subject: Re: [PATCH 1/2] bpf: add a bpf_override_function helper To: Josef Bacik , , , , , References: <1509464757-2222-1-git-send-email-josef@toxicpanda.com> <1509464757-2222-2-git-send-email-josef@toxicpanda.com> CC: Josef Bacik , Daniel Borkmann From: Alexei Starovoitov Message-ID: <0de2a952-8c41-c79b-f9cb-11b1dccf938a@fb.com> Date: Tue, 31 Oct 2017 21:47:21 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <1509464757-2222-2-git-send-email-josef@toxicpanda.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [2620:10d:c090:180::1:57b1] X-ClientProxiedBy: DM5PR19CA0025.namprd19.prod.outlook.com (2603:10b6:3:9a::11) To SN2PR15MB0976.namprd15.prod.outlook.com (2603:10b6:804:20::26) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e53b7dcf-4e13-4a09-5dd6-08d520e3a6a9 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(4534020)(4602075)(2017052603199);SRVR:SN2PR15MB0976; X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB0976;3:JqM8lBYFWtOuw2QIeyA+jt2N/PJFApD25JcHaRfKtmjNl+4+gHCLWrJtWunTQeSoHlZhNJ1KrXIveCtVb29ovIAoUrl3yBFSVKec1zZKRwRKb1srKzfsgtaD70fcNx2+yL1rn4GBHiETdve4EojM+2+Rc+50uhgsEkHWT96Xd+S66VMTXpEly336/0mJxBMoQotWRSyUBvP653rcukdELPi5byUHrGhUwm7/F1JqjfTunKnkApXzd7Wa1SUKemxa;25:Adbo55OwXMvA2MK/gftbIwA3blbpP908GjT2h0kDN8r7Rb3rcwpci1UWgkFE8tFlRLAOUhSpVDWDO92rD+tLlntUOjP6yyIGc+SfCGoKtP6da3Izuo+H/ov7obdoC3DGz4QgWzExPgPUua03HqTDWstu8GDnKpXnyfOXNZUS1mFWL2TbR2JKUKaNEzTfRX0RV6LlcxEwQXAD7cJ43kHTxaxkI3ok64UFzyOLt9OgJlaNbrmAVmHvH3RIUuDGqWf5YZg0v2s1TK8NdxqZvhG/NfzDRyKVE4wvAoAWLJ27cXWiLRqJ+2aSjBlZGmkMg69mbXidX+ZYD0Lw6COIsSPLBg==;31:gmPxp3jhkXWrFRR5mLc71/7e2j3Iu6I22A5Vs7kBGtOTwMOmQZZTGu4yFxh/yURsb/u4nEq12vWsL+eLpGFkmBImyrgRz4IszbechxRf9qU1jp7lJanj92BmdPChWVbKV8XiNmrWSG8IoZKBk+ao+TdSLOfMe8zLiWSH/b8sWsBjjXkTdIPfZ07kzQWvHF2sxlyIOrf0aulakdiVKLmq8VfiEjk3pq8+An7OE9o3cms= X-MS-TrafficTypeDiagnostic: SN2PR15MB0976: X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB0976;20:/8gOizV+uCR+oBIrPWvIE6WJneLO5qNzuZI7fzhYMvG/5e9Hab+BOraIyp/h01jmVb2Z89Ce5SlcpaPzY/flNiGAoCW8hW5nmdycsSCI+QWfD1KoaxyRLIqbY6qUKJvqigMv26yfd06FiWoriZBCrR7HnslGS2VuixPI/2sdZUG1gMH8eaP0OptC9R2JP+GwsV38x7rn6ZBZNsNhxuTDeoCc/RALreTELx93HGeJmUiz33+HVRofbV57G8OgTd8IXG5FuLdDr8pWVaQJaZQg0Pze8LR/Z9J718kMb/EltxPJGd8nH5shQvzRRGmFaD+XOjpG8YT/2Po4cUVjAAc13PZS1gZ5UIidwIZCbPWLaNB6ekQ3uh0MZT/TeMZ+GKT/KPnriAniDNZOOwdwaE7lM/8WGhwHIPV9P/0TxTnYDHz/AadMCr5mtELFApXY/WUR4MT00JkvrrQhqKdTFwylXkNyzvVRjovu9QrNiTFNAUQkDsIegWrPVca5XOimf0Wv;4:hpxiZm/jtkAva/DMPq5A+tyuFlSclt17OVElQEFQdAL+0XORggaiBj2eZgZ9G4UX7qqGxREz9M8eMIFCU5PQrYb1D5zIYOz/Mme/4PFyPyie2OYm//ntgrAJnKZ9zvqK4/WE/uOd3cs3vXGDu2hky67nq3dYqQ2cmk2OipCPV+L7bYZ9nItZYQ7v9h7ArkdQCO6p0uqZYr4d3n++EVxYoghe9ot/V0sf+HBzpdkJO8myuMdYSY3c+l+Kia8Q80K5hCj92Cbei3d+fw87zw+7wmytxIglN5vtKbqUqfaJZtQrjxiib21c7cQagmZ4mBlW X-Exchange-Antispam-Report-Test: UriScan:(67672495146484); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(100000703101)(100105400095)(3231020)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123555025)(20161123560025)(20161123564025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:SN2PR15MB0976;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:SN2PR15MB0976; X-Forefront-PRVS: 0478C23FE0 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(6009001)(376002)(346002)(189002)(199003)(24454002)(5660300001)(65826007)(6246003)(97736004)(53936002)(101416001)(2906002)(105586002)(4326008)(6486002)(25786009)(50986999)(76176999)(54356999)(229853002)(106356001)(53546010)(64126003)(81166006)(81156014)(23746002)(50466002)(31686004)(6116002)(67846002)(8676002)(68736007)(189998001)(1706002)(230700001)(8936002)(2950100002)(305945005)(6666003)(316002)(65806001)(6636002)(47776003)(33646002)(65956001)(86362001)(575784001)(83506002)(31696002)(478600001)(36756003)(7736002)(58126008)(54906003)(5890100001)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:SN2PR15MB0976;H:[IPv6:2620:10d:c081:1131::112d];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;SN2PR15MB0976;23:WaF7c6TZ5jVMrofrQwVl0eKO7YEDsrp1KwhdD?= =?Windows-1252?Q?Qg/qESyy6ZLsRPJ4C1todfX5s4vsBfGOcn1Yibs5/P/gigAOgf7kcZ0O?= =?Windows-1252?Q?2znXOG1UCN1M6gkC2MbXeMqHfQledhqK3fOAeFGKa6Sri4Y4r7axx1h8?= =?Windows-1252?Q?R0yHibYUu0IOCoQ6VoD1nDwzeefD+k00hiKWKkfaF/fjArW5w+fwRx2W?= =?Windows-1252?Q?fcExSm8HEqO9hNNCaEZziuUEmYyLvnnbQe/EECZ2XOa1C5Y2Or9S+DDM?= =?Windows-1252?Q?4NCxHh/ObPUN4H+DDDJTNtbdxG5rbCG5RcfX+D8o1y1zlWIvTAgIkeI/?= =?Windows-1252?Q?LdOunMUpQPs5XF2NdSTG9QydPR5P+m7LRaVZsQEStVmedzhi4ZVMCMsb?= =?Windows-1252?Q?4EUOVhYoitcMBEkXCUv02N/vjtHzihyaceDZTUyq6umTrzwcv5Y38f43?= =?Windows-1252?Q?m+FE7wwR0ty1e9AqXVN3TPUp3AJwCDq6FymgFdq8xPxCk394BPDU6cVl?= =?Windows-1252?Q?K0AYLlsxhsgnYRzTt/w3tnwEau/xBKNG6Y86ekf8lAbDNYNCCPRVFE4k?= =?Windows-1252?Q?Z5xQ/UNCyP90MeMARaV6rD4ZkYMsbWqb9inr2oa6o/3BZ67wRnJRapS1?= =?Windows-1252?Q?vKpVrAD7bcKvTUYhl8tnqSts2ExcIkFKipgzlnHPHdRZn6ijDEDRhjl7?= =?Windows-1252?Q?HOwlMoANthsUixyVP4v8rZWoIZ1ofAFMTPZDL5Y/RDK+LXAbxGzNx/U4?= =?Windows-1252?Q?LySsKaN3A9/UHiyZO5g+pPIgf29ATca4MwU3rwAwxRN1qckSPh5BQdtJ?= =?Windows-1252?Q?Q91vMMoxW8KTsvIrGBi8M1BY+KGyU80bslwUnJCMBje0id9DIlPbYUBa?= =?Windows-1252?Q?VY9hpSeMAyC42bXrgqDjSCHw90JE3QeYKWSMsVAl4QM+WMtoRKVUMuD6?= =?Windows-1252?Q?/S/8UtmHkGbUrnjiGrxwqC/dI8+Q2GSB7pCwCOOArfHGIbn44QlrYNdN?= =?Windows-1252?Q?tPVImMVJNT4KQuyejSjN3RAhJqeFW7MzX+PY3eHbQHprK0Qyamob7SgY?= =?Windows-1252?Q?FeqBx8W5K7YtfrgJyOqRHwMYpKcr7CkW+ktvqJIAXzcDIdDOuebBXM4k?= =?Windows-1252?Q?ZS+PpKbCZw00F4VWNfHq1w4Orl95iexfyIZ471AviorsygU31kk/FGkc?= =?Windows-1252?Q?bQjc7oxLq/y61FtworER+PjJqmVmgGfMCjkzsTGQd9R+0UQfxlYE7AVP?= =?Windows-1252?Q?eKoxS2AyKDpMAGqHJNvP+Z7ARG9KvU57zj/a4uJNoYnnOpzjzvs9PkjP?= =?Windows-1252?Q?cR6lALLxrOubuxwIDsl/TIokekLx27kcNoxAmq81IkRvRGTRp1mvymaU?= =?Windows-1252?Q?LAO5q1750Rv/IZynQoF03ygXCrBnObl6ZehDJxaEniLpKf5itWLzDlQV?= =?Windows-1252?Q?KxHwHqIC3pIrW1jTmps?= X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB0976;6:oHiISEpM3NoWDnVSisoF9duuFbQlOm+g+0ReLoMlq7jyo0KJdAsNkieglc9CVXeq7JuyatY+dEBKqIsRr2nX1xaWcqMcrSzjW0xQGYRDyAOt1McYLZWXWbgXkWLv3KK9A3nJakOWXsWStZiaW9pQa4He0/5PMg3U+pvj4gdG8nEYmtd2n906eOshpt/VNyWdWc2OVBblY63OJbDl0Cn1xm3BC+DMjOTaYjCqSe7NK0cK/zpq383ugP6uWm2LqccArpcw+5/JM32uBe89Re6D71u1GW1lN1irhk2iLcD1zFML7vb5BIyiiWfeznvZjBhMjdSyLRR1xqH6sUV5MNxi8cCMj+7yDh9/HBV9XS1EFxk=;5:PMXKtY8R6Sls3R7KRiAWvMiLUuyQjemhBEQEoh7u2ph3cJTfJCuW2vIW2pHCyMjEryw9vnwJ9Szoga5nqy1yxIdgCu8h3jitvEEDly2bxFyCTYOF4QOYncp1c1hFHMJ3e2musFEVimXCEFWWzEKCl5NFtt+GoPBC+TuaMUZxJ7I=;24:67k9Ya0tpPlzheIiUU55v58lbrauiuEZkUx/7BDL7rS9eW6TRyE9m/civ/lQcf95KNx6RePuUA2TtQhP/W8xltlsorL7+Wybtr721ndjQ1s=;7:KPvNFWZTPL0jIUNF3vMjbCNGeRgCwZhIlY3s2GeoBp8TF9Fl9ec7aJfZAdPwDCOqD2o1MJbINCVDt6KUn7DbGBxw9kWTRq66hSYDZKU10egERCj26jmQ7OPqB4ZPzABLlBHOw8iaMr0n/4soSIat3kUgk0ebxFpiRowQGjhMi+DDO4sEx3FDv5piR6/K7TJjVy/vO3xJf/KI+Yr76wv6GiVKCnCij/u+H+rAb4PWKhTo7oLeh0DSkO60Yi/JxJMF SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB0976;20:Qpa8z0b+R40CQj2pWXi1dBL8S8Hafi7Im6XAcZ/+49FoFPueQe1CEDm8nfJEdjswBbKpSSeqayr845aM2VpA3F35bnJYcj/cC1ZehD2F/FgBXcmCVHY9ySKSgVxGlcEbslyVFerzmryv9jARmPguVVP+dGWzejRny8Ffq1FErLA= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2017 04:47:25.7012 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e53b7dcf-4e13-4a09-5dd6-08d520e3a6a9 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR15MB0976 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-11-01_01:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/31/17 8:45 AM, Josef Bacik wrote: > From: Josef Bacik > > Error injection is sloppy and very ad-hoc. BPF could fill this niche > perfectly with it's kprobe functionality. We could make sure errors are > only triggered in specific call chains that we care about with very > specific situations. Accomplish this with the bpf_override_funciton > helper. This will modify the probe'd callers return value to the > specified value and set the PC to an override function that simply > returns, bypassing the originally probed function. This gives us a nice > clean way to implement systematic error injection for all of our code > paths. > > Signed-off-by: Josef Bacik > --- > arch/Kconfig | 3 +++ > arch/x86/Kconfig | 1 + > arch/x86/include/asm/kprobes.h | 4 ++++ > arch/x86/include/asm/ptrace.h | 5 +++++ > arch/x86/kernel/kprobes/ftrace.c | 14 ++++++++++++++ > include/linux/trace_events.h | 7 +++++++ > include/uapi/linux/bpf.h | 7 ++++++- > kernel/trace/Kconfig | 11 +++++++++++ > kernel/trace/bpf_trace.c | 30 ++++++++++++++++++++++++++++ > kernel/trace/trace_kprobe.c | 42 +++++++++++++++++++++++++++++++++------- > 10 files changed, 116 insertions(+), 8 deletions(-) > > diff --git a/arch/Kconfig b/arch/Kconfig > index d789a89cb32c..4fb618082259 100644 > --- a/arch/Kconfig > +++ b/arch/Kconfig > @@ -195,6 +195,9 @@ config HAVE_OPTPROBES > config HAVE_KPROBES_ON_FTRACE > bool > > +config HAVE_KPROBE_OVERRIDE > + bool > + > config HAVE_NMI > bool > > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig > index 971feac13506..5126d2750dd0 100644 > --- a/arch/x86/Kconfig > +++ b/arch/x86/Kconfig > @@ -152,6 +152,7 @@ config X86 > select HAVE_KERNEL_XZ > select HAVE_KPROBES > select HAVE_KPROBES_ON_FTRACE > + select HAVE_KPROBE_OVERRIDE > select HAVE_KRETPROBES > select HAVE_KVM > select HAVE_LIVEPATCH if X86_64 > diff --git a/arch/x86/include/asm/kprobes.h b/arch/x86/include/asm/kprobes.h > index 6cf65437b5e5..c6c3b1f4306a 100644 > --- a/arch/x86/include/asm/kprobes.h > +++ b/arch/x86/include/asm/kprobes.h > @@ -67,6 +67,10 @@ extern const int kretprobe_blacklist_size; > void arch_remove_kprobe(struct kprobe *p); > asmlinkage void kretprobe_trampoline(void); > > +#ifdef CONFIG_KPROBES_ON_FTRACE > +extern void arch_ftrace_kprobe_override_function(struct pt_regs *regs); > +#endif > + > /* Architecture specific copy of original instruction*/ > struct arch_specific_insn { > /* copy of the original instruction */ > diff --git a/arch/x86/include/asm/ptrace.h b/arch/x86/include/asm/ptrace.h > index 91c04c8e67fa..f04e71800c2f 100644 > --- a/arch/x86/include/asm/ptrace.h > +++ b/arch/x86/include/asm/ptrace.h > @@ -108,6 +108,11 @@ static inline unsigned long regs_return_value(struct pt_regs *regs) > return regs->ax; > } > > +static inline void regs_set_return_value(struct pt_regs *regs, unsigned long rc) > +{ > + regs->ax = rc; > +} > + > /* > * user_mode(regs) determines whether a register set came from user > * mode. On x86_32, this is true if V8086 mode was enabled OR if the > diff --git a/arch/x86/kernel/kprobes/ftrace.c b/arch/x86/kernel/kprobes/ftrace.c > index 041f7b6dfa0f..3c455bf490cb 100644 > --- a/arch/x86/kernel/kprobes/ftrace.c > +++ b/arch/x86/kernel/kprobes/ftrace.c > @@ -97,3 +97,17 @@ int arch_prepare_kprobe_ftrace(struct kprobe *p) > p->ainsn.boostable = false; > return 0; > } > + > +asmlinkage void override_func(void); > +asm( > + ".type override_func, @function\n" > + "override_func:\n" > + " ret\n" > + ".size override_func, .-override_func\n" > +); > + > +void arch_ftrace_kprobe_override_function(struct pt_regs *regs) > +{ > + regs->ip = (unsigned long)&override_func; > +} > +NOKPROBE_SYMBOL(arch_ftrace_kprobe_override_function); > diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h > index fc6aeca945db..9179f109c49b 100644 > --- a/include/linux/trace_events.h > +++ b/include/linux/trace_events.h > @@ -521,7 +521,14 @@ do { \ > #ifdef CONFIG_PERF_EVENTS > struct perf_event; > > +enum { > + BPF_STATE_NORMAL_KPROBE = 0, > + BPF_STATE_FTRACE_KPROBE, > + BPF_STATE_MODIFIED_PC, > +}; > + > DECLARE_PER_CPU(struct pt_regs, perf_trace_regs); > +DECLARE_PER_CPU(int, bpf_kprobe_state); > > extern int perf_trace_init(struct perf_event *event); > extern void perf_trace_destroy(struct perf_event *event); > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index 0b7b54d898bd..1ad5b87a42f6 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -673,6 +673,10 @@ union bpf_attr { > * @buf: buf to fill > * @buf_size: size of the buf > * Return : 0 on success or negative error code > + * > + * int bpf_override_return(pt_regs, rc) > + * @pt_regs: pointer to struct pt_regs > + * @rc: the return value to set > */ > #define __BPF_FUNC_MAPPER(FN) \ > FN(unspec), \ > @@ -732,7 +736,8 @@ union bpf_attr { > FN(xdp_adjust_meta), \ > FN(perf_event_read_value), \ > FN(perf_prog_read_value), \ > - FN(getsockopt), > + FN(getsockopt), \ > + FN(override_return), > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper > * function eBPF program intends to call > diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig > index 434c840e2d82..9dc0deeaad2b 100644 > --- a/kernel/trace/Kconfig > +++ b/kernel/trace/Kconfig > @@ -518,6 +518,17 @@ config FUNCTION_PROFILER > > If in doubt, say N. > > +config BPF_KPROBE_OVERRIDE > + bool "Enable BPF programs to override a kprobed function" > + depends on BPF_EVENTS > + depends on KPROBES_ON_FTRACE > + depends on HAVE_KPROBE_OVERRIDE > + depends on DYNAMIC_FTRACE_WITH_REGS > + default n > + help > + Allows BPF to override the execution of a probed function and > + set a different return value. This is used for error injection. > + > config FTRACE_MCOUNT_RECORD > def_bool y > depends on DYNAMIC_FTRACE > diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c > index 136aa6bb0422..e9a7ae9c3a78 100644 > --- a/kernel/trace/bpf_trace.c > +++ b/kernel/trace/bpf_trace.c > @@ -13,6 +13,8 @@ > #include > #include > #include > +#include > + > #include "trace.h" > > u64 bpf_get_stackid(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); > @@ -76,6 +78,32 @@ unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) > } > EXPORT_SYMBOL_GPL(trace_call_bpf); > > +#ifdef CONFIG_BPF_KPROBE_OVERRIDE > +BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc) > +{ > + /* We can only override ftrace kprobes at the moment. */ > + if (__this_cpu_read(bpf_kprobe_state) != BPF_STATE_FTRACE_KPROBE) > + return -EINVAL; > + __this_cpu_write(bpf_kprobe_state, BPF_STATE_MODIFIED_PC); > + regs_set_return_value(regs, rc); > + arch_ftrace_kprobe_override_function(regs); > + return 0; > +} > +#else > +BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc) > +{ > + return -EINVAL; > +} > +#endif > + > +static const struct bpf_func_proto bpf_override_return_proto = { > + .func = bpf_override_return, > + .gpl_only = true, > + .ret_type = RET_INTEGER, > + .arg1_type = ARG_PTR_TO_CTX, > + .arg2_type = ARG_ANYTHING, > +}; > + > BPF_CALL_3(bpf_probe_read, void *, dst, u32, size, const void *, unsafe_ptr) > { > int ret; > @@ -551,6 +579,8 @@ static const struct bpf_func_proto *kprobe_prog_func_proto(enum bpf_func_id func > return &bpf_get_stackid_proto; > case BPF_FUNC_perf_event_read_value: > return &bpf_perf_event_read_value_proto; > + case BPF_FUNC_override_return: > + return &bpf_override_return_proto; > default: > return tracing_func_proto(func_id); > } > diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c > index abf92e478cfb..b7173e44e2e7 100644 > --- a/kernel/trace/trace_kprobe.c > +++ b/kernel/trace/trace_kprobe.c > @@ -42,6 +42,7 @@ struct trace_kprobe { > (offsetof(struct trace_kprobe, tp.args) + \ > (sizeof(struct probe_arg) * (n))) > > +DEFINE_PER_CPU(int, bpf_kprobe_state); > > static nokprobe_inline bool trace_kprobe_is_return(struct trace_kprobe *tk) > { > @@ -1170,7 +1171,7 @@ static int kretprobe_event_define_fields(struct trace_event_call *event_call) > #ifdef CONFIG_PERF_EVENTS > > /* Kprobe profile handler */ > -static void > +static int > kprobe_perf_func(struct trace_kprobe *tk, struct pt_regs *regs) > { > struct trace_event_call *call = &tk->tp.call; > @@ -1179,12 +1180,37 @@ kprobe_perf_func(struct trace_kprobe *tk, struct pt_regs *regs) > int size, __size, dsize; > int rctx; > > - if (bpf_prog_array_valid(call) && !trace_call_bpf(call, regs)) > - return; > + if (bpf_prog_array_valid(call)) { > + int ret, state; > + > + /* > + * If we are an ftrace kprobe then we are allowed to do > + * overrides for this kprobe. > + */ > + if (unlikely(kprobe_ftrace(&tk->rp.kp))) > + __this_cpu_write(bpf_kprobe_state, > + BPF_STATE_FTRACE_KPROBE); > + ret = trace_call_bpf(call, regs); > + > + /* > + * We need to check and see if we modified the pc of the > + * pt_regs, and if so clear the kprobe and return 1 so that we > + * don't do the instruction skipping. Also reset our state so > + * we are clean the next pass through. > + */ > + state = __this_cpu_read(bpf_kprobe_state); > + __this_cpu_write(bpf_kprobe_state, BPF_STATE_NORMAL_KPROBE); > + if (state == BPF_STATE_MODIFIED_PC) { > + reset_current_kprobe(); > + return 1; > + } have been thinking whether it's possible to reduce runtime penalty in all bpf-kprobes for this feature... The first kprobe_ftrace() check can be done at attach time. Like a bit in bpf_prog can indicate whether it attempts to use bpf_override_return() (similar to cb_access) and in perf_event_attach_bpf_prog() do if (event->tp_event->flags & TRACE_EVENT_FL_KPROBE) kprobe_ftrace(&((struct trace_kprobe *) (event->tp_event->data))->rp.kp) may be there are helpers to make it pretty. Then after trace_call_bpf() we don't have to do __this_cpu_read+write Only read will be enough. The rest looks great. Looks like amazing feature. > + if (!ret) > + return 0; > + } > > head = this_cpu_ptr(call->perf_events); > if (hlist_empty(head)) > - return; > + return 0; > > dsize = __get_data_size(&tk->tp, regs); > __size = sizeof(*entry) + tk->tp.size + dsize; > @@ -1193,13 +1219,14 @@ kprobe_perf_func(struct trace_kprobe *tk, struct pt_regs *regs) > > entry = perf_trace_buf_alloc(size, NULL, &rctx); > if (!entry) > - return; > + return 0; > > entry->ip = (unsigned long)tk->rp.kp.addr; > memset(&entry[1], 0, dsize); > store_trace_args(sizeof(*entry), &tk->tp, regs, (u8 *)&entry[1], dsize); > perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, > head, NULL, NULL); > + return 0; > } > NOKPROBE_SYMBOL(kprobe_perf_func); > > @@ -1275,6 +1302,7 @@ static int kprobe_register(struct trace_event_call *event, > static int kprobe_dispatcher(struct kprobe *kp, struct pt_regs *regs) > { > struct trace_kprobe *tk = container_of(kp, struct trace_kprobe, rp.kp); > + int ret = 0; > > raw_cpu_inc(*tk->nhit); > > @@ -1282,9 +1310,9 @@ static int kprobe_dispatcher(struct kprobe *kp, struct pt_regs *regs) > kprobe_trace_func(tk, regs); > #ifdef CONFIG_PERF_EVENTS > if (tk->tp.flags & TP_FLAG_PROFILE) > - kprobe_perf_func(tk, regs); > + ret = kprobe_perf_func(tk, regs); > #endif > - return 0; /* We don't tweek kernel, so just return 0 */ > + return ret; > } > NOKPROBE_SYMBOL(kprobe_dispatcher); > >