From: Jordan Rife <jrife@google.com>
To: jrife@google.com
Cc: acme@kernel.org, alexander.shishkin@linux.intel.com,
andrii.nakryiko@gmail.com, ast@kernel.org, bpf@vger.kernel.org,
joel@joelfernandes.org, linux-kernel@vger.kernel.org,
mark.rutland@arm.com, mathieu.desnoyers@efficios.com,
mhiramat@kernel.org, mingo@redhat.com, mjeanson@efficios.com,
namhyung@kernel.org, paulmck@kernel.org, peterz@infradead.org,
rostedt@goodmis.org, tglx@linutronix.de, yhs@fb.com
Subject: Re: [RFC PATCH v4 4/4] tracing: Add might_fault() check in __DO_TRACE() for syscall
Date: Tue, 29 Oct 2024 00:28:14 +0000 [thread overview]
Message-ID: <20241029002814.505389-1-jrife@google.com> (raw)
In-Reply-To: <CADKFtnT59wzKxob03OOOfvVh67MQkpWvzvfmzv3D-_bGeM=rJA@mail.gmail.com>
> I can test this later today. Considering there needs to be a fix on
> the BPF side to fully resolve the use-after-free issue reported by
> syzbot, I may combine your v4 patch with the bandaid fix which chains
> call_rcu->call_rcu_tasks_trace I made earlier while running the
> reproducer locally.
Testing this way, the series LGTM. Here's what I did starting from
linux-next tag next-20241028.
1. Applied my patch from [1] to prevent any failures resulting from the
as-of-yet unpatched BPF code that uses call_rcu(). This lets us
focus on the effect's of Mathieu's patch series.
2. Ran the reproducer [3] from the original syzbot report [2] on a
kernel build /without/ Mathieu's v4 patch to confirm that we hit
a use-after-free bug resulting from the use of call_rcu() inside
release_probe().
3. Applied the patch series and rebuilt the kernel.
4. Reran the reproducer on the new kernel build to ensure that we don't
hit the same use-after-free bug anymore.
[1]: https://lore.kernel.org/bpf/20241023145640.1499722-1-jrife@google.com/
[2]: https://lore.kernel.org/bpf/67121037.050a0220.10f4f4.000f.GAE@google.com/
[3]: https://syzkaller.appspot.com/x/repro.syz?x=153ef887980000
Without Mathieu's Patch
=======================
A crash occurs after a few minutes.
jordan@t14:~/contexts/use-after-free$ ssh \
-p 10022 \
-o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
-o IdentitiesOnly=yes \
root@127.0.0.1 './syz-execprog -repeat=0 -procs=5 ./repro.syz.txt'
Warning: Permanently added '[127.0.0.1]:10022' (ED25519) to the list of known hosts.
2024/10/28 23:15:39 parsed 1 programs
2024/10/28 23:15:52 executed programs: 0
2024/10/28 23:15:57 executed programs: 34
2024/10/28 23:16:02 executed programs: 90
2024/10/28 23:16:07 executed programs: 121
2024/10/28 23:16:12 executed programs: 152
2024/10/28 23:16:17 executed programs: 165
2024/10/28 23:16:23 executed programs: 177
2024/10/28 23:16:28 executed programs: 209
2024/10/28 23:16:33 executed programs: 228
2024/10/28 23:16:38 executed programs: 251
2024/10/28 23:16:44 executed programs: 273
2024/10/28 23:16:49 executed programs: 316
2024/10/28 23:16:54 executed programs: 338
2024/10/28 23:16:59 executed programs: 352
2024/10/28 23:17:04 executed programs: 376
2024/10/28 23:17:10 executed programs: 404
2024/10/28 23:17:16 executed programs: 419
2024/10/28 23:17:21 executed programs: 433
2024/10/28 23:17:26 executed programs: 460
[ 687.323615][T16276] ==================================================================
[ 687.325235][T16276] BUG: KFENCE: use-after-free read in __traceiter_sys_enter+0x30/0x50
[ 687.325235][T16276]
[ 687.327193][T16276] Use-after-free read at 0xffff88807ec60028 (in kfence-#47):
[ 687.328404][T16276] __traceiter_sys_enter+0x30/0x50
[ 687.329338][T16276] syscall_trace_enter+0x1ea/0x2b0
[ 687.330021][T16276] do_syscall_64+0x1ec/0x250
[ 687.330816][T16276] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 687.331826][T16276]
[ 687.332291][T16276] kfence-#47: 0xffff88807ec60000-0xffff88807ec60057, size=88, cache=kmalloc-96
[ 687.332291][T16276]
[ 687.334265][T16276] allocated by task 16281 on cpu 1 at 683.953385s (3.380878s ago):
[ 687.335615][T16276] tracepoint_add_func+0x28a/0xd90
[ 687.336424][T16276] tracepoint_probe_register_prio_may_exist+0xa2/0xf0
[ 687.337416][T16276] bpf_probe_register+0x186/0x200
[ 687.338174][T16276] bpf_raw_tp_link_attach+0x21f/0x540
[ 687.339233][T16276] __sys_bpf+0x393/0x4fa0
[ 687.340042][T16276] __x64_sys_bpf+0x78/0xc0
[ 687.340801][T16276] do_syscall_64+0xcb/0x250
[ 687.341623][T16276] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 687.342697][T16276]
[ 687.343147][T16276] freed by task 14317 on cpu 1 at 687.273223s (0.069923s ago):
[ 687.344352][T16276] rcu_core+0x7a2/0x14f0
[ 687.344996][T16276] handle_softirqs+0x1d4/0x870
[ 687.345797][T16276] irq_exit_rcu+0xbb/0x120
[ 687.346519][T16276] sysvec_apic_timer_interrupt+0xa8/0xc0
[ 687.347432][T16276] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 687.348574][T16276] check_preemption_disabled+0x22/0x170
[ 687.349904][T16276] rcu_is_watching+0x12/0xc0
[ 687.350849][T16276] lock_release+0x51e/0x6f0
[ 687.351758][T16276] bpf_trace_run2+0x25a/0x580
[ 687.352675][T16276] __bpf_trace_sys_enter+0x6e/0xa0
[ 687.353625][T16276] syscall_trace_enter+0x1ea/0x2b0
[ 687.354570][T16276] do_syscall_64+0x1ec/0x250
[ 687.355447][T16276] entry_SYSCALL_64_after_hwframe+0x77/0x7f
With Mathieu's Patch
====================
No crash after 10m.
jordan@t14:~/contexts/use-after-free$ ssh \
-p 10022 \
-o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
-o IdentitiesOnly=yes root@127.0.0.1 \
'./syz-execprog -repeat=0 -procs=5 ./repro.syz.txt'
Warning: Permanently added '[127.0.0.1]:10022' (ED25519) to the list of known hosts.
2024/10/29 00:09:00 parsed 1 programs
2024/10/29 00:09:07 executed programs: 0
2024/10/29 00:09:12 executed programs: 10
2024/10/29 00:09:17 executed programs: 51
2024/10/29 00:09:23 executed programs: 82
2024/10/29 00:09:28 executed programs: 103
2024/10/29 00:09:33 executed programs: 125
2024/10/29 00:09:39 executed programs: 134
2024/10/29 00:09:44 executed programs: 147
...
2024/10/29 00:18:06 executed programs: 2671
2024/10/29 00:18:11 executed programs: 2725
2024/10/29 00:18:17 executed programs: 2743
2024/10/29 00:18:22 executed programs: 2772
2024/10/29 00:18:30 executed programs: 2784
2024/10/29 00:18:35 executed programs: 2816
2024/10/29 00:18:40 executed programs: 2842
2024/10/29 00:18:46 executed programs: 2881
2024/10/29 00:18:52 executed programs: 2923
2024/10/29 00:18:57 executed programs: 2947
2024/10/29 00:19:03 executed programs: 2991
2024/10/29 00:19:09 executed programs: 3013
2024/10/29 00:19:16 executed programs: 3052
Tested-by: Jordan Rife <jrife@google.com>
next prev parent reply other threads:[~2024-10-29 0:28 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-10-28 19:09 [RFC PATCH v4 0/4] Faultable syscall tracepoints updates Mathieu Desnoyers
2024-10-28 19:09 ` [RFC PATCH v4 1/4] tracing: Introduce tracepoint extended structure Mathieu Desnoyers
2024-10-28 19:09 ` [RFC PATCH v4 2/4] tracing: Introduce tracepoint_is_faultable() Mathieu Desnoyers
2024-10-28 20:19 ` Andrii Nakryiko
2024-10-28 20:24 ` Mathieu Desnoyers
2024-10-28 19:09 ` [RFC PATCH v4 3/4] tracing: Fix syscall tracepoint use-after-free Mathieu Desnoyers
2024-10-28 19:09 ` [RFC PATCH v4 4/4] tracing: Add might_fault() check in __DO_TRACE() for syscall Mathieu Desnoyers
2024-10-28 19:58 ` Mathieu Desnoyers
2024-10-28 20:20 ` Andrii Nakryiko
2024-10-28 20:25 ` Mathieu Desnoyers
2024-10-28 20:40 ` Jordan Rife
2024-10-29 0:28 ` Jordan Rife [this message]
2024-10-30 14:42 ` Mathieu Desnoyers
2024-10-30 14:59 ` Alexei Starovoitov
2024-10-30 17:34 ` Jordan Rife
2024-10-30 17:39 ` Andrii Nakryiko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241029002814.505389-1-jrife@google.com \
--to=jrife@google.com \
--cc=acme@kernel.org \
--cc=alexander.shishkin@linux.intel.com \
--cc=andrii.nakryiko@gmail.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=joel@joelfernandes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mathieu.desnoyers@efficios.com \
--cc=mhiramat@kernel.org \
--cc=mingo@redhat.com \
--cc=mjeanson@efficios.com \
--cc=namhyung@kernel.org \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®