* [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs [not found] <d85ece6e-27f9-4d9f-83af-4101bc76fef6@app.fastmail.com> @ 2026-11-06 13:38 ` roblabla 2026-11-06 15:41 ` Paul Moore 0 siblings, 1 reply; 3+ messages in thread From: roblabla @ 2026-11-06 13:38 UTC (permalink / raw) To: Daan De Meyer, Paul Moore Cc: bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko Resubmitting as plain text. Sorry for the noise. Hello, First of all, I'm sorry if I'm sending this email to the wrong mailing lists, it's my first time reporting a linux issue ^^'. In commit 16959c469f232f5c76cfb628297cafbf6dbdba79 (lsm: expose mount idmaps to inode hooks), merged in 7.4-rc1, a new argument was added in position 0 of various inode-related `security_` functions. Specifically, it added this argument to inode_create, inode_link, inode_symlink, inode_mkdir, inode_mknod and inode_permission. This has the immediate effect of breaking eBPF LSM programs relying on these hooks that were compiled with BPF CO-RE and using BPF_CORE_READ. Those programs don't simply stop loading (which would be fine enough) - instead, they will misinterpret the argument from one type to another, which can cause them to behave erratically. This has already bitten me in the past, when inode_setattr got a similar argument change in Linux 6.9. This caused our eBPF LSM to misbehave and deny actions that should have been ordinarily accepted, leading to huge production outages on our customers' machines. Here's a simple reproducer that shows the problem (full repro with the loader, makefile and run.sh is at https://github.com/roblabla/kernel-7.4-bpf-bug-reproducer): ``` #include "vmlinux.h" #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> char LICENSE[] SEC("license") = "GPL"; #define EPERM 1 #define NAME_LEN 32 static const char forbidden[] = "forbidden"; SEC("lsm/inode_create") int BPF_PROG(deny_forbidden, struct inode *dir, struct dentry *dentry) { char name[NAME_LEN] = {}; const unsigned char *name_ptr; unsigned long dir_ino; dir_ino = BPF_CORE_READ(dir, i_ino); name_ptr = BPF_CORE_READ(dentry, d_name.name); bpf_probe_read_kernel_str(name, sizeof(name), name_ptr); bpf_printk("inode_create: dir ino=%lu name='%s'", dir_ino, name); for (int i = 0; i < sizeof(forbidden); i++) if (name[i] != forbidden[i]) return 0; bpf_printk("inode_create: denied '%s'", name); return -EPERM; } ``` Compile it under a kernel before 7.4 release, then run it on mainline. Here's the output: - run on 7.1-rc7: touch forbidden: DENIED (dir ino=59 name='forbidden') - run on 7.4-rc1: touch forbidden: created (dir ino=0 name='') Before Linux 7.4, this program will properly deny the creation of any file named "forbidden". But in 7.4, the name_ptr BPF_CORE_READ will read garbage, and the check will fail. This is a bit of a contrived example, of course, but should show that there's a real problem in the way we handle eBPF loading in the face of changing signatures, that is prone to causing big compatibility issues. Now, I'm not sure how the Linux Kernel treats eBPF LSM hooks - if they're considered part of the userspace that shouldn't be broken. My understanding is that kprobes are not part of the Linux Kernel stability guarantees, for rather obvious reasons (they allow watching the internals of the kernel) - but I'm not sure if this extends to the LSM Hooks. I assume not. Still, this kind of breakage is really painful for us, and I hope we can find a solution. Could we fix this type of breakage by making sure new arguments are only added at the _end_ of the LSM hooks signatures, such that eBPF written for an older version of the kernel do not start misinterpreting arguments? eBPFs don't care about having "too few" arguments in the hook, so this would solve the situation. Separately, I think the current lack of error handling in case of argument mismatch should really be improved. Could we perhaps have a mechanism such that the kernel (or maybe libbpf?) could check the arguments of the eBPF program, and refuse loading if it mismatches what the kernel expects? Thanks, roblabla ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-06 13:38 ` [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs roblabla @ 2026-11-06 15:41 ` Paul Moore 2026-11-06 19:33 ` Andrii Nakryiko 0 siblings, 1 reply; 3+ messages in thread From: Paul Moore @ 2026-11-06 15:41 UTC (permalink / raw) To: roblabla Cc: Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko On Fri, Nov 6, 2026 at 8:39 AM roblabla <linux@roblab.la> wrote: > > Hello, > First of all, I'm sorry if I'm sending this email to the wrong mailing lists, it's my first time reporting a linux issue ^^'. Hi, It looks like you sent this email to the right people/lists, well done on the first hurdle :) > In commit 16959c469f232f5c76cfb628297cafbf6dbdba79 (lsm: expose mount idmaps to inode hooks), merged in 7.4-rc1, a new argument was added in position 0 of various inode-related `security_` functions. Specifically, it added this argument to inode_create, inode_link, inode_symlink, inode_mkdir, inode_mknod and inode_permission. > > This has the immediate effect of breaking eBPF LSM programs relying on these hooks that were compiled with BPF CO-RE and using BPF_CORE_READ. ... > Now, I'm not sure how the Linux Kernel treats eBPF LSM hooks - if they're considered part of the userspace that shouldn't be broken. My understanding is that kprobes are not part of the Linux Kernel stability guarantees, for rather obvious reasons (they allow watching the internals of the kernel) - but I'm not sure if this extends to the LSM Hooks. I assume not. Still, this kind of breakage is really painful for us, and I hope we can find a solution. Like kprobes, the LSM hooks are intentionally not part of any stable API promise. While we try not to needlessly churn the LSM hooks, we reserve the right to modify parameters, the LSM callbacks associated with a hook, and even the basic hook semantics from one kernel release to another. The reasoning is very similar to that of kprobes: the LSM hooks are linked to kernel internals and thus need the ability to change as the kernel changes. LSMs that need to provide a stable API must provide an abstraction layer between the LSM hook callbacks and their stable API. The LSM framework is not responsible for the stability of individual LSM APIs. > Could we fix this type of breakage by making sure new arguments are only added at the _end_ of the LSM hooks signatures, I'm sorry, but I can't make that promise at the LSM framework layer. It is impossible to predict how the kernel internals will continue to evolve and thus I can't promise how the LSM hooks and callbacks will need to change as a result. Regarding the specifics of how the BPF LSM has dealt with this in the past, and any plans for the future, I'll let others speak to that; hopefully they will have some tricks you can use to make these transitions easier. -- paul-moore.com ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-06 15:41 ` Paul Moore @ 2026-11-06 19:33 ` Andrii Nakryiko 0 siblings, 0 replies; 3+ messages in thread From: Andrii Nakryiko @ 2026-11-06 19:33 UTC (permalink / raw) To: Paul Moore, Daan De Meyer Cc: roblabla, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko On Fri, Nov 6, 2026 at 7:41 AM Paul Moore <paul@paul-moore.com> wrote: > > On Fri, Nov 6, 2026 at 8:39 AM roblabla <linux@roblab.la> wrote: > > > > Hello, > > First of all, I'm sorry if I'm sending this email to the wrong mailing lists, it's my first time reporting a linux issue ^^'. > > Hi, > > It looks like you sent this email to the right people/lists, well done > on the first hurdle :) > > > In commit 16959c469f232f5c76cfb628297cafbf6dbdba79 (lsm: expose mount idmaps to inode hooks), merged in 7.4-rc1, a new argument was added in position 0 of various inode-related `security_` functions. Specifically, it added this argument to inode_create, inode_link, inode_symlink, inode_mkdir, inode_mknod and inode_permission. > > > > This has the immediate effect of breaking eBPF LSM programs relying on these hooks that were compiled with BPF CO-RE and using BPF_CORE_READ. > > ... > > > Now, I'm not sure how the Linux Kernel treats eBPF LSM hooks - if they're considered part of the userspace that shouldn't be broken. My understanding is that kprobes are not part of the Linux Kernel stability guarantees, for rather obvious reasons (they allow watching the internals of the kernel) - but I'm not sure if this extends to the LSM Hooks. I assume not. Still, this kind of breakage is really painful for us, and I hope we can find a solution. > > Like kprobes, the LSM hooks are intentionally not part of any stable > API promise. While we try not to needlessly churn the LSM hooks, we > reserve the right to modify parameters, the LSM callbacks associated > with a hook, and even the basic hook semantics from one kernel release > to another. The reasoning is very similar to that of kprobes: the LSM > hooks are linked to kernel internals and thus need the ability to > change as the kernel changes. > > LSMs that need to provide a stable API must provide an abstraction > layer between the LSM hook callbacks and their stable API. The LSM > framework is not responsible for the stability of individual LSM APIs. > > > Could we fix this type of breakage by making sure new arguments are only added at the _end_ of the LSM hooks signatures, > > I'm sorry, but I can't make that promise at the LSM framework layer. > It is impossible to predict how the kernel internals will continue to > evolve and thus I can't promise how the LSM hooks and callbacks will > need to change as a result. Yeah, I think that's a reasonable stance and we went through similar discussion about tracepoints not being stable APIs, no questions about that. But I'm wondering if it's not too late (as far as I understand, this hasn't gotten into final release just yet), would it be ok to just move newly added argument to be the last ones? We had similar problem in not too distant past with one of scheduler tracepoints, where new argument was added in the middle of argument list, and that causes just unnecessary friction when using BPF with old and new kernels. We did move the argument to the end and resolved this (without really mandating any stability guarantees). So, in short, would it be possible to send a follow up to Daan's commit and move `struct mnt_idmap *idmap` to the end of LSM argument list? > > Regarding the specifics of how the BPF LSM has dealt with this in the > past, and any plans for the future, I'll let others speak to that; > hopefully they will have some tricks you can use to make these > transitions easier. > (this is answer to roblabla, I don't think you, Paul, should care) The answer here is nuanced. On one hand, libbpf and kernel (BPF verifier) don't really know exactly what BPF program assumes argument types are. libbpf-provided BPF_PROG() macro that everyone is using with BPF LSM just assumes user specifies correct sequence and types of arguments, and libbpf just provides a bit of syntactic sugar with casting untyped input arguments (they are just "u64" types) into expected types. So it's not easy to detect this automatically neither in libbpf nor in BPF verifier. Having said that, there is a way to handle this, although ugly and *everyone* using specific LSMs would need to do this on their own. Instead of using convenience BPF_PROG() macro to define BPF program, you just use raw `u64 ctx[]` access, and then cast ctx[0] to either `struct mnt_idmap *` on new kernel, or `struct dentry *` on old kernel. And you do that for all arguments. E.g., something like this SEC("...") int my_prog(u64 ctx[]) { bool new_kernel = ...; struct mnt_idmap *idmap = new_kernel ? bpf_core_cast(ctx[0], struct mnt_idmap) : NULL; struct dentry *old_dentry = bpf_core_cast(new_kernel ? ctx[1] : ctx[0]); ... and so on ... } How you determine "new_kernel" can be done in various ways as well, I don't want to overburden this email with details, but it's doable as well. so, tl;dr, it's possible to handle this LSM signature change, but it's painful and requires knowing about breakage and doing a bunch of extra plumbing. ideally, if it's just a new argument addition, it should go to the end of arg list and then all these issues go away > -- > paul-moore.com ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-06 19:33 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <d85ece6e-27f9-4d9f-83af-4101bc76fef6@app.fastmail.com>
2026-11-06 13:38 ` [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs roblabla
2026-11-06 15:41 ` Paul Moore
2026-11-06 19:33 ` Andrii Nakryiko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®