[REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Ameer Hamza]

Hi,

Since commit 97d34aa65c29 ("mm/secretmem: properly account locked
pages"), in v7.3-rc2 and the 6.18.52 and 7.2.5 backports, touching a
memfd_secret() page for the first time fails with SIGBUS while the
same user is running perf record, and read() into such a page fails
with EFAULT. This happens for root as well as for an unprivileged
user running its own perf record. It reproduces on v7.3-rc4 and
6.18.52, and the same test passes without the recording or with the
commit reverted.

The commit charges secretmem pages to the per-user user->locked_vm
and checks that counter against RLIMIT_MEMLOCK, with no CAP_IPC_LOCK
exemption because the fd can be passed to other processes. perf has
charged its ring buffers to the same counter since 2009, up to
perf_event_mlock_kb per online CPU, and only what exceeds that
allowance is checked against RLIMIT_MEMLOCK. sysctl/kernel.rst
describes the allowance as not counted against the mlock limit, yet
it sits in the counter that secretmem now enforces. perf record sizes
its buffers to the allowance by default, 516 KiB per CPU, so on 16 or
more CPUs one recording uses up the default 8 MiB limit on its own,
and on fewer CPUs it leaves correspondingly less for secretmem.

Reproducer, as root on a 16 vCPU x86_64 guest with the default
ulimit -l 8192, while "perf record -o /tmp/p.data -- sleep 300" runs
in another shell:

  fd = syscall(SYS_memfd_secret, 0);
  ftruncate(fd, 4096);
  p = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  read(open("/dev/zero", O_RDONLY), p, 4096);   /* EFAULT */
  p[0] = 1;                                     /* SIGBUS */

The commit closes a real hole, unbounded pinning of unevictable memory
by unprivileged users. The question is whether the perf_event_mlock_kb
allowance is meant to count against the RLIMIT_MEMLOCK budget that
secretmem and the other subsystems accounting to user->locked_vm
enforce, or whether perf should keep it in a counter of its own, which
would leave the hole closed and secretmem usable during a recording.

#regzbot introduced: 97d34aa65c29

Thanks,
Ameer Hamza

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Sasha Levin]

> Since commit 97d34aa65c29 ("mm/secretmem: properly account locked
> pages"), in v7.3-rc2 and the 6.18.52 and 7.2.5 backports, touching a
> memfd_secret() page for the first time fails with SIGBUS while the
> same user is running perf record, and read() into such a page fails

Thanks for the report. I've dropped 97d34aa65c29 from the 6.12 queue,
and also removed the copies that were staged for 6.6 and 6.1.

7.2.y and 6.18.y already shipped it, so those need an upstream fix,
which stable will pick up once it lands.

-- 
Thanks,
Sasha

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Lorenzo Stoakes (ARM)]

On Fri, Oct 02, 2026 at 09:30:23PM -0400, Sasha Levin wrote:
> > Since commit 97d34aa65c29 ("mm/secretmem: properly account locked
> > pages"), in v7.3-rc2 and the 6.18.52 and 7.2.5 backports, touching a
> > memfd_secret() page for the first time fails with SIGBUS while the
> > same user is running perf record, and read() into such a page fails
>
> Thanks for the report. I've dropped 97d34aa65c29 from the 6.12 queue,
> and also removed the copies that were staged for 6.6 and 6.1.
>
> 7.2.y and 6.18.y already shipped it, so those need an upstream fix,
> which stable will pick up once it lands.

Sasha,

Could we perhaps take a breath and pause?

This patch fixes a really quite serious vulnerability that allows an
unprivileged process to allocate arbitrary, unreclaimable (i.e. OOMK cannot
save you) memory.

Allow me to assess the validity of this report _before_ dropping things
from queues perhaps? :)

It was sent late on Friday in a week that I was ill and it's Saturday
pre-LPC, so that's hardly a timeframe that I could reasonably have replied.

Thanks!

--
Cheers, Lorenzo

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Sasha Levin]

On Sat, Oct 03, 2026 at 10:18:34AM +0100, Lorenzo Stoakes (ARM) wrote:
>On Fri, Oct 02, 2026 at 09:30:23PM -0400, Sasha Levin wrote:
>> > Since commit 97d34aa65c29 ("mm/secretmem: properly account locked
>> > pages"), in v7.3-rc2 and the 6.18.52 and 7.2.5 backports, touching a
>> > memfd_secret() page for the first time fails with SIGBUS while the
>> > same user is running perf record, and read() into such a page fails
>>
>> Thanks for the report. I've dropped 97d34aa65c29 from the 6.12 queue,
>> and also removed the copies that were staged for 6.6 and 6.1.
>>
>> 7.2.y and 6.18.y already shipped it, so those need an upstream fix,
>> which stable will pick up once it lands.
>
>Sasha,
>
>Could we perhaps take a breath and pause?

I pulled this out quickly because I didn't want this issue to go into the kernels that were released this morning.

In general, our policy around this type of reports is that if the issue is in a
released LTS kernel, we will wait for a fix upstream, and if the issue is still
in our queues, we will simply drop the patch from the queue.

>This patch fixes a really quite serious vulnerability that allows an
>unprivileged process to allocate arbitrary, unreclaimable (i.e. OOMK cannot
>save you) memory.
>
>Allow me to assess the validity of this report _before_ dropping things
>from queues perhaps? :)

No disagreement that this would have been better - in this case it was timing
as the releases were about to happen the following day, so the safest choice
was to drop it while we figure it out.

>It was sent late on Friday in a week that I was ill and it's Saturday
>pre-LPC, so that's hardly a timeframe that I could reasonably have replied.

Keep in mind that our release cycles are fairly short, so this shouldn't be too
big of a deal?

We will generally go for the safe option of dropping a commit that introduces a
regression as it's fairly easy to queue it back up very soon after a
diagnosis/fix shows up.

-- 
Thanks,
Sasha

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Lorenzo Stoakes (ARM)]

On Sat, Oct 03, 2026 at 10:30:10AM -0400, Sasha Levin wrote:
> On Sat, Oct 03, 2026 at 10:18:34AM +0100, Lorenzo Stoakes (ARM) wrote:

> > It was sent late on Friday in a week that I was ill and it's Saturday
> > pre-LPC, so that's hardly a timeframe that I could reasonably have replied.
>
> Keep in mind that our release cycles are fairly short, so this shouldn't be too
> big of a deal?

I mean the same argument could be made both ways :)

>
> We will generally go for the safe option of dropping a commit that introduces a
> regression as it's fairly easy to queue it back up very soon after a
> diagnosis/fix shows up.

I think we'll have to agree to disagree about what was safe in this situation!

Anyway, the commit is fine as-is, and fine to be queued.

I am writing a patch for what seems to be the underlying issue which appears to
be related to how perf accounts things.

I guess I'm going to have to pack for LPC really late today :)

--
Cheers, Lorenzo

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Sasha Levin]

On Sat, Oct 03, 2026 at 05:23:53PM +0100, Lorenzo Stoakes (ARM) wrote:
>On Sat, Oct 03, 2026 at 10:30:10AM -0400, Sasha Levin wrote:
>> On Sat, Oct 03, 2026 at 10:18:34AM +0100, Lorenzo Stoakes (ARM) wrote:
>
>> > It was sent late on Friday in a week that I was ill and it's Saturday
>> > pre-LPC, so that's hardly a timeframe that I could reasonably have replied.
>>
>> Keep in mind that our release cycles are fairly short, so this shouldn't be too
>> big of a deal?
>
>I mean the same argument could be made both ways :)
>
>>
>> We will generally go for the safe option of dropping a commit that introduces a
>> regression as it's fairly easy to queue it back up very soon after a
>> diagnosis/fix shows up.
>
>I think we'll have to agree to disagree about what was safe in this situation!

Happy to chat about it at LPC. Our reasoning is that users are better off with
a regression they are aware of, instead of getting surprised every time they
upgrade.

I can see the other side of this, and I think we can do some work to accomodate
this type of preferences from subsystems/authors.

-- 
Thanks,
Sasha

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[David Hildenbrand (Arm)]

On 10/2/26 19:56, Ameer Hamza wrote:
> Hi,
> 

Hi!

> Since commit 97d34aa65c29 ("mm/secretmem: properly account locked
> pages"), in v7.3-rc2 and the 6.18.52 and 7.2.5 backports, touching a
> memfd_secret() page for the first time fails with SIGBUS while the
> same user is running perf record, and read() into such a page fails
> with EFAULT. This happens for root as well as for an unprivileged
> user running its own perf record. It reproduces on v7.3-rc4 and
> 6.18.52, and the same test passes without the recording or with the
> commit reverted.

Thanks for the report!

May I know whether this problem was found by

(a) A real workload

(b) A test case

(c) Complains by some tooling

I am asking, because so far I was under the assumption that secretmem isn't
heavily used in the wild.

-- 
Cheers,

David

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Ameer Hamza]

On Sat, Oct 03, 2026 at 04:14:49PM +0200, David Hildenbrand (Arm) wrote:
> May I know whether this problem was found by
> 
> (a) A real workload
> 
> (b) A test case
> 
> (c) Complains by some tooling

(a). A root userspace process that keeps secrets in memfd_secret()
failed while perf record ran on the same machine. I reduced it to the
snippet in the report.

Thanks,
Ameer Hamza

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Lorenzo Stoakes (ARM)]

On Fri, Oct 02, 2026 at 10:56:51PM +0500, Ameer Hamza wrote:
> Hi,
>
> Since commit 97d34aa65c29 ("mm/secretmem: properly account locked
> pages"), in v7.3-rc2 and the 6.18.52 and 7.2.5 backports, touching a
> memfd_secret() page for the first time fails with SIGBUS while the
> same user is running perf record, and read() into such a page fails
> with EFAULT. This happens for root as well as for an unprivileged
> user running its own perf record. It reproduces on v7.3-rc4 and
> 6.18.52, and the same test passes without the recording or with the
> commit reverted.
>
> The commit charges secretmem pages to the per-user user->locked_vm
> and checks that counter against RLIMIT_MEMLOCK, with no CAP_IPC_LOCK
> exemption because the fd can be passed to other processes. perf has
> charged its ring buffers to the same counter since 2009, up to
> perf_event_mlock_kb per online CPU, and only what exceeds that
> allowance is checked against RLIMIT_MEMLOCK. sysctl/kernel.rst
> describes the allowance as not counted against the mlock limit, yet
> it sits in the counter that secretmem now enforces. perf record sizes
> its buffers to the allowance by default, 516 KiB per CPU, so on 16 or
> more CPUs one recording uses up the default 8 MiB limit on its own,
> and on fewer CPUs it leaves correspondingly less for secretmem.
>
> Reproducer, as root on a 16 vCPU x86_64 guest with the default
> ulimit -l 8192, while "perf record -o /tmp/p.data -- sleep 300" runs
> in another shell:
>
>   fd = syscall(SYS_memfd_secret, 0);
>   ftruncate(fd, 4096);
>   p = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
>   read(open("/dev/zero", O_RDONLY), p, 4096);   /* EFAULT */
>   p[0] = 1;                                     /* SIGBUS */
>
> The commit closes a real hole, unbounded pinning of unevictable memory
> by unprivileged users. The question is whether the perf_event_mlock_kb
> allowance is meant to count against the RLIMIT_MEMLOCK budget that
> secretmem and the other subsystems accounting to user->locked_vm
> enforce, or whether perf should keep it in a counter of its own, which
> would leave the hole closed and secretmem usable during a recording.
>
> #regzbot introduced: 97d34aa65c29
>

Hi,

Thanks for the report, however this is not a regression in secretmem.

The field that is being used for tracking the allowance (user->locked_vm)
is used by everything-but-perf to track against RLIMIT_MEMLOCK, but perf is
using it for something else (pinned_vm).

So this is a bug in perf, which should use its own counter for this,
AFAICT.

Actually it seems perf invented the field (user->locked_vm) to calculate
memory that actually isn't mlock()'d, but is kernel-allocated and pinned so
treated 'as if' it were.

It tracks a per-process (rather than per-mm) RLIMIT_MEMLOCK allowance plus
a 'gift' of extra available pages which exceeds this.

Every other user since is checking against RLIMIT_MEMLOCK (and you'd hit
the same issues with them too) - io_uring, MSG_ZEROCOPY, AF_XDP, iommufd,
s390 KVM zCPI and now secretmem.

So this is a long-standing bug it seems.

Other things:

The SIGBUS is unfortunately necessary - since the region is populated on
fault, it is only then that it can be determined whether the limit is
violated.

Not being unlimited on CAP_IPC_LOCK is also, unfortunately, necessary to
resolve the security hole - often privileged processes hand out secretmem
to other unprivileged processes.

If those processes then didn't apply the limit check, the mitigation can't
work.

The commit message goes into detail about all this.

The way to address this right now, as I see you've applied yourselves as
far as I can see (in [0]), is to change the limits to account for this.

I will work on a patch for perf that fixes this specific issue.

Sasha - you can re-queue the fix. I will send out another fix for perf.

--
Cheers, Lorenzo

[0]:https://github.com/truenas/truenas_ros/pull/128

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Lorenzo Stoakes (ARM)]

Fix for perf sent at
https://lore.kernel.org/linux-perf-users/20261003-perf-locked-vm-fix-v1-1-d214dcaf0d76@kernel.org/

I cc-d everybody on the thread also.

--
Cheers, Lorenzo

Re: [REGRESSION] secretmem: SIGBUS on memfd_secret() pages while perf record runs

[Ameer Hamza]

On Sat, Oct 03, 2026 at 04:56:32PM +0100, Lorenzo Stoakes (ARM) wrote:
> So this is a bug in perf, which should use its own counter for this,
> AFAICT.

Agreed, and the report's closing question pointed the same way, since
the commit itself closes a real hole. Glad the longstanding perf bug is
getting fixed as a result. Thanks for the analysis and for turning the
fix around so quickly.

Thanks,
Ameer Hamza