Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

[Heyne, Maximilian]

On Fri, Jul 04, 2025 at 05:48:18AM +0000, Kuniyuki Iwashima wrote:
> Netlink has this pattern in some places
> 
>   if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
>   	atomic_add(skb->truesize, &sk->sk_rmem_alloc);
> 
> , which has the same problem fixed by commit 5a465a0da13e ("udp:
> Fix multiple wraparounds of sk->sk_rmem_alloc.").
> 
> For example, if we set INT_MAX to SO_RCVBUFFORCE, the condition
> is always false as the two operands are of int.
> 
> Then, a single socket can eat as many skb as possible until OOM
> happens, and we can see multiple wraparounds of sk->sk_rmem_alloc.
> 
> Let's fix it by using atomic_add_return() and comparing the two
> variables as unsigned int.
> 
> Before:
>   [root@fedora ~]# ss -f netlink
>   Recv-Q      Send-Q Local Address:Port                Peer Address:Port
>   -1668710080 0               rtnl:nl_wraparound/293               *
> 
> After:
>   [root@fedora ~]# ss -f netlink
>   Recv-Q     Send-Q Local Address:Port                Peer Address:Port
>   2147483072 0               rtnl:nl_wraparound/290               *
>   ^
>   `--- INT_MAX - 576
> 
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Reported-by: Jason Baron <jbaron@akamai.com>
> Closes: https://lore.kernel.org/netdev/cover.1750285100.git.jbaron@akamai.com/
> Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>

Hi Kuniyuki,

We're seeing soft lockups with this patch in a variety of (stable)
kernel versions.

I was able to reproduce it on a couple of different EC2 instances also
with the latest linux kernel 6.16-rc7 using the following steps:

systemctl start auditd
sudo auditctl -D
sudo auditctl -b 512
sudo auditctl -a always,exit -F arch=b64 -S mmap,munmap,mprotect,brk -k memory_ops
sudo auditctl -e 1

Then execute some programs that call some of these memory functions,
such as repeated calls of "sudo auditctl -s" or logging in via SSH.

These settings are set in a way to create a lot audit messages. Once the
backlog (see auditctl -s) overshoots the backlog_limit, the system soft
lockups:

[  460.056244] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kauditd:32]
[  460.056249] Modules linked in: mousedev(E) nls_ascii(E) nls_cp437(E) sunrpc(E) vfat(E) fat(E) psmouse(E) atkbd(E) libps2(E) vivaldi_fmap(E) i8042(E) serio(E) skx_edac_common(E) button(E) ena(E) ghash_clmulni_intel(E) sch_fq_codel(E) drm(E) i2c_core(E) dm_mod(E) drm_panel_orientation_quirks(E) backlight(E) fuse(E) loop(E) dax(E) configfs(E) dmi_sysfs(E) efivarfs(E)
[  460.056272] CPU: 1 UID: 0 PID: 32 Comm: kauditd Tainted: G            EL      6.16.0-rc7+ #3 PREEMPT(none)
[  460.056275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[  460.056276] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
[  460.056277] RIP: 0010:_raw_spin_unlock_irqrestore+0x1b/0x30
[  460.056284] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 16 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d b5 23 b6 01 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 e9 14 23 00 00 0f 1f 40 00 90
[  460.056285] RSP: 0018:ffffb762c0123d70 EFLAGS: 00000246
[  460.056287] RAX: 0000000000000001 RBX: ffff9b14c08eafc0 RCX: ffff9b14c3337348
[  460.056288] RDX: ffff9b14c3337348 RSI: 0000000000000246 RDI: ffff9b14c3337340
[  460.056289] RBP: ffff9b14c3337000 R08: ffffffff93cea880 R09: 0000000000000001
[  460.056290] R10: 0000000000000001 R11: 0000000000000080 R12: ffff9b14c1b72500
[  460.056291] R13: ffffb762c0123de0 R14: ffff9b14c3337340 R15: ffff9b14c3337080
[  460.056294] FS:  0000000000000000(0000) GS:ffff9b1563788000(0000) knlGS:0000000000000000
[  460.056296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  460.056297] CR2: 00007f36fd5d21b4 CR3: 000000010241a002 CR4: 00000000007706f0
[  460.056298] PKRU: 55555554
[  460.056299] Call Trace:
[  460.056300]  <TASK>
[  460.056302]  netlink_attachskb+0xb7/0x2f0
[  460.056308]  ? __pfx_default_wake_function+0x10/0x10
[  460.056313]  netlink_unicast+0xea/0x3b0
[  460.056315]  kauditd_send_queue+0xaf/0x170
[  460.056318]  ? __pfx_kauditd_send_multicast_skb+0x10/0x10
[  460.056320]  ? __pfx_kauditd_retry_skb+0x10/0x10
[  460.056321]  kauditd_thread+0x132/0x2b0
[  460.056323]  ? __pfx_autoremove_wake_function+0x10/0x10
[  460.056327]  ? __pfx_kauditd_thread+0x10/0x10
[  460.056328]  kthread+0xfb/0x230
[  460.056331]  ? __pfx_kthread+0x10/0x10
[  460.056332]  ? __pfx_kthread+0x10/0x10
[  460.056334]  ret_from_fork+0x142/0x160
[  460.056338]  ? __pfx_kthread+0x10/0x10
[  460.056339]  ret_from_fork_asm+0x1a/0x30
[  460.056343]  </TASK>
[  469.011800] audit_log_start: 120 callbacks suppressed
[  469.011805] audit: audit_backlog=513 > audit_backlog_limit=512
[  469.013154] audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=512
[  469.013967] audit: backlog limit exceeded
[  469.014617] audit: audit_backlog=513 > audit_backlog_limit=512
[  469.015313] audit: audit_lost=2 audit_rate_limit=0 audit_backlog_limit=512
[  469.016112] audit: backlog limit exceeded

We've bug reports from many users, so the issue is rather wide-spread.

So far I don't know why the commit is causing this issue and will keep
investigating. However, when reverted (together with its 2 follow-up
patches), the issue goes away and host do not lock up.

Thanks,
Maximilian



Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

[Kuniyuki Iwashima]

On Fri, Aug 8, 2025 at 6:59 AM Heyne, Maximilian <mheyne@amazon.de> wrote:
>
> On Fri, Jul 04, 2025 at 05:48:18AM +0000, Kuniyuki Iwashima wrote:
> > Netlink has this pattern in some places
> >
> >   if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
> >       atomic_add(skb->truesize, &sk->sk_rmem_alloc);
> >
> > , which has the same problem fixed by commit 5a465a0da13e ("udp:
> > Fix multiple wraparounds of sk->sk_rmem_alloc.").
> >
> > For example, if we set INT_MAX to SO_RCVBUFFORCE, the condition
> > is always false as the two operands are of int.
> >
> > Then, a single socket can eat as many skb as possible until OOM
> > happens, and we can see multiple wraparounds of sk->sk_rmem_alloc.
> >
> > Let's fix it by using atomic_add_return() and comparing the two
> > variables as unsigned int.
> >
> > Before:
> >   [root@fedora ~]# ss -f netlink
> >   Recv-Q      Send-Q Local Address:Port                Peer Address:Port
> >   -1668710080 0               rtnl:nl_wraparound/293               *
> >
> > After:
> >   [root@fedora ~]# ss -f netlink
> >   Recv-Q     Send-Q Local Address:Port                Peer Address:Port
> >   2147483072 0               rtnl:nl_wraparound/290               *
> >   ^
> >   `--- INT_MAX - 576
> >
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Reported-by: Jason Baron <jbaron@akamai.com>
> > Closes: https://lore.kernel.org/netdev/cover.1750285100.git.jbaron@akamai.com/
> > Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
>
> Hi Kuniyuki,
>
> We're seeing soft lockups with this patch in a variety of (stable)
> kernel versions.
>
> I was able to reproduce it on a couple of different EC2 instances also
> with the latest linux kernel 6.16-rc7 using the following steps:
>
> systemctl start auditd
> sudo auditctl -D
> sudo auditctl -b 512
> sudo auditctl -a always,exit -F arch=b64 -S mmap,munmap,mprotect,brk -k memory_ops
> sudo auditctl -e 1
>
> Then execute some programs that call some of these memory functions,
> such as repeated calls of "sudo auditctl -s" or logging in via SSH.
>
> These settings are set in a way to create a lot audit messages. Once the
> backlog (see auditctl -s) overshoots the backlog_limit, the system soft
> lockups:
>
> [  460.056244] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kauditd:32]
> [  460.056249] Modules linked in: mousedev(E) nls_ascii(E) nls_cp437(E) sunrpc(E) vfat(E) fat(E) psmouse(E) atkbd(E) libps2(E) vivaldi_fmap(E) i8042(E) serio(E) skx_edac_common(E) button(E) ena(E) ghash_clmulni_intel(E) sch_fq_codel(E) drm(E) i2c_core(E) dm_mod(E) drm_panel_orientation_quirks(E) backlight(E) fuse(E) loop(E) dax(E) configfs(E) dmi_sysfs(E) efivarfs(E)
> [  460.056272] CPU: 1 UID: 0 PID: 32 Comm: kauditd Tainted: G            EL      6.16.0-rc7+ #3 PREEMPT(none)
> [  460.056275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
> [  460.056276] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
> [  460.056277] RIP: 0010:_raw_spin_unlock_irqrestore+0x1b/0x30
> [  460.056284] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 16 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d b5 23 b6 01 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 e9 14 23 00 00 0f 1f 40 00 90
> [  460.056285] RSP: 0018:ffffb762c0123d70 EFLAGS: 00000246
> [  460.056287] RAX: 0000000000000001 RBX: ffff9b14c08eafc0 RCX: ffff9b14c3337348
> [  460.056288] RDX: ffff9b14c3337348 RSI: 0000000000000246 RDI: ffff9b14c3337340
> [  460.056289] RBP: ffff9b14c3337000 R08: ffffffff93cea880 R09: 0000000000000001
> [  460.056290] R10: 0000000000000001 R11: 0000000000000080 R12: ffff9b14c1b72500
> [  460.056291] R13: ffffb762c0123de0 R14: ffff9b14c3337340 R15: ffff9b14c3337080
> [  460.056294] FS:  0000000000000000(0000) GS:ffff9b1563788000(0000) knlGS:0000000000000000
> [  460.056296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  460.056297] CR2: 00007f36fd5d21b4 CR3: 000000010241a002 CR4: 00000000007706f0
> [  460.056298] PKRU: 55555554
> [  460.056299] Call Trace:
> [  460.056300]  <TASK>
> [  460.056302]  netlink_attachskb+0xb7/0x2f0
> [  460.056308]  ? __pfx_default_wake_function+0x10/0x10
> [  460.056313]  netlink_unicast+0xea/0x3b0
...
>
> We've bug reports from many users, so the issue is rather wide-spread.
>
> So far I don't know why the commit is causing this issue and will keep
> investigating. However, when reverted (together with its 2 follow-up
> patches), the issue goes away and host do not lock up.

Thanks for the report, Max!

Does your tree have this commit ?  This is the 3rd follow-up patch.

commit 759dfc7d04bab1b0b86113f1164dc1fec192b859
Author: Fedor Pchelkin <pchelkin@ispras.ru>
Date:   Mon Jul 28 08:06:47 2025

    netlink: avoid infinite retry looping in netlink_unicast()

Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

[Paul Moore]

On Fri, Aug 8, 2025 at 11:54 AM Kuniyuki Iwashima <kuniyu@google.com> wrote:
> On Fri, Aug 8, 2025 at 6:59 AM Heyne, Maximilian <mheyne@amazon.de> wrote:
> > On Fri, Jul 04, 2025 at 05:48:18AM +0000, Kuniyuki Iwashima wrote:
> > > Netlink has this pattern in some places
> > >
> > >   if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
> > >       atomic_add(skb->truesize, &sk->sk_rmem_alloc);
> > >
> > > , which has the same problem fixed by commit 5a465a0da13e ("udp:
> > > Fix multiple wraparounds of sk->sk_rmem_alloc.").
> > >
> > > For example, if we set INT_MAX to SO_RCVBUFFORCE, the condition
> > > is always false as the two operands are of int.
> > >
> > > Then, a single socket can eat as many skb as possible until OOM
> > > happens, and we can see multiple wraparounds of sk->sk_rmem_alloc.
> > >
> > > Let's fix it by using atomic_add_return() and comparing the two
> > > variables as unsigned int.
> > >
> > > Before:
> > >   [root@fedora ~]# ss -f netlink
> > >   Recv-Q      Send-Q Local Address:Port                Peer Address:Port
> > >   -1668710080 0               rtnl:nl_wraparound/293               *
> > >
> > > After:
> > >   [root@fedora ~]# ss -f netlink
> > >   Recv-Q     Send-Q Local Address:Port                Peer Address:Port
> > >   2147483072 0               rtnl:nl_wraparound/290               *
> > >   ^
> > >   `--- INT_MAX - 576
> > >
> > > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > > Reported-by: Jason Baron <jbaron@akamai.com>
> > > Closes: https://lore.kernel.org/netdev/cover.1750285100.git.jbaron@akamai.com/
> > > Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
> >
> > Hi Kuniyuki,
> >
> > We're seeing soft lockups with this patch in a variety of (stable)
> > kernel versions.
> >
> > I was able to reproduce it on a couple of different EC2 instances also
> > with the latest linux kernel 6.16-rc7 using the following steps:
> >
> > systemctl start auditd
> > sudo auditctl -D
> > sudo auditctl -b 512
> > sudo auditctl -a always,exit -F arch=b64 -S mmap,munmap,mprotect,brk -k memory_ops
> > sudo auditctl -e 1
> >
> > Then execute some programs that call some of these memory functions,
> > such as repeated calls of "sudo auditctl -s" or logging in via SSH.
> >
> > These settings are set in a way to create a lot audit messages. Once the
> > backlog (see auditctl -s) overshoots the backlog_limit, the system soft
> > lockups:
> >
> > [  460.056244] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kauditd:32]
> > [  460.056249] Modules linked in: mousedev(E) nls_ascii(E) nls_cp437(E) sunrpc(E) vfat(E) fat(E) psmouse(E) atkbd(E) libps2(E) vivaldi_fmap(E) i8042(E) serio(E) skx_edac_common(E) button(E) ena(E) ghash_clmulni_intel(E) sch_fq_codel(E) drm(E) i2c_core(E) dm_mod(E) drm_panel_orientation_quirks(E) backlight(E) fuse(E) loop(E) dax(E) configfs(E) dmi_sysfs(E) efivarfs(E)
> > [  460.056272] CPU: 1 UID: 0 PID: 32 Comm: kauditd Tainted: G            EL      6.16.0-rc7+ #3 PREEMPT(none)
> > [  460.056275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
> > [  460.056276] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
> > [  460.056277] RIP: 0010:_raw_spin_unlock_irqrestore+0x1b/0x30
> > [  460.056284] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 16 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d b5 23 b6 01 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 e9 14 23 00 00 0f 1f 40 00 90
> > [  460.056285] RSP: 0018:ffffb762c0123d70 EFLAGS: 00000246
> > [  460.056287] RAX: 0000000000000001 RBX: ffff9b14c08eafc0 RCX: ffff9b14c3337348
> > [  460.056288] RDX: ffff9b14c3337348 RSI: 0000000000000246 RDI: ffff9b14c3337340
> > [  460.056289] RBP: ffff9b14c3337000 R08: ffffffff93cea880 R09: 0000000000000001
> > [  460.056290] R10: 0000000000000001 R11: 0000000000000080 R12: ffff9b14c1b72500
> > [  460.056291] R13: ffffb762c0123de0 R14: ffff9b14c3337340 R15: ffff9b14c3337080
> > [  460.056294] FS:  0000000000000000(0000) GS:ffff9b1563788000(0000) knlGS:0000000000000000
> > [  460.056296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  460.056297] CR2: 00007f36fd5d21b4 CR3: 000000010241a002 CR4: 00000000007706f0
> > [  460.056298] PKRU: 55555554
> > [  460.056299] Call Trace:
> > [  460.056300]  <TASK>
> > [  460.056302]  netlink_attachskb+0xb7/0x2f0
> > [  460.056308]  ? __pfx_default_wake_function+0x10/0x10
> > [  460.056313]  netlink_unicast+0xea/0x3b0
> ...
> >
> > We've bug reports from many users, so the issue is rather wide-spread.
> >
> > So far I don't know why the commit is causing this issue and will keep
> > investigating. However, when reverted (together with its 2 follow-up
> > patches), the issue goes away and host do not lock up.
>
> Thanks for the report, Max!
>
> Does your tree have this commit ?  This is the 3rd follow-up patch.
>
> commit 759dfc7d04bab1b0b86113f1164dc1fec192b859
> Author: Fedor Pchelkin <pchelkin@ispras.ru>
> Date:   Mon Jul 28 08:06:47 2025
>
>     netlink: avoid infinite retry looping in netlink_unicast()

Hopefully that resolves the problem, Maximilian?

Normally the audit subsystem is reasonably robust when faced with
significant audit loads.  An example I use for testing is to enable
logging for *every* syscall (from the command line, don't make this
persist via the config file!) and then shutdown the system; the system
will obviously slow quite a bit under the absurd load, but it should
shutdown gracefully without any lockups.

-- 
paul-moore.com

Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

[Heyne, Maximilian]

On Wed, Aug 13, 2025 at 03:00:29PM -0400, Paul Moore wrote:
> On Fri, Aug 8, 2025 at 11:54???AM Kuniyuki Iwashima <kuniyu@google.com> wrote:
> > On Fri, Aug 8, 2025 at 6:59???AM Heyne, Maximilian <mheyne@amazon.de> wrote:
> > > On Fri, Jul 04, 2025 at 05:48:18AM +0000, Kuniyuki Iwashima wrote:
> > > > Netlink has this pattern in some places
> > > >
> > > >   if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
> > > >       atomic_add(skb->truesize, &sk->sk_rmem_alloc);
> > > >
> > > > , which has the same problem fixed by commit 5a465a0da13e ("udp:
> > > > Fix multiple wraparounds of sk->sk_rmem_alloc.").
> > > >
> > > > For example, if we set INT_MAX to SO_RCVBUFFORCE, the condition
> > > > is always false as the two operands are of int.
> > > >
> > > > Then, a single socket can eat as many skb as possible until OOM
> > > > happens, and we can see multiple wraparounds of sk->sk_rmem_alloc.
> > > >
> > > > Let's fix it by using atomic_add_return() and comparing the two
> > > > variables as unsigned int.
> > > >
> > > > Before:
> > > >   [root@fedora ~]# ss -f netlink
> > > >   Recv-Q      Send-Q Local Address:Port                Peer Address:Port
> > > >   -1668710080 0               rtnl:nl_wraparound/293               *
> > > >
> > > > After:
> > > >   [root@fedora ~]# ss -f netlink
> > > >   Recv-Q     Send-Q Local Address:Port                Peer Address:Port
> > > >   2147483072 0               rtnl:nl_wraparound/290               *
> > > >   ^
> > > >   `--- INT_MAX - 576
> > > >
> > > > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > > > Reported-by: Jason Baron <jbaron@akamai.com>
> > > > Closes: https://lore.kernel.org/netdev/cover.1750285100.git.jbaron@akamai.com/
> > > > Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
> > >
> > > Hi Kuniyuki,
> > >
> > > We're seeing soft lockups with this patch in a variety of (stable)
> > > kernel versions.
> > >
> > > I was able to reproduce it on a couple of different EC2 instances also
> > > with the latest linux kernel 6.16-rc7 using the following steps:
> > >
> > > systemctl start auditd
> > > sudo auditctl -D
> > > sudo auditctl -b 512
> > > sudo auditctl -a always,exit -F arch=b64 -S mmap,munmap,mprotect,brk -k memory_ops
> > > sudo auditctl -e 1
> > >
> > > Then execute some programs that call some of these memory functions,
> > > such as repeated calls of "sudo auditctl -s" or logging in via SSH.
> > >
> > > These settings are set in a way to create a lot audit messages. Once the
> > > backlog (see auditctl -s) overshoots the backlog_limit, the system soft
> > > lockups:
> > >
> > > [  460.056244] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kauditd:32]
> > > [  460.056249] Modules linked in: mousedev(E) nls_ascii(E) nls_cp437(E) sunrpc(E) vfat(E) fat(E) psmouse(E) atkbd(E) libps2(E) vivaldi_fmap(E) i8042(E) serio(E) skx_edac_common(E) button(E) ena(E) ghash_clmulni_intel(E) sch_fq_codel(E) drm(E) i2c_core(E) dm_mod(E) drm_panel_orientation_quirks(E) backlight(E) fuse(E) loop(E) dax(E) configfs(E) dmi_sysfs(E) efivarfs(E)
> > > [  460.056272] CPU: 1 UID: 0 PID: 32 Comm: kauditd Tainted: G            EL      6.16.0-rc7+ #3 PREEMPT(none)
> > > [  460.056275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
> > > [  460.056276] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
> > > [  460.056277] RIP: 0010:_raw_spin_unlock_irqrestore+0x1b/0x30
> > > [  460.056284] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 16 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d b5 23 b6 01 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 e9 14 23 00 00 0f 1f 40 00 90
> > > [  460.056285] RSP: 0018:ffffb762c0123d70 EFLAGS: 00000246
> > > [  460.056287] RAX: 0000000000000001 RBX: ffff9b14c08eafc0 RCX: ffff9b14c3337348
> > > [  460.056288] RDX: ffff9b14c3337348 RSI: 0000000000000246 RDI: ffff9b14c3337340
> > > [  460.056289] RBP: ffff9b14c3337000 R08: ffffffff93cea880 R09: 0000000000000001
> > > [  460.056290] R10: 0000000000000001 R11: 0000000000000080 R12: ffff9b14c1b72500
> > > [  460.056291] R13: ffffb762c0123de0 R14: ffff9b14c3337340 R15: ffff9b14c3337080
> > > [  460.056294] FS:  0000000000000000(0000) GS:ffff9b1563788000(0000) knlGS:0000000000000000
> > > [  460.056296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [  460.056297] CR2: 00007f36fd5d21b4 CR3: 000000010241a002 CR4: 00000000007706f0
> > > [  460.056298] PKRU: 55555554
> > > [  460.056299] Call Trace:
> > > [  460.056300]  <TASK>
> > > [  460.056302]  netlink_attachskb+0xb7/0x2f0
> > > [  460.056308]  ? __pfx_default_wake_function+0x10/0x10
> > > [  460.056313]  netlink_unicast+0xea/0x3b0
> > ...
> > >
> > > We've bug reports from many users, so the issue is rather wide-spread.
> > >
> > > So far I don't know why the commit is causing this issue and will keep
> > > investigating. However, when reverted (together with its 2 follow-up
> > > patches), the issue goes away and host do not lock up.
> >
> > Thanks for the report, Max!
> >
> > Does your tree have this commit ?  This is the 3rd follow-up patch.
> >
> > commit 759dfc7d04bab1b0b86113f1164dc1fec192b859
> > Author: Fedor Pchelkin <pchelkin@ispras.ru>
> > Date:   Mon Jul 28 08:06:47 2025
> >
> >     netlink: avoid infinite retry looping in netlink_unicast()
> 

Hi Paul,

> Hopefully that resolves the problem, Maximilian?

sorry for the late reply. Just tested the commit yesterday and I can
confirm that this fixes our issues.

> Normally the audit subsystem is reasonably robust when faced with
> significant audit loads.  An example I use for testing is to enable
> logging for *every* syscall (from the command line, don't make this
> persist via the config file!) and then shutdown the system; the system
> will obviously slow quite a bit under the absurd load, but it should
> shutdown gracefully without any lockups.
> 

Thank you for suggesting this. Will add something like this to our
internal testing. Do you know whether there is already some stress test
that covers the audit subsystem or would have any selftest found this
issue?

Regards,
Maximilian



Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

[Paul Moore]

On Fri, Aug 15, 2025 at 6:00 AM Heyne, Maximilian <mheyne@amazon.de> wrote:
> On Wed, Aug 13, 2025 at 03:00:29PM -0400, Paul Moore wrote:

...

> > Hopefully that resolves the problem, Maximilian?
>
> sorry for the late reply. Just tested the commit yesterday and I can
> confirm that this fixes our issues.

Great, thanks for confirming that.

> > Normally the audit subsystem is reasonably robust when faced with
> > significant audit loads.  An example I use for testing is to enable
> > logging for *every* syscall (from the command line, don't make this
> > persist via the config file!) and then shutdown the system; the system
> > will obviously slow quite a bit under the absurd load, but it should
> > shutdown gracefully without any lockups.
>
> Thank you for suggesting this. Will add something like this to our
> internal testing.

I wish I could say I regularly stress the audit subsystem in that way,
but I typically only do that when I make a related change or happen to
notice something in a related subsystem which might have an impact.
Additional testing is always welcome!

> Do you know whether there is already some stress test
> that covers the audit subsystem ...

Aside from the manual test that I already mentioned, which is my
preferred mechanism for stressing the logging/queuing mechanism, there
are two (?) contributed stress tests in the audit-testsuite, but I
don't run them and I doubt anyone does on a regular basis (look in the
tests_manual directory).

* https://github.com/linux-audit/audit-testsuite

> ... or would have any selftest found this issue?

Not having looked at the root cause, as that work was done before I
dug into this thread, I honestly can't say.

-- 
paul-moore.com