Re: CVE-2024-50219: mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves

Re: CVE-2024-50219: mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves

[Vlastimil Babka]

On 11/9/24 11:15, Greg Kroah-Hartman wrote:
> Description
> ===========
> 
> In the Linux kernel, the following vulnerability has been resolved:
> 
> mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves
> 
> Under memory pressure it's possible for GFP_ATOMIC order-0 allocations to
> fail even though free pages are available in the highatomic reserves. 
> GFP_ATOMIC allocations cannot trigger unreserve_highatomic_pageblock()
> since it's only run from reclaim.
> 
> Given that such allocations will pass the watermarks in
> __zone_watermark_unusable_free(), it makes sense to fallback to highatomic
> reserves the same way that ALLOC_OOM can.
> 
> This fixes order-0 page allocation failures observed on Cloudflare's fleet
> when handling network packets:

Hi,

I would like to dispute the CVE. GFP_ATOMIC page allocations failures can
generally happen (typically from network receive path, like here) and should
always have a fallback. The impact could be somewhat worse performance at
worst. AFAIK they are not affected by panic_on_warn nor panic_on_oom either,
it's a pr_warn(), so I don't think there's a DoS vector.

It would be also nice to adjust any automatic flagging to distinguish this
for the future.

Thanks,
Vlastimil

>   kswapd1: page allocation failure: order:0, mode:0x820(GFP_ATOMIC),
>   nodemask=(null),cpuset=/,mems_allowed=0-7
>   CPU: 10 PID: 696 Comm: kswapd1 Kdump: loaded Tainted: G           O 6.6.43-CUSTOM #1
>   Hardware name: MACHINE
>   Call Trace:
>    <IRQ>
>    dump_stack_lvl+0x3c/0x50
>    warn_alloc+0x13a/0x1c0
>    __alloc_pages_slowpath.constprop.0+0xc9d/0xd10
>    __alloc_pages+0x327/0x340
>    __napi_alloc_skb+0x16d/0x1f0
>    bnxt_rx_page_skb+0x96/0x1b0 [bnxt_en]
>    bnxt_rx_pkt+0x201/0x15e0 [bnxt_en]
>    __bnxt_poll_work+0x156/0x2b0 [bnxt_en]
>    bnxt_poll+0xd9/0x1c0 [bnxt_en]
>    __napi_poll+0x2b/0x1b0
>    bpf_trampoline_6442524138+0x7d/0x1000
>    __napi_poll+0x5/0x1b0
>    net_rx_action+0x342/0x740
>    handle_softirqs+0xcf/0x2b0
>    irq_exit_rcu+0x6c/0x90
>    sysvec_apic_timer_interrupt+0x72/0x90
>    </IRQ>
> 
> [mfleming@cloudflare.com: update comment]
>   Link: https://lkml.kernel.org/r/20241015125158.3597702-1-matt@readmodwrite.com
> 
> The Linux kernel CVE team has assigned CVE-2024-50219 to this issue.
> 
> 
> Affected and fixed versions
> ===========================
> 
> 	Issue introduced in 5.9 with commit 1d91df85f399 and fixed in 5.15.171 with commit 4c4e238d3ada
> 	Issue introduced in 5.9 with commit 1d91df85f399 and fixed in 6.1.116 with commit 189b954469cf
> 	Issue introduced in 5.9 with commit 1d91df85f399 and fixed in 6.6.60 with commit b958948ae1cb
> 	Issue introduced in 5.9 with commit 1d91df85f399 and fixed in 6.11.7 with commit d0fdacfb85a3
> 	Issue introduced in 5.9 with commit 1d91df85f399 and fixed in 6.12-rc6 with commit 281dd25c1a01
> 
> Please see https://www.kernel.org for a full list of currently supported
> kernel versions by the kernel community.
> 
> Unaffected versions might change over time as fixes are backported to
> older supported kernel versions.  The official CVE entry at
> 	https://cve.org/CVERecord/?id=CVE-2024-50219
> will be updated if fixes are backported, please check that for the most
> up to date information about this issue.
> 
> 
> Affected files
> ==============
> 
> The file(s) affected by this issue are:
> 	mm/page_alloc.c
> 
> 
> Mitigation
> ==========
> 
> The Linux kernel CVE team recommends that you update to the latest
> stable kernel version for this, and many other bugfixes.  Individual
> changes are never tested alone, but rather are part of a larger kernel
> release.  Cherry-picking individual commits is not recommended or
> supported by the Linux kernel community at all.  If however, updating to
> the latest release is impossible, the individual changes to resolve this
> issue can be found at these commits:
> 	https://git.kernel.org/stable/c/4c4e238d3adad3c94bb255d0f117d3685bbfdd33
> 	https://git.kernel.org/stable/c/189b954469cf82f8b8cf496f8de94b006d2d4746
> 	https://git.kernel.org/stable/c/b958948ae1cb3e39c48e9f805436fd652103c71e
> 	https://git.kernel.org/stable/c/d0fdacfb85a3e1223b894cc6e60091ec91049e9e
> 	https://git.kernel.org/stable/c/281dd25c1a018261a04d1b8bf41a0674000bfe38
> 

Re: CVE-2024-50219: mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves

[Greg Kroah-Hartman]

On Mon, Nov 11, 2024 at 11:40:49AM +0100, Vlastimil Babka wrote:
> On 11/9/24 11:15, Greg Kroah-Hartman wrote:
> > Description
> > ===========
> > 
> > In the Linux kernel, the following vulnerability has been resolved:
> > 
> > mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves
> > 
> > Under memory pressure it's possible for GFP_ATOMIC order-0 allocations to
> > fail even though free pages are available in the highatomic reserves. 
> > GFP_ATOMIC allocations cannot trigger unreserve_highatomic_pageblock()
> > since it's only run from reclaim.
> > 
> > Given that such allocations will pass the watermarks in
> > __zone_watermark_unusable_free(), it makes sense to fallback to highatomic
> > reserves the same way that ALLOC_OOM can.
> > 
> > This fixes order-0 page allocation failures observed on Cloudflare's fleet
> > when handling network packets:
> 
> Hi,
> 
> I would like to dispute the CVE. GFP_ATOMIC page allocations failures can
> generally happen (typically from network receive path, like here) and should
> always have a fallback. The impact could be somewhat worse performance at
> worst. AFAIK they are not affected by panic_on_warn nor panic_on_oom either,
> it's a pr_warn(), so I don't think there's a DoS vector.

I read this as "there was a failure, with no fallback", but in looking
at the traceback:

> >   kswapd1: page allocation failure: order:0, mode:0x820(GFP_ATOMIC),
> >   nodemask=(null),cpuset=/,mems_allowed=0-7
> >   CPU: 10 PID: 696 Comm: kswapd1 Kdump: loaded Tainted: G           O 6.6.43-CUSTOM #1
> >   Hardware name: MACHINE
> >   Call Trace:
> >    <IRQ>
> >    dump_stack_lvl+0x3c/0x50
> >    warn_alloc+0x13a/0x1c0
> >    __alloc_pages_slowpath.constprop.0+0xc9d/0xd10
> >    __alloc_pages+0x327/0x340
> >    __napi_alloc_skb+0x16d/0x1f0

This function DOES have a fallback if this failed, so it's ok here.
Many other ATOMIC allocations in the kernel do not have fallbacks, which
would cause a crash.

Note, it is setting the NOWARN flag, so shouldn't this not be warning?

Anyway, you are right, I'll go reject this one, thanks for the review!

thanks,

greg k-h

Re: CVE-2024-50219: mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves

[Vlastimil Babka]

On 11/11/24 15:04, Greg Kroah-Hartman wrote:
> On Mon, Nov 11, 2024 at 11:40:49AM +0100, Vlastimil Babka wrote:
>> On 11/9/24 11:15, Greg Kroah-Hartman wrote:
>> > Description
>> > ===========
>> > 
>> > In the Linux kernel, the following vulnerability has been resolved:
>> > 
>> > mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves
>> > 
>> > Under memory pressure it's possible for GFP_ATOMIC order-0 allocations to
>> > fail even though free pages are available in the highatomic reserves. 
>> > GFP_ATOMIC allocations cannot trigger unreserve_highatomic_pageblock()
>> > since it's only run from reclaim.
>> > 
>> > Given that such allocations will pass the watermarks in
>> > __zone_watermark_unusable_free(), it makes sense to fallback to highatomic
>> > reserves the same way that ALLOC_OOM can.
>> > 
>> > This fixes order-0 page allocation failures observed on Cloudflare's fleet
>> > when handling network packets:
>> 
>> Hi,
>> 
>> I would like to dispute the CVE. GFP_ATOMIC page allocations failures can
>> generally happen (typically from network receive path, like here) and should
>> always have a fallback. The impact could be somewhat worse performance at
>> worst. AFAIK they are not affected by panic_on_warn nor panic_on_oom either,
>> it's a pr_warn(), so I don't think there's a DoS vector.
> 
> I read this as "there was a failure, with no fallback", but in looking
> at the traceback:
> 
>> >   kswapd1: page allocation failure: order:0, mode:0x820(GFP_ATOMIC),
>> >   nodemask=(null),cpuset=/,mems_allowed=0-7
>> >   CPU: 10 PID: 696 Comm: kswapd1 Kdump: loaded Tainted: G           O 6.6.43-CUSTOM #1
>> >   Hardware name: MACHINE
>> >   Call Trace:
>> >    <IRQ>
>> >    dump_stack_lvl+0x3c/0x50
>> >    warn_alloc+0x13a/0x1c0
>> >    __alloc_pages_slowpath.constprop.0+0xc9d/0xd10
>> >    __alloc_pages+0x327/0x340
>> >    __napi_alloc_skb+0x16d/0x1f0
> 
> This function DOES have a fallback if this failed, so it's ok here.
> Many other ATOMIC allocations in the kernel do not have fallbacks, which
> would cause a crash.

That would make them buggy and those should either use __GFP_NOFAIL or
handle the failure gracefully.

> Note, it is setting the NOWARN flag, so shouldn't this not be warning?

Hmm seems it does now, but AFAICS it's since 6e9b01909a81 ("net: remove
gfp_mask from napi_alloc_skb()") (since v6.10?) but the report above was
from 6.6.
Wasn't aware they decided to silence the warnings, I thought it was
intentional to make the admin aware they might need to increase the atomic
memory reserves for better performance (it's not trivial to tune AFAIK,
depends on bursty arrivals of packets etc). Also if there's a suboptimal
behavior in the implementation like the one commit fixed, it won't be so
easy to spot it anymore. On the other hand, less need to explain the
occasional warning :)

> Anyway, you are right, I'll go reject this one, thanks for the review!

Thanks!
Vlastimil

> 
> thanks,
> 
> greg k-h