* [PATCH] mm/vma: update VMA iterator state on error in new VMA merge
@ 2026-10-27 11:25 Lorenzo Stoakes (ARM)
2026-11-12 22:01 ` Liam R. Howlett
0 siblings, 1 reply; 3+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-10-27 11:25 UTC (permalink / raw)
To: Andrew Morton, Liam R. Howlett, Vlastimil Babka, Jann Horn,
Pedro Falcato
Cc: linux-mm, linux-kernel, ljs, stable, Li Jinyue
Commit cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs") shared
code between vma_merge_new_range() and vma_expand().
However in doing so, it failed to correctly restore the VMA iterator when
vma_expand() errors out.
This is a problem when the immediately preceding VMA is mergeable and
vmg->just_expand isn't specified, at which point the VMA iterator is moved
to the previous VMA.
(Note that vmg->just_expand is a special case for brk(), where the
iterator will already be positioned at the VMA to be expanded.)
Resolve things by restoring the VMA iterator on error. Use
vma_iter_next_range() to do this, so it will correctly stop on the gap
after the previous VMA, rather than skip to the next VMA after it (this is
a new VMA merge, so is being applied at a gap).
To isolate the error handling, return early when a merge isn't
possible (what the vmg->target check was doing before), then handle the
merge success/failure case explicitly.
While here, update vma_expand() to jump to the nomem case when
dup_anon_vma() fails, so vmg->state is correctly updated.
This doesn't impact anything in practice, but is the correct course and
since it's part of the error path that this patch addresses, it makes sense
to update it also.
Users are very unlikely to have observed the issue this patch addresses in
practice, as it is near-impossible to fail an allocation for an anon_vma,
but a process under memory pressure that receives a fatal signal could
trigger it.
This issue was originally reported by Li Jinyue for which I am grateful. He
indicated this was uncovered by a private syzkaller instance.
Fixes: cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs")
Cc: stable@vger.kernel.org
Reported-by: Li Jinyue <jinyue.li@linux.alibaba.com>
Closes: https://lore.kernel.org/all/20261025093921.1011927-1-jinyue.li@linux.alibaba.com/
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
mm/vma.c | 28 ++++++++++++++++++----------
1 file changed, 18 insertions(+), 10 deletions(-)
diff --git a/mm/vma.c b/mm/vma.c
index af8e65ada23b..d416ba58eb4c 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -1141,6 +1141,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
struct vm_area_struct *next = vmg->next;
unsigned long end = vmg->end;
bool can_merge_left, can_merge_right;
+ bool move_iter_back = false;
mmap_assert_write_locked(vmg->mm);
VM_WARN_ON_VMG(vmg->middle, vmg);
@@ -1182,23 +1183,30 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
vmg->end = end;
/* In expand-only case we are already positioned at prev. */
- if (!vmg->just_expand) {
- /* Equivalent to going to the previous range. */
- vma_prev(vmg->vmi);
- }
+ if (!vmg->just_expand)
+ move_iter_back = true;
}
+ if (!can_merge_left && !can_merge_right)
+ return NULL;
+
+ if (move_iter_back)
+ vma_prev(vmg->vmi);
+
/*
* Now try to expand adjacent VMA(s). This takes care of removing the
* following VMA if we have VMAs on both sides.
*/
- if (vmg->target && !vma_expand(vmg)) {
- khugepaged_enter_vma(vmg->target, vmg->vm_flags);
- vmg->state = VMA_MERGE_SUCCESS;
- return vmg->target;
+ if (vma_expand(vmg)) {
+ /* Error occurred, restore iterator. */
+ if (move_iter_back)
+ vma_iter_next_range(vmg->vmi);
+ return NULL;
}
- return NULL;
+ khugepaged_enter_vma(vmg->target, vmg->vm_flags);
+ vmg->state = VMA_MERGE_SUCCESS;
+ return vmg->target;
}
/*
@@ -1288,7 +1296,7 @@ int vma_expand(struct vma_merge_struct *vmg)
if (!ret && vmg->copied_from)
ret = dup_anon_vma(target, vmg->copied_from, &anon_dup);
if (ret)
- return ret;
+ goto nomem;
if (remove_next) {
vma_start_write(next);
---
base-commit: 1eb36316305767b54fac2cd40b88b3499791bdf2
change-id: 20261027-fix-new-merge-bug-71c2200d40fb
Cheers,
--
Lorenzo Stoakes (ARM) <ljs@kernel.org>
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] mm/vma: update VMA iterator state on error in new VMA merge
2026-10-27 11:25 [PATCH] mm/vma: update VMA iterator state on error in new VMA merge Lorenzo Stoakes (ARM)
@ 2026-11-12 22:01 ` Liam R. Howlett
2026-11-16 15:46 ` Lorenzo Stoakes (ARM)
0 siblings, 1 reply; 3+ messages in thread
From: Liam R. Howlett @ 2026-11-12 22:01 UTC (permalink / raw)
To: Lorenzo Stoakes (ARM)
Cc: Andrew Morton, Vlastimil Babka, Jann Horn, Pedro Falcato,
linux-mm, linux-kernel, stable, Li Jinyue
On 26/10/27 11:25AM, Lorenzo Stoakes (ARM) wrote:
I'm a bit late on this one..
> Commit cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs") shared
> code between vma_merge_new_range() and vma_expand().
>
> However in doing so, it failed to correctly restore the VMA iterator when
> vma_expand() errors out.
>
> This is a problem when the immediately preceding VMA is mergeable and
> vmg->just_expand isn't specified, at which point the VMA iterator is moved
> to the previous VMA.
>
> (Note that vmg->just_expand is a special case for brk(), where the
> iterator will already be positioned at the VMA to be expanded.)
>
> Resolve things by restoring the VMA iterator on error. Use
> vma_iter_next_range() to do this, so it will correctly stop on the gap
> after the previous VMA, rather than skip to the next VMA after it (this is
> a new VMA merge, so is being applied at a gap).
>
> To isolate the error handling, return early when a merge isn't
> possible (what the vmg->target check was doing before), then handle the
> merge success/failure case explicitly.
>
> While here, update vma_expand() to jump to the nomem case when
> dup_anon_vma() fails, so vmg->state is correctly updated.
>
> This doesn't impact anything in practice, but is the correct course and
> since it's part of the error path that this patch addresses, it makes sense
> to update it also.
>
> Users are very unlikely to have observed the issue this patch addresses in
> practice, as it is near-impossible to fail an allocation for an anon_vma,
> but a process under memory pressure that receives a fatal signal could
> trigger it.
>
> This issue was originally reported by Li Jinyue for which I am grateful. He
> indicated this was uncovered by a private syzkaller instance.
>
> Fixes: cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs")
> Cc: stable@vger.kernel.org
> Reported-by: Li Jinyue <jinyue.li@linux.alibaba.com>
> Closes: https://lore.kernel.org/all/20261025093921.1011927-1-jinyue.li@linux.alibaba.com/
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> ---
> mm/vma.c | 28 ++++++++++++++++++----------
> 1 file changed, 18 insertions(+), 10 deletions(-)
>
> diff --git a/mm/vma.c b/mm/vma.c
> index af8e65ada23b..d416ba58eb4c 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -1141,6 +1141,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
> struct vm_area_struct *next = vmg->next;
> unsigned long end = vmg->end;
> bool can_merge_left, can_merge_right;
> + bool move_iter_back = false;
The name move_iter_back is a bit confusing because you use it to move it
to the next entry. I get what you mean, move it back to where it was, not
go to prev, but the name is a bit off when used to go to the next entry.
reposition_iter, maybe?
>
> mmap_assert_write_locked(vmg->mm);
> VM_WARN_ON_VMG(vmg->middle, vmg);
> @@ -1182,23 +1183,30 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
> vmg->end = end;
>
> /* In expand-only case we are already positioned at prev. */
> - if (!vmg->just_expand) {
> - /* Equivalent to going to the previous range. */
> - vma_prev(vmg->vmi);
> - }
> + if (!vmg->just_expand)
> + move_iter_back = true;
> }
>
> + if (!can_merge_left && !can_merge_right)
> + return NULL;
> +
> + if (move_iter_back)
> + vma_prev(vmg->vmi);
> +
> /*
> * Now try to expand adjacent VMA(s). This takes care of removing the
> * following VMA if we have VMAs on both sides.
> */
> - if (vmg->target && !vma_expand(vmg)) {
> - khugepaged_enter_vma(vmg->target, vmg->vm_flags);
> - vmg->state = VMA_MERGE_SUCCESS;
> - return vmg->target;
> + if (vma_expand(vmg)) {
> + /* Error occurred, restore iterator. */
> + if (move_iter_back)
> + vma_iter_next_range(vmg->vmi);
> + return NULL;
> }
>
> - return NULL;
> + khugepaged_enter_vma(vmg->target, vmg->vm_flags);
> + vmg->state = VMA_MERGE_SUCCESS;
> + return vmg->target;
> }
>
> /*
> @@ -1288,7 +1296,7 @@ int vma_expand(struct vma_merge_struct *vmg)
> if (!ret && vmg->copied_from)
> ret = dup_anon_vma(target, vmg->copied_from, &anon_dup);
> if (ret)
> - return ret;
> + goto nomem;
>
> if (remove_next) {
> vma_start_write(next);
>
> ---
> base-commit: 1eb36316305767b54fac2cd40b88b3499791bdf2
> change-id: 20261027-fix-new-merge-bug-71c2200d40fb
>
> Cheers,
> --
> Lorenzo Stoakes (ARM) <ljs@kernel.org>
>
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] mm/vma: update VMA iterator state on error in new VMA merge
2026-11-12 22:01 ` Liam R. Howlett
@ 2026-11-16 15:46 ` Lorenzo Stoakes (ARM)
0 siblings, 0 replies; 3+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-11-16 15:46 UTC (permalink / raw)
To: Liam R. Howlett
Cc: Andrew Morton, Vlastimil Babka, Jann Horn, Pedro Falcato,
linux-mm, linux-kernel, stable, Li Jinyue
On Thu, Nov 12, 2026 at 05:01:34PM -0500, Liam R. Howlett wrote:
> On 26/10/27 11:25AM, Lorenzo Stoakes (ARM) wrote:
>
> I'm a bit late on this one..
>
> > Commit cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs") shared
> > code between vma_merge_new_range() and vma_expand().
> >
> > However in doing so, it failed to correctly restore the VMA iterator when
> > vma_expand() errors out.
> >
> > This is a problem when the immediately preceding VMA is mergeable and
> > vmg->just_expand isn't specified, at which point the VMA iterator is moved
> > to the previous VMA.
> >
> > (Note that vmg->just_expand is a special case for brk(), where the
> > iterator will already be positioned at the VMA to be expanded.)
> >
> > Resolve things by restoring the VMA iterator on error. Use
> > vma_iter_next_range() to do this, so it will correctly stop on the gap
> > after the previous VMA, rather than skip to the next VMA after it (this is
> > a new VMA merge, so is being applied at a gap).
> >
> > To isolate the error handling, return early when a merge isn't
> > possible (what the vmg->target check was doing before), then handle the
> > merge success/failure case explicitly.
> >
> > While here, update vma_expand() to jump to the nomem case when
> > dup_anon_vma() fails, so vmg->state is correctly updated.
> >
> > This doesn't impact anything in practice, but is the correct course and
> > since it's part of the error path that this patch addresses, it makes sense
> > to update it also.
> >
> > Users are very unlikely to have observed the issue this patch addresses in
> > practice, as it is near-impossible to fail an allocation for an anon_vma,
> > but a process under memory pressure that receives a fatal signal could
> > trigger it.
> >
> > This issue was originally reported by Li Jinyue for which I am grateful. He
> > indicated this was uncovered by a private syzkaller instance.
> >
> > Fixes: cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs")
> > Cc: stable@vger.kernel.org
> > Reported-by: Li Jinyue <jinyue.li@linux.alibaba.com>
> > Closes: https://lore.kernel.org/all/20261025093921.1011927-1-jinyue.li@linux.alibaba.com/
> > Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> > ---
> > mm/vma.c | 28 ++++++++++++++++++----------
> > 1 file changed, 18 insertions(+), 10 deletions(-)
> >
> > diff --git a/mm/vma.c b/mm/vma.c
> > index af8e65ada23b..d416ba58eb4c 100644
> > --- a/mm/vma.c
> > +++ b/mm/vma.c
> > @@ -1141,6 +1141,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
> > struct vm_area_struct *next = vmg->next;
> > unsigned long end = vmg->end;
> > bool can_merge_left, can_merge_right;
> > + bool move_iter_back = false;
>
> The name move_iter_back is a bit confusing because you use it to move it
> to the next entry. I get what you mean, move it back to where it was, not
> go to prev, but the name is a bit off when used to go to the next entry.
>
> reposition_iter, maybe?
Ahh too late this is merged :P
I think it's hard to get this quite right.
So in the first instance move_iter_back = do a vma_prev() i.e. literally move
the iterator backwards.
Then, if the iterator was moved back, we want to restore it on error, I hoped
that the comment would make that clear.
So it's just separating noting that it needs doing (move_iter_back = true) and
the actual action (vma_prev()) + the restore.
So I think arguably OK as-is? But could obviously also follow up if needed.
>
> >
> > mmap_assert_write_locked(vmg->mm);
> > VM_WARN_ON_VMG(vmg->middle, vmg);
> > @@ -1182,23 +1183,30 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
> > vmg->end = end;
> >
> > /* In expand-only case we are already positioned at prev. */
> > - if (!vmg->just_expand) {
> > - /* Equivalent to going to the previous range. */
> > - vma_prev(vmg->vmi);
> > - }
> > + if (!vmg->just_expand)
> > + move_iter_back = true;
> > }
> >
> > + if (!can_merge_left && !can_merge_right)
> > + return NULL;
> > +
> > + if (move_iter_back)
> > + vma_prev(vmg->vmi);
> > +
> > /*
> > * Now try to expand adjacent VMA(s). This takes care of removing the
> > * following VMA if we have VMAs on both sides.
> > */
> > - if (vmg->target && !vma_expand(vmg)) {
> > - khugepaged_enter_vma(vmg->target, vmg->vm_flags);
> > - vmg->state = VMA_MERGE_SUCCESS;
> > - return vmg->target;
> > + if (vma_expand(vmg)) {
> > + /* Error occurred, restore iterator. */
> > + if (move_iter_back)
> > + vma_iter_next_range(vmg->vmi);
> > + return NULL;
> > }
> >
> > - return NULL;
> > + khugepaged_enter_vma(vmg->target, vmg->vm_flags);
> > + vmg->state = VMA_MERGE_SUCCESS;
> > + return vmg->target;
> > }
> >
> > /*
> > @@ -1288,7 +1296,7 @@ int vma_expand(struct vma_merge_struct *vmg)
> > if (!ret && vmg->copied_from)
> > ret = dup_anon_vma(target, vmg->copied_from, &anon_dup);
> > if (ret)
> > - return ret;
> > + goto nomem;
> >
> > if (remove_next) {
> > vma_start_write(next);
> >
> > ---
> > base-commit: 1eb36316305767b54fac2cd40b88b3499791bdf2
> > change-id: 20261027-fix-new-merge-bug-71c2200d40fb
> >
> > Cheers,
> > --
> > Lorenzo Stoakes (ARM) <ljs@kernel.org>
> >
--
Cheers, Lorenzo
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-16 15:46 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-27 11:25 [PATCH] mm/vma: update VMA iterator state on error in new VMA merge Lorenzo Stoakes (ARM)
2026-11-12 22:01 ` Liam R. Howlett
2026-11-16 15:46 ` Lorenzo Stoakes (ARM)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®