[PATCH v3 0/2] kselftest: mm: fix intermittent failure khugepaged test

[PATCH v3 0/2] kselftest: mm: fix intermittent failure khugepaged test

[Yeoreum Yun]

There are intermittent failures in collapse_max_ptes_swap() and
collapse_max_ptes_shared() when using the khugepaged_context:

  # Run test: collapse_max_ptes_shared (khugepaged:anon)
  # Allocate huge page... OK
  # Share huge page over fork()... OK
  # Trigger CoW on page 1023 of 2048... OK
  # Maybe collapse with max_ptes_shared exceeded.... OK
  # Trigger CoW on page 1024 of 2048... Fail
  Bail out! Unexpected huge page
  # Planned tests != run tests (26 != 23)
  # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0

  # Run test: collapse_max_ptes_swap (khugepaged:anon)
  # Swapout 257 of 2048 pages... OK
  # Maybe collapse with max_ptes_swap exceeded.... OK
  # Swapout 256 of 2048 pages... OK
  Bail out! Unexpected huge page
  # Planned tests != run tests (26 != 17)
  # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0

This happens because khugepaged may collapse the pages before wait_for_scan()
is called, causing a sanity check that expects uncollapsed pages to fail.

For example, in collapse_max_ptes_swap(), after faulting the pages back in
and paging out up to max_ptes_swap pages, khugepaged may collapse them again
before c->collapse() is called.

To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been
collapsed by wait_for_scan() for anon. This prevents khugepaged from
collapsing it again before c->collapse() is called.

Also, fix false-positive results when a child process fails in tests
such as collapse_fork*() or collapse_max_ptes_shared():

  # -------------------------
  # running ./khugepaged -s 2
  # -------------------------
  #
  # Run test: collapse_max_ptes_shared (khugepaged:anon)
  # Allocate huge page... OK
  # Share huge page over fork()... OK
  # Trigger CoW on page 1023 of 2048... OK
  # Maybe collapse with max_ptes_shared exceeded.... OK
  # Trigger CoW on page 1024 of 2048... Fail
  Bail out! Unexpected huge page
  # Planned tests != run tests (26 != 23)
  # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0  // child failed.
  # Check if parent still has huge page... OK              // parent hpage success
  ok 24 collapse_max_ptes_shared                           // considered as success
  ...
  # Totals: pass:26 fail:0 xfail:0 xpass:0 skip:0 error:0

This failure was observed on NVIDIA Spark with 16KB page.

---
Changes in v3:
- using is_anon() instead of comparing name of mem_ops
- add r-b tag.
- Link to v2: https://lore.kernel.org/all/20260921-fix_khugepagd_fail-v2-0-3c2877beef61@arm.com/

Changes in v2:
- remove temporary enabled setup.
- Link to v1: https://lore.kernel.org/r/20260915-fix_khugepagd_fail-v1-0-bb6f04c8759f@arm.com

---
Yeoreum Yun (2):
      kselftest: mm: return fail when child test result is fail in khugepaged
      kselftest: mm: fix intermittent failure khugepaged test

 tools/testing/selftests/mm/khugepaged.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
---
base-commit: 685086170033a6ab334f0068cd25dae90fd1bb4e
change-id: 20260923-fix_khugepagd_fail-172362f58d17

Best regards,
-- 
Sincerely,
Yeoreum Yun

[PATCH v3 1/2] kselftest: mm: return fail when child test result is fail in khugepaged

[Yeoreum Yun]

Although the child process in collapse_fork*() or collapse_max_ptes_shared()
reports `KSFT_FAIL`, the result is ignored because the test only checks
whether the parent’s page was collapsed into a huge page.

As a result, the test is considered successful whenever the parent’s page
is a huge page, even if the child test fails, as shown below:

  #
  # Run test: collapse_max_ptes_shared (khugepaged:anon)
  # Allocate huge page... OK
  # Share huge page over fork()... OK
  # Trigger CoW on page 1023 of 2048... OK
  # Maybe collapse with max_ptes_shared exceeded.... OK
  # Trigger CoW on page 1024 of 2048... Fail
  Bail out! Unexpected huge page
  # Planned tests != run tests (26 != 23)
  # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0  // child failed.
  # Check if parent still has huge page... OK              // parent hpage success
  ok 24 collapse_max_ptes_shared                           // considered as success
  ...
  # Totals: pass:26 fail:0 xfail:0 xpass:0 skip:0 error:0

To address this, propagate the child’s failure and skip the subsequent
check in the parent.

Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 tools/testing/selftests/mm/khugepaged.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c
index a2ac3b3ca5de..2aa7c9197158 100644
--- a/tools/testing/selftests/mm/khugepaged.c
+++ b/tools/testing/selftests/mm/khugepaged.c
@@ -1053,6 +1053,8 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops)
 
 	wait(&wstatus);
 	exit_status = WEXITSTATUS(wstatus);
+	if (exit_status == KSFT_FAIL)
+		goto out;
 
 	ksft_print_msg("Check if parent still has small page...");
 	if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
@@ -1060,6 +1062,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops)
 	else
 		fail("Fail");
 	validate_memory(p, 0, page_size);
+out:
 	ops->cleanup_area(p, hpage_pmd_size);
 	ksft_test_result_report(exit_status, "%s\n", __func__);
 }
@@ -1100,6 +1103,8 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o
 
 	wait(&wstatus);
 	exit_status = WEXITSTATUS(wstatus);
+	if (exit_status == KSFT_FAIL)
+		goto out;
 
 	ksft_print_msg("Check if parent still has huge page...");
 	if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
@@ -1107,6 +1112,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o
 	else
 		fail("Fail");
 	validate_memory(p, 0, hpage_pmd_size);
+out:
 	ops->cleanup_area(p, hpage_pmd_size);
 	ksft_test_result_report(exit_status, "%s\n", __func__);
 }
@@ -1158,6 +1164,8 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops
 
 	wait(&wstatus);
 	exit_status = WEXITSTATUS(wstatus);
+	if (exit_status == KSFT_FAIL)
+		goto out;
 
 	ksft_print_msg("Check if parent still has huge page...");
 	if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
@@ -1165,6 +1173,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops
 	else
 		fail("Fail");
 	validate_memory(p, 0, hpage_pmd_size);
+out:
 	ops->cleanup_area(p, hpage_pmd_size);
 	ksft_test_result_report(exit_status, "%s\n", __func__);
 }

-- 
2.43.0

[PATCH v3 2/2] kselftest: mm: fix intermittent failure khugepaged test

[Yeoreum Yun]

There are intermittent failures in collapse_max_ptes_swap() and
collapse_max_ptes_shared() when using the khugepaged_context:

  // while running ./khugepaged -s 2

  # Run test: collapse_max_ptes_shared (khugepaged:anon)
  # Allocate huge page... OK
  # Share huge page over fork()... OK
  # Trigger CoW on page 1023 of 2048... OK
  # Maybe collapse with max_ptes_shared exceeded.... OK
  # Trigger CoW on page 1024 of 2048... Fail
  Bail out! Unexpected huge page
  # Planned tests != run tests (26 != 23)
  # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0

  # Run test: collapse_max_ptes_swap (khugepaged:anon)
  # Swapout 257 of 2048 pages... OK
  # Maybe collapse with max_ptes_swap exceeded.... OK
  # Swapout 256 of 2048 pages... OK
  Bail out! Unexpected huge page
  # Planned tests != run tests (26 != 17)
  # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0

This happens because khugepaged may collapse the pages before wait_for_scan()
is called, causing a sanity check that expects uncollapsed pages to fail.

For example, in collapse_max_ptes_swap(), after faulting the pages back in
and paging out up to max_ptes_swap pages, khugepaged may collapse them again
before c->collapse() is called.

To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been
collapsed by wait_for_scan() for anon. This prevents khugepaged from
collapsing it again before c->collapse() is called.

This failure was observed on NVIDIA Spark with 16KB page.

Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 tools/testing/selftests/mm/khugepaged.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c
index 2aa7c9197158..b0cb02bf1a73 100644
--- a/tools/testing/selftests/mm/khugepaged.c
+++ b/tools/testing/selftests/mm/khugepaged.c
@@ -618,6 +618,9 @@ static bool wait_for_scan(const char *msg, char *p, size_t len,
 		usleep(TICK);
 	}
 
+	if (is_anon(ops))
+		madvise(p, len, MADV_NOHUGEPAGE);
+
 	return timeout == -1;
 }
 

-- 
2.43.0

Re: [PATCH v3 1/2] kselftest: mm: return fail when child test result is fail in khugepaged

[Gregory Price]

On Wed, Sep 23, 2026 at 04:29:51PM +0100, Yeoreum Yun wrote:
> Although the child process in collapse_fork*() or collapse_max_ptes_shared()
> reports `KSFT_FAIL`, the result is ignored because the test only checks
> whether the parent’s page was collapsed into a huge page.
> 
> As a result, the test is considered successful whenever the parent’s page
> is a huge page, even if the child test fails, as shown below:
> 
>   #
>   # Run test: collapse_max_ptes_shared (khugepaged:anon)
>   # Allocate huge page... OK
>   # Share huge page over fork()... OK
>   # Trigger CoW on page 1023 of 2048... OK
>   # Maybe collapse with max_ptes_shared exceeded.... OK
>   # Trigger CoW on page 1024 of 2048... Fail
>   Bail out! Unexpected huge page
>   # Planned tests != run tests (26 != 23)
>   # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0  // child failed.
>   # Check if parent still has huge page... OK              // parent hpage success
>   ok 24 collapse_max_ptes_shared                           // considered as success
>   ...
>   # Totals: pass:26 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
> To address this, propagate the child’s failure and skip the subsequent
> check in the parent.
> 
> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>

Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>

Re: [PATCH v3 2/2] kselftest: mm: fix intermittent failure khugepaged test

[Gregory Price]

On Wed, Sep 23, 2026 at 04:29:52PM +0100, Yeoreum Yun wrote:
> There are intermittent failures in collapse_max_ptes_swap() and
> collapse_max_ptes_shared() when using the khugepaged_context:
> 
>   // while running ./khugepaged -s 2
> 
>   # Run test: collapse_max_ptes_shared (khugepaged:anon)
>   # Allocate huge page... OK
>   # Share huge page over fork()... OK
>   # Trigger CoW on page 1023 of 2048... OK
>   # Maybe collapse with max_ptes_shared exceeded.... OK
>   # Trigger CoW on page 1024 of 2048... Fail
>   Bail out! Unexpected huge page
>   # Planned tests != run tests (26 != 23)
>   # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
>   # Run test: collapse_max_ptes_swap (khugepaged:anon)
>   # Swapout 257 of 2048 pages... OK
>   # Maybe collapse with max_ptes_swap exceeded.... OK
>   # Swapout 256 of 2048 pages... OK
>   Bail out! Unexpected huge page
>   # Planned tests != run tests (26 != 17)
>   # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
> This happens because khugepaged may collapse the pages before wait_for_scan()
> is called, causing a sanity check that expects uncollapsed pages to fail.
> 
> For example, in collapse_max_ptes_swap(), after faulting the pages back in
> and paging out up to max_ptes_swap pages, khugepaged may collapse them again
> before c->collapse() is called.
> 
> To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been
> collapsed by wait_for_scan() for anon. This prevents khugepaged from
> collapsing it again before c->collapse() is called.
> 
> This failure was observed on NVIDIA Spark with 16KB page.
> 
> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>

Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>

Re: [PATCH v3 1/2] kselftest: mm: return fail when child test result is fail in khugepaged

[David Hildenbrand (Arm)]

On 9/23/26 17:29, Yeoreum Yun wrote:
> Although the child process in collapse_fork*() or collapse_max_ptes_shared()
> reports `KSFT_FAIL`, the result is ignored because the test only checks
> whether the parent’s page was collapsed into a huge page.
> 
> As a result, the test is considered successful whenever the parent’s page
> is a huge page, even if the child test fails, as shown below:
> 
>   #
>   # Run test: collapse_max_ptes_shared (khugepaged:anon)
>   # Allocate huge page... OK
>   # Share huge page over fork()... OK
>   # Trigger CoW on page 1023 of 2048... OK
>   # Maybe collapse with max_ptes_shared exceeded.... OK
>   # Trigger CoW on page 1024 of 2048... Fail
>   Bail out! Unexpected huge page
>   # Planned tests != run tests (26 != 23)
>   # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0  // child failed.
>   # Check if parent still has huge page... OK              // parent hpage success
>   ok 24 collapse_max_ptes_shared                           // considered as success
>   ...
>   # Totals: pass:26 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
> To address this, propagate the child’s failure and skip the subsequent
> check in the parent.
> 
> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> ---

That test implementation is rather messy ... :)

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v3 1/2] kselftest: mm: return fail when child test result is fail in khugepaged

[Lorenzo Stoakes (ARM)]

On Tue, Sep 29, 2026 at 10:21:53AM +0200, David Hildenbrand (Arm) wrote:
> On 9/23/26 17:29, Yeoreum Yun wrote:
> > Although the child process in collapse_fork*() or collapse_max_ptes_shared()
> > reports `KSFT_FAIL`, the result is ignored because the test only checks
> > whether the parent’s page was collapsed into a huge page.
> >
> > As a result, the test is considered successful whenever the parent’s page
> > is a huge page, even if the child test fails, as shown below:
> >
> >   #
> >   # Run test: collapse_max_ptes_shared (khugepaged:anon)
> >   # Allocate huge page... OK
> >   # Share huge page over fork()... OK
> >   # Trigger CoW on page 1023 of 2048... OK
> >   # Maybe collapse with max_ptes_shared exceeded.... OK
> >   # Trigger CoW on page 1024 of 2048... Fail
> >   Bail out! Unexpected huge page
> >   # Planned tests != run tests (26 != 23)
> >   # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0  // child failed.
> >   # Check if parent still has huge page... OK              // parent hpage success
> >   ok 24 collapse_max_ptes_shared                           // considered as success
> >   ...
> >   # Totals: pass:26 fail:0 xfail:0 xpass:0 skip:0 error:0
> >
> > To address this, propagate the child’s failure and skip the subsequent
> > check in the parent.
> >
> > Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> > Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> > ---
>
> That test implementation is rather messy ... :)

Yeah, I wish we could just convert more and more tests to the harness. Makes
everything easier...!

>
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>

Patch LGTM so:

Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>

>
> --
> Cheers,
>
> David

--
Cheers, Lorenzo

Re: [PATCH v3 2/2] kselftest: mm: fix intermittent failure khugepaged test

[David Hildenbrand (Arm)]

On 9/23/26 17:29, Yeoreum Yun wrote:
> There are intermittent failures in collapse_max_ptes_swap() and
> collapse_max_ptes_shared() when using the khugepaged_context:
> 
>   // while running ./khugepaged -s 2
> 
>   # Run test: collapse_max_ptes_shared (khugepaged:anon)
>   # Allocate huge page... OK
>   # Share huge page over fork()... OK
>   # Trigger CoW on page 1023 of 2048... OK
>   # Maybe collapse with max_ptes_shared exceeded.... OK
>   # Trigger CoW on page 1024 of 2048... Fail
>   Bail out! Unexpected huge page
>   # Planned tests != run tests (26 != 23)
>   # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
>   # Run test: collapse_max_ptes_swap (khugepaged:anon)
>   # Swapout 257 of 2048 pages... OK
>   # Maybe collapse with max_ptes_swap exceeded.... OK
>   # Swapout 256 of 2048 pages... OK
>   Bail out! Unexpected huge page
>   # Planned tests != run tests (26 != 17)
>   # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
> This happens because khugepaged may collapse the pages before wait_for_scan()
> is called, causing a sanity check that expects uncollapsed pages to fail.
> 
> For example, in collapse_max_ptes_swap(), after faulting the pages back in
> and paging out up to max_ptes_swap pages, khugepaged may collapse them again
> before c->collapse() is called.
> 
> To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been
> collapsed by wait_for_scan() for anon. This prevents khugepaged from
> collapsing it again before c->collapse() is called.
> 
> This failure was observed on NVIDIA Spark with 16KB page.
> 
> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> ---
>  tools/testing/selftests/mm/khugepaged.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c
> index 2aa7c9197158..b0cb02bf1a73 100644
> --- a/tools/testing/selftests/mm/khugepaged.c
> +++ b/tools/testing/selftests/mm/khugepaged.c
> @@ -618,6 +618,9 @@ static bool wait_for_scan(const char *msg, char *p, size_t len,
>  		usleep(TICK);
>  	}
>  
> +	if (is_anon(ops))
> +		madvise(p, len, MADV_NOHUGEPAGE);
> +

Any reason we just do that unconditionally?

After all, we set the

	madvise(p, len, MADV_HUGEPAGE);

unconditionally?


-- 
Cheers,

David

Re: [PATCH v3 2/2] kselftest: mm: fix intermittent failure khugepaged test

[Baolin Wang]

On 9/29/26 4:38 PM, David Hildenbrand (Arm) wrote:
> On 9/23/26 17:29, Yeoreum Yun wrote:
>> There are intermittent failures in collapse_max_ptes_swap() and
>> collapse_max_ptes_shared() when using the khugepaged_context:
>>
>>    // while running ./khugepaged -s 2
>>
>>    # Run test: collapse_max_ptes_shared (khugepaged:anon)
>>    # Allocate huge page... OK
>>    # Share huge page over fork()... OK
>>    # Trigger CoW on page 1023 of 2048... OK
>>    # Maybe collapse with max_ptes_shared exceeded.... OK
>>    # Trigger CoW on page 1024 of 2048... Fail
>>    Bail out! Unexpected huge page
>>    # Planned tests != run tests (26 != 23)
>>    # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0
>>
>>    # Run test: collapse_max_ptes_swap (khugepaged:anon)
>>    # Swapout 257 of 2048 pages... OK
>>    # Maybe collapse with max_ptes_swap exceeded.... OK
>>    # Swapout 256 of 2048 pages... OK
>>    Bail out! Unexpected huge page
>>    # Planned tests != run tests (26 != 17)
>>    # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0
>>
>> This happens because khugepaged may collapse the pages before wait_for_scan()
>> is called, causing a sanity check that expects uncollapsed pages to fail.
>>
>> For example, in collapse_max_ptes_swap(), after faulting the pages back in
>> and paging out up to max_ptes_swap pages, khugepaged may collapse them again
>> before c->collapse() is called.
>>
>> To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been
>> collapsed by wait_for_scan() for anon. This prevents khugepaged from
>> collapsing it again before c->collapse() is called.
>>
>> This failure was observed on NVIDIA Spark with 16KB page.
>>
>> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
>> ---
>>   tools/testing/selftests/mm/khugepaged.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c
>> index 2aa7c9197158..b0cb02bf1a73 100644
>> --- a/tools/testing/selftests/mm/khugepaged.c
>> +++ b/tools/testing/selftests/mm/khugepaged.c
>> @@ -618,6 +618,9 @@ static bool wait_for_scan(const char *msg, char *p, size_t len,
>>   		usleep(TICK);
>>   	}
>>   
>> +	if (is_anon(ops))
>> +		madvise(p, len, MADV_NOHUGEPAGE);
>> +
> 
> Any reason we just do that unconditionally?

Although it's a bit messy, as I mentioned before [1], unconditionally 
setting MADV_NOHUGEPAGE will break shmem testing. Maybe add some comments.

[1] 
https://lore.kernel.org/all/24690e82-3aab-4f2d-95a7-3bba332ca5bc@linux.alibaba.com/

Re: [PATCH v3 2/2] kselftest: mm: fix intermittent failure khugepaged test

[David Hildenbrand (Arm)]

On 9/29/26 10:43, Baolin Wang wrote:
> 
> 
> On 9/29/26 4:38 PM, David Hildenbrand (Arm) wrote:
>> On 9/23/26 17:29, Yeoreum Yun wrote:
>>> There are intermittent failures in collapse_max_ptes_swap() and
>>> collapse_max_ptes_shared() when using the khugepaged_context:
>>>
>>>    // while running ./khugepaged -s 2
>>>
>>>    # Run test: collapse_max_ptes_shared (khugepaged:anon)
>>>    # Allocate huge page... OK
>>>    # Share huge page over fork()... OK
>>>    # Trigger CoW on page 1023 of 2048... OK
>>>    # Maybe collapse with max_ptes_shared exceeded.... OK
>>>    # Trigger CoW on page 1024 of 2048... Fail
>>>    Bail out! Unexpected huge page
>>>    # Planned tests != run tests (26 != 23)
>>>    # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0
>>>
>>>    # Run test: collapse_max_ptes_swap (khugepaged:anon)
>>>    # Swapout 257 of 2048 pages... OK
>>>    # Maybe collapse with max_ptes_swap exceeded.... OK
>>>    # Swapout 256 of 2048 pages... OK
>>>    Bail out! Unexpected huge page
>>>    # Planned tests != run tests (26 != 17)
>>>    # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0
>>>
>>> This happens because khugepaged may collapse the pages before wait_for_scan()
>>> is called, causing a sanity check that expects uncollapsed pages to fail.
>>>
>>> For example, in collapse_max_ptes_swap(), after faulting the pages back in
>>> and paging out up to max_ptes_swap pages, khugepaged may collapse them again
>>> before c->collapse() is called.
>>>
>>> To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been
>>> collapsed by wait_for_scan() for anon. This prevents khugepaged from
>>> collapsing it again before c->collapse() is called.
>>>
>>> This failure was observed on NVIDIA Spark with 16KB page.
>>>
>>> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>>> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>>> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
>>> ---
>>>   tools/testing/selftests/mm/khugepaged.c | 3 +++
>>>   1 file changed, 3 insertions(+)
>>>
>>> diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/
>>> selftests/mm/khugepaged.c
>>> index 2aa7c9197158..b0cb02bf1a73 100644
>>> --- a/tools/testing/selftests/mm/khugepaged.c
>>> +++ b/tools/testing/selftests/mm/khugepaged.c
>>> @@ -618,6 +618,9 @@ static bool wait_for_scan(const char *msg, char *p,
>>> size_t len,
>>>           usleep(TICK);
>>>       }
>>>   +    if (is_anon(ops))
>>> +        madvise(p, len, MADV_NOHUGEPAGE);
>>> +
>>
>> Any reason we just do that unconditionally?
> 
> Although it's a bit messy, as I mentioned before [1], unconditionally setting
> MADV_NOHUGEPAGE will break shmem testing. Maybe add some comments.

Ah, thanks for clarifying. The problem really is that we cannot undo a
MADV_HUGEPAGE (give me hugepages) cleanly. We can only go to the other extreme
(no huge pages).

Yes, let's please add a comment describing why we limit it to anon.

-- 
Cheers,

David

Re: [PATCH v3 2/2] kselftest: mm: fix intermittent failure khugepaged test

[Yeoreum Yun]

On Tue, Sep 29, 2026 at 10:46:50AM +0200, David Hildenbrand (Arm) wrote:
> On 9/29/26 10:43, Baolin Wang wrote:
> > 
> > 
> > On 9/29/26 4:38 PM, David Hildenbrand (Arm) wrote:
> >> On 9/23/26 17:29, Yeoreum Yun wrote:
> >>> There are intermittent failures in collapse_max_ptes_swap() and
> >>> collapse_max_ptes_shared() when using the khugepaged_context:
> >>>
> >>>    // while running ./khugepaged -s 2
> >>>
> >>>    # Run test: collapse_max_ptes_shared (khugepaged:anon)
> >>>    # Allocate huge page... OK
> >>>    # Share huge page over fork()... OK
> >>>    # Trigger CoW on page 1023 of 2048... OK
> >>>    # Maybe collapse with max_ptes_shared exceeded.... OK
> >>>    # Trigger CoW on page 1024 of 2048... Fail
> >>>    Bail out! Unexpected huge page
> >>>    # Planned tests != run tests (26 != 23)
> >>>    # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0
> >>>
> >>>    # Run test: collapse_max_ptes_swap (khugepaged:anon)
> >>>    # Swapout 257 of 2048 pages... OK
> >>>    # Maybe collapse with max_ptes_swap exceeded.... OK
> >>>    # Swapout 256 of 2048 pages... OK
> >>>    Bail out! Unexpected huge page
> >>>    # Planned tests != run tests (26 != 17)
> >>>    # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0
> >>>
> >>> This happens because khugepaged may collapse the pages before wait_for_scan()
> >>> is called, causing a sanity check that expects uncollapsed pages to fail.
> >>>
> >>> For example, in collapse_max_ptes_swap(), after faulting the pages back in
> >>> and paging out up to max_ptes_swap pages, khugepaged may collapse them again
> >>> before c->collapse() is called.
> >>>
> >>> To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been
> >>> collapsed by wait_for_scan() for anon. This prevents khugepaged from
> >>> collapsing it again before c->collapse() is called.
> >>>
> >>> This failure was observed on NVIDIA Spark with 16KB page.
> >>>
> >>> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> >>> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> >>> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> >>> ---
> >>>   tools/testing/selftests/mm/khugepaged.c | 3 +++
> >>>   1 file changed, 3 insertions(+)
> >>>
> >>> diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/
> >>> selftests/mm/khugepaged.c
> >>> index 2aa7c9197158..b0cb02bf1a73 100644
> >>> --- a/tools/testing/selftests/mm/khugepaged.c
> >>> +++ b/tools/testing/selftests/mm/khugepaged.c
> >>> @@ -618,6 +618,9 @@ static bool wait_for_scan(const char *msg, char *p,
> >>> size_t len,
> >>>           usleep(TICK);
> >>>       }
> >>>   +    if (is_anon(ops))
> >>> +        madvise(p, len, MADV_NOHUGEPAGE);
> >>> +
> >>
> >> Any reason we just do that unconditionally?
> > 
> > Although it's a bit messy, as I mentioned before [1], unconditionally setting
> > MADV_NOHUGEPAGE will break shmem testing. Maybe add some comments.
> 
> Ah, thanks for clarifying. The problem really is that we cannot undo a
> MADV_HUGEPAGE (give me hugepages) cleanly. We can only go to the other extreme
> (no huge pages).
> 
> Yes, let's please add a comment describing why we limit it to anon.

Okay.

-- 
Sincerely,
Yeoreum Yun