[BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi all,

I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
involving a NULL pointer dereference in `__migrate_swap_task()` during
CPU migration. This occurred on various AMD and Intel systems while
running a CPU-intensive workload (Linpack, Stress_ng - it's not
specific to a benchmark).

Full trace below:
---
BUG: kernel NULL pointer dereference, address: 00000000000004c8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 4078b99067 P4D 4078b99067 PUD 0
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
RIP: 0010:__migrate_swap_task+0x2f/0x170
Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
PKRU: 55555554
Call Trace:
 <TASK>
 migrate_swap_stop+0xe8/0x190
 multi_cpu_stop+0xf3/0x130
 ? __pfx_multi_cpu_stop+0x10/0x10
 cpu_stopper_thread+0x97/0x140
 ? __pfx_smpboot_thread_fn+0x10/0x10
 smpboot_thread_fn+0xf3/0x220
 kthread+0xfc/0x240
 ? __pfx_kthread+0x10/0x10
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0xf0/0x110
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>
---

**Kernel Version:**
6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950

**Reproducibility:**
Happened multiple times during routine CPU-intensive operations. It
happens with various benchmarks (Stress_ng, Linpack) after several
hours of performance testing. `migration/*` kernel threads hit a NULL
dereference in `__migrate_swap_task`.

**System Info:**
- Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
- BIOS: M07 09/03/2021
- Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)

**Crash Cause (tentative):**
NULL dereference at offset `0x4c8` from a task struct pointer in
`__migrate_swap_task`. Possibly an uninitialized or freed
`task_struct` field.

Please let me know if you’d like me to test a patch or if you need
more details.

Thanks,
Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Aithal, Srikanth]

[-- Attachment #1: Type: text/plain, Size: 3431 bytes --]

Hi all,

I am hitting similar issue starting next-20250616 as well with 
next-20250617 
[https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git].

I have hit the issue on two of our CI machines [AMD EPYC Milan and AMD 
EPYC Genoa] while running normal as well as SEV guest boot tests.



I am attaching both run logs here.



Thanks and Regards,
Srikanth Aithal
Srikanth.Aithal@amd.com



On 6/18/2025 1:35 AM, Jirka Hladky wrote:
> Hi all,
> 
> I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
> involving a NULL pointer dereference in `__migrate_swap_task()` during
> CPU migration. This occurred on various AMD and Intel systems while
> running a CPU-intensive workload (Linpack, Stress_ng - it's not
> specific to a benchmark).
> 
> Full trace below:
> ---
> BUG: kernel NULL pointer dereference, address: 00000000000004c8
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 4078b99067 P4D 4078b99067 PUD 0
> Oops: Oops: 0000 [#1] SMP NOPTI
> CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
> 6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
> Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
> Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
> RIP: 0010:__migrate_swap_task+0x2f/0x170
> Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
> 48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
> bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
> RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
> RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
> RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
> RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
> R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
> R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
> FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
> PKRU: 55555554
> Call Trace:
>   <TASK>
>   migrate_swap_stop+0xe8/0x190
>   multi_cpu_stop+0xf3/0x130
>   ? __pfx_multi_cpu_stop+0x10/0x10
>   cpu_stopper_thread+0x97/0x140
>   ? __pfx_smpboot_thread_fn+0x10/0x10
>   smpboot_thread_fn+0xf3/0x220
>   kthread+0xfc/0x240
>   ? __pfx_kthread+0x10/0x10
>   ? __pfx_kthread+0x10/0x10
>   ret_from_fork+0xf0/0x110
>   ? __pfx_kthread+0x10/0x10
>   ret_from_fork_asm+0x1a/0x30
>   </TASK>
> ---
> 
> **Kernel Version:**
> 6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
> https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950
> 
> **Reproducibility:**
> Happened multiple times during routine CPU-intensive operations. It
> happens with various benchmarks (Stress_ng, Linpack) after several
> hours of performance testing. `migration/*` kernel threads hit a NULL
> dereference in `__migrate_swap_task`.
> 
> **System Info:**
> - Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
> - BIOS: M07 09/03/2021
> - Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)
> 
> **Crash Cause (tentative):**
> NULL dereference at offset `0x4c8` from a task struct pointer in
> `__migrate_swap_task`. Possibly an uninitialized or freed
> `task_struct` field.
> 
> Please let me know if you’d like me to test a patch or if you need
> more details.
> 
> Thanks,
> Jirka
> 

[-- Attachment #2: Genoa_log.tar.gz --]
[-- Type: application/gzip, Size: 91106 bytes --]

[-- Attachment #3: Milan_log.tar.gz --]
[-- Type: application/gzip, Size: 831857 bytes --]

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Abhigyan ghosh]

Hi Jirka,

Thanks for the detailed report.

I'm curious about the specific setup in which this panic was triggered. Could you share more about the exact configuration or parameters you used for running `stress-ng` or Linpack? For instance:

- How many threads/cores were used?
- Was it running inside a VM, container, or bare-metal?
- Was this under any thermal throttling or power-saving mode?

I'd like to try reproducing it locally to study the failure further.

Best regards,  
Abhigyan Ghosh

On 18 June 2025 1:35:30 am IST, Jirka Hladky <jhladky@redhat.com> wrote:
>Hi all,
>
>I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
>involving a NULL pointer dereference in `__migrate_swap_task()` during
>CPU migration. This occurred on various AMD and Intel systems while
>running a CPU-intensive workload (Linpack, Stress_ng - it's not
>specific to a benchmark).
>
>Full trace below:
>---
>BUG: kernel NULL pointer dereference, address: 00000000000004c8
>#PF: supervisor read access in kernel mode
>#PF: error_code(0x0000) - not-present page
>PGD 4078b99067 P4D 4078b99067 PUD 0
>Oops: Oops: 0000 [#1] SMP NOPTI
>CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
>6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
>Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
>Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
>RIP: 0010:__migrate_swap_task+0x2f/0x170
>Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
>48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
>bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
>RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
>RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
>RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
>RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
>R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
>R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
>FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
>CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
>PKRU: 55555554
>Call Trace:
> <TASK>
> migrate_swap_stop+0xe8/0x190
> multi_cpu_stop+0xf3/0x130
> ? __pfx_multi_cpu_stop+0x10/0x10
> cpu_stopper_thread+0x97/0x140
> ? __pfx_smpboot_thread_fn+0x10/0x10
> smpboot_thread_fn+0xf3/0x220
> kthread+0xfc/0x240
> ? __pfx_kthread+0x10/0x10
> ? __pfx_kthread+0x10/0x10
> ret_from_fork+0xf0/0x110
> ? __pfx_kthread+0x10/0x10
> ret_from_fork_asm+0x1a/0x30
> </TASK>
>---
>
>**Kernel Version:**
>6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
>https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950
>
>**Reproducibility:**
>Happened multiple times during routine CPU-intensive operations. It
>happens with various benchmarks (Stress_ng, Linpack) after several
>hours of performance testing. `migration/*` kernel threads hit a NULL
>dereference in `__migrate_swap_task`.
>
>**System Info:**
>- Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
>- BIOS: M07 09/03/2021
>- Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)
>
>**Crash Cause (tentative):**
>NULL dereference at offset `0x4c8` from a task struct pointer in
>`__migrate_swap_task`. Possibly an uninitialized or freed
>`task_struct` field.
>
>Please let me know if you’d like me to test a patch or if you need
>more details.
>
>Thanks,
>Jirka
>
>

aghosh

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi Abhigyan,

The testing is done on bare metal. The kernel panics occur after
several hours of benchmarking.

Out of 20 servers, the problem has occurred on 6 of them:
intel-sapphire-rapids-gold-6448y-2s
intel-emerald-rapids-platinum-8558-2s
amd-epyc5-turin-9655p-1s
amd-epyc4-zen4c-bergamo-9754-1s
amd-epyc3-milan-7713-2s
intel-skylake-2s

The number in the name is the CPU model. 1s: single socket, 2s: dual socket.

We were not able to find a clear pattern. It appears to be a race
condition of some kind.

We run various performance benchmarks, including Linpack, Stream, NAS
(https://www.nas.nasa.gov/software/npb.html), and Stress-ng. Testing
is conducted with various thread counts and settings. All benchmarks
together are running ~24 hours. One benchmark takes ~4 hours. Please
also note that we repeat the benchmarks to collect performance
statistics. In many cases, kernel panic has occurred when the
benchmark was repeated.

Crash occurred while running these tests:
Stress_ng: Starting test 'fork' (#29 out of 41), number of threads 32,
iteration 1 out of 5
SPECjbb2005: Starting DEFAULT run with 4 SPECJBB2005 instances, each
with 24 warehouses, iteration 2 out of 3
Stress_ng: test 'sem' (#30 out of 41), number of threads 24, iteration
2 out of 5
Stress_ng: test 'sem' (#30 out of 41), number of threads 64, iteration
4 out of 5
SPECjbb2005: SINGLE run with 1 SPECJBB2005 instances, each with 128
warehouses, iteration 2 out of 3
Linpack: Benchmark-utils/linpackd, iteration 3, testType affinityRun,
number of threads 128
NAS: NPB_sources/bin/is.D.x

There is no clear benchmark triggering the kernel panic. Looping
Stress_ng's sem test looks, however, like it's worth trying.

I hope this helps. Please let me know if there's anything I can help
with to pinpoint the problem.

Thanks
Jirka


On Wed, Jun 18, 2025 at 7:19 AM Abhigyan ghosh
<zscript.team.zs@gmail.com> wrote:
>
> Hi Jirka,
>
> Thanks for the detailed report.
>
> I'm curious about the specific setup in which this panic was triggered. Could you share more about the exact configuration or parameters you used for running `stress-ng` or Linpack? For instance:
>
> - How many threads/cores were used?
> - Was it running inside a VM, container, or bare-metal?
> - Was this under any thermal throttling or power-saving mode?
>
> I'd like to try reproducing it locally to study the failure further.
>
> Best regards,
> Abhigyan Ghosh
>
> On 18 June 2025 1:35:30 am IST, Jirka Hladky <jhladky@redhat.com> wrote:
> >Hi all,
> >
> >I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
> >involving a NULL pointer dereference in `__migrate_swap_task()` during
> >CPU migration. This occurred on various AMD and Intel systems while
> >running a CPU-intensive workload (Linpack, Stress_ng - it's not
> >specific to a benchmark).
> >
> >Full trace below:
> >---
> >BUG: kernel NULL pointer dereference, address: 00000000000004c8
> >#PF: supervisor read access in kernel mode
> >#PF: error_code(0x0000) - not-present page
> >PGD 4078b99067 P4D 4078b99067 PUD 0
> >Oops: Oops: 0000 [#1] SMP NOPTI
> >CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
> >6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
> >Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
> >Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
> >RIP: 0010:__migrate_swap_task+0x2f/0x170
> >Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
> >48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
> >bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
> >RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
> >RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
> >RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
> >RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
> >R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
> >R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
> >FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
> >CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
> >PKRU: 55555554
> >Call Trace:
> > <TASK>
> > migrate_swap_stop+0xe8/0x190
> > multi_cpu_stop+0xf3/0x130
> > ? __pfx_multi_cpu_stop+0x10/0x10
> > cpu_stopper_thread+0x97/0x140
> > ? __pfx_smpboot_thread_fn+0x10/0x10
> > smpboot_thread_fn+0xf3/0x220
> > kthread+0xfc/0x240
> > ? __pfx_kthread+0x10/0x10
> > ? __pfx_kthread+0x10/0x10
> > ret_from_fork+0xf0/0x110
> > ? __pfx_kthread+0x10/0x10
> > ret_from_fork_asm+0x1a/0x30
> > </TASK>
> >---
> >
> >**Kernel Version:**
> >6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
> >https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950
> >
> >**Reproducibility:**
> >Happened multiple times during routine CPU-intensive operations. It
> >happens with various benchmarks (Stress_ng, Linpack) after several
> >hours of performance testing. `migration/*` kernel threads hit a NULL
> >dereference in `__migrate_swap_task`.
> >
> >**System Info:**
> >- Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
> >- BIOS: M07 09/03/2021
> >- Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)
> >
> >**Crash Cause (tentative):**
> >NULL dereference at offset `0x4c8` from a task struct pointer in
> >`__migrate_swap_task`. Possibly an uninitialized or freed
> >`task_struct` field.
> >
> >Please let me know if you’d like me to test a patch or if you need
> >more details.
> >
> >Thanks,
> >Jirka
> >
> >
>
> aghosh
>


-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi Chen and all,

we have now verified that the following commit causes a kernel panic
discussed in this thread:

ad6b26b6a0a79 sched/numa: add statistics of numa balance task

Reverting this commit fixes the issue.

I'm happy to help debug this further or test a proposed fix.

Thank you!
Jirka

On Wed, Jun 18, 2025 at 1:34 PM Jirka Hladky <jhladky@redhat.com> wrote:
>
> Hi Abhigyan,
>
> The testing is done on bare metal. The kernel panics occur after
> several hours of benchmarking.
>
> Out of 20 servers, the problem has occurred on 6 of them:
> intel-sapphire-rapids-gold-6448y-2s
> intel-emerald-rapids-platinum-8558-2s
> amd-epyc5-turin-9655p-1s
> amd-epyc4-zen4c-bergamo-9754-1s
> amd-epyc3-milan-7713-2s
> intel-skylake-2s
>
> The number in the name is the CPU model. 1s: single socket, 2s: dual socket.
>
> We were not able to find a clear pattern. It appears to be a race
> condition of some kind.
>
> We run various performance benchmarks, including Linpack, Stream, NAS
> (https://www.nas.nasa.gov/software/npb.html), and Stress-ng. Testing
> is conducted with various thread counts and settings. All benchmarks
> together are running ~24 hours. One benchmark takes ~4 hours. Please
> also note that we repeat the benchmarks to collect performance
> statistics. In many cases, kernel panic has occurred when the
> benchmark was repeated.
>
> Crash occurred while running these tests:
> Stress_ng: Starting test 'fork' (#29 out of 41), number of threads 32,
> iteration 1 out of 5
> SPECjbb2005: Starting DEFAULT run with 4 SPECJBB2005 instances, each
> with 24 warehouses, iteration 2 out of 3
> Stress_ng: test 'sem' (#30 out of 41), number of threads 24, iteration
> 2 out of 5
> Stress_ng: test 'sem' (#30 out of 41), number of threads 64, iteration
> 4 out of 5
> SPECjbb2005: SINGLE run with 1 SPECJBB2005 instances, each with 128
> warehouses, iteration 2 out of 3
> Linpack: Benchmark-utils/linpackd, iteration 3, testType affinityRun,
> number of threads 128
> NAS: NPB_sources/bin/is.D.x
>
> There is no clear benchmark triggering the kernel panic. Looping
> Stress_ng's sem test looks, however, like it's worth trying.
>
> I hope this helps. Please let me know if there's anything I can help
> with to pinpoint the problem.
>
> Thanks
> Jirka
>
>
> On Wed, Jun 18, 2025 at 7:19 AM Abhigyan ghosh
> <zscript.team.zs@gmail.com> wrote:
> >
> > Hi Jirka,
> >
> > Thanks for the detailed report.
> >
> > I'm curious about the specific setup in which this panic was triggered. Could you share more about the exact configuration or parameters you used for running `stress-ng` or Linpack? For instance:
> >
> > - How many threads/cores were used?
> > - Was it running inside a VM, container, or bare-metal?
> > - Was this under any thermal throttling or power-saving mode?
> >
> > I'd like to try reproducing it locally to study the failure further.
> >
> > Best regards,
> > Abhigyan Ghosh
> >
> > On 18 June 2025 1:35:30 am IST, Jirka Hladky <jhladky@redhat.com> wrote:
> > >Hi all,
> > >
> > >I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
> > >involving a NULL pointer dereference in `__migrate_swap_task()` during
> > >CPU migration. This occurred on various AMD and Intel systems while
> > >running a CPU-intensive workload (Linpack, Stress_ng - it's not
> > >specific to a benchmark).
> > >
> > >Full trace below:
> > >---
> > >BUG: kernel NULL pointer dereference, address: 00000000000004c8
> > >#PF: supervisor read access in kernel mode
> > >#PF: error_code(0x0000) - not-present page
> > >PGD 4078b99067 P4D 4078b99067 PUD 0
> > >Oops: Oops: 0000 [#1] SMP NOPTI
> > >CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
> > >6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
> > >Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
> > >Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
> > >RIP: 0010:__migrate_swap_task+0x2f/0x170
> > >Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
> > >48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
> > >bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
> > >RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
> > >RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
> > >RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
> > >RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
> > >R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
> > >R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
> > >FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
> > >CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > >CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
> > >PKRU: 55555554
> > >Call Trace:
> > > <TASK>
> > > migrate_swap_stop+0xe8/0x190
> > > multi_cpu_stop+0xf3/0x130
> > > ? __pfx_multi_cpu_stop+0x10/0x10
> > > cpu_stopper_thread+0x97/0x140
> > > ? __pfx_smpboot_thread_fn+0x10/0x10
> > > smpboot_thread_fn+0xf3/0x220
> > > kthread+0xfc/0x240
> > > ? __pfx_kthread+0x10/0x10
> > > ? __pfx_kthread+0x10/0x10
> > > ret_from_fork+0xf0/0x110
> > > ? __pfx_kthread+0x10/0x10
> > > ret_from_fork_asm+0x1a/0x30
> > > </TASK>
> > >---
> > >
> > >**Kernel Version:**
> > >6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
> > >https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950
> > >
> > >**Reproducibility:**
> > >Happened multiple times during routine CPU-intensive operations. It
> > >happens with various benchmarks (Stress_ng, Linpack) after several
> > >hours of performance testing. `migration/*` kernel threads hit a NULL
> > >dereference in `__migrate_swap_task`.
> > >
> > >**System Info:**
> > >- Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
> > >- BIOS: M07 09/03/2021
> > >- Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)
> > >
> > >**Crash Cause (tentative):**
> > >NULL dereference at offset `0x4c8` from a task struct pointer in
> > >`__migrate_swap_task`. Possibly an uninitialized or freed
> > >`task_struct` field.
> > >
> > >Please let me know if you’d like me to test a patch or if you need
> > >more details.
> > >
> > >Thanks,
> > >Jirka
> > >
> > >
> >
> > aghosh
> >
>
>
> --
> -Jirka



-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Chen, Yu C]

Hi Jirka,

On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> Hi Chen and all,
> 
> we have now verified that the following commit causes a kernel panic
> discussed in this thread:
> 
> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> 
> Reverting this commit fixes the issue.
> 
> I'm happy to help debug this further or test a proposed fix.
> 

Thanks very much for your report, it seems that there is a
race condition that when the swap task candidate was chosen,
but its mm_struct get released due to task exit, then later
when doing the task swaping, the p->mm is NULL which caused
the problem:

CPU0                                   CPU1
:
...
task_numa_migrate
   task_numa_find_cpu
    task_numa_compare
      # a normal task p is chosen
      env->best_task = p

                                        # p exit:
                                        exit_signals(p);
                                           p->flags |= PF_EXITING
                                        exit_mm
                                           p->mm = NULL;

    migrate_swap_stop
      __migrate_swap_task((arg->src_task, arg->dst_cpu)
       count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL

Could you please help check if the following debug patch works,
and if there is no issue found after you ran several tests,
could you please provide the
/sys/kernel/debug/tracing/trace

BTW, is it possible to share your test script for stress-ng,
stream? It looks like the stress-ng's fork test case would
trigger this issue easier in theory.

thanks,
Chenyu


> Thank you!
> Jirka
> 
> On Wed, Jun 18, 2025 at 1:34 PM Jirka Hladky <jhladky@redhat.com> wrote:
>>
>> Hi Abhigyan,
>>
>> The testing is done on bare metal. The kernel panics occur after
>> several hours of benchmarking.
>>
>> Out of 20 servers, the problem has occurred on 6 of them:
>> intel-sapphire-rapids-gold-6448y-2s
>> intel-emerald-rapids-platinum-8558-2s
>> amd-epyc5-turin-9655p-1s
>> amd-epyc4-zen4c-bergamo-9754-1s
>> amd-epyc3-milan-7713-2s
>> intel-skylake-2s
>>
>> The number in the name is the CPU model. 1s: single socket, 2s: dual socket.
>>
>> We were not able to find a clear pattern. It appears to be a race
>> condition of some kind.
>>
>> We run various performance benchmarks, including Linpack, Stream, NAS
>> (https://www.nas.nasa.gov/software/npb.html), and Stress-ng. Testing
>> is conducted with various thread counts and settings. All benchmarks
>> together are running ~24 hours. One benchmark takes ~4 hours. Please
>> also note that we repeat the benchmarks to collect performance
>> statistics. In many cases, kernel panic has occurred when the
>> benchmark was repeated.
>>
>> Crash occurred while running these tests:
>> Stress_ng: Starting test 'fork' (#29 out of 41), number of threads 32,
>> iteration 1 out of 5
>> SPECjbb2005: Starting DEFAULT run with 4 SPECJBB2005 instances, each
>> with 24 warehouses, iteration 2 out of 3
>> Stress_ng: test 'sem' (#30 out of 41), number of threads 24, iteration
>> 2 out of 5
>> Stress_ng: test 'sem' (#30 out of 41), number of threads 64, iteration
>> 4 out of 5
>> SPECjbb2005: SINGLE run with 1 SPECJBB2005 instances, each with 128
>> warehouses, iteration 2 out of 3
>> Linpack: Benchmark-utils/linpackd, iteration 3, testType affinityRun,
>> number of threads 128
>> NAS: NPB_sources/bin/is.D.x
>>
>> There is no clear benchmark triggering the kernel panic. Looping
>> Stress_ng's sem test looks, however, like it's worth trying.
>>
>> I hope this helps. Please let me know if there's anything I can help
>> with to pinpoint the problem.
>>
>> Thanks
>> Jirka
>>
>>
>> On Wed, Jun 18, 2025 at 7:19 AM Abhigyan ghosh
>> <zscript.team.zs@gmail.com> wrote:
>>>
>>> Hi Jirka,
>>>
>>> Thanks for the detailed report.
>>>
>>> I'm curious about the specific setup in which this panic was triggered. Could you share more about the exact configuration or parameters you used for running `stress-ng` or Linpack? For instance:
>>>
>>> - How many threads/cores were used?
>>> - Was it running inside a VM, container, or bare-metal?
>>> - Was this under any thermal throttling or power-saving mode?
>>>
>>> I'd like to try reproducing it locally to study the failure further.
>>>
>>> Best regards,
>>> Abhigyan Ghosh
>>>
>>> On 18 June 2025 1:35:30 am IST, Jirka Hladky <jhladky@redhat.com> wrote:
>>>> Hi all,
>>>>
>>>> I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
>>>> involving a NULL pointer dereference in `__migrate_swap_task()` during
>>>> CPU migration. This occurred on various AMD and Intel systems while
>>>> running a CPU-intensive workload (Linpack, Stress_ng - it's not
>>>> specific to a benchmark).
>>>>
>>>> Full trace below:
>>>> ---
>>>> BUG: kernel NULL pointer dereference, address: 00000000000004c8
>>>> #PF: supervisor read access in kernel mode
>>>> #PF: error_code(0x0000) - not-present page
>>>> PGD 4078b99067 P4D 4078b99067 PUD 0
>>>> Oops: Oops: 0000 [#1] SMP NOPTI
>>>> CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
>>>> 6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
>>>> Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
>>>> Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
>>>> RIP: 0010:__migrate_swap_task+0x2f/0x170
>>>> Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
>>>> 48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
>>>> bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
>>>> RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
>>>> RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
>>>> RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
>>>> RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
>>>> R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
>>>> R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
>>>> FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
>>>> PKRU: 55555554
>>>> Call Trace:
>>>> <TASK>
>>>> migrate_swap_stop+0xe8/0x190
>>>> multi_cpu_stop+0xf3/0x130
>>>> ? __pfx_multi_cpu_stop+0x10/0x10
>>>> cpu_stopper_thread+0x97/0x140
>>>> ? __pfx_smpboot_thread_fn+0x10/0x10
>>>> smpboot_thread_fn+0xf3/0x220
>>>> kthread+0xfc/0x240
>>>> ? __pfx_kthread+0x10/0x10
>>>> ? __pfx_kthread+0x10/0x10
>>>> ret_from_fork+0xf0/0x110
>>>> ? __pfx_kthread+0x10/0x10
>>>> ret_from_fork_asm+0x1a/0x30
>>>> </TASK>
>>>> ---
>>>>
>>>> **Kernel Version:**
>>>> 6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
>>>> https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950
>>>>
>>>> **Reproducibility:**
>>>> Happened multiple times during routine CPU-intensive operations. It
>>>> happens with various benchmarks (Stress_ng, Linpack) after several
>>>> hours of performance testing. `migration/*` kernel threads hit a NULL
>>>> dereference in `__migrate_swap_task`.
>>>>
>>>> **System Info:**
>>>> - Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
>>>> - BIOS: M07 09/03/2021
>>>> - Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)
>>>>
>>>> **Crash Cause (tentative):**
>>>> NULL dereference at offset `0x4c8` from a task struct pointer in
>>>> `__migrate_swap_task`. Possibly an uninitialized or freed
>>>> `task_struct` field.
>>>>
>>>> Please let me know if you’d like me to test a patch or if you need
>>>> more details.
>>>>
>>>> Thanks,
>>>> Jirka
>>>>
>>>>
>>>
>>> aghosh
>>>
>>
>>
>> --
>> -Jirka
> 
> 
> 

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Chen, Yu C]

On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> Hi Jirka,
> 
> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>> Hi Chen and all,
>>
>> we have now verified that the following commit causes a kernel panic
>> discussed in this thread:
>>
>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>
>> Reverting this commit fixes the issue.
>>
>> I'm happy to help debug this further or test a proposed fix.
>>
> 
> Thanks very much for your report, it seems that there is a
> race condition that when the swap task candidate was chosen,
> but its mm_struct get released due to task exit, then later
> when doing the task swaping, the p->mm is NULL which caused
> the problem:
> 
> CPU0                                   CPU1
> :
> ...
> task_numa_migrate
>    task_numa_find_cpu
>     task_numa_compare
>       # a normal task p is chosen
>       env->best_task = p
> 
>                                         # p exit:
>                                         exit_signals(p);
>                                            p->flags |= PF_EXITING
>                                         exit_mm
>                                            p->mm = NULL;
> 
>     migrate_swap_stop
>       __migrate_swap_task((arg->src_task, arg->dst_cpu)
>        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> 
> Could you please help check if the following debug patch works,

Attached the patch:

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8988d38d46a3..82fc966b390c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct 
task_struct *p, int cpu)
  {
  	__schedstat_inc(p->stats.numa_task_swapped);
  	count_vm_numa_event(NUMA_TASK_SWAP);
-	count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
+	if (unlikely(!p->mm)) {
+		trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
+				p->flags);
+	} else {
+		count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
+	}

  	if (task_on_rq_queued(p)) {
  		struct rq *src_rq, *dst_rq;
-- 
2.25.1

> and if there is no issue found after you ran several tests,
> could you please provide the
> /sys/kernel/debug/tracing/trace
> 
> BTW, is it possible to share your test script for stress-ng,
> stream? It looks like the stress-ng's fork test case would
> trigger this issue easier in theory.
> 
> thanks,
> Chenyu
> 
> 

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Libo Chen]

Hi Chenyu,

On 6/27/25 00:33, Chen, Yu C wrote:
> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>> Hi Jirka,
>>
>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>> Hi Chen and all,
>>>
>>> we have now verified that the following commit causes a kernel panic
>>> discussed in this thread:
>>>
>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>
>>> Reverting this commit fixes the issue.
>>>
>>> I'm happy to help debug this further or test a proposed fix.
>>>
>>
>> Thanks very much for your report, it seems that there is a
>> race condition that when the swap task candidate was chosen,
>> but its mm_struct get released due to task exit, then later
>> when doing the task swaping, the p->mm is NULL which caused
>> the problem:
>>
>> CPU0                                   CPU1
>> :
>> ...
>> task_numa_migrate
>>    task_numa_find_cpu
>>     task_numa_compare
>>       # a normal task p is chosen
>>       env->best_task = p
>>
>>                                         # p exit:
>>                                         exit_signals(p);
>>                                            p->flags |= PF_EXITING
>>                                         exit_mm
>>                                            p->mm = NULL;
>>
>>     migrate_swap_stop
>>       __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>
>> Could you please help check if the following debug patch works,
> 
> Attached the patch:
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 8988d38d46a3..82fc966b390c 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
>  {
>      __schedstat_inc(p->stats.numa_task_swapped);
>      count_vm_numa_event(NUMA_TASK_SWAP);
> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> +    if (unlikely(!p->mm)) {

I am starting to wonder if we should keep this check and add a big fat warning
like  "there is a bug here, please report it!" rather than brick the kernel.
A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
so it's a tradeoff I guess. What do you think?


Thanks,
Libo   

> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> +                p->flags);
> +    } else {
> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> +    }
> 
>      if (task_on_rq_queued(p)) {
>          struct rq *src_rq, *dst_rq;

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Chen, Yu C]

Hi Libo,

On 7/1/2025 3:32 PM, Libo Chen wrote:
> Hi Chenyu,
> 
> On 6/27/25 00:33, Chen, Yu C wrote:
>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>>> Hi Jirka,
>>>
>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>>> Hi Chen and all,
>>>>
>>>> we have now verified that the following commit causes a kernel panic
>>>> discussed in this thread:
>>>>
>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>>
>>>> Reverting this commit fixes the issue.
>>>>
>>>> I'm happy to help debug this further or test a proposed fix.
>>>>
>>>
>>> Thanks very much for your report, it seems that there is a
>>> race condition that when the swap task candidate was chosen,
>>> but its mm_struct get released due to task exit, then later
>>> when doing the task swaping, the p->mm is NULL which caused
>>> the problem:
>>>
>>> CPU0                                   CPU1
>>> :
>>> ...
>>> task_numa_migrate
>>>     task_numa_find_cpu
>>>      task_numa_compare
>>>        # a normal task p is chosen
>>>        env->best_task = p
>>>
>>>                                          # p exit:
>>>                                          exit_signals(p);
>>>                                             p->flags |= PF_EXITING
>>>                                          exit_mm
>>>                                             p->mm = NULL;
>>>
>>>      migrate_swap_stop
>>>        __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>>         count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>>
>>> Could you please help check if the following debug patch works,
>>
>> Attached the patch:
>>
>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>> index 8988d38d46a3..82fc966b390c 100644
>> --- a/kernel/sched/core.c
>> +++ b/kernel/sched/core.c
>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
>>   {
>>       __schedstat_inc(p->stats.numa_task_swapped);
>>       count_vm_numa_event(NUMA_TASK_SWAP);
>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>> +    if (unlikely(!p->mm)) {
> 
> I am starting to wonder if we should keep this check and add a big fat warning
> like  "there is a bug here, please report it!" rather than brick the kernel.
> A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
> so it's a tradeoff I guess. What do you think?
> 

I agree with you that adding something like WARN_ON_ONCE()
could help, and it seems that the kernel panic is not expected
in this scenario because this feature is a statistic calculation
rather than the critical logic. Maybe waiting for Jirka's feedback
to decide the next step.

thanks,
Chenyu



> 
> Thanks,
> Libo
> 
>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
>> +                p->flags);
>> +    } else {
>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>> +    }
>>
>>       if (task_on_rq_queued(p)) {
>>           struct rq *src_rq, *dst_rq;
> 

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi Chenyu and Libo,

I agree, let's wait for the results. Hopefully, we can fix the root
cause based on the debug messages.

There was a problem with the infrastructure over the weekend, so I had
to restart the jobs yesterday. I should have results tomorrow.

Stay tuned.
Jirka


On Tue, Jul 1, 2025 at 4:59 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
>
> Hi Libo,
>
> On 7/1/2025 3:32 PM, Libo Chen wrote:
> > Hi Chenyu,
> >
> > On 6/27/25 00:33, Chen, Yu C wrote:
> >> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> >>> Hi Jirka,
> >>>
> >>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> >>>> Hi Chen and all,
> >>>>
> >>>> we have now verified that the following commit causes a kernel panic
> >>>> discussed in this thread:
> >>>>
> >>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> >>>>
> >>>> Reverting this commit fixes the issue.
> >>>>
> >>>> I'm happy to help debug this further or test a proposed fix.
> >>>>
> >>>
> >>> Thanks very much for your report, it seems that there is a
> >>> race condition that when the swap task candidate was chosen,
> >>> but its mm_struct get released due to task exit, then later
> >>> when doing the task swaping, the p->mm is NULL which caused
> >>> the problem:
> >>>
> >>> CPU0                                   CPU1
> >>> :
> >>> ...
> >>> task_numa_migrate
> >>>     task_numa_find_cpu
> >>>      task_numa_compare
> >>>        # a normal task p is chosen
> >>>        env->best_task = p
> >>>
> >>>                                          # p exit:
> >>>                                          exit_signals(p);
> >>>                                             p->flags |= PF_EXITING
> >>>                                          exit_mm
> >>>                                             p->mm = NULL;
> >>>
> >>>      migrate_swap_stop
> >>>        __migrate_swap_task((arg->src_task, arg->dst_cpu)
> >>>         count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> >>>
> >>> Could you please help check if the following debug patch works,
> >>
> >> Attached the patch:
> >>
> >> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >> index 8988d38d46a3..82fc966b390c 100644
> >> --- a/kernel/sched/core.c
> >> +++ b/kernel/sched/core.c
> >> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
> >>   {
> >>       __schedstat_inc(p->stats.numa_task_swapped);
> >>       count_vm_numa_event(NUMA_TASK_SWAP);
> >> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >> +    if (unlikely(!p->mm)) {
> >
> > I am starting to wonder if we should keep this check and add a big fat warning
> > like  "there is a bug here, please report it!" rather than brick the kernel.
> > A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
> > so it's a tradeoff I guess. What do you think?
> >
>
> I agree with you that adding something like WARN_ON_ONCE()
> could help, and it seems that the kernel panic is not expected
> in this scenario because this feature is a statistic calculation
> rather than the critical logic. Maybe waiting for Jirka's feedback
> to decide the next step.
>
> thanks,
> Chenyu
>
>
>
> >
> > Thanks,
> > Libo
> >
> >> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> >> +                p->flags);
> >> +    } else {
> >> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >> +    }
> >>
> >>       if (task_on_rq_queued(p)) {
> >>           struct rq *src_rq, *dst_rq;
> >
>


-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi Chenyu,

I have the first

if (unlikely(!p->mm)) {
trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm, p->flags);

log from dual socket AMD Milanx 7573X server:

[74615.841401] # entries-in-buffer/entries-written: 1/1   #P:128
[74615.841401] #
[74615.841401] #                                _-----=> irqs-off/BH-disabled
[74615.841401] #                               / _----=> need-resched
[74615.841401] #                              | / _---=> hardirq/softirq
[74615.841401] #                              || / _--=> preempt-depth
[74615.841401] #                              ||| / _-=> migrate-disable
[74615.841401] #                              |||| /     delay
[74615.841401] #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
[74615.841401] #              | |         |   |||||     |         |
[74615.841401]      migration/6-54      [006] d..5. 58590.368364:
__migrate_swap_task: !! (2939199 stress-ng-sem) flags=40004c

debug code caught a task with p->mm == NULL during __migrate_swap_task().

Trace shows:
  * It was a stress-ng-sem process (PID 2939199),
  * That process was in the process of exiting (PF_EXITING | PF_EXITPIDONE),
  * And was a workqueue worker (PF_WQ_WORKER),
  * It explains why p->mm had already been freed

I'm curious as to why the command is stress-ng-sem, and the flag
indicates it is a workqueue worker.

More jobs are still running. I will keep an eye on them and report the
other cases as they finish.

Jirka

On Tue, Jul 1, 2025 at 5:10 PM Jirka Hladky <jhladky@redhat.com> wrote:
>
> Hi Chenyu and Libo,
>
> I agree, let's wait for the results. Hopefully, we can fix the root
> cause based on the debug messages.
>
> There was a problem with the infrastructure over the weekend, so I had
> to restart the jobs yesterday. I should have results tomorrow.
>
> Stay tuned.
> Jirka
>
>
> On Tue, Jul 1, 2025 at 4:59 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
> >
> > Hi Libo,
> >
> > On 7/1/2025 3:32 PM, Libo Chen wrote:
> > > Hi Chenyu,
> > >
> > > On 6/27/25 00:33, Chen, Yu C wrote:
> > >> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> > >>> Hi Jirka,
> > >>>
> > >>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> > >>>> Hi Chen and all,
> > >>>>
> > >>>> we have now verified that the following commit causes a kernel panic
> > >>>> discussed in this thread:
> > >>>>
> > >>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> > >>>>
> > >>>> Reverting this commit fixes the issue.
> > >>>>
> > >>>> I'm happy to help debug this further or test a proposed fix.
> > >>>>
> > >>>
> > >>> Thanks very much for your report, it seems that there is a
> > >>> race condition that when the swap task candidate was chosen,
> > >>> but its mm_struct get released due to task exit, then later
> > >>> when doing the task swaping, the p->mm is NULL which caused
> > >>> the problem:
> > >>>
> > >>> CPU0                                   CPU1
> > >>> :
> > >>> ...
> > >>> task_numa_migrate
> > >>>     task_numa_find_cpu
> > >>>      task_numa_compare
> > >>>        # a normal task p is chosen
> > >>>        env->best_task = p
> > >>>
> > >>>                                          # p exit:
> > >>>                                          exit_signals(p);
> > >>>                                             p->flags |= PF_EXITING
> > >>>                                          exit_mm
> > >>>                                             p->mm = NULL;
> > >>>
> > >>>      migrate_swap_stop
> > >>>        __migrate_swap_task((arg->src_task, arg->dst_cpu)
> > >>>         count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> > >>>
> > >>> Could you please help check if the following debug patch works,
> > >>
> > >> Attached the patch:
> > >>
> > >> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > >> index 8988d38d46a3..82fc966b390c 100644
> > >> --- a/kernel/sched/core.c
> > >> +++ b/kernel/sched/core.c
> > >> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
> > >>   {
> > >>       __schedstat_inc(p->stats.numa_task_swapped);
> > >>       count_vm_numa_event(NUMA_TASK_SWAP);
> > >> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >> +    if (unlikely(!p->mm)) {
> > >
> > > I am starting to wonder if we should keep this check and add a big fat warning
> > > like  "there is a bug here, please report it!" rather than brick the kernel.
> > > A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
> > > so it's a tradeoff I guess. What do you think?
> > >
> >
> > I agree with you that adding something like WARN_ON_ONCE()
> > could help, and it seems that the kernel panic is not expected
> > in this scenario because this feature is a statistic calculation
> > rather than the critical logic. Maybe waiting for Jirka's feedback
> > to decide the next step.
> >
> > thanks,
> > Chenyu
> >
> >
> >
> > >
> > > Thanks,
> > > Libo
> > >
> > >> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> > >> +                p->flags);
> > >> +    } else {
> > >> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >> +    }
> > >>
> > >>       if (task_on_rq_queued(p)) {
> > >>           struct rq *src_rq, *dst_rq;
> > >
> >
>
>
> --
> -Jirka



-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Chen, Yu C]

On 7/2/2025 5:05 AM, Jirka Hladky wrote:
> Hi Chenyu,
> 
> I have the first
> 
> if (unlikely(!p->mm)) {
> trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm, p->flags);
> 
> log from dual socket AMD Milanx 7573X server:
> 
> [74615.841401] # entries-in-buffer/entries-written: 1/1   #P:128
> [74615.841401] #
> [74615.841401] #                                _-----=> irqs-off/BH-disabled
> [74615.841401] #                               / _----=> need-resched
> [74615.841401] #                              | / _---=> hardirq/softirq
> [74615.841401] #                              || / _--=> preempt-depth
> [74615.841401] #                              ||| / _-=> migrate-disable
> [74615.841401] #                              |||| /     delay
> [74615.841401] #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
> [74615.841401] #              | |         |   |||||     |         |
> [74615.841401]      migration/6-54      [006] d..5. 58590.368364:
> __migrate_swap_task: !! (2939199 stress-ng-sem) flags=40004c
> 
> debug code caught a task with p->mm == NULL during __migrate_swap_task().
> 
> Trace shows:
>    * It was a stress-ng-sem process (PID 2939199),
>    * That process was in the process of exiting (PF_EXITING | PF_EXITPIDONE),

Thanks Jirka. So this race condition is related to task exit.

>    * And was a workqueue worker (PF_WQ_WORKER),
>    * It explains why p->mm had already been freed
> 
> I'm curious as to why the command is stress-ng-sem, and the flag
> indicates it is a workqueue worker.

The flag 0x40004c is
PF_EXITING PF_POSTCOREDUMP PF_FORKNOEXEC PF_RANDOMIZE
according to latest
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/sched.h?h=v6.16-rc4

thanks,
Chenyu>
> More jobs are still running. I will keep an eye on them and report the
> other cases as they finish.
> 
> Jirka
> 
> On Tue, Jul 1, 2025 at 5:10 PM Jirka Hladky <jhladky@redhat.com> wrote:
>>
>> Hi Chenyu and Libo,
>>
>> I agree, let's wait for the results. Hopefully, we can fix the root
>> cause based on the debug messages.
>>
>> There was a problem with the infrastructure over the weekend, so I had
>> to restart the jobs yesterday. I should have results tomorrow.
>>
>> Stay tuned.
>> Jirka
>>
>>
>> On Tue, Jul 1, 2025 at 4:59 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
>>>
>>> Hi Libo,
>>>
>>> On 7/1/2025 3:32 PM, Libo Chen wrote:
>>>> Hi Chenyu,
>>>>
>>>> On 6/27/25 00:33, Chen, Yu C wrote:
>>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>>>>>> Hi Jirka,
>>>>>>
>>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>>>>>> Hi Chen and all,
>>>>>>>
>>>>>>> we have now verified that the following commit causes a kernel panic
>>>>>>> discussed in this thread:
>>>>>>>
>>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>>>>>
>>>>>>> Reverting this commit fixes the issue.
>>>>>>>
>>>>>>> I'm happy to help debug this further or test a proposed fix.
>>>>>>>
>>>>>>
>>>>>> Thanks very much for your report, it seems that there is a
>>>>>> race condition that when the swap task candidate was chosen,
>>>>>> but its mm_struct get released due to task exit, then later
>>>>>> when doing the task swaping, the p->mm is NULL which caused
>>>>>> the problem:
>>>>>>
>>>>>> CPU0                                   CPU1
>>>>>> :
>>>>>> ...
>>>>>> task_numa_migrate
>>>>>>      task_numa_find_cpu
>>>>>>       task_numa_compare
>>>>>>         # a normal task p is chosen
>>>>>>         env->best_task = p
>>>>>>
>>>>>>                                           # p exit:
>>>>>>                                           exit_signals(p);
>>>>>>                                              p->flags |= PF_EXITING
>>>>>>                                           exit_mm
>>>>>>                                              p->mm = NULL;
>>>>>>
>>>>>>       migrate_swap_stop
>>>>>>         __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>>>>>          count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>>>>>
>>>>>> Could you please help check if the following debug patch works,
>>>>>
>>>>> Attached the patch:
>>>>>
>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>> index 8988d38d46a3..82fc966b390c 100644
>>>>> --- a/kernel/sched/core.c
>>>>> +++ b/kernel/sched/core.c
>>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
>>>>>    {
>>>>>        __schedstat_inc(p->stats.numa_task_swapped);
>>>>>        count_vm_numa_event(NUMA_TASK_SWAP);
>>>>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>>>> +    if (unlikely(!p->mm)) {
>>>>
>>>> I am starting to wonder if we should keep this check and add a big fat warning
>>>> like  "there is a bug here, please report it!" rather than brick the kernel.
>>>> A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
>>>> so it's a tradeoff I guess. What do you think?
>>>>
>>>
>>> I agree with you that adding something like WARN_ON_ONCE()
>>> could help, and it seems that the kernel panic is not expected
>>> in this scenario because this feature is a statistic calculation
>>> rather than the critical logic. Maybe waiting for Jirka's feedback
>>> to decide the next step.
>>>
>>> thanks,
>>> Chenyu
>>>
>>>
>>>
>>>>
>>>> Thanks,
>>>> Libo
>>>>
>>>>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
>>>>> +                p->flags);
>>>>> +    } else {
>>>>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>>>> +    }
>>>>>
>>>>>        if (task_on_rq_queued(p)) {
>>>>>            struct rq *src_rq, *dst_rq;
>>>>
>>>
>>
>>
>> --
>> -Jirka
> 
> 
> 

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

> Thanks Jirka. So this race condition is related to task exit.

Yes! I'm glad we've found it! :-)

> The flag 0x40004c is
> PF_EXITING PF_POSTCOREDUMP PF_FORKNOEXEC PF_RANDOMIZE
> according to latest
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/sched.h?h=v6.16-rc4

Oh, my bad! Thank you for the correct pointer.


On Wed, Jul 2, 2025 at 1:14 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
>
> On 7/2/2025 5:05 AM, Jirka Hladky wrote:
> > Hi Chenyu,
> >
> > I have the first
> >
> > if (unlikely(!p->mm)) {
> > trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm, p->flags);
> >
> > log from dual socket AMD Milanx 7573X server:
> >
> > [74615.841401] # entries-in-buffer/entries-written: 1/1   #P:128
> > [74615.841401] #
> > [74615.841401] #                                _-----=> irqs-off/BH-disabled
> > [74615.841401] #                               / _----=> need-resched
> > [74615.841401] #                              | / _---=> hardirq/softirq
> > [74615.841401] #                              || / _--=> preempt-depth
> > [74615.841401] #                              ||| / _-=> migrate-disable
> > [74615.841401] #                              |||| /     delay
> > [74615.841401] #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
> > [74615.841401] #              | |         |   |||||     |         |
> > [74615.841401]      migration/6-54      [006] d..5. 58590.368364:
> > __migrate_swap_task: !! (2939199 stress-ng-sem) flags=40004c
> >
> > debug code caught a task with p->mm == NULL during __migrate_swap_task().
> >
> > Trace shows:
> >    * It was a stress-ng-sem process (PID 2939199),
> >    * That process was in the process of exiting (PF_EXITING | PF_EXITPIDONE),
>
> Thanks Jirka. So this race condition is related to task exit.
>
> >    * And was a workqueue worker (PF_WQ_WORKER),
> >    * It explains why p->mm had already been freed
> >
> > I'm curious as to why the command is stress-ng-sem, and the flag
> > indicates it is a workqueue worker.
>
> The flag 0x40004c is
> PF_EXITING PF_POSTCOREDUMP PF_FORKNOEXEC PF_RANDOMIZE
> according to latest
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/sched.h?h=v6.16-rc4
>
> thanks,
> Chenyu>
> > More jobs are still running. I will keep an eye on them and report the
> > other cases as they finish.
> >
> > Jirka
> >
> > On Tue, Jul 1, 2025 at 5:10 PM Jirka Hladky <jhladky@redhat.com> wrote:
> >>
> >> Hi Chenyu and Libo,
> >>
> >> I agree, let's wait for the results. Hopefully, we can fix the root
> >> cause based on the debug messages.
> >>
> >> There was a problem with the infrastructure over the weekend, so I had
> >> to restart the jobs yesterday. I should have results tomorrow.
> >>
> >> Stay tuned.
> >> Jirka
> >>
> >>
> >> On Tue, Jul 1, 2025 at 4:59 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
> >>>
> >>> Hi Libo,
> >>>
> >>> On 7/1/2025 3:32 PM, Libo Chen wrote:
> >>>> Hi Chenyu,
> >>>>
> >>>> On 6/27/25 00:33, Chen, Yu C wrote:
> >>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> >>>>>> Hi Jirka,
> >>>>>>
> >>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> >>>>>>> Hi Chen and all,
> >>>>>>>
> >>>>>>> we have now verified that the following commit causes a kernel panic
> >>>>>>> discussed in this thread:
> >>>>>>>
> >>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> >>>>>>>
> >>>>>>> Reverting this commit fixes the issue.
> >>>>>>>
> >>>>>>> I'm happy to help debug this further or test a proposed fix.
> >>>>>>>
> >>>>>>
> >>>>>> Thanks very much for your report, it seems that there is a
> >>>>>> race condition that when the swap task candidate was chosen,
> >>>>>> but its mm_struct get released due to task exit, then later
> >>>>>> when doing the task swaping, the p->mm is NULL which caused
> >>>>>> the problem:
> >>>>>>
> >>>>>> CPU0                                   CPU1
> >>>>>> :
> >>>>>> ...
> >>>>>> task_numa_migrate
> >>>>>>      task_numa_find_cpu
> >>>>>>       task_numa_compare
> >>>>>>         # a normal task p is chosen
> >>>>>>         env->best_task = p
> >>>>>>
> >>>>>>                                           # p exit:
> >>>>>>                                           exit_signals(p);
> >>>>>>                                              p->flags |= PF_EXITING
> >>>>>>                                           exit_mm
> >>>>>>                                              p->mm = NULL;
> >>>>>>
> >>>>>>       migrate_swap_stop
> >>>>>>         __migrate_swap_task((arg->src_task, arg->dst_cpu)
> >>>>>>          count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> >>>>>>
> >>>>>> Could you please help check if the following debug patch works,
> >>>>>
> >>>>> Attached the patch:
> >>>>>
> >>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>>>> index 8988d38d46a3..82fc966b390c 100644
> >>>>> --- a/kernel/sched/core.c
> >>>>> +++ b/kernel/sched/core.c
> >>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
> >>>>>    {
> >>>>>        __schedstat_inc(p->stats.numa_task_swapped);
> >>>>>        count_vm_numa_event(NUMA_TASK_SWAP);
> >>>>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>>>> +    if (unlikely(!p->mm)) {
> >>>>
> >>>> I am starting to wonder if we should keep this check and add a big fat warning
> >>>> like  "there is a bug here, please report it!" rather than brick the kernel.
> >>>> A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
> >>>> so it's a tradeoff I guess. What do you think?
> >>>>
> >>>
> >>> I agree with you that adding something like WARN_ON_ONCE()
> >>> could help, and it seems that the kernel panic is not expected
> >>> in this scenario because this feature is a statistic calculation
> >>> rather than the critical logic. Maybe waiting for Jirka's feedback
> >>> to decide the next step.
> >>>
> >>> thanks,
> >>> Chenyu
> >>>
> >>>
> >>>
> >>>>
> >>>> Thanks,
> >>>> Libo
> >>>>
> >>>>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> >>>>> +                p->flags);
> >>>>> +    } else {
> >>>>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>>>> +    }
> >>>>>
> >>>>>        if (task_on_rq_queued(p)) {
> >>>>>            struct rq *src_rq, *dst_rq;
> >>>>
> >>>
> >>
> >>
> >> --
> >> -Jirka
> >
> >
> >
>


-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Aithal, Srikanth]

On 6/27/2025 1:03 PM, Chen, Yu C wrote:
> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>> Hi Jirka,
>>
>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>> Hi Chen and all,
>>>
>>> we have now verified that the following commit causes a kernel panic
>>> discussed in this thread:
>>>
>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>
>>> Reverting this commit fixes the issue.
>>>
>>> I'm happy to help debug this further or test a proposed fix.
>>>
>>
>> Thanks very much for your report, it seems that there is a
>> race condition that when the swap task candidate was chosen,
>> but its mm_struct get released due to task exit, then later
>> when doing the task swaping, the p->mm is NULL which caused
>> the problem:
>>
>> CPU0                                   CPU1
>> :
>> ...
>> task_numa_migrate
>>    task_numa_find_cpu
>>     task_numa_compare
>>       # a normal task p is chosen
>>       env->best_task = p
>>
>>                                         # p exit:
>>                                         exit_signals(p);
>>                                            p->flags |= PF_EXITING
>>                                         exit_mm
>>                                            p->mm = NULL;
>>
>>     migrate_swap_stop
>>       __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>
>> Could you please help check if the following debug patch works,
> 
> Attached the patch:
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 8988d38d46a3..82fc966b390c 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct 
> task_struct *p, int cpu)
>   {
>       __schedstat_inc(p->stats.numa_task_swapped);
>       count_vm_numa_event(NUMA_TASK_SWAP);
> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> +    if (unlikely(!p->mm)) {
> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> +                p->flags);
> +    } else {
> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> +    }
> 
>       if (task_on_rq_queued(p)) {
>           struct rq *src_rq, *dst_rq;

I was encountering the same issue as mentioned earlier in this thread, 
which has been recurring in our daily linux-next CI builds within our 
virtualization CI stream where we observed this BUG appearing randomly 
during the runs.

Additionally, we were able to reproduce this issue while running the 
autonuma benchmark. As mentioned earlier, the BUG would occur randomly 
across iterations, typically between the 5th and 10th iterations.

We consistently encountered this issue up to the 
6.16.0-rc4-next-20250630 build 
[https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git].

After applying the aforementioned patch ontop of next-20250630 build, I 
tested it in our virtualization CI and with the autonuma benchmark 
reproducer described below, and the issue no longer occurred. The patch 
appears to have resolved the reported problem.


git clone https://github.com/pholasek/autonuma-benchmark.git
cd autonuma-benchmark
for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
Note: The server running the autonuma-benchmark must have at least two 
nodes.

If the provided fix is final, please feel free to include the following 
Tested-by tag:

Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
Tested-by: Suneeth D <Suneeth.D@amd.com>

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Aithal, Srikanth]

On 7/2/2025 10:02 AM, Aithal, Srikanth wrote:
> On 6/27/2025 1:03 PM, Chen, Yu C wrote:
>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>>> Hi Jirka,
>>>
>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>>> Hi Chen and all,
>>>>
>>>> we have now verified that the following commit causes a kernel panic
>>>> discussed in this thread:
>>>>
>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>>
>>>> Reverting this commit fixes the issue.
>>>>
>>>> I'm happy to help debug this further or test a proposed fix.
>>>>
>>>
>>> Thanks very much for your report, it seems that there is a
>>> race condition that when the swap task candidate was chosen,
>>> but its mm_struct get released due to task exit, then later
>>> when doing the task swaping, the p->mm is NULL which caused
>>> the problem:
>>>
>>> CPU0                                   CPU1
>>> :
>>> ...
>>> task_numa_migrate
>>>    task_numa_find_cpu
>>>     task_numa_compare
>>>       # a normal task p is chosen
>>>       env->best_task = p
>>>
>>>                                         # p exit:
>>>                                         exit_signals(p);
>>>                                            p->flags |= PF_EXITING
>>>                                         exit_mm
>>>                                            p->mm = NULL;
>>>
>>>     migrate_swap_stop
>>>       __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>>        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>>
>>> Could you please help check if the following debug patch works,
>>
>> Attached the patch:
>>
>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>> index 8988d38d46a3..82fc966b390c 100644
>> --- a/kernel/sched/core.c
>> +++ b/kernel/sched/core.c
>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct 
>> task_struct *p, int cpu)
>>   {
>>       __schedstat_inc(p->stats.numa_task_swapped);
>>       count_vm_numa_event(NUMA_TASK_SWAP);
>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>> +    if (unlikely(!p->mm)) {
>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
>> +                p->flags);
>> +    } else {
>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>> +    }
>>
>>       if (task_on_rq_queued(p)) {
>>           struct rq *src_rq, *dst_rq;
> 
> I was encountering the same issue as mentioned earlier in this thread, 
> which has been recurring in our daily linux-next CI builds within our 
> virtualization CI stream where we observed this BUG appearing randomly 
> during the runs.
> 
> Additionally, we were able to reproduce this issue while running the 
> autonuma benchmark. As mentioned earlier, the BUG would occur randomly 
> across iterations, typically between the 5th and 10th iterations.
> 
> We consistently encountered this issue up to the 6.16.0-rc4- 
> next-20250630 build [https://git.kernel.org/pub/scm/linux/kernel/git/ 
> next/linux-next.git].
> 
> After applying the aforementioned patch ontop of next-20250630 build, I 
> tested it in our virtualization CI and with the autonuma benchmark 
> reproducer described below, and the issue no longer occurred. The patch 
> appears to have resolved the reported problem.
> 
> 
> git clone https://github.com/pholasek/autonuma-benchmark.git
> cd autonuma-benchmark
> for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
> Note: The server running the autonuma-benchmark must have at least two 
> nodes.
> 
> If the provided fix is final, please feel free to include the following 
> Tested-by tag:
> 
> Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
> Tested-by: Suneeth D <Suneeth.D@amd.com>
> 


I apologize for overlooking that this was a debug patch. With this debug 
patch applied on top of 6.16.0-rc4-next-20250630, I am unable to 
reproduce the issue. However, when I revert this debug patch, I 
encounter the issue again on 6.16.0-rc4-next-20250630.

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Chen, Yu C]

On 7/2/2025 3:36 PM, Aithal, Srikanth wrote:
> 
> 
> On 7/2/2025 10:02 AM, Aithal, Srikanth wrote:
>> On 6/27/2025 1:03 PM, Chen, Yu C wrote:
>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>>>> Hi Jirka,
>>>>
>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>>>> Hi Chen and all,
>>>>>
>>>>> we have now verified that the following commit causes a kernel panic
>>>>> discussed in this thread:
>>>>>
>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>>>
>>>>> Reverting this commit fixes the issue.
>>>>>
>>>>> I'm happy to help debug this further or test a proposed fix.
>>>>>
>>>>
>>>> Thanks very much for your report, it seems that there is a
>>>> race condition that when the swap task candidate was chosen,
>>>> but its mm_struct get released due to task exit, then later
>>>> when doing the task swaping, the p->mm is NULL which caused
>>>> the problem:
>>>>
>>>> CPU0                                   CPU1
>>>> :
>>>> ...
>>>> task_numa_migrate
>>>>    task_numa_find_cpu
>>>>     task_numa_compare
>>>>       # a normal task p is chosen
>>>>       env->best_task = p
>>>>
>>>>                                         # p exit:
>>>>                                         exit_signals(p);
>>>>                                            p->flags |= PF_EXITING
>>>>                                         exit_mm
>>>>                                            p->mm = NULL;
>>>>
>>>>     migrate_swap_stop
>>>>       __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>>>        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>>>
>>>> Could you please help check if the following debug patch works,
>>>
>>> Attached the patch:
>>>
>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>> index 8988d38d46a3..82fc966b390c 100644
>>> --- a/kernel/sched/core.c
>>> +++ b/kernel/sched/core.c
>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct 
>>> task_struct *p, int cpu)
>>>   {
>>>       __schedstat_inc(p->stats.numa_task_swapped);
>>>       count_vm_numa_event(NUMA_TASK_SWAP);
>>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>> +    if (unlikely(!p->mm)) {
>>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
>>> +                p->flags);
>>> +    } else {
>>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>> +    }
>>>
>>>       if (task_on_rq_queued(p)) {
>>>           struct rq *src_rq, *dst_rq;
>>
>> I was encountering the same issue as mentioned earlier in this thread, 
>> which has been recurring in our daily linux-next CI builds within our 
>> virtualization CI stream where we observed this BUG appearing randomly 
>> during the runs.
>>
>> Additionally, we were able to reproduce this issue while running the 
>> autonuma benchmark. As mentioned earlier, the BUG would occur randomly 
>> across iterations, typically between the 5th and 10th iterations.
>>
>> We consistently encountered this issue up to the 6.16.0-rc4- 
>> next-20250630 build [https://git.kernel.org/pub/scm/linux/kernel/git/ 
>> next/linux-next.git].
>>
>> After applying the aforementioned patch ontop of next-20250630 build, 
>> I tested it in our virtualization CI and with the autonuma benchmark 
>> reproducer described below, and the issue no longer occurred. The 
>> patch appears to have resolved the reported problem.
>>
>>
>> git clone https://github.com/pholasek/autonuma-benchmark.git
>> cd autonuma-benchmark
>> for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
>> Note: The server running the autonuma-benchmark must have at least two 
>> nodes.
>>
>> If the provided fix is final, please feel free to include the 
>> following Tested-by tag:
>>
>> Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
>> Tested-by: Suneeth D <Suneeth.D@amd.com>
>>
> 
> 
> I apologize for overlooking that this was a debug patch. With this debug 
> patch applied on top of 6.16.0-rc4-next-20250630, I am unable to 
> reproduce the issue. However, when I revert this debug patch, I 
> encounter the issue again on 6.16.0-rc4-next-20250630.
> 
> 

Thank you Aithal, this is a debug + fix patch. And Jirka has confirmed
that this helps somehow. I'll post a formal one and Cc you.

Thanks,
Chenyu

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi Aithal,

thank you for the reproducer.

With the patched kernel, after running a reproducer, please do

cat /sys/kernel/debug/tracing/trace

If the counter is nonzero like this:
entries-in-buffer/entries-written: 1/1

please post the `cat /sys/kernel/debug/tracing/trace` here.

It would be helpful to check if there are any other race conditions
besides the task exit.

Thank you!
Jirka

On Wed, Jul 2, 2025 at 1:17 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
>
> On 7/2/2025 3:36 PM, Aithal, Srikanth wrote:
> >
> >
> > On 7/2/2025 10:02 AM, Aithal, Srikanth wrote:
> >> On 6/27/2025 1:03 PM, Chen, Yu C wrote:
> >>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> >>>> Hi Jirka,
> >>>>
> >>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> >>>>> Hi Chen and all,
> >>>>>
> >>>>> we have now verified that the following commit causes a kernel panic
> >>>>> discussed in this thread:
> >>>>>
> >>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> >>>>>
> >>>>> Reverting this commit fixes the issue.
> >>>>>
> >>>>> I'm happy to help debug this further or test a proposed fix.
> >>>>>
> >>>>
> >>>> Thanks very much for your report, it seems that there is a
> >>>> race condition that when the swap task candidate was chosen,
> >>>> but its mm_struct get released due to task exit, then later
> >>>> when doing the task swaping, the p->mm is NULL which caused
> >>>> the problem:
> >>>>
> >>>> CPU0                                   CPU1
> >>>> :
> >>>> ...
> >>>> task_numa_migrate
> >>>>    task_numa_find_cpu
> >>>>     task_numa_compare
> >>>>       # a normal task p is chosen
> >>>>       env->best_task = p
> >>>>
> >>>>                                         # p exit:
> >>>>                                         exit_signals(p);
> >>>>                                            p->flags |= PF_EXITING
> >>>>                                         exit_mm
> >>>>                                            p->mm = NULL;
> >>>>
> >>>>     migrate_swap_stop
> >>>>       __migrate_swap_task((arg->src_task, arg->dst_cpu)
> >>>>        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> >>>>
> >>>> Could you please help check if the following debug patch works,
> >>>
> >>> Attached the patch:
> >>>
> >>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>> index 8988d38d46a3..82fc966b390c 100644
> >>> --- a/kernel/sched/core.c
> >>> +++ b/kernel/sched/core.c
> >>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct
> >>> task_struct *p, int cpu)
> >>>   {
> >>>       __schedstat_inc(p->stats.numa_task_swapped);
> >>>       count_vm_numa_event(NUMA_TASK_SWAP);
> >>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>> +    if (unlikely(!p->mm)) {
> >>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> >>> +                p->flags);
> >>> +    } else {
> >>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>> +    }
> >>>
> >>>       if (task_on_rq_queued(p)) {
> >>>           struct rq *src_rq, *dst_rq;
> >>
> >> I was encountering the same issue as mentioned earlier in this thread,
> >> which has been recurring in our daily linux-next CI builds within our
> >> virtualization CI stream where we observed this BUG appearing randomly
> >> during the runs.
> >>
> >> Additionally, we were able to reproduce this issue while running the
> >> autonuma benchmark. As mentioned earlier, the BUG would occur randomly
> >> across iterations, typically between the 5th and 10th iterations.
> >>
> >> We consistently encountered this issue up to the 6.16.0-rc4-
> >> next-20250630 build [https://git.kernel.org/pub/scm/linux/kernel/git/
> >> next/linux-next.git].
> >>
> >> After applying the aforementioned patch ontop of next-20250630 build,
> >> I tested it in our virtualization CI and with the autonuma benchmark
> >> reproducer described below, and the issue no longer occurred. The
> >> patch appears to have resolved the reported problem.
> >>
> >>
> >> git clone https://github.com/pholasek/autonuma-benchmark.git
> >> cd autonuma-benchmark
> >> for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
> >> Note: The server running the autonuma-benchmark must have at least two
> >> nodes.
> >>
> >> If the provided fix is final, please feel free to include the
> >> following Tested-by tag:
> >>
> >> Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
> >> Tested-by: Suneeth D <Suneeth.D@amd.com>
> >>
> >
> >
> > I apologize for overlooking that this was a debug patch. With this debug
> > patch applied on top of 6.16.0-rc4-next-20250630, I am unable to
> > reproduce the issue. However, when I revert this debug patch, I
> > encounter the issue again on 6.16.0-rc4-next-20250630.
> >
> >
>
> Thank you Aithal, this is a debug + fix patch. And Jirka has confirmed
> that this helps somehow. I'll post a formal one and Cc you.
>
> Thanks,
> Chenyu
>


-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Aithal, Srikanth]

On 7/2/2025 5:23 PM, Jirka Hladky wrote:
> Hi Aithal,
> 
> thank you for the reproducer.
> 
> With the patched kernel, after running a reproducer, please do
> 
> cat /sys/kernel/debug/tracing/trace
> 
> If the counter is nonzero like this:
> entries-in-buffer/entries-written: 1/1
> 
> please post the `cat /sys/kernel/debug/tracing/trace` here.
> 
> It would be helpful to check if there are any other race conditions
> besides the task exit.
> 
> Thank you!
> Jirka
> 


None of the four machines running our workload [with the debug patch] 
showed any entries.
Entries in Buffer/Entries Written: 0/0


> On Wed, Jul 2, 2025 at 1:17 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
>>
>> On 7/2/2025 3:36 PM, Aithal, Srikanth wrote:
>>>
>>>
>>> On 7/2/2025 10:02 AM, Aithal, Srikanth wrote:
>>>> On 6/27/2025 1:03 PM, Chen, Yu C wrote:
>>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>>>>>> Hi Jirka,
>>>>>>
>>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>>>>>> Hi Chen and all,
>>>>>>>
>>>>>>> we have now verified that the following commit causes a kernel panic
>>>>>>> discussed in this thread:
>>>>>>>
>>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>>>>>
>>>>>>> Reverting this commit fixes the issue.
>>>>>>>
>>>>>>> I'm happy to help debug this further or test a proposed fix.
>>>>>>>
>>>>>>
>>>>>> Thanks very much for your report, it seems that there is a
>>>>>> race condition that when the swap task candidate was chosen,
>>>>>> but its mm_struct get released due to task exit, then later
>>>>>> when doing the task swaping, the p->mm is NULL which caused
>>>>>> the problem:
>>>>>>
>>>>>> CPU0                                   CPU1
>>>>>> :
>>>>>> ...
>>>>>> task_numa_migrate
>>>>>>     task_numa_find_cpu
>>>>>>      task_numa_compare
>>>>>>        # a normal task p is chosen
>>>>>>        env->best_task = p
>>>>>>
>>>>>>                                          # p exit:
>>>>>>                                          exit_signals(p);
>>>>>>                                             p->flags |= PF_EXITING
>>>>>>                                          exit_mm
>>>>>>                                             p->mm = NULL;
>>>>>>
>>>>>>      migrate_swap_stop
>>>>>>        __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>>>>>         count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>>>>>
>>>>>> Could you please help check if the following debug patch works,
>>>>>
>>>>> Attached the patch:
>>>>>
>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>> index 8988d38d46a3..82fc966b390c 100644
>>>>> --- a/kernel/sched/core.c
>>>>> +++ b/kernel/sched/core.c
>>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct
>>>>> task_struct *p, int cpu)
>>>>>    {
>>>>>        __schedstat_inc(p->stats.numa_task_swapped);
>>>>>        count_vm_numa_event(NUMA_TASK_SWAP);
>>>>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>>>> +    if (unlikely(!p->mm)) {
>>>>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
>>>>> +                p->flags);
>>>>> +    } else {
>>>>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>>>> +    }
>>>>>
>>>>>        if (task_on_rq_queued(p)) {
>>>>>            struct rq *src_rq, *dst_rq;
>>>>
>>>> I was encountering the same issue as mentioned earlier in this thread,
>>>> which has been recurring in our daily linux-next CI builds within our
>>>> virtualization CI stream where we observed this BUG appearing randomly
>>>> during the runs.
>>>>
>>>> Additionally, we were able to reproduce this issue while running the
>>>> autonuma benchmark. As mentioned earlier, the BUG would occur randomly
>>>> across iterations, typically between the 5th and 10th iterations.
>>>>
>>>> We consistently encountered this issue up to the 6.16.0-rc4-
>>>> next-20250630 build [https://git.kernel.org/pub/scm/linux/kernel/git/
>>>> next/linux-next.git].
>>>>
>>>> After applying the aforementioned patch ontop of next-20250630 build,
>>>> I tested it in our virtualization CI and with the autonuma benchmark
>>>> reproducer described below, and the issue no longer occurred. The
>>>> patch appears to have resolved the reported problem.
>>>>
>>>>
>>>> git clone https://github.com/pholasek/autonuma-benchmark.git
>>>> cd autonuma-benchmark
>>>> for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
>>>> Note: The server running the autonuma-benchmark must have at least two
>>>> nodes.
>>>>
>>>> If the provided fix is final, please feel free to include the
>>>> following Tested-by tag:
>>>>
>>>> Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
>>>> Tested-by: Suneeth D <Suneeth.D@amd.com>
>>>>
>>>
>>>
>>> I apologize for overlooking that this was a debug patch. With this debug
>>> patch applied on top of 6.16.0-rc4-next-20250630, I am unable to
>>> reproduce the issue. However, when I revert this debug patch, I
>>> encounter the issue again on 6.16.0-rc4-next-20250630.
>>>
>>>
>>
>> Thank you Aithal, this is a debug + fix patch. And Jirka has confirmed
>> that this helps somehow. I'll post a formal one and Cc you.
>>
>> Thanks,
>> Chenyu
>>
> 
> 

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Thank you, Aithal!

Chenyu, here is another debug log - the flag is essentially the same
as before, except that PF_NOFREEZE is also set, so it points to the
same race condition (task exit).

This happened on an AMD Bergamo 9754 single socket server.

cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 1/1   #P:256
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
   migration/96-599     [096] dN.5. 84281.647548: __migrate_swap_task:
!! (2886754 sysbench-binary) flags=40804c

On Wed, Jul 2, 2025 at 2:19 PM Aithal, Srikanth <sraithal@amd.com> wrote:
>
> On 7/2/2025 5:23 PM, Jirka Hladky wrote:
> > Hi Aithal,
> >
> > thank you for the reproducer.
> >
> > With the patched kernel, after running a reproducer, please do
> >
> > cat /sys/kernel/debug/tracing/trace
> >
> > If the counter is nonzero like this:
> > entries-in-buffer/entries-written: 1/1
> >
> > please post the `cat /sys/kernel/debug/tracing/trace` here.
> >
> > It would be helpful to check if there are any other race conditions
> > besides the task exit.
> >
> > Thank you!
> > Jirka
> >
>
>
> None of the four machines running our workload [with the debug patch]
> showed any entries.
> Entries in Buffer/Entries Written: 0/0
>
>
> > On Wed, Jul 2, 2025 at 1:17 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
> >>
> >> On 7/2/2025 3:36 PM, Aithal, Srikanth wrote:
> >>>
> >>>
> >>> On 7/2/2025 10:02 AM, Aithal, Srikanth wrote:
> >>>> On 6/27/2025 1:03 PM, Chen, Yu C wrote:
> >>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> >>>>>> Hi Jirka,
> >>>>>>
> >>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> >>>>>>> Hi Chen and all,
> >>>>>>>
> >>>>>>> we have now verified that the following commit causes a kernel panic
> >>>>>>> discussed in this thread:
> >>>>>>>
> >>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> >>>>>>>
> >>>>>>> Reverting this commit fixes the issue.
> >>>>>>>
> >>>>>>> I'm happy to help debug this further or test a proposed fix.
> >>>>>>>
> >>>>>>
> >>>>>> Thanks very much for your report, it seems that there is a
> >>>>>> race condition that when the swap task candidate was chosen,
> >>>>>> but its mm_struct get released due to task exit, then later
> >>>>>> when doing the task swaping, the p->mm is NULL which caused
> >>>>>> the problem:
> >>>>>>
> >>>>>> CPU0                                   CPU1
> >>>>>> :
> >>>>>> ...
> >>>>>> task_numa_migrate
> >>>>>>     task_numa_find_cpu
> >>>>>>      task_numa_compare
> >>>>>>        # a normal task p is chosen
> >>>>>>        env->best_task = p
> >>>>>>
> >>>>>>                                          # p exit:
> >>>>>>                                          exit_signals(p);
> >>>>>>                                             p->flags |= PF_EXITING
> >>>>>>                                          exit_mm
> >>>>>>                                             p->mm = NULL;
> >>>>>>
> >>>>>>      migrate_swap_stop
> >>>>>>        __migrate_swap_task((arg->src_task, arg->dst_cpu)
> >>>>>>         count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> >>>>>>
> >>>>>> Could you please help check if the following debug patch works,
> >>>>>
> >>>>> Attached the patch:
> >>>>>
> >>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>>>> index 8988d38d46a3..82fc966b390c 100644
> >>>>> --- a/kernel/sched/core.c
> >>>>> +++ b/kernel/sched/core.c
> >>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct
> >>>>> task_struct *p, int cpu)
> >>>>>    {
> >>>>>        __schedstat_inc(p->stats.numa_task_swapped);
> >>>>>        count_vm_numa_event(NUMA_TASK_SWAP);
> >>>>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>>>> +    if (unlikely(!p->mm)) {
> >>>>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> >>>>> +                p->flags);
> >>>>> +    } else {
> >>>>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>>>> +    }
> >>>>>
> >>>>>        if (task_on_rq_queued(p)) {
> >>>>>            struct rq *src_rq, *dst_rq;
> >>>>
> >>>> I was encountering the same issue as mentioned earlier in this thread,
> >>>> which has been recurring in our daily linux-next CI builds within our
> >>>> virtualization CI stream where we observed this BUG appearing randomly
> >>>> during the runs.
> >>>>
> >>>> Additionally, we were able to reproduce this issue while running the
> >>>> autonuma benchmark. As mentioned earlier, the BUG would occur randomly
> >>>> across iterations, typically between the 5th and 10th iterations.
> >>>>
> >>>> We consistently encountered this issue up to the 6.16.0-rc4-
> >>>> next-20250630 build [https://git.kernel.org/pub/scm/linux/kernel/git/
> >>>> next/linux-next.git].
> >>>>
> >>>> After applying the aforementioned patch ontop of next-20250630 build,
> >>>> I tested it in our virtualization CI and with the autonuma benchmark
> >>>> reproducer described below, and the issue no longer occurred. The
> >>>> patch appears to have resolved the reported problem.
> >>>>
> >>>>
> >>>> git clone https://github.com/pholasek/autonuma-benchmark.git
> >>>> cd autonuma-benchmark
> >>>> for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
> >>>> Note: The server running the autonuma-benchmark must have at least two
> >>>> nodes.
> >>>>
> >>>> If the provided fix is final, please feel free to include the
> >>>> following Tested-by tag:
> >>>>
> >>>> Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
> >>>> Tested-by: Suneeth D <Suneeth.D@amd.com>
> >>>>
> >>>
> >>>
> >>> I apologize for overlooking that this was a debug patch. With this debug
> >>> patch applied on top of 6.16.0-rc4-next-20250630, I am unable to
> >>> reproduce the issue. However, when I revert this debug patch, I
> >>> encounter the issue again on 6.16.0-rc4-next-20250630.
> >>>
> >>>
> >>
> >> Thank you Aithal, this is a debug + fix patch. And Jirka has confirmed
> >> that this helps somehow. I'll post a formal one and Cc you.
> >>
> >> Thanks,
> >> Chenyu
> >>
> >
> >
>


-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Let me close this loop.

All debug messages I have collected were connected with the task exit.

Thank you
Jirka


On Wed, Jul 2, 2025 at 3:24 PM Jirka Hladky <jhladky@redhat.com> wrote:
>
> Thank you, Aithal!
>
> Chenyu, here is another debug log - the flag is essentially the same
> as before, except that PF_NOFREEZE is also set, so it points to the
> same race condition (task exit).
>
> This happened on an AMD Bergamo 9754 single socket server.
>
> cat /sys/kernel/debug/tracing/trace
> # tracer: nop
> #
> # entries-in-buffer/entries-written: 1/1   #P:256
> #
> #                                _-----=> irqs-off/BH-disabled
> #                               / _----=> need-resched
> #                              | / _---=> hardirq/softirq
> #                              || / _--=> preempt-depth
> #                              ||| / _-=> migrate-disable
> #                              |||| /     delay
> #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
> #              | |         |   |||||     |         |
>    migration/96-599     [096] dN.5. 84281.647548: __migrate_swap_task:
> !! (2886754 sysbench-binary) flags=40804c
>
> On Wed, Jul 2, 2025 at 2:19 PM Aithal, Srikanth <sraithal@amd.com> wrote:
> >
> > On 7/2/2025 5:23 PM, Jirka Hladky wrote:
> > > Hi Aithal,
> > >
> > > thank you for the reproducer.
> > >
> > > With the patched kernel, after running a reproducer, please do
> > >
> > > cat /sys/kernel/debug/tracing/trace
> > >
> > > If the counter is nonzero like this:
> > > entries-in-buffer/entries-written: 1/1
> > >
> > > please post the `cat /sys/kernel/debug/tracing/trace` here.
> > >
> > > It would be helpful to check if there are any other race conditions
> > > besides the task exit.
> > >
> > > Thank you!
> > > Jirka
> > >
> >
> >
> > None of the four machines running our workload [with the debug patch]
> > showed any entries.
> > Entries in Buffer/Entries Written: 0/0
> >
> >
> > > On Wed, Jul 2, 2025 at 1:17 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
> > >>
> > >> On 7/2/2025 3:36 PM, Aithal, Srikanth wrote:
> > >>>
> > >>>
> > >>> On 7/2/2025 10:02 AM, Aithal, Srikanth wrote:
> > >>>> On 6/27/2025 1:03 PM, Chen, Yu C wrote:
> > >>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> > >>>>>> Hi Jirka,
> > >>>>>>
> > >>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> > >>>>>>> Hi Chen and all,
> > >>>>>>>
> > >>>>>>> we have now verified that the following commit causes a kernel panic
> > >>>>>>> discussed in this thread:
> > >>>>>>>
> > >>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> > >>>>>>>
> > >>>>>>> Reverting this commit fixes the issue.
> > >>>>>>>
> > >>>>>>> I'm happy to help debug this further or test a proposed fix.
> > >>>>>>>
> > >>>>>>
> > >>>>>> Thanks very much for your report, it seems that there is a
> > >>>>>> race condition that when the swap task candidate was chosen,
> > >>>>>> but its mm_struct get released due to task exit, then later
> > >>>>>> when doing the task swaping, the p->mm is NULL which caused
> > >>>>>> the problem:
> > >>>>>>
> > >>>>>> CPU0                                   CPU1
> > >>>>>> :
> > >>>>>> ...
> > >>>>>> task_numa_migrate
> > >>>>>>     task_numa_find_cpu
> > >>>>>>      task_numa_compare
> > >>>>>>        # a normal task p is chosen
> > >>>>>>        env->best_task = p
> > >>>>>>
> > >>>>>>                                          # p exit:
> > >>>>>>                                          exit_signals(p);
> > >>>>>>                                             p->flags |= PF_EXITING
> > >>>>>>                                          exit_mm
> > >>>>>>                                             p->mm = NULL;
> > >>>>>>
> > >>>>>>      migrate_swap_stop
> > >>>>>>        __migrate_swap_task((arg->src_task, arg->dst_cpu)
> > >>>>>>         count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> > >>>>>>
> > >>>>>> Could you please help check if the following debug patch works,
> > >>>>>
> > >>>>> Attached the patch:
> > >>>>>
> > >>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > >>>>> index 8988d38d46a3..82fc966b390c 100644
> > >>>>> --- a/kernel/sched/core.c
> > >>>>> +++ b/kernel/sched/core.c
> > >>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct
> > >>>>> task_struct *p, int cpu)
> > >>>>>    {
> > >>>>>        __schedstat_inc(p->stats.numa_task_swapped);
> > >>>>>        count_vm_numa_event(NUMA_TASK_SWAP);
> > >>>>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >>>>> +    if (unlikely(!p->mm)) {
> > >>>>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> > >>>>> +                p->flags);
> > >>>>> +    } else {
> > >>>>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >>>>> +    }
> > >>>>>
> > >>>>>        if (task_on_rq_queued(p)) {
> > >>>>>            struct rq *src_rq, *dst_rq;
> > >>>>
> > >>>> I was encountering the same issue as mentioned earlier in this thread,
> > >>>> which has been recurring in our daily linux-next CI builds within our
> > >>>> virtualization CI stream where we observed this BUG appearing randomly
> > >>>> during the runs.
> > >>>>
> > >>>> Additionally, we were able to reproduce this issue while running the
> > >>>> autonuma benchmark. As mentioned earlier, the BUG would occur randomly
> > >>>> across iterations, typically between the 5th and 10th iterations.
> > >>>>
> > >>>> We consistently encountered this issue up to the 6.16.0-rc4-
> > >>>> next-20250630 build [https://git.kernel.org/pub/scm/linux/kernel/git/
> > >>>> next/linux-next.git].
> > >>>>
> > >>>> After applying the aforementioned patch ontop of next-20250630 build,
> > >>>> I tested it in our virtualization CI and with the autonuma benchmark
> > >>>> reproducer described below, and the issue no longer occurred. The
> > >>>> patch appears to have resolved the reported problem.
> > >>>>
> > >>>>
> > >>>> git clone https://github.com/pholasek/autonuma-benchmark.git
> > >>>> cd autonuma-benchmark
> > >>>> for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
> > >>>> Note: The server running the autonuma-benchmark must have at least two
> > >>>> nodes.
> > >>>>
> > >>>> If the provided fix is final, please feel free to include the
> > >>>> following Tested-by tag:
> > >>>>
> > >>>> Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
> > >>>> Tested-by: Suneeth D <Suneeth.D@amd.com>
> > >>>>
> > >>>
> > >>>
> > >>> I apologize for overlooking that this was a debug patch. With this debug
> > >>> patch applied on top of 6.16.0-rc4-next-20250630, I am unable to
> > >>> reproduce the issue. However, when I revert this debug patch, I
> > >>> encounter the issue again on 6.16.0-rc4-next-20250630.
> > >>>
> > >>>
> > >>
> > >> Thank you Aithal, this is a debug + fix patch. And Jirka has confirmed
> > >> that this helps somehow. I'll post a formal one and Cc you.
> > >>
> > >> Thanks,
> > >> Chenyu
> > >>
> > >
> > >
> >
>
>
> --
> -Jirka



-- 
-Jirka

Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi Chenyu,

thank you for the patch! I will test it and get back to you next week.

The issue is quite challenging to reproduce, and it indeed points to a
race condition. I see different benchmarks running when hitting the
kernel panic.

As for stress_ng, we run the subtests like this:

sync; sync; echo 3 > /proc/sys/vm/drop_caches
./stress-ng --fork 24 --verbose --oomable --metrics-brief -t 23 --yaml
$(uname -r)_fork.yaml | tee $(uname -r)_fork.log

We vary the number of threads (in the example above, 24 threads) up to
the maximum number of available CPUs and repeat the tests several
times to record the runtime statistics and variations.

Try to run the test on several servers in parallel to increase the
chances of hitting the problem in a reasonable time.

Thank you
Jirka

On Fri, Jun 27, 2025 at 9:16 AM Chen, Yu C <yu.c.chen@intel.com> wrote:
>
> Hi Jirka,
>
> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> > Hi Chen and all,
> >
> > we have now verified that the following commit causes a kernel panic
> > discussed in this thread:
> >
> > ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> >
> > Reverting this commit fixes the issue.
> >
> > I'm happy to help debug this further or test a proposed fix.
> >
>
> Thanks very much for your report, it seems that there is a
> race condition that when the swap task candidate was chosen,
> but its mm_struct get released due to task exit, then later
> when doing the task swaping, the p->mm is NULL which caused
> the problem:
>
> CPU0                                   CPU1
> :
> ...
> task_numa_migrate
>    task_numa_find_cpu
>     task_numa_compare
>       # a normal task p is chosen
>       env->best_task = p
>
>                                         # p exit:
>                                         exit_signals(p);
>                                            p->flags |= PF_EXITING
>                                         exit_mm
>                                            p->mm = NULL;
>
>     migrate_swap_stop
>       __migrate_swap_task((arg->src_task, arg->dst_cpu)
>        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>
> Could you please help check if the following debug patch works,
> and if there is no issue found after you ran several tests,
> could you please provide the
> /sys/kernel/debug/tracing/trace
>
> BTW, is it possible to share your test script for stress-ng,
> stream? It looks like the stress-ng's fork test case would
> trigger this issue easier in theory.
>
> thanks,
> Chenyu
>
>
> > Thank you!
> > Jirka
> >
> > On Wed, Jun 18, 2025 at 1:34 PM Jirka Hladky <jhladky@redhat.com> wrote:
> >>
> >> Hi Abhigyan,
> >>
> >> The testing is done on bare metal. The kernel panics occur after
> >> several hours of benchmarking.
> >>
> >> Out of 20 servers, the problem has occurred on 6 of them:
> >> intel-sapphire-rapids-gold-6448y-2s
> >> intel-emerald-rapids-platinum-8558-2s
> >> amd-epyc5-turin-9655p-1s
> >> amd-epyc4-zen4c-bergamo-9754-1s
> >> amd-epyc3-milan-7713-2s
> >> intel-skylake-2s
> >>
> >> The number in the name is the CPU model. 1s: single socket, 2s: dual socket.
> >>
> >> We were not able to find a clear pattern. It appears to be a race
> >> condition of some kind.
> >>
> >> We run various performance benchmarks, including Linpack, Stream, NAS
> >> (https://www.nas.nasa.gov/software/npb.html), and Stress-ng. Testing
> >> is conducted with various thread counts and settings. All benchmarks
> >> together are running ~24 hours. One benchmark takes ~4 hours. Please
> >> also note that we repeat the benchmarks to collect performance
> >> statistics. In many cases, kernel panic has occurred when the
> >> benchmark was repeated.
> >>
> >> Crash occurred while running these tests:
> >> Stress_ng: Starting test 'fork' (#29 out of 41), number of threads 32,
> >> iteration 1 out of 5
> >> SPECjbb2005: Starting DEFAULT run with 4 SPECJBB2005 instances, each
> >> with 24 warehouses, iteration 2 out of 3
> >> Stress_ng: test 'sem' (#30 out of 41), number of threads 24, iteration
> >> 2 out of 5
> >> Stress_ng: test 'sem' (#30 out of 41), number of threads 64, iteration
> >> 4 out of 5
> >> SPECjbb2005: SINGLE run with 1 SPECJBB2005 instances, each with 128
> >> warehouses, iteration 2 out of 3
> >> Linpack: Benchmark-utils/linpackd, iteration 3, testType affinityRun,
> >> number of threads 128
> >> NAS: NPB_sources/bin/is.D.x
> >>
> >> There is no clear benchmark triggering the kernel panic. Looping
> >> Stress_ng's sem test looks, however, like it's worth trying.
> >>
> >> I hope this helps. Please let me know if there's anything I can help
> >> with to pinpoint the problem.
> >>
> >> Thanks
> >> Jirka
> >>
> >>
> >> On Wed, Jun 18, 2025 at 7:19 AM Abhigyan ghosh
> >> <zscript.team.zs@gmail.com> wrote:
> >>>
> >>> Hi Jirka,
> >>>
> >>> Thanks for the detailed report.
> >>>
> >>> I'm curious about the specific setup in which this panic was triggered. Could you share more about the exact configuration or parameters you used for running `stress-ng` or Linpack? For instance:
> >>>
> >>> - How many threads/cores were used?
> >>> - Was it running inside a VM, container, or bare-metal?
> >>> - Was this under any thermal throttling or power-saving mode?
> >>>
> >>> I'd like to try reproducing it locally to study the failure further.
> >>>
> >>> Best regards,
> >>> Abhigyan Ghosh
> >>>
> >>> On 18 June 2025 1:35:30 am IST, Jirka Hladky <jhladky@redhat.com> wrote:
> >>>> Hi all,
> >>>>
> >>>> I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
> >>>> involving a NULL pointer dereference in `__migrate_swap_task()` during
> >>>> CPU migration. This occurred on various AMD and Intel systems while
> >>>> running a CPU-intensive workload (Linpack, Stress_ng - it's not
> >>>> specific to a benchmark).
> >>>>
> >>>> Full trace below:
> >>>> ---
> >>>> BUG: kernel NULL pointer dereference, address: 00000000000004c8
> >>>> #PF: supervisor read access in kernel mode
> >>>> #PF: error_code(0x0000) - not-present page
> >>>> PGD 4078b99067 P4D 4078b99067 PUD 0
> >>>> Oops: Oops: 0000 [#1] SMP NOPTI
> >>>> CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
> >>>> 6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
> >>>> Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
> >>>> Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
> >>>> RIP: 0010:__migrate_swap_task+0x2f/0x170
> >>>> Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
> >>>> 48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
> >>>> bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
> >>>> RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
> >>>> RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
> >>>> RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
> >>>> RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
> >>>> R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
> >>>> R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
> >>>> FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
> >>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>>> CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
> >>>> PKRU: 55555554
> >>>> Call Trace:
> >>>> <TASK>
> >>>> migrate_swap_stop+0xe8/0x190
> >>>> multi_cpu_stop+0xf3/0x130
> >>>> ? __pfx_multi_cpu_stop+0x10/0x10
> >>>> cpu_stopper_thread+0x97/0x140
> >>>> ? __pfx_smpboot_thread_fn+0x10/0x10
> >>>> smpboot_thread_fn+0xf3/0x220
> >>>> kthread+0xfc/0x240
> >>>> ? __pfx_kthread+0x10/0x10
> >>>> ? __pfx_kthread+0x10/0x10
> >>>> ret_from_fork+0xf0/0x110
> >>>> ? __pfx_kthread+0x10/0x10
> >>>> ret_from_fork_asm+0x1a/0x30
> >>>> </TASK>
> >>>> ---
> >>>>
> >>>> **Kernel Version:**
> >>>> 6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
> >>>> https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950
> >>>>
> >>>> **Reproducibility:**
> >>>> Happened multiple times during routine CPU-intensive operations. It
> >>>> happens with various benchmarks (Stress_ng, Linpack) after several
> >>>> hours of performance testing. `migration/*` kernel threads hit a NULL
> >>>> dereference in `__migrate_swap_task`.
> >>>>
> >>>> **System Info:**
> >>>> - Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
> >>>> - BIOS: M07 09/03/2021
> >>>> - Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)
> >>>>
> >>>> **Crash Cause (tentative):**
> >>>> NULL dereference at offset `0x4c8` from a task struct pointer in
> >>>> `__migrate_swap_task`. Possibly an uninitialized or freed
> >>>> `task_struct` field.
> >>>>
> >>>> Please let me know if you’d like me to test a patch or if you need
> >>>> more details.
> >>>>
> >>>> Thanks,
> >>>> Jirka
> >>>>
> >>>>
> >>>
> >>> aghosh
> >>>
> >>
> >>
> >> --
> >> -Jirka
> >
> >
> >
>


-- 
-Jirka