Boot regression: next-20251215: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end

Boot regression: next-20251215: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end

[Naresh Kamboju]

[Re-sending with unwrapping]

The following boot warning is noticed on qemu-arm64 booting the
Linux next-20251215 and next-20251216.

Regression Analysis:
- New regression? Yes
- Reproducibility? Yes

First seen on next-20251215
Bad:  next-20251215 and next-20251216
Good: next-20251212

Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end

Are these expected warnings ? Anyone noticed ?

Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>

Full boot log links provided at the bottom of the email.

## Boot log
[    0.000000] Linux version 6.19.0-rc1-next-20251215
(tuxmake@tuxmake) (aarch64-linux-gnu-gcc (Debian 13.3.0-16) 13.3.0,
GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT @1765798467
...
[   14.659447] mlx5_core 0000:01:00.1 enp1s0f1np1: renamed from eth1
[   14.691267] CPPC Cpufreq:FIE not enabled on systems with registers in PCC
** replaying previous printk message **
[   14.691267] CPPC Cpufreq:FIE not enabled on systems with registers in PCC
[   14.694920] mlx5_core 0000:01:00.0 enp1s0f0np0: renamed from eth0 (while UP)
[   14.696414] ------------[ cut here ]------------
[   14.696418] WARNING: kernel/sched/core.c:10851 at
sched_change_end+0x168/0x188, CPU#12: ktimers/12/117
[   14.729321] Modules linked in: cppc_cpufreq(+) arm_dsu_pmu(+) fuse
drm backlight
[   14.736718] CPU: 12 UID: 0 PID: 117 Comm: ktimers/12 Not tainted
6.19.0-rc1-next-20251216 #1 PREEMPT_RT
[   14.746190] Hardware name: WIWYNN Mt.Jade Server System
B81.030Z1.0010/Mt.Jade Motherboard, BIOS 2.10.20250506-1P (SCP:
2.10.20250506) 2025/05/06
[   14.759217] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   14.766169] pc : sched_change_end (kernel/sched/core.c:10851
(discriminator 7))
[   14.770519] lr : sched_change_end (kernel/sched/core.c:10842
(discriminator 1))
[   14.774781] sp : ffff800080d33b90
[   14.778084] x29: ffff800080d33b90 x28: ffff07ff822f5500 x27: ffff80008e0c3660
[   14.785213] x26: 00000000000000f0 x25: ffff07ff822f5500 x24: 0000000000000000
[   14.792342] x23: 00000000ffffffff x22: ffffb4099c0d64d0 x21: ffff07ff822f5500
[   14.799469] x20: ffff083e5eeaf140 x19: ffff083e5ee9b290 x18: 0000000000000000
[   14.806597] x17: 0000000000000000 x16: ffffb4099a40ee68 x15: 0000000000000000
[   14.813725] x14: 0000000000000000 x13: 0000000000000000 x12: 0000020000000000
[   14.820853] x11: 00000000000000c0 x10: 0000000000000c60 x9 : ffffb40999b8d640
[   14.827981] x8 : 000000000000000c x7 : ffff07ff92ad8400 x6 : 00000000ffffffff
[   14.835109] x5 : 00000000000000ee x4 : 000000036b338800 x3 : 000000036b33b800
[   14.842238] x2 : 0000000000000001 x1 : ffffb4099c0d63d8 x0 : 0000000000000010
[   14.849366] Call trace:
[   14.851802]  sched_change_end (kernel/sched/core.c:10851
(discriminator 7)) (P)
[   14.856153]  rt_mutex_setprio (kernel/sched/sched.h:1813
kernel/sched/core.c:7368)
[   14.860157]  rt_mutex_slowunlock (kernel/locking/rtmutex.c:1341
kernel/locking/rtmutex.c:1466)
[   14.864422]  rt_spin_unlock (kernel/locking/spinlock_rt.c:86)
[   14.868165]  __hrtimer_run_queues (include/linux/spinlock_rt.h:44
kernel/time/hrtimer.c:1398 kernel/time/hrtimer.c:1843)
[   14.872513]  hrtimer_run_softirq (kernel/time/hrtimer.c:579
kernel/time/hrtimer.c:1193 kernel/time/hrtimer.c:1861)
[   14.876600]  handle_softirqs.isra.0
(arch/arm64/include/asm/jump_label.h:36 include/trace/events/irq.h:142
kernel/softirq.c:627)
[   14.881125]  run_ktimerd (kernel/softirq.c:325 kernel/softirq.c:1144)
[   14.884520]  smpboot_thread_fn (kernel/smpboot.c:160)
[   14.888608]  kthread (kernel/kthread.c:463)
[   14.891828]  ret_from_fork (arch/arm64/kernel/entry.S:861)
[   14.895396] ---[ end trace 0000000000000000 ]---

## Source
* Kernel version: 6.19.0-rc1
* Git tree: https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git
* Git commit: 4a5663c04bb679631985a15efab774da58c37815
* Git Describe: next-20251215
* Test Details:
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215

## Boot details
* Boot log: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215/testrun/30561916/suite/log-parser-test/test/exception-warning-kernelschedcore-at-sched_change_end/log
* Test details:
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215/testrun/30561916/suite/log-parser-test/test/exception-warning-kernelschedcore-at-sched_change_end/details/
* Test plan: https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/36sbHUVfnUKPzZ4LNkOCLdiFOin
* Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/36sbH5DQckAagO0V1UEWBBkOB54/
* Kernel config:
https://storage.tuxsuite.com/public/linaro/lkft/builds/36sbH5DQckAagO0V1UEWBBkOB54/config

--
Linaro LKFT
https://lkft.linaro.org

Re: Boot regression: next-20251215: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end

[Peter Zijlstra]

On Tue, Dec 16, 2025 at 06:11:11PM +0530, Naresh Kamboju wrote:
> [Re-sending with unwrapping]
> 
> The following boot warning is noticed on qemu-arm64 booting the
> Linux next-20251215 and next-20251216.
> 
> Regression Analysis:
> - New regression? Yes
> - Reproducibility? Yes
> 
> First seen on next-20251215
> Bad:  next-20251215 and next-20251216
> Good: next-20251212
> 
> Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
> 
> Are these expected warnings ? Anyone noticed ?
> 
> Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>

Still a wrapped mess :/

Anyway, afaik you're the first to notice.

> Full boot log links provided at the bottom of the email.
> 
> ## Boot log
> [    0.000000] Linux version 6.19.0-rc1-next-20251215
> (tuxmake@tuxmake) (aarch64-linux-gnu-gcc (Debian 13.3.0-16) 13.3.0,
> GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT @1765798467
> ...
> [   14.659447] mlx5_core 0000:01:00.1 enp1s0f1np1: renamed from eth1
> [   14.691267] CPPC Cpufreq:FIE not enabled on systems with registers in PCC
> ** replaying previous printk message **
> [   14.691267] CPPC Cpufreq:FIE not enabled on systems with registers in PCC
> [   14.694920] mlx5_core 0000:01:00.0 enp1s0f0np0: renamed from eth0 (while UP)
> [   14.696414] ------------[ cut here ]------------
> [   14.696418] WARNING: kernel/sched/core.c:10851 at
> sched_change_end+0x168/0x188, CPU#12: ktimers/12/117
> [   14.729321] Modules linked in: cppc_cpufreq(+) arm_dsu_pmu(+) fuse
> drm backlight
> [   14.736718] CPU: 12 UID: 0 PID: 117 Comm: ktimers/12 Not tainted
> 6.19.0-rc1-next-20251216 #1 PREEMPT_RT
> [   14.746190] Hardware name: WIWYNN Mt.Jade Server System
> B81.030Z1.0010/Mt.Jade Motherboard, BIOS 2.10.20250506-1P (SCP:
> 2.10.20250506) 2025/05/06
> [   14.759217] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [   14.766169] pc : sched_change_end (kernel/sched/core.c:10851
> (discriminator 7))
> [   14.770519] lr : sched_change_end (kernel/sched/core.c:10842
> (discriminator 1))
> [   14.774781] sp : ffff800080d33b90
> [   14.778084] x29: ffff800080d33b90 x28: ffff07ff822f5500 x27: ffff80008e0c3660
> [   14.785213] x26: 00000000000000f0 x25: ffff07ff822f5500 x24: 0000000000000000
> [   14.792342] x23: 00000000ffffffff x22: ffffb4099c0d64d0 x21: ffff07ff822f5500
> [   14.799469] x20: ffff083e5eeaf140 x19: ffff083e5ee9b290 x18: 0000000000000000
> [   14.806597] x17: 0000000000000000 x16: ffffb4099a40ee68 x15: 0000000000000000
> [   14.813725] x14: 0000000000000000 x13: 0000000000000000 x12: 0000020000000000
> [   14.820853] x11: 00000000000000c0 x10: 0000000000000c60 x9 : ffffb40999b8d640
> [   14.827981] x8 : 000000000000000c x7 : ffff07ff92ad8400 x6 : 00000000ffffffff
> [   14.835109] x5 : 00000000000000ee x4 : 000000036b338800 x3 : 000000036b33b800
> [   14.842238] x2 : 0000000000000001 x1 : ffffb4099c0d63d8 x0 : 0000000000000010
> [   14.849366] Call trace:
> [   14.851802]  sched_change_end (kernel/sched/core.c:10851
> (discriminator 7)) (P)
> [   14.856153]  rt_mutex_setprio (kernel/sched/sched.h:1813
> kernel/sched/core.c:7368)
> [   14.860157]  rt_mutex_slowunlock (kernel/locking/rtmutex.c:1341
> kernel/locking/rtmutex.c:1466)
> [   14.864422]  rt_spin_unlock (kernel/locking/spinlock_rt.c:86)
> [   14.868165]  __hrtimer_run_queues (include/linux/spinlock_rt.h:44
> kernel/time/hrtimer.c:1398 kernel/time/hrtimer.c:1843)
> [   14.872513]  hrtimer_run_softirq (kernel/time/hrtimer.c:579
> kernel/time/hrtimer.c:1193 kernel/time/hrtimer.c:1861)
> [   14.876600]  handle_softirqs.isra.0
> (arch/arm64/include/asm/jump_label.h:36 include/trace/events/irq.h:142
> kernel/softirq.c:627)
> [   14.881125]  run_ktimerd (kernel/softirq.c:325 kernel/softirq.c:1144)
> [   14.884520]  smpboot_thread_fn (kernel/smpboot.c:160)
> [   14.888608]  kthread (kernel/kthread.c:463)
> [   14.891828]  ret_from_fork (arch/arm64/kernel/entry.S:861)
> [   14.895396] ---[ end trace 0000000000000000 ]---

This is:

	WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) && 
			!test_tsk_need_resched(p));

And rt_mutex_setprio() doesn't set need_resched() on demotion; but
worse, reading it now, I realize that if p != current, nobody ever will.

I suppose something like so will cure things?

---
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 7d0a862a8c75..5b17d8e3cb55 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10833,23 +10833,24 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		if (p->sched_class->switched_to)
 			p->sched_class->switched_to(rq, p);
 
-		/*
-		 * If this was a class promotion; let the old class know it
-		 * got preempted. Note that none of the switch*_from() methods
-		 * know the new class and none of the switch*_to() methods
-		 * know the old class.
-		 */
-		if (ctx->running && sched_class_above(p->sched_class, ctx->class)) {
-			rq->next_class->wakeup_preempt(rq, p, 0);
-			rq->next_class = p->sched_class;
+		if (ctx->running) {
+			/*
+			 * If this was a class promotion; let the old class
+			 * know it got preempted. Note that none of the
+			 * switch*_from() methods know the new class and none
+			 * of the switch*_to() methods know the old class.
+			 */
+			if (sched_class_above(p->sched_class, ctx->class)) {
+				rq->next_class->wakeup_preempt(rq, p, 0);
+				rq->next_class = p->sched_class;
+			}
+			/*
+			 * If this was a degradation in class; make sure to
+			 * reschedule.
+			 */
+			if (sched_class_above(ctx->class, p->sched_class))
+				resched_curr(rq);
 		}
-
-		/*
-		 * If this was a degradation in class someone should have set
-		 * need_resched by now.
-		 */
-		WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) &&
-			     !test_tsk_need_resched(p));
 	} else {
 		p->sched_class->prio_changed(rq, p, ctx->prio);
 	}

Re: Boot regression: next-20251215: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end

[Naresh Kamboju]

On Tue, 16 Dec 2025 at 19:47, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Tue, Dec 16, 2025 at 06:11:11PM +0530, Naresh Kamboju wrote:
> > [Re-sending with unwrapping]
> >
> > The following boot warning is noticed on qemu-arm64 booting the
> > Linux next-20251215 and next-20251216.
> >
> > Regression Analysis:
> > - New regression? Yes
> > - Reproducibility? Yes
> >
> > First seen on next-20251215
> > Bad:  next-20251215 and next-20251216
> > Good: next-20251212
> >
> > Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
> >
> > Are these expected warnings ? Anyone noticed ?
> >
> > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
>
> Still a wrapped mess :/
>
> Anyway, afaik you're the first to notice.

<trim>

> This is:
>
>         WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) &&
>                         !test_tsk_need_resched(p));
>
> And rt_mutex_setprio() doesn't set need_resched() on demotion; but
> worse, reading it now, I realize that if p != current, nobody ever will.
>
> I suppose something like so will cure things?

I have tested this patch and did not encounter any regressions.

Tested-by:  Linux Kernel Functional Testing <lkft@linaro.org>

>
> ---
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 7d0a862a8c75..5b17d8e3cb55 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -10833,23 +10833,24 @@ void sched_change_end(struct sched_change_ctx *ctx)
>                 if (p->sched_class->switched_to)
>                         p->sched_class->switched_to(rq, p);
>
> -               /*
> -                * If this was a class promotion; let the old class know it
> -                * got preempted. Note that none of the switch*_from() methods
> -                * know the new class and none of the switch*_to() methods
> -                * know the old class.
> -                */
> -               if (ctx->running && sched_class_above(p->sched_class, ctx->class)) {
> -                       rq->next_class->wakeup_preempt(rq, p, 0);
> -                       rq->next_class = p->sched_class;
> +               if (ctx->running) {
> +                       /*
> +                        * If this was a class promotion; let the old class
> +                        * know it got preempted. Note that none of the
> +                        * switch*_from() methods know the new class and none
> +                        * of the switch*_to() methods know the old class.
> +                        */
> +                       if (sched_class_above(p->sched_class, ctx->class)) {
> +                               rq->next_class->wakeup_preempt(rq, p, 0);
> +                               rq->next_class = p->sched_class;
> +                       }
> +                       /*
> +                        * If this was a degradation in class; make sure to
> +                        * reschedule.
> +                        */
> +                       if (sched_class_above(ctx->class, p->sched_class))
> +                               resched_curr(rq);
>                 }
> -
> -               /*
> -                * If this was a degradation in class someone should have set
> -                * need_resched by now.
> -                */
> -               WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) &&
> -                            !test_tsk_need_resched(p));
>         } else {
>                 p->sched_class->prio_changed(rq, p, ctx->prio);
>         }
>

 - https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/36wC2DzDokQEutv2FqXwjRjSv82

- Naresh

[tip: sched/core] sched: Fix faulty assertion in sched_change_end()

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     1862d8e264def8425d682f1177e22f9fe7d947ea
Gitweb:        https://git.kernel.org/tip/1862d8e264def8425d682f1177e22f9fe7d947ea
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Wed, 17 Dec 2025 11:24:11 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Wed, 17 Dec 2025 11:41:18 +01:00

sched: Fix faulty assertion in sched_change_end()

Commit 47efe2ddccb1f ("sched/core: Add assertions to QUEUE_CLASS") added an
assert to sched_change_end() verifying that a class demotion would result in a
reschedule.

As it turns out; rt_mutex_setprio() does not force a resched on class
demontion. Furthermore, this is only relevant to running tasks.

Change the warning into a reschedule and make sure to only do so for running
tasks.

Fixes: 47efe2ddccb1f ("sched/core: Add assertions to QUEUE_CLASS")
Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Tested-by:  Linux Kernel Functional Testing <lkft@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20251216141725.GW3707837@noisy.programming.kicks-ass.net
---
 kernel/sched/core.c | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 7d0a862..5b17d8e 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10833,23 +10833,24 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		if (p->sched_class->switched_to)
 			p->sched_class->switched_to(rq, p);
 
-		/*
-		 * If this was a class promotion; let the old class know it
-		 * got preempted. Note that none of the switch*_from() methods
-		 * know the new class and none of the switch*_to() methods
-		 * know the old class.
-		 */
-		if (ctx->running && sched_class_above(p->sched_class, ctx->class)) {
-			rq->next_class->wakeup_preempt(rq, p, 0);
-			rq->next_class = p->sched_class;
+		if (ctx->running) {
+			/*
+			 * If this was a class promotion; let the old class
+			 * know it got preempted. Note that none of the
+			 * switch*_from() methods know the new class and none
+			 * of the switch*_to() methods know the old class.
+			 */
+			if (sched_class_above(p->sched_class, ctx->class)) {
+				rq->next_class->wakeup_preempt(rq, p, 0);
+				rq->next_class = p->sched_class;
+			}
+			/*
+			 * If this was a degradation in class; make sure to
+			 * reschedule.
+			 */
+			if (sched_class_above(ctx->class, p->sched_class))
+				resched_curr(rq);
 		}
-
-		/*
-		 * If this was a degradation in class someone should have set
-		 * need_resched by now.
-		 */
-		WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) &&
-			     !test_tsk_need_resched(p));
 	} else {
 		p->sched_class->prio_changed(rq, p, ctx->prio);
 	}