Shutdown-time hangs in -next in locktorture

Shutdown-time hangs in -next in locktorture

[Paul E. McKenney]

Hello, Peter,

I started hitting shutdown-time hangs in next-20251217 which persist
in next-20251219.  This hang happens on both x86 and arm64.  Once I
figured out that the failure is high probability, but not deterministic,
bisection converged here:

5d1f0b2f278e ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")

This commit reverts cleanly, and doing so restores hang-free operation.

The reproducer is shown below.

Thoughts?

							Thanx, Paul

------------------------------------------------------------------------

for i in 1 2 3 4 5
do
	tools/testing/selftests/rcutorture/bin/torture.sh --duration 20 --do-none --do-normal --do-locktorture --do-kasan --configs-locktorture "LOCK09"
	ret=$?
	if test "$ret" -ne 0
	then
		exit "$ret"
	fi
	echo Test $i succeeded
done
exit 0

Re: Shutdown-time hangs in -next in locktorture

[Paul E. McKenney]

On Fri, Dec 19, 2025 at 04:29:27PM -0800, Paul E. McKenney wrote:
> Hello, Peter,
> 
> I started hitting shutdown-time hangs in next-20251217 which persist
> in next-20251219.  This hang happens on both x86 and arm64.  Once I
> figured out that the failure is high probability, but not deterministic,
> bisection converged here:
> 
> 5d1f0b2f278e ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")
> 
> This commit reverts cleanly, and doing so restores hang-free operation.
> 
> The reproducer is shown below.
> 
> Thoughts?

With Chris Mason's help, I checked with a friendly local LLM, which
noted that a call to rq_modified_above() remains in kernel/sched/ext.c
in function do_pick_task_scx().  Of course, that does not explain a
locktorture hang, especially given that locktorture does not build
that file.  But in case it is helpful.

							Thanx, Paul

> ------------------------------------------------------------------------
> 
> for i in 1 2 3 4 5
> do
> 	tools/testing/selftests/rcutorture/bin/torture.sh --duration 20 --do-none --do-normal --do-locktorture --do-kasan --configs-locktorture "LOCK09"
> 	ret=$?
> 	if test "$ret" -ne 0
> 	then
> 		exit "$ret"
> 	fi
> 	echo Test $i succeeded
> done
> exit 0

Re: Shutdown-time hangs in -next in locktorture

[Peter Zijlstra]

On Fri, Dec 19, 2025 at 04:29:26PM -0800, Paul E. McKenney wrote:
> Hello, Peter,
> 
> I started hitting shutdown-time hangs in next-20251217 which persist
> in next-20251219.  This hang happens on both x86 and arm64.  Once I
> figured out that the failure is high probability, but not deterministic,
> bisection converged here:
> 
> 5d1f0b2f278e ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")

That commit no longer exists in tip/sched/core; it was fixed a few days
ago, except other problems made -next use an old tip branch which has
caused this fix to have delayed visibility :-(

Re: Shutdown-time hangs in -next in locktorture

[Paul E. McKenney]

On Sat, Dec 20, 2025 at 01:52:01PM +0100, Peter Zijlstra wrote:
> On Fri, Dec 19, 2025 at 04:29:26PM -0800, Paul E. McKenney wrote:
> > Hello, Peter,
> > 
> > I started hitting shutdown-time hangs in next-20251217 which persist
> > in next-20251219.  This hang happens on both x86 and arm64.  Once I
> > figured out that the failure is high probability, but not deterministic,
> > bisection converged here:
> > 
> > 5d1f0b2f278e ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")
> 
> That commit no longer exists in tip/sched/core; it was fixed a few days
> ago, except other problems made -next use an old tip branch which has
> caused this fix to have delayed visibility :-(

Very good, I will retry later.

							Thanx, Paul

Re: Shutdown-time hangs in -next in locktorture

[Paul E. McKenney]

On Sat, Dec 20, 2025 at 08:49:01AM -0800, Paul E. McKenney wrote:
> On Sat, Dec 20, 2025 at 01:52:01PM +0100, Peter Zijlstra wrote:
> > On Fri, Dec 19, 2025 at 04:29:26PM -0800, Paul E. McKenney wrote:
> > > Hello, Peter,
> > > 
> > > I started hitting shutdown-time hangs in next-20251217 which persist
> > > in next-20251219.  This hang happens on both x86 and arm64.  Once I
> > > figured out that the failure is high probability, but not deterministic,
> > > bisection converged here:
> > > 
> > > 5d1f0b2f278e ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")
> > 
> > That commit no longer exists in tip/sched/core; it was fixed a few days
> > ago, except other problems made -next use an old tip branch which has
> > caused this fix to have delayed visibility :-(
> 
> Very good, I will retry later.

A bit later than I was planning, but here we are on next-20260113.  This
has a very similar failure on arm64 with that same repeat-by as before:

tools/testing/selftests/rcutorture/bin/torture.sh --duration 20 --do-none --do-normal --do-locktorture --do-kasan --configs-locktorture "LOCK09"

Bisection converges here:

704069649b5b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")

This commit does not revert cleanly, but retesting on both this commit
and the previous commit confirms the bisection result.

I have not yet checked this carefully on x86.

Or is this another case of stale commits in -next?  If not, please let
me know if there are debug options/patches that would be helpful.



							Thanx, Paul