[RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Move the proxy_needs_return() check earlier in ttwu_runnable() to avoid
unnecessary scheduling operations when a proxy task requires return
migration to its original CPU.

The current implementation performs several operations (rq clock update,
enqueue, and wakeup preemption checks) before checking for return
migration needs. This is inefficient because:

1. For tasks needing return migration, these operations are redundant
   since the task will be dequeued from current rq anyway
2. The task may not even be allowed to run on current CPU due to
   possible affinity changes during blocking
3. The proper CPU selection will be handled by select_task_rq() in
   the subsequent try_to_wake_up() logic

By moving the proxy_needs_return() check to the beginning, we:
- Avoid unnecessary rq clock updates
- Skip redundant enqueue operations
- Eliminate meaningless wakeup preemption checks
- Let the normal wakeup path handle proper CPU selection

This optimization is particularly valuable in proxy execution scenarios
where tasks frequently migrate between CPUs.

Signed-off-by: hupu <hupu.gm@gmail.com>
---
 kernel/sched/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ca4ca739eb85..ebb4bc1800e3 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4162,6 +4162,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
 	rq = __task_rq_lock(p, &rf);
 	if (task_on_rq_queued(p)) {
+		if (proxy_needs_return(rq, p)) {
+			_trace_sched_pe_return_migration(p);
+			goto out;
+		}
 		update_rq_clock(rq);
 		if (p->se.sched_delayed) {
 			proxy_remove_from_sleeping_owner(p);
@@ -4174,10 +4178,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 			 */
 			wakeup_preempt(rq, p, wake_flags);
 		}
-		if (proxy_needs_return(rq, p)) {
-			_trace_sched_pe_return_migration(p);
-			goto out;
-		}
 		ttwu_do_wakeup(p);
 		ret = 1;
 	}
-- 
2.47.0

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Dear maintainer,
My sincere apologies for the oversight in my previous submission. I
realized I failed to provide the repository and branch details for the
patch context. Here are the specifics:

The patch is based on the following repository:
https://github.com/johnstultz-work/linux-dev.git

Specifically, the changes are present in these Proxy Execution
development branches:
a) proxy-exec-WIP
b) proxy-exec-v15-WIP

The changes are part of the ongoing Proxy Execution feature
development in these branches.
Please let me know if you need further clarifications. Thank you for
your understanding and patience.

On Mon, Apr 7, 2025 at 9:47 PM hupu <hupu.gm@gmail.com> wrote:
>
> Move the proxy_needs_return() check earlier in ttwu_runnable() to avoid
> unnecessary scheduling operations when a proxy task requires return
> migration to its original CPU.
>
> The current implementation performs several operations (rq clock update,
> enqueue, and wakeup preemption checks) before checking for return
> migration needs. This is inefficient because:
>
> 1. For tasks needing return migration, these operations are redundant
>    since the task will be dequeued from current rq anyway
> 2. The task may not even be allowed to run on current CPU due to
>    possible affinity changes during blocking
> 3. The proper CPU selection will be handled by select_task_rq() in
>    the subsequent try_to_wake_up() logic
>
> By moving the proxy_needs_return() check to the beginning, we:
> - Avoid unnecessary rq clock updates
> - Skip redundant enqueue operations
> - Eliminate meaningless wakeup preemption checks
> - Let the normal wakeup path handle proper CPU selection
>
> This optimization is particularly valuable in proxy execution scenarios
> where tasks frequently migrate between CPUs.
>
> Signed-off-by: hupu <hupu.gm@gmail.com>
> ---
>  kernel/sched/core.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index ca4ca739eb85..ebb4bc1800e3 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4162,6 +4162,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
>
>         rq = __task_rq_lock(p, &rf);
>         if (task_on_rq_queued(p)) {
> +               if (proxy_needs_return(rq, p)) {
> +                       _trace_sched_pe_return_migration(p);
> +                       goto out;
> +               }
>                 update_rq_clock(rq);
>                 if (p->se.sched_delayed) {
>                         proxy_remove_from_sleeping_owner(p);
> @@ -4174,10 +4178,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
>                          */
>                         wakeup_preempt(rq, p, wake_flags);
>                 }
> -               if (proxy_needs_return(rq, p)) {
> -                       _trace_sched_pe_return_migration(p);
> -                       goto out;
> -               }
>                 ttwu_do_wakeup(p);
>                 ret = 1;
>         }
> --
> 2.47.0
>

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[John Stultz]

On Mon, Apr 7, 2025 at 6:47 AM hupu <hupu.gm@gmail.com> wrote:
>
> Move the proxy_needs_return() check earlier in ttwu_runnable() to avoid
> unnecessary scheduling operations when a proxy task requires return
> migration to its original CPU.
>
> The current implementation performs several operations (rq clock update,
> enqueue, and wakeup preemption checks) before checking for return
> migration needs. This is inefficient because:
>
> 1. For tasks needing return migration, these operations are redundant
>    since the task will be dequeued from current rq anyway
> 2. The task may not even be allowed to run on current CPU due to
>    possible affinity changes during blocking
> 3. The proper CPU selection will be handled by select_task_rq() in
>    the subsequent try_to_wake_up() logic
>
> By moving the proxy_needs_return() check to the beginning, we:
> - Avoid unnecessary rq clock updates
> - Skip redundant enqueue operations
> - Eliminate meaningless wakeup preemption checks
> - Let the normal wakeup path handle proper CPU selection
>
> This optimization is particularly valuable in proxy execution scenarios
> where tasks frequently migrate between CPUs.

Hey hupu! Thanks again for your review of the proxy series and your
sharing of this improvement!

The proposal sounds attractive, however...

> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index ca4ca739eb85..ebb4bc1800e3 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4162,6 +4162,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
>
>         rq = __task_rq_lock(p, &rf);
>         if (task_on_rq_queued(p)) {
> +               if (proxy_needs_return(rq, p)) {
> +                       _trace_sched_pe_return_migration(p);
> +                       goto out;
> +               }
>                 update_rq_clock(rq);
>                 if (p->se.sched_delayed) {
>                         proxy_remove_from_sleeping_owner(p);
> @@ -4174,10 +4178,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
>                          */
>                         wakeup_preempt(rq, p, wake_flags);
>                 }
> -               if (proxy_needs_return(rq, p)) {
> -                       _trace_sched_pe_return_migration(p);
> -                       goto out;
> -               }
>                 ttwu_do_wakeup(p);
>                 ret = 1;
>         }
> --

Unfortunately this patch crashes pretty quickly in my testing. The
first issue was proxy_needs_return() calls deactivate_task() w/
DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock()
hasn't been called. Preserving the update_rq_clock() line before
checking proxy_needs_return() avoided that issue, but then I saw hangs
during bootup, which I suspect is due to us shortcutting over the
sched_delayed case.

Moving the proxy_needs_return above the if(task_on_cpu())
wakeup_preempt() logic booted ok, but I'm still a little hesitant of
what side-effects that might cause.

Part of the approach with proxy_needs_return() from ttwu_runnable() is
making sure everything is ready for the queued task to run, but in the
case where we need return migration, to dequeue the task and cause
ttwu_runnable() to fail, forcing the rest of try_to_wake_up() to
complete.  I do really appreciate your thoughts here though, as I
agree that the potential enqueue/dequeue/enqueue could be wasteful.

I'll take a bit more time to see how we might be able to avoid this
extra work without skipping needed steps. If you have further ideas,
I'd love to hear them!

thanks again!
-john

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John:
Thank you for your feedback.

On Thu, Apr 10, 2025 at 10:41 AM John Stultz <jstultz@google.com> wrote:
>
> Unfortunately this patch crashes pretty quickly in my testing. The
> first issue was proxy_needs_return() calls deactivate_task() w/
> DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock()
> hasn't been called. Preserving the update_rq_clock() line before
> checking proxy_needs_return() avoided that issue, but then I saw hangs
> during bootup, which I suspect is due to us shortcutting over the
> sched_delayed case.
>
> Moving the proxy_needs_return above the if(task_on_cpu())
> wakeup_preempt() logic booted ok, but I'm still a little hesitant of
> what side-effects that might cause.

I’m sorry for the confusion caused by this patch. Here is the
rationale behind my approach:

To ensure that donor tasks can get a suitable CPU and avoid negative
impacts from the Proxy-Execution on load balancing,
`proxy_needs_return()` in `ttwu_runnable()` should return false for
all donor tasks. This allows `try_to_wake_up()` to use `set_task_cpu`
to reselect a CPU for the donor tasks, unless the donor is already
running on a CPU.

This patch worked correctly on my QEMU-based test platform, it seems
our testing methods might differ. Could you please share the details
of your testing environment and methodology? I’ll try to replicate the
issue using the same approach.

In the meantime, I will carefully revisit the logic in this patch to
ensure its correctness and consistency. Once I’ve completed the
review, I look forward to further discussing the details with you.

Thank you again for your valuable feedback!

Best regards,
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[John Stultz]

On Thu, Apr 10, 2025 at 2:51 AM hupu <hupu.gm@gmail.com> wrote:
>
> Hi John:
> Thank you for your feedback.
>
> On Thu, Apr 10, 2025 at 10:41 AM John Stultz <jstultz@google.com> wrote:
> >
> > Unfortunately this patch crashes pretty quickly in my testing. The
> > first issue was proxy_needs_return() calls deactivate_task() w/
> > DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock()
> > hasn't been called. Preserving the update_rq_clock() line before
> > checking proxy_needs_return() avoided that issue, but then I saw hangs
> > during bootup, which I suspect is due to us shortcutting over the
> > sched_delayed case.
...
> This patch worked correctly on my QEMU-based test platform, it seems
> our testing methods might differ. Could you please share the details
> of your testing environment and methodology? I’ll try to replicate the
> issue using the same approach.

Usually I boot qemu with 64 cores, and have found stress testing
running the following separate programs frequently uncovers issues:

#cyclictest from the rt-test suite to add some rt load
cyclictest -t -p90

# rerunning the test_ww_mtuex logic (enabled by a patch in my full
proxy-exec series)
while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests ; sleep 5; done

# Along with my prio-inversion-demo (which ensures we do the proxying paths)
#   https://github.com/johnstultz-work/priority-inversion-demo
while true; do ./run.sh; sleep 10 ; done

# Sometimes I also throw in the ltp sched_football test:
# https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/func/sched_football
while true; do ./sched_football -m | grep Result ; sleep 1; done

But I'm always looking for other scheduling loads to trip up bugs, so
open to suggestions.

thanks
-john

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[John Stultz]

On Wed, Apr 16, 2025 at 5:23 PM John Stultz <jstultz@google.com> wrote:
>
> On Thu, Apr 10, 2025 at 2:51 AM hupu <hupu.gm@gmail.com> wrote:
> >
> > Hi John:
> > Thank you for your feedback.
> >
> > On Thu, Apr 10, 2025 at 10:41 AM John Stultz <jstultz@google.com> wrote:
> > >
> > > Unfortunately this patch crashes pretty quickly in my testing. The
> > > first issue was proxy_needs_return() calls deactivate_task() w/
> > > DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock()
> > > hasn't been called. Preserving the update_rq_clock() line before
> > > checking proxy_needs_return() avoided that issue, but then I saw hangs
> > > during bootup, which I suspect is due to us shortcutting over the
> > > sched_delayed case.
> ...
> > This patch worked correctly on my QEMU-based test platform, it seems
> > our testing methods might differ. Could you please share the details
> > of your testing environment and methodology? I’ll try to replicate the
> > issue using the same approach.
>
> Usually I boot qemu with 64 cores, and have found stress testing
> running the following separate programs frequently uncovers issues:

Oh, also I enable locktorture and boot with:
"torture.random_shuffle=1 locktorture.writer_fifo=1
locktorture.torture_type=mutex_lock locktorture.nested_locks=8
locktorture.rt_boost=1 locktorture.rt_boost_factor=50
locktorture.stutter=0 "

thanks
-john

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

On Thu, Apr 17, 2025 at 8:24 AM John Stultz <jstultz@google.com> wrote:
>
> Oh, also I enable locktorture and boot with:
> "torture.random_shuffle=1 locktorture.writer_fifo=1
> locktorture.torture_type=mutex_lock locktorture.nested_locks=8
> locktorture.rt_boost=1 locktorture.rt_boost_factor=50
> locktorture.stutter=0 "
>

Hi John,

Thank you so much for your feedback.

I’ll follow the methods you’ve outlined to try and reproduce the same
issue, and will get back to you for further discussion.

Thanks again,
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John,
Sorry for the delayed response.

> Usually I boot qemu with 64 cores, and have found stress testing
> running the following separate programs frequently uncovers issues:
>
> #cyclictest from the rt-test suite to add some rt load
> cyclictest -t -p90
>
> # rerunning the test_ww_mtuex logic (enabled by a patch in my full
> proxy-exec series)
> while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests ; sleep 5; done
>
> # Along with my prio-inversion-demo (which ensures we do the proxying paths)
> # ? https://github.com/johnstultz-work/priority-inversion-demo
> while true; do ./run.sh; sleep 10 ; done
>
> # Sometimes I also throw in the ltp sched_football test:
> # https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/func/sched_football
> while true; do ./sched_football -m | grep Result ; sleep 1; done
>


I followed your suggested approach and successfully reproduced the
issue you mentioned. The system experienced a hang during bootup and
triggered a crash due to a watchdog timeout. This issue is related to
the logic of sched_delayed. I will provide a detailed description of
the debugging steps below. I apologize in advance if the explanation
becomes somewhat lengthy.

I added debugging information to capture the task execution logic
prior to the crash. The corresponding patch is provided below:

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
old mode 100644
new mode 100755
index 06e9924d3f77..c9adc17f53e5
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2098,6 +2098,14 @@ static inline void __activate_task(struct rq
*rq, struct task_struct *p, int fla
  WRITE_ONCE(p->on_rq, TASK_ON_RQ_QUEUED);
  ASSERT_EXCLUSIVE_WRITER(p->on_rq);
+
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
 }

 #ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4160,6 +4168,19 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  rq = __task_rq_lock(p, &rf);
  if (task_on_rq_queued(p)) {
  update_rq_clock(rq);
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+ goto out;
+ }
+
  if (p->se.sched_delayed) {
  proxy_remove_from_sleeping_owner(p);
  enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
@@ -4171,10 +4192,6 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  */
  wakeup_preempt(rq, p, wake_flags);
  }
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
  ttwu_do_wakeup(p);
  ret = 1;
  }
@@ -4377,6 +4394,14 @@ static void ttwu_queue(struct task_struct *p,
int cpu, int wake_flags)
  struct rq *rq = cpu_rq(cpu);
  struct rq_flags rf;

+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+
  if (ttwu_queue_wakelist(p, cpu, wake_flags))
  return;

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
old mode 100644
new mode 100755
index 1f707b614142..a3857b058ed6
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5623,6 +5623,8 @@ static int dequeue_entities(struct rq *rq,
struct sched_entity *se, int flags);
 static struct sched_entity *
 pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)
 {
+ struct task_struct *p;
+
  /*
  * Enabling NEXT_BUDDY will affect latency but not fairness.
  */
@@ -5635,7 +5637,25 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)

  struct sched_entity *se = pick_eevdf(cfs_rq);
  if (se->sched_delayed) {
+ p = task_of(se);
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+
  dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+
  /*
  * Must not reference @se again, see __block_task().
  */


At the time of the crash, the dmesg logs included multiple warnings
similar to the following. For analysis, I will use the task with PID
151 as an example.

[  124.912061][   T61] INFO: task kworker/u32:8:151 blocked for more
than 62 seconds.
[  124.912708][   T61]      on_rq: 0 rq: 4 state: 0x2  bo_state: 1
[  124.913133][   T61]       Not tainted 6.12.0-00050-gf6c6e0ea8a1d-dirty #93
[  124.913535][   T61] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  124.991597][   T61] task:kworker/u32:8   state:D stack:0
pid:151   tgid:151   ppid:2      flags:0x00000008
[  124.992829][   T61] Workqueue: test-ww_mutex stress_reorder_work
[  124.993361][   T61] Call trace:
[  124.993677][   T61]  __switch_to+0xe4/0x1a4
[  125.027050][   T61]  __schedule+0x4e0/0x1960
[  125.031483][   T61]  schedule+0x4c/0x130
[  125.031811][   T61]  schedule_preempt_disabled+0x28/0x40
[  125.032150][   T61]  __ww_mutex_lock.constprop.28+0xbdc/0x1960
[  125.032539][   T61]  ww_mutex_lock+0x48/0x98
[  125.032870][   T61]  stress_reorder_work+0x268/0x3c4
[  125.033287][   T61]  process_scheduled_works+0x254/0x694
[  125.033728][   T61]  worker_thread+0x16c/0x310
[  125.034078][   T61]  kthread+0x128/0x134
[  125.034419][   T61]  ret_from_fork+0x10/0x20


Switching the crash environment to the context of task 151, the `info
stack` command reveals that task 151 is blocked in __ww_mutex_lock
while waiting for the lock `lock=lock@entry=0xffff000006ab5060`.

crash_arm64> set 151
    PID: 151
COMMAND: "kworker/u32:8"
   TASK: ffff0000060ec180  [THREAD_INFO: ffff0000060ec180]
    CPU: 4
  STATE: TASK_UNINTERRUPTIBLE
crash_arm64> info stack
#0  __switch_to (prev=prev@entry=0xffff0000060ec180,
next=next@entry=0xffff000005444180) at arch/arm64/kernel/process.c:609
#1  0xffff8000813bf6ac in context_switch (rf=0xffff800086123b40,
next=0xffff000005444180, prev=0xffff0000060ec180,
rq=0xffff00002cbf48c0) at kernel/sched/core.c:5831
#2  __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:7637
#3  0xffff8000813c0b78 in __schedule_loop (sched_mode=<optimised out>)
at kernel/sched/core.c:7718
#4  schedule () at kernel/sched/core.c:7733
#5  0xffff8000813c0c84 in schedule_preempt_disabled () at
kernel/sched/core.c:7790
#6  0xffff8000813c6054 in __mutex_lock_common (use_ww_ctx=true,
ww_ctx=0xffff800086123cd0, ip=18446603338369690404,
nest_lock=<optimised out>, subclass=0, state=2203262976,
lock=0xffff000006ab5060) at kernel/locking/mutex.c:665
#7  __ww_mutex_lock (lock=lock@entry=0xffff000006ab5060,
state=2203262976, state@entry=2, ip=ip@entry=18446603338369690404,
ww_ctx=ww_ctx@entry=0xffff800086123cd0, subclass=0) at
kernel/locking/mutex.c:762
#8  0xffff8000813c6ea0 in ww_mutex_lock
(lock=lock@entry=0xffff000006ab5060, ctx=0xffff800086123cd0,
ctx@entry=0xffff800086123d40) at kernel/locking/mutex.c:879
#9  0xffff8000800f6b24 in ww_mutex_lock_slow (ctx=0xffff800086123d40,
lock=0xffff000006ab5060) at ./include/linux/ww_mutex.h:299
#10 stress_reorder_work (work=0xffff0000065938f0) at
kernel/locking/test-ww_mutex.c:532
#11 0xffff80008007ef84 in process_one_work (work=0xffff0000065938f0,
worker=0xffff000005db1300) at kernel/workqueue.c:3229
#12 process_scheduled_works (worker=worker@entry=0xffff000005db1300)
at kernel/workqueue.c:3310
#13 0xffff80008007fa3c in worker_thread (__worker=0xffff000005db1300)
at kernel/workqueue.c:3391
#14 0xffff80008008b2e8 in kthread (_create=0xffff000006848400) at
kernel/kthread.c:389
#15 0xffff8000800162dc in ret_from_fork () at arch/arm64/kernel/entry.S:860
crash_arm64>


Using the following commands, it was determined that the mutex lock is
held by the task with PID 165. The field `__state = 0` indicates that
the task is in the TASK_RUNNING state, while `on_rq = 0` indicates
that the task has been removed from the runqueue. These two fields are
contradictory, preventing the task with PID 165 from ever getting a
chance to run, which is the direct cause of the system hang.

crash_arm64> struct -x mutex.owner 0xffff000006ab5060
  owner = {
    counter = 0xffff0000063ba0c1
  },
crash_arm64> struct
task_struct.comm,pid,__state,thread_info.cpu,se.on_rq,on_rq,on_cpu,blocked_on,blocked_on_state
0xffff0000063ba0c0
  comm = "kworker/u32:22\000",
  pid = 165,
  __state = 0,
  thread_info.cpu = 4
  se.on_rq = 0 '\000',
  on_rq = 0,
  on_cpu = 0,
  blocked_on = 0xffff000006aa3520,
  blocked_on_state = BO_RUNNABLE,
crash_arm64>

The following trace information provides the execution logic of the
task with PID 165 before the system hang occurred.

Line 993:   kworker/u32:20-163   [004]    60.082229: bprint:
    try_to_wake_up: DEBUG BYHP[4179]: p=kworker/u32:22$165, on_cpu=0,
on_rq=2, __state=2, sched_delayed=1
Line 994:   kworker/u32:20-163   [004]    60.082250: bprint:
    try_to_wake_up: DEBUG BYHP[4402]: p=kworker/u32:22$165, on_cpu=0,
on_rq=2, __state=512, sched_delayed=1
Line 995:   kworker/u32:20-163   [004]    60.082268: bprint:
    activate_task: DEBUG BYHP[2107]: p=kworker/u32:22$165, on_cpu=0,
on_rq=1, __state=512, sched_delayed=1
Line 1002:           <idle>-0     [004]    60.087364: bprint:
     pick_task_fair: DEBUG BYHP[5646]: p=kworker/u32:22$165, on_cpu=0,
on_rq=1, __state=0, sched_delayed=1
Line 1003:           <idle>-0     [004]    60.087375: bprint:
     pick_task_fair: DEBUG BYHP[5656]: p=kworker/u32:22$165, on_cpu=0,
on_rq=0, __state=0, sched_delayed=0


The trace information indicates that the task with PID 165 has
`p->se.sched_delayed = 1`. From the trace logs at lines 4179, 4402,
and 2107, it is evident that the task with PID 165 was successfully
removed from the runqueue during the try_to_wake_up process,
reassigned to a new CPU, and successfully completed the enqueue
operation. This behavior seems to align with our expectations.

try_to_wake_up
  |-- ttwu_runnable
      |-- proxy_needs_return    // line: 4179
  |-- cpu = select_task_rq()
  |-- ttwu_queue(p, cpu)        // line: 4402
      |-- __activate_task(p)    // line: 2107


However, the logs at lines 5646 and 5656 reveal that tasks with
`sched_delayed = 1` are removed from the runqueue again during the
pick_next_entity operation. The corresponding code logic in
pick_next_entity is as follows.

struct sched_entity *se = pick_eevdf(cfs_rq);
if (se->sched_delayed) {
dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
/*
* Must not reference @se again, see __block_task().
*/
return NULL;
}

This means that in `pick_task_fair -> pick_next_entity`, a task that
has already been successfully woken up and enqueued will be removed
from the runqueue again if its `sched_delayed = 1`. As a result, the
task will never get a chance to run, ultimately causing the system to
experience hangs.

In the original code logic, the sched_delayed operation is placed
before proxy_needs_return, allowing the `se->sched_delayed` variable
to be set to 0 through enqueue_task(ENQUEUE_DELAYED) ->
enqueue_task_fair -> requeue_delayed_entity. This prevents the already
woken-up task from being removed from the runqueue again in the
pick_next_entity function.

To verify the correctness of the above analysis, we forcibly cleared
`p->se.sched_delayed` after the execution of the proxy_needs_return
function. The corresponding patch is as follows, and with this change,
the system can bootup successfully.

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
old mode 100644
new mode 100755
index 06e9924d3f77..0da8d308ec4b
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c

 #ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4160,6 +4168,15 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  rq = __task_rq_lock(p, &rf);
  if (task_on_rq_queued(p)) {
  update_rq_clock(rq);
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ /*
+ * Force clearing to verify the correctness of the analysis logic
+ */
+ p->se.sched_delayed = 0;
+ goto out;
+ }
+
  if (p->se.sched_delayed) {
  proxy_remove_from_sleeping_owner(p);
  enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
@@ -4171,10 +4188,6 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  */
  wakeup_preempt(rq, p, wake_flags);
  }
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
  ttwu_do_wakeup(p);
  ret = 1;
  }


The above patch was solely intended to verify the correctness of the
analysis, as directly clearing `p->se.sched_delayed` is unreasonable.
Doing so would cause tasks with non-zero lag to be dequeued, thereby
undermining the fairness enforced by EEVDF. Moving the
proxy_needs_return operation after the handling of sched_delayed
appears to be more reasonable, as for a donor that is about to be
dequeued, executing the subsequent wakeup_preempt operation is
redundant. Therefore, the following patch seems reasonable, and I
propose adopting it as version 2.0.

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
old mode 100644
new mode 100755
index 06e9924d3f77..f804c681b861
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
 #ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  proxy_remove_from_sleeping_owner(p);
  enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
  }
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
  if (!task_on_cpu(rq, p)) {
  /*
  * When on_rq && !on_cpu the task is preempted, see if
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  */
  wakeup_preempt(rq, p, wake_flags);
  }
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
  ttwu_do_wakeup(p);
  ret = 1;
  }

However, there is a potential issue. Whether it is the original code
logic or the v2.0 patch mentioned above, directly removing a task from
the runqueue in proxy_needs_return undermines the fairness enforced by
EEVDF. In fact, clearing the task's `p->se.sched_delayed` variable
first (via enqueue_task(ENQUEUE_DELAYED) -> enqueue_task_fair ->
requeue_delayed_entity) and then removing task from the runqueue in
proxy_needs_return is essentially no different from directly removing
the task while retaining its lag. This is because the non-zero lag
will eventually be discarded. Therefore, regardless of the method
used, the fairness maintained by EEVDF will be compromised.

I have not yet determined the best approach to address the issue of
EEVDF fairness being compromised. Perhaps we could preserve the lag
during the dequeue process and handle it during the next enqueue. I
look forward to further discussing this issue with you.

Finally, I have decided to adopt the following patch as version 2.0.
This patch is specifically designed to avoid unnecessary
wakeup_preempt checks. Regarding the fairness issue discussed earlier,
I plan to submit a new patch to address it.

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
old mode 100644
new mode 100755
index 06e9924d3f77..f804c681b861
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
 #ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  proxy_remove_from_sleeping_owner(p);
  enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
  }
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
  if (!task_on_cpu(rq, p)) {
  /*
  * When on_rq && !on_cpu the task is preempted, see if
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  */
  wakeup_preempt(rq, p, wake_flags);
  }
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
  ttwu_do_wakeup(p);
  ret = 1;
  }

Thanks. Looking forward to your response.
hupu

Re: [RESEND RFC V2 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John,
It seems this email might have been overlooked. I am resending it to
bring it back to your attention and continue the discussion.

Looking forward to your response!
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[John Stultz]

On Sun, May 18, 2025 at 7:51 AM hupu <hupu.gm@gmail.com> wrote:
>
> Hi John,
> Sorry for the delayed response.
>
> > Usually I boot qemu with 64 cores, and have found stress testing
> > running the following separate programs frequently uncovers issues:
> >
> > #cyclictest from the rt-test suite to add some rt load
> > cyclictest -t -p90
> >
> > # rerunning the test_ww_mtuex logic (enabled by a patch in my full
> > proxy-exec series)
> > while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests ; sleep 5; done
> >
> > # Along with my prio-inversion-demo (which ensures we do the proxying paths)
> > # ? https://github.com/johnstultz-work/priority-inversion-demo
> > while true; do ./run.sh; sleep 10 ; done
> >
> > # Sometimes I also throw in the ltp sched_football test:
> > # https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/func/sched_football
> > while true; do ./sched_football -m | grep Result ; sleep 1; done
> >
>
>
> I followed your suggested approach and successfully reproduced the
> issue you mentioned. The system experienced a hang during bootup and
> triggered a crash due to a watchdog timeout. This issue is related to
> the logic of sched_delayed. I will provide a detailed description of
> the debugging steps below. I apologize in advance if the explanation
> becomes somewhat lengthy.
>

Yeah. To be honest, this is a little tough to follow, particularly
trying to distinguish between the debug changes and the actual change
you are proposing. Especially as they are whitespace damaged.

[snip]
> The above patch was solely intended to verify the correctness of the
> analysis, as directly clearing `p->se.sched_delayed` is unreasonable.
> Doing so would cause tasks with non-zero lag to be dequeued, thereby
> undermining the fairness enforced by EEVDF. Moving the
> proxy_needs_return operation after the handling of sched_delayed
> appears to be more reasonable, as for a donor that is about to be
> dequeued, executing the subsequent wakeup_preempt operation is
> redundant. Therefore, the following patch seems reasonable, and I
> propose adopting it as version 2.0.
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> old mode 100644
> new mode 100755
> index 06e9924d3f77..f804c681b861
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
>  #ifdef CONFIG_SCHED_PROXY_EXEC
> @@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p,
> int wake_flags)
>   proxy_remove_from_sleeping_owner(p);
>   enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
>   }
> + if (proxy_needs_return(rq, p)) {
> + _trace_sched_pe_return_migration(p);
> + goto out;
> + }
>   if (!task_on_cpu(rq, p)) {
>   /*
>   * When on_rq && !on_cpu the task is preempted, see if
> @@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
> int wake_flags)
>   */
>   wakeup_preempt(rq, p, wake_flags);
>   }
> - if (proxy_needs_return(rq, p)) {
> - _trace_sched_pe_return_migration(p);
> - goto out;
> - }
>   ttwu_do_wakeup(p);
>   ret = 1;
>   }

Ok, so this matches what I had tested before modifying your earlier patch.


> However, there is a potential issue. Whether it is the original code
> logic or the v2.0 patch mentioned above, directly removing a task from
> the runqueue in proxy_needs_return undermines the fairness enforced by
> EEVDF. In fact, clearing the task's `p->se.sched_delayed` variable
> first (via enqueue_task(ENQUEUE_DELAYED) -> enqueue_task_fair ->
> requeue_delayed_entity) and then removing task from the runqueue in
> proxy_needs_return is essentially no different from directly removing
> the task while retaining its lag. This is because the non-zero lag
> will eventually be discarded. Therefore, regardless of the method
> used, the fairness maintained by EEVDF will be compromised.

Hrm. Can you walk me through the specific case you're thinking about here?

Is the idea something like:  a mutex blocked task (not sched_delayed)
gets migrated to a rq, where it acts as a donor so that a lock holder
can be run.
If the lock holder sleeps, it might be set as sched_delayed, but the
donor will be dequeued from the rq and enqueued onto the sched_delayed
sleeping owner.

And the concern is that in doing this, the donor's lag from the rq it
was migrated to won't be preserved (since it isn't set as
sched_delayed)?

I'll need to think on this a bit, as I don't quite have my head around
how mutex blocked tasks might also end up sched_delayed.

> I have not yet determined the best approach to address the issue of
> EEVDF fairness being compromised. Perhaps we could preserve the lag
> during the dequeue process and handle it during the next enqueue. I
> look forward to further discussing this issue with you.
>
> Finally, I have decided to adopt the following patch as version 2.0.
> This patch is specifically designed to avoid unnecessary
> wakeup_preempt checks. Regarding the fairness issue discussed earlier,
> I plan to submit a new patch to address it.
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> old mode 100644
> new mode 100755
> index 06e9924d3f77..f804c681b861
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
>  #ifdef CONFIG_SCHED_PROXY_EXEC
> @@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p,
> int wake_flags)
>   proxy_remove_from_sleeping_owner(p);
>   enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
>   }
> + if (proxy_needs_return(rq, p)) {
> + _trace_sched_pe_return_migration(p);
> + goto out;
> + }
>   if (!task_on_cpu(rq, p)) {
>   /*
>   * When on_rq && !on_cpu the task is preempted, see if
> @@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
> int wake_flags)
>   */
>   wakeup_preempt(rq, p, wake_flags);
>   }
> - if (proxy_needs_return(rq, p)) {
> - _trace_sched_pe_return_migration(p);
> - goto out;
> - }
>   ttwu_do_wakeup(p);
>   ret = 1;
>   }

This looks identical to the version above, or am I missing something?

thanks
-john

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John
Thank you for your response.

>
> This looks identical to the version above, or am I missing something?
>

I sincerely apologize for the confusion caused by my unclear
explanation. The complete patch for version 2.0 is as follows. In this
patch, I relocated the `proxy_needs_return()` check to execute after
`sched_delayed` processing but before the `wakeup_preempt()` judgment.
This optimization allows skipping redundant `wakeup_preempt()`
operations when a donor task must migrate back to its original CPU, as
it becomes unnecessary in such cases.



Subject: [RFC] sched: Skip redundant operations when donor needs return.

Move the proxy_needs_return() check earlier in ttwu_runnable()
to minimize unnecessary operations, particularly in cases
where a donor task needs to migrate back to its original CPU.

Signed-off-by: hupu <hupu.gm@gmail.com>
---
 kernel/sched/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
 mode change 100644 => 100755 kernel/sched/core.c

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
old mode 100644
new mode 100755
index 06e9924d3f77..2c863ad53173
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4164,6 +4164,10 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  proxy_remove_from_sleeping_owner(p);
  enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
  }
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
  if (!task_on_cpu(rq, p)) {
  /*
  * When on_rq && !on_cpu the task is preempted, see if
@@ -4171,10 +4175,6 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
  */
  wakeup_preempt(rq, p, wake_flags);
  }
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
  ttwu_do_wakeup(p);
  ret = 1;
  }
--
2.17.1



> Hrm. Can you walk me through the specific case you're thinking about here?
>
> Is the idea something like:  a mutex blocked task (not sched_delayed)
> gets migrated to a rq, where it acts as a donor so that a lock holder
> can be run.
> If the lock holder sleeps, it might be set as sched_delayed, but the
> donor will be dequeued from the rq and enqueued onto the sched_delayed
> sleeping owner.
>
> And the concern is that in doing this, the donor's lag from the rq it
> was migrated to won't be preserved (since it isn't set as
> sched_delayed)?
>
> I'll need to think on this a bit, as I don't quite have my head around
> how mutex blocked tasks might also end up sched_delayed.
>

I need to add some debugging logs to further investigate this issue.
This may take a bit of time, and I will get back to you shortly.

Thanks.
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John

>
> > However, there is a potential issue. Whether it is the original code
> > logic or the v2.0 patch mentioned above, directly removing a task from
> > the runqueue in proxy_needs_return undermines the fairness enforced by
> > EEVDF. In fact, clearing the task's `p->se.sched_delayed` variable
> > first (via enqueue_task(ENQUEUE_DELAYED) -> enqueue_task_fair ->
> > requeue_delayed_entity) and then removing task from the runqueue in
> > proxy_needs_return is essentially no different from directly removing
> > the task while retaining its lag. This is because the non-zero lag
> > will eventually be discarded. Therefore, regardless of the method
> > used, the fairness maintained by EEVDF will be compromised.
>
> Hrm. Can you walk me through the specific case you're thinking about here?
>
> Is the idea something like:  a mutex blocked task (not sched_delayed)
> gets migrated to a rq, where it acts as a donor so that a lock holder
> can be run.
> If the lock holder sleeps, it might be set as sched_delayed, but the
> donor will be dequeued from the rq and enqueued onto the sched_delayed
> sleeping owner.
>
> And the concern is that in doing this, the donor's lag from the rq it
> was migrated to won't be preserved (since it isn't set as
> sched_delayed)?
>
> I'll need to think on this a bit, as I don't quite have my head around
> how mutex blocked tasks might also end up sched_delayed.
>

There may have been an issue with my earlier analysis, and I apologize
for my oversight.

During previous debugging, I noticed that when handling the
sched_delayed logic in the ttwu_runnable function, there were
instances where the donor's vlag was reset to zero. This raised
concerns for me, as I worried that directly resetting the vlag might
compromise the fairness ensured by EEVDF.

However, upon closer inspection of the code, I realized that this is
actually a normal adjustment of the vlag. The purpose of this process
is to check whether the sched_delayed task has repaid its past "debt"
and determine whether its position in the red-black tree needs to be
updated. The corresponding call stack is as follows:

ttwu_runnable
  |-- if (p->se.sched_delayed) enqueue_task(flags=ENQUEUE_DELAYED)
      |-- enqueue_task_fair(flags=ENQUEUE_DELAYED)
          |-- requeue_delayed_entity(se)
              |-- update_entity_lag()
                  |-- se->vlag = entity_lag(avg_vruntime(cfs_rq), se);

Therefore, the logic in ttwu_runnable does not compromise the fairness of EEVDF.


However, the above call stack gives me another idea: Executing the
sched_delayed logic seems redundant if a donor task needs to return
back. This is because the two main works in the sched_delayed logic —
"updating the vlag" and "adjusting the task's position in the
red-black tree" — can be deferred to later stages, as shown in the
call stack below:

a) STEP#1: Update the donor's vlag.
b) STEP#2: Update the donor's vruntime and deadline.
c) STEP#3: Update the donor's position in the red-black tree.

try_to_wake_up
  |-- ttwu_runnable
  |   |-- proxy_needs_return
  |       |-- deactivate_task
  |           |-- dequeue_task
  |               |-- dequeue_task_fair
  |                   |-- dequeue_entities
  |                       |-- dequeue_entity
  |                           |-- update_entity_lag         (STEP #1)
  |                               |-- se->vlag = entity_lag()
  |
  |-- new_cpu = select_task_rq  //Re-select a new CPU for the donor task
  |-- ttwu_queue(p, new_cpu)
      |-- ttwu_do_activate
          |-- activate_task(flags=ENQUEUE_WAKEUP)
              |-- __activate_task(flags=ENQUEUE_WAKEUP)
                  |-- enqueue_task(flags=ENQUEUE_WAKEUP)
                      |-- enqueue_task_fair(flags=ENQUEUE_WAKEUP)
                          |-- enqueue_entity(flags=ENQUEUE_WAKEUP)
                              |-- place_entity              (STEP #2)
                              |-- __enqueue_entity          (STEP #3)


In other words, if the donor task needs to be migrated back, the
sched_delayed logic can be skipped entirely. Therefore, the final
PATCH should be as follows.
In this PATCH, proxy_needs_return is moved ahead of the sched_delayed
logic, and the p->se.sched_delayed variable is forcefully set to 0.

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
old mode 100644
new mode 100755
index 06e9924d3f77..f585e28b19eb
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4160,6 +4160,18 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
        rq = __task_rq_lock(p, &rf);
        if (task_on_rq_queued(p)) {
                update_rq_clock(rq);
+               if (proxy_needs_return(rq, p)) {
+                       /*
+                        * Force clear, as the work of updating the donor's vlag
+                        * and its position in the red-black tree can
be deferred
+                        * to later stages.
+                        */
+                       p->se.sched_delayed = 0;
+
+                       _trace_sched_pe_return_migration(p);
+                       goto out;
+               }
+
                if (p->se.sched_delayed) {
                        proxy_remove_from_sleeping_owner(p);
                        enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
                         */
                        wakeup_preempt(rq, p, wake_flags);
                }
-               if (proxy_needs_return(rq, p)) {
-                       _trace_sched_pe_return_migration(p);
-                       goto out;
-               }
                ttwu_do_wakeup(p);
                ret = 1;
        }


This PATCH can boot up successfully. Furthermore, I triggered the
mutex-related test cases by executing the following command, and it
has been running for approximately 20 hours without any issues.
# while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests; sleep 10; done


Looking forward to your reply. Thanks.
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John,
It has been a few days without any response to this email. I'm
resending it for further discussion.

Best regards,
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[John Stultz]

On Tue, Jun 10, 2025 at 7:32 PM hupu <hupu.gm@gmail.com> wrote:
>
> Hi John,
> It has been a few days without any response to this email. I'm
> resending it for further discussion.

Apologies, I'm swamped at the moment with some other items and haven't
had time to give your suggestions enough serious thought to reply. :(

I have made some slow progress on a few items. Suggested by Suleiman,
I've reworked the migration logic to be considered after the owner
on_rq check, so we avoid migrating tasks to only realize they are
waiting on a sleeping owner and pulling them off the rq. This should
help reduce the number of migrations we make.

Unfortunately I've also been tripping over a crash with the full
series (not the portion I'm currently submitting upstream) that often
takes 30+ hours to reproduce and extra debugging logic seems to make
it even harder to hit. With all the other tasks, keeping enough
context in my head to make progress on this has been frustratingly
slow.

I'll hopefully have something to share soon and I can

thanks
-john

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John
Thank you for your response.

On Wed, Jun 11, 2025 at 1:05 PM John Stultz <jstultz@google.com> wrote:
>
> On Tue, Jun 10, 2025 at 7:32 PM hupu <hupu.gm@gmail.com> wrote:
> >
> > Hi John,
> > It has been a few days without any response to this email. I'm
> > resending it for further discussion.
>
> Apologies, I'm swamped at the moment with some other items and haven't
> had time to give your suggestions enough serious thought to reply. :(
>
> I have made some slow progress on a few items. Suggested by Suleiman,
> I've reworked the migration logic to be considered after the owner
> on_rq check, so we avoid migrating tasks to only realize they are
> waiting on a sleeping owner and pulling them off the rq. This should
> help reduce the number of migrations we make.
>
> Unfortunately I've also been tripping over a crash with the full
> series (not the portion I'm currently submitting upstream) that often
> takes 30+ hours to reproduce and extra debugging logic seems to make
> it even harder to hit. With all the other tasks, keeping enough
> context in my head to make progress on this has been frustratingly
> slow.
>

If you'd like, I'd be happy to help investigate this crash issue. :)

To do that, I would need some additional information, such as the
vmlinux file, coredump, and the source code at the point where the
crash is triggered.

> I'll hopefully have something to share soon and I can
>

Look forward to hearing from you with good news.


Thanks.
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[hupu]

Hi John

> Unfortunately I've also been tripping over a crash with the full
> series (not the portion I'm currently submitting upstream) that often
> takes 30+ hours to reproduce and extra debugging logic seems to make
> it even harder to hit.
>

I encountered a strange crash while running the code from the
proxy-exec-v17-WIP branch for over 150+ hours. I'm currently unsure if
this issue is the same as the one you mentioned.

Here is the dmesg log at the time of the crash:

[550195.352234][    C2] watchdog: BUG: soft lockup - CPU#2 stuck for
32s! [lock_torture_wr:139]
[550195.374670][    C2] Modules linked in:
[550195.375936][    C2] irq event stamp: 189580674
[550195.376275][    C2] hardirqs last  enabled at (189580673):
[<ffff8000813b7840>] exit_to_kernel_mode.isra.6+0xf8/0x130
[550195.393675][    C2] hardirqs last disabled at (189580674):
[<ffff8000813b78a4>] el1_interrupt+0x2c/0x70
[550195.393779][    C2] softirqs last  enabled at (189580644):
[<ffff80008005f7dc>] handle_softirqs+0x4b4/0x4cc
[550195.396778][    C2] softirqs last disabled at (189580639):
[<ffff800080010654>] __do_softirq+0x18/0x20
[550195.406109][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
Not tainted 6.12.0-00050-gf6c6e0ea8a1d-dirty #133
[550195.406661][    C2] Hardware name: linux,dummy-virt (DT)
[550195.407172][    C2] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO
-DIT -SSBS BTYPE=--)
[550195.407396][    C2] pc : queued_spin_lock_slowpath+0x270/0x528
[550195.408270][    C2] lr : queued_spin_lock_slowpath+0x3a8/0x528
[550195.408358][    C2] sp : ffff800084d83d20
[550195.408494][    C2] x29: ffff800084d83d20 x28: 28f5c28f5c28f5c3
x27: 0000000000000000
[550195.408985][    C2] x26: ffff800081ef3000 x25: 0000000000000000
x24: 0000000000000002
[550195.409158][    C2] x23: ffff800083532000 x22: ffff800082b55000
x21: ffff00002c286740
[550195.409322][    C2] x20: ffff800081ef3000 x19: ffff80008355b348
x18: ffff800081ce6e48
[550195.409522][    C2] x17: ffff7fffa954e000 x16: 000000006c0014e7
x15: ffff8000835321f0
[550195.409681][    C2] x14: 0001f45e95248cf0 x13: 0000000000000100
x12: 0000000000000000
[550195.409875][    C2] x11: 00000000002dc6c0 x10: 0000000000000000 x9
: ffff7fffa9731000
[550195.410102][    C2] x8 : ffff800081ef3fb8 x7 : 0000000000000000 x6
: 00000000000c0000
[550195.410262][    C2] x5 : ffff8000835365e8 x4 : 0000000000000000 x3
: 0000000000000000
[550195.410451][    C2] x2 : 0000000000000000 x1 : ffff00002c286740 x0
: ffff00002c286748
[550195.410849][    C2] Call trace:
[550195.410995][    C2]  queued_spin_lock_slowpath+0x270/0x528
[550195.411265][    C2]  do_raw_spin_lock+0xac/0x11c
[550195.421183][    C2]  _raw_spin_lock+0x74/0x90
[550195.421264][    C2]  torture_spin_lock_write_lock+0x1c/0x28
[550195.421333][    C2]  lock_torture_writer+0x104/0x2f4
[550195.421397][    C2]  kthread+0x128/0x134
[550195.421600][    C2]  ret_from_fork+0x10/0x20
[550195.422007][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for
32s! [lock_torture_wr:144]
[550195.422780][    C2] Kernel panic - not syncing: softlockup: hung tasks
[550195.423616][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
Tainted: G             L     6.12.0-00050-gf6c6e0ea8a1d-dirty #133
[550195.423813][    C2] Tainted: [L]=SOFTLOCKUP
[550195.423929][    C2] Hardware name: linux,dummy-virt (DT)
[550195.424123][    C2] Call trace:
[550195.424174][    C2]  dump_backtrace.part.8+0x8c/0xe8
[550195.424232][    C2]  show_stack+0x2c/0x38
[550195.424256][    C2]  dump_stack_lvl+0x6c/0xcc
[550195.424999][    C2]  dump_stack+0x14/0x1c
[550195.425050][    C2]  panic+0x150/0x428
[550195.425121][    C2]  watchdog_timer_fn+0x258/0x2e8
[550195.426192][    C2]  __hrtimer_run_queues+0x3bc/0x4a0
[550195.426651][    C2]  hrtimer_interrupt+0xd8/0x248
[550195.426680][    C2]  arch_timer_handler_virt+0x38/0x58
[550195.427608][    C2]  handle_percpu_devid_irq+0xac/0x2cc
[550195.427676][    C2]  generic_handle_domain_irq+0x40/0x58
[550195.427743][    C2]  gic_handle_irq+0x64/0x124
[550195.427771][    C2]  call_on_irq_stack+0x24/0x34
[550195.427832][    C2]  do_interrupt_handler+0x90/0x94
[550195.427865][    C2]  el1_interrupt+0x3c/0x70
[550195.427899][    C2]  el1h_64_irq_handler+0x14/0x1c
[550195.427960][    C2]  el1h_64_irq+0x64/0x68
[550195.427989][    C2]  queued_spin_lock_slowpath+0x270/0x528
[550195.428021][    C2]  do_raw_spin_lock+0xac/0x11c
[550195.428051][    C2]  _raw_spin_lock+0x74/0x90
[550195.428082][    C2]  torture_spin_lock_write_lock+0x1c/0x28
[550195.428115][    C2]  lock_torture_writer+0x104/0x2f4
[550195.428147][    C2]  kthread+0x128/0x134
[550195.428182][    C2]  ret_from_fork+0x10/0x20
[550195.428564][    C2] SMP: stopping secondary CPUs
[550195.448256][    C2] Kernel Offset: disabled
[550195.450640][    C2] CPU features: 0x00,00000019,40203020,560042af
[550195.451534][    C2] Memory Limit: none
[550195.452853][    C2] ---[ end Kernel panic - not syncing:
softlockup: hung tasks ]---


At the time of the crash, all CPU call stacks were identical. From the
call stack at level #15, it is evident that all threads are waiting
for the `torture_spinlock` lock in kernel/locking/locktorture.c.

-000|cpu_park_loop(inline)
-000|local_cpu_stop(cpu = 0)
-001|panic_smp_self_stop()
-002|panic(:fmt = 0xFFFF800081CFC838)
-003|watchdog_timer_fn(:hrtimer = 0xFFFF00002BEAEE20)
-004|__run_hrtimer(inline)
-004|__hrtimer_run_queues(:cpu_base = 0xFFFF00002BEAE0C0, :now =
550186225371571, :flags = 192, :active_mask = 15)
-005|hrtimer_interrupt(:dev = 0xFFFF00002BEC4A80)
-006|timer_handler(inline)
-006|arch_timer_handler_virt(:irq = 11, dev_id = 0xFFFF00002BEC4A80)
-007|handle_percpu_devid_irq(desc = 0xFFFF000004C2C400)
-008|generic_handle_irq_desc(inline)
-008|handle_irq_desc(inline)
-008|generic_handle_domain_irq(domain = 0xFFFF000004E50000, :hwirq = 27)
-009|__gic_handle_irq(inline)
-009|__gic_handle_irq_from_irqson(inline)
-009|gic_handle_irq(regs = ?)
-010|call_on_irq_stack(asm)
-011|do_interrupt_handler(:regs = 0xFFFF800084D4BBD0, :handler =
0xFFFF80008001020C)
-012|__el1_irq(inline)
-012|el1_interrupt(regs = 0xFFFF800084D4BBD0, handler = 0xFFFF80008001020C)
-013|el1h_64_irq_handler(regs = ?)
-014|el1h_64_irq(asm)
-015|trace_contention_begin(inline)
-015|queued_spin_lock_slowpath(:lock = 0xFFFF80008355B348, val = ?)
-016|queued_spin_lock(inline)
-016|do_raw_spin_lock(:lock = 0xFFFF80008355B348)
-017|__raw_spin_lock(inline)
-017|_raw_spin_lock(:lock = 0xFFFF80008355B348)
-018|spin_lock(inline)
-018|torture_spin_lock_write_lock(:tid = 0)
-019|lock_torture_writer(:arg = 0xFFFF0000060A4800)
-020|kthread(_create = 0xFFFF000006074500)
-021|ret_from_fork(asm)
-022|NSX:0x0::0xFFFFFFFFFF5F8000(asm)
 ---|end of frame

crash_arm64> sym torture_spinlock
ffff80008355b348 (d) torture_spinlock
crash_arm64>


However, based on the crash site, it appears that no thread is
currently holding the `torture_spinlock` lock, yet other threads on
different CPUs are still waiting for this lock, which is confusing to
me.

crash_arm64> struct -x  raw_spinlock 0xFFFF80008355B348
struct raw_spinlock {
  raw_lock = {
    {
      val = {
        counter = 0x180000
      },
      {
        locked = 0x0,
        pending = 0x0
      },
      {
        locked_pending = 0x0,
        tail = 0x18
      }
    }
  },
  magic = 0xdead4ead,
  owner_cpu = 0xffffffff,
  owner = 0xffffffffffffffff,
  dep_map = {
    key = 0xffff80008355b360 <torture_spinlock+24>,
    class_cache = {0xffff8000841d9e58 <lock_classes+127248>, 0x0},
    name = 0xffff800081cebe90 "torture_spinlock",
    wait_type_outer = 0x0,
    wait_type_inner = 0x3,
    lock_type = 0x0,
    cpu = 0x5,
    ip = 0xffff8000800f30b8
  }
}
crash_arm64>

Although I haven't yet pinpointed the root cause of this issue, I
would like to confirm whether the problem I'm encountering is the same
as the one you described.

If possible, could you share the dmesg log and call stack information
of the crash you encountered?

Thanks.
hupu

Re: [RFC 1/1] sched: Skip redundant operations for proxy tasks needing return migration

[John Stultz]

On Thu, Jun 12, 2025 at 2:57 AM hupu <hupu.gm@gmail.com> wrote:
>
> Hi John
>
> > Unfortunately I've also been tripping over a crash with the full
> > series (not the portion I'm currently submitting upstream) that often
> > takes 30+ hours to reproduce and extra debugging logic seems to make
> > it even harder to hit.
> >
>
> I encountered a strange crash while running the code from the
> proxy-exec-v17-WIP branch for over 150+ hours. I'm currently unsure if
> this issue is the same as the one you mentioned.
>
> Here is the dmesg log at the time of the crash:
>
> [550195.352234][    C2] watchdog: BUG: soft lockup - CPU#2 stuck for
> 32s! [lock_torture_wr:139]
> [550195.374670][    C2] Modules linked in:
> [550195.375936][    C2] irq event stamp: 189580674
> [550195.376275][    C2] hardirqs last  enabled at (189580673):
> [<ffff8000813b7840>] exit_to_kernel_mode.isra.6+0xf8/0x130
> [550195.393675][    C2] hardirqs last disabled at (189580674):
> [<ffff8000813b78a4>] el1_interrupt+0x2c/0x70
> [550195.393779][    C2] softirqs last  enabled at (189580644):
> [<ffff80008005f7dc>] handle_softirqs+0x4b4/0x4cc
> [550195.396778][    C2] softirqs last disabled at (189580639):
> [<ffff800080010654>] __do_softirq+0x18/0x20
> [550195.406109][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
> Not tainted 6.12.0-00050-gf6c6e0ea8a1d-dirty #133
> [550195.406661][    C2] Hardware name: linux,dummy-virt (DT)
> [550195.407172][    C2] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO
> -DIT -SSBS BTYPE=--)
> [550195.407396][    C2] pc : queued_spin_lock_slowpath+0x270/0x528
> [550195.408270][    C2] lr : queued_spin_lock_slowpath+0x3a8/0x528
> [550195.408358][    C2] sp : ffff800084d83d20
> [550195.408494][    C2] x29: ffff800084d83d20 x28: 28f5c28f5c28f5c3
> x27: 0000000000000000
> [550195.408985][    C2] x26: ffff800081ef3000 x25: 0000000000000000
> x24: 0000000000000002
> [550195.409158][    C2] x23: ffff800083532000 x22: ffff800082b55000
> x21: ffff00002c286740
> [550195.409322][    C2] x20: ffff800081ef3000 x19: ffff80008355b348
> x18: ffff800081ce6e48
> [550195.409522][    C2] x17: ffff7fffa954e000 x16: 000000006c0014e7
> x15: ffff8000835321f0
> [550195.409681][    C2] x14: 0001f45e95248cf0 x13: 0000000000000100
> x12: 0000000000000000
> [550195.409875][    C2] x11: 00000000002dc6c0 x10: 0000000000000000 x9
> : ffff7fffa9731000
> [550195.410102][    C2] x8 : ffff800081ef3fb8 x7 : 0000000000000000 x6
> : 00000000000c0000
> [550195.410262][    C2] x5 : ffff8000835365e8 x4 : 0000000000000000 x3
> : 0000000000000000
> [550195.410451][    C2] x2 : 0000000000000000 x1 : ffff00002c286740 x0
> : ffff00002c286748
> [550195.410849][    C2] Call trace:
> [550195.410995][    C2]  queued_spin_lock_slowpath+0x270/0x528
> [550195.411265][    C2]  do_raw_spin_lock+0xac/0x11c
> [550195.421183][    C2]  _raw_spin_lock+0x74/0x90
> [550195.421264][    C2]  torture_spin_lock_write_lock+0x1c/0x28
> [550195.421333][    C2]  lock_torture_writer+0x104/0x2f4
> [550195.421397][    C2]  kthread+0x128/0x134
> [550195.421600][    C2]  ret_from_fork+0x10/0x20
> [550195.422007][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for
> 32s! [lock_torture_wr:144]
> [550195.422780][    C2] Kernel panic - not syncing: softlockup: hung tasks
> [550195.423616][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
> Tainted: G             L     6.12.0-00050-gf6c6e0ea8a1d-dirty #133
> [550195.423813][    C2] Tainted: [L]=SOFTLOCKUP
> [550195.423929][    C2] Hardware name: linux,dummy-virt (DT)
> [550195.424123][    C2] Call trace:
> [550195.424174][    C2]  dump_backtrace.part.8+0x8c/0xe8
> [550195.424232][    C2]  show_stack+0x2c/0x38
> [550195.424256][    C2]  dump_stack_lvl+0x6c/0xcc
> [550195.424999][    C2]  dump_stack+0x14/0x1c
> [550195.425050][    C2]  panic+0x150/0x428
> [550195.425121][    C2]  watchdog_timer_fn+0x258/0x2e8
> [550195.426192][    C2]  __hrtimer_run_queues+0x3bc/0x4a0
> [550195.426651][    C2]  hrtimer_interrupt+0xd8/0x248
> [550195.426680][    C2]  arch_timer_handler_virt+0x38/0x58
> [550195.427608][    C2]  handle_percpu_devid_irq+0xac/0x2cc
> [550195.427676][    C2]  generic_handle_domain_irq+0x40/0x58
> [550195.427743][    C2]  gic_handle_irq+0x64/0x124
> [550195.427771][    C2]  call_on_irq_stack+0x24/0x34
> [550195.427832][    C2]  do_interrupt_handler+0x90/0x94
> [550195.427865][    C2]  el1_interrupt+0x3c/0x70
> [550195.427899][    C2]  el1h_64_irq_handler+0x14/0x1c
> [550195.427960][    C2]  el1h_64_irq+0x64/0x68
> [550195.427989][    C2]  queued_spin_lock_slowpath+0x270/0x528
> [550195.428021][    C2]  do_raw_spin_lock+0xac/0x11c
> [550195.428051][    C2]  _raw_spin_lock+0x74/0x90
> [550195.428082][    C2]  torture_spin_lock_write_lock+0x1c/0x28
> [550195.428115][    C2]  lock_torture_writer+0x104/0x2f4
> [550195.428147][    C2]  kthread+0x128/0x134
> [550195.428182][    C2]  ret_from_fork+0x10/0x20
> [550195.428564][    C2] SMP: stopping secondary CPUs
> [550195.448256][    C2] Kernel Offset: disabled
> [550195.450640][    C2] CPU features: 0x00,00000019,40203020,560042af
> [550195.451534][    C2] Memory Limit: none
> [550195.452853][    C2] ---[ end Kernel panic - not syncing:
> softlockup: hung tasks ]---
>

So, while I'm not sure if this is the same issue, I have found fixes
for a few issues (but unfortunately not all) I was hitting with v17.

Not exactly what your seeing, but  I was seeing some issues where due
to a spurious wakeup we would end up with an owner that was woken but
was still in the BO_WAKING state and hadn't been return migrated. This
could cause us to get stuck looping in and out of find_proxy_task(),
which could cause tasks to lock up in a similar fashion.  So I've
re-added the return migration for that case and it seems to be
resolved.

I've not seen a case where a spinlock was unowned but everyone was
spinning on it (particularly because my patch series doesn't really
mess with spinlocks, assuming the runnable/spinning tasks get
scheduled to run). The owner being stuck and not letting the lock go
seems more likely, so that's curious.

I've got my current WIP set here:
  https://github.com/johnstultz-work/linux-dev/commits/proxy-exec-WIP/
(Still need to write proper commit messages for many of the new changes)

The problem I see right now, which can sometimes take 30+ hours to
trip on an 64core qemu system running cyclictest, my
prio-inversion-demo test, and running the in-kernel test_ww_mutex
tests in loops.

The backtrace will look something like:
[18071.849807] ------------[ cut here ]------------
[18071.849829]
[18071.849830] ======================================================
[18071.849831] WARNING: possible circular locking dependency detected
[18071.849832] 6.12.0-00060-g6b88c4fed2ba #1362 Not tainted
[18071.849834] ------------------------------------------------------
[18071.849835] kworker/u256:8/940 is trying to acquire lock:
[18071.849837] ffffffff82ffaf38 ((console_sem).lock){-...}-{2:2}
[18071.849887] BUG: unable to handle page fault for address: 0000004800153ad9
[18071.878733] #PF: supervisor read access in kernel mode
[18071.881809] #PF: error_code(0x0000) - not-present page
[18071.884840] PGD 0 P4D 0
[18071.886351] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
[18071.889305] CPU: 11 UID: 0 PID: 940 Comm: kworker/u256:8 Not
tainted 6.12.0-00060-g6b88c4fed2ba #1362
[18071.894553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[18071.900009] Workqueue: test-ww_mutex stress_one_work
[18071.902902] RIP: 0010:pick_next_task_fair+0x2c5/0x3f0
[18071.905860] Code: b5 90 00 00 00 45 39 f7 7e 5c 49 8b bc 24 a0 00
00 00 4c 89 e6 e8 4b a3 ff ff 4d 8b a4 24 98 00 00 00 4d 8b bc 24 a0
00 00 00 <49> 8b bd a0 00 00 00 49 39 ff 75 c0 4d 85 ff 74 bb 4c 89 ee
4c 89
[18071.916706] RSP: 0018:ffffc90002877c38 EFLAGS: 00010006
[18071.919729] RAX: ffff888105810000 RBX: ffff88810087a140 RCX: 0000000000000000
[18071.923810] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[18071.927894] RBP: ffff8881b9c6fa00 R08: 0000000000000000 R09: ffff888103fbc280
[18071.931983] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff810ec0c0
[18071.936051] R13: 0000004800153a39 R14: 0000000000000000 R15: 00441f0ffa1e0ff3
[18071.940143] FS:  0000000000000000(0000) GS:ffff8881b90c0000(0000)
knlGS:0000000000000000
[18071.944776] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18071.948085] CR2: 0000004800153ad9 CR3: 0000000112512004 CR4: 0000000000370ef0
[18071.952210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18071.956277] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[18071.960375] Call Trace:
[18071.961826]  <TASK>
[18071.963097]  ? __die+0x56/0x92
[18071.964919]  ? page_fault_oops+0xca/0x260
[18071.967246]  ? down_trylock+0x13/0x40
[18071.969400]  ? _printk+0x6c/0x87
[18071.971246]  ? __down_trylock_console_sem+0x23/0x90
[18071.974005]  ? exc_page_fault+0x525/0x8e0
[18071.976276]  ? asm_exc_page_fault+0x26/0x30
[18071.978614]  ? put_prev_entity+0x70/0xb0
[18071.980861]  ? pick_next_task_fair+0x2c5/0x3f0
[18071.983409]  ? pick_next_task_fair+0x309/0x3f0
[18071.985977]  __schedule+0x22f/0x20c0
[18071.988041]  ? lock_release+0x16d/0x2e0
[18071.990248]  schedule+0x3d/0x150
[18071.992197]  schedule_hrtimeout_range_clock+0x110/0x160
[18071.995155]  ? __pfx_hrtimer_wakeup+0x10/0x10
[18071.997679]  ? usleep_range_state+0x59/0xc0
[18072.000063]  usleep_range_state+0x87/0xc0
[18072.002341]  stress_one_work+0x41/0x90
[18072.004465]  process_one_work+0x204/0x5a0
[18072.006793]  ? lock_is_held_type+0xca/0x130
[18072.009195]  worker_thread+0x18e/0x350
[18072.011328]  ? __pfx_worker_thread+0x10/0x10
[18072.013773]  kthread+0xee/0x120
[18072.015593]  ? __pfx_kthread+0x10/0x10
[18072.017805]  ret_from_fork+0x31/0x50
[18072.019803]  ? __pfx_kthread+0x10/0x10
[18072.021948]  ret_from_fork_asm+0x1a/0x30
[18072.024214]  </TASK>

Which is due to tripping the:
  SCHED_WARN_ON(cfs_rq->curr != prev);
check in put_prev_entity().

And then hitting some sort of memory trouble as the lockdep warning is
being printed.

I suspect the issue is that the cfs_rq->curr (which should match the
rq->donor set from the previous pick_next_task()) ends up as a
dangling reference when we migrate or otherwise re-do the pick_again
loop.

Unfortunately as I add more trace_printks to debug the issue, it
becomes even more rare to trip, or the log somehow ends up incomplete
(missing final lines I'd expect given the backtrace in the splat),  so
it has been slow progress. :(

Anyway, let me know if you run into anything similar or have ideas.
I'll be continuing to try to trap the behavior better, running over
this weekend.

thanks
-john