[PATCH v30 0/7] Sleeping Owner Handling for Proxy Execution (v30)

[PATCH v30 0/7] Sleeping Owner Handling for Proxy Execution (v30)

[John Stultz]

Hey All,

With some nice progress made recently, I wanted to move on to
sending out the next major component of Proxy Execution:
Sleeping Owner Handling.

As always, I’m trying to submit this larger work in smallish
digestible pieces. A quick overview of the journey so far:
1) prep patches
2) single rq proxying
3) simple donor migration
4) optimized donor migration
5) sleeping owner handling  <- We are here!
6) chain level balancing
7) proxy rwsems
8) ...

In this chunk, there is more than just sleeping owner handling:
I'm including some core-scheduling fixes from Vasily Gorbik and
K Prateek, as well as an optimization to do chain migration left
over from the last chunk.

But the substantial part of this chunk is the sleeping owner
handling, which addresses what to do when a task is blocked on a
mutex who’s owner is sleeping. Since there is nothing we can do
to boost the sleeping owner at that point, we instead deactivate
and queue the waiter on a list attached to the owner. Then when
the owner wakes up, we will activate the waiters on the same
runqueue, so they can then boost the owner to run.

But since the simple sounding things hide subtlety, this ends up
being a pretty complex bit of logic. You can effectively get
trees of waiters, and its possible to have tasks in the middle
of the tree be woken up. Then handling wakeups as they cascade
down the tree properly is also a bit complex, as there are extra
lists to manage the recursive style work without actually
recursing.

Hopefully the earlier patches will be easy to queue, but I
suspect there will be lots of review feedback for me to address
in the last one.  :)

I'd love to get further feedback on any place where these
patches are confusing, or could use additional clarifications.

Additionally I’d appreciate any testing or comments that folks
have with the full Proxy Execution series! You can find the full
Proxy Exec series here:
  https://github.com/johnstultz-work/linux-dev/commits/proxy-exec-v30-7.2-rc1/
  https://github.com/johnstultz-work/linux-dev.git proxy-exec-v30-7.2-rc1


New changes to the full patch series in this revision include:
* Rebased on top of Peter’s cleanups and improvements which
  landed in 7.2-rc1

* Added an optimization to the activate_blocked_waiters logic
  which was causing performance impact compared to the !proxy
  case

* Pulled in some of Andrea Righi’s sched_ext + proxy
  compatibility improvements (I’ve unfortunately not had much
  time to focus on this yet, so I’ve not done much testing!)

Issues still to address with the full series:
* Need to do more work integrating and testing Andrea’s
  sched_ext changes

* Reevaluate performance regression K Prateek Nayak found with
  the full series.

* The chain migration functionality needs further iterations and
  better validation to ensure it truly maintains the RT/DL load
  balancing invariants (despite this being broken in vanilla
  upstream with RT_PUSH_IPI currently)

Future work:
* Expand to more locking primitives: Suleiman is looking at
  pi-futexes, and using proxy for Binder PI is something else
  we’re exploring.

* Eventually: Work to replace rt_mutexes and get things happy
  with PREEMPT_RT


Credit/Disclaimer:
—--------------------
As always, this Proxy Execution series has a long history with
lots of developers that deserve credit:

First described in a paper[1] by Watkins, Straub, Niehaus, then
from patches from Peter Zijlstra, extended with lots of work by
Juri Lelli, Valentin Schneider, and Connor O'Brien. (and thank
you to Steven Rostedt for providing additional details here!).
Thanks also to Joel Fernandes, Dietmar Eggemann, Metin Kaya,
K Prateek Nayak and Suleiman Souhlal for their substantial
review, suggestion, and patch contributions.

So again, many thanks to those above, as all the credit for this
series really is due to them - while the mistakes are surely
mine.

Thanks so much!
-john

[1] https://static.lwn.net/images/conf/rtlws11/papers/proc/p38.pdf

Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com

John Stultz (4):
  sched/core: Avoid migrating blocked_on tasks
  sched: Switch rq->next_class in proxy_reset_donor()
  sched: Break out core of attach_tasks() helper into sched.h
  sched: Migrate whole chain in proxy_migrate_task()

Peter Zijlstra (1):
  sched: Add deactivated (sleeping) owner handling to find_proxy_task()

Vasily Gorbik (2):
  sched/core: Don't steal a proxy-exec donor
  sched/core: Don't proxy-exec unmatched cookie lock owners

 include/linux/sched.h |   7 +
 init/init_task.c      |   6 +
 kernel/fork.c         |   6 +
 kernel/sched/core.c   | 338 ++++++++++++++++++++++++++++++++++++++----
 kernel/sched/fair.c   |  16 +-
 kernel/sched/sched.h  |  19 +++
 6 files changed, 349 insertions(+), 43 deletions(-)

-- 
2.55.0.rc0.799.gd6f94ed593-goog

[PATCH v30 1/7] sched/core: Don't steal a proxy-exec donor

[John Stultz]

From: Vasily Gorbik <gor@linux.ibm.com>

try_steal_cookie() avoids stealing src->core_pick and src->curr before
moving a task with the same cookie via move_queued_task_locked().

With proxy-exec, src->donor is the current scheduling context and may
differ from src->curr. Stealing it migrates a task that the source rq
still treats as current, leaving src's scheduler state for that task
stale. For CFS this means cfs_rq->curr points at the stolen entity,
and the next pick on the source rq hits the WARN_ON_ONCE in
put_prev_entity().

Commit 7de9d4f94638 ("sched: Start blocked_on chain processing in
find_proxy_task()") tweaked the fair class logic so that the donor task
isn't migrated away while we're running the proxy. Do it similarly for
try_steal_cookie() and skip src->donor as well.

Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()")
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com
---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 96226707c2f61..e7074ba54a91f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6474,7 +6474,7 @@ static bool try_steal_cookie(int this, int that)
 		return false;
 
 	do {
-		if (p == src->core_pick || p == src->curr)
+		if (p == src->core_pick || p == src->curr || p == src->donor)
 			goto next;
 
 		if (!is_cpu_allowed(p, this))
-- 
2.55.0.rc0.799.gd6f94ed593-goog

[PATCH v30 2/7] sched/core: Avoid migrating blocked_on tasks

[John Stultz]

Its not useful to migrate blocked_on tasks from other
runqueues as the proxy logic will just migrate it
back to the owner's rq. So skip blocked_on tasks here.

Signed-off-by: John Stultz <jstultz@google.com>
---
Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com
---
 kernel/sched/core.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e7074ba54a91f..8e661b5f133d7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6477,6 +6477,9 @@ static bool try_steal_cookie(int this, int that)
 		if (p == src->core_pick || p == src->curr || p == src->donor)
 			goto next;
 
+		if (task_is_blocked(p))
+			goto next;
+
 		if (!is_cpu_allowed(p, this))
 			goto next;
 
-- 
2.55.0.rc0.799.gd6f94ed593-goog

[PATCH v30 3/7] sched/core: Don't proxy-exec unmatched cookie lock owners

[John Stultz]

From: Vasily Gorbik <gor@linux.ibm.com>

Core scheduling chooses a core-wide cookie before __schedule()
installs the next task. With proxy-exec enabled, that task becomes the
donor/scheduling context, and find_proxy_task() may then replace the
execution context with the runnable mutex owner. If its cookie differs
from the selected core cookie, running it would bypass core scheduling's
cookie selection.

When the final mutex owner found by find_proxy_task() does not match the
selected core cookie, stop proxying the donor. If the current execution
context is already in the blocked chain, fall back to idle like the
existing proxy-exec retry paths do. Otherwise deactivate the donor and
let __schedule() pick again. The mutex owner can be picked later under
its own cookie.

Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()")
Reported-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://lore.kernel.org/lkml/10282ce9-f4ae-498f-9b57-f4e1e61fffbc@amd.com/
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
[jstultz: Added tweak to ensure we deactivate donor, not runnable owner]
Signed-off-by: John Stultz <jstultz@google.com>
---
Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com
---
 kernel/sched/core.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8e661b5f133d7..36e1db67a8374 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7004,6 +7004,14 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 		owner->blocked_donor = p;
 	}
 	WARN_ON_ONCE(owner && !owner->on_rq);
+
+	if (owner && !sched_cpu_cookie_match(rq, owner)) {
+		if (curr_in_chain)
+			return proxy_resched_idle(rq);
+		p = donor; /* Deactivate the donor, not the runnable owner */
+		clear_task_blocked_on(p, NULL);
+		goto deactivate;
+	}
 	return owner;
 
 deactivate:
-- 
2.55.0.rc0.799.gd6f94ed593-goog

Re: [PATCH v30 3/7] sched/core: Don't proxy-exec unmatched cookie lock owners

[K Prateek Nayak]

Hello John,

On 7/2/2026 3:15 AM, John Stultz wrote:
> From: Vasily Gorbik <gor@linux.ibm.com>
> 
> Core scheduling chooses a core-wide cookie before __schedule()
> installs the next task. With proxy-exec enabled, that task becomes the
> donor/scheduling context, and find_proxy_task() may then replace the
> execution context with the runnable mutex owner. If its cookie differs
> from the selected core cookie, running it would bypass core scheduling's
> cookie selection.
> 
> When the final mutex owner found by find_proxy_task() does not match the
> selected core cookie, stop proxying the donor. If the current execution
> context is already in the blocked chain, fall back to idle like the
> existing proxy-exec retry paths do. Otherwise deactivate the donor and
> let __schedule() pick again. The mutex owner can be picked later under
> its own cookie.

So I had a interesting (maybe a bit insane) way to make this work by
using the lock owner for rq->core_cookie when the blocked donor is
the core-wide preferred selection.

I was waiting on Peter's core-sched cleanup to post an official series
but it is currently present as one big RFC patch at
https://lore.kernel.org/lkml/9ceb2af0-33c6-40ca-b855-5167a9c5ae0f@amd.com/

I can streamline that more on top of current tip to do it all in
pick_next_task() but the bigger question is, is it worth all the
complexity?

John, Peter, what do you think?

[..snip..]

> @@ -7004,6 +7004,14 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
>  		owner->blocked_donor = p;
>  	}
>  	WARN_ON_ONCE(owner && !owner->on_rq);
> +
> +	if (owner && !sched_cpu_cookie_match(rq, owner)) {
> +		if (curr_in_chain)
> +			return proxy_resched_idle(rq);
> +		p = donor; /* Deactivate the donor, not the runnable owner */
> +		clear_task_blocked_on(p, NULL);
> +		goto deactivate;
> +	}

Otherwise, this is acceptable if we don't want to touch core-sched
bits juts yet.

>  	return owner;
>  
>  deactivate:

-- 
Thanks and Regards,
Prateek

Re: [PATCH v30 3/7] sched/core: Don't proxy-exec unmatched cookie lock owners

[Peter Zijlstra]

On Thu, Jul 02, 2026 at 10:43:14AM +0530, K Prateek Nayak wrote:
> Hello John,
> 
> On 7/2/2026 3:15 AM, John Stultz wrote:
> > From: Vasily Gorbik <gor@linux.ibm.com>
> > 
> > Core scheduling chooses a core-wide cookie before __schedule()
> > installs the next task. With proxy-exec enabled, that task becomes the
> > donor/scheduling context, and find_proxy_task() may then replace the
> > execution context with the runnable mutex owner. If its cookie differs
> > from the selected core cookie, running it would bypass core scheduling's
> > cookie selection.
> > 
> > When the final mutex owner found by find_proxy_task() does not match the
> > selected core cookie, stop proxying the donor. If the current execution
> > context is already in the blocked chain, fall back to idle like the
> > existing proxy-exec retry paths do. Otherwise deactivate the donor and
> > let __schedule() pick again. The mutex owner can be picked later under
> > its own cookie.
> 
> So I had a interesting (maybe a bit insane) way to make this work by
> using the lock owner for rq->core_cookie when the blocked donor is
> the core-wide preferred selection.
> 
> I was waiting on Peter's core-sched cleanup to post an official series
> but it is currently present as one big RFC patch at
> https://lore.kernel.org/lkml/9ceb2af0-33c6-40ca-b855-5167a9c5ae0f@amd.com/
> 
> I can streamline that more on top of current tip to do it all in
> pick_next_task() but the bigger question is, is it worth all the
> complexity?
> 
> John, Peter, what do you think?

I am going to excuse myself and say I'll look at all this in about 4
weeks when I'm back from holidays.

Re: [PATCH v30 3/7] sched/core: Don't proxy-exec unmatched cookie lock owners

[John Stultz]

On Wed, Jul 1, 2026 at 10:13 PM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
> On 7/2/2026 3:15 AM, John Stultz wrote:
> > From: Vasily Gorbik <gor@linux.ibm.com>
> >
> > Core scheduling chooses a core-wide cookie before __schedule()
> > installs the next task. With proxy-exec enabled, that task becomes the
> > donor/scheduling context, and find_proxy_task() may then replace the
> > execution context with the runnable mutex owner. If its cookie differs
> > from the selected core cookie, running it would bypass core scheduling's
> > cookie selection.
> >
> > When the final mutex owner found by find_proxy_task() does not match the
> > selected core cookie, stop proxying the donor. If the current execution
> > context is already in the blocked chain, fall back to idle like the
> > existing proxy-exec retry paths do. Otherwise deactivate the donor and
> > let __schedule() pick again. The mutex owner can be picked later under
> > its own cookie.
>
> So I had a interesting (maybe a bit insane) way to make this work by
> using the lock owner for rq->core_cookie when the blocked donor is
> the core-wide preferred selection.
>
> I was waiting on Peter's core-sched cleanup to post an official series
> but it is currently present as one big RFC patch at
> https://lore.kernel.org/lkml/9ceb2af0-33c6-40ca-b855-5167a9c5ae0f@amd.com/
>

Oh! Very interesting! My apologies for somehow having overlooked this mail!


> I can streamline that more on top of current tip to do it all in
> pick_next_task() but the bigger question is, is it worth all the
> complexity?
>
> John, Peter, what do you think?
>
> [..snip..]
>
> > @@ -7004,6 +7004,14 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
> >               owner->blocked_donor = p;
> >       }
> >       WARN_ON_ONCE(owner && !owner->on_rq);
> > +
> > +     if (owner && !sched_cpu_cookie_match(rq, owner)) {
> > +             if (curr_in_chain)
> > +                     return proxy_resched_idle(rq);
> > +             p = donor; /* Deactivate the donor, not the runnable owner */
> > +             clear_task_blocked_on(p, NULL);
> > +             goto deactivate;
> > +     }
>
> Otherwise, this is acceptable if we don't want to touch core-sched
> bits juts yet.

Vasily's patch I think is a reasonable short-term solution just to
avoid breaking core-scheduling with proxy, but it's not a great
solution as it just disables proxy in the problematic cases.

So I'm very interested in your proposal, as I was thinking I'd have to
pick folks' brains at LPC to figure out a better approach. But it
might be worth taking Vasily's patch while yours gets polished and
tested?
I'll try pulling your change in to test with my full tree.

thanks
-john

Re: [PATCH v30 3/7] sched/core: Don't proxy-exec unmatched cookie lock owners

[K Prateek Nayak]

Hello John,

On 7/8/2026 10:30 AM, John Stultz wrote:
>> https://lore.kernel.org/lkml/9ceb2af0-33c6-40ca-b855-5167a9c5ae0f@amd.com/
>>
> 
> Oh! Very interesting! My apologies for somehow having overlooked this mail!
> 
> 
>> I can streamline that more on top of current tip to do it all in
>> pick_next_task() but the bigger question is, is it worth all the
>> complexity?
>>
>> John, Peter, what do you think?
>>
>> [..snip..]
>>
>>> @@ -7004,6 +7004,14 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
>>>               owner->blocked_donor = p;
>>>       }
>>>       WARN_ON_ONCE(owner && !owner->on_rq);
>>> +
>>> +     if (owner && !sched_cpu_cookie_match(rq, owner)) {
>>> +             if (curr_in_chain)
>>> +                     return proxy_resched_idle(rq);
>>> +             p = donor; /* Deactivate the donor, not the runnable owner */
>>> +             clear_task_blocked_on(p, NULL);
>>> +             goto deactivate;
>>> +     }
>>
>> Otherwise, this is acceptable if we don't want to touch core-sched
>> bits juts yet.
> 
> Vasily's patch I think is a reasonable short-term solution just to
> avoid breaking core-scheduling with proxy, but it's not a great
> solution as it just disables proxy in the problematic cases.

Ack! We can add this in and work on the CORE_SCHED bits before we drop
the EXPERT dependency perhaps?

I doubt if anyone would notice if PROXY is missing with CORE_SCHED but
if and when we plan to retire RT_MUTEXES in favor of PROXY_EXEC, we
should have it in for correctness.

> So I'm very interested in your proposal, as I was thinking I'd have to
> pick folks' brains at LPC to figure out a better approach. But it
> might be worth taking Vasily's patch while yours gets polished and
> tested?
> I'll try pulling your change in to test with my full tree.

I'll send a clean series out by early next week after some more testing
on top of your WIP branch.

-- 
Thanks and Regards,
Prateek

Re: [PATCH v30 3/7] sched/core: Don't proxy-exec unmatched cookie lock owners

[John Stultz]

On Thu, Jul 9, 2026 at 8:14 PM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
> On 7/8/2026 10:30 AM, John Stultz wrote:
> >> https://lore.kernel.org/lkml/9ceb2af0-33c6-40ca-b855-5167a9c5ae0f@amd.com/
> >>
> >
> > Oh! Very interesting! My apologies for somehow having overlooked this mail!
> >
> >
> >> I can streamline that more on top of current tip to do it all in
> >> pick_next_task() but the bigger question is, is it worth all the
> >> complexity?
> >>
> >> John, Peter, what do you think?
> >>
> >> [..snip..]
> >>
> >>> @@ -7004,6 +7004,14 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
> >>>               owner->blocked_donor = p;
> >>>       }
> >>>       WARN_ON_ONCE(owner && !owner->on_rq);
> >>> +
> >>> +     if (owner && !sched_cpu_cookie_match(rq, owner)) {
> >>> +             if (curr_in_chain)
> >>> +                     return proxy_resched_idle(rq);
> >>> +             p = donor; /* Deactivate the donor, not the runnable owner */
> >>> +             clear_task_blocked_on(p, NULL);
> >>> +             goto deactivate;
> >>> +     }
> >>
> >> Otherwise, this is acceptable if we don't want to touch core-sched
> >> bits juts yet.
> >
> > Vasily's patch I think is a reasonable short-term solution just to
> > avoid breaking core-scheduling with proxy, but it's not a great
> > solution as it just disables proxy in the problematic cases.
>
> Ack! We can add this in and work on the CORE_SCHED bits before we drop
> the EXPERT dependency perhaps?
>

Of course! I'm not in any rush to drop EXPERT.


> I doubt if anyone would notice if PROXY is missing with CORE_SCHED but
> if and when we plan to retire RT_MUTEXES in favor of PROXY_EXEC, we
> should have it in for correctness.
>

Agreed.

> > So I'm very interested in your proposal, as I was thinking I'd have to
> > pick folks' brains at LPC to figure out a better approach. But it
> > might be worth taking Vasily's patch while yours gets polished and
> > tested?
> > I'll try pulling your change in to test with my full tree.
>
> I'll send a clean series out by early next week after some more testing
> on top of your WIP branch.

Ok, I've been tinkering with your patch a bit already today. Though, I
need to figure out some better testing tools for core-sched so I can
properly validate the correctness.

thanks!
-john

[RFC PATCH 0/3] sched/core: Allow proxy execution to work with core scheduling

[K Prateek Nayak]

Proxy execution runs the lock owner in place of blocked donor returned
by pick_next_task() but the lock owner can have a different core-cookie
compared to the blocked donor and simply running the lock owner under
those circumstances breaks the guarantees of core scheduling.

Allow proxy execution to exist alognside core scheduling by forcing a
re-pick once the blocked_on chain is found to be stable in __schedule()
and then swapping the rq->core_pick with the lock owner allowing the
execution context to dictate the core-cookie.

Lightly tested with John's priority-inversion-demo [1] patched with
"coresched new -t pid --" and the plotted chart seems to suggest proxy
is working [2]. Your mileage may vary :-)

I'll update the thread if I find any nasty bugs over the weekend run.
Patches are prepared on top of:

  https://github.com/johnstultz-work/linux-dev.git proxy-exec-v30-7.2-rc1

at commit abb09adc985b ("HACK: scx: Add scx_prio_inversion test
scheduler") with commit 93bd6418a6ea ("sched/core: Skip
put_prev_task/set_next_task re-entry for sched_ext donors") reverted on
top to make !EXT builds happy.

This is essentially a formal version of [3] broken down to be slightly
more readable. Comments and feedback are always appreciated!

References
==========

[1] https://github.com/johnstultz-work/priority-inversion-demo/
[2] https://imgur.com/NsgWqRx
[3] https://lore.kernel.org/lkml/9ceb2af0-33c6-40ca-b855-5167a9c5ae0f@amd.com/

-- 
K Prateek Nayak (3):
  sched/core: Track the core-wide pick leader
  sched/core: Fix forceidle when lock owner has mismatchd cookie
  sched/core: Swap to lock owner for core-wide pick on core-cookie
    mismatch

 kernel/sched/core.c  | 129 ++++++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h |   4 ++
 2 files changed, 119 insertions(+), 14 deletions(-)

-- 
2.43.0

[RFC PATCH 1/3] sched/core: Track the core-wide pick leader

[K Prateek Nayak]

Track the rq whose task is dictating the core-wide cookie. If no
core-cookie is set (!needs_sync), every rq on the core is considered a
pick leader as it can individually try to dictate a core cookie later.

This detail will be used in subsequent commits to extend proxy execution
to remain compatible with core scheduling.

No functional changes intended.

Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 kernel/sched/core.c  | 11 +++++++++--
 kernel/sched/sched.h |  3 +++
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 25851bb5489d..31fa6bcee616 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6649,9 +6649,9 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	const struct cpumask *smt_mask;
 	bool fi_before = false;
 	bool core_clock_updated = (rq == rq->core);
+	struct rq *rq_i, *rq_max;
 	unsigned long cookie;
 	int i, cpu, occ = 0;
-	struct rq *rq_i;
 	bool need_sync;
 
 	if (!sched_core_enabled(rq))
@@ -6668,6 +6668,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 		 */
 		rq->core_pick = NULL;
 		rq->core_dl_server = NULL;
+		rq->core_pick_leader = true;
 		return __pick_next_task(rq, rf);
 	}
 
@@ -6737,6 +6738,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 		if (!next->core_cookie) {
 			rq->core_pick = NULL;
 			rq->core_dl_server = NULL;
+			rq->core_pick_leader = true;
 			/*
 			 * For robustness, update the min_vruntime_fi for
 			 * unconstrained picks as well.
@@ -6755,6 +6757,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	 */
 restart_multi:
 	max = NULL;
+	rq_max = NULL;
 	for_each_cpu_wrap(i, smt_mask, cpu) {
 		rq_i = cpu_rq(i);
 
@@ -6772,11 +6775,15 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 
 		rq_i->core_pick = p;
 		rq_i->core_dl_server = rq_i->dl_server;
+		rq_i->core_pick_leader = false;
 
-		if (!max || prio_less(max, p, fi_before))
+		if (!max || prio_less(max, p, fi_before)) {
 			max = p;
+			rq_max = rq_i;
+		}
 	}
 
+	rq_max->core_pick_leader = true;
 	cookie = rq->core->core_cookie = max->core_cookie;
 
 	/*
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 9539867e9e0f..b80a5eb384f0 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1350,6 +1350,9 @@ struct rq {
 	unsigned int		core_sched_seq;
 	struct rb_root		core_tree;
 
+	/* private state for sched_proxy_exec()  */
+	bool			core_pick_leader;
+
 	/* shared state -- careful with sched_core_cpu_deactivate() */
 	unsigned int		core_task_seq;
 	unsigned int		core_pick_seq;
-- 
2.43.0

[RFC PATCH 2/3] sched/core: Fix forceidle when lock owner has mismatchd cookie

[K Prateek Nayak]

Move Vasily's fix from 7cd4088f70e0 ("sched/core: Don't proxy-exec
unmatched cookie lock owners") into a separate helper and fix the
forceidle stats, balance callback queuing when rq is not the one
dictating the core cookie.

The helper will be extended in subsequent commits to allow for re-pick
with cookie from the lock owner if rq is the pick leader.

Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 kernel/sched/core.c | 43 ++++++++++++++++++++++++++++++++++++-------
 1 file changed, 36 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 31fa6bcee616..ed11410f1416 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6598,6 +6598,8 @@ __pick_next_task(struct rq *rq, struct rq_flags *rf)
 	BUG(); /* The idle class should always have a runnable task. */
 }
 
+static void proxy_deactivate(struct rq *rq, struct task_struct *donor);
+
 #ifdef CONFIG_SCHED_CORE
 static inline bool is_task_rq_idle(struct task_struct *t)
 {
@@ -7100,6 +7102,23 @@ static inline void sched_core_cpu_dying(unsigned int cpu)
 		rq->core = rq;
 }
 
+static struct task_struct *
+sched_core_swap_pick(struct rq *rq, struct task_struct *next)
+{
+	if (!rq->core_pick_leader) {
+		if (!rq->core->core_forceidle_count++)
+			rq->core->core_forceidle_seq++;
+
+		proxy_resched_idle(rq);
+		queue_core_balance(rq);
+		return rq->idle;
+	}
+
+	clear_task_blocked_on(rq->donor, NULL);
+	proxy_deactivate(rq, rq->donor);
+	return RETRY_TASK;
+}
+
 #else /* !CONFIG_SCHED_CORE: */
 
 static inline void sched_core_cpu_starting(unsigned int cpu) {}
@@ -7113,6 +7132,18 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	return __pick_next_task(rq, rf);
 }
 
+static struct task_struct *
+sched_core_swap_pick(struct rq *rq, struct task_struct *next)
+{
+	/*
+	 * Caller should always check !sched_cpu_cookie_match(rq, next)
+	 * which is always false for !CONFIG_SCHED_CORE and execution
+	 * should never reach here.
+	 */
+	BUG_ON();
+	return next;
+}
+
 #endif /* !CONFIG_SCHED_CORE */
 
 /*
@@ -7512,13 +7543,6 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	}
 	WARN_ON_ONCE(owner && !owner->on_rq);
 
-	if (owner && !sched_cpu_cookie_match(rq, owner)) {
-		if (curr_in_chain)
-			return proxy_resched_idle(rq);
-		p = donor; /* Deactivate the donor, not the runnable owner */
-		clear_task_blocked_on(p, NULL);
-		goto deactivate;
-	}
 	return owner;
 
 deactivate:
@@ -7686,6 +7710,11 @@ static void __sched notrace __schedule(int sched_mode)
 				zap_balance_callbacks(rq);
 				goto keep_resched;
 			}
+			if (!sched_cpu_cookie_match(rq, next)) {
+				next = sched_core_swap_pick(rq, next);
+				if (next == RETRY_TASK)
+					goto pick_again;
+			}
 			trace_sched_found_proxy_task(rq->donor, next, cpu);
 		}
 		if (rq->donor == prev_donor && prev != next) {
-- 
2.43.0

[RFC PATCH 3/3] sched/core: Swap to lock owner for core-wide pick on core-cookie mismatch

[K Prateek Nayak]

Proxy donor can have a different core cookie compared to the lock owner.
Simply running a lock owner with a mismatched core-cookie in not an
option as this breaks the core scheduling guarantees.

To prevent a cookie mismatch, use the "rq->core_pick_leader" indicator
to know if the CPU with the blocked donor is the one with the highest
priority task. If the rq contains the highest priority task, it is
that rq's pick which dictates the core-cookie.

Once the find_proxy_task() chain stabilizes on the rq (find_proxy_task()
returns the valid lock owner queued on the same rq), set a second
"rq->core_pick_blocked_donor" and retry the pick.

The flattened flow looks as follows:

  donor = pick_next_task()
    rq_set_donor(rq, donor)

  if (donor->is_blocked)
     next = find_proxy_task()

  /*
   * next has a different cookie compared to doner and
   * rq->core_pick_leader is set.
   */
  sched_core_swap_task()
    rq->core_pick_blocked_donor = 1; /* rq->donor was the last pick */
    return RETRY_TASK;

  pick_next_task()
    donor = rq->donor;
    owner = NULL;

    if (rq->core_pick_blocked_donor)
      owner = find_proxy_task(donor);
      goto restart_multi;

    /*
     * Finish the core-pick and validate that this rq
     * is still the pick leaader.
     */
    if (rq->core_pick_blocked_donor &&
        rq->core_pick_leader &&
        rq->core_pick == donor)
      rq->core_pick = owner; /* Swapped donor with owner. */

    rq->core->coore_cookie = rq->core_pick->cookie; /* owner cookie */

    /* Continue with the rest */

    return rq->core_pick; /* Owner */

    if (sched_core_retain_donor(rq))
       rq->sched_core_proxy_pick = 0 /* Consume */

Since the owner is not blocked, it continues to run and rq->donor is
preserved while the "rq->core_pick_blocked_donor" gets consumed.

Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 kernel/sched/core.c  | 81 +++++++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h |  1 +
 2 files changed, 74 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ed11410f1416..ab3b3104cd86 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6598,7 +6598,8 @@ __pick_next_task(struct rq *rq, struct rq_flags *rf)
 	BUG(); /* The idle class should always have a runnable task. */
 }
 
-static void proxy_deactivate(struct rq *rq, struct task_struct *donor);
+static struct task_struct *
+find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf);
 
 #ifdef CONFIG_SCHED_CORE
 static inline bool is_task_rq_idle(struct task_struct *t)
@@ -6643,10 +6644,16 @@ extern void task_vruntime_update(struct rq *rq, struct task_struct *p, bool in_f
 
 static void queue_core_balance(struct rq *rq);
 
+static __always_inline bool sched_core_proxy_pick(struct rq *rq)
+{
+	return sched_proxy_exec() && unlikely(rq->core_pick_blocked_donor);
+}
+
 static struct task_struct *
 pick_next_task(struct rq *rq, struct rq_flags *rf)
 	__must_hold(__rq_lockp(rq))
 {
+	struct task_struct *owner = NULL, *donor = rq->donor;
 	struct task_struct *next, *p, *max;
 	const struct cpumask *smt_mask;
 	bool fi_before = false;
@@ -6728,6 +6735,24 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	 */
 	rq->core->core_task_seq++;
 
+	/* Last core_pick resolved to a blocked_donor! */
+	if (sched_core_proxy_pick(rq)) {
+		WARN_ON_ONCE(!donor->is_blocked);
+
+		donor->blocked_donor = NULL;
+		owner = find_proxy_task(rq, donor, rf);
+		if (owner && owner != rq->idle)
+			goto restart_multi;
+		/*
+		 * Something changed in the proxy chain!
+		 * Retry pick like normal. Increment core_task_seq
+		 * since the core-wide lock might have been dropped
+		 * during proxy-migration in find_proxy_task().
+		 */
+		rq->core_pick_blocked_donor = false;
+		rq->core->core_task_seq++;
+	}
+
 	/*
 	 * Optimize for common case where this CPU has no cookies
 	 * and there are no cookied tasks running on siblings.
@@ -6786,6 +6811,21 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	}
 
 	rq_max->core_pick_leader = true;
+
+	if (sched_core_proxy_pick(rq)) {
+		WARN_ON_ONCE(!owner);
+
+		/*
+		 * Core-wide pick resolved to the same state as last time.
+		 * Swap the donor with the lock owner and continue the
+		 * rest of the pick sequence.
+		 */
+		if (rq->core_pick_leader && rq->core_pick == donor) {
+			rq_max->core_pick = owner;
+			max = owner;
+		}
+	}
+
 	cookie = rq->core->core_cookie = max->core_cookie;
 
 	/*
@@ -6883,9 +6923,16 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	}
 
 out_set_next:
-	put_prev_set_next_task(rq, rq->donor, next);
-	if (rq->core->core_forceidle_count && next == rq->idle)
-		queue_core_balance(rq);
+	/*
+	 * If this is a redo for setting the owner's core-cookie,
+	 * put_prev_set_next_task() was already done during the
+	 * last pick and there is nothing to do now.
+	 */
+	if (!sched_core_proxy_pick(rq)) {
+		put_prev_set_next_task(rq, rq->donor, next);
+		if (rq->core->core_forceidle_count && next == rq->idle)
+			queue_core_balance(rq);
+	}
 
 	return next;
 }
@@ -7114,11 +7161,21 @@ sched_core_swap_pick(struct rq *rq, struct task_struct *next)
 		return rq->idle;
 	}
 
-	clear_task_blocked_on(rq->donor, NULL);
-	proxy_deactivate(rq, rq->donor);
+	rq->core_pick_blocked_donor = true;
 	return RETRY_TASK;
 }
 
+static bool sched_core_retain_donor(struct rq *rq)
+{
+	bool retain = rq->core_pick_blocked_donor;
+
+	if (!sched_core_enabled(rq))
+		return false;
+
+	rq->core_pick_blocked_donor = false;
+	return retain;
+}
+
 #else /* !CONFIG_SCHED_CORE: */
 
 static inline void sched_core_cpu_starting(unsigned int cpu) {}
@@ -7144,6 +7201,11 @@ sched_core_swap_pick(struct rq *rq, struct task_struct *next)
 	return next;
 }
 
+static bool sched_core_retain_donor(struct rq *rq)
+{
+	return false;
+}
+
 #endif /* !CONFIG_SCHED_CORE */
 
 /*
@@ -7694,8 +7756,11 @@ static void __sched notrace __schedule(int sched_mode)
 	if (sched_proxy_exec()) {
 		struct task_struct *prev_donor = rq->donor;
 
-		rq_set_donor(rq, next);
-		next->blocked_donor = NULL;
+		if (!sched_core_retain_donor(rq)) {
+			rq_set_donor(rq, next);
+			next->blocked_donor = NULL;
+		}
+
 		if (unlikely(next->is_blocked)) {
 			next = find_proxy_task(rq, next, &rf);
 			if (!next) {
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index b80a5eb384f0..541b2ae673d8 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1352,6 +1352,7 @@ struct rq {
 
 	/* private state for sched_proxy_exec()  */
 	bool			core_pick_leader;
+	bool			core_pick_blocked_donor;
 
 	/* shared state -- careful with sched_core_cpu_deactivate() */
 	unsigned int		core_task_seq;
-- 
2.43.0

Re: [RFC PATCH 3/3] sched/core: Swap to lock owner for core-wide pick on core-cookie mismatch

[K Prateek Nayak]

Hello folks,

On 7/17/2026 4:46 PM, K Prateek Nayak wrote:
> @@ -6728,6 +6735,24 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
>  	 */
>  	rq->core->core_task_seq++;
>  
> +	/* Last core_pick resolved to a blocked_donor! */
> +	if (sched_core_proxy_pick(rq)) {
> +		WARN_ON_ONCE(!donor->is_blocked);
> +
> +		donor->blocked_donor = NULL;
> +		owner = find_proxy_task(rq, donor, rf);
> +		if (owner && owner != rq->idle)
> +			goto restart_multi;
> +		/*
> +		 * Something changed in the proxy chain!
> +		 * Retry pick like normal. Increment core_task_seq
> +		 * since the core-wide lock might have been dropped
> +		 * during proxy-migration in find_proxy_task().
> +		 */
> +		rq->core_pick_blocked_donor = false;
> +		rq->core->core_task_seq++;
> +	}
> +
>  	/*
>  	 * Optimize for common case where this CPU has no cookies
>  	 * and there are no cookied tasks running on siblings.
> @@ -6786,6 +6811,21 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
>  	}
>  
>  	rq_max->core_pick_leader = true;
> +
> +	if (sched_core_proxy_pick(rq)) {
> +		WARN_ON_ONCE(!owner);
> +
> +		/*
> +		 * Core-wide pick resolved to the same state as last time.
> +		 * Swap the donor with the lock owner and continue the
> +		 * rest of the pick sequence.
> +		 */
> +		if (rq->core_pick_leader && rq->core_pick == donor) {
> +			rq_max->core_pick = owner;
> +			max = owner;
> +		}
> +	}

And I missed:

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ab3b3104cd868..f12064c8222e6 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6823,6 +6823,13 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 		if (rq->core_pick_leader && rq->core_pick == donor) {
 			rq_max->core_pick = owner;
 			max = owner;
+		} else {
+			/*
+			 * Pick resolved to a different leader / donor task.
+			 * Clear the "core_pick_blocked_donor" indicator and
+			 * proceed like normal.
+			 */
+			rq->core_pick_blocked_donor = false;
 		}
 	}
 
---

I seemed to have managed without it because, once the proxy
chain has stabilized, we never drop the core-wide lock, and
pick always resolves to the same state for most part but a
->balance() on the re-pick can bring tasks in and that can
alter how the pick resolves.

priority-inversion-demo with the fix: https://imgur.com/ioppJWC

> +
>  	cookie = rq->core->core_cookie = max->core_cookie;
>  
>  	/*

-- 
Thanks and Regards,
Prateek

Re: [RFC PATCH 0/3] sched/core: Allow proxy execution to work with core scheduling

[John Stultz]

On Fri, Jul 17, 2026 at 4:16 AM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Proxy execution runs the lock owner in place of blocked donor returned
> by pick_next_task() but the lock owner can have a different core-cookie
> compared to the blocked donor and simply running the lock owner under
> those circumstances breaks the guarantees of core scheduling.
>
> Allow proxy execution to exist alognside core scheduling by forcing a
> re-pick once the blocked_on chain is found to be stable in __schedule()
> and then swapping the rq->core_pick with the lock owner allowing the
> execution context to dictate the core-cookie.
>
> Lightly tested with John's priority-inversion-demo [1] patched with
> "coresched new -t pid --" and the plotted chart seems to suggest proxy
> is working [2]. Your mileage may vary :-)
>
> I'll update the thread if I find any nasty bugs over the weekend run.
> Patches are prepared on top of:
>
>   https://github.com/johnstultz-work/linux-dev.git proxy-exec-v30-7.2-rc1
>
> at commit abb09adc985b ("HACK: scx: Add scx_prio_inversion test
> scheduler") with commit 93bd6418a6ea ("sched/core: Skip
> put_prev_task/set_next_task re-entry for sched_ext donors") reverted on
> top to make !EXT builds happy.
>
> This is essentially a formal version of [3] broken down to be slightly
> more readable. Comments and feedback are always appreciated!

This is very cool! Thank you so much for sending this out!

I unfortunately didn't get a chance to queue it up in my tree for
testing today, but I'm eager to do so monday!

Hopefully I can get my tree cleaned up and get v31 sent out, as
there's been a number of issues fixed since v30. And between you,
Andrea and Suleiman, there's a lot of cool new functionality to
include in the full series!

thanks
-john

[PATCH v30 4/7] sched: Switch rq->next_class in proxy_reset_donor()

[John Stultz]

Similar to commit 04f80f8b12a0 ("sched: Switch rq->next_class on
proxy_resched_idle()"), we should also set the rq->next_class
when we call proxy_reset_donor().

Fixes: f13beb010e4a ("sched: Have try_to_wake_up() handle return-migration for PROXY_WAKING case")
Signed-off-by: John Stultz <jstultz@google.com>
---
Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com
---
 kernel/sched/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 36e1db67a8374..564762ed36f2d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3750,6 +3750,7 @@ static inline void proxy_reset_donor(struct rq *rq)
 	WARN_ON_ONCE(rq->donor == rq->curr);
 
 	put_prev_set_next_task(rq, rq->donor, rq->curr);
+	rq->next_class = rq->curr->sched_class;
 	rq_set_donor(rq, rq->curr);
 	zap_balance_callbacks(rq);
 	resched_curr(rq);
-- 
2.55.0.rc0.799.gd6f94ed593-goog

[PATCH v30 5/7] sched: Break out core of attach_tasks() helper into sched.h

[John Stultz]

Pull the core of attach_tasks() out into sched.h so
it can be used more generically.

Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com
---
 kernel/sched/fair.c  | 16 +---------------
 kernel/sched/sched.h | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 15 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee13..6d2fa0cd77779 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11035,21 +11035,7 @@ static int detach_tasks(struct lb_env *env)
  */
 static void attach_tasks(struct lb_env *env)
 {
-	struct list_head *tasks = &env->tasks;
-	struct task_struct *p;
-	struct rq_flags rf;
-
-	rq_lock(env->dst_rq, &rf);
-	update_rq_clock(env->dst_rq);
-
-	while (!list_empty(tasks)) {
-		p = list_first_entry(tasks, struct task_struct, se.group_node);
-		list_del_init(&p->se.group_node);
-
-		attach_task(env->dst_rq, p);
-	}
-
-	rq_unlock(env->dst_rq, &rf);
+	__attach_tasks(env->dst_rq, &env->tasks);
 }
 
 #ifdef CONFIG_NO_HZ_COMMON
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 56acf502ba260..56d9c09c485e6 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3100,6 +3100,25 @@ static inline void attach_one_task(struct rq *rq, struct task_struct *p)
 	attach_task(rq, p);
 }
 
+/*
+ * __attach_tasks() - attaches a list of tasks (using se.group_node) to
+ * the new rq
+ */
+static inline void __attach_tasks(struct rq *rq, struct list_head *tasks)
+{
+	guard(rq_lock)(rq);
+	update_rq_clock(rq);
+
+	while (!list_empty(tasks)) {
+		struct task_struct *p;
+
+		p = list_first_entry(tasks, struct task_struct, se.group_node);
+		list_del_init(&p->se.group_node);
+
+		attach_task(rq, p);
+	}
+}
+
 #ifdef CONFIG_PREEMPT_RT
 # define SCHED_NR_MIGRATE_BREAK 8
 #else
-- 
2.55.0.rc0.799.gd6f94ed593-goog

[PATCH v30 6/7] sched: Migrate whole chain in proxy_migrate_task()

[John Stultz]

Instead of migrating one task each time through find_proxy_task(),
we can walk up the blocked_donor ptrs and migrate the entire
current chain in one go.

This was broken out of earlier patches and held back while the
series was being stabilized, but I wanted to re-introduce it.

Signed-off-by: John Stultz <jstultz@google.com>
---
v12:
* Earlier this was re-using blocked_node, but I hit
  a race with activating blocked entities, and to
  avoid it introduced a new migration_node listhead
v18:
* Add init_task initialization of migration_node as suggested
  by Suleiman
v22:
* Move migration_node under CONFIG_SCHED_PROXY_EXEC as suggested
  by K Prateek
v25:
* Use se.group_node instead of adding migration_node, as
  suggsested by K Prateek
* Integrated attach_tasks() cleanups suggested by K Prateek

Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com
---
 kernel/sched/core.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 564762ed36f2d..1bf60d78c9208 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6819,9 +6819,9 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
 	__must_hold(__rq_lockp(rq))
 {
 	struct rq *target_rq = cpu_rq(target_cpu);
+	LIST_HEAD(migrate_list);
 
 	lockdep_assert_rq_held(rq);
-	WARN_ON(p == rq->curr);
 	/*
 	 * Since we are migrating a blocked donor, it could be rq->donor,
 	 * and we want to make sure there aren't any references from this
@@ -6834,13 +6834,20 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
 	 * before we release the lock.
 	 */
 	proxy_resched_idle(rq);
-
-	deactivate_task(rq, p, DEQUEUE_NOCLOCK);
-	proxy_set_task_cpu(p, target_cpu);
-
+	for (; p; p = p->blocked_donor) {
+		WARN_ON(p == rq->curr);
+		deactivate_task(rq, p, DEQUEUE_NOCLOCK);
+		proxy_set_task_cpu(p, target_cpu);
+		/*
+		 * We can re-use se.group_node to migrate the thing,
+		 * because @p is deactivated (won't be balanced) and
+		 * we hold the rq_lock.
+		 */
+		list_add(&p->se.group_node, &migrate_list);
+	}
 	proxy_release_rq_lock(rq, rf);
 
-	attach_one_task(target_rq, p);
+	__attach_tasks(target_rq, &migrate_list);
 
 	proxy_reacquire_rq_lock(rq, rf);
 }
-- 
2.55.0.rc0.799.gd6f94ed593-goog

[PATCH v30 7/7] sched: Add deactivated (sleeping) owner handling to find_proxy_task()

[John Stultz]

From: Peter Zijlstra <peterz@infradead.org>

If the blocked_on chain resolves to a sleeping owner, deactivate
the donor task, and enqueue it on the sleeping owner task.
Then re-activate it later when the owner is woken up.

NOTE: This has been particularly challenging to get working
properly, and some of the locking is particularly awkward. I'd
very much appreciate review and feedback for ways to simplify
this.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Connor O'Brien <connoro@google.com>
[jstultz: This was broken out from the larger proxy() patch]
Signed-off-by: John Stultz <jstultz@google.com>
---
v5:
* Split out from larger proxy patch
v6:
* Major rework, replacing the single list head per task with
  per-task list head and nodes, creating a tree structure so
  we only wake up descendants of the task woken.
* Reworked the locking to take the task->pi_lock, so we can
  avoid mid-chain wakeup races from try_to_wake_up() called by
  the ww_mutex logic.
v7:
* Drop unnecessary __nested lock annotation, as we already drop
  the lock prior.
* Add comments on #else & #endif lines, and clearer function
  names, and commit message tweaks as suggested by Metin Kaya
* Move activate_blocked_entities() call from ttwu_queue to
  try_to_wake_up() to simplify locking. Thanks to questions from
  Metin Kaya
* Fix irqsave/irqrestore usage now we call this outside where
  the pi_lock is held
* Fix activate_blocked_entitites not preserving wake_cpu
* Fix for UP builds
v8:
* Minor checkpatch fixup
* Drop proxy_deactivate and cleanups suggested by Metin
v9:
* Fix bug causing possibly uninitialized cpu value to be used with
  activate_blocked_entities()
* Improved comment around preserving wake_cpu suggested by Metin
* Add additional lockdep asserts, suggested by Metin
* Tweaked placement of lockdep assert, suggested by Metin
* Fixed comment referring to structure entry name
* Fix to call proxy_resched_idle() _prior_ to calling
  proxy_enqueue_on_owner() where we deactivate the task, this
  avoids stale references to rq_selected() when the task may
  have been migrated to another rq.
* Fix to remove the blocked_head list at the start of
  activate_blocked_entities() so we only do a finite amount
  of work, avoiding a potential livelock of two cpus removing
  and adding tasks to the list at the same time if the owner
  went back to sleep while blocked entities were being woken.
v11:
* Big rework to get rid of recursion. Had to add another list
  item to the task_stuct to do this as we are in atomic context
  and cannot allocate memory while activating blocked entities.
  Will need to watch carefully for bugs, as switching to a
  list_head in the task_struct instead of a pointer on the
  stack opens up the potential for races on the shared state,
  but I think I've got the locking sorted.
* Moved proxy_set_task_cpu helper to earlier in the series
* Minor rework for try_to_deactivate_task changes
* Minor variable name cleanups suggested by Metin
v13:
* Switch to use donor from next for proxy_enqueue_on_owner
* Switch to using block_task instead of deactivate_task
v14:
* Ensure we call block_task() last in proxy_enqueue_on_owner
  and not touch it again to avoid races where it might be
  activated on another cpu
* Make sure we activate blocked_entities when we exit from ttwu
* Fix to enqueue the last task in the chain (p) on the blocked
  owner instead of donor, so that we preserve the chain
  structure so mid-chain wakeups propagate properly
* Rework of sleeping_owner handling so that we properly deal
  with delayed-dequeued (sched_delayed) tasks (also removes
  now unused proxy_deactivate() logic)
v15:
* Rework do_activate_task to be activate_task() and have it
  call __activate_task(), suggested by Carlos Llamas
* Put task_struct additions under CONFIG_SCHED_PROXY_EXEC
v16:
* Rework do_activate_blocked_waiter locking to use scoped_guard
* Rework find_proxy_task() logic to use guard
v18:
* Integrate Suleiman's suggested optimization to check sleeping
  owner status before task_cpu, to avoid unnecssarily
  proxy-migrating tasks to then just dequeue them.
* Add on_cpu check to fix for a very hard to reproduce race where
  a late blocked_task_activation() happens while the task is already
  on_cpu elsewhere. When we get to  __schedule() we block the task
  (!on_rq) but haven't yet switched away. blocked_task_activation()
  would then incorrectly activate on a different runqueue.
* Add init_task initialization for sleeping owner lists, as
  suggested by Suleiman
v19:
* Build fixup for !CONFIG_SMP
v22:
* Rework to avoid gotos in guard() scopes, using break and
  switch() on action values. Suggested by K Prateek.
v25:
* Fix elevated nr_uninterruptible and nr_iowait counts,
  which were causing bad loadavg values. Reported and
  fixed by David Stevens <stevensd@google.com>
v28:
* In __proxy_remove_from_sleeping_owner() we call
  put_task(), which might free the owner while we are
  still holing the owners->blocked_lock. So be sure
  to get_task()/put_task() around the owner usage to
  ensure we don't prematurely free the task.
* Also reorder the put_task/unlock lines in
  activate_blocked_waiters(), to avoid a similar issue
v30:
* Optimize activate_blocked_waiters() so we don't do so much
  unnecessary work when proxy-exec is enabled. If there are
  no blocked waiters, we can return early.

Cc: Joel Fernandes <joelagnelf@nvidia.com>
Cc: Qais Yousef <qyousef@layalina.io>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Zimuzo Ezeozue <zezeozue@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Will Deacon <will@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Metin Kaya <Metin.Kaya@arm.com>
Cc: Xuewen Yan <xuewen.yan94@gmail.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: kuyo chang <kuyo.chang@mediatek.com>
Cc: hupu <hupu.gm@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: kernel-team@android.com
---
 include/linux/sched.h |   7 +
 init/init_task.c      |   6 +
 kernel/fork.c         |   6 +
 kernel/sched/core.c   | 305 +++++++++++++++++++++++++++++++++++++++---
 4 files changed, 303 insertions(+), 21 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 373bcc0598d10..8c2ba6dce58f4 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1251,6 +1251,13 @@ struct task_struct {
 
 	struct mutex			*blocked_on;	/* lock we're blocked on */
 	raw_spinlock_t			blocked_lock;
+#ifdef CONFIG_SCHED_PROXY_EXEC
+	struct list_head		blocked_head;  /* tasks blocked on this task */
+	struct list_head		blocked_node;  /* our entry on someone elses blocked_head */
+	/* Node for list of tasks to process blocked_head list for blocked entitiy activations */
+	struct list_head		blocked_activation_node;
+	struct task_struct		*sleeping_owner; /* task our blocked_node is enqueued on */
+#endif
 
 	/*
 	 * The task that is boosting this task; a back link for the current
diff --git a/init/init_task.c b/init/init_task.c
index b67ef6040a655..809282a2741d7 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -211,6 +211,12 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 						 &init_task.alloc_lock),
 #endif
 	.blocked_donor = NULL,
+#ifdef CONFIG_SCHED_PROXY_EXEC
+	.blocked_head = LIST_HEAD_INIT(init_task.blocked_head),
+	.blocked_node = LIST_HEAD_INIT(init_task.blocked_node),
+	.blocked_activation_node = LIST_HEAD_INIT(init_task.blocked_activation_node),
+	.sleeping_owner = NULL,
+#endif
 #ifdef CONFIG_RT_MUTEXES
 	.pi_waiters	= RB_ROOT_CACHED,
 	.pi_top_task	= NULL,
diff --git a/kernel/fork.c b/kernel/fork.c
index 13e38e89a1f30..fd35f20e955e8 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -2242,6 +2242,12 @@ __latent_entropy struct task_struct *copy_process(
 
 	p->blocked_on = NULL; /* not blocked yet */
 	p->blocked_donor = NULL; /* nobody is boosting p yet */
+#ifdef CONFIG_SCHED_PROXY_EXEC
+	INIT_LIST_HEAD(&p->blocked_head);
+	INIT_LIST_HEAD(&p->blocked_node);
+	INIT_LIST_HEAD(&p->blocked_activation_node);
+	p->sleeping_owner = NULL;
+#endif
 
 #ifdef CONFIG_BCACHE
 	p->sequential_io	= 0;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1bf60d78c9208..ec23de56b5271 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2216,7 +2216,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	return p->sched_class->dequeue_task(rq, p, flags);
 }
 
-void activate_task(struct rq *rq, struct task_struct *p, int flags)
+static inline void __activate_task(struct rq *rq, struct task_struct *p, int flags)
 {
 	if (task_on_rq_migrating(p))
 		flags |= ENQUEUE_MIGRATED;
@@ -2227,6 +2227,71 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags)
 	ASSERT_EXCLUSIVE_WRITER(p->on_rq);
 }
 
+#ifdef CONFIG_SCHED_PROXY_EXEC
+static inline
+void __proxy_remove_from_sleeping_owner(struct task_struct *owner, struct task_struct *p)
+{
+	lockdep_assert_held(&owner->blocked_lock);
+
+	if (p->sleeping_owner == owner) {
+		list_del_init(&p->blocked_node);
+		WRITE_ONCE(p->sleeping_owner, NULL);
+		put_task_struct(owner); // matches get in proxy_enqueue_on_owner
+	}
+}
+
+static inline void proxy_remove_from_sleeping_owner(struct task_struct *p)
+{
+	struct task_struct *owner = READ_ONCE(p->sleeping_owner);
+
+	if (owner) {
+		/*
+		 * __proxy_remove_from_sleeping_owner() does a
+		 * put on owner to match the get done in
+		 * proxy_enqueue_on_owner(). If that put is the
+		 * last one and it frees owner, we'd be freeing
+		 * a lock we held. So get/put owner around its
+		 * usage her to ensure that doesn't happen.
+		 */
+		get_task_struct(owner);
+		raw_spin_lock(&owner->blocked_lock);
+		__proxy_remove_from_sleeping_owner(owner, p);
+		raw_spin_unlock(&owner->blocked_lock);
+		put_task_struct(owner);
+	}
+}
+
+void activate_task(struct rq *rq, struct task_struct *p, int en_flags)
+{
+	if (!sched_proxy_exec()) {
+		__activate_task(rq, p, en_flags);
+		return;
+	}
+
+	lockdep_assert_rq_held(rq);
+	proxy_remove_from_sleeping_owner(p);
+	/*
+	 * By calling __activate_task() with blocked_lock held, we
+	 * order against the find_proxy_task() blocked_task case
+	 * such that no more blocked tasks will be enqueued on p
+	 * once we release p->blocked_lock.
+	 */
+	raw_spin_lock(&p->blocked_lock);
+	WARN_ON(task_cpu(p) != cpu_of(rq));
+	__activate_task(rq, p, en_flags);
+	raw_spin_unlock(&p->blocked_lock);
+}
+#else
+static inline void proxy_remove_from_sleeping_owner(struct task_struct *p)
+{
+}
+
+void activate_task(struct rq *rq, struct task_struct *p, int en_flags)
+{
+	__activate_task(rq, p, en_flags);
+}
+#endif
+
 void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 {
 	WARN_ON_ONCE(flags & DEQUEUE_SLEEP);
@@ -3756,6 +3821,163 @@ static inline void proxy_reset_donor(struct rq *rq)
 	resched_curr(rq);
 }
 
+static inline void proxy_set_task_cpu(struct task_struct *p, int cpu)
+{
+	unsigned int wake_cpu;
+
+	/*
+	 * Since we are enqueuing a blocked task on a cpu it may
+	 * not be able to run on, preserve wake_cpu when we
+	 * __set_task_cpu so we can return the task to where it
+	 * was previously runnable.
+	 */
+	wake_cpu = p->wake_cpu;
+	__set_task_cpu(p, cpu);
+	p->wake_cpu = wake_cpu;
+}
+
+static void do_activate_blocked_waiter(struct rq *target_rq, struct task_struct *p, int en_flags)
+{
+	unsigned int state;
+	struct rq_flags rf;
+	int target_cpu = cpu_of(target_rq);
+
+	scoped_guard (raw_spinlock_irqsave, &p->pi_lock) {
+		state = READ_ONCE(p->__state);
+		/* Avoid racing with ttwu */
+		if (state == TASK_WAKING)
+			return;
+
+		if (READ_ONCE(p->on_rq)) {
+			/*
+			 * We raced with a non mutex handoff activation of p.
+			 * That activation will also take care of activating
+			 * all of the tasks after p in the blocked_head list,
+			 * so we're done here.
+			 */
+			return;
+		}
+		if (task_on_cpu(task_rq(p), p)) {
+			/*
+			 * Its possible this activation is very late, and
+			 * we already were woken up and are running on a
+			 * different cpu. If that task blocked, it could be
+			 * dequeued (so on_rq == 0), but still on_cpu.
+			 * Bail in this case, as we definitely don't want to
+			 * activate a task when its on_cpu elsewhere.
+			 */
+			return;
+		}
+		proxy_set_task_cpu(p, target_cpu);
+		rq_lock_irqsave(target_rq, &rf);
+		/*
+		 * proxy_enqueue_on_owner() called block_task() which
+		 * increments nr_uninterruptible/nr_iowait, so we need
+		 * to reverse that when we activate the blocked waiter
+		 */
+		if (p->sched_contributes_to_load)
+			target_rq->nr_uninterruptible--;
+		if (p->in_iowait) {
+			delayacct_blkio_end(p);
+			atomic_dec(&task_rq(p)->nr_iowait);
+		}
+		update_rq_clock(target_rq);
+		activate_task(target_rq, p, en_flags);
+		resched_curr(target_rq);
+		rq_unlock_irqrestore(target_rq, &rf);
+	}
+}
+
+static void activate_blocked_waiters(struct rq *target_rq,
+				     struct task_struct *owner,
+				     int wake_flags)
+{
+	struct list_head bal_head;
+	unsigned long flags;
+	int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
+
+	if (!sched_proxy_exec())
+		return;
+
+	/*
+	 * A whole bunch of waiting donor tasks back this blocked
+	 * lock owner task, wake them all up to give this task its
+	 * 'fair' share.
+	 *
+	 * This is a little unique here and the locking is messy.
+	 * At this point we only hold the blocked_lock, so the
+	 * owner task may be able to run and do all sorts of
+	 * things while we are processing the blocked_head list,
+	 * including going back to sleep, which can cause tasks
+	 * to be added to the owners->blocked_head while we are
+	 * processing it!
+	 * Thus, we pull the entire list off the owner->blocked_head
+	 * here so that we will only process a finite amount of
+	 * tasks. Tasks added after this will be processed by the
+	 * future wake events.
+	 * Even though we have pulled the list off the blocked_head
+	 * the removed list is *still* "owned" and serialized by
+	 * the owner->blocked_lock! As we have to serialize against
+	 * mid-chain wakeups, who may try to remove themselves from
+	 * the list.
+	 */
+	raw_spin_lock_irqsave(&owner->blocked_lock, flags);
+	if (!list_empty(&owner->blocked_activation_node)) {
+		raw_spin_unlock_irqrestore(&owner->blocked_lock, flags);
+		return;
+	}
+
+	if (list_empty(&owner->blocked_head)) {
+		raw_spin_unlock_irqrestore(&owner->blocked_lock, flags);
+		return;
+	}
+
+	get_task_struct(owner);
+	INIT_LIST_HEAD(&bal_head);
+	list_add_tail(&owner->blocked_activation_node, &bal_head);
+	raw_spin_unlock_irqrestore(&owner->blocked_lock, flags);
+
+	if (wake_flags & WF_MIGRATED)
+		en_flags |= ENQUEUE_MIGRATED;
+
+	while (!list_empty(&bal_head)) {
+		struct list_head tmp_head;
+
+		INIT_LIST_HEAD(&tmp_head);
+		owner = list_first_entry(&bal_head, struct task_struct, blocked_activation_node);
+
+		raw_spin_lock_irqsave(&owner->blocked_lock, flags);
+		list_replace_init(&owner->blocked_head, &tmp_head);
+		list_del_init(&owner->blocked_activation_node);
+		while (!list_empty(&tmp_head)) {
+			struct task_struct *p;
+
+			p = list_first_entry(&tmp_head,
+					     struct task_struct,
+					     blocked_node);
+			WARN_ON(p == owner);
+			WARN_ON(p->sleeping_owner != owner);
+			__proxy_remove_from_sleeping_owner(owner, p);
+			raw_spin_unlock_irqrestore(&owner->blocked_lock, flags);
+
+			do_activate_blocked_waiter(target_rq, p, en_flags);
+
+			raw_spin_lock_irqsave(&p->blocked_lock, flags);
+			if (list_empty(&p->blocked_activation_node)) {
+				get_task_struct(p);
+				list_add_tail(&p->blocked_activation_node, &bal_head);
+			}
+			raw_spin_unlock_irqrestore(&p->blocked_lock, flags);
+
+			raw_spin_lock_irqsave(&owner->blocked_lock, flags);
+		}
+		raw_spin_unlock_irqrestore(&owner->blocked_lock, flags);
+		put_task_struct(owner); // put matches get prior to adding to local bal_head
+	}
+}
+
+static inline struct task_struct *proxy_resched_idle(struct rq *rq);
+
 /*
  * Checks to see if task p has been proxy-migrated to another rq
  * and needs to be returned. If so, we deactivate the task here
@@ -3800,6 +4022,11 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
 {
 	return false;
 }
+static inline void activate_blocked_waiters(struct rq *target_rq,
+					    struct task_struct *owner,
+					    int wake_flags)
+{
+}
 #endif /* CONFIG_SCHED_PROXY_EXEC */
 
 static void
@@ -3873,8 +4100,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
 	update_rq_clock(rq);
 	if (p->is_blocked) {
-		if (p->se.sched_delayed)
+		if (p->se.sched_delayed) {
+			proxy_remove_from_sleeping_owner(p);
 			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+		}
 		if (proxy_needs_return(rq, p))
 			return 0;
 	}
@@ -3903,13 +4132,19 @@ void sched_ttwu_pending(void *arg)
 	update_rq_clock(rq);
 
 	llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+		int wake_flags;
 		if (WARN_ON_ONCE(p->on_cpu))
 			smp_cond_load_acquire(&p->on_cpu, !VAL);
 
 		if (WARN_ON_ONCE(task_cpu(p) != cpu_of(rq)))
 			set_task_cpu(p, cpu_of(rq));
 
-		ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
+		wake_flags = p->sched_remote_wakeup ? WF_MIGRATED : 0;
+		ttwu_do_activate(rq, p, wake_flags, &rf);
+		rq_unlock(rq, &rf);
+		activate_blocked_waiters(rq, p, wake_flags);
+		rq_lock(rq, &rf);
+		update_rq_clock(rq);
 	}
 
 	/*
@@ -4416,6 +4651,7 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		ttwu_queue(p, cpu, wake_flags);
 	}
 out:
+	activate_blocked_waiters(cpu_rq(task_cpu(p)), p, wake_flags);
 	if (success)
 		ttwu_stat(p, task_cpu(p), wake_flags);
 
@@ -6731,21 +6967,6 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,
 }
 
 #ifdef CONFIG_SCHED_PROXY_EXEC
-static inline void proxy_set_task_cpu(struct task_struct *p, int cpu)
-{
-	unsigned int wake_cpu;
-
-	/*
-	 * Since we are enqueuing a blocked task on a cpu it may
-	 * not be able to run on, preserve wake_cpu when we
-	 * __set_task_cpu so we can return the task to where it
-	 * was previously runnable.
-	 */
-	wake_cpu = p->wake_cpu;
-	__set_task_cpu(p, cpu);
-	p->wake_cpu = wake_cpu;
-}
-
 static inline struct task_struct *proxy_resched_idle(struct rq *rq)
 {
 	put_prev_set_next_task(rq, rq->donor, rq->idle);
@@ -6852,6 +7073,28 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
 	proxy_reacquire_rq_lock(rq, rf);
 }
 
+static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owner,
+				   struct task_struct *p)
+{
+	lockdep_assert_rq_held(rq);
+	lockdep_assert_held(&owner->blocked_lock);
+	/*
+	 * ttwu_activate() will pick them up and place them on whatever rq
+	 * @owner will run next.
+	 */
+	WARN_ON(p == owner);
+	WARN_ON(!p->on_rq);
+	WARN_ON(p->sleeping_owner);
+	get_task_struct(owner);
+	WRITE_ONCE(p->sleeping_owner, owner);
+	/*
+	 * ttwu_do_activate must not have a chance to activate p
+	 * elsewhere before it's fully extricated from its old rq.
+	 */
+	list_add(&p->blocked_node, &owner->blocked_head);
+	block_task(rq, p, READ_ONCE(p->__state));
+}
+
 /*
  * Find runnable lock owner to proxy for mutex blocked donor
  *
@@ -6938,11 +7181,31 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 		}
 
 		if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {
-			/* XXX Don't handle blocked owners/delayed dequeue yet */
+			/*
+			 * rq->curr must not be added to the blocked_head list or else
+			 * ttwu_do_activate could enqueue it elsewhere before it switches
+			 * out here. The approach to avoid this is the same as in the
+			 * migrate_task case.
+			 */
 			if (curr_in_chain)
 				return proxy_resched_idle(rq);
-			__clear_task_blocked_on(p, NULL);
-			goto deactivate;
+			/*
+			 * If !@owner->on_rq, holding @rq->lock will not pin the task,
+			 * so we cannot drop @mutex->wait_lock until we're sure its a blocked
+			 * task on this rq.
+			 *
+			 * We use @owner->blocked_lock to serialize against ttwu_activate().
+			 * Either we see its new owner->on_rq or it will see our list_add().
+			 */
+			WARN_ON(owner == p);
+			raw_spin_unlock(&p->blocked_lock);
+			raw_spin_lock(&owner->blocked_lock);
+			proxy_resched_idle(rq);
+			proxy_enqueue_on_owner(rq, owner, p);
+			raw_spin_unlock(&owner->blocked_lock);
+			raw_spin_lock(&p->blocked_lock);
+
+			return NULL; /* retry task selection */
 		}
 
 		owner_cpu = task_cpu(owner);
-- 
2.55.0.rc0.799.gd6f94ed593-goog

Re: [PATCH v30 7/7] sched: Add deactivated (sleeping) owner handling to find_proxy_task()

[K Prateek Nayak]

Hello John,

On 7/2/2026 3:16 AM, John Stultz wrote:
> @@ -6852,6 +7073,28 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
>  	proxy_reacquire_rq_lock(rq, rf);
>  }
>  
> +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owner,
> +				   struct task_struct *p)
> +{
> +	lockdep_assert_rq_held(rq);
> +	lockdep_assert_held(&owner->blocked_lock);
> +	/*
> +	 * ttwu_activate() will pick them up and place them on whatever rq
> +	 * @owner will run next.
> +	 */
> +	WARN_ON(p == owner);
> +	WARN_ON(!p->on_rq);
> +	WARN_ON(p->sleeping_owner);
> +	get_task_struct(owner);
> +	WRITE_ONCE(p->sleeping_owner, owner);
> +	/*
> +	 * ttwu_do_activate must not have a chance to activate p
> +	 * elsewhere before it's fully extricated from its old rq.
> +	 */
> +	list_add(&p->blocked_node, &owner->blocked_head);

I'll refer to this list_add() as (1) below ...

> +	block_task(rq, p, READ_ONCE(p->__state));
> +}
> +
>  /*
>   * Find runnable lock owner to proxy for mutex blocked donor
>   *
> @@ -6938,11 +7181,31 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
>  		}
>  
>  		if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {

I'm having a sneaky feeling that, with enough bad luck, we
can go from seeing !owner->on_rq and here to actually finish
enqueuing the owner by the time we get to (1) ...

> -			/* XXX Don't handle blocked owners/delayed dequeue yet */
> +			/*
> +			 * rq->curr must not be added to the blocked_head list or else
> +			 * ttwu_do_activate could enqueue it elsewhere before it switches
> +			 * out here. The approach to avoid this is the same as in the
> +			 * migrate_task case.
> +			 */
>  			if (curr_in_chain)
>  				return proxy_resched_idle(rq);
> -			__clear_task_blocked_on(p, NULL);
> -			goto deactivate;
> +			/*
> +			 * If !@owner->on_rq, holding @rq->lock will not pin the task,
> +			 * so we cannot drop @mutex->wait_lock until we're sure its a blocked
> +			 * task on this rq.
> +			 *
> +			 * We use @owner->blocked_lock to serialize against ttwu_activate().
> +			 * Either we see its new owner->on_rq or it will see our list_add().
> +			 */
> +			WARN_ON(owner == p);
> +			raw_spin_unlock(&p->blocked_lock);
> +			raw_spin_lock(&owner->blocked_lock);

... by the time we get here, activate_blocked_waiters() could have
already bailed out for:

  !list_empty(&owner->blocked_activation_node)

and then we go and add the task to list. I feel, like most conditionally
adding to list patterns, we should add the task to list, check the
condition once again under owner->blocked_lock, and then proceed.

That way we can sure the activate will definitely see the queued task.
Sorry in advance in case I've been blind and missed something
obvious :-)

On a separate note, is it worth tracking if the task is a mutex / lock
owner and only go through the activate_blocked_waiters() path only when
a __mutex_owner() somewhere can return the task being activated? Perhaps
a per-task counter that increments at every mutex_lock() and decrements
at a mutex_unlock()?

> +			proxy_resched_idle(rq);
> +			proxy_enqueue_on_owner(rq, owner, p);
> +			raw_spin_unlock(&owner->blocked_lock);
> +			raw_spin_lock(&p->blocked_lock);
> +
> +			return NULL; /* retry task selection */
>  		}
>  
>  		owner_cpu = task_cpu(owner);

-- 
Thanks and Regards,
Prateek

Re: [PATCH v30 7/7] sched: Add deactivated (sleeping) owner handling to find_proxy_task()

[John Stultz]

On Thu, Jul 2, 2026 at 8:07 PM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
> On 7/2/2026 3:16 AM, John Stultz wrote:
> > @@ -6852,6 +7073,28 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
> >       proxy_reacquire_rq_lock(rq, rf);
> >  }
> >
> > +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owner,
> > +                                struct task_struct *p)
> > +{
> > +     lockdep_assert_rq_held(rq);
> > +     lockdep_assert_held(&owner->blocked_lock);
> > +     /*
> > +      * ttwu_activate() will pick them up and place them on whatever rq
> > +      * @owner will run next.
> > +      */
> > +     WARN_ON(p == owner);
> > +     WARN_ON(!p->on_rq);
> > +     WARN_ON(p->sleeping_owner);
> > +     get_task_struct(owner);
> > +     WRITE_ONCE(p->sleeping_owner, owner);
> > +     /*
> > +      * ttwu_do_activate must not have a chance to activate p
> > +      * elsewhere before it's fully extricated from its old rq.
> > +      */
> > +     list_add(&p->blocked_node, &owner->blocked_head);
>
> I'll refer to this list_add() as (1) below ...
>
> > +     block_task(rq, p, READ_ONCE(p->__state));
> > +}
> > +
> >  /*
> >   * Find runnable lock owner to proxy for mutex blocked donor
> >   *
> > @@ -6938,11 +7181,31 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
> >               }
> >
> >               if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {
>
> I'm having a sneaky feeling that, with enough bad luck, we
> can go from seeing !owner->on_rq and here to actually finish
> enqueuing the owner by the time we get to (1) ...
>
> > -                     /* XXX Don't handle blocked owners/delayed dequeue yet */
> > +                     /*
> > +                      * rq->curr must not be added to the blocked_head list or else
> > +                      * ttwu_do_activate could enqueue it elsewhere before it switches
> > +                      * out here. The approach to avoid this is the same as in the
> > +                      * migrate_task case.
> > +                      */
> >                       if (curr_in_chain)
> >                               return proxy_resched_idle(rq);
> > -                     __clear_task_blocked_on(p, NULL);
> > -                     goto deactivate;
> > +                     /*
> > +                      * If !@owner->on_rq, holding @rq->lock will not pin the task,
> > +                      * so we cannot drop @mutex->wait_lock until we're sure its a blocked
> > +                      * task on this rq.
> > +                      *
> > +                      * We use @owner->blocked_lock to serialize against ttwu_activate().
> > +                      * Either we see its new owner->on_rq or it will see our list_add().
> > +                      */
> > +                     WARN_ON(owner == p);
> > +                     raw_spin_unlock(&p->blocked_lock);
> > +                     raw_spin_lock(&owner->blocked_lock);
>
> ... by the time we get here, activate_blocked_waiters() could have
> already bailed out for:
>
>   !list_empty(&owner->blocked_activation_node)
>
> and then we go and add the task to list. I feel, like most conditionally
> adding to list patterns, we should add the task to list, check the
> condition once again under owner->blocked_lock, and then proceed.
>
> That way we can sure the activate will definitely see the queued task.
> Sorry in advance in case I've been blind and missed something
> obvious :-)

Hrm.  That's a good point!

Let me take a stab at fixing that.  I'm thinking it might be easier to
check on_rq again after we take the blocked lock before calling
proxy_enqueue_on_owner(), because otherwise unwinding that is a little
complicated (pulling it off the list isn't, but the task will have
been blocked, which isn't what we want if we are going to pick again
to figure out if we can proxy or have to migrate the waiter).  Am I
still missing a race with that approach (instead of adding to the list
before the check)?

> On a separate note, is it worth tracking if the task is a mutex / lock
> owner and only go through the activate_blocked_waiters() path only when
> a __mutex_owner() somewhere can return the task being activated? Perhaps
> a per-task counter that increments at every mutex_lock() and decrements
> at a mutex_unlock()?

This as an optimization to avoid taking the owner->blocked_lock and
the list checking?
Let me think on that a bit.
For normal mutexes it seems easy enough but I want to make sure it
doesn't get complicated with ww_mutexes or the later proxy-rwsem
support from the full series.

thanks
-john

Re: [PATCH v30 7/7] sched: Add deactivated (sleeping) owner handling to find_proxy_task()

[K Prateek Nayak]

Hello John,

On 7/8/2026 10:04 AM, John Stultz wrote:
> On Thu, Jul 2, 2026 at 8:07 PM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>> On 7/2/2026 3:16 AM, John Stultz wrote:
>>> @@ -6852,6 +7073,28 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
>>>       proxy_reacquire_rq_lock(rq, rf);
>>>  }
>>>
>>> +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owner,
>>> +                                struct task_struct *p)
>>> +{
>>> +     lockdep_assert_rq_held(rq);
>>> +     lockdep_assert_held(&owner->blocked_lock);
>>> +     /*
>>> +      * ttwu_activate() will pick them up and place them on whatever rq
>>> +      * @owner will run next.
>>> +      */
>>> +     WARN_ON(p == owner);
>>> +     WARN_ON(!p->on_rq);
>>> +     WARN_ON(p->sleeping_owner);
>>> +     get_task_struct(owner);
>>> +     WRITE_ONCE(p->sleeping_owner, owner);
>>> +     /*
>>> +      * ttwu_do_activate must not have a chance to activate p
>>> +      * elsewhere before it's fully extricated from its old rq.
>>> +      */
>>> +     list_add(&p->blocked_node, &owner->blocked_head);
>>
>> I'll refer to this list_add() as (1) below ...
>>
>>> +     block_task(rq, p, READ_ONCE(p->__state));
>>> +}
>>> +
>>>  /*
>>>   * Find runnable lock owner to proxy for mutex blocked donor
>>>   *
>>> @@ -6938,11 +7181,31 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
>>>               }
>>>
>>>               if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {
>>
>> I'm having a sneaky feeling that, with enough bad luck, we
>> can go from seeing !owner->on_rq and here to actually finish
>> enqueuing the owner by the time we get to (1) ...
>>
>>> -                     /* XXX Don't handle blocked owners/delayed dequeue yet */
>>> +                     /*
>>> +                      * rq->curr must not be added to the blocked_head list or else
>>> +                      * ttwu_do_activate could enqueue it elsewhere before it switches
>>> +                      * out here. The approach to avoid this is the same as in the
>>> +                      * migrate_task case.
>>> +                      */
>>>                       if (curr_in_chain)
>>>                               return proxy_resched_idle(rq);
>>> -                     __clear_task_blocked_on(p, NULL);
>>> -                     goto deactivate;
>>> +                     /*
>>> +                      * If !@owner->on_rq, holding @rq->lock will not pin the task,
>>> +                      * so we cannot drop @mutex->wait_lock until we're sure its a blocked
>>> +                      * task on this rq.
>>> +                      *
>>> +                      * We use @owner->blocked_lock to serialize against ttwu_activate().
>>> +                      * Either we see its new owner->on_rq or it will see our list_add().
>>> +                      */
>>> +                     WARN_ON(owner == p);
>>> +                     raw_spin_unlock(&p->blocked_lock);
>>> +                     raw_spin_lock(&owner->blocked_lock);
>>
>> ... by the time we get here, activate_blocked_waiters() could have
>> already bailed out for:
>>
>>   !list_empty(&owner->blocked_activation_node)
>>
>> and then we go and add the task to list. I feel, like most conditionally
>> adding to list patterns, we should add the task to list, check the
>> condition once again under owner->blocked_lock, and then proceed.
>>
>> That way we can sure the activate will definitely see the queued task.
>> Sorry in advance in case I've been blind and missed something
>> obvious :-)
> 
> Hrm.  That's a good point!
> 
> Let me take a stab at fixing that.  I'm thinking it might be easier to
> check on_rq again after we take the blocked lock before calling
> proxy_enqueue_on_owner(), because otherwise unwinding that is a little
> complicated (pulling it off the list isn't, but the task will have
> been blocked, which isn't what we want if we are going to pick again
> to figure out if we can proxy or have to migrate the waiter).  Am I
> still missing a race with that approach (instead of adding to the list
> before the check)?

I think it is safe since the unlock on owner->blocked_on will act as a
release barrier ordering owner->on_rq = TASK_ON_RQ_QUEUED  when the
owner wakes up and the grabbing the owner->blocked_lock in
find_proxy_task() adds acquire semantic before the onwer->on_rq load.

Only downside is you'll need to check list_empty() within the
blocked_lock lock critical section.

> 
>> On a separate note, is it worth tracking if the task is a mutex / lock
>> owner and only go through the activate_blocked_waiters() path only when
>> a __mutex_owner() somewhere can return the task being activated? Perhaps
>> a per-task counter that increments at every mutex_lock() and decrements
>> at a mutex_unlock()?
> 
> This as an optimization to avoid taking the owner->blocked_lock and
> the list checking?

Ack! I had some more interesting (read stupid) ideas on top with that
but I'll try to see if it survives a boot before bombarding  you with
them :-)

> Let me think on that a bit.
> For normal mutexes it seems easy enough but I want to make sure it
> doesn't get complicated with ww_mutexes or the later proxy-rwsem
> support from the full series.

So, with rwsem, I recently realized there is {down,up}_read_non_owner()
which is interesting, but I don't think we set a p->blocked_on for those
cases and since the "owner" is always NULL, I doubt the limit usage of
it in the kernel even cares for priority inheritance.

Apart from that case, I don't think there is any other extra creative
locking patterns possible that don't have symmetry for lock and unlock.

> 
> thanks
> -john

-- 
Thanks and Regards,
Prateek

Re: [PATCH v30 7/7] sched: Add deactivated (sleeping) owner handling to find_proxy_task()

[John Stultz]

On Thu, Jul 9, 2026 at 9:04 PM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
> On 7/8/2026 10:04 AM, John Stultz wrote:
> > Let me take a stab at fixing that.  I'm thinking it might be easier to
> > check on_rq again after we take the blocked lock before calling
> > proxy_enqueue_on_owner(), because otherwise unwinding that is a little
> > complicated (pulling it off the list isn't, but the task will have
> > been blocked, which isn't what we want if we are going to pick again
> > to figure out if we can proxy or have to migrate the waiter).  Am I
> > still missing a race with that approach (instead of adding to the list
> > before the check)?
>
> I think it is safe since the unlock on owner->blocked_on will act as a
> release barrier ordering owner->on_rq = TASK_ON_RQ_QUEUED  when the
> owner wakes up and the grabbing the owner->blocked_lock in
> find_proxy_task() adds acquire semantic before the onwer->on_rq load.
>
> Only downside is you'll need to check list_empty() within the
> blocked_lock lock critical section.

I believe we currently do, but let me know if I'm missing something there.


> >> On a separate note, is it worth tracking if the task is a mutex / lock
> >> owner and only go through the activate_blocked_waiters() path only when
> >> a __mutex_owner() somewhere can return the task being activated? Perhaps
> >> a per-task counter that increments at every mutex_lock() and decrements
> >> at a mutex_unlock()?
> >
> > This as an optimization to avoid taking the owner->blocked_lock and
> > the list checking?
>
> Ack! I had some more interesting (read stupid) ideas on top with that
> but I'll try to see if it survives a boot before bombarding  you with
> them :-)

Ok, looking forward to it!

> > Let me think on that a bit.
> > For normal mutexes it seems easy enough but I want to make sure it
> > doesn't get complicated with ww_mutexes or the later proxy-rwsem
> > support from the full series.
>
> So, with rwsem, I recently realized there is {down,up}_read_non_owner()
> which is interesting, but I don't think we set a p->blocked_on for those
> cases and since the "owner" is always NULL, I doubt the limit usage of
> it in the kernel even cares for priority inheritance.
>
> Apart from that case, I don't think there is any other extra creative
> locking patterns possible that don't have symmetry for lock and unlock.

Thanks as always for your suggestions and guidance here! Very much appreciated!
-john