From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C598E3876BD for ; Tue, 11 Aug 2026 19:09:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786475382; cv=none; b=aenD1TTdUzQT0hHWEzHGyccDAcAc7svg9+y04F8y1ctwwgZUj2bUkh1bym7EGHW2OXVv8hBTDx7MJtP5i5YLo1o3QlvrBxfATJExHdNzcfG4+aQgFWfgeoYZ/ucKbAK/+N1XAir3kUx6p4s9RlOD3VixCQMi1h5u3Ire0ch0HsY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786475382; c=relaxed/simple; bh=Fnz9381Zxch8tIQaxNCH6+xs09Wqbk7JxCiTUg89YvM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=N/GKNzS9ZucGvXAH1LNOe6bOtdTNlvps7gTE7xkTuPlL9a7XOL909XM7rAA3EndO3m6Ls/j5ke6sQeYBkC6RI21RWL1p89TsJOpWWBHQ6iVMOCfDSeJHeq9ng5o1I9rxCXcxeK9tBBqHwc9a0w6kLegUuZIONwRz1sKiRpFpuSA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bMpg9WUs; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bMpg9WUs" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2cc61541f8cso18526115ad.0 for ; Tue, 11 Aug 2026 12:09:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786475379; x=1787080179; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=CH2kn2ZeoL6mmqRvDkFx9DwwHDwyqtUCvnZGPS0Pyhg=; b=bMpg9WUsIq5V9/2GHl2CpTY5RSg6eNhOcfwg3ipzRQBgn0N0mM+zPedwF2j3ZAk6AY x8LMkAl66CqsJ6LrsCAgX3crEH9I/MyTi4CEuhpWkgFnJukC9wcTxsZF150/RLdXtPl2 6aETNdy6C6W2dFvu5RPZXpx6thXC/omB5MKITL0OtDOWUHCB/9OUAzMelmCEDFdBhdxu XGdIxYZ5i37FtIRIvkpOZGeit7sbS08gzm9vsiSI5pufd0iPc3rNd/EGjPa83Ze64aQ4 6U3C3wbIolWJbfyOWOOpAFlz1lvgW5c8xr+XmTWIMX9qWbZ0HJOFc6bWRN+YGZ+S9NMt vlUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786475379; x=1787080179; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=CH2kn2ZeoL6mmqRvDkFx9DwwHDwyqtUCvnZGPS0Pyhg=; b=AOhPaPxR/vSF1y4vr/8dT+/lP+S8pLaCpstebZDMGGh9JHryNbKbyXVWFygtBc6weg 7BIdxuJrbtHoUe5PSH0QpwsAOeeJDKyyrD8jb17oYkdbNIjgYcnpCVxJrmU8u0tzb9mK ObmGmWeLrXwc+j12kT2ziwB+HwCPg264hA1/ZfGSNwN1cQ3Fx4IaalsKkp+eqELRZ4/O KRQWzH6RGuMzl9tjvt9I8ETYRHS+bVJqPwqfHyZDMkeeyZeHFKEElAc/2e6rcei05PAa SYU8+x8In4Jk625wiVXtlmrS/DbkCUmF3hla4jnnFcfihzNh/leiuONHbXCIKjVfn7Ld A3Rg== X-Gm-Message-State: AOJu0YxxEmhBGCA5kECR8itIPlB6nH/i+XNKr/xXPl6+iGCMpCs5RZJb ivCqRZu6ldfovOA4SfQy8pvHmC9nG8WWFBUF/jaBDArXnmzy7yru7cY5 X-Gm-Gg: AR+sD11pY3tV60PX1vY4aMFjgI2xaS6wqFgNk1qobSwo0D84WS02+abatebBTrP8fOW gYi2wWg4pr4NuiKp0k1bQ6RG+o6d2AdAwHDt8vZxXJMVPlWF3uoBy2a8MK4dV9etjJWR0V0Ljf/ OiOodelT6hYNpWkDGMUP6m8gsmYzv0Tgdu+k08GWuGxa3uTgJ0UaD3QopKQNyL7P3Bw4nzvgBnm vI4fAINg7YeRxcfS5UyOpLMafmwBZ3WNRt0afkiYGUCRpUlWx7S0ceUCbGdSSVdVZevh6DmGniQ O4A+BqVts0mAywxhOKDyrz7Qx8otmpXdOTrpL5LZcqHFtjueuBATn4F8eNHDV0qwC1TdQfsA950 P5Z/lwdBYPj1bhRdEjGZGSW2zO+S5L8yeHtN+5OBGScfNOh6YjfZ5LH/3qyyeogmaNnpprREjeJ hfeJgQhu43Mxw0U3spKsYWlroG2I4uTVtvLIDvDAAtOWK1dXjxEacSZkGcpUvmsSvUCw== X-Received: by 2002:a17:90b:2801:b0:38e:a52e:a3c9 with SMTP id 98e67ed59e1d1-392f3f6bf0bmr1357793a91.8.1786475378844; Tue, 11 Aug 2026 12:09:38 -0700 (PDT) Received: from atom0118 ([2401:4900:1f25:7356:bfe2:e97a:e2db:b6c9]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-1412458461bsm3171114c88.13.2026.08.11.12.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 12:09:38 -0700 (PDT) Date: Wed, 12 Aug 2026 00:38:47 +0530 From: Atul Kumar Pant To: John Stultz Cc: LKML , Peter Zijlstra , Juri Lelli , Valentin Schneider , Connor O'Brien , Joel Fernandes , Qais Yousef , Ingo Molnar , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Subject: Re: [RESEND][PATCH v31 8/9] sched: Add deactivated (sleeping) owner handling to find_proxy_task() Message-ID: References: <20260807035232.1881495-1-jstultz@google.com> <20260807035232.1881495-9-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260807035232.1881495-9-jstultz@google.com> On Fri, Aug 07, 2026 at 03:52:14AM +0000, John Stultz wrote: > From: Peter Zijlstra > > If the blocked_on chain resolves to a sleeping owner, deactivate > the donor task, and enqueue it on the sleeping owner task. > Then re-activate it later when the owner is woken up. > > NOTE: This has been particularly challenging to get working > properly, and some of the locking is particularly awkward. I'd > very much appreciate review and feedback for ways to simplify > this. > > Signed-off-by: Peter Zijlstra (Intel) > Signed-off-by: Juri Lelli > Signed-off-by: Valentin Schneider > Signed-off-by: Connor O'Brien > [jstultz: This was broken out from the larger proxy() patch] > Signed-off-by: John Stultz > --- > v5: > * Split out from larger proxy patch > v6: > * Major rework, replacing the single list head per task with > per-task list head and nodes, creating a tree structure so > we only wake up descendants of the task woken. > * Reworked the locking to take the task->pi_lock, so we can > avoid mid-chain wakeup races from try_to_wake_up() called by > the ww_mutex logic. > v7: > * Drop unnecessary __nested lock annotation, as we already drop > the lock prior. > * Add comments on #else & #endif lines, and clearer function > names, and commit message tweaks as suggested by Metin Kaya > * Move activate_blocked_entities() call from ttwu_queue to > try_to_wake_up() to simplify locking. Thanks to questions from > Metin Kaya > * Fix irqsave/irqrestore usage now we call this outside where > the pi_lock is held > * Fix activate_blocked_entitites not preserving wake_cpu > * Fix for UP builds > v8: > * Minor checkpatch fixup > * Drop proxy_deactivate and cleanups suggested by Metin > v9: > * Fix bug causing possibly uninitialized cpu value to be used with > activate_blocked_entities() > * Improved comment around preserving wake_cpu suggested by Metin > * Add additional lockdep asserts, suggested by Metin > * Tweaked placement of lockdep assert, suggested by Metin > * Fixed comment referring to structure entry name > * Fix to call proxy_resched_idle() _prior_ to calling > proxy_enqueue_on_owner() where we deactivate the task, this > avoids stale references to rq_selected() when the task may > have been migrated to another rq. > * Fix to remove the blocked_head list at the start of > activate_blocked_entities() so we only do a finite amount > of work, avoiding a potential livelock of two cpus removing > and adding tasks to the list at the same time if the owner > went back to sleep while blocked entities were being woken. > v11: > * Big rework to get rid of recursion. Had to add another list > item to the task_stuct to do this as we are in atomic context > and cannot allocate memory while activating blocked entities. > Will need to watch carefully for bugs, as switching to a > list_head in the task_struct instead of a pointer on the > stack opens up the potential for races on the shared state, > but I think I've got the locking sorted. > * Moved proxy_set_task_cpu helper to earlier in the series > * Minor rework for try_to_deactivate_task changes > * Minor variable name cleanups suggested by Metin > v13: > * Switch to use donor from next for proxy_enqueue_on_owner > * Switch to using block_task instead of deactivate_task > v14: > * Ensure we call block_task() last in proxy_enqueue_on_owner > and not touch it again to avoid races where it might be > activated on another cpu > * Make sure we activate blocked_entities when we exit from ttwu > * Fix to enqueue the last task in the chain (p) on the blocked > owner instead of donor, so that we preserve the chain > structure so mid-chain wakeups propagate properly > * Rework of sleeping_owner handling so that we properly deal > with delayed-dequeued (sched_delayed) tasks (also removes > now unused proxy_deactivate() logic) > v15: > * Rework do_activate_task to be activate_task() and have it > call __activate_task(), suggested by Carlos Llamas > * Put task_struct additions under CONFIG_SCHED_PROXY_EXEC > v16: > * Rework do_activate_blocked_waiter locking to use scoped_guard > * Rework find_proxy_task() logic to use guard > v18: > * Integrate Suleiman's suggested optimization to check sleeping > owner status before task_cpu, to avoid unnecssarily > proxy-migrating tasks to then just dequeue them. > * Add on_cpu check to fix for a very hard to reproduce race where > a late blocked_task_activation() happens while the task is already > on_cpu elsewhere. When we get to __schedule() we block the task > (!on_rq) but haven't yet switched away. blocked_task_activation() > would then incorrectly activate on a different runqueue. > * Add init_task initialization for sleeping owner lists, as > suggested by Suleiman > v19: > * Build fixup for !CONFIG_SMP > v22: > * Rework to avoid gotos in guard() scopes, using break and > switch() on action values. Suggested by K Prateek. > v25: > * Fix elevated nr_uninterruptible and nr_iowait counts, > which were causing bad loadavg values. Reported and > fixed by David Stevens > v28: > * In __proxy_remove_from_sleeping_owner() we call > put_task(), which might free the owner while we are > still holing the owners->blocked_lock. So be sure > to get_task()/put_task() around the owner usage to > ensure we don't prematurely free the task. > * Also reorder the put_task/unlock lines in > activate_blocked_waiters(), to avoid a similar issue > v30: > * Optimize activate_blocked_waiters() so we don't do so much > unnecessary work when proxy-exec is enabled. If there are > no blocked waiters, we can return early. > v31: > * K Prateek's noticed a potential race where if a sleeping owner > is woken in paralell with the waiter enquing itself on the > sleeping owner, the waiting task could get stuck on that owner > until it sleeps and is woken up again. So double check after > we grab the blocked_lock to avoid this. > * Maria Yu and Tengfei Fan reported an issue with rq->nr_iowait > values getting out of balance, and provided a helpful > reproducer. To fix this, make sure we do the atomic_dec() in > do_activate_blocked_waiter() before we call > proxy_set_task_cpu() so we always dec on the same rq we > inc'ed. > > Cc: Joel Fernandes > Cc: Qais Yousef > Cc: Ingo Molnar > Cc: Peter Zijlstra > Cc: Juri Lelli > Cc: Vincent Guittot > Cc: Dietmar Eggemann > Cc: Valentin Schneider > Cc: Steven Rostedt > Cc: Ben Segall > Cc: Zimuzo Ezeozue > Cc: Mel Gorman > Cc: Will Deacon > Cc: Waiman Long > Cc: Boqun Feng > Cc: "Paul E. McKenney" > Cc: Metin Kaya > Cc: Xuewen Yan > Cc: K Prateek Nayak > Cc: Thomas Gleixner > Cc: Daniel Lezcano > Cc: Suleiman Souhlal > Cc: Andrea Righi > Cc: kuyo chang > Cc: hupu > Cc: kernel-team@android.com > --- > include/linux/sched.h | 7 + > init/init_task.c | 6 + > kernel/fork.c | 6 + > kernel/sched/core.c | 317 +++++++++++++++++++++++++++++++++++++++--- > 4 files changed, 315 insertions(+), 21 deletions(-) > > diff --git a/include/linux/sched.h b/include/linux/sched.h > index 373bcc0598d10..8c2ba6dce58f4 100644 > --- a/include/linux/sched.h > +++ b/include/linux/sched.h > @@ -1251,6 +1251,13 @@ struct task_struct { > > struct mutex *blocked_on; /* lock we're blocked on */ > raw_spinlock_t blocked_lock; > +#ifdef CONFIG_SCHED_PROXY_EXEC > + struct list_head blocked_head; /* tasks blocked on this task */ > + struct list_head blocked_node; /* our entry on someone elses blocked_head */ > + /* Node for list of tasks to process blocked_head list for blocked entitiy activations */ > + struct list_head blocked_activation_node; > + struct task_struct *sleeping_owner; /* task our blocked_node is enqueued on */ > +#endif > > /* > * The task that is boosting this task; a back link for the current > diff --git a/init/init_task.c b/init/init_task.c > index b67ef6040a655..809282a2741d7 100644 > --- a/init/init_task.c > +++ b/init/init_task.c > @@ -211,6 +211,12 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { > &init_task.alloc_lock), > #endif > .blocked_donor = NULL, > +#ifdef CONFIG_SCHED_PROXY_EXEC > + .blocked_head = LIST_HEAD_INIT(init_task.blocked_head), > + .blocked_node = LIST_HEAD_INIT(init_task.blocked_node), > + .blocked_activation_node = LIST_HEAD_INIT(init_task.blocked_activation_node), > + .sleeping_owner = NULL, > +#endif > #ifdef CONFIG_RT_MUTEXES > .pi_waiters = RB_ROOT_CACHED, > .pi_top_task = NULL, > diff --git a/kernel/fork.c b/kernel/fork.c > index f0e2e131a9a5a..6dfde115fe34e 100644 > --- a/kernel/fork.c > +++ b/kernel/fork.c > @@ -2247,6 +2247,12 @@ __latent_entropy struct task_struct *copy_process( > > p->blocked_on = NULL; /* not blocked yet */ > p->blocked_donor = NULL; /* nobody is boosting p yet */ > +#ifdef CONFIG_SCHED_PROXY_EXEC > + INIT_LIST_HEAD(&p->blocked_head); > + INIT_LIST_HEAD(&p->blocked_node); > + INIT_LIST_HEAD(&p->blocked_activation_node); > + p->sleeping_owner = NULL; > +#endif > > #ifdef CONFIG_BCACHE > p->sequential_io = 0; > diff --git a/kernel/sched/core.c b/kernel/sched/core.c > index 1bf60d78c9208..d813a9c2f0c35 100644 > --- a/kernel/sched/core.c > +++ b/kernel/sched/core.c > @@ -2216,7 +2216,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) > return p->sched_class->dequeue_task(rq, p, flags); > } > > -void activate_task(struct rq *rq, struct task_struct *p, int flags) > +static inline void __activate_task(struct rq *rq, struct task_struct *p, int flags) > { > if (task_on_rq_migrating(p)) > flags |= ENQUEUE_MIGRATED; > @@ -2227,6 +2227,71 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags) > ASSERT_EXCLUSIVE_WRITER(p->on_rq); > } > > +#ifdef CONFIG_SCHED_PROXY_EXEC > +static inline > +void __proxy_remove_from_sleeping_owner(struct task_struct *owner, struct task_struct *p) > +{ > + lockdep_assert_held(&owner->blocked_lock); > + > + if (p->sleeping_owner == owner) { > + list_del_init(&p->blocked_node); > + WRITE_ONCE(p->sleeping_owner, NULL); > + put_task_struct(owner); // matches get in proxy_enqueue_on_owner > + } > +} > + > +static inline void proxy_remove_from_sleeping_owner(struct task_struct *p) > +{ > + struct task_struct *owner = READ_ONCE(p->sleeping_owner); > + > + if (owner) { > + /* > + * __proxy_remove_from_sleeping_owner() does a > + * put on owner to match the get done in > + * proxy_enqueue_on_owner(). If that put is the > + * last one and it frees owner, we'd be freeing > + * a lock we held. So get/put owner around its > + * usage her to ensure that doesn't happen. > + */ > + get_task_struct(owner); > + raw_spin_lock(&owner->blocked_lock); > + __proxy_remove_from_sleeping_owner(owner, p); > + raw_spin_unlock(&owner->blocked_lock); > + put_task_struct(owner); > + } > +} > + > +void activate_task(struct rq *rq, struct task_struct *p, int en_flags) > +{ > + if (!sched_proxy_exec()) { > + __activate_task(rq, p, en_flags); > + return; > + } > + > + lockdep_assert_rq_held(rq); > + proxy_remove_from_sleeping_owner(p); > + /* > + * By calling __activate_task() with blocked_lock held, we > + * order against the find_proxy_task() blocked_task case > + * such that no more blocked tasks will be enqueued on p > + * once we release p->blocked_lock. > + */ > + raw_spin_lock(&p->blocked_lock); > + WARN_ON(task_cpu(p) != cpu_of(rq)); > + __activate_task(rq, p, en_flags); > + raw_spin_unlock(&p->blocked_lock); > +} > +#else > +static inline void proxy_remove_from_sleeping_owner(struct task_struct *p) > +{ > +} > + > +void activate_task(struct rq *rq, struct task_struct *p, int en_flags) > +{ > + __activate_task(rq, p, en_flags); > +} > +#endif > + > void deactivate_task(struct rq *rq, struct task_struct *p, int flags) > { > WARN_ON_ONCE(flags & DEQUEUE_SLEEP); > @@ -3756,6 +3821,169 @@ static inline void proxy_reset_donor(struct rq *rq) > resched_curr(rq); > } > > +static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) > +{ > + unsigned int wake_cpu; > + > + /* > + * Since we are enqueuing a blocked task on a cpu it may > + * not be able to run on, preserve wake_cpu when we > + * __set_task_cpu so we can return the task to where it > + * was previously runnable. > + */ > + wake_cpu = p->wake_cpu; > + __set_task_cpu(p, cpu); > + p->wake_cpu = wake_cpu; > +} > + > +static void do_activate_blocked_waiter(struct rq *target_rq, struct task_struct *p, int en_flags) > +{ > + unsigned int state; > + struct rq_flags rf; > + int target_cpu = cpu_of(target_rq); > + > + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { > + state = READ_ONCE(p->__state); > + /* Avoid racing with ttwu */ > + if (state == TASK_WAKING) > + return; > + > + if (READ_ONCE(p->on_rq)) { > + /* > + * We raced with a non mutex handoff activation of p. > + * That activation will also take care of activating > + * all of the tasks after p in the blocked_head list, > + * so we're done here. > + */ > + return; > + } > + if (task_on_cpu(task_rq(p), p)) { > + /* > + * Its possible this activation is very late, and > + * we already were woken up and are running on a > + * different cpu. If that task blocked, it could be > + * dequeued (so on_rq == 0), but still on_cpu. > + * Bail in this case, as we definitely don't want to > + * activate a task when its on_cpu elsewhere. > + */ > + return; > + } Hi John, one doubt, can It happen that this task 'p' has already finished running in between the time when it was was picked from blocked list (activate_blocked_waiters()) and this function? I mean, is it possible for 'p' that task_is_blocked() is false? Thanks, Atul > + /* > + * Have to make sure we handle nr_iowait adjustment before > + * we call proxy_set_task_cpu() to ensure we are adjusting > + * the same runqueue we left (where block_task() > + * incremented nr_iowait). > + */ > + if (p->in_iowait) { > + delayacct_blkio_end(p); > + atomic_dec(&task_rq(p)->nr_iowait); > + } > + proxy_set_task_cpu(p, target_cpu); > + rq_lock_irqsave(target_rq, &rf); > + /* > + * proxy_enqueue_on_owner() called block_task() which > + * increments nr_uninterruptible, so we need to reverse > + * that when we activate the blocked waiter > + */ > + if (p->sched_contributes_to_load) > + target_rq->nr_uninterruptible--; > + update_rq_clock(target_rq); > + activate_task(target_rq, p, en_flags); > + resched_curr(target_rq); > + rq_unlock_irqrestore(target_rq, &rf); > + } > +} > + > +static void activate_blocked_waiters(struct rq *target_rq, > + struct task_struct *owner, > + int wake_flags) > +{ > + struct list_head bal_head; > + unsigned long flags; > + int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK; > + > + if (!sched_proxy_exec()) > + return; > + > + /* > + * A whole bunch of waiting donor tasks back this blocked > + * lock owner task, wake them all up to give this task its > + * 'fair' share. > + * > + * This is a little unique here and the locking is messy. > + * At this point we only hold the blocked_lock, so the > + * owner task may be able to run and do all sorts of > + * things while we are processing the blocked_head list, > + * including going back to sleep, which can cause tasks > + * to be added to the owners->blocked_head while we are > + * processing it! > + * Thus, we pull the entire list off the owner->blocked_head > + * here so that we will only process a finite amount of > + * tasks. Tasks added after this will be processed by the > + * future wake events. > + * Even though we have pulled the list off the blocked_head > + * the removed list is *still* "owned" and serialized by > + * the owner->blocked_lock! As we have to serialize against > + * mid-chain wakeups, who may try to remove themselves from > + * the list. > + */ > + raw_spin_lock_irqsave(&owner->blocked_lock, flags); > + if (!list_empty(&owner->blocked_activation_node)) { > + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); > + return; > + } > + > + if (list_empty(&owner->blocked_head)) { > + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); > + return; > + } > + > + get_task_struct(owner); > + INIT_LIST_HEAD(&bal_head); > + list_add_tail(&owner->blocked_activation_node, &bal_head); > + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); > + > + if (wake_flags & WF_MIGRATED) > + en_flags |= ENQUEUE_MIGRATED; > + > + while (!list_empty(&bal_head)) { > + struct list_head tmp_head; > + > + INIT_LIST_HEAD(&tmp_head); > + owner = list_first_entry(&bal_head, struct task_struct, blocked_activation_node); > + > + raw_spin_lock_irqsave(&owner->blocked_lock, flags); > + list_replace_init(&owner->blocked_head, &tmp_head); > + list_del_init(&owner->blocked_activation_node); > + while (!list_empty(&tmp_head)) { > + struct task_struct *p; > + > + p = list_first_entry(&tmp_head, > + struct task_struct, > + blocked_node); > + WARN_ON(p == owner); > + WARN_ON(p->sleeping_owner != owner); > + __proxy_remove_from_sleeping_owner(owner, p); > + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); > + > + do_activate_blocked_waiter(target_rq, p, en_flags); > + > + raw_spin_lock_irqsave(&p->blocked_lock, flags); > + if (list_empty(&p->blocked_activation_node)) { > + get_task_struct(p); > + list_add_tail(&p->blocked_activation_node, &bal_head); > + } > + raw_spin_unlock_irqrestore(&p->blocked_lock, flags); > + > + raw_spin_lock_irqsave(&owner->blocked_lock, flags); > + } > + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); > + put_task_struct(owner); // put matches get prior to adding to local bal_head > + } > +} > + > +static inline struct task_struct *proxy_resched_idle(struct rq *rq); > + > /* > * Checks to see if task p has been proxy-migrated to another rq > * and needs to be returned. If so, we deactivate the task here > @@ -3800,6 +4028,10 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p) > { > return false; > } > + > +static inline void activate_blocked_waiters(struct rq *target_rq, > + struct task_struct *owner, > + int wake_flags) {} > #endif /* CONFIG_SCHED_PROXY_EXEC */ > > static void > @@ -3873,8 +4105,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags) > > update_rq_clock(rq); > if (p->is_blocked) { > - if (p->se.sched_delayed) > + if (p->se.sched_delayed) { > + proxy_remove_from_sleeping_owner(p); > enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED); > + } > if (proxy_needs_return(rq, p)) > return 0; > } > @@ -3903,13 +4137,19 @@ void sched_ttwu_pending(void *arg) > update_rq_clock(rq); > > llist_for_each_entry_safe(p, t, llist, wake_entry.llist) { > + int wake_flags; > if (WARN_ON_ONCE(p->on_cpu)) > smp_cond_load_acquire(&p->on_cpu, !VAL); > > if (WARN_ON_ONCE(task_cpu(p) != cpu_of(rq))) > set_task_cpu(p, cpu_of(rq)); > > - ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); > + wake_flags = p->sched_remote_wakeup ? WF_MIGRATED : 0; > + ttwu_do_activate(rq, p, wake_flags, &rf); > + rq_unlock(rq, &rf); > + activate_blocked_waiters(rq, p, wake_flags); > + rq_lock(rq, &rf); > + update_rq_clock(rq); > } > > /* > @@ -4416,6 +4656,7 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) > ttwu_queue(p, cpu, wake_flags); > } > out: > + activate_blocked_waiters(cpu_rq(task_cpu(p)), p, wake_flags); > if (success) > ttwu_stat(p, task_cpu(p), wake_flags); > > @@ -6731,21 +6972,6 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p, > } > > #ifdef CONFIG_SCHED_PROXY_EXEC > -static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) > -{ > - unsigned int wake_cpu; > - > - /* > - * Since we are enqueuing a blocked task on a cpu it may > - * not be able to run on, preserve wake_cpu when we > - * __set_task_cpu so we can return the task to where it > - * was previously runnable. > - */ > - wake_cpu = p->wake_cpu; > - __set_task_cpu(p, cpu); > - p->wake_cpu = wake_cpu; > -} > - > static inline struct task_struct *proxy_resched_idle(struct rq *rq) > { > put_prev_set_next_task(rq, rq->donor, rq->idle); > @@ -6852,6 +7078,29 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf, > proxy_reacquire_rq_lock(rq, rf); > } > > +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owner, > + struct task_struct *p) > +{ > + lockdep_assert_rq_held(rq); > + lockdep_assert_held(&owner->blocked_lock); > + /* > + * ttwu_activate() will pick them up and place them on whatever rq > + * @owner will run next. > + */ > + WARN_ON(p == owner); > + WARN_ON(!p->on_rq); > + WARN_ON(p->sleeping_owner); > + get_task_struct(owner); > + WRITE_ONCE(p->sleeping_owner, owner); > + /* > + * ttwu_do_activate must not have a chance to activate p > + * elsewhere before it's fully extricated from its old rq. > + */ > + list_add(&p->blocked_node, &owner->blocked_head); > + proxy_resched_idle(rq); > + block_task(rq, p, READ_ONCE(p->__state)); > +} > + > /* > * Find runnable lock owner to proxy for mutex blocked donor > * > @@ -6938,11 +7187,37 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) > } > > if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) { > - /* XXX Don't handle blocked owners/delayed dequeue yet */ > + /* > + * rq->curr must not be added to the blocked_head list or else > + * ttwu_do_activate could enqueue it elsewhere before it switches > + * out here. The approach to avoid this is the same as in the > + * migrate_task case. > + */ > if (curr_in_chain) > return proxy_resched_idle(rq); > - __clear_task_blocked_on(p, NULL); > - goto deactivate; > + /* > + * If !@owner->on_rq, holding @rq->lock will not pin the task, > + * so we cannot drop @mutex->wait_lock until we're sure its a blocked > + * task on this rq. > + * > + * We use @owner->blocked_lock to serialize against ttwu_activate(). > + * Either we see its new owner->on_rq or it will see our list_add(). > + */ > + WARN_ON(owner == p); > + raw_spin_unlock(&p->blocked_lock); > + raw_spin_lock(&owner->blocked_lock); > + /* > + * Before actually adding to the sleeping owner, double check > + * we didn't race with an owner wakeup before grabbing the > + * owner's blocked_lock. > + */ > + if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) > + proxy_enqueue_on_owner(rq, owner, p); > + > + raw_spin_unlock(&owner->blocked_lock); > + raw_spin_lock(&p->blocked_lock); > + > + return NULL; /* retry task selection */ > } > > owner_cpu = task_cpu(owner); > -- > 2.55.0.654.g21b8a5bc05-goog >