Re: [PATCH 17/18] sched_ext: scx_qmap: Add proxy execution support

[Andrea Righi <arighi@nvidia.com>]

Hi Richard,

On Tue, Sep 01, 2026 at 03:52:05PM +0800, Richard Cheng wrote:
> On Mon, Aug 31, 2026 at 03:42:27PM +0800, Andrea Righi wrote:
> > Add a -X option to opt scx_qmap into queueing mutex-blocked tasks for
> > proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear
> > and mutex waiters block normally. With -X, blocked donors are passed to
> > qmap_enqueue() with SCX_ENQ_BLOCKED.
> > 
> > When scx_qmap receives a blocked donor, select a cid allowed by the task
> > and held by qmap, preferring its current cid. Dispatch the donor to that
> > cid's local DSQ with a fresh slice and SCX_ENQ_PREEMPT, adding
> > SCX_ENQ_IMMED when the cid is time-shared. This places the donor at the
> > head of an eligible DSQ and requests an immediate reschedule, allowing
> > the core proxy-exec path to run the mutex owner using the donor's
> > scheduling context as soon as the donor is selected.
> > 
> > The blocked policy is intentionally unfair and can strongly prioritize
> > tasks using contended mutexes, but scx_qmap is a demo scheduler and such
> > aggressive behavior makes proxy-exec support easy to observe. Count all
> > blocked dispatch attempts in nr_enq_blocked and report their per-interval
> > delta.
> > 
> > Acked-by: John Stultz <jstultz@google.com>
> > Signed-off-by: Andrea Righi <arighi@nvidia.com>
> > ---
> >  tools/sched_ext/scx_qmap.bpf.c | 53 +++++++++++++++++++++++++++++++++-
> >  tools/sched_ext/scx_qmap.c     | 13 +++++++--
> >  tools/sched_ext/scx_qmap.h     |  1 +
> >  3 files changed, 63 insertions(+), 4 deletions(-)
> > 
> > diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
> > index 5bb8b90a275a3..21f8a9be9650c 100644
> > --- a/tools/sched_ext/scx_qmap.bpf.c
> > +++ b/tools/sched_ext/scx_qmap.bpf.c
> > @@ -439,6 +439,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
> >  	 */
> >  	taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;
> >  
> > +	if (enq_flags & SCX_ENQ_BLOCKED)
> > +		__sync_fetch_and_add(&qa.nr_enq_blocked, 1);
> > +
> >  	/*
> >  	 * A task of ours that can run on none of our self cids - the parent
> >  	 * didn't grant them or we delegated them to children - would starve in
> > @@ -446,7 +449,8 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
> >  	 *
> >  	 * Force it onto its first allowed cid's local DSQ. If we hold that cid
> >  	 * it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel
> > -	 * diverts the task to its rescue path.
> > +	 * diverts the task to its rescue path. Do this before the blocked-donor
> > +	 * fast paths, which also require an eligible self cid to make progress.
> >  	 */
> >  	if (!cmask_intersects(&taskc->cpus_allowed, &qa.self_cids.mask)) {
> >  		s32 c = cmask_next_set_wrap(&taskc->cpus_allowed, 0);
> > @@ -460,6 +464,53 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
> >  		}
> >  	}
> >  
> > +	/*
> > +	 * SCX_OPS_ALWAYS_ENQ_IMMED makes the local insertion below implicitly
> > +	 * carry SCX_ENQ_IMMED. If the CPU can't run the blocked donor immediately,
> > +	 * the core returns it through ops.enqueue() with SCX_ENQ_REENQ. Inserting
> > +	 * it into the same local DSQ would repeat the IMMED handback until the
> > +	 * scheduler is ejected. Move reenqueued blocked donors to the shared DSQ,
> > +	 * which doesn't carry SCX_ENQ_IMMED, so another CPU can consume them.
> > +	 */
> > +	if ((enq_flags & (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) ==
> > +	    (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) {
> > +		taskc->force_local = false;
> > +		scx_bpf_dsq_insert(p, SHARED_DSQ, 0, enq_flags);
> > +		cid = cmask_next_and2_set_wrap(&taskc->cpus_allowed,
> > +					       &qa.idle_cids.mask,
> > +					       &qa.self_cids.mask, 0);
> > +		if (cid < scx_bpf_nr_cids())
> > +			scx_bpf_kick_cid(cid, SCX_KICK_IDLE);
> > +		return;
> > +	}
> > +
> > +	/*
> > +	 * Insert a blocked mutex donor at the head of an eligible local DSQ with
> > +	 * a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate reschedule.
> > +	 * The test above guarantees that cpus_allowed intersects self_cids, but
> > +	 * the donor's current cid may have been delegated to a child. Search the
> > +	 * intersection starting at the current cid, preserving it when qmap still
> > +	 * holds it and wrapping to another eligible self cid otherwise.
> > +	 *
> > +	 * A self cid may be held exclusively with SCX_CAP_ENQ or time-shared with
> > +	 * only SCX_CAP_ENQ_IMMED. Add needs_immed() so either kind can accept the
> > +	 * local insertion instead of rejecting and reenqueuing the donor for a
> > +	 * capability miss. Once selected, the core proxy-exec path can run the
> > +	 * mutex owner using the donor's scheduling context.
> > +	 *
> > +	 * This policy is intentionally unfair and can strongly prioritize tasks
> > +	 * using contended mutexes; scx_qmap is a demonstration scheduler and
> > +	 * this behavior makes proxy-exec support easy to observe.
> > +	 */
> > +	if (enq_flags & SCX_ENQ_BLOCKED) {
> > +		cid = cmask_next_and_set_wrap(&taskc->cpus_allowed,
> > +					      &qa.self_cids.mask,
> > +					      scx_bpf_task_cid(p));
> > +		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cid, slice_ns,
> > +				   enq_flags | needs_immed(cid) | SCX_ENQ_PREEMPT);
> > +		return;
> > +	}
> > +
> 
> Hi Andrea,
> 
> Do we need to do serialization between ops.enqueue() and the paths which rebuild qa.self_cids.mask ?
> 
> IMHO, appy_partition() can clear or rebuild self_cids after the check but before cmask_next_and_set_wrap().
> If the intersection is NULL, the helper return CID, and then the value gets passed to needs_immed(), hiwch uses it as
> an index and do "SCX_DSQ_LOCAL_ON | cid". This will be an invalid CPU.
> 
> Maybe do "cid < scx_bpf_nr_cids()" before using it ?

I think you're right, the mask can change between the intersection check and
cmask_next_and_set_wrap(), so the returned cid needs to be validated before it
is passed to needs_immed() or encoded into the local DSQ id.

I don't think serialization is necessary here. If ownership changes after a
valid cid has been selected, the core's capability validation should handle the
race by rejecting and reenqueuing the task.

I'll add the bounds check and factor out the existing rescue placement.

Thanks,
-Andrea