From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2C0D363C61 for ; Sat, 31 Jan 2026 17:53:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769882011; cv=none; b=pYuhjksQMFFM6s5OBZGG7sIpNC1024Vea88TCBvzIOZ2z1FUKkp82+guii7m5ZmyrAvbgXJBZDgTIRb/8+zljmgC2KnE7qxgWWyGVgVd11vey1pM6Biue87FXeADzxupNC4oD+pDsHXzldDkrW8xuC5zkQa0oM5e93Nc757jxTg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769882011; c=relaxed/simple; bh=0irGEZ08+V/qRYUwn27aProVewgb11GukO6MAntRIcM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VhXZs0eSp7c9VRy2ujMvGiZ8OODAMo1p4T+8glIXU4sFMdcjnwaJDmH6ooNBwX5T5m36z4L7/ktSZEhMYUlQe8LAXg6fKdZlr34PgLOXPYFkC8cxq697Yk0XmmbbdaY9EJrWRXjv4zl/Mfb04cpXljKsaH+wddelulUZgD7m0K0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jpiecuch.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WZjzqyv3; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jpiecuch.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WZjzqyv3" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4801bceb317so28890355e9.1 for ; Sat, 31 Jan 2026 09:53:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1769882008; x=1770486808; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=jAEV9l2nl/oDQDI4VQvAn/MQQD4yxX75EO20H6p8bGQ=; b=WZjzqyv3Q0mUAh2E8/a71XN7ZIg8Tf1Thxj7t7l+rfg0FnfaWJ3EFxU6X1oF60+65H S58ZmyfWy0p2aU7LF90GTWrkuzuWtxd2TCpThM0LTVRj/uWBe2nOJHlvHQtANogzeqCr xCngVpZDzY9ZnDYvOV1MWsora+L6U7J75wblgdt/9dhpXFwKf0lICImMBhLFrWcC7bqW 6jcyWVqacvlKeu+q8kDvUSL3LdJ0KGjFvIkj7kSK5xkEx+d41YmMeOkYE6fmrxRj8Jzx Qkol1OyXil/GulYck8DzAFTfahrdJs2AAmSja+Wf/IzUm1yE+q7rZzdvdlPJEp8UgYzr p+ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769882008; x=1770486808; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=jAEV9l2nl/oDQDI4VQvAn/MQQD4yxX75EO20H6p8bGQ=; b=WZ4opa2dbmFraKIZKzcvtK9P1t2iuAZBgXpJyZCOzWdySQrhD7QzZfzX3feVbipcUc Vk+nXQTe2yms/PWncmI18swn+lxSylQhiFWuPTwkMu9Up00Ejqt0i5JijQYYcl6UYaZj TRhu24FoNr8bMZ6fa/Kfmpd+YnFtmSCqFnSCdRyx59iSGjnGqoLVoxfVDcTcBXM3+OzA Vu3bXqipl9rLMZY04e3C0QG0l1cYLy58RMnPRami8/mMIUg/wE6lUkVjoBZAXgUMBVFO U3vC9VdZOvyvzklkPVoBvIu+cDHpnnFlXH3b7vrl2Tqr+hhGCHyJGqxgAOlYktf4fSR/ haZQ== X-Forwarded-Encrypted: i=1; AJvYcCW8h/WZTeUje9YiVHSGss7wvWMgFHAxA5DugGXSePeeipyqqYyfC3Y2BbaqMuvV0rVScOdksRp79gRSouw=@vger.kernel.org X-Gm-Message-State: AOJu0YyF9K+XTMW2HvlRB98yRSp503DIZZwaEUOM5TL61kDLXH+bCNCC YEqw/EAq3a3KyRpL29d4vDQf+s5cfZnVNl+NXghJl4w6fZ+3xJvc0iLbMnT/9Ww5YzWOntom2O9 hKYJyyTJPQSTpsA== X-Received: from wmaj4.prod.google.com ([2002:a05:600c:6c04:b0:477:98b9:1e26]) (user=jpiecuch job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:45c5:b0:47d:403e:9cd5 with SMTP id 5b1f17b1804b1-482db45fcb4mr78570415e9.11.1769882008267; Sat, 31 Jan 2026 09:53:28 -0800 (PST) Date: Sat, 31 Jan 2026 17:53:27 +0000 In-Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260126084258.3798129-1-arighi@nvidia.com> <20260126084258.3798129-2-arighi@nvidia.com> X-Mailer: aerc 0.21.0-0-g5549850facc2 Message-ID: Subject: Re: [PATCH 1/2] sched_ext: Fix ops.dequeue() semantics From: Kuba Piecuch To: Andrea Righi , Kuba Piecuch Cc: Tejun Heo , David Vernet , Changwoo Min , Christian Loehle , Daniel Hodges , , , Emil Tsalapatis Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat Jan 31, 2026 at 9:02 AM UTC, Andrea Righi wrote: > On Fri, Jan 30, 2026 at 11:54:00AM +0000, Kuba Piecuch wrote: >> Is "local" short for "local or global", i.e. not user-created? >> Direct dispatching into the global DSQ also shouldn't trigger ops.dequeu= e(), >> since dispatch isn't necessary for the task to run. This follows from th= e last >> paragraph: >>=20 >> Note that, this way, whether ops.dequeue() needs to be called agrees w= ith >> whether the task needs to be dispatched to run. >>=20 >> I agree with your points, just wanted to clarify this one thing. > > I think this should be interpreted as local DSQs only > (SCX_DSQ_LOCAL / SCX_DSQ_LOCAL_ON), not any built-in DSQ. SCX_DSQ_GLOBAL = is > essentially a built-in user DSQ, provided for convenience, it's not reall= y > a "direct dispatch" DSQ. SCX_DSQ_GLOBAL is significantly different from user DSQs, because balance_o= ne() can pull tasks directly from SCX_DSQ_GLOBAL, while it cannot pull tasks fro= m user-created DSQs. If a BPF scheduler puts a task onto SCX_DSQ_GLOBAL, then it _must_ be ok wi= th balance_one() coming along and pulling that task without the BPF scheduler'= s intervention, so in that way I believe SCX_DSQ_GLOBAL is semantically quite similar to local DSQs. >> Here's my attempt at documenting this behavior: >>=20 >> After ops.enqueue() is called on a task, the task is owned by the BPF >> scheduler, provided the task wasn't direct-dispatched to a local/global = DSQ. >> When a task is owned by the BPF scheduler, the scheduler needs to dispat= ch the >> task to a local/global DSQ in order for it to run. >> When the BPF scheduler loses ownership of the task, either due to dispat= ching it >> to a local/global DSQ or due to external events (core-sched pick, CPU >> migration, scheduling property changes), the BPF scheduler is notified t= hrough >> ops.dequeue() with appropriate flags (TBD). > > This looks good overall, except for the global DSQ part. Also, it might b= e > better to avoid the term =E2=80=9Cowned=E2=80=9D, internally the kernel a= lready uses the > concept of "task ownership" with a different meaning (see > https://lore.kernel.org/all/aVHAZNbIJLLBHEXY@slm.duckdns.org), and reusin= g > it here could be misleading. > > With that in mind, I'd probably rephrase your documentation along these > lines: > > After ops.enqueue() is called, the task is considered *enqueued* by the B= PF > scheduler, unless it is directly dispatched to a local DSQ (via > SCX_DSQ_LOCAL or SCX_DSQ_LOCAL_ON). > > While a task is enqueued, the BPF scheduler must explicitly dispatch it t= o > a DSQ in order for it to run. > > When a task leaves the enqueued state (either because it is dispatched to= a > non-local DSQ, or due to external events such as a core-sched pick, CPU Shouldn't it be "dispatched to a local DSQ"? > migration, or scheduling property changes), ops.dequeue() is invoked to > notify the BPF scheduler, with flags indicating the reason for the dequeu= e: > regular dispatch dequeues have no flags set, whereas dequeues triggered b= y > scheduling property changes are reported with SCX_DEQ_SCHED_CHANGE. Core-sched dequeues also have a dedicated flag, it should probably be inclu= ded here. > > What do you think? I think using the term "enqueued" isn't very good either since it results i= n two ways in which a task can be considered enqueued: 1. Between ops.enqueue() and ops.dequeue() 2. Between enqueue_task_scx() and dequeue_task_scx() The two are not equivalent, since a task that's running is not enqueued according to 1. but is enqueued according to 2. I would be ok with it if we change it to something unambiguous, e.g. "BPF-enqueued", although that poses a risk of people getting lazy and using "enqueued" anyway. Some potential alternative terms: "resident"/"BPF-resident", "managed"/"BPF-managed", "dispatchable", "pending dispatch", or simply "pending". Thanks, Kuba