From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 501D536EA8A for ; Fri, 30 Jan 2026 11:54:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769774045; cv=none; b=Vpzp5n8wdci48pIOplVHtLvmjcLw6nu1nzznwCcmlTaKrSNJTds/Vhvwxo1keEi0ktxngWiAkcriehUdFd4o2H7TXk3RARrlrYx7FHbeBNo++2scN7bfVk3EicjTYj12W+SomhVfvgKANNSuFswlSQQ0dHZVZqvxefMIPiBe8q8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769774045; c=relaxed/simple; bh=TK+mA/mAevtJOngvguRrsd6I7cJZvzDA2qawlfZBSDg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DIuSpgykh5IrQa5DAy6OYeSmrKoViuZQhBcEgt6uSFj0TFdrn5YfLt60iKqTghfLJV7iEeDlmFjeya/yK6cv31GAiYvzqmkdAp0FlQH2dJ3xwk8+oneu4ME+Hj7E0KamlK2SiZ4amJVoU/u+KTdmxgwwc5a8ZbXsMJDGrc854aU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jpiecuch.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2sCr6uI0; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jpiecuch.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2sCr6uI0" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-48069a43217so19547545e9.1 for ; Fri, 30 Jan 2026 03:54:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1769774041; x=1770378841; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yWLfQaMCxd/sRhYR0V/QyqItxfbBpgyNHdrxmNX7bC8=; b=2sCr6uI0p8JIm/RkKN4nJdC9mRtf2utWQ80gJTe/Xmo4nZwVXdERjbc59gYClgWdoT qsByxoOA5K+s0OXSFw52hghOk4i5SHN3sxdGmbwuA45mZYxBSDbudCqY8APFKQSmv8fo x/w0QECimEy0ltHWEl0l0bnet+4vWWj2TBZO9YcqymSQ/COn2uR/4gFEvc8uNaPm8yAY E3S53iEl3ET53a6yH4+z4n+X0oMrQMo1qn5l/I7mTte1Gzca20IEey8yP2p2DFZVooBD H5T/brLENtmZxeSwgGxah3GuhHEqZXojlYvXeiGQ/L/VbZuOO0aMw+UkrJSkD6dIAqgI jQqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769774041; x=1770378841; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yWLfQaMCxd/sRhYR0V/QyqItxfbBpgyNHdrxmNX7bC8=; b=UChnSxMFbRtvDCbkGeoPSujSoWf+01UF+aa66ODM2ZB/+1H00fwqhRUMH1snQESLvk nIoh39WBliF+bMC1j31HYuqTyTYWy6VjvFSSSzo4pmkM3MCZp0YNaEAvyOYWIx71TEpC qYLdCeqIEZArYNvP4YjspVOIOlp0e9MxPcxkMeX5OaqLYlPtpY6dEZNqmhcuDRILJKZp xODPvBRX9GRItvHaGjA1OZgOGf664mbsmJi3mXQ58IkUrZU7ghXOku1vGK37af3uIBNM oPu1HNhvW2klr9/Y6d63rDF9elTun5B1svmGDyyw5wgy9Rk1rbjhNfiw//rS/Mr4Oabp VR/w== X-Forwarded-Encrypted: i=1; AJvYcCU3+hZwks5N9kHQX6+98siD1GoTSCnG2Rx6+AOafXRFeaBTkUqxi6myMBOrw9p1l7NmpW+eNvyoR6u+eTI=@vger.kernel.org X-Gm-Message-State: AOJu0Yz7iupmCD+Aq+7CclgaKAjd/Yoa1FkmDNUbiOB/4GEpB9S5/Z4+ muj107ohMLMbs/8JtQIOqJwyRC3yDpLepAO6sB7bKEVwPpKJF8CZlGLpaQM0tGxs0eMx7sA02it IxmTeHeoEmZ5J3g== X-Received: from wmfv6.prod.google.com ([2002:a05:600c:15c6:b0:480:6c48:f9ad]) (user=jpiecuch job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:310f:b0:477:7ae0:cd6e with SMTP id 5b1f17b1804b1-482db493e6bmr29179215e9.5.1769774041293; Fri, 30 Jan 2026 03:54:01 -0800 (PST) Date: Fri, 30 Jan 2026 11:54:00 +0000 In-Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260126084258.3798129-1-arighi@nvidia.com> <20260126084258.3798129-2-arighi@nvidia.com> X-Mailer: aerc 0.21.0-0-g5549850facc2 Message-ID: Subject: Re: [PATCH 1/2] sched_ext: Fix ops.dequeue() semantics From: Kuba Piecuch To: Tejun Heo , Andrea Righi Cc: David Vernet , Changwoo Min , Kuba Piecuch , Christian Loehle , Daniel Hodges , , , Emil Tsalapatis Content-Type: text/plain; charset="UTF-8" Hi Tejun, On Wed Jan 28, 2026 at 9:21 PM UTC, Tejun Heo wrote: ... > 1. When to call ops.dequeue()? > > I'm not sure whether deciding whether to call ops.dequeue() solely onwhether > ops.enqueue() was called. Direct dispatch has been expanded to include other > DSQs but was originally added as a way to shortcut the dispatch path and > "dispatch directly" for execution from ops.select_cpu/enqueue() paths. ie. > When a task is dispatched directly to a local DSQ, the BPF scheduler is done > with that task - the task is now in the same state with tasks that get > dispatched to a local DSQ from ops.dispatch(). > > ie. What effectively decides whether a task left the BPF scheduler is > whether the task reached a local DSQ or not, and direct dispatching into a > local DSQ shouldn't trigger ops.dequeue() - the task never really "queues" > on the BPF scheduler. Is "local" short for "local or global", i.e. not user-created? Direct dispatching into the global DSQ also shouldn't trigger ops.dequeue(), since dispatch isn't necessary for the task to run. This follows from the last paragraph: Note that, this way, whether ops.dequeue() needs to be called agrees with whether the task needs to be dispatched to run. I agree with your points, just wanted to clarify this one thing. > > This creates another discrepancy - From ops.enqueue(), direct dispatching > into a non-local DSQ clearly makes the task enter the BPF scheduler and thus > its departure should trigger ops.dequeue(). What about a task which is > direct dispatched to a non-local DSQ from ops.select_cpu()? Superficially, > the right thing to do seems to skip ops.dequeue(). After all, the task has > never been ops.enqueue()'d. However, I think this is another case where > what's obvious doesn't agree with what's happening underneath. > > ops.select_cpu() cannot actually queue anything. It's too early. Direct > dispatch from ops.select_cpu() is a shortcut to schedule direct dispatch > once the enqueue path is invoked so that the BPF scheudler can avoid > invocation of ops.enqueue() when the decision has already been made. While > this shortcut was added for convenience (so that e.g. the BPF scheduler > doesn't have to pass a note from ops.select_cpu() to ops.enqueue()), it has > real performance implications as it does save a roundtrip through > ops.enqueue() and we know that such overheads do matter for some use cases > (e.g. maximizing FPS on certain games). > > So, while more subtle on the surface, I think the right thing to do is > basing the decision to call ops.dequeue() on the task's actual state - > ops.dequeue() should be called if the task is "on" the BPF scheduler - ie. > if the task ran ops.select_cpu/enqueue() paths and ended up in a non-local > DSQ or on the BPF side. > > The subtlety would need clear documentation and we probably want to allow > ops.dequeue() to distinguish different cases. If you boil it down to the > actual task state, I don't think it's that subtle - if a task is in the > custody of the BPF scheduler, ops.dequeue() will be called. Otherwise, not. > Note that, this way, whether ops.dequeue() needs to be called agrees with > whether the task needs to be dispatched to run. Here's my attempt at documenting this behavior: After ops.enqueue() is called on a task, the task is owned by the BPF scheduler, provided the task wasn't direct-dispatched to a local/global DSQ. When a task is owned by the BPF scheduler, the scheduler needs to dispatch the task to a local/global DSQ in order for it to run. When the BPF scheduler loses ownership of the task, either due to dispatching it to a local/global DSQ or due to external events (core-sched pick, CPU migration, scheduling property changes), the BPF scheduler is notified through ops.dequeue() with appropriate flags (TBD). Thanks, Kuba