From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E9343FB046 for ; Fri, 1 May 2026 16:20:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777652406; cv=none; b=jKGSwwpM3r9DEi9a/MrkR5cQx5h3aEyxmyAVYzX9MvWoEiCpf2lqmXJStdX7iPpNaXeM0+4BlHWENe6k5Hr8Q1WdOhvMzg80Mn8gsfKdF050E1VtA3vT7tFG6e8PTG8aeZrVq8bMObwCsMl20CJ24ErkxoQgezf3WmJQ+6WHm68= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777652406; c=relaxed/simple; bh=pFHhAGjrWqlE3zlouvyixgqPtCaNd1s68oDlvq1IYa0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=nE/ZVvNQHBO45D1QL9L20FpUBgN77dqhYec6V9oPwl27jTmf7fDQ8hHvKM/uLwLjuqgBFK/l03/ksNyQT9/Or36vjuIJofXiXrG0Rjx07BS0o1TMlbNhy3T7ShlqNyQzo58au3QI3gGY8IIYyXaIpx5mnHWl2TUUWREN6341/kc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OX44vEzV; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OX44vEzV" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-35da1af3e10so2275861a91.3 for ; Fri, 01 May 2026 09:20:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1777652397; x=1778257197; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=OartFn9vmQoF2kzuRmda/6if5JbGEfaXSR/oNU+kPz0=; b=OX44vEzVGGIpIxeS19VjT80sTjvcSqQhgBoHGO4ep0Fs/ARvmJiMLZJngqdDNP1bvG GiarWscRgqt54jkBk6QHgpE6F5vzbtJcpvWNc8N7WpVIR0UENyGh5nUDijb11gVP6GOj rFx7HEAH2d5JuSCTKtXGzDe7h1azoBs1SunW7KyrcH1yocv1qa7acEGdPVIpuXAOphvy VyqeM8jlmv1+Wl1OXGnY/yPcaBTGlqi1VMjNC6o2km1LUMMT76yYHwlsXTvFXfGtLyFJ lnkvSJDm0cXPVUvcul/bFMlM52UNS3byUbxsOp15Gh1b0zV2pUouPSu2iZrE4UGBTTVh Rk7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777652397; x=1778257197; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OartFn9vmQoF2kzuRmda/6if5JbGEfaXSR/oNU+kPz0=; b=lilYtEGL5PWA4dpn9m0PpegbDkvbO6Iwt9K5ITZ47875cy9LWMw1mLjKdovbWqqNfl 81tMs1UmvR/j0aBh6flAftlSZ3KgdsZZ4pYJc6fdmJVF2wznKmBZEu/0fHMMRtY5Q5HP HuNN0lVqP4QrpET4aWWTUH6tBHUKK2Kv9I9WWI07ANpK6i4yDGWutoXUBP1ZMOLqBuT6 lU5hDpqSwXfndGWi+UcciPOQCPCmGAn3zUreFYrUbbNyt5dV9eVm+tQp+H84pPZI0pGJ 6T2pwXPB+3umtoOZ09WJdiqGW4zRh5pFtIbXTaW07Sjuil4KMXU2Sx+8rwAYI4GwbJ8d 4V+g== X-Forwarded-Encrypted: i=1; AFNElJ89scNeR4uvKmZEFSM7UKDiygHcZF4zps58f5dROD5eMT5gaYCfhinoGewADpOob6T+mCQAxpINLZLX56g=@vger.kernel.org X-Gm-Message-State: AOJu0YzaULtd4asvbCLRGdodNxVI/rDVLaeKyzc7w4Pq3gqgiIa/eyMY PrB7OjG+fJh035C88aZkV7gfz+nef3nYYXz2f6dd0/W1VEFdccEOpvLL X-Gm-Gg: AeBDieszw148DTnJhRF0r2ZRiQ9B9uncnZAKTpAYLI5UHqa+wDLQC02YiPQ+rGYd/PW hX+F7RlS20PQZctYQ0S1mWq3OK0E0srO/KeICcjKNmIaRHoRCj49OQeTJeEgHI3jSLrU6Xf4Fsk Rrd0mhxL4QmkgXc0Po4/wE0fT36/5dbyFPI2prRdUCNk2EUIhLo2BZwRxLe4n67bsAbuUUkLbvM 7mROUcxllVBsYngiP5nGH2rnADKp/Xsg5RtPRbAPVII1AZ6znFzXMtCQxr3XLiWtM3w9P5WjYaH Hk3EzB9/IoeE5uN3Lp9KcQz0obENjbQGgV3Z7S0k4keTmaT42cB06g/xFzuJhGPTCRiZi7R3PNH vvRfPQbd2b8dwpKwRk/m4124p00D4eCMyAc0aUZflz+FqLLWbYUfhq0ew8RdbJOdpub3DbmS9C5 en3Z3fbR4cNxo352c1kWz4X0Bjfee1e4d7cFD//lB/RUQFpj0VCu+KfuOMsDDdeSmMuP+cfORtB IMs+vaQNdcJb0nb X-Received: by 2002:a17:90b:1dca:b0:35f:b293:7ac6 with SMTP id 98e67ed59e1d1-364c2fe3ec9mr8416481a91.6.1777652396759; Fri, 01 May 2026 09:19:56 -0700 (PDT) Received: from cchengyang.duckdns.org (36-225-77-225.dynamic-ip.hinet.net. [36.225.77.225]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2b9caa7ebe8sm31062495ad.2.2026.05.01.09.19.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 May 2026 09:19:56 -0700 (PDT) Date: Sat, 2 May 2026 00:19:52 +0800 From: Cheng-Yang Chou To: Kuba Piecuch Cc: Tejun Heo , Andrea Righi , David Vernet , Changwoo Min , Emil Tsalapatis , Christian Loehle , Daniel Hodges , sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, Ching-Chun Huang , Chia-Ping Tsai Subject: Re: [PATCH v2 sched_ext/for-7.1] sched_ext: Invalidate dispatch decisions on CPU affinity changes Message-ID: <20260502000039.Ga94c@cchengyang.duckdns.org> References: <20260319083518.94673-1-arighi@nvidia.com> <20260422142633.G7180@cchengyang.duckdns.org> <20260426093756.Gd781@cchengyang.duckdns.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Hi Kuba, On Mon, Apr 27, 2026 at 09:06:21AM +0000, Kuba Piecuch wrote: > > On Thu, Apr 23, 2026 at 01:32:20PM +0000, Kuba Piecuch wrote: > >> > On Mon, Mar 23, 2026 at 01:13:20PM -1000, Tejun Heo wrote: > >> >> > The simple way to do this is to do scx_bpf_dsq_insert() at the very beginning, > >> >> > once we know which task we would like to dispatch, and cancel the pending > >> >> > dispatch via scx_bpf_dispatch_cancel() if any of the pre-dispatch checks fail > >> >> > on the BPF side. This way, the "critical section" includes BPF-side checks, and > >> >> > SCX will ignore the dispatch if there was a dequeue/enqueue racing with the > >> >> > critical section. > >> >> > > >> >> > With this solution, we can throw an error if task_can_run_on_remote_rq() is > >> >> > false, because we know that there was no racing cpumask change (if there was, > >> >> > it would have been caught earlier, in finish_dispatch()). > >> >> > >> >> Yeah, I think this makes more sense. qseq is already there to provide > >> >> protection against these events. It's just that the capturing of qseq is too > >> >> late. If insert/cancel is too ugly, we can introduce another kfunc to > >> >> capture the qseq - scx_bpf_dsq_insert_begin() or something like that - and > >> >> stash it in a per-cpu variable. That way, qseq would be cover the "current" > >> >> queued instance and the existing qseq mechanism would be able to reliably > >> >> ignore the ones that lost race to dequeue. > >> > > >> > Since this has been stale for a while, I prepared a patch to implement > >> > scx_bpf_dsq_insert_begin() as suggested. > >> > >> Thanks for creating the patch. A couple of thoughts: > >> > >> 1. Do we have a use case that requires dsq_insert_begin() that isn't > >> satisfied using the "insert and then cancel if needed" approach? > > > > IIUC, yes. scx_bpf_dispatch_cancel() is only registered in > > scx_kfunc_ids_dispatch, so it is only callable from ops.dispatch(). > > dsq_insert_begin(), on the other hand, is available from both > > ops.enqueue() and ops.dispatch() (SCX_KF_ENQUEUE | SCX_KF_DISPATCH). > > Since there is nothing to cancel in ops.enqueue(), the insert-and-cancel > > approach simply doesn't work there. > > Wouldn't the natural thing then be to extend scx_bpf_dispatch_cancel() to > work for direct dispatch? Instead of introducing a whole new mechanism, let's > extend the one we have by functionality that it (arguably) should have had > from the beginning. I see. You're right that dispatch_cancel() could be extended to work in the enqueue context. I'm happy to go either direction, your approach or Tejun's suggestion. Tejun, Andrea, sched-ext folks, thoughts? > > > > >> > >> 2. Do we want to restrict ourselves through the one qseq slot provided by > >> dsq_insert_begin()? The most flexible approach IMO would be to simply > >> allow BPF to read the qseq directly via a kfunc and then supply it to > >> dsq_insert() later. With this, we can have multiple qseqs saved at the > >> same time, and we can even pass them between CPUs, e.g. if one CPU > >> dequeues a task for a sibling CPU, but we want the checks to be made inside > >> the sibling's ops.dispatch() (I just made this use case it up, it may not > >> be practical.) > >> That said, exposing an internal thing like qseq to BPF may be a step too far. > > > > In Tejun's reply back in [1], he suggested dsq_insert_begin() precisely > > to avoid promoting qseq into the BPF ABI — which matches your own concern. > > The single per-CPU slot is sufficient for the one-task-per-iteration > > dispatch loops used by existing schedulers (e.g., scx_central). > > If a concrete cross-CPU use case materializes later, we can always extend > > dsq_insert() to accept an explicit qseq without breaking the current, > > simpler path. > > > > [1]: https://lore.kernel.org/all/acHJED4iAeytdC2l@slm.duckdns.org/ > > > > Well, Tejun doesn't explicitly say there that he's against exposing qseq, but > I won't be surprised if he is. > > FWIW, ghOSt (our Google-internal BPF scheduling solution) uses exactly this > approach to guard the dispatch path against racing dequeues/enqueues. > Every task has a seqnum that gets incremented on each "event" pertaining to > the task. In the dispatch path, the BPF scheduler reads the task seqnum, > does whatever checks it needs to do, and passes the seqnum to ghOSt at the end. > > Admittedly, what works downstream doesn't have to work upstream, but I still > wanted to provide this data point :-) The ghOSt data point is appreciated. If a concrete use case emerges where the single-slot approach falls short, extending dsq_insert() to accept an explicit qseq seems like a natural next step. Tejun, Andrea, sched-ext folks, any preferences? -- Cheers, Cheng-Yang