From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F8CE2D6E4B for ; Sun, 8 Feb 2026 17:59:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770573579; cv=none; b=Ub3tPZgqiTRN+nKsTfk+tHf0yT4GZCYL1DYWrBYV2zedtD2xiuU9brPISFufa/x50oV5V2BCWsjghv+SagKGvS/hSXf9xRpGw6ACVVXZhvWabJBbTyc2Bb/qwnmLK0fh2ojoYTGrlYpEwYcOd+Yo+gWMRkJp2OvpJZ9FSmNVFHE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770573579; c=relaxed/simple; bh=zMcfca0X5VThFA9j0nAmI4dEJJ0zfPZ6AWVf8vT0QBo=; h=Mime-Version:Content-Type:Date:Message-Id:Subject:From:To:Cc: References:In-Reply-To; b=gPG6Mx5ddhDLmL+KKpVOaM2leCbEJdPxKDZCRUTXEowRKZpvqRkrUmw8F+HwffoV6ytsS8vkPxebVIv3QJqTPo9KG6LBBgPTTooXPCgOjzlNl1rp4uRqYSpnZtRU0+nb2jgVSM6Il2d1Y9F6uPOyX3Ha8hKjsMoLkeCaXxIrjco= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=etsalapatis.com; spf=pass smtp.mailfrom=etsalapatis.com; dkim=pass (2048-bit key) header.d=etsalapatis-com.20230601.gappssmtp.com header.i=@etsalapatis-com.20230601.gappssmtp.com header.b=gFVONsUf; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=etsalapatis.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=etsalapatis.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=etsalapatis-com.20230601.gappssmtp.com header.i=@etsalapatis-com.20230601.gappssmtp.com header.b="gFVONsUf" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-8ca01dc7d40so398688685a.1 for ; Sun, 08 Feb 2026 09:59:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=etsalapatis-com.20230601.gappssmtp.com; s=20230601; t=1770573578; x=1771178378; darn=vger.kernel.org; h=in-reply-to:references:cc:to:from:subject:message-id:date :content-transfer-encoding:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Z+LW08Zb0i/6FTaN/ZmjBAf4wAOOpN1TIoml3WmBDAM=; b=gFVONsUfZKIz2PqWZj0rodIgBifCr6QR1vnVu90j9RRVdv38gi+EzeRSU+GL8OEDcG kzTeOFN+CP8rb2YqTJ2gf0wkOfkrkkzNmrZ8iaf/HLxlAQX3pELqekF6yAXQPUIAyU8Y EsirBZlAnDH/MfEGq5JBZHVytA6Bc/6jI20ATADKdkfM7wtjcekxBc+Dnldk0+rygACl 25YsKjfohHN+TBSV7CZwiXghZlOC3a1YX8oMP4IeLM/qAiYIV2Amssy050vVqdJUdpDk OXozq8beIB+FoPjMaiKn7LIHNd+9yYH+V4c6wKSPpbE1xGGhBkjIN7LphpK66qzU+aTJ HeOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770573578; x=1771178378; h=in-reply-to:references:cc:to:from:subject:message-id:date :content-transfer-encoding:mime-version:x-gm-gg:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Z+LW08Zb0i/6FTaN/ZmjBAf4wAOOpN1TIoml3WmBDAM=; b=TwsTLoouCq4o+BCWcT4Vvbf3W1IqW3Lsecr1qxevZ/fa9TzDjnzvUATVVoT0ZUuJ/K gX5qKJjQgPJpiH0I0S07s/0ULWH6AS0Z4HbPbnVjqKtwtd57RlydzVvP3oQBkthaHGPE X95ltE4zTYrzKZSF1/pdcaz1RaJkwIHd7ip/+nIbUxAwfiEFxAnUiGbtwSbe0U7Mbn0V wjUjpgAhjOhnAQTpq9WkQ0BB/Zfz3iL3z0wRoTzQmKp/CnMA9N0Tc8YYTADslIXgm8qp YPKRurXL/CkKIuTZc2eefnOBz0s8AK+8oJbtNhM5ugFOMlRFy/6uN636pGCpDxT2L5j6 I+Vw== X-Forwarded-Encrypted: i=1; AJvYcCXm5AZa8rZ1mnQ3UwXackVlN7JpPrbT03Rr+5PrPCIT+JhOfNGPM5SMrdJX2rpgG8BcumdnN1IDOcFAoIQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxPBWhuse94Mq/hYUb2SNrvIHTiwZ1jKv6xMKHrV6avuE9pNnRs 64f5h3MbFc10NgfMLFDEdmF6ZyfmeZD3ooDFtnCoAPSFSdNmMLYrz15fmolwXPKIiJjP0pCYSeD np30PQnU= X-Gm-Gg: AZuq6aJtWCU4vz/Ikaqwjs+bJyC+Q9ZN00znGCV1I++C75Co+03cV81M/3LalFAv83w +Y2vQUTsJQMGYrqd7e8RXqFnA9bnzp6rIMECjZ0IPIKYMFPe3kXf50T/YbxtCOAph5olwtMC6L7 SwcceaIrZPuTC8QOo3CUDrYiIcJS44sTs7+rni9Pz4cg7dk5RWCVUUhRyK+CEI2eptNp/au+slO PNjHMpeu/DO88bV8beeuTzzMdyzmNyIJyybVj7ZBth6CchqnGbwHRz1kjyKpm9dnQL0AEHBoZKe MaEA6HFl0vNwREsK7Ok34d//gyd82T0iCmuciMpZw0+0sXwc1Ns+JE1ctD+bdum2DuJj43uL/xu kmBQZDWYA2AXMpo33iVUi7MthHOLD7rsjBPt71+XILyie1wTJs3Ytj6Xed9EnqcZlBpLvr/jEh0 9PqTAqzrQ8/L4= X-Received: by 2002:a05:620a:2a07:b0:8c9:ea6f:6a4a with SMTP id af79cd13be357-8caf2594195mr1190220285a.78.1770573577818; Sun, 08 Feb 2026 09:59:37 -0800 (PST) Received: from localhost ([140.174.219.137]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8953bf3c740sm61559046d6.11.2026.02.08.09.59.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 08 Feb 2026 09:59:37 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Sun, 08 Feb 2026 12:59:36 -0500 Message-Id: Subject: Re: [PATCH 2/2] selftests/sched_ext: Add test to validate ops.dequeue() semantics From: "Emil Tsalapatis" To: "Andrea Righi" Cc: "Tejun Heo" , "David Vernet" , "Changwoo Min" , "Kuba Piecuch" , "Christian Loehle" , "Daniel Hodges" , , X-Mailer: aerc 0.20.1 References: <20260206135742.2339918-1-arighi@nvidia.com> <20260206135742.2339918-3-arighi@nvidia.com> In-Reply-To: On Sun Feb 8, 2026 at 8:55 AM EST, Andrea Righi wrote: > On Sun, Feb 08, 2026 at 11:26:13AM +0100, Andrea Righi wrote: >> On Sun, Feb 08, 2026 at 10:02:41AM +0100, Andrea Righi wrote: >> ... >> > > >> > - From ops.select_cpu(): >> > > >> > - scenario 0 (local DSQ): tasks dispatched to the local DS= Q bypass >> > > >> > the BPF scheduler entirely; they never enter BPF custody= , so >> > > >> > ops.dequeue() is not called, >> > > >> > - scenario 1 (global DSQ): tasks dispatched to SCX_DSQ_GLO= BAL also >> > > >> > bypass the BPF scheduler, like the local DSQ; ops.dequeu= e() is >> > > >> > not called, >> > > >> > - scenario 2 (user DSQ): tasks enter BPF scheduler custody= with full >> > > >> > enqueue/dequeue lifecycle tracking and state machine val= idation >> > > >> > (expects 1:1 enqueue/dequeue pairing). >> > > >>=20 >> > > >> Could you add a note here about why there's no equivalent to scen= ario 6? >> > > >> The differentiating factor between that and scenario 2 (nontermin= al queue) is=20 >> > > >> that scx_dsq_insert_commit() is called regardless of whether the = queue is terminal. >> > > >> And this makes sense since for non-DSQ queues the BPF scheduler c= an do its >> > > >> own tracking of enqueue/dequeue (plus it does not make too much s= ense to >> > > >> do BPF-internal enqueueing in select_cpu). >> > > >>=20 >> > > >> What do you think? If the above makes sense, maybe we should spel= l it out=20 >> > > >> in the documentation too. Maybe also add it makes no sense to enq= ueue >> > > >> in an internal BPF structure from select_cpu - the task is not ye= t >> > > >> enqueued, and would have to go through enqueue anyway. >> > > > >> > > > Oh, I just didn't think about it, we can definitely add to ops.sel= ect_cpu() >> > > > a scenario equivalent to scenario 6 (push task to the BPF queue). >> > > > >> > > > From a practical standpoint the benefits are questionable, but in = the scope >> > > > of the kselftest I think it makes sense to better validate the ent= ire state >> > > > machine in all cases. I'll add this scenario as well. >> > > > >> > >=20 >> > > That makes sense! Let's add it for completeness. Even if it doesn't = make >> > > sense right now that may change in the future. For example, if we en= d >> > > up finding a good reason to add the task into an internal structure = from >> > > .select_cpu(), we may allow the task to be explicitly marked as bein= g in >> > > the BPF scheduler's custody from a kfunc. Right now we can't do that >> > > from select_cpu() unless we direct dispatch IIUC. >> >=20 >> > Ok, I'll send a new patch later with the new scenario included. It sho= uld >> > work already (if done properly in the test case), I think we don't nee= d to >> > change anything in the kernel. >>=20 >> Actually I take that back. The internal BPF queue from ops.select_cpu() >> scenario is a bit tricky, because when we return from ops.select_cpu() >> without p->scx.ddsp_dsq_id being set, we don't know if the scheduler add= ed >> the task to an internal BPF queue or simply did nothing. >>=20 >> We need to add some special logic here, preferably without introducing >> overhead just to handle this particular (really uncommon) case. I'll tak= e a >> look. > > The more I think about this, the more it feels wrong to consider a task a= s > being "in BPF scheduler custody" if it is stored in a BPF internal data > structure from ops.select_cpu(). > > At the point where ops.select_cpu() runs, the task has not yet entered th= e > BPF scheduler's queues. While it is technically possible to stash the tas= k > in some BPF-managed structure from there, doing so should not imply full > scheduler custody. > > In particular, we should not trigger ops.dequeue(), because the task has > not reached the "enqueue" stage of its lifecycle. ops.select_cpu() is > effectively a pre-enqueue hook, primarily intended as a fast path to bypa= ss > the scheduler altogether. As such, triggering ops.dequeue() in this case > would not make sense IMHO. > > I think it would make more sense to document this behavior explicitly and > leave the kselftest as is. > > Thoughts? I am going back and forth on this but I think the problem is that the enque= ue()=20 and dequeue() BPF callbacks we have are not actually symmetrical?=20 1) ops.enqueue() is "sched-ext specific work for the scheduler core's enque= ue method". This is independent on whether the task ends up in BPF custody or = not. It could be in a terminal DSQ, a non-terminal DSQ, or a BPF data structure. 2) ops.dequeue() is "remove task from BPF custody". E.g., it is used by the BPF scheduler to signal whether it should keep a task within its internal tracking structures. So the edge case of ops.select_cpu() placing the task in BPF custody is currently valid. The way I see it, we have two choices in terms of semantics: 1) ops.dequeue() must be the equivalent of ops.enqueue(). If the BPF scheduler writer decides to place a task into BPF custody during the ops.select_cpu() that's on them. ops.select_cpu() is supposed to be a pure function providing a hint, anyway. Using it to place a task into BPF is a bit of an abuse even if allowed. 2) We interpret ops.dequeue() to mean "dequeue from the BPF scheduler". In that case we allow the edge case and interpret ops.dequeue() as "the function that must be called to clear the NEEDS_DEQ/IN_BPF flag", not as the complement of ops.enqueue(). In most cases both will be true, and in the cases where not then it's up to the scheduler writer to understand the nuance. I think while 2) is cleaner, it is more involved and honestly kinda speculative. However, I think it's fair game since once we settle on the semantics it will be more difficult to change them. Which one do you=20 think makes more sense? > > Thanks, > -Andrea