From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD9D2388E6B for ; Fri, 17 Jul 2026 11:00:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784286016; cv=none; b=nTRhl30iueIS2mGRintllzIFinJuE/gDDpgXeBl9LfZhlOx8SJ6zX3THjnJLPxQPwP9P+vd6fapePlAvUVqllML57xb65T1DW8K2B/DwkNd/vd7VO9lVXHkZAzi/TOncb4W1fxEbIeqCZ5IR/8gCizYabkmusLiarqq1W2Cnoi4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784286016; c=relaxed/simple; bh=WUfkrN2jJ4I/dDIVImEZIvDfOXlv3ygceMZjOuDD1qM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=JZSfF/u53L2KY3CvScYebd6OdDkbnQi0nSU3xqcrwsafksI525gFmfXZ1OdXdcRK84TSbIB/7wTsadSFByxMdAumQv/LFa2S5rv3UEzmYkxhyL1QMFQpp2tnoJTbl6qdCpAtINZwLoalrEZ6lJTcFy70hQUVFIInuwHvPicpwAY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=auGeaacw; arc=none smtp.client-ip=95.215.58.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="auGeaacw" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1784285996; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=1gUpsrVHCUjWR5C/g4ANxCpopkxj9399TS4L77cpB8s=; b=auGeaacwSblmS6+Zzm1qcWFFeiXN65ymm5/SmCHWOE0SRy9FTrWS2K/kpjc5EJkyCpO5uM W3REkCDaYdQc7xZZ6x54PC12KhXLTV32+rx5USwmY0386NiVgHDynn9qt84V9vy136rMNs nVekZgTN3p1AbK8SX3w27QPlc3WsW0M= From: Usama Arif To: bsegall@google.com, dietmar.eggemann@arm.com, hannes@cmpxchg.org, juri.lelli@redhat.com, kprateek.nayak@amd.com, linux-kernel@vger.kernel.org, mgorman@suse.de, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, surenb@google.com, vincent.guittot@linaro.org, vschneid@redhat.com, shakeel.butt@linux.dev, riel@surriel.com, kernel-team@meta.com Cc: Usama Arif Subject: [PATCH v2] sched/psi: use __ffs() to walk task-count bitmasks in psi_group_change() Date: Fri, 17 Jul 2026 03:59:39 -0700 Message-ID: <20260717105939.203685-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT psi_group_change() walks the @clear and @set bitmasks to decrement/increment groupc->tasks[t]. Both masks are at most NR_PSI_TASK_COUNTS (=4) wide and typically have one or two bits set. Today's form visits every position up to the highest set bit: for (t = 0, m = clear; m; m &= ~(1 << t), t++) { if (!(m & (1 << t))) continue; ... } so a mask with only bit 3 set still spins four times through the skip path. Switch both walks to __ffs() + m &= m-1 form: while (clear) { t = __ffs(clear); clear &= clear - 1; ... } which iterates only over the set bits and terminates naturally on m == 0. m & (m - 1) clears the lowest set bit. This code is easier to read as well. An in-kernel microbench (noinline, same body, IRQs off, pinned CPU on Zen4c, min-of-10 cyc/call) over mask distributions produced by common scheduler PSI paths: mask pattern old new delta empty (clear=0x0, set=0x0) 3.68 3.68 +0% sleep (clear=0x4, set=0x0) 9.60 3.74 -61% iowait-sleep (clear=0x4, set=0x1) 12.32 4.45 -63% memstall-sleep (clear=0xc, set=0x0) 11.87 5.54 -53% wake (clear=0x0, set=0x4) 7.10 3.70 -47% iowait-wake (clear=0x1, set=0x4) 10.83 4.48 -58% Every non-empty case wins 47-63%: old cost tracks the highest set bit (linear walk), new cost tracks the count of set bits (skip zeros via TZCNT). Single-bit patterns run at the empty-case floor. The generated psi_group_change() text also shrinks by 67 bytes under -O2 -march=x86-64 (756 -> 689): no scratch register for a "constant 1" (only __ffs's operand is needed), simpler bit-clear (LEA+AND vs SHL+NOT+AND after the test), and no skip-if-unset check per position. Acked-by: Johannes Weiner Signed-off-by: Usama Arif --- v1 -> v2 - Switch from using for_each_set_bit() to __ffs(). Based on the benchmark: https://gist.github.com/uarif1/e1bf78b54f50099b354b84684f880fda and code size reduction for psi_group_change(). --- kernel/sched/psi.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c index d9c9d9480a45..2951614cae17 100644 --- a/kernel/sched/psi.c +++ b/kernel/sched/psi.c @@ -798,7 +798,7 @@ static void psi_group_change(struct psi_group *group, int cpu, u64 now, bool wake_clock) { struct psi_group_cpu *groupc; - unsigned int t, m; + unsigned int t, clear_orig; u32 state_mask; lockdep_assert_rq_held(cpu_rq(cpu)); @@ -820,27 +820,31 @@ static void psi_group_change(struct psi_group *group, int cpu, state_mask = groupc->state_mask & PSI_ONCPU; } + clear_orig = clear; + /* * The rest of the state mask is calculated based on the task * counts. Update those first, then construct the mask. */ - for (t = 0, m = clear; m; m &= ~(1 << t), t++) { - if (!(m & (1 << t))) - continue; + while (clear) { + t = __ffs(clear); + clear &= clear - 1; if (groupc->tasks[t]) { groupc->tasks[t]--; } else if (!psi_bug) { printk_deferred(KERN_ERR "psi: task underflow! cpu=%d t=%d tasks=[%u %u %u %u] clear=%x set=%x\n", cpu, t, groupc->tasks[0], groupc->tasks[1], groupc->tasks[2], - groupc->tasks[3], clear, set); + groupc->tasks[3], clear_orig, set); psi_bug = 1; } } - for (t = 0; set; set &= ~(1 << t), t++) - if (set & (1 << t)) - groupc->tasks[t]++; + while (set) { + t = __ffs(set); + set &= set - 1; + groupc->tasks[t]++; + } if (!group->enabled) { /* -- 2.53.0-Meta