From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DABE363C5A for ; Wed, 19 Aug 2026 09:51:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.72 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133099; cv=none; b=FlitjObruZiWByrBlh+SUt0Op5NqE2uT/eDVinJ84iOk1oQHzz6QCn7AggARP36ki9xGFDI5h76naZX37MwPv1rUzgeDqVznwjMDc9BY8ipLn+93MkbtskKSQ0Yqbw1j/nad9xsPaGN947oPtJka3QWGPcRBhUKTaVgzLqKHw/0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133099; c=relaxed/simple; bh=TIc/H50tP8+DBqQHM2YIzpGzbEF4v8hjcmCYGb+YPeQ=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=HXmX7+VF/K3BDUpMeGIUdssPTWEm8kBCjNnkeTOPOpMHDpUBZJ2V621Qwc5goTqbJwJuAkru04Us+qBWLSwS96yZjUcCucIhHP7pSjh8tgQJzQ2MN5ErgQlOwtibdI2m36joGdLhxlvnGC4w6uFEJ9xjpR1dk0WwcD4jvLdIjGE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JlHvMLy6; arc=none smtp.client-ip=209.85.218.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JlHvMLy6" Received: by mail-ej1-f72.google.com with SMTP id a640c23a62f3a-c20262b5e10so84098666b.1 for ; Wed, 19 Aug 2026 02:51:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787133096; x=1787737896; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:mime-version:date:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6TjVwG0tgmancN9djPW8S7CTb1ZHSRZsU9D/9uGPQqY=; b=JlHvMLy68WPIBDqehSwq+uaPZl7Z6e4iNXGlQ5MNnxuIEEbRHQhiGIcBl5KcQ3bMkA 1GrBXkvG70N/l9DfqfqCTj9+3G5wTX0BlsVL/ErMWhSQ4k4v5YG0dZ+8pBNoXPSJG2Hf OG6PzDBg8ZAoFSbP0t449smMjKVozzLJeZYihMr9ihMoEPShMbwWwwFC8Q4qgPDPPH7J oiqCzMj2uTpLLTjAXaXt0cYrQZhntpgR227DYsSI+D8wcK5Ik6qOYRiU8+8GKP6gHvhD M/+0Zha8z2DkR8NOXYiboBBQUwppymyBHDytZ3ub/BOqk9k/EajpOXjHjgeOX0hnjHOK mvvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133096; x=1787737896; h=content-type:cc:to:from:subject:message-id:mime-version:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=6TjVwG0tgmancN9djPW8S7CTb1ZHSRZsU9D/9uGPQqY=; b=ggIzBG+xVimoZ7TXH9YN/uyMkEa+FfRCMlESMD9pGI17y05FwUXZ0eaUiJf5JAvtvA WmztwCT9ejiMNLEUQH8avLXYpDxKIKImsSzXrTnORWkJ7Q5Oi43JQqYC4XWY4ooelpPG +yKZj4Us687t6vlJxNyInd9YNgChdIGG59GR03/jY9eM2ofQAKo4f2iganbCq14AMY/O RkpcL3NET9qYYz+sCwpcJHH78Jlp/RyLwd14mjd8FN7j8IdMrwAXMfFCMmr3i/o8r6FY Mu5UBafFnns2XHGySv0CQzrP3nDKvf+xj0bEeE1HOiA9sZfzlCEgR0xB/Ul/vZgfIIGn Zu9w== X-Forwarded-Encrypted: i=1; AHgh+RojkVSP//EUPoV+0IMV/ulP61dGvvIG8aWukm66tXOmBJPqq3TMSk0C1lbQjGEkWF/eGrNj1WkEGatfvg8=@vger.kernel.org X-Gm-Message-State: AOJu0YzF5bJf8Cce42TAzp7eWVdd2jRItHd0XlrFwPiRGGyf3SptP093 XjwCxS1lxzldcuj6E5N1zVFGF/cVbwyyjTMYHC+B3mP8z1H//69gTlOzYRN0RZeKnJg49BSH62T yE3EH0Q== X-Received: from ejc6-n2.prod.google.com ([2002:a05:6938:a006:20b0:c12:8c27:d430]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:ea94:b0:c19:49ee:19b9 with SMTP id a640c23a62f3a-c242e215190mr192610466b.15.1787133095874; Wed, 19 Aug 2026 02:51:35 -0700 (PDT) Date: Wed, 19 Aug 2026 09:51:28 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.55.0.737.g08866a6d13-goog Message-ID: <20260819095129.4056035-1-lrizzo@google.com> Subject: [PATCH] sched/cpupri: Remove count field from struct cpupri_vec From: Luigi Rizzo To: Luigi Rizzo , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" Under heavy I/O workloads (such as fio using threaded IRQs), irq_thread runs as SCHED_FIFO 50 and undergoes frequent enqueue/dequeue cycles. Every dequeue invokes cpupri_set() to transition the CPU priority back to CPUPRI_NORMAL. The "count" field in struct cpupri_vec was designed as an early-exit heuristic for __cpupri_find(). However, maintaining count in sync with mask requires atomics and expensive barriers (e.g. "dmb ish" on ARM), which on large multicore systems become extremely expensive: Experiment on a dual socket ARM with 220 CPUs: - runtime for cpupri_set(), nanoseconds: with count: p10: 700 p50: 1915 p90: 6619 p98 11150 p99: 13200 without: p10: 290 p50: 425 p90: 567 p98: 685 p99: 821 - cpupri_set() usage on softirq CPU, measured with perf top with count: ~ 25% without: < 2% Remove the count field from struct cpupri_vec and code manipulating it. This eliminates the cache-line contention and pipeline stalls while preserving scheduler correctness. Signed-off-by: Luigi Rizzo --- kernel/sched/cpupri.c | 61 ++----------------------------------------- kernel/sched/cpupri.h | 2 -- 2 files changed, 2 insertions(+), 61 deletions(-) diff --git a/kernel/sched/cpupri.c b/kernel/sched/cpupri.c index 8f2237e8b484f..e9025f47880d9 100644 --- a/kernel/sched/cpupri.c +++ b/kernel/sched/cpupri.c @@ -69,33 +69,6 @@ static inline int __cpupri_find(struct cpupri *cp, struct task_struct *p, struct cpumask *lowest_mask, int idx) { struct cpupri_vec *vec = &cp->pri_to_cpu[idx]; - int skip = 0; - - if (!atomic_read(&(vec)->count)) - skip = 1; - /* - * When looking at the vector, we need to read the counter, - * do a memory barrier, then read the mask. - * - * Note: This is still all racy, but we can deal with it. - * Ideally, we only want to look at masks that are set. - * - * If a mask is not set, then the only thing wrong is that we - * did a little more work than necessary. - * - * If we read a zero count but the mask is set, because of the - * memory barriers, that can only happen when the highest prio - * task for a run queue has left the run queue, in which case, - * it will be followed by a pull. If the task we are processing - * fails to find a proper place to go, that pull request will - * pull this task if the run queue is running at a lower - * priority. - */ - smp_rmb(); - - /* Need to do the rmb for every iteration */ - if (skip) - return 0; if (cpumask_any_and(&p->cpus_mask, vec->mask) >= nr_cpu_ids) return 0; @@ -212,7 +185,6 @@ void cpupri_set(struct cpupri *cp, int cpu, int newpri) { int *currpri = &cp->cpu_to_pri[cpu]; int oldpri = *currpri; - int do_mb = 0; newpri = convert_prio(newpri); @@ -231,39 +203,11 @@ void cpupri_set(struct cpupri *cp, int cpu, int newpri) struct cpupri_vec *vec = &cp->pri_to_cpu[newpri]; cpumask_set_cpu(cpu, vec->mask); - /* - * When adding a new vector, we update the mask first, - * do a write memory barrier, and then update the count, to - * make sure the vector is visible when count is set. - */ - smp_mb__before_atomic(); - atomic_inc(&(vec)->count); - do_mb = 1; } - if (likely(oldpri != CPUPRI_INVALID)) { - struct cpupri_vec *vec = &cp->pri_to_cpu[oldpri]; - /* - * Because the order of modification of the vec->count - * is important, we must make sure that the update - * of the new prio is seen before we decrement the - * old prio. This makes sure that the loop sees - * one or the other when we raise the priority of - * the run queue. We don't care about when we lower the - * priority, as that will trigger an rt pull anyway. - * - * We only need to do a memory barrier if we updated - * the new priority vec. - */ - if (do_mb) - smp_mb__after_atomic(); + if (likely(oldpri != CPUPRI_INVALID)) { + struct cpupri_vec *vec = &cp->pri_to_cpu[oldpri]; - /* - * When removing from the vector, we decrement the counter first - * do a memory barrier and then clear the mask. - */ - atomic_dec(&(vec)->count); - smp_mb__after_atomic(); cpumask_clear_cpu(cpu, vec->mask); } @@ -283,7 +227,6 @@ int cpupri_init(struct cpupri *cp) for (i = 0; i < CPUPRI_NR_PRIORITIES; i++) { struct cpupri_vec *vec = &cp->pri_to_cpu[i]; - atomic_set(&vec->count, 0); if (!zalloc_cpumask_var(&vec->mask, GFP_KERNEL)) goto cleanup; } diff --git a/kernel/sched/cpupri.h b/kernel/sched/cpupri.h index 6f562088c0565..78516c8428761 100644 --- a/kernel/sched/cpupri.h +++ b/kernel/sched/cpupri.h @@ -1,5 +1,4 @@ /* SPDX-License-Identifier: GPL-2.0 */ -#include #include #include @@ -11,7 +10,6 @@ #define CPUPRI_HIGHER 100 struct cpupri_vec { - atomic_t count; cpumask_var_t mask; }; -- 2.55.0.737.g08866a6d13-goog