From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from canpmsgout02.his.huawei.com (canpmsgout02.his.huawei.com [113.46.200.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64BCD4071E9 for ; Mon, 20 Jul 2026 11:56:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=113.46.200.217 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784548590; cv=none; b=H6xRPYCDvIZKt1d0gf/8oEzBIfW3YdiDltmsBck4ZeKEYySLbJ/Ty/9DnCI6ibGJozeC5d3jzjmjUJC1OKJMrozBAx6H4DyixT/jd+klSlY00mJe9g8KXqNoFc4zz7YCRevZvHl2/Su8tutxTUIb3q2gHwitWrQ3naH56GjXKds= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784548590; c=relaxed/simple; bh=18UyYnu7LBBSQHA0CtLLjYwE5QWvHd79SBOXPtztVNk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=paia+qoQyCj8G47nRD2FrkYdPgCC7irrLho8sjpNNubqNmPk1OIBBX6U/sbgkbR/fOZ5AM8t4ASQsPdZmp0x/TxsQ0v0DB80sLrTgPvypxzDviSdSIggHnS4eC/aZIMJQsUEe5D0B008HrIrZm56FYlWoD9Bva5DiqTL1v9BJE8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=HabMtLGy; arc=none smtp.client-ip=113.46.200.217 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="HabMtLGy" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=lbBBqjAHE+LfDowRD+KwQdEARTYH39Su+MY/jo4iNn0=; b=HabMtLGycmmScDfkCsC6WazmBhUWzeI1OvLzP1xgyB1WZVQXYgu8To7MzACCxov+jW2XdWZpt rTLyp6wgymsYdDxybLd5vEMogB70htWfWtB+rELZwTRR3EKLpWnyhFijMo3LlkfO+BRe0HLS5DM 8hCTl+lS7acMcIQLQe8Jceg= Received: from mail.maildlp.com (unknown [172.19.162.197]) by canpmsgout02.his.huawei.com (SkyGuard) with ESMTPS id 4h3dz54F0BzcZyH; Mon, 20 Jul 2026 19:46:57 +0800 (CST) Received: from kwepemj100017.china.huawei.com (unknown [7.202.194.11]) by mail.maildlp.com (Postfix) with ESMTPS id DD5B040579; Mon, 20 Jul 2026 19:56:22 +0800 (CST) Received: from huawei.com (10.67.174.193) by kwepemj100017.china.huawei.com (7.202.194.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Mon, 20 Jul 2026 19:56:22 +0800 From: Luo Gengkun To: , , , , , CC: , , , , , , Subject: [PATCH v7 linux 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Date: Mon, 20 Jul 2026 12:22:13 +0000 Message-ID: <20260720122214.3977092-2-luogengkun2@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260720122214.3977092-1-luogengkun2@huawei.com> References: <20260720122214.3977092-1-luogengkun2@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: kwepems200001.china.huawei.com (7.221.188.67) To kwepemj100017.china.huawei.com (7.202.194.11) The overhead of task_cache_work() is high, especially in multi-NUMA systems. Currently, task_cache_work() tries to find the pref_llc by scanning all CPUs in the system. However, most of these scans are meaningless, such as those for CPUs that have never been visited or were accessed a long time ago. To address this problem, introduce visited_cpus to track the visited CPUs and evict them once they have not been accessed for a duration exceeding llc_epoch_affinity_timeout. Signed-off-by: Luo Gengkun --- include/linux/mm_types.h | 6 +++++ include/linux/sched.h | 2 ++ kernel/sched/fair.c | 48 +++++++++++++++++++++++++++++++--------- 3 files changed, 46 insertions(+), 10 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index b18c2b2e7d2c..35559079e4d4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1620,6 +1620,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm) if (!pcpu_sched) return -ENOMEM; + if (!zalloc_cpumask_var(&mm->sc_stat.visited_cpus, GFP_KERNEL)) { + free_percpu(pcpu_sched); + return -ENOMEM; + } + mm_init_sched(mm, pcpu_sched); return 0; } @@ -1630,6 +1635,7 @@ static inline void mm_destroy_sched(struct mm_struct *mm) { free_percpu(mm->sc_stat.pcpu_sched); mm->sc_stat.pcpu_sched = NULL; + free_cpumask_var(mm->sc_stat.visited_cpus); } #else /* !CONFIG_SCHED_CACHE */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..b461a71a65da 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2388,6 +2388,7 @@ static __always_inline int task_mm_cid(struct task_struct *t) struct sched_cache_time { u64 runtime; unsigned long epoch; + unsigned long epoch_last_visit; }; struct sched_cache_stat { @@ -2398,6 +2399,7 @@ struct sched_cache_stat { unsigned long next_scan; unsigned long footprint; int cpu; + cpumask_var_t visited_cpus; } ____cacheline_aligned_in_smp; #else diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1..c56b2df66de8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1585,6 +1585,7 @@ void mm_init_sched(struct mm_struct *mm, pcpu_sched->runtime = 0; /* a slightly stale cpu epoch is acceptible */ pcpu_sched->epoch = rq->cpu_epoch; + pcpu_sched->epoch_last_visit = rq->cpu_epoch; epoch = rq->cpu_epoch; } @@ -1635,13 +1636,23 @@ static inline void __update_mm_sched(struct rq *rq, } } -static unsigned long fraction_mm_sched(struct rq *rq, - struct sched_cache_time *pcpu_sched) +static unsigned long fraction_mm_sched(int cpu, + struct mm_struct *mm) { + struct sched_cache_time *pcpu_sched = + per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); + struct rq *rq = cpu_rq(cpu); + guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); __update_mm_sched(rq, pcpu_sched); + /* Skip the rq that has not been hit for a long time */ + if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) { + cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus); + return 0; + } + /* * Runtime is a geometric series (r=0.5) and as such will sum to twice * the accumulation period, this means the multiplcation here should @@ -1711,6 +1722,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) pcpu_sched->runtime += delta_exec; rq->cpu_runtime += delta_exec; epoch = rq->cpu_epoch; + pcpu_sched->epoch_last_visit = epoch; + if (!cpumask_test_cpu(cpu_of(rq), mm->sc_stat.visited_cpus)) + cpumask_set_cpu(cpu_of(rq), mm->sc_stat.visited_cpus); } /* @@ -1867,6 +1881,18 @@ static void task_cache_work(struct callback_head *work) guard(rcu)(); get_scan_cpumasks(cpus, p); + /* + * Data race: While evaluating the visited_cpus without + * a lock, a CPU could be concurrently set by + * account_mm_sched(), meaning the scan might skip the newly + * visited CPU if the bit changes during the scan. This is + * a deliberate trade-off between accuracy and efficiency: + * locking would prevent this race but incur extra overhead. + * The missed runtime contribution is negligible because it + * implies this process hasn't run on that CPU for a long + * time, and will be captured in the next cycle. + */ + cpumask_and(cpus, cpus, mm->sc_stat.visited_cpus); for_each_cpu(cpu, cpus) { /* XXX sched_cluster_active */ @@ -1877,19 +1903,21 @@ static void task_cache_work(struct callback_head *work) if (!sd) continue; - for_each_cpu(i, sched_domain_span(sd)) { - occ = fraction_mm_sched(cpu_rq(i), - per_cpu_ptr(mm->sc_stat.pcpu_sched, i)); + for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) { + cur = rcu_dereference_all(cpu_rq(i)->curr); + if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) && + cur->mm == mm) + nr_running++; + + occ = fraction_mm_sched(i, mm); + if (occ == 0) + continue; + a_occ += occ; if (occ > m_occ) { m_occ = occ; m_cpu = i; } - - cur = rcu_dereference_all(cpu_rq(i)->curr); - if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) && - cur->mm == mm) - nr_running++; } /* -- 2.34.1