From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 985693AFCEA for ; Wed, 19 Aug 2026 17:49:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787161801; cv=none; b=eytmE9OkWzGmsMopXSgrtONkmjngZGj5PW9TuyJmxImL0k7YR8k1b/yBe8mrTDN7+6tbip6e07qoIdm83Ltl/pqmcnbIkLWNu/1M5eu18p4j0rkl+iLSW3aoNnY0HIOqtn1TlGH4uDO6ehghlkfNBKx9oXLMee3l6mb+n7x3l4c= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787161801; c=relaxed/simple; bh=J0kWI3E6Oyso6cMlwyesS9c83pFoNPGtJpagN0Xh5VQ=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=Z9qqIlgrbNqvCGSbJ3dcckX2GYB0ADDq6hWBw28A5Qw8ojuO2OZvF8yK+OT4WJkgdPTVns1m8gkiwsInvYQ3WERXXc6i6Qu8NbIkbTomEN0VbFQd0u4Ua+dX+PIOxQDxIUB+J+6uG1qaOc+rn7VAVxPCu+hoP28zgWRUTPbv1Rw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=abn32U88; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="abn32U88" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 67JF1XCF3757915; Wed, 19 Aug 2026 17:49:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=k2zxWo U5EJ9yIGPUU5uLKUPTfeyQttbz25Doh9m9kek=; b=abn32U881VCPhRqTpXJg8J wWUNQH3gPyHe/J99VR3mX2Ub7MHO6qx0ZH+1L9wMk4kqbkF9NWVH1de7Om7LZzfL +s0AtUwkNVvqoavA4K8tgEX2nT6VYlbmyZf9YX4owkB/E5X1TevgRoc5+cDEaezH Jqg5N0eGndwDPcB3h4SAi5GkZLpTf06oAJum4z/QzYDt/WzxHH+aE7axxFnVThGy iEcWfnW6MViFVGuSgNrnkNYnR4bH7L0XqWN+HRXtXvxMkUYDFLrAzxkzYmVGZxCE JeTUXcfGNruQuh51n+ARZrazaqrloCuH2LPrAboovQ6RjesbUa24te+i6qY5Qjkg == Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4g4yu1dbwt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Aug 2026 17:49:24 +0000 (GMT) Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 67JHfGMr013966; Wed, 19 Aug 2026 17:49:23 GMT Received: from smtprelay01.fra02v.mail.ibm.com ([9.218.2.227]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 4g32eqabka-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Aug 2026 17:49:23 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay01.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 67JHnLT041615762 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 19 Aug 2026 17:49:21 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7C5972004B; Wed, 19 Aug 2026 17:49:21 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D3C142004E; Wed, 19 Aug 2026 17:49:17 +0000 (GMT) Received: from [9.124.213.40] (unknown [9.124.213.40]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 19 Aug 2026 17:49:17 +0000 (GMT) Message-ID: <020329f9-7cd4-41d0-8e8a-0a8babacd0c1@linux.ibm.com> Date: Wed, 19 Aug 2026 23:19:16 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing From: Shrikanth Hegde To: Peter Zijlstra Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Phil Auld , Mete Durlu , linux-kernel@vger.kernel.org, Ingo Molnar , Juri Lelli , Vincent Guittot , Andrea Righi , K Prateek Nayak References: <20260804151324.918020-1-arighi@nvidia.com> <9153b101-583a-43ac-831a-6f4d78468b25@linux.ibm.com> <4914e57a-f15b-4354-9fe4-5504efcebe24@linux.ibm.com> Content-Language: en-US In-Reply-To: <4914e57a-f15b-4354-9fe4-5504efcebe24@linux.ibm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Authority-Analysis: v=2.4 cv=LsCiDHdc c=1 sm=1 tr=0 ts=6a85eca5 cx=c_pps a=bLidbwmWQ0KltjZqbj+ezA==:117 a=bLidbwmWQ0KltjZqbj+ezA==:17 a=IkcTkHD0fZMA:10 a=Sv0fKeRqtYgA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=903iWD1lD_9MPf0Z5M8A:9 a=QEXdDO2ut3YA:10 X-Proofpoint-GUID: a3-LpkkIA1c6UtiJmGzWrLtT8J8whgUP X-Proofpoint-Spam-Info: AW1haW4tMjYwODE5MDEzNiBTYWx0ZWRfX/PgzXz7dn10M gZzVVPKbH2tamqbYzr73KkaVfGutzdmAl7EH+s6gTPuiMHyM2mSygzyWfKOd9Nj45rWN4wSgQxd 42vZ3vwdwkBAWvYn/quaBzCDKnQQAz8= X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwODE5MDEzNiBTYWx0ZWRfX72oNEuDJg6d/ MmqCTmqsw6mQCA10vD6LxMLbMr/+C3iDnlAPjlDEQRTG9e3FsQLUv8b6qF50CF8zAsiS4QSyfdJ TQJkdxuteZsHmdF/oyEDLc3Fq4Hx900YUlm4yiFTewTgjivKgkcytS7Nz4SW6Q0R3tYlbdgo1FT NNpnMSUFrmgp3bCZips3q34e79u7h4kS+8VPfbAqtKYCpexNTMExcBlGnYvwfkug+FWKWgLoWjv A7xIcppmnNOFQCdugR4HoXPNonoAhIYNjSZtRHbuvLXIHQ9r+92xyJK8HfdWUwmruRbRfgUcIdn jCLPJYFymeeQZR2UW13A1QDGEd10kZY81UNesueuzdaR5XnNdYVVdwP2eiGgxIQnxBx7PHzcTsS PyXKbp3JllP4YpV0+T+aef5sXhJMnzsAII3fCh7wHmszE7TgchZ7jg+KlLLzDzTtFak9sptD+ql loAi52A8uEogeus73wQ== X-Proofpoint-ORIG-GUID: 2XNFXfTQcm2o29n20zmIVCu3gy0khGaW X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-08-19_04,2026-08-19_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 spamscore=0 priorityscore=1501 lowpriorityscore=0 bulkscore=0 adultscore=0 suspectscore=0 impostorscore=0 clxscore=1015 phishscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2608190136 >> Separate would be fine. Perhaps another way to do it is using the fancy >> new clang context analysis. > while going through some other patch around context analysis this issue came across to my mind again. Peter, is below is what you meant? Without using guard it gets ugly pretty quickly in select_idle_cpu, find_energy_efficient_cpu etc with too many goto's. Should I work this up as a series post some ifdefs/cleanups etc? From: Shrikanth Hegde Date: Fri, 7 Aug 2026 05:58:48 -0400 Subject: [PATCH] sched: introduce generic scratch cpumask --- kernel/sched/core.c | 18 ++++- kernel/sched/deadline.c | 17 +---- kernel/sched/ext/idle.c | 27 ++----- kernel/sched/fair.c | 39 +++-------- kernel/sched/rt.c | 14 +--- kernel/sched/sched.h | 152 +++++++++++++++++++++++++++++++++++++++- 6 files changed, 184 insertions(+), 83 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2e7cde033a31..eb44b562abfb 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -130,6 +130,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_dl_server_stop_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); DEFINE_PER_CPU(struct rnd_state, sched_rnd_state); +DEFINE_PER_CPU(struct sched_scratchmask_pool, sched_scratchmask_irq); #ifdef CONFIG_SCHED_PROXY_EXEC DEFINE_STATIC_KEY_TRUE(__sched_proxy_exec); @@ -8875,9 +8876,6 @@ void __init sched_init_smp(void) current->flags &= ~PF_NO_SETAFFINITY; sched_init_granularity(); - init_sched_rt_class(); - init_sched_dl_class(); - sched_init_dl_servers(); sched_smp_initialized = true; @@ -8897,6 +8895,18 @@ int in_sched_functions(unsigned long addr) && addr < (unsigned long)__sched_text_end); } +void __init init_scratchmasks(void) +{ + struct sched_scratchmask_pool *pool; + int cpu, slot; + + for_each_possible_cpu(cpu) { + pool = per_cpu_ptr(&sched_scratchmask_irq, cpu); + for (slot = 0; slot < SCHED_SCRATCHMASK_IRQ_SLOTS; slot++) + zalloc_cpumask_var_node(&pool->mask[slot], GFP_KERNEL, cpu_to_node(cpu)); + } +} + #ifdef CONFIG_CGROUP_SCHED /* * Default task group. @@ -8946,6 +8956,8 @@ void __init sched_init(void) #endif /* CONFIG_RT_GROUP_SCHED */ + init_scratchmasks(); + init_defrootdomain(); #ifdef CONFIG_RT_GROUP_SCHED diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 857dbe3519a8..0b4ee736ffb6 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2927,13 +2927,10 @@ static struct task_struct *pick_earliest_pushable_dl_task(struct rq *rq, int cpu return NULL; } -/* Access rule: must be called on local CPU with preemption disabled */ -static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask_dl); - static int find_later_rq(struct task_struct *task) { struct sched_domain *sd; - struct cpumask *later_mask = this_cpu_cpumask_var_ptr(local_cpu_mask_dl); + guard_sched_scratchmask_irq0(later_mask); int this_cpu = smp_processor_id(); int cpu = task_cpu(task); @@ -3382,15 +3379,6 @@ static void rq_offline_dl(struct rq *rq) cpudl_clear(&rq->rd->cpudl, rq->cpu, false); } -void __init init_sched_dl_class(void) -{ - unsigned int i; - - for_each_possible_cpu(i) - zalloc_cpumask_var_node(&per_cpu(local_cpu_mask_dl, i), - GFP_KERNEL, cpu_to_node(i)); -} - /* * This function always returns a non-empty bitmap in @cpus. This is because * if a root domain has reserved bandwidth for DL tasks, the DL bandwidth @@ -3427,7 +3415,6 @@ void dl_add_task_root_domain(struct task_struct *p) struct rq *rq; struct dl_bw *dl_b; unsigned int cpu; - struct cpumask *msk; raw_spin_lock_irqsave(&p->pi_lock, rf.flags); if (!dl_task(p) || dl_entity_is_special(&p->dl)) { @@ -3435,7 +3422,7 @@ void dl_add_task_root_domain(struct task_struct *p) return; } - msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl); + guard_sched_scratchmask_irq0(msk); dl_get_task_effective_cpus(p, msk); cpu = cpumask_first_and(cpu_active_mask, msk); BUG_ON(cpu >= nr_cpu_ids); diff --git a/kernel/sched/ext/idle.c b/kernel/sched/ext/idle.c index 6f93cc32b650..4d417ee66578 100644 --- a/kernel/sched/ext/idle.c +++ b/kernel/sched/ext/idle.c @@ -47,13 +47,6 @@ static struct scx_idle_cpus scx_idle_global_masks; */ static struct scx_idle_cpus **scx_idle_node_masks; -/* - * Local per-CPU cpumasks (used to generate temporary idle cpumasks). - */ -static DEFINE_PER_CPU(cpumask_var_t, local_idle_cpumask); -static DEFINE_PER_CPU(cpumask_var_t, local_llc_idle_cpumask); -static DEFINE_PER_CPU(cpumask_var_t, local_numa_idle_cpumask); - /* * Return the idle masks associated to a target @node. * @@ -468,8 +461,12 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, /* * Determine the subset of CPUs usable by @p within @cpus_allowed. */ + guard_sched_scratchmask_irq0(local_idle_cpumask); + guard_sched_scratchmask_irq1(local_numa_idle_cpumask); + guard_sched_scratchmask_irq2(local_llc_idle_cpumask); + if (allowed != p->cpus_ptr) { - struct cpumask *local_cpus = this_cpu_cpumask_var_ptr(local_idle_cpumask); + struct cpumask *local_cpus = local_idle_cpumask; if (task_affinity_all(p)) { allowed = cpus_allowed; @@ -500,7 +497,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, * directly. */ if (static_branch_maybe(CONFIG_NUMA, &scx_selcpu_topo_numa)) { - struct cpumask *local_cpus = this_cpu_cpumask_var_ptr(local_numa_idle_cpumask); + struct cpumask *local_cpus = local_numa_idle_cpumask; const struct cpumask *cpus = numa_span(prev_cpu); if (allowed == p->cpus_ptr && task_affinity_all(p)) @@ -510,7 +507,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, } if (static_branch_maybe(CONFIG_SCHED_MC, &scx_selcpu_topo_llc)) { - struct cpumask *local_cpus = this_cpu_cpumask_var_ptr(local_llc_idle_cpumask); + struct cpumask *local_cpus = local_llc_idle_cpumask; const struct cpumask *cpus = llc_span(prev_cpu); if (allowed == p->cpus_ptr && task_affinity_all(p)) @@ -695,16 +692,6 @@ void scx_idle_init_masks(void) BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[i]->cpu, GFP_KERNEL, i)); BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[i]->smt, GFP_KERNEL, i)); } - - /* Allocate local per-cpu idle cpumasks */ - for_each_possible_cpu(i) { - BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_idle_cpumask, i), - GFP_KERNEL, cpu_to_node(i))); - BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_llc_idle_cpumask, i), - GFP_KERNEL, cpu_to_node(i))); - BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_numa_idle_cpumask, i), - GFP_KERNEL, cpu_to_node(i))); - } } static void update_builtin_idle(int cpu, bool idle) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f79fcba4afec..b1f41619c74d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8199,11 +8199,6 @@ static inline unsigned int cfs_h_nr_delayed(struct rq *rq) return (rq->cfs.h_nr_queued - rq->cfs.h_nr_runnable); } -/* Working cpumask for: sched_balance_rq(), sched_balance_newidle(). */ -static DEFINE_PER_CPU(cpumask_var_t, load_balance_mask); -static DEFINE_PER_CPU(cpumask_var_t, select_rq_mask); -static DEFINE_PER_CPU(cpumask_var_t, should_we_balance_tmpmask); - #ifdef CONFIG_NO_HZ_COMMON static struct { @@ -8674,7 +8669,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t */ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target) { - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask); + guard_sched_scratchmask_irq0(cpus); int i, cpu, idle_cpu = -1, nr = INT_MAX; if (sched_feat(SIS_UTIL) && sd->shared) { @@ -8806,10 +8801,9 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) unsigned long task_util, util_min, util_max, best_cap = 0; int fits, best_fits = ASYM_IDLE_THREAD_MISFIT; int cpu, best_cpu = -1; - struct cpumask *cpus; int nr = INT_MAX; - cpus = this_cpu_cpumask_var_ptr(select_rq_mask); + guard_sched_scratchmask_irq0(cpus); cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); task_util = task_util_est(p); @@ -8957,9 +8951,6 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) util_max = uclamp_eff_value(p, UCLAMP_MAX); } - /* - * per-cpu select_rq_mask usage - */ lockdep_assert_irqs_disabled(); if (choose_idle_cpu(target, p) && @@ -9496,7 +9487,7 @@ compute_energy(struct energy_env *eenv, struct perf_domain *pd, */ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu) { - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask); + guard_sched_scratchmask_irq0(cpus); unsigned long prev_delta = ULONG_MAX, best_delta = ULONG_MAX; unsigned long p_util_min = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MIN) : 0; unsigned long p_util_max = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MAX) : 1024; @@ -13309,8 +13300,9 @@ static int need_active_balance(struct lb_env *env) static int active_load_balance_cpu_stop(void *data); static int should_we_balance(struct lb_env *env) + __must_hold(sched_scratchmask_irq0) { - struct cpumask *swb_cpus = this_cpu_cpumask_var_ptr(should_we_balance_tmpmask); + guard_sched_scratchmask_irq1(swb_cpus); struct sched_group *sg = env->sd->groups; int cpu, idle_smt = -1; @@ -13426,7 +13418,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, struct sched_group *group; struct rq *busiest; struct rq_flags rf; - struct cpumask *cpus = this_cpu_cpumask_var_ptr(load_balance_mask); + guard_sched_scratchmask_irq0(cpus); struct lb_env env = { .sd = sd, .dst_cpu = this_cpu, @@ -14003,16 +13995,9 @@ static inline int on_null_domain(struct rq *rq) */ static inline int find_new_ilb(void) { - struct cpumask *ilb_cpus; int ilb_cpu, fallback = -1; + guard_sched_scratchmask_irq0(ilb_cpus); - lockdep_assert_irqs_disabled(); - - /* - * Reuse the per-CPU select_rq_mask, which is protected from concurrent - * use on this CPU by having interrupts disabled. - */ - ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask); cpumask_and(ilb_cpus, nohz.idle_cpus_mask, housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)); @@ -15613,18 +15598,12 @@ __init void init_sched_fair_class(void) { int i; - for_each_possible_cpu(i) { - zalloc_cpumask_var_node(&per_cpu(load_balance_mask, i), GFP_KERNEL, cpu_to_node(i)); - zalloc_cpumask_var_node(&per_cpu(select_rq_mask, i), GFP_KERNEL, cpu_to_node(i)); - zalloc_cpumask_var_node(&per_cpu(should_we_balance_tmpmask, i), - GFP_KERNEL, cpu_to_node(i)); - #ifdef CONFIG_CFS_BANDWIDTH + for_each_possible_cpu(i) { INIT_CSD(&cpu_rq(i)->cfsb_csd, __cfsb_csd_unthrottle, cpu_rq(i)); INIT_LIST_HEAD(&cpu_rq(i)->cfsb_csd_list); -#endif } - +#endif open_softirq(SCHED_SOFTIRQ, sched_balance_softirq); #ifdef CONFIG_NO_HZ_COMMON diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index e6e5f8a2caaf..8e17c4dfc1f6 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1770,12 +1770,10 @@ static struct task_struct *pick_highest_pushable_task(struct rq *rq, int cpu) return NULL; } -static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask); - static int find_lowest_rq(struct task_struct *task) { struct sched_domain *sd; - struct cpumask *lowest_mask = this_cpu_cpumask_var_ptr(local_cpu_mask); + guard_sched_scratchmask_irq0(lowest_mask); int this_cpu = smp_processor_id(); int cpu = task_cpu(task); int ret; @@ -2425,16 +2423,6 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p) rt_queue_pull_task(rq); } -void __init init_sched_rt_class(void) -{ - unsigned int i; - - for_each_possible_cpu(i) { - zalloc_cpumask_var_node(&per_cpu(local_cpu_mask, i), - GFP_KERNEL, cpu_to_node(i)); - } -} - /* * When switching a task to RT, we may overload the runqueue * with RT tasks. In this case we try to push them off to diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 26ae13c86b69..778e0342d636 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1064,6 +1064,7 @@ struct root_domain { }; extern void init_defrootdomain(void); +extern void init_scratchmasks(void); extern int sched_init_domains(const struct cpumask *cpu_map); extern void rq_attach_root(struct rq *rq, struct root_domain *rd); extern void sched_get_rd(struct root_domain *rd); @@ -2968,8 +2969,6 @@ extern void sysrq_sched_debug_show(void); extern void sched_init_granularity(void); extern void update_max_interval(void); -extern void init_sched_dl_class(void); -extern void init_sched_rt_class(void); extern void init_sched_fair_class(void); extern void resched_curr(struct rq *rq); @@ -4230,4 +4229,153 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change) #include "ext/ext.h" +enum sched_scratchmask_irq_slot { + SCHED_SCRATCHMASK_IRQ_0, + SCHED_SCRATCHMASK_IRQ_1, + SCHED_SCRATCHMASK_IRQ_2, + SCHED_SCRATCHMASK_IRQ_SLOTS, +}; + +struct sched_scratchmask_pool { + cpumask_var_t mask[SCHED_SCRATCHMASK_IRQ_SLOTS]; + unsigned long in_use; +}; + +DECLARE_PER_CPU(struct sched_scratchmask_pool, sched_scratchmask_irq); + +static __always_inline struct cpumask * +__sched_get_scratchmask_irq(unsigned int slot) +{ + struct sched_scratchmask_pool *pool; + + lockdep_assert_irqs_disabled(); + + if (WARN_ON_ONCE(slot >= SCHED_SCRATCHMASK_IRQ_SLOTS)) + return NULL; + + pool = this_cpu_ptr(&sched_scratchmask_irq); + + if (WARN_ON_ONCE(!pool->mask[slot])) + return NULL; + + if (WARN_ON_ONCE(test_and_set_bit(slot, &pool->in_use))) + return NULL; + + return pool->mask[slot]; +} + +static __always_inline void +__sched_put_scratchmask_irq(unsigned int slot) +{ + struct sched_scratchmask_pool *pool; + + lockdep_assert_irqs_disabled(); + + if (WARN_ON_ONCE(slot >= SCHED_SCRATCHMASK_IRQ_SLOTS)) + return; + + pool = this_cpu_ptr(&sched_scratchmask_irq); + + WARN_ON_ONCE(!test_and_clear_bit(slot, &pool->in_use)); +} + +#include +#include + +token_context_lock(sched_scratchmask_irq0); +token_context_lock(sched_scratchmask_irq1); +token_context_lock(sched_scratchmask_irq2); + +struct sched_scratchmask { + struct cpumask *mask; + unsigned int slot; +}; + +static __always_inline struct sched_scratchmask +sched_scratchmask_acquire_irq0(void) +{ + struct sched_scratchmask scratch = { + .mask = __sched_get_scratchmask_irq(SCHED_SCRATCHMASK_IRQ_0), + .slot = SCHED_SCRATCHMASK_IRQ_0, + }; + + __acquire(sched_scratchmask_irq0); + return scratch; +} + +static __always_inline void +sched_scratchmask_release_irq0(struct sched_scratchmask *scratch) +{ + if (scratch->mask) + __sched_put_scratchmask_irq(SCHED_SCRATCHMASK_IRQ_0); + + __release(sched_scratchmask_irq0); +} + +static __always_inline struct sched_scratchmask +sched_scratchmask_acquire_irq1(void) +{ + struct sched_scratchmask scratch = { + .mask = __sched_get_scratchmask_irq(SCHED_SCRATCHMASK_IRQ_1), + .slot = SCHED_SCRATCHMASK_IRQ_1, + }; + + __acquire(sched_scratchmask_irq1); + return scratch; +} + +static __always_inline void +sched_scratchmask_release_irq1(struct sched_scratchmask *scratch) +{ + if (scratch->mask) + __sched_put_scratchmask_irq(SCHED_SCRATCHMASK_IRQ_1); + + __release(sched_scratchmask_irq1); +} + +static __always_inline struct sched_scratchmask +sched_scratchmask_acquire_irq2(void) +{ + struct sched_scratchmask scratch = { + .mask = __sched_get_scratchmask_irq(SCHED_SCRATCHMASK_IRQ_2), + .slot = SCHED_SCRATCHMASK_IRQ_2, + }; + + __acquire(sched_scratchmask_irq2); + return scratch; +} + +static __always_inline void +sched_scratchmask_release_irq2(struct sched_scratchmask *scratch) +{ + if (scratch->mask) + __sched_put_scratchmask_irq(SCHED_SCRATCHMASK_IRQ_2); + + __release(sched_scratchmask_irq2); +} + +DEFINE_CLASS(sched_scratchmask_irq0, struct sched_scratchmask, + sched_scratchmask_release_irq0(&_T), + sched_scratchmask_acquire_irq0(), void); + +DEFINE_CLASS(sched_scratchmask_irq1, struct sched_scratchmask, + sched_scratchmask_release_irq1(&_T), + sched_scratchmask_acquire_irq1(), void); + +DEFINE_CLASS(sched_scratchmask_irq2, struct sched_scratchmask, + sched_scratchmask_release_irq2(&_T), + sched_scratchmask_acquire_irq2(), void); + +#define guard_sched_scratchmask_irq0(_mask) \ + CLASS(sched_scratchmask_irq0, __scratch_##_mask)(); \ + struct cpumask *_mask = __scratch_##_mask.mask + +#define guard_sched_scratchmask_irq1(_mask) \ + CLASS(sched_scratchmask_irq1, __scratch_##_mask)(); \ + struct cpumask *_mask = __scratch_##_mask.mask + +#define guard_sched_scratchmask_irq2(_mask) \ + CLASS(sched_scratchmask_irq2, __scratch_##_mask)(); \ + struct cpumask *_mask = __scratch_##_mask.mask + #endif /* _KERNEL_SCHED_SCHED_H */ -- 2.47.3