From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D5EF336883 for ; Tue, 7 Apr 2026 19:21:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775589710; cv=none; b=e86NQjwG0tDaubCcK7wgPxzoqoBRJD8D3pyB7c84505o0AyniaSm+5P8sTUpFd75hpH2GyD0+pBfGcmhcgoRKKsAWwIEhxKPpXcxxFT54OQk6wbpks4ef/J/EccUO+/rqOHpiSD3jpxiAc04QKnJg8SWQM3NybuydZbmBPWhV4o= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775589710; c=relaxed/simple; bh=fPTTyUEMxfQbWwUbPDJilRu93V+38OSMoU0Y2zdPJ4o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kLaLwoVBMPZ4kO5vKVO8k7MQiwRTs5iZnOuGdPcC7LvJd1vN6eLhL6gCNfexJuxrzCadFt5D4uyYo2Wyn8Qobn52peXbHT/ps1EC8XKHhlz1B5RkEneLWWhMtSiBV9pDNaM4qurrgtReiYJF3pUlwV8Hu79+hHz3GYX+3Q+lxuM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=gruMaw8r; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="gruMaw8r" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 637DmSZh2213384; Tue, 7 Apr 2026 19:21:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=mFY4RzirG9nq9H7ZA i1EavPiIOO9ccONp4UqWS4Nw5Y=; b=gruMaw8r8iv7BM8W60VBqUeMXro+RicnU gumVlGdMG5/g2r6haSXrKieKU8ztyT7NhuemP0jMujhvxjr8dGcrz5aNnMsN/luZ iYzoechGmvwerR2vDSBlRAO2uqw2j94nWGEOkzsGwZVirrz9iPvHyUtcES9sXnaC xT7FeZcVc54BI/YyTlLx8d1c5f7JEAM0hnrfXUwbgBVx61vU9c+/FKqLpsg5Am5s fYxaoPoSL6gmW70/SrVysK4gZgAPSgNdvWlIwFdRYlf6s7xG5HPML5xS+2SXIobJ KnuSGwXaIMVJyLW1eElEqOG31tXPGt9CdTEqPNzdVlTuCNxQJ5AbQ== Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4dcn2hccf7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2026 19:21:05 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 637IlaN1014348; Tue, 7 Apr 2026 19:21:04 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 4dcmg4mhv2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2026 19:21:04 +0000 Received: from smtpav05.fra02v.mail.ibm.com (smtpav05.fra02v.mail.ibm.com [10.20.54.104]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 637JL0rI25952734 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2026 19:21:01 GMT Received: from smtpav05.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D0E8120043; Tue, 7 Apr 2026 19:21:00 +0000 (GMT) Received: from smtpav05.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 63DFE20040; Tue, 7 Apr 2026 19:20:56 +0000 (GMT) Received: from li-7bb28a4c-2dab-11b2-a85c-887b5c60d769.ibm.com.com (unknown [9.39.20.194]) by smtpav05.fra02v.mail.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2026 19:20:56 +0000 (GMT) From: Shrikanth Hegde To: linux-kernel@vger.kernel.org, mingo@kernel.org, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, tglx@linutronix.de, yury.norov@gmail.com, gregkh@linuxfoundation.org Cc: sshegde@linux.ibm.com, pbonzini@redhat.com, seanjc@google.com, kprateek.nayak@amd.com, vschneid@redhat.com, iii@linux.ibm.com, huschle@linux.ibm.com, rostedt@goodmis.org, dietmar.eggemann@arm.com, mgorman@suse.de, bsegall@google.com, maddy@linux.ibm.com, srikar@linux.ibm.com, hdanton@sina.com, chleroy@kernel.org, vineeth@bitbyteword.org, joelagnelf@nvidia.com Subject: [PATCH v2 10/17] sched/core: Push current task from non preferred CPU Date: Wed, 8 Apr 2026 00:49:43 +0530 Message-ID: <20260407191950.643549-11-sshegde@linux.ibm.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260407191950.643549-1-sshegde@linux.ibm.com> References: <20260407191950.643549-1-sshegde@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNDA3MDE3NSBTYWx0ZWRfX+89yVyDUm2Bz zAw53BLkxIVl4k9pVgJF3KobYYguofMuineyFN+ZqU8SurqhYVKUOQmjk7b/ad/y++Sl0oxooRk tryUMsM4eI9C7FAvzKdEDtEV984Cgx/psR2l2Hjf4bukGCxgQeBJ5D8C8w8/Wi3I3baQmZK1AHN SK0tFjRbjg2eA6aR69v8gL09Us6K+LVF1w5hd0QBz8B9aO/2gdW9LhAwECwGB7i2gTS19CKHFCw 6jQNSYH6IB2PVWievDjkzo0F1XjD/52FB2s4uGfUMLUsFZhiNNHuvrT0kurIU+cnsR7Acg2aVKY CuFyOgmd73PVgsg2FqLXxDN9FMOHhshWau4mBkKg8xZEAPwMVjPWt8aHq69ZRZf6YNTFB2M7Xh/ V2kL1WAInlb4rZjEAgMTbWLU6umqnCPaaDHAeHPzDTe4UzAC0i/SIizKXRAlVO1PjLH2EBlXMXB 9sBHWvNiRs79/xHy3QA== X-Proofpoint-GUID: XcP599203HowhfKAocaMUVMEUt3XZb3I X-Authority-Analysis: v=2.4 cv=a/wAM0SF c=1 sm=1 tr=0 ts=69d55921 cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=A5OVakUREuEA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=V8glGbnc2Ofi9Qvn3v5h:22 a=VnNF1IyMAAAA:8 a=DHV1smzMkSx7QAEspXQA:9 X-Proofpoint-ORIG-GUID: U_44uhBUXbgWOhKrWQ9ulQ8AAFhml1DJ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-04-07_04,2026-04-07_05,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 clxscore=1015 spamscore=0 impostorscore=0 priorityscore=1501 phishscore=0 lowpriorityscore=0 adultscore=0 malwarescore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2604010000 definitions=main-2604070175 Actively push out RT/CFS running on a non-preferred CPU. Since the task is running on the CPU, need to stop the cpu and push the task out. However, if the task in pinned only to non-preferred CPUs, it will continue running there. This will help in maintaining the userspace affinities unlike CPU hotplug or isolated cpusets. Though code is almost same as __balance_push_cpu_stop and quite close to push_cpu_stop, it is being kept separate as it provides a cleaner implementation w.r.t to PARAVIRT config. Add push_task_work_done flag to protect work buffer. This currently works only fair and rt class. Signed-off-by: Shrikanth Hegde --- kernel/sched/core.c | 84 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 8 +++++ 2 files changed, 92 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index c7f046443dc5..b375c500d49e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5652,6 +5652,10 @@ void sched_tick(void) unsigned long hw_pressure; u64 resched_latency; + /* push the current CFS/RT task out if its on non-preferred CPU */ + if (!cpu_preferred(cpu)) + sched_push_current_non_preferred_cpu(rq); + if (housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE)) arch_scale_freq_tick(); @@ -11247,4 +11251,84 @@ void sched_change_end(struct sched_change_ctx *ctx) #ifdef CONFIG_PARAVIRT struct cpumask __cpu_preferred_mask __read_mostly; EXPORT_SYMBOL(__cpu_preferred_mask); + +/* npc - non preferred CPU */ +static DEFINE_PER_CPU(struct cpu_stop_work, npc_push_task_work); + +static int sched_non_preferred_cpu_push_stop(void *arg) +{ + struct task_struct *p = arg; + struct rq *rq = this_rq(); + struct rq_flags rf; + int cpu; + + raw_spin_lock_irq(&p->pi_lock); + rq_lock(rq, &rf); + rq->push_task_work_done = 0; + + update_rq_clock(rq); + + if (task_rq(p) == rq && task_on_rq_queued(p)) { + cpu = select_fallback_rq(rq->cpu, p); + rq = __migrate_task(rq, &rf, p, cpu); + } + + rq_unlock(rq, &rf); + raw_spin_unlock_irq(&p->pi_lock); + put_task_struct(p); + + return 0; +} + +/* Using this CPU will lead to more hypervisor preemptions. + * It is better not to use this CPU. + * + * In case any task is scheduled on such CPU, move it out. In + * select_fallback_rq a preferred CPU will be chosen and henceforth + * task shouldn't come back to this CPU + */ +void sched_push_current_non_preferred_cpu(struct rq *rq) +{ + struct task_struct *push_task = rq->curr; + unsigned long flags; + struct rq_flags rf; + + /* sanity check */ + if (cpu_preferred(rq->cpu)) + return; + + /* Idle task can't be pused out */ + if (rq->curr == rq->idle) + return; + + /* Do for only SCHED_NORMAL AND RT for now */ + if (push_task->sched_class != &fair_sched_class && + push_task->sched_class != &rt_sched_class) + return; + + if (kthread_is_per_cpu(push_task) || + is_migration_disabled(push_task)) + return; + + /* Is there any preferred CPU in the affinity list */ + if (!task_can_run_on_preferred_cpu(push_task)) + return; + + /* There is already a stopper thread for this. Dont race with it */ + if (rq->push_task_work_done == 1) + return; + + local_irq_save(flags); + + get_task_struct(push_task); + + rq_lock(rq, &rf); + rq->push_task_work_done = 1; + rq_unlock(rq, &rf); + + stop_one_cpu_nowait(rq->cpu, sched_non_preferred_cpu_push_stop, + push_task, this_cpu_ptr(&npc_push_task_work)); + local_irq_restore(flags); +} + #endif diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 4c45092b2fce..c1d037f11c62 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1239,6 +1239,10 @@ struct rq { unsigned char nohz_idle_balance; unsigned char idle_balance; +#ifdef CONFIG_PARAVIRT + bool push_task_work_done; +#endif + unsigned long misfit_task_load; /* For active balancing */ @@ -4138,11 +4142,15 @@ static inline bool task_can_run_on_preferred_cpu(struct task_struct *p) { return cpumask_intersects(p->cpus_ptr, cpu_preferred_mask); } + +void sched_push_current_non_preferred_cpu(struct rq *rq); #else static inline bool task_can_run_on_preferred_cpu(struct task_struct *p) { return true; } + +static inline void sched_push_current_non_preferred_cpu(struct rq *rq) { } #endif #endif /* _KERNEL_SCHED_SCHED_H */ -- 2.47.3