From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A29E3B8D71 for ; Sun, 7 Dec 2025 12:13:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765109636; cv=none; b=MZxeQXyMrEqhgI0OWBUTgSQaHGba0fZ74f2fKL+ja8F69dieB/eWnhY6FPdTeYeq2JvHboyrFQyAdF03PUQH1ojYMFR7y5ntT/bm5S+VeFtc3jXF8IYs3Oesu5JvEWV5nuPZlisCTszZD6ylJnCvWsZihz2f8AM5xDsjqjhQy5U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765109636; c=relaxed/simple; bh=TwGu0+pvHlF/f9H9Ux15apW0PFRp0ta5N9yO46oyZls=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=YHFDiK+RiDWUtNA0pIp9rXRULi0pGsZQhNkA2lcUOhRozSqXp3hXeo4ObfdgGM2EqvsbiSe2fDZ2TDorQZZzZ5zqemW9lbWjR4SPZF5dQwQyuh6Ot8pB/XLHwPGkw0/syW3S/zdWOxE261FU6bx1/p+6qSEavjkGrXQrnPaf6kA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=h/jI78Zl; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="h/jI78Zl" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 5B6Ns7Pm026752; Sun, 7 Dec 2025 12:13:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=xgOzTt xarv1t9cCzMfXaNF4yuV1bTsuVssAMc2abksQ=; b=h/jI78ZlsmYTTP6AGYxMMM W6Jw1LJ1x0lOLwzYRGAkE+KkgzpBPNEi8ZIRaGqxpNEp3PayMMFFKByX3YNz8RTB Kdh8MARPjvDSFW0eJMKscpB0lCzidBcd51cwgEhjCB+16e/Z2TOJ/ytDkTyhOqfQ iKIfX9af2fgv2bGYiCesZkmi9EHwejwG4IlC5d2uFp6dpF4FQ2/+LTlpZD44A2is castg7deCC2UNVmD8CY2maAQiYNNI/4koZAET9N+Ldog08TiI1DZOc5kHEQqCo73 Ki/N68XOQoUyleuIXljzflF8+abzwKrArBl9cjWHKnZZ6J1inZLQ4APuot9u5WRw == Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4av9wvc06c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 07 Dec 2025 12:13:36 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 5B7CDZhW026020; Sun, 7 Dec 2025 12:13:35 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4av9wvc06a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 07 Dec 2025 12:13:35 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 5B79eImi026799; Sun, 7 Dec 2025 12:13:34 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 4aw1h0s6hy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 07 Dec 2025 12:13:34 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 5B7CDW2X32178490 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 7 Dec 2025 12:13:33 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D34E320043; Sun, 7 Dec 2025 12:13:32 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4DE2D20040; Sun, 7 Dec 2025 12:13:29 +0000 (GMT) Received: from [9.39.31.222] (unknown [9.39.31.222]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 7 Dec 2025 12:13:29 +0000 (GMT) Message-ID: Date: Sun, 7 Dec 2025 17:43:28 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 4/6 v8] sched/fair: Add push task mechanism for fair To: Vincent Guittot , mingo@redhat.com, peterz@infradead.org, vschneid@redhat.com, juri.lelli@redhat.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com References: <20251202181242.1536213-1-vincent.guittot@linaro.org> <20251202181242.1536213-5-vincent.guittot@linaro.org> Content-Language: en-US From: Shrikanth Hegde In-Reply-To: <20251202181242.1536213-5-vincent.guittot@linaro.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: pnNp-zOOFuIX6N926YS4BMEGW5aH7qoK X-Proofpoint-ORIG-GUID: EoH5supS6EnDAxJY0ur-Bpdl7XX8RRbT X-Authority-Analysis: v=2.4 cv=AdS83nXG c=1 sm=1 tr=0 ts=69356f70 cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=IkcTkHD0fZMA:10 a=wP3pNCr1ah4A:10 a=VkNPw1HP01LnGYTKEx00:22 a=KKAkSRfTAAAA:8 a=Y8hsWrfxbevIX5lAADUA:9 a=QEXdDO2ut3YA:10 a=cvBusfyB2V15izCimMoJ:22 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUxMjA2MDAwMCBTYWx0ZWRfX6u7nyyBafm4+ JHo8mL6Q+VzStr44LycE6d4pWo6KNE+X2QKyTRotYb8BLnbFQMJTG3HXUHi4p27kO5dfLvWCXk+ PqS+f2wrsEfPr5lYuf+WF5edw28Pj/f7qrj8wdZLgobc2ONy6PDJ4PtkYEuK6pWOjoLptrh84mX cOi6Sy4BdBCvmhP6qWRg1t9p9pOE9NV1QcpCrravHMTHpvHRjPHGqORwPRMWOMrr/AWKOp7SX6I G1CzI3X9zWvE4DxH0JwCGMZXR/1Pi9rCZpJPfNqw1EKnb1kmYL1WNCIOFyNpTa2F7w8T9ARgXwp jwGGS7aeRj1L1xsSr+Si5L+/v2Lr2kNg4Fo7eE15Z4UncbpmPUzkFclWQfFUgwgYChjeQzKjrt4 nJ8yANW/1rsmmyzFeIF8vW29csnD0A== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1121,Hydra:6.1.9,FMLib:17.12.100.49 definitions=2025-12-06_02,2025-12-04_04,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 adultscore=0 priorityscore=1501 spamscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 clxscore=1011 malwarescore=0 impostorscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2510240000 definitions=main-2512060000 On 12/2/25 11:42 PM, Vincent Guittot wrote: > EAS is based on wakeup events to efficiently place tasks on the system, but > there are cases where a task doesn't have wakeup events anymore or at a far > too low pace. For such situation, we can take advantage of the task being > put back in the enqueued list to check if it should be pushed on another > CPU. > When the task is alone on the CPU, it's never put back in the enqueued > list; In this special case, we use the tick to run the check. > > Add a push task mechanism that enables fair scheduler to push runnable > tasks. EAS will be one user but other feature like filling idle CPUs > can also take advantage of it. > > Signed-off-by: Vincent Guittot > --- > kernel/sched/fair.c | 212 ++++++++++++++++++++++++++++++++++++++++++- > kernel/sched/sched.h | 4 + > 2 files changed, 214 insertions(+), 2 deletions(-) > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 80c4131fb35b..252254168c92 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -6989,6 +6989,8 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) > hrtick_update(rq); > } > > +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p); > + > /* > * Basically dequeue_task_fair(), except it can deal with dequeue_entity() > * failing half-way through and resume the dequeue later. > @@ -7017,6 +7019,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) > h_nr_idle = task_has_idle_policy(p); > if (task_sleep || task_delayed || !se->sched_delayed) > h_nr_runnable = 1; > + > + fair_remove_pushable_task(rq, p); > } > > for_each_sched_entity(se) { > @@ -8504,6 +8508,187 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu) > return target; > } > > +DEFINE_STATIC_KEY_FALSE(sched_push_task); > + > +static inline bool sched_push_task_enabled(void) > +{ > + return static_branch_unlikely(&sched_push_task); > +} > + > +static bool fair_push_task(struct rq *rq, struct task_struct *p) > +{ > + return false; > +} > + > +static inline int has_pushable_tasks(struct rq *rq) > +{ > + return !plist_head_empty(&rq->cfs.pushable_tasks); > +} > + > +static struct task_struct *pick_next_pushable_fair_task(struct rq *rq) > +{ > + struct task_struct *p; > + > + if (!has_pushable_tasks(rq)) > + return NULL; > + > + p = plist_first_entry(&rq->cfs.pushable_tasks, > + struct task_struct, pushable_tasks); > + > + WARN_ON_ONCE(rq->cpu != task_cpu(p)); > + WARN_ON_ONCE(task_current(rq, p)); > + WARN_ON_ONCE(p->nr_cpus_allowed <= 1); > + WARN_ON_ONCE(!task_on_rq_queued(p)); > + > + /* > + * Remove task from the pushable list as we try only once after that > + * the task has been put back in enqueued list. > + */ > + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); > + > + return p; > +} > + > +static int > +select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags); > + > +/* > + * See if the non running fair tasks on this rq can be sent on other CPUs > + * that fits better with their profile. > + */ > +static bool push_fair_task(struct rq *rq) > +{ > + struct task_struct *next_task; > + int prev_cpu, new_cpu; > + struct rq *new_rq; > + > + next_task = pick_next_pushable_fair_task(rq); > + if (!next_task) > + return false; > + > + if (is_migration_disabled(next_task)) > + return true; > + > + /* We might release rq lock */ > + get_task_struct(next_task); > + > + prev_cpu = rq->cpu; > + > + new_cpu = select_task_rq_fair(next_task, prev_cpu, 0); > + > + if (new_cpu == prev_cpu) > + goto out; > + > + new_rq = cpu_rq(new_cpu); > + > + if (double_lock_balance(rq, new_rq)) { > + /* The task has already migrated in between */ > + if (task_cpu(next_task) != rq->cpu) { > + double_unlock_balance(rq, new_rq); > + goto out; > + } > + > + deactivate_task(rq, next_task, 0); > + set_task_cpu(next_task, new_cpu); > + activate_task(new_rq, next_task, 0); > + > + resched_curr(new_rq); > + > + double_unlock_balance(rq, new_rq); > + } > + > +out: > + put_task_struct(next_task); > + > + return true; > +} > + > +static void push_fair_tasks(struct rq *rq) > +{ > + /* push_fair_task() will return true if it moved a fair task */ > + while (push_fair_task(rq)) > + ; > +} > + > +static DEFINE_PER_CPU(struct balance_callback, fair_push_head); > + > +static inline void fair_queue_pushable_tasks(struct rq *rq) > +{ > + if (!sched_push_task_enabled() || !has_pushable_tasks(rq)) > + return; > + > + queue_balance_callback(rq, &per_cpu(fair_push_head, rq->cpu), push_fair_tasks); > +} > + > +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p) > +{ > + if (sched_push_task_enabled()) > + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); > +} > + > +static void fair_add_pushable_task(struct rq *rq, struct task_struct *p) > +{ > + if (sched_push_task_enabled() && fair_push_task(rq, p)) { > + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); > + plist_node_init(&p->pushable_tasks, p->prio); > + plist_add(&p->pushable_tasks, &rq->cfs.pushable_tasks); > + } > +} > + > +static int active_load_balance_cpu_stop(void *data); > + > +/* > + * See if the alone task running on the CPU should migrate on a better than > + * the local one. > + */ > +static inline bool check_pushable_task(struct task_struct *p, struct rq *rq) > +{ > + int new_cpu, cpu = cpu_of(rq); > + > + if (!sched_push_task_enabled()) > + return false; > + > + if (WARN_ON(!p)) > + return false; > + > + if (WARN_ON(!task_current(rq, p))) > + return false; > + > + if (is_migration_disabled(p)) > + return false; > + > + /* If there are several task, wait for being put back */ > + if (rq->nr_running > 1) > + return false; > + > + if (!fair_push_task(rq, p)) > + return false; > + RT matters for EAS too? or only CFS? Since we have quite a few patches floating around push task framework, can we generalize the framework for pushing the current task out? push_current_task(rq, CFS|RT|DL|IDLE|EXT|ALL) - Depending on the second argument push the task out after doing necessary class specific checks? Maybe a new method be added per class. - current cpu hotplug code can make use of this infra with (ALL) - push_rt_task with (RT), sched_balance_rq (CFS) - push_current_from_paravirt_cpu (CFS|RT) (Patch series which i sent few days ago) I know it is tricky right now due to specific checks in each path and the way new cpu is found is different and all that. affine_move_task seems quite complicated to fit in. Maybe i thinking too far. > + new_cpu = select_task_rq_fair(p, cpu, 0); > + > + if (new_cpu == cpu) > + return false; > + > + /* > + * ->active_balance synchronizes accesses to > + * ->active_balance_work. Once set, it's cleared > + * only after active load balance is finished. > + */ > + if (!rq->active_balance) { > + rq->active_balance = 1; > + rq->push_cpu = new_cpu; > + } else > + return false; > + > + raw_spin_rq_unlock(rq); can this race with sched_balance_rq? I think it is okay since rq->active_balance = 0 at the end. so work buffer should be protected. > + stop_one_cpu_nowait(cpu, > + active_load_balance_cpu_stop, rq, > + &rq->active_balance_work); > + raw_spin_rq_lock(rq); > + > + return true; > +} > + > /* > * select_task_rq_fair: Select target runqueue for the waking task in domains > * that have the relevant SD flag set. In practice, this is SD_BALANCE_WAKE, > @@ -8973,6 +9158,12 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf > put_prev_entity(cfs_rq, pse); > set_next_entity(cfs_rq, se); > > + /* > + * The previous task might be eligible for being pushed on > + * another cpu if it is still active. > + */ > + fair_add_pushable_task(rq, prev); > + > __set_next_task_fair(rq, p, true); > } > > @@ -9036,6 +9227,13 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t > cfs_rq = cfs_rq_of(se); > put_prev_entity(cfs_rq, se); > } > + > + /* > + * The previous task might be eligible for being pushed on another cpu > + * if it is still active. > + */ > + fair_add_pushable_task(rq, prev); > + > } > > /* > @@ -13390,8 +13588,10 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued) > if (static_branch_unlikely(&sched_numa_balancing)) > task_tick_numa(rq, curr); > > - update_misfit_status(curr, rq); > - check_update_overutilized_status(task_rq(curr)); > + if (!check_pushable_task(curr, rq)) { > + update_misfit_status(curr, rq); > + check_update_overutilized_status(task_rq(curr)); > + } > > task_tick_core(rq, curr); > } > @@ -13552,6 +13752,8 @@ static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool firs > { > struct sched_entity *se = &p->se; > > + fair_remove_pushable_task(rq, p); > + > if (task_on_rq_queued(p)) { > /* > * Move the next running task to the front of the list, so our > @@ -13567,6 +13769,11 @@ static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool firs > if (hrtick_enabled_fair(rq)) > hrtick_start_fair(rq, p); > > + /* > + * Try to push prev task before checking misfit for next task as > + * the migration of prev can make next fitting the CPU > + */ > + fair_queue_pushable_tasks(rq); > update_misfit_status(p, rq); > sched_fair_update_stop_tick(rq, p); > } > @@ -13596,6 +13803,7 @@ void init_cfs_rq(struct cfs_rq *cfs_rq) > { > cfs_rq->tasks_timeline = RB_ROOT_CACHED; > cfs_rq->zero_vruntime = (u64)(-(1LL << 20)); > + plist_head_init(&cfs_rq->pushable_tasks); > raw_spin_lock_init(&cfs_rq->removed.lock); > } > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > index b419a4d98461..697bd654298a 100644 > --- a/kernel/sched/sched.h > +++ b/kernel/sched/sched.h > @@ -711,6 +711,8 @@ struct cfs_rq { > unsigned long runnable_avg; > } removed; > > + struct plist_head pushable_tasks; > + > #ifdef CONFIG_FAIR_GROUP_SCHED > u64 last_update_tg_load_avg; > unsigned long tg_load_avg_contrib; > @@ -3620,6 +3622,8 @@ static inline bool sched_energy_enabled(void) { return false; } > > #endif /* !(CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL) */ > > +DECLARE_STATIC_KEY_FALSE(sched_push_task); > + You have sched_energy_present which is also enabled at the same point. Do you see more usecases for sched_push_task?