From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 490F8374183 for ; Wed, 17 Jun 2026 17:44:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781718286; cv=none; b=Fex5zJAA5Ybwa7pCzkGIaUpGSYxesxkGUTrKp/UYHvo9MP6Cc6FfIw0s4XDabvSjHGEaWCChRtOR9YL2/8/ik0dEV9gW2HHIXbOS+IXqonpoVFeUpRxfM2SO0Eg7uyL+iH4aeMC3bYK2oA60DmMi3mXpJBZEh0ePqFThOgZboko= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781718286; c=relaxed/simple; bh=hHwSrnhYfuJHxEwmoYAoWpdViuI18lMTUBcMy77tVsM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UwQQoVMDjpLLgXEsk76QuEsK5GjQxm6xeRYkh4KJPwdiUDTuRjoKkEbev1I/VLCcAbem5i5SL9JS7CQ6amJBpw/qYMQFC8CGO+wAYsLZTtBt8HjelzXyuAqJhYo3Jog8LdLcytM0Ze3XBROrZ0d1BDWZWYRa1CXnS2d9dbXwVoU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=OKicYRPj; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="OKicYRPj" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 65HFqBOE762100; Wed, 17 Jun 2026 17:44:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=5/hpKIX30DLYdWiTF yixEnshnMZdtQ8uSkRwaat7cxI=; b=OKicYRPj9+tsomKgphWy3DCrL1HgVYlZ5 iA9r4whMTG/Ylpv0eDm6pLx9qWZvBMZ2tQA86B7Sr821T0evDoOP5OeNwuVg06S4 s5rwHRpZCfm2BXa1Q2F/JP6sjhSExq5jyAlgLR5Y7Ked2NAqu/pjrAPF1lLURrxd b4/g7nnqJCSe/i+dc6JZv0ti4Imtz/jiUXaI53z90ngXoQSd7EA6zi5YySaFP1NC 6uHSZmOfDDnItzAYJU57Af9EUz/E8CRa36pMpTE/qePabX1CU7jjCPw3e8IKDhgh ZkJ51vadwiT/1t3glczkfdtzn91FqiCrvj8quePgtM9b2w/G6k/fQ== Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4eueqxc6j2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 17 Jun 2026 17:44:28 +0000 (GMT) Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 65HHYflW013003; Wed, 17 Jun 2026 17:44:27 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 4eudva4cmr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 17 Jun 2026 17:44:27 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (smtpav06.fra02v.mail.ibm.com [10.20.54.105]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 65HHiNMM52429176 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 17 Jun 2026 17:44:24 GMT Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DD14020049; Wed, 17 Jun 2026 17:44:23 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D070020040; Wed, 17 Jun 2026 17:44:16 +0000 (GMT) Received: from li-7bb28a4c-2dab-11b2-a85c-887b5c60d769.ibm.com.com (unknown [9.39.28.77]) by smtpav06.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 17 Jun 2026 17:44:16 +0000 (GMT) From: Shrikanth Hegde To: linux-kernel@vger.kernel.org, mingo@kernel.org, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, yury.norov@gmail.com, kprateek.nayak@amd.com, iii@linux.ibm.com Cc: sshegde@linux.ibm.com, tglx@kernel.org, gregkh@linuxfoundation.org, pbonzini@redhat.com, seanjc@google.com, vschneid@redhat.com, huschle@linux.ibm.com, rostedt@goodmis.org, dietmar.eggemann@arm.com, mgorman@suse.de, bsegall@google.com, maddy@linux.ibm.com, srikar@linux.ibm.com, hdanton@sina.com, chleroy@kernel.org, vineeth@bitbyteword.org, frederic@kernel.org, arighi@nvidia.com, pauld@redhat.com, christian.loehle@arm.com, tj@kernel.org, tommaso.cucinotta@gmail.com, maz@kernel.org, rafael@kernel.org Subject: [PATCH v4 15/20] sched/core: Compute steal values at regular intervals Date: Wed, 17 Jun 2026 23:11:34 +0530 Message-ID: <20260617174139.155540-16-sshegde@linux.ibm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260617174139.155540-1-sshegde@linux.ibm.com> References: <20260617174139.155540-1-sshegde@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Authority-Analysis: v=2.4 cv=OcSoyBTY c=1 sm=1 tr=0 ts=6a32dcfc cx=c_pps a=bLidbwmWQ0KltjZqbj+ezA==:117 a=bLidbwmWQ0KltjZqbj+ezA==:17 a=FelO9ux0wxsA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=V8glGbnc2Ofi9Qvn3v5h:22 a=VnNF1IyMAAAA:8 a=53LOTjAdL3b_QjPWis8A:9 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNjE3MDE2NiBTYWx0ZWRfX3fIF1kCrZICz KSnMViUYl85T/mdzzC5//8J0uFRs+WTPXyI90qHP74O3D67DmiGylEMaukcO1J2DqEeYNJu5xVZ GZOrszXDFlaaZCPjdOyq9KgbAGCZJNHcGEENnuSJR4UtMx2PbjGkoq+WQvNNsFbx0G9+5hcZ7jz 699E3R5CnwLtM+TC8dsUh60cPnyj3KHHz+ZzhKUswhyixJuItL29+ZwvIcNHZbf9Yha/uyEWIDl x0tWYWumCVFCJHsbimR2zqStm1yqyQ1ungqZrIrpPvnAYrJgcnlBXWa8smAxANbGJVeHeaAqQTh M+kF9PJ6xIY8m/OLyWDHXPi8kDUuMA59SkYc/AUUeKs2lkxyWBL+gwrF8pch/3/R3DHRXBO3fOE S0aBv4ESi391pU6s4RPFFLgtFQ1KngU8A1a2oqq4BZRhPHOYUZXeQMb70maR2zyG0NRZ53c6t6/ ZZ++uEDekbyGl8hnXzg== X-Proofpoint-GUID: SMEu1g8S4Q3p784oEgpsJ42ib0EDMMc7 X-Proofpoint-Spam-Info: AW1haW4tMjYwNjE3MDE2NiBTYWx0ZWRfX7YJASQZ4ciw+ uukhxLfpB99pBvg++LEQ25iFbL78YL7R454vkxKQ/5Ddo6ntEsFOrspznPhxe8oaSgtvfalYskm Q6HgxoVz9silZfM1mGyRN1LZtLVul4U= X-Proofpoint-ORIG-GUID: Jo7mpdAsBuMu1D1_mLLoXIWDymPiiPhg X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.125,FMLib:17.12.100.49 definitions=2026-06-17_02,2026-06-17_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 phishscore=0 bulkscore=0 suspectscore=0 lowpriorityscore=0 impostorscore=0 malwarescore=0 spamscore=0 adultscore=0 clxscore=1015 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2606170166 Kick off the work to compute the steal time at regular interval. Gated with steal monitor enabled static key check to avoid any overhead when its disabled. The sampling period can changed at runtime using steal_mon/sampling_period. By default is 1000 milliseconds. I.e. 1 second This work is done by first active housekeeping CPU only. Hence it won't need any complicated synchronization. Now, that sched_steal_mon_enabled() is available which is a static branch, add this to hotpath such as wakeup and load balance. This will make them effectively nop when the feature is disabled. Signed-off-by: Shrikanth Hegde --- v3->v4: - Add static key check in hotpaths. Could be split into a separate patch. Let me know if thats better. include/linux/sched.h | 2 ++ kernel/sched/core.c | 28 +++++++++++++++++++++++++++- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 3 ++- kernel/sched/sched.h | 10 +++++++++- 5 files changed, 41 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index ce6bc8a22eb1..5b15353ed7ef 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2527,5 +2527,7 @@ struct steal_monitor_t { unsigned int high_threshold; unsigned int sampling_period_ms; }; + +extern struct steal_monitor_t steal_mon; #endif #endif diff --git a/kernel/sched/core.c b/kernel/sched/core.c index cc48632dd42d..f1a91021e357 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5793,7 +5793,7 @@ void sched_tick(void) unsigned long hw_pressure; u64 resched_latency; - if (!cpu_preferred(cpu)) + if (sched_steal_mon_enabled() && !cpu_preferred(cpu)) sched_push_current_non_preferred_cpu(rq); if (housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE)) @@ -5834,6 +5834,9 @@ void sched_tick(void) rq->idle_balance = idle_cpu(cpu); sched_balance_trigger(rq); } + + if (sched_steal_mon_enabled()) + sched_trigger_steal_computation(cpu); } #ifdef CONFIG_NO_HZ_FULL @@ -11407,4 +11410,27 @@ void sched_steal_detection_work(struct work_struct *work) now = ktime_get(); sm->prev_time = now; } + +void sched_trigger_steal_computation(int cpu) +{ + int first_hk_cpu = cpumask_first_and(housekeeping_cpumask(HK_TYPE_KERNEL_NOISE), + cpu_active_mask); + ktime_t now; + + /* Done by first active housekeeping CPU only */ + if (likely(cpu != first_hk_cpu)) + return; + + /* + * Since everything is updated by first housekeeping CPU, + * There is no need for complex syncronization. + */ + now = ktime_get(); + + /* Default is once per second */ + if (likely(ktime_ms_delta(now, steal_mon.prev_time) < steal_mon.sampling_period_ms)) + return; + + schedule_work_on(first_hk_cpu, &steal_mon.work); +} #endif diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 2d62858f9cc0..55b8beb42574 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -649,6 +649,7 @@ static ssize_t sched_sm_en_write(struct file *filp, const char __user *ubuf, static_branch_enable(&__sched_sm_enable); } else if (!sched_sm_wr_enable && orig) { static_branch_disable(&__sched_sm_enable); + cancel_work_sync(&steal_mon.work); cpumask_copy(&__cpu_preferred_mask, cpu_active_mask); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3f3c7f0ca489..b02a414ffaae 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13292,7 +13292,8 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, cpumask_and(cpus, sched_domain_span(sd), cpu_active_mask); /* Spread load among preferred CPUs */ - cpumask_and(cpus, cpus, cpu_preferred_mask); + if (sched_steal_mon_enabled()) + cpumask_and(cpus, cpus, cpu_preferred_mask); schedstat_inc(sd->lb_count[idle]); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 984da3827f19..f3814099cc0b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1060,6 +1060,7 @@ struct root_domain { struct perf_domain __rcu *pd; }; +static inline bool sched_steal_mon_enabled(void); extern void init_defrootdomain(void); extern int sched_init_domains(const struct cpumask *cpu_map); extern void rq_attach_root(struct rq *rq, struct root_domain *rd); @@ -1436,7 +1437,7 @@ static inline bool available_idle_cpu(int cpu) if (!idle_rq(cpu_rq(cpu))) return 0; - if (!cpu_preferred(cpu)) + if (sched_steal_mon_enabled() && !cpu_preferred(cpu)) return 0; if (vcpu_is_preempted(cpu)) @@ -4243,8 +4244,15 @@ DECLARE_STATIC_KEY_FALSE(__sched_sm_enable); void sched_init_steal_monitor(void); void sched_steal_detection_work(struct work_struct *work); void sched_push_current_non_preferred_cpu(struct rq *rq); +void sched_trigger_steal_computation(int cpu); +static inline bool sched_steal_mon_enabled(void) +{ + return static_branch_unlikely(&__sched_sm_enable); +} #else /* !CONFIG_PREFERRED_CPU */ static inline void sched_push_current_non_preferred_cpu(struct rq *rq) { } static inline void sched_init_steal_monitor(void) { } +static inline void sched_trigger_steal_computation(int cpu) { } +static inline bool sched_steal_mon_enabled(void) { return false; } #endif #endif /* _KERNEL_SCHED_SCHED_H */ -- 2.47.3