mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH][V2] sched/rt: Print curr when RT throttling activated
@ 2023-05-16 12:22 Alexandru Ardelean
  2023-07-13 20:07 ` Alexandru Ardelean
  0 siblings, 1 reply; 5+ messages in thread
From: Alexandru Ardelean @ 2023-05-16 12:22 UTC (permalink / raw)
  To: linux-kernel
  Cc: mingo, peterz, juri.lelli, vincent.guittot, dietmar.eggemann,
	rostedt, bsegall, mgorman, bristot, vschneid, tian.xianting,
	steffen.aschbacher, Alexandru Ardelean

From: Xianting Tian <tian.xianting@h3c.com>

We may meet the issue, that one RT thread occupied the cpu by 950ms/1s,
The RT thread maybe is a business thread or other unknown thread.

Currently, it only outputs the print "sched: RT throttling activated"
when RT throttling happen. It is hard to know what is the RT thread,
For further analysis, we need add more prints.

This patch is to print current RT task when RT throttling activated,
It help us to know what is the RT thread in the first time.

Tested-by: Alexandru Ardelean <alex@shruggie.ro>
Signed-off-by: Xianting Tian <tian.xianting@h3c.com>
---

Initial patch submission:
  https://lore.kernel.org/all/f3265adc26d4416dacf157f61fa60ad6@h3c.com/T/

We've been having some issues of our own with some applications + some RT
configuration == some threads endded up taking too much CPU time.
This patch came in quite in handy to see in logs which thread is more
problematic.

We've applied this patch onto a 5.10.116 tree. It did need a bit of
re-applying since some context has changed since the initial version (hence
the V2 tag).
Since 5.10.116 (where we used it), it applied nicely to the latest/current
linux-next tree (hence the Tested-by tag).

It would be nice to apply this to the mainline kernel and have this handy.

 kernel/sched/rt.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 00e0e5074115..44b161e42733 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -995,7 +995,7 @@ static inline int rt_se_prio(struct sched_rt_entity *rt_se)
 	return rt_task_of(rt_se)->prio;
 }
 
-static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)
+static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq, struct task_struct *curr)
 {
 	u64 runtime = sched_rt_runtime(rt_rq);
 
@@ -1019,7 +1019,8 @@ static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)
 		 */
 		if (likely(rt_b->rt_runtime)) {
 			rt_rq->rt_throttled = 1;
-			printk_deferred_once("sched: RT throttling activated\n");
+			printk_deferred_once("sched: RT throttling activated (curr: pid %d, comm %s)\n",
+						curr->pid, curr->comm);
 		} else {
 			/*
 			 * In case we did anyway, make it go away,
@@ -1074,7 +1075,7 @@ static void update_curr_rt(struct rq *rq)
 		if (sched_rt_runtime(rt_rq) != RUNTIME_INF) {
 			raw_spin_lock(&rt_rq->rt_runtime_lock);
 			rt_rq->rt_time += delta_exec;
-			exceeded = sched_rt_runtime_exceeded(rt_rq);
+			exceeded = sched_rt_runtime_exceeded(rt_rq, curr);
 			if (exceeded)
 				resched_curr(rq);
 			raw_spin_unlock(&rt_rq->rt_runtime_lock);
-- 
2.40.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH][V2] sched/rt: Print curr when RT throttling activated
  2023-05-16 12:22 [PATCH][V2] sched/rt: Print curr when RT throttling activated Alexandru Ardelean
@ 2023-07-13 20:07 ` Alexandru Ardelean
  2024-04-01 15:47   ` Alexandru Ardelean
  0 siblings, 1 reply; 5+ messages in thread
From: Alexandru Ardelean @ 2023-07-13 20:07 UTC (permalink / raw)
  To: linux-kernel
  Cc: mingo, peterz, juri.lelli, vincent.guittot, dietmar.eggemann,
	rostedt, bsegall, mgorman, bristot, vschneid, tian.xianting,
	steffen.aschbacher, gregkh

On Tue, May 16, 2023 at 3:22 PM Alexandru Ardelean <alex@shruggie.ro> wrote:
>
> From: Xianting Tian <tian.xianting@h3c.com>
>
> We may meet the issue, that one RT thread occupied the cpu by 950ms/1s,
> The RT thread maybe is a business thread or other unknown thread.
>
> Currently, it only outputs the print "sched: RT throttling activated"
> when RT throttling happen. It is hard to know what is the RT thread,
> For further analysis, we need add more prints.
>
> This patch is to print current RT task when RT throttling activated,
> It help us to know what is the RT thread in the first time.
>

Adding Greg on this patch, since it 's been a while.
And also: ping :)

> Tested-by: Alexandru Ardelean <alex@shruggie.ro>
> Signed-off-by: Xianting Tian <tian.xianting@h3c.com>
> ---
>
> Initial patch submission:
>   https://lore.kernel.org/all/f3265adc26d4416dacf157f61fa60ad6@h3c.com/T/
>
> We've been having some issues of our own with some applications + some RT
> configuration == some threads endded up taking too much CPU time.
> This patch came in quite in handy to see in logs which thread is more
> problematic.
>
> We've applied this patch onto a 5.10.116 tree. It did need a bit of
> re-applying since some context has changed since the initial version (hence
> the V2 tag).
> Since 5.10.116 (where we used it), it applied nicely to the latest/current
> linux-next tree (hence the Tested-by tag).
>
> It would be nice to apply this to the mainline kernel and have this handy.
>
>  kernel/sched/rt.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
> index 00e0e5074115..44b161e42733 100644
> --- a/kernel/sched/rt.c
> +++ b/kernel/sched/rt.c
> @@ -995,7 +995,7 @@ static inline int rt_se_prio(struct sched_rt_entity *rt_se)
>         return rt_task_of(rt_se)->prio;
>  }
>
> -static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)
> +static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq, struct task_struct *curr)
>  {
>         u64 runtime = sched_rt_runtime(rt_rq);
>
> @@ -1019,7 +1019,8 @@ static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)
>                  */
>                 if (likely(rt_b->rt_runtime)) {
>                         rt_rq->rt_throttled = 1;
> -                       printk_deferred_once("sched: RT throttling activated\n");
> +                       printk_deferred_once("sched: RT throttling activated (curr: pid %d, comm %s)\n",
> +                                               curr->pid, curr->comm);
>                 } else {
>                         /*
>                          * In case we did anyway, make it go away,
> @@ -1074,7 +1075,7 @@ static void update_curr_rt(struct rq *rq)
>                 if (sched_rt_runtime(rt_rq) != RUNTIME_INF) {
>                         raw_spin_lock(&rt_rq->rt_runtime_lock);
>                         rt_rq->rt_time += delta_exec;
> -                       exceeded = sched_rt_runtime_exceeded(rt_rq);
> +                       exceeded = sched_rt_runtime_exceeded(rt_rq, curr);
>                         if (exceeded)
>                                 resched_curr(rq);
>                         raw_spin_unlock(&rt_rq->rt_runtime_lock);
> --
> 2.40.1
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH][V2] sched/rt: Print curr when RT throttling activated
  2023-07-13 20:07 ` Alexandru Ardelean
@ 2024-04-01 15:47   ` Alexandru Ardelean
  2024-04-08 15:25     ` Valentin Schneider
  0 siblings, 1 reply; 5+ messages in thread
From: Alexandru Ardelean @ 2024-04-01 15:47 UTC (permalink / raw)
  To: linux-kernel
  Cc: mingo, peterz, juri.lelli, vincent.guittot, dietmar.eggemann,
	rostedt, bsegall, mgorman, bristot, vschneid, tian.xianting,
	steffen.aschbacher, gregkh

On Thu, Jul 13, 2023 at 11:07 PM Alexandru Ardelean <alex@shruggie.ro> wrote:
>
> On Tue, May 16, 2023 at 3:22 PM Alexandru Ardelean <alex@shruggie.ro> wrote:
> >
> > From: Xianting Tian <tian.xianting@h3c.com>
> >
> > We may meet the issue, that one RT thread occupied the cpu by 950ms/1s,
> > The RT thread maybe is a business thread or other unknown thread.
> >
> > Currently, it only outputs the print "sched: RT throttling activated"
> > when RT throttling happen. It is hard to know what is the RT thread,
> > For further analysis, we need add more prints.
> >
> > This patch is to print current RT task when RT throttling activated,
> > It help us to know what is the RT thread in the first time.
> >
>
> Adding Greg on this patch, since it 's been a while.
> And also: ping :)

Ping on this :)

Thanks
Alex

>
> > Tested-by: Alexandru Ardelean <alex@shruggie.ro>
> > Signed-off-by: Xianting Tian <tian.xianting@h3c.com>
> > ---
> >
> > Initial patch submission:
> >   https://lore.kernel.org/all/f3265adc26d4416dacf157f61fa60ad6@h3c.com/T/
> >
> > We've been having some issues of our own with some applications + some RT
> > configuration == some threads endded up taking too much CPU time.
> > This patch came in quite in handy to see in logs which thread is more
> > problematic.
> >
> > We've applied this patch onto a 5.10.116 tree. It did need a bit of
> > re-applying since some context has changed since the initial version (hence
> > the V2 tag).
> > Since 5.10.116 (where we used it), it applied nicely to the latest/current
> > linux-next tree (hence the Tested-by tag).
> >
> > It would be nice to apply this to the mainline kernel and have this handy.
> >
> >  kernel/sched/rt.c | 7 ++++---
> >  1 file changed, 4 insertions(+), 3 deletions(-)
> >
> > diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
> > index 00e0e5074115..44b161e42733 100644
> > --- a/kernel/sched/rt.c
> > +++ b/kernel/sched/rt.c
> > @@ -995,7 +995,7 @@ static inline int rt_se_prio(struct sched_rt_entity *rt_se)
> >         return rt_task_of(rt_se)->prio;
> >  }
> >
> > -static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)
> > +static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq, struct task_struct *curr)
> >  {
> >         u64 runtime = sched_rt_runtime(rt_rq);
> >
> > @@ -1019,7 +1019,8 @@ static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)
> >                  */
> >                 if (likely(rt_b->rt_runtime)) {
> >                         rt_rq->rt_throttled = 1;
> > -                       printk_deferred_once("sched: RT throttling activated\n");
> > +                       printk_deferred_once("sched: RT throttling activated (curr: pid %d, comm %s)\n",
> > +                                               curr->pid, curr->comm);
> >                 } else {
> >                         /*
> >                          * In case we did anyway, make it go away,
> > @@ -1074,7 +1075,7 @@ static void update_curr_rt(struct rq *rq)
> >                 if (sched_rt_runtime(rt_rq) != RUNTIME_INF) {
> >                         raw_spin_lock(&rt_rq->rt_runtime_lock);
> >                         rt_rq->rt_time += delta_exec;
> > -                       exceeded = sched_rt_runtime_exceeded(rt_rq);
> > +                       exceeded = sched_rt_runtime_exceeded(rt_rq, curr);
> >                         if (exceeded)
> >                                 resched_curr(rq);
> >                         raw_spin_unlock(&rt_rq->rt_runtime_lock);
> > --
> > 2.40.1
> >

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH][V2] sched/rt: Print curr when RT throttling activated
  2024-04-01 15:47   ` Alexandru Ardelean
@ 2024-04-08 15:25     ` Valentin Schneider
  2024-04-17  6:48       ` Alexandru Ardelean
  0 siblings, 1 reply; 5+ messages in thread
From: Valentin Schneider @ 2024-04-08 15:25 UTC (permalink / raw)
  To: Alexandru Ardelean, linux-kernel
  Cc: mingo, peterz, juri.lelli, vincent.guittot, dietmar.eggemann,
	rostedt, bsegall, mgorman, bristot, tian.xianting,
	steffen.aschbacher, gregkh

On 01/04/24 18:47, Alexandru Ardelean wrote:
> On Thu, Jul 13, 2023 at 11:07 PM Alexandru Ardelean <alex@shruggie.ro> wrote:
>>
>> On Tue, May 16, 2023 at 3:22 PM Alexandru Ardelean <alex@shruggie.ro> wrote:
>> >
>> > From: Xianting Tian <tian.xianting@h3c.com>
>> >
>> > We may meet the issue, that one RT thread occupied the cpu by 950ms/1s,
>> > The RT thread maybe is a business thread or other unknown thread.
>> >
>> > Currently, it only outputs the print "sched: RT throttling activated"
>> > when RT throttling happen. It is hard to know what is the RT thread,
>> > For further analysis, we need add more prints.
>> >
>> > This patch is to print current RT task when RT throttling activated,
>> > It help us to know what is the RT thread in the first time.
>> >
>>
>> Adding Greg on this patch, since it 's been a while.
>> And also: ping :)
>
> Ping on this :)
>

AFAIA this has been proposed a few times in the past, the problem is that
printing the current task isn't the right thing to do as it may not be the
one that contributed most of the runtime that lead to throttling.

See https://lore.kernel.org/lkml/20221209163606.53a2370e@gandalf.local.home/


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH][V2] sched/rt: Print curr when RT throttling activated
  2024-04-08 15:25     ` Valentin Schneider
@ 2024-04-17  6:48       ` Alexandru Ardelean
  0 siblings, 0 replies; 5+ messages in thread
From: Alexandru Ardelean @ 2024-04-17  6:48 UTC (permalink / raw)
  To: Valentin Schneider
  Cc: linux-kernel, mingo, peterz, juri.lelli, vincent.guittot,
	dietmar.eggemann, rostedt, bsegall, mgorman, bristot,
	tian.xianting, steffen.aschbacher, gregkh

On Mon, Apr 8, 2024 at 6:25 PM Valentin Schneider <vschneid@redhat.com> wrote:
>
> On 01/04/24 18:47, Alexandru Ardelean wrote:
> > On Thu, Jul 13, 2023 at 11:07 PM Alexandru Ardelean <alex@shruggie.ro> wrote:
> >>
> >> On Tue, May 16, 2023 at 3:22 PM Alexandru Ardelean <alex@shruggie.ro> wrote:
> >> >
> >> > From: Xianting Tian <tian.xianting@h3c.com>
> >> >
> >> > We may meet the issue, that one RT thread occupied the cpu by 950ms/1s,
> >> > The RT thread maybe is a business thread or other unknown thread.
> >> >
> >> > Currently, it only outputs the print "sched: RT throttling activated"
> >> > when RT throttling happen. It is hard to know what is the RT thread,
> >> > For further analysis, we need add more prints.
> >> >
> >> > This patch is to print current RT task when RT throttling activated,
> >> > It help us to know what is the RT thread in the first time.
> >> >
> >>
> >> Adding Greg on this patch, since it 's been a while.
> >> And also: ping :)
> >
> > Ping on this :)
> >
>
> AFAIA this has been proposed a few times in the past, the problem is that
> printing the current task isn't the right thing to do as it may not be the
> one that contributed most of the runtime that lead to throttling.
>
> See https://lore.kernel.org/lkml/20221209163606.53a2370e@gandalf.local.home/

Ah.
Thanks for the input :)
I'll shut up on this.

Thanks
Alex

>

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2024-04-17  6:48 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-16 12:22 [PATCH][V2] sched/rt: Print curr when RT throttling activated Alexandru Ardelean
2023-07-13 20:07 ` Alexandru Ardelean
2024-04-01 15:47   ` Alexandru Ardelean
2024-04-08 15:25     ` Valentin Schneider
2024-04-17  6:48       ` Alexandru Ardelean

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®