mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: Clark Williams <williams@redhat.com>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
	LKML <linux-kernel@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@kernel.org>,
	Daniel Bristot de Oliveira <bristot@redhat.com>,
	John Kacur <jkacur@redhat.com>, Scott Wood <swood@redhat.com>,
	Mike Galbraith <efault@gmx.de>
Subject: Re: [PATCH RT] fix IPI balancing for 4.14-rt
Date: Tue, 21 Nov 2017 10:24:36 -0500	[thread overview]
Message-ID: <20171121102436.3e25655f@gandalf.local.home> (raw)
In-Reply-To: <20171121091425.7577dc02@tagon>

On Tue, 21 Nov 2017 09:14:25 -0600
Clark Williams <williams@redhat.com> wrote:

> From 8ea8311b75a40bdea03e7f8228a0578b6367e9d1 Mon Sep 17 00:00:00 2001
> From: Clark Williams <williams@redhat.com>
> Date: Mon, 20 Nov 2017 14:26:12 -0600
> Subject: [PATCH] [rt] sched/rt: fix panic in double_lock_balance with
>  simplified IPI RT balancing
> 
> I was testing 4.14-rt1 on a large system (cores == 96) and saw that
> we were getting into an rt balancing storm, so I tried applying Steven's
> patch (not upstream yet):
> 
>     sched/rt: Simplify the IPI rt balancing logic
> 
> Booting the resulting kernel yielded a panic in
> double_lock_balance() due to irqs not being disabled.
> 
> This patch changes the calls to raw_spin_{lock,unlock} in the
> function rto_push_irq_work_function, to be raw_spin_{lock,unlock}_irq.
> Not sure if that's too heavy a hammer, but the resulting kernel boots
> and runs and survives 12h runs of rteval. Once Steven's patch goes in 
> upstream, we'll need something like this in RT.
> 
> Signed-off-by: Clark Williams <williams@redhat.com>
> ---
>  kernel/sched/rt.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
> index 57fb251dd8ce..a5cd0cea2f0f 100644
> --- a/kernel/sched/rt.c
> +++ b/kernel/sched/rt.c
> @@ -2008,9 +2008,9 @@ void rto_push_irq_work_func(struct irq_work *work)
>  	 * When it gets updated, a check is made if a push is possible.
>  	 */
>  	if (has_pushable_tasks(rq)) {
> -		raw_spin_lock(&rq->lock);
> +		raw_spin_lock_irq(&rq->lock);
>  		push_rt_tasks(rq);
> -		raw_spin_unlock(&rq->lock);
> +		raw_spin_unlock_irq(&rq->lock);

This looks buggy to me. You know you just indiscriminately enabled
interrupts here.
 
>  	}
>  
>  	raw_spin_lock(&rq->rd->rto_lock);

Why is this patch necessary?

Is it because you have the irq_work running in non hard irq context? I
think you need something like this instead (if you haven't already
added it):

-- Steve

Index: linux-rt.git/kernel/sched/topology.c
===================================================================
--- linux-rt.git.orig/kernel/sched/topology.c
+++ linux-rt.git/kernel/sched/topology.c
@@ -257,6 +257,7 @@ static int init_rootdomain(struct root_d
 	rd->rto_cpu = -1;
 	raw_spin_lock_init(&rd->rto_lock);
 	init_irq_work(&rd->rto_push_work, rto_push_irq_work_func);
+	rd->rto_push_work.flags |= IRQ_WORK_HARD_IRQ;
 #endif
 
 	init_dl_bw(&rd->dl_bw);

  reply	other threads:[~2017-11-21 15:24 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-10-06 18:05 [PATCH tip/sched/core v3] sched/rt: Simplify the IPI rt balancing logic Steven Rostedt
2017-11-21 15:14 ` [PATCH RT] fix IPI balancing for 4.14-rt Clark Williams
2017-11-21 15:24   ` Steven Rostedt [this message]
2017-11-23 17:25     ` Sebastian Andrzej Siewior
2017-11-27 22:25       ` Steven Rostedt
2017-11-28  8:31         ` Sebastian Andrzej Siewior
2017-11-28 15:32           ` Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20171121102436.3e25655f@gandalf.local.home \
    --to=rostedt@goodmis.org \
    --cc=bigeasy@linutronix.de \
    --cc=bristot@redhat.com \
    --cc=efault@gmx.de \
    --cc=jkacur@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=swood@redhat.com \
    --cc=tglx@linutronix.de \
    --cc=williams@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®