mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: bugzilla-daemon@bugzilla.kernel.org,
	LKML <linux-kernel@vger.kernel.org>,
	wen.yang99@zte.com.cn, Petr Mladek <pmladek@suse.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Christoph Hellwig <hch@infradead.org>
Subject: Re: [Bug 199003] console stalled, cause Hard LOCKUP.
Date: Thu, 22 Mar 2018 18:25:17 -0400	[thread overview]
Message-ID: <20180322182517.511aad9d@gandalf.local.home> (raw)
In-Reply-To: <20180322021437.GA3181@jagdpanzerIV>

On Thu, 22 Mar 2018 11:14:37 +0900
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> wrote:


> Looking at
>  printk()->call_console_drivers()->serial8250_console_putchar()->wait_for_xmitr()
> 
> ... wait_for_xmitr() can spin for over 1 second waiting for the UART_MSR_CTS
> bit.
> 
> static void wait_for_xmitr(struct uart_8250_port *up, int bits)
> {
> 	unsigned int status, tmout = 10000;
> 
> 	/* Wait up to 10ms for the character(s) to be sent. */
> 	for (;;) {
> 		status = serial_in(up, UART_LSR);
> 
> 		up->lsr_saved_flags |= status & LSR_SAVE_FLAGS;
> 
> 		if ((status & bits) == bits)
> 			break;
> 		if (--tmout == 0)
> 			break;
> 		udelay(1);
> 		touch_nmi_watchdog();
> 	}
> 
> 	/* Wait up to 1s for flow control if necessary */
> 	if (up->port.flags & UPF_CONS_FLOW) {
> 		for (tmout = 1000000; tmout; tmout--) {
> 			unsigned int msr = serial_in(up, UART_MSR);
> 			up->msr_saved_flags |= msr & MSR_SAVE_FLAGS;
> 			if (msr & UART_MSR_CTS)
> 				break;
> 			udelay(1);
> 			touch_nmi_watchdog();
> 		}
> 	}
> 	...
> }
> 
> a 1+ second long busy loop in the console driver is quite close to

Yeah that's nasty but shouldn't ever hit 1 second unless there's
hardware issues.

> "problems guaranteed". But, wait, there is even more. This wait_for_xmitr()
> busy wait is happening after every character we print on the console. So
> printk("foo") will generate 5 * wait_for_xmitr() busy loops [foo + \r + \n].
> They punch&touch watchdog a lot, so at the least the system won't get killed
> by the hardlockup detector. But at the same time, it's still potentially a
> 1+ second busy loop in the console driver * strlen(message).
> 
> Sometimes I really wish we had detached consoles. Direct printk()->console
> is nice and cool, but... we can't have it. I don't want to pressure for
> printk_kthread, but look at all those consoles. There are not so many ways
> out. What do you think?

If anything, perhaps we could have a printk thread that is triggered to
wake up if any print is taking a long time, or if there's a lot of
prints happening. And then it could try to print anything in the
buffer. Just the mere printk() in the thread should do the hand off.

I wonder how bad it would be to wake the printk thread whenever a
printk() is executed. Printks really shouldn't be that common.

-- Steve

  parent reply	other threads:[~2018-03-22 22:25 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <bug-199003-8243@https.bugzilla.kernel.org/>
     [not found] ` <bug-199003-8243-Txkdkdv5Cm@https.bugzilla.kernel.org/>
2018-03-21 13:44   ` Steven Rostedt
2018-03-22  2:14     ` Sergey Senozhatsky
2018-03-22  2:34       ` Sergey Senozhatsky
2018-03-22 22:25       ` Steven Rostedt [this message]
2018-03-23 12:06         ` Sergey Senozhatsky
2018-03-23 13:16           ` Petr Mladek
2018-03-26  5:12             ` Sergey Senozhatsky
2018-03-26  9:26               ` Petr Mladek
     [not found] <bug-199003-14532@https.bugzilla.kernel.org/>
     [not found] ` <bug-199003-14532-lzL5ySZS5x@https.bugzilla.kernel.org/>
2018-03-05  9:27   ` Sergey Senozhatsky
     [not found] ` <bug-199003-14532-4aWzZJgGHz@https.bugzilla.kernel.org/>
2018-03-21  7:28   ` Sergey Senozhatsky
     [not found] ` <bug-199003-14532-DgJdGAHEOE@https.bugzilla.kernel.org/>
2018-03-26  5:18   ` Sergey Senozhatsky
     [not found] ` <bug-199003-14532-IBTfRpvy7t@https.bugzilla.kernel.org/>
2018-03-27 10:37   ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180322182517.511aad9d@gandalf.local.home \
    --to=rostedt@goodmis.org \
    --cc=akpm@linux-foundation.org \
    --cc=bugzilla-daemon@bugzilla.kernel.org \
    --cc=hch@infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=pmladek@suse.com \
    --cc=sergey.senozhatsky.work@gmail.com \
    --cc=wen.yang99@zte.com.cn \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®