From: Steven Rostedt <rostedt@goodmis.org>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: bugzilla-daemon@bugzilla.kernel.org,
LKML <linux-kernel@vger.kernel.org>,
wen.yang99@zte.com.cn, Petr Mladek <pmladek@suse.com>,
Peter Zijlstra <peterz@infradead.org>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Hellwig <hch@infradead.org>
Subject: Re: [Bug 199003] console stalled, cause Hard LOCKUP.
Date: Thu, 22 Mar 2018 18:25:17 -0400 [thread overview]
Message-ID: <20180322182517.511aad9d@gandalf.local.home> (raw)
In-Reply-To: <20180322021437.GA3181@jagdpanzerIV>
On Thu, 22 Mar 2018 11:14:37 +0900
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> wrote:
> Looking at
> printk()->call_console_drivers()->serial8250_console_putchar()->wait_for_xmitr()
>
> ... wait_for_xmitr() can spin for over 1 second waiting for the UART_MSR_CTS
> bit.
>
> static void wait_for_xmitr(struct uart_8250_port *up, int bits)
> {
> unsigned int status, tmout = 10000;
>
> /* Wait up to 10ms for the character(s) to be sent. */
> for (;;) {
> status = serial_in(up, UART_LSR);
>
> up->lsr_saved_flags |= status & LSR_SAVE_FLAGS;
>
> if ((status & bits) == bits)
> break;
> if (--tmout == 0)
> break;
> udelay(1);
> touch_nmi_watchdog();
> }
>
> /* Wait up to 1s for flow control if necessary */
> if (up->port.flags & UPF_CONS_FLOW) {
> for (tmout = 1000000; tmout; tmout--) {
> unsigned int msr = serial_in(up, UART_MSR);
> up->msr_saved_flags |= msr & MSR_SAVE_FLAGS;
> if (msr & UART_MSR_CTS)
> break;
> udelay(1);
> touch_nmi_watchdog();
> }
> }
> ...
> }
>
> a 1+ second long busy loop in the console driver is quite close to
Yeah that's nasty but shouldn't ever hit 1 second unless there's
hardware issues.
> "problems guaranteed". But, wait, there is even more. This wait_for_xmitr()
> busy wait is happening after every character we print on the console. So
> printk("foo") will generate 5 * wait_for_xmitr() busy loops [foo + \r + \n].
> They punch&touch watchdog a lot, so at the least the system won't get killed
> by the hardlockup detector. But at the same time, it's still potentially a
> 1+ second busy loop in the console driver * strlen(message).
>
> Sometimes I really wish we had detached consoles. Direct printk()->console
> is nice and cool, but... we can't have it. I don't want to pressure for
> printk_kthread, but look at all those consoles. There are not so many ways
> out. What do you think?
If anything, perhaps we could have a printk thread that is triggered to
wake up if any print is taking a long time, or if there's a lot of
prints happening. And then it could try to print anything in the
buffer. Just the mere printk() in the thread should do the hand off.
I wonder how bad it would be to wake the printk thread whenever a
printk() is executed. Printks really shouldn't be that common.
-- Steve
next prev parent reply other threads:[~2018-03-22 22:25 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <bug-199003-8243@https.bugzilla.kernel.org/>
[not found] ` <bug-199003-8243-Txkdkdv5Cm@https.bugzilla.kernel.org/>
2018-03-21 13:44 ` Steven Rostedt
2018-03-22 2:14 ` Sergey Senozhatsky
2018-03-22 2:34 ` Sergey Senozhatsky
2018-03-22 22:25 ` Steven Rostedt [this message]
2018-03-23 12:06 ` Sergey Senozhatsky
2018-03-23 13:16 ` Petr Mladek
2018-03-26 5:12 ` Sergey Senozhatsky
2018-03-26 9:26 ` Petr Mladek
[not found] <bug-199003-14532@https.bugzilla.kernel.org/>
[not found] ` <bug-199003-14532-lzL5ySZS5x@https.bugzilla.kernel.org/>
2018-03-05 9:27 ` Sergey Senozhatsky
[not found] ` <bug-199003-14532-4aWzZJgGHz@https.bugzilla.kernel.org/>
2018-03-21 7:28 ` Sergey Senozhatsky
[not found] ` <bug-199003-14532-DgJdGAHEOE@https.bugzilla.kernel.org/>
2018-03-26 5:18 ` Sergey Senozhatsky
[not found] ` <bug-199003-14532-IBTfRpvy7t@https.bugzilla.kernel.org/>
2018-03-27 10:37 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180322182517.511aad9d@gandalf.local.home \
--to=rostedt@goodmis.org \
--cc=akpm@linux-foundation.org \
--cc=bugzilla-daemon@bugzilla.kernel.org \
--cc=hch@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=wen.yang99@zte.com.cn \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®