mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Pavel Machek <pavel@ucw.cz>
Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Petr Mladek <pmladek@suse.com>, Jan Kara <jack@suse.cz>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Peter Zijlstra <peterz@infradead.org>,
	"Rafael J . Wysocki" <rjw@rjwysocki.net>,
	Eric Biederman <ebiederm@xmission.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Jiri Slaby <jslaby@suse.com>, Len Brown <len.brown@intel.com>,
	linux-kernel@vger.kernel.org,
	Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Subject: Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
Date: Fri, 7 Apr 2017 14:12:44 +0900	[thread overview]
Message-ID: <20170407051244.GB487@jagdpanzerIV.localdomain> (raw)
In-Reply-To: <20170406171430.GB10363@amd>

Hello,

On (04/06/17 19:14), Pavel Machek wrote:
[..]
> > @@ -1765,17 +1803,40 @@ asmlinkage int vprintk_emit(int facility, int level,
> >  
> >  	printed_len += log_output(facility, level, lflags, dict, dictlen, text, text_len);
> >  
> > +	/*
> > +	 * Emergency level indicates that the system is unstable and, thus,
> > +	 * we better stop relying on wake_up(printk_kthread) and try to do
> > +	 * a direct printing.
> > +	 */
> > +	if (level == LOGLEVEL_EMERG)
> > +		printk_kthread_disabled = true;
> > +
> > +	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
> 
> Messages lower then _EMERG may be important, too.. and usually are,
> for debugging.
> 
> And you keep both code paths, anyway, so they have to work. So you did
> not really "fix" issues you are pointing out -- they still remain
> there for _EMERG and above.

we don't drop messages of lower levels. we just print then from a
schedulable context. once the things go off the rails, and EMERG
is a good hint, I think, we stop being optimismitcs and switch to
a "best effort" mode. that is sort of reasonable. if there is a
flood of EMERG messages that are not actually important and,
basically, are result of a coding error, then, I think, the we
must fix that coding error. I mean, there should be some common
sense, and doing
		while (1)
			printk(KERN_EMERG "hello\n");
is probably not.


> I agree that printing too much is a problem. Could you just print
> "(messages delayed)" in that case, then wake a kernel thread to [rint
> the rest?

sorry, but what difference would it make?

it's really unclear at what point we should offload printing if we begin
that "we will offload sometimes". for example, I've seen many spin-lock
lockups where printk was involved.

	CPU0					CPU1		CPU2					CPU3

	spin_lock(&lock)					spin_lock(&lock)			spin_lock(&lock)
	printk("foo") // grabs the console_sem
	printk("bar")				printk("a")
						printk("b")
						printk("c")
						...
						printk("z")
								spin_dump()				spin_dump()
	  call_console_drivers()				 printk()				 printk()
	   serial_driver_write()				 printk()				 printk()
	    spin_lock_irqsave(port->lock)			 ...					 ...
	     uart_console_write(...)				 trigger_all_cpu_backtrace()		 trigger_all_cpu_backtrace()
	      serial_driver_putchar()
	       while (!txrdy(...))
	         cpu_relax()


spin_dump() and trigger_all_cpu_backtrace() result in a bunch of
additional printk()-s so CPU0 has even more job to do in console_unlock(),
while it still holds the contended spin_lock. and so on; there are
many other examples.

so should we declare a "we can spend only 2 seconds in direct printk()
and then must offload printing" rule? I don't think it's much better
than a simpler "we always offload, as long as we think it's safe".

	-ss

  reply	other threads:[~2017-04-07  5:13 UTC|newest]

Thread overview: 75+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-03-29  9:25 [RFC][PATCHv2 0/8] " Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu Sergey Senozhatsky
2017-03-31 13:09   ` Petr Mladek
2017-03-31 13:33     ` Peter Zijlstra
2017-04-03 11:23       ` Sergey Senozhatsky
2017-04-03 12:43         ` Petr Mladek
2017-03-29  9:25 ` [RFC][PATCHv2 2/8] printk: introduce printing kernel thread Sergey Senozhatsky
2017-04-04  9:01   ` Petr Mladek
2017-04-04  9:36     ` Sergey Senozhatsky
2017-04-06 17:14   ` Pavel Machek
2017-04-07  5:12     ` Sergey Senozhatsky [this message]
2017-04-07  7:21       ` Pavel Machek
2017-04-07  8:15         ` Sergey Senozhatsky
2017-04-07 12:06           ` Pavel Machek
2017-03-29  9:25 ` [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func() Sergey Senozhatsky
2017-03-31 14:56   ` Petr Mladek
2017-04-04 16:15     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places Sergey Senozhatsky
2017-03-31 15:06   ` Petr Mladek
2017-04-06 17:20   ` Pavel Machek
2017-04-09 10:59     ` Andreas Mohr
2017-04-10 12:20       ` Petr Mladek
2017-04-10 14:38         ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 5/8] sysrq: " Sergey Senozhatsky
2017-03-31 15:37   ` Petr Mladek
2017-04-01  0:04     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 6/8] kexec: " Sergey Senozhatsky
2017-03-31 15:39   ` Petr Mladek
2017-03-29  9:25 ` [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter Sergey Senozhatsky
2017-04-03 15:29   ` Petr Mladek
2017-04-04  8:29     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 8/8] printk: enable printk offloading Sergey Senozhatsky
2017-03-30 21:38   ` [printk] fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage kernel test robot
2017-03-31  2:35     ` Sergey Senozhatsky
2017-03-31  4:04       ` Sergey Senozhatsky
2017-03-31  6:39         ` Ye Xiaolong
2017-03-31 14:47           ` Sergey Senozhatsky
2017-03-31 15:28             ` Eric W. Biederman
2017-04-03  9:31               ` Jan Kara
2017-04-03 10:06                 ` Petr Mladek
2017-04-06 17:33                 ` Pavel Machek
2017-04-07  4:44                   ` Sergey Senozhatsky
2017-04-07  7:15                     ` Pavel Machek
2017-04-07  7:46                       ` Sergey Senozhatsky
2017-04-07  8:14                         ` Pavel Machek
2017-04-07 12:10                           ` Sergey Senozhatsky
2017-04-07 12:44                             ` Pavel Machek
2017-04-07 14:40                               ` Steven Rostedt
2017-05-08  6:37                                 ` Sergey Senozhatsky
2017-05-17 13:13                                   ` Petr Mladek
2017-04-07 15:13                               ` Sergey Senozhatsky
2017-04-07 15:23                                 ` Peter Zijlstra
2017-04-07 15:40                                   ` Sergey Senozhatsky
2017-04-09 18:21                                     ` Eric W. Biederman
2017-04-10  4:46                                       ` Sergey Senozhatsky
2017-04-09 10:12                                 ` Pavel Machek
2017-04-10  4:53                                   ` Sergey Senozhatsky
2017-04-10 11:54                                     ` Petr Mladek
2017-04-10 15:08                                       ` Sergey Senozhatsky
2017-04-10 18:48                                     ` Pavel Machek
2017-04-11  1:46                                       ` Sergey Senozhatsky
2017-04-11 16:19                                         ` Sergey Senozhatsky
2017-04-12 18:43                                           ` Pavel Machek
2017-04-13  4:34                                             ` Sergey Senozhatsky
2017-04-13  5:50                                           ` Sergey Senozhatsky
2017-04-13  8:19                                             ` Sergey Senozhatsky
2017-04-13 14:03                                           ` Petr Mladek
2017-04-14  4:42                                             ` Sergey Senozhatsky
2017-04-07 14:29                           ` Steven Rostedt
2017-04-09  9:57                             ` Pavel Machek
2017-04-03 10:51               ` Sergey Senozhatsky
2017-04-05  7:29           ` Ye Xiaolong
2017-04-05  8:40             ` Sergey Senozhatsky
2017-04-03 15:42   ` [RFC][PATCHv2 8/8] printk: enable printk offloading Petr Mladek
2017-04-04 13:20     ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170407051244.GB487@jagdpanzerIV.localdomain \
    --to=sergey.senozhatsky.work@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=ebiederm@xmission.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jack@suse.cz \
    --cc=jslaby@suse.com \
    --cc=len.brown@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pavel@ucw.cz \
    --cc=peterz@infradead.org \
    --cc=pmladek@suse.com \
    --cc=rjw@rjwysocki.net \
    --cc=rostedt@goodmis.org \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®