From: Steven Rostedt <rostedt@goodmis.org>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: Tejun Heo <tj@kernel.org>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Petr Mladek <pmladek@suse.com>, Jan Kara <jack@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Rafael Wysocki <rjw@rjwysocki.net>, Pavel Machek <pavel@ucw.cz>,
Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
linux-kernel@vger.kernel.org
Subject: Re: [RFC][PATCHv6 00/12] printk: introduce printing kernel thread
Date: Mon, 18 Dec 2017 20:03:51 -0500 [thread overview]
Message-ID: <20171218200351.09164cdd@gandalf.local.home> (raw)
In-Reply-To: <20171219005248.GA8892@jagdpanzerIV>
On Tue, 19 Dec 2017 09:52:48 +0900
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> wrote:
> > The case here, you are talking about a CPU doing console_lock() from a
> > non printk() case. Which is what I was asking about how often this
> > happens.
>
> I'd say often enough. but the point I was trying to make is that we can
> have non-atomic CPUs which can do the print out, instead of "sharing the
> load" between atomic CPUs.
We don't even know if sharing between "atomic" and "non-atomic" is an
issue. Anything that does a printk() in an atomic location, is going to
have latency to begin with.
>
> > As for why there's no handoff. Does the non printk()
> > console_lock/unlock ever happen from a critical location? I don't think
> > it does (but I haven't checked). Then it is the perfect candidate to do
> > all the printing.
>
> that's right. that is the point I was trying making. we can have better
> candidates to do all the printing.
Sure, but we don't even know if we have to. A problem scenario hasn't
come up that wasn't due to the current implementation (which my patch
changes).
> I did tests yesterday, traces are available. I can't conclude that
> the patch fixes the unfairness of printk().
It doesn't fix the "unfairness" it fixes the unboundedness of printk.
That is what has been triggering all the issues from before.
> consider the following case
>
> we have console_lock() from non-atomic context. console_sem owner is
> getting preempted, under console_sem. which is totally possible and
> happens a lot. in the mean time we have OOM, which can print a lot of
> info. by the time console_sem returns back to TASK_RUNNING logbuf
> contains some number of pending messages [lets say 10 seconds worth
> of printing]. console owner goes to console_unlock(). accidentally
> we have printk from IRQ on CPUz. console_owner hands over printing
> duty to CPUz. so now we have to print 10 seconds worth of OOM messages
> from irq.
Yes that can happen. But printk's from irq context is not nice to have
either, and should only happen when things are going wrong to begin
with.
>
>
>
> CPU0 CPU1 ~ CPUx CPUz
>
> console_lock
>
> << preempted >>
>
>
> OOM OOM printouts, lots
> of OOM traces, etc.
>
> OOM end [progress done].
>
> << back to RUNNING >>
>
> console_unlock()
>
> for (;;)
> sets console_owner
> call_console_drivers() IRQ
> printk
> sees console_owner
> sets console_waiter
>
> clears console_owner
> sees console_waier
> handoff
> for (;;) {
> call_console_drivers()
> ??? lockup
> }
> up()
>
> this is how we down() from non-atomic and up() from atomic [if we make
> it to up(). we might end up in NMI panic]. this scenario is totally possible,
The printk buffer needs to be very big, and bad things have to happen
first. This is a theoretical scenario, and I'd like to see it happen in
the real world before we try to fix it. My patch should make printk
behave *MUCH BETTER* than it currently does.
If you are worried about NMI panics, then we could add a touch nmi
within the printk loop.
> isn't it? the optimistic expectation here is that some other printk() from
> non-atomic CPU will jump in and take over printing from atomic CPUz. but I
> don't see why we are counting on it.
I don't see why we even care. Placing a printk in an atomic context is
a problem to begin with, and should only happen if there's issues in
the system.
-- Steve
next prev parent reply other threads:[~2017-12-19 1:03 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-04 13:48 Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 01/12] printk: move printk_pending out of per-cpu Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 02/12] printk: introduce printing kernel thread Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 03/12] printk: consider watchdogs thresholds for offloading Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 04/12] printk: add sync printk_emergency API Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 05/12] printk: enable printk offloading Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 06/12] PM: switch between printk emergency modes Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 07/12] printk: register syscore notifier Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 08/12] printk: force printk_kthread to offload printing Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 09/12] printk: do not cond_resched() when we can offload Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 10/12] printk: move offloading logic to per-cpu Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 11/12] printk: add offloading watchdog API Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 12/12] printk: improve printk offloading mechanism Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 0/4] printk: offloading testing module/trace events Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 1/4] printk/lib: add offloading trace events and test_printk module Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 2/4] printk/lib: simulate slow consoles Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 3/4] printk: add offloading takeover traces Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 4/4] printk: add task name and CPU to console messages Sergey Senozhatsky
2017-12-14 14:27 ` [RFC][PATCHv6 00/12] printk: introduce printing kernel thread Petr Mladek
2017-12-14 14:39 ` Sergey Senozhatsky
2017-12-15 15:55 ` Steven Rostedt
2017-12-14 15:25 ` Tejun Heo
2017-12-14 17:55 ` Steven Rostedt
2017-12-14 18:11 ` Tejun Heo
2017-12-14 18:21 ` Steven Rostedt
2017-12-22 0:09 ` Tejun Heo
2017-12-22 4:19 ` Steven Rostedt
2017-12-28 6:48 ` Sergey Senozhatsky
2017-12-28 10:07 ` Sergey Senozhatsky
2017-12-29 13:59 ` Tetsuo Handa
2017-12-31 1:44 ` Sergey Senozhatsky
2018-01-09 20:06 ` Tejun Heo
2018-01-09 22:08 ` Tetsuo Handa
2018-01-09 22:17 ` Tejun Heo
2018-01-11 11:14 ` Tetsuo Handa
2018-01-09 22:08 ` Steven Rostedt
2018-01-09 22:17 ` Tejun Heo
2018-01-09 22:47 ` Steven Rostedt
2018-01-09 22:53 ` Tejun Heo
2018-01-10 7:18 ` Steven Rostedt
2018-01-10 14:04 ` Tejun Heo
2017-12-15 2:10 ` Sergey Senozhatsky
2017-12-15 3:18 ` Steven Rostedt
2017-12-15 5:06 ` Sergey Senozhatsky
2017-12-15 6:52 ` Sergey Senozhatsky
2017-12-15 15:39 ` Steven Rostedt
2017-12-15 8:31 ` Petr Mladek
2017-12-15 8:42 ` Sergey Senozhatsky
2017-12-15 9:08 ` Petr Mladek
2017-12-15 15:47 ` Steven Rostedt
2017-12-18 9:36 ` Sergey Senozhatsky
2017-12-18 10:36 ` Sergey Senozhatsky
2017-12-18 12:35 ` Sergey Senozhatsky
2017-12-18 13:51 ` Petr Mladek
2017-12-18 13:31 ` Petr Mladek
2017-12-18 13:39 ` Sergey Senozhatsky
2017-12-18 14:13 ` Petr Mladek
2017-12-18 17:46 ` Steven Rostedt
2017-12-19 1:03 ` Sergey Senozhatsky
2017-12-19 1:08 ` Steven Rostedt
2017-12-19 1:24 ` Sergey Senozhatsky
2017-12-19 2:03 ` Steven Rostedt
2017-12-19 2:46 ` Sergey Senozhatsky
2017-12-19 3:38 ` Steven Rostedt
2017-12-19 4:58 ` Sergey Senozhatsky
2017-12-19 14:40 ` Steven Rostedt
2017-12-20 7:46 ` Sergey Senozhatsky
2017-12-19 14:31 ` Michal Hocko
2017-12-20 7:10 ` Sergey Senozhatsky
2017-12-20 12:06 ` Tetsuo Handa
2017-12-21 6:52 ` Sergey Senozhatsky
2017-12-19 4:36 ` Sergey Senozhatsky
2017-12-18 14:10 ` Petr Mladek
2017-12-19 1:09 ` Sergey Senozhatsky
2017-12-15 15:42 ` Steven Rostedt
2017-12-15 15:19 ` Steven Rostedt
2017-12-19 0:52 ` Sergey Senozhatsky
2017-12-19 1:03 ` Steven Rostedt [this message]
2018-01-05 2:54 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171218200351.09164cdd@gandalf.local.home \
--to=rostedt@goodmis.org \
--cc=akpm@linux-foundation.org \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=pavel@ucw.cz \
--cc=penguin-kernel@I-love.SAKURA.ne.jp \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rjw@rjwysocki.net \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=sergey.senozhatsky@gmail.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®