From: "Paul E. McKenney" <paulmck@kernel.org>
To: Feng Tang <feng.tang@linux.alibaba.com>
Cc: Petr Mladek <pmladek@suse.com>, Lance Yang <lance.yang@linux.dev>,
Andrew Morton <akpm@linux-foundation.org>,
Steven Rostedt <rostedt@goodmis.org>,
linux-kernel@vger.kernel.org, mhiramat@kernel.org,
llong@redhat.com, John Ogness <john.ogness@linutronix.de>,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Tomasz Figa <tfiga@chromium.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Mel Gorman <mgorman@suse.de>,
Thomas Gleixner <tglx@linutronix.de>,
Michal Hocko <mhocko@suse.com>, Tejun Heo <tj@kernel.org>,
Douglas Anderson <dianders@chromium.org>
Subject: Re: [PATCH v1 0/3] generalize panic_print's dump function to be used by other kernel parts
Date: Wed, 14 May 2025 06:55:45 -0700 [thread overview]
Message-ID: <4b7d9ea8-330f-486e-957b-aadbf366e3fe@paulmck-laptop> (raw)
In-Reply-To: <aCQPA7-5uSSlPAV-@U-2FWC9VHC-2323.local>
On Wed, May 14, 2025 at 11:33:23AM +0800, Feng Tang wrote:
> On Tue, May 13, 2025 at 10:09:51AM -0700, Paul E. McKenney wrote:
> > On Tue, May 13, 2025 at 03:27:33PM +0200, Petr Mladek wrote:
> > > On Mon 2025-05-12 16:23:30, Lance Yang wrote:
> > > >
> > > >
> > > > On 2025/5/12 11:14, Feng Tang wrote:
> > > > > Hi Andrew,
> > > > >
> > > > > Thanks for the review!
> > > > >
> > > > > On Sun, May 11, 2025 at 06:46:17PM -0700, Andrew Morton wrote:
> > > > > > On Sun, 11 May 2025 16:52:51 +0800 Feng Tang <feng.tang@linux.alibaba.com> wrote:
> > > > > >
> > > > > > > When working on kernel stability issues, panic, task-hung and
> > > > > > > software/hardware lockup are frequently met. And to debug them, user
> > > > > > > may need lots of system information at that time, like task call stacks,
> > > > > > > lock info, memory info etc.
> > > > > > >
> > > > > > > panic case already has panic_print_sys_info() for this purpose, and has
> > > > > > > a 'panic_print' bitmask to control what kinds of information is needed,
> > > > > > > which is also helpful to debug other task-hung and lockup cases.
> > > > > > >
> > > > > > > So this patchset extract the function out, and make it usable for other
> > > > > > > cases which also need system info for debugging.
> > > > > > >
> > > > > > > Locally these have been used in our bug chasing for stablility issues
> > > > > > > and was helpful.
> > > > > >
> > > > > > Truth. Our responses to panics, oopses, WARNs, BUGs, OOMs etc seem
> > > > > > quite poorly organized. Some effort to clean up (and document!) all of
> > > > > > this sounds good.
> > > > > >
> > > > > > My vote is to permit the display of every scrap of information we can
> > > > > > think of in all situations. And then to permit users to select which of
> > > > > > that information is to be displayed under each situation.
> > > >
> > > > Completely agreed. The tricky part is making a global knob that works for
> > > > all situations without breaking userspace, but it's a better system-wide
> > > > approach ;)
> > > >
> > > > >
> > > > > Good point! Maybe one future todo is to add a gloabl system info dump
> > > > > function with ONE global knob for selecting different kinds of information,
> > > > > which could be embedded into some cases you mentioned above.
> > > >
> > > > IMHO, for features with their own knobs, we need:
> > > > a) The global knob (if enabled) turns on all related feature-level knobs,
> > > > b) while still allowing users to manually override individual knobs.
> > > >
> > > > Something like:
> > > >
> > > > If SYS_PRINT_ALL_CPU_BT (global knob) is on, it enables
> > > > hung_task_all_cpu_backtrace
> > > > for hung-task situation automatically. But users can still disable it via
> > > > hung_task_all_cpu_backtrace.
> > >
> > > I am all for unifying the options for printing debug information
> > > in various emergency situations. I am just not sure whether we really
> > > want to do the same in all situations.
> > >
> > > Some lockup detectors tries to be more clever, for example:
> > >
> > > + RCU stall detector prints backtraces only from CPUs which are
> > > involved in the stall, see print_other_cpu_stall().
> > >
> > > + Workqueues watchdog shows backtraces from tasks which are
> > > preventing forward progress, see show_cpu_pool_hog().
> > >
> > > And stalls are about scheduling (disabled preemption, disabled IRQ,
> > > deadlocks, too long uninterruptible sleep). OOM is about memory
> > > usage. Oops is about an invalid memory access. WARNs() are
> > > completely random stuff.
> > >
> > > Also I am afraid of printing too much information when the system
> > > is supposed to continue running. It would make sense to print it in
> > > nbcon_cpu_emergency_enter()/exit() context which disables
> > > preemption. And it might cause softlockups on its own.
> >
> > And we did do some of the cleverness that Petr points out because of
> > problems caused by flooding the console log. We first ran into this
> > sort of thing on embedded systems with slow serial consoles (where 115K
> > baud is now way slow), but it also shows up in other environments, for
> > example, those committing large numbers of console logs to stable storage,
> > multiplexing large numbers of logs across networks that sometimes get
> > congested, and so on.
> >
> > So I second the call for individual knobs, either in addition to or
> > instead of the global knob.
>
> Thanks for the detail elaboration! RCU stall case is also a main target
> in the stability issues I have worked on, besides panic/taskhung/lockup.
> I noticed it has its own mature handling, and dare not to touch it in
> this patchset :)
Uhhh...
Please do dare to touch it, as that is the only way that it can possibly
improve. Just please also be very careful *how* you touch it. ;-)
Thanx, Paul
> Thanks,
> Feng
>
> >
> > > Finally, I wonder whether ftrace_dump() might cause a livelock when ftrace
> > > is adding new messages in parallel.
> >
> > It definitely can cause problems, and me learning this the hard way is
> > why rcutorture calls tracing_off() before calling ftrace_dump().
> >
> > > The situation is much easier during panic() because the system is
> > > going to die() anyway, non-panic CPUs are stopped, ...
> > >
> > > That said, I could understand that people might want to see as much
> > > information as possible when the console is fast and the range of
> > > possible problems is big.
> >
> > No argument here.
> >
> > Thanx, Paul
> >
> > > Anyway, I have added few more people into Cc who are interested into
> > > the various watchdogs.
> > >
> > > And there is parallel initiative which tries to unify the loglevel or
> > > somehow make the filtering easier, see
> > > https://lore.kernel.org/r/20250424070436.2380215-1-senozhatsky@chromium.org
> > >
> > > Best Regards,
> > > Petr
next prev parent reply other threads:[~2025-05-14 13:55 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-11 8:52 Feng Tang
2025-05-11 8:52 ` [PATCH v1 1/3] kernel/panic: generalize panic_print's function to show sys info Feng Tang
2025-05-13 9:57 ` Petr Mladek
2025-05-13 13:23 ` Feng Tang
2025-05-15 10:32 ` Petr Mladek
2025-05-15 11:28 ` Lance Yang
2025-05-16 5:38 ` Feng Tang
2025-05-16 13:39 ` Lance Yang
2025-05-21 5:31 ` Feng Tang
2025-06-10 15:44 ` Petr Mladek
2025-06-11 13:43 ` Feng Tang
2025-05-11 8:52 ` [PATCH v1 2/3] kernel/hung_task: add option to dump system info when hung task detected Feng Tang
2025-05-11 8:52 ` [PATCH v1 3/3] kernel/watchdog: add option to dump system info when system is locked up Feng Tang
2025-05-12 1:46 ` [PATCH v1 0/3] generalize panic_print's dump function to be used by other kernel parts Andrew Morton
2025-05-12 3:14 ` Feng Tang
2025-05-12 8:23 ` Lance Yang
2025-05-12 9:31 ` Feng Tang
2025-05-13 13:27 ` Petr Mladek
2025-05-13 17:09 ` Paul E. McKenney
2025-05-14 3:33 ` Feng Tang
2025-05-14 13:55 ` Paul E. McKenney [this message]
2025-05-14 3:22 ` Feng Tang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4b7d9ea8-330f-486e-957b-aadbf366e3fe@paulmck-laptop \
--to=paulmck@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=dianders@chromium.org \
--cc=feng.tang@linux.alibaba.com \
--cc=john.ogness@linutronix.de \
--cc=lance.yang@linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=llong@redhat.com \
--cc=mgorman@suse.de \
--cc=mhiramat@kernel.org \
--cc=mhocko@suse.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=senozhatsky@chromium.org \
--cc=tfiga@chromium.org \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®