Using tracing_off() in __schedule_bug()

Using tracing_off() in __schedule_bug()

[Chase Douglas]

We're considering using tracing_off() in __schedule_bug() in the
Ubuntu kernel to help us solve "scheduling while atomic" bugs without
needing to compile and install a special kernel to do so. Using the
tracing_off() call enables us to generate smaller function traces when
debugging, and find the bug easier since it should be at the end of
any traces. The patch we are thinking of using can be found at [1].
You can find an overview of how bugs are solved using this approach
there as well.

Are there any thoughts as to the appropriateness of this patch in
Ubuntu kernels? My guess is that this isn't done in the upstream
vanilla kernels because it would be annoying to have a stock kernel
littered with tracing_off() calls if you hit one when you weren't
meaning to. However, we're interested in hearing if there's some
negative consequence we haven't thought of.

Thanks,
Chase Douglas

[1] http://lkml.org/lkml/2009/4/10/310

Re: Using tracing_off() in __schedule_bug()

[Steven Rostedt]

On Fri, 2010-03-12 at 10:32 -0500, Chase Douglas wrote:
> We're considering using tracing_off() in __schedule_bug() in the
> Ubuntu kernel to help us solve "scheduling while atomic" bugs without
> needing to compile and install a special kernel to do so. Using the
> tracing_off() call enables us to generate smaller function traces when
> debugging, and find the bug easier since it should be at the end of
> any traces. The patch we are thinking of using can be found at [1].
> You can find an overview of how bugs are solved using this approach
> there as well.
> 
> Are there any thoughts as to the appropriateness of this patch in
> Ubuntu kernels? My guess is that this isn't done in the upstream
> vanilla kernels because it would be annoying to have a stock kernel
> littered with tracing_off() calls if you hit one when you weren't
> meaning to. However, we're interested in hearing if there's some
> negative consequence we haven't thought of.

Actually, it has been on my TODO list to litter tracing_off() all over
the kernel ;-)

No, by all means, I would definitely take a patch that turn off tracing
at critical areas. One of which I would like to do is with the BUG()
call.

Hmm, thinking about it more, I would rather have a separate function,
that would call tracing_off() if some variable is set. By default it
would be set, but in case you want to keep tracing after a bug is hit,
you can have a way to disable it.

I need to write up a patch soon. Thanks for bring this up.

-- Steve

Re: Using tracing_off() in __schedule_bug()

[Chase Douglas]

On Fri, Mar 12, 2010 at 6:34 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 2010-03-12 at 10:32 -0500, Chase Douglas wrote:
>> We're considering using tracing_off() in __schedule_bug() in the
>> Ubuntu kernel to help us solve "scheduling while atomic" bugs without
>> needing to compile and install a special kernel to do so. Using the
>> tracing_off() call enables us to generate smaller function traces when
>> debugging, and find the bug easier since it should be at the end of
>> any traces. The patch we are thinking of using can be found at [1].
>> You can find an overview of how bugs are solved using this approach
>> there as well.
>>
>> Are there any thoughts as to the appropriateness of this patch in
>> Ubuntu kernels? My guess is that this isn't done in the upstream
>> vanilla kernels because it would be annoying to have a stock kernel
>> littered with tracing_off() calls if you hit one when you weren't
>> meaning to. However, we're interested in hearing if there's some
>> negative consequence we haven't thought of.
>
> Actually, it has been on my TODO list to litter tracing_off() all over
> the kernel ;-)
>
> No, by all means, I would definitely take a patch that turn off tracing
> at critical areas. One of which I would like to do is with the BUG()
> call.

Sounds good to me. The only place I currently see tracing_off called
is in oops_enter.

> Hmm, thinking about it more, I would rather have a separate function,
> that would call tracing_off() if some variable is set. By default it
> would be set, but in case you want to keep tracing after a bug is hit,
> you can have a way to disable it.
>
> I need to write up a patch soon. Thanks for bring this up.

I'd be happy to help out in this endeavor if you'd like. I'm wondering
if there shouldn't be multiple levels of tracing_off support specified
at boot time (disabled on WARNING, BUG, __schedule_bug, OOPS) in an
ordered priority way. I.e. tracing_off_bug would leave tracing on for
WARNING's, but turn it off for BUG's, schedule bugs, and oopses. The
default would be tracing_off_warn, which would call tracing_off on all
of the above.

Your thoughts?

-- Chase

Re: Using tracing_off() in __schedule_bug()

[Steven Rostedt]

On Fri, 2010-03-12 at 21:12 -0500, Chase Douglas wrote:
> On Fri, Mar 12, 2010 at 6:34 PM, Steven Rostedt <rostedt@goodmis.org> wrote:

> > Hmm, thinking about it more, I would rather have a separate function,
> > that would call tracing_off() if some variable is set. By default it
> > would be set, but in case you want to keep tracing after a bug is hit,
> > you can have a way to disable it.
> >
> > I need to write up a patch soon. Thanks for bring this up.
> 
> I'd be happy to help out in this endeavor if you'd like. I'm wondering
> if there shouldn't be multiple levels of tracing_off support specified
> at boot time (disabled on WARNING, BUG, __schedule_bug, OOPS) in an
> ordered priority way. I.e. tracing_off_bug would leave tracing on for
> WARNING's, but turn it off for BUG's, schedule bugs, and oopses. The
> default would be tracing_off_warn, which would call tracing_off on all
> of the above.

I think that's a bit over-engineering. I'd suggest that you either want
to disable tracing on an error or you don't. I could add a tracing
option that lets you stop it. Keep it simple. If it becomes complex, no
one will use it.

-- Steve

Re: Using tracing_off() in __schedule_bug()

[Chase Douglas]

On Fri, Mar 12, 2010 at 9:30 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 2010-03-12 at 21:12 -0500, Chase Douglas wrote:
>> On Fri, Mar 12, 2010 at 6:34 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
>> > Hmm, thinking about it more, I would rather have a separate function,
>> > that would call tracing_off() if some variable is set. By default it
>> > would be set, but in case you want to keep tracing after a bug is hit,
>> > you can have a way to disable it.
>> >
>> > I need to write up a patch soon. Thanks for bring this up.
>>
>> I'd be happy to help out in this endeavor if you'd like. I'm wondering
>> if there shouldn't be multiple levels of tracing_off support specified
>> at boot time (disabled on WARNING, BUG, __schedule_bug, OOPS) in an
>> ordered priority way. I.e. tracing_off_bug would leave tracing on for
>> WARNING's, but turn it off for BUG's, schedule bugs, and oopses. The
>> default would be tracing_off_warn, which would call tracing_off on all
>> of the above.
>
> I think that's a bit over-engineering. I'd suggest that you either want
> to disable tracing on an error or you don't. I could add a tracing
> option that lets you stop it. Keep it simple. If it becomes complex, no
> one will use it.

I was thinking that there may be times where you want to skip warnings
to trace real bugs. For example, there's a WARNING that you hit if
your resume takes too long. I may want to skip that warning for the
oops that occurs just after it. As a distro, we also want to be
flexible in our official kernels so we don't have to build special
ones when people hit bugs. It's not as though it would be very
difficult to design with a few priorities, so unless it's really
unnecessary I don't see why we shouldn't. The default would also fire
tracing_off in all cases, so most people wouldn't have to modify it
unless they hit a corner case.

-- Chase

Re: Using tracing_off() in __schedule_bug()

[Steven Rostedt]

On Fri, 2010-03-12 at 21:50 -0500, Chase Douglas wrote:
> On Fri, Mar 12, 2010 at 9:30 PM, Steven Rostedt <rostedt@goodmis.org> wrote:

> I was thinking that there may be times where you want to skip warnings
> to trace real bugs. For example, there's a WARNING that you hit if
> your resume takes too long. I may want to skip that warning for the
> oops that occurs just after it. As a distro, we also want to be
> flexible in our official kernels so we don't have to build special
> ones when people hit bugs. It's not as though it would be very
> difficult to design with a few priorities, so unless it's really
> unnecessary I don't see why we shouldn't. The default would also fire
> tracing_off in all cases, so most people wouldn't have to modify it
> unless they hit a corner case.

I'm fine with having a two layer. Stop tracing on bugs and/or on
warnings. I would actually have disable on warnings be default off.
There are too many kernel warnings that trigger too easily, having your
resume warning be one of them.

I'm still not sure we need a separate one for critical errors. A oops,
panic and BUG should all be the same. Since any of them can cause the
system to halt.

-- Steve

Re: Using tracing_off() in __schedule_bug()

[Chase Douglas]

On Fri, Mar 12, 2010 at 10:09 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 2010-03-12 at 21:50 -0500, Chase Douglas wrote:
>> On Fri, Mar 12, 2010 at 9:30 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
>> I was thinking that there may be times where you want to skip warnings
>> to trace real bugs. For example, there's a WARNING that you hit if
>> your resume takes too long. I may want to skip that warning for the
>> oops that occurs just after it. As a distro, we also want to be
>> flexible in our official kernels so we don't have to build special
>> ones when people hit bugs. It's not as though it would be very
>> difficult to design with a few priorities, so unless it's really
>> unnecessary I don't see why we shouldn't. The default would also fire
>> tracing_off in all cases, so most people wouldn't have to modify it
>> unless they hit a corner case.
>
> I'm fine with having a two layer. Stop tracing on bugs and/or on
> warnings. I would actually have disable on warnings be default off.
> There are too many kernel warnings that trigger too easily, having your
> resume warning be one of them.
>
> I'm still not sure we need a separate one for critical errors. A oops,
> panic and BUG should all be the same. Since any of them can cause the
> system to halt.

That sounds fine to me. I'll work on a patch for this.

-- Chase