From: Peter Hurley <peter@hurleysoftware.com>
To: Andrew Morton <akpm@linux-foundation.org>,
Xiaotian Feng <xtfeng@gmail.com>
Cc: linux-kernel@vger.kernel.org,
Xiaotian Feng <dannyfeng@tencent.com>,
Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@elte.hu>
Subject: [BUG -next-20121127] kernel BUG at kernel/softirq.c:471!
Date: Wed, 28 Nov 2012 13:00:23 -0500 [thread overview]
Message-ID: <1354125623.2788.23.camel@thor> (raw)
In-Reply-To: <20121105173707.94602896.akpm@linux-foundation.org>
On Mon, 2012-11-05 at 17:37 -0800, Andrew Morton wrote:
> On Tue, 6 Nov 2012 09:22:16 +0800 Xiaotian Feng <xtfeng@gmail.com> wrote:
>
> > On Tue, Nov 6, 2012 at 6:52 AM, Andrew Morton <akpm@linux-foundation.org> wrote:
> > > On Fri, 2 Nov 2012 10:48:54 +0800
> > > Xiaotian Feng <xtfeng@gmail.com> wrote:
> > >
> > >> We met a ksoftirqd 100% issue, the perf top shows kernel is busy
> > >> with tasklet_action(), but no actual action is shown. From dumped
> > >> kernel, there's only one disabled tasklet on the tasklet_vec.
> > >>
> > >> tasklet_action might be handled after tasklet is disabled, this will
> > >> make disabled tasklet stayed on tasklet_vec. tasklet_action will not
> > >> handle disabled tasklet, but place it on the tail of tasklet_vec,
> > >> still raise softirq for this tasklet. Things will become worse if
> > >> device driver uses tasklet_disable on its device remove/close code.
> > >> The disabled tasklet will stay on the vec, frequently __raise_softirq_off()
> > >> and make ksoftirqd wakeup even if no tasklets need to be handled.
> > >>
> > >> This patch introduced a new TASKLET_STATE_HI bit to indicate HI_SOFTIRQ,
> > >> in tasklet_action(), simply ignore the disabled tasklet and don't raise
> > >> the softirq nr. In my previous patch, I remove tasklet_hi_enable() since
> > >> it is the same as tasklet_enable(). So only tasklet_enable() needs to be
> > >> modified, if tasklet state is changed from disable to enable, use
> > >> __tasklet_schedule() to put it on the right vec.
> > >
> > > gee, I haven't looked at the tasklet code in 100 years. I think I'll
> > > send this in Thomas's direction ;)
> > >
> > > The race description seems real and the patch looks sane to me. Are
> > > you sure we can get away with never clearing TASKLET_STATE_HI? For
> > > example, what would happen if code does a tasklet_hi_schedule(t) and
> > > later does a tasklet_schedule(t)?
> >
> > hmm, that will be a nightmare...
> > tasklet_schedule(t)/tasklet_hi_schedule(t) doesn't use list_head, they
> > simply
> > make t->next = NULL, then put t on the tail of
> > tasklet_vec/tasklet_hi_vec. If the code does a tasklet_hi_schedule()
> > and then a tasklet_schedule, the tasklet will stay on tasklet_vec and
> > tasklet_hi_vec .... tasklet_hi_action will handle it first and clear
> > the TASKLET_SCHED_SCHED bit, later, in tasklet_action, it will be
> > handled again and hit a BUG_ON ...
>
> Well, actually I meant if the caller reuses the tassklet_struct after
> its softirq has been run.
>
> > But if code does a tasklet_hi_schedule(), then tasklet_kil and later
> > does a tasklet_schedule(), we do need clear the TASKLET_STATE_HI.
>
> That as well ;)
>
> > Also
> > we need to remove the tasklet_hi_enable() as it is the same as
> > tasklet_enable() and there's
> > only one user..
> >
> > I'll send you V2 patch soon, thanks.
>
> Sounds good.
Hi all,
I couldn't find the v2 patch of this on linux-kernel but this commit
4660e32 "tasklet: ignore disabled tasklet in tasklet_action()"
BUGS in -next-20121127.
-----------[cut here ]----------
kernel BUG at /home/peter/src/kernels/next/kernel/softirq.c:471!
invalid opcode: 0000 [#1] PREEMPT SMP
....
The registers/stack dump isn't useful so I didn't include it here.
I'm still trying to track down the execution sequence that causes this,
but the high-level trigger is a firewire bus reset.
Hopefully I'll have more information soon.
Regards,
Peter Hurley
PS - My new staging/fwserial driver isn't to blame because it isn't
loaded when this happens ;)
next prev parent reply other threads:[~2012-11-28 18:00 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-02 2:48 [RFC PATCH] tasklet: ignore disabled tasklet in tasklet_action Xiaotian Feng
2012-11-05 22:52 ` Andrew Morton
2012-11-06 1:22 ` Xiaotian Feng
2012-11-06 1:37 ` Andrew Morton
2012-11-28 18:00 ` Peter Hurley [this message]
2012-11-28 22:12 ` [BUG -next-20121127] kernel BUG at kernel/softirq.c:471! Peter Hurley
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1354125623.2788.23.camel@thor \
--to=peter@hurleysoftware.com \
--cc=akpm@linux-foundation.org \
--cc=dannyfeng@tencent.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=tglx@linutronix.de \
--cc=xtfeng@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®