From: Tejun Heo <tj@kernel.org>
To: Lei Wen <adrian.wenl@gmail.com>
Cc: linux-kernel@vger.kernel.org, leiwen@marvell.com,
wwang27@marvell.com, Thomas Gleixner <tglx@linutronix.de>
Subject: Re: workqueue panic in 3.4 kernel
Date: Thu, 7 Mar 2013 07:49:07 -0800 [thread overview]
Message-ID: <20130307154907.GB29601@htj.dyndns.org> (raw)
In-Reply-To: <CALZhoSRJ+HV93tDJo-57M-SdXi8Z6qW_bRV9FzSfGWuz8j-t8Q@mail.gmail.com>
(cc'ing Thomas, hi!)
Hello,
Lei is seeing a problem where a delayed_work item gets corrupted (its
work->data gets cleared while still queued on the timer). He thinks
what's going on is that del_timer() is returning 1 but the timer
function still gets executed.
On Thu, Mar 07, 2013 at 11:22:40PM +0800, Lei Wen wrote:
> >> If del_timer() happens after the timer starts running, del_timer()
> >> would return NULL and try_to_grab_pending() will be called which will
> >> return >=0 iff if successfully steals the PENDING bit (ie. it's the
> >> sole owner of the work item). If del_timer() happens before the timer
> >> starts running, the timer function would never run.
> >
> > If del_timer() happen before __run_timers() is called, while timer irq
> > already happen,
> > would it return 1 for the timer is still not detached in __run_timers()?
> > If it is possible, then we would call try_to_grab_pending(), so that
> > work->data would
> > be cleared in this way.
> >
> >>
> >> clear_work_data() happens iff the work item is confirmed to be idle.
> >> At this point, I'm pretty skeptical this is a bug in workqueue itself
> >> and strongly suggest looking at the crashing workqueue user.
> >
> > Also I am not very familiar with workqueue mechanism, how many place
> > in kernel would
> > clear the work->data beside the clear_work_data()?
Work item initialization and clear_work_data() are the only places and
from the looks of it you definitely seem to be hitting
clear_work_data().
> > From the memory, I cannot find any hint for work structure being destroyed.
> > So the only possible seems to me is the work->data be set by someone on purpose.
> >
> > crash> struct delayed_work 0xbf03d544 -x
> > struct delayed_work {
> > work = {
> > data = {
> > counter = 0x300
> > },
> > entry = {
> > next = 0xbf03d548,
> > prev = 0xbf03d548
> > },
> > func = 0xbf014b00
> > },
> > timer = {
> > entry = {
> > next = 0x0,
> > prev = 0x200200
> > },
> > expires = 0x12b638b,
> > base = 0xc0844e01,
> > function = 0xc014c7a0 <delayed_work_timer_fn>,
> > data = 0xbf03d544,
> > slack = 0xffffffff,
> > start_pid = 0xffffffff,
> > start_site = 0x0,
> > start_comm = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> > }
> > }
>
> I captured a trace log, which shows my previous suspicion is true:
> __cancel_work_timer is called before delayed_work_timer_fn, so that
> work->data is cleared.
>
> And when __cancel_work_timer is called, the timer is still pending,
> so del_timer would return 1, thus no try_to_grab_pending would be called.
>
> But it is very strange that in __run_timers, it still get the same timer.
> Then its callback, delayed_work_timer_fn, would be called, which cause
> the issue.
>
> The detach_timer in __cancel_work_timer should already move the timer
> from all list, am I right?
Yes.
> Could it happen for the timer_list be queued twice, like queue over two cpu?
> If not, how could it happen?
I can't see how something like that would happen and still find it
quite unlikely this would be a generic problem in either timer or
workqueue given how widely those are used and your case is the only
similar case that came up till now (and 3.4 is a long time ago).
Thomas, any ideas?
Thanks.
--
tejun
next prev parent reply other threads:[~2013-03-07 15:49 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-03-05 7:31 Lei Wen
2013-03-05 16:32 ` Tejun Heo
2013-03-06 14:39 ` Lei Wen
2013-03-06 19:14 ` Tejun Heo
2013-03-07 1:15 ` Lei Wen
2013-03-07 15:22 ` Lei Wen
2013-03-07 15:49 ` Tejun Heo [this message]
2013-03-07 16:07 ` Thomas Gleixner
[not found] ` <CALZhoSQBH3RxSoaVDCYoCyRVRndht-Rk0rh_w5Dbp6+5T_auSw@mail.gmail.com>
2013-03-12 5:12 ` Tejun Heo
2013-03-12 5:18 ` Lei Wen
2013-03-12 5:24 ` Tejun Heo
2013-03-12 5:34 ` Lei Wen
2013-03-12 5:40 ` Tejun Heo
2013-03-12 6:01 ` Lei Wen
2013-03-12 6:13 ` Tejun Heo
2013-03-12 6:41 ` Lei Wen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130307154907.GB29601@htj.dyndns.org \
--to=tj@kernel.org \
--cc=adrian.wenl@gmail.com \
--cc=leiwen@marvell.com \
--cc=linux-kernel@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=wwang27@marvell.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®