* [GIT PULL] workqueue: Fixes for v7.4-rc2
@ 2026-11-15 20:45 Tejun Heo
2026-11-15 22:14 ` Linus Torvalds
0 siblings, 1 reply; 4+ messages in thread
From: Tejun Heo @ 2026-11-15 20:45 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Lai Jiangshan, linux-kernel
Hello,
The following changes since commit d89e1d186ec9271bf78e62771779e9fc03bf0fc3:
Merge tag 'fscrypt-for-linus' of git://git.kernel.org/pub/scm/fs/fscrypt/linux (2026-10-29 14:00:23 -1000)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git tags/wq-for-7.4-rc2-fixes
for you to fetch changes up to c7d64f52bffaca093acfb34277877138f6ec243a:
workqueue: Fix drain_workqueue() missing work items chained across pwqs (2026-11-02 08:06:21 -1000)
----------------------------------------------------------------
workqueue: Fixes for v7.4-rc2
- Draining a workqueue could return while a work item queued by another work
item was still pending, making workqueue destruction warn and leak the
workqueue.
----------------------------------------------------------------
Tejun Heo (1):
workqueue: Fix drain_workqueue() missing work items chained across pwqs
kernel/workqueue.c | 68 ++++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 51 insertions(+), 17 deletions(-)
--
tejun
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [GIT PULL] workqueue: Fixes for v7.4-rc2 2026-11-15 20:45 [GIT PULL] workqueue: Fixes for v7.4-rc2 Tejun Heo @ 2026-11-15 22:14 ` Linus Torvalds 2026-11-16 17:09 ` Tejun Heo 0 siblings, 1 reply; 4+ messages in thread From: Linus Torvalds @ 2026-11-15 22:14 UTC (permalink / raw) To: Tejun Heo; +Cc: Lai Jiangshan, linux-kernel On Sun, 15 Nov 2026 at 12:45, Tejun Heo <tj@kernel.org> wrote: > > - Draining a workqueue could return while a work item queued by another work > item was still pending, making workqueue destruction warn and leak the > workqueue. Why is this re-creating - badly - the existing sequence counter code? Not only does it do the memory barriers by hand - possibly getting them wrong, but it seems actively broken. There's a reason proper sequence counters have that "odd/even" rule, and a writer increments the sequence counter *twice* - once when it starts the modification and once when it is done. As far as I can tell, your "fix" is still broken: drain_workqueue: mutex_lock(&wq->mutex); if (!wq->nr_drainers++) wq->flags |= __WQ_DRAINING; mutex_unlock(&wq->mutex); seq = atomic_read(&wq->drain_seq); __queue_work: if (unlikely(wq->flags & (__WQ_DESTROYING | __WQ_DRAINING))) { ... smp_mb__before_atomic(); atomic_inc(&wq->drain_seq); smp_mb__after_atomic(); but what if __queue_work() was called before __WQ_DRAINING was set and is running concurrently and never incremented the sequence number? Of even if it *did* increment the sequence number, what keeps it from doing that *before* the atomic_read() in drain_workqueue() and then Either way drain_workqueue() never sees that drain_seq change, and thinks everythings is fine, even though __queue_work added an entry? Is there some other serialization that I'm missing? Anyway, sequence numbers almost never work with a single write, exactly beause the single write is itself then not ordered with the changes that the writer does (ie the sequence point update happens before them or after them, but either way the reader might miss the sequence number update, becuase it's not atomic wrt the actual change it is supposed to be about. The actual 8real* sequence numbers we have do that two-step change exactly because of these issues. The writer increments once before it starts writing, and increments it again after it is done. That way the reader actually can check whether something changed (an odd value means that change is happening and needs retrying regardless - which is why read_seqcount_begin(() will loop on odd numbers) It is possible that I'm missing something that makes your ad-hoc version ok, but this really does look just plain broken. I pulled it and then unpulled it, waiting for explanations. Linus ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [GIT PULL] workqueue: Fixes for v7.4-rc2 2026-11-15 22:14 ` Linus Torvalds @ 2026-11-16 17:09 ` Tejun Heo 2026-11-16 17:29 ` Linus Torvalds 0 siblings, 1 reply; 4+ messages in thread From: Tejun Heo @ 2026-11-16 17:09 UTC (permalink / raw) To: Linus Torvalds; +Cc: Lai Jiangshan, linux-kernel Hello, Linus. On Sun, Nov 15, 2026 at 02:14:41PM -0800, Linus Torvalds wrote: > On Sun, 15 Nov 2026 at 12:45, Tejun Heo <tj@kernel.org> wrote: > > > > - Draining a workqueue could return while a work item queued by another work > > item was still pending, making workqueue destruction warn and leak the > > workqueue. > > Why is this re-creating - badly - the existing sequence counter code? I was trying to plug the hole minimally and it just didn't trigger "oh this is seqcount". > Not only does it do the memory barriers by hand - possibly getting > them wrong, but it seems actively broken. > > There's a reason proper sequence counters have that "odd/even" rule, > and a writer increments the sequence counter *twice* - once when it > starts the modification and once when it is done. > > As far as I can tell, your "fix" is still broken: > > drain_workqueue: > > mutex_lock(&wq->mutex); > if (!wq->nr_drainers++) > wq->flags |= __WQ_DRAINING; > mutex_unlock(&wq->mutex); > > seq = atomic_read(&wq->drain_seq); > > > __queue_work: > > if (unlikely(wq->flags & (__WQ_DESTROYING | __WQ_DRAINING))) { > ... > smp_mb__before_atomic(); > atomic_inc(&wq->drain_seq); > smp_mb__after_atomic(); > > but what if __queue_work() was called before __WQ_DRAINING was set and > is running concurrently and never incremented the sequence number? > > Of even if it *did* increment the sequence number, what keeps it from > doing that *before* the atomic_read() in drain_workqueue() and then > > Either way drain_workqueue() never sees that drain_seq change, and > thinks everythings is fine, even though __queue_work added an entry? > > Is there some other serialization that I'm missing? Hmm... so, draining looks like this: set DRAINING; read seq; flush wq: for each pwq: flush them; for each pwq: lock and check whether empty; if not drained or seq bumped, try again; and the queueing side tests DRAINING and incs the seq while the queueing work item is still in flight on its pwq. The chained insertion is under the pool lock. Any work item that hasn't seen DRAINING or inc'd the seq before draining reads it should be visible to the flush wq step, and by the time flush finishes, all the immediate chained work items it queued should be visible to the following pwq empty test step. A chained work item queued after the flush started saw DRAINING and inc'd the seq after the read, so if it chains further and finishes before the empty test, the seq check catches it. > Anyway, sequence numbers almost never work with a single write, > exactly beause the single write is itself then not ordered with the > changes that the writer does (ie the sequence point update happens > before them or after them, but either way the reader might miss the > sequence number update, becuase it's not atomic wrt the actual change > it is supposed to be about. > > The actual 8real* sequence numbers we have do that two-step change > exactly because of these issues. The writer increments once before it > starts writing, and increments it again after it is done. That way the > reader actually can check whether something changed (an odd value > means that change is happening and needs retrying regardless - which > is why read_seqcount_begin(() will loop on odd numbers) > > It is possible that I'm missing something that makes your ad-hoc > version ok, but this really does look just plain broken. > > I pulled it and then unpulled it, waiting for explanations. I don't think it's broken but it is adhoc and difficult to get one's head around including my own. It'd be a lot simpler w/ seqcount. I'm traveling this week, so might be slower but will redo it. Thanks. -- tejun ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [GIT PULL] workqueue: Fixes for v7.4-rc2 2026-11-16 17:09 ` Tejun Heo @ 2026-11-16 17:29 ` Linus Torvalds 0 siblings, 0 replies; 4+ messages in thread From: Linus Torvalds @ 2026-11-16 17:29 UTC (permalink / raw) To: Tejun Heo; +Cc: Lai Jiangshan, linux-kernel On Mon, 16 Nov 2026 at 09:09, Tejun Heo <tj@kernel.org> wrote: > > Hmm... so, draining looks like this: > > set DRAINING; > read seq; > flush wq: > for each pwq: > flush them; > for each pwq: > lock and check whether empty; > if not drained or seq bumped, try again; > > and the queueing side tests DRAINING and incs the seq while the queueing > work item is still in flight on its pwq. The chained insertion is under the > pool lock. So now I don't have the patch in my tree any more (I could fetch it again, but I'm lazy), but from memory and my lkimited quoting the queueing side only incremented the seq number *if* it saw DRAINING. And that seemed to be the biggest issue. If the queueing side did *not* see DRAINING, it also wouldn't increment the sequence number, and it also didn't have any memory barriers. So it looked to me like the code basically fell back to the exact old sequence that the patch claimed to fix: if the queuing side missed seeing that "set DRAINING" part, all the changes in the commit became no-ops. Yes, the draining side would re-read the sequence counter and retry, but the sequence counter didn't actually change because the queueing side wouldn't be updating it. But I'll freely admit that I really only looked at the "shape" of the code, and it all felt wrong to me. This was not some deep-dive understanding, it was a "this doesn't feel right". > I don't think it's broken but it is adhoc and difficult to get one's head > around including my own. It'd be a lot simpler w/ seqcount. I'm traveling > this week, so might be slower but will redo it. Thanks. The main problem with a seqcount might be performance - I'm assuming you tried to avoid the write for the common cases. I assume *draining* is the slow and uncommon case, and typically *that* side is where you want to do the writer side of the seqcount, and then the fast and common case is the one that wants to do the sequence number read. So maybe the generic seqcount code isn't what you want, and maybe your current code is the right thing, and my "it feels wrong" is the wrong reaction. Put another way: you should also just feel free to tell me that I was wrong, and explain *why* it all works the way you did it. Use small words and call me names. Linus ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-16 17:29 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-11-15 20:45 [GIT PULL] workqueue: Fixes for v7.4-rc2 Tejun Heo 2026-11-15 22:14 ` Linus Torvalds 2026-11-16 17:09 ` Tejun Heo 2026-11-16 17:29 ` Linus Torvalds
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®