mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [GIT PULL] workqueue: Fixes for v7.4-rc2
@ 2026-11-15 20:45 Tejun Heo
  2026-11-15 22:14 ` Linus Torvalds
  0 siblings, 1 reply; 4+ messages in thread
From: Tejun Heo @ 2026-11-15 20:45 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Lai Jiangshan, linux-kernel

Hello,

The following changes since commit d89e1d186ec9271bf78e62771779e9fc03bf0fc3:

  Merge tag 'fscrypt-for-linus' of git://git.kernel.org/pub/scm/fs/fscrypt/linux (2026-10-29 14:00:23 -1000)

are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git tags/wq-for-7.4-rc2-fixes

for you to fetch changes up to c7d64f52bffaca093acfb34277877138f6ec243a:

  workqueue: Fix drain_workqueue() missing work items chained across pwqs (2026-11-02 08:06:21 -1000)

----------------------------------------------------------------
workqueue: Fixes for v7.4-rc2

- Draining a workqueue could return while a work item queued by another work
  item was still pending, making workqueue destruction warn and leak the
  workqueue.

----------------------------------------------------------------
Tejun Heo (1):
      workqueue: Fix drain_workqueue() missing work items chained across pwqs

 kernel/workqueue.c | 68 ++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 51 insertions(+), 17 deletions(-)

-- 
tejun

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [GIT PULL] workqueue: Fixes for v7.4-rc2
  2026-11-15 20:45 [GIT PULL] workqueue: Fixes for v7.4-rc2 Tejun Heo
@ 2026-11-15 22:14 ` Linus Torvalds
  2026-11-16 17:09   ` Tejun Heo
  0 siblings, 1 reply; 4+ messages in thread
From: Linus Torvalds @ 2026-11-15 22:14 UTC (permalink / raw)
  To: Tejun Heo; +Cc: Lai Jiangshan, linux-kernel

On Sun, 15 Nov 2026 at 12:45, Tejun Heo <tj@kernel.org> wrote:
>
> - Draining a workqueue could return while a work item queued by another work
>   item was still pending, making workqueue destruction warn and leak the
>   workqueue.

Why is this re-creating - badly - the existing sequence counter code?

Not only does it do the memory barriers by hand - possibly getting
them wrong, but it seems actively broken.

There's a reason proper sequence counters have that "odd/even" rule,
and a writer increments the sequence counter *twice* - once when it
starts the modification and once when it is done.

As far as I can tell, your "fix" is still broken:

  drain_workqueue:

        mutex_lock(&wq->mutex);
        if (!wq->nr_drainers++)
                wq->flags |= __WQ_DRAINING;
        mutex_unlock(&wq->mutex);

        seq = atomic_read(&wq->drain_seq);


__queue_work:

        if (unlikely(wq->flags & (__WQ_DESTROYING | __WQ_DRAINING))) {
   ...
                smp_mb__before_atomic();
                atomic_inc(&wq->drain_seq);
                smp_mb__after_atomic();

but what if __queue_work() was called before __WQ_DRAINING was set and
is running concurrently and never incremented the sequence number?

Of even if it *did* increment the sequence number, what keeps it from
doing that *before* the atomic_read() in drain_workqueue() and then

Either way drain_workqueue() never sees that drain_seq change, and
thinks everythings is fine, even though __queue_work added an entry?

Is there some other serialization that I'm missing?

Anyway, sequence numbers almost never work with a single write,
exactly beause the single write is itself then not ordered with the
changes that the writer does (ie the sequence point update happens
before them or after them, but either way the reader might miss the
sequence number update, becuase it's not atomic wrt the actual change
it is supposed to be about.

The actual 8real* sequence numbers we have do that two-step change
exactly because of these issues. The writer increments once before it
starts writing, and increments it again after it is done. That way the
reader actually can check whether something changed (an odd value
means that change is happening and needs retrying regardless - which
is why read_seqcount_begin(() will loop on odd numbers)

It is possible that I'm missing something that makes your ad-hoc
version ok, but this really does look just plain broken.

I pulled it and then unpulled it, waiting for explanations.

             Linus

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [GIT PULL] workqueue: Fixes for v7.4-rc2
  2026-11-15 22:14 ` Linus Torvalds
@ 2026-11-16 17:09   ` Tejun Heo
  2026-11-16 17:29     ` Linus Torvalds
  0 siblings, 1 reply; 4+ messages in thread
From: Tejun Heo @ 2026-11-16 17:09 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Lai Jiangshan, linux-kernel

Hello, Linus.

On Sun, Nov 15, 2026 at 02:14:41PM -0800, Linus Torvalds wrote:
> On Sun, 15 Nov 2026 at 12:45, Tejun Heo <tj@kernel.org> wrote:
> >
> > - Draining a workqueue could return while a work item queued by another work
> >   item was still pending, making workqueue destruction warn and leak the
> >   workqueue.
> 
> Why is this re-creating - badly - the existing sequence counter code?

I was trying to plug the hole minimally and it just didn't trigger "oh this
is seqcount".

> Not only does it do the memory barriers by hand - possibly getting
> them wrong, but it seems actively broken.
> 
> There's a reason proper sequence counters have that "odd/even" rule,
> and a writer increments the sequence counter *twice* - once when it
> starts the modification and once when it is done.
> 
> As far as I can tell, your "fix" is still broken:
> 
>   drain_workqueue:
> 
>         mutex_lock(&wq->mutex);
>         if (!wq->nr_drainers++)
>                 wq->flags |= __WQ_DRAINING;
>         mutex_unlock(&wq->mutex);
> 
>         seq = atomic_read(&wq->drain_seq);
> 
> 
> __queue_work:
> 
>         if (unlikely(wq->flags & (__WQ_DESTROYING | __WQ_DRAINING))) {
>    ...
>                 smp_mb__before_atomic();
>                 atomic_inc(&wq->drain_seq);
>                 smp_mb__after_atomic();
> 
> but what if __queue_work() was called before __WQ_DRAINING was set and
> is running concurrently and never incremented the sequence number?
> 
> Of even if it *did* increment the sequence number, what keeps it from
> doing that *before* the atomic_read() in drain_workqueue() and then
> 
> Either way drain_workqueue() never sees that drain_seq change, and
> thinks everythings is fine, even though __queue_work added an entry?
> 
> Is there some other serialization that I'm missing?

Hmm... so, draining looks like this:

        set DRAINING;
        read seq;
        flush wq:
                for each pwq:
                        flush them;
        for each pwq:
                lock and check whether empty;
        if not drained or seq bumped, try again;

and the queueing side tests DRAINING and incs the seq while the queueing
work item is still in flight on its pwq. The chained insertion is under the
pool lock.

Any work item that hasn't seen DRAINING or inc'd the seq before draining
reads it should be visible to the flush wq step, and by the time flush
finishes, all the immediate chained work items it queued should be visible
to the following pwq empty test step.

A chained work item queued after the flush started saw DRAINING and inc'd
the seq after the read, so if it chains further and finishes before the
empty test, the seq check catches it.

> Anyway, sequence numbers almost never work with a single write,
> exactly beause the single write is itself then not ordered with the
> changes that the writer does (ie the sequence point update happens
> before them or after them, but either way the reader might miss the
> sequence number update, becuase it's not atomic wrt the actual change
> it is supposed to be about.
> 
> The actual 8real* sequence numbers we have do that two-step change
> exactly because of these issues. The writer increments once before it
> starts writing, and increments it again after it is done. That way the
> reader actually can check whether something changed (an odd value
> means that change is happening and needs retrying regardless - which
> is why read_seqcount_begin(() will loop on odd numbers)
> 
> It is possible that I'm missing something that makes your ad-hoc
> version ok, but this really does look just plain broken.
> 
> I pulled it and then unpulled it, waiting for explanations.

I don't think it's broken but it is adhoc and difficult to get one's head
around including my own. It'd be a lot simpler w/ seqcount. I'm traveling
this week, so might be slower but will redo it.

Thanks.

-- 
tejun

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [GIT PULL] workqueue: Fixes for v7.4-rc2
  2026-11-16 17:09   ` Tejun Heo
@ 2026-11-16 17:29     ` Linus Torvalds
  0 siblings, 0 replies; 4+ messages in thread
From: Linus Torvalds @ 2026-11-16 17:29 UTC (permalink / raw)
  To: Tejun Heo; +Cc: Lai Jiangshan, linux-kernel

On Mon, 16 Nov 2026 at 09:09, Tejun Heo <tj@kernel.org> wrote:
>
> Hmm... so, draining looks like this:
>
>         set DRAINING;
>         read seq;
>         flush wq:
>                 for each pwq:
>                         flush them;
>         for each pwq:
>                 lock and check whether empty;
>         if not drained or seq bumped, try again;
>
> and the queueing side tests DRAINING and incs the seq while the queueing
> work item is still in flight on its pwq. The chained insertion is under the
> pool lock.

So now I don't have the patch in my tree any more (I could fetch it
again, but I'm lazy), but from memory and my lkimited quoting the
queueing side only incremented the seq number *if* it saw DRAINING.

And that seemed to be the biggest issue.

If the queueing side did *not* see DRAINING, it also wouldn't
increment the sequence number, and it also didn't have any memory
barriers.

So it looked to me like the code basically fell back to the exact old
sequence that the patch claimed to fix: if the queuing side missed
seeing that "set DRAINING" part, all the changes in the commit became
no-ops. Yes, the draining side would re-read the sequence counter and
retry, but the sequence counter didn't actually change because the
queueing side wouldn't be updating it.

But I'll freely admit that I really only looked at the "shape" of the
code, and it all felt wrong to me. This was not some deep-dive
understanding, it was a "this doesn't feel right".

> I don't think it's broken but it is adhoc and difficult to get one's head
> around including my own. It'd be a lot simpler w/ seqcount. I'm traveling
> this week, so might be slower but will redo it.

Thanks.

The main problem with a seqcount might be performance - I'm assuming
you tried to avoid the write for the common cases. I assume *draining*
is the slow and uncommon case, and typically *that* side is where you
want to do the writer side of the seqcount, and then the fast and
common case is the one that wants to do the sequence number read.

So maybe the generic seqcount code isn't what you want, and maybe your
current code is the right thing, and my "it feels wrong" is the wrong
reaction.

Put another way: you should also just feel free to tell me that I was
wrong, and explain *why* it all works the way you did it. Use small
words and call me names.

             Linus

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-16 17:29 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-15 20:45 [GIT PULL] workqueue: Fixes for v7.4-rc2 Tejun Heo
2026-11-15 22:14 ` Linus Torvalds
2026-11-16 17:09   ` Tejun Heo
2026-11-16 17:29     ` Linus Torvalds

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®