[PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Marco Crivellari]

Hi,

Currently system_wq and system_unbound_wq can still be used and no message
is printed. To avoid further use of them, add a warning. The detection of
deprecated workqueue is done using a new internal flag, called __WQ_DEPRECATED.

Similar scenario for WQ_PERCPU and WQ_UNBOUND. Currently there are still
users that use alloc_workqueue(,0,). To avoid such situations, a couple
of checks have been added:

  - if neither of the flag is present, set WQ_PERCPU
  - if both are present, remove WQ_PERCPU

Along with these, WARN_ONCE() is used.

Thanks!


Marco Crivellari (2):
  workqueue: Add warnings and fallback if system_{unbound}_wq is used
  workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND
    is present

 include/linux/workqueue.h |  1 +
 kernel/workqueue.c        | 31 ++++++++++++++++++++++++++++---
 2 files changed, 29 insertions(+), 3 deletions(-)

---
Changes in v4:
- P2: instead of pr_warn_once() use WARN_ONCE()

Link to v3: https://lore.kernel.org/all/20260528160159.471265-1-marco.crivellari@suse.com/

Changes in v3:
- system_wq / system_unbond_wq not routed to the newer. The warnings have been simpliified
  keeping only one of them inside __queue_work() printing the work function.

- the deprecated status is detected using a new internal flag, called __WQ_DEPRECATED.

Link to v2: https://lore.kernel.org/all/20260526150041.365392-1-marco.crivellari@suse.com/

Changes in v2:
- rebased on v7.1-rc5

- fixed typo in the text

- cleared WQ_PERCPU when wq_pwer_efficient flag is set, to avoid triggering a warning

- if statement to route old workqueue to newer also added in queue_rcu_work()

Link to v1: https://lore.kernel.org/all/20260514092354.125149-1-marco.crivellari@suse.com/

-- 
2.54.0

[PATCH v4 1/2] workqueue: Add warnings and fallback if system_{unbound}_wq is used

[Marco Crivellari]

Currently many users transitioned already to the new introduced workqueue
(system_percpu_wq, system_dfl_wq), but there are new users who still use the
older system_wq and system_unbound_wq.

This change try to push this transition forward, by warning whether the old
workqueues are used.

Link: https://lore.kernel.org/all/20250221112003.1dSuoGyc@linutronix.de/
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 include/linux/workqueue.h |  1 +
 kernel/workqueue.c        | 12 ++++++++++--
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
index 6177624539b3..a283766a192a 100644
--- a/include/linux/workqueue.h
+++ b/include/linux/workqueue.h
@@ -409,6 +409,7 @@ enum wq_flags {
 	__WQ_DRAINING		= 1 << 16, /* internal: workqueue is draining */
 	__WQ_ORDERED		= 1 << 17, /* internal: workqueue is ordered */
 	__WQ_LEGACY		= 1 << 18, /* internal: create*_workqueue() */
+	__WQ_DEPRECATED		= 1 << 19, /* internal: workqueue is deprecated */
 
 	/* BH wq only allows the following flags */
 	__WQ_BH_ALLOWS		= WQ_BH | WQ_HIGHPRI | WQ_PERCPU,
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 33b721a9af02..0b55f8009eed 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -2280,6 +2280,14 @@ static void __queue_work(int cpu, struct workqueue_struct *wq,
 	unsigned int work_flags;
 	unsigned int req_cpu = cpu;
 
+	/*
+	 * NOTE: Check whether the used workqueue is deprecated and warn
+	 */
+	if (unlikely(wq->flags & __WQ_DEPRECATED))
+		pr_warn_once("workqueue: work func %ps enqueued on deprecated workqueue. "
+			"Use system_{percpu|dfl}_wq instead.\n",
+			work->func);
+
 	/*
 	 * While a work item is PENDING && off queue, a task trying to
 	 * steal the PENDING will busy-loop waiting for it to either get
@@ -8037,12 +8045,12 @@ void __init workqueue_init_early(void)
 		ordered_wq_attrs[i] = attrs;
 	}
 
-	system_wq = alloc_workqueue("events", WQ_PERCPU, 0);
+	system_wq = alloc_workqueue("events", WQ_PERCPU | __WQ_DEPRECATED, 0);
 	system_percpu_wq = alloc_workqueue("events", WQ_PERCPU, 0);
 	system_highpri_wq = alloc_workqueue("events_highpri",
 					    WQ_HIGHPRI | WQ_PERCPU, 0);
 	system_long_wq = alloc_workqueue("events_long", WQ_PERCPU, 0);
-	system_unbound_wq = alloc_workqueue("events_unbound", WQ_UNBOUND, WQ_MAX_ACTIVE);
+	system_unbound_wq = alloc_workqueue("events_unbound", WQ_UNBOUND | __WQ_DEPRECATED, WQ_MAX_ACTIVE);
 	system_dfl_wq = alloc_workqueue("events_unbound", WQ_UNBOUND, WQ_MAX_ACTIVE);
 	system_freezable_wq = alloc_workqueue("events_freezable",
 					      WQ_FREEZABLE | WQ_PERCPU, 0);
-- 
2.54.0

[PATCH v4 2/2] workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is present

[Marco Crivellari]

Currently there are no checks in order to enforce the use of one between
WQ_PERCPU or WQ_UNBOUND.

So act as following:
- if neither of them is present, set WQ_PERCPU
- if both are present, remove WQ_PERCPU

Along with this change, WARN_ONCE(), so that the code still uses both or
neither of them, can be changed.

Link: https://lore.kernel.org/all/20250221112003.1dSuoGyc@linutronix.de/
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 kernel/workqueue.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 0b55f8009eed..d4799e93d10f 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -5826,7 +5826,7 @@ static struct workqueue_struct *__alloc_workqueue(const char *fmt,
 
 	/* see the comment above the definition of WQ_POWER_EFFICIENT */
 	if ((flags & WQ_POWER_EFFICIENT) && wq_power_efficient)
-		flags |= WQ_UNBOUND;
+		flags = (flags & ~WQ_PERCPU) | WQ_UNBOUND;
 
 	/* allocate wq and format name */
 	if (flags & WQ_UNBOUND)
@@ -5850,6 +5850,23 @@ static struct workqueue_struct *__alloc_workqueue(const char *fmt,
 		pr_warn_once("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n",
 			     wq->name);
 
+	/*
+	 * One among WQ_PERCPU and WQ_UNBOUND must be set, but not both.
+	 * - If neither is set, default to WQ_PERCPU
+	 * - If both are set, default to WQ_UNBOUND
+	 *
+	 * This code can be removed after workqueue are unbound by default
+	 */
+	if (unlikely(!(flags & (WQ_UNBOUND | WQ_PERCPU)))) {
+		WARN_ONCE(1, "workqueue: %s is using neither WQ_PERCPU or WQ_UNBOUND. "
+			  "Setting WQ_PERCPU.\n", wq->name);
+		flags |= WQ_PERCPU;
+	} else if(unlikely((flags & WQ_PERCPU) && (flags & WQ_UNBOUND))) {
+		WARN_ONCE(1, "workqueue: %s uses both WQ_PERCPU and WQ_UNBOUND. "
+			  "Dropped WQ_PERCPU, keeping WQ_UNBOUND.\n", wq->name);
+		flags &= (~WQ_PERCPU);
+	}
+
 	if (flags & WQ_BH) {
 		/*
 		 * BH workqueues always share a single execution context per CPU
-- 
2.54.0

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tejun Heo]

Hello,

On Fri, May 29, 2026 at 03:06:38PM +0200, Marco Crivellari wrote:
> Marco Crivellari (2):
>   workqueue: Add warnings and fallback if system_{unbound}_wq is used
>   workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND
>     is present

Applied 1-2 to wq/for-7.2. I fixed up minor whitespace on 2/2: added the
missing space in "else if (" and dropped the redundant parentheses around
~WQ_PERCPU.

Thanks.

-- 
tejun

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tetsuo Handa]

On 2026/05/30 3:07, Tejun Heo wrote:
> Hello,
> 
> On Fri, May 29, 2026 at 03:06:38PM +0200, Marco Crivellari wrote:
>> Marco Crivellari (2):
>>   workqueue: Add warnings and fallback if system_{unbound}_wq is used
>>   workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND
>>     is present
> 
> Applied 1-2 to wq/for-7.2. I fixed up minor whitespace on 2/2: added the
> missing space in "else if (" and dropped the redundant parentheses around
> ~WQ_PERCPU.

Please don't use WARN_ONCE(), please use printk() but don't emit WARNING: or BUG: in the string.
syzbot is failing to test linux-next kernels due to commit 21c05ca88a54 ("workqueue:
Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is present").
We can't tolerate linux-next being untested until all callers are fixed.

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tejun Heo]

Sorry for the slow reply - I'm off at the moment so my latency is high.

> syzbot is failing to test linux-next kernels due to commit 21c05ca88a54
> ("workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is
> present").
> We can't tolerate linux-next being untested until all callers are fixed.

Which caller is breaking which tests?

Reverting 21c05ca88a54 is easy, but there's no clean point to do it: the
warning is in the wq tree while the caller fixes are scattered across several
subsystem trees, and there's no reasonable way to synchronize all of that
outside mainline. If it's a specific caller breaking a test, fixing that one
is the better path.

Thanks.

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tetsuo Handa]

On 2026/06/01 1:48, Tejun Heo wrote:
> Sorry for the slow reply - I'm off at the moment so my latency is high.
> 
>> syzbot is failing to test linux-next kernels due to commit 21c05ca88a54
>> ("workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is
>> present").
>> We can't tolerate linux-next being untested until all callers are fixed.
> 
> Which caller is breaking which tests?

It is https://syzkaller.appspot.com/bug?extid=d078cba4418e65f61984 .

Even 10 days of being unable to test linux-next tree
( https://lkml.kernel.org/r/1a9f53d4-6f48-4df8-a3d8-2b0e442a163a@I-love.SAKURA.ne.jp )
can cause difficulty of identifying a culprit.

> 
> Reverting 21c05ca88a54 is easy, but there's no clean point to do it: the
> warning is in the wq tree while the caller fixes are scattered across several
> subsystem trees, and there's no reasonable way to synchronize all of that
> outside mainline. If it's a specific caller breaking a test, fixing that one
> is the better path.
> 

I've spent more than one year for updating all in-tree users who are
flushing kernel global workqueues. Please see a message from Linus at
https://lkml.kernel.org/r/CAHk-=whWreGjEQ6yasspzBrNnS7EQiL+SknToWt=SzUh4XomyQ@mail.gmail.com .

Please never try to emit WARNING: or BUG: message just for letting developers
update their code. If you can't use panic() instead of WARN*() or BUG*(), you
should not use WARN*() or BUG*(). The patch author who changes the behavior of
in-tree code is responsible for updating all in-tree code.

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tejun Heo]

On Mon, Jun 01, 2026 at 07:12:28AM +0900, Tetsuo Handa wrote:
> I've spent more than one year for updating all in-tree users who are
> flushing kernel global workqueues. Please see a message from Linus at
> https://lkml.kernel.org/r/CAHk-=whWreGjEQ6yasspzBrNnS7EQiL+SknToWt=SzUh4XomyQ@mail.gmail.com .
> 
> Please never try to emit WARNING: or BUG: message just for letting developers
> update their code. If you can't use panic() instead of WARN*() or BUG*(), you
> should not use WARN*() or BUG*(). The patch author who changes the behavior of
> in-tree code is responsible for updating all in-tree code.

I don't think a statement this blanket makes sense. I scanned the existing
linux-next tree and there were a handful that were missed (I didn't find out
whether they are new ones in linux-next or ones that were misseds from
mainline tho). Let's just apply the nvme-tcp patch and move on.

Thanks.

-- 
tejun

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Marco Crivellari]

On Fri, May 29, 2026 at 8:07 PM Tejun Heo <tj@kernel.org> wrote:
>
> Hello,
>
> On Fri, May 29, 2026 at 03:06:38PM +0200, Marco Crivellari wrote:
> > Marco Crivellari (2):
> >   workqueue: Add warnings and fallback if system_{unbound}_wq is used
> >   workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND
> >     is present
>
> Applied 1-2 to wq/for-7.2. I fixed up minor whitespace on 2/2: added the
> missing space in "else if (" and dropped the redundant parentheses around
> ~WQ_PERCPU.

Hello Tejun,

Many thanks!

-- 

Marco Crivellari

SUSE Labs

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tetsuo Handa]

On 2026/06/01 23:47, Tejun Heo wrote:
> On Mon, Jun 01, 2026 at 07:12:28AM +0900, Tetsuo Handa wrote:
>> I've spent more than one year for updating all in-tree users who are
>> flushing kernel global workqueues. Please see a message from Linus at
>> https://lkml.kernel.org/r/CAHk-=whWreGjEQ6yasspzBrNnS7EQiL+SknToWt=SzUh4XomyQ@mail.gmail.com .
>>
>> Please never try to emit WARNING: or BUG: message just for letting developers
>> update their code. If you can't use panic() instead of WARN*() or BUG*(), you
>> should not use WARN*() or BUG*(). The patch author who changes the behavior of
>> in-tree code is responsible for updating all in-tree code.
> 
> I don't think a statement this blanket makes sense. I scanned the existing
> linux-next tree and there were a handful that were missed (I didn't find out
> whether they are new ones in linux-next or ones that were misseds from
> mainline tho). Let's just apply the nvme-tcp patch and move on.

Not all users are quickly accepting patches. Please apply all fixes *before* you start
emitting WARNING: message. You are currently making linux-next tree untestable by syzbot.

> 
> Thanks.
> 

Mark Brown, please skip WQ changes from linux-next until all fixes are applied.

Linus, please be sure to skip WQ changes in the upcoming merge window if some fix
does not get applied in time for the merge window. (Of course, applying all fixes
before we start emitting WARNING: message is desirable for bisectability, even if
all fixes are get applied in time for the merge window...)

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tetsuo Handa]

On 2026/06/02 2:51, Tetsuo Handa wrote:
> On 2026/06/01 23:47, Tejun Heo wrote:
>> On Mon, Jun 01, 2026 at 07:12:28AM +0900, Tetsuo Handa wrote:
>>> I've spent more than one year for updating all in-tree users who are
>>> flushing kernel global workqueues. Please see a message from Linus at
>>> https://lkml.kernel.org/r/CAHk-=whWreGjEQ6yasspzBrNnS7EQiL+SknToWt=SzUh4XomyQ@mail.gmail.com .
>>>
>>> Please never try to emit WARNING: or BUG: message just for letting developers
>>> update their code. If you can't use panic() instead of WARN*() or BUG*(), you
>>> should not use WARN*() or BUG*(). The patch author who changes the behavior of
>>> in-tree code is responsible for updating all in-tree code.
>>
>> I don't think a statement this blanket makes sense. I scanned the existing
>> linux-next tree and there were a handful that were missed (I didn't find out
>> whether they are new ones in linux-next or ones that were misseds from
>> mainline tho). Let's just apply the nvme-tcp patch and move on.
> 
> Not all users are quickly accepting patches. Please apply all fixes *before* you start
> emitting WARNING: message. You are currently making linux-next tree untestable by syzbot.
> 
>>
>> Thanks.
>>
> 
> Mark Brown, please skip WQ changes from linux-next until all fixes are applied.
> 
> Linus, please be sure to skip WQ changes in the upcoming merge window if some fix
> does not get applied in time for the merge window. (Of course, applying all fixes
> before we start emitting WARNING: message is desirable for bisectability, even if
> all fixes are get applied in time for the merge window...)
> 

After 10 days of untestable period, the third breakage report came from btrfs
caused by commit 69635d7f4b34 ("fs: WQ_PERCPU added to alloc_workqueue users").
That commit added WQ_PERCPU to fs/btrfs/disk-io.c despite the "ordered_flags" is
passed to alloc_ordered_workqueue() which implicitly applies WQ_UNBOUND.

That commit says

  Both flags coexist for one release cycle to allow callers to transition
  their calls.

  Once migration is complete, WQ_UNBOUND can be removed and unbound will
  become the implicit default.

and you definitely started emitting WARNING: messages before allowing callers
to transition their calls. I consider that commit 21c05ca88a54 ("workqueue: Add
warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is present") should be
excluded from the linux-next and the upcoming merge window. The next one or two
cycles for transition should be printk() without WARNING: string.

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

On Wed, Jun 10, 2026 at 09:48:05PM +0900, Tetsuo Handa wrote:
> On 2026/06/02 2:51, Tetsuo Handa wrote:
> > On 2026/06/01 23:47, Tejun Heo wrote:

> > Not all users are quickly accepting patches. Please apply all fixes *before* you start
> > emitting WARNING: message. You are currently making linux-next tree untestable by syzbot.

> > Mark Brown, please skip WQ changes from linux-next until all fixes are applied.

So that's the workqueues tree so extra disruption would be limited but TBH
as far as kicking things out goes I'd be more inclined to take a steer
from Linus on his thinking here rather than making that decision for
myself, especially this close to the merge window.  If there's changes
that can't be applied directly to the workqueues tree one thing I could
do that's a bit less disruptive is carry those fixups in -next, do the
appropriate patches exist?  We'd obviously need to make sure they get
into mainline promptly.

Just as a note here if you put something at the top of the mail when you
add me it increases my chances of seeing the bit directed at me - I get
CCed on a *lot* of mail that's not at all relevant to me, unfortunately
that means I'm a bit lossy when it comes to triaging my inbox.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Nathan Chancellor]

On Wed, Jun 10, 2026 at 09:48:05PM +0900, Tetsuo Handa wrote:
> After 10 days of untestable period, the third breakage report came from btrfs
> caused by commit 69635d7f4b34 ("fs: WQ_PERCPU added to alloc_workqueue users").
> That commit added WQ_PERCPU to fs/btrfs/disk-io.c despite the "ordered_flags" is
> passed to alloc_ordered_workqueue() which implicitly applies WQ_UNBOUND.
> 
> That commit says
> 
>   Both flags coexist for one release cycle to allow callers to transition
>   their calls.
> 
>   Once migration is complete, WQ_UNBOUND can be removed and unbound will
>   become the implicit default.
> 
> and you definitely started emitting WARNING: messages before allowing callers
> to transition their calls. I consider that commit 21c05ca88a54 ("workqueue: Add
> warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is present") should be
> excluded from the linux-next and the upcoming merge window. The next one or two
> cycles for transition should be printk() without WARNING: string.

FWIW, I sent a patch for the btrfs issue:

  https://lore.kernel.org/20260601-btrfs-fix-wq-warning-qgroup-rescan-v1-1-aff9a1128f27@kernel.org/

With that applied to next-20260610, I don't see any other instances of
this workqueue warning on any of my test machines. I do agree that
enabling warnings prematurely is disruptive for testing and should be
avoided but it seems like this one is close? That change should be
applicable to the workqueue tree, it just needs to be taken at this
point.

-- 
Cheers,
Nathan

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

On Wed, Jun 10, 2026 at 10:28:23AM -0700, Nathan Chancellor wrote:

> With that applied to next-20260610, I don't see any other instances of
> this workqueue warning on any of my test machines. I do agree that
> enabling warnings prematurely is disruptive for testing and should be
> avoided but it seems like this one is close? That change should be
> applicable to the workqueue tree, it just needs to be taken at this
> point.

Yeah, if things are directly applicable to the workqueue tree they
should just go there - I was assuming the fixes that were causing issues
were ones that were for things newly added in -next.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 0/2] workqueue: Add warnings and check WQ flags usage

[Tetsuo Handa]

On 2026/06/11 2:28, Nathan Chancellor wrote:
> FWIW, I sent a patch for the btrfs issue:
> 
>   https://lore.kernel.org/20260601-btrfs-fix-wq-warning-qgroup-rescan-v1-1-aff9a1128f27@kernel.org/

Then, please immediately add that patch and
https://lkml.kernel.org/r/20260611015545.111157-1-wuyankun@uniontech.com
to the wq tree.

> 
> With that applied to next-20260610, I don't see any other instances of
> this workqueue warning on any of my test machines. I do agree that
> enabling warnings prematurely is disruptive for testing and should be
> avoided but it seems like this one is close? That change should be
> applicable to the workqueue tree, it just needs to be taken at this
> point.
>