[QUESTION] workqueue: Reducing flush_workqueue() overhead for unbound workqueues

[QUESTION] workqueue: Reducing flush_workqueue() overhead for unbound workqueues

[Yao Kai]

Hi,

I am seeing significant flush_workqueue() overhead for an unbound
workqueue with a very small concurrency requirement.

The original XFS discussion is here:

https://lore.kernel.org/all/20260827033654.1172495-1-ranhongyun1@huawei.com/T/#u

XFS creates its CIL push workqueue as:

         alloc_workqueue("xfs-cil/%s",
                 WQ_FREEZABLE | WQ_MEM_RECLAIM | WQ_UNBOUND, 4, ...);

The CIL pipeline has at most four push works scheduled or running
concurrently. During a synchronous log force, XFS calls
flush_workqueue() before queueing the new push work. This completes
previously queued pushes and reduces the latency of waiting for the
target checkpoint.

Since commit 636b927eba5b ("workqueue: Make unbound workqueues to use
per-cpu pool_workqueues"), flush_workqueue_prep_pwqs() walks the
per-CPU pwqs of the workqueue. Commit 85f0d8e39aff ("workqueue: Reduce
expensive locks for unbound workqueue") substantially reduced the
number of pool lock operations, but the O(nr_cpu_ids) pwq traversal
remains.

On a 128-CPU, 4-NUMA-node system, with 16 threads performing sequential
writes followed by fsync, I observed:

Function                      per-CPU pwqs     per-NUMA pwqs
-------------------------    ---------------  --------------
xfs_fsync_flush_log            256-1000 us      32-256 us
flush_workqueue_prep_pwqs        2-64 us          2-8 us

Dave pointed out that removing it may increase scheduling variance and
long-tail latency, and may move ordering latency to journal I/O
completion. He also suggested that changing the XFS caller would be a
workaround for an infrastructure regression, as other low-concurrency
unbound workqueues could have the same problem.

Is there a recommended way to avoid this overhead for such case,
I would appreciate guidance on how this problem should be solved.
I'm happy to prototype and benchmark the suggested approach.

Thanks,
Yao Kai

Re: [QUESTION] workqueue: Reducing flush_workqueue() overhead for unbound workqueues

[Tejun Heo]

Hello,

On Tue, Sep 01, 2026 at 04:46:06PM +0800, Yao Kai wrote:
> Is there a recommended way to avoid this overhead for such case,
> I would appreciate guidance on how this problem should be solved.

Posted a patchset which makes flush_workqueue() visit only the pwqs which
have been active since the last flush:

  https://lore.kernel.org/all/20260901210929.3092513-1-tj@kernel.org/

Can you please test whether it resolves the fsync latency issue you're
seeing?

Thanks.

--
tejun

Re: [QUESTION] workqueue: Reducing flush_workqueue() overhead for unbound workqueues

[Yao Kai]

On 9/2/2026 5:11 AM, Tejun Heo wrote:
> Hello,
> 
> On Tue, Sep 01, 2026 at 04:46:06PM +0800, Yao Kai wrote:
>> Is there a recommended way to avoid this overhead for such case,
>> I would appreciate guidance on how this problem should be solved.
> 
> Posted a patchset which makes flush_workqueue() visit only the pwqs which
> have been active since the last flush:
> 
>    https://lore.kernel.org/all/20260901210929.3092513-1-tj@kernel.org/
> 
> Can you please test whether it resolves the fsync latency issue you're
> seeing?
> 
> Thanks.
> 
> --
> tejun

Sorry for the late reply. Because my server encountered a hardware failure,
I was unable to test the previous comparison data.

This issue was discovered when running MySQL sysbench on XFS. When we
migrated the kernel from 5.10 to 6.6, we observed a 60% performance drop
in MySQL sysbench. After investigation, we found that the time spent on
`flush_workqueue_prep_pwqs` had increased significantly.

After my colleague applied your new patches, the performance of MySQL
sysbench returned to the level of the 5.10 kernel, so I believe this
can solve the problem.

Thanks,
Yao

Re: [QUESTION] workqueue: Reducing flush_workqueue() overhead for unbound workqueues

[Tejun Heo]

Hello,

On Mon, Sep 07, 2026 at 10:19:20AM +0800, Yao Kai wrote:
> After my colleague applied your new patches, the performance of MySQL
> sysbench returned to the level of the 5.10 kernel, so I believe this
> can solve the problem.

Great to hear. Applied the patchset to wq/for-7.4.

Thanks.

--
tejun