[PATCH V4 0/4] workqueue: Factor the way to assign rescuer work

[PATCH V4 0/4] workqueue: Factor the way to assign rescuer work

[Lai Jiangshan]

From: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Previously, the rescuer scanned for all matching work items at once and
processed them within a single rescuer thread, which could cause one
blocking work item to stall all others.

Make the rescuer process work items one-by-one instead of slurping all
matches in a single pass using a cursor.

Changed from V3:
  check memory pressure
  update comments
  rename @limited to @throttled

v3: https://lore.kernel.org/lkml/20251121145720.342467-1-jiangshanlai@gmail.com/

Lai Jiangshan (4):
  workqueue: Loop over in rescuer until all its work is done
  workqueue: Process rescuer work items one-by-one using a cursor
  workqueue: Limit number of processed works in rescuer per turn
  workqueue: Process extra works in rescuer when there are no more to
    rescue

 kernel/workqueue.c | 141 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 108 insertions(+), 33 deletions(-)

-- 
2.19.1.6.gb485710b

[PATCH V4 1/4] workqueue: Loop over in rescuer until all its work is done

[Lai Jiangshan]

From: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Simplify the rescuer work by looping directly in the rescuer rather than
adding the pwq back to the maydays list. This also helps when
max_requests is 1 or small but pwq->inactive_works has a large number of
pending work items.

This might hurt fairness among PWQs and the rescuer could end up being
stuck on one PWQ indefinitely, but the rescuer's objective is to make
forward progress rather than ensure fairness.

Fairness can be further improved in future by assigning work items to
the rescuer one by one and this is a temporary change to ease the
transition.

Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
---
 kernel/workqueue.c | 24 +-----------------------
 1 file changed, 1 insertion(+), 23 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 2654fbd481a1..02386e6eb409 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -3520,31 +3520,9 @@ static int rescuer_thread(void *__rescuer)
 
 		WARN_ON_ONCE(!list_empty(&rescuer->scheduled));
 
-		if (assign_rescuer_work(pwq, rescuer)) {
+		while (assign_rescuer_work(pwq, rescuer))
 			process_scheduled_works(rescuer);
 
-			/*
-			 * The above execution of rescued work items could
-			 * have created more to rescue through
-			 * pwq_activate_first_inactive() or chained
-			 * queueing.  Let's put @pwq back on mayday list so
-			 * that such back-to-back work items, which may be
-			 * being used to relieve memory pressure, don't
-			 * incur MAYDAY_INTERVAL delay inbetween.
-			 */
-			if (pwq->nr_active && need_to_create_worker(pool)) {
-				raw_spin_lock(&wq_mayday_lock);
-				/*
-				 * Queue iff somebody else hasn't queued it already.
-				 */
-				if (list_empty(&pwq->mayday_node)) {
-					get_pwq(pwq);
-					list_add_tail(&pwq->mayday_node, &wq->maydays);
-				}
-				raw_spin_unlock(&wq_mayday_lock);
-			}
-		}
-
 		/*
 		 * Leave this pool. Notify regular workers; otherwise, we end up
 		 * with 0 concurrency and stalling the execution.
-- 
2.19.1.6.gb485710b

[PATCH V4 2/4] workqueue: Process rescuer work items one-by-one using a cursor

[Lai Jiangshan]

From: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Previously, the rescuer scanned for all matching work items at once and
processed them within a single rescuer thread, which could cause one
blocking work item to stall all others.

Make the rescuer process work items one-by-one instead of slurping all
matches in a single pass.

Break the rescuer loop after finding and processing the first matching
work item, then restart the search to pick up the next. This gives
normal worker threads a chance to process other items which gives them
the opportinity to be processed instead of waiting on the rescuer's
queue and prevents a blocking work item from stalling the rest once
memory pressure is relieved.

Introduce a dummy cursor  work item to avoid potentially O(N^2)
rescans of the work list.  The marker records the resume position for
the next scan, eliminating redundant traversals.

Cc: ying chen <yc1082463@gmail.com>
Reported-by: ying chen <yc1082463@gmail.com>
Fixes: e22bee782b3b ("workqueue: implement concurrency managed dynamic worker pool")
Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
---
 kernel/workqueue.c | 55 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 48 insertions(+), 7 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 02386e6eb409..06cd3d6ff7e1 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -286,6 +286,7 @@ struct pool_workqueue {
 	struct list_head	pending_node;	/* LN: node on wq_node_nr_active->pending_pwqs */
 	struct list_head	pwqs_node;	/* WR: node on wq->pwqs */
 	struct list_head	mayday_node;	/* MD: node on wq->maydays */
+	struct work_struct	mayday_cursor;	/* L: cursor on pool->worklist */
 
 	u64			stats[PWQ_NR_STATS];
 
@@ -1120,6 +1121,12 @@ static struct worker *find_worker_executing_work(struct worker_pool *pool,
 	return NULL;
 }
 
+static void mayday_cursor_func(struct work_struct *work)
+{
+	/* should not be processed, only for marking position */
+	BUG();
+}
+
 /**
  * move_linked_works - move linked works to a list
  * @work: start of series of works to be scheduled
@@ -1182,6 +1189,16 @@ static bool assign_work(struct work_struct *work, struct worker *worker,
 
 	lockdep_assert_held(&pool->lock);
 
+	/* The cursor work should not be processed */
+	if (unlikely(work->func == mayday_cursor_func)) {
+		/* only worker_thread() can possibly take this branch */
+		WARN_ON_ONCE(worker->rescue_wq);
+		if (nextp)
+			*nextp = list_next_entry(work, entry);
+		list_del_init(&work->entry);
+		return false;
+	}
+
 	/*
 	 * A single work shouldn't be executed concurrently by multiple workers.
 	 * __queue_work() ensures that @work doesn't jump to a different pool
@@ -3436,22 +3453,33 @@ static int worker_thread(void *__worker)
 static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescuer)
 {
 	struct worker_pool *pool = pwq->pool;
+	struct work_struct *cursor = &pwq->mayday_cursor;
 	struct work_struct *work, *n;
 
+	/* search from the start or cursor if available */
+	if (list_empty(&cursor->entry)) {
+		work = list_first_entry(&pool->worklist, struct work_struct, entry);
+	} else {
+		work = list_next_entry(cursor, entry);
+		/* It will be at a new position or not need cursor anymore */
+		list_del_init(&cursor->entry);
+	}
+
 	/* need rescue? */
 	if (!pwq->nr_active || !need_to_create_worker(pool))
 		return false;
 
-	/*
-	 * Slurp in all works issued via this workqueue and
-	 * process'em.
-	 */
-	list_for_each_entry_safe(work, n, &pool->worklist, entry) {
-		if (get_work_pwq(work) == pwq && assign_work(work, rescuer, &n))
+	/* find the next work item to rescue */
+	list_for_each_entry_safe_from(work, n, &pool->worklist, entry) {
+		if (get_work_pwq(work) == pwq && assign_work(work, rescuer, &n)) {
 			pwq->stats[PWQ_STAT_RESCUED]++;
+			/* put the cursor for next search */
+			list_add_tail(&cursor->entry, &n->entry);
+			return true;
+		}
 	}
 
-	return !list_empty(&rescuer->scheduled);
+	return false;
 }
 
 /**
@@ -5135,6 +5163,19 @@ static void init_pwq(struct pool_workqueue *pwq, struct workqueue_struct *wq,
 	INIT_LIST_HEAD(&pwq->pwqs_node);
 	INIT_LIST_HEAD(&pwq->mayday_node);
 	kthread_init_work(&pwq->release_work, pwq_release_workfn);
+
+	/*
+	 * Set the dummy cursor work with valid function and get_work_pwq().
+	 *
+	 * The cursor work should only be in the pwq->pool->worklist, and
+	 * should not be treated as a processable work item.
+	 *
+	 * WORK_STRUCT_PENDING and WORK_STRUCT_INACTIVE just make it less
+	 * surprise for kernel debuging tools and reviewers.
+	 */
+	INIT_WORK(&pwq->mayday_cursor, mayday_cursor_func);
+	atomic_long_set(&pwq->mayday_cursor.data, (unsigned long)pwq |
+			WORK_STRUCT_PENDING | WORK_STRUCT_PWQ | WORK_STRUCT_INACTIVE);
 }
 
 /* sync @pwq with the current state of its associated wq and link it */
-- 
2.19.1.6.gb485710b

[PATCH V4 3/4] workqueue: Limit number of processed works in rescuer per turn

[Lai Jiangshan]

From: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Currently the rescuer keeps looping until all work on a PWQ is done, and
this may hurt fairness among PWQs, as the rescuer could remain stuck on
one PWQ indefinitely.

Introduce RESCUER_BATCH to control the maximum number of work items the
rescuer processes in each turn, and move on to other PWQs when the limit
is reached.

Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
---
 kernel/workqueue.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 06cd3d6ff7e1..7cec9755b4e1 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -117,6 +117,8 @@ enum wq_internal_consts {
 	MAYDAY_INTERVAL		= HZ / 10,	/* and then every 100ms */
 	CREATE_COOLDOWN		= HZ,		/* time to breath after fail */
 
+	RESCUER_BATCH		= 16,		/* process items per turn */
+
 	/*
 	 * Rescue workers are used only on emergencies and shared by
 	 * all cpus.  Give MIN_NICE.
@@ -3450,7 +3452,15 @@ static int worker_thread(void *__worker)
 	goto woke_up;
 }
 
-static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescuer)
+/*
+ * Try to assign one work item from @pwq to @rescuer.
+ *
+ * Returns true if a work item was successfully assigned, false otherwise.
+ * If @throttled and other PWQs are in mayday, requeue mayday for this PWQ
+ * and let the rescuer handle other PWQs first.
+ * If this is the only PWQ in mayday, process it regardless of @throttled.
+ */
+static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescuer, bool throttled)
 {
 	struct worker_pool *pool = pwq->pool;
 	struct work_struct *cursor = &pwq->mayday_cursor;
@@ -3471,7 +3481,21 @@ static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescu
 
 	/* find the next work item to rescue */
 	list_for_each_entry_safe_from(work, n, &pool->worklist, entry) {
-		if (get_work_pwq(work) == pwq && assign_work(work, rescuer, &n)) {
+		if (get_work_pwq(work) != pwq)
+		       continue;
+		/*
+		 * If throttled, update the cursor, requeue a mayday for this
+		 * PWQ, and move on to other PWQs. If there are no other PWQs
+		 * in mayday, continue processing this one.
+		 */
+		if (throttled && !list_empty(&pwq->wq->maydays)) {
+			list_add_tail(&cursor->entry, &work->entry);
+			raw_spin_lock(&wq_mayday_lock);		/* for wq->maydays */
+			send_mayday(work);
+			raw_spin_unlock(&wq_mayday_lock);
+			return false;
+		}
+		if (assign_work(work, rescuer, &n)) {
 			pwq->stats[PWQ_STAT_RESCUED]++;
 			/* put the cursor for next search */
 			list_add_tail(&cursor->entry, &n->entry);
@@ -3536,6 +3560,7 @@ static int rescuer_thread(void *__rescuer)
 		struct pool_workqueue *pwq = list_first_entry(&wq->maydays,
 					struct pool_workqueue, mayday_node);
 		struct worker_pool *pool = pwq->pool;
+		unsigned int count = 0;
 
 		__set_current_state(TASK_RUNNING);
 		list_del_init(&pwq->mayday_node);
@@ -3548,7 +3573,7 @@ static int rescuer_thread(void *__rescuer)
 
 		WARN_ON_ONCE(!list_empty(&rescuer->scheduled));
 
-		while (assign_rescuer_work(pwq, rescuer))
+		while (assign_rescuer_work(pwq, rescuer, ++count > RESCUER_BATCH))
 			process_scheduled_works(rescuer);
 
 		/*
-- 
2.19.1.6.gb485710b

[PATCH V4 4/4] workqueue: Process extra works in rescuer when there are no more to rescue

[Lai Jiangshan]

From: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Make the rescuer process more work on the last pwq when there are no
more to rescue for the whole workqueue to help the regular workers in
case it is a temporary memory pressure relief and to reduce relapse.

Also throttle the number of extra works.

Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
---
 kernel/workqueue.c | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 7cec9755b4e1..12d7d6503482 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -3459,6 +3459,9 @@ static int worker_thread(void *__worker)
  * If @throttled and other PWQs are in mayday, requeue mayday for this PWQ
  * and let the rescuer handle other PWQs first.
  * If this is the only PWQ in mayday, process it regardless of @throttled.
+ * If this is the last PWQ leaving mayday, but the pool is in semi-mayday
+ * (idle workers exist while memory pressure is still on), process it unless
+ * @throttled.
  */
 static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescuer, bool throttled)
 {
@@ -3475,10 +3478,38 @@ static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescu
 		list_del_init(&cursor->entry);
 	}
 
-	/* need rescue? */
-	if (!pwq->nr_active || !need_to_create_worker(pool))
+	/* have work items to rescue? */
+	if (!pwq->nr_active)
 		return false;
 
+	/* need rescue? */
+	if (!need_to_create_worker(pool)) {
+		/*
+		 * The pool has idle workers and doesn't need the rescuer, so it
+		 * could simply return false here.
+		 *
+		 * However, the memory pressure might not be fully relieved.
+		 * In PERCPU pool with concurrency enabled, having idle workers
+		 * does not necessarily mean memory pressure is gone; it may
+		 * simply mean regular workers have woken up, completed their
+		 * work, and gone idle again due to concurrency limits.
+		 *
+		 * In this case, those working workers may later sleep again,
+		 * the pool may run out of idle workers, and it will have to
+		 * allocate new ones and wait for the timer to send mayday,
+		 * causing unnecessary delay - especially if memory pressure
+		 * was never resolved throughout.
+		 *
+		 * Do more work if memory pressure is still on to reduce
+		 * relapse, using (pool->flags & POOL_MANAGER_ACTIVE), though
+		 * not precisely, unless there are other PWQs needing help or
+		 * this PWQ is throttled.
+		 */
+		if (!(pool->flags & POOL_MANAGER_ACTIVE) ||
+		    !list_empty(&pwq->wq->maydays) || throttled)
+			return false;
+	}
+
 	/* find the next work item to rescue */
 	list_for_each_entry_safe_from(work, n, &pool->worklist, entry) {
 		if (get_work_pwq(work) != pwq)
-- 
2.19.1.6.gb485710b

Re: [PATCH V4 1/4] workqueue: Loop over in rescuer until all its work is done

[Tejun Heo]

On Tue, Nov 25, 2025 at 02:36:14PM +0800, Lai Jiangshan wrote:
> From: Lai Jiangshan <jiangshan.ljs@antgroup.com>
> 
> Simplify the rescuer work by looping directly in the rescuer rather than
> adding the pwq back to the maydays list. This also helps when
> max_requests is 1 or small but pwq->inactive_works has a large number of
> pending work items.
> 
> This might hurt fairness among PWQs and the rescuer could end up being
> stuck on one PWQ indefinitely, but the rescuer's objective is to make
> forward progress rather than ensure fairness.
> 
> Fairness can be further improved in future by assigning work items to
> the rescuer one by one and this is a temporary change to ease the
> transition.

Again, I'd just note that this is temporary removal rather than trying
justify that fairness doesn't matter because I think it does (not in terms
of theoretical correctness but in the practical approach of avoiding
unnecessary surprises) and the behavior is restored by a later patch.

Thanks.

-- 
tejun

Re: [PATCH V4 3/4] workqueue: Limit number of processed works in rescuer per turn

[Tejun Heo]

On Tue, Nov 25, 2025 at 02:36:16PM +0800, Lai Jiangshan wrote:
...
> +		while (assign_rescuer_work(pwq, rescuer, ++count > RESCUER_BATCH))
>  			process_scheduled_works(rescuer);

Can we something like the following instead?

        while (assign_rescuer_work(pwq, rescuer, &count))

It just feels odd to for the caller to decide "you should stop" and then
taking actions on the return value of the callee. Alternatively, just
separate out the pwq rotation into a separate function, so that the caller
can do

        while (assign_rescuer_work(..)) {
                process_scheduled_works(rescuer);
                if (++count > RESCUER_BATCH) {
                        rotate mayday list;
                        break;
                }
        }

Thanks.

-- 
tejun

Re: [PATCH V4 3/4] workqueue: Limit number of processed works in rescuer per turn

[Lai Jiangshan]

On Wed, Dec 3, 2025 at 2:16 AM Tejun Heo <tj@kernel.org> wrote:
>
> On Tue, Nov 25, 2025 at 02:36:16PM +0800, Lai Jiangshan wrote:
> ...
> > +             while (assign_rescuer_work(pwq, rescuer, ++count > RESCUER_BATCH))
> >                       process_scheduled_works(rescuer);
>
> Can we something like the following instead?
>
>         while (assign_rescuer_work(pwq, rescuer, &count))
>
> It just feels odd to for the caller to decide "you should stop" and then
> taking actions on the return value of the callee.

I think the nanming of "limited" or "throttled" gives the wrong
impression of "you should stop". How about renaming it again to
"rotate" or "prefer_rotate", meaning "prefer to rotate instead of
assigning work from this pwq"?

> Alternatively, just
> separate out the pwq rotation into a separate function, so that the caller
> can do
>
>         while (assign_rescuer_work(..)) {
>                 process_scheduled_works(rescuer);
>                 if (++count > RESCUER_BATCH) {
>                         rotate mayday list;

The current implementation of the cursor cannot be processed by workers as a
regular work item, which means it has no pwq reference and cannot be left
behind if the pwq is not put back on the mayday list. As a result, the code
here has to handle the cursor explicitly, which adds extra burden to
understanding the code.

I prefer using send_mayday() as in this patch (which is naturally available
in assign_rescuer_work() but requires extra code here) rather than the
rotate code used before this patchset.

In other words, I prefer the rotate logic - which decides whether to assign work
or not - to live in assign_rescuer_work(). I’m also okay with moving the
counting logic into assign_rescuer_work(),  as
assign_rescuer_work(pwq, rescuer, &count);

Thanks
Lai

>                         break;
>                 }
>         }
>
> Thanks.
>
> --
> tejun

Re: [PATCH V4 3/4] workqueue: Limit number of processed works in rescuer per turn

[Tejun Heo]

Hello, Lai.

On Wed, Dec 03, 2025 at 11:22:32AM +0800, Lai Jiangshan wrote:
> On Wed, Dec 3, 2025 at 2:16 AM Tejun Heo <tj@kernel.org> wrote:
...
> > Alternatively, just
> > separate out the pwq rotation into a separate function, so that the caller
> > can do
> >
> >         while (assign_rescuer_work(..)) {
> >                 process_scheduled_works(rescuer);
> >                 if (++count > RESCUER_BATCH) {
> >                         rotate mayday list;
> 
> The current implementation of the cursor cannot be processed by workers as a
> regular work item, which means it has no pwq reference and cannot be left
> behind if the pwq is not put back on the mayday list. As a result, the code
> here has to handle the cursor explicitly, which adds extra burden to
> understanding the code.

I'm probably missing something but what prevents rescuer_thread() from just
not calling assign_rescuer_work() and just call send_mayday() on the pwq?
send_mayday() currently takes @work but all it cares about is pwq, so we can
just pass in pwq.

Thanks.

-- 
tejun