[PATCH V4 4/4] workqueue: Process extra works in rescuer when there are no more to rescue

[Lai Jiangshan <jiangshanlai@gmail.com>]

From: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Make the rescuer process more work on the last pwq when there are no
more to rescue for the whole workqueue to help the regular workers in
case it is a temporary memory pressure relief and to reduce relapse.

Also throttle the number of extra works.

Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
---
 kernel/workqueue.c | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 7cec9755b4e1..12d7d6503482 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -3459,6 +3459,9 @@ static int worker_thread(void *__worker)
  * If @throttled and other PWQs are in mayday, requeue mayday for this PWQ
  * and let the rescuer handle other PWQs first.
  * If this is the only PWQ in mayday, process it regardless of @throttled.
+ * If this is the last PWQ leaving mayday, but the pool is in semi-mayday
+ * (idle workers exist while memory pressure is still on), process it unless
+ * @throttled.
  */
 static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescuer, bool throttled)
 {
@@ -3475,10 +3478,38 @@ static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker *rescu
 		list_del_init(&cursor->entry);
 	}
 
-	/* need rescue? */
-	if (!pwq->nr_active || !need_to_create_worker(pool))
+	/* have work items to rescue? */
+	if (!pwq->nr_active)
 		return false;
 
+	/* need rescue? */
+	if (!need_to_create_worker(pool)) {
+		/*
+		 * The pool has idle workers and doesn't need the rescuer, so it
+		 * could simply return false here.
+		 *
+		 * However, the memory pressure might not be fully relieved.
+		 * In PERCPU pool with concurrency enabled, having idle workers
+		 * does not necessarily mean memory pressure is gone; it may
+		 * simply mean regular workers have woken up, completed their
+		 * work, and gone idle again due to concurrency limits.
+		 *
+		 * In this case, those working workers may later sleep again,
+		 * the pool may run out of idle workers, and it will have to
+		 * allocate new ones and wait for the timer to send mayday,
+		 * causing unnecessary delay - especially if memory pressure
+		 * was never resolved throughout.
+		 *
+		 * Do more work if memory pressure is still on to reduce
+		 * relapse, using (pool->flags & POOL_MANAGER_ACTIVE), though
+		 * not precisely, unless there are other PWQs needing help or
+		 * this PWQ is throttled.
+		 */
+		if (!(pool->flags & POOL_MANAGER_ACTIVE) ||
+		    !list_empty(&pwq->wq->maydays) || throttled)
+			return false;
+	}
+
 	/* find the next work item to rescue */
 	list_for_each_entry_safe_from(work, n, &pool->worklist, entry) {
 		if (get_work_pwq(work) != pwq)
-- 
2.19.1.6.gb485710b