From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1713A2E88A4 for ; Thu, 13 Aug 2026 04:53:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786596804; cv=none; b=dUu6gBpgEa3rEK1iGyX0AfRpj9WkZxS/KEadZq3dy90E0hSsuAD/1O78LBXQLKXSdsCEOW4OBKt3TYxuaHgw+QOuFhLnJwDb2WZpssPXk3Kgp9YI1UuoEjwLftglUm9LChgckwGpl4HkaDUkRjH1Tlwni7R22lMW31lvsZBk7go= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786596804; c=relaxed/simple; bh=Nx5utJ30XGoOQYw2fNLjz+zcUVg/hemONiUPe/7xbFk=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=ZGKzFjoL0HB1mLTmxFpq8qTsgcdIY+hoW3/OoAzS4o1mW9UDyb0yIHMfyKa7gq+WfdyYMFvNZsOfIJ7H5GyPrKRDZG5lGL0FQcrHqp4Em6Tgwm+0Klqf0XB5ROcOveY1jqbQrtgaKZCbkl3SnGGw5G9LEirK5v4xB/Rzkj7+dqo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JEkXzOZ1; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JEkXzOZ1" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-84e507b079dso1257344b3a.0 for ; Wed, 12 Aug 2026 21:53:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786596802; x=1787201602; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=xNOICvrU8VxNji6mzFKbtvng8Ry1jjvkuzRDfCw8NGw=; b=JEkXzOZ1p0kuGx3UR1yv1nb30hJ5vFcJ7FyV8d949619OAIszEjgd6wbajcMIxjGNk 1ikCJH41hwHeGpNBFcnWVgBC7ln0EieQrRwq+8nBl5gXZXyFX6KEKMQnWT+J767dkOLc z13QWJL/K3miUXJzjQLlfFkCs1VSlHsSA3I2uD8H3bfMqJ1gPT8G1D96eb/Tx0CX/0yt L1DzOuy0zf3aXbJ/BPMZAfsg7HPdXL3XEfxCRzu9/I6Qo97rguY+SCbt4pAosHJZcQUX taI/xoz9Bzb7l/69vWrtRpCi3nHit462t8n52VT9ju+6sXCKoQYjqsF/ppNJ8DGm51FK 0x/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786596802; x=1787201602; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xNOICvrU8VxNji6mzFKbtvng8Ry1jjvkuzRDfCw8NGw=; b=dtV06rtMf4D2/X5tinnRaepBamZB9CjXMXj9yErLX9WGkMaCNadzZD3Uh7yU3IIenk 2ZldEvMnqQEBeqFfVlEqcJ1WLBGlGg2oEeJvTk6qd+kAMhD5V6EL9B6fOu6u3ZNaSZK+ ulvSuY8rZUnebF2KD5IzJnMcLIun/ABZWIOLlRLPdjHmyVpJ6raDYi9b7IPwTQnfE5Es K61NwJMNzEg+0h1w/X6/vg0HiMXCiv10BBJCXcuNns2KDV2HBBupwx8jdraEPgeSLect /Ehn5qszfRnpIyZkHfOZvG9V3etuirvzmnc+BH0Qzju6nCiTr5zXBdkOfZljfKTBLuI0 /fWQ== X-Gm-Message-State: AOJu0YyIWILJJyrfH1FwZzevQOXNjUjCvA7tG2tDNTVana9vveiNWnyw vvQNo01cWiRBmrGecqoDFRfzh3eXGUpJ7h0vttHBD68Vfyms+4CcMxPdKbzaWw== X-Gm-Gg: AR+sD101Ilqqohizy8z2Ydr5n0FSBgqylf9/brGP8TxtZpPXQQt57I60yQsfRjakgml Y0nbk7I4CBw+EedE+BMjvzbUC2m3XJ9KxyxXdgEUUk/DYYYnlSHW5N8lsGGKRec8y+4+PHqXkgI 3fmHaN6f6ymcUHpnvOI6RufDgb1ZiDgqDkjbkpSEWHsL2HrsDCRi1k0LixRJdxt60W2EHEG8dsj BHQ8U9bncmnAxZSkuLm4ENWU8uOW4VRSh9VAdSLQPiRhRsa71cDBYMc7OnpVSNNsJ3+fxrByq7E 8ePFk3pTUGPnIr0GU9kQ4Zg+MgcISyGhFgfMqDfqqMblsuUSYC57Zrsx/2JNNzeWWkmr7epxxmx CjqA1/lOUKcVtEzKlz3Wffu+k8+4xpnYucI3PvxFaRFqDGQuGcu+GlTxVgXU4cyxcuu3t+Pt6Ln eSzzDLNDR2Kj1/9CS7gCZVl2Lzu7Ye0N2JKx1VCj+n1oJWqJ2S3Q/p7PWE8FRxvF0vlVCGuqWX+ 2IE97n94lKWIlA= X-Received: by 2002:a05:6a00:299a:b0:84e:2722:5d98 with SMTP id d2e1a72fcca58-84fc7514ce0mr3108402b3a.19.1786596802052; Wed, 12 Aug 2026 21:53:22 -0700 (PDT) Received: from PC-2B0BA19500175.company.local ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84fcb6a371csm140633b3a.1.2026.08.12.21.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Aug 2026 21:53:21 -0700 (PDT) From: Lu Wang To: linux-kernel@vger.kernel.org Cc: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, yu.c.chen@intel.com, tim.c.chen@linux.intel.com, chen.yu@linux.dev, kprateek.nayak@amd.com, Lu Wang Subject: [PATCH v3] sched/cache: honor migrate_llc_task semantics in active load balance Date: Thu, 13 Aug 2026 12:52:41 +0800 Message-ID: <20260813045241.3039862-1-wanglu.priv@gmail.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit CAS introduced the migrate_llc_task migration type to direct tasks toward their preferred LLC, but its semantics can be lost when passive load balance falls back to active load balance. This may allow ALB to select a candidate whose preferred LLC does not match the destination, moving it away from its preferred LLC. Example scenario: src_rq has two runnable tasks, p1 and p2. p1 prefers dst_rq (dst_llc), while p2 prefers src_rq (src_llc). In this case, migrate_llc_task is set because src_rq has at least one task, p1, that wants to migrate to dst_rq. In ALB, can_migrate_task() finds p2 and returns true for it, thus moving p2 out of its preferred LLC. Solution: The CPU stopper in ALB constructs a fresh lb_env that does not inherit migration_type from the passive load-balance pass. Two approaches are possible: (a) Add a new member to struct rq so ALB can inherit migrate_llc_task from the passive LB that triggered it. (b) Define a new flag LBF_ACTIVE_LB_LLC and select the stopper callback at kick time to preserve the migration semantics across the asynchronous boundary. We choose (b) because it avoids passing migration_type through the stopper, which would affect the meaning of migration_type for delayed-dequeue tasks. Fixes: e4c9a4cb244a ("sched/cache: Add migrate_llc_task migration type for cache-aware balancing") Suggested-by: "Chen, Yu C" Reviewed-by: Tim Chen Reviewed-by: Chen Yu Signed-off-by: Lu Wang --- Changes in v3: - Updated commit message and helper comment description, no code changes - Link to V2: https://lore.kernel.org/all/20260809105343.1189051-1-wanglu.priv@gmail.com/ Changes in v2: - Select the stopper callback at kick time to preserve migrate_llc_task semantics in active load balance without passing migration_type across the stopper, which affects delayed-dequeue tasks. - Link to V1: https://lore.kernel.org/all/20260801122252.2476258-1-wanglu.priv@gmail.com/ kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 6 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6..13f002a96 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10254,6 +10254,7 @@ enum migration_type { #define LBF_SOME_PINNED 0x08 #define LBF_ACTIVE_LB 0x10 #define LBF_LLC_PINNED 0x20 +#define LBF_ACTIVE_LB_LLC 0x40 struct lb_env { struct sched_domain *sd; @@ -10645,6 +10646,21 @@ alb_break_llc(struct lb_env *env) return false; } +/* + * Returns true if p's preferred LLC does not match the destination CPU + * under migrate_llc_task semantics. Passive LB passes migrate_llc_task + * in env->migration_type, while active LB carries LBF_ACTIVE_LB_LLC in + * env->flags to avoid overwriting env->migration_type. + */ +static inline bool +migrate_llc_task_wrong_dst(struct task_struct *p, struct lb_env *env) +{ + return sched_cache_enabled() && + (env->migration_type == migrate_llc_task || + env->flags & LBF_ACTIVE_LB_LLC) && + READ_ONCE(p->preferred_llc) != llc_id(env->dst_cpu); +} + /* * Check if migrating task p from env->src_cpu to * env->dst_cpu breaks LLC localiy. @@ -10673,8 +10689,7 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env) * run on env->dst_cpu, skip the tasks do not prefer * env->dst_cpu, and find the one that prefers. */ - if (env->migration_type == migrate_llc_task && - READ_ONCE(p->preferred_llc) != llc_id(env->dst_cpu)) + if (migrate_llc_task_wrong_dst(p, env)) return true; if (can_migrate_llc_task(env->src_cpu, @@ -10697,6 +10712,12 @@ alb_break_llc(struct lb_env *env) return false; } +static inline bool +migrate_llc_task_wrong_dst(struct task_struct *p, struct lb_env *env) +{ + return false; +} + static inline bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env) { @@ -10796,7 +10817,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) * 4) too many balance attempts have failed. */ if (env->flags & LBF_ACTIVE_LB) - return 1; + return !migrate_llc_task_wrong_dst(p, env); degrades = migrate_degrades_locality(p, env); if (!degrades) { @@ -13156,6 +13177,20 @@ static int need_active_balance(struct lb_env *env) } static int active_load_balance_cpu_stop(void *data); +static int active_load_balance_llc_cpu_stop(void *data); + +/* + * migration_type is checked elsewhere to decide migration policy, so + * it shouldn't be repurposed just to flag an LLC-directed active + * balance across the stopper. Pick the callback here instead. + */ +static inline cpu_stop_fn_t alb_stop_fn(struct lb_env *env) +{ + if (env->migration_type == migrate_llc_task) + return active_load_balance_llc_cpu_stop; + + return active_load_balance_cpu_stop; +} static int should_we_balance(struct lb_env *env) { @@ -13492,7 +13527,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, raw_spin_rq_unlock_irqrestore(busiest, flags); if (active_balance) { stop_one_cpu_nowait(cpu_of(busiest), - active_load_balance_cpu_stop, busiest, + alb_stop_fn(&env), busiest, &busiest->active_balance_work); } preempt_enable(); @@ -13603,7 +13638,7 @@ update_next_balance(struct sched_domain *sd, unsigned long *next_balance) * least 1 task to be running on each physical CPU where possible, and * avoids physical / logical imbalances. */ -static int active_load_balance_cpu_stop(void *data) +static int __active_load_balance_cpu_stop(void *data, unsigned int lb_flags) { struct rq *busiest_rq = data; int busiest_cpu = cpu_of(busiest_rq); @@ -13653,7 +13688,7 @@ static int active_load_balance_cpu_stop(void *data) .src_cpu = busiest_rq->cpu, .src_rq = busiest_rq, .idle = CPU_IDLE, - .flags = LBF_ACTIVE_LB, + .flags = LBF_ACTIVE_LB | lb_flags, }; schedstat_inc(sd->alb_count); @@ -13681,6 +13716,16 @@ static int active_load_balance_cpu_stop(void *data) return 0; } +static int active_load_balance_cpu_stop(void *data) +{ + return __active_load_balance_cpu_stop(data, 0); +} + +static int active_load_balance_llc_cpu_stop(void *data) +{ + return __active_load_balance_cpu_stop(data, LBF_ACTIVE_LB_LLC); +} + /* * Scale the max sched_balance_rq interval with the number of CPUs in the system. * This trades load-balance latency on larger machines for less cross talk. -- 2.43.0