* [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
@ 2025-05-21 3:45 Zhongkun He
2025-05-21 17:15 ` Tejun Heo
0 siblings, 1 reply; 6+ messages in thread
From: Zhongkun He @ 2025-05-21 3:45 UTC (permalink / raw)
To: tj, hannes, longman; +Cc: cgroups, linux-kernel, muchun.song, Zhongkun He
Setting the cpuset.mems in cgroup v2 can trigger memory
migrate in cpuset. This behavior is fine for newly created
cgroups but it can cause issues for the existing cgroups.
In our scenario, modifying the cpuset.mems setting during
peak times frequently leads to noticeable service latency
or stuttering.
It is important to have a consistent set of behavior for
both cpus and memory. But it does cause issues at times,
so we would like to have a flexible option.
This idea is from the non-blocking limit setting option in
memory control.
https://lore.kernel.org/all/20250506232833.3109790-1-shakeel.butt@linux.dev/
Signed-off-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
---
Documentation/admin-guide/cgroup-v2.rst | 7 +++++++
kernel/cgroup/cpuset-internal.h | 6 ++++++
kernel/cgroup/cpuset.c | 7 +++++++
3 files changed, 20 insertions(+)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 1a16ce68a4d7..d9e8e2a770af 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2408,6 +2408,13 @@ Cpuset Interface Files
a need to change "cpuset.mems" with active tasks, it shouldn't
be done frequently.
+ If cpuset.mems is opened with O_NONBLOCK then the migration is
+ bypassed. This is useful for admin processes that need to adjust
+ the cpuset.mems dynamically without blocking. However, there is
+ a risk that previously allocated pages are not within the new
+ cpuset.mems range, which may be altered by move_pages syscall or
+ numa_balance.
+
cpuset.mems.effective
A read-only multiple values file which exists on all
cpuset-enabled cgroups.
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 383963e28ac6..5686bb08c4fe 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -162,6 +162,9 @@ struct cpuset {
/* partition root state */
int partition_root_state;
+ /* Do not migrate memory when modifying cpuset.mems this time */
+ bool skip_migration_once;
+
/*
* number of SCHED_DEADLINE tasks attached to this cpuset, so that we
* know when to rebuild associated root domain bandwidth information.
@@ -227,6 +230,9 @@ static inline int is_sched_load_balance(const struct cpuset *cs)
static inline int is_memory_migrate(const struct cpuset *cs)
{
+ if (cs->skip_migration_once)
+ return 0;
+
return test_bit(CS_MEMORY_MIGRATE, &cs->flags);
}
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 24b70ea3e6ce..f43d7b291cde 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3208,7 +3208,14 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
retval = update_exclusive_cpumask(cs, trialcs, buf);
break;
case FILE_MEMLIST:
+ if (of->file->f_flags & O_NONBLOCK)
+ cs->skip_migration_once = true;
+
retval = update_nodemask(cs, trialcs, buf);
+
+ /* Restore skip_migration */
+ if (cs->skip_migration_once)
+ cs->skip_migration_once = false;
break;
default:
retval = -EINVAL;
--
2.39.5
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
2025-05-21 3:45 [PATCH] cpuset: introduce non-blocking cpuset.mems setting option Zhongkun He
@ 2025-05-21 17:15 ` Tejun Heo
0 siblings, 0 replies; 6+ messages in thread
From: Tejun Heo @ 2025-05-21 17:15 UTC (permalink / raw)
To: Zhongkun He; +Cc: hannes, longman, cgroups, linux-kernel, muchun.song
On Wed, May 21, 2025 at 11:45:27AM +0800, Zhongkun He wrote:
> Setting the cpuset.mems in cgroup v2 can trigger memory
> migrate in cpuset. This behavior is fine for newly created
> cgroups but it can cause issues for the existing cgroups.
> In our scenario, modifying the cpuset.mems setting during
> peak times frequently leads to noticeable service latency
> or stuttering.
>
> It is important to have a consistent set of behavior for
> both cpus and memory. But it does cause issues at times,
> so we would like to have a flexible option.
>
> This idea is from the non-blocking limit setting option in
> memory control.
>
> https://lore.kernel.org/all/20250506232833.3109790-1-shakeel.butt@linux.dev/
>
> Signed-off-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
> ---
> Documentation/admin-guide/cgroup-v2.rst | 7 +++++++
> kernel/cgroup/cpuset-internal.h | 6 ++++++
> kernel/cgroup/cpuset.c | 7 +++++++
> 3 files changed, 20 insertions(+)
>
> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index 1a16ce68a4d7..d9e8e2a770af 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> @@ -2408,6 +2408,13 @@ Cpuset Interface Files
> a need to change "cpuset.mems" with active tasks, it shouldn't
> be done frequently.
>
> + If cpuset.mems is opened with O_NONBLOCK then the migration is
> + bypassed. This is useful for admin processes that need to adjust
> + the cpuset.mems dynamically without blocking. However, there is
> + a risk that previously allocated pages are not within the new
> + cpuset.mems range, which may be altered by move_pages syscall or
> + numa_balance.
As said in the other thread, nack on this approach.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
2025-05-20 3:15 Zhongkun He
2025-05-20 13:13 ` kernel test robot
2025-05-20 13:25 ` kernel test robot
@ 2025-05-20 13:34 ` Waiman Long
2 siblings, 0 replies; 6+ messages in thread
From: Waiman Long @ 2025-05-20 13:34 UTC (permalink / raw)
To: Zhongkun He, tj, hannes; +Cc: cgroups, linux-kernel, muchun.song
On 5/19/25 11:15 PM, Zhongkun He wrote:
> Setting the cpuset.mems in cgroup v2 can trigger memory
> migrate in cpuset. This behavior is fine for newly created
> cgroups but it can cause issues for the existing cgroups.
> In our scenario, modifying the cpuset.mems setting during
> peak times frequently leads to noticeable service latency
> or stuttering.
>
> It is important to have a consistent set of behavior for
> both cpus and memory. But it does cause issues at times,
> so we would hope to have a flexible option.
>
> This idea is from the non-blocking limit setting option in
> memory control.
>
> https://lore.kernel.org/all/20250506232833.3109790-1-shakeel.butt@linux.dev/
>
> Signed-off-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
> ---
> Documentation/admin-guide/cgroup-v2.rst | 7 +++++++
> kernel/cgroup/cpuset.c | 11 +++++++++++
> 2 files changed, 18 insertions(+)
>
> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index 1a16ce68a4d7..d9e8e2a770af 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> @@ -2408,6 +2408,13 @@ Cpuset Interface Files
> a need to change "cpuset.mems" with active tasks, it shouldn't
> be done frequently.
>
> + If cpuset.mems is opened with O_NONBLOCK then the migration is
> + bypassed. This is useful for admin processes that need to adjust
> + the cpuset.mems dynamically without blocking. However, there is
> + a risk that previously allocated pages are not within the new
> + cpuset.mems range, which may be altered by move_pages syscall or
> + numa_balance.
> +
> cpuset.mems.effective
> A read-only multiple values file which exists on all
> cpuset-enabled cgroups.
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 24b70ea3e6ce..2a0867e0c6d2 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -3208,7 +3208,18 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
> retval = update_exclusive_cpumask(cs, trialcs, buf);
> break;
> case FILE_MEMLIST:
> + bool skip_migrate_once = false;
> +
> + if ((of->file->f_flags & O_NONBLOCK) &&
> + is_memory_migrate(cs) &&
> + !cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 0))
> + skip_migrate_once = true;
> +
> retval = update_nodemask(cs, trialcs, buf);
> +
> + /* Restore the migrate flag */
> + if (skip_migrate_once)
> + cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 1);
> break;
> default:
> retval = -EINVAL;
I would prefer to temporarily make is_memory_migrate() helper return
false by also checking an internal variable, for example, instead of
messing with the cpuset flags.
Cheers,
Longman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
2025-05-20 3:15 Zhongkun He
2025-05-20 13:13 ` kernel test robot
@ 2025-05-20 13:25 ` kernel test robot
2025-05-20 13:34 ` Waiman Long
2 siblings, 0 replies; 6+ messages in thread
From: kernel test robot @ 2025-05-20 13:25 UTC (permalink / raw)
To: Zhongkun He, tj, hannes, longman
Cc: llvm, oe-kbuild-all, cgroups, linux-kernel, muchun.song, Zhongkun He
Hi Zhongkun,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tj-cgroup/for-next]
[also build test WARNING on linus/master v6.15-rc7 next-20250516]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Zhongkun-He/cpuset-introduce-non-blocking-cpuset-mems-setting-option/20250520-111737
base: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next
patch link: https://lore.kernel.org/r/20250520031552.1931598-1-hezhongkun.hzk%40bytedance.com
patch subject: [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
config: s390-randconfig-002-20250520 (https://download.01.org/0day-ci/archive/20250520/202505202112.tmU9BTzA-lkp@intel.com/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project f819f46284f2a79790038e1f6649172789734ae8)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250520/202505202112.tmU9BTzA-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202505202112.tmU9BTzA-lkp@intel.com/
All warnings (new ones prefixed by >>):
>> kernel/cgroup/cpuset.c:3246:3: warning: label followed by a declaration is a C23 extension [-Wc23-extensions]
3246 | bool skip_migrate_once = false;
| ^
1 warning generated.
vim +3246 kernel/cgroup/cpuset.c
3215
3216 /*
3217 * Common handling for a write to a "cpus" or "mems" file.
3218 */
3219 ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
3220 char *buf, size_t nbytes, loff_t off)
3221 {
3222 struct cpuset *cs = css_cs(of_css(of));
3223 struct cpuset *trialcs;
3224 int retval = -ENODEV;
3225
3226 buf = strstrip(buf);
3227 cpus_read_lock();
3228 mutex_lock(&cpuset_mutex);
3229 if (!is_cpuset_online(cs))
3230 goto out_unlock;
3231
3232 trialcs = alloc_trial_cpuset(cs);
3233 if (!trialcs) {
3234 retval = -ENOMEM;
3235 goto out_unlock;
3236 }
3237
3238 switch (of_cft(of)->private) {
3239 case FILE_CPULIST:
3240 retval = update_cpumask(cs, trialcs, buf);
3241 break;
3242 case FILE_EXCLUSIVE_CPULIST:
3243 retval = update_exclusive_cpumask(cs, trialcs, buf);
3244 break;
3245 case FILE_MEMLIST:
> 3246 bool skip_migrate_once = false;
3247
3248 if ((of->file->f_flags & O_NONBLOCK) &&
3249 is_memory_migrate(cs) &&
3250 !cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 0))
3251 skip_migrate_once = true;
3252
3253 retval = update_nodemask(cs, trialcs, buf);
3254
3255 /* Restore the migrate flag */
3256 if (skip_migrate_once)
3257 cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 1);
3258 break;
3259 default:
3260 retval = -EINVAL;
3261 break;
3262 }
3263
3264 free_cpuset(trialcs);
3265 if (force_sd_rebuild)
3266 rebuild_sched_domains_locked();
3267 out_unlock:
3268 mutex_unlock(&cpuset_mutex);
3269 cpus_read_unlock();
3270 flush_workqueue(cpuset_migrate_mm_wq);
3271 return retval ?: nbytes;
3272 }
3273
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
2025-05-20 3:15 Zhongkun He
@ 2025-05-20 13:13 ` kernel test robot
2025-05-20 13:25 ` kernel test robot
2025-05-20 13:34 ` Waiman Long
2 siblings, 0 replies; 6+ messages in thread
From: kernel test robot @ 2025-05-20 13:13 UTC (permalink / raw)
To: Zhongkun He, tj, hannes, longman
Cc: oe-kbuild-all, cgroups, linux-kernel, muchun.song, Zhongkun He
Hi Zhongkun,
kernel test robot noticed the following build errors:
[auto build test ERROR on tj-cgroup/for-next]
[also build test ERROR on linus/master v6.15-rc7 next-20250516]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Zhongkun-He/cpuset-introduce-non-blocking-cpuset-mems-setting-option/20250520-111737
base: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next
patch link: https://lore.kernel.org/r/20250520031552.1931598-1-hezhongkun.hzk%40bytedance.com
patch subject: [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
config: sparc64-randconfig-001-20250520 (https://download.01.org/0day-ci/archive/20250520/202505202106.sXzGXeU4-lkp@intel.com/config)
compiler: sparc64-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250520/202505202106.sXzGXeU4-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202505202106.sXzGXeU4-lkp@intel.com/
All errors (new ones prefixed by >>):
kernel/cgroup/cpuset.c: In function 'cpuset_write_resmask':
>> kernel/cgroup/cpuset.c:3246:3: error: a label can only be part of a statement and a declaration is not a statement
bool skip_migrate_once = false;
^~~~
vim +3246 kernel/cgroup/cpuset.c
3215
3216 /*
3217 * Common handling for a write to a "cpus" or "mems" file.
3218 */
3219 ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
3220 char *buf, size_t nbytes, loff_t off)
3221 {
3222 struct cpuset *cs = css_cs(of_css(of));
3223 struct cpuset *trialcs;
3224 int retval = -ENODEV;
3225
3226 buf = strstrip(buf);
3227 cpus_read_lock();
3228 mutex_lock(&cpuset_mutex);
3229 if (!is_cpuset_online(cs))
3230 goto out_unlock;
3231
3232 trialcs = alloc_trial_cpuset(cs);
3233 if (!trialcs) {
3234 retval = -ENOMEM;
3235 goto out_unlock;
3236 }
3237
3238 switch (of_cft(of)->private) {
3239 case FILE_CPULIST:
3240 retval = update_cpumask(cs, trialcs, buf);
3241 break;
3242 case FILE_EXCLUSIVE_CPULIST:
3243 retval = update_exclusive_cpumask(cs, trialcs, buf);
3244 break;
3245 case FILE_MEMLIST:
> 3246 bool skip_migrate_once = false;
3247
3248 if ((of->file->f_flags & O_NONBLOCK) &&
3249 is_memory_migrate(cs) &&
3250 !cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 0))
3251 skip_migrate_once = true;
3252
3253 retval = update_nodemask(cs, trialcs, buf);
3254
3255 /* Restore the migrate flag */
3256 if (skip_migrate_once)
3257 cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 1);
3258 break;
3259 default:
3260 retval = -EINVAL;
3261 break;
3262 }
3263
3264 free_cpuset(trialcs);
3265 if (force_sd_rebuild)
3266 rebuild_sched_domains_locked();
3267 out_unlock:
3268 mutex_unlock(&cpuset_mutex);
3269 cpus_read_unlock();
3270 flush_workqueue(cpuset_migrate_mm_wq);
3271 return retval ?: nbytes;
3272 }
3273
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] cpuset: introduce non-blocking cpuset.mems setting option
@ 2025-05-20 3:15 Zhongkun He
2025-05-20 13:13 ` kernel test robot
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Zhongkun He @ 2025-05-20 3:15 UTC (permalink / raw)
To: tj, hannes, longman; +Cc: cgroups, linux-kernel, muchun.song, Zhongkun He
Setting the cpuset.mems in cgroup v2 can trigger memory
migrate in cpuset. This behavior is fine for newly created
cgroups but it can cause issues for the existing cgroups.
In our scenario, modifying the cpuset.mems setting during
peak times frequently leads to noticeable service latency
or stuttering.
It is important to have a consistent set of behavior for
both cpus and memory. But it does cause issues at times,
so we would hope to have a flexible option.
This idea is from the non-blocking limit setting option in
memory control.
https://lore.kernel.org/all/20250506232833.3109790-1-shakeel.butt@linux.dev/
Signed-off-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
---
Documentation/admin-guide/cgroup-v2.rst | 7 +++++++
kernel/cgroup/cpuset.c | 11 +++++++++++
2 files changed, 18 insertions(+)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 1a16ce68a4d7..d9e8e2a770af 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2408,6 +2408,13 @@ Cpuset Interface Files
a need to change "cpuset.mems" with active tasks, it shouldn't
be done frequently.
+ If cpuset.mems is opened with O_NONBLOCK then the migration is
+ bypassed. This is useful for admin processes that need to adjust
+ the cpuset.mems dynamically without blocking. However, there is
+ a risk that previously allocated pages are not within the new
+ cpuset.mems range, which may be altered by move_pages syscall or
+ numa_balance.
+
cpuset.mems.effective
A read-only multiple values file which exists on all
cpuset-enabled cgroups.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 24b70ea3e6ce..2a0867e0c6d2 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3208,7 +3208,18 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
retval = update_exclusive_cpumask(cs, trialcs, buf);
break;
case FILE_MEMLIST:
+ bool skip_migrate_once = false;
+
+ if ((of->file->f_flags & O_NONBLOCK) &&
+ is_memory_migrate(cs) &&
+ !cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 0))
+ skip_migrate_once = true;
+
retval = update_nodemask(cs, trialcs, buf);
+
+ /* Restore the migrate flag */
+ if (skip_migrate_once)
+ cpuset_update_flag(CS_MEMORY_MIGRATE, cs, 1);
break;
default:
retval = -EINVAL;
--
2.39.5
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-05-21 17:15 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-05-21 3:45 [PATCH] cpuset: introduce non-blocking cpuset.mems setting option Zhongkun He
2025-05-21 17:15 ` Tejun Heo
-- strict thread matches above, loose matches on Subject: below --
2025-05-20 3:15 Zhongkun He
2025-05-20 13:13 ` kernel test robot
2025-05-20 13:25 ` kernel test robot
2025-05-20 13:34 ` Waiman Long
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®