Re: [PATCH v5 19/29] cgroup/cpuset: Wait for an attach before rebuilding the sched domains

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

Hi Waiman,

> I agree that rebuild_sched_domains() should wait for an in-progress
> attach operation. The change to cpuset_full_lock(), however, is not
> really necessary. This cpuset_full_lock() is now mainly used for
> operations like cpuset_css_online() that are initiated from the cgroup
> side where cgroup_mutex should have been held. IOW, an attach operation
> which requires holding cgroup_mutex cannot happen in parallel. The only
> exception is hk_sd_workfn() which is a work item for updating the
> housekeeping cpumasks from hotplug which has no dependency on an attach
> operation.

I agree for cpuset_css_online(), cpuset_css_offline() and
cpuset_css_killed(). They hold cgroup_mutex, so they never wait there.

 From reading, three other callers of cpuset_full_lock() can rebuild the
sched domains without cgroup_mutex:

- cpuset_write_u64(), for the cgroup v1 file sched_load_balance. It
   calls cpuset_update_flag(), which calls
   rebuild_sched_domains_locked() when the flag changes.
- cpuset_write_s64(), for the cgroup v1 file sched_relax_domain_level.
   update_relax_domain_level() calls rebuild_sched_domains_locked().
- hk_sd_workfn(). cpuset_update_sd_hk_unlock() calls
   rebuild_sched_domains_locked() when force_sd_rebuild is set.

The two write handlers are called from cgroup_file_write(), which does
not take cgroup_mutex. The work item does not take it either.

The dependency on the attach is in the rebuild, not in what these
callers change. rebuild_sched_domains_locked() ends in
dl_rebuild_rd_accounting(). That clears total_bw of every root domain
and counts the deadline tasks again, each where it is now. A
reservation that cpuset_can_attach() made for a task that has not moved
yet is lost. That is the diagram of the changelog, with one of these
three in the place of thread B.

I did not reproduce it through these three callers. The test of this
patch goes through sched_rt_runtime_us, and my test kernels have no
cgroup v1 cpuset.

If you prefer cpuset_full_lock() unchanged, I can make these three
callers wait instead, and leave the css callbacks alone. Tell me which
you prefer.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com