Re: [PATCH 02/27] sched/isolation: Introduce housekeeping per-cpu rwsem

[Waiman Long <llong@redhat.com>]

On 6/25/25 11:50 AM, Phil Auld wrote:
> On Wed, Jun 25, 2025 at 04:34:18PM +0200 Frederic Weisbecker wrote:
>> Le Wed, Jun 25, 2025 at 08:18:50AM -0400, Phil Auld a écrit :
>>> Hi Waiman,
>>>
>>> On Mon, Jun 23, 2025 at 01:34:58PM -0400 Waiman Long wrote:
>>>> On 6/20/25 11:22 AM, Frederic Weisbecker wrote:
>>>>> The HK_TYPE_DOMAIN isolation cpumask, and further the
>>>>> HK_TYPE_KERNEL_NOISE cpumask will be made modifiable at runtime in the
>>>>> future.
>>>>>
>>>>> The affected subsystems will need to synchronize against those cpumask
>>>>> changes so that:
>>>>>
>>>>> * The reader get a coherent snapshot
>>>>> * The housekeeping subsystem can safely propagate a cpumask update to
>>>>>     the susbsytems after it has been published.
>>>>>
>>>>> Protect against readsides that can sleep with per-cpu rwsem. Updates are
>>>>> expected to be very rare given that CPU isolation is a niche usecase and
>>>>> related cpuset setup happen only in preparation work. On the other hand
>>>>> read sides can occur in more frequent paths.
>>>>>
>>>>> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
>>>> Thanks for the patch series and it certainly has some good ideas. However I
>>>> am a bit concern about the overhead of using percpu-rwsem for
>>>> synchronization especially when the readers have to wait for the completion
>>>> on the writer side. From my point of view, during the transition period when
>>>> new isolated CPUs are being added or old ones being removed, the reader will
>>>> either get the old CPU data or the new one depending on the exact timing.
>>>> The effect the CPU selection may persist for a while after the end of the
>>>> critical section.
>>>>
>>>> Can we just rely on RCU to make sure that it either get the new one or the
>>>> old one but nothing in between without the additional overhead?
>>>>
>>>> My current thinking is to make use CPU hotplug to enable better CPU
>>>> isolation. IOW, I would shut down the affected CPUs, change the housekeeping
>>>> masks and then bring them back online again. That means the writer side will
>>>> take a while to complete.
>>> The problem with this approach is that offlining a cpu effects all the other
>>> cpus and causes latency spikes on other low latency tasks which may already be
>>> running on other parts of the system.
>>>
>>> I just don't want us to finally get to dynamic isolation and have it not
>>> usable for the usecases asking for it.
>> We'll have to discuss that eventually because that's the plan for nohz_full.
>> We can work around the stop machine rendez-vous on nohz_full if that's the
>> problem. If the issue is not to interrupt common RT-tasks, then that's a
>> different problem for which I don't have a solution.
>>
> My understanding is that it's the stop machine issue. If you have a way
> around that then great!

My current thinking is to just run a selected set of CPUHP teardown and 
startup methods relevant to housekeeping cpumasks usage without calling 
the full set from CPUHP_ONLINE to CPUHP_OFFLINE. I don't know if it is 
possible or not or how much additional changes will be needed to make 
that possible. That will skip the CPUHP_TEARDOWN_CPU teardown method 
that is likely the cause of most the latency spike experienced by other 
CPUs.

Cheers,
Longman