[PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pingfan Liu]

When testing kexec-reboot on a 144 cpus machine with
isolcpus=managed_irq,domain,1-71,73-143 in kernel command line, I
encounter the following bug:

[   97.114759] psci: CPU142 killed (polled 0 ms)
[   97.333236] Failed to offline CPU143 - error=-16
[   97.333246] ------------[ cut here ]------------
[   97.342682] kernel BUG at kernel/cpu.c:1569!
[   97.347049] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
[   97.353281] Modules linked in: rfkill sunrpc dax_hmem cxl_acpi cxl_port cxl_core einj vfat fat arm_smmuv3_pmu nvidia_cspmu arm_spe_pmu coresight_trbe arm_cspmu_module rndis_host ipmi_ssif cdc_ether i2c_smbus spi_nor usbnet ast coresight_tmc mii ixgbe i2c_algo_bit mdio mtd coresight_funnel coresight_stm stm_core coresight_etm4x coresight cppc_cpufreq loop fuse nfnetlink xfs crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce sbsa_gwdt nvme nvme_core nvme_auth i2c_tegra acpi_power_meter acpi_ipmi ipmi_devintf ipmi_msghandler dm_mirror dm_region_hash dm_log dm_mod
[   97.404119] CPU: 0 UID: 0 PID: 2583 Comm: kexec Kdump: loaded Not tainted 6.12.0-41.el10.aarch64 #1
[   97.413371] Hardware name: Supermicro MBD-G1SMH/G1SMH, BIOS 2.0 07/12/2024
[   97.420400] pstate: 23400009 (nzCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[   97.427518] pc : smp_shutdown_nonboot_cpus+0x104/0x128
[   97.432778] lr : smp_shutdown_nonboot_cpus+0x11c/0x128
[   97.438028] sp : ffff800097c6b9a0
[   97.441411] x29: ffff800097c6b9a0 x28: ffff0000a099d800 x27: 0000000000000000
[   97.448708] x26: 0000000000000000 x25: 0000000000000000 x24: ffffb94aaaa8f218
[   97.456004] x23: ffffb94aaaabaae0 x22: ffffb94aaaa8f018 x21: 0000000000000000
[   97.463301] x20: ffffb94aaaa8fc10 x19: 000000000000008f x18: 00000000fffffffe
[   97.470598] x17: 0000000000000000 x16: ffffb94aa958fcd0 x15: ffff103acfca0b64
[   97.477894] x14: ffff800097c6b520 x13: 36312d3d726f7272 x12: ffff103acfc6ffa8
[   97.485191] x11: ffff103acf6f0000 x10: ffff103bc085c400 x9 : ffffb94aa88a0eb0
[   97.492488] x8 : 0000000000000001 x7 : 000000000017ffe8 x6 : c0000000fffeffff
[   97.499784] x5 : ffff003bdf62b408 x4 : 0000000000000000 x3 : 0000000000000000
[   97.507081] x2 : 0000000000000000 x1 : ffff0000a099d800 x0 : 0000000000000002
[   97.514379] Call trace:
[   97.516874]  smp_shutdown_nonboot_cpus+0x104/0x128
[   97.521769]  machine_shutdown+0x20/0x38
[   97.525693]  kernel_kexec+0xc4/0xf0
[   97.529260]  __do_sys_reboot+0x24c/0x278
[   97.533272]  __arm64_sys_reboot+0x2c/0x40
[   97.537370]  invoke_syscall.constprop.0+0x74/0xd0
[   97.542179]  do_el0_svc+0xb0/0xe8
[   97.545562]  el0_svc+0x44/0x1d0
[   97.548772]  el0t_64_sync_handler+0x120/0x130
[   97.553222]  el0t_64_sync+0x1a4/0x1a8
[   97.556963] Code: a94363f7 a8c47bfd d50323bf d65f03c0 (d4210000)
[   97.563191] ---[ end trace 0000000000000000 ]---
[   97.595854] Kernel panic - not syncing: Oops - BUG: Fatal exception
[   97.602275] Kernel Offset: 0x394a28600000 from 0xffff800080000000
[   97.608502] PHYS_OFFSET: 0x80000000
[   97.612062] CPU features: 0x10,0000000d,002a6928,5667fea7
[   97.617580] Memory Limit: none
[   97.648626] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception ]

Tracking down this issue, I found that dl_bw_deactivate() returned
-EBUSY, which caused sched_cpu_deactivate() to fail on the last CPU.
When a CPU is inactive, its rd is set to def_root_domain. For an S-state
deadline task (in this case, "cppc_fie"), it was not migrated to CPU0,
and its task_rq() information is stale. As a result, its bandwidth is
wrongly accounted into def_root_domain during domain rebuild.

This patch uses the rd from the run queue of still-active CPU to get the
correct root domain.

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Valentin Schneider <vschneid@redhat.com>
To: linux-kernel@vger.kernel.org
---
 kernel/sched/deadline.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index f25301267e47..bb42b82d6366 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2913,6 +2913,7 @@ void dl_add_task_root_domain(struct task_struct *p)
 	struct rq_flags rf;
 	struct rq *rq;
 	struct dl_bw *dl_b;
+	unsigned int cpu;
 
 	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
 	if (!dl_task(p) || dl_entity_is_special(&p->dl)) {
@@ -2920,16 +2921,23 @@ void dl_add_task_root_domain(struct task_struct *p)
 		return;
 	}
 
-	rq = __task_rq_lock(p, &rf);
-
+	lockdep_assert_cpus_held();
+	/*
+	 * If @p is not in R state, task_cpu() may be not active.  task_rq()'s
+	 * root_domain may be invalid. But the rest active cpus on cpus_ptr
+	 * share the same root domain.
+	 */
+	cpu = cpumask_first_and(cpu_active_mask, p->cpus_ptr);
+	rq = cpu_rq(cpu);
+	/*
+	 * This point is under the protection of cpu_hotplug_lock. Hence
+	 * rq->rd is stable.
+	 */
 	dl_b = &rq->rd->dl_bw;
 	raw_spin_lock(&dl_b->lock);
-
 	__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
-
 	raw_spin_unlock(&dl_b->lock);
-
-	task_rq_unlock(rq, p, &rf);
+	raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
 }
 
 void dl_clear_root_domain(struct root_domain *rd)
-- 
2.49.0

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Peter Zijlstra]

On Mon, Sep 29, 2025 at 09:36:02PM +0800, Pingfan Liu wrote:
> When testing kexec-reboot on a 144 cpus machine with
> isolcpus=managed_irq,domain,1-71,73-143 in kernel command line, I
> encounter the following bug:
> 
> [   97.114759] psci: CPU142 killed (polled 0 ms)
> [   97.333236] Failed to offline CPU143 - error=-16
> [   97.333246] ------------[ cut here ]------------
> [   97.342682] kernel BUG at kernel/cpu.c:1569!

> [   97.514379] Call trace:
> [   97.516874]  smp_shutdown_nonboot_cpus+0x104/0x128
> [   97.521769]  machine_shutdown+0x20/0x38
> [   97.525693]  kernel_kexec+0xc4/0xf0
> [   97.529260]  __do_sys_reboot+0x24c/0x278
> [   97.533272]  __arm64_sys_reboot+0x2c/0x40

> Tracking down this issue, I found that dl_bw_deactivate() returned
> -EBUSY, which caused sched_cpu_deactivate() to fail on the last CPU.
> When a CPU is inactive, its rd is set to def_root_domain. For an S-state

You mean a blocked task?

> deadline task (in this case, "cppc_fie"), it was not migrated to CPU0,
> and its task_rq() information is stale. As a result, its bandwidth is
> wrongly accounted into def_root_domain during domain rebuild.
> 
> This patch uses the rd from the run queue of still-active CPU to get the
> correct root domain.

That doesn't seem right in general. What if there are multiple root
domains; how does it know which to use?

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pingfan Liu]

Hi Peter,

On Mon, Sep 29, 2025 at 9:54 PM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Mon, Sep 29, 2025 at 09:36:02PM +0800, Pingfan Liu wrote:
> > When testing kexec-reboot on a 144 cpus machine with
> > isolcpus=managed_irq,domain,1-71,73-143 in kernel command line, I
> > encounter the following bug:
> >
> > [   97.114759] psci: CPU142 killed (polled 0 ms)
> > [   97.333236] Failed to offline CPU143 - error=-16
> > [   97.333246] ------------[ cut here ]------------
> > [   97.342682] kernel BUG at kernel/cpu.c:1569!
>
> > [   97.514379] Call trace:
> > [   97.516874]  smp_shutdown_nonboot_cpus+0x104/0x128
> > [   97.521769]  machine_shutdown+0x20/0x38
> > [   97.525693]  kernel_kexec+0xc4/0xf0
> > [   97.529260]  __do_sys_reboot+0x24c/0x278
> > [   97.533272]  __arm64_sys_reboot+0x2c/0x40
>
> > Tracking down this issue, I found that dl_bw_deactivate() returned
> > -EBUSY, which caused sched_cpu_deactivate() to fail on the last CPU.
> > When a CPU is inactive, its rd is set to def_root_domain. For an S-state
>
> You mean a blocked task?
>

Yes.

> > deadline task (in this case, "cppc_fie"), it was not migrated to CPU0,
> > and its task_rq() information is stale. As a result, its bandwidth is
> > wrongly accounted into def_root_domain during domain rebuild.
> >
> > This patch uses the rd from the run queue of still-active CPU to get the
> > correct root domain.
>

Sorry that I haven't explained it clearly. I mean the still-active CPU
in task->cpus_ptr,

> That doesn't seem right in general. What if there are multiple root
> domains; how does it know which to use?
>

In the case of task->cpus_ptr, there should be only one root domain, right?

Thanks,

Pingfan

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Juri Lelli]

On 30/09/25 09:47, Pingfan Liu wrote:
> Hi Peter,
> 
> On Mon, Sep 29, 2025 at 9:54 PM Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Mon, Sep 29, 2025 at 09:36:02PM +0800, Pingfan Liu wrote:
> > > When testing kexec-reboot on a 144 cpus machine with
> > > isolcpus=managed_irq,domain,1-71,73-143 in kernel command line, I
> > > encounter the following bug:
> > >
> > > [   97.114759] psci: CPU142 killed (polled 0 ms)
> > > [   97.333236] Failed to offline CPU143 - error=-16
> > > [   97.333246] ------------[ cut here ]------------
> > > [   97.342682] kernel BUG at kernel/cpu.c:1569!
> >
> > > [   97.514379] Call trace:
> > > [   97.516874]  smp_shutdown_nonboot_cpus+0x104/0x128
> > > [   97.521769]  machine_shutdown+0x20/0x38
> > > [   97.525693]  kernel_kexec+0xc4/0xf0
> > > [   97.529260]  __do_sys_reboot+0x24c/0x278
> > > [   97.533272]  __arm64_sys_reboot+0x2c/0x40
> >
> > > Tracking down this issue, I found that dl_bw_deactivate() returned
> > > -EBUSY, which caused sched_cpu_deactivate() to fail on the last CPU.
> > > When a CPU is inactive, its rd is set to def_root_domain. For an S-state
> >
> > You mean a blocked task?
> >
> 
> Yes.
> 
> > > deadline task (in this case, "cppc_fie"), it was not migrated to CPU0,
> > > and its task_rq() information is stale. As a result, its bandwidth is
> > > wrongly accounted into def_root_domain during domain rebuild.
> > >
> > > This patch uses the rd from the run queue of still-active CPU to get the
> > > correct root domain.
> >
> 
> Sorry that I haven't explained it clearly. I mean the still-active CPU
> in task->cpus_ptr,
> 
> > That doesn't seem right in general. What if there are multiple root
> > domains; how does it know which to use?
> >

I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
tasks (like schedutil [1]). IIUC that is how it is thought to behave
already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
it is not "transparent" from a bandwidth tracking point of view.

1 - https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
2 - https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Peter Zijlstra]

On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:

> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
> tasks (like schedutil [1]). IIUC that is how it is thought to behave
> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
> it is not "transparent" from a bandwidth tracking point of view.
> 
> 1 - https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
> 2 - https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198

Right, I remember that hack. Bit sad its spreading, but this CPPC thing
is very much like the schedutil one, so might as well do that I suppose.

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pierre Gondois]

On 9/30/25 11:04, Peter Zijlstra wrote:
> On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
>
>> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
>> tasks (like schedutil [1]). IIUC that is how it is thought to behave
>> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
>> it is not "transparent" from a bandwidth tracking point of view.
>>
>> 1 - https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
>> 2 - https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
> Right, I remember that hack. Bit sad its spreading, but this CPPC thing
> is very much like the schedutil one, so might as well do that I suppose.
>
IIUC, the sugov thread was switched to deadline to allow frequency updates
when deadline tasks start to run. I.e. there should be no point updating 
the
freq. after the deadline task finished running, cf [1] and [2]

The CPPC FIE worker should not require to run that quickly as it seems to be
more like a freq. maintenance work (the call comes from the sched tick)

sched_tick()
\-arch_scale_freq_tick() / topology_scale_freq_tick()
   \-set_freq_scale() / cppc_scale_freq_tick()
     \-irq_work_queue()


[1] https://lore.kernel.org/all/20171204102325.5110-3-juri.lelli@redhat.com/
[2] https://lore.kernel.org/all/20171204102325.5110-1-juri.lelli@redhat.com/
"""
  o 03/08      it's a temporary solution to make possible (on ARM) to change

               frequency for DEADLINE tasks (that would possibly delay 
the SCHED_FIFO
               worker kthread); proper solution would be to be able to 
issue frequency
               transition from an atomic ctx
"""

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Juri Lelli]

On 06/10/25 12:13, Pierre Gondois wrote:
> 
> On 9/30/25 11:04, Peter Zijlstra wrote:
> > On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
> > 
> > > I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
> > > tasks (like schedutil [1]). IIUC that is how it is thought to behave
> > > already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
> > > it is not "transparent" from a bandwidth tracking point of view.
> > > 
> > > 1 -https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
> > > 2 -https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
> > Right, I remember that hack. Bit sad its spreading, but this CPPC thing
> > is very much like the schedutil one, so might as well do that I suppose.
> 
> IIUC, the sugov thread was switched to deadline to allow frequency updates
> when deadline tasks start to run. I.e. there should be no point updating the
> freq. after the deadline task finished running, cf [1] and [2]
> 
> The CPPC FIE worker should not require to run that quickly as it seems to be
> more like a freq. maintenance work (the call comes from the sched tick)
> 
> sched_tick()
> \-arch_scale_freq_tick() / topology_scale_freq_tick()
>   \-set_freq_scale() / cppc_scale_freq_tick()
>     \-irq_work_queue()

OK, but how much bandwidth is enough for it (on different platforms)?
Also, I am not sure the worker follows cpusets/root domain changes.

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pierre Gondois]

On 10/6/25 14:12, Juri Lelli wrote:
> On 06/10/25 12:13, Pierre Gondois wrote:
>> On 9/30/25 11:04, Peter Zijlstra wrote:
>>> On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
>>>
>>>> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
>>>> tasks (like schedutil [1]). IIUC that is how it is thought to behave
>>>> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
>>>> it is not "transparent" from a bandwidth tracking point of view.
>>>>
>>>> 1 -https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
>>>> 2 -https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
>>> Right, I remember that hack. Bit sad its spreading, but this CPPC thing
>>> is very much like the schedutil one, so might as well do that I suppose.
>> IIUC, the sugov thread was switched to deadline to allow frequency updates
>> when deadline tasks start to run. I.e. there should be no point updating the
>> freq. after the deadline task finished running, cf [1] and [2]
>>
>> The CPPC FIE worker should not require to run that quickly as it seems to be
>> more like a freq. maintenance work (the call comes from the sched tick)
>>
>> sched_tick()
>> \-arch_scale_freq_tick() / topology_scale_freq_tick()
>>    \-set_freq_scale() / cppc_scale_freq_tick()
>>      \-irq_work_queue()
> OK, but how much bandwidth is enough for it (on different platforms)?
> Also, I am not sure the worker follows cpusets/root domain changes.
>
>
To share some additional information, I could to reproduce the issue by
creating as many deadline tasks with a huge bandwidth that the platform
allows it:
chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &

Then kexec to another kernel. The available bandwidth of the root domain
gradually decreases with the number of CPUs unplugged.
At some point, there is not enough bandwidth and an overflow is detected.
(Same call stack as in the original message).

So I'm not sure this is really related to the cppc_fie thread.
I think it's more related to checking the available bandwidth in a context
which is not appropriate. The deadline bandwidth might lack when the 
platform
is reset, but this should not be that important.

---

Question:
Since the cppc_fie worker doesn't have the SCHED_FLAG_SUGOV flag,
is this comment actually correct ?
/*
  * Fake (unused) bandwidth; workaround to "fix"
  * priority inheritance.
  */

---

On a non-deadline related topic, the CPPC drivers creates a cppc_fie 
worker in
case the CPPC counters to estimate the current frequency are in PCC 
channels.
Accessing these channels requires to go through sleeping sections,
that's why a worker is used.

However, CPPC counters might be accessed through FFH, which doesn't go 
through
sleeping sections. In such case, the cppc_fie worker is never used and never
removed, so it would be nice to remote it.

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pingfan Liu]

Hi Pierre,

Thanks for sharing your perspective.

On Sat, Oct 11, 2025 at 12:26 AM Pierre Gondois <pierre.gondois@arm.com> wrote:
>
>
> On 10/6/25 14:12, Juri Lelli wrote:
> > On 06/10/25 12:13, Pierre Gondois wrote:
> >> On 9/30/25 11:04, Peter Zijlstra wrote:
> >>> On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
> >>>
> >>>> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
> >>>> tasks (like schedutil [1]). IIUC that is how it is thought to behave
> >>>> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
> >>>> it is not "transparent" from a bandwidth tracking point of view.
> >>>>
> >>>> 1 -https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
> >>>> 2 -https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
> >>> Right, I remember that hack. Bit sad its spreading, but this CPPC thing
> >>> is very much like the schedutil one, so might as well do that I suppose.
> >> IIUC, the sugov thread was switched to deadline to allow frequency updates
> >> when deadline tasks start to run. I.e. there should be no point updating the
> >> freq. after the deadline task finished running, cf [1] and [2]
> >>
> >> The CPPC FIE worker should not require to run that quickly as it seems to be
> >> more like a freq. maintenance work (the call comes from the sched tick)
> >>
> >> sched_tick()
> >> \-arch_scale_freq_tick() / topology_scale_freq_tick()
> >>    \-set_freq_scale() / cppc_scale_freq_tick()
> >>      \-irq_work_queue()
> > OK, but how much bandwidth is enough for it (on different platforms)?
> > Also, I am not sure the worker follows cpusets/root domain changes.
> >
> >
> To share some additional information, I could to reproduce the issue by
> creating as many deadline tasks with a huge bandwidth that the platform
> allows it:
> chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &
>
> Then kexec to another kernel. The available bandwidth of the root domain
> gradually decreases with the number of CPUs unplugged.
> At some point, there is not enough bandwidth and an overflow is detected.
> (Same call stack as in the original message).
>
> So I'm not sure this is really related to the cppc_fie thread.
> I think it's more related to checking the available bandwidth in a context
> which is not appropriate. The deadline bandwidth might lack when the
> platform
> is reset, but this should not be that important.
>

I think there are two independent issues.

In your experiment, as CPUs are hot-removed one by one, at some point
the hot-removal will fail due to insufficient DL bandwidth. There
should be a warning message to inform users about what's happening,
and users can then remove some DL tasks to continue the CPU
hot-removal.

Meanwhile, in the kexec case, this checking can be skipped since the
system cannot roll back to a working state anyway


Thanks,

Pingfan
> ---
>
> Question:
> Since the cppc_fie worker doesn't have the SCHED_FLAG_SUGOV flag,
> is this comment actually correct ?
> /*
>   * Fake (unused) bandwidth; workaround to "fix"
>   * priority inheritance.
>   */
>
> ---
>
> On a non-deadline related topic, the CPPC drivers creates a cppc_fie
> worker in
> case the CPPC counters to estimate the current frequency are in PCC
> channels.
> Accessing these channels requires to go through sleeping sections,
> that's why a worker is used.
>
> However, CPPC counters might be accessed through FFH, which doesn't go
> through
> sleeping sections. In such case, the cppc_fie worker is never used and never
> removed, so it would be nice to remote it.
>

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Juri Lelli]

On 14/10/25 21:09, Pingfan Liu wrote:
> Hi Pierre,
> 
> Thanks for sharing your perspective.
> 
> On Sat, Oct 11, 2025 at 12:26 AM Pierre Gondois <pierre.gondois@arm.com> wrote:
> >
> >
> > On 10/6/25 14:12, Juri Lelli wrote:
> > > On 06/10/25 12:13, Pierre Gondois wrote:
> > >> On 9/30/25 11:04, Peter Zijlstra wrote:
> > >>> On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
> > >>>
> > >>>> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
> > >>>> tasks (like schedutil [1]). IIUC that is how it is thought to behave
> > >>>> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
> > >>>> it is not "transparent" from a bandwidth tracking point of view.
> > >>>>
> > >>>> 1 -https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
> > >>>> 2 -https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
> > >>> Right, I remember that hack. Bit sad its spreading, but this CPPC thing
> > >>> is very much like the schedutil one, so might as well do that I suppose.
> > >> IIUC, the sugov thread was switched to deadline to allow frequency updates
> > >> when deadline tasks start to run. I.e. there should be no point updating the
> > >> freq. after the deadline task finished running, cf [1] and [2]
> > >>
> > >> The CPPC FIE worker should not require to run that quickly as it seems to be
> > >> more like a freq. maintenance work (the call comes from the sched tick)
> > >>
> > >> sched_tick()
> > >> \-arch_scale_freq_tick() / topology_scale_freq_tick()
> > >>    \-set_freq_scale() / cppc_scale_freq_tick()
> > >>      \-irq_work_queue()
> > > OK, but how much bandwidth is enough for it (on different platforms)?
> > > Also, I am not sure the worker follows cpusets/root domain changes.
> > >
> > >
> > To share some additional information, I could to reproduce the issue by
> > creating as many deadline tasks with a huge bandwidth that the platform
> > allows it:
> > chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &
> >
> > Then kexec to another kernel. The available bandwidth of the root domain
> > gradually decreases with the number of CPUs unplugged.
> > At some point, there is not enough bandwidth and an overflow is detected.
> > (Same call stack as in the original message).

I seem to agree with Pingfan below, kexec (kernel crash?) is a case
where all guarantees are out of the window anyway, so really no point in
keeping track of bandwidth and failing hotplug. Guess we should be
adding an ad-hoc check/bail for this case.

> > So I'm not sure this is really related to the cppc_fie thread.
> > I think it's more related to checking the available bandwidth in a context
> > which is not appropriate. The deadline bandwidth might lack when the
> > platform
> > is reset, but this should not be that important.
> >
> 
> I think there are two independent issues.
> 
> In your experiment, as CPUs are hot-removed one by one, at some point
> the hot-removal will fail due to insufficient DL bandwidth. There
> should be a warning message to inform users about what's happening,
> and users can then remove some DL tasks to continue the CPU
> hot-removal.
> 
> Meanwhile, in the kexec case, this checking can be skipped since the
> system cannot roll back to a working state anyway
> 
> 
> Thanks,
> 
> Pingfan
> > ---
> >
> > Question:
> > Since the cppc_fie worker doesn't have the SCHED_FLAG_SUGOV flag,
> > is this comment actually correct ?
> > /*
> >   * Fake (unused) bandwidth; workaround to "fix"
> >   * priority inheritance.
> >   */
> >
> > ---
> >
> > On a non-deadline related topic, the CPPC drivers creates a cppc_fie
> > worker in
> > case the CPPC counters to estimate the current frequency are in PCC
> > channels.
> > Accessing these channels requires to go through sleeping sections,
> > that's why a worker is used.
> >
> > However, CPPC counters might be accessed through FFH, which doesn't go
> > through
> > sleeping sections. In such case, the cppc_fie worker is never used and never
> > removed, so it would be nice to remote it.
> >
> 

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pierre Gondois]

On 10/15/25 11:35, Juri Lelli wrote:
> On 14/10/25 21:09, Pingfan Liu wrote:
>> Hi Pierre,
>>
>> Thanks for sharing your perspective.
>>
>> On Sat, Oct 11, 2025 at 12:26 AM Pierre Gondois <pierre.gondois@arm.com> wrote:
>>>
>>> On 10/6/25 14:12, Juri Lelli wrote:
>>>> On 06/10/25 12:13, Pierre Gondois wrote:
>>>>> On 9/30/25 11:04, Peter Zijlstra wrote:
>>>>>> On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
>>>>>>
>>>>>>> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
>>>>>>> tasks (like schedutil [1]). IIUC that is how it is thought to behave
>>>>>>> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
>>>>>>> it is not "transparent" from a bandwidth tracking point of view.
>>>>>>>
>>>>>>> 1 -https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
>>>>>>> 2 -https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
>>>>>> Right, I remember that hack. Bit sad its spreading, but this CPPC thing
>>>>>> is very much like the schedutil one, so might as well do that I suppose.
>>>>> IIUC, the sugov thread was switched to deadline to allow frequency updates
>>>>> when deadline tasks start to run. I.e. there should be no point updating the
>>>>> freq. after the deadline task finished running, cf [1] and [2]
>>>>>
>>>>> The CPPC FIE worker should not require to run that quickly as it seems to be
>>>>> more like a freq. maintenance work (the call comes from the sched tick)
>>>>>
>>>>> sched_tick()
>>>>> \-arch_scale_freq_tick() / topology_scale_freq_tick()
>>>>>     \-set_freq_scale() / cppc_scale_freq_tick()
>>>>>       \-irq_work_queue()
>>>> OK, but how much bandwidth is enough for it (on different platforms)?
>>>> Also, I am not sure the worker follows cpusets/root domain changes.
>>>>
>>>>
>>> To share some additional information, I could to reproduce the issue by
>>> creating as many deadline tasks with a huge bandwidth that the platform
>>> allows it:
>>> chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &
>>>
>>> Then kexec to another kernel. The available bandwidth of the root domain
>>> gradually decreases with the number of CPUs unplugged.
>>> At some point, there is not enough bandwidth and an overflow is detected.
>>> (Same call stack as in the original message).
> I seem to agree with Pingfan below, kexec (kernel crash?) is a case
> where all guarantees are out of the window anyway, so really no point in
> keeping track of bandwidth and failing hotplug. Guess we should be
> adding an ad-hoc check/bail for this case.

Yes right

>>> So I'm not sure this is really related to the cppc_fie thread.
>>> I think it's more related to checking the available bandwidth in a context
>>> which is not appropriate. The deadline bandwidth might lack when the
>>> platform
>>> is reset, but this should not be that important.
>>>
>> I think there are two independent issues.
>>
>> In your experiment, as CPUs are hot-removed one by one, at some point
>> the hot-removal will fail due to insufficient DL bandwidth. There
>> should be a warning message to inform users about what's happening,
>> and users can then remove some DL tasks to continue the CPU
>> hot-removal.
>>
>> Meanwhile, in the kexec case, this checking can be skipped since the
>> system cannot roll back to a working state anyway

Yes right, I meant that:
-
when using kexec, the kernel crashes
-
when manually unplugging CPUs with:
`echo 0 > /sys/devices/system/cpu/cpuX/online`
The kernel returns `write error: Device or resource busy` at some point 
to prevent
from reducing the DL bandwidth too much.

------

I  could not reproduce the issue you reported initially. I am using 
a radxa orion o6
which has a cppc_fie worker.

AFAIU it should not be possible to add/remove bandwidth to the 
def_root_domain.
During kexec, the following is happening to all CPUs:
\-dl_bw_manage(dl_bw_req_deactivate, cpu)
   \- // Check if there is enough bandwidth
\-dl_clear_root_domain_cpu(cpu)
   \- // Recompute the available bandwidth based on the remaining CPUs

So I'm not sure to understand why accounting some bandwidth to the 
def_root_domain
is problematic in practice as the def_root_domain seems to have some DL 
bandwidth.

IIUC the problem seems to be that for some reason there is not enough 
bandwidth in the
def_root_domain aswell, which triggers the bandwidth overflow detection.

>>
>> Thanks,
>>
>> Pingfan
>>> ---
>>>
>>> Question:
>>> Since the cppc_fie worker doesn't have the SCHED_FLAG_SUGOV flag,
>>> is this comment actually correct ?
>>> /*
>>>    * Fake (unused) bandwidth; workaround to "fix"
>>>    * priority inheritance.
>>>    */
>>>
>>> ---
>>>
>>> On a non-deadline related topic, the CPPC drivers creates a cppc_fie
>>> worker in
>>> case the CPPC counters to estimate the current frequency are in PCC
>>> channels.
>>> Accessing these channels requires to go through sleeping sections,
>>> that's why a worker is used.
>>>
>>> However, CPPC counters might be accessed through FFH, which doesn't go
>>> through
>>> sleeping sections. In such case, the cppc_fie worker is never used and never
>>> removed, so it would be nice to remote it.
>>>
>

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pingfan Liu]

On Thu, Oct 16, 2025 at 7:38 PM Pierre Gondois <pierre.gondois@arm.com> wrote:
>
>
> On 10/15/25 11:35, Juri Lelli wrote:
> > On 14/10/25 21:09, Pingfan Liu wrote:
> >> Hi Pierre,
> >>
> >> Thanks for sharing your perspective.
> >>
> >> On Sat, Oct 11, 2025 at 12:26 AM Pierre Gondois <pierre.gondois@arm.com> wrote:
> >>>
> >>> On 10/6/25 14:12, Juri Lelli wrote:
> >>>> On 06/10/25 12:13, Pierre Gondois wrote:
> >>>>> On 9/30/25 11:04, Peter Zijlstra wrote:
> >>>>>> On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
> >>>>>>
> >>>>>>> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
> >>>>>>> tasks (like schedutil [1]). IIUC that is how it is thought to behave
> >>>>>>> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
> >>>>>>> it is not "transparent" from a bandwidth tracking point of view.
> >>>>>>>
> >>>>>>> 1 -https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
> >>>>>>> 2 -https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
> >>>>>> Right, I remember that hack. Bit sad its spreading, but this CPPC thing
> >>>>>> is very much like the schedutil one, so might as well do that I suppose.
> >>>>> IIUC, the sugov thread was switched to deadline to allow frequency updates
> >>>>> when deadline tasks start to run. I.e. there should be no point updating the
> >>>>> freq. after the deadline task finished running, cf [1] and [2]
> >>>>>
> >>>>> The CPPC FIE worker should not require to run that quickly as it seems to be
> >>>>> more like a freq. maintenance work (the call comes from the sched tick)
> >>>>>
> >>>>> sched_tick()
> >>>>> \-arch_scale_freq_tick() / topology_scale_freq_tick()
> >>>>>     \-set_freq_scale() / cppc_scale_freq_tick()
> >>>>>       \-irq_work_queue()
> >>>> OK, but how much bandwidth is enough for it (on different platforms)?
> >>>> Also, I am not sure the worker follows cpusets/root domain changes.
> >>>>
> >>>>
> >>> To share some additional information, I could to reproduce the issue by
> >>> creating as many deadline tasks with a huge bandwidth that the platform
> >>> allows it:
> >>> chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &
> >>>
> >>> Then kexec to another kernel. The available bandwidth of the root domain
> >>> gradually decreases with the number of CPUs unplugged.
> >>> At some point, there is not enough bandwidth and an overflow is detected.
> >>> (Same call stack as in the original message).
> > I seem to agree with Pingfan below, kexec (kernel crash?) is a case
> > where all guarantees are out of the window anyway, so really no point in
> > keeping track of bandwidth and failing hotplug. Guess we should be
> > adding an ad-hoc check/bail for this case.
>
> Yes right
>
> >>> So I'm not sure this is really related to the cppc_fie thread.
> >>> I think it's more related to checking the available bandwidth in a context
> >>> which is not appropriate. The deadline bandwidth might lack when the
> >>> platform
> >>> is reset, but this should not be that important.
> >>>
> >> I think there are two independent issues.
> >>
> >> In your experiment, as CPUs are hot-removed one by one, at some point
> >> the hot-removal will fail due to insufficient DL bandwidth. There
> >> should be a warning message to inform users about what's happening,
> >> and users can then remove some DL tasks to continue the CPU
> >> hot-removal.
> >>
> >> Meanwhile, in the kexec case, this checking can be skipped since the
> >> system cannot roll back to a working state anyway
>
> Yes right, I meant that:
> -
> when using kexec, the kernel crashes
> -
> when manually unplugging CPUs with:
> `echo 0 > /sys/devices/system/cpu/cpuX/online`
> The kernel returns `write error: Device or resource busy` at some point
> to prevent
> from reducing the DL bandwidth too much.
>
> ------
>
> I  could not reproduce the issue you reported initially. I am using
> a radxa orion o6
> which has a cppc_fie worker.
>

I speculate that you miss something like
"isolcpus=managed_irq,domain,1-71,73-143" in the kernel command line.
That is critical to reproduce the bug. In that case, cpus
[1,71],[73,143] are in def_root_domain, while cpu0 and 72 are in the
other new root_domain. The bug is triggered if cppc_fie worker is
scheduled on cpu72.

> AFAIU it should not be possible to add/remove bandwidth to the
> def_root_domain.
> During kexec, the following is happening to all CPUs:
> \-dl_bw_manage(dl_bw_req_deactivate, cpu)
>    \- // Check if there is enough bandwidth
> \-dl_clear_root_domain_cpu(cpu)
>    \- // Recompute the available bandwidth based on the remaining CPUs
>
> So I'm not sure to understand why accounting some bandwidth to the
> def_root_domain
> is problematic in practice as the def_root_domain seems to have some DL
> bandwidth.
>
> IIUC the problem seems to be that for some reason there is not enough
> bandwidth in the
> def_root_domain aswell, which triggers the bandwidth overflow detection.
>

The problem is caused by accounting the blocked-state DL task's
bandwidth to a wrong root_domain. Let me refer to the previous
example.  cpus [1,71],[73,143] belong to def_root_domain, cpu0,72
belong to root_domainA. In the kernel, the root_domain is traced in
cpu_rq(cpu)->rd. But for an offline rq, rq->rd points to
def_root_domain. Hence the reserved bandwidth of cppc_fie is wrongly
accounted into the  def_root_domain instead of root_domainA. So
finally, cpu143 refuses to be offlined since def_root_domain
demonstrates there should be reserved DL bandwidth.


Thanks,

Pingfan
> >>
> >> Thanks,
> >>
> >> Pingfan
> >>> ---
> >>>
> >>> Question:
> >>> Since the cppc_fie worker doesn't have the SCHED_FLAG_SUGOV flag,
> >>> is this comment actually correct ?
> >>> /*
> >>>    * Fake (unused) bandwidth; workaround to "fix"
> >>>    * priority inheritance.
> >>>    */
> >>>
> >>> ---
> >>>
> >>> On a non-deadline related topic, the CPPC drivers creates a cppc_fie
> >>> worker in
> >>> case the CPPC counters to estimate the current frequency are in PCC
> >>> channels.
> >>> Accessing these channels requires to go through sleeping sections,
> >>> that's why a worker is used.
> >>>
> >>> However, CPPC counters might be accessed through FFH, which doesn't go
> >>> through
> >>> sleeping sections. In such case, the cppc_fie worker is never used and never
> >>> removed, so it would be nice to remote it.
> >>>
> >
>

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pierre Gondois]

On 10/16/25 14:17, Pingfan Liu wrote:
> On Thu, Oct 16, 2025 at 7:38 PM Pierre Gondois <pierre.gondois@arm.com> wrote:
>>
>> On 10/15/25 11:35, Juri Lelli wrote:
>>> On 14/10/25 21:09, Pingfan Liu wrote:
>>>> Hi Pierre,
>>>>
>>>> Thanks for sharing your perspective.
>>>>
>>>> On Sat, Oct 11, 2025 at 12:26 AM Pierre Gondois <pierre.gondois@arm.com> wrote:
>>>>> On 10/6/25 14:12, Juri Lelli wrote:
>>>>>> On 06/10/25 12:13, Pierre Gondois wrote:
>>>>>>> On 9/30/25 11:04, Peter Zijlstra wrote:
>>>>>>>> On Tue, Sep 30, 2025 at 08:20:06AM +0100, Juri Lelli wrote:
>>>>>>>>
>>>>>>>>> I actually wonder if we shouldn't make cppc_fie a "special" DEADLINE
>>>>>>>>> tasks (like schedutil [1]). IIUC that is how it is thought to behave
>>>>>>>>> already [2], but, since it's missing the SCHED_FLAG_SUGOV flag(/hack),
>>>>>>>>> it is not "transparent" from a bandwidth tracking point of view.
>>>>>>>>>
>>>>>>>>> 1 -https://elixir.bootlin.com/linux/v6.17/source/kernel/sched/cpufreq_schedutil.c#L661
>>>>>>>>> 2 -https://elixir.bootlin.com/linux/v6.17/source/drivers/cpufreq/cppc_cpufreq.c#L198
>>>>>>>> Right, I remember that hack. Bit sad its spreading, but this CPPC thing
>>>>>>>> is very much like the schedutil one, so might as well do that I suppose.
>>>>>>> IIUC, the sugov thread was switched to deadline to allow frequency updates
>>>>>>> when deadline tasks start to run. I.e. there should be no point updating the
>>>>>>> freq. after the deadline task finished running, cf [1] and [2]
>>>>>>>
>>>>>>> The CPPC FIE worker should not require to run that quickly as it seems to be
>>>>>>> more like a freq. maintenance work (the call comes from the sched tick)
>>>>>>>
>>>>>>> sched_tick()
>>>>>>> \-arch_scale_freq_tick() / topology_scale_freq_tick()
>>>>>>>      \-set_freq_scale() / cppc_scale_freq_tick()
>>>>>>>        \-irq_work_queue()
>>>>>> OK, but how much bandwidth is enough for it (on different platforms)?
>>>>>> Also, I am not sure the worker follows cpusets/root domain changes.
>>>>>>
>>>>>>
>>>>> To share some additional information, I could to reproduce the issue by
>>>>> creating as many deadline tasks with a huge bandwidth that the platform
>>>>> allows it:
>>>>> chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &
>>>>>
>>>>> Then kexec to another kernel. The available bandwidth of the root domain
>>>>> gradually decreases with the number of CPUs unplugged.
>>>>> At some point, there is not enough bandwidth and an overflow is detected.
>>>>> (Same call stack as in the original message).
>>> I seem to agree with Pingfan below, kexec (kernel crash?) is a case
>>> where all guarantees are out of the window anyway, so really no point in
>>> keeping track of bandwidth and failing hotplug. Guess we should be
>>> adding an ad-hoc check/bail for this case.
>> Yes right
>>
>>>>> So I'm not sure this is really related to the cppc_fie thread.
>>>>> I think it's more related to checking the available bandwidth in a context
>>>>> which is not appropriate. The deadline bandwidth might lack when the
>>>>> platform
>>>>> is reset, but this should not be that important.
>>>>>
>>>> I think there are two independent issues.
>>>>
>>>> In your experiment, as CPUs are hot-removed one by one, at some point
>>>> the hot-removal will fail due to insufficient DL bandwidth. There
>>>> should be a warning message to inform users about what's happening,
>>>> and users can then remove some DL tasks to continue the CPU
>>>> hot-removal.
>>>>
>>>> Meanwhile, in the kexec case, this checking can be skipped since the
>>>> system cannot roll back to a working state anyway
>> Yes right, I meant that:
>> -
>> when using kexec, the kernel crashes
>> -
>> when manually unplugging CPUs with:
>> `echo 0 > /sys/devices/system/cpu/cpuX/online`
>> The kernel returns `write error: Device or resource busy` at some point
>> to prevent
>> from reducing the DL bandwidth too much.
>>
>> ------
>>
>> I  could not reproduce the issue you reported initially. I am using
>> a radxa orion o6
>> which has a cppc_fie worker.
>>
> I speculate that you miss something like
> "isolcpus=managed_irq,domain,1-71,73-143" in the kernel command line.
> That is critical to reproduce the bug. In that case, cpus
> [1,71],[73,143] are in def_root_domain, while cpu0 and 72 are in the
> other new root_domain. The bug is triggered if cppc_fie worker is
> scheduled on cpu72.

I could finally reproduce the same issue.

The orion o6 has only 12 CPUs. I use:
   isolcpus=managed_irq,domain,1-9,11
Just before kexec, I check that the cppc_fie worker is on CPU10.

During kexec, I can see what you signal:

Upon unplugging CPU10, in this order:
- dl_bw_manage() is called and no overflow is detected as the 
non-default root domain is used - def_root_domain is attached to CPU10 - 
dl_add_task_root_domain(CPU10) is called for cppc_fie
- the resulting root domain for CPU10 is def_root_domain
- dl_clear_root_domain_cpu is called and a new bandwidth
is computed for the CPUs that are part of def_root_domain, i.e. CPU11.
(I don't know how correct this is to do that).

Upon unplugging CPU11:
- dl_bw_manage() is called, but now that there is only one CPU left in 
def_root_domain, an overflow is detected. - dl_bw_deactivate() returns 
an error code and the platform crashes

I could not see the issue previously as the cppc_fie worker was not on
the penultimate CPU (say CPU7). When unplugging CPU7:
- there is enough bandwidth on the CPUs that are part of def_root_domain,
i.e. CPU8,9, 10,11, so there is no overflow
- I assume the cppc_fie worker had the time to wake-up or be migrated
to CPU0 before reaching the last isolated CPU ...

Even by reducing the frequency at which the worker is used it seems to
only be triggered when the CPUs 1-9,11 are isolated...


>
>> AFAIU it should not be possible to add/remove bandwidth to the
>> def_root_domain.
>> During kexec, the following is happening to all CPUs:
>> \-dl_bw_manage(dl_bw_req_deactivate, cpu)
>>     \- // Check if there is enough bandwidth
>> \-dl_clear_root_domain_cpu(cpu)
>>     \- // Recompute the available bandwidth based on the remaining CPUs
>>
>> So I'm not sure to understand why accounting some bandwidth to the
>> def_root_domain
>> is problematic in practice as the def_root_domain seems to have some DL
>> bandwidth.
>>
>> IIUC the problem seems to be that for some reason there is not enough
>> bandwidth in the
>> def_root_domain aswell, which triggers the bandwidth overflow detection.
>>
> The problem is caused by accounting the blocked-state DL task's
> bandwidth to a wrong root_domain. Let me refer to the previous
> example.  cpus [1,71],[73,143] belong to def_root_domain, cpu0,72
> belong to root_domainA. In the kernel, the root_domain is traced in
> cpu_rq(cpu)->rd. But for an offline rq, rq->rd points to
> def_root_domain. Hence the reserved bandwidth of cppc_fie is wrongly
> accounted into the  def_root_domain instead of root_domainA. So
> finally, cpu143 refuses to be offlined since def_root_domain
> demonstrates there should be reserved DL bandwidth.

Yes right, I think we agree on what is happening

>
>
> Thanks,
>
> Pingfan
>>>> Thanks,
>>>>
>>>> Pingfan
>>>>> ---
>>>>>
>>>>> Question:
>>>>> Since the cppc_fie worker doesn't have the SCHED_FLAG_SUGOV flag,
>>>>> is this comment actually correct ?
>>>>> /*
>>>>>     * Fake (unused) bandwidth; workaround to "fix"
>>>>>     * priority inheritance.
>>>>>     */
>>>>>
>>>>> ---
>>>>>
>>>>> On a non-deadline related topic, the CPPC drivers creates a cppc_fie
>>>>> worker in
>>>>> case the CPPC counters to estimate the current frequency are in PCC
>>>>> channels.
>>>>> Accessing these channels requires to go through sleeping sections,
>>>>> that's why a worker is used.
>>>>>
>>>>> However, CPPC counters might be accessed through FFH, which doesn't go
>>>>> through
>>>>> sleeping sections. In such case, the cppc_fie worker is never used and never
>>>>> removed, so it would be nice to remote it.
>>>>>
>

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pingfan Liu]

On Wed, Oct 15, 2025 at 5:35 PM Juri Lelli <juri.lelli@redhat.com> wrote:
>
[...]
> > > Then kexec to another kernel. The available bandwidth of the root domain
> > > gradually decreases with the number of CPUs unplugged.
> > > At some point, there is not enough bandwidth and an overflow is detected.
> > > (Same call stack as in the original message).
>
> I seem to agree with Pingfan below, kexec (kernel crash?) is a case

It is kexec-rebooting, not crashing. The crash just leaves the other
cpus in loop instead of offlining them.

> where all guarantees are out of the window anyway, so really no point in
> keeping track of bandwidth and failing hotplug. Guess we should be
> adding an ad-hoc check/bail for this case.
>

I have finished patches for this issue, but have some trouble getting
a machine to verify it.  I will send it out later.


Thanks,

Pingfan

> > > So I'm not sure this is really related to the cppc_fie thread.
> > > I think it's more related to checking the available bandwidth in a context
> > > which is not appropriate. The deadline bandwidth might lack when the
> > > platform
> > > is reset, but this should not be that important.
> > >
> >
> > I think there are two independent issues.
> >
> > In your experiment, as CPUs are hot-removed one by one, at some point
> > the hot-removal will fail due to insufficient DL bandwidth. There
> > should be a warning message to inform users about what's happening,
> > and users can then remove some DL tasks to continue the CPU
> > hot-removal.
> >
> > Meanwhile, in the kexec case, this checking can be skipped since the
> > system cannot roll back to a working state anyway
> >
> >
> > Thanks,
> >
> > Pingfan
> > > ---
> > >
> > > Question:
> > > Since the cppc_fie worker doesn't have the SCHED_FLAG_SUGOV flag,
> > > is this comment actually correct ?
> > > /*
> > >   * Fake (unused) bandwidth; workaround to "fix"
> > >   * priority inheritance.
> > >   */
> > >
> > > ---
> > >
> > > On a non-deadline related topic, the CPPC drivers creates a cppc_fie
> > > worker in
> > > case the CPPC counters to estimate the current frequency are in PCC
> > > channels.
> > > Accessing these channels requires to go through sleeping sections,
> > > that's why a worker is used.
> > >
> > > However, CPPC counters might be accessed through FFH, which doesn't go
> > > through
> > > sleeping sections. In such case, the cppc_fie worker is never used and never
> > > removed, so it would be nice to remote it.
> > >
> >
>

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Peter Zijlstra]

On Tue, Sep 30, 2025 at 09:47:33AM +0800, Pingfan Liu wrote:

> > > This patch uses the rd from the run queue of still-active CPU to get the
> > > correct root domain.
> >
> 
> Sorry that I haven't explained it clearly. I mean the still-active CPU
> in task->cpus_ptr,
> 
> > That doesn't seem right in general. What if there are multiple root
> > domains; how does it know which to use?
> >
> 
> In the case of task->cpus_ptr, there should be only one root domain, right?

IIRC there was a corner case somewhere; something like clearing the old
cpuset load_balance flag on the root domain would not iterate all tasks
or so.

The result would be tasks with all-set cpumasks (the default value)
spread over multiple root domains. Every task would be caught in
whatever root domain it was at the time of toggle.

This might have been fixed, but I can't remember.

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pingfan Liu]

 f

On Tue, Sep 30, 2025 at 5:03 PM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Tue, Sep 30, 2025 at 09:47:33AM +0800, Pingfan Liu wrote:
>
> > > > This patch uses the rd from the run queue of still-active CPU to get the
> > > > correct root domain.
> > >
> >
> > Sorry that I haven't explained it clearly. I mean the still-active CPU
> > in task->cpus_ptr,
> >
> > > That doesn't seem right in general. What if there are multiple root
> > > domains; how does it know which to use?
> > >
> >
> > In the case of task->cpus_ptr, there should be only one root domain, right?
>
> IIRC there was a corner case somewhere; something like clearing the old
> cpuset load_balance flag on the root domain would not iterate all tasks
> or so.
>
According to the current implementation, root_domain is the toppest
cpuset, except top_cpuset, with load_balance flag. So at the top
level, it should be several disjoint CPU sets. If a top level cpuset's
load_balance flag is cleared, the rebuilt root domain which covers
this cpuset's CPU should be the one corresponding to top_cpuset. If
this is true, I think there is always one root domain.

> The result would be tasks with all-set cpumasks (the default value)
> spread over multiple root domains. Every task would be caught in
> whatever root domain it was at the time of toggle.
>

If the above is true, the tasks will have top_cpuset's
root_domain->span in cpus_ptr. And this corner case will be avoided.

Does that make sense?

Thanks,

Pingfan

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Juri Lelli]

Hello!

On 29/09/25 21:36, Pingfan Liu wrote:
> When testing kexec-reboot on a 144 cpus machine with
> isolcpus=managed_irq,domain,1-71,73-143 in kernel command line, I
> encounter the following bug:
> 
> [   97.114759] psci: CPU142 killed (polled 0 ms)
> [   97.333236] Failed to offline CPU143 - error=-16
> [   97.333246] ------------[ cut here ]------------
> [   97.342682] kernel BUG at kernel/cpu.c:1569!
> [   97.347049] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
> [   97.353281] Modules linked in: rfkill sunrpc dax_hmem cxl_acpi cxl_port cxl_core einj vfat fat arm_smmuv3_pmu nvidia_cspmu arm_spe_pmu coresight_trbe arm_cspmu_module rndis_host ipmi_ssif cdc_ether i2c_smbus spi_nor usbnet ast coresight_tmc mii ixgbe i2c_algo_bit mdio mtd coresight_funnel coresight_stm stm_core coresight_etm4x coresight cppc_cpufreq loop fuse nfnetlink xfs crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce sbsa_gwdt nvme nvme_core nvme_auth i2c_tegra acpi_power_meter acpi_ipmi ipmi_devintf ipmi_msghandler dm_mirror dm_region_hash dm_log dm_mod
> [   97.404119] CPU: 0 UID: 0 PID: 2583 Comm: kexec Kdump: loaded Not tainted 6.12.0-41.el10.aarch64 #1

Could you please confirm this is still reproducible with plain upstream
(e5f0a698b34e ("Linux 6.17") as of today)? I just wonder if we might be
missing some of the recent fixes around SCHED_DEADLINE.

Thanks,
Juri

Re: [PATCH] sched/deadline: Derive root domain from active cpu in task's cpus_ptr

[Pingfan Liu]

Hi Juri,

On Mon, Sep 29, 2025 at 10:37 PM Juri Lelli <juri.lelli@redhat.com> wrote:
>
> Hello!
>
> On 29/09/25 21:36, Pingfan Liu wrote:
> > When testing kexec-reboot on a 144 cpus machine with
> > isolcpus=managed_irq,domain,1-71,73-143 in kernel command line, I
> > encounter the following bug:
> >
> > [   97.114759] psci: CPU142 killed (polled 0 ms)
> > [   97.333236] Failed to offline CPU143 - error=-16
> > [   97.333246] ------------[ cut here ]------------
> > [   97.342682] kernel BUG at kernel/cpu.c:1569!
> > [   97.347049] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
> > [   97.353281] Modules linked in: rfkill sunrpc dax_hmem cxl_acpi cxl_port cxl_core einj vfat fat arm_smmuv3_pmu nvidia_cspmu arm_spe_pmu coresight_trbe arm_cspmu_module rndis_host ipmi_ssif cdc_ether i2c_smbus spi_nor usbnet ast coresight_tmc mii ixgbe i2c_algo_bit mdio mtd coresight_funnel coresight_stm stm_core coresight_etm4x coresight cppc_cpufreq loop fuse nfnetlink xfs crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce sbsa_gwdt nvme nvme_core nvme_auth i2c_tegra acpi_power_meter acpi_ipmi ipmi_devintf ipmi_msghandler dm_mirror dm_region_hash dm_log dm_mod
> > [   97.404119] CPU: 0 UID: 0 PID: 2583 Comm: kexec Kdump: loaded Not tainted 6.12.0-41.el10.aarch64 #1
>
> Could you please confirm this is still reproducible with plain upstream
> (e5f0a698b34e ("Linux 6.17") as of today)? I just wonder if we might be
> missing some of the recent fixes around SCHED_DEADLINE.
>

I can reproduce this bug with (9087e52ce85e Linux 6.17-rc7). I thought
that the last fix for SCHED_DEADLINE should be (a3a70caf79067
sched/deadline: Fix dl_server behaviour), which is included by -rc7
tag.

Is it good enough or should I have a test against  (e5f0a698b34e ("Linux 6.17")

Thanks,

Pingfan