[PATCH v3] sched/core: Adapt WARN_DOUBLE_CLOCK machinery for core-sched

[PATCH v3] sched/core: Adapt WARN_DOUBLE_CLOCK machinery for core-sched

[Hao Jia]

When sched_core_enabled(), we sometimes need to call update_rq_clock()
to update the rq clock of sibling CPUs on the same core, before that we
need to clear RQCF_UPDATED of rq->clock_update_flags to avoid the
WARN_DOUBLE_CLOCK warning. Because at this time the rq->clock_update_flags
of sibling CPUs may be RQCF_UPDATED. If sched_core_enabled(), we will get
a core wide rq->lock, so at this point we can safely clear RQCF_UPDATED of
rq->clock_update_flags of all CPUs on this core to avoid the
WARN_DOUBLE_CLOCK warning.

We sometimes use rq_pin_lock() and raw_spin_rq_lock() separately,
For example newidle_balance() and _double_lock_balance(). We will
temporarily give up core wide rq->lock, and then use raw_spin_rq_lock()
to reacquire core wide rq->lock without rq_pin_lock(), so We can not
clear RQCF_UPDATED of rq->clock_update_flags of other cpus on the
same core in rq_pin_lock().

Steps to reproduce:
1. Enable CONFIG_SCHED_DEBUG and CONFIG_SCHED_CORE when compiling
   the kernel
2. echo 1 > /sys/kernel/debug/clear_warn_once
   echo "WARN_DOUBLE_CLOCK" > /sys/kernel/debug/sched/features
3. Run the linux/tools/testing/selftests/sched/cs_prctl_test test

Signed-off-by: Hao Jia <jiahao.os@bytedance.com>
---
v2->v3:
 - Modify the function name to sched_core_clear_rqcf_updated,
   and add function comments.
 - Modify commit information.
 [v2] https://lore.kernel.org/all/20230215073927.97802-1-jiahao.os@bytedance.com

v1->v2:
 - Adapt WARN_DOUBLE_CLOCK machinery for core-sched instead of clearing
   WARN_DOUBLE_CLOCK warning one by one.
 - Modify commit information
 [v1] https://lore.kernel.org/all/20221206070550.31763-1-jiahao.os@bytedance.com

 kernel/sched/core.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0d18c3969f90..5e06da2f07cb 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -429,11 +429,32 @@ void sched_core_put(void)
 		schedule_work(&_work);
 }
 
+/*
+ * Now, we have obtained a core wide rq->lock, then we need to clear
+ * RQCF_UPDATED of rq->clock_update_flags of the sibiling CPU
+ * on this core to avoid the WARN_DOUBLE_CLOCK warning.
+ */
+static inline void sched_core_clear_rqcf_updated(struct rq *rq)
+{
+#ifdef CONFIG_SCHED_DEBUG
+	const struct cpumask *smt_mask;
+	int i;
+
+	if (rq->core_enabled) {
+		smt_mask = cpu_smt_mask(rq->cpu);
+		for_each_cpu(i, smt_mask) {
+			if (rq->cpu != i)
+				cpu_rq(i)->clock_update_flags &= (RQCF_REQ_SKIP|RQCF_ACT_SKIP);
+		}
+	}
+#endif
+}
 #else /* !CONFIG_SCHED_CORE */
 
 static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { }
 static inline void
 sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }
+static inline void sched_core_clear_rqcf_updated(struct rq *rq) { }
 
 #endif /* CONFIG_SCHED_CORE */
 
@@ -548,6 +569,7 @@ void raw_spin_rq_lock_nested(struct rq *rq, int subclass)
 		if (likely(lock == __rq_lockp(rq))) {
 			/* preempt_count *MUST* be > 1 */
 			preempt_enable_no_resched();
+			sched_core_clear_rqcf_updated(rq);
 			return;
 		}
 		raw_spin_unlock(lock);
-- 
2.37.0

Re: [PATCH v3] sched/core: Adapt WARN_DOUBLE_CLOCK machinery for core-sched

[Phil Auld]

Hi,

On Thu, Mar 30, 2023 at 11:58:27AM +0800 Hao Jia wrote:
> When sched_core_enabled(), we sometimes need to call update_rq_clock()
> to update the rq clock of sibling CPUs on the same core, before that we
> need to clear RQCF_UPDATED of rq->clock_update_flags to avoid the
> WARN_DOUBLE_CLOCK warning. Because at this time the rq->clock_update_flags
> of sibling CPUs may be RQCF_UPDATED. If sched_core_enabled(), we will get
> a core wide rq->lock, so at this point we can safely clear RQCF_UPDATED of
> rq->clock_update_flags of all CPUs on this core to avoid the
> WARN_DOUBLE_CLOCK warning.
> 
> We sometimes use rq_pin_lock() and raw_spin_rq_lock() separately,
> For example newidle_balance() and _double_lock_balance(). We will
> temporarily give up core wide rq->lock, and then use raw_spin_rq_lock()
> to reacquire core wide rq->lock without rq_pin_lock(), so We can not
> clear RQCF_UPDATED of rq->clock_update_flags of other cpus on the
> same core in rq_pin_lock().
> 
> Steps to reproduce:
> 1. Enable CONFIG_SCHED_DEBUG and CONFIG_SCHED_CORE when compiling
>    the kernel
> 2. echo 1 > /sys/kernel/debug/clear_warn_once
>    echo "WARN_DOUBLE_CLOCK" > /sys/kernel/debug/sched/features
> 3. Run the linux/tools/testing/selftests/sched/cs_prctl_test test
> 
> Signed-off-by: Hao Jia <jiahao.os@bytedance.com>

I think this looks good. One small nit is that I think you might
replace "core wide" with "core-wide" everywhere.


Reviewed-by: Phil Auld <pauld@redhat.com>

> ---
> v2->v3:
>  - Modify the function name to sched_core_clear_rqcf_updated,
>    and add function comments.
>  - Modify commit information.
>  [v2] https://lore.kernel.org/all/20230215073927.97802-1-jiahao.os@bytedance.com
> 
> v1->v2:
>  - Adapt WARN_DOUBLE_CLOCK machinery for core-sched instead of clearing
>    WARN_DOUBLE_CLOCK warning one by one.
>  - Modify commit information
>  [v1] https://lore.kernel.org/all/20221206070550.31763-1-jiahao.os@bytedance.com
> 
>  kernel/sched/core.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 0d18c3969f90..5e06da2f07cb 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -429,11 +429,32 @@ void sched_core_put(void)
>  		schedule_work(&_work);
>  }
>  
> +/*
> + * Now, we have obtained a core wide rq->lock, then we need to clear
> + * RQCF_UPDATED of rq->clock_update_flags of the sibiling CPU
> + * on this core to avoid the WARN_DOUBLE_CLOCK warning.
> + */
> +static inline void sched_core_clear_rqcf_updated(struct rq *rq)
> +{
> +#ifdef CONFIG_SCHED_DEBUG
> +	const struct cpumask *smt_mask;
> +	int i;
> +
> +	if (rq->core_enabled) {
> +		smt_mask = cpu_smt_mask(rq->cpu);
> +		for_each_cpu(i, smt_mask) {
> +			if (rq->cpu != i)
> +				cpu_rq(i)->clock_update_flags &= (RQCF_REQ_SKIP|RQCF_ACT_SKIP);
> +		}
> +	}
> +#endif
> +}
>  #else /* !CONFIG_SCHED_CORE */
>  
>  static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { }
>  static inline void
>  sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }
> +static inline void sched_core_clear_rqcf_updated(struct rq *rq) { }
>  
>  #endif /* CONFIG_SCHED_CORE */
>  
> @@ -548,6 +569,7 @@ void raw_spin_rq_lock_nested(struct rq *rq, int subclass)
>  		if (likely(lock == __rq_lockp(rq))) {
>  			/* preempt_count *MUST* be > 1 */
>  			preempt_enable_no_resched();
> +			sched_core_clear_rqcf_updated(rq);
>  			return;
>  		}
>  		raw_spin_unlock(lock);
> -- 
> 2.37.0
> 

-- 

Re: [External] Re: [PATCH v3] sched/core: Adapt WARN_DOUBLE_CLOCK machinery for core-sched

[Hao Jia]

On 2023/3/31 Phil Auld wrote:
> Hi,
> 
> On Thu, Mar 30, 2023 at 11:58:27AM +0800 Hao Jia wrote:
>> When sched_core_enabled(), we sometimes need to call update_rq_clock()
>> to update the rq clock of sibling CPUs on the same core, before that we
>> need to clear RQCF_UPDATED of rq->clock_update_flags to avoid the
>> WARN_DOUBLE_CLOCK warning. Because at this time the rq->clock_update_flags
>> of sibling CPUs may be RQCF_UPDATED. If sched_core_enabled(), we will get
>> a core wide rq->lock, so at this point we can safely clear RQCF_UPDATED of
>> rq->clock_update_flags of all CPUs on this core to avoid the
>> WARN_DOUBLE_CLOCK warning.
>>
>> We sometimes use rq_pin_lock() and raw_spin_rq_lock() separately,
>> For example newidle_balance() and _double_lock_balance(). We will
>> temporarily give up core wide rq->lock, and then use raw_spin_rq_lock()
>> to reacquire core wide rq->lock without rq_pin_lock(), so We can not
>> clear RQCF_UPDATED of rq->clock_update_flags of other cpus on the
>> same core in rq_pin_lock().
>>
>> Steps to reproduce:
>> 1. Enable CONFIG_SCHED_DEBUG and CONFIG_SCHED_CORE when compiling
>>     the kernel
>> 2. echo 1 > /sys/kernel/debug/clear_warn_once
>>     echo "WARN_DOUBLE_CLOCK" > /sys/kernel/debug/sched/features
>> 3. Run the linux/tools/testing/selftests/sched/cs_prctl_test test
>>
>> Signed-off-by: Hao Jia <jiahao.os@bytedance.com>
> 
> I think this looks good. One small nit is that I think you might
> replace "core wide" with "core-wide" everywhere.
> 
> 
> Reviewed-by: Phil Auld <pauld@redhat.com>
> 

Thank you for your review, I will do it in the next version.

Thanks,
Hao