[PATCH] rcu/nocb: Keep lazy timer intact on second lazy callback

[PATCH] rcu/nocb: Keep lazy timer intact on second lazy callback

[Chris Wiseman]

wake_nocb_gp_defer() checks waketype == RCU_NOCB_WAKE_LAZY and
rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT in one if-statement.

When the first lazy call_rcu() in a NOCB group arms rdp_gp->nocb_timer
for the 10-second lazy flush delay and sets rdp_gp->nocb_defer_wakeup to
RCU_NOCB_WAKE_LAZY, the next CPU in the group that queues its first lazy
callback calls wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, ...). Because
rdp_gp->nocb_defer_wakeup is no longer RCU_NOCB_WAKE_NOT, it misses the
first branch and falls into the else block for RCU_NOCB_WAKE, calling
mod_timer(&rdp_gp->nocb_timer, jiffies + 1) and resetting the 10-second
lazy timer down to 1 jiffy. The same fallthrough also shortens a pending
2-jiffy RCU_NOCB_WAKE_BYPASS timer to 1 jiffy.

Commit 3cb278e73be5 ("rcu: Make call_rcu() lazy to save power") wrote
this check against rdp->nocb_defer_wakeup, and commit 463d46044f04
("rcu: Fix delayed execution of hurry callbacks") switched it to
rdp_gp->nocb_defer_wakeup, which exposed the fallthrough across all CPUs
sharing rdp_gp.

Give waketype == RCU_NOCB_WAKE_LAZY its own branch and only arm
nocb_timer when rdp_gp->nocb_defer_wakeup is RCU_NOCB_WAKE_NOT.

Fixes: 3cb278e73be5 ("rcu: Make call_rcu() lazy to save power")
Fixes: 463d46044f04 ("rcu: Fix delayed execution of hurry callbacks")
Cc: stable@vger.kernel.org
Signed-off-by: Chris Wiseman <chriswiseman@google.com>
---
 kernel/rcu/tree_nocb.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index b6cfee4e5039..347d718befd0 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -335,10 +335,12 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype,
 	 * Bypass wakeup overrides previous deferments. In case of
 	 * callback storms, no need to wake up too early.
 	 */
-	if (waketype == RCU_NOCB_WAKE_LAZY &&
-	    rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
-		mod_timer(&rdp_gp->nocb_timer, jiffies + rcu_get_jiffies_lazy_flush());
-		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
+	if (waketype == RCU_NOCB_WAKE_LAZY) {
+		if (rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
+			mod_timer(&rdp_gp->nocb_timer,
+				  jiffies + rcu_get_jiffies_lazy_flush());
+			WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
+		}
 	} else if (waketype == RCU_NOCB_WAKE_BYPASS) {
 		mod_timer(&rdp_gp->nocb_timer, jiffies + 2);
 		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
-- 
2.52.0.rc1

Re: [PATCH] rcu/nocb: Keep lazy timer intact on second lazy callback

[jfern]

On chriswiseman@google.com wrote:
> wake_nocb_gp_defer() checks waketype == RCU_NOCB_WAKE_LAZY and
> rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT in one if-statement.
> 
> When the first lazy call_rcu() in a NOCB group arms rdp_gp->nocb_timer
> for the 10-second lazy flush delay and sets rdp_gp->nocb_defer_wakeup to
> RCU_NOCB_WAKE_LAZY, the next CPU in the group that queues its first lazy
> callback calls wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, ...). Because
> rdp_gp->nocb_defer_wakeup is no longer RCU_NOCB_WAKE_NOT, it misses the
> first branch and falls into the else block for RCU_NOCB_WAKE, calling
> mod_timer(&rdp_gp->nocb_timer, jiffies + 1) and resetting the 10-second
> lazy timer down to 1 jiffy. The same fallthrough also shortens a pending
> 2-jiffy RCU_NOCB_WAKE_BYPASS timer to 1 jiffy.
> 
> Commit 3cb278e73be5 ("rcu: Make call_rcu() lazy to save power") wrote
> this check against rdp->nocb_defer_wakeup, and commit 463d46044f04
> ("rcu: Fix delayed execution of hurry callbacks") switched it to
> rdp_gp->nocb_defer_wakeup, which exposed the fallthrough across all CPUs
> sharing rdp_gp.
> 
> Give waketype == RCU_NOCB_WAKE_LAZY its own branch and only arm
> nocb_timer when rdp_gp->nocb_defer_wakeup is RCU_NOCB_WAKE_NOT.
> 
> Fixes: 3cb278e73be5 ("rcu: Make call_rcu() lazy to save power")
> Fixes: 463d46044f04 ("rcu: Fix delayed execution of hurry callbacks")
> Cc: stable@vger.kernel.org
> Signed-off-by: Chris Wiseman <chriswiseman@google.com>

Looks reasonable to me. Is there a test case that you added to find this, or any
power savings that you found with this added patch?

thanks,

Joel Fernandes

> ---
>  kernel/rcu/tree_nocb.h | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> index b6cfee4e5039..347d718befd0 100644
> --- a/kernel/rcu/tree_nocb.h
> +++ b/kernel/rcu/tree_nocb.h
> @@ -335,10 +335,12 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype,
>  	 * Bypass wakeup overrides previous deferments. In case of
>  	 * callback storms, no need to wake up too early.
>  	 */
> -	if (waketype == RCU_NOCB_WAKE_LAZY &&
> -	    rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
> -		mod_timer(&rdp_gp->nocb_timer, jiffies + rcu_get_jiffies_lazy_flush());
> -		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
> +	if (waketype == RCU_NOCB_WAKE_LAZY) {
> +		if (rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
> +			mod_timer(&rdp_gp->nocb_timer,
> +				  jiffies + rcu_get_jiffies_lazy_flush());
> +			WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
> +		}
>  	} else if (waketype == RCU_NOCB_WAKE_BYPASS) {
>  		mod_timer(&rdp_gp->nocb_timer, jiffies + 2);
>  		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
> -- 
> 2.52.0.rc1

Re: [PATCH] rcu/nocb: Keep lazy timer intact on second lazy callback

[Chris Wiseman]

Hi Joel,

> Looks reasonable to me. Is there a test case that you added to find this, or any
> power savings that you found with this added patch?

I spotted this while tracing idle rcuog wakeups with CONFIG_RCU_LAZY=y
and rcu_nocbs=all, where rcuog kept waking up after 1 jiffy whenever a
second CPU in the group queued a lazy callback.  I verified it with
rcutorture (NOCB01) and measured a light background callback workload on
an arm64 system:

                        Before          After           Delta
rcuog wakeups:          1,885           1,174           -37.7%
1-jiffy timer resets:     739               0          -100.0%
SoC power (CPU+DSU):   94.1 mW         76.5 mW          -18.6% (-17.5 mW)

Thanks,
Chris

Re: [PATCH] rcu/nocb: Keep lazy timer intact on second lazy callback

[Joel Fernandes]

On chriswiseman@google.com wrote:
> Hi Joel,
> 
> > Looks reasonable to me. Is there a test case that you added to find this, or any
> > power savings that you found with this added patch?
> 
> I spotted this while tracing idle rcuog wakeups with CONFIG_RCU_LAZY=y
> and rcu_nocbs=all, where rcuog kept waking up after 1 jiffy whenever a
> second CPU in the group queued a lazy callback.  I verified it with
> rcutorture (NOCB01) and measured a light background callback workload on
> an arm64 system:
> 
>                         Before          After           Delta
> rcuog wakeups:          1,885           1,174           -37.7%
> 1-jiffy timer resets:     739               0          -100.0%
> SoC power (CPU+DSU):   94.1 mW         76.5 mW          -18.6% (-17.5 mW)
> 

Looks nice! I will apply the patch and run some tests.

Reviewed-by: Joel Fernandes <jfern@kernel.org>

> Thanks,
> Chris