[PATCH v3 06/14] rcu: set need_resched on softirq deferred-QS arming path

[Joel Fernandes <joelagnelf@nvidia.com>]

The arming code in rcu_read_unlock_special() has two paths for
deferring QS reporting: a softirq raise and an irq_work/set_need_resched
combination.

The irq_work path always calls set_need_resched_current() before
queuing irq_work. The softirq path does not, relying solely on the
softirq firing to do the rightthing.

This results in a problem as follows:

Consider 2 rcu_read_lock/unlock segments:

        rcu_read_lock();            // segment 1 starts
        // needs_exp becomes true
        preempt_disable();
        rcu_read_unlock();          // segment 1 ends; IRQs on, preempt
                                    // off, needs_exp=true =>
                                    //   raise_softirq(RCU_SOFTIRQ);
                                    //   arms defer_qs_pending.
                                    // Before this fix: no
                                    // set_need_resched_current().
        local_irq_disable();
        preempt_enable();           // softirq pending but IRQs disabled
                                    // hold it off.
        rcu_read_lock();            // segment 2 starts
        local_irq_enable();         // softirq fires: rcu_core runs, but
                                    // we are inside a reader (depth>0)
                                    // so no QS report; on softirq-exit
                                    // preempt-check finds no
                                    // need_resched -- still no nudge.
        preempt_disable();
        rcu_read_unlock();          // arming attempt suppressed
                                    // incorrectly: defer_qs_pending
                                    // already PENDING.  Without this
                                    // fix, no fresh
                                    // set_need_resched_current() on
                                    // this path either.
        preempt_enable();

Therefore, add set_need_resched_current() to the softirq deferral path to
avoid long latencies in situations where GP needs to end sooner.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 kernel/rcu/tree_plugin.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index d17c2d97ebcc..7045f0deee17 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -765,6 +765,7 @@ static void rcu_read_unlock_special(struct task_struct *t)
 			// Using softirq, safe to awaken, and either the
 			// wakeup is free or there is either an expedited
 			// GP in flight or a potential need to deboost.
+			set_need_resched_current();
 			if (rdp->defer_qs_pending != DEFER_QS_PENDING) {
 				rdp->defer_qs_pending = DEFER_QS_PENDING;
 				raise_softirq_irqoff(RCU_SOFTIRQ);
-- 
2.34.1