Re: [RFC PATCH 12/30] rcu: Prepare rcu_read_[un]lock_bh() for handling softirq mask

[Joel Fernandes <joel@joelfernandes.org>]

On Thu, Oct 11, 2018 at 01:11:59AM +0200, Frederic Weisbecker wrote:
> This pair of function is implemented on top of local_bh_disable() that
> is going to handle a softirq mask in order to apply finegrained vector
> disablement. The lock function is going to return the previous vectors
> enabled mask prior to the last call to local_bh_disable(), following a
> similar model to that of local_irq_save/restore. Subsequent calls to
> local_bh_disable() and friends can then stack up:
> 
> 	bh = local_bh_disable(vec_mask);
> 		bh2 = rcu_read_lock_bh() {
> 			bh2 = local_bh_disable(...)
>             return bh2;
> 		}
> 		...
> 		rcu_read_unlock_bh(bh2) {
> 			local_bh_enable(bh2);
> 		}
> 	local_bh_enable(bh);
> 
> To prepare for that, make rcu_read_lock_bh() able to return a saved vector
> enabled mask and pass it back to rcu_read_unlock_bh(). We'll plug it
> to local_bh_disable() in a subsequent patch.

> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: David S. Miller <davem@davemloft.net>
> Cc: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  crypto/pcrypt.c                           |  5 ++--
>  drivers/infiniband/ulp/ipoib/ipoib_main.c |  5 ++--
>  drivers/net/hyperv/rndis_filter.c         |  5 ++--
>  drivers/net/macsec.c                      | 12 +++++----
>  drivers/net/vrf.c                         | 19 ++++++++------
>  drivers/vhost/net.c                       |  5 ++--
>  include/linux/rcupdate.h                  |  5 ++--
>  include/net/arp.h                         | 10 ++++---
>  include/net/ip6_fib.h                     |  1 +
>  include/net/ndisc.h                       | 10 ++++---
>  include/net/neighbour.h                   |  1 +
>  kernel/padata.c                           |  5 ++--
>  kernel/rcu/rcuperf.c                      |  2 +-
>  kernel/rcu/rcutorture.c                   |  2 +-
>  net/caif/caif_dev.c                       |  5 ++--
>  net/core/dev.c                            |  7 ++---
>  net/core/neighbour.c                      | 37 +++++++++++++++-----------
>  net/core/pktgen.c                         |  5 ++--
>  net/decnet/dn_route.c                     | 27 +++++++++++--------
>  net/ipv4/fib_semantics.c                  |  5 ++--
>  net/ipv4/ip_output.c                      |  7 ++---
>  net/ipv4/netfilter/ipt_CLUSTERIP.c        |  5 ++--
>  net/ipv6/addrconf.c                       | 21 ++++++++-------
>  net/ipv6/ip6_fib.c                        |  4 +--
>  net/ipv6/ip6_flowlabel.c                  | 43 ++++++++++++++++++-------------
>  net/ipv6/ip6_output.c                     | 12 +++++----
>  net/ipv6/route.c                          | 15 ++++++-----
>  net/ipv6/xfrm6_tunnel.c                   |  5 ++--
>  net/l2tp/l2tp_core.c                      | 33 ++++++++++++++----------
>  net/llc/llc_core.c                        |  5 ++--
>  net/llc/llc_proc.c                        | 13 +++++++---
>  net/llc/llc_sap.c                         |  5 ++--
>  net/netfilter/ipset/ip_set_core.c         | 10 ++++---
>  net/netfilter/ipset/ip_set_hash_gen.h     | 15 ++++++-----
>  net/netfilter/nfnetlink_log.c             | 17 ++++++++----
>  35 files changed, 229 insertions(+), 154 deletions(-)
> 
> diff --git a/crypto/pcrypt.c b/crypto/pcrypt.c
> index f8ec3d4..490358c 100644
> --- a/crypto/pcrypt.c
> +++ b/crypto/pcrypt.c
> @@ -73,12 +73,13 @@ struct pcrypt_aead_ctx {
>  static int pcrypt_do_parallel(struct padata_priv *padata, unsigned int *cb_cpu,
>  			      struct padata_pcrypt *pcrypt)
>  {
> +	unsigned int bh;
>  	unsigned int cpu_index, cpu, i;
>  	struct pcrypt_cpumask *cpumask;
>  
>  	cpu = *cb_cpu;
>  
> -	rcu_read_lock_bh();
> +	bh = rcu_read_lock_bh();
>  	cpumask = rcu_dereference_bh(pcrypt->cb_cpumask);
>  	if (cpumask_test_cpu(cpu, cpumask->mask))
>  			goto out;
> @@ -95,7 +96,7 @@ static int pcrypt_do_parallel(struct padata_priv *padata, unsigned int *cb_cpu,
>  	*cb_cpu = cpu;
>  
>  out:
> -	rcu_read_unlock_bh();
> +	rcu_read_unlock_bh(bh);
>  	return padata_do_parallel(pcrypt->pinst, padata, cpu);
>  }

This complicates the RCU API for -bh and doesn't look pretty at all. Is there
anything better we can do so we don't have to touch existing readers at all?

Also, I thought softirqs were kind of a thing of the past, and threaded
interrupts are the more preferred interrupt bottom halves these days,
especially for -rt. Maybe that was just wishful thinking on my part :-)

thanks,

 - Joel