Paul E. McKenney wrote:
> +/*
> + * If the specified CPU is offline, tell the caller that it is in
> + * a quiescent state.  Otherwise, whack it with a reschedule IPI.
> + * Grace periods can end up waiting on an offline CPU when that
> + * CPU is in the process of coming online -- it will be added to the
> + * rcu_node bitmasks before it actually makes it online.  Because this
> + * race is quite rare, we check for it after detecting that the grace
> + * period has been delayed rather than checking each and every CPU
> + * each and every time we start a new grace period.
> + */
> +static int rcu_implicit_offline_qs(struct rcu_data *rdp)
> +{
> +	/*
> +	 * If the CPU is offline, it is in a quiescent state.  We can
> +	 * trust its state not to change because interrupts are disabled.
> +	 */
> +	if (cpu_is_offline(rdp->cpu)) {
> +		rdp->offline_fqs++;
> +		return 1;
> +	}
>   
I fear that this won't work.
E.g. look at x86, smp_callin() [arch/x86/kernel/smpboot.c]:
The boot code must enable local interrupts around calibrate_delay(), 
otherwise the NMI watchdog would complain.
That means the first interrupts, the first read side critical sections 
can run way before the cpu bit is set within cpu_online_map.
cpus are just started, we are not within stop_machine. Thus we cannot 
make any assumption about what the remaining cpus are doing.

What about introducing a CPU_STARTING notifier call, similar to CPU_DYING:
- called with disabled interrupts
- called before interrupts are enabled
- must not sleep
- called on the new cpu.

This might also be useful for something like kvm. I'm not sure if it's 
guaranteed that hardware_enable() runs early enough.

Attached is a patch proposal. Note that it doesn't work correctly: On 
x86-64, I have seen that CPU_STARTING is called after CPU_ONLINE. Thus 
frozen_cpus could already be cleared, then _FROZEN would be wrong.

--
Manfred