Re: [BUG] Boot hangs at clocksource_done_booting on large configs

[Thomas Gleixner <tglx@linutronix.de>]

On Tue, 1 Sep 2015, Alex Thorlton wrote:
> On Mon, Aug 31, 2015 at 11:12:12PM +0200, Thomas Gleixner wrote:
> > But they are obviously held off by something. And that something looks
> > like the timekeeper sequence lock. 
> 
> Ok.  IIRC, that lock prevents reads to the clocksource until all
> necessary write operations have finished, correct?  i.e. tk_core.seq?
> 
> I looked at that as being a possible place to get stuck in ktime_get:
> 
>         do {
>                 seq = read_seqcount_begin(&tk_core.seq);
>                 base = tk->tkr_mono.base;
>                 nsecs = timekeeping_get_ns(&tk->tkr_mono);
> 
>         } while (read_seqcount_retry(&tk_core.seq, seq));
> 
> But I don't have enough evidence to point there for sure.

Right and it might be a red herring. If the access to HPET takes long
enough for each CPU then you get into a situation where you never get
out of those timer interrupts anymore.
 
> > Your patch shifts some timing around so the issue does not happen, but
> > that's certainly not a solution.
> 
> I was fairly sure of that, but it's good to know that you agree.
> 
> I'll try and get a backtrace from a completely unmodified kernel here in
> the next few days.  Unfortunately, as mentioned, we only hit the issue
> intermittently.  I was able to reproduce pretty regularly on kernels
> from 4.0 and before, but it seems to be less common on newer kernels,
> though we definitely still manage to hit it from time to time.

Well, I doubt that backtraces will tell us much. Some moderate form of
tracing (a few tracepoints enabled) and a trace_dump(DUMP_ALL); call
before or instead of the NMI backtrace stuff might give us the really
interesting information. i.e. what the cpus have been doing for the
last second or such. The backtraces are just momentary snapshots and
not necessarily helpful.

You might have to hack the trace dump code to limit the output to a
few cpus if you don't want to wait a day or two until the information
has spilled out over serial :)

Thanks,

	tglx