Re: [RFC patch 15/15] LTTng timestamp x86

[Linus Torvalds <torvalds@linux-foundation.org>]

On Tue, 21 Oct 2008, Bjorn Helgaas wrote:

> On Monday 20 October 2008 04:29:07 pm H. Peter Anvin wrote:
> > Linus Torvalds wrote:
> > > 
> > > But it's not going to solve the "hey, I have 512 CPU's, they are all on 
> > > different boards, and no, they are _not_ synchronized to one global 
> > > clock!".
> > > 
> > > That's why I'd suggest making _purely_ local time, and then aiming for 
> > > something NTP-like. But maybe there are better solutions out there.
> > 
> > At the same time, it would definitely be nice to encourage vendors of 
> > large SMP systems to provide a common root crystal (frequency standard) 
> > for a single SMP domain.  Preferrably a really good one, TCXO or better.
> 
> A single root crystal is nice for us software guys.  But it often
> also turns into a single point of failure, which the hardware guys
> are always trying to eliminate.  So I think multiple crystals are
> inevitable for the really large machines.

They are almost inevitable for another reason too: the interconnect seldom 
has a concept of "clock signal" other than for the signalling itself, and 
the signal clock is designed for the signal itself and is designed for 
signal integrity rather than "stable clock".

Does _any_ common interconnect have integral support for clock 
distribution?

And no, nobody is going to add another clock network for just clock 
distribution. 

So even ignoring redundancy issues, and the fact that people want to 
hot-plug things (and yes, that would make a central clock interesting), I 
doubt any hw manufacturer really looks at it the way we do.

The best we could hope for is some hardware assist for helping distribute 
a common clock. Ie not a real single crystal, but having time-packets in 
the interconnect that are used to synchronize nodes whenever there is 
communication between them. It's hard to do that in software, because the 
overhead is fairly high, but if hardware does at least some of it you 
could probably get a damn fine distributed clock source.

But I don't know if any hw people are worried enough about it to do it...

		Linus