Re: [PATCH] Drivers: hv: util: Implement Time Synchronization using host time sample

Re: [PATCH] Drivers: hv: util: Implement Time Synchronization using host time sample

[Olaf Hering]

On Tue, Sep 23, Thomas Shao wrote:

> In current hyper-v time sync service,it only gets the initial clock time
> from the host. It didn't process the following time samples. This change
> introduced a module parameter called host_time_sync. If it is set to true,
> the guest will periodically sychronize it's time with the host clock using
> host time sample. By default it is disabled, because we still recommend
> user to configure NTP for time synchronization.

Is that new knob a temporary thing to aid testing and development?
Because once it sneaks in it becomes some sort of ABIT.
So if the outcome of all that is that 'sync with host every once in a
while' will be the default, the new knob is not required.

Olaf

RE: [PATCH] Drivers: hv: util: Implement Time Synchronization using host time sample

[Thomas Shao]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1647 bytes --]


> -----Original Message-----
> From: Olaf Hering [mailto:olaf@aepfle.de]
> Sent: Tuesday, September 23, 2014 3:56 PM
> To: Thomas Shao
> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> driverdev-devel@linuxdriverproject.org; apw@canonical.com;
> jasowang@redhat.com; KY Srinivasan; Haiyang Zhang
> Subject: Re: [PATCH] Drivers: hv: util: Implement Time Synchronization using
> host time sample
> 
> On Tue, Sep 23, Thomas Shao wrote:
> 
> > In current hyper-v time sync service,it only gets the initial clock
> > time from the host. It didn't process the following time samples. This
> > change introduced a module parameter called host_time_sync. If it is
> > set to true, the guest will periodically sychronize it's time with the
> > host clock using host time sample. By default it is disabled, because
> > we still recommend user to configure NTP for time synchronization.
> 
> Is that new knob a temporary thing to aid testing and development?
> Because once it sneaks in it becomes some sort of ABIT.
> So if the outcome of all that is that 'sync with host every once in a while' will be
> the default, the new knob is not required.
> 

Hi Olaf, 
If the guest is configured with other time sync source, like NTP, we don't want the 
guest to periodically sync with host. This feature is useful when the guest didn't have
Internet connection to setup NTP. So the new knob is to allow users to only turn on the
host-guest sync when they need.

Regards
Thomas

> Olaf
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [PATCH] Drivers: hv: util: Implement Time Synchronization using host time sample

[Sitsofe Wheeler]

On Tue, Sep 23, 2014 at 09:56:10AM +0200, Olaf Hering wrote:
> On Tue, Sep 23, Thomas Shao wrote:
> 
> > In current hyper-v time sync service,it only gets the initial clock time
> > from the host. It didn't process the following time samples. This change
> > introduced a module parameter called host_time_sync. If it is set to true,
> > the guest will periodically sychronize it's time with the host clock using
> > host time sample. By default it is disabled, because we still recommend
> > user to configure NTP for time synchronization.

You [Microsoft?] do? Can you link to public sources where is this stated
please? I don't see any mention of doing this on
http://technet.microsoft.com/en-us/library/dn720239.aspx .. The only
official Microsoft information with respect to Hyper-V guest time
synchronisation I've seen has been for Windows guests but perhaps I've
looked in the wrong places.

The reason I ask is because regular ntpd is not enough to discipline a
Linux Hyper-V guest's clock. So much drift can occur under load that
ntpd can't bring the clock under sync. For now, I've been using Chrony
which has a higher tolerance for correcting drifting clocks. I'm not the
only one seeing this either (see http://serverfault.com/a/488528/203726
and
http://serverfault.com/questions/523389/linux-clock-loses-10-minutes-every-week
).

It would be good to something official about this issue as it is painful
when happens.

-- 
Sitsofe | http://sucs.org/~sits/

RE: [PATCH] Drivers: hv: util: Implement Time Synchronization using host time sample

[Thomas Shao]

> -----Original Message-----
> From: Sitsofe Wheeler [mailto:sitsofe@gmail.com]
> Sent: Thursday, September 25, 2014 2:08 PM
> To: Olaf Hering
> Cc: Thomas Shao; gregkh@linuxfoundation.org; linux-
> kernel@vger.kernel.org; driverdev-devel@linuxdriverproject.org;
> apw@canonical.com; jasowang@redhat.com; KY Srinivasan; Haiyang Zhang
> Subject: Re: [PATCH] Drivers: hv: util: Implement Time Synchronization using
> host time sample
> 
> On Tue, Sep 23, 2014 at 09:56:10AM +0200, Olaf Hering wrote:
> > On Tue, Sep 23, Thomas Shao wrote:
> >
> > > In current hyper-v time sync service,it only gets the initial clock
> > > time from the host. It didn't process the following time samples.
> > > This change introduced a module parameter called host_time_sync. If
> > > it is set to true, the guest will periodically sychronize it's time
> > > with the host clock using host time sample. By default it is
> > > disabled, because we still recommend user to configure NTP for time
> synchronization.
> 
> You [Microsoft?] do? Can you link to public sources where is this stated
> please? I don't see any mention of doing this on
> http://technet.microsoft.com/en-us/library/dn720239.aspx .. The only
> official Microsoft information with respect to Hyper-V guest time
> synchronisation I've seen has been for Windows guests but perhaps I've
> looked in the wrong places.
> 
> The reason I ask is because regular ntpd is not enough to discipline a Linux
> Hyper-V guest's clock. So much drift can occur under load that ntpd can't
> bring the clock under sync. For now, I've been using Chrony which has a
> higher tolerance for correcting drifting clocks. I'm not the only one seeing this
> either (see http://serverfault.com/a/488528/203726
> and
> http://serverfault.com/questions/523389/linux-clock-loses-10-minutes-
> every-week
> ).
> 
> It would be good to something official about this issue as it is painful when
> happens.
> 

Hi Sitsofe,

As far as I know, currently the document about time sync in Hyper-V is only for Windows. Below is a doc from MSDN:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx 
We are working on the document about Time Sync on Hyper-V to cover both Linux and Windows.

> --
> Sitsofe | http://sucs.org/~sits/

Re: Time keeps on slipping... on Hyper-V

[Sitsofe Wheeler]

On Thu, Sep 25, 2014 at 09:40:42AM +0000, Thomas Shao wrote:
> > > On Tue, Sep 23, Thomas Shao wrote:
> > >
> > > with the host clock using host time sample. By default it is
> > > disabled, because we still recommend user to configure NTP for time
> >
> > -----Original Message-----
> > From: Sitsofe Wheeler [mailto:sitsofe@gmail.com]
> > Sent: Thursday, September 25, 2014 2:08 PM
> >
> > You [Microsoft?] do? Can you link to public sources where is this stated
> 
> As far as I know, currently the document about time sync in Hyper-V is
> only for Windows. Below is a doc from MSDN:
> http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx

That post is problematic and not quite enough because it doesn't say "we
recommend you always configure an external time source for a guest" but
rather gives a more complicated statement saying "because guest time
drifts we have a time synchronisation service that will sort things out.
If you happen to be running a domain controller we recommend you
configure regular synchronisation to an external time source in
addition". This leads to questions like: if my (Linux) guest's sole
purpose is to run a web server am I in the domain controller situation
(domain controllers have to re-serve time so perhaps they are special)?
It also doesn't cover the "clock drift is so bad ntpd can't correct it"
issue.

Have you seen this document on TechNet -
http://blogs.technet.com/b/enterprise_admin/archive/2010/04/04/linux-on-hyper-v-timesync.aspx
"With the new Integration Services, Linux virtual machines can! 

TimeSync means that you don’t have to hassle with configur[ing]
network-based time sync for your Linux VMs on Hyper-V…they can get the
correct time from the host!"

This document gives simpler advice looks quite compelling at first
glance.

> We are working on the document about Time Sync on Hyper-V to cover
> both Linux and Windows.

The sooner you can get such an official document out the better. All
sources of guest time synchronisation information (including Microsoft
blogs and non-Microsoft Q&A sites) need updating so they say the same
(correct!) thing in a clear way.

For the meantime can you state what you believe the recommend practice
is for Linux Hyper-V guests with regards to time synchronisation?

-- 
Sitsofe | http://sucs.org/~sits/

RE: Time keeps on slipping... on Hyper-V

[Thomas Shao]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 3614 bytes --]



> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Sitsofe Wheeler
> Sent: Thursday, September 25, 2014 9:28 PM
> To: Thomas Shao
> Cc: Olaf Hering; gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> driverdev-devel@linuxdriverproject.org; apw@canonical.com;
> jasowang@redhat.com; KY Srinivasan; Haiyang Zhang
> Subject: Re: Time keeps on slipping... on Hyper-V
> 
> On Thu, Sep 25, 2014 at 09:40:42AM +0000, Thomas Shao wrote:
> > > > On Tue, Sep 23, Thomas Shao wrote:
> > > >
> > > > with the host clock using host time sample. By default it is
> > > > disabled, because we still recommend user to configure NTP for
> > > > time
> > >
> > > -----Original Message-----
> > > From: Sitsofe Wheeler [mailto:sitsofe@gmail.com]
> > > Sent: Thursday, September 25, 2014 2:08 PM
> > >
> > > You [Microsoft?] do? Can you link to public sources where is this
> > > stated
> >
> > As far as I know, currently the document about time sync in Hyper-V is
> > only for Windows. Below is a doc from MSDN:
> > http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchro
> > nization-in-hyper-v.aspx
> 
> That post is problematic and not quite enough because it doesn't say "we
> recommend you always configure an external time source for a guest" but
> rather gives a more complicated statement saying "because guest time drifts
> we have a time synchronisation service that will sort things out.
> If you happen to be running a domain controller we recommend you configure
> regular synchronisation to an external time source in addition". This leads to
> questions like: if my (Linux) guest's sole purpose is to run a web server am I in
> the domain controller situation (domain controllers have to re-serve time so
> perhaps they are special)?
> It also doesn't cover the "clock drift is so bad ntpd can't correct it"
> issue.
> 
> Have you seen this document on TechNet -
> http://blogs.technet.com/b/enterprise_admin/archive/2010/04/04/linux-on-hy
> per-v-timesync.aspx
> "With the new Integration Services, Linux virtual machines can!
> 
> TimeSync means that you don’t have to hassle with configur[ing]
> network-based time sync for your Linux VMs on Hyper-V…they can get the
> correct time from the host!"
> 
> This document gives simpler advice looks quite compelling at first glance.
> 
> > We are working on the document about Time Sync on Hyper-V to cover
> > both Linux and Windows.
> 
> The sooner you can get such an official document out the better. All sources of
> guest time synchronisation information (including Microsoft blogs and
> non-Microsoft Q&A sites) need updating so they say the same
> (correct!) thing in a clear way.
> 
> For the meantime can you state what you believe the recommend practice is
> for Linux Hyper-V guests with regards to time synchronisation?
> 

We still recommend user to configure NTP in the guest VM. With the new time sync feature in this patch, 
you could have one more option to enable the guest-host sync, if the NTP didn't work in the environment.
For example the guest VM didn't have network connection.

> --
> Sitsofe | http://sucs.org/~sits/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: Time keeps on slipping... on Hyper-V

[Mike Surcouf]

> We still recommend user to configure NTP in the guest VM. With the new time sync feature in this patch,
> you could have one more option to enable the guest-host sync, if the NTP didn't work in the environment.
> For example the guest VM didn't have network connection.

Microsoft used to use host time-samples in their older drivers but
this was dropped (when I don't know).
Now we must use NTP to correct hyperv_clocksource which suffers from
the usual problems associated with virtual environment  cpu loading.

Host time-samples in conjunction with an effective clock stability
algorithm with slews rather than steps should be the default.
NTP is a workaround and should not be the primary solution.

I have seen a lots of posts for RHEL CENTOS  Linux that have a
FAST.hyper_clocksource confirmed by myself on CENTOS 6.5 and 7.0
(also confirmed by Olaf on SLES).

If this patch steps the clock once in a while without any form of
slewing then it has the potential to break things as files and logs
travel BACK in time.

Apologies Thomas could you send me the patch via email as lkml.org
does not have it in the archives.

Regards Mike

RE: Time keeps on slipping... on Hyper-V

[Thomas Shao]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1985 bytes --]



> -----Original Message-----
> From: Mike Surcouf [mailto:mps.surcouf.lkml@gmail.com]
> Sent: Friday, September 26, 2014 5:24 PM
> To: Thomas Shao
> Cc: Sitsofe Wheeler; Olaf Hering; gregkh@linuxfoundation.org; linux-
> kernel@vger.kernel.org; driverdev-devel@linuxdriverproject.org;
> apw@canonical.com; jasowang@redhat.com; KY Srinivasan; Haiyang Zhang
> Subject: Re: Time keeps on slipping... on Hyper-V
> 
> > We still recommend user to configure NTP in the guest VM. With the new
> > time sync feature in this patch, you could have one more option to enable
> the guest-host sync, if the NTP didn't work in the environment.
> > For example the guest VM didn't have network connection.
> 
> Microsoft used to use host time-samples in their older drivers but this was
> dropped (when I don't know).
> Now we must use NTP to correct hyperv_clocksource which suffers from the
> usual problems associated with virtual environment  cpu loading.
> 
> Host time-samples in conjunction with an effective clock stability algorithm
> with slews rather than steps should be the default.
> NTP is a workaround and should not be the primary solution.
> 
> I have seen a lots of posts for RHEL CENTOS  Linux that have a
> FAST.hyper_clocksource confirmed by myself on CENTOS 6.5 and 7.0 (also
> confirmed by Olaf on SLES).
> 
> If this patch steps the clock once in a while without any form of slewing then
> it has the potential to break things as files and logs travel BACK in time.
> 
> Apologies Thomas could you send me the patch via email as lkml.org does
> not have it in the archives.
> 
Sure. I'll send the package again. It's strange that this patch didn't in the archives. And in my patch, it will slew the time, if the time drift is less than 1 s. It only steps the clock if the drift is larger than 1 s.

> Regards Mike
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥