Weird timer handler used with LAPIC selected

Weird timer handler used with LAPIC selected

[limp]

Hi all,

I have configured my kernel (2.6.31.6) so that I get periodic ticks from the
timer interrupt (i.e. tickless is not configured) with a rate determined by
the HZ value. I have also selected the "Local APIC support on uniprocessors"
option so that the LAPIC timer is used as a tick source. The problem is that
the handler used for the timer is not the "local_apic_timer_interrupt"
handler as I was excepted but the "tick_handle_periodic" handler. This can
be verified by seeing the output of /proc/timer_list:

Tick Device: mode:     0
Per CPU device: 0
Clock Event Device: lapic
 max_delta_ns:   1006457290
 min_delta_ns:   1799
 mult:           35797637
 shift:          32
 mode:           1
 next_event:     9223372036854775807 nsecs
 set_next_event: lapic_next_event
 set_mode:       lapic_timer_setup
 event_handler:  tick_handle_periodic

Is there an explanation for that? The board in which I am running the kernel
is a Kontron 986LCD-M/mITX one with an Intel Celeron M 440 installed on it.
I have verified that Intel Celeron M Processors have a LAPIC and
consequently, a LAPIC timer.

--------------------------------------------
Another weird thing that I noticed is that by running the same kernel but
configured for "Core 2/newer Xeon" processors, the timer handler used is the
"hrtimer_interrupt" handler. Again, this is verified by /proc/timer_list: 

Tick Device: mode:     1
Per CPU device: 0
Clock Event Device: lapic
event_handler:  hrtimer_interrupt

I have put some printks inside "local_apic_timer_interrupt" handler which
should never be executed as the handler selected for the timer is
"hrtimer_interrupt" and not "local_apic_timer_interrupt" as verified.
However, when I execute a user process which is just a long time delay, the
printks from the "local_apic_timer_interrupt" are printed. 

If you need more info about something, please ask.

Any help will be much appreciated.

Thanks in advance.

P.S. please CC me.

-John

Re: Weird timer handler used with LAPIC selected

[john stultz]

On Fri, May 7, 2010 at 4:10 AM, limp <johnkyr83@hotmail.com> wrote:
> Hi all,
>
> I have configured my kernel (2.6.31.6) so that I get periodic ticks from the
> timer interrupt (i.e. tickless is not configured) with a rate determined by
> the HZ value. I have also selected the "Local APIC support on uniprocessors"
> option so that the LAPIC timer is used as a tick source. The problem is that
> the handler used for the timer is not the "local_apic_timer_interrupt"
> handler as I was excepted but the "tick_handle_periodic" handler. This can
> be verified by seeing the output of /proc/timer_list:
>
> Tick Device: mode:     0
> Per CPU device: 0
> Clock Event Device: lapic
>  max_delta_ns:   1006457290
>  min_delta_ns:   1799
>  mult:           35797637
>  shift:          32
>  mode:           1
>  next_event:     9223372036854775807 nsecs
>  set_next_event: lapic_next_event
>  set_mode:       lapic_timer_setup
>  event_handler:  tick_handle_periodic

You might want to read the code in further detail. The clockevent
device is an abstraction on top of the actual hardware, so the
event_handler method is not mapped into the irq handler directly.

So the local_apic_timer_interrupt() is called by the apic hardware
interrupt, it then calls lapic clockevent device's event_handler()
method, which is set to tick_handle_periodic(). Since you're in
_PERIODIC mode, not _ONESHOT (hrtimers/nohz), this seems correct to
me.



> Is there an explanation for that? The board in which I am running the kernel
> is a Kontron 986LCD-M/mITX one with an Intel Celeron M 440 installed on it.
> I have verified that Intel Celeron M Processors have a LAPIC and
> consequently, a LAPIC timer.
>
> --------------------------------------------
> Another weird thing that I noticed is that by running the same kernel but
> configured for "Core 2/newer Xeon" processors, the timer handler used is the
> "hrtimer_interrupt" handler. Again, this is verified by /proc/timer_list:
>
> Tick Device: mode:     1
> Per CPU device: 0
> Clock Event Device: lapic
> event_handler:  hrtimer_interrupt

Yep. On this system I'm guessing you have hrtimers or nohz/tickless enabled?


> I have put some printks inside "local_apic_timer_interrupt" handler which
> should never be executed as the handler selected for the timer is
> "hrtimer_interrupt" and not "local_apic_timer_interrupt" as verified.

Nope. Again, local_apic_timer_interrupt is the actual irq hook. It
then calls the lapic clockevent device's event_handler() method, which
is hrtimer_interrupt.

thanks
-john

RE: Weird timer handler used with LAPIC selected

[limp]

Hi John and thanks for your reply,

>You might want to read the code in further detail. The clockevent
>device is an abstraction on top of the actual hardware, so the
>event_handler method is not mapped into the irq handler directly.
>
>So the local_apic_timer_interrupt() is called by the apic hardware
>interrupt, it then calls lapic clockevent device's event_handler()
>method, which is set to tick_handle_periodic(). Since you're in
>_PERIODIC mode, not _ONESHOT (hrtimers/nohz), this seems correct to
>me.

I have examined the code in more detail and now I see what you're saying.
The thing that is confusing me a bit is why the LAPIC interrupt triggers
the local_apic_timer_interrupt() and not the timer_interrupt() in time.c?
ASAIU, every timer IRQ should call timer_interrupt() and for there, 
according to the selected clocksource, the appropriate event_handler should
be called but for LAPIC this is not the case...

>> I have put some printks inside "local_apic_timer_interrupt" handler which
>> should never be executed as the handler selected for the timer is
>> "hrtimer_interrupt" and not "local_apic_timer_interrupt" as verified.
>
>Nope. Again, local_apic_timer_interrupt is the actual irq hook. It
>then calls the lapic clockevent device's event_handler() method, which
>is hrtimer_interrupt.

Hmm, I am quite sure that hrtimer_interrupt() event_handler is called from
timer_interrupt() and not from local_apic_timer_interrupt()..

The reason for not getting any printks from the "local_apic_timer_interrupt"
irq hook was because I was getting about 318-360 LOC (Local timer
interrupts)
- the number was varied at every system boot - and then they were stopped
and the
LOC number was not incremented!!!! I've put a printk in lapic_timer_setup
and it
seems that LAPIC was going to shutdown mode after a while. 

\x0301[14:43] <limp> ...
\x0301[14:43] <limp> [    0.046999] LIMP: lapic_timer_setup: the mode used is 1
\x0301[14:43] <limp> [    0.046999] LIMP: lapic_timer_setup: the mode used is 2
\x0301[14:43] <limp> ...
\x0301[14:43] <limp> [    0.699077] LIMP: lapic_timer_setup: the mode used is 1

This solved by disabling ACPI from the kernel command line. After that, I
was
Getting LOCs at HZ rate and irq0_timer were stalled at boot time after some
initial ones (I guess there is no sense of getting LOCs and irq0_timer ticks
simultaneously on a uniprocessor target).  

It seems that the system were entering a deep sleep state and was never
waking
up. Again, this seems a bit weird as the irq0 timer interrupt was still
triggered
periodically at HZ rate and should have not let ACPI to put LAPIC timer into
sleep...

Another question is this: Is there a way to know when my target enters an
ACPI
power state? Also, is there a way to prevent my target for entering into
power
states, without having to disable the whole ACPI support (something like 
acpi_sleep=false)? 

Many thanks for the useful help.

Kind Regards

John K.

RE: Weird timer handler used with LAPIC selected

[john stultz]

On Thu, 2010-05-13 at 11:42 +0100, limp wrote:
> >> I have put some printks inside "local_apic_timer_interrupt" handler which
> >> should never be executed as the handler selected for the timer is
> >> "hrtimer_interrupt" and not "local_apic_timer_interrupt" as verified.
> >
> >Nope. Again, local_apic_timer_interrupt is the actual irq hook. It
> >then calls the lapic clockevent device's event_handler() method, which
> >is hrtimer_interrupt.
> 
> Hmm, I am quite sure that hrtimer_interrupt() event_handler is called from
> timer_interrupt() and not from local_apic_timer_interrupt()..

It can actually be called from both. From your earlier mail:
	Tick Device: mode:     1
	Per CPU device: 0
	Clock Event Device: lapic
	event_handler:  hrtimer_interrupt

That means when local_apic_timer_interrupt calls the lapic clockevent
device's event_handler() method, it will call hrtimer_interrupt.


> The reason for not getting any printks from the "local_apic_timer_interrupt"
> irq hook was because I was getting about 318-360 LOC (Local timer
> interrupts)
> - the number was varied at every system boot - and then they were stopped
> and the
> LOC number was not incremented!!!! I've put a printk in lapic_timer_setup
> and it
> seems that LAPIC was going to shutdown mode after a while. 
> 
> \x0301[14:43] <limp> ...
> \x0301[14:43] <limp> [    0.046999] LIMP: lapic_timer_setup: the mode used is 1
> \x0301[14:43] <limp> [    0.046999] LIMP: lapic_timer_setup: the mode used is 2
> \x0301[14:43] <limp> ...
> \x0301[14:43] <limp> [    0.699077] LIMP: lapic_timer_setup: the mode used is 1
> 
> This solved by disabling ACPI from the kernel command line. After that, I
> was
> Getting LOCs at HZ rate and irq0_timer were stalled at boot time after some
> initial ones (I guess there is no sense of getting LOCs and irq0_timer ticks
> simultaneously on a uniprocessor target).  

Right, depending on the system and the configuration, we try to not have
duplicate irqs landing on the same cpu. So either the irq0 will be
shutdown once the lapic is up, or the lapic will be stopped if we're in
broadcast mode.

> It seems that the system were entering a deep sleep state and was never
> waking
> up. Again, this seems a bit weird as the irq0 timer interrupt was still
> triggered
> periodically at HZ rate and should have not let ACPI to put LAPIC timer into
> sleep...

Right. So on this system, its likely the LAPIC halts in deep idle states
(normally C3). So there will be another tick device (hpet or pit
usually) which is configured in broadcast mode and will trigger
frequently to wake the specified idled cpus up. If you look at your
timer_list in detail it should provide some info to that effect.


> Another question is this: Is there a way to know when my target enters an
> ACPI
> power state? Also, is there a way to prevent my target for entering into
> power
> states, without having to disable the whole ACPI support (something like 
> acpi_sleep=false)? 

There's a couple of ways. idle=poll will keep the system from going into
deep sleep modes, but at the cost of burning energy when there's nothing
to do. You might read over the idle= options in
Documentation/kernel-parameters.txt

Good luck!

thanks
-john