NMI watchdog logic on X86-64

NMI watchdog logic on X86-64

[Stephane Eranian]

Hello,

In the context of the perfmon2 interface, I am trying to understand
the NMI watchdog logic on X86-64 (I think this is identical on IA-32).

I am looking at the case where the NMI watchdog uses the Local APIC
and the performance counters (AMD or P4). The code of reference
is in 2.6.15.rc5-git1 arch/x86_64/kernel/nmi.c.

It is not clear to me what is going in the case of an SMP machine.

I would like to use reserve_lapic_nmi() and release_lapic_nmi()
from perfmon2 to start/stop the NMI watchdog while we have active
monitoring (i.e., get the performance counters back). The perfctr
subsystem is coded similarly.

The call reserve_lapic_nmi() invokes enable_lapic_nmi_watchdog() which
ends up in setup_apic_nmi_watchdog(). That function is executed only
on the calling processor. In SMP it means one CPU does the NMI watchdog.

In the case of the LAPIC APIC, this means that the call to release_lapic_nmi()
MUST be done on the same CPU as the one used for reserve_lapic_nmi() otherwise
the NMI watchdog setup for the performance counters will not be quiesced.

I would like to make sure that the logic of the code is that ONLY one processor
does the NMI watchdog timer?

In that case, I think it would be good to document or enforce the fact that when
Local APIC is used, calls to reserve and release must be made on the same processor.

Am I missing something here?

-- 
-Stephane

Re: NMI watchdog logic on X86-64

[Stephane Eranian]

Eric,

On Sat, Dec 17, 2005 at 02:10:42PM -0700, Eric W. Biederman wrote:
> >> I would like to make sure that the logic of the code is that ONLY one
> >> processor does the NMI watchdog timer?
> 
> That much I answer.  The logic is that each processor does the NMI watchdog.
> 
I checked again and there is something that looks broken to me. I agree with you
that the NMI watchdog must be per-CPU. In fact it is setup per local APIC in
smpboot.c. Yet what is confusing is that the owner variable lapic_nmi_owner
is shared with spinlock. So you either setup NMI on all CPUs or none. 

That also implies that the reserve_lapic_nmi() function must be called only
once. If called for each CPU, it will fail for the second caller because
it will see the lapic_nmi_owner already set to RESERVED. But if called only
once, then the the disable logic in reserve_lapic_nmi() is wrong:

int reserve_lapic_nmi(void)
{
        unsigned int old_owner;

        spin_lock(&lapic_nmi_owner_lock);
        old_owner = lapic_nmi_owner;
        lapic_nmi_owner |= LAPIC_NMI_RESERVED;
        spin_unlock(&lapic_nmi_owner_lock);
        if (old_owner & LAPIC_NMI_RESERVED)
                return -EBUSY;
        if (old_owner & LAPIC_NMI_WATCHDOG)
                disable_lapic_nmi_watchdog();
        return 0;
}

If reserve_lapic_nmi() is called only once, then it will call
disable_lapic_nmi_watchdog() on ONLY one CPU even though it was
setup on all CPUs in smpboot.c. The function disable_lapic_nmi_watchdog()
should be called on all CPUs (once nmi_active, nmi_watchdog are moved
out of it). 

It appears to me that the lapic_nmi_owner variable somehow needs
to become a per-cpu variable. Then the reserve_lapic_nmi() would
have to be called on CPUs as necessary.

-- 
-Stephane