Re: [PATCH] watchdog: Make sure the watchdog thread gets CPU on loaded system

[Michal Hocko <mhocko@suse.cz>]

On Thu 15-03-12 11:54:13, Don Zickus wrote:
> On Thu, Mar 15, 2012 at 09:02:32AM +0100, Michal Hocko wrote:
> > On Wed 14-03-12 16:19:06, Andrew Morton wrote:
> > > On Wed, 14 Mar 2012 16:38:45 -0400
> > > Don Zickus <dzickus@redhat.com> wrote:
> > > 
> > > > From: Michal Hocko <mhocko@suse.cz>
> > > 
> > > This changelog is awful.
> 
> My apologies too, Andrew for not being more diligent.
> 
> Some nitpicks below (hopefully it isn't too picky :-( )

Thanks! Updated
---
>From a8da58750ba78d737136a4df24af805cb936ee00 Mon Sep 17 00:00:00 2001
From: Michal Hocko <mhocko@suse.cz>
Date: Tue, 13 Mar 2012 10:34:44 +0100
Subject: [PATCH] watchdog: make sure the watchdog thread gets CPU on loaded
 system

If the system is heavily loaded while hotplugging a CPU, we might end up
with a bogus hardlockup detection. This has been seen during LTP pounder
test executed in parallel with the hotplug test.

Hard lockup detector consist of two parts
	- watchdog_overflow_callback (executed as a perf counter callback
	  from NMI) which checks whether per-cpu hrtimer_interrupts changed
	  since the last time it run and panics if not
	- watchdog kernel thread which starts watchdog_hrtimer which
	  periodically updates hrtimer_interrupts.

The main problem is that watchdog_enable (called when a CPU is brought up)
registers a perf event but the hrtimer is started later when the watchdog
thread gets a chance to run.

The watchdog thread starts with a normal priority currently and boosts
itself as soon as it gets to a CPU. This might be, however, already too
late as demonstrated with the LTP pounder test executed in parallel by
LTP hotplug test. There are zillions of userspace processes sitting in
the runque while the number of online CPUs gets down to 1. CPUs are
onlined back in the second stage where the issue triggers.

When we online a CPU and create the watchdog kernel thread it will take
some time until it gets to a CPU. On the other hand the perf counter
callback is executed in the timely fashion so we explode the first time
it finds out that the hrtimer_interrupts wasn't incremented.

Let's fix this by boosting the watchdog thread priority before we wake it up
rather than when it's already running.
This still doesn't handle a case where we have the same amount of high prio
FIFO tasks but that doesn't seem to be common. The current implementation
doesn't handle that case anyway so this is no worse at least.

Unfortunately, we cannot start perf counter from the watchdog thread
because we could miss a real lock up and also we cannot start the
hrtimer from watchdog_enable because we there is no way (at least I
don't know any) to start a hrtimer from a different CPU.
-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic