RE: Why yield in coredump_wait? [was: Re: Resent: BUG in RT 45-01whenRT program dumps core]

RE: Why yield in coredump_wait? [was: Re: Resent: BUG in RT 45-01whenRT program dumps core]

[kus Kusche Klaus]

> On Thu, 2005-05-19 at 10:37 -0400, Steven Rostedt wrote:
> > On Thu, 2005-05-19 at 16:23 +0200, kus Kusche Klaus wrote:
> > > Does that mean that the core dump is written 
> > > with the rt prio of the task which dumps?
> > 
> > Yes, since the process itself that crashed is what is 
> writing the core.
> > So if a RT process crashes, it writes the core as whatever it was.
> > 
> > > I'm not sure if this is a good idea: 
> > > Dumping a big core might take *ages* (at least w.r.t. realtime),
> > > especially because it usually goes to flash memory, a CF card,
> > > or some other really slow device.
> > > 
> > 
> > This is interesting, since if a RT task is dumping core, 
> that usually
> > means that it crashed, and therefore there's a bug in the 
> system.  Also,
> > unless the processes is writing to something that requires 
> a busy wait
> > (which the serial might do, and probably some flashes), 
> this shouldn't
> > effect the system.
> 
> Interesting indeed.  This could be caused by (possibly transient)
> hardware failure as well as a bug.  How do mission critical hard RT
> applications typically handle disasters like the RT process dumping
> core?  Presumably you have a hardware or software watchdog, and drop
> into some kind of safe mode.  It seems that you would need redundant
> systems if you wanted to continue to handle the RT constraint while
> recovering.
> 
> Lee

First of all, yes, we are talking about busy waits:
The CF cards run in PIO mode, i.e. the CPU polls and copies the data,
and we know from measurements that this causes 100 % CPU load and
blocks anything at lower prio for extended periods.

Of course, in some cases you are in big trouble anyway 
if some RT process in a control system dumps core,
and it will not matter any more at which prio core is dumped.

However, it might also be a non-critical part of the system,
and in this case, other parts should continue at normal rate.
If some part of the system which was designed to occupy the cpu
exclusively for let's say at most 50 microseconds suddenly 
monopolizes it for 5 seconds, that will cause surprises...

Moreover, the lowest-pri parts of a control system are usually the
graphical user interfaces. We are talking about core dump times
in the order of 5 or 10 seconds, and the operators will panic madly
if the system does not react to any user interaction and does not
update its display for that long.

Similarly, log daemons should spread word about the problem 
immediately, not 10 seconds late.

So we are not yet at the question how to recover and how to
continue. 

First of all, 
we want to survive the core dumping itself gracefully, 
we don't want to increase damage by dumping core, 
and we want to be able to inform and take action about the trouble
before the core has finished dumping.

Greetings

-- 
Klaus Kusche                 (Software Development - Control Systems)
KEBA AG             Gewerbepark Urfahr, A-4041 Linz, Austria (Europe)
Tel: +43 / 732 / 7090-3120                 Fax: +43 / 732 / 7090-6301
E-Mail: kus@keba.com                                WWW: www.keba.com

RE: Why yield in coredump_wait? [was: Re: Resent: BUG in RT 45-01whenRT program dumps core]

[Steven Rostedt]

On Fri, 2005-05-20 at 13:30 +0200, kus Kusche Klaus wrote:
> 
> First of all, yes, we are talking about busy waits:
> The CF cards run in PIO mode, i.e. the CPU polls and copies the data,
> and we know from measurements that this causes 100 % CPU load and
> blocks anything at lower prio for extended periods.
> 

Yeah, I figured as much.  I started looking at some of the flash code
and did see busy waits, usually with timeouts, but that's still not
acceptable in RT.

> Of course, in some cases you are in big trouble anyway 
> if some RT process in a control system dumps core,
> and it will not matter any more at which prio core is dumped.
> 

I'll send a patch to Ingo to at least drop the real time priority of a
process dumping core. But then the next question is, what should the
core priority be? Meaning, what nice value?  Should this be a sysctl
value?  For now it should probably just drop to nice 0.

> However, it might also be a non-critical part of the system,
> and in this case, other parts should continue at normal rate.
> If some part of the system which was designed to occupy the cpu
> exclusively for let's say at most 50 microseconds suddenly 
> monopolizes it for 5 seconds, that will cause surprises...
> 

I totally agree.

> Moreover, the lowest-pri parts of a control system are usually the
> graphical user interfaces. We are talking about core dump times
> in the order of 5 or 10 seconds, and the operators will panic madly
> if the system does not react to any user interaction and does not
> update its display for that long.

If the core dump drops the task out of RT, then the user interface
should still be responding. Here the nice value may be a factor to, but
you shouldn't see 5 or 10 seconds waiting (unless you are running
Evolution, but then that's just normal! :-)

> Similarly, log daemons should spread word about the problem 
> immediately, not 10 seconds late.

I'm not sure that this should be a kernel feature.  Maybe someday there
can be a notify event of when a process dumps core, but I suspect that
would be a low priority.  For now, since a core dump causes all tasks
sharing the memory to exit immediately, you may be able to figure out
how to monitor this externally. Or at least, check for activity of
something writing to core. 

Also, since a core dump is usually caused by a signal (is there any
other way?) you can have all the RT tasks catching these signals and try
to do something appropriately first.  But then again, a SIGSEGV is
usually something that makes the program unpredictable in continuing.
Just make sure your notifier doesn't do much (allocation or anything)
and is pretty bug free.

> So we are not yet at the question how to recover and how to
> continue. 
> 
> First of all, 
> we want to survive the core dumping itself gracefully, 

Capture the signal.

> we don't want to increase damage by dumping core, 

I'll work on lowering the priority.

> and we want to be able to inform and take action about the trouble
> before the core has finished dumping.

Again, do this via the signal handler.

> Greetings
> 

-- Steve