processes/threads monitor

processes/threads monitor

[Angelo Borsotti]

Hello,

I wish to be personally CC'ed the answers/comments posted to the list
in response to this posting


In 2004, a thread appeared in the mailing list, stating the need for
some means to
monitor processes and threads. Patches were proposed, and alternatives solutions
hinted. However, the thread ended with no final solution to the problem.
I would like to bring the problem up again.

The need occurs on Linux installations in which there are some process
control applications,
or some continuous sever applications that need some resilience
towards process failures.
When a process providing an essential service terminates, come
"process monitor" would
create it again, restoring the service automatically (and without the
need for human intervention).
Traditionally this has been implemented in a very inefficient way by
having some process
reading periodically the list of running processes (e.g. by execing
"ps") and checking if the
processes providing essential service are still there.
This is clearly very inefficient, and all the more, it needs to be
executed repeatedly polling
processes.
A much more efficient solution would be for the process monitor to be
allowed to wait
for the termination of any processes/threads belonging to a specified list.
Note that this would mean waiting for one among several events to occur.

Are there any solution to this?

-Angelo Borsotti

Re: processes/threads monitor

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1807 bytes --]

On Sat, 31 Jan 2009 22:57:36 +0100, Angelo Borsotti said:

> The need occurs on Linux installations in which there are some process
> control applications,
> or some continuous sever applications that need some resilience
> towards process failures.
> When a process providing an essential service terminates, come
> "process monitor" would
> create it again, restoring the service automatically (and without the
> need for human intervention).

Usually, this is easily done with any sane /sbin/init, by sticking a
line in /etc/inittab that looks like:

vip:12345:respawn:/usr/bin/my-important-process

It terminates, it gets respawned.

The *tricky* case isn't dealing with a process that manages to terminate,
the real uglyness starts when you have a process that becomes wedged up
but failing to terminate (which can happen for any number of reasons - it
went into an infinite loop, or two threads managed to deadlock, or....)

And unfortunately, there's no clean general-purpose way to do *that*, mostly
because there's no good config language that allows you to code stuff like
"if Apache wedges up and fails to respond in 0.5 seconds or less, shoot it
and restart it, unless you have detected that the failure is actually due to
unrelated cause A, in which case do X, or if B happened, then do Y, or if
C happened, then do Z, or...."

If you have real requirements for all-the-time continuous operations, then
you don't *really* want a monitor anyhow.  What you *want* is redundant
hardware with some good high-availability clustering software to do hot failover.

Because if your server comes to a screeching halt because a memory card or
other hardware just bit the dust, that monitor isn't going to do *anything* for you.

Plus - if you can't do failover, how do you do a system upgrade if needed?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: processes/threads monitor

[Willy Tarreau]

On Sat, Jan 31, 2009 at 10:57:36PM +0100, Angelo Borsotti wrote:
(...)
> A much more efficient solution would be for the process monitor to be
> allowed to wait
> for the termination of any processes/threads belonging to a specified list.
> Note that this would mean waiting for one among several events to occur.
> 
> Are there any solution to this?

This is called a launcher or wrapper. Your process just has to fork/exec
the process to monitor, and it receives a sigchild when that one dies.
One well-known process does that all the time, it's called init :-)

But this does not save you from checking that your processes are properly
working. A process which just dies is the trivial case. Most unhandled
errors are from stuck processes, which need to be killed and respawned.
So if you have to check that a process is working correctly and you can
kill it, it's often easy to restart it, and you don't need your wrapper.

You're back to the script you found inefficient ;-)

Cheers,
Willy