RFC : remote driver debugging efforts

RFC : remote driver debugging efforts

[Om N.]

I am trying to debug a driver ported from 2.4 to 2.6 for a friend. But
the device is custom made and not available anywhere else, only with
my friend.
he suggested me that I do remote debugging using telnet/ssh.

For all driver development efforts I have carried out so far, I had
free and close proximity access to the device, the target machine and
a serial port. Now
in this case, without a serial port console, and without remote power off/on
facilities, I would like to ask the list about the options I have.

(I do not have a remote power on/off switch. The driver panics so
often that somebody has to babysit the machine to switch it off and
on. We are in different time zones and things are not moving forward at all)

Regards,
Om.

Re: RFC : remote driver debugging efforts

[Jesse Brandeburg]

On 8/10/06, Om N. <xhandle@gmail.com> wrote:
> (I do not have a remote power on/off switch. The driver panics so
> often that somebody has to babysit the machine to switch it off and
> on. We are in different time zones and things are not moving forward at all)

two (or three) things I've done to help this, when I'm working remotely

add panic=30 to your kernel options in grub (or echo 30 >
/proc/sys/kernel/panic) to reboot the system automatically on panic.

set grub to automatically boot the safe kernel by default, and when
making a new kernel, set grub to boot it only once with (say your
default is 0 and the new kernel is 1 in grub)

echo 'savedefault --default=1 --once' | grub --batch

set up netconsole so that you can see the kernel messages (optional) on oops.

finding out about all these was incredibly hard and obtuse :-)  So
hope this helps.

Re: RFC : remote driver debugging efforts

[Jesper Juhl]

On 12/08/06, Jesse Brandeburg <jesse.brandeburg@gmail.com> wrote:
> On 8/10/06, Om N. <xhandle@gmail.com> wrote:
> > (I do not have a remote power on/off switch. The driver panics so
> > often that somebody has to babysit the machine to switch it off and
> > on. We are in different time zones and things are not moving forward at all)
>
> two (or three) things I've done to help this, when I'm working remotely
>
> add panic=30 to your kernel options in grub (or echo 30 >
> /proc/sys/kernel/panic) to reboot the system automatically on panic.
>
> set grub to automatically boot the safe kernel by default, and when
> making a new kernel, set grub to boot it only once with (say your
> default is 0 and the new kernel is 1 in grub)
>
> echo 'savedefault --default=1 --once' | grub --batch
>
> set up netconsole so that you can see the kernel messages (optional) on oops.
>
> finding out about all these was incredibly hard and obtuse :-)  So
> hope this helps.

If you have remote access to the keyboard/display etc, for example via
a network enabled KVM switch or similar, then magic sysrq can also be
very helpful to remotely sync & unmount filesystems, and do emergency
reboots... It can also be used to get stack dumps etc...  Read
Documentation/sysrq.txt for all the details.

I'm not sure, but I think there was a patch floating around at one
point that would let you trigger sysrq remotely over the network
without even logging into the box or having remote keyboard access -
could be very useful in your case although it is ofcourse extremely
insecure.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: RFC : remote driver debugging efforts

[Jesper Juhl]

On 12/08/06, Jesper Juhl <jesper.juhl@gmail.com> wrote:
> On 12/08/06, Jesse Brandeburg <jesse.brandeburg@gmail.com> wrote:
> > On 8/10/06, Om N. <xhandle@gmail.com> wrote:
> > > (I do not have a remote power on/off switch. The driver panics so
> > > often that somebody has to babysit the machine to switch it off and
> > > on. We are in different time zones and things are not moving forward at all)
> >
> > two (or three) things I've done to help this, when I'm working remotely
> >
> > add panic=30 to your kernel options in grub (or echo 30 >
> > /proc/sys/kernel/panic) to reboot the system automatically on panic.
> >
> > set grub to automatically boot the safe kernel by default, and when
> > making a new kernel, set grub to boot it only once with (say your
> > default is 0 and the new kernel is 1 in grub)
> >
> > echo 'savedefault --default=1 --once' | grub --batch
> >
> > set up netconsole so that you can see the kernel messages (optional) on oops.
> >
> > finding out about all these was incredibly hard and obtuse :-)  So
> > hope this helps.
>
> If you have remote access to the keyboard/display etc, for example via
> a network enabled KVM switch or similar, then magic sysrq can also be
> very helpful to remotely sync & unmount filesystems, and do emergency
> reboots... It can also be used to get stack dumps etc...  Read
> Documentation/sysrq.txt for all the details.
>
> I'm not sure, but I think there was a patch floating around at one
> point that would let you trigger sysrq remotely over the network
> without even logging into the box or having remote keyboard access -
> could be very useful in your case although it is ofcourse extremely
> insecure.
>
One other thing, putting a watchdog card in the box to automatically
reboot it if the machine hangs would also be a good thing. If you
don't have (or don't want to buy) such a card, then building the
kernel with the software watchdog is better than nothing - there are
situations where it'll fail to reboot the machine that a real watchdog
card would be able to handle, but it's a lot better than nothing.
See;  Device Drivers --> Character devices --> Watchdog Cards in menuconfig.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: RFC : remote driver debugging efforts

[Frederik Deweerdt]

On Sat, Aug 12, 2006 at 11:43:13AM -0700, Jesse Brandeburg wrote:
> On 8/10/06, Om N. <xhandle@gmail.com> wrote:
> >(I do not have a remote power on/off switch. The driver panics so
> >often that somebody has to babysit the machine to switch it off and
> >on. We are in different time zones and things are not moving forward at all)
> 
> add panic=30 to your kernel options in grub (or echo 30 >
> /proc/sys/kernel/panic) to reboot the system automatically on panic.
> 
On a similar note, kexec -p might help too.

Regards,
Frederik