console problem?, close(2) hangs /sbin/init

console problem?, close(2) hangs /sbin/init

[John Linn]

I have a kernel that runs fine with a busybox root file system using
inittab.  When I move to a non-busybox init (sysvinit), I see init
hanging when doing a close of standard error very early when it starts.
I have debugged it with GDB but not found the source of the hang as it's
in the kernel it appears.

To debug this, I use init=/bin/sh on the kernel command line, then exec
init to start it. The same root file system works on another board with
a different kernel.  I have also noticed that building init with
INITDEBUG on so that it forks for debugging makes it work fine. This
seems to indicate the device nodes are all fine to me.

It seems related to PID 1 and the console device.

Adding debug to the kernel in sys_close() shows it hanging in
filp_close() and I'm still digging more (maybe locking issue). The
console is a serial device with a newer driver which I suspected to be
the issue, but can't see anything wrong.

Any hints or similar problems seen would be much appreciated.
Thanks,
John


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

Re: console problem?, close(2) hangs /sbin/init

[Jesper Juhl]

On Sat, 22 Jan 2011, John Linn wrote:

> I have a kernel that runs fine 

What kernel exactely?

>with a busybox root file system using
> inittab.  When I move to a non-busybox init (sysvinit), I see init
> hanging when doing a close of standard error very early when it starts.
> I have debugged it with GDB but not found the source of the hang as it's
> in the kernel it appears.
> 
> To debug this, I use init=/bin/sh on the kernel command line, then exec
> init to start it. The same root file system works on another board with
> a different kernel. 

What other kernel version exactely?

> I have also noticed that building init with
> INITDEBUG on so that it forks for debugging makes it work fine. This
> seems to indicate the device nodes are all fine to me.
> 
> It seems related to PID 1 and the console device.
> 
> Adding debug to the kernel in sys_close() shows it hanging in
> filp_close() and I'm still digging more (maybe locking issue). The
> console is a serial device with a newer driver which I suspected to be
> the issue, but can't see anything wrong.
> 
> Any hints or similar problems seen would be much appreciated.
> Thanks,
> John
> 

-- 
Jesper Juhl <jj@chaosbits.net>            http://www.chaosbits.net/
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please.

RE: console problem?, close(2) hangs /sbin/init

[John Linn]

> -----Original Message-----
> From: Jesper Juhl [mailto:jj@chaosbits.net]
> Sent: Saturday, January 22, 2011 3:14 PM
> To: John Linn
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: console problem?, close(2) hangs /sbin/init
> 
> On Sat, 22 Jan 2011, John Linn wrote:
> 
> > I have a kernel that runs fine
> 
> What kernel exactely?

2.6.37, but I also tried a 2.6.35 build.

> 
> >with a busybox root file system using
> > inittab.  When I move to a non-busybox init (sysvinit), I see init
> > hanging when doing a close of standard error very early when it
starts.
> > I have debugged it with GDB but not found the source of the hang as
it's
> > in the kernel it appears.
> >
> > To debug this, I use init=/bin/sh on the kernel command line, then
exec
> > init to start it. The same root file system works on another board
with
> > a different kernel.
> 
> What other kernel version exactely?

2.6.28, but others have told them tested the same root file system on
2.6.37 kernels on other boards fine.

It's a poky based root file system for ARM.

I've tried comparing kernel configs to try look for differences without
much luck yet.

Thanks,
John

> 
> > I have also noticed that building init with
> > INITDEBUG on so that it forks for debugging makes it work fine. This
> > seems to indicate the device nodes are all fine to me.
> >
> > It seems related to PID 1 and the console device.
> >
> > Adding debug to the kernel in sys_close() shows it hanging in
> > filp_close() and I'm still digging more (maybe locking issue). The
> > console is a serial device with a newer driver which I suspected to
be
> > the issue, but can't see anything wrong.
> >
> > Any hints or similar problems seen would be much appreciated.
> > Thanks,
> > John
> >
> 
> --
> Jesper Juhl <jj@chaosbits.net>            http://www.chaosbits.net/
> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
> Plain text mails only, please.
> 


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

RE: console problem?, close(2) hangs /sbin/init

[John Linn]

> -----Original Message-----
> From: Jesper Juhl [mailto:jj@chaosbits.net]
> Sent: Saturday, January 22, 2011 3:14 PM
> To: John Linn
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: console problem?, close(2) hangs /sbin/init
> 
> On Sat, 22 Jan 2011, John Linn wrote:
> 
> > I have a kernel that runs fine
> 
> What kernel exactely?
> 
> >with a busybox root file system using
> > inittab.  When I move to a non-busybox init (sysvinit), I see init
> > hanging when doing a close of standard error very early when it
starts.
> > I have debugged it with GDB but not found the source of the hang as
it's
> > in the kernel it appears.

I seems like it's any of the descriptors for the console, 0, 1, or 2 as
I try to isolate it.
Init closes all 3 when it begins and this is where the problem seems to
start.

At one point during some debug, I saw the kernel spit out a message
about a hung task which
showed tty_lock/mutex_lock_slowpath from a sys_open origin.

I had written it off in the beginning because I don't always see it, but
maybe this is also
a symptom I should be debugging.

> >
> > To debug this, I use init=/bin/sh on the kernel command line, then
exec
> > init to start it. The same root file system works on another board
with
> > a different kernel.
> 
> What other kernel version exactely?
> 
> > I have also noticed that building init with
> > INITDEBUG on so that it forks for debugging makes it work fine. This
> > seems to indicate the device nodes are all fine to me.
> >
> > It seems related to PID 1 and the console device.
> >
> > Adding debug to the kernel in sys_close() shows it hanging in
> > filp_close() and I'm still digging more (maybe locking issue). The
> > console is a serial device with a newer driver which I suspected to
be
> > the issue, but can't see anything wrong.
> >
> > Any hints or similar problems seen would be much appreciated.
> > Thanks,
> > John
> >
> 
> --
> Jesper Juhl <jj@chaosbits.net>            http://www.chaosbits.net/
> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
> Plain text mails only, please.
> 


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.