Oops with 2.4.3-XFS

Oops with 2.4.3-XFS

[Matt Bernstein]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

[ I'm not subscribed to linux-xfs, please cc me ]

We have managed to get a Debian potato system (with the 2.4 updates from
http://people.debian.org/~bunk/debian plus xfs-tools which we imported
from woody) to run 2.4.3-XFS.

However, in testing a directory with lots (~177000) of files, we get the
following oops (copied by hand, and run through ksymoops on a Red Hat box
since the Debian one segfaulted :( )

HTH,

Matt

r2-pc:~/oopsie$ ksymoops -m System.map -v vmlinux -K -L -O -S <oops
ksymoops 2.4.0 on i686 2.4.2-2.  Options used
     -v vmlinux (specified)
     -K (specified)
     -L (specified)
     -O (specified)
     -m System.map (specified)
     -S

Unable to handle kernel NULL pointer dereference at virtual address 00000008 printing eip:
c01950e9
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c01950e9>]
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010256
eax: 00000000   ebx: cff25c00   ecx: 00000008   edx: 00000008
esi: 00000000   edi: 00000005   ebp: 00000000   esp: cee59b94
ds: 0018   es: 0018   ss: 0018
Process xntpd (pid: 364, stackpage=cee59000)
Stack: c0411e90 00000fe6 00000005 c0411e90 c0198682 cd2b06c8 c01986e6 00000000
       02683f18 00000000 00000010 c0411e08 0000e016 c01980d7 cff25c00 00000000
       cbf894b0 cee59c04 cee59c08 00000000 00000000 cbf894b0 00000fe6 00000bb0
Call Trace: [<c0198682>] [<c01986e6>] [<c01980d7>] [<c0199074>] [<c01992e0>] [<c01a95b2>] [<c019c33e>]
       [<c019b7a1>] [<c01a85d4>] [<c0196c24>] [<c01aed54>] [<c01af5af>] [<c0106fe0>] [<c01bf124>] [<c01be417>]
       [<c014182b>] [<c013fb4e>] [<c013fe31>] [<c0127fef>] [<c0128162>] [<c0128e3e>] [<c0128ef4>] [<c013c183>]
       [<c01f3c67>] [<c01f0773>] [<c013c3cf>] [<c013c91d>] [<c0106f37>]
Code: f6 45 08 04 74 09 8b 55 30 01 c2 eb 12 89 f6 50 55 e8 51 9f

>>EIP; c01950e9 <xfs_itobp+139/1e0>   <=====
Trace; c0198682 <xfs_iflush_int+272/300>
Trace; c01986e6 <xfs_iflush_int+2d6/300>
Trace; c01980d7 <xfs_iflush+b7/3f0>
Trace; c0199074 <xfs_inode_item_trylock+24/80>
Trace; c01992e0 <xfs_inode_item_push+10/20>
Trace; c01a95b2 <xfs_trans_push_ail+122/200>
Trace; c019c33e <xlog_grant_push_ail+15e/170>
Trace; c019b7a1 <xfs_log_reserve+41/90>
Trace; c01a85d4 <xfs_trans_reserve+84/140>
Trace; c0196c24 <xfs_itruncate_finish+354/3f0>
Trace; c01aed54 <xfs_inactive_free_eofblocks+284/2d0>
Trace; c01af5af <xfs_inactive+10f/480>
Trace; c0106fe0 <ret_from_intr+0/20>
Trace; c01bf124 <vn_put+34/50>
Trace; c01be417 <linvfs_put_inode+17/20>
Trace; c014182b <iput+2b/150>
Trace; c013fb4e <prune_dcache+de/150>
Trace; c013fe31 <shrink_dcache_memory+21/30>
Trace; c0127fef <do_try_to_free_pages+5f/80>
Trace; c0128162 <try_to_free_pages+22/30>
Trace; c0128e3e <__alloc_pages+24e/2f0>
Trace; c0128ef4 <__get_free_pages+14/20>
Trace; c013c183 <__pollwait+33/a0>
Trace; c01f3c67 <datagram_poll+27/f0>
Trace; c01f0773 <sock_poll+23/30>
Trace; c013c3cf <do_select+11f/210>
Trace; c013c91d <sys_select+42d/5a0>
Trace; c0106f37 <system_call+33/38>
Code;  c01950e9 <xfs_itobp+139/1e0>            00000000 <_EIP>:
Code;  c01950e9 <xfs_itobp+139/1e0>               0:   f6 45 08 04               testb  $0x4,0x8(%ebp)   <=====
Code;  c01950ed <xfs_itobp+13d/1e0>               4:   74 09                     je     f <_EIP+0xf> c01950f8 <xfs_itobp+148/1e0>
Code;  c01950ef <xfs_itobp+13f/1e0>               6:   8b 55 30                  mov    0x30(%ebp),%edx
Code;  c01950f2 <xfs_itobp+142/1e0>               9:   01 c2                     add    %eax,%edx
Code;  c01950f4 <xfs_itobp+144/1e0>               b:   eb 12                     jmp    1f <_EIP+0x1f> c0195108 <xfs_itobp+158/1e0>
Code;  c01950f6 <xfs_itobp+146/1e0>               d:   89 f6                     mov    %esi,%esi
Code;  c01950f8 <xfs_itobp+148/1e0>               f:   50                        push   %eax
Code;  c01950f9 <xfs_itobp+149/1e0>              10:   55                        push   %ebp
Code;  c01950fa <xfs_itobp+14a/1e0>              11:   e8 51 9f 00 00            call   9f67 <_EIP+0x9f67> c019f050 <xlog_find_zeroed+150/200>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE7Al/D1vU/2MhEp5cRApvnAKCWumj3KYcd1ucNLVrPwZOWzVxwWQCfaqVg
6L3NvlsLoQgvkPbru2dPYJs=
=z9ET
-----END PGP SIGNATURE-----

Re: Oops with 2.4.3-XFS

[Steve Lord]

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi,
> 
> [ I'm not subscribed to linux-xfs, please cc me ]
> 
> We have managed to get a Debian potato system (with the 2.4 updates from
> http://people.debian.org/~bunk/debian plus xfs-tools which we imported
> from woody) to run 2.4.3-XFS.
> 
> However, in testing a directory with lots (~177000) of files, we get the
> following oops (copied by hand, and run through ksymoops on a Red Hat box
> since the Debian one segfaulted :( )
> 
> HTH,
> 
> Matt
> 

Can you describe your testing beyond using a directory with 177000 files
in it?

Also, can you explain how you obtained the xfs code, from a patch, from
the cvs development tree, or from somewhere else?

Thanks

   Steve

Re: Oops with 2.4.3-XFS

[Matt Bernstein]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> However, in testing a directory with lots (~177000) of files, we get the
>> following oops (copied by hand, and run through ksymoops on a Red Hat box
>> since the Debian one segfaulted :( )

>Can you describe your testing beyond using a directory with 177000 files
>in it?
>Also, can you explain how you obtained the xfs code, from a patch, from
>the cvs development tree, or from somewhere else?

Certainly :) (have been investigating further through the day)

The code was obtained as the patches for 2.4.3, (it was also patched for a
scsi changer, but all the scsi stuff is modules that weren't loaded), and
we used your Red Hat ISO in rescue mode to convert our root partition to
XFS.

I'd been trying to hammer a partition remotely. I'd exported it with
knfsd, but the oops I posted was caused by trying to tar said directory
(locally--no NFS involvement) to a file in its parent (which does contain
some large (~4GB) files.

kernel built with egcs-1.1.2, not very much in it, modules loaded at the
time were nfs, sunrpc, lockd, autofs (but I will test again without any of
those loaded when the box is less busy)--I had removed nfsd and friends.

We tried to recreate this on a second box a couple of hours ago and
failed, so we can't rule out a hardware/other non-XFS problem. Two
immediate potential culprits are the VIA KT133 UDMA and the drive itself
(NB the second box has the same motherboard, but not a WDC drive):

# hdparm -i /dev/hda

/dev/hda:

 Model=WDC WD400BB-00AUA1, FwRev=18.20D18, SerialNo=WD-WMA6R1239029
 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
 RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=40
 BuffType=3(DualPortCache), BuffSize=2048kB, MaxMultSect=16, MultSect=off
 DblWordIO=no, OldPIO=2, DMA=yes, OldDMA=0
 CurCHS=4047/16/255, CurSects=16511760, LBA=yes, LBAsects=78165360
 tDMA={min:120,rec:120}, DMA modes: mword0 mword1 mword2
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4
 UDMA modes: mode0 mode1 mode2 mode3 *mode4 mode5

Thanks for the reply. Any more information on request :)

Matt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE7Apqx1vU/2MhEp5cRAgecAJ0bAm1Jlay2AjHjGaQ1Zck7/1vOewCgujgD
mAnEXzuyabuUwcPy22e1avM=
=41+h
-----END PGP SIGNATURE-----

Re: Oops with 2.4.3-XFS

[Matt Bernstein]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

After some more testing it's entirely plausible (though unfortunately we
don't have anything conclusive) that we have a hardware problem.

Having added a new drive (hdc as reported below) and booted root=/dev/hdc2
with nothing mounted on hda, we have thus far failed to recreate the crash.

VP_IDE: IDE controller on PCI bus 00 dev 39
VP_IDE: chipset revision 16
VP_IDE: not 100% native mode: will probe irqs later
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
VP_IDE: VIA vt82c686a (rev 22) IDE UDMA66 controller on pci00:07.1
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
hda: WDC WD400BB-00AUA1, ATA DISK drive
hdc: FUJITSU MPG3102AT E, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: 78165360 sectors (40021 MB) w/2048KiB Cache, CHS=4865/255/63, UDMA(66)
hdc: 20015856 sectors (10248 MB) w/512KiB Cache, CHS=19857/16/63, UDMA(33)

On May 16 Matt Bernstein wrote:

>We have managed to get a Debian potato system (with the 2.4 updates from
>http://people.debian.org/~bunk/debian plus xfs-tools which we imported
>from woody) to run 2.4.3-XFS.
>
>However, in testing a directory with lots (~177000) of files, we get the
>following oops (copied by hand, and run through ksymoops on a Red Hat box
>since the Debian one segfaulted :( )
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE7FRSV1vU/2MhEp5cRApffAKDJcNLh6pp/iXyHjdWRC/x/zpabfwCgszG0
6mClfT2pCZwNjo2etYuI0mI=
=hE/Z
-----END PGP SIGNATURE-----