BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

[Zwane Mwaikambo]

Hi,
	Hope you all enjoyed your Christmas break, if anyone's still around heres one to
look at :) (oops message and decode at the end).

I got the oops+panic by doing "mount /dev/hdd /cdrom -o loop"
Box is highmem 64G with 192M ram and SMP kernel on UP. The oops is
reproducible and produces a similar trace each time (a bit of noise from
other subsystems). And yes the symbol resolving is all correct.

ll_rw_blk.c: end_that_request_first()
<--snip-->
	if (!bio->bi_size) {
			req->bio = bio->bi_next;
			if (unlikely(bio_endio(bio, uptodate, total_nsect)))
				BUG(); <==== [1]
			total_nsect = 0;
		}
<--snip-->

[1] looks like we're hitting it there but bio_endio _never_ returns non zero, did we just
drive off the edge of the world!? =)

eip: cc916780 <== is this because we're in an interrupt handler?
kernel BUG at /usr/src/linux-2.5.2-pre1.orig/include/asm/spinlock.h:133! <= ??
invalid operand: 0000
CPU:    0
EIP:    0010:[<cc9167ba>]    Not tainted
EFLAGS: 00010082
eax: 00000049   ebx: cb889b00   ecx: c0303c84   edx: 00058231
esi: cbf2c924   edi: 00000096   ebp: c156e5d4   esp: c0319c28
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0, stackpage=c0319000)
Stack: cc918140 00000085 cb889b00 cbf2c924 00000004 cc916997 cb889b00 cbf2c924
       cbf2c924 00000004 c014d9ee cbf2c924 00000004 cbf2c924 00000004 c020b7de
       cbf2c924 00000001 00000004 00000004 c0318000 c0318000 c1519cb0 c90f7f2c
Call Trace: [<cc918140>] [<cc916997>] [<c014d9ee>] [<c020b7de>] [<c021b7db>]
   [<c0222dde>] [<c02a4400>] [<c01453d4>] [<c014cfdd>] [<c02a4422>] [<c01453d4>]
   [<c014cfdd>] [<c0126a30>] [<c010e707>] [<c0126540>] [<c010a99e>] [<cc835198>]
   [<cc83ad45>] [<cc83889a>] [<c0222d40>] [<c0218a62>] [<c010a99e>] [<c010acf9>]
   [<c0106dc0>] [<c0106dc0>] [<c01e0018>] [<c01ec713>] [<c0106dc0>] [<c01ec600>]
   [<c0106e72>] [<c0105000>]

Code: 0f 0b 5a 59 89 f6 f0 fe 8b cc 00 00 00 0f 88 31 18 00 00 8b
 <0>Kernel panic: Aiee, killing interrupt handler!
In interrupt handler - not syncing

>>EIP; cc9187ba <END_OF_CODE+9215a/????>   <=====
Trace; cc91a140 <END_OF_CODE+93ae0/????>
Trace; cc918997 <END_OF_CODE+92337/????>
Trace; c014db03 <bio_endio+23/30>
Trace; c020b8ee <end_that_request_first+19e/2a0>
Trace; c024b125 <qdisc_restart+15/290>
Trace; c021b8eb <__ide_end_request+bb/150>
Trace; c0222eee <cdrom_read_intr+9e/330>
Trace; c010aeda <do_IRQ+1aa/1c0>
Trace; c013b9c6 <kfree+196/1a0>
Trace; c023ed8c <kfree_skbmem+c/70>
Trace; cc87aa2d <[8139too]rtl8139_start_xmit+fd/170>
Trace; cc87d229 <[8139too].text.end+c8a/de1>
Trace; c024b125 <qdisc_restart+15/290>
Trace; c02430de <dev_queue_xmit+19e/4e0>
Trace; c023e7b7 <sock_def_readable+57/90>
Trace; c0253ff3 <ip_output+113/180>
Trace; c023ff94 <skb_checksum+54/3d0>
Trace; c02543a4 <ip_queue_xmit+344/4a0>
Trace; c026b12f <tcp_v4_checksum_init+7f/110>
Trace; c026b7ad <tcp_v4_rcv+46d/740>
Trace; c026a19e <tcp_v4_send_check+6e/b0>
Trace; c0264b35 <tcp_transmit_skb+565/620>
Trace; cc835198 <[usb-ohci]roothub_portstatus+28/70>
Trace; cc83ad45 <[usb-ohci].text.end+876/8e1>
Trace; cc83889a <[usb-ohci]rh_send_irq+fa/1a0>
Trace; cc838940 <[usb-ohci]rh_int_timer_do+0/70>
Trace; c0222e50 <cdrom_read_intr+0/330>
Trace; c0218b72 <ide_intr+162/240>
Trace; c010a99e <handle_IRQ_event+5e/90>
Trace; c010ae39 <do_IRQ+109/1c0>
Trace; c0106dc0 <default_idle+0/40>
Trace; c0106dc0 <default_idle+0/40>
Trace; c01e0018 <acpi_rs_address64_stream+148/160>
Trace; c01ec823 <pr_power_idle+113/270>
Trace; c0106dc0 <default_idle+0/40>
Trace; c01ec710 <pr_power_idle+0/270>
Trace; c0106e72 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Code;  cc9187ba <END_OF_CODE+9215a/????>
00000000 <_EIP>:
Code;  cc9187ba <END_OF_CODE+9215a/????>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  cc9187bc <END_OF_CODE+9215c/????>
   2:   5a                        pop    %edx
Code;  cc9187bd <END_OF_CODE+9215d/????>
   3:   59                        pop    %ecx
Code;  cc9187be <END_OF_CODE+9215e/????>
   4:   89 f6                     mov    %esi,%esi
Code;  cc9187c0 <END_OF_CODE+92160/????>
   6:   f0 fe 8b cc 00 00 00      lock decb 0xcc(%ebx)
Code;  cc9187c7 <END_OF_CODE+92167/????>
   d:   0f 88 31 18 00 00         js     1844 <_EIP+0x1844> cc919ffe <END_OF_CODE+9399e/????>
Code;  cc9187cd <END_OF_CODE+9216d/????>
  13:   8b 00                     mov    (%eax),%eax

 <0>Kernel panic: Aiee, killing interrupt handler

Re: BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

[Keith Owens]

On Wed, 26 Dec 2001 20:33:07 +0200 (SAST), 
Zwane Mwaikambo <zwane@linux.realnet.co.sz> wrote:
>eip: cc916780 <== is this because we're in an interrupt handler?

Probably because your module structure after reboot is not the same as
the panic.  Try using the module data saved in /var/log/ksymoops, man
insmod and look for ksymoops assistance.

Re: BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

[Zwane Mwaikambo]

On Thu, 27 Dec 2001, Keith Owens wrote:

> On Wed, 26 Dec 2001 20:33:07 +0200 (SAST),
> Zwane Mwaikambo <zwane@linux.realnet.co.sz> wrote:
> >eip: cc916780 <== is this because we're in an interrupt handler?
>
> Probably because your module structure after reboot is not the same as
> the panic.  Try using the module data saved in /var/log/ksymoops, man
> insmod and look for ksymoops assistance.

I AM AN IDIOT
come to think of it, loop was built as a module!! argh!

Thanks,
	Zwane Mwaikambo

Re: BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

[Zwane Mwaikambo]

Hi Jens,
mount /dev/fd0 /floppy -o loop gives a "better" dump :) This is on
2.5.2-pre2 (The END_OF_CODE is because i didn't do modprobe loop before
running ksymoops. Thanks to Keith Owens for that) but we know where the
bug really is now.

invalid operand: 0000
CPU:    0
EIP:    0010:[<c020c065>]    Not tainted
EFLAGS: 00010286
eax: 00000020   ebx: c88db1a4   ecx: c0304cc4   edx: 000025e1
esi: cae38c64   edi: cae95000   ebp: 00000000   esp: c8adff9c
ds: 0018   es: 0018   ss: 0018
Process loop0 (pid: 697, stackpage=c8adf000)
Stack: c02e1c03 00000554 cae38c64 cae38f64 c014dd33 cae38c64 00000000 cc9170a2
       cae38c64 00000001 00000000 cae95000 cae38f64 cae38c64 00000000 00000000
       00000000 c1560018 00000f00 ca13bf30 00000000 c0107286 cae95000 cc916e80
Call Trace: [<c014dd33>] [<cc9170a2>] [<c0107286>] [<cc916e80>]

Code: 0f 0b 58 5a 8b 46 0c 83 e0 01 50 53 ff 53 34 56 e8 b6 14 f4

 Jan  1 02:12:34 mondecino kernel: kernel BUG at ll_rw_blk.c:1364!
Jan  1 02:12:34 mondecino kernel: invalid operand: 0000
Jan  1 02:12:34 mondecino kernel: CPU:    0
Jan  1 02:12:34 mondecino kernel: EIP:    0010:[end_bio_bh_io_sync+37/80]    Not tainted
Jan  1 02:12:34 mondecino kernel: EIP:    0010:[<c020c065>]    Not tainted
Jan  1 02:12:34 mondecino kernel: EFLAGS: 00010286
Jan  1 02:12:34 mondecino kernel: eax: 00000020   ebx: c88db1a4   ecx: c0304cc4   edx: 000025e1
Jan  1 02:12:34 mondecino kernel: esi: cae38c64   edi: cae95000   ebp: 00000000   esp: c8adff9c
Jan  1 02:12:34 mondecino kernel: ds: 0018   es: 0018   ss: 0018
Jan  1 02:12:34 mondecino kernel: Process loop0 (pid: 697, stackpage=c8adf000)
Jan  1 02:12:34 mondecino kernel: Stack: c02e1c03 00000554 cae38c64 cae38f64 c014dd33 cae38c64 000
Jan  1 02:12:34 mondecino kernel:        cae38c64 00000001 00000000 cae95000 cae38f64 cae38c64 000
Jan  1 02:12:34 mondecino kernel:        00000000 c1560018 00000f00 ca13bf30 00000000 c0107286 cae
Jan  1 02:12:34 mondecino kernel: Call Trace: [bio_endio+35/48] [sound:num_midis_Rsmp_a1eae7cf+814
Jan  1 02:12:34 mondecino kernel: Call Trace: [<c014dd33>] [<cc9170a2>] [<c0107286>] [<cc916e80>]
Jan  1 02:12:34 mondecino kernel:
Jan  1 02:12:34 mondecino kernel: Code: 0f 0b 58 5a 8b 46 0c 83 e0 01 50 53 ff 53 34 56 e8 b6 14 f

>>EIP; c020c065 <end_bio_bh_io_sync+25/50>   <=====
Trace; c014dd33 <bio_endio+23/30>
Trace; cc9170a2 <END_OF_CODE+90a42/????>
Trace; c0107286 <kernel_thread+26/30>
Trace; cc916e80 <END_OF_CODE+90820/????>
Code;  c020c065 <end_bio_bh_io_sync+25/50>
00000000 <_EIP>:
Code;  c020c065 <end_bio_bh_io_sync+25/50>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c020c067 <end_bio_bh_io_sync+27/50>
   2:   58                        pop    %eax
Code;  c020c068 <end_bio_bh_io_sync+28/50>
   3:   5a                        pop    %edx
Code;  c020c069 <end_bio_bh_io_sync+29/50>
   4:   8b 46 0c                  mov    0xc(%esi),%eax
Code;  c020c06c <end_bio_bh_io_sync+2c/50>
   7:   83 e0 01                  and    $0x1,%eax
Code;  c020c06f <end_bio_bh_io_sync+2f/50>
   a:   50                        push   %eax
Code;  c020c070 <end_bio_bh_io_sync+30/50>
   b:   53                        push   %ebx
Code;  c020c071 <end_bio_bh_io_sync+31/50>
   c:   ff 53 34                  call   *0x34(%ebx)
Code;  c020c074 <end_bio_bh_io_sync+34/50>
   f:   56                        push   %esi
Code;  c020c075 <end_bio_bh_io_sync+35/50>
  10:   e8 b6 14 f4 00            call   f414cb <_EIP+0xf414cb> c114d530 <_end+d6c100/c444c30>

Re: BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

[Zwane Mwaikambo]

Sorry the previous emails didn't help much, here is a properly decoded oops as well as a
printout of the values before the BIO_BUG_ON.

ll_rw_blk.c
static int end_bio_bh_io_sync(struct bio *bio, int nr_sectors)
{
	struct buffer_head *bh = bio->bi_private;
	printk(KERN_DEBUG "%s: nr_sectors=%d bh->b_size=%d\n", __FUNCTION__, nr_sectors, (bh->b_size >> 9));
	BIO_BUG_ON(nr_sectors != (bh->b_size >> 9)); <== BUG tripped here
	bh->b_end_io(bh, test_bit(BIO_UPTODATE, &bio->bi_flags));
	bio_put(bio);
	return 0;
}

Jan  1 02:01:02 mondecino kernel: loop: loaded (max 8 devices)
Jan  1 02:01:02 mondecino kernel: VFS: Disk change detected on device fd(2,0)
Jan  1 02:01:02 mondecino kernel: end_bio_bh_io_sync: nr_sectors=8 bh->b_size=8
Jan  1 02:01:02 mondecino last message repeated 3 times
Jan  1 02:01:03 mondecino kernel: end_bio_bh_io_sync: nr_sectors=2 bh->b_size=2
Jan  1 02:01:03 mondecino kernel: end_bio_bh_io_sync: nr_sectors=8 bh->b_size=8

invalid operand: 0000
CPU:    0
EIP:    0010:[<c020c065>]    Not tainted
EFLAGS: 00010286
eax: 00000020   ebx: c88db1a4   ecx: c0304cc4   edx: 000025e1
esi: cae38c64   edi: cae95000   ebp: 00000000   esp: c8adff9c
ds: 0018   es: 0018   ss: 0018
Process loop0 (pid: 697, stackpage=c8adf000)
Stack: c02e1c03 00000554 cae38c64 cae38f64 c014dd33 cae38c64 00000000 cc9170a2
       cae38c64 00000001 00000000 cae95000 cae38f64 cae38c64 00000000 00000000
       00000000 c1560018 00000f00 ca13bf30 00000000 c0107286 cae95000 cc916e80
Call Trace: [<c014dd33>] [<cc9170a2>] [<c0107286>] [<cc916e80>]

Code: 0f 0b 58 5a 8b 46 0c 83 e0 01 50 53 ff 53 34 56 e8 b6 14 f4

>>EIP; c020c065 <end_bio_bh_io_sync+25/50>   <=====
Trace; c014dd33 <bio_endio+23/30>
Trace; cc9170a2 <[loop]loop_thread+222/280>
Trace; c0107286 <kernel_thread+26/30>
Trace; cc916e80 <[loop]loop_thread+0/280>
Code;  c020c065 <end_bio_bh_io_sync+25/50>
00000000 <_EIP>:
Code;  c020c065 <end_bio_bh_io_sync+25/50>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c020c067 <end_bio_bh_io_sync+27/50>
   2:   58                        pop    %eax
Code;  c020c068 <end_bio_bh_io_sync+28/50>
   3:   5a                        pop    %edx
Code;  c020c069 <end_bio_bh_io_sync+29/50>
   4:   8b 46 0c                  mov    0xc(%esi),%eax
Code;  c020c06c <end_bio_bh_io_sync+2c/50>
   7:   83 e0 01                  and    $0x1,%eax
Code;  c020c06f <end_bio_bh_io_sync+2f/50>
   a:   50                        push   %eax
Code;  c020c070 <end_bio_bh_io_sync+30/50>
   b:   53                        push   %ebx
Code;  c020c071 <end_bio_bh_io_sync+31/50>
   c:   ff 53 34                  call   *0x34(%ebx)
Code;  c020c074 <end_bio_bh_io_sync+34/50>
   f:   56                        push   %esi
Code;  c020c075 <end_bio_bh_io_sync+35/50>
  10:   e8 b6 14 f4 00            call   f414cb <_EIP+0xf414cb> c114d530 <_end+d6c100/c444c30>

Re: BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

[Zwane Mwaikambo]

Ok looks like the floppy and the cdrom oops are two different thing entirely.
A newly decoded cdrom oops is at the end. I'm a bit confused here because
it looks like a spinlock magic thing, but then again that spinlock is
_definately_ initialised properly (spin_lock_init)

loop.c
static int loop_end_io_transfer(struct bio *bio, int nr_sectors)
{
	struct loop_device *lo = &loop_dev[MINOR(bio->bi_dev)]; <- evals to loop_dev[0]
	int uptodate = test_bit(BIO_UPTODATE, &bio->bi_flags);
	if (!uptodate || bio_rw(bio) == WRITE) {
		struct bio *rbh = bio->bi_private;
		bio_endio(rbh, uptodate, nr_sectors);
		if (atomic_dec_and_test(&lo->lo_pending))
			up(&lo->lo_bh_mutex);
		loop_put_buffer(bio);
	} else
		loop_add_bio(lo, bio); <==
	return 0;
}

static void loop_add_bio(struct loop_device *lo, struct bio *bio)
{
	unsigned long flags;
	spin_lock_irqsave(&lo->lo_lock, flags);	<== we puke here i think
	if (lo->lo_biotail) {
		lo->lo_biotail->bi_next = bio;
		lo->lo_biotail = bio;
	} else
		lo->lo_bio = lo->lo_biotail = bio;
	spin_unlock_irqrestore(&lo->lo_lock, flags);
	up(&lo->lo_bh_mutex);
}

(gdb) disassemble loop_add_bio
Dump of assembler code for function loop_add_bio:
0x740 <loop_add_bio>:   push   %edi
0x741 <loop_add_bio+1>: push   %esi
0x742 <loop_add_bio+2>: push   %ebx
0x743 <loop_add_bio+3>: mov    0x10(%esp,1),%ebx
0x747 <loop_add_bio+7>: mov    0x14(%esp,1),%esi
0x74b <loop_add_bio+11>:        pushf
0x74c <loop_add_bio+12>:        pop    %edi
0x74d <loop_add_bio+13>:        cli
0x74e <loop_add_bio+14>:        cmpl   $0xdead4ead,0xd0(%ebx)
0x758 <loop_add_bio+24>:        je     0x780 <loop_add_bio+64>
0x75a <loop_add_bio+26>:        push   $0x740
0x75f <loop_add_bio+31>:        push   $0x0
0x764 <loop_add_bio+36>:        call   0x765 <loop_add_bio+37>
0x769 <loop_add_bio+41>:        pop    %edx
0x76a <loop_add_bio+42>:        pop    %ecx
0x76b <loop_add_bio+43>:        push   $0x85
0x770 <loop_add_bio+48>:        push   $0x0
0x775 <loop_add_bio+53>:        call   0x776 <loop_add_bio+54>
0x77a <loop_add_bio+58>:        ud2a   <== we fail at spin_lock_irqsave magic
0x77c <loop_add_bio+60>:        pop    %ecx
0x77d <loop_add_bio+61>:        pop    %eax
0x77e <loop_add_bio+62>:        mov    %esi,%esi
<--snip-->

invalid operand: 0000
CPU:    0
EIP:    0010:[<cc9167da>]    Not tainted
EFLAGS: 00010086
eax: 00000044   ebx: cb18ab00   ecx: c0304d24   edx: 0003a5a9
esi: cb281d24   edi: 00000096   ebp: c156d5d4   esp: c031bc30
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0, stackpage=c031b000)
Stack: cc918240 00000085 cb18ab00 cb281d24 00000004 cc9169f7 cb18ab00 cb281d24
       cb281d24 00000004 c014dd33 cb281d24 00000004 c020c56e cb281d24 00000001
       00000004 00000004 00000000 00000008 c031bcc0 20000000 c156d5d4 00000001
Call Trace: [<cc918240>] [<cc9169f7>] [<c014dd33>] [<c020c56e>] [<c021c56b>]
   [<c0223b6e>] [<c014d39d>] [<c014d565>] [<c020c098>] [<c014dd33>] [<c0145794>]
   [<c014d39d>] [<c014d565>] [<c020c098>] [<c014dd33>] [<c01f71e1>] [<cc835198>]
   [<cc83ad45>] [<cc83889a>] [<cc838940>] [<c0223ad0>] [<c02197f2>] [<c010a9ae>]
   [<c010ad09>] [<c0106dd0>] [<c0106dd0>] [<c01e0018>] [<c01ed533>] [<c0106dd0>]
   [<c01ed420>] [<c0106e82>] [<c0105000>]

Code: 0f 0b 59 58 89 f6 f0 fe 8b cc 00 00 00 0f 88 21 19 00 00 8b
 <0>Kernel panic: Aiee, killing interrupt handler!
In interrupt handler - not syncing

>>EIP; cc9167da <[loop]loop_add_bio+3a/f0>   <=====
Trace; cc918240 <[loop].text.end+161/861>
Trace; cc9169f7 <[loop]loop_end_io_transfer+77/80>
Trace; c014dd33 <bio_endio+23/30>
Trace; c020c56e <end_that_request_first+19e/2a0>
Trace; c021c56b <__ide_end_request+bb/150>
Trace; c0223b6e <cdrom_read_intr+9e/330>
Trace; c014d39d <bio_destructor+4d/60>
Trace; c014d565 <bio_put+35/40>
Trace; c020c098 <end_bio_bh_io_sync+58/70>
Trace; c014dd33 <bio_endio+23/30>
Trace; c0145794 <mempool_free+104/110>
Trace; c014d39d <bio_destructor+4d/60>
Trace; c014d565 <bio_put+35/40>
Trace; c020c098 <end_bio_bh_io_sync+58/70>
Trace; c014dd33 <bio_endio+23/30>
Trace; c01f71e1 <add_timer_randomness+d1/e0>
Trace; cc835198 <[usb-ohci]roothub_portstatus+28/70>
Trace; cc83ad45 <[usb-ohci].text.end+876/8e1>
Trace; cc83889a <[usb-ohci]rh_send_irq+fa/1a0>
Trace; cc838940 <[usb-ohci]rh_int_timer_do+0/70>
Trace; c0223ad0 <cdrom_read_intr+0/330>
Trace; c02197f2 <ide_intr+162/240>
Trace; c010a9ae <handle_IRQ_event+5e/90>
Trace; c010ad09 <do_IRQ+109/1c0>
Trace; c0106dd0 <default_idle+0/40>
Trace; c0106dd0 <default_idle+0/40>
Trace; c01e0018 <acpi_ps_get_next_walk_op+1a8/1d0>
Trace; c01ed533 <pr_power_idle+113/270>
Trace; c0106dd0 <default_idle+0/40>
Trace; c01ed420 <pr_power_idle+0/270>
Trace; c0106e82 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Code;  cc9167da <[loop]loop_add_bio+3a/f0>
00000000 <_EIP>:
Code;  cc9167da <[loop]loop_add_bio+3a/f0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  cc9167dc <[loop]loop_add_bio+3c/f0>
   2:   59                        pop    %ecx
Code;  cc9167dd <[loop]loop_add_bio+3d/f0>
   3:   58                        pop    %eax
Code;  cc9167de <[loop]loop_add_bio+3e/f0>
   4:   89 f6                     mov    %esi,%esi
Code;  cc9167e0 <[loop]loop_add_bio+40/f0>
   6:   f0 fe 8b cc 00 00 00      lock decb 0xcc(%ebx)
Code;  cc9167e7 <[loop]loop_add_bio+47/f0>
   d:   0f 88 21 19 00 00         js     1934 <_EIP+0x1934> cc91810e <[loop].text.end+2f/861>
Code;  cc9167ed <[loop]loop_add_bio+4d/f0>
  13:   8b 00                     mov    (%eax),%eax

 <0>Kernel panic: Aiee, killing interrupt handler!

Re: BUG and Kernel Panic on 2.5.2-pre1 with loop and cdrom

[Zwane Mwaikambo]

Here is the floppy case with some debugging stuff put in. My kernel log is first, orignal
oops second and decoded oops last. Triggered with mount /dev/fd0 /floppy -o loop.
The ksymoops was run on the machine *right* after the oops (it was still
alive). If you want me to track down where the bh->b_size and nr_sectors
began to differ i can put extra checks further down.

Thanks,
	Zwane Mwaikambo

Dec 31 02:09:27 mondecino kernel: end_bio_bh_io_sync: nr_sectors=0 bh->b_size=2 <== the values before the BUG check
Dec 31 02:09:27 mondecino kernel: kernel BUG at ll_rw_blk.c:1367!
Dec 31 02:09:27 mondecino kernel: invalid operand: 0000
Dec 31 02:09:27 mondecino kernel: CPU:    0
Dec 31 02:09:27 mondecino kernel: EIP:    0010:[end_bio_bh_io_sync+77/112]    Not tainted
Dec 31 02:09:27 mondecino kernel: EIP:    0010:[<c020c08d>]    Not tainted
Dec 31 02:09:27 mondecino kernel: EFLAGS: 00010282
Dec 31 02:09:27 mondecino kernel: eax: 00000020   ebx: c8a92384   ecx: c0304d24   edx: 00002b24
Dec 31 02:09:27 mondecino kernel: esi: 00000000   edi: cb6fcc64   ebp: 00000000   esp: c8bfbf88
Dec 31 02:09:27 mondecino kernel: ds: 0018   es: 0018   ss: 0018
Dec 31 02:09:27 mondecino kernel: Process loop0 (pid: 727, stackpage=c8bfb000)
Dec 31 02:09:27 mondecino kernel: Stack: c02e1c63 00000557 cb6fcc64 caf4ac24 c9116000 c014dd33 cb6fcc64 00000000
Dec 31 02:09:27 mondecino kernel:        cc9171f8 cb6fcc64 00000001 00000000 cc9187c8 cc91879d 00000000 c9116000
Dec 31 02:09:27 mondecino kernel:        caf4ac24 cb6fcc64 c8fcbf30 00000000 00000000 00000000 00000f00 c8fcbf30
Dec 31 02:09:27 mondecino kernel: Call Trace: [bio_endio+35/48] [sound:num_midis_Rsmp_a1eae7cf+815152/51255793] [sound:nu
m_midis_Rsmp_a1eae7cf+820736/51250209] [sound:num_midis_Rsmp_a1eae7cf+820693/51250252] [kernel_thread+38/48]
Dec 31 02:09:27 mondecino kernel: Call Trace: [<c014dd33>] [<cc9171f8>] [<cc9187c8>] [<cc91879d>] [<c0107286>]
Dec 31 02:09:27 mondecino kernel:    [sound:num_midis_Rsmp_a1eae7cf+814520/51256425]
Dec 31 02:09:27 mondecino kernel:    [<cc916f80>]
Dec 31 02:09:27 mondecino kernel:
Dec 31 02:09:27 mondecino kernel: Code: 0f 0b 5e 58 8b 47 0c 83 e0 01 50 53 ff 53 34 57 e8 8e 14 f4

invalid operand: 0000
CPU:    0
EIP:    0010:[<c020c08d>]    Not tainted
EFLAGS: 00010282
eax: 00000020   ebx: c8a92384   ecx: c0304d24   edx: 00002b24
esi: 00000000   edi: cb6fcc64   ebp: 00000000   esp: c8bfbf88
ds: 0018   es: 0018   ss: 0018
Process loop0 (pid: 727, stackpage=c8bfb000)
Stack: c02e1c63 00000557 cb6fcc64 caf4ac24 c9116000 c014dd33 cb6fcc64 00000000
       cc9171f8 cb6fcc64 00000001 00000000 cc9187c8 cc91879d 00000000 c9116000
       caf4ac24 cb6fcc64 c8fcbf30 00000000 00000000 00000000 00000f00 c8fcbf30
Call Trace: [<c014dd33>] [<cc9171f8>] [<cc9187c8>] [<cc91879d>] [<c0107286>]
   [<cc916f80>]

>>EIP; c020c08d <end_bio_bh_io_sync+4d/70>   <=====
Trace; c014dd33 <bio_endio+23/30>
Trace; cc9171f8 <[loop]loop_thread+278/2d0>
Trace; cc9187c8 <[loop].text.end+689/8a1>
Trace; cc91879d <[loop].text.end+65e/8a1>
Trace; c0107286 <kernel_thread+26/30>
Trace; cc916f80 <[loop]loop_thread+0/2d0>
Code;  c020c08d <end_bio_bh_io_sync+4d/70>
00000000 <_EIP>:
Code;  c020c08d <end_bio_bh_io_sync+4d/70>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c020c08f <end_bio_bh_io_sync+4f/70>
   2:   5e                        pop    %esi
Code;  c020c090 <end_bio_bh_io_sync+50/70>
   3:   58                        pop    %eax
Code;  c020c091 <end_bio_bh_io_sync+51/70>
   4:   8b 47 0c                  mov    0xc(%edi),%eax
Code;  c020c094 <end_bio_bh_io_sync+54/70>
   7:   83 e0 01                  and    $0x1,%eax
Code;  c020c097 <end_bio_bh_io_sync+57/70>
   a:   50                        push   %eax
Code;  c020c098 <end_bio_bh_io_sync+58/70>
   b:   53                        push   %ebx
Code;  c020c099 <end_bio_bh_io_sync+59/70>
   c:   ff 53 34                  call   *0x34(%ebx)
Code;  c020c09c <end_bio_bh_io_sync+5c/70>
   f:   57                        push   %edi
Code;  c020c09d <end_bio_bh_io_sync+5d/70>
  10:   e8 8e 14 f4 00            call   f414a3 <_EIP+0xf414a3> c114d530 <_end+d6c100/c444c30>