Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Maneesh Soni]

Quoting Greg KH <greg@kroah.com>:

> On Fri, Jun 03, 2005 at 04:55:24PM +0530, Vivek Goyal wrote:
> > Hi,
> > 
> > In kdump, sometimes, general driver initialization issues seems to be
> cropping 
> > in second kernel due to devices not being shutdown during crash and these
> 
> > devices are sending interrupts while second kernel is booting and drivers
> are 
> > not expecting any interrupts yet.
> 
> What are the errors you are seeing?
> How would the drivers be able to be getting interrupts delivered to them
> if they haven't registered the irq handler yet?

Probably the boot log with error messages could have been inlined instead
of putting as attachement. This is from attachement in Vievk's post


irq 11: nobody cared (try booting with the "irqpoll" option. 
 [<c103a3ef>] __report_bad_irq+0x2a/0x8d 
 [<c1039c28>] handle_IRQ_event+0x39/0x6d 
 [<c103a4eb>] note_interrupt+0x7f/0xe4 
 [<c1039dc5>] __do_IRQ+0x169/0x194 
 [<c1004cc6>] do_IRQ+0x1b/0x28 
 [<c1003256>] common_interrupt+0x1a/0x20 
 [<c101d61b>] __do_softirq+0x2b/0x88 
 [<c101d69e>] do_softirq+0x26/0x2a 
 [<c101d759>] irq_exit+0x33/0x35 
 [<c1004ccb>] do_IRQ+0x20/0x28 
 [<c1003256>] common_interrupt+0x1a/0x20 
 [<c1018916>] release_console_sem+0x43/0xf6 
 [<c101875f>] vprintk+0x16c/0x277 
 [<c1074dd8>] d_lookup+0x23/0x46 
 [<c10748ac>] d_alloc+0x136/0x1a6 
 [<c10185ef>] printk+0x17/0x1b 
 [<c120f33f>] hub_probe+0xa0/0x168 
 [<c1096dd1>] sysfs_new_dirent+0x1f/0x64 
 [<c120d37f>] usb_probe_interface+0x59/0x71 
 [<c116c9b2>] driver_probe_device+0x2f/0xa7 
 [<c116ca2a>] __device_attach+0x0/0x5 
 [<c116c259>] bus_for_each_drv+0x58/0x78 
 [<c116ca8f>] device_attach+0x60/0x64 
 [<c116ca2a>] __device_attach+0x0/0x5 
 [<c116c383>] bus_add_device+0x29/0xa6 
 [<c116f9c7>] device_pm_add+0x56/0x9c 
 [<c116b7c3>] device_add+0xc5/0x14a 
 [<c121557e>] usb_set_configuration+0x346/0x528 
 [<c120fa47>] usb_new_device+0xab/0x1be 
 [<c120007b>] ohci_iso_recv_set_channel_mask+0x3/0xc6 
 [<c12124e1>] register_root_hub+0xaf/0x162 
 [<c12133bf>] usb_add_hcd+0x172/0x396 
 [<c1217bce>] usb_hcd_pci_probe+0x26e/0x375 
 [<c10284e7>] __call_usermodehelper+0x0/0x61 
 [<c110cbe4>] pci_device_probe_static+0x40/0x54 
 [<c110cc27>] __pci_device_probe+0x2f/0x42 
 [<c110cc63>] pci_device_probe+0x29/0x47 
 [<c116c9b2>] driver_probe_device+0x2f/0xa7 
 [<c116ca93>] __driver_attach+0x0/0x43 
 [<c116cad4>] __driver_attach+0x41/0x43 
 [<c116c1c2>] bus_for_each_dev+0x5a/0x7a 
 [<c116cafc>] driver_attach+0x26/0x2a 
 [<c116ca93>] __driver_attach+0x0/0x43 
 [<c116c5cb>] bus_add_driver+0x6b/0xa5 
 [<c110ce97>] pci_register_driver+0x5e/0x7c 
 [<c14246f5>] init+0x1d/0x25 
 [<c140c7ed>] do_initcalls+0x54/0xb6 
 [<c100029c>] init+0x0/0x10c 
 [<c100029c>] init+0x0/0x10c 
 [<c10002c6>] init+0x2a/0x10c 
 [<c1001348>] kernel_thread_helper+0x0/0xb 
 [<c100134d>] kernel_thread_helper+0x5/0xb 
handlers: 
[<c11e3421>] (ahd_linux_isr+0x0/0x284) 
Disabling IRQ #11 


Thanks
Maneesh


-- 
Maneesh Soni
IBM Linux Technology Center
IBM India Software Labs,
Bangalore, India
Ph. 91-80-25044990
email: maneesh@in.ibm.com

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Vivek Goyal]

> > 
> > > > Shared interrupts are an interesting case.  The simplest solution I can
> > > > think of for a crash dump capture kernel is to periodically poll
> > > > the hardware, as if all interrupts are shared.  At that level
> > > > I think we could get away with ignoring all hardware interrupt sources.
> > > 
> > > Yes, that's perfectly ok. We are no longer in a multitasking env.
> > 
> > Well we are at least capable of multitasking but that is no longer the
> > primary focus.  Having polling as at least an option should make
> > debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> > to do something very like this.
> > 
> 
> If I understand this right, the idea is that let all irqs be masked (except
> timer one) and invoke all the irq handlers whenever a timer interrupt occurs.
> This will automatcally be equivalent to drivers polling their devices for
> any interrupt.
> 
> As you mentioned that irqpoll option comes close. If enabled, it invokes
> all the irq handlers on every timer interrupt (IRQ0). The only difference is 
> that irqs are not masked (until and unless kernel masks these due to excessive
> unhandled interrupts). 
> 
> I tried booting kdump kernel with irqpoll option. It seems to be going 
> little bit ahead than previous point of failure (boot without irqpoll) but
> panics later. Following is the stack trace.
> 

Second kernel booted fine with MPT_DEBUG_IRQ enabled (with irqpoll option). 
There were few warning messages though spitted by the code under MPT_DEBUG_IRQ.

Looks like drivers need to be hardened on case to case basis to initialize 
properly even if underlying device is not in a reset state.

Thanks
Vivek

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Vivek Goyal]

On Tue, Jun 07, 2005 at 12:42:42PM -0600, Eric W. Biederman wrote:
> Grant Grundler <grundler@parisc-linux.org> writes:
> 
> > On Tue, Jun 07, 2005 at 03:59:18AM -0600, Eric W. Biederman wrote:
> > > > *lots* of PCI devices predate PCI2.3. Possibly even the majority.
> > > 
> > > In general generic hardware bits for disabling DMA, disabling interrupts
> > > and the like are all advisory.  With the current architecture things
> > > will work properly even if you don't manage to disable DMA (assuming
> > > you don't reassign IOMMU entries at least).
> > 
> > ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require
> > use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed.
> > Probably want to make a choice to ignore those arches for now
> > or sort out how to deal with an IOMMU.
> 
> The howto deal with an IOMMU has been sorted out but so far no one 
> has actually done it.  What has been discussed previously is simply
> reserving a handful of IOMMU entries, and then only using those
> in the crash recover kernel.  This is essentially what we do with DMA
> on architectures that don't have an IOMMU and it seems quite safe
> enough there.
> 
> > > Shared interrupts are an interesting case.  The simplest solution I can
> > > think of for a crash dump capture kernel is to periodically poll
> > > the hardware, as if all interrupts are shared.  At that level
> > > I think we could get away with ignoring all hardware interrupt sources.
> > 
> > Yes, that's perfectly ok. We are no longer in a multitasking env.
> 
> Well we are at least capable of multitasking but that is no longer the
> primary focus.  Having polling as at least an option should make
> debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> to do something very like this.
> 

If I understand this right, the idea is that let all irqs be masked (except
timer one) and invoke all the irq handlers whenever a timer interrupt occurs.
This will automatcally be equivalent to drivers polling their devices for
any interrupt.

As you mentioned that irqpoll option comes close. If enabled, it invokes
all the irq handlers on every timer interrupt (IRQ0). The only difference is 
that irqs are not masked (until and unless kernel masks these due to excessive
unhandled interrupts). 

I tried booting kdump kernel with irqpoll option. It seems to be going 
little bit ahead than previous point of failure (boot without irqpoll) but
panics later. Following is the stack trace.

mptbase: Initiating ioc0 bringup
Unable to handle kernel NULL pointer dereference at virtual address 00000608
 printing eip:
c11e1b73
*pde = 00000000
Oops: 0000 [#1]
Modules linked in:
CPU:    0
EIP:    0060:[<c11e1b73>]    Not tainted VLI
EFLAGS: 00010006   (2.6.12-rc6-mm1-16M)
EIP is at mptscsih_io_done+0x23/0x350
eax: c1778400   ebx: 00000000   ecx: 00000600   edx: 00000250
esi: 00000000   edi: 0000000e   ebp: 00000001   esp: c15efcbc
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 1, threadinfo=c15ee000 task=c147fa00)
Stack: c15f02d1 c1116cfd c1116d60 c15df660 00000000 0000006c 00000250 00000000
       00000000 0000000e 00000001 c11db9ea c1778400 00000600 00000000 00000000
       00000600 c1788aa0 c1410e80 00000009 00000001 c1039e7c 00000009 c1778400
Call Trace:
 [<c1116cfd>] acpi_ev_gpe_detect+0x83/0x10f
 [<c1116d60>] acpi_ev_gpe_detect+0xe6/0x10f
 [<c11db9ea>] mpt_interrupt+0xfa/0x1e0
 [<c1039e7c>] misrouted_irq+0xec/0x100
 [<c103a007>] note_interrupt+0xb7/0xf0
 [<c10399c4>] __do_IRQ+0xe4/0xf0
 [<c1004e09>] do_IRQ+0x19/0x30
 [<c1003246>] common_interrupt+0x1a/0x20
 [<c1018d8f>] release_console_sem+0x3f/0xa0
 [<c1018c27>] vprintk+0x177/0x220
 [<c126ecbd>] pci_read+0x3d/0x50
 [<c1101a0a>] kobject_get+0x1a/0x30
 [<c116b676>] get_device+0x16/0x30
 [<c110b95a>] pci_dev_get+0x1a/0x30
 [<c1018aa7>] printk+0x17/0x20
 [<c11dcbeb>] mpt_do_ioc_recovery+0x4b/0x540
 [<c11dc4ff>] mpt_attach+0x2ef/0x690
 [<c11e6e03>] mptspi_probe+0x23/0x3e0
 [<c110b4f2>] pci_device_probe_static+0x52/0x70
 [<c110b54c>] __pci_device_probe+0x3c/0x50
 [<c110b58f>] pci_device_probe+0x2f/0x50
 [<c116cb68>] driver_probe_device+0x38/0xb0
 [<c116cc60>] __driver_attach+0x0/0x50
 [<c116cca7>] __driver_attach+0x47/0x50
 [<c116c229>] bus_for_each_dev+0x69/0x80
 [<c116ccd6>] driver_attach+0x26/0x30
 [<c116cc60>] __driver_attach+0x0/0x50
 [<c116c6d8>] bus_add_driver+0x88/0xc0
 [<c110b6d0>] pci_device_shutdown+0x0/0x30
 [<c110b857>] pci_register_driver+0x67/0x90
 [<c142a890>] mptspi_init+0xa0/0xb0
 [<c11e37a0>] mptscsih_ioc_reset+0x0/0x170
 [<c141483c>] do_initcalls+0x2c/0xc0
 [<c142d5fa>] sock_init+0x2a/0x40
 [<c1000290>] init+0x0/0x100
 [<c10002b5>] init+0x25/0x100
 [<c10013a0>] kernel_thread_helper+0x0/0x10
 [<c10013a5>] kernel_thread_helper+0x5/0x10
Code: 00 8d bc 27 00 00 00 00 55 57 56 53 83 ec 1c 8b 44 24 30 8b 4c 24 34 8b 7
 <0>Kernel panic - not syncing: Fatal exception in interrupt


Thanks
Vivek

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Eric W. Biederman]

Grant Grundler <grundler@parisc-linux.org> writes:

> On Tue, Jun 07, 2005 at 12:42:42PM -0600, Eric W. Biederman wrote:
> > The howto deal with an IOMMU has been sorted out but so far no one 
> > has actually done it.  What has been discussed previously is simply
> > reserving a handful of IOMMU entries,
> 
> How? with dma_alloc_consistent() or some special hook?
> I'm just curious.

We didn't get that far but I believe the idea was a special hook.

> ...
> >  and then only using those
> > in the crash recover kernel.  This is essentially what we do with DMA
> > on architectures that don't have an IOMMU and it seems quite safe
> > enough there.
> 
> Yeah, in general that should be feasible.
> 
> One might be able to trivially allocate a small, seperate IO PDIR
> just for KDUMP and switch to that. Key thing is it be physically
> contiguous in memory. Very little code is involved with IO Pdir
> setup for both parisc and IA64. I can't speak for Alpha/sparc/ppc/et al.

Cool.
 
> ...
> > Well we are at least capable of multitasking but that is no longer the
> > primary focus.  Having polling as at least an option should make
> > debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> > to do something very like this.
> 
> You could run the itimer but I don't see why you should.
> Kdump is essentially an embedded linux kernel. It really
> doesn't need to be premptive multitasking either.

It is mostly a matter of minimizing differences from the norm.

> Anyway, sounds like you guys are on the right track.

Thanks.   It just takes a while for the simple solutions to
get there.

Eric

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Grant Grundler]

On Tue, Jun 07, 2005 at 12:42:42PM -0600, Eric W. Biederman wrote:
> The howto deal with an IOMMU has been sorted out but so far no one 
> has actually done it.  What has been discussed previously is simply
> reserving a handful of IOMMU entries,

How? with dma_alloc_consistent() or some special hook?
I'm just curious.

...
>  and then only using those
> in the crash recover kernel.  This is essentially what we do with DMA
> on architectures that don't have an IOMMU and it seems quite safe
> enough there.

Yeah, in general that should be feasible.

One might be able to trivially allocate a small, seperate IO PDIR
just for KDUMP and switch to that. Key thing is it be physically
contiguous in memory. Very little code is involved with IO Pdir
setup for both parisc and IA64. I can't speak for Alpha/sparc/ppc/et al.

...
> Well we are at least capable of multitasking but that is no longer the
> primary focus.  Having polling as at least an option should make
> debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> to do something very like this.

You could run the itimer but I don't see why you should.
Kdump is essentially an embedded linux kernel. It really
doesn't need to be premptive multitasking either.

Anyway, sounds like you guys are on the right track.

thanks,
grant

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Eric W. Biederman]

Grant Grundler <grundler@parisc-linux.org> writes:

> On Tue, Jun 07, 2005 at 03:59:18AM -0600, Eric W. Biederman wrote:
> > > *lots* of PCI devices predate PCI2.3. Possibly even the majority.
> > 
> > In general generic hardware bits for disabling DMA, disabling interrupts
> > and the like are all advisory.  With the current architecture things
> > will work properly even if you don't manage to disable DMA (assuming
> > you don't reassign IOMMU entries at least).
> 
> ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require
> use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed.
> Probably want to make a choice to ignore those arches for now
> or sort out how to deal with an IOMMU.

The howto deal with an IOMMU has been sorted out but so far no one 
has actually done it.  What has been discussed previously is simply
reserving a handful of IOMMU entries, and then only using those
in the crash recover kernel.  This is essentially what we do with DMA
on architectures that don't have an IOMMU and it seems quite safe
enough there.

> > Shared interrupts are an interesting case.  The simplest solution I can
> > think of for a crash dump capture kernel is to periodically poll
> > the hardware, as if all interrupts are shared.  At that level
> > I think we could get away with ignoring all hardware interrupt sources.
> 
> Yes, that's perfectly ok. We are no longer in a multitasking env.

Well we are at least capable of multitasking but that is no longer the
primary focus.  Having polling as at least an option should make
debugging easier.  Last I looked Andrews kernel hand an irqpoll option
to do something very like this.

> > Does anyone know of a anything that would break by always polling
> > the hardware?  I guess there could be a problem with drivers
> > that don't understand shared interrupts, are there enough of those
> > to be an issue.
> 
> PCI requires drivers support Shared IRQs.
> A few oddballs might be broken but I expect networking/mass storage
> drivers get this right.

Agreed.  Which means any drivers we really need for dumping the system
should be fine.  If the drivers don't work in that mode at least the
concept of fixing it won't be controversial.


Eric

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Eric W. Biederman]

Dipankar Sarma <dipankar@in.ibm.com> writes:

> On Sat, Jun 04, 2005 at 07:13:06PM +0530, Dipankar Sarma wrote:
> > That said, I am not sure what is the issue with the console
> > drivers. What good is the irq for the console driver if
> > it hasn't requested for it ? Why should disabling it affect
> > consoles ? The interrupt will get enabled as soon as the driver
> > requests for it as per Vivek's patch. Am I missing something here ?
> 
> Doh! The answer is in earlier emails - fw controlled pci consoles.

There is still the question do fw controlled pci consoles
intersect with consoles used during kdump.  I would be
very surprised if the intersected but they might.

Eric

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Dipankar Sarma]

On Sat, Jun 04, 2005 at 07:13:06PM +0530, Dipankar Sarma wrote:
> That said, I am not sure what is the issue with the console
> drivers. What good is the irq for the console driver if
> it hasn't requested for it ? Why should disabling it affect
> consoles ? The interrupt will get enabled as soon as the driver
> requests for it as per Vivek's patch. Am I missing something here ?

Doh! The answer is in earlier emails - fw controlled pci consoles.

Thanks
Dipankar

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Dipankar Sarma]

On Sat, Jun 04, 2005 at 04:18:24PM +0300, Denis Vlasenko wrote:
> On Friday 03 June 2005 21:36, Eric W. Biederman wrote:
> > 
> > As I recall the drivers were not getting the interrupts but the interrupts
> > were happening.  To stop being spammed the kernel disables the irq line,
> > at the interrupt controller.  Then when the driver registered the
> > interrupt it would never receive the interrupt.
> 
> Shouldn't kernel keep all interrupt lines initially disabled 
> (sans platform-specific magic), and enable each like only when
> a device driver requests IRQ? This sounds simpler to do...

This doesn't help kdump folks. Interrupt pending from a device
from the first boot can flood the system when another driver
sharing the irq gets enabled in the second boot (kdump boot).
The disabling of interrupts need to be done on a per-device
basis for kdump to avoid this problem.

That said, I am not sure what is the issue with the console
drivers. What good is the irq for the console driver if
it hasn't requested for it ? Why should disabling it affect
consoles ? The interrupt will get enabled as soon as the driver
requests for it as per Vivek's patch. Am I missing something here ?

Thanks
Dipankar