Re: kexec and IRQ sharing

Re: kexec and IRQ sharing

[Eric W. Biederman]

Alan Stern <stern@rowland.harvard.edu> writes:

> On 7 Mar 2005, Eric W. Biederman wrote:
> 
> > Alan Stern <stern@rowland.harvard.edu> writes:
> > 
> > > Shared IRQs will always be a problem.  And the PCI subsystem doesn't
> > > implement shutdown() methods; that seems like a pretty big hole.  I guess 
> > > individual drivers can always create their own, however.
> > 
> > Not having seen this in practice what is the bad effect
> > that happens with shared irqs.  I know there is a warning messages
> > but is there anything beyond this, such as the irq being stuck on?
> 
> The problem comes when two devices share an IRQ line and at least one of 
> them is actively generating interrupt requests.  An ethernet interface or 
> a USB controller would make a good example, if they weren't shut down 
> properly during a kexec reboot.
> 
> Let's say devices A and B share the same IRQ, and B has an interrupt
> request pending.  Initially all the IRQ lines are disabled; they only get
> enabled as drivers request them.  So consider what happens when the driver
> for A is initialized.  It will probe A and will request the IRQ line be
> enabled.  B's outstanding request will immediately interrupt the
> processor.  But since there's no driver loaded for B, nobody will claim
> the interrupt or clear its source.
> 
> Result: The kernel warns about interrupts not owned by a driver, and 
> automatically disables the IRQ line.  Now device A is in trouble.  
> Obviously it can't function correctly without interrupts.  B will be in 
> trouble too, when its driver loads.
> 
> This is exactly what happened to Laurent Riffard in
> 
> http://bugme.osdl.org/show_bug.cgi?id=4294
> 
> Read his full dmesg attachment; you can see where IRQ 5 was triggered as 
> soon as one of the USB controllers enabled it.  Apparently the other USB 
> controller had a pending interrupt.
> 
> Things wouldn't be quite so bad if setup_irq() would re-enable a disabled
> shared IRQ when another driver requests it, but it doesn't.  Is this
> behavior a bug or is it deliberate?

Very good question.  I'm going to bounce this discussion towards
linux-kernel and because it appears that the kernels interrupt
handling behavior could be made more robust.  It looks like
irqpoll command line option is a step in that direction.

/sbin/kexec downs network interfaces before calling sys_reboot()
so for network interfaces this is largely solved.

The ongoing DMA transfers which are companions of the irq generating
events are what really concern me, as we could get all kinds of
interesting memory stomps.  Do you think you could implement
a reboot notifier or a shutdown() method to handle that case?

This appears to be yet another case of kexec transforming theoretical
bugs into actual ones.  Groan.

Eric

Re: [linux-usb-devel] Re: kexec and IRQ sharing

[David Brownell]

On Monday 07 March 2005 2:08 pm, Eric W. Biederman wrote:

> 
> The ongoing DMA transfers which are companions of the irq generating
> events are what really concern me, as we could get all kinds of
> interesting memory stomps.  Do you think you could implement
> a reboot notifier or a shutdown() method to handle that case?
> 
> This appears to be yet another case of kexec transforming theoretical
> bugs into actual ones.  Groan.

Reboot notifiers seem to be a more general solution than the
driver model shutdown() hooks, since as Alan noted not every
bus framework uses shutdown() ... and although remove() would
solve the issue too, for essentially all busses, it doesn't
kick in here (and linker tricks may have removed it anyway).

Of possible relevance:  Documentation/arm/Booting lists, at
the end, requirements the boot firmware must meet before it
runs Linux.  (Stuff like U-Boot, BLOB, HaRET, or dozens of
other open-sourced analogues of x86 BIOS code.)  Seems like
the same rules will apply before kexec ... yes?  That mentions
DMA, though not IRQs (possibly an omission).

- Dave

Re: kexec and IRQ sharing

[Eric W. Biederman]

Alan Stern <stern@rowland.harvard.edu> writes:

> I can implement either a shutdown method or a reboot notifier for
> uhci-hcd.  Note however that the upcoming changes to the PM core include a
> suspend call (PMSG_FREEZE) that does exactly what you want: quiesce the
> device, disable IRQ and DMA, but don't necessarily switch the power level.  
> (See Documentation/power/devices.txt.)  Unfortunately it won't be
> implemented until another few kernel releases have gone by.

Thanks.
 
> The idea of using kexec to recover from a partially-damaged kernel seems 
> unreliable.  Even attempting to quiesce devices prior to rebooting may 
> not work because of locks whose owners have crashed.

There is still a final bit of feeling our way through the weirdness,
in that case.  But in that case no devices will be quiesced before
we start the next kernel.  Basically very little beyond a jump
jump instruction will be executed at that point.  What is loaded must
be a very robust kernel capable of handling the few devices it needs
from an active state.  The irq weirdness you just ran into is
something that really hasn't been looked at.

In that case I suspect we will probably simply want to specify irqpoll
on the command line by default.  We are as safe as possible from DMA
traffic because we run out of a completely reserved area of memory
that the primary kernel never uses.

And the target is not a complete system recover but rather being able
to do something useful in the event of a crash like write a core
dump.

So in the crash dump case with a non-hardened driver that can't cope
the system may lock up, but I don't see it corrupting your data.
Which is the important bit.  And with drivers that have been
sufficiently hardened you will be able to do things like write
to disk etc.

Eric