Re: Question about interrupt prioriyt of ARM GICv3/4

[Mark Kettenis <mark.kettenis@xs4all.nl>]

> Date: Thu, 12 Dec 2024 10:12:32 +0000
> From: Marc Zyngier <maz@kernel.org>

Hi Marc, Richard,

> On Thu, 12 Dec 2024 09:18:56 +0000,
> richard clark <richard.xnu.clark@gmail.com> wrote:
> > 
> > Hi M,
> 
> Hi r,
> 
> > 
> > On Fri, Dec 6, 2024 at 5:37 PM Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > On Fri, 06 Dec 2024 08:33:11 +0000,
> > > richard clark <richard.xnu.clark@gmail.com> wrote:
> > > >
> > > > Hi,
> > > > Currently seems the GICv3/4 irqchip configures all the interrupts as
> > > > the same priority, I am thinking about to minimize the latency of the
> > > > interrupt for a particular device, e.g, the arm arch_timer in the RTL
> > > > system. The question is,
> > > > 1. Why don't we provide a /proc or /sys interface for the enduser to
> > > > set the priority of a specific interrupt(SPI/PPI)?
> > >
> > > I'm afraid this really has nothing to do with any particular interrupt
> > > architecture.
> > >
> > > Before thinking of exposing the interrupt priority to userspace, you
> > > should look at what this translates into for the kernel once you allow
> > > interrupts to be preempted by another one with a higher priority.
> > >
> > Interrupt priority doesn't necessarily mean the preemption, seems
> > you're talking about the interrupt preemption harm according to your
> > statement followed.Frankly I am just thinking higher priority will win
> > the lower ones in case massive external interrupts received in the GIC
> > level (you see I am still talking about GIC, not kernel)
> 
> As I stated at the end of my email, the GIC only gives guarantee that
> you will ack the highest priority interrupt in finite time. Not right
> when it is made pending. Yes, it has the concept of HPPI. But that
> from the PoV of the CPU interface, not that of the distributor. Factor
> in the Stream interface, and you realise that expecting to always ack
> the highest priority pending interrupt is akin to expecting no
> reordering of packets in a network.
> 
> > >
> > > This means that at every point where you would normally see a
> > > local_irq_save(), spinlock_irqsave() or equivalent, you would need to
> > > explicitly specify the priority that you allow for preemption. You
> > > should then make sure that any code that can be run during an
> > > interrupt is reentrant. You need to define which data structures can
> > > be manipulated at which priority level... The list goes on.
> > >
> > irqsave just masks the interrupt from the point of cpu, I don't think
> > it will mess things up if preemption really happens (no? then what the
> > side-effect is for the nested interrupt handling in the softirq part.
> > damage the semantic of the lock primitives?)
> > >
> > > If you want a small taste of the complexity, just look at what
> > > handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
> > > generalise it to not just two, but an arbitrary range of priorities.
> > >
> > > If you want the full blown experience, look at the BSDs and their use
> > > of spl*(). I don't think anyone has any plan to get there, and the RT
> > > patches have shown that there is little need for it.
> > >
> > As supplement，the fiq is suggested to be used as an alternative to the
> > higher priority in the RT area...
> 
> <PulpFiction>
> FIQ's dead, baby. FIQ's dead.
> </PulpFiction>

Hah, tell that to Apple! ;).

> > > > 2. Is there any way to verify the higher priority interrupt will have
> > > > more dominant to be selected to the CPU (IOW, the priority is really
> > > > working) in case of multiple different interrupts asserted to the GIC
> > > > at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> > > > to record higher priority wins)?
> > >
> > > The GIC architecture makes no promise that the interrupt you
> > > acknowledge is the highest priority pending interrupt, because this is
> > > by definition a very racy process.
> > >
> > > Also, even on busy systems, you will very rarely see two interrupts
> > > targeting the same CPU being made pending at the same time, so that
> > > the interrupt delivery system would have to arbitrate between the two.
> > > That's because interrupts are vanishingly rare in the grand scheme of
> > > things.
> > >
> > 1. I am trying to stress the external interrupts to the core#0 via the
> > stress-ng tool with one of interrupt being configured as higher
> > priority to see the benchmark data, it's time consuming as you can
> > image, still is in progress(BTW, I can't see any lockup similar hang
> > in the system with a higher priority configured)
> 
> If you don't have preemption, I don't think anything wrong will
> happen. But I don't expect any benefit either.

Based on my experience with OpenBSD, I'm not even sure there is much
benefit even if you have preemtion.

And regarding anything wrong happening: there is an interesting bug in
the RK3399 GIC integration where it gets the priorities wrong:

https://github.com/openbsd/src/blob/feb3ea439d8f49b3c0e33f54c34631a611b98e21/sys/arch/arm64/dev/agintc.c#L395

(that comment is my interpretation of what's happening; I might be
misinterpreting what's really going on)

As far as I can tell the Linux code doesn't handle that quirk.
Probably it doesn't matter because Linux only uses the priority
mechanisms to implement pseudo-NMI functionality and/or doesn't do
preemption of interrupts.

> > 2. This raises a very interesting question and I am also very curious
> > about is, what is the purpose for the GIC to introduce the interrupt
> > priority features, a placeholder feature reserved for the future? Ah,
> > hardware prefer to provide more functionalities than its being
> > actually used by software, any other justification except that?
> 
> You realise that the HW is not exclusively designed for Linux, right?
> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.
> 
>