SMMUv3 interrupt handling via custom logic

SMMUv3 interrupt handling via custom logic

[Michal Simek]

Hi Will and Robin, (+Stefano, Anirudha)

We are using smmu-v3 in our SOC and I would like to ask you for recommendation 
how to handle our interrupt cases.

here is description which we are using

smmu: iommu@ec000000 {
	compatible = "arm,smmu-v3";
	reg = <...>;
	#iommu-cells = <1>;
	interrupt-names = "combined";
	interrupts = <0 169 4>;
};

but it is missing one important detail which just arise that actually there is 
additional HW logic which deals with SMMU interrupts separately.
There is a secure part (global, cmd, event - gerror, cmdq-sync, eventq in DT)
and non secure part (pri, global, cmd, event - priq, gerror, cmdq-sync, eventq 
in DT).
Based on my information all these interrupts should be acked once handled to be 
able to get another one.
The driver itself is able to handle them separately but we didn't create any 
solution to reach custom HW to do it.

I looked at f935448acf46 ("iommu/arm-smmu-v3: Add workaround for Cavium 
ThunderX2 erratum #126") which introduced combined IRQs but it looks like that 
there is no need for additional ACK of that IRQs.

The HW logic itself is handling secure and non secure settings for SMMU that's 
why would be the best to avoid directly mapping it in Linux.

One way to go is to create secondary interrupt controller driver
a) ioremap one with notice about secure part because we are using SMMU only with 
NS world
b) firmware based to tunnel accesses via SMCs and allow only access to limited 
amount of registers

The second way is likely create any hooks in the driver to be able to provide 
additional SOC specific hooks.

I am not quite sure which way would be the best that's why I would like to get 
some recommendation from you.

Stefano: please correct me if any of my description is not accurate.

Thanks,
Michal

Re: SMMUv3 interrupt handling via custom logic

[Robin Murphy]

On 2025-06-27 8:19 am, Michal Simek wrote:
> Hi Will and Robin, (+Stefano, Anirudha)
> 
> We are using smmu-v3 in our SOC and I would like to ask you for 
> recommendation how to handle our interrupt cases.
> 
> here is description which we are using
> 
> smmu: iommu@ec000000 {
>      compatible = "arm,smmu-v3";
>      reg = <...>;
>      #iommu-cells = <1>;
>      interrupt-names = "combined";
>      interrupts = <0 169 4>;
> };
> 
> but it is missing one important detail which just arise that actually 
> there is additional HW logic which deals with SMMU interrupts separately.
> There is a secure part (global, cmd, event - gerror, cmdq-sync, eventq 
> in DT)
> and non secure part (pri, global, cmd, event - priq, gerror, cmdq-sync, 
> eventq in DT).
> Based on my information all these interrupts should be acked once 
> handled to be able to get another one.
> The driver itself is able to handle them separately but we didn't create 
> any solution to reach custom HW to do it.
> 
> I looked at f935448acf46 ("iommu/arm-smmu-v3: Add workaround for Cavium 
> ThunderX2 erratum #126") which introduced combined IRQs but it looks 
> like that there is no need for additional ACK of that IRQs.

Per the architecture, SMMU interrupts are logically edge-triggered so 
there is nothing to clear at the SMMU end (the "interrupt status" is 
implicit in whatever condition caused an interrupt to be sent, e.g. the 
event queue becoming non-empty, SMMU_GERROR becoming different from 
SMMU_GERRORN, etc.)

If this is an Arm SMMU IP (MMU-600/700/S3) then the physical interrupt 
outputs are most definitely rising-edge. If somone's stuck some 
interrupt combiner in between those and the main interrupt controller, 
then yes, that interrupt combiner really should have its own driver.

> The HW logic itself is handling secure and non secure settings for SMMU 
> that's why would be the best to avoid directly mapping it in Linux.
> 
> One way to go is to create secondary interrupt controller driver
> a) ioremap one with notice about secure part because we are using SMMU 
> only with NS world
> b) firmware based to tunnel accesses via SMCs and allow only access to 
> limited amount of registers
> 
> The second way is likely create any hooks in the driver to be able to 
> provide additional SOC specific hooks.

If this thing is munging *all* the SMMU interrupt outputs as I suspect, 
then the big problem with that idea is that "the driver" is at least two 
separate drivers (SMMU and PMU), 3 if it has RAS and you ever want to 
entertain the idea of kernel-first handling.

Thanks,
Robin.

> 
> I am not quite sure which way would be the best that's why I would like 
> to get some recommendation from you.
> 
> Stefano: please correct me if any of my description is not accurate.
> 
> Thanks,
> Michal

Re: SMMUv3 interrupt handling via custom logic

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3397 bytes --]

Hi Will, Robin,

Long time no see!

On Fri, 27 Jun 2025, Robin Murphy wrote:
> On 2025-06-27 8:19 am, Michal Simek wrote:
> > Hi Will and Robin, (+Stefano, Anirudha)
> > 
> > We are using smmu-v3 in our SOC and I would like to ask you for
> > recommendation how to handle our interrupt cases.
> > 
> > here is description which we are using
> > 
> > smmu: iommu@ec000000 {
> >      compatible = "arm,smmu-v3";
> >      reg = <...>;
> >      #iommu-cells = <1>;
> >      interrupt-names = "combined";
> >      interrupts = <0 169 4>;
> > };
> > 
> > but it is missing one important detail which just arise that actually there
> > is additional HW logic which deals with SMMU interrupts separately.
> > There is a secure part (global, cmd, event - gerror, cmdq-sync, eventq in
> > DT)
> > and non secure part (pri, global, cmd, event - priq, gerror, cmdq-sync,
> > eventq in DT).
> > Based on my information all these interrupts should be acked once handled to
> > be able to get another one.
> > The driver itself is able to handle them separately but we didn't create any
> > solution to reach custom HW to do it.
> > 
> > I looked at f935448acf46 ("iommu/arm-smmu-v3: Add workaround for Cavium
> > ThunderX2 erratum #126") which introduced combined IRQs but it looks like
> > that there is no need for additional ACK of that IRQs.
> 
> Per the architecture, SMMU interrupts are logically edge-triggered so there is
> nothing to clear at the SMMU end (the "interrupt status" is implicit in
> whatever condition caused an interrupt to be sent, e.g. the event queue
> becoming non-empty, SMMU_GERROR becoming different from SMMU_GERRORN, etc.)
> 
> If this is an Arm SMMU IP (MMU-600/700/S3) then the physical interrupt outputs
> are most definitely rising-edge. If somone's stuck some interrupt combiner in
> between those and the main interrupt controller, then yes, that interrupt
> combiner really should have its own driver.
> 
> > The HW logic itself is handling secure and non secure settings for SMMU
> > that's why would be the best to avoid directly mapping it in Linux.
> > 
> > One way to go is to create secondary interrupt controller driver
> > a) ioremap one with notice about secure part because we are using SMMU only
> > with NS world
> > b) firmware based to tunnel accesses via SMCs and allow only access to
> > limited amount of registers
> > 
> > The second way is likely create any hooks in the driver to be able to
> > provide additional SOC specific hooks.
> 
> If this thing is munging *all* the SMMU interrupt outputs as I suspect, then
> the big problem with that idea is that "the driver" is at least two separate
> drivers (SMMU and PMU), 3 if it has RAS and you ever want to entertain the
> idea of kernel-first handling.

Yeah... I tend to favor simple solutions when possible and the secondary
interrupt controller driver approach is looking increasingly complex.

In addition to what Robin mentioned, I understand that this email is
directed to linux-arm-kernel and the LKML, so the focus is naturally to
solve the problem for Linux. However, let me point out that this issue
also affects Xen, all hypervisors, and proprietary operating systems.
The "big problem" is even bigger :-(  Complexity will multiply very
quickly.

Do you have any creative ideas for how this could be handled more
simply?  Maybe with help from the firmware (e.g. TF-A)?

Re: SMMUv3 interrupt handling via custom logic

[Will Deacon]

Hey Stefano, [+Marc]

On Fri, Jun 27, 2025 at 12:20:37PM -0700, Stefano Stabellini wrote:
> On Fri, 27 Jun 2025, Robin Murphy wrote:
> > On 2025-06-27 8:19 am, Michal Simek wrote:
> > > We are using smmu-v3 in our SOC and I would like to ask you for
> > > recommendation how to handle our interrupt cases.
> > > 
> > > here is description which we are using
> > > 
> > > smmu: iommu@ec000000 {
> > >      compatible = "arm,smmu-v3";
> > >      reg = <...>;
> > >      #iommu-cells = <1>;
> > >      interrupt-names = "combined";
> > >      interrupts = <0 169 4>;
> > > };
> > > 
> > > but it is missing one important detail which just arise that actually there
> > > is additional HW logic which deals with SMMU interrupts separately.
> > > There is a secure part (global, cmd, event - gerror, cmdq-sync, eventq in
> > > DT)
> > > and non secure part (pri, global, cmd, event - priq, gerror, cmdq-sync,
> > > eventq in DT).
> > > Based on my information all these interrupts should be acked once handled to
> > > be able to get another one.
> > > The driver itself is able to handle them separately but we didn't create any
> > > solution to reach custom HW to do it.
> > > 
> > > I looked at f935448acf46 ("iommu/arm-smmu-v3: Add workaround for Cavium
> > > ThunderX2 erratum #126") which introduced combined IRQs but it looks like
> > > that there is no need for additional ACK of that IRQs.
> > 
> > Per the architecture, SMMU interrupts are logically edge-triggered so there is
> > nothing to clear at the SMMU end (the "interrupt status" is implicit in
> > whatever condition caused an interrupt to be sent, e.g. the event queue
> > becoming non-empty, SMMU_GERROR becoming different from SMMU_GERRORN, etc.)
> > 
> > If this is an Arm SMMU IP (MMU-600/700/S3) then the physical interrupt outputs
> > are most definitely rising-edge. If somone's stuck some interrupt combiner in
> > between those and the main interrupt controller, then yes, that interrupt
> > combiner really should have its own driver.
> > 
> > > The HW logic itself is handling secure and non secure settings for SMMU
> > > that's why would be the best to avoid directly mapping it in Linux.
> > > 
> > > One way to go is to create secondary interrupt controller driver
> > > a) ioremap one with notice about secure part because we are using SMMU only
> > > with NS world
> > > b) firmware based to tunnel accesses via SMCs and allow only access to
> > > limited amount of registers
> > > 
> > > The second way is likely create any hooks in the driver to be able to
> > > provide additional SOC specific hooks.
> > 
> > If this thing is munging *all* the SMMU interrupt outputs as I suspect, then
> > the big problem with that idea is that "the driver" is at least two separate
> > drivers (SMMU and PMU), 3 if it has RAS and you ever want to entertain the
> > idea of kernel-first handling.
> 
> Yeah... I tend to favor simple solutions when possible and the secondary
> interrupt controller driver approach is looking increasingly complex.

I think that's probably the right way to go, though. Moving the
configuration to firmware just means you now have two problems instead
of one.

It looks like MTK may have done something similar:

https://lore.kernel.org/r/20250616025628.25454-7-xueqi.zhang@mediatek.com

so if changes are needed to irqchip to handle the limited capabilities
of the extra logic (as per Marc's comments in the thread above), it
would be good to make sure that's reusable. What I really _don't_ want
is a half-baked, custom interrupt handling mechanism in the IOMMU code.

> In addition to what Robin mentioned, I understand that this email is
> directed to linux-arm-kernel and the LKML, so the focus is naturally to
> solve the problem for Linux. However, let me point out that this issue
> also affects Xen, all hypervisors, and proprietary operating systems.
> The "big problem" is even bigger :-(  Complexity will multiply very
> quickly.

Sounds like you need to feed that back to the geniuses who designed this
hardware :). If you want your hardware to work well with existing
software, it's generally best to build it to spec.

Will