From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751658AbdJBVLc (ORCPT ); Mon, 2 Oct 2017 17:11:32 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:50890 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751128AbdJBVLa (ORCPT ); Mon, 2 Oct 2017 17:11:30 -0400 Date: Mon, 2 Oct 2017 23:11:25 +0200 (CEST) From: Thomas Gleixner To: LKML cc: x86@kernel.org, Peter Zijlstra , Arjan van de Veen , Tony Luck , Borislav Petkov Subject: [RFC] x86: Moving edge interrupt affinity in process context Message-ID: User-Agent: Alpine 2.20 (DEB 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Linutronix-Spam-Score: -1.0 X-Linutronix-Spam-Level: - X-Linutronix-Spam-Status: No , -1.0 points, 5.0 required, ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi! I'm looking into the constraint of moving interrupt affinities in interrupt context only which we have for non remapped interrupts. After staring into manuals and application notes for quite a while I came to the conclusion that this is only required for level type interrupts. The reason for this is the Remote IRR bit in the IOAPIC. From the SDM: "This bit is used for level triggered interrupts. Its meaning is undefined for edge triggered interrupts. For level triggered interrupts, this bit is set to 1 when local APIC(s) accept the level interrupt sent by the IOAPIC. The Remote IRR bit is set to 0 when an EOI message with a matching interrupt vector is received from a local APIC." Moving the interrupt affinity in non interrupt context might cause a stale remote IRR in the following case: IOAPIC APIC pin A triggers sends message vector X accepts message, sets APIC IRR bit for vector X sets IOAPIC remote IRR bit --> change vector for pin A to vector Y interrupt handler runs EOI() sends EOI message for vector X ===> Does not clear the pending remote IRR bit in the IOAPIC because vector X != vector Y, which is now associated to pin A, which makes that thing stale. That's at least how I understand the issue from the not really elaborative documentation. Though for edge interrupts I'm not really understanding why we would have to move them in interrupt context. EOI is not sent out to the originating device for edge type interrupts, as there is no related state in the device. Of course we'd need to keep the previous vector around and active up to the point where the next interrupt is delivered or any other synchronization point makes sure that it cannot be on flight to the previous destination. But that's something we need to do anyway. Lifting the 'only move in interrupt context' restriction would remove quite some work from interrupt context in case of changing the interrupt affinity of an edge interrupt, which is the vast majority of interrupts today (MSI/MSI-X). I started to look into that initialy to check the feasability of multi-MSI support w/o interrupt remapping, but supporting that has other implications as we'd need to support moving affinity of all MSI vectors in one go, which requires rather intrusive changes to both x86 and the core interrupt code. Not sure, that this is really worth the trouble. It still needs to be analyzed how much gain this really brings, but I won't start with looking into that without confirmation from the HW folks that this is actually a safe thing to do. Even if it turns out that it's not worth it, it would at least be good to have it documented how all of this works and why we are using the 'move in interrupt context' mode unconditionally. Thanks, tglx