From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760272AbXEROVT (ORCPT ); Fri, 18 May 2007 10:21:19 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754665AbXEROVG (ORCPT ); Fri, 18 May 2007 10:21:06 -0400 Received: from ebiederm.dsl.xmission.com ([166.70.28.69]:44930 "EHLO ebiederm.dsl.xmission.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755237AbXEROVF (ORCPT ); Fri, 18 May 2007 10:21:05 -0400 From: ebiederm@xmission.com (Eric W. Biederman) To: Andi Kleen Cc: "Siddha, Suresh B" , mingo@elte.hu, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, nanhai.zou@intel.com, asit.k.mallick@intel.com, keith.packard@intel.com Subject: Re: [patch] x86_64, irq: check remote IRR bit before migrating level triggered irq References: <20070517230324.GB8089@linux-os.sc.intel.com> <200705181400.12811.ak@suse.de> Date: Fri, 18 May 2007 08:19:42 -0600 In-Reply-To: <200705181400.12811.ak@suse.de> (Andi Kleen's message of "Fri, 18 May 2007 14:00:12 +0200") Message-ID: User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Andi Kleen writes: > On Friday 18 May 2007 01:03, Siddha, Suresh B wrote: > >> Normally, the EOI generated by local APIC for level trigger interrupt >> contains vector number. The IOAPIC will take this vector number and >> search the IOAPIC RTE entries for an entry with matching vector number and >> clear the remote IRR bit (indicate EOI). However, if the vector number is >> changed (as in step 3) the IOAPIC will not find the RTE entry when the EOI >> is received later. This will cause the remote IRR to get stuck causing the >> interrupt hang (no more interrupt from this RTE). > > Does this happen often or did you only see it in some extreme or obscure > case? It is obscure. Someone may have a case that makes it easy to reproduce but it won't happen frequently. So we should have enough time to review and fix this carefully, instead of needing to rush a fix out. >> + /* >> + * If the EOI still didn't reach the RTE corresponding to the >> + * level triggered irq, postpone the irq migration to the next >> + * irq arrival event. >> + */ >> + if (pending_eoi(irq)) { >> + irq_desc[irq].status |= IRQ_MOVE_PENDING; >> + return; > > Other code seems to have similar problems, but we don't have any lock > protecting that bitmap against parallel updates outside the irq itself, don't > we? Perhaps it needs to be all set_bit() Huh? The irq lock is sufficient for this. This is not a code software problem, this is a software hardware interaction problem. This is a problem in that there is no way to guarantee that ioapics see a series of manipulations in the same order that the cpu issues those manipulations. In particular the ioapic is seeing the ack after we reprogram the ioapic, which causes the ioapic to not accept the ack (because it's vector number changed) and then the ioapic waits forever for an ack that isn't coming. The other half of the problem is pending_eoi() is testing a bit in the hardware that in my experience is not sufficient to guarantee that the hardware is in a state where action may be taken. This is a very delicate problem because there is little if any guaranteed ordering between cpu operations and ioapic operations, especially because there are two channels of communication the ``apic bus'' and the memory mapped ioapic registers. So what happens on one channel has no ordering requirements of any kind with what happens on the other channel. Eric