From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 422B0C282D9 for ; Thu, 31 Jan 2019 20:58:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1AC1720B1F for ; Thu, 31 Jan 2019 20:58:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729008AbfAaU6U (ORCPT ); Thu, 31 Jan 2019 15:58:20 -0500 Received: from mga12.intel.com ([192.55.52.136]:24553 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726802AbfAaU6U (ORCPT ); Thu, 31 Jan 2019 15:58:20 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 31 Jan 2019 12:58:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,545,1539673200"; d="scan'208";a="315311785" Received: from djiang5-mobl2.amr.corp.intel.com (HELO [10.255.72.219]) ([10.255.72.219]) by fmsmga006.fm.intel.com with ESMTP; 31 Jan 2019 12:58:19 -0800 Subject: Re: [PATCH 0/9] Support using MSI interrupts in ntb_transport To: Logan Gunthorpe , linux-kernel@vger.kernel.org, linux-ntb@googlegroups.com, linux-pci@vger.kernel.org, iommu@lists.linux-foundation.org, linux-kselftest@vger.kernel.org, Jon Mason , Bjorn Helgaas , Joerg Roedel Cc: Allen Hubbe , Serge Semin , Eric Pilmore References: <20190131185656.17972-1-logang@deltatee.com> <345197a6-89e6-c0de-5f7b-a646b5f396c9@intel.com> From: Dave Jiang Message-ID: Date: Thu, 31 Jan 2019 13:58:18 -0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:65.0) Gecko/20100101 Thunderbird/65.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 1/31/2019 1:48 PM, Logan Gunthorpe wrote: > > On 2019-01-31 1:20 p.m., Dave Jiang wrote: >> Does this work when the system moves the MSI vector either via software >> (irqbalance) or BIOS APIC programming (some modes cause round robin >> behavior)? > > I don't know how irqbalance works, and I'm not sure what you are > referring to by BIOS APIC programming, however I would expect these > things would not be a problem. > > The MSI code I'm presenting here doesn't do anything crazy with the > interrupts, it allocates and uses them just as any PCI driver would. The > only real difference here is that instead of a piece of hardware sending > the IRQ TLP, it will be sent through the memory window (which, from the > OS's perspective, is just coming from an NTB hardware proxy alias). > > Logan Right. I did that as a hack a while back for some silicon errata workaround. When the vector moves, the address for the LAPIC changes. So unless it gets updated, you end up writing to the old location and lose all the new interrupts. irqbalance is a user daemon that rotates the system interrupts around to ensure that not all interrupts are pinned on a single core. I think it's enabled by default on several distros. Although MSIX has nothing to do with the IOAPIC, the mode that the APIC is programmed can have an influence on how the interrupts are delivered. There are certain Intel platforms (I don't know if AMD does anything like that) puts the IOAPIC in a certain configuration that causes the interrupts to be moved in a round robin fashion. I think it's physical flat mode? I don't quite recall. Normally on the low end Xeons. It's probably worth doing a test run with the irqbalance daemon running and make sure you traffic stream doesn't all of sudden stop. >