From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752601AbbGORTj (ORCPT ); Wed, 15 Jul 2015 13:19:39 -0400 Received: from mga14.intel.com ([192.55.52.115]:20462 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751940AbbGORTh (ORCPT ); Wed, 15 Jul 2015 13:19:37 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.15,481,1432623600"; d="scan'208";a="606825327" Date: Wed, 15 Jul 2015 17:19:33 +0000 (UTC) From: Keith Busch X-X-Sender: vmware@localhost.lm.intel.com To: Bart Van Assche cc: Thomas Gleixner , Christoph Hellwig , linux-rdma@vger.kernel.org, Jens Axboe , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, ksummit-discuss@lists.linuxfoundation.org Subject: Re: [Ksummit-discuss] [TECH TOPIC] IRQ affinity In-Reply-To: <55A67F11.1030709@sandisk.com> Message-ID: References: <20150715120708.GA24534@infradead.org> <55A67F11.1030709@sandisk.com> User-Agent: Alpine 2.00 (LNX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 15 Jul 2015, Bart Van Assche wrote: > * With blk-mq and scsi-mq optimal performance can only be achieved if > the relationship between MSI-X vector and NUMA node does not change > over time. This is necessary to allow a blk-mq/scsi-mq driver to > ensure that interrupts are processed on the same NUMA node as the > node on which the data structures for a communication channel have > been allocated. However, today there is no API that allows > blk-mq/scsi-mq drivers and irqbalanced to exchange information > about the relationship between MSI-X vector ranges and NUMA nodes. We could have low-level drivers provide blk-mq the controller's irq associated with a particular h/w context, and the block layer can provide the context's cpumask to irqbalance with the smp affinity hint. The nvme driver already uses the hwctx cpumask to set hints, but this doesn't seems like it should be a driver responsibility. It currently doesn't work correctly anyway with hot-cpu since blk-mq could rebalance the h/w contexts without syncing with the low-level driver. If we can add this to blk-mq, one additional case to consider is if the same interrupt vector is used with multiple h/w contexts. Blk-mq's cpu assignment needs to be aware of this to prevent sharing a vector across NUMA nodes. > The only approach I know of that works today to define IRQ affinity > for blk-mq/scsi-mq drivers is to disable irqbalanced and to run a > custom script that defines IRQ affinity (see e.g. the > spread-mlx4-ib-interrupts attachment of > http://thread.gmane.org/gmane.linux.kernel.device-mapper.devel/21312/focus=98409).