From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752177AbdLACJH (ORCPT ); Thu, 30 Nov 2017 21:09:07 -0500 Received: from mail.kernel.org ([198.145.29.99]:56602 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752035AbdLACJF (ORCPT ); Thu, 30 Nov 2017 21:09:05 -0500 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 58F3521890 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=goodmis.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=rostedt@goodmis.org Date: Thu, 30 Nov 2017 21:08:57 -0500 From: Steven Rostedt To: David Su Cc: linux-kernel@vger.kernel.org, bigeasy@linutronix.de, tglx@linutronix.de Subject: Re: [PATCH RT] vfio-pci: Set MSI/MSI-X ISR to non-threaded Message-ID: <20171130210857.53b81ba3@gandalf.local.home> In-Reply-To: <1512090335-6888-1-git-send-email-david.w.su@intel.com> References: <1512090335-6888-1-git-send-email-david.w.su@intel.com> X-Mailer: Claws Mail 3.14.0 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 30 Nov 2017 17:05:35 -0800 David Su wrote: > Setting MSI/MSI-X ISR to be non-threaded will result in shorter and more > deterministic IRQ delivery latencies to VFIO applications, because > context switches to the ISR thread are eliminated. This is important > for applications with low latency requirement running in virtual > machines on RT Linux host with assigned devices through vfio-pci. > > A FPGA based interrupt testing device was used to compare latencies with > threaded and non-threaded vfio-pci ISR. The device has a free running > time stamp counter and a register recording the time an interrupt was > sent to the host. With these registers the device driver and test > application for the device are able to calculate and record the latency > between the time an interrupt was sent and the time the ISR in the > device's driver was invoked. > > The result is with non-threaded vfio-pci ISR the average latency is > reduced by about 54% and the maximum-minimum latency range is reduced by > about 65%. > > Non-threaded vfio-pci ISR: > Minimum 4.18us, Average 4.47us, Maximum 10.26us > > Threaded vfio-pci ISR: > Minimum 8.97us, Average 9.65us, Maximum 26.11us > > Signed-off-by: David Su > --- > drivers/vfio/pci/vfio_pci_intrs.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c > index 1c46045..4c54e56 100644 > --- a/drivers/vfio/pci/vfio_pci_intrs.c > +++ b/drivers/vfio/pci/vfio_pci_intrs.c > @@ -333,7 +333,7 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_device *vdev, > pci_write_msi_msg(irq, &msg); > } > > - ret = request_irq(irq, vfio_msihandler, 0, > + ret = request_irq(irq, vfio_msihandler, IRQF_NO_THREAD, > vdev->ctx[vector].name, trigger); Hmm, but we have this: static irqreturn_t vfio_msihandler(int irq, void *arg) { struct eventfd_ctx *trigger = arg; eventfd_signal(trigger, 1); return IRQ_HANDLED; } __u64 eventfd_signal(struct eventfd_ctx *ctx, __u64 n) { unsigned long flags; spin_lock_irqsave(&ctx->wqh.lock, flags); if (ULLONG_MAX - ctx->count < n) n = ULLONG_MAX - ctx->count; ctx->count += n; if (waitqueue_active(&ctx->wqh)) wake_up_locked_poll(&ctx->wqh, POLLIN); spin_unlock_irqrestore(&ctx->wqh.lock, flags); return n; } And spin_lock() turns into a mutex in PREEMPT_RT, which means it can sleep. You can't sleep in hard interrupt context. This will eventually crash the kernel. And no, we are not going to convert the ctx->wqh.lock into a raw_spin_lock. -- Steve > if (ret) { > kfree(vdev->ctx[vector].name);