From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner+w=401wt.eu-S932148AbZHKStU@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S932148AbZHKStU (ORCPT <rfc822;w@1wt.eu>);
	Tue, 11 Aug 2009 14:49:20 -0400
Received: (majordomo@vger.kernel.org) by vger.kernel.org id S932117AbZHKStS
	(ORCPT <rfc822;linux-kernel-outgoing>);
	Tue, 11 Aug 2009 14:49:18 -0400
Received: from mail-ew0-f214.google.com ([209.85.219.214]:42538 "EHLO
	mail-ew0-f214.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1755455AbZHKStP (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Tue, 11 Aug 2009 14:49:15 -0400
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=Ur5fm70n30ayWop0ZoDcjS1NGyOFyYZgSPNWOHM0E6zKnIlTQpPzkF8EIomCUNGNc3
         ynM8Q8hGV7pQhHzUQSLUK89YoPsGaKzvZj+N6zxhsnahKTbQY41zH6Q6k/L5XV7vh/Gx
         thciP06bIU72ibaepMjVv80/CJkS3T6rTSKtI=
MIME-Version: 1.0
In-Reply-To: <20090811184118.GW12579@kernel.dk>
References: <1249588685-4662-1-git-send-email-jens.axboe@oracle.com>
	 <e2e108260908110335tc500412mc6354359d85ae9b6@mail.gmail.com>
	 <20090811143919.GU12579@kernel.dk>
	 <e2e108260908110759w4fbbb5dejf852ba272fa29d75@mail.gmail.com>
	 <20090811171427.GV12579@kernel.dk>
	 <e2e108260908111137k1e0d3418qeaa4cd9775bd0f38@mail.gmail.com>
	 <20090811184118.GW12579@kernel.dk>
Date: Tue, 11 Aug 2009 20:49:16 +0200
Message-ID: <e2e108260908111149j7c51cb57sc651e4ab7a58a2b2@mail.gmail.com>
Subject: Re: [PATCH 0/3]: blk-iopoll, a polled completion API for block 
	devices
From: Bart Van Assche <bart.vanassche@gmail.com>
To: Jens Axboe <jens.axboe@oracle.com>
Cc: linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
       Eric.Moore@lsi.com, jeff@garzik.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Aug 11, 2009 at 8:41 PM, Jens Axboe<jens.axboe@oracle.com> wrote:
> On Tue, Aug 11 2009, Bart Van Assche wrote:
>> On Tue, Aug 11, 2009 at 7:14 PM, Jens Axboe<jens.axboe@oracle.com> wrote:
>> > Did you profile this? Where did it burn all the CPU time on the
>> > initiator side?
>>
>> The test I ran involved a Linux SRP initiator and a Linux SRP target
>> (SCST) using a RAM disk as backstorage. Read throughput is about 1700
>> MB/s for block sizes of 8 MB and above. But with a block size of 4 KB,
>> the read throughput on the initiator drops to 100 MB/s. At this block
>> size there are about 50.000 interrupts per second generated by the
>> InfiniBand HCA in the initiator system. On the same setup the
>> ib_send_bw tool reports a throughput of 1850 MB/s for a block size of
>> 4 KB. This last tool is not interrupt driven but uses polling.
>
> OK, so that looks promising at least. Which hw driver does it use? If I
> look under infiniband/, I see nes, amso, ehca, various ipath and mthca.
> That's where it needs to be hooked up, the srp above mostly looks like
> library helpers and the target hook to the scsi layer.

The above numbers have been obtained on Mellanox ConnectX hardware.
This hardware is controlled by the mlx4_core and mlx4_ib kernel
modules. Source code for these drivers can be found in
drivers/infiniband/hw/mlx4 and drivers/net/mlx4.

Bart.