Re: queue_nr_requests needs to be selective

["Jeff V. Merkey" <jmerkey@vger.timpanogas.org>]

:-)

They're great cards.

Jeff


On Sun, Mar 03, 2002 at 11:16:34PM -0800, Mike Anderson wrote:
> Thanks Jeff for the data. I went back and re-read the code. 
> 
> It looks like the template cmd_per_lun value is recalculated before the
> scsi-register call is made by the driver. The number is 255 / number of
> units (which you said from your previous post is one). This then matches
> your proc info. From the numbers these look like pretty nice cards :-). 
> 
> -Mike
> 
> Jeff V. Merkey [jmerkey@vger.timpanogas.org] wrote:
> > 
> > 
> > Mike,
> > 
> > Here are some numbers from the running system.  This system is 
> > running at 120 MB/S on a single 3Ware adapter.  Stats attached.  You
> > will note that the max commands hitting the adapter are way above
> > 15.  I can only presume this is due to caching behavior on the card.
> > I do have these cards enabled with caching.  I have had these numbers
> > as high as 319 MB/S with multiple cards in separate buses.  The system
> > this test is running on has 3 PCI buses.  2 x 33 Mhz and 1 x 66 Mhz.  
> > with Serverwork HE Chipset.
> > 
> > We are averaging complete saturation of the inbound command queue
> > on the 3Ware adapter and are posting 64K writes (128 * 512) * 3 on
> > aeverage with each command.  I have no idea why the 15 queue depth
> > limit is being exceeded here, possibly due to the fact the adapter 
> > is caching, but the stats report up to 255 requests being posted 
> > per driver pass.
> > 
> > cat /proc/scsi/3w-xxxx/3
> > 
> > scsi3: 3ware Storage Controller
> > Driver version: 1.02.00.016
> > Current commands posted:       205
> > Max commands posted:           255
> > Current pending commands:        0
> > Max pending commands:            0
> > Last sgl length:                 0
> > Max sgl length:                  3
> > Last sector count:             128
> > Max sector count:              256
> > Resets:                          0
> > Aborts:                          0
> > AEN's:                           0
> > 
> > 
> > cat /proc/scsi/3w-xxxx/3
> > 
> > scsi3: 3ware Storage Controller
> > Driver version: 1.02.00.016
> > Current commands posted:         7
> > Max commands posted:           255
> > Current pending commands:        0
> > Max pending commands:            0
> > Last sgl length:                 0
> > Max sgl length:                  3
> > Last sector count:             128
> > Max sector count:              256
> > Resets:                          0
> > Aborts:                          0
> > AEN's:                           0
> > 
> > 
> > This test is posting a max of 54,000 4K buffer heads to a single 3Ware
> > adapter.  See below:
> > 
> > 
> > trxinfo -a  
> > 
> > ioctl TRXDRV_QUERY_AIO ret 0
> > TRXDRV-AIO STATS
> > aio_submitted-3 aio_completed-25601331 aio_error-0
> > disk_aio_submitted-25601539 disk_aio_completed-25601331
> > sync_active [ x x x x x x x x ]
> > async_active [ x x x x x x x x ]
> > cb_active-0 aio_sequence-0
> > bh_count-65536 bh_inuse-32208 bh_max_inuse-54593 bh_waiters-0
> > hash_hits-0 hash_misses-0 hash_fill-3 hash_total-3
> > probe_avg-0 probe_max-1
> > total_read_req-0 total_write_req-25601536 total_fill_req-0
> > total_complete-25739847
> > req_sec-0 seconds-12925
> > 
> > trxinfo -a  
> > 
> > ioctl TRXDRV_QUERY_AIO ret 0
> > TRXDRV-AIO STATS
> > aio_submitted-3 aio_completed-25605732 aio_error-0
> > disk_aio_submitted-25605891 disk_aio_completed-25605732
> > sync_active [ x x x x x x x x ]
> > async_active [ x x x x x x x x ]
> > cb_active-0 aio_sequence-0
> > bh_count-65536 bh_inuse-31440 bh_max_inuse-54593 bh_waiters-0
> > hash_hits-0 hash_misses-0 hash_fill-3 hash_total-3
> > probe_avg-0 probe_max-1
> > total_read_req-0 total_write_req-25605888 total_fill_req-0
> > total_complete-25744271
> > req_sec-0 seconds-12927
> > 
> > This test is posting a max of 54,000 4K buffer heads to a single 3Ware
> > adapter.
> > 
> > 
> > Jeff
> > 
> > 
> > 
> > 
> > On Fri, Mar 01, 2002 at 09:39:08PM -0700, Jeff V. Merkey wrote:
> > > 
> > > 
> > > We are going to sleep a lot in __get_request_wait().  This 
> > > means the write queue has no free request blocks.  We are mostly writing 
> > > to the adapter in this test case, and the data we are writing 
> > > is already in order when it's posted.    
> > > 
> > > We are also posting via submit_bh() so you should trace 
> > > that path.  I am seeing 22,000+ buffer heads posted concurrently
> > > on each 3Ware card of 4K each with this application
> > > on the patch for 2.4.19-pre2.  I will post the actual data
> > > for you.  Stand by. 
> > > 
> > > These 3Ware cards are incredible. 
> > > 
> > > Jeff
> > > 
> > > 
> > > On Fri, Mar 01, 2002 at 04:51:04PM -0800, Mike Anderson wrote:
> > > > Jeff V. Merkey [jmerkey@vger.timpanogas.org] wrote:
> > > > > 
> > > > > ..snip..
> > > > >
> > > > > What is really needed here is to allow queue_nr_requests to be 
> > > > > configurable on a per adapter/device basis for these high end 
> > > > > raid cards like 3Ware since in a RAID 0 configuration, 8 drives
> > > > > are in essence a terabyte (1.3 terrabytes in our configuration) 
> > > > > and each adapter is showing up as a 1.3 TB device.  64/128
> > > > > requests are simply not enough to get the full spectrum of 
> > > > > performance atainable with these cards.
> > > > > 
> > > > Not having direct experience on this card it appears that increasing the
> > > > queue_nr_requests number will not allow you to have more ios in flight.
> > > > 
> > > > Unless I am reading the driver wrong you will be limited to
> > > > TW_MAX_CMDS_PER_LUN (15). This value is used by scsi_build_commandblocks
> > > > to allocate scsi commands for your scsi_device. This driver does not provide
> > > > a select_queue_depths function which allows for increase to the default
> > > > template value. 
> > > > 
> > > > Could it be that the experimentation of increasing this number has
> > > > allowed for better merging.
> > > > 
> > > > -Mike
> > > > -- 
> > > > Michael Anderson
> > > > andmike@us.ibm.com
> > > > 
> > > > -
> > > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > Please read the FAQ at  http://www.tux.org/lkml/
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 
> -- 
> Michael Anderson
> andmike@us.ibm.com