Re: [PATCH] queue barrier support

Re: [PATCH] queue barrier support

[James Bottomley]

axboe@suse.de said:
> ChangeSet@1.297, 2002-02-13 13:42:39+01:00, axboe@burns.home.kernel.dk
>   Add support for SCSI drivers to indicate support for ordered tags
>   http://bitmover.com:8888//tmp/v2_logging/athlon.transmeta.com/
> torvalds-2002020517305 \ 6-16047-c1d11a41ed024864/cset@1.133.114.4?nav=
> index.html|ChangeSet@-1h

> ChangeSet@1.298, 2002-02-13 13:43:04+01:00, axboe@burns.home.kernel.dk
>   Add ordered tag support to the aic7xxx scsi driver

The rest of the aic7xxx code uses MSG_ORDERED_TASK rather than 
MSG_ORDERED_Q_TAG.  You have to scan through the headers to see that these are 
#defined the same.

A problem (that is probably only an issue for older drives) is that while 
technically the standard requires all 3 types of TAG to be supported if tag 
queueing is, some drives really only have simple tag support in their 
firmware, so you may need to add a blacklist for ordered tags on certain 
drives.

A further issue is that you haven't added anything to the error recovery code 
for this.  If error recovery is activated for the device at the reset level, 
all tags will be discarded by the device.  The eh will retry the failing 
command and then the other tagged commands will be re-issued from the 
scsi_bottom_half_handler (assuming the low level device driver immediately 
fails them with DID_RESET) in the order in which the low level driver failed 
them.  Thus you have potentially completely messed up the ordering when the 
commands all get retried.

James Bottomley

Re: [PATCH] queue barrier support

[Jens Axboe]

On Wed, Feb 13 2002, James Bottomley wrote:
> axboe@suse.de said:
> > ChangeSet@1.297, 2002-02-13 13:42:39+01:00, axboe@burns.home.kernel.dk
> >   Add support for SCSI drivers to indicate support for ordered tags
> >   http://bitmover.com:8888//tmp/v2_logging/athlon.transmeta.com/
> > torvalds-2002020517305 \ 6-16047-c1d11a41ed024864/cset@1.133.114.4?nav=
> > index.html|ChangeSet@-1h
> 
> > ChangeSet@1.298, 2002-02-13 13:43:04+01:00, axboe@burns.home.kernel.dk
> >   Add ordered tag support to the aic7xxx scsi driver
> 
> The rest of the aic7xxx code uses MSG_ORDERED_TASK rather than 
> MSG_ORDERED_Q_TAG.  You have to scan through the headers to see that
> these are #defined the same.

Ah, the MSG_ORDERED_TASK did probably not exist when I originally did
the patch (a year or so ago, iirc). I'll change the define to follow the
rest of the code.

> A problem (that is probably only an issue for older drives) is that
> while technically the standard requires all 3 types of TAG to be
> supported if tag queueing is, some drives really only have simple tag
> support in their firmware, so you may need to add a blacklist for
> ordered tags on certain drives.

Yes you are right.

> A further issue is that you haven't added anything to the error
> recovery code for this.  If error recovery is activated for the device
> at the reset level, all tags will be discarded by the device.  The eh
> will retry the failing command and then the other tagged commands will
> be re-issued from the scsi_bottom_half_handler (assuming the low level
> device driver immediately fails them with DID_RESET) in the order in
> which the low level driver failed them.  Thus you have potentially
> completely messed up the ordering when the commands all get retried.

I already have this fixed locally by just maintaining a fifo list of
queued commands so we can retry them in the correct order. For 2.5
there's a busy and free list (so no more scanning for a free command
either), I would imagine that the easiest for 2.4 is to just maintain a
busy list in addition to the current array of pending/free commands.

-- 
Jens Axboe

Re: [PATCH] queue barrier support

[James Bottomley]

James.Bottomley@steeleye.com said:
> A further issue is that you haven't added anything to the error
> recovery code for this.  If error recovery is activated for the device
> at the reset level, all tags will be discarded by the device.  The eh
> will retry the failing command and then the other tagged commands will
> be re-issued from the scsi_bottom_half_handler (assuming the low level
> device driver immediately fails them with DID_RESET) in the order in
> which the low level driver failed them.  Thus you have potentially
> completely messed up the ordering when the commands all get retried.

axboe@suse.de said:
> I already have this fixed locally by just maintaining a fifo list of
> queued commands so we can retry them in the correct order. For 2.5
> there's a busy and free list (so no more scanning for a free command
> either), I would imagine that the easiest for 2.4 is to just maintain
> a busy list in addition to the current array of pending/free commands.

mason@suse.com said:
> I was wondering about this, we would need to change the error handler
> to  fail all the requests after the barrier.  I was hoping the driver
> did this for us ;-) 

Unfortunately, this is going to involve deep hackery inside the error handler. 
 The current initial premise is that it can simply retry the failing command 
by issuing an ABORT to the tag and resending it (which can cause a tag to move 
past your barrier).  In an error situation, it really wouldn't be wise to try 
to abort lots of potentially running tags to preserve the barrier ordering 
(because of the overload placed on a known failing component), so I think the 
error handler has to abandon the concept of aborting commands and move 
straight to device reset.  We then carefully resend the commands in FIFO order.

Additionally, you must handle the case that a device is reset by something 
else (in error handler terms, the cc_ua [check condition/unit attention]).  
Here also, the tags would have to be sent back down in FIFO order as soon as 
the condition is detected.

mason@suse.com said:
> Yes, this could get sticky.  Does anyone know if other OSes have
> already done this? 

Other OSs (well the ones I've heard about: Solaris and HP-UX) try to avoid 
ordered tags, mainly because of the performance impact they have---the drive 
tag service algorithms become inefficient in the presence of ordered tags 
since they're usually optimised for all simple tags.

James

Re: [PATCH] queue barrier support

[Chris Mason]

On Friday, February 15, 2002 10:15:58 AM -0500 James Bottomley <James.Bottomley@steeleye.com> wrote:

> mason@suse.com said:
>> I was wondering about this, we would need to change the error handler
>> to  fail all the requests after the barrier.  I was hoping the driver
>> did this for us ;-) 
> 
> Unfortunately, this is going to involve deep hackery inside the error handler. 
>  The current initial premise is that it can simply retry the failing command 
> by issuing an ABORT to the tag and resending it (which can cause a tag to move 
> past your barrier).  In an error situation, it really wouldn't be wise to try 
> to abort lots of potentially running tags to preserve the barrier ordering 
> (because of the overload placed on a known failing component), so I think the 
> error handler has to abandon the concept of aborting commands and move 
> straight to device reset.  We then carefully resend the commands in FIFO order.
> 

Ok, I'll try to narrow the barrier usage a bit, I'm waiting on the
barrier write once it is sent, so I'm not worried about anything done
after the ordered tag.

write X log blocks   (simple tag)
write 1 commit block (ordered tag)
wait on all of them.

All I care about is knowing that all of the log blocks hit the disk
before the commit.  So, if one of the log blocks aborts, I want it
to abort the commit too.  Is this a little easier to implement?

> Additionally, you must handle the case that a device is reset by something 
> else (in error handler terms, the cc_ua [check condition/unit attention]).  
> Here also, the tags would have to be sent back down in FIFO order as soon as 
> the condition is detected.
> 
> mason@suse.com said:
>> Yes, this could get sticky.  Does anyone know if other OSes have
>> already done this? 
> 
> Other OSs (well the ones I've heard about: Solaris and HP-UX) try to avoid 
> ordered tags, mainly because of the performance impact they have---the drive 
> tag service algorithms become inefficient in the presence of ordered tags 
> since they're usually optimised for all simple tags.

I do see that on my drives ;-)  The main reason I think its worth trying
is because the commit block is directly adjacent to the last log block,
so I'm hoping the drive can optimize the commit (even though it is ordered)
better when the OS sends it directly after the last log block.

While I've got linux-scsi cc'd, I'll reask a question from yesterday.
Do the targets with write back caches usually ignore the order tag, 
doing the write in the most efficient way possible instead?

-chris

Re: [PATCH] queue barrier support

[James Bottomley]

mason@suse.com said:
> While I've got linux-scsi cc'd, I'll reask a question from yesterday.
> Do the targets with write back caches usually ignore the order tag,
> doing the write in the most efficient way possible instead? 

I'll try to answer, although I'm not quite sure of the basis of the question.

No ordinary SCSI drive that's not on a battery backed circuit should *ever* 
use writeback caching.  They should all come by default as write through.  In 
this case, the tag is acknowleged as completed only when the write has made it 
to the medium.

If you alter the drive parameter page to turn on write back caching, it's 
caveat emptor.  Since you're now telling the drive that you consider hitting 
the cache to be equivalent to hitting the medium (because you know better 
about power failures and the like) it is undefined by the standards how the 
drives write from the cache to the medium---and you shouldn't care about this 
if you have arranged your system to do write back caching.  They will 
acknowlege the tag as completed as soon as it hits the cache, and the ordered 
tag will be order it commits to the cache.

Note, for high end disk arrays, the reverse is usually true since they often 
have internal battery backups for their caches and drives.

If you lose power to the drive that does write back caching before the cache 
flushes, all the indications you've given the journalling fs about write 
commits are voided and the fs is probably inconsistent and not recoverable by 
a journal replay.

Note also that on system shutdown, most devices that use write back caching 
are also expecting a cache flush instruction from the node, which Linux 
doesn't send.

James

Re: [PATCH] queue barrier support

[Chris Mason]

On Friday, February 15, 2002 11:51:45 AM -0500 James Bottomley <James.Bottomley@steeleye.com> wrote:

> mason@suse.com said:
>> While I've got linux-scsi cc'd, I'll reask a question from yesterday.
>> Do the targets with write back caches usually ignore the order tag,
>> doing the write in the most efficient way possible instead? 
> 
> I'll try to answer, although I'm not quite sure of the basis of the question.
> 
> No ordinary SCSI drive that's not on a battery backed circuit should *ever* 
> use writeback caching.  They should all come by default as write through.  In 
> this case, the tag is acknowleged as completed only when the write has made it 
> to the medium.

Right, friends don't let friends use non-battery back write back caches ;-)

> 
> If you alter the drive parameter page to turn on write back caching, it's 
> caveat emptor.  Since you're now telling the drive that you consider hitting 
> the cache to be equivalent to hitting the medium (because you know better 
> about power failures and the like) it is undefined by the standards how the 
> drives write from the cache to the medium---and you shouldn't care about this 
> if you have arranged your system to do write back caching.  They will 
> acknowlege the tag as completed as soon as it hits the cache, and the ordered 
> tag will be order it commits to the cache.

Ok, this is what I was expecting.  The performance improvement from
using the ordered tags on single drives doesn't seem to be very big, and
given the problems with failed requests, I'm thinking about dropping the
scsi parts of the 2.4 barrier patch, and just worrying about making ide 
drives flush things correctly.  The hard stuff on error recovery can
be tackled in 2.5 and (maybe) ported back later.

But, if high end controllers with write back caching see improvements
because they can combine the log writes better with the patch, it might
be worth keeping, but modified in reiserfs so it flushes IDE caches
by default, but only sends ordered tags when the admin mounts 
with -o barrier_ordered or something.

thanks for the info
-chris

Re: [PATCH] queue barrier support

[James Bottomley]

mason@suse.com said:
> I'm thinking about dropping the scsi parts of the 2.4 barrier patch,
> and just worrying about making ide  drives flush things correctly.
> The hard stuff on error recovery can be tackled in 2.5 and (maybe)
> ported back later.

That's probably best.  I do agree that 2.5 is the place to play around with 
SCSI error handling first.

I'm willing to help re-do the error handler, since I've always thought that 
abort isn't a good first line of defence because it actually adds to the 
command burden of a failing drive.

Jens, if you want to share the code you already have (or point me to the 
bitkeeper repository where you keep it) I'll look it over.

As far as the back-port to 2.4, as long as you only support SCSI drivers that 
use the new error handler (so we don't have to worry about the obsolete one) 
that should be reasonably easy.

James

Re: [PATCH] queue barrier support

[Matthias Andree]

On Fri, 15 Feb 2002, James Bottomley wrote:

> Note also that on system shutdown, most devices that use write back caching 
> are also expecting a cache flush instruction from the node, which Linux 
> doesn't send.

Hair splitting: it looks as though Andre Hedrick's IDE patch did this at
least. Of course, that does not affect SCSI drives, but since you wrote
"Linux", I thought this could be mentioned in this thread again. At
least, my ATA drives are powered down some seconds before my ATX PC is,
and the kernel says about flushing the cache.

-- 
Matthias Andree

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."         Benjamin Franklin

Re: [PATCH] queue barrier support

[James Bottomley]

mason@suse.com said:
> Ok, I'll try to narrow the barrier usage a bit, I'm waiting on the
> barrier write once it is sent, so I'm not worried about anything done
> after the ordered tag.

> write X log blocks   (simple tag) write 1 commit block (ordered tag)
> wait on all of them.

> All I care about is knowing that all of the log blocks hit the disk
> before the commit.  So, if one of the log blocks aborts, I want it to
> abort the commit too.  Is this a little easier to implement? 

Actually, I'm afraid not.  The FS has the notion of a transaction to help it 
with this.  All the SCSI driver sees is a list of IOs with some requests for 
ordered tags.  If we go into error recovery and have to abort a tag, how do we 
know which of the outstanding ordered tags is actually the commit block for 
this transaction? (Remember also that a FS sits on a partition, not a physical 
SCSI device, so even if you single thread your commits per FS, there could 
still be multiple FSs on the same physical device).

That's why I think it's safer to abandon the abort entirely.  If our first 
line of error recovery is device reset, we know we've just cleared the entire 
outstanding tag queue and we can resend all the tags in FIFO order which means 
that we still don't need a concept of "transaction" at the SCSI level (which 
is a good thing, I think), all we need to know is the order in which the tags 
came to us.

Even if some tags were committed but not acknowledged at reset time, it's 
still safe to resend them in the correct ordered sequence because of 
transaction idempotence of block writes.

James

Re: [PATCH] queue barrier support

[Mike Anderson]

James Bottomley [James.Bottomley@SteelEye.com] wrote:
> James.Bottomley@steeleye.com said:
> 
> Unfortunately, this is going to involve deep hackery inside the error handler. 
>  The current initial premise is that it can simply retry the failing command 
> by issuing an ABORT to the tag and resending it (which can cause a tag to move 
> past your barrier).  In an error situation, it really wouldn't be wise to try 
> to abort lots of potentially running tags to preserve the barrier ordering 
> (because of the overload placed on a known failing component), so I think the 
> error handler has to abandon the concept of aborting commands and move 
> straight to device reset.  We then carefully resend the commands in FIFO order.
> 
> Additionally, you must handle the case that a device is reset by something 
> else (in error handler terms, the cc_ua [check condition/unit attention]).  
> Here also, the tags would have to be sent back down in FIFO order as soon as 
> the condition is detected.

I agree on the hacker. You also will need to clean the pipe of the unit
attention post the reset or the first command you send down will
be coming right back into the error handler and could get you in to a
error recovery storm.


> 
> James
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Michael Anderson
andmike@us.ibm.com

Re: [PATCH] queue barrier support

[Chris Mason]

On Wednesday, February 13, 2002 01:26:12 PM -0500 James Bottomley <James.Bottomley@steeleye.com> wrote:

> axboe@suse.de said:
>> ChangeSet@1.297, 2002-02-13 13:42:39+01:00, axboe@burns.home.kernel.dk
>>   Add support for SCSI drivers to indicate support for ordered tags
>>   http://bitmover.com:8888//tmp/v2_logging/athlon.transmeta.com/
>> torvalds-2002020517305 \ 6-16047-c1d11a41ed024864/cset@1.133.114.4?nav=
>> index.html|ChangeSet@-1h
> 
>> ChangeSet@1.298, 2002-02-13 13:43:04+01:00, axboe@burns.home.kernel.dk
>>   Add ordered tag support to the aic7xxx scsi driver
> 
> The rest of the aic7xxx code uses MSG_ORDERED_TASK rather than 
> MSG_ORDERED_Q_TAG.  You have to scan through the headers to see that these are 
># defined the same.

Jens, my patch from yesterday has this fixed.

> 
> A problem (that is probably only an issue for older drives) is that while 
> technically the standard requires all 3 types of TAG to be supported if tag 
> queueing is, some drives really only have simple tag support in their 
> firmware, so you may need to add a blacklist for ordered tags on certain 
> drives.

Yes, this could get sticky.  Does anyone know if other OSes have already
done this?

> 
> A further issue is that you haven't added anything to the error recovery code 
> for this.  If error recovery is activated for the device at the reset level, 
> all tags will be discarded by the device.  The eh will retry the failing 
> command and then the other tagged commands will be re-issued from the 
> scsi_bottom_half_handler (assuming the low level device driver immediately 
> fails them with DID_RESET) in the order in which the low level driver failed 
> them.  Thus you have potentially completely messed up the ordering when the 
> commands all get retried.

I was wondering about this, we would need to change the error handler to 
fail all the requests after the barrier.  I was hoping the driver
did this for us ;-)

-chris

Re: [PATCH] queue barrier support

[Daniel Phillips]

On February 13, 2002 07:26 pm, James Bottomley wrote:
> A problem (that is probably only an issue for older drives) is that while 
> technically the standard requires all 3 types of TAG to be supported if tag 
> queueing is, some drives really only have simple tag support in their 
> firmware, so you may need to add a blacklist for ordered tags on certain 
> drives.

>From user space, I hope.

-- 
Daniel

Re: [PATCH] queue barrier support

[James Bottomley]

On February 13, 2002 07:26 pm, James Bottomley wrote:
> A problem (that is probably only an issue for older drives) is that while 
> technically the standard requires all 3 types of TAG to be supported if tag 
> queueing is, some drives really only have simple tag support in their 
> firmware, so you may need to add a blacklist for ordered tags on certain 
> drives.

phillips@bonn-fries.net said:
> From user space, I hope. 

Well, following the current SCSI black/white list procedure, they would be in 
the static device_list table in scsi_scan.c, so no.

I agree that it's not a nice or friendly thing (and is certainly prone to 
delays when you have to get entries into the actual kernel code), but fixing 
this particular annoyance of the scsi subsystem (although I have actually 
thought about doing it) would be a project separate from the queue barrier 
support.

James

929-Emulex, ABTS Command Anamoly!

[Cindy Sweet]

Is this the right group to discuss about the ABTS
command ??

__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: 929-Emulex, ABTS Command Anamoly!

[arjan]

In article <20020225205527.76533.qmail@web20908.mail.yahoo.com> you wrote:
> Is this the right group to discuss about the ABTS
> command ??

no.

emulex is a binary only driver so it should be discussed with emulex
if you have problems...