Re: SSD read latency negatively impacted by large writes (independent of choice of I/O scheduler)

[Stefan Richter <stefanr@s5r6.in-berlin.de>]

Jeff Moyer wrote:
> Zubin Dittia <zubin@tintri.com> writes:
[...]
>> about 30 seconds after the write (which is when the write is
>> actually written back to the device from buffer cache), I see a very
>> large spike in read latency: from 200 microseconds to 25 milliseconds.
>>  This seems to imply that the writes issued by the scheduler are not
>> being broken up into sufficiently small chunks with interspersed
>> reads; instead, the whole sequential write seems to be getting issued
>> while starving reads during that period.
[...]
>> Playing around with different I/O
>> schedulers and parameters doesn't seem to help at all.
[...]
> I haven't verified your findings, but if what you state is true, then
> you could try tuning max_sectors_kb for your device.  Making that
> smaller will decrease the total amount of I/O that can be queued in the
> device at any given time.  There's always a trade-off between bandwidth
> and latency, of course.

Maximum transfer size per request is indeed one factor; another one is
queue_depth.  With a deep queue, a read request between many write
requests will still be held up by many write requests queued up before
the read request.  (Once the scheduler issued the requests to the queue,
it can't reorder the requests any more --- only the disk's firmware
could reorder the requests if it is sophisticated enough and there are
no barriers in the mix.)

When transfer size and queue depth are set different from the default,
the various I/O schedulers should be tested again because then their
behaviors may vary more than before.
-- 
Stefan Richter
-=====-==--= =-== ---=-
http://arcgraph.de/sr/