Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)

["Dong-Jae Kang" <baramsori72@gmail.com>]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 12215 bytes --]

Hi, AndreaThank you for your comments
> Dong-Jae Kang wrote:>> Hi, Andrea>>>> thank you for your contribution to community>> these days, I am testing several IO controllers in container ML,>> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your>> io-throttle(v11)>> Thanks! this is surely a valuable task.>>>>> I have several question about io-throttle>> below is my test reusult of io-throttle(v11) with xdd 6.5>> But, I think I have something wrong, as showed in result>> In direct IO mode, Only read operation was controlled by io-throttle>> Can you check my  test procedure and result and comments to me about that>> Your procedure is correct. Anyway, you found a known bug in io-throttle> v11. If you want to properly use it you need to mount the memory> controller together with blockio, since currently blockio depends on it> to retrieve the owner of a page during writes in submit_bio().>> As reported in:>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>> this is no more than a hack and in perspective a more generic framework> able to provide this functionality should be used (i.e. bio-cgroup).
But, In my opinion,it is some strange that the bandwidth of write operation in direct IOmode was not controlled by io-throttleI think [PATCH -mm 4/6] is for control of buffered IO.Do I misunderstand it?
> I'll fix this issue in the next version of io-throttle (probably I'll> try to rewrite io-throttle on top of bio-cgroup), but for now the> workaround is to mount the cgroupfs using -o blockio,memory (at least).>
Oh, it sounds good.I look foward to your next io-throttle
>>>> additionally, your testing shell script(run_io_throttle_test.sh) for>> io-throttle was not updated for new io-throttle>> so, it could be operated after I fixed it>> The testing of iops limiting is not yet implemented and I don't have a> very good testcase for this, but I can share with you a small script that> I'm using to check if iops limiting is working or not, if you're interested.
thank you, Andrea.I want to check iops limiting of io-throttle is working well.Have a nice day...>>>>> ----------------------------------------------------------------------------------->> - Test System Information>>>> Computer Name, localhost.localdomain, User Name, root>> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008>> Machine hardware type, i686>> Number of processors on this system, 1>> Page size in bytes, 4096>> Number of physical pages, 515885>> Megabytes of physical memory, 2015>> Target[0] Q[0], /dev/sdb>> Per-pass time limit in seconds, 30>> Blocksize in bytes, 512>> Request size, 128, blocks, 65536, bytes>> Number of Requests, 16384>> Number of MegaBytes, 512 or 1024>> Direct I/O, disabled or enable>> Seek pattern, sequential>> Queue Depth, 1>>>> - Test Procedure>>>>      mkdir /dev/blockioctl>>      mount -t cgroup -o blockio cgroup /dev/blockioctl>>      mkdir /dev/blockioctl/cgroup-1>>      mkdir /dev/blockioctl/cgroup-2>>      mkdir /dev/blockioctl/cgroup-3>>      echo /dev/sdb:$((1024*1024)):0:0 >>> /dev/blockioctl/cgroup-1/blockio.bandwidth-max>>      echo /dev/sdb:$((2*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-2/blockio.bandwidth-max>>      echo /dev/sdb:$((3*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-3/blockio.bandwidth-max>>      in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks>>      in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks>>      in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks>>      in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb>> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 )  -timelimit 30>> -verbose –dio(enable or disable)>>>> - setting status information>>>> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max>> 8 16 1048576 0 0 0 13016>> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max>> 8 16 2097152 0 0 0 11763>> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max>> 8 16 3145728 0 0 0 11133>>>> - Test Result>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio -verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      31522816      481    30.005     1.051      16.03    0.0624>>   0.00   read       65536>> 0  1      31522816      481    30.005     1.051      16.03    0.0624>>   0.00   read       65536>> 1  1      31522816      481    30.005     1.051      16.03    0.0624>>   0.00   read       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      62980096      961    30.001     2.099      32.03    0.0312>>   0.00   read       65536>> 0  1      62980096      961    30.001     2.099      32.03    0.0312>>   0.00   read       65536>> 1  1      62980096      961    30.001     2.099      32.03    0.0312>>   0.00   read       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      94437376     1441    30.003     3.148      48.03    0.0208>>   0.00   read       65536>> 0  1      94437376     1441    30.003     3.148      48.03    0.0208>>   0.00   read       65536>> 1  1      94437376     1441    30.003     3.148      48.03    0.0208>>   0.00   read       65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio –verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     640221184     9769    30.097    21.272     324.58    0.0031>>   0.00   write       65536>> 0  1     640221184     9769    30.097    21.272     324.58    0.0031>>   0.00   write       65536>> 1  1     640221184     9769    30.097    21.272     324.58    0.0031>>   0.00   write       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     633798656     9671    30.001    21.126     322.36    0.0031>>   0.00   write       65536>> 0  1     633798656     9671    30.001    21.126     322.36    0.0031>>   0.00   write       65536>> 1  1     633798656     9671    30.001    21.126     322.36    0.0031>>   0.00   write       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     630652928     9623    30.001    21.021     320.76    0.0031>>   0.00   write       65536>> 0  1     630652928     9623    30.001    21.021     320.76    0.0031>>   0.00   write       65536>> 1  1     630652928     9623    30.001    21.021     320.76    0.0031>>   0.00   write       65536>>>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024  -timelimit 30  -verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      70123520     1070    30.150     2.326      35.49    0.0282>>   0.00   read       65536>> 0  1      70123520     1070    30.150     2.326      35.49    0.0282>>   0.00   read       65536>> 1  1      70123520     1070    30.150     2.326      35.49    0.0282>>   0.00   read       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      70844416     1081    30.063     2.357      35.96    0.0278>>   0.00   read       65536>> 0  1      70844416     1081    30.063     2.357      35.96    0.0278>>   0.00   read       65536>> 1  1      70844416     1081    30.063     2.357      35.96    0.0278>>   0.00   read       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      72155136     1101    30.204     2.389      36.45    0.0274>>   0.00   read       65536>> 0  1      72155136     1101    30.204     2.389      36.45    0.0274>>   0.00   read       65536>> 1  1      72155136     1101    30.204     2.389      36.45    0.0274>>   0.00   read       65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024  -timelimit 30  -verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     818610176    12491    30.031    27.258     415.93    0.0024>>   0.00   write       65536>> 0  1     818610176    12491    30.031    27.258     415.93    0.0024>>   0.00   write       65536>> 1  1     818610176    12491    30.031    27.258     415.93    0.0024>>   0.00   write       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     848494592    12947    30.066    28.221     430.62    0.0023>>   0.00   write       65536>> 0  1     848494592    12947    30.066    28.221     430.62    0.0023>>   0.00   write       65536>> 1  1     848494592    12947    30.066    28.221     430.62    0.0023>>   0.00   write       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     786563072    12002    30.078    26.151     399.03    0.0025>>   0.00   write       65536>> 0  1     786563072    12002    30.078    26.151     399.03    0.0025>>   0.00   write       65536>> 1  1     786563072    12002    30.078    26.151     399.03    0.0025>>   0.00   write       65536>>>> Best Regards,>> Dong-Jae Kang>>>>>> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>>> The objective of the i/o controller is to improve i/o performance>>> predictability of different cgroups sharing the same block devices.>>>>>> Respect to other priority/weight-based solutions the approach used by this>>> controller is to explicitly choke applications' requests that directly (or>>> indirectly) generate i/o activity in the system.>>>>>> The direct bandwidth and/or iops limiting method has the advantage of improving>>> the performance predictability at the cost of reducing, in general, the overall>>> performance of the system (in terms of throughput).>>>>>> Detailed informations about design, its goal and usage are described in the>>> documentation.>>>>>> Patchset against 2.6.27-rc5-mm1:>>>>>>  [PATCH 0/6] cgroup: block device i/o controller (v11)>>>  [PATCH 1/6] i/o controller documentation>>>  [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter>>>  [PATCH 3/6] i/o controller infrastructure>>>  [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>>>  [PATCH 5/6] i/o controller instrumentation: accounting and throttling>>>  [PATCH 6/6] export per-task i/o throttling statistics to userspace>>>>>> The all-in-one patch (and previous versions) can be found at:>>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>>>>>> There are no significant changes respect to v10, I've only implemented/fixed>>> some suggestions I received.>>>>>> Changelog: (v10 -> v11)>>>>>> * report per block device i/o statistics (total bytes read/written and iops)>>>  in blockio.stat for i/o limited cgroups>>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and>>>  /proc/PID/io-throttle-stat (suggested by David Radford)>>> * merge res_counter_ratelimit functionality into res_counter, to avoid code>>>  duplication (suggested by Paul Manage)>>> * use kernel-doc style for documenting struct res_counter attributes>>>  (suggested by Randy Dunalp)>>> * udpated documentation>>>>>> Thanks to all for the feedback!>>> -Andrea>


-- -------------------------------------------------------------------------------------------------   DONG-JAE, KANG   Senior Member of Engineering Staff   Internet Platform Research Dept, S/W Content Research Lab   Electronics and Telecommunications Research Institute(ETRI)   138 Gajeongno, Yuseong-gu, Daejeon, 305-700 KOREA   Phone : 82-42-860-1561 Fax : 82-42-860-6699   Mobile : 82-10-9919-2353 E-mail : djkang@etri.re.kr (MSN)-------------------------------------------------------------------------------------------------ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥