[PATCH 0/2] range-bw : Another I/O scheduling policy of dm-ioband (v1)

[PATCH 0/2] range-bw : Another I/O scheduling policy of dm-ioband (v1)

[Dong-Jae Kang]

Hi, all

* Overview
----------------
range-bw was implemented as another I/O scheduling policy of dm-ioband
to support predicable I/O bandwidth between minimum and maximum
bandwidth defined by administrator. So, basic advantages and defects
are same with dm-ioband. Here, minimum I/O bandwidth should be
guaranteed for stable performance or reliability of specific process
groups and I/O bandwidth over maximum should be throttled to protect
the limited I/O resource from over-provisioning in unnecessary usage
or to reserve the I/O bandwidth for another use.

range-bw has two kinds of operation modes, min-max and max mode.
Min-max mode is to supports guaranteeing the minimum I/O requirement
and limitation of unnecessary I/O bandwidth at the same time. And max
mode is to support only limitation. So in case of min-max mode, you
need to configure min-bw and max-bw values and in case of max mode,
configure only max-bw.
So, range-bw was implemented to include the two concepts, guaranteeing
of minimum bandwidth and limitation of maximum bandwidth according to
the importance or priority of specific process groups.

* Attention
-----------------
Range-BW supports the predicable I/O bandwidth, but it should be
configured in the scope of total I/O bandwidth of the I/O system to
guarantee the minimum I/O requirement. For example, if total I/O
bandwidth is 40Mbytes/sec, the summary of I/O bandwidth configured in
each process group should be equal or smaller than 40Mbytes/sec.
So, we need to check total I/O bandwidth before set it up.

* Setup information
------------------------------
range-bw is based on newest version of dm-ioband, bio-cgroup V7(4patch
files), dm-ioband-V1.10.3(1 patch file) and these can be
referred in
http://people.valinux.co.jp/~ryov/dm-ioband/
http://people.valinux.co.jp/~ryov/bio-cgroup/
and the below range-bw patch file(dm-ioband-rangebw-1.10.3.patch)
You have to apply this(dm-ioband-rangebw-1.10.3.patch) patch file
after applying dm-ioband and bio-cgroup patches

* Usage
--------------
It is very useful to refer the documentation for dm-ioband in
/Documentation/device-mapper/ioband.txt or
http://people.valinux.co.jp/~ryov/dm-ioband/, because Range-BW follows
the basic semantics of dm-ioband.
This example is for min-max mode.

# mount the cgroup for range-bw
mount -t cgroup -o bio none /root/cgroup/bio

# create the process groups (3 groups)
mkdir /root/cgroup/bio/bgroup1
mkdir /root/cgroup/bio/bgroup2
mkdir /root/cgroup/bio/bgroup3

# create the ioband device ( name : ioband1 )
echo "0 $(blockdev --getsize /dev/sdb2) ioband /dev/sdb2 1 0 0 none
range-bw 0 :0" | dmsetup create ioband1
: device name (/dev/sdb2) should be modified depending on your system

# init ioband device ( type and policy )
dmsetup message ioband1 0 type cgroup
dmsetup message ioband1 0 policy range-bw

# attach the groups to the ioband device
dmsetup message ioband1 0 attach 2
dmsetup message ioband1 0 attach 3
dmsetup message ioband1 0 attach 4
: group number can be referred in /root/cgroup/bio/bgroup1/bio.id

# allocate the values ( min-bw and max-bw ) : XXX Kbytes
: the sum of minimum I/O bandwidth in each group should be equal or
smaller than total bandwidth to be supported by your system

# range : about 100~500 Kbytes
dmsetup message ioband1 0 min-bw 2:100
dmsetup message ioband1 0 max-bw 2:500

# range : about 700~1000 Kbytes
dmsetup message ioband1 0 min-bw 3:700
dmsetup message ioband1 0 max-bw 3:1000

# range : about 50~60Mbytes
dmsetup message ioband1 0 min-bw 4:50000
dmsetup message ioband1 0 max-bw 4:60000

You can confirm the configuration of range-bw by using this command :
[root@localhost range-bw]# dmsetup table --target ioband
ioband1: 0 305235000 ioband 8:18 1 4 128 cgroup range-bw 16384 :0
2:100:500 3:700:1000 4:30000:60000

* Evaluation
------------------

range-bw supports the 2 kinds of configuration style, one is min-max
configuration
and the other is only-max configuration.
min-max is to supports guaranteeing the minimum I/O requirement and
limitation of
unnecessary I/O bandwidth at the same time. And only-max is to support
only bandwidth limitation.
the evaluation result of range-bw is as like below

-Testing tool : fio-1.26
-Total bandwidth in my test system :
  * Sequential read / write : about 60 Mbytes/sec
  * Random read / write : about 290 Kbytes/sec
- configuration semantics : X:Y:Z = process group : min-bw : max-bw
and min-bw, max-bw : Kbytes/sec

1. Sequential read / write (total bandwidth : about 60 Mbytes/sec)

1.1 Test1
* Configuration (min-max)
[root@localhost range-bw]# dmsetup table --target ioband
ioband1: 0 305235000 ioband 8:18 1 4 128 cgroup range-bw 16384 :0
2:500:700 3:3000:5000 4:50000:55000

* Command :
fio --time_based --runtime=30 --ioengine=libaio --iodepth=50
--direct=1  --rw=write --name=grp1 --filename=/dev/mapper/ioband1
--numjobs=1 --norandommap

* Result :
- Group2
Run status group 0 (all jobs):
  WRITE: io=21,008KiB, aggrb=693KiB/s, minb=693KiB/s, maxb=693KiB/s,
mint=30311msec, maxt=30311msec
- Group3
Run status group 0 (all jobs):
  WRITE: io=147MiB, aggrb=4,979KiB/s, minb=4,979KiB/s,
maxb=4,979KiB/s, mint=30128msec, maxt=30128msec
- Group4
Run status group 0 (all jobs):
WRITE: io=1,526MiB, aggrb=52,057KiB/s, minb=52,057KiB/s,
maxb=52,057KiB/s, mint=30010msec, maxt=30010msec

1.2 Test2
* Configuration (only-max)
[root@localhost range-bw]# dmsetup table --target ioband
ioband1: 0 305235000 ioband 8:18 1 4 128 cgroup range-bw 16384 :0
2:0:700 3:0:5000 4:0:10000

* Command :
fio --time_based --runtime=30 --ioengine=libaio --iodepth=50
--direct=1  --rw=read --name=grp1 --filename=/dev/mapper/ioband1
--numjobs=1 --norandommap

* Result :
- Group2
Run status group 0 (all jobs):
   READ: io=20,424KiB, aggrb=676KiB/s, minb=676KiB/s, maxb=676KiB/s,
mint=30169msec, maxt=30169msec
- Group3
Run status group 0 (all jobs):
   READ: io=142MiB, aggrb=4,836KiB/s, minb=4,836KiB/s,
maxb=4,836KiB/s, mint=30022msec, maxt=30022msec
- Group4
Run status group 0 (all jobs):
   READ: io=293MiB, aggrb=9,755KiB/s, minb=9,755KiB/s,
maxb=9,755KiB/s, mint=30752msec, maxt=30752msec

2. Random read / write (total bandwidth : about 290 Kbytes/sec)

2.1 Test1 (min-max)
* Configuration :
[root@localhost range-bw]# dmsetup table --target ioband
ioband1: 0 305235000 ioband 8:18 1 4 128 cgroup range-bw 16384 :0
2:80:90 3:180:200 4:0:0

* Command :
fio --time_based --runtime=30 --ioengine=libaio --iodepth=50
--direct=1  --rw=randread --name=grp1 --filename=/dev/mapper/ioband1
--numjobs=1 --norandommap

* Result :
Group2
Run status group 0 (all jobs):
   READ: io=2,780KiB, aggrb=92KiB/s, minb=92KiB/s, maxb=92KiB/s,
mint=30175msec, maxt=30175msec
Group3
Run status group 0 (all jobs):
   READ: io=6,008KiB, aggrb=193KiB/s, minb=193KiB/s, maxb=193KiB/s,
mint=30998msec, maxt=30998msec

2.2 Test2 (min-max)
* Configuration :
[root@localhost range-bw]# dmsetup table --target ioband
ioband1: 0 305235000 ioband 8:18 1 4 128 cgroup range-bw 16384 :0
2:50:60 3:200:220 4:0:0

* Command :
fio --time_based --runtime=30 --ioengine=libaio --iodepth=50
--direct=1  --rw=randwrite --name=grp1 --filename=/dev/mapper/ioband1
--numjobs=1 --norandommap

* Result :
- Group2
Run status group 0 (all jobs):
  WRITE: io=1,848KiB, aggrb=61KiB/s, minb=61KiB/s, maxb=61KiB/s,
mint=30158msec, maxt=30158msec
- Group3
Run status group 0 (all jobs):
  WRITE: io=6,612KiB, aggrb=213KiB/s, minb=213KiB/s, maxb=213KiB/s,
mint=30994msec, maxt=30994msec

-- 
Best Regards,
Dong-Jae Kang

Re: [PATCH 0/2] range-bw : Another I/O scheduling policy of dm-ioband (v1)

[Chris Snook]

On Mon, May 4, 2009 at 6:56 AM, Dong-Jae Kang <baramsori72@gmail.com> wrote:
> range-bw has two kinds of operation modes, min-max and max mode.
> Min-max mode is to supports guaranteeing the minimum I/O requirement
> and limitation of unnecessary I/O bandwidth at the same time. And max
> mode is to support only limitation. So in case of min-max mode, you
> need to configure min-bw and max-bw values and in case of max mode,
> configure only max-bw.

Please forgive me if I missed a previous discussion, but how is max
mode different from min-max mode with a minimum of zero?  I understand
that there may be some special-case optimizations you can do
internally, but it's unclear to me why you need two different
interfaces.

-- Chris

Re: [PATCH 0/2] range-bw : Another I/O scheduling policy of dm-ioband (v1)

[Dong-Jae Kang]

Hi, Chris Snook

2009/5/4 Chris Snook <chris.snook@gmail.com>:
> On Mon, May 4, 2009 at 6:56 AM, Dong-Jae Kang <baramsori72@gmail.com> wrote:
>> range-bw has two kinds of operation modes, min-max and max mode.
>> Min-max mode is to supports guaranteeing the minimum I/O requirement
>> and limitation of unnecessary I/O bandwidth at the same time. And max
>> mode is to support only limitation. So in case of min-max mode, you
>> need to configure min-bw and max-bw values and in case of max mode,
>> configure only max-bw.
>
> Please forgive me if I missed a previous discussion, but how is max
> mode different from min-max mode with a minimum of zero?  I understand
> that there may be some special-case optimizations you can do
> internally, but it's unclear to me why you need two different
> interfaces.
>

Thank you for your good comments,

As you asked,
max mode is actually same with min-max mode with a zero minimum value.
It is only to consider the aspect of user interface now.
the reason is as below
according to the usage case, some users may be interested in
guranteeing the minimum BW
and limitation of maximum BW at the same time (the  basic
fuctionalilty of range-bw), but, some
users can be interested in only limitation of BW to reserve the BW for
another service.

actually, I was overlooking that users are confused with the two kinds
of interface

So, your comments will be related with my future work :)
if two kinds of mode is confused or unnecessary to people,
I will re-consider about supporting the unified interfaces as you pointed
or I try to optimize the scheduling policy for max mode.
and this work will be applied in next release of range-bw.

Thank you, Chris Snook

-- 
Best Regards,
Dong-Jae Kang