From: Yu Kuai <yukuai1@huaweicloud.com>
To: "Dragan Milivojević" <galileo@pkm-inc.com>,
"Yu Kuai" <yukuai1@huaweicloud.com>,
"yukuai (C)" <yukuai3@huawei.com>
Cc: Song Liu <song@kernel.org>,
linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org,
yi.zhang@huawei.com, yangerkun@huawei.com, "Luse,
Paul E" <paul.e.luse@intel.com>
Subject: Re: [PATCH md-6.13] md: remove bitmap file support
Date: Mon, 11 Nov 2024 21:02:38 +0800 [thread overview]
Message-ID: <e9fd5484-619b-e8a9-984c-359bf5475b9f@huaweicloud.com> (raw)
In-Reply-To: <cf00703f-f4bd-4b6a-9626-72d839ebaf7b@pkm-inc.com>
Hi,
在 2024/11/11 19:59, Dragan Milivojević 写道:
> On 11/11/2024 03:04, Yu Kuai wrote:
>
>> Yes, as I said please show me you how you creat the array and your test
>> script. I must know what you are testing, like single threaded or high
>> concurrency. For example your result shows bitmap none close to bitmap
>> external, this is impossible in our previous results. I can only guess
>> that you're testing single threaded.
>
> All of that is included in the previously linked pastebin.
TBO, I don't know what is this. :(
> I will include the contents of that pastebin at the end of this email if
> that helps.
> Every test includes the mdadm create line, disk settings, md settings,
> fio test line
> used and the results and the typical iostat output during the test. I
> hope that is
> sufficient.
>
>> BTW, it'll be great if you can provide some perf results of the internal
>> bitmap in your case, that will show us directly where is the bottleneck.
>
> Not right now, this server is in production and I'm not sure if I will
> be able
> to get it to an idle state or to find the time to do it due to other
> projects.
>
>>> BTW do you guys do performance tests? All of the raid levels are
>>
>> We do, but we never test external bitmap.
>
> I wasn't referring to that, more to the fact that there is a huge
> difference in
> performance between no bitmap and bitmap or that raid (even "simple"
> levels like 0)
> do not scale with real world workloads.
Yes, this is a known problem, the gap here is that I don't think
external bitmap is much helpful, while your result disagree.
>
> The contents of that pastebin, hopefully my email client won't mess up
> the formating:
>
>
> 5 disk RAID5, 64K chunk
>
> Summary
>
> Test BW IOPS
> bitmap internal 64M 700KiB/s 174
> bitmap internal 128M 702KiB/s 175
> bitmap internal 512M 1142KiB/s 285
> bitmap internal 1024M 40.4MiB/s 10.3k
> bitmap internal 2G 66.5MiB/s 17.0k
> bitmap external 64M 67.8MiB/s 17.3k
> bitmap external 1024M 76.5MiB/s 19.6k
> bitmap none 80.6MiB/s 20.6k
> Single disk 1K 54.1MiB/s 55.4k
> Single disk 4K 269MiB/s 68.8k
>
>
>
>
> AlmaLinux release 9.4 (Seafoam Ocelot)
> 5.14.0-427.20.1.el9_4
>
>
> nvme list
> Node Generic SN
> Model Namespace
> Usage Format FW Rev
> --------------------- --------------------- --------------------
> ---------------------------------------- ----------
> -------------------------- ---------------- --------
> /dev/nvme0n1 /dev/ng0n1 1460A0F9TSTJ Dell DC
> NVMe CD8 U.2 960GB 0x1 122.33 GB / 960.20 GB
> 512 B + 0 B 2.0.0
> /dev/nvme1n1 /dev/ng1n1 S6WRNJ0WA04045P Samsung
> SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB
> 512 B + 0 B 5B2QGXA7
> /dev/nvme2n1 /dev/ng2n1 S6WRNJ0WA04048B Samsung
> SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB
> 512 B + 0 B 5B2QGXA7
> /dev/nvme3n1 /dev/ng3n1 S6WRNJ0W810396H Samsung
> SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB
> 512 B + 0 B 5B2QGXA7
> /dev/nvme4n1 /dev/ng4n1 S6WRNJ0W808149N Samsung
> SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB
> 512 B + 0 B 5B2QGXA7
> /dev/nvme5n1 /dev/ng5n1 S6WRNJ0WA04043Z Samsung
> SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB
> 512 B + 0 B 5B2QGXA7
> /dev/nvme6n1 /dev/ng6n1 PHBT909504AH016N INTEL
> MEMPEK1J016GAL 0x1 14.40 GB / 14.40 GB
> 512 B + 0 B K4110420
> /dev/nvme7n1 /dev/ng7n1 S6WRNJ0WA04036R Samsung
> SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB
> 512 B + 0 B 5B2QGXA7
> /dev/nvme8n1 /dev/ng8n1 S6WRNJ0WA04050H Samsung
> SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB
> 512 B + 0 B 5B2QGXA7
>
>
>
>
> bitmap internal 64M
> ================================================================
> mdadm --verbose --create --assume-clean --bitmap=internal
> --bitmap-chunk=64M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K
> --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
The array set up is fine. And the following external bitmap is using
/bitmap/bitmap.bin, does the back-end storage of this file the same as
test device?
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
Then this is what I suspected, the above test is quite limited and can't
replace the real world workload, 1 thread 1 iodepth with 4k randwrite.
I still can't believe your test result, and I can't figure out why
internal bitmap is so slow. Hence I use ramdisk(10GB) to create a raid5,
and use the same fio script to test, the result is quite different from
yours:
ram0: 981MiB/s
non-bitmap: 132MiB/s
internal-bitmap: 95.5MiB/s
>
> Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=716KiB/s][w=179 IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=4718: Sun Jun 30 02:18:30 2024
> write: IOPS=174, BW=700KiB/s (717kB/s)(41.0MiB/60005msec); 0 zone resets
> slat (usec): min=4, max=18062, avg=11.28, stdev=176.21
> clat (usec): min=46, max=13308, avg=5700.08, stdev=1194.59
> lat (usec): min=53, max=22717, avg=5711.36, stdev=1206.03
> clat percentiles (usec):
> | 1.00th=[ 51], 5.00th=[ 5800], 10.00th=[ 5800], 20.00th=[ 5866],
> | 30.00th=[ 5866], 40.00th=[ 5866], 50.00th=[ 5866], 60.00th=[ 5932],
> | 70.00th=[ 5932], 80.00th=[ 5932], 90.00th=[ 5932], 95.00th=[ 5997],
> | 99.00th=[ 6194], 99.50th=[ 8586], 99.90th=[10290], 99.95th=[13042],
> | 99.99th=[13042]
> bw ( KiB/s): min= 608, max= 752, per=100.00%, avg=700.03,
> stdev=20.93, samples=119
There is absolutely something wrong here, it doesn't make sense to me
that internal bitmap is so slow. However, I have no idea until you can
provide the perf result.
Thanks,
Kuai
> iops : min= 152, max= 188, avg=175.01, stdev= 5.23,
> samples=119
> lat (usec) : 50=0.68%, 100=3.23%
> lat (msec) : 10=95.99%, 20=0.10%
> cpu : usr=0.08%, sys=0.24%, ctx=10503, majf=0, minf=8
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,10499,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=700KiB/s (717kB/s), 700KiB/s-700KiB/s (717kB/s-717kB/s),
> io=41.0MiB (43.0MB), run=60005-60005msec
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 0.07 0.00 0.00 99.93
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s
> wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 175.47 0.69
> 0.00 0.00 5.68 4.00 0.00 0.00 1.00 100.00
> nvme1n1 69.40 0.27 0.00 0.00 0.01 4.00 237.80 0.93
> 0.00 0.00 3.55 4.00 168.47 0.81 0.98 95.59
> nvme2n1 69.20 0.27 0.00 0.00 0.01 4.00 237.60 0.93
> 0.00 0.00 3.55 4.00 168.47 0.81 0.98 95.61
> nvme3n1 72.20 0.28 0.00 0.00 0.01 4.00 240.60 0.94
> 0.00 0.00 3.51 4.00 168.47 0.83 0.98 95.29
> nvme4n1 68.07 0.27 0.00 0.00 0.02 4.00 236.53 0.92
> 0.00 0.00 3.57 4.00 168.47 0.81 0.98 95.65
> nvme5n1 72.07 0.28 0.00 0.00 0.02 4.00 240.53 0.94
> 0.00 0.00 3.52 4.00 168.47 0.83 0.99 95.31
>
>
> mdadm -X /dev/nvme1n1
> Filename : /dev/nvme1n1
> Magic : 6d746962
> Version : 4
> UUID : 77fa1a1b:2f0dd646:adc85c8e:985513a8
> Events : 3
> Events Cleared : 3
> State : OK
> Chunksize : 64 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
> Bitmap : 29807 bits (chunks), 1517 dirty (5.1%)
>
>
> bitmap internal 128M
> ================================================================
> for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
>
> mdadm --verbose --create --assume-clean --bitmap=internal
> --bitmap-chunk=128M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K
> --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
>
> Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=716KiB/s][w=179 IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=6283: Sun Jun 30 02:49:06 2024
> write: IOPS=175, BW=702KiB/s (719kB/s)(41.1MiB/60002msec); 0 zone resets
> slat (usec): min=8, max=18200, avg=16.06, stdev=177.21
> clat (usec): min=61, max=20048, avg=5675.78, stdev=1968.88
> lat (usec): min=74, max=22975, avg=5691.84, stdev=1976.14
> clat percentiles (usec):
> | 1.00th=[ 68], 5.00th=[ 73], 10.00th=[ 5866], 20.00th=[ 5932],
> | 30.00th=[ 5932], 40.00th=[ 5932], 50.00th=[ 5932], 60.00th=[ 5997],
> | 70.00th=[ 5997], 80.00th=[ 5997], 90.00th=[ 5997], 95.00th=[ 6063],
> | 99.00th=[14615], 99.50th=[15008], 99.90th=[16188], 99.95th=[16319],
> | 99.99th=[16319]
> bw ( KiB/s): min= 384, max= 816, per=99.97%, avg=702.12,
> stdev=72.52, samples=119
> iops : min= 96, max= 204, avg=175.53, stdev=18.13,
> samples=119
> lat (usec) : 100=7.62%, 250=0.01%
> lat (msec) : 10=90.80%, 20=1.56%, 50=0.01%
> cpu : usr=0.11%, sys=0.34%, ctx=10539, majf=0, minf=8
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,10534,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=702KiB/s (719kB/s), 702KiB/s-702KiB/s (719kB/s-719kB/s),
> io=41.1MiB (43.1MB), run=60002-60002msec
>
>
>
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 0.08 0.00 0.00 99.92
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s
> wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 173.73 0.68
> 0.00 0.00 5.73 4.00 0.00 0.00 1.00 99.99
> nvme1n1 65.87 0.26 0.00 0.00 0.01 4.00 226.07 0.65
> 0.00 0.00 3.60 2.94 160.20 0.81 0.94 92.46
> nvme2n1 71.33 0.28 0.00 0.00 0.02 4.00 231.53 0.67
> 0.00 0.00 3.50 2.96 160.27 0.84 0.95 91.79
> nvme3n1 68.60 0.27 0.00 0.00 0.02 4.00 228.80 0.66
> 0.00 0.00 3.68 2.95 160.27 0.93 0.99 94.37
> nvme4n1 68.87 0.27 0.00 0.00 0.02 4.00 229.07 0.66
> 0.00 0.00 3.52 2.95 160.20 0.81 0.94 91.59
> nvme5n1 72.80 0.28 0.00 0.00 0.02 4.00 233.00 0.68
> 0.00 0.00 3.53 2.97 160.27 0.87 0.96 92.29
>
>
> mdadm -X /dev/nvme1n1
> Filename : /dev/nvme1n1
> Magic : 6d746962
> Version : 4
> UUID : 93fdcd4b:ae61a1f8:4d809242:2cd4a4c7
> Events : 3
> Events Cleared : 3
> State : OK
> Chunksize : 128 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
> Bitmap : 14904 bits (chunks), 1617 dirty (10.8%)
>
>
>
>
>
> bitmap internal 512M
> ================================================================
> for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
>
> mdadm --verbose --create --assume-clean --bitmap=internal
> --bitmap-chunk=512M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K
> --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
>
>
> Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=1232KiB/s][w=308 IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=6661: Sun Jun 30 02:58:11 2024
> write: IOPS=285, BW=1142KiB/s (1169kB/s)(66.9MiB/60006msec); 0 zone
> resets
> slat (usec): min=4, max=18130, avg=10.80, stdev=138.54
> clat (usec): min=42, max=13261, avg=3490.08, stdev=2945.95
> lat (usec): min=50, max=22827, avg=3500.88, stdev=2949.63
> clat percentiles (usec):
> | 1.00th=[ 49], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 55],
> | 30.00th=[ 58], 40.00th=[ 72], 50.00th=[ 5866], 60.00th=[ 5932],
> | 70.00th=[ 5932], 80.00th=[ 5932], 90.00th=[ 5997], 95.00th=[ 5997],
> | 99.00th=[ 6128], 99.50th=[ 8586], 99.90th=[ 9896], 99.95th=[13042],
> | 99.99th=[13042]
> bw ( KiB/s): min= 600, max= 1648, per=99.68%, avg=1138.89,
> stdev=188.44, samples=119
> iops : min= 150, max= 412, avg=284.72, stdev=47.11,
> samples=119
> lat (usec) : 50=3.41%, 100=38.62%, 250=0.04%, 500=0.03%
> lat (msec) : 10=57.83%, 20=0.07%
> cpu : usr=0.09%, sys=0.40%, ctx=17130, majf=0, minf=9
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,17127,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=1142KiB/s (1169kB/s), 1142KiB/s-1142KiB/s
> (1169kB/s-1169kB/s), io=66.9MiB (70.2MB), run=60006-60006msec
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 0.10 0.00 0.00 99.90
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s
> wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 307.13 1.20
> 0.00 0.00 3.24 4.00 0.00 0.00 1.00 100.00
> nvme1n1 120.47 0.47 0.00 0.00 0.01 4.00 286.07 0.63
> 0.00 0.00 3.03 2.26 165.60 0.99 1.03 96.58
> nvme2n1 123.87 0.48 0.00 0.00 0.01 4.00 289.47 0.65
> 0.00 0.00 3.00 2.28 165.60 1.00 1.04 96.63
> nvme3n1 120.87 0.47 0.00 0.00 0.01 4.00 286.47 0.63
> 0.00 0.00 3.02 2.27 165.60 1.00 1.03 96.39
> nvme4n1 125.00 0.49 0.00 0.00 0.02 4.00 290.60 0.65
> 0.00 0.00 3.00 2.29 165.60 1.02 1.04 96.54
> nvme5n1 124.07 0.48 0.00 0.00 0.02 4.00 289.67 0.65
> 0.00 0.00 3.01 2.28 165.60 1.03 1.04 96.59
>
>
> mdadm -X /dev/nvme1n1
> Filename : /dev/nvme1n1
> Magic : 6d746962
> Version : 4
> UUID : 17eadc76:a367542a:feb6e24e:d650576c
> Events : 3
> Events Cleared : 3
> State : OK
> Chunksize : 512 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
> Bitmap : 3726 bits (chunks), 1977 dirty (53.1%)
>
>
> bitmap internal 1024M
> ================================================================
> for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
>
> mdadm --verbose --create --assume-clean --bitmap=internal
> --bitmap-chunk=1024M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K
> --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
>
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=51.0MiB/s][w=13.1k IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=7120: Sun Jun 30 03:08:12 2024
> write: IOPS=10.3k, BW=40.4MiB/s (42.4MB/s)(2425MiB/60001msec); 0 zone
> resets
> slat (usec): min=6, max=18135, avg= 8.93, stdev=23.41
> clat (usec): min=3, max=10459, avg=86.97, stdev=342.95
> lat (usec): min=63, max=22927, avg=95.90, stdev=344.33
> clat percentiles (usec):
> | 1.00th=[ 62], 5.00th=[ 63], 10.00th=[ 64], 20.00th=[ 65],
> | 30.00th=[ 65], 40.00th=[ 66], 50.00th=[ 67], 60.00th=[ 67],
> | 70.00th=[ 68], 80.00th=[ 69], 90.00th=[ 70], 95.00th=[ 74],
> | 99.00th=[ 133], 99.50th=[ 155], 99.90th=[ 5997], 99.95th=[ 5997],
> | 99.99th=[ 6063]
> bw ( KiB/s): min= 616, max=52968, per=99.80%, avg=41305.95,
> stdev=20465.79, samples=119
> iops : min= 154, max=13242, avg=10326.47, stdev=5116.44,
> samples=119
> lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 100=98.64%, 250=1.00%
> lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.01%, 10=0.33%, 20=0.01%
> cpu : usr=1.89%, sys=12.74%, ctx=620837, majf=0, minf=170751
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,620822,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=40.4MiB/s (42.4MB/s), 40.4MiB/s-40.4MiB/s
> (42.4MB/s-42.4MB/s), io=2425MiB (2543MB), run=60001-60001msec
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.04 0.00 1.27 0.00 0.00 98.70
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s
> wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 18216.93
> 71.16 0.00 0.00 0.05 4.00 0.00 0.00 0.88 100.00
> nvme1n1 7256.20 28.34 0.00 0.00 0.01 4.00 7256.27
> 28.34 0.00 0.00 0.01 4.00 0.00 0.00 0.19 99.71
> nvme2n1 7302.53 28.53 0.00 0.00 0.01 4.00 7302.53
> 28.53 0.00 0.00 0.01 4.00 0.00 0.00 0.17 99.73
> nvme3n1 7278.47 28.43 0.00 0.00 0.01 4.00 7278.53
> 28.43 0.00 0.00 0.01 4.00 0.00 0.00 0.18 99.57
> nvme4n1 7303.93 28.53 0.00 0.00 0.01 4.00 7303.93
> 28.53 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.74
> nvme5n1 7292.67 28.49 0.00 0.00 0.02 4.00 7292.60
> 28.49 0.00 0.00 0.02 4.00 0.00 0.00 0.22 99.69
>
>
>
> mdadm -X /dev/nvme1n1
> Filename : /dev/nvme1n1
> Magic : 6d746962
> Version : 4
> UUID : a0c7ad14:50689e41:e065a166:4935a186
> Events : 3
> Events Cleared : 3
> State : OK
> Chunksize : 1 GB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
> Bitmap : 1863 bits (chunks), 1863 dirty (100.0%)
>
>
> bitmap internal 2G
> ================================================================
> for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
>
> mdadm --verbose --create --assume-clean --bitmap=internal
> --bitmap-chunk=2G /dev/md/raid5 --name=raid5 --level=5 --chunk=64K
> --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
>
> Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=74.7MiB/s][w=19.1k IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=7696: Sun Jun 30 03:30:40 2024
> write: IOPS=17.0k, BW=66.5MiB/s (69.8MB/s)(3993MiB/60001msec); 0 zone
> resets
> slat (usec): min=2, max=18094, avg= 4.79, stdev=17.94
> clat (usec): min=5, max=10352, avg=53.37, stdev=181.29
> lat (usec): min=41, max=22883, avg=58.16, stdev=182.72
> clat percentiles (usec):
> | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 46],
> | 30.00th=[ 46], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 48],
> | 70.00th=[ 48], 80.00th=[ 49], 90.00th=[ 50], 95.00th=[ 52],
> | 99.00th=[ 90], 99.50th=[ 126], 99.90th=[ 873], 99.95th=[ 5997],
> | 99.99th=[ 6063]
> bw ( KiB/s): min= 640, max=80168, per=99.91%, avg=68080.94,
> stdev=21547.29, samples=119
> iops : min= 160, max=20042, avg=17020.24, stdev=5386.82,
> samples=119
> lat (usec) : 10=0.01%, 50=92.06%, 100=7.10%, 250=0.73%, 500=0.01%
> lat (usec) : 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.01%, 10=0.09%, 20=0.01%
> cpu : usr=2.22%, sys=11.55%, ctx=1022167, majf=0, minf=14
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,1022154,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=66.5MiB/s (69.8MB/s), 66.5MiB/s-66.5MiB/s
> (69.8MB/s-69.8MB/s), io=3993MiB (4187MB), run=60001-60001msec
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.04 0.00 1.15 0.00 0.00 98.81
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s
> wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 18836.40
> 73.58 0.00 0.00 0.05 4.00 0.00 0.00 0.87 100.00
> nvme1n1 7505.27 29.32 0.00 0.00 0.01 4.00 7505.40
> 29.32 0.00 0.00 0.01 4.00 0.00 0.00 0.19 99.93
> nvme2n1 7510.00 29.34 0.00 0.00 0.01 4.00 7510.07
> 29.34 0.00 0.00 0.01 4.00 0.00 0.00 0.17 99.90
> nvme3n1 7561.40 29.54 0.00 0.00 0.01 4.00 7561.47
> 29.54 0.00 0.00 0.01 4.00 0.00 0.00 0.19 100.00
> nvme4n1 7543.07 29.47 0.00 0.00 0.01 4.00 7543.07
> 29.47 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.91
> nvme5n1 7552.73 29.50 0.00 0.00 0.01 4.00 7552.80
> 29.50 0.00 0.00 0.01 4.00 0.00 0.00 0.22 99.91
>
>
>
> mdadm -X /dev/nvme1n1
> Filename : /dev/nvme1n1
> Magic : 6d746962
> Version : 4
> UUID : 7d8ed7e8:4c2c4b17:723a22e5:4e9b5200
> Events : 3
> Events Cleared : 3
> State : OK
> Chunksize : 2 GB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
> Bitmap : 932 bits (chunks), 932 dirty (100.0%)
>
>
>
>
>
>
>
>
>
> bitmap external 64M
> ================================================================
> for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
>
> mdadm --verbose --create --assume-clean --bitmap=/bitmap/bitmap.bin
> --bitmap-chunk=64M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K
> --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
>
> Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=67.3MiB/s][w=17.2k IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=7912: Sun Jun 30 03:39:11 2024
> write: IOPS=17.3k, BW=67.8MiB/s (71.1MB/s)(4066MiB/60001msec); 0 zone
> resets
> slat (usec): min=2, max=21987, avg= 6.11, stdev=22.04
> clat (usec): min=3, max=8410, avg=50.79, stdev=27.03
> lat (usec): min=42, max=22140, avg=56.90, stdev=35.13
> clat percentiles (usec):
> | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 46],
> | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 49], 60.00th=[ 50],
> | 70.00th=[ 51], 80.00th=[ 52], 90.00th=[ 56], 95.00th=[ 68],
> | 99.00th=[ 93], 99.50th=[ 124], 99.90th=[ 155], 99.95th=[ 237],
> | 99.99th=[ 1037]
> bw ( KiB/s): min=38120, max=82576, per=100.00%, avg=69402.96,
> stdev=7769.33, samples=119
> iops : min= 9530, max=20644, avg=17350.76, stdev=1942.33,
> samples=119
> lat (usec) : 4=0.01%, 20=0.01%, 50=67.87%, 100=31.34%, 250=0.76%
> lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
> cpu : usr=2.23%, sys=14.27%, ctx=1040947, majf=0, minf=233929
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,1040925,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s
> (71.1MB/s-71.1MB/s), io=4066MiB (4264MB), run=60001-60001msec
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.04 0.00 1.15 0.00 0.00 98.81
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s
> wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 18428.60
> 71.99 0.00 0.00 0.05 4.00 0.00 0.00 0.87 99.99
> nvme1n1 7399.40 28.90 0.00 0.00 0.01 4.00 7399.47
> 28.90 0.00 0.00 0.01 4.00 0.00 0.00 0.17 99.73
> nvme2n1 7361.20 28.75 0.00 0.00 0.01 4.00 7361.27
> 28.75 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.63
> nvme3n1 7376.67 28.82 0.00 0.00 0.01 4.00 7376.73
> 28.82 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.63
> nvme4n1 7367.27 28.78 0.00 0.00 0.01 4.00 7367.20
> 28.78 0.00 0.00 0.01 4.00 0.00 0.00 0.18 99.65
> nvme5n1 7352.47 28.72 0.00 0.00 0.01 4.00 7352.67
> 28.72 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.73
> nvme8n1 0.47 0.00 0.00 0.00 0.00 4.00 293.40
> 1.15 0.00 0.00 0.02 4.00 0.00 0.00 0.01 24.24
>
>
>
> mdadm -X /bitmap/bitmap.bin
> Filename : /bitmap/bitmap.bin
> Magic : 6d746962
> Version : 4
> UUID : 1e3480e5:1f9d8b8a:53ebc6b7:279afb73
> Events : 3
> Events Cleared : 3
> State : OK
> Chunksize : 64 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
> Bitmap : 29807 bits (chunks), 29665 dirty (99.5%)
>
>
>
> bitmap external 1024M
> ================================================================
> for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
>
> mdadm --verbose --create --assume-clean --bitmap=/bitmap/bitmap.bin
> --bitmap-chunk=1024M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K
> --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
>
> Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=70.6MiB/s][w=18.1k IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=8592: Sun Jun 30 03:54:11 2024
> write: IOPS=19.6k, BW=76.5MiB/s (80.2MB/s)(4590MiB/60001msec); 0 zone
> resets
> slat (usec): min=2, max=21819, avg= 4.12, stdev=20.16
> clat (usec): min=22, max=3706, avg=46.37, stdev=20.38
> lat (usec): min=40, max=21951, avg=50.49, stdev=28.81
> clat percentiles (usec):
> | 1.00th=[ 40], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 42],
> | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 47],
> | 70.00th=[ 48], 80.00th=[ 49], 90.00th=[ 50], 95.00th=[ 52],
> | 99.00th=[ 86], 99.50th=[ 120], 99.90th=[ 157], 99.95th=[ 233],
> | 99.99th=[ 906]
> bw ( KiB/s): min=61616, max=84728, per=100.00%, avg=78398.66,
> stdev=5410.81, samples=119
> iops : min=15404, max=21182, avg=19599.66, stdev=1352.70,
> samples=119
> lat (usec) : 50=90.10%, 100=9.16%, 250=0.72%, 500=0.01%, 750=0.01%
> lat (usec) : 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.01%
> cpu : usr=2.35%, sys=11.88%, ctx=1175104, majf=0, minf=11
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,1175086,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=76.5MiB/s (80.2MB/s), 76.5MiB/s-76.5MiB/s
> (80.2MB/s-80.2MB/s), io=4590MiB (4813MB), run=60001-60001msec
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.04 0.00 1.03 0.00 0.00 98.93
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s
> wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 20758.20
> 81.09 0.00 0.00 0.04 4.00 0.00 0.00 0.89 100.00
> nvme1n1 8291.67 32.39 0.00 0.00 0.01 4.00 8291.73
> 32.39 0.00 0.00 0.01 4.00 0.00 0.00 0.22 99.87
> nvme2n1 8270.93 32.31 0.00 0.00 0.01 4.00 8271.07
> 32.31 0.00 0.00 0.01 4.00 0.00 0.00 0.19 99.79
> nvme3n1 8310.67 32.46 0.00 0.00 0.01 4.00 8310.80
> 32.46 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.83
> nvme4n1 8300.67 32.42 0.00 0.00 0.01 4.00 8300.67
> 32.42 0.00 0.00 0.01 4.00 0.00 0.00 0.23 99.76
> nvme5n1 8342.13 32.59 0.00 0.00 0.02 4.00 8342.13
> 32.59 0.00 0.00 0.01 4.00 0.00 0.00 0.25 99.85
> nvme8n1 0.33 0.00 0.00 0.00 8.40 4.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33
>
>
> mdadm -X /bitmap/bitmap.bin
> Filename : /bitmap/bitmap.bin
> Magic : 6d746962
> Version : 4
> UUID : 30e6211d:31ac1204:e6cdadb3:9691d3ee
> Events : 3
> Events Cleared : 3
> State : OK
> Chunksize : 1 GB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
> Bitmap : 1863 bits (chunks), 1863 dirty (100.0%)
>
>
>
> bitmap none
> ================================================================
> for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
>
> mdadm --verbose --create --assume-clean --bitmap=none /dev/md/raid5
> --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
>
> for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
> blockdev --setra 1024 /dev/md/raid5
>
> echo 8 > /sys/block/md127/md/group_thread_cnt
> echo 8192 > /sys/block/md127/md/stripe_cache_size
>
>
> fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Raid5
>
> Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=82.5MiB/s][w=21.1k IOPS][eta 00m:00s]
> Raid5: (groupid=0, jobs=1): err= 0: pid=9158: Sun Jun 30 04:11:01 2024
> write: IOPS=20.6k, BW=80.6MiB/s (84.5MB/s)(4833MiB/60001msec); 0 zone
> resets
> slat (usec): min=2, max=13598, avg= 3.50, stdev=12.46
> clat (usec): min=4, max=3694, avg=44.31, stdev=21.60
> lat (usec): min=39, max=13681, avg=47.81, stdev=24.98
> clat percentiles (usec):
> | 1.00th=[ 39], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 41],
> | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 44],
> | 70.00th=[ 45], 80.00th=[ 46], 90.00th=[ 48], 95.00th=[ 50],
> | 99.00th=[ 87], 99.50th=[ 117], 99.90th=[ 157], 99.95th=[ 229],
> | 99.99th=[ 963]
> bw ( KiB/s): min=74112, max=86712, per=100.00%, avg=82486.43,
> stdev=3696.94, samples=119
> iops : min=18528, max=21678, avg=20621.59, stdev=924.23,
> samples=119
> lat (usec) : 10=0.01%, 50=95.91%, 100=3.33%, 250=0.74%, 500=0.01%
> lat (usec) : 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.01%
> cpu : usr=2.30%, sys=10.74%, ctx=1237375, majf=0, minf=179597
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,1237359,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=80.6MiB/s (84.5MB/s), 80.6MiB/s-80.6MiB/s
> (84.5MB/s-84.5MB/s), io=4833MiB (5068MB), run=60001-60001msec
>
>
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.04 0.00 1.06 0.00 0.00 98.91
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s
> wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> md127 0.00 0.00 0.00 0.00 0.00 0.00 20040.87
> 78.28 0.00 0.00 0.04 4.00 0.00 0.00 0.89 99.99
> nvme1n1 8016.80 31.32 0.00 0.00 0.01 4.00 8016.93
> 31.32 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.68
> nvme2n1 7983.20 31.18 0.00 0.00 0.01 4.00 7983.20
> 31.18 0.00 0.00 0.01 4.00 0.00 0.00 0.18 99.74
> nvme3n1 8030.07 31.37 0.00 0.00 0.01 4.00 8030.20
> 31.37 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.62
> nvme4n1 8016.40 31.31 0.00 0.00 0.01 4.00 8016.40
> 31.31 0.00 0.00 0.01 4.00 0.00 0.00 0.23 99.73
> nvme5n1 8034.87 31.39 0.00 0.00 0.02 4.00 8035.00
> 31.39 0.00 0.00 0.01 4.00 0.00 0.00 0.24 99.71
>
>
>
>
> single disk 1K RW
> ================================================================
> fio --filename=/dev/nvme8n1 --direct=1 --rw=randwrite --bs=1k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Single
> Single: (g=0): rw=randwrite, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T)
> 1024B-1024B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=54.3MiB/s][w=55.6k IOPS][eta 00m:00s]
> Single: (groupid=0, jobs=1): err= 0: pid=4471: Sun Jun 30 18:31:56 2024
> write: IOPS=55.4k, BW=54.1MiB/s (56.7MB/s)(3244MiB/60001msec); 0 zone
> resets
> slat (usec): min=2, max=2792, avg= 2.71, stdev= 2.12
> clat (nsec): min=651, max=8350.9k, avg=14864.41, stdev=5360.57
> lat (usec): min=15, max=8403, avg=17.57, stdev= 5.79
> clat percentiles (usec):
> | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
> | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
> | 70.00th=[ 15], 80.00th=[ 16], 90.00th=[ 16], 95.00th=[ 16],
> | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 32], 99.95th=[ 33],
> | 99.99th=[ 206]
> bw ( KiB/s): min=51884, max=56778, per=100.00%, avg=55394.37,
> stdev=561.60, samples=119
> iops : min=51884, max=56778, avg=55394.44, stdev=561.62,
> samples=119
> lat (nsec) : 750=0.01%, 1000=0.01%
> lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=99.43%, 50=0.54%
> lat (usec) : 100=0.01%, 250=0.02%, 500=0.01%
> lat (msec) : 10=0.01%
> cpu : usr=3.57%, sys=16.41%, ctx=3321571, majf=0, minf=180742
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,3321653,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=54.1MiB/s (56.7MB/s), 54.1MiB/s-54.1MiB/s
> (56.7MB/s-56.7MB/s), io=3244MiB (3401MB), run=60001-60001msec
>
> Disk stats (read/write):
> nvme8n1: ios=0/3309968, merge=0/0, ticks=0/44637, in_queue=44638,
> util=99.71%
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.04 0.00 0.42 0.00 0.00 99.54
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s
> wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> nvme8n1 0.00 0.00 0.00 0.00 0.00 0.00 55496.93 54.20
> 0.00 0.00 0.01 1.00 0.00 0.00 0.75 100.00
>
>
>
>
>
> single disk 4K RW
> ================================================================
> blockdev --setra 256 /dev/nvme8n1
>
> fio --filename=/dev/nvme8n1 --direct=1 --rw=randwrite --bs=4k
> --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting
> --time_based --name=Single
>
> Single: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.35
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=270MiB/s][w=69.0k IOPS][eta 00m:00s]
> Single: (groupid=0, jobs=1): err= 0: pid=4396: Sun Jun 30 18:21:52 2024
> write: IOPS=68.8k, BW=269MiB/s (282MB/s)(15.8GiB/60001msec); 0 zone
> resets
> slat (usec): min=2, max=796, avg= 2.45, stdev= 1.59
> clat (nsec): min=652, max=8343.1k, avg=11616.73, stdev=5088.99
> lat (usec): min=11, max=8410, avg=14.06, stdev= 5.36
> clat percentiles (usec):
> | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
> | 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
> | 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
> | 99.00th=[ 14], 99.50th=[ 17], 99.90th=[ 28], 99.95th=[ 34],
> | 99.99th=[ 204]
> bw ( KiB/s): min=264072, max=277568, per=100.00%, avg=275629.71,
> stdev=1902.45, samples=119
> iops : min=66018, max=69392, avg=68907.43, stdev=475.55,
> samples=119
> lat (nsec) : 750=0.01%, 1000=0.01%
> lat (usec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=99.55%, 50=0.38%
> lat (usec) : 100=0.01%, 250=0.02%, 1000=0.01%
> lat (msec) : 10=0.01%
> cpu : usr=5.20%, sys=21.28%, ctx=4129887, majf=0, minf=45204
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued rwts: total=0,4130258,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: bw=269MiB/s (282MB/s), 269MiB/s-269MiB/s (282MB/s-282MB/s),
> io=15.8GiB (16.9GB), run=60001-60001msec
>
> Disk stats (read/write):
> nvme8n1: ios=0/4119593, merge=0/0, ticks=0/40922, in_queue=40923,
> util=99.89%
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.08 0.00 0.57 0.00 0.00 99.35
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s
> wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
> nvme8n1 0.00 0.00 0.00 0.00 0.00 0.00 69041.33
> 269.69 0.00 0.00 0.01 4.00 0.00 0.00 0.68 100.00
>
> .
>
next prev parent reply other threads:[~2024-11-11 13:02 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-11-07 12:59 Yu Kuai
2024-11-07 23:41 ` Song Liu
2024-11-08 1:03 ` Yu Kuai
2024-11-08 1:28 ` Song Liu
2024-11-08 1:33 ` Yu Kuai
2024-11-08 5:15 ` Dragan Milivojević
2024-11-08 6:07 ` Yu Kuai
2024-11-08 22:19 ` Dragan Milivojević
2024-11-09 1:43 ` Yu Kuai
2024-11-09 2:15 ` Dragan Milivojević
2024-11-11 2:04 ` Yu Kuai
2024-11-11 11:59 ` Dragan Milivojević
2024-11-11 13:02 ` Yu Kuai [this message]
2024-11-11 14:07 ` Dragan Milivojević
2024-11-13 1:18 ` Yu Kuai
2024-11-13 1:25 ` Dragan Milivojević
2024-11-12 7:54 ` Mariusz Tkaczyk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e9fd5484-619b-e8a9-984c-359bf5475b9f@huaweicloud.com \
--to=yukuai1@huaweicloud.com \
--cc=galileo@pkm-inc.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=paul.e.luse@intel.com \
--cc=song@kernel.org \
--cc=yangerkun@huawei.com \
--cc=yi.zhang@huawei.com \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®