SUSE Linux で fio を使って iSCSI のパフォーマンスを測定してみた。

fio はよく Linux で利用されるオープンソースの I/O 測定ベンチマークツールです。 Windows 版もありますが、バイナリ配布されていないようなので、コンパイル環境が必要なようですね。

-SUSE Linux へのインストール-

http://software.opensuse.org　から fio を serach すると、各バージョンに適合した fio のリストが出てきました。



リモートX端末からブラウザを起動して 1 click install をクリックすると、ダウンロードするかインストーラを起動するか確認のダイアログが出ますので YaST インストーラを起動します。レポジトリの更新から始まり、インストールされます。レポジトリのダウンロードのため、途中で証明書を import します。インストールは全てGUIで行えます。

-iSCSI とローカルディスクで比べてみた-

smartctl で見ると /tmp は Hitachi HDS722020ALA330 どうやら 2Tb 7200rpm ものです。
/iscsi sumsung 製 HD103U1 5200rpm １Tb SATA 3G の古いHDDをこれまた懐かしい QNAP TS-110 を iSCSI でマウントしました。

コマンド打ちが面倒なのでスクリプトを使ってみます。

fio はコマンドパラメータを使うか、パラメータリストファイルを使ってジョブを作ります。
ここでは seq-write の結果を求めてみます。

sles11:~ # cat fiotest.sh
#! /bin/sh
echo -------- iscsi ----
fio --directory=/iscsi --direct=1 --rw=write --bs=4k --size=1G --numjobs=3 --runtime=100 --name=seq-write
rm /iscsi/seq-write.* <------ テンポラリファイルは自分で消す必要があります。
sleep 20 <---- 次のテストで「削除の余韻」の影響避けるためわずかな間 sleep させます。
echo -------- local ----
fio --directory=/tmp --direct=1 --rw=write --bs=4k --size=1G --numjobs=3 --runtime=100 --name=seq-write
rm /tmp/iseq-write.*
echo -------- end ----
sles11:~ #
sles11:~ #

実行結果

sles11:~ # ./fiotest.sh
-------- iscsi ----
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 3 processes
seq-write: Laying out IO file(s) (1 file(s) / 1024MB)
seq-write: Laying out IO file(s) (1 file(s) / 1024MB)
seq-write: Laying out IO file(s) (1 file(s) / 1024MB)
seq-write: Laying out IO file(s) (1 file(s) / 1024MB)
seq-write: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 3 (f=3): [WWW] [100.0% done] [0K/4589K /s] [0 /1120 iops] [eta 00m:00s]
seq-write: (groupid=0, jobs=1): err= 0: pid=30120
write: io=112228KB, bw=1122.3KB/s, iops=280 , runt=100002msec
clat (usec): min=485 , max=4059.9K, avg=3551.68, stdev=35576.86
lat (usec): min=486 , max=4059.9K, avg=3554.75, stdev=35576.90
clat percentiles (usec):
| 1.00th=[ 502], 5.00th=[ 540], 10.00th=[ 756], 20.00th=[ 804],
| 30.00th=[ 828], 40.00th=[ 852], 50.00th=[ 876], 60.00th=[ 900],
| 70.00th=[ 924], 80.00th=[ 964], 90.00th=[ 1048], 95.00th=[ 1240],
| 99.00th=[103936], 99.50th=[201728], 99.90th=[464896], 99.95th=[518144],
| 99.99th=[634880]
bw (KB/s) : min= 0, max= 2434, per=38.35%, avg=1186.94, stdev=465.50
lat (usec) : 500=0.70%, 750=9.00%, 1000=76.46%
lat (msec) : 2=11.02%, 4=0.71%, 10=0.75%, 20=0.18%, 50=0.11%
lat (msec) : 100=0.06%, 250=0.85%, 500=0.09%, 750=0.08%, >=2000=0.01%
cpu : usr=0.31%, sys=1.52%, ctx=28183, majf=0, minf=24
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=28057/d=0, short=r=0/w=0/d=0
seq-write: (groupid=0, jobs=1): err= 0: pid=30121
write: io=103572KB, bw=1035.8KB/s, iops=258 , runt=100001msec
clat (usec): min=565 , max=3898.5K, avg=3849.17, stdev=37243.20
lat (usec): min=569 , max=3898.5K, avg=3852.31, stdev=37243.40
clat percentiles (usec):
| 1.00th=[ 748], 5.00th=[ 772], 10.00th=[ 788], 20.00th=[ 820],
| 30.00th=[ 836], 40.00th=[ 860], 50.00th=[ 884], 60.00th=[ 908],
| 70.00th=[ 932], 80.00th=[ 964], 90.00th=[ 1048], 95.00th=[ 1272],
| 99.00th=[156672], 99.50th=[201728], 99.90th=[477184], 99.95th=[522240],
| 99.99th=[667648]
bw (KB/s) : min= 2, max= 2320, per=35.39%, avg=1095.22, stdev=398.96
lat (usec) : 750=1.54%, 1000=83.71%
lat (msec) : 2=11.67%, 4=0.43%, 10=1.21%, 20=0.27%, 50=0.08%
lat (msec) : 100=0.02%, 250=0.81%, 500=0.15%, 750=0.09%, >=2000=0.01%
cpu : usr=0.30%, sys=1.43%, ctx=25996, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=25893/d=0, short=r=0/w=0/d=0
seq-write: (groupid=0, jobs=1): err= 0: pid=30122
write: io=93788KB, bw=960360 B/s, iops=234 , runt=100003msec
clat (usec): min=509 , max=4034.5K, avg=4250.95, stdev=39938.84
lat (usec): min=510 , max=4034.5K, avg=4254.40, stdev=39940.64
clat percentiles (usec):
| 1.00th=[ 756], 5.00th=[ 780], 10.00th=[ 804], 20.00th=[ 836],
| 30.00th=[ 860], 40.00th=[ 884], 50.00th=[ 916], 60.00th=[ 940],
| 70.00th=[ 972], 80.00th=[ 1012], 90.00th=[ 1096], 95.00th=[ 1368],
| 99.00th=[199680], 99.50th=[203776], 99.90th=[493568], 99.95th=[522240],
| 99.99th=[634880]
bw (KB/s) : min= 4, max= 2184, per=31.98%, avg=989.78, stdev=408.18
lat (usec) : 750=0.87%, 1000=76.62%
lat (msec) : 2=19.04%, 4=0.37%, 10=1.46%, 20=0.30%, 50=0.11%
lat (msec) : 100=0.04%, 250=0.91%, 500=0.17%, 750=0.09%, 1000=0.01%
lat (msec) : >=2000=0.01%
cpu : usr=0.30%, sys=1.92%, ctx=31587, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=23447/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=309588KB, aggrb=3095KB/s, minb=937KB/s, maxb=1122KB/s, mint=100001msec, maxt=100003msec

Disk stats (read/write):
sde: ios=13/77800, merge=0/44990, ticks=1008/3260968, in_queue=2594160, util=100.00%
Jobs: 3 (f=3): [WWW] [100.0% done] [0K/3653K /s] [0 /891 iops] [eta 00m:00s]
seq-write: (groupid=0, jobs=1): err= 0: pid=30214
write: io=119372KB, bw=1193.8KB/s, iops=298 , runt=100001msec
clat (usec): min=480 , max=837463 , avg=3337.99, stdev=26091.77
lat (usec): min=481 , max=837465 , avg=3341.11, stdev=26091.88
clat percentiles (usec):
| 1.00th=[ 524], 5.00th=[ 596], 10.00th=[ 660], 20.00th=[ 788],
| 30.00th=[ 820], 40.00th=[ 844], 50.00th=[ 868], 60.00th=[ 892],
| 70.00th=[ 924], 80.00th=[ 956], 90.00th=[ 1032], 95.00th=[ 1192],
| 99.00th=[47872], 99.50th=[201728], 99.90th=[497664], 99.95th=[518144],
| 99.99th=[602112]
bw (KB/s) : min= 4, max= 2496, per=35.18%, avg=1230.01, stdev=437.89
lat (usec) : 500=0.28%, 750=13.20%, 1000=73.42%
lat (msec) : 2=10.38%, 4=0.67%, 10=0.80%, 20=0.16%, 50=0.08%
lat (msec) : 100=0.03%, 250=0.81%, 500=0.07%, 750=0.09%, 1000=0.01%
cpu : usr=0.40%, sys=1.61%, ctx=29994, majf=0, minf=23
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=29843/d=0, short=r=0/w=0/d=0
seq-write: (groupid=0, jobs=1): err= 0: pid=30215
write: io=116876KB, bw=1168.8KB/s,iops=292 , runt=100003msec
clat (usec): min=476 , max=951756 , avg=3409.88, stdev=26699.56
lat (usec): min=478 , max=951758 , avg=3412.95, stdev=26699.64
clat percentiles (usec):
| 1.00th=[ 540], 5.00th=[ 684], 10.00th=[ 764], 20.00th=[ 796],
| 30.00th=[ 828], 40.00th=[ 844], 50.00th=[ 868], 60.00th=[ 892],
| 70.00th=[ 916], 80.00th=[ 956], 90.00th=[ 1032], 95.00th=[ 1208],
| 99.00th=[61696], 99.50th=[201728], 99.90th=[497664], 99.95th=[522240],
| 99.99th=[569344]
bw (KB/s) : min= 4, max= 2504, per=34.45%, avg=1204.29, stdev=410.11
lat (usec) : 500=0.22%, 750=7.28%, 1000=79.96%
lat (msec) : 2=9.88%, 4=0.52%, 10=0.91%, 20=0.16%, 50=0.08%
lat (msec) : 100=0.01%, 250=0.81%, 500=0.08%, 750=0.10%, 1000=0.01%
cpu : usr=0.37%, sys=1.55%, ctx=29338, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=29219/d=0, short=r=0/w=0/d=0
seq-write: (groupid=0, jobs=1): err= 0: pid=30216
write: io=113416KB, bw=1134.2KB/s, iops=283 , runt=100005msec
clat (usec): min=482 , max=1097.7K, avg=3513.74, stdev=27293.60
lat (usec): min=483 , max=1097.7K, avg=3516.88, stdev=27293.64
clat percentiles (usec):
| 1.00th=[ 548], 5.00th=[ 716], 10.00th=[ 772], 20.00th=[ 812],
| 30.00th=[ 844], 40.00th=[ 868], 50.00th=[ 892], 60.00th=[ 916],
| 70.00th=[ 940], 80.00th=[ 980], 90.00th=[ 1064], 95.00th=[ 1224],
| 99.00th=[116224], 99.50th=[201728], 99.90th=[497664], 99.95th=[518144],
| 99.99th=[602112]
bw (KB/s) : min= 3, max= 2480, per=33.41%, avg=1168.02, stdev=363.35
lat (usec) : 500=0.15%, 750=5.86%, 1000=77.70%
lat (msec) : 2=13.55%, 4=0.28%, 10=1.19%, 20=0.14%, 50=0.09%
lat (msec) : 100=0.03%, 250=0.84%, 500=0.08%, 750=0.09%, 2000=0.01%
cpu : usr=0.37%, sys=1.73%, ctx=28515, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=28354/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=349664KB, aggrb=3496KB/s, minb=1134KB/s, maxb=1193KB/s, mint=100001msec, maxt=100005msec

Disk stats (read/write):
sde: ios=1/87531, merge=0/14355, ticks=0/3385260, in_queue=2789384, util=100.00%

-------- local ----
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 3 processes
seq-write: Laying out IO file(s) (1 file(s) / 1024MB)
seq-write: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 3 (f=3): [WWW] [100.0% done] [0K/14566K /s] [0 /3556 iops] [eta 00m:00s]
seq-write: (groupid=0, jobs=1): err= 0: pid=30222
write: io=438048KB, bw=4380.5KB/s, iops=1095 , runt=100001msec
clat (usec): min=159 , max=679274 , avg=904.70, stdev=12497.03
lat (usec): min=160 , max=679276 , avg=906.94, stdev=12497.04
clat percentiles (usec):
| 1.00th=[ 209], 5.00th=[ 221], 10.00th=[ 225], 20.00th=[ 229],
| 30.00th=[ 231], 40.00th=[ 235], 50.00th=[ 239], 60.00th=[ 243],
| 70.00th=[ 249], 80.00th=[ 262], 90.00th=[ 298], 95.00th=[ 402],
| 99.00th=[ 812], 99.50th=[ 3376], 99.90th=[224256], 99.95th=[301056],
| 99.99th=[337920]
bw (KB/s) : min= 111, max= 8744, per=34.68%, avg=4475.22, stdev=1731.04
lat (usec) : 250=70.03%, 500=26.71%, 750=2.13%, 1000=0.30%
lat (msec) : 2=0.19%, 4=0.16%, 10=0.12%, 20=0.07%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.18%, 500=0.08%, 750=0.01%
cpu : usr=0.58%, sys=3.42%, ctx=109670, majf=0, minf=23
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=109512/d=0, short=r=0/w=0/d=0
seq-write: (groupid=0, jobs=1): err= 0: pid=30223
write: io=434468KB, bw=4344.7KB/s, iops=1086 , runt=100001msec
clat (usec): min=179 , max=691006 , avg=911.99, stdev=12768.08
lat (usec): min=180 , max=691007 , avg=914.21, stdev=12768.08
clat percentiles (usec):
| 1.00th=[ 207], 5.00th=[ 221], 10.00th=[ 225], 20.00th=[ 231],
| 30.00th=[ 233], 40.00th=[ 237], 50.00th=[ 239], 60.00th=[ 245],
| 70.00th=[ 251], 80.00th=[ 262], 90.00th=[ 302], 95.00th=[ 402],
| 99.00th=[ 756], 99.50th=[ 2128], 99.90th=[238592], 99.95th=[301056],
| 99.99th=[342016]
bw (KB/s) : min= 5, max= 8600, per=34.71%, avg=4479.71, stdev=1762.32
lat (usec) : 250=69.17%, 500=27.74%, 750=2.05%, 1000=0.29%
lat (msec) : 2=0.20%, 4=0.11%, 10=0.10%, 20=0.05%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.17%, 500=0.09%, 750=0.01%
cpu : usr=0.68%, sys=3.71%, ctx=108938, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=108617/d=0, short=r=0/w=0/d=0
seq-write: (groupid=0, jobs=1): err= 0: pid=30224
write: io=418116KB, bw=4181.2KB/s, iops=1045 , runt=100001msec
clat (usec): min=174 , max=577159 , avg=948.32, stdev=13147.12
lat (usec): min=175 , max=577160 , avg=950.53, stdev=13147.13
clat percentiles (usec):
| 1.00th=[ 209], 5.00th=[ 221], 10.00th=[ 225], 20.00th=[ 229],
| 30.00th=[ 231], 40.00th=[ 235], 50.00th=[ 237], 60.00th=[ 243],
| 70.00th=[ 249], 80.00th=[ 262], 90.00th=[ 294], 95.00th=[ 402],
| 99.00th=[ 772], 99.50th=[ 3120], 99.90th=[252928], 99.95th=[305152],
| 99.99th=[354304]
bw (KB/s) : min= 105, max= 8640, per=33.47%, avg=4319.83, stdev=1646.15
lat (usec) : 250=71.38%, 500=25.58%, 750=1.98%, 1000=0.28%
lat (msec) : 2=0.20%, 4=0.11%, 10=0.11%, 20=0.06%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.17%, 500=0.10%, 750=0.01%
cpu : usr=0.72%, sys=3.02%, ctx=104679, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=104529/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=1260.4MB, aggrb=12906KB/s, minb=4181KB/s, maxb=4380KB/s, mint=100001msec, maxt=100001msec

Disk stats (read/write):
sda: ios=2583/323195, merge=14/28603, ticks=24160/4690956, in_queue=5321588, util=100.00%
rm: cannot remove `/tmp/iseq-write.*': No such file or directory
-------- end ----
sles11:~ #


なんともいえないのですは、この iSCSI ドライブは大体ローカルディスクの２割から３割程度のパフォーマンスしか出ていないようです。

ははーん、ボトルネックはここなんだ、と思うのは早計かも知れません。fioはあくまでもディスクの性能を測るものなので、システム全体の性能とは直結しているわけではありません。実際エンドユーザが利用したときのレスポンスとかが重要です。エンドユーザサイドで　iSCSI で恐ろしいパフォーマンスを出す機器もあれば、円盤そのものの性能や、SAN装置のキャッシュの容量、RAIDコントローラの性能など色々な要素があるのです。


iSCSI は SUSE 仮想環境でファイルサーバとして使い物になるか

仮想環境では、iSCSI は容易に容量を拡張できる手段です。実際に２年前にこのエントリを書いたときからあまり環境が変わっていないことを考えると、ディスクの性能＝システムの性能とは一概には言えません。

機器を納品した後、こうして初期性能を計っておくことはひとつの問題解決の準備段階として重要です。この数値を元に、後にクレームがきた場合のシステム全体の性能が低下したかどうかの基準として利用できるかも知れません。

一度、Raid の故障の後、性能が異常に低下したことがありました。原因はディスクのファームウェアにあったのですが、初期性能が正しく測定されており、機器の故障修理の後、同じパフォーマンスが出ているかを確認することも必要です。

ちなみに　SAS Raid の fio の参考値　もご覧ください。

※ ちなみに fio は実稼動しているサーバーでは実行しないでください。CPU負荷が大きなWindows 系の仮想マシンが全滅しました。 Linux 系仮想マシンは全部無事だったけど。ユーザの負荷が少ないときに実施してくださいね。

-Keyword-

仮想環境 SUSE Linux SLES XEN iSCSI fio ディスクベンチマーク

islandcenter.jp