RGW performance with FileStore and MemStore

Posted on | In | |

概述

针对我们的应用，ceph使用Filestore和Memstore的性能有多大差距呢？我们通过ceph的rest-bench工具对之进行了性能对比测试。

FileStore

分两次的测试结果如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# rest-bench --proxy=localhost --proxy-port=36666 --bucket=bug_fix --api-host=localhost --access-key=LC5O0HVGBZ1RJXBW4FZ4 --secret=2MsVUZ+h4lTrSAUx9ZCY4D4a874X+MPULzlUc0RB write
...
2015-05-21 15:08:16.741796min lat: 0.158293 max lat: 1.01379 avg lat: 0.359322
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    60      16      2681      2665   177.642       168   0.31313  0.359322
 Total time run:         60.523804
Total writes made:      2681
Write size:             4194304
Bandwidth (MB/sec):     177.186

Stddev Bandwidth:       30.3802
Max bandwidth (MB/sec): 208
Min bandwidth (MB/sec): 0
Average Latency:        0.360339
Stddev Latency:         0.131017
Max latency:            1.01379
Min latency:            0.158293
In sync_write(), oid: benchmark_last_metadata

# rest-bench --proxy=localhost --proxy-port=36666 --bucket=bug_fix --api-host=localhost --access-key=LC5O0HVGBZ1RJXBW4FZ4 --secret=2MsVUZ+h4lTrSAUx9ZCY4D4a874X+MPULzlUc0RB write
...
2015-05-21 15:18:32.523700min lat: 0.169844 max lat: 1.1016 avg lat: 0.338614
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    60      16      2842      2826   188.376       188  0.197176  0.338614
 Total time run:         60.473186
Total writes made:      2842
Write size:             4194304
Bandwidth (MB/sec):     187.984

Stddev Bandwidth:       27.4199
Max bandwidth (MB/sec): 208
Min bandwidth (MB/sec): 0
Average Latency:        0.339906
Stddev Latency:         0.131774
Max latency:            1.1016
Min latency:            0.169844
In sync_write(), oid: benchmark_last_metadata

MemStore

参考文章配置memstore：How to configure ceph use memstore?

分两次的测试结果如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# rest-bench --proxy=localhost --proxy-port=36666 --bucket=bug_fix --api-host=localhost --access-key=GR73W0UWWQ6U43DUHVCC --secret= GzgWGEp0hO0JdfdbrxLuopCv+ufHyHYgFSgjvU0U write
...
2015-05-21 16:03:21.806960min lat: 0.069719 max lat: 1.45503 avg lat: 0.302808
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    60      16      3177      3161   210.701       224  0.193723  0.302808
 Total time run:         60.187152
Total writes made:      3178
Write size:             4194304
Bandwidth (MB/sec):     211.208

Stddev Bandwidth:       34.5447
Max bandwidth (MB/sec): 244
Min bandwidth (MB/sec): 0
Average Latency:        0.302945
Stddev Latency:         0.101357
Max latency:            1.45503
Min latency:            0.069719
In sync_write(), oid: benchmark_last_metadata

# rest-bench --proxy=localhost --proxy-port=36666 --bucket=bug_fix --api-host=localhost --access-key=GR73W0UWWQ6U43DUHVCC --secret= GzgWGEp0hO0JdfdbrxLuopCv+ufHyHYgFSgjvU0U write
...
2015-05-21 16:04:45.235222min lat: 0.080655 max lat: 1.23022 avg lat: 0.287523
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    60      16      3344      3328   221.834       204  0.295268  0.287523
 Total time run:         60.175814
Total writes made:      3345
Write size:             4194304
Bandwidth (MB/sec):     222.348

Stddev Bandwidth:       30.089
Max bandwidth (MB/sec): 240
Min bandwidth (MB/sec): 0
Average Latency:        0.287704
Stddev Latency:         0.0919689
Max latency:            1.23022
Min latency:            0.080655
In sync_write(), oid: benchmark_last_metadata

测试结果

总结上述各两次的测试结果如下：

type filestore memstore
Bandwidth (MB/s) 177.186 211.208
187.984 222.348
Average Latency 0.360339 0.339906
0.302945 0.287704

从上述表格中看出，使用Memstore能提升约20%的Bandwidth，能提升约10%的Latency，相比于内存 vs 磁盘的性能对比，Memstore vs Filestore的提升不是很明显。

支持原创