radosgw配置user使用指定data pool

Posted on | In | |

概述

在使用radosgw提供s3服务的时候，有时候我们会需要指定user使用不同的data pool，或者不同的data index pool；

在Jewel版本里，我们可以通过配置placement target来实现；

首先介绍下Jewel版本里radosgw的几个概念：

缩略语 全称
Realm 域，同一域内的账户在其下属的zone上是通用的。一个域下只能有一个主zonegroup，从的zonegroup可以是0或多个。
Period 表示域的有效期，在域的结构发生变化时，其Period会相应变化。
Zonegroup Zone的集合，等价于之前的Region。一个Zonegroup下只能有一个主Zone。主Zone和从Zone可以部署在同一集群上，也可以部署在不同的集群上。
Zone 表示独立的一个对象存储区域。

代码解析

zone placement

Radosgw里的user bucket数据存放哪里，是有user里的placement确定的，每个placement info可以指定三个pool，如下所示：

1
2
3
4
5
6
7
struct RGWZonePlacementInfo {
    string index_pool;
    string data_pool;
    string data_extra_pool; /* if not set we should use data_pool */
    RGWBucketIndexType index_type;
...
}

每个zone里可以用多个placement，定义如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
struct RGWZoneParams : RGWSystemMetaObj {
    rgw_bucket domain_root;
    rgw_bucket metadata_heap;
    rgw_bucket control_pool;        // 每个zone里的下面一些pool是只能配置一个的
    rgw_bucket gc_pool;
    rgw_bucket log_pool;
    rgw_bucket intent_log_pool;
    rgw_bucket usage_log_pool;
    rgw_bucket user_keys_pool;
    rgw_bucket user_email_pool;
    rgw_bucket user_swift_pool;
    rgw_bucket user_uid_pool;
    RGWAccessKey system_key;
 
    map<string, RGWZonePlacementInfo> placement_pools;    // zone里支持配置多个placement
...
}

zonegroup placement

zonegroup里的placement target的定义如下，每个zonegroup可以配置多个placement targets，每个placement targets有name，tags（对应user info里的配置）标识；

这里的placement target name都对应着zone placement_pools里的string 关键字；

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
struct RGWZoneGroupPlacementTarget {
    string name;
    set<string> tags; // 可以有多个tags，对应RGWUserInfo里的placement_tags
...
};
  
struct RGWZoneGroup : public RGWSystemMetaObj {
    string api_name;
    list<string> endpoints;
    bool is_master;
 
    string master_zone;
    map<string, RGWZone> zones;       // 配置多个zones
 
    map<string, RGWZoneGroupPlacementTarget> placement_targets;       // zonegroup支持配置多个placement targets
    string default_placement;
...
};

user info

在每个user里，可以配置default placement，也支持配置多个placement tags；

user支持在创建bucket的时候指定placement；

1
2
3
4
5
6
7
8
9
struct RGWUserInfo
{
    uint64_t auid;
    rgw_user user_id;
...
    string default_placement;       // 指定默认的placement
    list<string> placement_tags;  // tags里的值需配置为placement target里tags集合里的值
...
};

操作步骤

下面通过增加一个test的placement target，让rgw user miketest使用pool default.rgw.buckets.data.tst 作为其bucket的data pool；

创建pool

根据需要通过ceph osd pool 命令创建要使用的pool，例如：

1
2
# ceph osd pool create default.rgw.buckets.data.tst 1024 1024
# ceph osd pool set <poolname> crush_ruleset 2    // 根据需要配置pool的crush rule

添加zone的placement

可以先list下当前的zone的placement

1
2
3
4
5
6
7
8
9
10
11
12
# radosgw-admin zone placement list  --rgw-zone=default
[
    {
        "key": "default-placement",
        "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.data",
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 0
        }
    }
]

zone添加placement的命令如下（指定zone placement的各个pool）：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# radosgw-admin zone placement add --rgw-zone=default --placement-id=test --index_pool=default.rgw.buckets.index --data_pool=default.rgw.buckets.data.tst --data_extra_pool=default.rgw.buckets.non-ec
[
    {
        "key": "default-placement",
        "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.data",
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 0
        }
    },
    {
        "key": "test",
        "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.data.tst",    // 这里仅仅指定了data_pool为新的pool
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 0
        }
    }
]

也可以通过如下命令来添加zone的placement：

1
2
3
# radosgw-admin zone get --rgw-zone=default > zone.info
# vim zone.info     // 添加 key为test的placement
# radosgw-admin zone set --rgw-zone=default < zone.info

添加zonegroup的placement

查看当前的zonegroup的placement

1
2
3
4
5
6
7
8
9
10
# radosgw-admin zonegroup placement list
[
    {
        "key": "default-placement",
        "val": {
            "name": "default-placement",
            "tags": []
        }
    }
]

zonegroup添加新的placement

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# radosgw-admin zonegroup placement add --rgw-zonegroup=default --placement-id=test
[
    {
        "key": "default-placement",
        "val": {
            "name": "default-placement",
            "tags": []
        }
    },
    {
        "key": "test",
        "val": {
            "name": "test",
            "tags": []
        }
    }
]

更新user info

查看user的信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# radosgw-admin user info --uid=miketest
{
    "user_id": "miketest",
    "display_name": "miketest",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "miketest",
            "access_key": "7J82YY2ODPQ22337N392",
            "secret_key": "IgRmuo11OrxRlWa7TEicHSbbUFNYP2MDwZ8St2PS"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "default-placement",
    "placement_tags": [
        "default-placement"
    ],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

导出user info，修改后导入新的信息，添加新的placement_tags：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# radosgw-admin metadata get user:miketest > miketest.userinfo
# vim miketest.userinfo
# radosgw-admin metadata put user:miketest < miketest.userinfo
# radosgw-admin user info --uid=miketest
{
    "user_id": "miketest",
    "display_name": "miketest",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "miketest",
            "access_key": "7J82YY2ODPQ22337N392",
            "secret_key": "IgRmuo11OrxRlWa7TEicHSbbUFNYP2MDwZ8St2PS"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "default-placement",   // 这里也可以配置default为 test placement
    "placement_tags": [
        "default-placement",
        "test"          // 添加了名为test的placement
    ],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

更新zonegroup map

更新完zonegroup的placement后，虽说也可以查看到新的placement，但这些并没有更新到zonegroup map里；

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# radosgw-admin zonegroup placement list
[
    {
        "key": "default-placement",
        "val": {
            "name": "default-placement",
            "tags": []
        }
    },
    {
        "key": "test",
        "val": {
            "name": "test",
            "tags": []
        }
    }
]
  
# radosgw-admin zonegroup-map get
{
    "zonegroups": [
        {
            "key": "342414eb-ee98-40b9-b9d5-971ad14d2dd0",
            "val": {
                "id": "342414eb-ee98-40b9-b9d5-971ad14d2dd0",
                "name": "default",
                "api_name": "",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "f31f020f-aaa2-4570-b6f7-19258064e5b5",
                "zones": [
                    {
                        "id": "f31f020f-aaa2-4570-b6f7-19258064e5b5",
                        "name": "default",
                        "endpoints": [],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [  // 看到这里并没有新加的test placement
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "7eeb01eb-3774-4abe-ba93-c205cc3c83af"
            }
        }
    ],
    "master_zonegroup": "342414eb-ee98-40b9-b9d5-971ad14d2dd0",
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    }
}

执行zonegroup map更新：radosgw-admin period update --commit

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
# radosgw-admin period update --commit
{
    "id": "bd50c829-f7eb-4aa9-ad0a-efa814dfc3cc",
    "epoch": 2,
    "predecessor_uuid": "da2e2b0d-a17a-46fb-8369-6cb373ffd08d",
    "sync_status": [
...
    ],
    "period_map": {
        "id": "bd50c829-f7eb-4aa9-ad0a-efa814dfc3cc",
        "zonegroups": [
            {
                "id": "342414eb-ee98-40b9-b9d5-971ad14d2dd0",
                "name": "default",
                "api_name": "",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "f31f020f-aaa2-4570-b6f7-19258064e5b5",
                "zones": [
                    {
                        "id": "f31f020f-aaa2-4570-b6f7-19258064e5b5",
                        "name": "default",
                        "endpoints": [],
                        "log_meta": "true",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    },
                    {
                        "name": "test", // 更新后有 test placement
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "7eeb01eb-3774-4abe-ba93-c205cc3c83af"
            }
        ],
        "short_zone_ids": [
            {
                "key": "f31f020f-aaa2-4570-b6f7-19258064e5b5",
                "val": 568702538
            }
        ]
    },
    "master_zonegroup": "342414eb-ee98-40b9-b9d5-971ad14d2dd0",
    "master_zone": "f31f020f-aaa2-4570-b6f7-19258064e5b5",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        }
    },
    "realm_id": "7eeb01eb-3774-4abe-ba93-c205cc3c83af",
    "realm_name": "default",
    "realm_epoch": 2
}

重启radosgw服务

重启radosgw服务，Centos里的命令为：

1
# systemctl restart ceph-radosgw.target

测试

通过s3cmd就可以测试，命令如下：

1
2
3
✗ s3cmd mb s3://foxtst --bucket-location=:test
Bucket 's3://foxtst/' created
✗ s3cmd put pg_by_osd.sh s3://foxtst

在ceph集群里，可以查看到上传的文件pg_by_osd.sh在指定的pool里：default.rgw.buckets.data.tst

1
2
# rados ls -p default.rgw.buckets.data.tst
f31f020f-aaa2-4570-b6f7-19258064e5b5.1224751.1_pg_by_osd.sh

参考

http://docs.ceph.com/docs/jewel/radosgw/multisite/
http://www.jianshu.com/p/31a6f8df9a8f
http://ceph.org.cn/2017/03/24/ceph-rgwj%E4%BB%A5%E4%B8%8A%E7%89%88%E6%9C%AC%E7%9A%84%E7%94%A8%E6%88%B7placement%E8%AE%BE%E7%BD%AE-by_%E7%A7%A6%E7%89%A7%E7%BE%8A/

支持原创