RDS配置使用Thin Provision LVM

概述

在RDS的存储中,我们使用LVM卷给Docker container使用,默认LVM是不使用thin的feature的;在LVM 的 2.02.89 版本后,引入了Thin Pool的概念,开始支持创建Thin Provision的volume;

我们线上的LVM版本为:

1
2
3
4
openstack@trove-server1:~$ sudo lvm version
LVM version: 2.02.98(2) (2012-10-15)
Library version: 1.02.77 (2012-10-15)
Driver version: 4.33.0

所以我们这里调整LVM的配置,来支持创建的volume为Thin Provision的volume,提高设备的资源使用率

LVM创建Thin Provision Volume

创建Thin Pool

1
2
3
4
5
6
7
8
9
root@trove-server1:/home/openstack# vgs
VG #PV #LV #SN Attr VSize VFree
volume-group-pcie 1 2 0 wz--n- 2.91t 2.61t
volume-group-sata 2 1 0 wz--n- 7.28t 372.44g
volume-group-ssd 3 1 0 wz--n- 1.85t 94.59g
root@trove-server1:/home/openstack# lvcreate -L 7168G --thinpool vg-sata-thin-pool volume-group-sata
root@trove-server1:/home/openstack# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
vg-sata-thin-pool volume-group-sata twi-a-tz- 7.00t 0.00

查看Thin Pool

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@trove-server1:/home/openstack# lvdisplay volume-group-sata/vg-sata-thin-pool
--- Logical volume ---
LV Name volume-group-sata-pool
VG Name volume-group-sata
LV UUID A3N8Bs-7mVk-VaAU-MwJp-J5RD-Dlfp-0pIHzK
LV Write Access read/write
LV Creation host, time trove-server1, 2016-11-01 14:11:12 +0800
LV Pool transaction ID 0
LV Pool metadata volume-group-sata-pool_tmeta
LV Pool data volume-group-sata-pool_tdata
LV Pool chunk size 4.00 MiB
LV Zero new blocks yes
LV Status available
# open 0
LV Size 7.00 TiB
Allocated pool data 0.00%
Allocated metadata 0.32%
Current LE 1812081
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:10

创建Thin Volume

1
2
3
4
5
6
root@trove-server1:/home/openstack# lvcreate -V 5G --thin -n thin_vol_test volume-group-sata/vg-sata-thin-pool
Logical volume "thin_vol_test" created
root@trove-server1:/home/openstack# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
thin_vol_test volume-group-sata Vwi-a-tz- 5.00g vg-sata-thin-pool 0.00
vg-sata-thin-pool volume-group-sata twi-a-tz- 7.00t 0.00

从上述的Data%中可以看出,创建5.00G的volume,vg-sata-thin-pool的Data%为0.00;

删除Thin Volume

1
2
3
root@trove-server1:/home/openstack# lvremove volume-group-sata/thin_vol_test
Do you really want to remove and DISCARD active logical volume thin_vol_test? [y/n]: y
Logical volume "thin_vol_test" successfully removed

Thin Volume创建时间

测试下Thin Volume的创建时间是否与volume size有关系

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
root@BJ-BGP03-001-001:/home/openstack/mikeyang# time lvcreate -V 30G --thin -n volume-test volume-group-sata/volume-group-sata-pool
Logical volume "volume-test" created
real 0m0.378s
user 0m0.008s
sys 0m0.028s

root@BJ-BGP03-001-001:/home/openstack/mikeyang# time lvcreate -V 300G --thin -n volume-test volume-group-sata/volume-group-sata-pool
Logical volume "volume-test" created
real 0m0.359s
user 0m0.000s
sys 0m0.036s

root@BJ-BGP03-001-001:/home/openstack/mikeyang# time lvcreate -V 1000G --thin -n volume-test volume-group-sata/volume-group-sata-pool
Logical volume "volume-test" created
real 0m0.374s
user 0m0.016s
sys 0m0.024s

从上述测试中可以看出,Thin Volume的创建时间很短,不受volume size大小影响

扩容Thin Pool

因为Thin Pool对应的其实也是一个LVM volume,所以通过lvextend命令即可扩容

1
2
3
4
5
6
7
8
# lvextend -L 2.5t volume-group-pcie/volume-group-pcie-pool
Extending logical volume volume-group-pcie-pool to 2.50 TiB
Logical volume volume-group-pcie-pool successfully resized

或者
# lvextend -L +1t volume-group-pcie/volume-group-pcie-pool
Extending logical volume volume-group-pcie-pool to 2.50 TiB
Logical volume volume-group-pcie-pool successfully resized

参考

https://linux.cn/article-4288-1.html
http://mathslinux.org/?p=379

Cinder支持创建Thin Provision volume

Cinder配置

查询Cinder代码,发现cinder是支持thin的lvm格式的;

需要在cinder中添加如下配置:

1
2
3
4
5
6
7
8
/opt/etc/cinder/cinder.conf
[lvm-sata]
volume_group = volume-group-sata
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=LVM_SATA
volume_clear = none
lvm_type = thin
...

如上述配置,只需要添加:lvm_type = thin 即可;LVM会自动创建名称为-pool的thin pool;

重启服务

1
openstack@trove-server1:~$ service cinder-api restart; service cinder-backup restart; service cinder-scheduler restart; service cinder-volume restart;

重启cinder服务后,测试创建RDS实例即可;

原有volumes迁移

修改为支持Thin Provision Volumes后,对之前创建的volume需要迁移到Thin Provision Volume上,这里给出迁移方案

在RDS的两个docker container中,一端一端的做volume的迁移;

Umount volume

1
2
3
4
5
6
7
8
9
10
11
openstack@trove-server1:~$ docker exec -it 583e869197cd bash
root@rds-vpie5bx2:/# df
Filesystem 1K-blocks Used Available Use% Mounted on
none 103081248 2906328 94915656 3% /
tmpfs 198079304 0 198079304 0% /dev
tmpfs 198079304 0 198079304 0% /sys/fs/cgroup
/dev/rbd0 51475068 54156 48783088 1% /var/log/mysql
/dev/rbd1 103081248 2906328 94915656 3% /var/log/rds/9709c88c-d392-4a52-8301-4135cadcdea6
shm 65536 0 65536 0% /dev/shm
/dev/vdb 5029504 118360 4649000 3% /var/lib/mysql
root@rds-vpie5bx2:/# umount /var/lib/mysql/

停止Docker Container

1
openstack@trove-server1:~$ docker stop 583e869197cd

rename原有volume

1
root@trove-server1:/home/openstack# lvrename volume-group-pcie/volume-ca880d25-c95f-4d19-993c-ee7ccfbf8d1c volume-group-pcie/backup-volume

创建新的Thin Provision Volume

1
root@trove-server1:/home/openstack# lvcreate -V 5G --thin -n volume-ca880d25-c95f-4d19-993c-ee7ccfbf8d1c volume-group-ssd/volume-group-ssd-pool

格式化新的Thin Provision Volume

1
root@trove-server1:/home/openstack# mkfs.ext3 /dev/volume-group-ssd/volume-ca880d25-c95f-4d19-993c-ee7ccfbf8d1c

volume数据迁移

1
2
3
4
5
6
root@trove-server1:/home/openstack# mount /dev/volume-group-ssd/volume-ca880d25-c95f-4d19-993c-ee7ccfbf8d1c /mnt/destination/
root@trove-server1:/home/openstack# mount /dev/volume-group-pcie/backup-volume /mnt/source/
root@trove-server1:/mnt/source# cp -r -p * /mnt/destination/
root@trove-server1:/mnt/source# cd /home/openstack/
root@trove-server1:/home/openstack# umount /mnt/source/
root@trove-server1:/home/openstack# umount /mnt/destination/

启动Docker Container

1
openstack@trove-server1:~$ nova reboot 9709c88c-d392-4a52-8301-4135cadcdea6

若报错,则先执行nova start ,然后再执行 nova reboot

检查trove实例状态

1
2
3
4
5
6
7
8
9
10
openstack@trove-server1:~$ docker exec -it 583e869197cd bash
root@rds-vpie5bx2:/# df
Filesystem 1K-blocks Used Available Use% Mounted on
none 103081248 2906328 94915656 3% /
tmpfs 198079304 0 198079304 0% /dev
tmpfs 198079304 0 198079304 0% /sys/fs/cgroup
/dev/rbd0 51475068 54156 48783088 1% /var/log/mysql
/dev/rbd1 103081248 2906328 94915656 3% /var/log/rds/9709c88c-d392-4a52-8301-4135cadcdea6
shm 65536 0 65536 0% /dev/shm
/dev/vdb 5029504 118360 4649000 3% /var/lib/mysql

若发现mysql没启动,可能是因为/var/lib/mysql/里的文件权限导致的,把/var/lib/mysql/的权限修改为mysql:mysql 即可;

注意事项

  1. 最好先迁移floating ip不在的trove实例上的volume;然后切换floating ip后,再迁移另外一个trove实例的volume;
  2. 最好在客户使用rds服务不频繁的时候做数据迁移;
  3. 迁移完一端的trove实例后,需要等待mysql为active-active状态后再迁移另外一端;
支持原创