一、简介
Openstack的后端存储对接Ceph分布式统一存储,将Glance镜像、Nova虚拟机及快照、Cinder-Volume的存储在Ceph统一存储上。Ceph在Openstack社区的带动下发展迅速,已经成为最受欢迎的开源Openstack后端存储,Ceph的相关内容将在后面的博客中进行学习总结,这里不在赘述。
安装简明过程:
- 在终端1中,SSH到控制节点,修改系统0-set-config.sh中关于ceph的配置,指定Mon、OSD、部署节点、存储网络、安装版本、安装源等相关信息;
- 在控制节点上执行install-configure-ceph-deploy.sh,提示执行/tmp/set-ssh-ceph-storage-nodes.sh时暂停操作;
- 新开一个终端2,SSH到存储部署节点,比如这里是compute01,执行/tmp/set-ssh-ceph-storage-nodes.sh完成部署节点到其他所有节点的SSH;
- 在新开的终端2上SSH到控制节点,执行test-ssh-compute-nodes.sh,检查部署节点到其他节点的SSH配置是否正确,如果正确继续下面操作;
- 回到终端1中,输入yes继续操作,完成ceph-deploy的安装;
- 在终端1中,继续执行 install-configure-ceph-storage-cluster.sh脚本,完成Ceph存储集群的安装;
- 查看安装后ceph集群状态,如果失败,查找原因,再次执行install-configure-ceph-storage-cluster.sh,脚本会自动卸载ceph并清空安装数据,必要时重启节点,通知内核读取最新磁盘分区表。
- 直到安装成功。
二、部署脚本
安装前请注意0-set-config.sh配置项是否正确
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
declare -A monitor_map=(["controller01"]="192.168.2.11" ["network01"]="192.168.2.14" ["compute01"]="192.168.2.17"); ### 计算节点默认部署节点(必须存在该主机名的节点) compute_host=compute01 ### 设置网络网段信息,分别对应管理网、虚拟网、存储网 store_network=11.11.11.0/24 ### 存储节点上OSD盘挂载目录 所有节点统一成一个 declare -A blks_map=(["osd01"]="sdb" ["osd02"]="sdc" ["osd03"]="sdd"); ### ceph安装版本 ceph_release=jewel ### 每个pool的pg_num,参考http://ceph.com/pgcalc/计算 pg_num_volumes=512 pg_num_images=1024 pg_num_backups=128 pg_num_vms=512 |
注意:对于每个pool的pg_num的设置参考http://ceph.com/pgcalc/,根据实际情况在线计算即可,这里不在赘述。
离线安装ceph-deploy,避免python包冲突,install-configure-ceph-deploy.sh
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
#!/bin/sh . ../0-set-config.sh ./style/print-split.sh "Ceph-deploy Installation" echo $compute_host scp ../0-set-config.sh $compute_host:/tmp scp ./set-ssh-ceph-storage-nodes.sh $compute_host:/tmp . ./style/print-warnning.sh "Have you execute \". /tmp/set-ssh-ceph-storage-nodes.sh\" to SSH in $compute_host! \nIf haven't, please press Ctrl+C to terminate this script and execute it in $compute_host!" . ./style/print-info.sh "continue?yes/Ctrl+C" read continue_tag base_location=../conf/wheel_ceph/ scp -r ../conf/wheel_ceph/ root@$compute_host:/tmp ssh root@$compute_host /bin/bash << EOF yum install --nogpgcheck -y epel-release rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 rm -rf /etc/yum.repos.d/epel* yum install -y python-pip yum install -y python-wheel pip install --use-wheel --no-index --trusted-host $(echo $ftp_info|awk -F "/" '{print $3}') --find-links=/tmp/wheel_ceph/ ceph-deploy ceph-deploy --version EOF |
配置存储部署节点到其他所有节点的SSH,配置脚本set-ssh-ceph-storage-nodes.sh,该脚本被自动拷贝至存储部署节点的/tmp/set-ssh-ceph-storage-nodes.sh目录下,需要手动SSH到该部署节点并执行该脚本,完成SSH配置,该脚本如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
#!/bin/sh . /tmp/0-set-config.sh nodes_name=(${!nodes_map[@]}); ### 重新设置/etc/hosts,切换主机名至ceph的存储网段 cp /etc/hosts /etc/hosts.bak2 ssh-keygen for ((i=0; i<${#nodes_map[@]}; i+=1)); do name=${nodes_name[$i]}; ip=${nodes_map[$name]}; echo "-------------$name------------" ssh-copy-id root@$ip ssh-copy-id root@$name ssh-copy-id root@$(echo $store_network|cut -d "." -f1-3).$(echo $ip|awk -F "." '{print $4}') done; |
SSH配置完成后,可以利用test-ssh-compute-nodes.sh脚本测试是否配置好存储部署节点到其他节点的SSH,test-ssh-compute-nodes.sh脚本如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
#!/bin/sh . ../0-set-config.sh ./style/print-split.sh "Test Compute Nodes SSH" scp pssh-exe root@$compute_host:/tmp/ scp -r ./hosts root@$compute_host:/tmp/ ssh root@$compute_host /bin/bash << EOF yum install -y pssh chmod +x /tmp/pssh-exe sed -i -e '/0-gen-hosts.sh/d' /tmp/pssh-exe cd /tmp /tmp/pssh-exe A date cp /etc/hosts /etc/hosts.bak2 sed -i -e 's#'"$(echo $local_network|cut -d "." -f1-3)"'#'"$(echo $store_network|cut -d "." -f1-3)"'#g' /etc/hosts EOF |
部署Ceph存储集群 install-configure-ceph-storage-cluster.sh,注意配置文件中OSD盘(一定是数据盘,千万不要选择系统盘,否则系统盘会被格式化)、Mon节点、Ceph存储网络的配置,一定要对应。
install-configure-ceph-storage-cluster.sh脚本如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
#!/bin/sh . ../0-set-config.sh ./style/print-split.sh "Ceph Storage Cluster Installation" base_location=$ftp_info deploy_node=$compute_host ### 获取Monitor信息,用于生成ceph配置文件 monitor_name=(${!monitor_map[@]}); mon_hostname="" mon_ip="" ### set mon nodes for ((i=0; i<${#monitor_map[@]}; i+=1)); do name=${monitor_name[$i]}; ip=${monitor_map[$name]}; if [ $name = $deploy_node ]; then echo $name" already is mon!" else mon_hostname=$mon_hostname","$name mon_ip=$mon_ip","$(echo $store_network|cut -d "." -f1-3).$(echo $ip|awk -F "." '{print $4}') fi done; #echo $mon_hostname" >>${#monitor_map[@]} "$mon_ip ### 获取OSD信息,用于生成并激活OSD blk_name=(${!blks_map[@]}); osds=""; echo $osds for ((i=0; i<${#nodes_map[@]}; i+=1)); do name=${nodes_name[$i]}; ip=${nodes_map[$name]}; for ((j=0; j<${#blks_map[@]}; j+=1)); do name2=${blk_name[$j]}; blk=${blks_map[$name2]}; osds=$osds" "$name":"$blk; done done echo $osds echo $deploy_node ###安装前所有OSD盘重置为裸盘 . ./style/print-warnning.sh "Do you want to zap/erase/destroy partition table and contents in all OSD Disks" . ./style/print-info.sh "Zap all?yes/no" read zap_tag for ((i=0; i<${#nodes_map[@]}; i+=1)); do name=${nodes_name[$i]}; ip=${nodes_map[$name]}; for ((j=0; j<${#blks_map[@]}; j+=1)); do name2=${blk_name[$j]}; blk=${blks_map[$name2]}; . style/print-info.sh "-------------$name:$blk------------"; if [[ "$zap_tag" = "yes" ]];then ssh root@$ip ceph-disk zap /dev/$blk ssh root@$ip partprobe fi done done . ./style/print-warnning.sh "Do you want to purge all previous ceph data before further installing?" . ./style/print-info.sh "Purge all?yes/no" read purge_tag ssh root@$compute_host /bin/bash << EOF sed -i -e 's#'"$(echo $local_network|cut -d "." -f1-3)"'#'"$(echo $store_network|cut -d "." -f1-3)"'#g' /etc/hosts ceph-deploy forgetkeys if [[ "$purge_tag" = "yes" ]];then ceph-deploy purge ${nodes_name[@]} ceph-deploy purgedata ${nodes_name[@]} fi mkdir -p /root/my-cluster cd /root/my-cluster rm -rf /root/my-cluster/* ceph-deploy new $deploy_node echo "public network ="$store_network>>ceph.conf ceph-deploy install --nogpgcheck --repo-url $base_location/download.ceph.com/rpm-$ceph_release/el7/ ${nodes_name[@]} --gpg-url $base_location/download.ceph.com/release.asc ceph-deploy mon create-initial ceph-deploy osd create $osds ceph-deploy admin ${nodes_name[@]} EOF ## set mon nodes for ((i=0; i<${#monitor_map[@]}; i+=1)); do name=${monitor_name[$i]}; ip=${monitor_map[$name]}; . style/print-info.sh "Set $name as a ceph monitor" if [ $name = $deploy_node ]; then echo $name" already is mon!" else ssh root@$deploy_node /bin/bash << EOF cd /root/my-cluster ceph-deploy mon add $name EOF fi done; ###查看集群状态 ceph管理节点创建Pool ssh root@$deploy_node /bin/bash << EOF cd /root/my-cluster sed -i -e 's#'"$(ssh root@$compute_host cat /root/my-cluster/ceph.conf |grep mon_initial_members)"'#'"$(ssh root@$compute_host cat /root/my-cluster/ceph.conf |grep mon_initial_members)$mon_hostname"'#g' /root/my-cluster/ceph.conf sed -i -e 's#'"$(ssh root@$compute_host cat /root/my-cluster/ceph.conf |grep mon_host )"'#'"$(ssh root@$compute_host cat /root/my-cluster/ceph.conf |grep mon_host )$mon_ip"'#g' /root/my-cluster/ceph.conf ceph-deploy --overwrite-conf config push ${nodes_name[@]} ceph -s ceph osd pool create volumes $pg_num_volumes ceph osd pool create images $pg_num_images ceph osd pool create backups $pg_num_backups ceph osd pool create vms $pg_num_vms ### [恢复部署节点hosts文件] sed -i -e 's#'"$(echo $store_network|cut -d "." -f1-3)"'#'"$(echo $local_network|cut -d "." -f1-3)"'#g' /etc/hosts EOF |
注意:脚本安装时会提示是否清除之前的数据盘分区以及之前的安装数据,如果首次安装失败时,重复安装需要执行磁盘zap及数据purge清除操作。
三、参考文档
http://docs.ceph.com/docs/master/start/quick-ceph-deploy/
四、源码
脚本源码:https://github.com/zjmeixinyanzhi/Openstack-HA-Install-Shells
五、系列文章
Openstack云平台脚本部署之Galera高可用集群配置(二)
Openstack云平台脚本部署之RabbitMQ高可用集群部署(三)
Openstack云平台脚本部署之Memcached配置(五)
Openstack云平台脚本部署之Keystone认证服务配置(六)
Openstack云平台脚本部署之Glance镜像服务配置(七)
Openstack云平台脚本部署之Nova计算服务配置(八)
Openstack云平台脚本部署之Neutron网络服务配置(九)
Openstack云平台脚本部署之Dashboard配置(十)
Openstack云平台脚本部署之Cinder块存储服务配置(十一)
Openstack云平台脚本部署之Ceilometer数据收集服务配置(十二)
Openstack云平台脚本部署之Aodh告警服务配置(十三)
code
more code
~~~~