GlusterFS是一种分布式分布式文件系统，默认采用无中心完全对等架构，搭建维护使用十分简单，是很受欢迎的分布式文件系统。
官网https://www.gluster.org/，官网上表示Gluster 5是最新版本，点进去发现CentOS 8的，有点超前，不过RHCE8测试版1发布了。那么我们还是要用长期支持版本Gluster 4.1了。
1.基本环境的准备
1 在三台机器上都执行 2 [root@g1 ~]# yum install -y centos-release-gluster glusterfs-server glusterfs-fuse glusterfs-rdma glusterfs 3 [root@g1 ~]# ls /etc/yum.repos.d/ 4 CentOS7-Base-163.repo CentOS-Base.repo CentOS-CR.repo CentOS-Debuginfo.repo CentOS-fasttrack.repo CentOS-Gluster-4.1.repo CentOS-Media.repo CentOS-Sources.repo CentOS-Storage-common.repo CentOS-Vault.repo epel.repo salt-latest.repo 5 [root@g1 ~]# cat /etc/hosts 6 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 7 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 8 9 192.168.56.11 g1 10 192.168.56.12 g2 11 192.168.56.13 g3 12 [root@g1 ~]# mkdir -p /glusterfs/data{1,2,3} # 我直接使用这三个目录了，因为虚拟机 13 [root@g1 ~]# systemctl start glusterfsd.service 14 [root@g1 ~]# systemctl enable glusterfsd.service 15 正规情况请先格式化磁盘并分别挂载 16 [root@g1 ~]# mkfs.xfs -i size=512 /dev/sdb 17 [root@g1 ~]# echo "/dev/sdb /glusterfs/data1 xfs defaults 1 2" >>/etc/fstab
2.池相关
我们将每一台机器看做一个池，池会提供后续块资源（某个磁盘空间），若干个池组成一个glusterfs集群
1 在任意一台机器上 2 [root@g1 ~]# gluster pool list # 只有自己一个 3 UUID Hostname State 4 ce160a74-6f02-4760-862c-1daf6cfa4300 localhost Connected 5 [root@g1 ~]# gluster peer status # 没有同等的小伙伴 6 Number of Peers: 0 7 [root@g1 ~]# gluster peer probe g2 # 将g2添加进来，这里主机名要解析，当然直接写ip添加也是可以的 8 peer probe: success. 9 [root@g1 ~]# gluster peer probe g3 # 将g3添加进来，这里注意不需要添加自己 10 peer probe: success. 11 [root@g1 ~]# gluster peer status # 此时集群内对等的池数量有2个 12 Number of Peers: 2 13 14 Hostname: g2 15 Uuid: 1da148ac-0c81-4434-830e-ab3f66c046ea 16 State: Peer in Cluster Connected) 17 18 Hostname: g3 19 Uuid: aff7232f-a731-4def-b996-db3723c1fc97 20 State: Peer in Cluster Connected) 21 [root@g1 ~]# gluster pool list # 池则为3个 22 UUID Hostname State 23 1da148ac-0c81-4434-830e-ab3f66c046ea g2 Connected 24 aff7232f-a731-4def-b996-db3723c1fc97 g3 Connected 25 ce160a74-6f02-4760-862c-1daf6cfa4300 localhost Connected 26 [root@g1 ~]# gluster peer detach g3 # 从集群中去掉一个池
3.卷相关
池是集群内部资源，无法直接提供给外部使用，我们将池中的块资源组成一个卷，卷是对外提供挂载使用的。一个集群可以提供多个卷。
一、分布卷: 将文件已hash算法随机分布到 一台服务器节点中存储
二、复制卷: 将文件复制到 replica x 个节点中。
三、条带卷：将文件切割成数据块，分别存储到 stripe x 个节点中，类似于raid0。条带卷的理念很好，将大文件分散至若干个磁盘后提升IO速度，但是实际情况中偶尔会出现丢失块情况，因此与条带卷相关的所以卷类型在生产都不可以使用！！！
四、分布式条带卷：分布卷与条带卷的组合。生产不使用。
五、分布式复制卷：分布卷与复制卷的组合。这是我们最常用的类型！！！
六、条带复制卷：条带卷与复制卷的组合。生产不使用。
七、混合卷：三种卷模式的组合。生产不使用。
1 [root@g1 ~]# gluster volume list # 目前集群没有卷 2 No volumes present in cluster 3 [root@g1 ~]# gluster volume create test replica 2 g1:/glusterfs/data1 g2:/glusterfs/data1 g3:/glusterfs/data1 g1:/glusterfs/data2 g2:/glusterfs/data2 # 我们用池内5块盘创建一个分布式复制卷，复制数为2 4 Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/. 5 Do you still want to continue? 6 y/n) y 7 number of bricks is not a multiple of replica count 8 9 Usage: 10 volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force] # 一共出现了2次问题，第一次提示说，复制数为2可能会造成脑裂，推荐复制三份，是否继续？生产要使用3，现在我就继续了。第二个说使用的块设备数量不是复制分数的倍数。因此像复制数为2的卷我们需要使用2468个块，复制数为3则为369，以此类推 11 [root@g1 ~]# gluster volume create test replica 2 g1:/glusterfs/data1 g2:/glusterfs/data1 g3:/glusterfs/data1 g1:/glusterfs/data2 12 Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/. 13 Do you still want to continue? 14 y/n) y 15 volume create: test: failed: The brick g1:/glusterfs/data1 is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior. # 第一个还是提示警告，第二个是因为我直接使用了/下的目录并不是挂载的盘，所以会有这个提示，正常挂载的磁盘不会有此报错 16 [root@g1 ~]# gluster volume create test replica 2 g1:/glusterfs/data1 g2:/glusterfs/data1 g3:/glusterfs/data1 g1:/glusterfs/data2 force # 强制操作一波 17 volume create: test: success: please start the volume to access data 18 [root@g1 ~]# gluster volume list # 可以看见我们的集群中存在的卷 19 test 20 [root@g1 ~]# gluster volume info test # 查看卷的信息 21 22 Volume Name: test 23 Type: Distributed-Replicate 24 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 25 Status: Created 26 Snapshot Count: 0 27 Number of Bricks: 2 x 2 = 4 28 Transport-type: tcp 29 Bricks: 30 Brick1: g1:/glusterfs/data1 31 Brick2: g2:/glusterfs/data1 32 Brick3: g3:/glusterfs/data1 33 Brick4: g1:/glusterfs/data2 34 Options Reconfigured: 35 transport.address-family: inet 36 nfs.disable: on 37 performance.client-io-threads: off 38 [root@g1 ~]# gluster volume start test # 将卷启动 39 volume start: test: success 40 [root@g1 ~]# gluster volume status test # 查看卷的状态 41 Status of volume: test 42 Gluster process TCP Port RDMA Port Online Pid 43 ------------------------------------------------------------------------------ 44 Brick g1:/glusterfs/data1 49152 0 Y 81315 45 Brick g2:/glusterfs/data1 49152 0 Y 72882 46 Brick g3:/glusterfs/data1 49152 0 Y 72834 47 Brick g1:/glusterfs/data2 49153 0 Y 81337 48 Self-heal Daemon on localhost N/A N/A Y 81360 49 Self-heal Daemon on g2 N/A N/A Y 72905 50 Self-heal Daemon on g3 N/A N/A Y 72857 51 52 Task Status of Volume test 53 ------------------------------------------------------------------------------ 54 There are no active volume tasks 55 [root@g1 ~]# netstat -tpln|grep gluster 56 tcp 0 0 0.0.0.0:49152 0.0.0.0:* LISTEN 81315/glusterfsd # 为磁盘打开的通讯端口 57 tcp 0 0 0.0.0.0:49153 0.0.0.0:* LISTEN 81337/glusterfsd # 为磁盘打开的通讯端口 58 tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 7381/glusterd # 服务本身 59 [root@g1 ~]# ps -ef|grep 81315 60 root 81315 1 0 17:12 ? 00:00:00 /usr/sbin/glusterfsd -s g1 --volfile-id test.g1.glusterfs-data1 -p /var/run/gluster/vols/test/g1-glusterfs-data1.pid -S /var/run/gluster/643869d38d7edd70.socket --brick-name /glusterfs/data1 -l /var/log/glusterfs/bricks/glusterfs-data1.log --xlator-option *-posix.glusterd-uuid=ce160a74-6f02-4760-862c-1daf6cfa4300 --process-name brick --brick-port 49152 --xlator-option test-server.listen-port=49152 61 root 81569 96704 0 17:14 pts/1 00:00:00 grep --color=auto 81315 62 63 服务端在默认情况卷已经打开了很多的优化参数，我们只需要根据服务器自身资源修改一些值即可 64 [root@g1 ~]# gluster volume set test performance.cache-size 1GB # 读缓存大小1G，如果此值超过机器内存，则在挂载时会报错 65 volume set: success 66 [root@g1 ~]# gluster volume set test performance.io-thread-count 32 # 设置io线程 67 volume set: success 68 [root@g1 ~]# gluster volume quota test enable # 打开限额功能，内部机房一般不用 69 quota command failed : Quota is already enabled 70 [root@g1 ~]# gluster volume quota test limit-usage / 10GB # 设置最多可使用量 71 volume quota : success 72 [root@g1 ~]# gluster volume set test performance.cache-refresh-timeout 5 # 设置缓存刷新时间 73 volume set: success 74 [root@g1 ~]# gluster volume info test 75 76 Volume Name: test 77 Type: Distributed-Replicate 78 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 79 Status: Started 80 Snapshot Count: 0 81 Number of Bricks: 2 x 2 = 4 82 Transport-type: tcp 83 Bricks: 84 Brick1: g1:/glusterfs/data1 85 Brick2: g2:/glusterfs/data1 86 Brick3: g3:/glusterfs/data1 87 Brick4: g1:/glusterfs/data2 88 Options Reconfigured: 89 performance.cache-refresh-timeout: 5 90 performance.io-thread-count: 32 91 performance.cache-size: 1GB 92 features.quota-deem-statfs: on 93 nfs.disable: on 94 features.inode-quota: on 95 features.quota: on 96 97 98 3.挂载相关 99 我们在一台可以进行主机名解析的机器上进行挂载测试 100 [root@c4 /]# yum install -y glusterfs glusterfs-fuse 101 [root@c4 /]# mount -t glusterfs -o backup-volfile-servers=g2:g3,log-level=WARNING g1:/test /mnt # 挂载时指定备用服务器 102 [root@c4 /]# df -h|grep mnt # 很明显我们的限额起了作用 103 g1:/test 10G 0 10G 0% /mnt 104 [root@c4 ~]# cd /mnt 105 [root@c4 mnt]# touch file{1..10} 106 可以看到确实均匀分布在了我们卷里的4块盘之上 107 [root@g1 glusterfs]# tree . 108 . 109 ├── data1 110 │ ├── file10 111 │ ├── file3 112 │ ├── file4 113 │ ├── file7 114 │ └── file9 115 ├── data2 116 │ ├── file1 117 │ ├── file2 118 │ ├── file5 119 │ ├── file6 120 │ └── file8 121 └── data3 122 [root@g2 glusterfs]# tree . 123 . 124 ├── data1 125 │ ├── file10 126 │ ├── file3 127 │ ├── file4 128 │ ├── file7 129 │ └── file9 130 ├── data2 131 └── data3 132 [root@g3 glusterfs]# tree . 133 . 134 ├── data1 135 │ ├── file1 136 │ ├── file2 137 │ ├── file5 138 │ ├── file6 139 │ └── file8 140 ├── data2 141 └── data3 142 当然我们还有nfs挂载与cifs挂载，只是这两种挂载我们几乎使用不到的，而且使用方法也很简单，就不多写了。使用常规的原生挂载方式是很不错的选择，可以获得更高的并发性能和透明的失效转移功能
4.运维相关
扩容操作
1 [root@g1 ~]# gluster peer probe g3 # 将新节点添加到集群里，如果是原本集群内的机器操作则省略 2 peer probe: success. Host g3 port 24007 already in peer list # 这台机器添加过了 3 [root@g1 ~]# gluster volume info test # 此时该卷块设备为4个 4 5 Volume Name: test 6 Type: Distributed-Replicate 7 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 8 Status: Started 9 Snapshot Count: 0 10 Number of Bricks: 2 x 2 = 4 11 Transport-type: tcp 12 Bricks: 13 Brick1: g1:/glusterfs/data1 14 Brick2: g2:/glusterfs/data1 15 Brick3: g3:/glusterfs/data1 16 Brick4: g1:/glusterfs/data2 17 [root@g1 ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 g3:/glusterfs/data3 18 volume add-brick: failed: Incorrect number of bricks supplied 5 with count 2 # 很明显又是之前的块设备与复制数备份问题。因此注意服务器上的磁盘数量要与卷复制数匹配问题，比如一个复制数为3的卷，买了10块盘其中一块是加不进来的 19 [root@g1 ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 20 volume add-brick: failed: The brick g1:/glusterfs/data3 is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior. # 我还是用的/所以要强制 21 [root@g1 ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 force 22 volume add-brick: success 23 [root@g1 ~]# gluster volume info test # 卷的块设备变多了 24 25 Volume Name: test 26 Type: Distributed-Replicate 27 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 28 Status: Started 29 Snapshot Count: 0 30 Number of Bricks: 4 x 2 = 8 31 Transport-type: tcp 32 Bricks: 33 Brick1: g1:/glusterfs/data1 34 Brick2: g2:/glusterfs/data1 35 Brick3: g3:/glusterfs/data1 36 Brick4: g1:/glusterfs/data2 37 Brick5: g2:/glusterfs/data2 38 Brick6: g3:/glusterfs/data2 39 Brick7: g1:/glusterfs/data3 40 Brick8: g2:/glusterfs/data3 41 [root@g1 ~]# gluster volume rebalance test start # 让以前的数据再次均匀分布 42 volume rebalance: test: success: Rebalance on test has been started successfully. Use rebalance status command to check status of the rebalance process. 43 ID: a2f4b603-283a-4303-8ad0-84db00adb5a5 44 [root@g1 ~]# gluster volume rebalance test status # 查看任务状态，要均衡文件较大时需要一段时间 45 Node Rebalanced-files size scanned failures skipped status run time in h:m:s 46 --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- 47 localhost 2 0Bytes 10 0 0 completed 0:00:00 48 g2 1 0Bytes 9 0 0 completed 0:00:00 49 g3 3 0Bytes 6 0 0 completed 0:00:00 50 volume rebalance: test: success 51 [root@g1 ~]# gluster volume rebalance test stop # 等所有状态completed就可以停了 52 Node Rebalanced-files size scanned failures skipped status run time in h:m:s 53 --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- 54 localhost 2 0Bytes 10 0 0 completed 0:00:00 55 g2 1 0Bytes 9 0 0 completed 0:00:00 56 g3 3 0Bytes 6 0 0 completed 0:00:00 57 volume rebalance: test: success: rebalance process may be in the middle of a file migration. 58 The process will be fully stopped once the migration of the file is complete. 59 Please check rebalance process for completion before doing any further brick related tasks on the volume. 60 61 [root@g1 ~]# gluster volume rebalance test status # 现在该卷上已经没有在均衡的任务了 62 volume rebalance: test: failed: Rebalance not started for volume test.
缩容操作
1 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 start # 去掉复制倍数的块设备，此时去除的设备上的数据会开始迁移 2 Running remove-brick with cluster.force-migration enabled can result in data corruption. It is safer to disable this option so that files that receive writes during migration are not migrated. 3 Files that are not migrated can then be manually copied after the remove-brick commit operation. 4 Do you want to continue with your current cluster.force-migration settings? y/n) y 5 volume remove-brick start: success 6 ID: 9b4657c0-ed29-4c75-8bb6-7b8f277f02ec 7 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 status # 查看迁移状态 8 Node Rebalanced-files size scanned failures skipped status run time in h:m:s 9 --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- 10 localhost 0 0Bytes 10 0 0 completed 0:00:00 11 g2 0 0Bytes 5 0 0 completed 0:00:00 12 g3 0 0Bytes 5 0 0 completed 0:00:00 13 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 commit # completed后从卷中将块设备去除掉 14 volume remove-brick commit: success 15 Check the removed bricks to ensure all files are migrated. 16 If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. 17 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 status # 没有任务 18 volume remove-brick status: failed: remove-brick not started for volume test. 19 [root@g1 ~]# gluster volume info test # 又变回了以前的4块盘 20 21 Volume Name: test 22 Type: Distributed-Replicate 23 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 24 Status: Started 25 Snapshot Count: 0 26 Number of Bricks: 2 x 2 = 4 27 Transport-type: tcp 28 Bricks: 29 Brick1: g1:/glusterfs/data1 30 Brick2: g2:/glusterfs/data1 31 Brick3: g3:/glusterfs/data1 32 Brick4: g1:/glusterfs/data2 33 [root@g1 ~]# ls -a /glusterfs/data3 # 将刚才所有被去除的块设备下隐藏目录删除，否则可能会影响到以后再次使用此块设备加入其它卷 34 . .. .glusterfs 35 [root@g1 ~]# rm -fr /glusterfs/data3/.glusterfs/
更换操作
1 [root@g1 ~]# gluster volume replace-brick test g3:/glusterfs/data2 g3:/glusterfs/data3 commit force # 使用g3:/glusterfs/data3替换掉g3:/glusterfs/data2 2 volume replace-brick: success: replace-brick commit force operation successful 3 [root@g1 ~]# gluster volume info test 4 5 Volume Name: test 6 Type: Distributed-Replicate 7 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 8 Status: Started 9 Snapshot Count: 0 10 Number of Bricks: 4 x 2 = 8 11 Transport-type: tcp 12 Bricks: 13 Brick1: g1:/glusterfs/data1 14 Brick2: g2:/glusterfs/data1 15 Brick3: g3:/glusterfs/data1 16 Brick4: g1:/glusterfs/data2 17 Brick5: g2:/glusterfs/data2 18 Brick6: g3:/glusterfs/data3 # 被换掉了 19 Brick7: g1:/glusterfs/data3 20 Brick8: g2:/glusterfs/data3