Kubernetes 系列（二十五）基于 velero & MinIO 对 kubernetes 进行备份和恢复
Velero 介绍
Velero 是由vmware公司团队研发开源工具，用于安全地备份、恢复和迁移 Kubernetes 集群和持久卷。它可以在本地和公共云中运行。Velero 由一个在您的 Kubernetes 集群中作为部署运行的服务器进程和一个命令行界面 (CLI) 组成，DevOps 团队和平台操作员可通过该界面配置计划备份、触发临时备份、执行恢复等。官网地址：https://velero.io/
Velero备份流程：
整体架构：
准备 MinIO 存储环境
安装 MinIO
docker pull minio/minio:latest
docker run -p 9000:9000 -p 9090:9090 \
--net=host \
--name minio \
-d --restart=always \
-e "MINIO_ACCESS_KEY=admin" \
-e "MINIO_SECRET_KEY=minio123" \
-v /home/minio/data:/data \
-v /home/minio/config:/root/.minio \
minio/minio server \
/data --console-address ":9090" -address ":9000"
[root@easzlab-minio-01 ~]# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:40963 0.0.0.0:* LISTEN 737/containerd
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 669/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 830/sshd: /usr/sbin
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 1437/sshd: root@pts
tcp6 0 0 :::9000 :::* LISTEN 5020/minio
tcp6 0 0 :::22 :::* LISTEN 830/sshd: /usr/sbin
tcp6 0 0 ::1:6010 :::* LISTEN 1437/sshd: root@pts
tcp6 0 0 :::9090 :::* LISTEN 5020/minio
[root@easzlab-minio-01 ~]#admin minio123
创建 velerodata buckets
在master-01节点部署velero
下载安装包
wget https://github.com/vmware-tanzu/velero/releases/download/v1.8.1/velero-v1.8.1-linux-amd64.tar.gz
tar -xvzf velero-v1.8.1-linux-amd64.tar.gz
mv velero /usr/local/bin配置 velero 认证环境
mkdir /data/velero -p
cd /data/velero
root@easzlab-k8s-master-01:/data/velero# cat velero-auth.txt
[default]
aws_access_key_id = admin
aws_secret_access_key = minio123将 velero 安装到 K8s 集群
root@easzlab-k8s-master-01:/data/velero# kubectl create ns velero-system
namespace/velero-system created
root@easzlab-k8s-master-01:/data/velero# velero --kubeconfig /root/.kube/config \
> install \
> --provider aws \
> --plugins velero/velero-plugin-for-aws:v1.3.1 \
> --bucket velerodata \
> --secret-file ./velero-auth.txt \
> --use-volume-snapshots=false \
> --namespace velero-system \
> --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://172.16.88.170:9000
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero-system: attempting to create resource
Namespace/velero-system: attempting to create resource client
Namespace/velero-system: already exists, proceeding
Namespace/velero-system: created
ClusterRoleBinding/velero-velero-system: attempting to create resource
ClusterRoleBinding/velero-velero-system: attempting to create resource client
ClusterRoleBinding/velero-velero-system: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ? Use 'kubectl logs deployment/velero -n velero-system' to view the status.
root@easzlab-k8s-master-01:/data/velero#root@easzlab-k8s-master-01:/data/velero# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c8bb696bb-fxbmr 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-2qtfm 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-8l78t 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-9b75m 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-k75jh 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-kmbhs 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-lxfk9 1/1 Running 1 (2d7h ago) 6d18h
kube-system coredns-69548bdd5f-6df7j 1/1 Running 1 (2d7h ago) 6d6h
kube-system coredns-69548bdd5f-nl5qc 1/1 Running 1 (2d7h ago) 6d6h
kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-2d275 1/1 Running 1 (2d7h ago) 6d6h
kubernetes-dashboard kubernetes-dashboard-5676d8b865-6l8n8 1/1 Running 1 (2d7h ago) 6d6h
linux70 linux70-tomcat-app1-deployment-5d666575cc-kbjhk 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-58ll2 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-6xcjk 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-cxg5m 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-gv2gk 1/1 Running 1 (2d7h ago) 4d7h
velero-system velero-858b9459f9-c7qc9 1/1 Running 0 2m29s
root@easzlab-k8s-master-01:/data/velero#测试备份功能
root@easzlab-k8s-master-01:~# DATE=`date +%Y%m%d%H%M%S`
root@easzlab-k8s-master-01:~# velero backup create default-backup-${DATE} --include-cluster-resources=true --include-namespaces kube-system --kubeconfig=/root/.kube/config --namespace velero-system
Backup request "default-backup-20220804202021" submitted successfully.
Run `velero backup describe default-backup-20220804202021` or `velero backup logs default-backup-20220804202021` for more details.
root@easzlab-k8s-master-01:~#
root@easzlab-k8s-master-01:~# velero backup create myserver-backup-${DATE} --include-cluster-resources=true --include-namespaces myserver --kubeconfig=/root/.kube/config --namespace velero-system
Backup request "myserver-backup-20220804202021" submitted successfully.
Run `velero backup describe myserver-backup-20220804202021` or `velero backup logs myserver-backup-20220804202021` for more details.
root@easzlab-k8s-master-01:~#检查是否备份成功
使用脚本备份
root@easzlab-k8s-master-01:~# cat velero-k8s-backup.sh
#!/bin/bash
NS_NAME=$(kubectl get ns |awk '{if(NR>1){print $1}}')
DATE=$(date +%Y%m%d%H%M%S)
cd /data/velero/
for i in $NS_NAME;do
velero backup create ${i}-ns-backup-${DATE} \
--include-cluster-resources=true \
--include-namespaces ${i} \
--kubeconfig=/root/.kube/config \
--namespace velero-system
done
root@easzlab-k8s-master-01:~#
root@easzlab-k8s-master-01:~# bash velero-k8s-backup.sh
Backup request "default-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe default-ns-backup-20220805165331` or `velero backup logs default-ns-backup-20220805165331` for more details.
Backup request "kube-node-lease-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe kube-node-lease-ns-backup-20220805165331` or `velero backup logs kube-node-lease-ns-backup-20220805165331` for more details.
Backup request "kube-public-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe kube-public-ns-backup-20220805165331` or `velero backup logs kube-public-ns-backup-20220805165331` for more details.
Backup request "kube-system-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe kube-system-ns-backup-20220805165331` or `velero backup logs kube-system-ns-backup-20220805165331` for more details.
Backup request "kubernetes-dashboard-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe kubernetes-dashboard-ns-backup-20220805165331` or `velero backup logs kubernetes-dashboard-ns-backup-20220805165331` for more details.
Backup request "linux70-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe linux70-ns-backup-20220805165331` or `velero backup logs linux70-ns-backup-20220805165331` for more details.
Backup request "myserver-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe myserver-ns-backup-20220805165331` or `velero backup logs myserver-ns-backup-20220805165331` for more details.
Backup request "velero-system-ns-backup-20220805165331" submitted successfully.
Run `velero backup describe velero-system-ns-backup-20220805165331` or `velero backup logs velero-system-ns-backup-20220805165331` for more details.
root@easzlab-k8s-master-01:~#恢复备份测试
删除备份 pod
root@easzlab-deploy:~/pod-test# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c8bb696bb-fxbmr 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-2qtfm 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-8l78t 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-9b75m 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-k75jh 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-kmbhs 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-lxfk9 1/1 Running 1 (2d10h ago) 6d21h
kube-system coredns-69548bdd5f-6df7j 1/1 Running 1 (2d10h ago) 6d9h
kube-system coredns-69548bdd5f-nl5qc 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-2d275 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard kubernetes-dashboard-5676d8b865-6l8n8 1/1 Running 1 (2d10h ago) 6d9h
linux70 linux70-tomcat-app1-deployment-5d666575cc-kbjhk 1/1 Running 1 (2d10h ago) 4d10h
myserver linux70-nginx-deployment-55dc5fdcf9-58ll2 1/1 Running 1 (2d10h ago) 4d10h
myserver linux70-nginx-deployment-55dc5fdcf9-6xcjk 1/1 Running 1 (2d10h ago) 4d10h
myserver linux70-nginx-deployment-55dc5fdcf9-cxg5m 1/1 Running 1 (2d10h ago) 4d10h
myserver linux70-nginx-deployment-55dc5fdcf9-gv2gk 1/1 Running 1 (2d10h ago) 4d10h
velero-system velero-858b9459f9-5mxxx 1/1 Running 0 23m
root@easzlab-deploy:~/pod-test# kubectl delete -f nginx.yaml
deployment.apps "linux70-nginx-deployment" deleted
service "linux70-nginx-service" deleted
root@easzlab-deploy:~/pod-test# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c8bb696bb-fxbmr 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-2qtfm 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-8l78t 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-9b75m 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-k75jh 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-kmbhs 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-lxfk9 1/1 Running 1 (2d10h ago) 6d21h
kube-system coredns-69548bdd5f-6df7j 1/1 Running 1 (2d10h ago) 6d9h
kube-system coredns-69548bdd5f-nl5qc 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-2d275 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard kubernetes-dashboard-5676d8b865-6l8n8 1/1 Running 1 (2d10h ago) 6d9h
linux70 linux70-tomcat-app1-deployment-5d666575cc-kbjhk 1/1 Running 1 (2d10h ago) 4d10h
velero-system velero-858b9459f9-5mxxx 1/1 Running 0 24m
root@easzlab-deploy:~/pod-test#使用备份数据进行恢复
root@easzlab-k8s-master-01:~# velero restore create --from-backup myserver-backup-20220804202021 --wait --kubeconfig=/root/.kube/config --namespace velero-system
Restore request "myserver-backup-20220804202021-20220804203424" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
.....................
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe myserver-backup-20220804202021-20220804203424` and `velero restore logs myserver-backup-20220804202021-20220804203424`.
root@easzlab-k8s-master-01:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c8bb696bb-fxbmr 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-2qtfm 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-8l78t 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-9b75m 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-k75jh 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-kmbhs 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-lxfk9 1/1 Running 1 (2d10h ago) 6d21h
kube-system coredns-69548bdd5f-6df7j 1/1 Running 1 (2d10h ago) 6d9h
kube-system coredns-69548bdd5f-nl5qc 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-2d275 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard kubernetes-dashboard-5676d8b865-6l8n8 1/1 Running 1 (2d10h ago) 6d9h
linux70 linux70-tomcat-app1-deployment-5d666575cc-kbjhk 1/1 Running 1 (2d10h ago) 4d10h
myserver linux70-nginx-deployment-55dc5fdcf9-58ll2 0/1 ContainerCreating 0 36s
myserver linux70-nginx-deployment-55dc5fdcf9-6xcjk 1/1 Running 0 36s
myserver linux70-nginx-deployment-55dc5fdcf9-cxg5m 0/1 ContainerCreating 0 36s
myserver linux70-nginx-deployment-55dc5fdcf9-gv2gk 1/1 Running 0 36s
velero-system velero-858b9459f9-5mxxx 1/1 Running 0 25m
root@easzlab-k8s-master-01:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c8bb696bb-fxbmr 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-2qtfm 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-8l78t 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-9b75m 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-k75jh 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-kmbhs 1/1 Running 1 (2d10h ago) 6d21h
kube-system calico-node-lxfk9 1/1 Running 1 (2d10h ago) 6d21h
kube-system coredns-69548bdd5f-6df7j 1/1 Running 1 (2d10h ago) 6d9h
kube-system coredns-69548bdd5f-nl5qc 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-2d275 1/1 Running 1 (2d10h ago) 6d9h
kubernetes-dashboard kubernetes-dashboard-5676d8b865-6l8n8 1/1 Running 1 (2d10h ago) 6d9h
linux70 linux70-tomcat-app1-deployment-5d666575cc-kbjhk 1/1 Running 1 (2d10h ago) 4d10h
myserver linux70-nginx-deployment-55dc5fdcf9-58ll2 1/1 Running 0 2m6s
myserver linux70-nginx-deployment-55dc5fdcf9-6xcjk 1/1 Running 0 2m6s
myserver linux70-nginx-deployment-55dc5fdcf9-cxg5m 1/1 Running 0 2m6s
myserver linux70-nginx-deployment-55dc5fdcf9-gv2gk 1/1 Running 0 2m6s
velero-system velero-858b9459f9-5mxxx 1/1 Running 0 27m
root@easzlab-k8s-master-01:~#卸载安装
root@easzlab-k8s-master-01:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c8bb696bb-fxbmr 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-2qtfm 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-8l78t 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-9b75m 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-k75jh 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-kmbhs 1/1 Running 1 (2d7h ago) 6d18h
kube-system calico-node-lxfk9 1/1 Running 1 (2d7h ago) 6d18h
kube-system coredns-69548bdd5f-6df7j 1/1 Running 1 (2d7h ago) 6d7h
kube-system coredns-69548bdd5f-nl5qc 1/1 Running 1 (2d7h ago) 6d7h
kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-2d275 1/1 Running 1 (2d7h ago) 6d7h
kubernetes-dashboard kubernetes-dashboard-5676d8b865-6l8n8 1/1 Running 1 (2d7h ago) 6d7h
linux70 linux70-tomcat-app1-deployment-5d666575cc-kbjhk 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-58ll2 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-6xcjk 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-cxg5m 1/1 Running 1 (2d7h ago) 4d7h
myserver linux70-nginx-deployment-55dc5fdcf9-gv2gk 1/1 Running 1 (2d7h ago) 4d7h
velero-system velero-858b9459f9-c7qc9 1/1 Running 0 15m
root@easzlab-k8s-master-01:~#
root@easzlab-k8s-master-01:~# velero --kubeconfig /root/.kube/config uninstall --namespace velero-system
You are about to uninstall Velero.
Are you sure you want to continue (Y/N)? y
Waiting for velero namespace "velero-system" to be deleted
....................................................................
Velero namespace "velero-system" deleted
Velero uninstalled ?
root@easzlab-k8s-master-01:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c8bb696bb-fxbmr 1/1 Running 1 (2d9h ago) 6d21h
kube-system calico-node-2qtfm 1/1 Running 1 (2d9h ago) 6d21h
kube-system calico-node-8l78t 1/1 Running 1 (2d9h ago) 6d21h
kube-system calico-node-9b75m 1/1 Running 1 (2d9h ago) 6d21h
kube-system calico-node-k75jh 1/1 Running 1 (2d9h ago) 6d21h
kube-system calico-node-kmbhs 1/1 Running 1 (2d9h ago) 6d21h
kube-system calico-node-lxfk9 1/1 Running 1 (2d9h ago) 6d21h
kube-system coredns-69548bdd5f-6df7j 1/1 Running 1 (2d9h ago) 6d9h
kube-system coredns-69548bdd5f-nl5qc 1/1 Running 1 (2d9h ago) 6d9h
kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-2d275 1/1 Running 1 (2d9h ago) 6d9h
kubernetes-dashboard kubernetes-dashboard-5676d8b865-6l8n8 1/1 Running 1 (2d9h ago) 6d9h
linux70 linux70-tomcat-app1-deployment-5d666575cc-kbjhk 1/1 Running 1 (2d9h ago) 4d9h
myserver linux70-nginx-deployment-55dc5fdcf9-58ll2 1/1 Running 1 (2d9h ago) 4d9h
myserver linux70-nginx-deployment-55dc5fdcf9-6xcjk 1/1 Running 1 (2d9h ago) 4d9h
myserver linux70-nginx-deployment-55dc5fdcf9-cxg5m 1/1 Running 1 (2d9h ago) 4d9h
myserver linux70-nginx-deployment-55dc5fdcf9-gv2gk 1/1 Running 1 (2d9h ago) 4d9h
root@easzlab-k8s-master-01:~#