1. 什么是openstack

openstack 是一个云操作系统，可控制整个数据中心内的大量计算、存储和网络资源，所有资源均通过具有通用身份验证机制的API进行管理和配置。

还提供仪表板，使管理员能够进行控制，同时使用户能够通过Web界面配置资源。

除了标准的基础设施即服务功能之外，其他组件还提供编排、故障管理和服务管理等服务，以确保用户应用程序的高可用性。

2. kubernetes与openstack的异同

kubernetes与 openstack由两个不同的社区创建，解决两个不同的技术问题。因此，尽管它们的功能有时会重叠，但运行原理却不同。

2.1 支持的工作负载类型

之前，openstack支持虚拟机（VM），而kubernetes曾经只支持容器。现在情况已大不相同。多年来，这些技术不断发展，可确保覆盖所有类型的工作负载。openstack现通过Ironic项目支持裸机工作负载，并通过Magnum项目支持容器化工作负载。而kubernetes用户可利用Metal等工具配置物理机和Kata容器，或通过kubevirt配置虚拟机。

2.2 裸机管理功能

openstack被设计为直接在裸机上运行。因此，它的裸机管理功能通常比kubernetes更加先进。尽管kubernetes也可直接在物理层上运行，但openstack已在这一领域深耕十三年，拥有更丰富的经验。openstack通常能更好地支持与现有各种储存平台和软件定义网络 (SDN) 控制器的集成，以及众多性能扩展（例如硬件卸载）。

2.3 应用程序管理功能

另一方面，kubernetes在应用程序管理方面是首屈一指的。kubernetes的设计考虑了大量的应用程序管理功能，包括ReplicaSet或DaemonSet等各种控制器以及云原生功能。例如，这些包括滚动升级和自动扩展。虽然openstack社区在过去努力用heat和murano项目来填补这一差距，但他们并未取得太大的成功，使得其在应用程序管理领域落后于kubernetes。

3. 结合k8s与openstack的优势

在这一点上，我们应该清楚不管是openstack还是kubernetes，都不能提供另一个项目所提供的全部功能。它们完全是为了不同的目的而生。因此，将它们捆绑在一起使用，更能发挥重大意义。

3.1 openstack上的kubernetes

在openstack上运行kubernetes具有诸多好处。首先，这比直接在裸机上运行更为简单。有一些工具支持在openstack上即时配置kubernetes集群，包括Juju、Magnum或Cluster API。受益于openstack提供的多租户技术，每个用户或实体可获得他们自己的云资源池，并在其中按需启动他们的k8s。此外，如前所述，这些k8s集群也可利用openstack的金属管理功能，例如通过Octavia实现适当的负载平衡，vGPU 资源配置等等。

3.2 kubernetes上的openstack

在kubernetes上运行openstack的控制平面（Control Plane）也具有其优势。最重要的是，它能够更好地隔离openstack服务，有效地将它们从底层操作系统（OS）中分离出来。此外，在这样的设置中，openstack可受益于kubernetes提供的生命周期管理功能，大大简化升级等历来就十分复杂的操作。最后，对k8s进行标准化可让openstack的架构变得相对轻盈，有效将其转变成所有人都可使用的东西，甚至在他们的工作站上直接转变。

3.3 堆叠栈

试想下，我们在kubernetes上的openstack上设置kubernetes呐？是不是很有趣！即使听起来有些别扭，但该架构设置具有最多优势，可有效的支持在一个平台上结合这两者的优势。

4. openstack格局

openstack被分解为多个服务，以便可以根据需要即插即用组件。

关于部分组件解释：

1）Nova

实施服务和相关库，以提供对计算资源（包括裸机、虚拟机和容器）的大规模可扩展、按需、自助服务访问。

2）Ironic

实现服务和相关库，以提供对计算资源（包括裸机、虚拟机和容器）的大规模可扩展、按需、自助访问。

3）Swift

Swift是一个高度可用的、分布式的、最终一致的对象/blob存储。组织可以使用Swift高效、安全、廉价地存储大量数据。它是为扩展而构建的，并针对整个数据集的耐用性、可用性和并发性进行了优化。Swift非常适合存储无约束增长的非结构化数据。

4）Neutron

OpenStack Neutron是一个SDN网络项目，专注于在虚拟计算环境中提供网络即服务（NaaS）。

5）Keystone

Keystone是一项OpenStack服务，通过实现OpenStack的Identity API，提供API客户端身份验证、服务发现和分布式多对端授权。它支持LDAP、OAuth、OpenIDConnect、SAML和SQL。

6）Placement

Placement是一个OpenStack服务，它提供了一个HTTP API来跟踪云资源清单和使用情况，以帮助其他服务有效地管理和分配其资源。

7）Glance

Glance镜像服务包括发现、注册和检索虚拟机镜像。Glance有一个RESTful API，它允许查询虚拟机映像元数据以及检索实际映像。通过Glance提供的虚拟机映像可以存储在各种位置，从简单的文件系统到像OpenStack Swift项目这样的对象存储系统。

8）Barbican

Barbican是OpenStack密钥管理器服务。它提供秘密数据的安全存储、供应和管理，如密码、加密密钥、X.509证书和原始二进制数据。

9）Heat

Heat基于可以像代码一样处理的文本文件形式的模板，为云应用程序编排基础设施资源。Heat提供了一个OpenStack-native ReST API和一个CloudFormation兼容的Query API。Heat还提供了一个与OpenStack Telemetry服务集成的自动缩放服务，因此可以将缩放组作为资源包含在模板中。

10）Horizon

Horizon是OpenStack仪表板的规范实现，它是可扩展的，并为OpenStack服务提供了一个基于web的用户界面。

5. 小试牛刀-安装openstack

1）在master节点上，为kubernetes各节点打上标签
kubectl label nodes k8s-master openstack-control-plane=enabled
kubectl label nodes k8s-slave1 openstack-compute-node=enabled
kubectl label nodes k8s-slave2 openvswitch=enabled

2）在master节点上，OpenStack-Helm 各个服务的 endpoints URL 都是以 *.openstack.svc.cluster.local 域名的形式，配置本地hosts解析：
cat >>/etc/hosts<<EOF
192.168.2.148 keystone.openstack.svc.cluster.local
192.168.2.148 heat.openstack.svc.cluster.local
192.168.2.148 glance.openstack.svc.cluster.local
192.168.2.148 nova.openstack.svc.cluster.local
192.168.2.148 neutron.openstack.svc.cluster.local
192.168.2.148 heat-api.openstack.svc.cluster.local
192.168.2.148 horizon-int.openstack.svc.cluster.local
EOF

3）在全部节点上，根据需要安装最新版本的git、make等
yum install -y ca-certificates git make jq nmap curl uuid-runtime bc python3-pip

4）在master节点上，克隆包含 OpenStack-Helm chart的git仓库
git clone https://opendev.org/openstack/openstack-helm-infra.git
git clone https://opendev.org/openstack/openstack-helm.git

5）定义openstack安装版本，本示例为xena
[root@k8s-master openstack-helm]# cd openstack-helm
[root@k8s-master openstack-helm]# cat tools/deployment/common/setup-client.sh | grep OPENSTACK_RELEASE
-c${UPPER_CONSTRAINTS_FILE:=https://releases.openstack.org/constraints/upper/${OPENSTACK_RELEASE:-xena}} \
[root@k8s-master openstack-helm]# cat tools/deployment/common/get-values-overrides.sh | grep OPENSTACK_RELEASE:
: “${OPENSTACK_RELEASE:=”xena”}”

6）创建命名空间
kubectl create ns openstack
kubectl create ns ceph

7）安装openstack客户端
给所有节点 配置国内pip源
mkdir ~/.pip
cat > ~/.pip/pip.conf << EOF
[global]
trusted-host=mirrors.aliyun.com
index-url=https://mirrors.aliyun.com/pypi/simple/
EOF

[root@k8s-master openstack-helm]# cd openstack-helm
[root@k8s-master openstack-helm]# ./tools/deployment/common/setup-client.sh

验证客户端安装
[root@k8s-master openstack-helm]# openstack –version
openstack 5.6.2

8）安装 ingress
[root@k8s-master openstack-helm]# ./tools/deployment/component/common/ingress.sh

查看创建的pods
kubectl get pods -n openstack -l application=ingress

9）安装 NFS Provisioner
[root@k8s-master openstack-helm]# mkdir -p /var/lib/openstack-helm/nfs
[root@k8s-master openstack-helm]# ./tools/deployment/component/nfs-provisioner/nfs-provisioner.sh

查看创建的pods
[root@k8s-master openstack-helm]# kubectl get pods -n nfs -l application=nfs
NAME READY STATUS RESTARTS AGE
nfs-provisioner-5fbd577c5b-9d5qr 1/1 Running 0 4m30s

查看创建的storageclass
[root@k8s-master openstack-helm]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
general nfs/nfs-provisioner Delete Immediate false 6m13s

所有节点安装nfs客户端
yum install nfs-utils -y

10）安装 MariaDB
[root@k8s-master openstack-helm]# ./tools/deployment/component/common/mariadb.sh

查看部署的mariadb pods
[root@k8s-master openstack-helm]# kubectl get pods -n openstack -l application=mariadb
NAME READY STATUS RESTARTS AGE
mariadb-ingress-779bdc967d-bl695 1/1 Running 0 3m14s
mariadb-ingress-779bdc967d-t67v2 1/1 Running 0 3m14s
mariadb-ingress-error-pages-85f967dc75-s4mkf 1/1 Running 0 3m14s
mariadb-server-0 1/1 Running 0 3m13s

11）安装 RabbitMQ
[root@k8s-master openstack-helm]# ./tools/deployment/component/common/rabbitmq.sh

查看部署的rabbitmq pods
[root@k8s-master openstack-helm]# kubectl get pods -n openstack -l application=rabbitmq
NAME READY STATUS RESTARTS AGE
rabbitmq-cluster-wait-5g59d 0/1 Completed 0 5m27s
rabbitmq-rabbitmq-0 1/1 Running 0 5m28s

12）安装 Memcached
[root@k8s-master openstack-helm]# ./tools/deployment/component/common/memcached.sh

查看部署的memcached pods
[root@k8s-master openstack-helm]# kubectl get pods -n openstack -l application=memcached
NAME READY STATUS RESTARTS AGE
memcached-memcached-c67796474-ph8sx 1/1 Running 0 62s

13）安装 Keystone
[root@k8s-master openstack-helm]# ./tools/deployment/component/keystone/keystone.sh

查看部署的keystone pods
[root@k8s-master openstack-helm]# kubectl get pods -n openstack -l application=keystone
NAME READY STATUS RESTARTS AGE
keystone-api-748569599b-schw7 1/1 Running 0 10m
keystone-bootstrap-jws2b 0/1 Completed 0 6m48s
keystone-credential-setup-x78fv 0/1 Completed 0 10m
keystone-db-init-ps58p 0/1 Completed 0 9m11s
keystone-db-sync-kd9k8 0/1 Completed 0 8m8s
keystone-domain-manage-r6xqh 0/1 Completed 0 7m37s
keystone-fernet-setup-4c48c 0/1 Completed 0 8m18s
keystone-rabbit-init-9jbpt 0/1 Completed 0 7m43s
keystone-test 0/1 Completed 0 5m54s

14）安装 Heat
[root@k8s-master openstack-helm]# ./tools/deployment/component/heat/heat.sh

查看部署的heat pods
[root@k8s-master openstack-helm]# kubectl get pods -n openstack -l application=heat
NAME READY STATUS RESTARTS AGE
heat-api-6bf8998ff5-rm48w 1/1 Running 0 9m50s
heat-bootstrap-mk9xv 0/1 Completed 0 7m43s
heat-cfn-548954864c-w4rrp 1/1 Running 0 9m50s
heat-db-init-4q6h2 0/1 Completed 0 9m49s
heat-db-sync-w65dk 0/1 Completed 0 9m41s
heat-domain-ks-user-xb8mb 0/1 Completed 0 7m30s
heat-engine-cleaner-28338480-x82bv 0/1 Completed 0 7m50s
heat-engine-cleaner-28338485-qndq7 0/1 Completed 0 2m50s
heat-engine-f7c855cf7-96wb7 1/1 Running 0 9m50s
heat-ks-endpoints-ddmlm 0/6 Completed 0 8m55s
heat-ks-service-xj9ps 0/2 Completed 0 9m21s
heat-ks-user-r8l7q 0/1 Completed 0 8m25s
heat-rabbit-init-qlkpw 0/1 Completed 0 9m29s
heat-trustee-ks-user-r6w6c 0/1 Completed 0 6m47s
heat-trusts-vg69f 0/1 Completed 0 6m5s

15）安装 Horizon
[root@k8s-master openstack-helm]# ./tools/deployment/component/horizon/horizon.sh

查看部署的horizon pods
[root@k8s-master openstack-helm]# kubectl get pods -n openstack -l application=horizon
NAME READY STATUS RESTARTS AGE
horizon-7dbcdf9f8b-vxr9n 1/1 Running 0 5m16s
horizon-db-init-lnt74 0/1 Completed 0 5m16s
horizon-db-sync-vvxq2 0/1 Completed 0 5m6s
horizon-test 0/1 Completed 0 2m29s

horizon-int 对应的就是 Dashboard 服务，它的类型是 NodePort，映射的端口号是 31000

16）安装 Glance
[root@k8s-master openstack-helm]# ./tools/deployment/component/glance/glance.sh

其中/tmp/glance.yaml定义了glance使用的存储类型，values.yaml中支持的存储类型如下：

# radosgw, rbd, swift or pvc

storage: swift

目前 Glance 支持几种 backend storage:

pvc: 一个简单的 Kubernetes PVC存储后端。
rbd: 使用 Ceph RBD 来存储 images。
radosgw: 使用 Ceph RGW 来存储 images。
swift: 另用 OpenStack switf 所提供的对象存储来存储 images.

查看部署的glance pods
[root@k8s-master ~]# kubectl get pods -n openstack -l application=glance
NAME READY STATUS RESTARTS AGE
glance-api-869959bffd-mgwc2 1/1 Running 0 19m
glance-bootstrap-xwkjv 0/1 Completed 3 18m
glance-db-init-twvsp 0/1 Completed 0 19m
glance-db-sync-bf2zl 0/1 Completed 0 19m
glance-ks-endpoints-mj97j 0/3 Completed 0 19m
glance-ks-service-x4dnn 0/1 Completed 0 19m
glance-ks-user-69plf 0/1 Completed 0 18m
glance-metadefs-load-xfldv 0/1 Completed 0 18m
glance-rabbit-init-6zvll 0/1 Completed 0 19m
glance-storage-init-jq256 0/1 Completed 0 18m

17）安装 OpenvSwitch
[root@k8s-master openstack-helm]# ./tools/deployment/component/compute-kit/openvswitch.sh

查看部署的openvswitch pods
[root@k8s-master ~]# kubectl get pods -n openstack -l application=openvswitch
NAME READY STATUS RESTARTS AGE
openvswitch-wwwk6 2/2 Running 0 2m43s

18）安装 Libvirt
[root@k8s-master openstack-helm]# ./tools/deployment/component/compute-kit/libvirt.sh

查看部署的libvirt pods
[root@k8s-master networks]# kubectl get po -n openstack -l application=libvirt

19）安装 Compute Kit (Nova and Neutron)
[root@k8s-master openstack-helm]# ./tools/deployment/component/compute-kit/compute-kit.sh

查看部署的placement pods
[root@k8s-master ~]# kubectl get pods -n openstack -l application=placement
NAME READY STATUS RESTARTS AGE
placement-api-556bf56cdf-zgwp9 1/1 Running 0 39m
placement-db-init-6g6ph 0/1 Completed 0 39m
placement-db-sync-krdnf 0/1 Completed 0 39m
placement-ks-endpoints-sxnnx 0/3 Completed 0 38m
placement-ks-service-zpk8j 0/1 Completed 0 39m
placement-ks-user-qv622 0/1 Completed 0 38m

查看部署的nova pods
[root@k8s-master ~]# kubectl get pods -n openstack -l application=nova
NAME READY STATUS RESTARTS AGE
nova-api-metadata-cd7bb68bc-p5qc2 1/1 Running 1 (35m ago) 39m
nova-api-osapi-7f7dc9f44-ccbkg 1/1 Running 0 39m
nova-bootstrap-26dbc 1/1 Running 0 39m
nova-cell-setup-28342680-wfjbw 0/1 Completed 0 17m
nova-cell-setup-r5vqh 0/1 Completed 0 39m
nova-compute-default-x9fnm 1/1 Running 0 39m
nova-conductor-57bbb4b999-8n95p 1/1 Running 0 39m
nova-db-init-ffxvj 0/3 Completed 0 39m
nova-db-sync-n9mz4 0/1 Completed 0 39m
nova-ks-endpoints-25xk6 0/3 Completed 0 37m
nova-ks-service-jqqbb 0/1 Completed 0 37m
nova-ks-user-v6fzg 0/1 Completed 0 36m
nova-novncproxy-67fbf56db4-b424b 1/1 Running 0 39m
nova-rabbit-init-459×9 0/1 Completed 0 37m
nova-scheduler-79ddc44bc9-t7rkq 1/1 Running 0 39m
nova-service-cleaner-28342680-72pxq 0/1 Completed 0 17m

查看部署的neutron pods
[root@k8s-master ~]# kubectl get pods -n openstack -l application=neutron
NAME READY STATUS RESTARTS AGE
neutron-db-init-tkqvb 0/1 Completed 0 36m
neutron-db-sync-h2zht 0/1 Completed 0 36m
neutron-dhcp-agent-default-p9855 1/1 Running 0 36m
neutron-ks-endpoints-p5mtm 0/3 Completed 0 35m
neutron-ks-service-cv2tb 0/1 Completed 0 36m
neutron-ks-user-7wtff 0/1 Completed 0 35m
neutron-metadata-agent-default-dnhkn 1/1 Running 0 36m
neutron-netns-cleanup-cron-default-j6z84 1/1 Running 0 36m
neutron-ovs-agent-default-stzrx 1/1 Running 0 36m
neutron-rabbit-init-rvtlk 0/1 Completed 0 36m
neutron-server-649f5f9fbb-96w5j 1/1 Running 0 36m

20）设置 public network的gateway
给每个节点安装依赖包
[root@k8s-master openstack-helm]# yum install net-tools -y
然后执行：
[root@k8s-master openstack-helm]# cat ./tools/deployment/common/setup-gateway.sh

#以下两个参数保持默认即可

#Assign IP address to br-ex

: ${OSH_EXT_SUBNET:=”172.24.4.0/24″}
: ${OSH_BR_EX_ADDR:=”172.24.4.1/24″}

[root@k8s-master openstack-helm]# ./tools/deployment/common/setup-gateway.sh

21）Exercise the Cloud
部署 OpenStack-Helm 后，可以使用 OpenStack 客户端或validation gates中使用的相同 heat 模板来运行云。
[root@k8s-master openstack-helm]# ./tools/deployment/common/use-it.sh

22）查看创建的资源

查看service
[root@k8s-master openstack-helm]# kubectl get svc -n openstack
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cloudformation ClusterIP 10.98.184.197 80/TCP,443/TCP 2d22h
glance ClusterIP 10.110.30.27 80/TCP,443/TCP 2d20h
glance-api ClusterIP 10.98.213.245 9292/TCP 2d20h
heat ClusterIP 10.100.247.210 80/TCP,443/TCP 2d22h
heat-api ClusterIP 10.103.161.38 8004/TCP 2d22h
heat-cfn ClusterIP 10.102.25.102 8000/TCP 2d22h
horizon ClusterIP 10.110.32.8 80/TCP,443/TCP 2d22h
horizon-int NodePort 10.107.94.4 80:31000/TCP 2d22h
ingress ClusterIP 10.96.122.66 80/TCP,443/TCP,10246/TCP 3d2h
ingress-error-pages ClusterIP None 80/TCP 3d2h
ingress-exporter ClusterIP 10.109.233.129 10254/TCP 3d2h
keystone ClusterIP 10.106.174.60 80/TCP,443/TCP 2d23h
keystone-api ClusterIP 10.100.56.3 5000/TCP 2d23h
mariadb ClusterIP 10.109.212.62 3306/TCP 3d
mariadb-discovery ClusterIP None 3306/TCP,4567/TCP 3d
mariadb-ingress-error-pages ClusterIP None 80/TCP 3d
mariadb-server ClusterIP 10.111.132.147 3306/TCP 3d
memcached ClusterIP 10.103.187.219 11211/TCP 2d23h
metadata ClusterIP 10.100.205.253 80/TCP,443/TCP 74m
neutron ClusterIP 10.105.22.8 80/TCP,443/TCP 71m
neutron-server ClusterIP 10.96.49.4 9696/TCP 71m
nova ClusterIP 10.97.111.65 80/TCP,443/TCP 74m
nova-api ClusterIP 10.107.112.60 8774/TCP 74m
nova-metadata ClusterIP 10.97.197.254 8775/TCP 74m
nova-novncproxy ClusterIP 10.111.65.13 6080/TCP 74m
novncproxy ClusterIP 10.105.220.245 80/TCP,443/TCP 74m
placement ClusterIP 10.96.59.125 80/TCP,443/TCP 76m
placement-api ClusterIP 10.97.99.41 8778/TCP 76m
rabbitmq ClusterIP None 5672/TCP,25672/TCP,15672/TCP,15692/TCP 3d
rabbitmq-mgr-7b1733 ClusterIP 10.99.162.173 80/TCP,443/TCP 3d

查看ingress
[root@k8s-master openstack-helm]# kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
ceph ceph-ingress-ceph nginx-ceph *.ceph.svc.cluster.local 192.168.2.148 80 3d2h
openstack cloudformation nginx cloudformation,cloudformation.openstack,cloudformation.openstack.svc.cluster.local 192.168.2.148 80 2d22h
openstack glance nginx glance,glance.openstack,glance.openstack.svc.cluster.local 192.168.2.148 80 2d20h
openstack heat nginx heat,heat.openstack,heat.openstack.svc.cluster.local 192.168.2.148 80 2d22h
openstack horizon nginx horizon,horizon.openstack,horizon.openstack.svc.cluster.local 192.168.2.148 80 2d22h
openstack keystone nginx keystone,keystone.openstack,keystone.openstack.svc.cluster.local 192.168.2.148 80 2d23h
openstack metadata nginx metadata,metadata.openstack,metadata.openstack.svc.cluster.local 192.168.2.148 80 76m
openstack neutron nginx neutron,neutron.openstack,neutron.openstack.svc.cluster.local 192.168.2.148 80 73m
openstack nova nginx nova,nova.openstack,nova.openstack.svc.cluster.local 192.168.2.148 80 76m
openstack novncproxy nginx novncproxy,novncproxy.openstack,novncproxy.openstack.svc.cluster.local 192.168.2.148 80 76m
openstack openstack-ingress-openstack nginx *.openstack.svc.cluster.local 192.168.2.148 80 3d2h
openstack placement nginx placement,placement.openstack,placement.openstack.svc.cluster.local 192.168.2.148 80 78m
openstack rabbitmq-mgr-7b1733 nginx rabbitmq-mgr-7b1733,rabbitmq-mgr-7b1733.openstack,rabbitmq-mgr-7b1733.openstack.svc.cluster.local 192.168.2.148 80 3d

23）访问horizon
在web中访问以下地址，用户名密码为：admin/password，域为default
http://192.168.2.148:31000
登陆horizon

参考链接：

https://www.openstack.org/software/
https://cn.ubuntu.com/blog/kubernetes-vs-openstack-cn
https://blog.csdn.net/networken/article/details/132527961