Microk8s Ingress是什么 Ingress是k8s的一种资源对象,用于管理外部对集群内服务的访问, 它通过提供一个统一的入口点,将外部流量路由到集群内部的不同服务。
Microk8s Ingress用于解决什么问题 k8s集群中服务默认只能在集群内访问。 如果需要从外部访问服务,通常需要使用NodePort或LoadBalancer类型服务,这两个服务都存在一些问题
NodePort会占用节点端口,可能导致端口冲突
LoadBalancer需要云提供商支持, 不适合本地环境
Ingress提供一种灵活的方式暴露服务,允许通过域名或路径规则将流量路由到不同的服务
Microk8s Ingress基本原理
Microk8s内置了一个Nginx的Ingress Controller, 负责监听k8s中的Ingress资源,当检测到Ingress资源更新时, 动态更新Nginx配置文件
外部流量先到达Nginx, 再基于域名和URL将请求转发到Service, Service再将流量分发到Pod
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # kubectl -n ingress get pod nginx-ingress-microk8s-controller-nrftt NAME READY STATUS RESTARTS AGE nginx-ingress-microk8s-controller-nrftt 1/1 Running 0 15m # ps axf 860646 ? Sl 0:00 /var/lib/snapd/snap/microk8s/7665/bin/containerd-shim-runc-v2 -namespace k8s.io -id 8fb71f54235bf26664240c8b9272b623a70d 860668 ? Ss 0:00 \_ /pause 860700 ? Ss 0:00 \_ /usr/bin/dumb-init -- /nginx-ingress-controller --configmap=ingress/nginx-load-balancer-microk8s-conf --tcp-services-configmap=ingress/nginx-ingress-tcp-microk8s-conf --udp-services-configmap=ingress/nginx-ingress-udp-microk8s-conf --ingress-class=public --publish-status-address=127.0.0.1 --default-ssl-certificate=default/example-tls 860712 ? Ssl 0:00 \_ /nginx-ingress-controller --configmap=ingress/nginx-load-balancer-microk8s-conf --tcp-services-configmap=ingress/nginx-ingress-tcp-microk8s-conf --udp-services-configmap=ingress/nginx-ingress-udp-microk8s-conf --ingress-class=public --publish-status-address=127.0.0.1 --default-ssl-certificate=default/example-tls 860782 ? S 0:00 \_ nginx: master process /usr/bin/nginx -c /etc/nginx/nginx.conf 860786 ? Sl 0:00 \_ nginx: worker process 860787 ? Sl 0:00 \_ nginx: worker process 860788 ? Sl 0:00 \_ nginx: worker process 860789 ? Sl 0:00 \_ nginx: worker process 860790 ? S 0:00 \_ nginx: cache manager process
实践: 使用Microk8s Ingress配置HTTP/TCP负载均衡 首先安装并启动Microk8s和Ingress插件, 参考: 在RockyLinux9.4上安装Microk8s 演示案例如下图:
支持HTTP负载均衡 用Flask写两个简单的HTTP服务
app.py
1 2 3 4 5 6 7 8 9 10 from flask import Flask app = Flask(__name__) @app.route('/foo/') def hello_world(): return "Hello, foo!" if __name__ == '__main__': app.run(host='0.0.0.0', port=8080)
Dockerfile
1 2 3 4 5 6 FROM python:3.9-slim WORKDIR /app COPY . /app RUN pip install flask EXPOSE 8080 CMD ["python", "app.py"]
构建docker镜像
1 2 docker build -t foo:1.0 -f Dockerfile . docker save foo:1.0 > foo.tar
创建k8s资源, 创建Deployment
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 apiVersion: apps/v1 kind: Deployment metadata: name: foo-deployment spec: replicas: 1 selector: matchLabels: app: foo template: metadata: labels: app: foo spec: containers: - name: foo image: foo:1.0 ports: - containerPort: 8080
创建Service
1 2 3 4 5 6 7 8 9 10 11 12 apiVersion: v1 kind: Service metadata: name: foo-service spec: selector: app: foo ports: - protocol: TCP port: 80 targetPort: 8080 type: ClusterIP
在Microk8s环境上导入镜像, 创建yaml, 先在集群内部测试一下服务OK
1 2 3 4 5 6 7 8 9 10 microk8s.ctr image *.tar kubectl create -f *.yaml # kubectl get svc default bar-service ClusterIP 10.152.183.208 <none> 80/TCP 11s default foo-service ClusterIP 10.152.183.189 <none> 80/TCP 25m # curl 10.152.183.189:80/foo/ Hello Foo! # curl 10.152.183.208/bar/ Hello Bar!
配置Ingress, 支持集群外访问 默认情况下, Ingress Controller是通过NodePort类型暴露的, 这里我们改成监听宿主机的80端口, 通过修改Ingress的DaemonSet实现: 创建nginx-ingress-microk8s-controller-patch.yaml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 apiVersion: apps/v1 kind: DaemonSet metadata: name: nginx-ingress-microk8s-controller namespace: ingress spec: template: spec: hostNetwork: true containers: - name: nginx-ingress-microk8s args: # must list all others here? - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf - --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf - --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf - --ingress-class=public - ' ' - --publish-status-address=127.0.0.1
修改Ingress的DaemonSet
1 kubectl -n ingress patch ds nginx-ingress-microk8s-controller --patch-file nginx-ingress-microk8s-controller-patch.yaml
确认宿主机80端口已经LISTEN
1 2 ss -antp | grep ":*80" LISTEN 0 4096 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=741444,fd=17),("nginx",pid=741437,fd=17))
创建Ingress资源, 支持从集群外访问两个HTTP服务
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-ingress spec: rules: - host: foo.example.com http: paths: - path: /foo/ pathType: Prefix backend: service: name: foo-service port: number: 80 - host: bar.example.com http: paths: - path: /bar/ pathType: Prefix backend: service: name: bar-service port: number: 80
测试, 从集群外部访问成功
1 2 3 4 curl foo.example.com/foo/ Hello Foo! curl bar.example.com/bar/ Hello Bar!
支持TCP负载均衡 Ingress同时支持四层(TCP,UDP)的负载均衡, 例如:
创建一个TCP服务, 监听8888端口 (tcp-service.yaml)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 apiVersion: apps/v1 kind: Deployment metadata: name: tcp-server namespace: service-foo spec: replicas: 1 selector: matchLabels: app: tcp-server template: metadata: labels: app: tcp-server spec: containers: - name: tcp-container image: busybox command: ["nc", "-lk", "8888"] ports: - containerPort: 8888 --- apiVersion: v1 kind: Service metadata: name: tcp-service namespace: service-foo spec: selector: app: tcp-server ports: - protocol: TCP port: 8888 targetPort: 8888 type: ClusterIP
配置Ingress的ConfigMap, 以支持TCP的暴露
创建nginx-ingress-tcp-microk8s-conf-patch.yaml
1 2 3 4 5 6 7 kind: ConfigMap apiVersion: v1 metadata: name: nginx-ingress-tcp-microk8s-conf namespace: ingress data: "8888": tcp-service:8888
修改Ingress的ConfigMap
1 kubectl -n ingress patch cm nginx-ingress-tcp-microk8s-conf --patch-file nginx-ingress-tcp-microk8s-conf-patch.yaml
从集群外部测试, 访问成功
1 2 3 # telnet foo.example.com 8888 Connected to foo.example.com. Escape character is '^]'
Ingress 支持HTTPS 生成证书和私钥
1 openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=example.com/O=Example Company"
创建kubernetes secret
1 microk8s kubectl create secret tls example-tls --cert=tls.crt --key=tls.key -n default
改一下Ingress的DaemonSet, 添加一行--default-ssl-certificate
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 apiVersion: apps/v1 kind: DaemonSet metadata: name: nginx-ingress-microk8s-controller namespace: ingress spec: template: spec: hostNetwork: true containers: - name: nginx-ingress-microk8s args: # must list all others here? - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf - --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf - --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf - --ingress-class=public - ' ' - --publish-status-address=127.0.0.1 - --default-ssl-certificate=default/example-tls
1 kubectl -n ingress patch ds nginx-ingress-microk8s-controller --patch-file nginx-ingress-microk8s-controller-patch.yaml
从集群外部测试, HTTP和HTTPS都可以访问
1 2 3 4 # curl foo.example.com/foo/ Hello Foo! # curl https://bar.example.com/bar/ -k Hello Bar!
问题 Q: 可能遇到导入Microk8s镜像失败, microk8s.ctr image list
查询镜像类型为text/html, 只有几百k A: 原因一般是之前导入镜像前就创建了pod, Microk8s尝试从网络获取镜像失败; 解决方法是先microk8s.ctr image rm
删除旧的镜像, 再重新导入即可
参考 【1】 https://microk8s.io/docs/addon-ingress 【2】 https://kubernetes.io/docs/concepts/services-networking/ingress/