一、K8S应用层面的服务暴露方式¶
K8S 在集群架构层面有三类IP：分别为 Node Network （物理节点网络）、 Pod Network（pod容器网络） 、 Cluster Network（集群网络，也称为 Service NetWork ）。
Pod 服务层面的暴露方式常用如下几种分类：
-
ClusterIP：当 service 的type 为 ClusterIP 的时候，服务只能在集群内部访问。 （默认的ServiceType）可以访问集群外接数据库。
-
NodePort: 当 service 的type 为 NodePort 的时候，会将service 的IP和端口映射到所有node节点的IP和端口。
-
hostPort: 当 service 的type 为 hostPort 的时候，类似 docker -p参数做的端口映射，将容器内端口映射到 当前宿主机上(hostPort)
-
LoadBalancer：使用云服务商的负载均衡器。外部的负载均衡器可以路由到 NodePort 服务 和 ClusterIP 服务。
- Ingress：创建一 个Ingress资源， 这是一 个完全不同的机制，通过一 个IP地址公开多个服务，就是一个网关入口，类似：springcloud的网关zuul、gateway
NodePort：
LoadBalance：
二、应用访问的具体链路（K8S层面）¶
当我们访问一个k8s 集群内部的服务的时候，集群内流量的走向为 ：
用户输入域名 --> 通过各种解析到 k8s 集群的工作节点上的 ingress --> ingress 将流量代理到 service:端口上 --> servic 通过寻找 endpointd 里面封装的 pod ip:端口 --> 通过 kube-proxy 在 ipvs 或者 iptables 中封装的规则调度到指定的 pod 上。
三、initContainer工作原理¶
初始化容器是在pod的主容器启动之前要运行的容器，主要是做一些 主容器的前置工作，它具有两大特征：
- 1、初始化容器必须运行完成直至结束，若某初始化容器运行失败，那么 kubernetes需要重启它直到成功完成；
- 2、初始化容器必须按照定义的顺序执行，当且仅当前一个成功之后，后面的一个才能运行，一旦失败，如果 Pod 对应的 restartPolicy 值为 Never，它不会重新启动；
初始化容器有很多的应用场景，下面列出的是最常见的几个：
- 提供主容器镜像中不具备的工具程序或自定义代码；
- 初始化容器要先于应用容器串行启动并运行完成，因此可用于延后应用容器的启动直至其依赖的条件得到满足；
四、initContainer数据共享¶
需求： 假设要以主容器来运行nginx，但是要求在运行nginx之前需要拿到最新的index主页；
创建pod-initcontainer.yaml，内容如下：
# 创建ns
[root@master01 1]# k create ns study
# 创建pod模板
[root@master01 1]# k run php-updated --image=php:7-fpm --dry-run=client -oyaml > pod-initcontainer.yaml
# 定义yaml文件
[root@master01 1]# vim pod-initcontainer.yaml
apiVersion: v1
kind: Pod
metadata:
name: php-updated
namespace: study
spec:
containers:
- name: php
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/php:7-fpm
volumeMounts:
- name: dir
mountPath: /var/www/html/
initContainers:
- name: install
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox:1.28
volumeMounts:
- name: dir
mountPath: /var/www/html/
command:
- wget
- "-O"
- "/var/www/html/index.php"
- https://gitee.com
volumes:
- name: dir
emptyDir: {}
# 应用
[root@master01 1]# kaf pod-initcontainer.yaml
# 查看
[root@master01 1]# kgp -nstudy
NAME READY STATUS RESTARTS AGE
php-updated 1/1 Running 0 8m30s
# 验证，查看到/var/www/html/目录下的index.html文件为init container所生成
[root@master01 1]# k exec -it php-updated -nstudy -c php -- bash
root@php-updated:/var/www/html# cat index.php | head -n1
<!DOCTYPE html><html lang="zh-CN"><head><meta charSet="utf-8"/><title>Gitee - 基于 Git 的代码托管和研发协作平台</title><meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1.0, user-scalable=no, viewport-fit=cover"/><meta name="keywords" content=" 码云,Gitee,Gitee企业版,代码管理,协作开发,敏捷开发,研发管理,DevOps,项目管理,迭代规划Git,SVN,代码托管,代码评审,CodeReview,任务管理,文档协作,持续集成,Bug管理,SaaS,本地部署,私有化,BUG,github,gitlab,开源,内源,知识库,git,持续集成,持续交付,持续部署,云端,IDE,任务管理,敏捷,瀑布,ones,华为云,腾讯云,阿里云,版本管理,企业级,公司,软件开发,运维,智能,国产化,信创,效率,鸿蒙,流水线,云效,Coding,简单云,TAPD,Jira,禅道,Teambition,Jenkins,WorkTile,甘特图,精益开发,开源社区"/><meta name="description" content="面向企业提供一站式研发管理解决方案，包括代码管理、项目管理、文档协作、测试管理、CICD、效能度量等多个模块，支持SaaS、私有化等多种部署方式，帮助企业有序规划和管理研发过程，提升研发效率和质量。"/><meta name="applicable-device" content="pc,mobile"/><meta name="referrer" content="always"/><meta content="Gitee - 基于 Git 的代码托管和研发协作平台" itemProp="name" property="og:title"/><meta content="Object" property="og:type"/><meta content="https://gitee.com" property="og:url"/><meta content="Gitee" property="og:site_name"/><meta content="面向企业提供一站式研发管理解决方案，包括代码管理、项目管理、文档协作、测试管理、CICD、效能度量等多个模块，支持SaaS、私有化等多种部署方式，帮助企业有序规划和管理研发过程，提升研发效率和质量。" property="og:description"/><meta content="/static/images/logo.svg" itemProp="image" property="og:image"/><link rel="shortcut icon" href="/favicon.ico"/><link rel="preconnect" href="https://e-assets.gitee.com"/><link rel="preconnect" href="https://portrait.gitee.com"/><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"/><meta name="renderer" content="webkit"/><meta name="force-rendering" content="webkit"/><meta name="browsermode" content="application"/><meta name="x5-page-mode" content="app"/><style id="disable-transition">
# 环境复原
[root@master01 1]# k delete -f pod-initcontainer.yaml
启动成功后，登陆进PHP容器，可以查看到/var/www/html/目录下的index.html文件为 init container所生成。
五、initContainer前置数据操作¶
初始化容器和PortStart的区别：
PostStart：依赖主应用的环境，而且并不一定先于Command运行
InitContainer：不依赖主应用的环境，可以有更高的权限和更多的工具，一定会在主应用启动之前完成。
Init 容器不支持 lifecycle、livenessProbe、readinessProbe 和 startupProbe。
需求：
假设主容器在运行前，需要依赖一个B应用，只有B应用成功启动后此容器才可以正常运行；
# 创建ns
[root@master01 1]# k create ns
# 创建deploy模板
[root@master01 1]# k create deploy nginx-blue-deployment --image=registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox:1.28 -r=2 --dry-run=client -oyaml > pod-initcontainer22.yaml
# 定义yaml文件
[root@master01 1]# vim pod-initcontainer22.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: my-app
name: my-app
spec:
replicas: 2
selector:
matchLabels:
run: my-app
template:
metadata:
labels:
run: my-app
spec:
restartPolicy: Always
containers:
- image: registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox:1.28
name: myapp-container
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init-myappb
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox:1.28
command: ['sh', '-c', "until nslookup myappb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myappb; sleep 2; done"]
# 应用
[root@master01 1]# kaf pod-initcontainer22.yaml
# 验证，观察到一直处于初始化
[root@master01 1]# kgp
NAME READY STATUS RESTARTS AGE
my-app-76c46b64b6-694nr 0/2 Init:0/2 0 6m9s
my-app-76c46b64b6-82tvh 0/2 Init:0/2 0 6m9s
# 进一步验证,观察到一直在探测myapp服务
[root@master01 1]# k logs -f my-app-76c46b64b6-694nr -c init-myappb
nslookup: can't resolve 'myappb.default.svc.cluster.local'
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
waiting for myappb
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
waiting for myappb
nslookup: can't resolve 'myappb.default.svc.cluster.local'
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
waiting for myappb
...
...
# 定义servcie的yaml文件
[root@master01 1]# vim myappb-service.yaml
apiVersion: v1
kind: Service
metadata:
name: myappb
spec:
ports:
- protocol: TCP
port: 80
targetPort: 9377
# 应用
[root@master01 1]# kaf myappb-service.yaml
# 验证，观察到服务正常
[root@master01 1]# kgp
NAME READY STATUS RESTARTS AGE
my-app-76c46b64b6-694nr 2/2 Running 0 14m
my-app-76c46b64b6-82tvh 2/2 Running 0 14m
# 进一步验证，查看日志信息
[root@master01 1]# k logs -f my-app-76c46b64b6-694nr -c myapp-container
The app is running!
# 环境复原
[root@master01 1]# k delete -f myappb-service.yaml
[root@master01 1]# k delete -f pod-initcontainer22.yaml
未创建svc前，initcontainer一直处于等待，可以从console端输出日志看到其状态，一 旦创建svc，initcontainer探测到svc正常后，即启动后续的mainContainer。
六、故障处理¶
6.1 故障描述¶
初始化容器一直处于Init:CrashLoopBackOff
[root@master01 1]# vim pod-initcontainer.yaml
apiVersion: v1
kind: Pod
metadata:
name: php-updated
namespace: study
spec:
containers:
- name: php
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/php:7-fpm
volumeMounts:
- name: dir
mountPath: /var/www/html/
initContainers:
- name: install
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox
volumeMounts:
- name: dir
mountPath: /var/www/html/
command:
- wget
- "-O"
- "/var/www/html/index.php"
- https://gitee.com
volumes:
- name: dir
emptyDir: {}
# 应用
[root@master01 1]# kaf pod-initcontainer.yaml
# 查看
[root@master01 1]# kgp -n study
NAME READY STATUS RESTARTS AGE
php-updated 0/1 Init:CrashLoopBackOff 1 (6s ago) 8s
6.2 故障排查¶
1、使用describe命令查看详细信息
[root@master01 1]# kdp php-updated -nstudy
Name: php-updated
Namespace: study
Priority: 0
Service Account: default
Node: node01/192.168.1.61
Start Time: Fri, 10 Nov 2023 08:54:34 +0800
Labels: <none>
Annotations: cni.projectcalico.org/containerID: aec198a7e28438e24f13cf5fde574fcf18f308c189a364923bcea9fe776f97c6
cni.projectcalico.org/podIP: 172.29.55.47/32
cni.projectcalico.org/podIPs: 172.29.55.47/32
Status: Pending
IP: 172.29.55.47
IPs:
IP: 172.29.55.47
Init Containers:
install:
Container ID: containerd://d2c362329804b626b91c0d59e0b9423b720b1e072fe202c61a6e385a4ffb7756
Image: registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox
Image ID: registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox@sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee
Port: <none>
Host Port: <none>
Command:
wget
-O
/var/www/html/index.php
https://gitee.com
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 10 Nov 2023 08:57:33 +0800
Finished: Fri, 10 Nov 2023 08:57:33 +0800
Ready: False
Restart Count: 5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8rk8x (ro)
/var/www/html/ from dir (rw)
Containers:
php:
Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/abroad_images/php:7-fpm
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8rk8x (ro)
/var/www/html/ from dir (rw)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-8rk8x:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m56s default-scheduler Successfully assigned study/php-updated to node01
Normal Pulled 3m55s kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox" in 396.025664ms (396.043724ms including waiting)
Normal Pulled 3m54s kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox" in 453.578638ms (453.586758ms including waiting)
Normal Pulled 3m37s kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox" in 430.515061ms (430.529231ms including waiting)
Normal Created 3m6s (x4 over 3m55s) kubelet Created container install
Normal Started 3m6s (x4 over 3m55s) kubelet Started container install
Normal Pulled 3m6s kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox" in 511.002671ms (511.009511ms including waiting)
Warning BackOff 2m37s (x7 over 3m53s) kubelet Back-off restarting failed container install in pod php-updated_study(b433389e-3ed2-4960-bc11-61c72b1142ec)
Normal Pulling 2m24s (x5 over 3m56s) kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox"
Normal Pulled 2m23s kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox" in 527.655048ms (527.659228ms including waiting)
2、查看install容器的详细日志
[root@master01 1]# k logs -f php-updated -nstudy -c install
Connecting to gitee.com (180.76.198.77:443)
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer
3、查找资料https://github.com/GoogleContainerTools/kaniko/issues/2452，定位到是目前busybox版本的问题。替换busybox的版本为1.28。重新创建即可
[root@master01 1]# cat pod-initcontainer.yaml
apiVersion: v1
kind: Pod
metadata:
name: php-updated
namespace: study
spec:
containers:
- name: php
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/php:7-fpm
imagePullPolicy: IfNotPresent
volumeMounts:
- name: dir
mountPath: /var/www/html/
initContainers:
- name: install
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/busybox:1.28
volumeMounts:
- name: dir
mountPath: /var/www/html/
command:
- wget
- "-O"
- "/var/www/html/index.php"
- https://gitee.com
volumes:
- name: dir
emptyDir: {}