Kubernetes扩展：Operator初次上手

目前主流的Operator开发框架有两个：kubebuilder和Operator-sdk，两者实际上并没有本质的区别，它们的核心都是使用官方的 controller-tools 和 controller-runtime。不过细节上稍有不同，比如 kubebuilder 有着更为完善的测试与部署以及代码生成的脚手架等；而 operator-sdk 对 ansible operator 这类上层操作的支持更好一些。

下面基于kubebuilder，讲解如何开发Operator

1. 环境准备

Kubebuilder工作依赖go环境，所以需要安装go，理应单独拿一台机器来安装Kubebuilder，但我们为了节省资源，就拿k8s-master02来安装。

1）安装go

Centos7.9使用yum安装

$ yum install -y epel-release
$ yum install -y golang.x86_64

检测版本

$ go version

设置代理

$ go env -w GOPROXY=https://goproxy.cn,direct

将go env GOPATH回显内容/root/go输出到/etc/profile,并再次执行

$ go env GOPATH
$ echo "export GOPATH=/root/go/" >> /etc/profile
$ source /etc/profile

2）安装docker

开发过程中会用到docker环境，由于我们部署k8s时，安装过containerd，当时配置过yum仓库，所以可以直接通过yum来安装docker

$ yum install -y docker-ce

启动服务

$ systemctl start docker

3）安装kubectl 以及配置直接访问k8s集群

由于是k8s-master02，已经安装过kubectl，如果没有安装请参考部署k8s集群的步骤来安装。而默认k8s-master02是无法直接访问k8s集群的，需要将k8s-master01下面的/root/.kube目录拷贝到k8s-master02才可以

$ scp -r /root/.kube  k8s-master02:/root/

测试，在k8s-master02上查看node节点

$ kubectl get node

4）安装kubebuilder

$ wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.10.0/kubebuilder_linux_amd64
$ mv kubebuilder_linux_amd64 kubebuilder && chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

测试,在k8s-master02上查看kubebuilder版本

$ kubebuilder version

Version: main.version{KubeBuilderVersion:"3.10.0", KubernetesVendor:"1.26.1", GitCommit:"0fa57405d4a892efceec3c5a902f634277e30732", BuildDate:"2023-04-15T08:10:35Z", GoOs:"linux", GoArch:"amd64"}

2. 创建Helloworld项目

1）在k8s-master02上初始化项目

$ export GOPATH=`go env GOPATH`
$ mkdir -p $GOPATH/src/helloworld
$ cd $GOPATH/src/helloworld
$ kubebuilder init --domain aminglinux.com

初始化完成后，目录结构是：

$ cd $GOPATH/src/helloworld
$ yum install -y tree
$ tree .

├── cmd
│   └── main.go
├── config
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   └── rbac
│       ├── auth_proxy_client_clusterrole.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_service.yaml
│       ├── kustomization.yaml
│       ├── leader_election_role_binding.yaml
│       ├── leader_election_role.yaml
│       ├── role_binding.yaml
│       └── service_account.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── Makefile
├── PROJECT
└── README.md

2）创建API（CRD + Controller）

先在k8s-master02上安装make

$ yum install -y make

创建API

$ cd $GOPATH/src/helloworld
$ kubebuilder create api --group webapp --version v1 --kind Guestbook  ##打两次y

...
...
...
Create Resource [y/n]
y
Create Controller [y/n]
y

3）构建和部署CRD

$ cd $GOPATH/src/helloworld
$ make install

这个过程会将CRD部署到k8s集群里,此时在k8s-master02上查看CRD

$ kubectl get crd |grep aminglinux.com

guestbooks.webapp.aminglinux.com                      2023-10-06T01:47:51Z

可以通过下面命令查看该CRD对应的yaml

$ cd $GOPATH/src/helloworld
$ /root/go/src/helloworld/bin/kustomize build config/crd

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: guestbooks.webapp.aminglinux.com
spec:
  group: webapp.aminglinux.com
  names:
    kind: Guestbook
    listKind: GuestbookList
    plural: guestbooks
    singular: guestbook
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Guestbook is the Schema for the guestbooks API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: GuestbookSpec defines the desired state of Guestbook
            properties:
              foo:
                description: Foo is an example field of Guestbook. Edit guestbook_types.go
                  to remove/update
                type: string
            type: object
          status:
            description: GuestbookStatus defines the observed state of Guestbook
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

4）编辑Controller对应的源码，并编译

如果是生产环境，此时就要去编辑Controller对应的go程序啦，由于这里是体验过程，所以只做简单改动

源码文件路径为：$GOPATH/src/helloworld/internal/controller/guestbook_controller.go

$ vi $GOPATH/src/helloworld/internal/controller/guestbook_controller.go

改动1：增加一个依赖包fmt

Operator初次上手-1

改动2：找到// TODO(user): your logic here，在下面增加一行代码，用来打印堆栈信息

fmt.Println("Helloworld.")

Operator初次上手-2

改完后，在k8s-master02上执行

$ cd $GOPATH/src/helloworld
$ make run

这样就可以将该Controller运行起来了。会显示如下信息

test -s /root/go/src/helloworld/bin/controller-gen && /root/go/src/helloworld/bin/controller-gen --version | grep -q v0.11.3 || \
GOBIN=/root/go/src/helloworld/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.11.3
/root/go/src/helloworld/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/go/src/helloworld/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./cmd/main.go
2023-10-06T12:27:28+08:00       INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
2023-10-06T12:27:28+08:00       INFO    setup   starting manager
2023-10-06T12:27:28+08:00       INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-10-06T12:27:28+08:00       INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-10-06T12:27:28+08:00       INFO    Starting EventSource    {"controller": "guestbook", "controllerGroup": "webapp.aminglinux.com", "controllerKind": "Guestbook", "source": "kind source: *v1.Guestbook"}
2023-10-06T12:27:28+08:00       INFO    Starting Controller     {"controller": "guestbook", "controllerGroup": "webapp.aminglinux.com", "controllerKind": "Guestbook"}
2023-10-06T12:27:28+08:00       INFO    Starting workers        {"controller": "guestbook", "controllerGroup": "webapp.aminglinux.com", "controllerKind": "Guestbook", "worker count": 1}

注意：不要按ctrc c中断，此时需要到k8s-master01上

5）到k8s创建Guestbook资源的实例

现在kubernetes已经部署了Guestbook类型的CRD，而且对应的controller也已正在运行中，可以尝试创建Guestbook类型的实例了(相当于有了pod的定义后，才可以创建pod)；

kubebuilder已经自动创建了一个类型的部署文件：$GOPATH/src/helloworld/config/samples/webapp_v1_guestbook.yaml ，内容如下，很简单，接下来在k8s-master01节点上用这个文件来创建Guestbook实例：

$ cat > guestbook.yaml <<EOF
apiVersion: webapp.aminglinux.com/v1
kind: Guestbook
metadata:
  labels:
    app.kubernetes.io/name: guestbook
    app.kubernetes.io/instance: guestbook-sample
    app.kubernetes.io/part-of: helloworld
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: helloworld
  name: guestbook-sample
spec:
  # TODO(user): Add fields here
  foo: bar
EOF

在k8s-master01节点上应用此yaml

$ k apply -f guestbook.yaml

回到k8s-master02节点的终端，可以看到多了一行输出

...
...

Helloworld.

6）将Controller制作成镜像，并上传到远程仓库

首先需要有一个私有镜像仓库，用来存储编译好的镜像。如果有harbor直接使用harbor最好，如果没有，就是用docker的镜像仓库hub.docker.com，假设你已经有账号了。

在编译镜像之前还需要登录到docker的镜像仓库

$ docker login  https://hub.docker.com

这里使用的是harbor,在k8s-master02节点上登录harbor

$ vi /etc/docker/daemon.json

{
  "registry-mirrors": ["https://y0araofw.mirror.aliyuncs.com"],
  "insecure-registries": ["192.168.1.35", "k8s-node02"]
}
$ systemctl daemon-reload
$ systemctl restart docker
$ echo "192.168.1.35 www.zhang-qing.com" >> /etc/hosts
$ docker login www.zhang-qing.com -u admin -p Harbor12345

在k8s-master02节点上给Dockerfile里增加GOPROXY设置

$ cd /root/go/src/helloworld
$ vi  Dockerfile

#在go mod download上面增加一行
RUN go env -w GOPROXY=https://goproxy.cn
将gcr.io/distroless/static:nonroot替换为registry.cn-hangzhou.aliyuncs.com/abroad_images/static:nonroot
registry.cn-hangzhou.aliyuncs.com/abroad_images/static:nonroot

Operator初次上手-3

浏览器输入www.zhang-qing.com打开harbor界面，点击【新建项目】后，输入项目名称-aming来创建项目

Operator初次上手-4

编译镜像

$ make docker-build docker-push IMG=www.zhang-qing.com/aming/guestbook:v1

刷新harbor界面，观察到镜像已上传完成

Operator初次上手-5

7）在k8s里部署该镜像

部署之前，需要把之前设置的代理取消，否则会出错

$ unset http_proxy
$ unset https_proxy

在k8s-master02节点上修改kube-rbac-proxy镜像地址，由gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1镜像修改为registry.cn-hangzhou.aliyuncs.com/abroad_images/kube-rbac-proxy:v0.13.1

$ cd /root/go/src/helloworld
$ vim config/default/manager_auth_proxy_patch.yaml

Operator初次上手-6

在k8s-master02节点上修改kube-rbac-proxy镜像地址，由controller:latest镜像修改为www.zhang-qing.com/aming/guestbook:v1

$ cd /root/go/src/helloworld
$ vim config/manager/manager.yaml

Operator初次上手-7

将harbor项目改为公开，方便下载

Operator初次上手-8

在五个k8s node上手动将镜像下载下来

$ docker pull www.zhang-qing.com/aming/guestbook:v1

在k8s-master02节点上部署

$ cd /root/go/src/helloworld
$ make deploy www.zhang-qing.com/aming/guestbook:v1

在k8s-master02节点上查看pod

$ kubectl get po -n helloworld-system

NAME                                            READY   STATUS    RESTARTS   AGE
helloworld-controller-manager-c6c6d744f-55j8d   2/2     Running   0          14s

在k8s-master02节点上查看pod日志

$ kubectl  logs -f -n helloworld-system helloworld-controller-manager-c6c6d744f-55j8d

2023-10-07T00:29:37Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
2023-10-07T00:29:37Z    INFO    setup   starting manager
I1007 00:29:37.508186       1 leaderelection.go:248] attempting to acquire leader lease helloworld-system/3b9f5c61.aminglinux.com...
2023-10-07T00:29:37Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-10-07T00:29:37Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
I1007 00:29:37.515674       1 leaderelection.go:258] successfully acquired lease helloworld-system/3b9f5c61.aminglinux.com
2023-10-07T00:29:37Z    DEBUG   events  helloworld-controller-manager-c6c6d744f-55j8d_1f14451e-0e6f-4030-8ff9-b5c6c571eab3 became leader        {"type": "Normal", "object": {"kind":"Lease","namespace":"helloworld-system","name":"3b9f5c61.aminglinux.com","uid":"75ce8221-33fc-4b38-a2c9-28a376396346","apiVersion":"coordination.k8s.io/v1","resourceVersion":"355710"}, "reason": "LeaderElection"}
2023-10-07T00:29:37Z    INFO    Starting EventSource    {"controller": "guestbook", "controllerGroup": "webapp.aminglinux.com", "controllerKind": "Guestbook", "source": "kind source: *v1.Guestbook"}
2023-10-07T00:29:37Z    INFO    Starting Controller     {"controller": "guestbook", "controllerGroup": "webapp.aminglinux.com", "controllerKind": "Guestbook"}
2023-10-07T00:29:37Z    INFO    Starting workers        {"controller": "guestbook", "controllerGroup": "webapp.aminglinux.com", "controllerKind": "Guestbook", "worker count": 1}
Helloworld.

在k8s-master01节点上再次去apply guestbook.yaml

$ kubectl delete -f guestbook.yaml
$ kubectl apply -f guestbook.yaml

在k8s-master02节点上再去查看helloworld-controller-manager-c6c6d744f-55j8d的log,观察到又输出了两次Helloworld.

$ kubectl  logs -f -n helloworld-system helloworld-controller-manager-c6c6d744f-55j8d

...
...
Helloworld.
Helloworld.
Helloworld.

8）清理

在k8s-master02节点上清理CRD资源

$ cd /root/go/src/helloworld
$ make uninstall

在k8s-master02节点上清理Controller

$ kubectl delete ns helloworld-system

评论区