The Operator Framework is an open source toolkit developed by CoreOS and later acquired by RedHat that manages Kubernetes native applications in an efficient, automated, and extensible manner. The framework includes the Operator SDK, which helps developers leverage their expertise to guide and build operators without having to understand the complexities of the Kubernetes API. Today we will learn how to create an Ansible-based Operator application (discussed briefly in the Loki Operator Tutorial). It leverages existing Ansible PlayBook and modules to deploy and manage Kubernetes resources as a unified application lifecycle, without writing any Go code.

The installation

  • Homebrew installation (MacOS)

If you are using Homebrew, you can install the SDK CLI tool using the following command.

$ brew install operator-sdk
Copy the code
  • Binary installation
$ export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac) $ export OS=$(uname | awk '{print tolower($0)}') $ export Download OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.10.0 # $curl operator - SDK -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH} $ chmod +x operator-sdk_${OS}_${ARCH} && sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdkCopy the code
  • Compile the installation
$ git clone https://github.com/operator-framework/operator-sdk
$ cd operator-sdk
$ git checkout master
$ make install
Copy the code

Initialize the

1. Initialize the project

Use the operer-sdk command to quickly initialize the space for this project.

$ mkdir loki-operator
$ cd loki-operator
$ operator-sdk init --plugins=ansible --domain cloudxiaobai.com
Copy the code

The file generated by this command contains a Kubebuilder PROJECT description file. From it, we can know that this project is based on Ansible implementation

$ cat PROJECT

domain: loki.cloudxiaobai.com
layout:
- ansible.sdk.operatorframework.io/v1
plugins:
  manifests.sdk.operatorframework.io/v2: {}
  scorecard.sdk.operatorframework.io/v2: {}
projectName: cloudxiaobai
version: "3"
Copy the code

2. Create API

$ operator-sdk create api \
      --group=plugins --version=v1 --kind=Loki \
      --generate-playbook \
      --generate-role
Copy the code

This command will quickly help us build the scaffolding for this project. It contains the following contents:

  • Generated the definition of Loki CRD, and a sample Loki CR;
  • A Controller is generated to manage the status of CR’s services on the Kubernetes cluster
  • The directory structure of Ansible Playbook is generated
  • A watches. Yaml file is generated, which is used to associate CR resources with Ansible Playbook.

Ansible Playbook

After completing the initialization of the project, we have the following main directories under the project.

├── config │ ├── samples │ ├─ ├─ ch.htm │ ├─ ch.htm Loki. Yml ├ ─ ─ roles / / the device file │ └ ─ ─ Loki │ ├ ─ ─ the README. Md │ ├ ─ ─ defaults │ │ └ ─ ─ the main, yml │ ├ ─ ─ files │ ├ ─ ─ handlers │ │ └ ─ ─ main. Yml │ ├ ─ ─ meta │ │ └ ─ ─ the main, yml │ ├ ─ ─ the tasks │ │ └ ─ ─ the main, yml │ ├ ─ ─ templates │ └ ─ ─ vars │ └ ─ ─ main. YmlCopy the code

If you’re familiar with Ansible, let’s start writing a simple Playbook file for the Loki service to demonstrate its basic logic.

1. Modify the roles/Loki/tasks/main yaml

- name: Loki Operator | Loki | StatefulSet
  community.kubernetes.k8s:
    definition: "{{ lookup('template', 'statefulset.yaml') | from_yaml }}"
Copy the code

You can see that Ansible interacts with Kubernetes through the community. Kubernetes K8S module. Go to the templates directory and look for the statefulset.yaml file. Render it as a YAML file and submit it to Kubernetes.

2. Create roles/Loki/template/statefulset yaml

Statefulset. yaml is the ansible template file that needs to be deployed. We need to create the following contents:

#jinja2:lstrip_blocks: True
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: '{{ansible_operator_meta.name}}'
  namespace: '{{ansible_operator_meta.namespace}}'
  labels:
     app.kubernetes.io/name: '{{ ansible_operator_meta.name }}-loki-system'
spec:
  replicas: {{ replicas }}
  serviceName:  '{{ansible_operator_meta.name}}-headless'
  selector:
    matchLabels:
       app.kubernetes.io/name: '{{ ansible_operator_meta.name }}-loki-system'
  template:
    metadata:
      labels:
         app.kubernetes.io/name: '{{ ansible_operator_meta.name }}-loki-system'
    spec:
      containers:
      - name: loki-system
        image: grafana/loki:2.2.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3100
          name: http-3100
          protocol: TCP
Copy the code

The Ansible application template above has several points worth highlighting:

  • #jinja2:lstrip_blocks: TrueKeep statements such as whitespace, indentation, and so on in YAML files
  • {{ansible_operator_meta.name}}Defines the name of the Loki resource controlled in CR
  • {{ansible_operator_meta.namespace}}Defines the namespace in which CR functions
  • {{ replicas }}Custom CR variables

You can see that we have declared a custom variable named Replicas that we need to control the number of copies of the service. The values of variables in Ansible are controlled by user-created CRS.

Tasks in the Ansible Roles file actually define the CR state. In Kubernetes when creating resources, we do not need to define the actual CR field type declaration in the CRD because it allows input of any field. Although it is not automatically generated in the Operator SDK, it is recommended to add a CRD field description for actual use, so that Kubernetes users can see its corresponding description when using the CR.

3. Create a CR

By default, the Operator SDK creates a default CR sample file, which is under the config/samples/ directory. In this article, our CR file name is plugins_v1_loki.yaml

apiVersion: plugins.cloudxiaobai.com/v1
kind: Loki
metadata:
  name: loki
spec:
  replicas: 1
Copy the code

Build and deploy

Once the playbook of the application is written, we can use the make command to build and upload the image. We can pre-edit the Makefile to define the image name,

IMAGE_TAG_BASE ? = cloudxiaobai.com/cloudxiaobai IMG ?= $(IMAGE_TAG_BASE):$(VERSION)Copy the code

This way, we generate a fixed image name each time we build the image.

1. Build an Operator image

$make docker-build docker-push VERSION="0.1"Copy the code

Build and upload the Operator image.

2. Deploy the Operator service

$ make deploy
# kustomize build config/default | kubectl apply -f -
Copy the code

You can see that kustomize is called at deployment time to build the files in the config/default directory and commit them to Kubernetes. The important thing to note here is that by default, Kustomize creates a new namespace named -system for deploying operators.

When the deployment is complete, we can verify the service status with Kubectl.

$ kubectl get deployment -n cloudxiaobai--system

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
loki-controller-manager   1/1     1            1           8m
Copy the code

3. Submit CR

Submit the CR statement above to Kubernetes

$ cat config/samples/plugins_v1_loki.yaml

apiVersion: plugins.cloudxiaobai.com/v1
kind: Loki
metadata:
  name: loki
spec:
  replicas: 1

$ kubectl apply -f config/samples/plugins_v1_loki.yaml
Copy the code


 then we can verify that the service is working as expected with the following command

$ kubectl get deployment

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
cloudxiaobai-31e1d3526-a1klm             1/1     1            1           1m
Copy the code

If we want to adjust the number of Loki copies, we only need to modify the REPLICas variable in CR or modify it through patch.

$ kubectl patch loki cloudxiaobai  -p '{"spec":{"size": 2    }}' --type=merge
Copy the code

4. Delete resources

Directly run the following commands to delete all resources associated with CR and Operator respectively

$ kubectl delete -f config/samples/plugins_v1_loki.yaml
$ make undeploy
Copy the code

tip

1. Transform Ansible variables

The names of all variables in the Spec field in the custom CR are converted by the operator to snake_case (underlined below) naming format. For example, if we write a variable serviceAccount in spec, it will be converted to service_account in Ansible. You can disable this case conversion by setting snakeCaseParameters to false in watches. Yaml.

2. Use the default value

In order to adapt Ansible Template to most scenarios, Xiao White recommends using the default value in the template to avoid playbook execution errors caused by no variables defined in CR. For example, we defined Loki mirroring information in CR as follows.

Spec: image: repository: grafana/ Loki tag: 2.1.1Copy the code

We can write this on the template:

containers:
      - name: loki-system
        image: '{{service.cluster.loki.image | default('grafana/loki')}}:{{version | default('latest')}}'
Copy the code

Here, if we don’t define the image variable in CR, when the Playbook executes here, it will use grafana/ Loki: Latest image.

3. Use opportunely items

Since the community.kubernetes.k8s module does not allow the submission of a YAML file for Allinone, when we actually write the Playbook, if we want to complete all YAML submissions inone task, we can use the following statement:

- name: Loki Operator | Loki | Deploy
  community.kubernetes.k8s:
    definition: "{{ lookup('template', item) | from_yaml }}"
  with_items:
    - statefulset.yaml
    - service.yaml
Copy the code

4. Enable or disable the control module

If we want to control whether a service is enabled or disabled in our CR, we usually use enabled: true to define it explicitly, such as in this CR

spec:
  redis:
    enabled: true
Copy the code

So, when we write the Playbook, we can determine whether to execute by judging the variable in CR in state, as follows:

- name: Loki Operator | Reids Cache | PVC
  community.kubernetes.k8s:
    state: '{{"present" if redis is defined and redis.enabled else "absent"}}'
    definition: "{{ lookup('template', 'redis/pvc.yaml') | from_yaml }}
Copy the code

5. Owner References

Owner References is the garbage removal mechanism in Kubernetes, which can clean up after CR is deleted. By default, ownerReferences is injected by the proxy at ansible execution time. If both CR and managed resources are in the same namespace, ownerReferences will be displayed in the resource as follows:

metadata:
  ownerReferences:
    - apiVersion: loki.cloudxiaobai.com/v1
      kind: loki
      name: loki
      uid:fda3b711-1f25-4d14-bf84-cf70f48e7ff4
Copy the code

When CR and the managed resource are not in the same namespace, ownerReferences are defined using the following annotation field:

operator-sdk/primary-resource: {metadata.namespace}/{metadata.name}
operator-sdk/primary-resource-type: {kind}.{group}
Copy the code

If you want to disable ownerReferences, modify the dockerfile as follows:

ENTRYPOINT ["/usr/local/bin/entrypoint", "--inject-owner-ref=false"]
Copy the code

6. Task concurrency

By default, Ansible calls Runtime.numcpu () to set the maximum concurrency coordination value, which you can see depends on the number of CPU cores on your host. The coordination performance of Ansible can be improved by increasing the number of concurrent coordination and allowing concurrent processing of events. We can set this manually in the operator startup parameter

- name: manager
  args:
    - "--max-concurrent-reconciles"
    - "3"
Copy the code

conclusion

This article uses a small example to show how to quickly build and manage Loki applications using the Operator SDK. And through 6 tips let the reader in the use of the process as far as possible to avoid the pit. At present, In practical work, Ansible Operator is mainly applied to the installation management of various plug-ins and third-party services during the initialization of Kubernetes platform, such as Prometheus, Loki, grafna, etc. I hope it can also help you in other aspects.

For more content, please pay attention to “Cloud native white”