Kubernetes Informer source code parsing and deep use [3/4]: implementation of custom resource (CRD) Informer

This paper is divided into four parts:

(1). Cache package source code analysis and the use of Informer;

(2). Informers package source code parsing and the use of SharedInformerFactory, and the best practice of Informer in the actual use;

(3). Implementation of custom resource (CRD) Informer;

(4). The dynamic package source code parsing and DynamicSharedInformerFactory use;

This is part 3.

All code used in this article can be found in Articles /archive/ dive-into-Kubernetes-Informer at Main · wbsnail/articles · GitHub.

Custom Resource Definition (CRD)

While the built-in components and resource types of Kubernetes cover a wide range of usage scenarios, users’ needs vary widely. As an infrastructure, Kubernetes provides a variety of ways to extend Kubernetes. Custom resources are an important way to extend Kubernetes. It is worth noting that custom resources have no different status in Kubernetes than built-in resource types: all resource types are equal in Kubernetes.

About the definition and resources to add and delete custom resources is outside the realm of this operation, the official document has detailed instructions: Extend the Kubernetes API with CustomResourceDefinitions | Kubernetes.

SharedInformerFactory

In the previous section, I analyzed the Informers package in the Kubernetes/Client-Go repository and explained the proper use of SharedInformerFactory. However, this package only involves Kubernetes built-in resource types, and we definitely need to use programs to maintain the state of custom resources. How to implement the controller for custom resources? I’m going to show you how to implement custom resource Informer.

👮♂️ If you haven’t seen the previous section, I suggest you finish it first.

Code generation

Code generation is a common pattern in major programming languages, and when I say code generation, I don’t mean the process of compiling and generating machine code, but the process of generating code in the same language. Why do I need to generate code? Reasons such as:

(1) Automatically generate some codes with the same pattern to improve efficiency;

(2) convenient unified code specification;

(3). Allow more complex logic to be described in a concise way.

Code-generator is a code generation suite for Kubernetes that provides a number of code generation commands for generating custom resource-related code, including the Informer of interest.

In fact all the Lister, Informer, SharedInformerFactory of Kubernetes built-in resources are generated using code-generator. Again: all resource types are equal in Kubernetes.

Create custom resource Informer using code-Generator

This section describes how to use code-generator

Code-generator is a set of simple command line tools for generating code related to custom resources. The command list is code-generator/ CMD at master · kubernetes/code-generator · GitHub.

For example, deepcopy-gen is used to generate deepcopy methods for types. The reason for this is that after Kubernetes 1.8, we added the DeepCopyObject method to runtime.Object for custom resource types. Unlike other runtime.Object methods, this method cannot be abstracted as a public method because it returns an Object of the corresponding type:

type Object interface {
	// ...

	DeepCopyObject() Object
}
Copy the code

This means that for each custom type, we need to define a DeepCopyObject method, such as type T:

type T struct {}

func (t *T) DeepCopy(a) *T {
    // ...
}

func (t *T) DeepCopyObject(a) runtime.Object {
    return t.DeepCopy()
}
Copy the code

🤔 DeepCopy () : *T () : *T (); Is it possible that I can abstract out a method that returns Runtime. Object and reuse it without the need for deepcopy-gen? You can think about it for yourself.

A typical DeepCopyObject ends up with logic like this:

func (in *T) DeepCopyInto(out *T) {
	*out = *in
	out.TypeMeta = in.TypeMeta
	in.ListMeta.DeepCopyInto(&out.ListMeta)
	ifin.Items ! =nil {
		in, out := &in.Items, &out.Items
		*out = make([]Service, len(*in))
		for i := range *in {
			(*in)[i].DeepCopyInto(&(*out)[i])
		}
	}
	return
}

func (in *T) DeepCopy(a) *T {
	if in == nil {
		return nil
	}
	out := new(T)
	in.DeepCopyInto(out)
	return out
}

func (in *T) DeepCopyObject(a) runtime.Object {
	ifc := in.DeepCopy(); c ! =nil {
		return c
	}
	return nil
}
Copy the code

It’s a deepcopy of each field recursively. It follows a pattern, so we can write a program to automatically generate this method. Deepcopy-gen generates this code.

All commands in code-generator are implemented based on another package, Kubernetes/Gengo, so they share some parameters. Notable ones are:

InputDirs (–input-dirs): import directory of packages of the type to be resolved;
OutputBase (–output-base): the base disk directory of the output file, default is “$GOPATH/ SRC “;
OutputPackagePath (–output-package): import path of output package;
GoHeaderFilePath (–go-header-file): specifies the path of the copyright header file.

In addition to shared parameters, each command has its own parameters.

All code used in this article can be found in Articles /archive/ dive-into-Kubernetes-Informer at Main · wbsnail/articles · GitHub.

The following operations are performed in the archive/dive-into-kubernetes-informer/11-crd-informer directory of the Wbsnail/Articles repository.

Due to the design of the code-generator command, the code needs to be stored in the Golang default SRC directory structure. If you use the default “$GOPATH/ SRC “, Above code needs to be cloned to $GOPATH/src/github.com/wbsnail/articles, if use the go module, also requires the directory structure for/somewhere/github.com/wbsnail/articles “, Also set –output-base=/somewhere, otherwise the generated code file location will be wrong. If you don’t want to worry about this, it is recommended to clone the repository to “$GOPATH/ SRC “.

The code-generator command needs to be installed using Golang. There is no release binary on Github. To do this, clone code-generator locally and go install./ CMD /deepcopy-gen.

Actually code – the generator synchronous since kubernetes/staging/SRC/k8s. IO/code – the generator at the master, kubernetes/kubernetes, making, changes have occurred in the latter, It then synchronizes to the code-generator repository. So if you have local kubernetes warehouse, also can go the staging/SRC/k8s. IO/code – the generator/CMD/deepcopy – gen.

Take deepcopy-gen as an example to see how to use the code-generator command.

Create a file/api/stable.wbsnail.com/v1/doc.go:

// +k8s.deepcopy-gen=package

package v1
Copy the code

And the. / api/stable.wbsnail.com/v1/types.go:

package v1

type Rabbit struct {
	Color string `json:"color,omitempty"`
}
Copy the code

And copyright header file./boilerplate.go.txt:

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */
Copy the code

The disk directory where the type files reside is./ API /{group}/{version}. This is not mandatory, but it is best to keep it official with Kubernetes, including the names “API” and “types.go”. But the file name “doc.go” is mandatory by code-generator, which declares some code generation parameters for the package.

“// +k8s:deepcopy-gen=package” is a comment, but those familiar with Golang should know that annotations are often used to do useful things, such as “// +build” used in go build. Go Embed uses “//go: Embed “and so on. // +k8s:deepcopy-gen=package” tells deepcopy-gen to generate all types of deep copies of the package.

Types.go needless to say, is the file in which we define the types.

Boilerplate.go. TXT is a copyright header file that is added to the beginning of every generated file.

Perform:

deepcopy-gen \
  --input-dirs github.com/wbsnail/articles/archive/dive-into-kubernetes-informer/11-crd-informer/api/stable.wbsnail.com/v1 \
  --output-file-base zz_generated.deepcopy \
  --go-header-file ./boilerplate.go.txt
Copy the code

Where –input-dirs declares the package import directory of the type we need to resolve. –output-file-base is the generated file name, the default is deepcopy.generated, we use zz_generated. –go-header-file Specifies the disk path of the copyright header file.

Again :(1). All operations are performed under archive/dive-into-kubernetes-informer/11-crd-informer in wbsnail/articles; (2). Because the code – the generator command design problem, the code needs to deposit to Golang default SRC directory structure, such as “$GOPATH/src/github.com/wbsnail/articles”.

After execution, 1 file is generated:

./api/stable.wbsnail.com/v1/zz_generated.deepcopy.go:
Copy the code

// +build ! ignore_autogenerated

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */

// Code generated by deepcopy-gen. DO NOT EDIT.

package v1

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Rabbit) DeepCopyInto(out *Rabbit) {
	*out = *in
	return
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Rabbit.
func (in *Rabbit) DeepCopy(a) *Rabbit {
	if in == nil {
		return nil
	}
	out := new(Rabbit)
	in.DeepCopyInto(out)
	return out
}
Copy the code

The generated file has four parts:

(1). “// +build ! Ignore_autogenerated “, stating that the file is automatically generated;

(2). Copyright information;

// Code generated by deepcopy-gen. DO NOT EDIT.” // Code generated by deepcopy-gen. DO NOT EDIT.”

(4). Generated deep copy method.

🤖 You may have noticed that there is no DeepCopyObject method in the generated code. Do you know why?

Code Generation principles

The code-generator is actually a Gengo package with the go/ AST package and the Go/Parser package, which parses Golang code and generates the corresponding abstract syntax tree (AST). With these two packages, Gengo gets the abstract syntax tree of the package we specified and is able to walk through all of its types. As for the generated snippet, it’s a simple (and perhaps not simple) template rendering.

The core method of gengo package exposure is * generatorargs.execute:

func (g *GeneratorArgs) Execute(
	nameSystems namer.NameSystems,
	defaultSystem string,
	pkgs func(*generator.Context, *GeneratorArgs) generator.Packages.
) error {}
Copy the code

The most important of the parameters it receives is PKGS, which allows the user to build packages to be generated from * generator.context and *GeneratorArgs. Generators have another important method in generator.Packages returned by PKGS:

type Generator interface {
	// There are many other methods, including generating Vars, Consts, Imports, and so on

	// GenerateType should emit the code for a particular type.
	GenerateType(*Context, *types.Type, io.Writer) error
}


// Package contains the contract for generating a package.
type Package interface {
	// ...

	// Generators returns the list of generators for this package. It is
	// allowed for more than one generator to write to the same file.
	// A Context is passed in case the list of generators depends on the
	// input types.
	Generators(*Context) []Generator
}
Copy the code

Generators the key to using the Gengo package is to define Generators, where each generator will generate a section of code based on the package information and context, which will be generated by gengo as a.go file and output to disk files.

Take a look at a snippet of Generator code implemented in deepcopy-Gen:

	sw := generator.NewSnippetWriter(w, c, "$"."$")
	args := argsFromType(t)

	if deepCopyIntoMethodOrDie(t) == nil {
		sw.Do("// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.\n", args)
		if isReference(t) {
			sw.Do("func (in $.type|raw$) DeepCopyInto(out *$.type|raw$) {\n", args)
			sw.Do("{in:=&in\n".nil)}else {
			sw.Do("func (in *$.type|raw$) DeepCopyInto(out *$.type|raw$) {\n", args)
		}
		ifdeepCopyMethodOrDie(t) ! =nil {
			if t.Methods["DeepCopy"].Signature.Receiver.Kind == types.Pointer {
				sw.Do("clone := in.DeepCopy()\n".nil)
				sw.Do("*out = *clone\n".nil)}else {
				sw.Do("*out = in.DeepCopy()\n".nil)
			}
			sw.Do("return\n".nil)}else {
			g.generateFor(t, sw)
			sw.Do("return\n".nil)}if isReference(t) {
			sw.Do("}\n".nil)
		}
		sw.Do("}\n\n".nil)}Copy the code

You can see how the DeepCopy method is generated.

More details on how the code is generated will not be covered here, but are interested in reading the code-Generator and Gengo source code for yourself. Next we will focus on the use of code-generator commands.

Custom resource: Rabbit

Before we proceed, let’s define a complete custom resource that fits our general impression:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: rabbits.stable.wbsnail.com
spec:
  group: stable.wbsnail.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              color:
                type: string
    additionalPrinterColumns:
    - name: Color
      type: string
      description: Color
      jsonPath: .spec.color
    - name: Age
      type: date
      description: Age
      jsonPath: .metadata.creationTimestamp
  scope: Namespaced
  names:
    plural: rabbits
    singular: rabbit
    kind: Rabbit
Copy the code

Apply it to the cluster and create two sample resources:

apiVersion: "stable.wbsnail.com/v1"
kind: Rabbit
metadata:
  name: judy
  namespace: tmp
spec:
  color: white
---
apiVersion: "stable.wbsnail.com/v1"
kind: Rabbit
metadata:
  name: bugs
  namespace: tmp
spec:
  color: gray
Copy the code

Try getting Rabbits.

kubectl get rabbits -n tmp
Copy the code

The output should look something like:

NAME   COLOR   AGE
bugs   gray    1h
judy   white   1h
Copy the code

You can play with the rabbits, try adding, deleting, changing, and checking, and then we’ll move on to the next step.

Adjust type file

Doc. Go, types. Go, boilerplate.go. TXT. Since we have adjusted the Rabbit structure, we need to modify the types.

package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type Rabbit struct {
	metav1.TypeMeta `json:",inline"`
	// +optional
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec RabbitSpec `json:"spec,omitempty"`
}

type RabbitSpec struct {
	Color string `json:"color,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type RabbitList struct {
	metav1.TypeMeta `json:",inline"`
	// +optional
	metav1.ListMeta `son:"metadata,omitempty"`

	Items []Rabbit `json:"items"`
}
Copy the code

Let’s find out what changes we made:

(1). “// + genClient “is declared to generate Clientset for the next type (from top to bottom, the one closest to this annotation, i.e. Rabbit);

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
Copy the code

(2) create a DeepCopyX method, where X is the interface specified;

Metav1. TypeMeta and metav1.ObjectMeta are embedded to implement other methods of Runtime. Object, while providing Kubernetes resource default many fields, Name, Namespace, Labels, Annotations, etc.;

(4) Field definitions are moved to RabbitSpec according to our definition. Some resources, such as ConfigMap, do not have specs, but Data;

(5). Add a RabbitList resource, which is the type of resource required to generate a Lister.

Then add a file/api/stable.wbsnail.com/v1/register.go:

package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: "stable.wbsnail.com", Version: "v1"}

// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
	// SchemeBuilder initializes a scheme builder
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	// AddToScheme is a global function that registers this API group & version to a scheme
	AddToScheme = SchemeBuilder.AddToScheme
)

// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Rabbit{},
		&RabbitList{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}
Copy the code

This file defines the registration method of the type. The template is taken from the sample project sample-Controller, which is provided by Kubernetes. All of these methods are useful for generating clientsets, which basically map Golang types to Kubernetes resources, so that the interface gets data (JSON unmarshal) to the corresponding Golang type. I haven’t delve too far into the details, but if you are interested in reading the source code you can start with the Apimachinery repository and the Clientset code generated later.

generate-groups.sh

If you look at deepcopy-gen, you might think that client-gen, lister-Gen and Informer gen are the next steps. That’s true, but code-Generator has been kind enough to prepare a script for us, deepcopy-gen, Clientset, Lister (lister-gen) and Informer (Informer -gen), that is the generate-groups.sh script file under the code-generator repository.

/path/to/code-generator/generate-groups.sh \
  all \
  github.com/wbsnail/articles/archive/dive-into-kubernetes-informer/11-crd-informer/client \
  github.com/wbsnail/articles/archive/dive-into-kubernetes-informer/11-crd-informer/api \
  stable.wbsnail.com:v1 \
  --go-header-file ./boilerplate.go.txt
Copy the code

Parameters 1, 2, 3, and 4 correspond to the generator, output package, input package, and resource group version, respectively. When a generator types all, four generators are called: deepcopy-Gen, client-gen, lister-Gen, and Informer -gen.

After successful execution we generate the Informer code for custom resource Rabbit (including Clientset and Lister).

I’m not going to list all the files that I generated, Articles /archive/dive-into-kubernetes-informer/11-crd-informer at main · wbsnail/articles · GitHub I won’t go into detail about the generated files because, like Deepcopy-Gen, they are generated in the same way as the Kubernetes built-in resource types, so the structure, usage, and even annotations are very much the same as the client-Go repository code.

For example, we want to use it:

package main

import (
	"fmt"
	"github.com/spongeprojects/magicconch"
	v1 "github.com/wbsnail/articles/archive/dive-into-kubernetes-informer/11-crd-informer/api/stable.wbsnail.com/v1"
	"github.com/wbsnail/articles/archive/dive-into-kubernetes-informer/11-crd-informer/client/clientset/versioned"
	"github.com/wbsnail/articles/archive/dive-into-kubernetes-informer/11-crd-informer/client/informers/externalversions"
	"k8s.io/client-go/tools/cache"
	"k8s.io/client-go/tools/clientcmd"
	"os"
)

func main(a) {
	kubeconfig := os.Getenv("KUBECONFIG")
	config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
	magicconch.Must(err)

	clientset, err := versioned.NewForConfig(config)
	magicconch.Must(err)

	informerFactory := externalversions.NewSharedInformerFactory(clientset, 0)
	rabbitInformer := informerFactory.Stable().V1().Rabbits().Informer()
	rabbitInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc: func(obj interface{}) {
			rabbit, ok := obj.(*v1.Rabbit)
			if! ok {return
			}
			fmt.Printf("A rabbit is created: %s\n", rabbit.Name)
		},
		UpdateFunc: func(oldObj, newObj interface{}) {
			newRabbit, ok := oldObj.(*v1.Rabbit)
			if! ok {return
			}
			fmt.Printf("A rabbit is updated: %s\n", newRabbit.Name)
		},
		DeleteFunc: func(obj interface{}) {
			rabbit, ok := obj.(*v1.Rabbit)
			if! ok {return
			}
			fmt.Printf("A rabbit is deleted: %s\n", rabbit.Name)
		},
	})

	stopCh := make(chan struct{})
	defer close(stopCh)

	fmt.Println("Start syncing....")

	go informerFactory.Start(stopCh)

	<-stopCh
}
Copy the code

Is it exactly the same as the Informers package introduced in the previous section?

The output of the above code looks like this:

Start syncing....
A rabbit is created: judy
A rabbit is created: bugs
A rabbit is updated: judy
A rabbit is updated: bugs
Copy the code

The controller for user-defined resources is created 🎉 🎉

conclusion

In the previous section, I introduced you to the implementation of custom resource Informer, as well as some of the other related content provided by code-Generator.

Next, in the next installment, I’ll introduce you to an advanced use of Informer: dynamic Informer.

The next section: the use of the dynamic package source code parsing and DynamicSharedInformerFactory.

Source code analysis and depth use [3/4]: implement custom resource (CRD) Informer

The resources

Bitnami Engineering: A deep dive into Kubernetes controllers

Bitnami Engineering: Kubewatch, an example of Kubernetes custom controller

Dynamic Kubernetes Informers | FireHydrant

Go at master · Kubernetes /client-go · GitHub

GitHub – kubernetes/sample-controller: Repository for sample controller. Complements sample-apiserver

Kubernetes Deep Dive: Code Generation for CustomResources

How to generate client codes for Kubernetes Custom Resource Definitions (CRD) | by Roger Liang | ITNEXT