This paper mainly introduces the kubelet function, core components, as well as the source analysis of the startup process, summed up the working principle of Kubelet.

Kubelet profile

Kubelet is easy to find in the official architecture diagram

Kubelet -h kubelet -h

  • Kubelet is the primary “Node agent” that runs on every Node. Use one of the following to register a Node with apiserver: hosthostname; coverhostThe parameters; Or the logic specified by the cloud provider.
  • Kubelet based onPodSpecWork.PodSpecIs to useYAMLorJSONObject to describe a Pod. Kubelet accepts a set provided through various mechanisms, mainly ApiserverPodSpecAnd make sure the container described inside is in good working order.

In addition to being provided by Apiserver, PodSpecs can also be provided by:

  • file
  • The HTTP endpoint
  • The HTTP server

Kubelet functions are summarized as reporting Node information and managing (creating and destroying) pods. The function looks simple, but it’s not. It takes a long time to explain each point. For example, the computing resources of Node Node are not only traditional CPU, memory and hard disk, but also extended to support resources such as GPU. Pods are not only containers, but also related networks, security policies, and so on.

Kubelet architecture

Important component

The architecture of Kubelet consists of more than N components. Here are some important ones:

PLEG

A Pod Lifecycle Event Generator, literally a Pod Lifecycle Event Generator (ContainerStarted, ContainerDied, ContainerRemoved, ContainerChanged).

It maintains the Pod cache; Periodically obtain Pod information from ContainerRuntime and compare it with the information in the cache to generate the preceding event. Writes events to the channels it maintains.

PodWorkers

Handles Pod synchronization in events. The core method managePodLoop() indirectly calls kubelet.syncPod() to synchronize pods:

  • If a Pod is being created, record its delay
  • Generate THE Pod API Status, that isv1.PodStatus: Converts runtime status to API status
  • Pod frompending 到 runningThe time-consuming
  • inStatusManagerUpdate the pod status in
  • Kill a Pod that should not be running
  • If the network plug-in is not ready, only the Pod that uses the host Network is started
  • If static Pod does not exist, create a Mirror pod for it
  • Create file system directories for Pod: Pod directory, volume directory, plug-in directory
  • useVolumeManagerMount a volume to Pod
  • Get Image pull Secrets
  • That calls the Container Runtime#SyncPod()methods

PodManager

Stores the expected state of pods from different channels of Kubelet service

StatsProvider

Provides statistics for nodes and containers, both cAdvisor and CRI implementations.

ContainerRuntime

As the name implies, the container runs. Interacts with high-level container runtimes that follow the CRI specification.

Deps.PodConfig

PodConfig is a configuration multiplexer that merges many Pod configuration sources into a single consistent structure and then passes incremental change notifications to listeners in sequence.

Configuration sources include file, apiserver, and HTTP

#syncLoop

Receive Pod change notifications from PodConfig, scheduled tasks, events from PLEG, and events from ProbeManager to synchronize pods to the desired state.

PodAdmitHandlers

Pod admission a series of processors called during the Pod admission process, like the eviction handler, which won’t expel pods with QoS set to BestEffort when the node memory is stressed, the shutdown admit handler, which won’t exclude pods with QoS set to BestEffort when the node is shutdown, Does not handle pod synchronization), etc.

OOMWatcher

Obtain the OOM log of the container from the system log, encapsulate it as an event, and record it.

VolumeManger

The VolumeManager runs a set of asynchronous loops that determine which volumes need to be attached/mounted/unmounted/detached and perform operations based on the PODS scheduled on this node.

CertificateManager

Handle certificate rotation.

ProbeManager

There are actually three types of probes that provide Probe result caches and channels.

  • LivenessManager
  • ReadinessManager
  • StartupManager

EvictionManager

Monitor resource usage of Node nodes and expel PODS based on expulsion rules to release resources and relieve pressure on nodes.

PluginManager

The PluginManager runs a set of asynchronous loops using this node to determine which plug-ins need to be registered/unregistered and executed. Examples include CSI drivers and Device plugins.

CSI

Container Storage Interface: a Storage driver implemented by a Storage vendor.

Device Manager Plugin

Kubernetes provides a device plug-in framework that you can use to publish system hardware resources to Kubelet.

Vendors can implement device plug-ins that you deploy manually or as DaemonSet without having to customize Kubernetes’ own code. Target devices include Gpus, high-performance nics, FPgas, InfiniBand adapters, and other similar computing resources that may require vendor-specific initialization and setup.

Kubelet startup process

To analyze the startup process of Kubelet, start with how Kubelet runs. Find a Node and it’s easy to find kubelet’s process. Since it starts in systemd mode, you can also view its status through systemctl.

Kubelet start command

Kubelet startup command (minikube environment)

$ ps -aux | grep '/kubelet' | grep -v grepRoot 4917 2.6 0.3 1857652 106152? Ssl 01:34 13:05 / var/lib/minikube/binaries/v1.21.0 / kubelet - the bootstrap - kubeconfig = / etc/kubernetes/bootstrap - kubelet. Conf - config = / var/lib/kubelet/config yaml - container - the runtime = docker - the hostname - override = 1.21.0 -- kubeconfig = / etc/kubernetes/kubelet. Conf - node - IP = 192.168.64.5Copy the code

or

$ systemctl status kubelet.serviceLow kubelet. Service - kubelet: The Kubernetes Node Agent The Loaded: The Loaded (/ usr/lib/systemd/system/kubelet. Service; disabled; Vendor preset: enabled) Drop - In: / etc/systemd/system/kubelet. Service. D └ ─ 10 - kubeadm. Conf Active: active (running) since Sun 2021-06-13 01:34:42 UTC; 11 h line Docs: http://kubernetes.io/docs/ Main PID: 4917 (kubelet) Tasks: 15 (38314) limit: the Memory: 39.4 M CGroup: / system. Slice/kubelet service └ ─ 4917 / var/lib/minikube/binaries/v1.21.0 / kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml - the container - the runtime = docker - the hostname - override = 1.21.0 - kubeconfig = / etc/kubernetes/kubelet conf - node - IP = 192.168.64Copy the code

Source code analysis

From [email protected]: kubernetes/kubernetes git repository for code, using the latest release 1.21 branch.

  • cmd/kubelet/kubelet.go:35 的 mainThe method is program entry.
    • callNewKubeletCommandMethod to create a command
    • Execute the command
      • cmd/kubelet/app/server.go:434 的 RunMethods.

        • Call the RunKubelet method.

          • callcreateAndInitKubeletMethod to create and initialize kubelet
            • pkg/kubelet/kubelet.go 的 NewMainKubeletMethod to create various components of Kubelet. A total of a dozen components, seeKubelet architecture.
            • callBirtyCryMethod: releaseStartingThe event
            • callStartGarbageCollectionMethod, openContainerGC 和 ImageGC
          • callstartKubeletMethods (heavy use of Goroutine and channels)
            • Goroutine:kubelet.Run()
              • Initialize the module
                • The metrics associated
                • Example Create a file system directory directory
                • Create a container log directory
                • Start theImageGCManager
                • Start theServerCertificateManager
                • Start theOOMWatcher
                • Start theResourceAnalyzer
              • Goroutine:VolumeManager.Run()Start handling Pod Volume unmounting and mounting
              • Goroutine: Status updatefastStatusUpdateOnce()Update Pod CIDR -> UpdateContainerRuntimeStatus -> Update Node Node status)
              • Goroutine:NodeLeaseController.Run()Renewing the Node Lease
              • Goroutine:podKiller.PerformPodKillingWorkKill pods that are not handled properly
              • StatusManager.Start()Start updating the Pod status to Apiserver
              • RuntimeClassManager.Start()
              • PLEG.Start(): continuous fromContainerRuntimeGet Pod/ container status and compare with kubelet local cache to generate the correspondingEvent
              • syncLoop()The key,Continuously monitor and process changes from files, Apiserver, HTTP. Including Pod addition, update, graceful delete, ungraceful delete, harmonize.

        • Start the server and expose the/HealthZ endpoint

        • Notifies Systemd kuberlet that the service is started

How Kubelet works

  1. Pod configuration changes to static files, Apiserver, and HTTP requests are sent tokubelet.syncLoop
  2. PLEG will periodically obtain the status of Pod on the node through the container runtime, compare the Pod information in its cache, encapsulate it into events, and enter the channel of PLEG
  3. Check the Pod in the work queue periodically
  4. The Pod in the ProbeManager channel
  5. All of the above 1 to 4 will entersyncLoopIterationAnd obtain the corresponding Pod from the corresponding channel and save the Pod information toPodManager; And then distribute it toPodWorker.Complete some column synchronization.

conclusion

So much for kubelet startup traffic, although complex, there are traces to follow. Once you understand kubelet’s position and role in Kubernetes, it’s easy to understand its work flow.

More on the Pod creation and startup process later.

read

  • Kubernetes source code analysis – Informer
  • How does HPA level auto Scaling work

The article is uniformly published in the public number cloud native refers to north