Understand the Kubernetes APIServer principle

preface

The whole Kubernetes technology system is composed of declarative API and Controller, and Kube-Apiserver is the declarative API server of Kubernetes, and provides a bridge for the interaction of other components. Therefore, it is very important to deepen the understanding of Kube-Apiserver.

Overall component function

Kube-apiserver, as the only entry for Kubernetes cluster to operate ETCD, is responsible for Kubernetes resources authentication & authentication, verification and CRUD operations, and provides RESTful APIs for other components to call:

Kube-apiserver contains three types of Apiserver:

aggregatorServer: Responsible for handlingapiregistration.k8s.ioGroup APIService resource requests, and aggregated requests from users to the Aggregated Server (AA)
KubeAPIServer: Responsible for some general handling of requests, including authentication, authentication, and REST services for various built-in resources (POD, Deployment, Service and etc.)
ApiExtensionsServer: Register the CustomResourceDefinition (CRD) apiResources and apiVersions, handle CRD and CustomResource (CR) REST requests (404 will be returned if CR cannot be processed), And the last part of Apiserver Delegation

In addition, bootstrap-Controller is responsible for creating and managing Kubernetes default Apiserver service.

The following provides an overview of the above components.

bootstrap-controller

Apiserver bootstrap – controller to create & operation logic in k8s. IO/kubernetes/PKG/master directory
Bootstrap-controller is used to create and maintain the internal Kubernetes default Apiserver service
Kubernetes default apiserver service spec.selector is empty. This is the biggest difference between default Apiserver Service and other normal services. The endpoints corresponding to this particular service are not controlled by the EndPoints controller. They are directly governed by kube-Apiserver Bootstrap-Controller (maintained by this code, not by the pod selector)
bootstrap-controllerThe following are the main features:
- Create the default, kube-system, kube-public, and kube-node-lease namespace
- Create and maintain kubernetes Default Apiserver Service and its endpoint
- Provides Service ClusterIP-based inspection and repair functions (--service-cluster-ip-rangeDesignated area)
- Provides Service NodePort-based check and repair (--service-node-port-rangeDesignated area)

// k8s.io/kubernetes/pkg/master/controller.go:142
// Start begins the core controller loops that must exist for bootstrapping
// a cluster.
func (c *Controller) Start(a) {
    ifc.runner ! =nil {
        return
    }
    // Reconcile during first run removing itself until server is ready.
    endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts)
    iferr := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err ! =nil {
        klog.Errorf("Unable to remove old endpoints from kubernetes service: %v", err)
    }
    repairClusterIPs := servicecontroller.NewRepair(c.ServiceClusterIPInterval, c.ServiceClient, c.EventClient, &c.ServiceClusterIPRange, c.ServiceClusterIPRegistry, &c.SecondaryServiceClusterIPRange, c.SecondaryServiceClusterIPRegistry)
    repairNodePorts := portallocatorcontroller.NewRepair(c.ServiceNodePortInterval, c.ServiceClient, c.EventClient, c.ServiceNodePortRange, c.ServiceNodePortRegistry)
    // run all of the controllers once prior to returning from Start.
    iferr := repairClusterIPs.RunOnce(); err ! =nil {
        // If we fail to repair cluster IPs apiserver is useless. We should restart and retry.
        klog.Fatalf("Unable to perform initial IP allocation check: %v", err)
    }
    iferr := repairNodePorts.RunOnce(); err ! =nil {
        // If we fail to repair node ports apiserver is useless. We should restart and retry.
        klog.Fatalf("Unable to perform initial service nodePort check: %v", err)
    }
    // Periodically executes the four main functions of the Bootstrap Controller (Trends)
    c.runner = async.NewRunner(c.RunKubernetesNamespaces, c.RunKubernetesService, repairClusterIPs.RunUntil, repairNodePorts.RunUntil)
    c.runner.Start()
}
Copy the code

For more details on how the code works, see Kubernetes-Reading-Notes.

kubeAPIServer

KubeAPIServer mainly provides internal API Resources operation request, registered routing information for each API Resources in Kubernetes, and exposed RESTful API. Enable services in and outside the cluster to use RESTful apis to operate resources in Kubernetes

In addition, kubeAPIServer is the core of Kubernetes Apiserver The aggregatorServer and apiExtensionsServer described below are built on top of kubeAPIServer (to complement Kubernetes’ ability to support user-defined resources).

The core function of kubeAPIServer is to add routes to Kubernetes built-in resources, as follows:

callm.InstallLegacyAPIAdd the core API Resources to the route, in apiserver/apiResource at the beginning;
callm.InstallAPIsAdd the extended API Resources to the route, in apiserver/apisResource at the beginning;

// k8s.io/kubernetes/pkg/master/master.go:332
// New returns a new instance of Master from the given config.
// Certain config fields will be set to a default value if unset.
// Certain config fields must be specified, including:
// KubeletClientConfig
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error){...// Install LegacyAPI(core API)
    // install legacy rest storage
    if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
        legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
            StorageFactory:              c.ExtraConfig.StorageFactory,
            ProxyTransport:              c.ExtraConfig.ProxyTransport,
            KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
            EventTTL:                    c.ExtraConfig.EventTTL,
            ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
            SecondaryServiceIPRange:     c.ExtraConfig.SecondaryServiceIPRange,
            ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
            LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
            ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
            ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
            APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
        }
        iferr := m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider); err ! =nil {
            return nil, err
        }
    }
    ...
    // Install APIs(named Groups APIs)
    iferr := m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...) ; err ! =nil {
        return nil, err
    }
    ...
    return m, nil
}
Copy the code

KubeAPIServer provides three types of API Resource interfaces:

Core group: mainly in/api/v1Under;
Named groups: its path is/apis/$GROUP/$VERSION;
Some apis for system state: for example/metrics 、/versionAnd so on;

/apis/{group}/{version}/namespaces/{namespace}/resource/{name}

KubeAPIServer creates a RESTStorage for each API resource. The purpose of the RESTStorage is to match the access path of each resource with the operations of the back-end Storage. The RESTStorage interface is constructed to determine which operations the resource can perform (for example: Each operation corresponds to a standard REST method. For example, create corresponds to THE REST Method POST, and Update corresponds to the REST Method PUT. Finally, it iterates through the actions array one by one, adds a handler for each operation (handler corresponds to the relevant interface of REST Storage implementation), registers the handler with route, and provides RESTful apis as follows:

// m.GenericAPIServer.InstallLegacyAPIGroup --> s.installAPIResources --> apiGroupVersion.InstallREST --> installer.Install --> a.registerResourceHandlers
// k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go:181
func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService) (*metav1.APIResource, error){...// Figure out what REST interfaces the Resource implements, and determine which Verbs it supports so you can add a route to them
    // what verbs are supported by the storage, used to know what verbs we support per path
    creater, isCreater := storage.(rest.Creater)
    namedCreater, isNamedCreater := storage.(rest.NamedCreater)
    lister, isLister := storage.(rest.Lister)
    getter, isGetter := storage.(rest.Getter)
    ...
    // add actions for resource (+ depending on whether namespace is supported)
    // Get the list of actions for the given scope.
    switch {
    case! namespaceScoped:// Handle non-namespace scoped resources like nodes.
        resourcePath := resource
        resourceParams := params
        itemPath := resourcePath + "/{name}"
        nameParams := append(params, nameParam)
        proxyParams := append(nameParams, pathParam)
        ...
        // Handler for standard REST verbs (GET, PUT, POST and DELETE).
        // Add actions at the resource path: /api/apiVersion/resource
        actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
        actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater) ... }...// 3. Mapping from rest.Storage to rest. Route
    // Add a handler for each operation
    for _, action := range actions {
        ...
        switch action.Verb {
        ...
        case "POST": // Create a resource.
            var handler restful.RouteFunction
            // 4. Initialize handler
            if isNamedCreater {
                handler = restfulCreateNamedResource(namedCreater, reqScope, admit)
            } else {
                handler = restfulCreateResource(creater, reqScope, admit)
            }
            handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
            ...
            // 5. Bind route to handler
            route := ws.POST(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty"."If 'true', then the output is pretty printed.")).
                Operation("create"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...) ...). . Returns(http.StatusOK,"OK", producedObject).
                // TODO: in some cases, the API may return a v1.Status instead of the versioned object
                // but currently go-restful can't handle multiple different objects being returned.
                Returns(http.StatusCreated, "Created", producedObject).
                Returns(http.StatusAccepted, "Accepted", producedObject).
                Reads(defaultVersionedObject).
                Writes(producedObject)
            iferr := AddObjectParams(ws, route, versionedCreateOptions); err ! =nil {
                return nil, err
            }
            addParams(route, action.Params)
            // 6. Add it to the route
            routes = append(routes, route)
        case "DELETE": // Delete a resource..default:
            return nil, fmt.Errorf("unrecognized action verb: %s", action.Verb)
        }
        for _, route := range routes {
            route.Metadata(ROUTE_META_GVK, metav1.GroupVersionKind{
                Group:   reqScope.Kind.Group,
                Version: reqScope.Kind.Version,
                Kind:    reqScope.Kind.Kind,
            })
            route.Metadata(ROUTE_META_ACTION, strings.ToLower(action.Verb))
            ws.Route(route)
        }
        // Note: update GetAuthorizerAttributes() when adding a custom handler.}... }Copy the code

KubeAPIServer code structure is as follows:

1. Total apiserver logic k8s start. IO/kubernetes/CMD/kube - 2 apiserver apiserver bootstrap - create & operation logic controller K8s. IO/kubernetes/PKG/master 3. API Resource corresponding to the back-end RESTStorage (on Genericregistry. Store) create k8s. IO/kubernetes/PKG/registry 4. Aggregated - apiserver create & processing logic K8s. IO/kubernetes/staging/SRC/k8s. IO/kube - aggregator 5. Extensions - apiserver create & processing logic K8s. IO/kubernetes/staging/SRC/k8s. IO/apiextensions - apiserver 6. Apiserver create & operation k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server 7. Register API Resource ResourceHandlers (InstallREST&Install®isterResourceHandlers) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints 8. Create the storage backend (etcdv3) k8s. IO/kubernetes/staging/SRC/k8s. IO/apiserver/PKG/storage. Genericregistry.Store.Com pleteWithOptions initialization k8s. IO/kubernetes/staging/SRC/k8s. IO/apiserver/PKG/registryCopy the code

The call chain is sorted as follows:

For more details on how the code works, see Kubernetes-Reading-Notes.

aggregatorServer

Aggregated APIServer(AA) aggregatorServer is primarily used to handle the second way to extend Kubernetes API Resources, Aggregated APIServer(AA), broker CR requests to AA:

Aggregated apiserver here, combining information about sample- Apiserver from Kubernetes, can be summarized as follows:

The aggregatorServer associates the APIServices object with a Service to forward requests. The associated Service type further determines the form of request forwarding. The aggregatorServer includes a GenericAPIServer and a Controller that maintains its own state. Where GenericAPIServer mainly processes APIService resource requests under apiregistration.k8s. IO group, while Controller includes:
- apiserviceRegistrationController: Builds a proxy based on the aggregated Server Service defined by APIService, and forwards CR requests to the aggregated Server on the backend
- availableConditionController: Maintain the availability of APIServices, including the availability of its reference Service.
- autoRegistrationController: a specific set of APIServices used to maintain an API presence;
- crdRegistrationController: Responsible for automatically registering CRD GroupVersions with APIServices;
- openAPIAggregationController: Synchronize changes to APIServices resources to the provided OpenAPI documentation;

ApiserviceRegistrationController responsible for according to the aggregated defined APIService server service build agent, will forward requests to the backend of CR aggregated server. ApiService has two types: Local(Service is null) and Service(Service is not null). ApiserviceRegistrationController apiService setting agent is responsible for the two types: the Local type direct route to kube – apiserver processing; Service types, on the other hand, set up proxies and aggregated requests about services (proxyPath := “/apis/” + apiservice.spec.group + “/” + ApiService. Spec. Version), Kube-apiserver (if service is kubernetes default apiserver service:443)=> Service ClusterIP:Port (default) or Access by randomly selecting Service Endpoint Backend:

func (s *APIAggregator) AddAPIService(apiService *v1.APIService) error{... proxyPath :="/apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version
    // v1. is a special case for the legacy API. It proxies to a wider set of endpoints.
    if apiService.Name == legacyAPIServiceName {
        proxyPath = "/api"
    }
    // register the proxy handler
    proxyHandler := &proxyHandler{
        localDelegate:   s.delegateHandler,
        proxyClientCert: s.proxyClientCert,
        proxyClientKey:  s.proxyClientKey,
        proxyTransport:  s.proxyTransport,
        serviceResolver: s.serviceResolver,
        egressSelector:  s.egressSelector,
    }
  ...
    s.proxyHandlers[apiService.Name] = proxyHandler
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(proxyPath, proxyHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandlePrefix(proxyPath+"/", proxyHandler)
  ...
    // it's time to register the group aggregation endpoint
    groupPath := "/apis/" + apiService.Spec.Group
    groupDiscoveryHandler := &apiGroupHandler{
        codecs:    aggregatorscheme.Codecs,
        groupName: apiService.Spec.Group,
        lister:    s.lister,
        delegate:  s.delegateHandler,
    }
    // aggregation is protected
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(groupPath, groupDiscoveryHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandle(groupPath+"/", groupDiscoveryHandler)
    s.handledGroups.Insert(apiService.Spec.Group)
    return nil
}
// k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:109
func (r *proxyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    // Load roxyHandlingInfo to process the request
    value := r.handlingInfo.Load()
    if value == nil {
        r.localDelegate.ServeHTTP(w, req)
        return
    }
    handlingInfo := value.(proxyHandlingInfo)
  ...
    // Check whether the APIService service is normal
    if! handlingInfo.serviceAvailable { proxyError(w, req,"service unavailable", http.StatusServiceUnavailable)
        return
    }
    // Convert the original request to a request for APIService
    // write a new location based on the existing request pointed at the target service
    location := &url.URL{}
    location.Scheme = "https"
    rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName, handlingInfo.servicePort)
    iferr ! =nil {
        klog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceNamespace, handlingInfo.serviceName, err)
        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)
        return
    }
    location.Host = rloc.Host
    location.Path = req.URL.Path
    location.RawQuery = req.URL.Query().Encode()
    newReq, cancelFn := newRequestForProxy(location, req)
    defer cancelFn()
   ...
    proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper)
    handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w})
    handler.ServeHTTP(w, newReq)
}
Copy the code

$ kubectl get APIService           
NAME                                   SERVICE                      AVAILABLE   AGE
...
v1.apps                                Local                        True        50d
...
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        50d
...
Copy the code

# default APIServices
$ kubectl get -o yaml APIService/v1.apps
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    kube-aggregator.kubernetes.io/automanaged: onstart
  name: v1.apps
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.apps
spec:
  group: apps
  groupPriorityMinimum: 17800
  version: v1
  versionPriority: 15
status:
  conditions:
  - lastTransitionTime: "2020-10-20T10:39:48Z"
    message: Local APIServices are always available
    reason: Local
    status: "True"
    type: Available

# aggregated server    
$ kubectl get -o yaml APIService/v1beta1.metrics.k8s.io
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: v1beta1.metrics.k8s.io
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-12-05T00:50:48Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

# CRD
$ kubectl get -o yaml APIService/v1.duyanghao.example.com
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    kube-aggregator.kubernetes.io/automanaged: "true"
  name: v1.duyanghao.example.com
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.duyanghao.example.com
spec:
  group: duyanghao.example.com
  groupPriorityMinimum: 1000
  version: v1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-12-11T08:45:37Z"
    message: Local APIServices are always available
    reason: Local
    status: "True"
    type: Available
Copy the code

AggregatorServer will create a default APIService list based on all API resources defined by Kube-Apiserver. The name is $VERSION.$GROUP. These APIService will have labels kube – aggregator. Kubernetes. IO/automanaged: onstart, for example: v1. Apps APIService. AutoRegistrationController APIService in creation and maintenance of the list, which we see the Local APIService; Information about customized APIService(aggregated Server) is not processed
Aggregated Server implements CR(custom API resources) CRUD apis, and can optionally optionally store information about back-end storage, including shared ETCD with Core Kube-Apiserver, and aggregated ETCD databases and other databases. Aggregated server implementation of CR API path: / apis/GROUP/GROUP/GROUP/VERSION, specific to the sample apiserver as follows: / apis/wardle.example.com/v1alpha1, the following resource types are: flunders and fischers
Aggregated Server You can deploy APIService resources, and then add information about service fields to corresponding aggregated Server service to integrate and interact with Core Kube-Apiserver

The sample-Apiserver directory structure is as follows, and information Server can be aggregated:

The staging/SRC/k8s. IO/sample - apiserver ├ ─ ─ artifacts │ ├ ─ ─ example │ │ ├ ─ ─ apiservice. Yaml... ├ ─ ─ hack ├ ─ ─ main. Go └ ─ ─ PKG ├ ─ ─ admission ├ ─ ─ apis ├ ─ ─ apiserver ├ ─ ─ CMD ├ ─ ─ generated │ ├ ─ ─ clientset │ │ └ ─ ─ versioned ... │ │ └ ─ ─ typed │ │ └ ─ ─ wardle │ │ ├ ─ ─ v1alpha1 │ │ └ ─ ─ v1beta1 │ ├ ─ ─ informers │ │ └ ─ ─ externalversions │ │ └ ─ ─ wardle │ │ ├ ─ ─ v1alpha1 │ │ └ ─ ─ v1beta1 │ ├ ─ ─ listers │ │ └ ─ ─ wardle │ │ ├ ─ ─ v1alpha1 │ │ └ ─ ─ v1beta1 └ ─ ─ registryCopy the code

Artifacts is used to deploy the YAML sample
Hack directory for automatic scripts (eg: update-codeGen)
Main. Go is the aggregated Server startup entry; PKG/CMD Starts aggregated Server logic. PKG /apiserver Used for aggregated Server initialization and route registration
PKG /apis are responsible for the structure definition of related CR, automatically generated (update-codeGen)
PKG/Admission is responsible for the relevant codes for admission
PKG /generated is responsible for generating clientsets, Informers, and listers that access CR
The PKG/Registry directory is responsible for cr-related RESTStorage implementations

For more details on how the code works, see Kubernetes-Reading-Notes.

apiExtensionsServer

ApiExtensionsServer is responsible for CustomResourceDefinition (CRD) apiResources and apiVersions registration. Handling both CRD and the corresponding CustomResource (CR) REST request (returning 404 if the corresponding CR cannot be processed) is the last part of apiserver Delegation

The principle is summarized as follows:

Custom Resource, CR for short, is a Kubernetes Custom Resource type, and it corresponds to Kubernetes built-in Resource types, such as Pod, Service, etc.. With CR we can define any resource type we want
CRD registers CR with Kubernetes in the form of YAML files to implement the custom API-resources, which is the second way to extend Kubernetes API resources, and is also the most commonly used one
APIExtensionServer is responsible for CustomResourceDefinition (CRD) apiResources and apiVersions registration. Handling both CRD and the corresponding CustomResource (CR) REST request (returning 404 if the corresponding CR cannot be processed) is the last part of apiserver Delegation
CrdRegistrationController responsible for CRD GroupVersions automatically registered to the APIServices. The specific logic is: Enumerate all CRDs, then according to the CRD definition of CRD. Spec. The Group as well as the CRD. Spec. Versions field building APIService, and add to the autoRegisterController. ApiServicesToSync, It is created and maintained by the autoRegisterController. Is that why CRD creates APIService objects

The Controllers and functions contained in the APIExtensionServer are as follows:

OpenapiController: Synchronizes CRD resource changes to OpenAPI documents. You can visit/OpenAPI /v2 to view CRD resource changes.
CrdController: Registers CRD information with apiVersions and apiResources, which can be viewed via kubectl api-versions and kubectl api-Resources.

Kubectl API – versions command returns all Kubernetes cluster resources version information (actual sent two requests that are https://127.0.0.1:6443/api and https://127.0.0.1:6443/apis, The results of the two requests are merged at the end.

$kubectl -v=8 API -versions I1211 11:44:50.276446 22493 loader.go:375] Config loaded from file / root/kube/config I1211 11:44:50. 277005 22493 round_trippers. Go: 420] GET https://127.0.0.1:6443/api? timeout=32s ... I1211 11:44:50.290265 22493 Request. Go :1068] Response Body: {" kind ":" APIVersions ", "versions:" [] "v1", "serverAddressByClientCIDRs" : [{" clientCIDR ":" 0.0.0.0/0 ", "serverAddress" : "X.X.X.X : 6443 "}}] I1211 11:44:50. 293673 22493 round_trippers. Go: 420] GET https://127.0.0.1:6443/apis? timeout=32s ... I1211 11:44:50.298360 22493 Request. Go :1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregi stration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVer sion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"ext ensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"n ame":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version ":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVer sion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupV ersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}] ,"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars] apiextensions.k8s.io/v1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 authentication.k8s.io/v1beta1 ... storage.k8s.io/v1 storage.k8s.io/v1beta1 v1Copy the code

The kubectl api-resources command is used to fetch all the API versions and then call the interface for each VERSION of the API to fetch all the API resource types under that version

$ kubectl -v=8 api-resources 5077 loader.go:375] Config loaded from file: / root/kube/config I1211 15:19:47. 593450 15077 round_trippers. Go: 420] GET https://127.0.0.1:6443/api? Timeout =32s I1211 15:19:47.602273 15077 Request. Go :1068] Response: {" kind ":" APIVersions ", "versions:" [] "v1", "serverAddressByClientCIDRs" : [{" clientCIDR ":" 0.0.0.0/0 ", "serverAddress" : "X.X.X.X : 6443 "}}] I1211 15:19:47. 606279 15077 round_trippers. Go: 420] GET https://127.0.0.1:6443/apis? Timeout =32s I1211 15:19:47.610333 15077 Request. Go :1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregi stration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVer sion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"ext ensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"n ame":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version ":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVer sion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupV ersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}] ,"preferredVersion":{"groupVersion":"authentication.k8s. IO /v1"," [truncated 4985 chars] I1211 15:19:47.614700 15077 Round_trippers. Go: 420] GET https://127.0.0.1:6443/apis/batch/v1? Timeout = 32 s I1211 15:19:47. 614804 15077 round_trippers. Go: 420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1? Timeout = 32 s I1211 15:19:47. 615687 15077 round_trippers. Go: 420] GET https://127.0.0.1:6443/apis/auth.tkestack.io/v1? Timeout = 32 s https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1? Timeout = 32 s I1211 15:19:47. 616794 15077 round_trippers. Go: 420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1? Timeout = 32 s I1211 15:19:47. 616863 15077 round_trippers. Go: 420] GET https://127.0.0.1:6443/apis/apps/v1? timeout=32s ... NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding endpoints ep true Endpoints events ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes no false Node ...Copy the code

- 'namingController' : check for naming conflicts in CRD obj in CRD '.status.conditions'; - 'establishingController' : check whether CRD is in normal state, which can be seen in CRD '.status.conditions'; - ` nonStructuralSchemaController ` : check the CRD obj structure is normal, but in the CRD. ` status. The conditions ` view; - 'apiApprovalController' : checks whether THE CRD follows the Kubernetes API declaration policy, which can be seen in CRD '.status.conditions'; - 'finalizingController' : functions that resemble finalizing, and that pertains to the deletion of CRs;Copy the code

The processing logic of CR CRUD APIServer is summarized as follows:

CreateAPIExtensionsServer = > NewCustomResourceDefinitionHandler = > crdHandler = > registered CR CRUD API interface:

// New returns a new instance of CustomResourceDefinitions from the given config.
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*CustomResourceDefinitions, error){... crdHandler, err := NewCustomResourceDefinitionHandler( versionDiscoveryHandler, groupDiscoveryHandler, s.Informers.Apiextensions().V1().CustomResourceDefinitions(), delegateHandler, c.ExtraConfig.CRDRESTOptionsGetter, c.GenericConfig.AdmissionControl, establishingController, c.ExtraConfig.ServiceResolver, c.ExtraConfig.AuthResolverWrapper, c.ExtraConfig.MasterCount, s.GenericAPIServer.Authorizer, c.GenericConfig.RequestTimeout, time.Duration(c.GenericConfig.MinRequestTimeout)*time.Second, apiGroupInfo.StaticOpenAPISpec, c.GenericConfig.MaxRequestBodyBytes, )iferr ! =nil {
        return nil, err
    }
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle("/apis", crdHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.HandlePrefix("/apis/", crdHandler)
    ...
    return s, nil
}

Copy the code

The crdHandler processing logic is as follows:

Parsing the req (GET/apis/duyanghao.example.com/v1/namespaces/default/students), according to the request path group (duyanghao.example.com), the version (v1), Get(CRD, err := r.crlister.get (crdName))
Through the CRD. The UID and CRD. Get crdInfo Name, if there is no criterion to create the corresponding crdInfo (r. gutierrez etOrCreateServingInfoFor crdInfo, err: = (CRD) UID, CRD. Name)). CrdInfo contains the CRD definition and the customResource.REST Storage for the Custom Resource
Customresource.REST storage is created by CR Group(duyanghao.example.com), Version(v1), Kind(Student), Resource(Students), etc. Because CR doesn’t have a specific structure defined in Kubernetes code, it initializes a generic Unstructured structure that holds Custom resources of all types. And perform SetGroupVersionKind on the structure (set the specific Custom Resource Type)

The Unstructured structure is retrieved from CustomResource.REST Storage, transformed and returned

// k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:223
func (r *crdHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
  ctx := req.Context()
  requestInfo, ok := apirequest.RequestInfoFrom(ctx)
  ...
  crdName := requestInfo.Resource + "." + requestInfo.APIGroup
  crd, err := r.crdLister.Get(crdName)
  ...
  crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name)
  verb := strings.ToUpper(requestInfo.Verb)
  resource := requestInfo.Resource
  subresource := requestInfo.Subresource
  scope := metrics.CleanScope(requestInfo)
  ...
  switch {
  case subresource == "status"&& subresources ! =nil&& subresources.Status ! =nil:
      handlerFunc = r.serveStatus(w, req, requestInfo, crdInfo, terminating, supportedTypes)
  case subresource == "scale"&& subresources ! =nil&& subresources.Scale ! =nil:
      handlerFunc = r.serveScale(w, req, requestInfo, crdInfo, terminating, supportedTypes)
  case len(subresource) == 0:
      handlerFunc = r.serveResource(w, req, requestInfo, crdInfo, terminating, supportedTypes)
  default:
      responsewriters.ErrorNegotiated(
          apierrors.NewNotFound(schema.GroupResource{Group: requestInfo.APIGroup, Resource: requestInfo.Resource}, requestInfo.Name),
          Codecs, schema.GroupVersion{Group: requestInfo.APIGroup, Version: requestInfo.APIVersion}, w, req,
      )
  }
  ifhandlerFunc ! =nil {
      handlerFunc = metrics.InstrumentHandlerFunc(verb, requestInfo.APIGroup, requestInfo.APIVersion, resource, subresource, scope, metrics.APIServerComponent, handlerFunc)
      handler := genericfilters.WithWaitGroup(handlerFunc, longRunningFilter, crdInfo.waitGroup)
      handler.ServeHTTP(w, req)
      return}}Copy the code

For more details on how the code works, see Kubernetes-Reading-Notes.

Conclusion

In this paper, the source code level of Kubernetes apiserver is summarized, including: aggregatorServer, kubeAPIServer, apiExtensionsServer and bootstrap-controller. Through reading this article, we can have a general understanding of the internal principle of Apiserver, which is also helpful for further research

Refs

kubernetes-reading-notes

Understand the Kubernetes APIServer principle

preface

Overall component function

bootstrap-controller

kubeAPIServer

aggregatorServer

apiExtensionsServer

Conclusion

Refs

Related Posts

There’s no simple thread pool

Data Link Layer (3)

Invalid authentication data is displayed when IDEA is connected to GitHub. error