Overview

This article based on k8s the release 1.17 branch code, the code is located in the PKG/controller/serviceaccount directory, code: tokens_controller. Go.

ServiceAccount AdmissionController ServiceAccount AdmissionController Knowing that a ServiceAccount object references a secret object of type=”kubernetes. IO /service-account-token”, Ca.crt, Namespace, and token data in this Secret object are mounted to each container in pod for authentication and authorization when calling apI-Server.

When a ServiceAccount object is created, the secret object referenced with type=”kubernetes. IO /service-account-token” is automatically created. Such as:

kubectl create sa test-sa1 -o yaml
kubectl get sa test-sa1 -o yaml
kubectl get secret test-sa1-token-jg6lm -o yaml
Copy the code

The question is, how does it work?

The source code parsing

TokensController instantiation

This is actually implemented by the TokenController of Kube-Controller-Manager, Kube-controller-manager kube-controller-manager kube-controller-manager kube-controller-manager kube-controller-manager kube-controller-manager kube-controller-manager –root-ca-file is ca. CRT data in the figure above, and –service-account-private-key-file is used to sign JWT token data, that is, the token field value in the figure above.

When kube-Controller-Manager starts, it instantiates the TokensController first and passes the parameters required for the instantiation. Among them, read from the boot parameters ca root certificate and private key file content, and use serviceaccount. JWTTokenGenerator () function to generate JWT token, code in L546 – L592:

func (c serviceAccountTokenControllerStarter) startServiceAccountTokenController(ctx ControllerContext) (http.Handler, bool, error) {
	// ...
	// Read --service-account-private-key-file private key file
	privateKey, err := keyutil.PrivateKeyFromFile(ctx.ComponentConfig.SAController.ServiceAccountKeyFile)
	iferr ! =nil {
		return nil.true, fmt.Errorf("error reading key for service account token controller: %v", err)
	}

	// Read --root-ca-file as ca. If not, use kubeconfig as CA
	var rootCA []byte
	ifctx.ComponentConfig.SAController.RootCAFile ! ="" {
		ifrootCA, err = readCA(ctx.ComponentConfig.SAController.RootCAFile); err ! =nil {
			return nil.true, fmt.Errorf("error parsing root-ca-file at %s: %v", ctx.ComponentConfig.SAController.RootCAFile, err)
		}
	} else {
		rootCA = c.rootClientBuilder.ConfigOrDie("tokens-controller").CAData
	}

	// Use tokenGenerator to generate the JWT token and use --service-account-private-key-file private key to sign the JWT token
	tokenGenerator, err := serviceaccount.JWTTokenGenerator(serviceaccount.LegacyIssuer, privateKey)
	/ /...
	
	// instantiate TokensController
	controller, err := serviceaccountcontroller.NewTokensController(
		ctx.InformerFactory.Core().V1().ServiceAccounts(), // ServiceAccount informer
		ctx.InformerFactory.Core().V1().Secrets(), // Secret informer
		c.rootClientBuilder.ClientOrDie("tokens-controller"),
		serviceaccountcontroller.TokensControllerOptions{
			TokenGenerator: tokenGenerator,
			RootCA:         rootCA,
		},
	)
	// ...
	// Consume queue data
	go controller.Run(int(ctx.ComponentConfig.SAController.ConcurrentSATokenSyncs), ctx.Stop)

	// Enable ServiceAccount Informer and Secret Informer
	ctx.InformerFactory.Start(ctx.Stop)

	return nil.true.nil
}
Copy the code

[TokensController] [TokensController] [ServiceAccount] [kubernetes. IO/service-Account-Token] [TokensController] [Kubernetes. IO/service-Account-Token] [kubernetes. IO/service-Account-Token] [kubernetes.

func NewTokensController(serviceAccounts informers.ServiceAccountInformer, secrets informers.SecretInformer, cl clientset.Interface, options TokensControllerOptions) (*TokensController, error) {
    e := &TokensController{
        // ...
    	// Create queue for service and secret to store event data
        syncServiceAccountQueue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "serviceaccount_tokens_service"),
        syncSecretQueue:         workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "serviceaccount_tokens_secret"),}// ...
	e.serviceAccounts = serviceAccounts.Lister()
	e.serviceAccountSynced = serviceAccounts.Informer().HasSynced
	// Register the event listener for the ServiceAccount resource object and add the event to the syncServiceAccountQueue
	serviceAccounts.Informer().AddEventHandlerWithResyncPeriod(
		cache.ResourceEventHandlerFuncs{
			AddFunc:    e.queueServiceAccountSync,
			UpdateFunc: e.queueServiceAccountUpdateSync,
			DeleteFunc: e.queueServiceAccountSync,
		},
		options.ServiceAccountResync,
	)

	// ...
	secrets.Informer().AddEventHandlerWithResyncPeriod(
		cache.FilteringResourceEventHandler{
			FilterFunc: func(obj interface{}) bool {
				switch t := obj.(type) {
				case *v1.Secret:
					return t.Type == v1.SecretTypeServiceAccountToken // Kubernetes. IO /service-account-token" secret is filtered
				default:
					utilruntime.HandleError(fmt.Errorf("object passed to %T that is not expected: %T", e, obj))
					return false}},// Register the event listener for the Secret resource object and place the event in the syncSecretQueue
			Handler: cache.ResourceEventHandlerFuncs{
				AddFunc:    e.queueSecretSync,
				UpdateFunc: e.queueSecretUpdateSync,
				DeleteFunc: e.queueSecretSync,
			},
		},
		options.SecretResync,
	)

	return e, nil
}
// Add the Service object to the syncServiceAccountQueue
func (e *TokensController) queueServiceAccountSync(obj interface{}) {
    if serviceAccount, ok := obj.(*v1.ServiceAccount); ok {
        e.syncServiceAccountQueue.Add(makeServiceAccountKey(serviceAccount))
    }
}
// Put the secret object in syncSecretQueue
func (e *TokensController) queueSecretSync(obj interface{}) {
    if secret, ok := obj.(*v1.Secret); ok {
        e.syncSecretQueue.Add(makeSecretQueueKey(secret))
    }
}
Copy the code

After the data is queued, Goroutine calls Controller.run () to consume the queued data, executing the specific business logic:

func (e *TokensController) Run(workers int, stopCh <-chan struct{}) {
	// ...
	for i := 0; i < workers; i++ {
		go wait.Until(e.syncServiceAccount, 0, stopCh)
		go wait.Until(e.syncSecret, 0, stopCh)
	}
	<-stopCh
	// ...
}
Copy the code

Controller Business logic

Add, delete, modify, and query ServiceAccount

When adding, deleting, modifying, or querying a ServiceAccount, you need to determine two service logics: To delete a ServiceAccount, you need to delete the Secret object referenced by it. When adding or updating a ServiceAccount, ensure that the Secret object referenced exists. If it does not, create a new Secret object. Visible code:

func (e *TokensController) syncServiceAccount(a) {
	// ...
	// Query the service Account object from the local cache
	sa, err := e.getServiceAccount(saInfo.namespace, saInfo.name, saInfo.uid, false)
	switch {
	caseerr ! =nil:
		klog.Error(err)
		retry = true
	case sa == nil:
		// The service account has been deleted and the secret object referenced by it needs to be deleted
		sa = &v1.ServiceAccount{ObjectMeta: metav1.ObjectMeta{Namespace: saInfo.namespace, Name: saInfo.name, UID: saInfo.uid}}
		retry, err = e.deleteTokens(sa)
	default:
		// When creating or updating a service account, ensure that the secret object referenced by the service account exists. If it does not, create a new secret object
		retry, err = e.ensureReferencedToken(sa)
		// ...}}Copy the code

Let’s see how to delete the business logic of the secret object that it references. Deleting the logic is also simple:

// Delete the secret object referenced by the Service Account
func (e *TokensController) deleteTokens(serviceAccount *v1.ServiceAccount) ( /*retry*/ bool, error) {
	// List all secret references to the service account
	tokens, err := e.listTokenSecrets(serviceAccount)
	// ...
	for _, token := range tokens {
		// Delete secret objects one by one
		r, err := e.deleteToken(token.Namespace, token.Name, token.UID)
		// ...
	}
	// ...
}
func (e *TokensController) deleteToken(ns, name string, uid types.UID) ( /*retry*/ bool, error) {
    // ...
	// Request the api-server to delete the secret object resource
    err := e.client.CoreV1().Secrets(ns).Delete(name, opts)
    // ...
}
Copy the code

The key here is to find all secret objects referenced by serviceAccount. You can’t use the ServiceAccount. secrets field, which is only a partial secret value. Cache secret =kubernetes. IO /service-account-token Then look for the kubernetes.io/ service-Account. name of the Secret annotation, which should be serviceAccount. name, IO /service-account.uid should be the serviceAccount. uid value. The secrets referenced by the serviceAccount can be obtained only when the preceding conditions are met. UpdatedSecrets uses LRU(Least Recently Used) Cache to reduce memory usage.

func (e *TokensController) listTokenSecrets(serviceAccount *v1.ServiceAccount) ([]*v1.Secret, error) {
	// Find all secrets in namespace from LRU cache
	namespaceSecrets, err := e.updatedSecrets.ByIndex("namespace", serviceAccount.Namespace)
	// ...
	items := []*v1.Secret{}
	for _, obj := range namespaceSecrets {
		secret := obj.(*v1.Secret)
		// Check that the secret referenced by the serviceAccount is the one that meets the corresponding conditions
		if serviceaccount.IsServiceAccountToken(secret, serviceAccount) {
			items = append(items, secret)
		}
	}
	return items, nil
}
// Determine the condition
func IsServiceAccountToken(secret *v1.Secret, sa *v1.ServiceAccount) bool {
    ifsecret.Type ! = v1.SecretTypeServiceAccountToken {return false
    }
    name := secret.Annotations[v1.ServiceAccountNameKey]
    uid := secret.Annotations[v1.ServiceAccountUIDKey]
    ifname ! = sa.Name {return false
    }
    if len(uid) > 0&& uid ! =string(sa.UID) {
        return false
    }
    return true
}
Copy the code

Therefore, when the ServiceAccount object is deleted, you need to delete all Secrets objects referenced by it.

Now look at how to create the business logic for the Secret object. When creating or updating a ServiceAccount object, ensure that the Secrets object referenced by the ServiceAccount object exists. If it does not, create a Secret object:

// Check that the Secrets object referenced by the ServiceAccount object exists. If it does not, create a new one
func (e *TokensController) ensureReferencedToken(serviceAccount *v1.ServiceAccount) (bool, error) {
	// First make sure secret is present in the serviceAccount.secrets field value
	ifhasToken, err := e.hasReferencedToken(serviceAccount); err ! =nil {
		return false, err
	} else if hasToken {
		return false.nil
	}

	// Make a request to apI-server to find the serviceAccount object
	serviceAccounts := e.client.CoreV1().ServiceAccounts(serviceAccount.Namespace)
	liveServiceAccount, err := serviceAccounts.Get(serviceAccount.Name, metav1.GetOptions{})
	// ...
	ifliveServiceAccount.ResourceVersion ! = serviceAccount.ResourceVersion {return true.nil
	}

	// For a new ServiceAccount, add a default generated secret object to the ServiceAccount.secrets field value
	secret := &v1.Secret{
		ObjectMeta: metav1.ObjectMeta{
			Name:      secret.Strategy.GenerateName(fmt.Sprintf("%s-token-", serviceAccount.Name)),
			Namespace: serviceAccount.Namespace,
			Annotations: map[string]string{
				v1.ServiceAccountNameKey: serviceAccount.Name, // ServiceAccount.name is used as the annotation
				v1.ServiceAccountUIDKey:  string(serviceAccount.UID), // ServiceAccount.uid is used as the annotation
			},
		},
		Type: v1.SecretTypeServiceAccountToken,
		Data: map[string] []byte{},}// Generates a JWT token that is signed with a private key
	token, err := e.token.GenerateToken(serviceaccount.LegacyClaims(*serviceAccount, *secret))
	// ...
	secret.Data[v1.ServiceAccountTokenKey] = []byte(token)
	secret.Data[v1.ServiceAccountNamespaceKey] = []byte(serviceAccount.Namespace)
	ife.rootCA ! =nil && len(e.rootCA) > 0 {
		secret.Data[v1.ServiceAccountRootCAKey] = e.rootCA
	}

	// Create the secret object in api-server
	createdToken, err := e.client.CoreV1().Secrets(serviceAccount.Namespace).Create(secret)
	// ...
	// Write to LRU cache
	e.updatedSecrets.Mutation(createdToken)

	err = clientretry.RetryOnConflict(clientretry.DefaultRetry, func(a) error {
		// ...
		// Put the new secrets object into the ServiceAccount. secrets field and update the ServiceAccount object
		liveServiceAccount.Secrets = append(liveServiceAccount.Secrets, v1.ObjectReference{Name: secret.Name})
		if_, err := serviceAccounts.Update(liveServiceAccount); err ! =nil {
			return err
		}
		// ...
	})

	// ...
}
Copy the code

Therefore, when the ServiceAccount object is created, a new Secret object needs to be created as a reference to the ServiceAccount object. The business code is relatively simple.

Secret add, delete, change and check

When adding, deleting, modifying, and checking secret, you need to delete the secrets field reference under serviceAccount object when deleting secret.

func (e *TokensController) syncSecret(a) {
	// ...
	// Find the secret from the LRU Cache
	secret, err := e.getSecret(secretInfo.namespace, secretInfo.name, secretInfo.uid, false)
	switch {
	caseerr ! =nil:
		klog.Error(err)
		retry = true
	case secret == nil:
		// Delete secret:
		// Check whether the serviceAccount object exists
		if sa, saErr := e.getServiceAccount(secretInfo.namespace, secretInfo.saName, secretInfo.saUID, false); saErr == nil&& sa ! =nil {
			// Remove its secret reference from the service
			if err := clientretry.RetryOnConflict(RemoveTokenBackoff, func(a) error {
				returne.removeSecretReference(secretInfo.namespace, secretInfo.saName, secretInfo.saUID, secretInfo.name) }); err ! =nil {
				klog.Error(err)
			}
		}
	default:
		// When creating or updating secret:
		// Check whether the serviceAccount object exists
		sa, saErr := e.getServiceAccount(secretInfo.namespace, secretInfo.saName, secretInfo.saUID, true)
		switch {
		casesaErr ! =nil:
			klog.Error(saErr)
			retry = true
		case sa == nil:
			// If serviceAccount does not exist, delete secret
			ifretriable, err := e.deleteToken(secretInfo.namespace, secretInfo.name, secretInfo.uid); err ! =nil {
                // ...
			}
		default:
			// When creating or updating secret and serviceAccount exists, check whether the CA, Namespace, and Token fields in Secret need to be updated
			// Of course, when you create a new secret, you will need to update it
			ifretriable, err := e.generateTokenIfNeeded(sa, secret); err ! =nil {
				// ...}}}}Copy the code

Therefore, the service logic of adding, deleting, modifying and checking kubernetes. IO /service-account-token secret is relatively simple. The focus is to learn the official Golang code writing and some of the use of K8S API, for their secondary development of K8S is of great benefit.

conclusion

This paper mainly learns how TokensController listens to the increase, deletion, change and check of ServiceAccount object and Kubernetes. IO/service-Account-Token Secret object, and makes corresponding business logic processing. For example, you need to create a Secret object to create a ServiceAccount, delete a Secret object to delete a ServiceAccount, and create a Secret object. You also need to fill in the ca.crt/ Namespace /token field values for the Secret object, some boundary condition handling logic, and so on.

Meanwhile, the official code writing specification of TokensController, the application of K8S API, the processing of boundary conditions, and the use of LRU Cache are all worthy of reference in my own project.

Learning points

Tokens_controller. go L106 uses the LRU cache.

reference

Configure the service account for the Pod

Service account token Secret

Serviceaccounts – Controller source code official website parsing