In Openyurt, Yurt-tunnel is an important component for constructing cloud side tunnels. In this paper, the basic principle of Yurt Tunnel is introduced in detail. How to solve the O&M monitoring challenge of K8s in cloud side collaboration. In this document, the DNS mode of yurt-Tunnel and specific configuration methods are introduced.

Defects of DNAT

Yurt Tunnel supports DNAT and DNS diversion modes. By default, yurt-tunnel-server forwards the requests sent to ports 10250 and 10255 of Kubelet to Yurt-tunnel-server by configuring DNAT rules for nodes. In other words, in DNAT mode, when kubectl logs/exec is executed, apiserver’s request to edge node Kubelet will be diverted to Yurt-tunnel-server to realize cloud side tunnel communication. DNAT provides seamless flow diversion without the need for additional configuration of cloud side components. However, the DNAT mode also has some disadvantages:

In edge computing scenarios, the NodeIP of edge nodes may be duplicated. An edge node cannot be uniquely identified by NodeIP:Port or Yurt-tunnel-server.
The DNAT rule is configured only on the node where the Yurt-tunnel-server resides. If you need to access the edge cloud components (such as Apiserver and Promethues) and the Yurt-Tunnel-server are not on the same node, the DNAT rule is not configured. Cloud components will not be able to access edge components through tunnels. For example, the following scenario:

To solve the preceding two problems, use the DNS mode of yurt-Tunnel.

Principles of DNS Mode

In this paper, the principle of DNS mode is introduced:

1. The yurt-tunnel-server creates or updates the Yurt-tunnel-Nodes configmap to kube-Apiserver. The format of the tunnel-Nodes field is: {x - tunnel - server - internal - SVC clusterIP} {nodeName}, Ensure that the mapping between all nodeName and Yurt-tunnel-server services is recorded. 2. Mount yurt-Tunnel-Nodes configMap to coreDNS Pod. Use DNS records from Configmap with host plugin 3. Configure port mapping in x-tunnel-server-internal-svC. 10250 is mapped to 10263. 10255 is mapped to 102644. Through the above configuration, http://{nodeName}:{port}/{path} requests can be seamlessly forwarded to Yurt-tunnel-ServersCopy the code

The process is roughly as follows:

When apiserver receives an exec/logs request, Apiserver uses the node hostname to access Kubelet. The hostname of the node is resolved to the service address of Yurt-Tunnel-server by the CoreDNS in the cluster to divert traffic.

To get through this process, the following problems need to be solved:

To configure Apiserver and other cloud components (for example, Prometheus), use the node hostname as the domain name to access Kubelet and other edge components.
To configure Apiserver and other cloud components (for example, Promethues), use CoreDNS as the DNS server.
To solve problem 2, a new problem arises: The cloud component needs to access CoreDNS through service IP. In order to prevent the cloud from accessing the edge CoreDNS address, we need to open the traffic closed-loop function for the cloud — that is, the cloud also needs to deploy Yurthub and configure kube-proxy to access apiserver through Yurthub. Through the flow closed-loop function, Yurthub filters out the edge CoreDNS address.

The sample

Next, we use an example to demonstrate how to configure the Yurt-Tunnel DNS mode so that the K8S native interface (Kubectl logs/exec) and Promethues can work properly. This example can be run in the local VIRTUAL machine environment, which is listed as follows:

One master node and one worker node;
K8S cluster version is V1.18.9, using kubeadm default configuration to deploy the native cluster;
Convert to An Openyurt cluster using YURT convert or manually deploy the Openyurt image using the current master branch (COMMIT ID: 33e3445)

Depending on some of Openyurt’s latest features and optimizations, using older versions of Openyurt images may cause the example to not work properly.

$kubectl get nodes -owide NAME STATUS ROLES AGE VERSION internal-ip external-ip Os-image kernel-version CONTAINER-RUNTIME master-1 Ready master 2D18h v1.18.9 192.168.33.220 < None > CentOS Linux 7 X86_64 docker://19.3.15 worker-1 Ready < None > 2d18h v1.18.9 192.168.33.221 < None > CentOS Linux 7 (Core) 3.10.0-1127. El7. X86_64 docker: / / 19.3.15Copy the code

Flow closed loop and CoreDNS configuration

Yurthub is deployed on the cloud

To solve problem 3, you need to deploy Yurthub in the cloud. Copy the Yurthub manifest from the worker-1 node. Add –disabled-resource-filters= discardCloudservice flag and save it in master-1’s manifest directory:

# vi /etc/kubernetes/manifests/yurt-hub.yaml ... command: - yurthub - - v = 2 - - server - addr = https://192.168.33.220:6443 - - node - name = $(NODE_NAME) - --access-server-through-hub=true - --disabled-resource-filters=discardcloudservice # Add - disabled - resource - filters = discardcloudservice...Copy the code

By default, the Yurthub data filtering framework enables the discardCloudService function to filter out some cloud-specific services, such as x-tunnel-server-internal-svC. However, in the cloud, we need to access yurt-tunnel-server through x-tunnel-server-internal-svC service, so we need to disable the filter. After Yurthub starts normally, kubelet configuration needs to be modified to enable Kubelet to access Apiserver through Yurthub. For details about how to configure kubelet, see github.com/openyurtio/…

Add RBAC to Yurthub

Some RBAC rules are missing in the current version of Yurthub. Before we fix them, we will manually fix them:

$ cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: yurt-hub
rules:
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - get
  - apiGroups:
      - apps.openyurt.io
    resources:
      - nodepools
    verbs:
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: yurt-hub
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: yurt-hub
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: system:nodes
EOF
Copy the code

Setting a Node Pool

The function of flow closed-loop depends on node pool. We create a node pool for each cloud side:

$ cat <<EOF | kubectl apply -f -
apiVersion: apps.openyurt.io/v1alpha1
kind: NodePool
metadata:
  name: master
spec:
  type: Cloud
---
apiVersion: apps.openyurt.io/v1alpha1
kind: NodePool
metadata:
  name: worker
spec:
  type: Edge
EOF
Copy the code

Add master-1 and worker-1 nodes to the battery of master and worker respectively.

$ kubectl label node master-1 apps.openyurt.io/desired-nodepool=master
node/master-1 labeled
$ kubectl label node worker-1  apps.openyurt.io/desired-nodepool=worker
node/worker-1 labeled
Copy the code

Modify CoreDNS configurations

Kubeadm deploys CoreDNS in a Deployment mode by default, and in a cloud-side scenario, each node pool requires CoreDNS capability. Here, CoreDNS was redeployed in Daemonset mode. During the modification process, the following volumes were added to the Daemonset:

        volumeMounts:
        - mountPath: /etc/edge
          name: hosts
          readOnly: true
...

      volumes:
      - configMap:
          defaultMode: 420
          name: yurt-tunnel-nodes
        name: hosts
Copy the code

The new Yurt-tunnel-Nodes ConfigMap volume stores DNS records of each node and is maintained by yurt-tunnel-Server:

$ kubectl get cm -n kube-system yurt-tunnel-nodes -oyaml apiVersion: v1 data: tunnel-nodes: "10.106.217.16 \ \ n192.168.33.220 \ tmaster tworker - 1-1" kind: ConfigMap...Copy the code

This configuration resolves worker-1 to 10.106.217.16, the x-tunnel-server-internal-svc service address. Resolve master-1 to 192.168.33.220, which is the IP address of master-1. Obviously, master-1 does not need to access itself through yurt-tunnel-server.

Next modify coreDNS ConfigMap and add hosts plugin:

$ kubectl edit cm -n kube-system coredns -oyaml apiVersion: v1 data: Corefile: |. 53 {errors: health} {lameduck 5 s ready hosts/etc/edge/tunnel - nodes {# add hosts plug-in reload 300 ms fallthrough} kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } ...Copy the code

After the modification, restart CoreDNS Pod for it to take effect.

Modify kube – DNS service

To make CoreDNS kube – DNS service support flow closed loop, needs to increase openyurt. IO/topologyKeys: openyurt. IO/nodepoolannotation:

$ kubectl edit svc kube-dns -n kube-system apiVersion: v1 kind: Service metadata: annotations: Openyurt. IO/topologyKeys: openyurt. IO/nodepool # add annotations...Copy the code

Configuration kube – proxy

To support traffic closed-loop, Kubernetes versions need >=1.18(relying on EndpointSlices), and kube-proxy EndpointSlices need to be enabled.

With EndpointSlices turned on, kube-proxy listens for EndpointSlices (instead of the original Endpoints resources) to configure proxy forwarding for back-end services. Yurthub filters EndpointSlices to return only the endpoint addresses in the node pool, enabling the closed-loop function of traffic.

EndpointSlices reference documentation: v1-18. The docs. Kubernetes. IO/docs/concep…

In K8S version v1.18.9 of the sample environment, the EndpointSlices feature is turned off by default, and we need to enable EndpointSliceProxying feature gate for kube-proxy.

How to open the EndpointSlices: v1-18. The docs. Kubernetes. IO/docs/tasks /…

The default installation of kubeadm will generate kubeconFig configuration for Kube-proxy. In order to support traffic closed loop, kube-Proxy will access Apiserver through Yurthub. Thanks to Yurthub’s masterService overwriting and bearer Token transparent transmission, kubeconFig configurations of Kube-proxy can be deleted directly.

Masterservice rewrite: After kubeconfig is deleted, kube-proxy will use the InClusterConfig configuration to access apiserver, The apiserver address of the InClusterConfig configuration (i.e., the environment variables KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT) is overwritten by Kubelet to the Yurthub address.

Bearer of tokens: Yurthub has been bearer of authenticated tokens through requests made by Yurthub agents. In this way, kube-proxy retains the RBAC permission of Kube-proxy ServiceAccount, so that Kube-Proxy can transparently use Yurthub without additional RBAC policy configuration.

Kube-proxy ConfigMap: kube-proxy ConfigMap

$ kubectl edit cm -n kube-system kube-proxy apiVersion: v1 data: config.conf: |- apiVersion: Kubeproxy. Config. K8s. IO/v1alpha1 bindAddress: 0.0.0.0 featureGates: # ① Enable EndpointSliceProxying feature gate. EndpointSliceProxying: true clientConnection: acceptContentTypes: "burst: #kubeconfig: /var/lib/kube-proxy/kubeconfig.conf # 10.244.0.0/16 configSyncPeriod: 0 s...Copy the code

After the modification, restart kube-proxy for it to take effect:

$ kubectl delete pods -l k8s-app=kube-proxy -n kube-system
Copy the code

Kube-proxy has enabled endpointSlice Controller. Kube-proxy has enabled endpointSlice Controller.

$kubectl logs -f kube-proxy-62b5k -n kube-system I0911 17:37:08.422052 1 server.go:548] Neither kubeconfig file nor Master URL was specified. Falling back to in-cluster config.w0911 17:37:08.423936 1 Server_others. go:559] Unknown proxy Model "", Assuming iptables Proxy I0911 17:37:08.472651 1 node.go:136] Successfully retrieved node IP: Go :186] Using iptables proxier. I0911 17:37:08.488403 1 server.go:583] Version: V1.18.9 I0911 17:37:08.491237 1 conntrack.go:52] Setting Nf_conntrack_max to 131072 I0911 17:37:08.492341 1 Config. Go :315] Starting Service config Controller I0911 17:37:08.492357 1 Shared_informer. Go :223] Waiting for caches to Sync for service config I0911 17:37:08.492384 1 config.go:224] Starting endpoint slice config Controller I0911 17:37:08.492387 1 shared_informer. Go :223] Waiting for caches to sync for endpoint slice config I0911 17:37:08.593107 1 Shared_informer. Go :230] Caches are synced for endpoint slice config I0911 17:37:08.593119 1 shared_informer. Go :230] Caches are synced for service configCopy the code

Configure apiserver to use domain names to access nodes

Apiserver uses the — kubelet-preferredaddress-types configuration to determine how to access kubelet. The default configuration of kubeadm sets the apiserver to use InternalIP to access kubelet.

$ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep 'kubelet-preferred-address-types'
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 
Copy the code

Apiserver needs to use Hostname first to access Kubelet: – kubelet – preferred – address – types = the Hostname, InternalIP, ExternalIP.

Apiserver is generally deployed in hostNetwork mode. In hostNetwork mode, DNS configurations on hosts are used by default. To enable apiserver to use CoreDNS in hostNetwork mode, the dnsPolicy of apiserver Pod needs to be set to ClusterFirstWithHostNet.

$ vi /etc/kubernetes/manifests/kube-apiserver.yaml apiVersion: v1 kind: Pod ... Spec: dnsPolicy: ClusterFirstWithHostNet containers - kube-apiserver ... - - kubelet - preferred - address - types = the Hostname, InternalIP, ExternalIP # (2) put the Hostname as the first...Copy the code

After all the above configuration, Kubectl logs/exec works in DNS mode of Yurt-Tunnel.

Prometheus configuration

After configuring the K8S native interface, we will configure Prometheus to also access the edge expoter address via yurt-tunnel. In this example, the community-popular Prometry-operator monitoring scheme is used and deployed using Kube-Prometheus (v0.5.0).

Prometheus uses node IP to access kubelet and Node-Exporter metric addresses by default. Prometheus provides relabel to modify the node IP to hostname. Promethues -operator uses the ServiceMonitor CRD to define the fetch configuration, and relabel rules are configured in the corresponding ServiceMonitor.

Kubelet monitoring configuration

Update kubelet ServiceMonitor to add relabel rule and replace node IP with __meta_kubernetes_endpoint_address_target_name:

$ kubectl edit serviceMonitor kubelet -n monitoring apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: . spec: endpoints: ... Relabelings: -sourcelabels: -__metrics_path__ targetLabel: metrics_path - Action: replace # ① Add relabel rule regex: (.*); .*:(.*) replacement: $1:$2 sourceLabels: - __meta_kubernetes_endpoint_address_target_name - __address__ targetLabel: __address__ ... - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token ... Relabelings: -sourcelabels: -__metrics_path__ targetLabel: metrics_path__ - Action: replace # ② Add relabel rule regex: (.*); .*:(.*) replacement: $1:$2 sourceLabels: - __meta_kubernetes_endpoint_address_target_name - __address__ targetLabel: __address__ ...Copy the code

Kubelet exposes two metric addresses, /metrics and /metrics/ cAdvisor. The default capture address __address__ is changed from IP+ port to domain name + port by using relabel rules (1) and (2).

Prometheus relabel promethues configuration method reference document. IO/docs/promet…

Node – exporter configuration

Similarly, we modify node-Exporter’s ServiceMonitor to replace the node IP with __meta_kubernetes_pod_node_name.

$ kubectl edit ServiceMonitor -n monitoring node-exporter apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: ... spec: endpoints: ... relabelings: - action: replace regex: (.*) replacement: $1 sourceLabels: - __meta_kubernetes_pod_node_name targetLabel: Instance-action: replace # ① Add relabel rule regex: (.*); .*:(.*) replacement: $1:$2 sourceLabels: - __meta_kubernetes_pod_node_name - __address__ targetLabel: __address__ ...Copy the code

Note that the node IP is __meta_kubernetes_endpoint_address_target_name and the node IP is __meta_kubernetes_endpoint_address_target_name. __meta_kubernetes_pod_node_name both specific differences can reference documentation: Prometheus. IO/docs/promet…

The port 9100 is a node-exporter. By default, yurt-tunnel-server forwards only kubelet-related ports 10250 and 10255. You can add other ports by modifying configurations. Modify yurt-tunnel-server-cfg ConfigMap to add port 9100 to HTTPS proxy-ports :(similarly, if HTTP port is added, modify http-proxy-ports.)

$ kubectl edit cm -n kube-system yurt-tunnel-server-cfg apiVersion: v1 data: dnat-ports-pair: "" http-proxy-ports: Localhost-proxy-ports: 10266, 10267 KIND: ConfigMap localhost-proxy-ports: 10266, 10267 KIND: ConfigMapCopy the code

When the DNSController in Yurt-tunnel-server detects that the configuration changes, it modifies the x-tunnel-server-internal-svcService and adds the mapping 9100 to 10263.

$ kubectl describe svc x-tunnel-server-internal-svc -n kube-system Name: x-tunnel-server-internal-svc Namespace: kube-system Labels: name=yurt-tunnel-server Annotations: <none> Selector: k8s-app=yurt-tunnel-server Type: ClusterIP IP: 10.106.217.16 Port: HTTP 10255/TCP TargetPort: 10264/TCP Endpoints: 192.168.33.220:10264 Port: HTTPS 10250/TCP TargetPort: 10263/TCP Endpoints: 192.168.33.220:10263 Port: TargetPort: 10263/TCP Endpoints: 192.168.33.220:10263 Session Affinity: dNAT-9100 9100/TCP None Events: <none>Copy the code

After the configuration is complete, from the Promethues console, you can see that Kubelet and Node-Exporter use the hostname of the node as the domain name to access and can pull and fetch data normally.

End

At this point, all configuration is complete. It is found that the process of manually configuring the Yurt-Tunnel DNS mode is complicated. You need to manually create a node pool, deploy cloud Yurthub, and configure Apiserver and Kube-proxy. We look forward to the continued optimization of the Openyurt community to bring you a more silky deployment experience.

reference

Deep interpretation of Openyurt Series Yurthub data filtering framework Proposal

Openyurt Indicates the DNS mode of the Yurt Tunnel