Welcome to pay attention to personal account “Operation and Maintenance Development Story”

instructions

This paper mainly includes the following contents:

  • Vxlan overview
  • Why use VXLAN
  • How can K8S use flannel to communicate with POD

Vxlan overview

Virtual eXtensible LAN(VXLAN) is a Virtual tunnel communication technology. It is a kind of overlay network technology that builds a virtual layer 2 network with three layers of networks. In simple terms, VXLAN is a logical network that uses tunnel technology on top of underlay and relies on UDP layer overlay. It decouples the logical network from the physical network to meet flexible networking requirements. It can be used not only in virtual machine environments, but also in container environments.

Why use VXLAN

  1. The VXLAN supports more subnets (a VLAN supports only 2 to the 12th subnets, and a VXLAN supports 2 to the 24th subnets). The Virtual Network Identifier (VNI) is used to distinguish different subnets, which is equivalent to the LAN ID of a VLAN
  2. Multi-tenancy network isolation. IP and MAC addresses must be independently assigned to different users
  3. Cloud computing services have high requirements on service flexibility. VMS may be migrated on a large scale and the network is always available. To solve this problem and ensure that the layer 2 broadcast domain does not expand too much, which is also the requirement of cloud computing network

Using flannel in K8S

Note: I use kubeadm installed K8S, version is 1.19, flannel network mode is VXLAN, you can modify according to your needs.

[root@master huazai]# kubectl version Client Version: Version. The Info {Major: "1", Minor: "19", GitVersion: "v1.19.0 GitCommit:" e19964183377d0ec2052d1f1fa930c4d7575bd50." BuildDate GitTreeState: "clean" : "the 2020-08-26 T14: shall Z," GoVersion: "go1.15", the Compiler: "gc", Platform:"linux/amd64"} Server Version: Version. The Info {Major: "1", Minor: "19", GitVersion: "v1.19.0 GitCommit:" e19964183377d0ec2052d1f1fa930c4d7575bd50." GitTreeState:"clean", BuildDate:" 2020-08-26T14:23:04z ", GoVersion:" GO1.15 ", Compiler:" GC ", Platform:" Linux/AMD64 "}Copy the code

Reference: kubernetes. IO/docs/concep…

Download the flannel. Yml

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Copy the code

Install the flannel

kubectl apply -f kube-flannel.yml
Copy the code

Viewing installation Results

[root@master huazai]# kubectl  get po -A |grep flannel
kube-system            kube-flannel-ds-f4x7m                        1/1     Running   0          15h
kube-system            kube-flannel-ds-ltr8h                          1/1     Running   0          15h
kube-system            kube-flannel-ds-mp76x                        1/1     Running   0          15h
Copy the code

Look at what flannel did to the host after it was installed

  1. Example Create a VXLAN nic named Flannel.1
[root@master huazai]# ip -d link show flannel.1 4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether Fe :be:87:93:06:e2 BRD ff:ff:ff:ff:ff:ff promiscuity 0 VXLAN ID 1 local 192.168.0.39 dev eth0 srcport 0 0 dstport 8472 nolearning ageing 300 noudpcsumCopy the code

The MTU value is 1450(the IP header, UDP header, MAC header, and VXLAN header account for 50). Dstport is 8472 and local IP is the node IP address. View the following information about flannel

[root@master huazai]# ifconfig flannel.1 flannel.1: Flags = 4163 < UP, BROADCAST, RUNNING, MULTICAST > mtu 1450 inet 10.244.0.0 netmask 255.255.255.255 BROADCAST 10.244.0.0 inet6 fe80::fcbe:87ff:fe93:6e2 prefixlen 64 scopeid 0x20<link> ether fe:be:87:93:06:e2 txqueuelen 0 (Ethernet) RX packets 2622 Bytes 465577 (454.6KIb) RX errors 0 Dropped 0 Overruns 0 Frame 0 TX packets 4765 bytes 1081070 (1.0 MiB) TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0Copy the code
  1. Create routing table to pod CIDRs of other nodes (known by kubectl get node master-o Yaml), mainly to route the traffic in POD to Flannel.1 interface
[root@master huazai]# route -n 10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1 10.244.2.0 10.244.2.0 255.255.255.0 UG 0 0 0 FlannelCopy the code
  1. Add the IP address of the node and the static ARP cache of the VTEP
[root@master huazai]# arp-n 10.244.1.0 ether 0e:61:06:ff:7a:73 CM Flannel.1 0a:72:bf:3f: CD :40 CM [root@master huazai]# Bridge FDB 0a:72:bf:3f: CD :40 dev Flannel.1 DST 192.168.0.8 self permanent 0e:61:06:ff:7a:73 Dev Flannel.1 DST 192.168.0.22 self permanentCopy the code

The preceding MAC addresses are MAC addresses of flannel.1 on corresponding nodes

How do PODS access each other

  1. How can THE POD of the same node be accessed

Take the following two pods as an example, both of which are located on node1 with IP addresses 10.244.1.8 and 10.244.1.9 respectively. Assume that the POD with IP address 10.244.1.8 is pinged from the pod with IP address 10.244.1.9

[root@master huazai]# kubectl get po-o wide nginx-deployments-66b6c48DD5-NZjGD 1/1 Running 0 35m 10.244.1.8node1 Nginx-deployment-66b6c48dd5-jcwc9 1/1 Running 0 35m 10.244.1.9 node1Copy the code

Enter the POD with pod IP 10.244.1.8

[root@master huazai]# kubectl exec -it  nginx-deployment-66b6c48dd5-jcwc9  -- /bin/bash
root@nginx-deployment-66b6c48dd5-jcwc9:/#
Copy the code

View its route

root@nginx-deployment-66b6c48dd5-jcwc9:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.244.1.1      0.0.0.0         UG    0      0        0 eth0
10.244.0.0      10.244.1.1      255.255.0.0     UG    0      0        0 eth0
10.244.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
Copy the code

It can be found that the POD on the same node is directly accessed (in the same network segment) without being forwarded. Enter another POD to view the route and find the same

[root@master ~]# kubectl exec -it nginx-deployment-66b6c48dd5-nzjgd  -- /bin/bash
root@nginx-deployment-66b6c48dd5-nzjgd:/#
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.244.1.1      0.0.0.0         UG    0      0        0 eth0
10.244.0.0      10.244.1.1      255.255.0.0     UG    0      0        0 eth0
10.244.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
Copy the code
  1. How can THE POD of different nodes be accessed

Take the following two pods as an example. One pod is on node1 with IP 10.244.1.8, and the other pod is on Node2 with IP 10.244.2.4

[root@master ~]# kubectl get Po -o wide nginx-deployments-66b6c48dd5-f7v9q 1/1 Running 0 60m 10.244.2.4node2 Nginx-deployment-66b6c48dd5-nzjgd 1/1 Running 0 60m 10.244.1.8node1Copy the code

Enter the POD with pod IP 10.244.1.8

[root@master huazai]# kubectl exec -it  nginx-deployment-66b6c48dd5-jcwc9  -- /bin/bash
root@nginx-deployment-66b6c48dd5-jcwc9:/#
Copy the code

root@nginx-deployment-66b6c48dd5-jcwc9:/# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.244.1.1 0.0.0.0 UG 0 0 0 eth0 10.244.0.0 10.244.1.1 255.255.0.0 UG 0 0 0 eth0 10.244.1.0 You can find that ping 10.244.2.4 requires 10.244.1.1. 10.244.1.1 is the IP address of CN0 on Node1. Cni0 is the bridge created by flannel

[root@node1 net.d]# ifconfig cni0: Flags = 4163 < UP, BROADCAST, RUNNING, MULTICAST > mtu 1450 inet 10.244.1.1 netmask 255.255.255.0 BROADCAST 10.244.1.255Copy the code

Look again at the route on Node1

[root@node1 net.d]# route -n 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0
10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.1.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
10.244.2.0      10.244.2.0      255.255.255.0   UG    0      0        0 flannel.1
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.0.0     0.0.0.0         255.255.255.192 U     0      0        0 eth0
Copy the code

The next hop of the packet whose destination ADDRESS is 10.244.2.0 is 10.244.2.0, and flannel.1 and Flannel.1 serve as a VTEP device. After receiving the packet, the FLANnel.1 will be sealed according to the VTEP configuration. View arp and FDB on node1

[root@node1 net.d]# arp-n 10.244.0.0 ether fe:be:87:93:06 e2 CM flannel.1 10.244.2.0 ether :72:bf:3f: CD :40 CM [root@node1 net.d]# Bridge FDB 0a:72:bf:3f: CD :40 dev Flannel.1 DST 192.168.0.8 self permanent Fe :be:87:93:06:e2 dev flannel.1 DST 192.168.0.39 self permanentCopy the code

In this case, etCD can be used to know that 10.244.2.4 is on Node2, and the IP address of Node2 can be obtained, and the MAC address of VTEP corresponding to Node2 can be known through the forwarding table of node1. Then configure VXLAN packets based on the parameters (VNI, local IP, and Port) set during flannel. The VXLAN packet reaches Node2 through port 8472. The VXLAN packet is forwarded to VTEP Flannel. The decapsulated IP packet matches the routing table (10.244.2.0) in Node2. The kernel forwards IP packets to CNI0.

[root@node2 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0 10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1 10.244.1.0 10.244.1.0 UG 0 0 0 flannel.1 10.244.2.0 0.0.0.0 255.255.255.0 U 0 0 0 Cni0Copy the code

Cni0 forwards IP packets to pods connected to cNI0

conclusion

According to the above findings, when pods on different nodes want to access each other, they need to go through host routing and the packet unpacking operation of the kernel. The whole process is shown as follows:Therefore, you can use tcpdump to judge and process packets captured by CN0, Flannel. 1, eth0, and veth pair on nodes when discovering related networks. In addition, check the ARP and FDB information on the node.