This is the 19th day of my participation in the Genwen Challenge
Xvi Network Communication
The network communication of K8S is completely realized by the plug-in on the CNI interface, which needs to realize the following centralized communication model.
The most popular plug-ins are: Flannel, Calico, Canel, Kube-Router…
- How to load plug-ins
During startup, K8S will go to /etc/cni/net.d/ to find the configuration file of the network plug-in. When POD is created, K8S will call this configuration file and the plug-in will create the network according to this configuration file.
16.1 Communication Model
- Inter-container communication: The communication between multiple containers in the same POD, using lo network card
- Communication between Pods: POD IP communicates with POD IP directly
- POD and Service: The POD IP address directly matches the Cluster IP address
- Service communicates with external clients of the cluster: ingress, NodePort, Loadbacer
16.2 Communication Model Bottom layer
No matter what kind of network plug-in it is, it uses the following underlying schemes:
- Virtual bridge: BRG, using pure software to achieve a virtual network card, one end on POD, the other end on the host computer access to the bridge or physical interface bridge, known as the tunnel network.
- Multiplex: MacVLAN: Vlans are created in Mac-based mode. Each virtual interface is configured with an independent MAC address, so that one physical NIC can bear multiple containers. In this way, containers directly use physical nics for cross-node communication based on MacVLAN.
- Hardware switching: A NIC supports hardware switching and SR-IOV (single-root -I/O virtualization). This NIC can directly virtualize multiple interfaces at the physical level, providing high performance.
16.3 K8S namespace
The K8S namespace is not in the same dimension as the POD network namespace, so even different pods created within different K8S cluster namespaces can communicate directly over the network.
However, the flannel network plug-in, which is widely used at present, does not support the network isolation strategy of different cluster namespaces.
Calico supports address allocation and network isolation policies for different cluster namespaces, but it is more complex to use. It supports BGP layer 3 network forwarding and has better performance than Flannel.
You can also use Flannel for network management and install Calico for cluster namespace network isolation.
16.4 K8S Network Topology
All the PODS are connected to the network of the cNI0 interface of the local machine. Packets sent by the CNI0 interface reach flannel.1, which encapsulates the packets as a tunnel protocol and sends them through the real physical network card of the local machine.
- View the local interface
1: lo: Local loopback
2: ens33: # Host physical nic
3: docker0: # Docker default bridge network, useless in K8S can be removed
4: dummy0: #
5: kube-ipvs0: #
6: flannel.1: # Flannel Encapsulates tunnel packets
7: cni0: All containers are on this bridge
8: veth0c014b8b@if3: The container's nic is connected to CNI0
9: veth97c048e5@if3: The container's nic is connected to CNI0
11: vethd2f0bf2b@if3: The container's nic is connected to CNI0
12: veth648a500f@if3: The container's nic is connected to CNI0
Copy the code- Download the bridge-utils package run the BRCTL show cni0 command to view the CNI0 interface
Bridge Name Bridge ID STP Enabled interfaces CNI0 8000.9 A6EC95F8285 no VETH0c014b8b VETH648A500f veth7a3F56b7 veth97c048e5 vethd2f0bf2bCopy the code16.5 flannel
Flannel is a three-tier networking solution tailored for Kubernetes to address cross-host communication issues in containers.
16.5.1 Flannel Working Mode
- Flannel.1 The virtual NIC supports multiple transmission modes: VxLAN, host-GW, Directrouting, and UDP
| model | introduce |
|---|---|
| VXLAN | VxLAN is used as a tunnel to encapsulate packets |
| host-gw | Instead of using the superimposed network, route entries to subnets of other hosts are created in the routing table of the host, which has better performance. However, the disadvantage is that all nodes must be in the same Layer 2 network. |
| DirectRouting | If the hosts reside on the same subnet, enable direct routing instead of rolling back to the VxLAN. |
| UDP | The UDP protocol is used directly, but the performance is poor |
16.5.2 VXLAN Communication Process
Flannel VXLAN is essentially an overlay network. It packages TCP data in another network package for routing, forwarding, and communication. Currently, VXLAN supports UDP, VXLAN, AWS VPC, and GCE routing.
- Flannel VXLAN communication process
On K8S, POD and POD communicate directly through each other’s IP address. The packets sent by POD reach Flannel through the CNI0 network bridge. Flannel encapsulates the packets in the head of VxLAN, and the outer layer is encapsulated in the head of UDP. Flannel was then sent to the local physical network adapter. The local physical network adapter then encapsulated the packet with the IP header and MAC frame header, and the packet was then sent by the network adapter. Another node received the packet and the kernel discovered that it was a VxLAN packet. Flannel disconnects the VxLAN header and sends the internal data to the CNI0 bridge. The CNI0 receives the data and forwards it to the POD.
| | | | < -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- VxLAN encapsulation -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - > | < -- -- -- -- -- -- -- -- -- -- - original message -- -- -- -- -- -- -- -- -- -- -- -- - > | + -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- + | | node network node network | | node network VxLan | | | POD MAC POD IP data | | frame first MAC first | | IP UDP header | first | first | first | Payload | +-----------+-----------+-----------+-----------+-----------+-----------+-----------+Copy the code16.5.3 Flannel Deployment Mode
- Before the K8S cluster starts, Flannel is directly deployed on the node and runs as a daemon.
Flannel should be deployed on any node where Kubelet is deployed, because Kubelet uses flannel to set up a network interface for podsCopy the code- Use kube-admin directly to include K8S own components including flannel’s static POD running on top of K8S.
The Flannel POD must be configured as a POD in the shared node network namespace. Therefore, the Flannel POD controller is DaemonSet.Copy the code16.5.4 Flannel Configuration File
- Meaning of configuration file options
{"Network": "10.244.0.0/16", // The Network address in CIDR format used by flannel to configure the POD Network function "SubnetLen": 24, // How long is the mask used when dividing a Network into subnets for each node? The default value is 24 "SubnetMin": "10.244.10.0/24", // The starting subnet address assigned to node, from which network "SubnetMax" is assigned: "10.244.255.0/24" // End location of the subnet allocated to nIDE. This is the maximum network allocated. "Backend": {// Specifies the flannel working mode for communicating with POD across nodes. "Vxlan ", // working mode "Directrouting": true // whether to use Directrouting mode}}Copy the code- Flannel A configuration file hosted on k8S in the kube-flannel-cfg configmap.
kubectl get configmap kube-flannel-cfg -n kube-system -o json
Copy the code16.5.5 Changing the Working Mode
- Modify the Flannel working mode and add Directrouting. This operation should be modified after deploying the K8S cluster
kubectl edit configmap kube-flannel-cfg -n kube-system
Copy the code"Backend": {
"Type": "vxlan"."Directrouting": true
}
Copy the code- View the local routing table
ip route show
Copy the codeDefault via 172.16.100.254 dev ens33 Proto static metric 100 10.244.1.0/24 via 10.244.1.0 dev ens33Directrouting is not set successfully unless it is a dev physical nic interface
10.244.2.0/24 via 10.244.2.0 dev ens33 Directrouting is not set successfully unless it is a dev physical nic interface172.16.100.0/24 dev ens33 proto kernel scope link SRC 172.16.100.101 metric 100 172.17.0.0/16 dev docker0 proto kernel The scope link SRC 172.17.0.1Copy the code16.6 the Calico
Calico creates and manages a flat three-tier network (no overlays are required), and each container is assigned a routable IP. Because there is no need to unpack or seal packets during communication, network performance loss is small, easy to troubleshoot, and easy to horizontal expansion.
In small-scale deployment, BGP clients can be used for direct interconnection. In large-scale deployment, BGP Route reflector can be used to ensure that all data traffic is interconnected through IP routes.
Calico also provides rich and flexible network policies based on Iptables that guarantee Workload multi-tenant isolation, security groups, and other accessibility restrictions through ACLs on individual nodes.
A new project, Canel, brings together the best of Flannel and Calico.
- Pay attention to
Calico does not currently support Kube-proxy working under iptables. The following describes the use of canal network policy
16.6.1 installation canal
- To download the manifest file, you need to climb the wall
Kubectl apply -f https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/canal/canal.yamlCopy the code16.6.2 List definition
- List format, see kubectl Explain NetworkPolicy
egress <[]Object> # list of objects for the outbound rule
ports <[]Object> The object list of the destination port
port <string> # A numeric or named port
protocol Protocol TCP, UDP
to <[]Object> List of destination address objects
ipBlock <Object> # a set of IP addresses
cidr <string> CIDR indicates the IP range
except <[]string> Exclude certain addresses in CIDR
namespaceSelector <Object> # namespace selector
podSelector <Object> The destination address can also be a set of pods
ingress <[]Object> Object list of inbound rules
from <[]Object> Source address object list
ipBlock <Object> # a set of IP addresses
cidr <string> CIDR indicates the IP range
except <[]string> Exclude certain addresses in CIDR
namespaceSelector <Object> # namespace selector
podSelector <Object> The source address can also be a set of pods
ports <[]Object> # POD a list of objects that control whether their ports are accessible
port # A numeric or named port
protocol Protocol TCP, UDP
podSelector <Object> The POD selector determines which pods the rule applies to
policyTypes <[]string> The value can be "Ingress", "Egress", or "Ingress,Egress", allowing access that meets these rules
Copy the code16.6.3 policyTypes
- First, define the namespace
kubectl create namespace dev
kubectl create namespace prod
Copy the code- Create a POD in each namespace
apiVersion: v1
kind: Pod
metadata:
name: pod1
namespace: dev
labels:
app: myapp
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
---
apiVersion: v1
kind: Pod
metadata:
name: pod1
namespace: prod
labels:
app: myapp
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
Copy the codekubectl apply -f pod-a.yaml -n dev
Copy the code- Denied all packets for the dev space
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-ingress
namespace: dev
spec:
podSelector: {} # {} An empty selector means select all
policyTypes:
- Ingress If the Ingress rule is valid, all matching Ingress will be allowed
# policyTypes No Egress indicates that the Egress is not controlled
Copy the code- Applies the rule file to the specified namespace
kubectl apply -f deny-all-ingress.yaml -n dev
Copy the code- View the rules
kubectl get networkpolicy -n dev
Copy the code- Look at the POD address in the dev space and access it, but it is not accessible because the namespace is closed to external access
kubectl get pods -n dev -o wide
Copy the codeThe curl 10.244.1.2Copy the code- Look at the POD address in the PROd space and access it. The results are accessible because there are no rules defined for this namespace
kubectl get pods -n dev -o wide
Copy the codeThe curl 10.244.2.2Copy the code- Allow pods on the specified network segment to access port 80 of this POD
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-80-ingress
namespace: dev
spec:
podSelector:
matchLabels:
app: myapp
ingress:
- from:
- ipBlock: # specify source address as IP address block
cidr: 10.244. 0. 0/ 16 The mask indicates the source IP address range
except: Exclude an address in cidR range
- 10.2441.2./ 32
ports:
- port: 80 If the destination port is 80, the value will match
protocol: TCP
- port: 443
protocol: TCP
policyTypes:
- Ingress If the Ingress is not defined, all matches will not be allowed
# policyTypes No Egress indicates that the Egress is not controlled
Copy the code- View the rules
kubectl get networkpolicy -n dev
Copy the code- Reject all requests for removal from the stack
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-egress
namespace: prod
spec:
podSelector: {} # {} An empty selector means select all
policyTypes:
- Egress If the Egress rule takes effect, all matched Egress is allowed. If the Egress is not specified, all matched Egress is denied
# policyTypes No Ingress indicates that the Egress is not controlled. This parameter is enabled by default
Copy the codeother
Send your notes to: github.com/redhatxl/aw… Welcome one button three links