When it comes to virtual environments, the VMware vSphere network is probably one of the most critical components.
How does your ESXi host communicate with VMS?
Since virtual networks are the key to everything else, you need to have a good understanding of how ESXi networks work.
1. Analyze the ESXi network
When it comes to our ESXi hosts and their network communication, the first thing we need to know is what they need to do. There are several different ways to communicate:
- Communicate with each other (think vMotion)
- Communicate with vCenter (think HA)
- Communicate with external resources they depend on, such as storage arrays or Active Directory
- Allow their resident virtual machines to communicate with other virtual machines and external resources that they may depend on
Therefore, several different components of the ESXi network in VMware vSphere must be configured in our environment.
- Virtual switches on ESXi hosts
- Port group on the virtual switch
In each of these categories, there are many things to consider.
2. Virtual switches on ESXi hosts
Think of the virtual switches on ESXi hosts as any other switches you might have encountered before, and when it comes to the network layer, the virtual switches in ESXi provide connectivity, redundancy, and load balancing for our ESXi hosts.
There are two types of switches, although you may hear them each using several different names.
- VSphere Standard switch
- VSphere Distributed switch (requires a vSphere Enterprise Plus license)
Let’s take a look at each virtual switch and its features.
3. Standard vSphere switch/standard vSwitch/vSwitch
Basic virtual switches, also known as VSwitches. It is the first Switch type available in VMware vSphere, and there is nothing wrong with using standard VSwitches, although they are harder to manage than vSphere Distributed Switch.
The standard vSwitch resides on ESXi hosts and must be configured separately on each host. There are several ways to simplify this process, such as using PowerCLI or host configuration files.
Network switches connect to network adapters on ESXi hosts, and network adapters on ESXi hosts connect to virtual switches on ESXi hosts. !
When we talk about the nics of ESXi hosts, we call them VMnics because that’s what they are called in ESXi. Vmincs always start at 0, so your first physical nic port will correspond to VMNIC0.
Depending on how you design your virtual network, you may have multiple VSwitches on your ESXi host.
VSphere Distributed switch/Distributed virtual switch /dvSwitch
In physical terms, there is no significant difference between standard virtual switches and distributed virtual switches. The network adapters on ESXi hosts are still connected to network switches in a redundant manner. The changes are the virtual layer and the switches in VMware vSphere itself.
Here is a picture of a distributed virtual switch:
As you can see, the switch configuration itself is located on the vCenter server and not on the ESXi host. Of course, if something goes wrong with vCenter, this could be a problem, right?
Each ESXi host has a copy of the distributed virtual switch configuration, as you can see in the bright blue switch on each host, so that if vCenter has problems, our virtual machines are not affected.
I remember when distributed virtual switches first appeared in vSphere 4, it took everyone by surprise because the way we are used to doing things has changed so dramatically, so let’s break down our vSphere DISTRIBUTED switches again:
Each VMNIC is connected to the corresponding dvUplink port on the distributed virtual switch. Then, the vCenter server is used to manage distributed virtual switches. The important part about this is that the distributed virtual switch configuration is the same design on each ESXi host.
Similar to standard VSwitches, distributed VSwitches can be managed using vSphere Client, PowerCLI, or host configuration files. The difference is that you need to configure a single distributed virtual switch on the vCenter server instead of a single virtual switch on the ESXi host.
5. Which VMware virtual switch is suitable for my environment?
It’s a good question and unfortunately there is no single answer. Your choice of standard or distributed virtual switches will depend on the unique requirements of your environment.
When making choices, it is important to keep infrastructure design quality in mind when determining what meets your requirements. They are:
- availability
- manageability
- performance
- recoverability
- security
It may also be helpful to rank these qualities in order of importance to the project.
Be sure to extensively test your VMware vSphere network configuration before declaring your VMware vSphere environment ready for production.
Keep in mind that vSphere Distributed Switch does require a VMware Enterprise Plus license.
6. NIC group policy in VMware
Another important part of the VMware network is the NIC combination. NIC Teaming is setting up our virtual switches in a way that promotes load balancing and prevents component failures.
For example, if my switch is connected to two physical network cards, ideally I want to ensure that traffic flows through both physical network cards and that my switch will remain operational if either network card or upstream switch fails.
There are several ways to do this in VMware vSphere Networking.
Now assuming that we have configured our switch, proceed to the next step.
7. Configure port groups on the virtual switch
When we connect to our virtual switch, we connect to something called a port group. A port group is, as its name suggests, a group of virtual ports on our virtual switch.
There are several different types of port groups:
- Vm port group
- The VMkernel port group
Virtual machine port groups are how we connect to virtual machines. For example, I might have a port group with a single VLAN, multiple vlans (VLAN relay), or in the case of distributed virtual switches, a private VLAN.
As for VMkernel port groups, you might ask…
8. What are VMkernel ports?
Good question! VMkernel ports are used for non-virtual machine traffic in VMware vSphere, and there are many different types of VMkernel ports, as you can see in the screenshot you took when configuring the network:
Each ESXi host has at least one VMkernel port for host management. If it is a member of the vSphere cluster, it will also have a VMkernel port for vMotion. If the cluster is a vSAN cluster, there will be a VMkernel port for vSAN.
If you are taking advantage of this feature, there will also be VMkernel ports for IP-based storage (such as iSCSI, NFS, or FCoE) and VMkernel ports for fault tolerance.
The biggest difference between a VIRTUAL machine port group and a VMkernel port group is the type of traffic it delivers. VMkernel ports are delivering VMware Vsphere-specific traffic, while a virtual machine port group just delivers your various virtual machines.
You can read more about VMkernel system traffic types in the official vSphere network documentation.
Each VMkernel port has its own unique IP address, and you can also change the default MTU 1500 on VMkernel ports.
9. VMware vSwitch security Settings
Now that we’ve covered most of the basics of vSphere networking, I’d like to dig a little deeper into some areas involving VMware switch configuration. VMware vSwitch security Settings are often misunderstood.
Each of these security Settings has two options: reject or accept.
By default, these security Settings are set to reject on the distributed virtual switch port group I just created, so let’s see what they really mean.
9.1 Promiscuous Mode VMware security policy
That sounds almost exactly what it means. If promiscuous mode is set to accept, the virtual machine’s nic accepts all frames sent to the active VLAN on the port group.
This is not safe, but it does have its use cases. If you are running a virtual machine that needs to check all packets, such as an intrusion detection system, you can set this option to accept.
By default, it is set to reject and should remain that way.
9.2 MAC Address Change VMware security policy
MAC address changes focus on inbound traffic to the virtual machine.
There are several different MAC addresses when it comes to the virtual machine’s network interface.
First, there is the MAC address specified in the VIRTUAL machine’s VMX file. Think of this as if your virtual machine has a physical NIC card. It is the MAC address of the NIC card specified by the manufacturer.
And the MAC address specified in the VIRTUAL machine, which is the same as the MAC address in the VMX file… Unless you need to change it for some reason.
You can change the MAC address of a VM. I have changed it in the past to ensure that the MAC address of the virtual machine is the same as the MAC address of the physical machine after P2V because the software license is associated with the MAC address.
In some valid use cases, these MAC addresses may be different, and you must set the MAC address change to accept.
Otherwise, set it to reject, because malicious actors can also wreak havoc on your environment with deceptive MAC addresses.
9.3. Forged Transmits VMware security policies
Forged Transmission also checks the MAC address of the virtual machine, but it handles outgoing traffic.
If the two MAC addresses do not match, it will lose frames, similar to a MAC address change.
Again, there are valid use cases for setting up bogus transports to accept, such as nested ESXi. In that case, we’ll be sending all kinds of crazy packets!
Forged transport is closely related to MAC address changes.
10. Private VLAN in VMware
When we talk about port groups, we also talk about private vlans.
Private VLANS are a more granular approach than isolating traffic at the VLAN level, which is why I brought it up after we started our security discussion.
Private vlans are dedicated to distributed virtual switches. The physical switches connected to your ESXi host must also support private vlans.
When we use PVLAN, the VLAN we use is decomposed in several ways:
- Master PVLAN, which is jumbled.
- Secondary Pvlans are of two types, Community and Isolated.
Let’s take a look at the figure below, where the Primary PVLAN is green and the secondary PVLAN is blue.
A VM port on the Isolated PVLAN can only communicate with the heterogeneous ports on the primary PVLAN, but cannot communicate with other ports on the isolated PVLAN.
Virtual machine ports on the community PVLAN and promiscuous ports on the main PVLAN can communicate with each other.
11. VMware vSwitch Security Summary
When it comes to security policy Settings for virtual switches, it is best to set them to reject unless you have a specific use case, and if you do have a reason to set promiscuous mode, change MAC addresses, or forge transport to accept, modify the security Settings accordingly.
Private vlans are another way to provide traffic segmentation in a virtual network environment, so be sure to pay special attention to the secondary PVLAN types to make sure things work as expected.
The good news is that the PVLAN Settings are easy to change for troubleshooting, as further described in the VMware vSphere networking documentation.
12. Basic knowledge of VMware vSphere network
Understanding the basics of VMware vSphere networking is important for VMware administrators, network administrators, IT security analysts, and almost anyone who needs access to a VMware vSphere environment. While many traditional networking concepts do apply to VMware vSphere, But it’s important to be familiar with how the virtualization layer transforms.