Centralized VXLAN Gateway Deployment in Static Mode

In centralized VXLAN gateway deployment in static mode, the control plane is responsible for VXLAN tunnel establishment and dynamic MAC address learning; the forwarding plane is responsible for intra-subnet known unicast packet forwarding, intra-subnet BUM (Broadcast&Unknown-unicast&Multicast) packet forwarding, and inter-subnet packet forwarding.

Deploying centralized VXLAN gateways in static mode involves heavy workload and is inflexible, and therefore is inapplicable to large-scale networks. As such, deploying centralized VXLAN gateways using BGP EVPN is recommended.

The following VXLAN tunnel establishment uses an IPv4 over IPv4 network as an example. Table 1 shows the implementation differences between the other combinations of underlay and overlay networks and IPv4 over IPv4.

**Table 1** Implementation differences
Combination Category	Implementation Difference
IPv6 over IPv4	During dynamic MAC address learning, a Layer 2 gateway learns the local host's MAC address using neighbor solicitation (NS) packets sent by the host. In the inter-subnet interworking scenario, an IPv6 address must be configured for the Layer 3 gateway's VBDIF interface. During inter-subnet packet forwarding, the Layer 3 gateway needs to search its IPv6 routing table for the next-hop address of the destination IPv6 address, queries the ND table based on the next-hop address, and then obtains information such as the destination MAC address.
IPv4 over IPv6	The VTEPs at both ends of a VXLAN tunnel use IPv6 addresses, and IPv6 Layer 3 route reachability must be implemented between the VTEPs.
IPv6 over IPv6	The VTEPs at both ends of a VXLAN tunnel use IPv6 addresses, and IPv6 Layer 3 route reachability must be implemented between the VTEPs. During dynamic MAC address learning, a Layer 2 gateway learns the local host's MAC address using NS packets sent by the host. In the inter-subnet interworking scenario, an IPv6 address must be configured for the Layer 3 gateway's VBDIF interface. During inter-subnet packet forwarding, the Layer 3 gateway needs to search its IPv6 routing table for the next-hop address of the destination IPv6 address, queries the ND table based on the next-hop address, and then obtains information such as the destination MAC address.

VXLAN Tunnel Establishment

A VXLAN tunnel is identified by a pair of VTEP IP addresses. A VXLAN tunnel can be statically created after you configure local and remote VNIs, VTEP IP addresses, and an ingress replication list, and the tunnel goes Up when the pair of VTEPs are reachable at Layer 3.

On the network shown in Figure 1, VTEP 2 connects to Host 1 and Host 3; VTEP 3 connects to Host 2; VTEP 1 functions as a Layer 3 gateway.

To allow Host 3 and Host 2 to communicate, Layer 2 VNIs and an ingress replication list must be configured on VTEP 2 and VTEP 3. The peer VTEPs' IP addresses must be specified in the ingress replication list. A VXLAN tunnel can be established between VTEP 2 and VTEP 3 if their VTEPs have Layer 3 routes to each other.
To allow Host 1 and Host 2 to communicate, Layer 2 VNIs and an ingress replication list must be configured on VTEP 2, VTEP 3, and also VTEP 1. The peer VTEPs' IP addresses must be specified in the ingress replication list. A VXLAN tunnel can be established between VTEP 2 and VTEP 1 and between VTEP 3 and VTEP 1 if they have Layer 3 routes to the IP addresses of the VTEPs of each other.

Although Host 1 and Host 3 both connect to VTEP 2, they belong to different subnets and must communicate through the Layer 3 gateway (VTEP 1). Therefore, a VXLAN tunnel is also required between VTEP 2 and VTEP 1.

Figure 1 VXLAN tunnel networking

Dynamic MAC Address Learning

VXLAN supports dynamic MAC address learning to allow communication between tenants. MAC address entries are dynamically created and do not need to be manually maintained, greatly reducing maintenance workload. The following example illustrates dynamic MAC address learning for intra-subnet communication on the network shown in Figure 2.

Figure 2 Dynamic MAC Address Learning

Host 3 sends an ARP request for Host 2's MAC address. The ARP request carries the source MAC address being MAC3, destination MAC address being all Fs, source IP address being IP3, and destination IP address being IP2.
Upon receipt of the ARP request, VTEP 2 determines that the Port1 receiving the ARP request belongs to a BD that has been bound to a VNI (20), meaning that the ARP request packet must be transmitted over the VXLAN tunnel identified by VNI 20. VTEP 2 then learns the mapping between Host 3's MAC address, BDID (Layer 2 broadcast domain ID), and inbound interface (Port1) that has received the ARP request and generates a MAC address entry for Host 3. The MAC address entry's outbound interface is Port1.
VTEP 2 then performs VXLAN encapsulation on the ARP request, with the VNI being the one bound to the BD, source IP address in the outer IP header being the VTEP's IP address of VTEP 2, destination IP address in the outer IP header being the VTEP's IP address of VTEP 3, source MAC address in the outer Ethernet header being MAC address of VTEP 2, and destination MAC address in the outer Ethernet header being the MAC address of the next hop pointing to the destination IP address. Figure 3 shows the VXLAN packet format. The VXLAN packet is then transmitted over the IP network based on the IP and MAC addresses in the outer headers and finally reaches VTEP 3.

Figure 3 VXLAN packet format
After VTEP 3 receives the VXLAN packet, it decapsulates the packet and obtains the ARP request originated from Host 3. VTEP 3 then learns the mapping between Host 3's MAC address, BDID, and VTEP's IP address of VTEP 2 and generates a MAC address entry for Host 3. Based on the next hop (VTEP's IP address of VTEP 2), the MAC address entry's outbound interface is iterated to the VXLAN tunnel destined for VTEP 2.
VTEP 3 broadcasts the ARP request in the Layer 2 domain. Upon receipt of the ARP request, Host 2 finds that the destination IP address is its own IP address and saves Host 3's MAC address to the local MAC address table. Host 2 then responds with an ARP reply.

So far, Host 2 has learned Host 3's MAC address. Therefore, Host 2 responds with a unicast ARP reply. The ARP reply is transmitted to Host 3 in the same manner. After Host 2 and Host 3 learn the MAC address of each other, they will subsequently communicate with each other in unicast mode.

Dynamic MAC address learning is required only between hosts and Layer 3 gateways in inter-subnet communication scenarios. The process is the same as that for intra-subnet communication.

Intra-Subnet Known Unicast Packet Forwarding

Intra-subnet known unicast packets are forwarded only through Layer 2 VXLAN gateways and are unknown to Layer 3 VXLAN gateways. Figure 4 shows the intra-subnet known unicast packet forwarding process.

Figure 4 Intra-subnet known unicast packet forwarding

After VTEP 2 receives Host 3's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information and searches for the outbound interface and encapsulation information in the BD.
VTEP 2 performs VXLAN encapsulation based on the encapsulation information obtained and forwards the packets through the outbound interface obtained.
Upon receipt of the VXLAN packet, VTEP 3 verifies the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. VTEP 3 obtains the Layer 2 BD based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
VTEP 3 obtains the destination MAC address of the inner Layer 2 packet, performs VLAN tags to the packets based on the outbound interface and encapsulation information in the local MAC address table, and forwards the packets to Host 2.

Host 2 sends packets to Host 3 in the same manner.

Intra-Subnet BUM Packet Forwarding

Intra-subnet BUM packet forwarding is completed between Layer 2 VXLAN gateways. Layer 3 VXLAN gateways do not need to be unaware of the process. Intra-subnet BUM packets can be forwarded in ingress replication mode.

In ingress replication mode, after a BUM packet enters a VXLAN tunnel, the ingress VTEP performs VXLAN encapsulation based on the ingress replication list and sends the packet to all the egress VTEPs in the list. When the BUM packet leaves the VXLAN tunnel, the egress VTEPs decapsulate the BUM packet. Figure 5 shows the forwarding process of a BUM packet in ingress replication mode.

Figure 5 Forwarding process of an intra-subnet BUM packet in ingress replication mode

After VTEP 1 receives Terminal A's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information.
VTEP 1 obtains the ingress replication list for the VNI, replicates packets based on the list, and performs VXLAN encapsulation by adding outer headers. VTEP 1 then forwards the VXLAN packet through the outbound interface.
Upon receipt of the VXLAN packet, VTEP 2's VTEP and VTEP 3's VTEP verify the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. VTEP 2/VTEP 3 obtains the Layer 2 BD based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
VTEP 2/VTEP 3 checks the destination MAC address of the inner Layer 2 packet and finds it a BUM MAC address. Therefore, VTEP 2/VTEP 3 broadcasts the packet onto the network connected to the terminals (not the VXLAN tunnel side) in the Layer 2 broadcast domain. Specifically, VTEP 2/VTEP 3 finds the outbound interfaces and encapsulation information not related to the VXLAN tunnel, performs VLAN tags to the packet, and forwards the packet to Terminal B/Terminal C.

Terminal B/Terminal C responds to Terminal A in the same process as intra-subnet known unicast packet forwarding.

Inter-Subnet Packet Forwarding

Inter-subnet packets must be forwarded through a Layer 3 gateway. Figure 6 shows inter-subnet packet forwarding in centralized VXLAN gateway scenarios.

Figure 6 Inter-subnet packet forwarding

After VTEP 2 receives Host 1's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information and searches for the outbound interface and encapsulation information in the BD.
VTEP 2 performs VXLAN encapsulation based on the outbound interface and encapsulation information and forwards the packets to VTEP 1.
After VTEP 1 receives the VXLAN packet, it decapsulates the packet and finds that the destination MAC address of the inner packet is the MAC address (MAC3) of the Layer 3 gateway interface (VBDIF10) so that the packet must be forwarded at Layer 3.
VTEP 1 removes the inner Ethernet header, parses the destination IP address, and searches the routing table for a next hop address. VTEP 1 then searches the ARP table based on the next hop address to obtain the destination MAC address, VXLAN tunnel's outbound interface, and VNI.
VTEP 1 performs VXLAN encapsulation on the inner packet again and forwards the VXLAN packet to VTEP 3, with the source MAC address in the inner Ethernet header being the MAC address (MAC4) of the Layer 3 gateway interface (VBDIF20).
Upon receipt of the VXLAN packet, VTEP 3 verifies the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. VTEP 3 then obtains the Layer 2 broadcast domain based on the VNI and removes the outer headers to obtain the inner Layer 2 packet. It then searches for the outbound interface and encapsulation information in the Layer 2 broadcast domain.
VTEP 3 performs VLAN tags to the packets based on the outbound interface and encapsulation information and forwards the packets to Host 2.

Host 2 sends packets to Host 1 in the same manner.