Centralized VXLAN Gateway Deployment Using BGP EVPN

In centralized VXLAN gateway deployment using BGP EVPN, the control plane is responsible for VXLAN tunnel establishment and dynamic MAC address learning; the forwarding plane is responsible for intra-subnet known unicast packet forwarding, intra-subnet broadcast, unknown-unicast, and multicast (BUM) packet forwarding, and inter-subnet packet forwarding. This deployment mode is flexible because EVPN allows dynamic VTEP discovery and VXLAN tunnel establishment, and is therefore applicable to large-scale networks. If centralized VXLAN gateway deployment is needed, using this mode is recommended.

The following VXLAN tunnel establishment uses an IPv4 over IPv4 network as an example. Table 1 shows the implementation differences between the other combinations of underlay and overlay networks and IPv4 over IPv4.

**Table 1** Implementation differences
Combination Category	Implementation Difference
IPv6 over IPv4	During dynamic MAC address learning, a Layer 2 gateway learns the local host's MAC address by the neighbor discovery function. Hosts at both ends learn each other's MAC addresses by exchanging neighbor solicitation (NS) or neighbor advertisement (NA) packets. In the inter-subnet interworking scenario, an IPv6 address must be configured for the Layer 3 gateway's VBDIF interface. During inter-subnet packet forwarding, the Layer 3 gateway needs to search its IPv6 routing table for the next-hop address of the destination IPv6 address, queries the ND table based on the next-hop address, and then obtains information such as the destination MAC address. During dynamic MAC address learning, a Layer 2 gateway learns the local host's MAC address using neighbor solicitation (NS) packets sent by the host.
IPv4 over IPv6	Not supported
IPv6 over IPv6	Not supported

VXLAN Tunnel Establishment

A VXLAN tunnel is identified by a pair of VTEP IP addresses. During VXLAN tunnel establishment, the local and remote VTEPs attempt to obtain the IP addresses of each other. A VXLAN tunnel can be established if the IP addresses obtained are reachable at Layer 3. When BGP EVPN is used to dynamically establish a VXLAN tunnel, the local and remote VTEPs first establish a BGP EVPN peer relationship and then exchange BGP EVPN routes to transmit VNIs and VTEPs' IP addresses.

On the network shown in Figure 1, VTEP 2 connects to Host 1 and Host 3; VTEP 3 connects to Host 2; VTEP 1 functions as a Layer 3 gateway. To allow Host 3 and Host 2 to communicate, establish a VXLAN tunnel between VTEP 2 and VTEP 3. To allow Host 1 and Host 2 to communicate, establish a VXLAN tunnel between VTEP 2 and VTEP 1 and between VTEP 1 and VTEP 3. Although Host 1 and Host 3 both connect to VTEP 2, they belong to different subnets and must communicate through the Layer 3 gateway (VTEP 1). Therefore, a VXLAN tunnel is also required between VTEP 2 and VTEP 1.

Figure 1 VXLAN tunnel networking

The following example illustrates how to use BGP EVPN to dynamically establish a VXLAN tunnel between VTEP 2 and VTEP 3.

Figure 2 Dynamic VXLAN tunnel establishment

VTEP 2 and VTEP 3 establish a BGP EVPN peer relationship. Then, local EVPN instances are created on VTEP 2 and VTEP 3, and an RD, export VPN targets (ERT), and import VPN targets (IRT) are configured for the EVPN instance. Layer 2 broadcast domains are created and bound to VNIs and EVPN instances. After the local VTEP's IP address is configured on VTEP 2 and VTEP 3, they generate a BGP EVPN route and send it to each other. The BGP EVPN route carries the local EVPN instance's export VPN target and an inclusive multicast route (Type 3 route defined in BGP EVPN). Figure 3 shows the format of an inclusive multicast route, which comprises a prefix and a PMSI attribute. VTEP IP addresses are stored in the Originating Router's IP Address field in the inclusive multicast route prefix, and VNIs are stored in the MPLS Label field in the PMSI attribute.

Figure 3 Format of an inclusive multicast route
After VTEP 2 and VTEP 3 receive a BGP EVPN route from each other, they match the export VPN targets of the route against the import VPN targets of the local EVPN instance. If a match is found, the route is accepted. If no match is found, the route is discarded. If the route is accepted, VTEP 2/VTEP 3 obtains the remote VTEP's IP address and VNI carried in the route. If the remote VTEP's IP address is reachable at Layer 3, a VXLAN tunnel to the remote VTEP is established. If the remote VNI is the same as the local VNI, an ingress replication list is created for subsequent BUM packet forwarding.

The processes for dynamic VXLAN tunnel establishment using BGP EVPN between VTEP 2 and VTEP 1 and between VTEP 1 and VTEP 3 are the same.

A VPN target is an extended community attribute of BGP for advertising VPN routes. An EVPN instance can have import and export VPN targets configured. The local EVPN instance's export VPN target must match the remote EVPN instance's import VPN target for EVPN route advertisement. If not, VXLAN tunnels cannot be dynamically established. If only one end can successfully accept the BGP EVPN route, this end can establish a VXLAN tunnel to the other end, but cannot exchange data packets with the other end. The other end drops packets after confirming that there is no VXLAN tunnel to the end that has sent these packets.

For details on VPN targets, see Basic Concepts of BGP/MPLS IP VPN in "BGP/MPLS IP VPN Configuration" in the S2720, S5700, and S6700 V200R019C10Configuration Guide - VPN.

Dynamic MAC Address Learning

VXLAN supports dynamic MAC address learning to allow communication between tenants. MAC address entries are dynamically created and do not need to be manually maintained, greatly reducing maintenance workload. The following example illustrates dynamic MAC address learning for intra-subnet communication on the network shown in Figure 4.

Figure 4 Dynamic MAC address learning

When Host 3 communicates with VTEP 2 for the first time, VTEP 2 learns the mapping between Host 3's MAC address, BDID (Layer 2 broadcast domain ID), and inbound interface (Port1) that has received the dynamic ARP packet and generates a MAC address entry for Host 3. The MAC address entry's outbound interface is Port1. VTEP 2 generates and sends a BGP EVPN route based on the ARP entry of Host 3 to VTEP 3. The BGP EVPN route carries the local EVPN instance's export VPN targets, Next_Hop attribute, and a Type 2 route (MAC/IP route) defined in BGP EVPN. The Next_Hop attribute carries the local VTEP's IP address. The MAC Address Length and MAC Address fields identify Host 3's MAC address. The Layer 2 VNI is stored in the MPLS Label1 field. Figure 5 shows the format of a MAC/IP route.

Figure 5 MAC/IP route
After VTEP 3 receives a BGP EVPN route from VTEP 2, VTEP 3 matches the export VPN targets of the route against the import VPN targets of the local EVPN instance. If a match is found, the route is accepted. If no match is found, the route is discarded. If the route is accepted, VTEP 3 obtains the mapping between Host 3's MAC address, BDID, VTEP 2's VTEP IP address (Next_Hop attribute) and generates a MAC address entry for Host 3. Based on the next hop, the MAC address entry's outbound interface is iterated to the VXLAN tunnel destined for VTEP 2.

VTEP 2 learns the MAC address of Host 2 in the same process.

When Host 3 communicates with Host 2 for the first time, Host 3 sends an ARP request for Host 2's MAC address. The ARP request carries the destination MAC address being all Fs and destination IP address being IP2. By default, VTEP 2 broadcasts the ARP request onto the network segment after receiving it. To reduce broadcast packets, ARP broadcast suppression can be enabled on VTEP 2. In the case ARP broadcast suppression is enabled and VTEP 2 receives the ARP request, VTEP 2 checks whether it has Host 2's MAC address based on the destination IP address of the ARP request. If VTEP 2 has Host 2's MAC address, it replaces the destination MAC address of the ARP request with Host 2's MAC address and unicasts the ARP request to VTEP 3 through the VXLAN tunnel. Upon receipt, VTEP 3 forwards the ARP request to Host 2, which then learns Host 3's MAC address and responds with an ARP reply in unicast mode. After Host 3 receives the ARP reply, it learns Host 2's MAC address. So far, Host 2 and Host 3 have learned the MAC address of each other, and will subsequently communicate with each other in unicast mode.

Dynamic MAC address learning is required only between hosts and Layer 3 gateways in inter-subnet communication scenarios. The process is the same as that for intra-subnet communication.
VTEP nodes can learn the MAC addresses of hosts during data forwarding, if this capability is enabled. If VXLAN tunnels are established using BGP EVPN, VTEP nodes can dynamically learn the MAC addresses of hosts through BGP EVPN routes, rather than data forwarding.

Intra-Subnet Known Unicast Packet Forwarding

Intra-subnet known unicast packets are forwarded only through Layer 2 VXLAN gateways and are unknown to Layer 3 VXLAN gateways. Figure 6 shows the intra-subnet known unicast packet forwarding process.

Figure 6 Intra-subnet known unicast packet forwarding

After VTEP 2 receives Host 3's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information and searches for the outbound interface and encapsulation information in the BD.
VTEP 2 performs VXLAN encapsulation based on the encapsulation information obtained and forwards the packets through the outbound interface obtained.
Upon receipt of the VXLAN packet, VTEP 3 verifies the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. VTEP 3 obtains the Layer 2 BD based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
VTEP 3 obtains the destination MAC address of the inner Layer 2 packet, performs VLAN tags to the packets based on the outbound interface and encapsulation information in the local MAC address table, and forwards the packets to Host 2.

Host 2 sends packets to Host 3 in the same manner.

Intra-Subnet BUM Packet Forwarding

Intra-subnet BUM packet forwarding is completed between Layer 2 VXLAN gateways. Layer 3 VXLAN gateways do not need to be unaware of the process. Intra-subnet BUM packets can be forwarded in ingress replication mode.

In ingress replication mode, after a BUM packet enters a VXLAN tunnel, the ingress VTEP performs VXLAN encapsulation based on the ingress replication list and sends the packet to all the egress VTEPs in the list. When the BUM packet leaves the VXLAN tunnel, the egress VTEPs decapsulate the BUM packet. Figure 7 shows the forwarding process of a BUM packet in ingress replication mode.

Figure 7 Forwarding process of an intra-subnet BUM packet in ingress replication mode

After VTEP 1 receives Terminal A's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information.
VTEP 1 obtains the ingress replication list for the VNI, replicates packets based on the list, and performs VXLAN encapsulation by adding outer headers. VTEP 1 then forwards the VXLAN packet through the outbound interface.
Upon receipt of the VXLAN packet, VTEP 2 and VTEP 3 verify the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. VTEP 2/VTEP 3 obtains the Layer 2 BD based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
VTEP 2/VTEP 3 checks the destination MAC address of the inner Layer 2 packet and finds it a BUM MAC address. Therefore, VTEP 2/VTEP 3 broadcasts the packet onto the network connected to the terminals (not the VXLAN tunnel side) in the Layer 2 broadcast domain. Specifically, VTEP 2/VTEP 3 finds the outbound interfaces and encapsulation information not related to the VXLAN tunnel, performs VLAN tags to the packet, and forwards the packet to Terminal B/Terminal C.

Terminal B/Terminal C responds to Terminal A in the same process as intra-subnet known unicast packet forwarding.

Inter-Subnet Packet Forwarding

Inter-subnet packets must be forwarded through a Layer 3 gateway. Figure 8 shows the inter-subnet packet forwarding process.

Figure 8 Inter-subnet packet forwarding

After VTEP 2 receives Host 1's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information and searches for the outbound interface and encapsulation information in the BD.
VTEP 2 performs VXLAN encapsulation based on the outbound interface and encapsulation information and forwards the packets to VTEP 1.
After VTEP 1 receives the VXLAN packet, it decapsulates the packet and finds that the destination MAC address of the inner packet is the MAC address (MAC3) of the Layer 3 gateway interface (VBDIF10) so that the packet must be forwarded at Layer 3.
VTEP 1 removes the inner Ethernet header, parses the destination IP address, and searches the routing table for a next hop address. VTEP 1 then searches the ARP table based on the next hop address to obtain the destination MAC address, VXLAN tunnel's outbound interface, and VNI.
VTEP 1 performs VXLAN encapsulation on the inner packet again and forwards the VXLAN packet to VTEP 3, with the source MAC address in the inner Ethernet header being the MAC address (MAC4) of the Layer 3 gateway interface (VBDIF20).
Upon receipt of the VXLAN packet, VTEP 3 verifies the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. VTEP 3 then obtains the Layer 2 broadcast domain based on the VNI and removes the outer headers to obtain the inner Layer 2 packet. It then searches for the outbound interface and encapsulation information in the Layer 2 broadcast domain.
VTEP 3 performs VLAN tags to the packets based on the outbound interface and encapsulation information and forwards the packets to Host 2.

Host 2 sends packets to Host 1 in the same manner.