Encapsulation Modes
Encapsulation is a process of adding AH or ESP fields to original IP packets for packet authentication and encryption. This process is implemented in either transport or tunnel mode.
Efficient VPN supports only the tunnel mode.
Transport Mode
IPSec transport mode works by inserting an AH or ESP header between an IP header and a transport-layer protocol header to protect the TCP, UDP, or ICMP payload. Because no additional IP header is added, IP addresses in the original packets are visible in the IP header of the post-encrypted packet. Figure 1 shows an example of TCP packet encapsulation in transport mode.
In transport mode, AH protects the IP header, but ESP does not.
Tunnel Mode
IPSec tunnel mode works by encrypting and authenticating an entire IP packet, including the IP header and payload. In this mode, an AH or ESP header is added before the raw IP header, and a new IP header is added before the AH or ESP header. Figure 2 shows an example of TCP packet encapsulation in tunnel mode.
In tunnel mode, AH protects the new IP header, but ESP does not.
Comparisons Between the Transport Mode and Tunnel Mode
The main differences between the transport mode and tunnel mode are as follows:
The tunnel mode is more secure because original IP packets are completely authenticated and encrypted. This mode hides the IP address, protocol type, and port number in an original IP packet.
The tunnel mode generates an additional IP header, occupying more bandwidth than the transport mode.
The transport mode is mainly used for communication between two hosts or between a host and a VPN gateway. The tunnel mode is mainly used for communication between two VPN gateways or between a host and a VPN gateway.
When both AH and ESP are used to protect traffic, they must use the same encapsulation mode.