Encapsulation Mode

Transport Mode

With the transport mode, the AH or ESP header is inserted following the IP header, but before all transport layer protocol headers or all other IPsec protocol headers, as shown in Figure 1.

Figure 1 Format of a packet in transport mode

The transport mode does not change the IP packet header. Only the IP protocol field is changed to 51 (AH) or 50 (ESP), and the checksum of the IP packet header is re-calculated. The source and destination addresses of the IPsec tunnel must be the source and destination addresses in the IP packet header. Therefore, the transport mode is applicable only to communications between hosts.

In transport mode, AH verifies the entire IP packet during integrity verification. If the content of the IP packet is changed, the AH verification on the receiving end fails. Therefore, AH cannot coexist with the NAT protocol that changes the IP address in the IP packet header. ESP checks the integrity of the ESP header, transport layer protocol header, data, and ESP tail, instead of the IP packet header. Therefore, ESP cannot ensure the security of the IP packet header but can coexist with the NAT protocol. ESP encryption covers the transport layer protocol header, data, and ESP tail.

Tunnel Mode

Figure 2 shows the packet format in tunnel mode. In this mode, the original IP packet is encapsulated into a new IP packet, and an IPsec header (AH or ESP) is inserted between the old (IP Header in Figure 2) and new (New IP Header in Figure 2) headers. The original IP address is protected by IPsec as a part of the payload.

Figure 2 Format of a packet in tunnel mode

In tunnel mode, the original IP packet header is hidden. Therefore, the tunnel mode is mainly applicable to communications between VPN gateways or between a host and a VPN gateway.

In tunnel mode, the AH protocol checks the integrity of the whole IP packets including the new IP headers. The ESP protocol checks the integrity of ESP header, original IP header, transport layer protocol header, data, and ESP packet trailer, but do not the new IP header. Therefore, the ESP protocol cannot ensure the security of the new IP header. The ESP protocol encrypts the transport layer protocol header, data, and ESP packet trailer.

If both AH and ESP are used, the two protocols must use the same encapsulation mode.

Comparison Between the Transport Mode and the Tunnel Mode

Comparison between the transport mode and the tunnel mode is as follows:

• The tunnel mode is more secure than the transport mode. In tunnel mode, the original IP packet can be authenticated and encrypted completely. Besides, the internal IP address, protocol type, and port are hidden.

• The tunnel mode occupies more bandwidth because of an extra IP header.