IPsec: IKEv2, ESP, and AH Protocols

IPsec is not a single protocol but a suite of standards for authenticating and encrypting IP traffic. The main components are IKE (Internet Key Exchange) for establishing and managing Security Associations, ESP (Encapsulating Security Payload) for encryption and integrity, and AH (Authentication Header) for integrity without encryption. IPsec operates at the network layer, protecting all traffic without changes to applications.

IKEv2 (RFC 7296) replaced IKEv1 with a simpler, more efficient design. IKEv1 required multiple modes and message exchanges; IKEv2 uses a single unified exchange model. IKEv2 also includes built-in NAT traversal, EAP support for user authentication, MOBIKE for mobile IP address changes, and stronger security defaults. Most modern VPN implementations use IKEv2.

The IKE_SA_INIT exchange establishes the IKE Security Association. The initiator sends supported algorithms, Diffie-Hellman key share, and a nonce. The responder selects algorithms, sends its DH key share and nonce. Both parties compute the same shared secret and derive keys for the IKE SA. This exchange protects the subsequent IKE_AUTH phase.

The IKE_AUTH exchange authenticates the peers and creates the first Child SA for actual traffic. Authentication can use X.509 certificates (most secure), pre-shared keys (PSK), or EAP methods for user-level authentication. The exchange also negotiates the Child SA parameters: the traffic selectors (which IP ranges to protect), encryption algorithm, and integrity algorithm.

ESP (Encapsulating Security Payload) is the workhorse of IPsec traffic protection. In tunnel mode, ESP encapsulates the entire original IP packet inside a new IP packet with an ESP header, providing both encryption and integrity. In transport mode, ESP protects only the payload of the original IP packet, preserving the original IP header. AES-256-GCM is the recommended algorithm for ESP in modern deployments.

AH (Authentication Header) provides integrity and authentication without encryption. It covers the original IP header (unlike ESP), but this makes AH incompatible with NAT, which modifies IP header fields. Because AH provides no confidentiality and breaks NAT traversal, it is rarely used in practice. ESP with strong AEAD algorithms provides better protection than AH plus encryption separately.

In transport mode, IPsec protects the payload between two endpoints but leaves IP headers visible. This is used for host-to-host communication where routing information need not be hidden. In tunnel mode, the entire original IP packet is encapsulated inside a new packet, hiding source and destination from observers. VPN gateways always use tunnel mode to connect networks or users to networks.

Standard ESP cannot pass through NAT because NAT modifies IP headers and port numbers that ESP integrity covers. IPsec solves this with NAT-T (NAT Traversal): it detects NAT in the IKE exchange and encapsulates ESP packets inside UDP port 4500. This UDP wrapper allows NAT to translate addresses without breaking ESP integrity verification.

IPsec maintains a Security Association Database (SAD) that stores all active SAs with their keys, algorithms, and lifetimes. The Security Policy Database (SPD) defines which traffic should be protected, bypassed, or discarded. When outbound traffic matches an SPD entry, the kernel looks up or creates the appropriate SA in the SAD and applies the specified protection.

EAP (Extensible Authentication Protocol) in IKEv2 enables user-level authentication methods beyond certificates and PSK. EAP-TLS uses client certificates, EAP-MSCHAPv2 uses username and password (common in Windows clients), and EAP-TTLS tunnels inner methods inside a TLS session. Many enterprise IPsec VPNs use certificate authentication for the gateway and EAP for individual users.

Security Associations have lifetime limits in bytes and time. When a Child SA approaches its lifetime, IKEv2 automatically creates a new SA with fresh keys through a CREATE_CHILD_SA exchange. This rekeying provides forward secrecy: compromise of the current key does not expose past traffic encrypted under previous keys. IKE SA lifetimes are typically longer than Child SA lifetimes.

Which IPsec mode encapsulates the entire original IP packet inside a new packet, hiding the original source and destination addresses?

IPsec recap: IKEv2 establishes Security Associations via IKE_SA_INIT (DH key exchange) and IKE_AUTH (peer authentication), ESP provides encrypted and authenticated tunnel mode for traffic, AH is rarely used due to NAT incompatibility, NAT-T wraps ESP in UDP for NAT traversal, and automatic rekeying provides forward secrecy throughout the VPN session lifetime.