Tunneling and Encryption Protocols

A VPN tunnel is built by specific protocols that define how traffic is encapsulated, encrypted, and authenticated. Different protocols suit different needs, balancing security, speed, and compatibility.

The two giants of VPN technology are IPsec and SSL/TLS. Understanding their roles helps you recognize how nearly every VPN you meet is actually built.

IPsec (Internet Protocol Security) is a suite of protocols that secures traffic at the network layer (Layer 3). It can encrypt and authenticate every packet between two endpoints, and is the backbone of most site-to-site VPNs.

Because it works at Layer 3, IPsec protects all traffic transparently, regardless of the application sending it, which makes it ideal for connecting whole networks.

IPsec uses several pieces:

• AH (Authentication Header): provides integrity and authentication but no encryption.
• ESP (Encapsulating Security Payload): provides encryption plus integrity, the part most used.
• IKE (Internet Key Exchange): negotiates keys and sets up the secure association.

ESP with IKE is the common combination that delivers full confidentiality and integrity.

IPsec runs in two modes. Transport mode encrypts only the packet payload, leaving the original IP header, used for host-to-host links. Tunnel mode encrypts the entire original packet and wraps it in a new one.

Tunnel mode is what site-to-site VPNs use, since it hides the internal addressing and protects the complete original packet between the two gateways.

SSL/TLS (Secure Sockets Layer / Transport Layer Security) VPNs secure traffic at a higher layer and often work right inside a web browser or a lightweight client. TLS is the same protocol that secures HTTPS websites.

SSL VPNs are popular for client remote access because they are easy to deploy, work through firewalls and NAT readily, and may need no special software beyond a browser.

The two excel at different jobs:

• IPsec: great for site-to-site and full-network access, but can struggle with NAT and firewalls and needs more configuration.
• SSL/TLS: great for client access to specific apps, browser-friendly, firewall-friendly.

Many organizations use IPsec between offices and SSL/TLS for remote users.

Newer protocols improve on the classics. WireGuard is a modern, fast, lightweight protocol with a small codebase that is easier to audit and very efficient.

OpenVPN is a flexible, widely used open-source option built on TLS. IKEv2/IPsec is favored on mobile devices because it reconnects quickly when the network changes, such as switching from Wi-Fi to cellular.

Some older VPN protocols are now insecure and should not be used. PPTP (Point-to-Point Tunneling Protocol) has known weaknesses and broken encryption. Plain L2TP (Layer 2 Tunneling Protocol) has no encryption on its own and must be paired with IPsec.

Recognizing outdated protocols matters: using them gives a false sense of security while leaving traffic exposed.

All these protocols rely on cryptography. They use symmetric encryption (one shared key) for bulk data because it is fast, and asymmetric encryption (key pairs) plus key-exchange algorithms to safely agree on that shared key.

Strong, modern ciphers and proper key exchange are what make a tunnel truly secure; weak or outdated cryptography undermines even a well-designed VPN.

VPN configuration centers on the endpoints, keys, and allowed traffic. This simplified snippet shows the kind of settings a WireGuard peer uses.

IPsec secures traffic at Layer 3 using ESP and IKE, with tunnel mode for site-to-site links. SSL/TLS VPNs secure client access and are browser- and firewall-friendly. Modern options like WireGuard, OpenVPN, and IKEv2 improve speed and mobility, while legacy PPTP should be avoided.

Test your VPN protocol knowledge.

IPsec secures traffic at the network layer with AH, ESP, and IKE, using tunnel mode for site-to-site VPNs. SSL/TLS VPNs suit browser-based client access. Modern protocols include WireGuard, OpenVPN, and IKEv2, while insecure legacy options like PPTP should be retired.