IPsec, VPN Protocols, and Remote Access Security

A Virtual Private Network (VPN) creates an encrypted tunnel over a public or untrusted network, allowing remote users and sites to communicate as if they were on the same private network. VPNs protect against eavesdropping on untrusted networks (hotel Wi-Fi, public hotspots), allow remote employees to access internal resources, and connect branch offices securely over the internet. The Security+ exam tests multiple VPN technologies — IPsec, SSL-VPN, and emerging protocols like WireGuard — along with the security considerations that apply to each.

IPsec (Internet Protocol Security) is a suite of protocols that secures IP traffic at Layer 3. IPsec operates in two modes: Transport mode encrypts only the payload of each IP packet (the original IP header is preserved — used for end-to-end host communication) and Tunnel mode encrypts the entire original packet and wraps it in a new IP header — used for site-to-site VPNs where gateways act as tunnel endpoints. IPsec uses two main protocols: AH (Authentication Header) for integrity and authentication only (no encryption), and ESP (Encapsulating Security Payload) for both encryption and authentication.

IKE (Internet Key Exchange) negotiates the security parameters (algorithms, keys) for an IPsec session. IKEv1 used two phases: Phase 1 (establish a secure channel to protect Phase 2 negotiation) and Phase 2 (negotiate the actual IPsec Security Associations). IKEv2 (current standard) is simpler, faster, more reliable, and adds built-in support for EAP authentication and MOBIKE (handling IP address changes mid-session — important for mobile devices). IPsec Security Associations (SAs) are unidirectional; each direction has its own SA with its own encryption key.

IPsec VPN operates at Layer 3, requiring client software and often needing specific firewall port openings (UDP 500, UDP 4500 for NAT traversal, ESP Protocol 50). It is harder to deploy but provides full network-layer access. SSL-VPN tunnels traffic over HTTPS (TCP 443), which passes through almost every firewall and can work from browsers (clientless mode) or thin clients. SSL-VPN uses TLS for the outer tunnel, so it benefits from TLS security improvements and is easier for users to connect with. Enterprise SSL-VPN products (Cisco AnyConnect, Palo Alto GlobalProtect) also support split tunneling and endpoint posture assessment.

Split tunneling allows VPN clients to route only corporate-bound traffic through the VPN tunnel while sending internet-bound traffic directly to the internet without passing through corporate security controls. The security risk: a compromised endpoint can simultaneously access the corporate network (via VPN) and communicate with malware C2 servers (direct internet) — effectively bridging the corporate network to the attacker. Full tunneling routes all traffic through the VPN gateway, where corporate security controls (web filtering, DLP, IPS) inspect it. Most security-conscious organizations enforce full tunneling, accepting the bandwidth overhead.

WireGuard is a modern VPN protocol designed to be simpler, faster, and more secure than IPsec and OpenVPN. WireGuard uses a fixed, minimal cryptographic stack: ChaCha20 for symmetric encryption, Poly1305 for authentication, Curve25519 for ECDH key exchange, BLAKE2s for hashing, and SipHash24 for hash table keys. Its code base is under 4,000 lines (versus 100,000+ for OpenVPN), making it auditable and reducing the attack surface. WireGuard is now built into the Linux kernel (5.6+) and is supported on all major platforms.

Site-to-site VPNs connect entire networks (branch offices, data centers, cloud VPCs) through persistent encrypted tunnels between gateway devices. Unlike remote-access VPNs where individual users connect, site-to-site tunnels are always-on between fixed endpoints. IPsec in tunnel mode is the most common technology. Configuration requires matching IKE proposals (encryption, integrity, DH group, lifetime) on both ends — mismatches are the most common site-to-site troubleshooting issue. Redundant tunnels with dead peer detection (DPD) provide failover when the primary path fails.

VPNs support multiple authentication mechanisms. Pre-shared keys (PSK): a shared secret configured on both endpoints — simple but creates a single point of compromise if the key is leaked; suitable for site-to-site. Certificate-based authentication: each client or gateway presents an X.509 certificate — stronger, scales to many users, certificates can be revoked. EAP (Extensible Authentication Protocol): supports RADIUS integration, enabling username/password, MFA (EAP-TTLS, EAP-TLS), and integration with Active Directory. Most enterprise deployments combine certificate authentication (device identity) with MFA (user identity).

Modern VPN gateways perform endpoint posture checks before granting access, ensuring that connecting devices meet security requirements. Checks include: OS version and patch level (reject end-of-life OS), antivirus installation and signature currency, disk encryption enabled, no unauthorized software installed, and firewall active. Devices that fail posture checks are placed in a quarantine VLAN with limited access — typically only remediation resources — until they meet policy. This prevents a compromised personal device from directly accessing the production corporate network, even with valid credentials.

Always-on VPN ensures that corporate endpoints maintain a persistent VPN connection whenever they are off the corporate network. Unlike traditional VPNs where users manually connect, always-on VPN connects before login (sometimes using machine certificate authentication) and remains connected throughout the session. This ensures all endpoint traffic is inspected by corporate security controls even for remote employees. Microsoft DirectAccess (predecessor) and solutions like Cisco AnyConnect Always-On and GlobalProtect implement this model. The tradeoff is increased gateway load — every remote endpoint maintains a permanent session.

A VPN concentrator is a dedicated appliance or virtual device that terminates large numbers of VPN sessions, handling the cryptographic processing load that would overwhelm a general-purpose router or firewall. Enterprise VPN concentrators support thousands of simultaneous sessions and integrate with RADIUS for authentication, LDAP/AD for user lookups, and certificate authorities for client certificate validation. High availability (HA) configurations use active/standby or active/active clusters with session synchronization so that users are not disconnected if one concentrator fails. Proper HA planning is essential for organizations that depend on VPN for critical operations.

Test your understanding of CompTIA Security+ (SY0-701) concepts from this lesson.

In this lesson you learned: IPsec secures IP traffic at Layer 3 using AH (integrity only) or ESP (encryption + integrity) in transport or tunnel mode with IKEv2 for key negotiation, SSL-VPN tunnels over HTTPS making it easier to deploy through firewalls compared to IPsec, and split tunneling creates a security risk by allowing malware on endpoints to reach the internet directly while the VPN provides corporate network access simultaneously. This completes the Secure Protocols course — next up we explore Zero Trust Architecture.