Summary
Legacy encryption standards can no longer meet the security demands of today’s VPN encryption protocols. In this guide, cybersecurity expert Matt Tadevich breaks down the cryptographic principles and configuration steps organizations need to strengthen VPN encryption. Learn how to align your policies with modern best practices, including IKEv2, AES-GCM-256, and CNSA-compliant ciphers built for a quantum-resilient future.
[Estimated read time: 7-9 minutes]
A few years ago, I was heavily involved in performing firewall refreshes and firewall migrations, often moving from legacy firewalls to next generation firewall (NGFW) platforms. A critical task during that process is modernizing VPN encryption.
Legacy ciphers like DES, 3DES, or deprecated Diffie-Hellman (DH) groups (e.g., Group 1 or 2) are no longer secure and must be replaced with robust, modern cryptographic standards. This guide demystifies VPN encryption details, explaining the “what” and “why” behind choosing between AES-256 and DH Group 21, avoiding SHA-1, and clarifying the differences between IKE Phase 1 and Phase 2.
VPN encryption and IKE protocols
VPN encryption enables site-to-site VPNs to establish secure, persistent tunnels between two network infrastructure devices (e.g., firewalls or routers), allowing remote networks to communicate in cases like County A accessing City B’s dispatching servers. These encrypted tunnels typically rely on the IP security (IPSec) protocol suite, which uses Internet Key Exchange (IKE) to manage cryptographic keys and session parameters.
IKE operates in two phases:
- Phase 1: Establishes a secure channel for key exchange (IKE Security Association, or IKE_SA).
- Phase 2: Sets up the IPSec tunnel for data encryption (CHILD_SA).
Two versions of IKE exist: IKEv1 and IKEv2. IKEv2 is preferred for modern deployments due to its streamlined design, improved security, and support for advanced cryptographic algorithms. IKEv2 was standardized in RFC 5996 (2010) and has been increasingly adopted for its robustness.
IKEv2 Overview
IKEv2 is more secure and efficient than IKEv1, supporting modern cryptographic tools. It operates in two phases:
Phase 1 (IKE_SA)
Phase 1 creates a secure communication channel for exchanging encapsulating security payload (ESP) or authentication header (AH) packets. Both VPN endpoints must agree on the following attributes:
- Encryption Algorithm: AES-GCM-256, preferred for integrated encryption and authentication
- Hashing (Integrity): SHA-384 or SHA-512; NULL when using GCM-based encryption
- Diffie-Hellman (DH) Group: Groups 19, 20, or 21, elliptic-curve-based for quantum resistance
- Pseudo-Random Function (PRF): SHA-384, for key derivation
- Authentication Method: Pre-shared key (PSK), digital certificates, or elliptic curve digital signatures (ECDSA)
Phase 2 (CHILD_SA)
Phase 2 establishes the IPSec tunnel for data transfer, typically using ESP. Key attributes include:
- ESP Encryption: AES-GCM-256
- ESP Hash (Integrity): NULL when using GCM, as it provides built-in integrity
- Perfect Forward Secrecy (PFS): Enabled with DH Group 19, 20, or 21 for session-specific key exchanges, ensuring compromised session keys don’t affect other sessions
Cryptography Best Practices
Modern VPN cryptography has evolved significantly, driven by advancements like the Commercial National Security Algorithm (CNSA) suite which was developed by the NSA to replace Suite B. CNSA is designed for high-security environments and aligns with quantum-resistant standards. As of 2025, NIST continues to refine quantum-resistant algorithms (NIST, 2022).
Recommended CNSA Cipher Suite
- Encryption: AES-GCM-256, provides both confidentiality and integrity
- Hashing: NULL when using GCM; otherwise, SHA-384 or SHA-512
- Key Exchange: Elliptic Curve Diffie-Hellman (ECDH) with P-256, P-384, or P-521 curves from DH Groups 19, 20, 21
Fallback Options
If a VPN termination point doesn’t support CNSA ciphers, use the next strongest available:
- Encryption: AES-256 in CBC mode, if GCM is unavailable
- Key Exchange: DH Group 14 or higher, with preference for the highest supported group
Practices to Avoid
- Deprecated Ciphers: DES, 3DES, and AES-128 are all vulnerable to brute-force attacks.
- Weak Hashing: MD5 and SHA-1 are susceptible to collision attacks.
- Weak DH Groups: Groups 1, 2, and 5 have insufficient key strength.
Platform-Agnostic Configuration Guidance
Site-to-site VPN configuration, including VPN encryption protocols, varies by platform (e.g., Cisco Secure Firewall, Palo Alto, Fortinet, SonicWall), but the steps below outline a general approach for IKEv2 with CNSA-compliant settings:
- Define VPN Parameters:
- Specify the protected networks (subnets) and tunnel endpoints (public IPs) for both sides.
- Ensure both endpoints use matching IKEv2 policies.
2. Configure IKEv2 Phase 1 (IKE_SA):
- Encryption: Set to AES-GCM-256.
- Integrity (Hash): Set to NULL for GCM or SHA-384.
- DH Group: Set to Group 21, or 19/20 if 21 is unsupported.
- Authentication: Use a strong, randomly generated pre-shared key (PSK, minimum 16 characters) or digital certificates for enhanced security.
3. Configure IKEv2 Phase 2 (IPSec Proposal):
- ESP Encryption: Set to AES-GCM-256.
- ESP Hash: Set to NULL for GCM.
- PFS: Enable with DH Group 21, or 19/20 if necessary.
4. Test and Validate:
- Verify tunnel establishment and test connectivity between networks.
- Check logs for errors related to mismatched ciphers or authentication failures.
Conclusion
VPN encryption protocols have evolved significantly over the past two decades. The complexity arises from the wide range of cipher options, but the CNSA suite simplifies standardization with a strong, quantum-resistant cryptographic posture. For further questions or to connect with peers, feel free to reach out!
Sources
