UsMan's WoRkSpAce

Tuesday, June 17, 2008

IPSec technical details

IPSec tunnel consists of a pair of unidirectional security associations (SA) (one at each end). SA specify security parameter index (SPI), destination IP and security protocol (AH or ESP). VPN traffic can be authenticated, encrypted, integrity protected or all of them. Key creation can be manual or via AutoKey IKE with a pre-shared key or a certificate. IPSec works at IP packet layer. It consists of transport and tunnel modes and two protocols, AH & ESP for authentication and encryption. When both end of a tunnel are hosts, both transport or tunnel mode can be used. Tunnel mode must be used when at least one of endpoint is a gateway such as firewall or router. Netscreen operates in tunnel mode for IPSec tunnels and transport mode for L2TP-over-IPsec tunnels.

Transport mode: Original IP packet is not encapsulated within another IP packet. Entire packet is authenticated with AH, payload can be encrypted with ESP and original header remains in plaintext across the WAN/internet.
Tunnel mode: Entire original IP packet (payload + header) is encapsulated within another IP payload and a new header appended to it. Entire original packet can be encrypted, authenticated or both. With AH the entire packet including AH and new headers can also be authenticated. In site-to-site VPN, source and destination addresses of new header are IP address of outgoing interface (NAT/route mode) or VLAN1 IP (transparent mode). Source and destination IPs within encapsulated packets are those of ultimate endpoints.

Authentication Header (AH): Authenticates source of a packet and verifies integrity of its content. Checksum is generated via a hash-based message authentication code (HMAC) using a secret key with either MD5 or SHA1 hash functions. SHA1 is considered more secure than MD5 because of its larger hashes (160 bit to 128 bit) and larger security key (20 byte to 16 byte)
Encapsulating Security Payload (ESP): Encrypts the entire IP packet and authenticates its content. ESP can be used to authenticate or encrypt or both at the same time. Supported encryption algorithms are DES (block algorithm), Triple DES, AES
Key management: Distribution and management of keys are critical to any VPN solution. IPSec supports manual and automatic key distribution methods. With manual keys, administrators manually configure parameters at each end of tunnel. There is no SA negotiation as all SA parameters are already defined. Autokey IKE supports automated generation and negotiation of keys and security associations. Autokey IKE works with preshared keys and certificates. IKE can use preshared keys to authenticate participant in an IKE session. IKE changes key frequently at pre-determined intervals. There are two phases of negotiation for IKE. In phase 1, participants establish a secure channel to negotiate SAs. In phase 2, participants negotiate SAs for encryption and/or authentication. Proposal exchange in phase 1 can be done by aggressive or main modes. In main mode, both parties send three 2-way exchanges (total six messages). First exchange, propose and accept encryption and authentication algorithm. Second exchange is about diffie-hellman and parties present identities in third exchange. Aggressive mode works in only two exchanges and a total of three messages. Aggressive mode does not provide identify authentication as they are exchanged in clear text. Aggressive mode must be used for dial-up VPN. Diffie-hellman produces a shared secret value. It is created without sending the secret over the network. Larger modulus create more secure key. SAs are negotiated in phase 2 to secure the data through the tunnel. It works like phase 1 negotiation in the sense that there are proposals. Phase 2 proposal also includes decision over AH and ESP. Phase 2 always operates in quick mode and negotiation is done by exchange of three messages. Netscreen also has a replay protection feature. Custom phase-2 proposals can be created in netscreen. In phase 2 peers also exchange proxy IDs, a three-part tuple consisting of localIP-remote-IP-service. Perfect forward secrecy (PFS) is used for deriving phase 2 keys independent of phase 1 keys. ISAKMP is an IKE packet

Security Association (SA): unidirectional agreement between VPN participant regarding methods and parameters to use in securing communication. Full bidirectional communication requires at least two SAs, one for each direction. SA groups security algorithms and keys, protocol mode, key management method and SA lifetime. SA can be associated with a VPN tunnel