IP Security

Users have security concerns that cut across protocol layers. For example, an enterprise can run
a secure, private IP network by disallowing links to untrusted sites, encrypting packets that leave the premises,
and authenticating packets that enter the premises. By implementing security at the IP level, an organization can
ensure secure networking not only for applications that have security mechanisms but also for the many security-
ignorant applications.
IP-level security encompasses three functional areas: authentication, confidentiality, and key management.
IP SECURITY OVERVIEW:
Applications of IPsec
IPsec provides the capability to secure communications across a LAN, across private and public WANs, and
across the Internet. Examples of its use include:
■ Secure branch office connectivity over the Internet: A company can build a secure virtual private network
over the Internet or over a public WAN. This enables a business to rely heavily on the Internet and reduce its need
for private networks, saving costs and network management overhead.
■ Secure remote access over the Internet: An end user whose system is equipped with IP security protocols can
make a local call to an Internet Service Provider (ISP) and gain secure access to a company network. This reduces
the cost of toll charges for traveling employees and telecommuters.
■ Establishing extranet and intranet connectivity with partners: IPsec can be used to secure communication
with other organizations, ensuring authentication and confidentiality and providing a key exchange mechanism.
■ Enhancing electronic commerce security: Even though some Web and electronic commerce applications
have built-in security protocols, the use of IPsec enhances that security. IPsec guarantees that all traffic designated
by the network administrator is both encrypted and authenticated.
Figure 8.1 is a typical scenario of IPsec usage. An organization maintains LANs at dispersed locations.
Non secure IP traffic is conducted on each LAN. For traffic offsite, through some sort of private or public WAN,
IPsec protocols are used. These protocols operate in networking devices, such as a router or firewall that connect
each LAN to the outside world. The IPsec networking device will typically encrypt all traffic going into the WAN
and decrypt traffic coming from the WAN; these operations are transparent to workstations and servers on the
LAN. Secure transmission is also possible with individual users who dial into the WAN. Such user workstations
must implement the IPsec protocols to provide security.

Figure 8.1: A typical scenario of IPsec usage

 Benefits of IPsec:
Some of the benefits of IPsec:
■ When IPsec is implemented in a firewall or router, it provides strong security that can be applied to all traffic
crossing the perimeter. Traffic within a company or workgroup does not incur the overhead of security-related
processing.
■ IPsec in a firewall is resistant to bypass if all traffic from the outside must use IP and the firewall is the only
means of entrance from the Internet into the organization.
■ IPsec is below the transport layer (TCP, UDP) and so is transparent to applications. There is no need to change
software on a user or server system when IPsec is implemented in the firewall or router. Even if IPsec is
implemented in end systems, upper-layer software, including applications, is not affected.
■ IPsec can be transparent to end users. There is no need to train users on security mechanisms, issue keying
material on a per-user basis, or revoke keying material when users leave the organization.
■ IPsec can provide security for individual users if needed. This is useful for offsite workers and for setting up a
secure virtual sub network within an organization for sensitive applications.

Routing Applications:
In addition to supporting end users and protecting premises systems and networks, IPsec can play a vital role in
the routing architecture required for internetworking.
IPsec can assure that
■ A router advertisement (a new router advertises its presence) comes from an authorized router.
■ A neighbor advertisement (a router seeks to establish or maintain a neighbor relationship with a router in another
routing domain) comes from an authorized router.
■ A redirect message comes from the router to which the initial IP packet was sent.
■ A routing update is not forged.
Without such security measures, an opponent can disrupt communications or divert some traffic. Routing
protocols such as Open Shortest Path First (OSPF) should be run on top of security associations between routers
that are defined by IPsec.

IPsec Services:
IPsec provides security services at the IP layer by enabling a system to select required security protocols,
determine the algorithm(s) to use for the service(s), and put in place any cryptographic keys required to provide
the requested services. Two protocols are used to provide security: an authentication protocol designated by the
header of the protocol, Authentication Header (AH); and a combined encryption/authentication protocol
designated by the format of the packet for that protocol, Encapsulating Security Payload (ESP). The following
services provided IPSec:
■ Access control
■ Connectionless integrity
■ Data origin authentication
■ Rejection of replayed packets (a form of partial sequence integrity)
■ Confidentiality (encryption)
■ Limited traffic flow confidentiality

Transport and Tunnel Modes:

Both AH and ESP support two modes of use: transport and tunnel mode.
 Transport mode provides protection primarily for upper-layer protocols. That is, transport mode protection
extends to the payload of an IP packet. When a host runs AH or ESP over IPv4, the payload is the data that
normally follow the IP header. For IPv6, the payload is the data that normally follow both the IP header and
any IPv6 extensions headers that are present, with the possible exception of the destination options header,
which may be included in the protection.
 ESP in transport mode encrypts and optionally authenticates the IP payload but not the IP header. AH in
transport mode authenticates the IP payload and selected portions of the IP header.
  Tunnel mode provides protection to the entire IP packet. To achieve this, after the AH or ESP fields are
added to the IP packet, the entire packet plus security fields is treated as the payload of new outer IP packet
with a new outer IP header. The entire original, inner, packet travels through a tunnel from one point of an
IP network to another; no routers along the way are able to examine the inner IP header. Because the
original packet is encapsulated, the new, larger packet may have totally different source and destination
addresses, adding to the security. Tunnel mode is used when one or both ends of a security association
(SA) are a security gateway, such as a firewall or router that implements IPsec.

 Here is an example of how tunnel mode IPsec operates. Host A on a network generates an IP packet with
the destination address of host B on another network. This packet is routed from the originating host to a
firewall or secure router at the boundary of A’s network. The firewall filters all outgoing packets to
determine the need for IPsec processing. If this packet from A to B requires IPsec, the firewall performs
IPsec processing and encapsulates the packet with an outer IP header. The source IP address of this outer
IP packet is this firewall, and the destination address may be a firewall that forms the boundary to B’s local
network. This packet is now routed to B’s firewall, with intermediate routers examining only the outer IP
header. At B’s firewall, the outer IP header is stripped off, and the inner packet is delivered to B.

 ESP in tunnel mode encrypts and optionally authenticates the entire inner IP packet, including the inner IP
header. AH in tunnel mode authenticates the entire inner IP packet and selected portions of the outer IP
header.

IP SECURITY POLICY:
Fundamental to the operation of IPsec is the concept of a security policy applied to each IP packet that transits
from a source to a destination. IPsec policy is determined primarily by the interaction of two databases, the
security association database (SAD) and the security policy database (SPD). Figure 8.2 illustrates the relevant
relationships (IPSec Architecture).

Figure 8.2: IPsec Architecture

Security Associations (SA):
A key concept that appears in both the authentication and confidentiality mechanisms for IP is the security
association (SA). An SA is a one-way logical connection between a sender and a receiver that affords security
services to the traffic carried on it. If a peer relationship is needed for two-way secure exchange, then two security
associations are required.
A security association is uniquely identified by three parameters.
■ Security Parameters Index (SPI): A 32-bit unsigned integer assigned to this SA and having local significance
only. The SPI is carried in AH and ESP headers to enable the receiving system to select the SA under which a
received packet will be processed.
■ IP Destination Address: This is the address of the destination endpoint of the SA, which may be an end-user
system or a network system such as a firewall or router.
■ Security Protocol Identifier: This field from the outer IP header indicates whether the association is an AH or
ESP security association.

Security Association Database (SAD):

In each IPsec implementation, there is a Security Association Database that defines the parameters associated
with each SA. A security association is normally defined by the following parameters in an SAD entry.
■ Security Parameter Index: A 32-bit value selected by the receiving end of an SA to uniquely identify the SA.
In an SAD entry for an outbound SA, the SPI is used to construct the packet’s AH or ESP header. In an SAD
entry for an inbound SA, the SPI is used to map traffic to the appropriate SA.
■ Sequence Number Counter: A 32-bit value used to generate the Sequence Number field in AH or ESP headers.
■ Sequence Counter Overflow: A flag indicating whether overflow of the Sequence Number Counter should
generate an auditable event and prevent further transmission of packets on this SA.
■ Anti-Replay Window: Used to determine whether an inbound AH or ESP packet is a replay.
■ AH Information: Authentication algorithm, keys, key lifetimes, and related parameters being used with AH.
■ ESP Information: Encryption and authentication algorithm, keys, initialization values, key lifetimes, and
related parameters being used with ESP.
■ Lifetime of this Security Association: A time interval or byte count after which an SA must be replaced with
a new SA (and new SPI) or terminated, plus an indication of which of these actions should occur.
■ IPsec Protocol Mode: Tunnel, transport, or wildcard.
■ Path MTU: Any observed path maximum transmission unit (maximum size of a packet that can be transmitted
without fragmentation) and aging variables.

Security Policy Database (SPD):

o The means by which IP traffic is related to specific SAs (or no SA in the case of traffic allowed to bypass
IPsec) is the nominal Security Policy Database (SPD). In its simplest form, an SPD contains entries, each of
which defines a subset of IP traffic and points to an SA for that traffic.
o Each SPD entry is defined by a set of IP and upper-layer protocol field values, called selectors. In effect, these
selectors are used to filter outgoing traffic in order to map it into a particular SA.
The following selectors determine an SPD entry:
■ Remote IP Address: This may be a single IP address, an enumerated list or range of addresses, or a wildcard
(mask) address. The latter two are required to support more than one destination system sharing the same SA
(e.g., behind a firewall).
■ Local IP Address: This may be a single IP address, an enumerated list or range of addresses, or a wildcard
(mask) address. The latter two are required to support more than one source system sharing the same SA (e.g.,
behind a firewall).
■ Next Layer Protocol: The IP protocol header (IPv4, IPv6, or IPv6 Extension) includes a field (Protocol for
IPv4, Next Header for IPv6 or IPv6 Extension) that designates the protocol operating over IP. If AH or ESP is
used, then this IP protocol header immediately precedes the AH or ESP header in the packet.
■ Name: A user identifier from the operating system. This is not a field in the IP or upper-layer headers but is
available if IPsec is running on the same operating system as the user.
■ Local and Remote Ports: These may be individual TCP or UDP port values, an enumerated list of ports, or a
wildcard port.

IP Traffic Processing:
IPsec is executed on a packet-by-packet basis. When IPsec is implemented, each outbound IP packet is
processed by the IPsec logic before transmission, and each inbound packet is processed by the IPsec logic after
reception and before passing the packet contents on to the next higher layer (e.g., TCP or UDP).
OUTBOUND PACKETS Figure 8.3 highlights the main elements of IPsec processing for outbound traffic. A
block of data from a higher layer, such as TCP is passed down to the IP layer and an IP packet is formed, consisting
of an IP header and an IP body. Then the following steps occur:
1. IPsec searches the SPD for a match to this packet.
2. If no match is found, then the packet is discarded and an error message is generated.
3. If a match is found, further processing is determined by the first matching entry in the SPD. If the policy for
this packet is DISCARD, then the packet is discarded. If the policy is BYPASS, then there is no further IPsec
processing; the packet is forwarded to the network for transmission.
4. If the policy is PROTECT, then a search is made of the SAD for a matching entry. If no entry is found, then
IKE is invoked to create an SA with the appropriate keys and an entry is made in the SA.
5. The matching entry in the SAD determines the processing for this packet. Either encryption, authentication,
or both can be performed, and either transport or tunnel mode can be used. The packet is then forwarded to
the network for transmission.

Figure 8.3: Processing Model for Outbound Packets

Figure 8.4: Processing Model for Inbound Packets

INBOUND PACKETS Figure 8.4 highlights the main elements of IPsec processing for inbound traffic. An
incoming IP packet triggers the IPsec processing. The following steps occur:
1. IPsec determines whether this is an unsecured IP packet or one that has ESP or AH headers/trailers, by
examining the IP Protocol field (IPv4) or Next Header field (IPv6).
2. If the packet is unsecured, IPsec searches the SPD for a match to this packet. If the first matching entry has a
policy of BYPASS, the IP header is processed and stripped off and the packet body is delivered to the next
higher layer, such as TCP. If the first matching entry has a policy of PROTECT or DISCARD, or if there is
no matching entry, the packet is discarded.
3. For a secured packet, IPsec searches the SAD. If no match is found, the packet is discarded. Otherwise, IPsec
applies the appropriate ESP or AH processing. Then, the IP header is processed and stripped off and the packet
body is delivered to the next higher layer, such as TCP.

ENCAPSULATING SECURITY PAYLOAD:

ESP can be used to provide confidentiality, data origin authentication, connectionless integrity, an anti-replay
service (a form of partial sequence integrity), and (limited) traffic flow confidentiality. The set of services
provided depends on options selected at the time of Security Association (SA) establishment and on the location
of the implementation in a network topology.
ESP can work with a variety of encryption and authentication algorithms, including authenticated encryption
algorithms such as GCM (Galois/Counter Mode).

ESP Format:
Figure 8.5 shows the top-level format of an ESP packet. It contains the following fields.
■ Security Parameters Index (32 bits): Identifies a security association.
■ Sequence Number (32 bits): A monotonically increasing counter value; this provides an anti-replay function,
as discussed for AH.
■ Payload Data (variable): This is a transport-level segment (transport mode) or IP packet (tunnel mode) that is
protected by encryption.
■ Padding (0–255 bytes): If an encryption algorithm requires the plaintext to be a multiple of some number of
bytes (the multiple of a single block for a block cipher), the Padding field is used to expand the plaintext to the
required length.
■ Pad Length (8 bits): Indicates the number of pad bytes immediately preceding this field.
■ Next Header (8 bits): Identifies the type of data contained in the payload data field by identifying the first
header in that payload (e.g., an extension header in IPv6, or an upper-layer protocol such as TCP).
■ Integrity Check Value (variable): A variable-length field (must be an integral number of 32-bit words) that
contains the Integrity Check Value computed over the ESP packet minus the Authentication Data field.

Figure 8.5:Top-level format of an ESP Packet

When any combined mode algorithm is employed, the algorithm itself is expected to return both decrypted
plaintext and a pass/fail indication for the integrity check. For combined mode algorithms, the ICV that would
normally appear at the end of the ESP packet (when integrity is selected) may be omitted. When the ICV is omitted
and integrity is selected, it is the responsibility of the combined mode algorithm to encode within the Payload
Data an ICV-equivalent means of verifying the integrity of the packet.
Two additional fields may be present in the payload part. An initialization value (IV), or nonce is present if
this is required by the encryption or authenticated encryption algorithm used for ESP. If tunnel mode is being
used, then the IPsec implementation may add traffic flow confidentiality (TFC) padding after the Payload Data
and before the Padding field.

Anti-Replay Service:
 A replay attack is one in which an attacker obtains a copy of an authenticated packet and later transmits
it to the intended destination. The receipt of duplicate, authenticated IP packets may disrupt service in
some way or may have some other undesired consequence.
 The Sequence Number field is designed to thwart such attacks.When a new SA is established, the sender
initializes a sequence number counter to 0. Each time that a packet is sent on this SA, the sender increments
the counter and places the value in the Sequence Number field. Thus, the first value to be used is 1. If anti-
replay is enabled (the default), the sender must not allow the sequence number to cycle past 232 - 1 back
to zero. Otherwise, there would be multiple valid packets with the same sequence number. If the limit of
232 - 1 is reached, the sender should terminate this SA and negotiate a new SA with a new key.
 The IPsec authentication document dictates that the receiver should implement a window of size W, with
a default of W = 64. The right edge of the window represents the highest sequence number, N, so far
received for a valid packet. For any packet with a sequence number in the range from N - W + 1 to N that
has been correctly received (i.e., properly authenticated), the corresponding slot in the window is marked
(Figure 8.6).
Inbound processing proceeds as follows when a packet is received:
1. If the received packet falls within the window and is new, the MAC is checked. If the packet is authenticated,
the corresponding slot in the window is marked.
2. If the received packet is to the right of the window and is new, the MAC is checked. If the packet is
authenticated, the window is advanced so that this sequence number is the right edge of the window, and the
corresponding slot in the window is marked.
3. If the received packet is to the left of the window or if authentication fails, the packet is discarded; this is an
auditable event.

Figure 8.6: Anti-replay Mechanism

Transport and Tunnel Modes:
 TRANSPORT MODE ESP: Transport mode ESP is used to encrypt and optionally authenticate the data
carried by IP (e.g., a TCP segment), as shown in Figure 8.7b. For this mode using IPv4, the ESP header is
inserted into the IP packet immediately prior to the transport-layer header (e.g., TCP, UDP, ICMP), and
an ESP trailer (Padding, Pad Length, and Next Header fields) is placed after the IP packet.
 If authentication is selected, the ESP Authentication Data field is added after the ESP trailer. The entire
transport-level segment plus the ESP trailer are encrypted. Authentication covers all of the ciphertext plus
the ESP header. For IPv6, encryption covers the entire transport-level segment plus the ESP trailer plus
 the destination options extension header if it occurs after the ESP header. Again, authentication covers the
ciphertext plus the ESP header.
 Transport mode operation provides confidentiality for any application that uses it. One drawback to this
mode is that it is possible to do traffic analysis on the transmitted packets.

(a) Before Applying ESP

(b) Transport Mode

(c) Tunnel Mode

Figure 8.7: Scope of ESP Encryption and Authentication

 TUNNEL MODE ESP: Tunnel mode ESP is used to encrypt an entire IP packet (Figure 8.7c). For this mode,
the ESP header is prefixed to the packet and then the packet plus the ESP trailer is encrypted. This method
can be used to counter traffic analysis.
 Because the IP header contains the destination address and possibly source routing directives and hop-by-
hop option information, it is not possible simply to transmit the encrypted IP packet prefixed by the ESP
header. Intermediate routers would be unable to process such a packet. Therefore, it is necessary to
encapsulate the entire block (ESP header plus ciphertext plus Authentication Data, if present) with a new IP
header that will contain sufficient information for routing but not for traffic analysis.
 The tunnel mode is useful in a configuration that includes a firewall or other sort of security gateway that
protects a trusted network from external networks. In this case, encryption occurs only between an external
host and the security gateway or between two security gateways. This relieves hosts on the internal network
of the processing burden of encryption and simplifies the key distribution task by reducing the number of
needed keys. Further, it thwarts traffic analysis based on ultimate destination.