Network Security: Lecture 3

300143 Network Security

3. Internet Protocol Security (IPsec)

• Reference: [Chapter 9 IP Security]

 Especially, for this lecture, Sections 9.1 to 9.4
 or Directly related RFCs for IPsec version 1
 RFC2401: Security Architecture for the Internet Protocol
 RFC2402: IP Authentication Header
 RFC2406: IP Encapsulating Security Payload (ESP)

• IPsec has two parts:

 Establishment of a secure communication channel by using the IKE
protocol
 To be discussed in Lecture 4
 Operation of the secure channel
 This lecture is mainly on the operation.

© 2020.2 WSU Lecture No. 3-1

 Network Security: Lecture 3

Secure Internetworking Protocols …

• A secure networking protocol may be implemented at one

of the layers of the OSI network model.
• For internetworking, secure networking protocols are
implemented at Layer 3, Layer 4, or at a higher layer
 Basic scope of protection at an OSI layer (conceptually)

Hdr = Header Layer 3

Layer 4

Layer 7

L2Hdr L3Hdr L4Hdr … L7Hdr Layer 7 Payload

© 2020.2 WSU Lecture No. 3-2

 Network Security: Lecture 3

… Secure Internetworking Protocols …

• Both Layers 3 and 4 are usually in-built in an operating

system.
 Incorporation of security into Layer 3 or 4
 may require modifications to the OS
 automatically forces applications to be protected without major
modifications to the applications, especially for Layer 3
• Protection implemented at the Layer 3 (or IP layer) can:
 protect all Internet protocols, and even IP itself
 be transparent to upper layer protocols and applications
 be in a network device outside users’ computers, i.e., transparent to
individual users/computers
• A secure networking protocol, at Layer 4, only protects the
particular transport-layer application/service, e.g.,
 only the TCP service using the particular TCP port allocated to the
protocol.

© 2020.2 WSU Lecture No. 3-3

 Network Security: Lecture 3

… Secure Internetworking Protocols

All packets can be configured to

go through the protection, It can also be
especially those packets from implemented in a network
applications which are not self- device, and hence out of
protecting. the control of the sender.

Protection
at IP layer

It needs to be
implemented in a sender.
Protection via Internet
a dedicated
All packets TCP port
IP
No Protection
via other TCP
ports

© 2020.2 WSU Lecture No. 3-4

 Network Security: Lecture 3

Secure Networking Between Entities and/or Sites

Secure transport:
Internal connection: • Only protecting the message
• May or may not need protection contents

Router

Public network,
Network Site 1 e.g. the Internet. Network Site 2
Secure tunnel:
Leased line: • Protecting the whole message
• A private network (PN) including header and contents
• At physical layer, expensive, but • It may emulate a leased line, i.e.,
physically isolated and therefore secure virtual private network (VPN).
© 2020.2 WSU Lecture No. 3-5
 Network Security: Lecture 3

Internet Protocol Security (IPsec) …

• IPsec is an IETF security protocol for real time
communications.
 It operates at Layer 3 of the OSI model.
 The end entities negotiate, securely, in real-time to establish a
secure communication channel by:
 Authenticating each other

• End Entity authentication

 Establishing session keys for:

• Message origin authentication

 Including integrity protection

• Confidentiality protection
• IPsec provides methods to implement secure
 transport between two end entities
 tunnel between two network sites

© 2020.2 WSU Lecture No. 3-6

 Network Security: Lecture 3

… IPsec …
• [Figure 9.1 An IPsec VPN Scenario] IPsec Trailer

IP Header IP Payload IP Header IPsec Header IP Payload

Public network,
Network Site 1 e.g. the Internet. Network Site 2
= protected
= added
new IP Header IPsec Header IP Header IP Payload

© 2020.2 WSU Lecture No. 3-7

 Network Security: Lecture 3

… IPsec
• For this lecture, we will assume that the two
communicating entities have already:
 Authenticated each other
 Established session keys
 Negotiated and configured a secure communication channel
• The above are usually done by using the corresponding
IPsec IKE protocols to be discussed in the next lecture.
 Related RFCs for IKE version 1:
 RFC2407: The Internet IP Security Domain of Interpretation
for ISAKMP
 RFC2408: Internet Security Association and Key Management
Protocol (ISAKMP)
 RFC2409: The Internet Key Exchange (IKE)

• IPsec works with IPv4 or IPv6.

© 2020.2 WSU Lecture No. 3-8

 Network Security: Lecture 3

IPsec Services

• IPsec provides the following security services at the

network layer:
 Access control
 Access control to the IPsec-secured communication channel or
the system protected by IPsec
 Connectionless integrity
 Protection against modification of individual IP packets
 Data (or message) origin authentication
 Providing origin verification for each IP packet
 Rejection of replayed packets
 Protection against replay attacks outside a small time window
 Confidentiality
 Protection against passive attacks
 Limited traffic flow confidentiality
 Limited protection against traffic analysis, i.e., no protection
for secure transport and some protection for secure tunnel

© 2020.2 WSU Lecture No. 3-9

 Network Security: Lecture 3

IPsec Headers
• Security services are provided by one of the IPsec
protocols.
 A generic format of an IP packet after it is processed by IPsec:

New IP header or
IPsec IPsec
slightly modified (New) IP payload
header trailer
original IP header

• An IPsec header in the above diagram can be an:

AH Header (Authentication Header, defined in RFC 2402)
It provides most of the IPsec services, except confidentiality.


ESP Header (Encapsulating Security Payload, defined in RFC

2406)
 Provides all IPsec services
 Integrity protection does not protect the (outer) IP header, if
any. (Structure of an ESP packet to be discussed.)
 AH is not often used and therefore, in this unit, we focus on ESP.

© 2020.2 WSU Lecture No. 3-10

 Network Security: Lecture 3

Security Associations

• An IPsec Security Association (SA) is used to create a one-

way secure connection.
 An IPsec SA is identified by its Security Parameter Index (SPI)
• A security association refers to the set of security
parameters for an IPsec-secured connection.

IPsec-secured connection
using an IPsec SA

• Both sides store IPsec SA parameters for the secured connection.

• These parameters are used during the operation of the secure
connection for the security services provided by IPsec.

© 2020.2 WSU Lecture No. 3-11

 Network Security: Lecture 3

IPsec SA Parameters …

• Sequence Number Counter

 A 32-bit counter used to generate a number for the Sequence
Number field in an IPsec header
 Range of the sequence number is from 0 to 232 – 1.
 Used to provide timing and order the IPsec packets
• Sequence Counter Overflow
 A flag indicating whether overflow of the Sequence Number
Counter should generate an auditable event and further
transmission of packets using this SA should be terminated.
 If the secure channel is terminated, a new secure channel needs
to be created if secure communication is to be continued.
• Anti-Replay Window
 Used to determine whether an incoming IPsec packet is a replay
 It allows for (variations in) transmission delays, but replayed
packets cannot be detected within the window.

© 2020.2 WSU Lecture No. 3-12

 Network Security: Lecture 3

… IPsec SA Parameters
• AH Information
 Message authentication algorithm, keys, key lifetimes, and related
parameters being used by the AH protocol
• ESP Information
 Encryption and message authentication algorithms, keys, initial
values, key lifetimes, and ESP related parameters
• Lifetime of this Security Association
 When it expires, a secure channel using this SA is terminated
• IPsec Protocol Mode
 Tunnel, transport, or wildcard
 wildcard means which mode to use is determined during the
actual operation of the secure connection.
• Path MTU
 Any observed path maximum transmission unit.

© 2020.2 WSU Lecture No. 3-13

 Network Security: Lecture 3

Mode of Operation …

• [Figure 9.7 Transport-Mode vs Tunnel-Mode Encryption]

• There are two modes of operation: transport and tunnel
Only payload is
protected
IP Header IP Payload IP Header IP Payload

IP Header ESP Header IP Payload ESP Trailer

Transport mode Transport mode

encapsulation decapsulation

Tunnel mode Tunnel mode

encapsulation decapsulation

new IP Header ESP Header IP Header IP Payload ESP Trailer

© 2020.2 WSU Lecture No. 3-14

 Network Security: Lecture 3

… Mode of Operation

• Transport mode
 provides secure transport
 only protects the message contents, i.e., the IP packet payload
 does not protect other information about the message, i.e., IP
header
• Tunnel mode
 provides secure tunnel
 protects the whole IP packet including IP header and payload
 needs to add a new IP header, since the original IP header is hidden
by encryption
 is, usually, used by router/firewall to protect network traffic which
is transmitted to other networks
 protects the original end entities against traffic analysis.
 It hides the original IP header if encryption is used.

© 2020.2 WSU Lecture No. 3-15

 Network Security: Lecture 3

Transport Mode Encapsulation

Input IP Packet to IP Header IP Payload

IPsec Transport
Mode Processing
Transport Mode
Encapsulation

IP Header is ESP
IP Header IP Payload ESP Trailer
slightly modified Header

Output IP Packet IP Header New IP Payload

 The resulted packet has the same structure of an IP packet.

 ESP Header consists of just two header fields.
 ESP Trailer consists of one or two trailer items.

© 2020.2 WSU Lecture No. 3-16

 Network Security: Lecture 3

Tunnel Mode Encapsulation

Input IP Packet IP Header IP Payload

New IP Header Tunnel Mode

is added Encapsulation

New IP ESP
IP Header IP Payload ESP Trailer
Header Header

Output IP Packet

New IP
New IP Payload
Header

• Similarly, a new IP packet is produced.

© 2020.2 WSU Lecture No. 3-17

 Network Security: Lecture 3

Encapsulating Security Payload Scope

• [Figure 9.8 Scope of ESP Encryption and Authentication]

 Only IPv4 is considered in this unit.

Contains an upper layer protocol

IPv4 Packet IP Header IP Payload
ESP Auth is only present when
ESP authentication is used.
ESP authenticated
ESP encrypted

Transport mode IP ESP ESP ESP

IP Payload
Header Header Trlr Auth

ESP authenticated
ESP encrypted

Tunnel mode new IP ESP IP ESP ESP

IP Payload
Header Header Header Trlr Auth

 Green-coloured items become ESP Payload.

 ESP Trailer includes ESP Trlr and ESP Auth in the above diagrams.
© 2020.2 WSU Lecture No. 3-18
 Network Security: Lecture 3

ESP Encapsulation Packet Format …

• [Figure 9.5 (a) Top-level format of an ESP Packet]

Header
ESP
Bit: 0 8 16 24
Security Parameter Index (SPI)

ESP Payload
Sequence Number
ESP Payload (variable length)

Padding (variable) Pad Length Next Header

ESP
Trlr
ESP Authentication Data (variable length)

Auth
ESP
• ESP inserts ESP fields around the original input data.
• ESP should always do encryption, although a null
encryption algorithm may be specified.
• ESP Authentication Data is called Integrity Check value
(ICV) in the text and in version 2&3’s documentations
© 2020.2 WSU Lecture No. 3-19
 Network Security: Lecture 3

… ESP Encapsulation Packet Format

• Security Parameter Index and Sequence Number have been

discussed before.
• Payload Data is the IP payload of the input IP packet or
the whole input IP packet depending on what mode is used.
• Padding is
 for the block cipher, e.g., DES, 3DES or AES
 also used to protect against cryptanalysis, especially when the size
of Payload Data is small, or traffic analysis to hide payload size.
• Next Header points to the protocol carried in the Payload
Data.
• ESP Authentication Data is a keyed-hash created from
the Payload Data and the session key from the IPsec SA.

© 2020.2 WSU Lecture No. 3-20

 Network Security: Lecture 3

IPsec and Routers …

• Firewalls need to look at Layer 4 information, e.g.

TCP/UDP Ports to perform packet filtering.
• Transport layer segment is usually the payload of an IP
datagram.
 Encrypting an IP datagram’s payload will prevent firewalls from
access to Layer 4 information.
 A firewall may reject the packet.

Firewall is not able to inspect.

IP Header ESP Header TCP Header TCP Payload ESP Trailer

Incoming packet
Firewall

© 2020.2 WSU Lecture No. 3-21

 Network Security: Lecture 3

… IPsec and Routers

• Network address port translation (NAPT)
 A technique to overcome the limited number of IPv4 addresses.
 It translates between an internal source IP address and an external
source IP address.
 It can translate many internal IP addresses to one external IP
address.
• The source TCP port number and some other parameters
may need to be changed.

Router cannot alter values

protected by IPsec encryption.

IP Header ESP Header TCP Header TCP Payload ESP Trailer

Outgoing packet Router with NAPT

© 2020.2 WSU Lecture No. 3-22

 Network Security: Lecture 3

More Advanced Features

• Bundle of SAs: combining security associations
 An IP packet can be protected by a combination of SAs.
 [Figure 9.10 Basic Combinations of Security Associations]
• An example SA bundle: [Figure 9.10(c)]

IP Header IP Payload IP Header IP Payload

ESP encrypted ESP encrypted

IP Header ESP Hdr IP Payload ESP Trlr IP Header ESP Hdr IP Payload ESP Trlr

new IP Header ESP Hdr IP Header ESP Hdr IP Payload ESP Trlr ESP Trlr ESP Auth

ESP encrypted
ESP Authenticated

© 2020.2 WSU Lecture No. 3-23

 Network Security: Lecture 3

Wireshark Capture of ESP Packets in Transport Mode

Wireshark is not able to

interpret the captured data
here since some of the
data are encrypted.

© 2020.2 WSU Lecture No. 3-24

 Network Security: Lecture 3

Anti-Replay ...

Replay
Message is delayed
Attacker
and resent.

IPsec packet m
Bob IPsec packet m Alice

• [Figure 9.6 Anti-Replay Mechanism] at the receiving end

Window moves forward if a IPsec packet with
IPsec packets with seq. a larger than N seq. no. is received.
no. ≤ (N – W) will be
Window of fixed size W
rejected.

... N ...
Packet is N-W
considered lost. IPsec packets have N is the largest seq. no.
not arrived. received so far.

© 2020.2 WSU Lecture No. 3-25

 Network Security: Lecture 3

... Anti-Replay

• To protect an IPsec connection against replay attacks:

 A 32-bit sequence number is placed in each outgoing IPsec packet
from the sender.
 A ‘sliding’ window is used, by the receiver, to keep track of
received IPsec packets.
 IPsec packets may arrive out of order, e.g. a packet with a
smaller sequence number may arrive after one with a large
number.
 Within this window, this out-of-order arrival is allowed.
Replay attacks may still be possible within that window.
• False out-of-order arrivals happen, if the sequence counter
overflows, i.e., counting from zero again.
 In this case, an arriving IPsec packet always has a smaller sequence
number than the current largest sequence number.
 A secure channel can be configured to terminate when the sequence
counter reaches its maximum value, i.e., 232 ‒ 1 and a new secure
channel is to be created to continue secure communications.

© 2020.2 WSU Lecture No. 3-26

 Network Security: Lecture 3

A Typical IPsec System

• [Figure 9.2 IPsec Architecture]
• In addition to the underlying IPsec components, such as
encryption algorithms, key exchange algorithms, and so on,
a typical IPsec implementation should also have:
 Security Policy Database (SPD)
 Security Association Database (SAD)
• Relation between these databases:
 Logically, outbound and inbound traffic have their own databases.

Outbound
or
Inbound

SAD
SPD
© 2020.2 WSU Lecture No. 3-27
 Network Security: Lecture 3

SPD

• [Table 9.2 Host SPD Example]

• A typical SPD entry has:
 An identity number to identify the entry in the database
 Traffic Selectors to match a packet, including source and
destination IP addresses, protocol, source and destination ports, etc.
 Action is performed if these selectors are matched to the
parameters in a packet and other outside relevant parameters.
• Example actions:
• Dropped, forwarded or accepted with or without IPsec,
and if IPsec, which SA(s)
• An typical SPD entry format:

Traffic Selectors Action(s) Other fields

© 2020.2 WSU Lecture No. 3-28

 Network Security: Lecture 3

IPsec Traffic Processing with SPD and SAD …

SPD
Selectors Action Matched SPD
Outgoing IP packet Other entry, with Action
parameters requiring IPsec

… …
IP Header TCP Header TCP Payload

Search
No IPsec SA
IPsec SA entry
IPsec ?

Yes …
IPsec SA entry
IPsec Processing
Assuming an
… IPsec SA is
already created;
if not IKE needs
SAD to be done.
IP packet IPsec packet

© 2020.2 WSU Lecture No. 3-29

 Network Security: Lecture 3

… IPsec Traffic Processing with SPD and SAD

• [Figure 9.3 Processing Model for Outbound Packets]

• For an outgoing IP packet,
 IPsec searches the SPD for an entry whose selectors match the
values in the TCP/IP headers and other parameter values, if any
 If a match is found and if protection is required, the SAD
database is searched for the SA to implement the SPD action.
• If the SA entry related to the SPD entry can be located
from the SAD database, the packet is then processed by
IPsec, according to the entry;
• If there is no SA to implement the actions specified in the
SPD entry, the IKE protocol is used to create the SA.
• Other outcomes for the search of the SPD depend on how
the SPD is designed and if other filtering is implemented.
• For an incoming IP packet at the receiving end, a similar
search process of the SPD and SAD is required.
© 2020.2 WSU Lecture No. 3-30
 Network Security: Lecture 3

SAD

• Security Association Database

 More dynamic in nature since SAs are created and removed during
an IPsec operation
 Logically, there is one SAD for inbound and one for outbound
traffic.
• An SAD entry should have:
 An entry identification
 In some implementations, the actual SA for this SAD entry may be
stored in another database, referred to as an SA List.
 An SAD entry, instead of containing the details of an SA, will
have a pointer pointing to the SA on the SA List.

© 2020.2 WSU Lecture No. 3-31

 Network Security: Lecture 3

Windows 7 IPsec Configuration

• One way to configure IPsec on a Windows 7 computer is to
use the IP Security Policy Management snap-in in the
Microsoft Management Console (mmc.exe)

IP Security Policy
Management

© 2020.2 WSU Lecture No. 3-32

 Network Security: Lecture 3

Windows 7 IKE Settings

• Life times of the policy and session keys for the IKE
• List of cryptographic methods used to protect the IKE

© 2020.2 WSU Lecture No. 3-33

 Network Security: Lecture 3

Windows 7 IPsec Access Control

• Traffic selector
 Specifying what traffic to be protected.

© 2020.2 WSU Lecture No. 3-34

 Network Security: Lecture 3

IPsec Protection Methods

• Specifying the list of cryptographic methods for protecting
the IPsec secure channel and other parameters

© 2020.2 WSU Lecture No. 3-35

 Network Security: Lecture 3

IKE Authentication Method

• IKE end entity authentication methods will be discussed in

the next lecture.

*** End ***

© 2020.2 WSU Lecture No. 3-36