Analysis of problems associated with IPSec VPN Technology
Olalekan Adeyinka
School of Computing and Technology,
University of East London
adeyinka@acm.org
Abstract
The original goal of Internet Protocol Security (IPSec) is
to enable the protection of all types of Internet protocol
(IP) communications by protecting multiple peers at the
network layer, in both the IPv4 and IPv6 environments.
IPSec is a standard for securing internet communication
and a widely deployed mechanism for implementing
Virtual Private Networks (VPNs).
Most organisations deploy IPSec VPNs that provide
enterprise-level secure remote access by protecting the
IP packet exchanged between remote networks or host
and an IPSec gateway located at the edge of a private
network.. This paper investigates the security issues of
IPSec VPN technology with respect to remote access
communication.
1.0 Introduction
The internet was designed to be an open and distributed
environment without any central instance controlling the
communication among the users and thus mutual mistrust
was not of primary concern. The internet is in general an
adversarial environment where attacks can be quite easy,
inexpensive and may be hard to prevent, detect or trace.
In general, it is difficult to ensure the main security goals
regarding confidentiality, integrity and availability. The
Internet has become a popular, low-cost backbone
infrastructure. Its universal reach has led many
businesses to consider constructing a secure virtual
private network (VPN) over the public Internet. The
challenge in designing a VPN for today’s global business
environment will be to exploit the public Internet
backbone for both intra-company and inter-company
communication while still providing the security of the
traditional private, self-administered corporate network.
IPSec VPNs were created as a cost-effective encrypted
transport alternative to private or leased lines.
978-1-4244-1643-1/08/$25.00 ©2008 IEEE
001903
The original goal of IPSec [1] is to enable the protection
of all types of Internet Protocol (IP) communication. The
IPSec uses the Internet Key Exchange (IKE) [2] protocol
to negotiate protocols between peers and generate
encryption and authentication keys. Many commercial
IPSec clients are available for windows operating
systems and also there are many Unix-like operating
systems that have IPSec capabilities. IPSec have been
widely used recently to provide the most versatile
gateway solution in web services, hence it is designed to
allow encryption and authentication network traffic
between the host machines over an existing TCP/IP
network. Therefore, its associated authentication
mechanism will manage the authentication and
encryption phases between the client and the gateway
computers. This process will allow the administrator of a
gateway to choose or select the best authentication
mechanism for each case. Authentication could be
identified as pre-shared secret keys, or it could utilise a
certificate based authentication scheme [3].
Using an IPSec is a way of providing additional security
of IPv4 based on networks, while maintaining
compatibility with the exiting networks infrastructure. It
provides two network protocols, the Authentication
Header and the Encapsulating secure payload and these
works together as they encrypt and authenticate network
data packets [4]. There are two operation modes of IPSec
which are the tunnel mode and the transport mode. The
tunnel mode as it is known protects the IP header and
reveals only the IP address of the IPSec gateway
machine, while the transport mode does not protect the
original IP header but encrypts only the payload. Though
IPSec is extremely complex, is a flexible suit of
protocols. It relies on asymmetric key algorithms, a
public/private key pair-one to encrypt and to decrypt.
Applications requiring standards-based security services
such as the World Wide Web (WWW), email, IP
Security (IPSec) and secure commerce transactions are
being enhanced to take full advantage of this technology,
[5]. It is known that asymmetric algorithms are preferable
for encrypting the keys, but they are more CPU-intensive
than symmetric algorithms and may not be the best
means of encrypting the actual message. The biggest
difference between Secure Socket Layer (SSL) VPNs and
traditional IP Security remote access VPNs is that IPSec
standard requires installation of client software on the
end user’s system, while SSL VPNs focus on making
applications available through any web browser [6]. SSL
VPNs are easier to set up and use for the end user than
the traditional IPSec VPNs, but sometimes harder for the
network manager to maintain.
2.0 IPSec Protocols.
In today’s IT environment, it is critical to protect data
traffic between disparate host systems in multi-tier
applications. Information security teams look for security
vulnerabilities and try to manage the risk of data
tampering, snooping, and eavesdropping. Plain text data
flow of critical information like passwords, credit card
numbers, and privacy information between systems is
highly vulnerable to misuse. The technology that brings
secure communications to the Internet Protocol (IP) is
called Internet Protocol Security, commonly abbreviated
IPSec. It is an Internet Engineering Task Force (IETF)
standard, and many operating systems vendors have
incorporated IPSec in their product offerings. The
Internet needs more and better security due to the
inherent security weakness in the TCP/IP protocols suite
and it identified key areas of security improvements.
IPSec protects IP packets, supports a strong encryption
and data integrity mechanisms and is a network layer
VPN technology, meaning it operates independent of the
application(s) that may use it. The VPNs relied on dial-
up connections initially, but the increasing availability
and decreasing cost of broadband internet connectivity
has led companies to develop internet VPNs in an effort
to provide a more flexible and cost effective solution.
IPSec is a framework that provides security services at
the IP layer by enabling a system to select required
security protocols, determine the algorithm(s) to use for
the service(s), and put in place any cryptographic keys
required to provide the requested services [8]. IPSec can
protect host-to-host, gateway-to-gateway or host-to-
gateway / gateway-to-host packet communications.
The security services provided by IPSec are
connectionless integrity, data origin authentication,
protection against replays, and confidentiality. These
services are provided at the IP layer, thus offering
protection for IP and all upper layer protocols. Optional
in IPv4, IPSec is mandatory for any implementation of
IPv6. Once IPv6 is widely spread, it will be possible for
any user wishing to use security functions to use IPSec.
In the meantime, we must rely on IPSec implementations
under IPv4.
It must be acknowledged that the IPSec is perhaps one of
the most complicated and confusing security standards
ever put forward for universal implementation. There are
001904
two security mechanism of IPSec. The ESP
(Encapsulating Security Payload) mechanism [9]
provides confidentiality, authentication, and integrity, the
AH (Authentication Header) mechanism [10] provides
Authentication and integrity. It protects against data
tampering and unauthorized retransmission of packets.
The last component is the IKE (Internet Key Exchange)
that provides key management and security association
management
IPSec can be used in two modes, namely, transport mode
and tunnel modes. Transport mode provides protection
primarily for upper layer protocols. It is used for end-to-
end communication between two hosts. ESP encrypts and
optionally authenticates the IP payload but not the IP
header. AH authenticates the IP payload and non-mutable
portions of the header. AH and ESP header are inserted
after the original IP header and before the IP payload.
The ESP trailer is inserted after the IP datagram and after
at the optional ESP authentication data field can be
placed.
The IPSec protocols include AH, ESP, IKE,
ISAKMP/Oakley, and transform. In order to understand,
implement and use IPSec, it is necessary to understand
the relationship among these components. The IPSec
roadmap defines how various components of IPSec
interact with each other. The ESP and the AH documents
defines the protocol, the payload header format, and the
services they provide. In addition these documents define
the packet processing rule. IKE generates keys for the
IPSec protocols. IKE is also used to negotiate keys for
other protocols that need keys. The parameters that are
negotiated are documented in a separate document called
the IPSec Domain of Interpretation [4]. Authentication
Header (AH) and encapsulating security payload (ESP)
are two components of IPSec that are added to the plain
internet protocol to meet the security requirements. Both
mechanisms add a new header to the IP datagram.
Important fundamental concept in the IPSec architecture
is the security association (SA). It is a one-way
relationship between the sender and a receiver which
contains all the necessary information for secured
communication, such as negotiated encryption algorithms
being used, and encryption keys. For a two-way
relationship, two SA are needed. This is also the case
when both AH and ESP are needed for communication.
An SA is uniquely identified by three parameters: SPI
(security parameters index), Destination IP address and
security protocol (AH or ESP). SAs are kept in an SA
database (SAD) and when a datagram is sent, its
destination address is looked for in the SAD and security
policy database (SPD) is used to decide whether the
datagram is described or accepted [11].
2.1 Security Mechanism (AH and ESP Protocol)

The two security mechanisms use are the AH and ESP

"protocols", which are added to traditional IP processing.
Authentication Header (AH) is conceived to ensure
integrity and authentication of IP datagrams, without data
encryption (i.e. without confidentiality). The principle of
AH is to add an additional field to the traditional IP
datagram; this field makes it possible for the receiver to
check the authenticity of the data included in the
datagram [10]. The authentication header guarantees data
integrity and it authenticates the sending peer. Message
authentication code (MAC) is used in AH and the hash
function is applied to the whole data part of the IP
datagram and to non-mutable fields of the IP header. A Fig 2 Position of AH in tunnel mode (IPv4)
compliant AH implementation must support the HMAC
hash algorithm with MD5 or SHA- 1 algorithms. 2.1.2 Encapsulating Security Payload (ESP)
Encapsulating Security Payload (ESP) is primarily
designed for ensuring confidentiality, but can also The figures below indicate the position of ESP and the
provide data authenticity. The principle of ESP is to services it provides, depending on the selected mode
generate, from a traditional IP datagram, a new datagram (transport or tunnel).
in which the data and eventually the original header are
encrypted [9]. The encapsulating security payload
provides confidentiality, optional integrity and
authentication. The original datagram is encrypted before
it is sent. There are multiple encryption algorithms (e.g.
DES, 3DES) that can be employed of for any DES IPSec
implementation.

2.1.1 Authentication Header (AH)

The figures below indicate the position of AH and the

service brought depending on the selected mode
(transport or tunnel).

Fig 3 Position of ESP in transport mode (IPv4)

Fig1. Position of AH in transport mode (IPv4)

Fig 4 Position of ESP in tunnel mode (IPv4)

001905
2.2 IKE Overview
The purpose of Internet Key Exchange (IKE) is to allow
devices to exchange information that’s required for
secure communication. It operates by allowing security
associations to be negotiated through a series of
ISAKMP, a generic protocol that supports many different
key exchange methods. In IKE, the ISAKMP framework
is used as the basis for a specific key exchange method
that combines features from two key exchange protocols.
It also used for negotiating the SAs develop for IPSec.
This includes cryptographic keys that are used for
encoding authentication information and performing
payload encryption [2]. IKE works by allowing IPSec-
capable devices to exchange SAs, which populate their
SADs. These SADs are then used for the actual
exchange of secured datagrams with the AH and ESP
protocols. IKE is considered a hybrid protocol because it
combines (and supplements) the functions of three other
protocols. The Internet Security Association and Key
Management Protocol (ISAKMP). The protocol
ISAKMP is not usable alone, it’s a framework which
allows the use of several key exchange protocols and
which can be used for other security mechanisms than
those of IPSec. Within the framework of the
standardization of IPSec, ISAKMP is associated with
part of the SKEME and Oakley protocols to result in a
protocol called IKE (Internet Key Exchange).
3.0 IPSec VPN Overview
IPSec based VPNs are the deployment-proven remote
access technology used by most organizations today to
establish connection using pre-installed VPN client
software to connects hosts to entire private networks by
protecting the IP packet exchanged between remote
networks or hosts and an IPSec gateway located at the
edge of the private network. IPSec VPNs can support all
IP-based applications to an IPSec VPN product. All IP
packets are the same, [7] With the IPSec client software,
organizations can control the function of the VPN client
fuse in applications such as unattended kiosks,
integration with other desktop applications, and other
special use cases. IPSec authentication employs Internet
Key Exchange (IKE), using digital certificates or pre
shared secrets for two-way authentication. They differ
significantly on how these extensions are implemented.
Many organizations find that IPSec meets the
requirements of users already using the technology. But
the advantages of dynamic self-updating desktop
software, ease of access for non-company-managed
desktops, and highly customizable user access make SSL
VPNs a compelling choice for reducing remote-access
VPN operations costs and extending network access to
hard-to-serve users like contractors and business partners.
As such, organizations often deploy a combination of
001906
SSL and IPSec approaches. The IPSec VPNs technology
provides the robust solution for the Internet based
connectivity. IPSec encapsulates the original IP data
packet with its own packet, thus hiding all application
protocol information. Once an IPSec tunnel is negotiated,
any number of connections and types (web, email, file
transfer, VoIP) can flow over it, each destined for
different servers behind the VPN gateway.
IT administrators must determine who should have
remote access to the network because IPSec VPNs
require a client to be installed on each user machine this
entails deployment, configuration and maintenance. The
solution becomes resource intensive and cost prohibitive
when deployed across large enterprise [12].Traditional
VPNs rely on IPSec to tunnel between the two endpoints.
IPSec works on the Network Layer of the OSI Model-
securing all data that travels between the two endpoints
without an association to any specific application. When
connected on an IPSec VPN the client computer is
“virtually” a full member of the corporate network- able
to see and potentially access the entire network. The
majority of IPSec VPN solutions require third-party
hardware and / or software. In order to access an IPSec
VPN, the workstation or device in question must have an
IPSec client software application installed.
Fig 5 IPSec VPN gateway technology
IPSec VPNs in Fig 5 provide access to the entire subnets
of a network. A remote user with VPN client software
installed communicate through the firewall or VPN
gateway and initiates a key exchange called the Internet
key Exchange (IKE).
Another factor to consider with IPSec VPNs is the level
of management resources required for deployment and
maintenance. All remote or mobile users not at an
aggregation point must have client software on their
remote PC. For organizations trying to provide remote
access to hundreds or thousands of mobile users,
deploying, updating, configuring and managing all of
these clients can be very time consuming and costly. If
remote partners or customers are considered, the
difficulties are multiplied. While a necessary and
appropriate investment for regional, branch and remote
offices where the enterprise needs reliable, “always on”
connectivity and only has to manage a few network VPN
devices, IPSec clients are, in many ways, impractical
investment to meet the needs of mobile/remote workers,
business partners or customers. For example, because
VPN client software is required to connect remote users,
those users are restricted to devices where the software is
installed; i.e., corporate laptops. This does not
accommodate additional methods of access, such as
Internet kiosks, PDA’s, etc. Tunnelling IPSec is the
method usually used in conjunction with a VPN, but has
more overheads as it generally requires that a public key
infrastructure be deployed on the server side. Running
IPSec VPNs provides better authentication security and
user access on almost any application available on the
corporate intranet. Actually, it requires special client
software and more infrastructure overhead than SSL
VPNs. However, SSL VPNs is simpler to deploy but
limits users to browser based applications. Secure back-
end connectivity is enhanced with the addition of IPSec
and VPN as standard features, [13]. SSL VPN was
adopted for secure delivery between a web server and a
user (web browser), and the reason was to eliminate the
complexity and cost of IPSec-based VPN delivery of
applications to remote users. IPSec VPN technology was
designed exclusively for IP networks, and IPSec based
VPNs offers reasonable performance, standard-based
security and application transparency, though it suffers
from technical complexity for both users of the VPN and
administrators of the network infrastructure.
4.0 Problems Associated with IPSec VPN
Internet-based IPSec remote access VPNs require
software on each remote PC that has to be installed
configured and updated for the VPN to work properly
and this can be cumbersome for small companies. Since
IPSec has been approved by all as a standard of security
on the Internet, there is a wide believe that it has all
security solution. IPSec is considered as a solution to all
security concerns. Due to lack of understanding of IPSec
standard, recently people seek other alternative such as
SSL VPNs due to the complexities of setting up and
maintaining IPSec VPNs. It shows that other services like
the SSL VPN are easier to set up and use than IPSec
VPNs.
A growing number of organisations looking for a fast,
secure way to link remote users and business partners are
turning away from traditional IP Security-based VPNs
and toward products and services based on sockets layer
technology. The reason is: Browser-based SSL
alternative require little or no software on remote PCs,
and in most cases any PC with a browser can be used to
make the secure connection, as long as the user can
001907
authenticate to a central server and SSL firewall ports
that the traffic uses are generally left open, so firewall
reconfiguring is usually unnecessary. The idea is that
SSLs simplicity translates into an easier installation and
long term cost savings because of simpler ongoing
support, [14]. Among the problems of IPSec is that is
difficult to distribute client software and to administer,
and the gear requires reconfiguration of employee’s
home firewalls.
Major drawback of IPSec VPNs is the fact that it
provides access to the entire subnet within the corporate
network. This means that the client PC can potentially be
used as a means to enter the network by a hacker and also
if client PC becomes infected with virus or Trojan, it
could potentially spread to the entire network. Once a
client has a tunnel (effectively a “PVC”) into an
organization, this can be a target of hackers (i.e. the
remote access client can effectively be turned into a
router into the organization, unless mitigated by personal
firewalls and/or access controls at the VPN gateway).
Access control can also be an issue with IPSec VPNs
since they rely on network access controls. A VPN
gateway is solely responsible for creating the VPN tunnel
with the client. Once the tunnel is created the information
that passes through is not reviewed for any type of user
rights or permissions. The permissions and the user rights
are governed by whatever controls the network.
The Internet has not been designed to deliver
performance guarantees, the complexities introduced by
VPNs and the requirement to provide QoS have made the
job of the ISPs and systems administrators extremely
difficult, and as today’s network infrastructure continues
to grow, the ability to manage increasing complexity is a
crucial factor for VPN solutions. But, at the same time,
this also opens the possibility for ISPs to sell VPN
services to mostly corporate end users. Because VPN
data networks are a critical element of the overall
business process, VPN must provides a reliable service to
users and their applications.
Many clients do not have versions that run in multiple
operating systems such as UNIX, Linux and Mac.
Interoperability between vendors is virtually non-
existent, meaning VPN clients from one vendor are not
compatible with other VPN appliances. This could be a
problem for users who need to connect to different VPN
sites that use different VPN gateways.
Connectivity can be adversely affected by Network
Address Translation (NAT) or Proxy devices between the
client and gateway as to require client configuration
before the tunnel is established. Interoperability of
different vendors’ IPSec clients to other vendors’ IPSec
servers/gateways is weak, mainly due to configuration
issues. In a large distributed system or inter-domain
environment, the diversified regional security policy
enforcement can create significant problems for end-to
end communication.
One of the biggest drawbacks of IPSec is its complexity.
While IPSec flexibility has contributed to its popularity,
it also leads to confusion and has led to security experts
to state that IPSec contains too many options and too
much flexibility. Much of IPSec flexibility and
complexity may be attributed to the fact that IPSec was
developed via IETF Working groups.
5.0 Conclusion
Although IPSec VPN technology has already addressed
the requirements for site-to-site network connectivity, for
mobile users, they were often too costly, while for
business partners or customers they were impossible to
deploy as they require client software to be installed and
configured on each endpoint. Also security and efficiency
are conflicting requirements for real-time multimedia
applications that impose time constraints on packet
delivery in order to produce the original source of
information whose functionality may be compromised by
security controls. IPSec based VPN requires client VPN
software to be installed on the remote PC or notebook,
which means that kiosk and cyber café, cannot be used
for VPN access and that anyone who is permitted to
access the VPN must either be a corporate owned
notebook. However IPSec VPNs connection comes
through the firewall; it requires reconfiguration of
firewall policies and may require the opening of ports on
the firewall. The opening of ports on the firewall may
present a security breach as it opens up door through
which malicious users can enter the entire network.
IPSec is a versatile and functional protocol for securing
IP. It is not a complete solution for all deployment, and
there are still other issue that may need to be addressed.
Many protocols desired to use IPSec to satisfy their
security concerts and these have different problems that
IPSec addresses incompletely. While IPSec is a solution
to a large problems it is a problem to other protocol for
instance network address translation (NAT). A distinct
advantage of IPSec is that it can be deployed
incrementally, unlike other network related protocols or
technologies; it is not an all-or-nothing proposition [15].
There are some disadvantages, the use of IPSec carries a
cost: additional processing and increased packet size, that
includes IKE traffic that precedes the IPSec protected
communications, as well as the additional information
added to each IPSec protected packet. Despite the above
stated issues, IPSec is currently the most secure VPN
solution available in the market. The ability to dictate the
requirement of current antivirus and firewall software
and to ensure the operating systems are patched virtually
eliminates the risk of malicious intent. The requirements
of VPN client software reduce the risk of security breach.
001908
References
[1] S. Kent and R. Atkinson, “IP encapsulating security
payload,” RFC 2406 (Proposed Standard), Internet
Engineering Task Force, Nov. 1998.
[2] D. Harkins and D. Carrel, “The Internet Key
Exchange (IKE),” RFC 2409 (Proposed Standard),
Internet Engineering Task Force, Nov. 1998.
[3] Thai L. Thuan and Lam Hoang, (2006), Web
services”
http://www.ondotnet.com/pub/a/dotnet/excerpt/netessenti
als2_6/index4.html Date Accessed: 19/12/07
[4] Doraswamy, N., and Harkins, D., 2003, ‘The New
Standard for the Internet, Intranets, and Virtual Private
Networks: Prentice-Hall’ ISBN: 013046189-X
[5] King M. Christopher, (1997), ‘Public key
infrastructure: End-to-end security’ ABI/INFORM
Global: Business Communication Review, Vol. 27, No.
11, pg. 50
[6] Snyder Joel, (2004), ‘Netscreen, Nokia top the
growing field of products that target simplified secure
remote access’ ABI/INFORM Global: Network World,
Vol. 21, No.2; pg. 43
[7] Phifer, L., 2003, ‘Tunnel Visions: How do SSL VPNs
match up with their older cousins?’ Information Security
Magazine: Pgs. 31-43.
[8] Roland, J.F., and Newcomb, M.J., 2003, CSVPN
Certification Guide, CISCO Press
[9] S. Kent and R. Atkinson, “Security architecture for
the internet protocol,” RFC 2401 (Proposed Standard),
Internet Engineering Task Force, Nov. 1998, updated by
RFC 3168
[10] S. Kent and R. Atkinson “IP authentication header,”
RFC 2402 (Proposed Standard), Internet Engineering
Task Force, Nov. 1998.
[11] Niemi, A., “End-to-end web security – protocol
overview”, Department of Computer Science University
of Helsinki, Finland, December 2003
[12] Harding, A., 2003, ‘SSL Virtual Private Networks’,
Computer and Security: Vol. 22, No. 5, Pgs. 416-420
[13] Yager Tom, (2004), ‘Mobility’s mayflower sets sail’
ABI/INFORM Global: InfoWorld, Vol. 26, No. 40; pg.
58
[14] Greene Tim, (2004), ‘Net6 offers twist on remote
access’ ABI/INFORM Global: Network World, Vol. 21,
No. 11; pg. 17
[15] Frankel, S., 2001 ‘Demystifying the IPSec Puzzle’,
Arctech house computer series’ ISBN: 1-58053-079-6
.