Professional Documents
Culture Documents
Exploring Sec Strat For 3wayhandshake
Exploring Sec Strat For 3wayhandshake
Exploring Sec Strat For 3wayhandshake
Dissertation Manuscript
W
Requirements for the Degree of
DOCTOR OF PHILOSOPHY
IE
by
EV
Deember 2023
Approval Page
By
W
Efosa Osayamwen
Dissertation Chair: INSERT NAME Degree Held Date
IE PhD 01/21/2024 | 10:52:30 MST
INSERT
Committee Member: Milton NAME Degree Held
Kabia Date
Abstract
Due to TCP/IP network protocol vulnerabilities, various risks reside with the 3-way handshake,
affecting the protocol's inability to establish a secure connection. The problem addressed in this
study focuses on vulnerabilities with the 3-way handshake, which could lead to various attacks
against system confidentiality, integrity, and availability. This situation poses a global challenge
for organizations and users, which could cause data breaches, system compromise, and a denial
of service. The purpose of this qualitative research methodology and design was to propose a
strategy to address vulnerabilities found within the 3-way handshake. The systems theory was
W
the theoretical framework used in this study. The researcher used the grounded theory research
design and employed network security experts with 5-7 years of experience to participate in
IE
surveys as a research instrument. The security of the 3-way handshake needs to be improved, as
network security. The results indicate the necessity for increased security with the conventional
3-way handshake to meet the demands of modern network environments. The implications
drawn from the survey results indicate that inherent weaknesses within the 3-way handshake
could cause information systems to be susceptible to ongoing attacks. The Internet Corporation
for Assigned Names and Numbers should adopt a secure 3S Handshake featuring a dedicated
security flag for verification. Additional verification flags should be included in the 3-way
i
Acknowledgements
I want to take this opportunity to express my deep gratitude and appreciation to the
individuals and organizations who have contributed to the completion of my dissertation. Their
academic journey.
First and foremost, I want to thank the Elumogo family and close friends, your
unwavering support and understanding during this journey have been my pillar of strength. Your
encouragement motivated me, and I am profoundly grateful for your love and patience.
W
I extend my heartfelt thanks to my academic advisor, Dr. Efosa Osayamwen, for his
unwavering support, mentorship, and the countless hours he dedicated to refining this research.
IE
His expertise and insights significantly enhanced the quality of my work.
I also want to thank Dr. Brian Aunkst, my Academic Success Center coach who
EV
supported me throughout the dissertation process, helping me learn and develop my written
communication skills. His collective wisdom has been instrumental in shaping my dissertation.
PR
I would like to recognize National University and the School of Technology and
Engineering for providing access to invaluable resources and research facilities that greatly
facilitated my work.
organizations, and I am truly fortunate to have had their support. Thank you all for being a part
ii
Table of Contents
W
Documentation ........................................................................................................................ 11
Theoretical Framework ........................................................................................................... 12
Cloud Security ........................................................................................................................ 15
Network Security .................................................................................................................... 17
IE
Software Security .................................................................................................................... 33
Summary ................................................................................................................................. 43
EV
Chapter 3: Research Method ......................................................................................................... 45
Implications............................................................................................................................. 82
iii
Recommendations for Practice ............................................................................................... 86
Recommendations for Future Research .................................................................................. 89
Conclusions ............................................................................................................................. 90
References ..................................................................................................................................... 92
W
IE
EV
PR
iv
List of Tables
Table 1 Gender Distribution Among Survey Participants ........................................................... 64
Table 2 Job Titles of Survey Participants .................................................................................... 64
Table 3 Primary Industries of Survey Participants ..................................................................... 65
Table 4 Educational Background of Survey Participants ............................................................ 65
Table 5 Network Security Certifications of Survey Participants ................................................. 66
Table 6 Frequency of Working with Network Security Protocols ................................................ 66
Table 7 Security Perception of the 3-Way Handshake for Trusted Connection .......................... 67
Table 8 Perspectives on Verification in the 3-Way Handshake ................................................... 67
Table 9 Perspectives on Improving the 3-Way Handshake.......................................................... 68
Table 10 Security Control Measures for Enhancing the 3-Way Handshake Design ................... 68
Table 11 Demographic Information of Survey Participants ........................................................ 69
Table 12 Security Control Measures for Enhancing the 3-Way Handshake Design ................... 71
W
IE
EV
PR
v
List of Figures
Figure 1 Security Perception of the 3-Way Handshake for Trusted Connection ........................ 70
Figure 2 Perspectives on Verification in the 3-Way Handshake ................................................. 73
Figure 3 Perspectives on Improving the 3-Way Handshake ........................................................ 74
Figure 4 3S Handshake Source Code to Improve the 3-Way Handshake .................................... 75
Figure 5 Shell Output - 3-Way Handshake Simulation with SEC Flag ....................................... 77
W
IE
EV
PR
vi
1
Chapter 1: Introduction
(TCP/IP) suite and provides a process to establish a network connection between devices at the
transport layer of the open systems intercommunication (OSI) model (Amponis et al., 2023).
Communication is crucial for network interoperability. The 3-way handshake ensures that the
client and the server know each other's intentions, capabilities, and readiness to establish a
reliable and synchronized connection. Networking systems know how to interpret and
understand the rules of the 3-way handshake through the network protocol in the networking
W
stack (Iren, Amer, & Conrad, 1999). Network security is vital to protect the networks that make
handshake. This security process determines who should be granted or denied requests for data,
EV
applications, and resources based on access privileges. Authentication and authorization are
concepts that work together and are applied to ensure controlled access to resources. With
PR
authentication, users, systems, or applications must prove their identity with valid credentials,
i.e., usernames, passwords, biometrics, and smart cards. With authorization based on identity that
has been verified, they will receive access rights to specific resources (Kizza & Kizza, 2008).
Technical controls often follow access control policy standards (i.e., NIST 800-162), which are
sets of guidelines and rules that organizations follow to implement access control in their
consistent, effective, and secure access control measures to protect sensitive information from
This research topic was relevant because the 3-way handshake provides host access to
2
web servers via the internet. The handshake communication process is essential for computer
systems to connect and transfer data. Application messages transmit segments of data to the
network layer. If the handshake process is not successful, the connections will drop. TCP is part
of the 3-way handshake, and it is responsible for data integrity with flow and error control to
ensure that messages are delivered. The 3-way handshake establishes the rules and standard
procedures for communicating information online. Without it, users would be unable to access
The internet comprises multiple networks, trusted and untrusted, which connect
W
computers worldwide (Townes, 2012). The author explained the vulnerabilities associated with
the 3-way handshake (Kovacs, 2022). Hackers are often anonymous and take advantage of
IE
network access with malicious software and attacks (Krishna et al., 2022). Malicious activity
should be stopped at the transport layer with the 3-way handshake (Heinz et al., 2021). Common
EV
attacks include Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks from
the internet (Zakwan et al., 2023). This leads to a server overloading and, eventually a system
PR
crash. If one layer is hacked, the other layers are unaware of the hack and the entire
Kupreev et al. (2020) stated how DDoS attacks have affected companies in 2022 with
quarter trends:
In Q2 2022, DDoS attacks dropped by 13.72% (to 78, 558) as compared to the previous
reporting period. Activity increased steadily throughout the quarter: from 731 attacks per
It is vital to address vulnerabilities with the TCP/IP protocol due to the impact this attack
has on users and organizations. Many organizations become victims of attack vectors due to
3
system. The 3-way handshake could cause systems to be vulnerable to malware to disrupt,
damage, or gain unauthorized access to a computer system. Software must be improved with
secure coding practices for developing technologies designed with security in the requirements
phase (Jayachandra et al., 2018). The TCP/IP protocol stack has over 100 vulnerabilities
exploited with remote code execution for DoS and other attacks (Kovacs, 2022). The problem
with the 3-way handshake is its design as well as its implementation. Significant vulnerabilities
put users and organizations at risk for system compromise and data breaches. The 3-way
W
handshake verifies device connectivity but does not include security checks for authorization and
The problem addressed in this study was the 3-way handshake vulnerabilities found in
EV
the TCP/IP network connection. i.e., Client-side security may be compromised if the intrusion
prevention system (IPS) only scans all incoming packets. The software must distinguish when
PR
the host acts as a client and acts like a server when sending SYN/ACK packets. Adversaries can
exploit these vulnerabilities, which is called the split handshake attack. This type of attack is
example of one of many that affect the 3-way handshake (Kak, 2022).
The TCP/IP protocol stack has over 100 vulnerabilities exploited with remote code
execution for DoS and other attacks (Kovacs, 2022). This attack could put users at risk,
especially if a firewall is not configured correctly for security (Fu-Hau, 2016). DDoS attacks
have caused significant shutdowns for companies running critical services connected to
networks. This service attack sends massive SYN requests to a server to overwhelm it with open
The design and implementation of the 3-way handshake does not include enforced
security controls for identity and access management. The 3-way handshake standard remains
the same as a legacy standard. The 4-way handshake provides Wi-fi security to encrypt data in
transit with asymmetric keys and other rulesets for authentication but does not fix the security
problems associated with the 3-way handshake (Singh et al., 2020). Users and organizations that
use the internet are at risk of cyber-attacks due to the vulnerabilities found within the 3-way
handshake. Connecting to untrusted systems and networks will expose entities to malware,
service failures, and zero-day attacks (Kumar & Paul, 2018). Security flags with authentication
W
are needed and must be known about with the 3-way handshake protocol. The potential negative
consequences will remain with attack vectors if this study does not address the problem.
IE
Purpose of the Proposal
The purpose of this qualitative exploratory research methodology and grounded theory
EV
design was to propose a strategy to address the vulnerabilities found within the 3-way handshake
(Kak, 2022). The overarching goal is to develop a simulated program to improve the protocol
PR
design. Then, the design will be sent to the Internet Corporation for Assigned Names and
Numbers (ICANN) for consideration to improve the 3-way handshake. Rulesets from existing
protocols and whitepapers were researched as a reference standard. Qualitative data were
collected from security experts and multiple literature reviews. Data analysis occurred to
improve the 3-way handshake for developing a simulation. The next step will be to develop and
demonstrate risk-driven innovation to prove the interest of investors and encourage them to
invest money in the 3Secure (3S) handshake project. The target population for this investigation
comprised expert network security professionals, and the sample size was determined based on
specific criteria and qualifications of individuals selected through LinkedIn. The primary
5
objective of this research was to provide valuable insights into enhancing the security of
In this research, the system theory was used to explain and find solutions for complex
systems. The theoretical framework was based on the systems theory, which states that any
(Lalande et al., 2020). The systems theory helped identify how the 3S handshake works to
W
accomplish a successful outcome. This protocol included open system software since it will be
affected by its environment with inputs, throughputs, and outputs. Other TCP/IP suite functions
IE
and the networks and computer devices affected are part of this system (Dostálek & Kabelová,
2006). Applying this system theory to understand the distinct parts that interact to form a
EV
complex system is essential (Keating et al., 2018). Throughout the development process,
negative and positive feedback loops will be monitored to make necessary changes to locate
PR
problems that need to be corrected. This framework helped guide the research decisions,
including developing the problem statement, purpose statement, and research questions.
Using the systems theory provided practical strategies to improve the 3-way handshake
with trusted networks while considering the interconnectivity within digital ecosystems. This
researcher used the systems theory to understand the interactions and interdependencies between
the internet and its users and organizations (Lalande et al., 2018). The system theory was applied
in this research to address the network security problem with the 3-way handshake with a
proposed strategy of a 3S handshake process with security verification. The researcher aimed to
understand how the current 3-way handshake connection affected organizations and how the
6
proposed strategy improved network communications for users and organizations. The
researcher also explored implementing the 3S handshake within the context of the systems
theory.
The methodology for this research was a qualitative approach, as it allows for an in-depth
exploration of the limitations and vulnerabilities of the current 3-way handshake connection and
how it affects organizations. This approach was also helpful in understanding participants'
W
Additionally, a research design was chosen to examine this phenomenon more thoroughly within
its real-life context. These methods provided further insight into what is happening and why
IE
these connections are so vulnerable to attack or misuse by those with malicious intent (Bhangu et
al., 2022). Therefore, utilizing qualitative research methods helped users gain invaluable
EV
knowledge on how best to protect their information systems from potential threats while still
The data collection procedure for this research combines surveys from security experts
with literature reviews (Bhangu et al., 2022). A survey of security experts provided valuable
insights into their experiences and perceptions regarding the current 3-way handshake
connection, as well as their opinions on improving the protocol's design. Document analysis,
meanwhile, gathered information from relevant literature, such as articles and research studies to
understand better any limitations or vulnerabilities associated with the existing 3-way system. To
analyze this data effectively, a thematic analysis was used to identify themes and patterns to gain
approach provided an effective way for researchers to collect reliable qualitative data that helped
7
inform new strategies for improving network security protocols like those related to handshaking
In summary, a qualitative research methodology was ideal for improving the 3-way
handshake for users and organizations (Bhangu et al., 2022). Through the development of a 3S
handshake process with a security flag for verification, the researcher will provide an effective
way to address current vulnerabilities in 3-way handshake connections. The survey of security
experts, document analysis, and thematic analysis were employed during the data collection
W
while also offering potential strategies to address these issues. This approach was invaluable in
In this research, the author sought to explore the limitations and vulnerabilities of the
EV
research was employed to collect in-depth evidence to comprehensively understand this issue.
PR
RQ1
How secure is the 3-way handshake for establishing client-server connections via the TCP/IP
network?
RQ2
What are the security requirements for constructing a new security protocol to protect against
untrusted networks?
RQ3
How can a security flag for verification improve network security for users and organizations?
8
This proposed research is important in its potential to revolutionize network security for
organizations by developing a 3S handshake process with security flags for verification. In this
study, the researcher explained the issues with 3-way handshake connections, which have been
proven vulnerable, as well as their effect on organizations (Kak, 2022). Researching possible
strategies offered guidance that will help protect users and companies from cyber threats.
W
continues to advance, so does the need for secure communication systems, thus making this
particular research invaluable in providing a practical answer tailored to specific problems within
IE
the field of cybersecurity (Chaudhary et al., 2022). The implications are clear: if implemented
effectively, these strategies could lead us into an era where individuals’ network is better
EV
This proposed research was essential for advancing network security, particularly for
PR
organizations. By examining the current limitations and vulnerabilities of the 3-way handshake
connection, the research provided a unique opportunity to examine how these issues can be
addressed with proposed strategies that apply to real-world situations. Moreover, it contributed
significantly to the existing literature on internet management security and user-level security,
ultimately provided users with more secure online experiences. Therefore, this research was
prioritized due to its potential implications for improving network security and overall cyber
Cyber Security
Cybersecurity - the practice of protecting critical systems and sensitive information from
Internet
W
communication protocols (Oxford University Press, 2006).
Examples of ISPs are Comcast. AT& T. Time Warner Cable (Mohn et al., 2018).
EV
Malware
Malware is intrusive software that is designed to damage and destroy computers and
PR
Synchronize (SYN)
connection be established between them. If the second machine receives the SYN, an
SYN/ACK is sent back to the address requested by the SYN (Kabelova & Dostalek,
2006).
Summary
research and its objectives. The problem to be addressed in this study was the 3-way handshake
vulnerabilities found in the TCP/IP network connection. The purpose of this qualitative research
methodology and design was to propose a strategy to address the vulnerabilities found within the
3-way handshake, which significantly impacts network security for users and organizations. The
W
researcher aimed to create a 3S handshake process with security flags for verification to increase
network security and reduce the risk associated with online communication. By exploring how
IE
authentication can be improved upon existing protocols, this research will likely lead to more
secure communication systems that protect individual users and larger organizations from
EV
malicious actors. In this research, the system theory was applied in assessing the limitations and
Lastly, in this chapter the researcher defined several key terms related to cyber security, such as
the 3-way handshake connection and 3S handshake process with a security flag verification.
11
The purpose of this research was to present a strategy to improve the 3-way handshake
for secure, trusted network communications. The problem addressed in this study was the 3-way
handshake vulnerabilities found in the TCP/IP network connection (Kak, 2022). The study
included a literature review of related topic areas to improve the scope and design of the 3-way
handshake to include cloud security, network security, and software. In this literature review, the
researcher used the National University Library and Google Scholar search engines to identify
W
Documentation
A systematic approach was employed to search for and identify relevant literature to
IE
support the development of a strategy for the TCP/IP 3-way handshake. A comprehensive set of
studies, articles, and other sources of information were reviewed to help address the research
EV
questions related to the study. The search parameters included themes such as the 3-way
handshake, TCP/IP suite, OSI model, cloud security, network security, software security, and
PR
system theory, utilizing Boolean operators. An extensive review was conducted, focusing on
literary sources published within the past five years, including academic sources such as books,
journal articles, conferences, and expert reports. To document emerging scientific knowledge,
both practical and theoretical ideas were considered and evaluated. Keyword searches were
conducted using Boolean operators to refine the focus of the results. The phrases used included
"TCP handshake AND security AND performance," "network AND security," "network
Theoretical Framework
In this section, the researcher examined system theories to build, support, and provide a
structure for a secure network communication protocol to replace the 3-way handshake.
Strategies were used to help develop a secure protocol architecture. Several research theories
were applied to the TCP segment structure to analyze patterns from available data. Key concepts
were identified to prevent system compromise. Recent theories and recommendations were used
for information security research. The system theory was a theoretical framework used to
analyze complex systems and understand the relationships and interactions among its
W
components. The concept of this framework was relevant to improve the system components that
make up the 3-way handshake to enforce security control measures. In the context of network
IE
communication protocols, system theory can be applied as a framework to identify and analyze
the various components and their interactions that contribute to the overall performance and
EV
Ludwig von Bertalanffy was the originator of the system theory. He focused on the idea
PR
of integrating various levels of science. He developed the general system theory in biology,
establish theoretical disciplines. Other philosophers like Paul Weiss helped evolve this topic in
his Ph.D. thesis to study animal behavior. At the same time, Bertalanffy applied system thinking
on several levels to study areas from molecules to ecosystems in biology to identify and
The system theory breaks down the rules and patterns of interacting components within a
system to outline basic laws that can be applied virtually to every scientific field. Bertalanffy
proposed this theory as general framework to analyze systems logically (Drack, 2009). Lalande
13
(2023) used this theory to study social systems to uncover patterns with psychological outcomes.
The researcher broke down a complex system into parts instead of observing a system in
isolation. Interconnected system designs have elements that work together in a flow of
Systems theory was applied when the first electronic computers were developed. It
provided a framework for understanding the behavior of complex systems. This theory enables
engineers to design and optimize computer systems more effectively. Claude Shannon (1948)
W
information theory, a key component in modern computer science. It was used to analyze the
foundation for the research study. The grounded theory is used to conduct qualitative research
EV
for theory development to support both communication protocols. This theory was developed by
Barney and Anselm (1965). It is significant because the authors provided explicit guidelines for
PR
conducting quality research. It offers specific strategies for handling the analytic phases of
inquiry, streamlines and integrates data collection and analysis, advances theoretical analysis of
qualitative data, and legitimizes qualitative research as a scientific inquiry (Turner & Astin,
2021). The grounded research theory has been previously used in education and health research.
The grounded and systems theories are similar in their focus and approach. The system theory
can be used to understand and analyze complex systems, while the grounded theory can support
the analysis approach to generate new theories from empirical data, which is why the systems
There were frameworks with a different viewpoint than the chosen one, which could have
been used to examine the problem. The diffusion of innovations theory describes how new ideas,
practices, or products spread throughout societies and cultures for widespread adoption. This
theory was developed by E.M Rogers (1962) to understand how innovations were adopted over
time by the leading players in the view, including innovators, early adopters, early majority, late
Different societies are likely to have different adoption rates. The innovation and systems
theories are distinct approaches to understanding the relationship between technology and
W
society. The innovation theory focuses on creating and diffusing new ideas and technologies to
understand how new technologies and ideas emerge, diffuse, and transform society. In contrast,
IE
the system theory is only used to understand and analyze complex components to improve
system functionality.
EV
David Clark (1970) is a computer scientist who developed the TCP/IP 3-way handshake
PR
and a team of software engineers at the Massachusetts Institute of Technology (MIT). The
TCP/IP protocol suite was developed to establish a reliable connection between two devices over
a network (Clark, 1982). The 3-way handshake has strengths that include reliability, security,
and compatibility. Reliability ensures that a reliable connection is established between two
devices before data is transmitted to reduce the risk of data loss or corruption. Security helps
prevent unauthorized access to a network by verifying the identity of the connecting devices.
Compatibility is included for most networked devices. Weaknesses of the 3-way handshake
includes system overhead, vulnerability attacks, delay, and firewall issues. Extra overhead is
added to the data transmission process, which slows down performance. The 3-way handshake is
15
vulnerable to attacks like SYN flooding, which can overwhelm a server with fake connection
requests to cause a crash. Delays occur before a connection is established, which can be a
problem for time-sensitive applications. Firewall issues and other security measures can block
the 3-way handshake to prevent a connection from being established (Shaikh & Shehzad, 2018).
The 3-way handshake has undergone some minor changes and improvement over time
since it was first developed to include TCP Fast Open, an extension of the TCP protocol that
allows data to be exchanged during the initial SYN packet to reduce the latency of the 3-way
handshake, SYN Cookies, a technique to help prevent SYN flooding attacks by modifying the
W
way that servers respond to incoming SYN packets, Explicit Congestion Notification, a feature
that allows routers to notify TCP endpoints of impending network congestion and Multipath TCP
IE
an extension of the TCP protocol that allows data to be transmitted over multiple paths
Cloud Security
Oncioiu et al. (2018) suggested a solution to prevent SYN Flood attacks from affecting
systems in a cloud environment. This Denial of Service (DoS) attack exploits the 3-way
handshake by overwhelming a server with SYN requests (Gülşen et al., 2021). Vint Cerf and
Bob Kahn (1970) are computer scientists that designed the TCP/IP suite that contains the 3-way
handshake (Huitema, 2014). The 3-way handshake was developed to establish a reliable,
ordered, error-checked data transfer between two devices over a network. The protocol consists
of three steps: The initiating device sends an SYN packet to the receiving device, indicating it
wants to establish a connection. The receiving device responds with an SYN-ACK packet,
indicating that it received the SYN packet and is willing to establish a connection. The initiating
16
device responds with an ACK packet, indicating that it received the SYN-ACK packet and is
ready to transmit data (Nandi et al., 2012). Oncioiu et al. (2018) described limitations to firewalls
and Intrusion Detection Systems (IDS), which provide a layer of security but do not safeguard
systems against all attacks. The authors proposed a threshold-based mechanism that blocks
connections from IP addresses that exceed a certain threshold. This proposed solution effectively
prevents DoS attacks while allowing legitimate connections to be established. In this paper,
simulation experiments are used to evaluate the effectiveness of their solution utilizing a
W
Encrypted Traffic Analytic using Identity Based Encryption
There are limitations in cloud computing as it pertains to encrypted traffic and analytics.
IE
Malware embedded in encrypted data poses new challenges for network threat detection systems.
Alornyo et al. (2018) proposed a new method for encrypted traffic analysis that uses Identity-
EV
Based Encryption (IBE) with an Equality Test (ET) for cloud computing environments. IBE is a
type of public-key encryption that allows an individual to use an easily remembered identifier,
PR
such as an email address or a name, as their public key. It was first proposed by Adi Shamir
(1984) and later developed by Dan Boneh and Matt Franklin in the early 2000s. Alornyo et al.
(2018) conducted a simulation experiment using quantitative evidence with a real-world dataset.
The results show that his proposed approach can provide accurate and efficient encrypted traffic
Zhao et al. (2022) proposed a new approach to zero trust access authorization and control
of network boundary using cloud sea big data fuzzy clustering. The authors argued that
traditional security measures are insufficient for protecting networks against increasingly
Reproduced with permission of copyright owner. Further reproduction prohibited without permission.