TY CO/CM/IF-NIS

Chapter 4: Firewall and Intrusion Detection System

FIREWALL: A network security domain is a contiguous region of a network that operates under a
single, uniform security policy. Whenever domains intersect, there is a potential need for security to
control traffic allowed into the network. Firewall technology can be used to filter this traffic. The most
common boundary where firewalls are applied is between an organization’s internal network and the
internet.

A firewall is a hardware or software device which is configured to permit, deny, or proxy data through
a computer network which has different levels of trust. The most common boundary where firewalls
are applied is between an organization’s internal network and the internet.

Firewall-working: There are two access denial methodologies used by firewalls. A firewall may allow
all traffic through unless it meets certain criteria, or it may deny all traffic unless it meets certain
criteria. The type of criteria used to determine whether traffic should be allowed through varies from
one type of firewall to another. Firewalls may be concerned with the type of traffic, or with source or
destination addresses and ports. They may also use complex rule bases that analyze the application
data to determine if the traffic should be allowed through. How a firewall determines what traffic to
let through depends on which network layer it operates at.

Firewall Interaction with the OSI and TCP/IP Network Models:

Network Firewalls operate at different layers of the OSI and TCP/IP network models. The lowest layer
at which a firewall can operate is the third level which is the network layer for the OSI model and the
Internet Protocol layer for TCP/IP. At this layer a firewall can determine if a packet is from a trusted
source but cannot grant or deny access based on what it contains. Firewalls that operate at the highest
layer, which is the application layer, know a large amount of information including the source and the
packet contents. Therefore, they can be much more selective in granting access.

Firewalls fall into four broad categories:

➢ Packet filters: Packet filtering firewalls operate at the network level of the OSI model or the
IP layer of TCP/IP. In a packet filtering firewall, each packet is compared to a set of rules before it
is forwarded. The firewall can drop the packet, forward it, or send a message to the source.

Page 1 of 13
 TY CO/CM/IF-NIS

➢ Circuit level gateways: Circuit level gateways operate at the session layer of the OSI model,
or the TCP layer of TCP/IP. Circuit level gateways examine each connection setup to ensure that it
follows legitimate TCP handshaking.

➢ Application level gateways: Application level gateways or proxies operate at the application
layer. Packets received or leaving cannot access services for which there is no proxy.

➢ Stateful multilayer inspection firewalls: Stateful multilayer inspection firewalls combine

aspects of the other three types of firewalls. They filter packets at the network layer, determine
whether packets are valid at the session layer, and assess the contents of packets at the application
layer.

Firewall Architectures: There are two classes of firewall architectures:

❖ Single layer: In a single layer architecture, one host is allocated all firewall functions. This
method is usually chosen when either cost is a key factor or if there are only two networks
to connect.
The advantage to this architecture is any changes to the firewall need only to be done at a
single host.
The biggest disadvantage of the single layer approach it provides single entry point. If this
entry point is breached, the entire network becomes vulnerable to an intruder.

❖ Multiple Layers: In multiple layer architecture the firewall functions are distributed among
two or more hosts normally connected in series. This method is more difficult to design
and manage, it is also more costly, but can provide significantly greater security by
diversifying the firewall defense. A common design approach for this type of architecture
using two firewall hosts with a demilitarized network (DMZ) between them separating the
Internet and the internal network. Using this setup traffic between the internal network and
the Internet must pass through two firewalls and the DMZ.

Page 2 of 13
 TY CO/CM/IF-NIS
Firewall Types: After the security requirements are established, a basic architecture is
selected then Firewall functions can be chosen to meet these needs. The following is a detailed
discussion of the 4 firewall categories:

Packet Filtering Firewalls: The first generation of firewall architectures appeared around 1985
and came out of Cisco's IOS software division. These are called packet filter firewalls. Packet
Filtering is usually performed by a router as part of a firewall. A normal router decides where
to direct the data, a packet filtering router decides if it should forward the data at all. Packet
filtering rules can be set on the following: physical network interface the packet arrives on;
source or destination IP address, the type of transport layer (TCP, UDP, ICMP), or the transport
layer source or destination ports. Packet filtering firewalls are low cost, have only a small effect
on the network performance, and do not require client computers to be configured in any
particular way.

However packet filtering firewalls is not considered to be very secure on their own because
they do not understand application layer protocols. Therefore, they cannot make content-based
decisions on the packets, which makes them less secure than application layer and circuit level
firewalls. Another disadvantage of Packet filtering firewalls are they are stateless and do not
retain the state of a connection. They also have very little or no logging capability which makes
it hard to detect if the network is under attack.

Circuit Level Gateways: The second generation of firewall architectures with research in circuit
relays which were called circuit level gateways. Circuit level gateways are used for TCP
connections to observe handshaking between packets to ensure a requested session is
legitimate. Normally, it would store the following information: a unique session identifier, the
state of the connection (i.e., handshake established or closing), sequencing information, source
or destination IP address, and the physical network interface through which the packet arrives
or departs. The firewall then checks to see if the sending host has permission to send to the
destination, and that the receiving host has permission to receive from the sender. If the
connection is acceptable, all packets are routed through the firewall with no more security tests.

Page 3 of 13
 TY CO/CM/IF-NIS

The advantages of circuit level gateways is that they are usually faster than application layer
firewalls because they perform less evaluations and they can also protect a network by blocking
connections between specific Internet sources and internal hosts.
The main disadvantages to circuit level gateways are that they cannot restrict access to protocol
subsets other than TCP and similarly to packet filtering, testing the grant and deny rules can be
difficult which may leave the network vulnerable or incorrectly configured.

Application Level Gateways: The third generation of firewall architectures called Application
level gateways was independently researched and developed during the late 1980s and early
1990s. Application level gateways or proxy firewalls are software applications with two
primary modes (proxy server or proxy client). When a user on a trusted network wants to
connect to a service on an untrusted network such as the Internet, the request is directed to the
proxy server on the firewall. The proxy server pretends to be the real server on the Internet. It
checks the request and decides whether to permit or deny the request based on a set of rules. If
the request is approved, the server passes the request to the proxy client, which contacts the
real server on the Internet. Connections from the Internet are made to the proxy client, which
then passes them on to the proxy server for delivery to the real client. This method ensures that
all incoming connections are always made with the proxy client, while outgoing connections
are always made with the proxy server. Therefore, there is no direct connection between the
trusted and untrusted networks.
The main advantages are that application level gateways can set rules based on high level
protocols, maintain state information about the communications passing through the firewall
server, and can keep detailed activity records.
The main disadvantages are its complex filtering and access control decisions can require
significant computing resources which can cause performance delays and its vulnerability to
operating system and application level bugs.

Page 4 of 13
 TY CO/CM/IF-NIS

Stateful Multilayer Inspection Firewalls: Check Point Software released the first commercial
product based on this fourth generation architecture in 1994 called stateful multilayer
inspection
Firewalls. Stateful multilayer inspection firewalls provide the best security of the four firewall
types by monitoring the data being communicated at application socket or port layer as well as
the protocol and address level to verify that the request is functioning as expected.

An example is if during an FTP session the port numbers being used or an IP address were to
change, the firewall would not permit the connection to continue.
Another advantage is when a specific session is complete; any ports that were being used are
closed. Stateful inspection systems can dynamically open and close ports for each session
which differs from basic packet filtering that leaves ports in a constant opened or closed state.
The main disadvantage to stateful multilayer inspection firewalls is that they can be costly
because they require the purchase of additional hardware and/or software that is not normally
packaged with a network device.

Page 5 of 13
 TY CO/CM/IF-NIS
Firewall Configurations:

Demilitarized zone (DMZ): Systems or sections of a private local network that are made
publicly available and visible from outside networks such as the Internet. Usually, a publicly-
accessible resource (such as an HTTP server) is placed on a DMZ to protect the internal
network while maintaining proper functionality for outside users.
In computer networks, a DMZ (demilitarized zone) is a computer host or small network
inserted as a "neutral zone" between a company's private network and the outside public
network. It prevents outside users from getting direct access to a server that has company data.
A DMZ is an optional and more secure approach to a firewall and effectively acts as a proxy
server as well.

In a typical DMZ configuration for a small company, a separate computer (or host in network
terms) receives requests from users within the private network for access to Web sites or other
companies accessible on the public network. The DMZ host then initiates sessions for these
requests on the public network. However, the DMZ host is not able to initiate a session back
into the private network. It can only forward packets that have already been requested.

Users of the public network outside the company can access only the DMZ host. The DMZ
may typically also have the company's Web pages so these could be served to the outside world.
However, the DMZ provides access to no other company data. In the event that an outside user

Page 6 of 13
 TY CO/CM/IF-NIS
penetrated the DMZ host's security, the Web pages might be corrupted but no other company
information would be exposed.

Intruder: An Intruder is a person that enters in to the area that does not belong to him. The
objective of the Intruder is to gain access to a system or to increase the range of privileges
accessible on a system. There are three classes of intruder:-
❖ Masquerader: An individual who is not authorized to use the computer and who enters
in to system to use a legitimate user’s account.
❖ Misfeasor: A legitimate user who accesses the data, programs or resources from whom
this access is not authorized or who is authorized for such access but misuse his / her
privileges.
❖ Clandestine or Secret User: An individual who hold managerial control of the system
and uses this control to avoid auditing and access controls or to suppress audit logs.
Generally, the masquerader is an outsider, the misfeasor is an insider and the
clandestine user can be an outsider or an insider.
Because of the growing awareness of the intruder’s problem, a number of Computer
Emergency Response Team (CERT) has been established. This team collects information about
system vulnerabilities and distributes it to the systems administrators. In addition to run,
password cracking program the intruders also attempt to modify login software which enables
them to capture user’s passwords. This makes the intruders possible to build up a collection of
password which can be used in future to access the system. An intruder can login to the system
and enjoy all the privileges of the legitimate users using the password. So a system must
maintain password file and protect it. A password file can be protected using encryption &
decryption technique and by providing access control. If both or any one of these in place, then
intruder need to do some extra efforts to learn a password.

IDS (Intrusion Detection System): Intrusion Detection is a process of monitoring the events
occurring in a computer system or network and analyzing them for possible incident like
violation of computer security policies and standard security practices. An IDS provides much
the same purpose as a burglar alarm installed in a house. In CASe of an intrusion, the IDS
system will issue some type of warning or alert. An operator will then tag events of interests

Page 7 of 13
 TY CO/CM/IF-NIS
for in further investigation by the incident handling team. An IDS watches the surrounding
activity and tries to identify undesirable activity. The main purpose of IDS is to identify
suspicious or malicious activity which deviates from normal behavior. IDS are mainly divided
into two categories depending on the monitoring activity:-
a. Host based IDS: This examines activity on an individual system like a mail server, web
server or an individual PC. It is concerned only with the individual system and usually
does not look in to the activity on the n/w or systems around it.
b. N/w based IDS: This examines activity on the network itself. It will monitor only the
traffic across the n/w link and usually does not look in to the activity of individual
systems.
Typically, an IDS will have the following components:

Critical Signature
Files Database

Network Traffic Analysis User

Traffic Collector Engine Interface

Log Alarm Reports

Files Storage

Component1: Traffic collector: This component collects activity or events from the IDS to
examine. On host based IDS the data collected can be log files or traffic coming or leaving the
specific system. On n/w based IDS, the data collected can be traffic across a network link.

Component 2: Analysis Engine: This component examines the collected network traffic and
compares it to known patterns of suspicious or malicious activity stored in the signature
database. The analysis engine acts like a brain of the IDS.

Component 3: Signature database: It is a collection of patterns and definition of known

suspicious or malicious activities.

Component 4: User Interface & Reporting: This is the component that interfaces with the
human element, providing alerts when suitable and allows the user to interact with and operate
the IDS.

Page 8 of 13
 TY CO/CM/IF-NIS

Host based IDS:

Critical Signature
Files Database

Traffic Analysis User

Collector Engine Interface

Log Files Alarm Reports

Storage

Components of the Host based IDS:

The host based IDS check log files and n/w traffic coming in to or leaving a specific post. It
can operate in Real time, looking for activity as it arises or batch mode, looking for activity on
a periodic basis. Typically host based systems are self contained but many new systems are
designed for reporting to and managed by a central system. These systems are also taking local
system resources to operate. Older versions of IDS were designed to operate in batch mode. It
was looking for a suspicious activity on an hourly or daily basis and particular events in the
systems log files. In the new version of host based IDS, processor speed is increased and IDS
looks through the log files in Real time. The ability to examine the data traffic coming and
leaving the host is also added. Some host based IDS have the ability to cover specific
applications by examining the logs produced by that specific applications. They also have the
ability of examining the traffic from the web services.
Host based IDS looks for certain activities in the log file and these activities are:-
1. Logins at odd hours
2. Logins authentication failure
3. Adding new user account
4. Modification or access of critical system files.
5. Modification or removal of binary files.
6. Starting or stopping processes.
7. Privilege modification
8. Use of certain programs.

Network based IDS:

Signature
Database

N/W Traffic Traffic Analysis User Interface

Collector Engine

Alarm Storage Reports

Page 9 of 13
 TY CO/CM/IF-NIS
N/w based IDS focuses on n/w traffic traveling along the cable and wires that interconnect the
system. A n/w IDS should be able to check the n/w traffic and analysis that traffic according
to protocol amount, time, source, destination etc. such an analysis occur quickly and the IDS
must be able to handle the traffic at n/w speed. N/w based IDS are generally deployed so that
they can monitor traffic in & out of an organization major links like internet connection and
remote connection.

Vulnerability assessment
Vulnerability assessment in Intrusion Detection Systems (IDS) is a critical component in the
cybersecurity posture of organizations. This process involves identifying, quantifying, and
prioritizing (or ranking) the vulnerabilities in a system. Here's an overview of how
vulnerability assessment plays a role in IDS and the steps involved:

Purpose of Vulnerability Assessment in IDS

1. Identifying Weak Points: The primary aim is to uncover any weak spots in the system that
could potentially be exploited by attackers. These vulnerabilities might exist in software,
hardware, or configurations.

2. Improving Security Posture: By understanding where vulnerabilities lie, organizations can

take proactive measures to strengthen their defenses, thus enhancing their overall security
posture.

3. Compliance and Regulations: Many industries require regular vulnerability assessments as

part of their compliance standards. Identifying and mitigating vulnerabilities helps in
adhering to these regulations.

Steps in Vulnerability Assessment for IDS

1. Asset Inventory: The first step is to compile a comprehensive list of all assets within the
system. This includes hardware, software, and any network infrastructure components.

2. Vulnerability Scanning: Using specialized tools, the system is scanned for known
vulnerabilities. These tools can range from basic network scanners to more advanced
vulnerability assessment software that can detect intricate software flaws.

3. Vulnerability Analysis: Once vulnerabilities are identified, they are analyzed to determine
their severity and potential impact on the system. This analysis considers the exploitability
of the vulnerability, the complexity of an attack, and the potential damage or data loss.

4. Risk Assessment: This involves assessing the risks associated with each vulnerability,
considering the likelihood of exploitation and the potential impact. This step helps in
prioritizing the vulnerabilities for remediation.

5. Mitigation and Remediation: Based on the priority, appropriate measures are taken to
mitigate or remediate the vulnerabilities. This could involve patching software, changing
configurations, or even replacing vulnerable hardware.

6. Reporting and Documentation: Finally, a detailed report is produced outlining the

vulnerabilities found, their severity, and the actions taken to address them. This
documentation is vital for audit purposes and for planning future assessments.

Page 10 of 13
 TY CO/CM/IF-NIS
Misuse detection:
Misuse detection, also known as signature-based detection, is a method used in Intrusion
Detection Systems (IDS) and other security mechanisms to identify known threats and
malicious activities within a network or system. It relies on a database of known attack patterns
or signatures, which are essentially detailed descriptions of previously identified threats. Here's
an overview of how misuse detection works and its key aspects:

How Misuse Detection Works

1. Signature Database: At the core of misuse detection is a comprehensive database that
contains the signatures of known threats. These signatures can include specific byte
sequences in network traffic, known malicious payload structures, or particular
behaviors indicative of an attack.
2. Monitoring and Analysis: The IDS continuously monitors network traffic or system
activities, analyzing them for patterns that match any of the signatures in the database.
3. Detection: When a match is found, the system flags the activity as a potential threat.
The action taken can vary from logging the event for later review, sending alerts to
administrators, or automatically blocking the activity.
4. Update Mechanism: For the detection to be effective against new threats, the signature
database must be regularly updated with new signatures as they are discovered.

Advantages of Misuse Detection

1. High Accuracy for Known Threats: Misuse detection is highly effective at
identifying and mitigating known threats with a low rate of false positives for those
specific signatures.
2. Straightforward Implementation: The concept behind misuse detection is relatively
simple, making it easier to implement and maintain compared to more complex
detection methodologies.
3. Immediate Response: Since this method relies on known patterns, it allows for
immediate action against recognized threats without the need for extensive analysis.

Limitations
1. Ineffective Against New or Modified Threats: Misuse detection cannot identify new
threats or even slightly modified versions of known threats that do not match existing
signatures.
2. Regular Updates Required: The signature database requires regular updates to remain
effective. This dependency means that the system is always playing catch-up with threat
actors.
3. Resource Intensive: Maintaining an extensive signature database and continuously
scanning for matches can be resource-intensive, potentially impacting system
performance.

Anomaly detection:
Anomaly detection, refers to the process of identifying patterns in network or system behaviour
that deviate from what is considered normal. This method relies on establishing a baseline of
regular activity and then using statistical models, machine learning, or other analytical
techniques to detect deviations from this baseline. Anomaly detection is particularly useful for
identifying previously unknown threats or sophisticated attacks that may not have a defined
signature.

Page 11 of 13
 TY CO/CM/IF-NIS
How Anomaly Detection Works
1. Establishing a Baseline: The first step is to establish what normal behavior looks like
within a network or system. This involves analyzing traffic patterns, system
performance metrics, user behavior, and other relevant data over time.
2. Continuous Monitoring and Analysis: Once a baseline is established, the system
continuously monitors for activity that deviates from this norm. This can involve
complex statistical analysis, machine learning models, or other data analysis techniques
to identify outliers.
3. Alerting and Response: When an anomaly is detected, the system can alert security
personnel to the potential threat. Depending on the configuration, the system may also
take automated actions, such as blocking suspicious traffic.
4. Tuning and Optimization: Anomaly detection systems require ongoing tuning to
minimize false positives and negatives. This involves adjusting the sensitivity of the
detection algorithms and refining the baseline model as the network or system evolves.

Advantages of Anomaly Detection

1. Detection of Unknown Threats: Anomaly detection can identify new or sophisticated
threats that do not match any known signatures or patterns.
2. Adaptability: Since it is based on behavior rather than signatures, anomaly detection
can adapt to changes in the network environment and emerging threat landscapes.
3. Comprehensive Coverage: This method can potentially identify a wide range of
threats, from network intrusions and malware to insider threats and data exfiltration.

Honeypots:
Honeypots are a cybersecurity strategy designed to act as decoys for detecting, deflecting, or
studying hacking attempts and other malicious activities. They are intentionally vulnerable
systems, applications, or data that appear to be part of a network's infrastructure but are actually
isolated and monitored closely. By mimicking real systems that contain no actual production
data, honeypots aim to lure attackers away from valuable targets and gather intelligence about
their tactics, techniques, and procedures. Here's a detailed look at honeypots and their
significance in cybersecurity:

Types of Honeypots
1. Low-Interaction Honeypots: These are simple systems that simulate only the services
frequently scanned by attackers. They are relatively easy to deploy and maintain but
offer limited interaction, capturing basic information about potential threats.
2. High-Interaction Honeypots: These are complex systems that simulate a real
operating environment, allowing attackers to engage with the system as they would
with a genuine target. High-interaction honeypots can collect more detailed information
but are more challenging to set up and maintain.
3. Research Honeypots: Deployed by educational or research institutions to gather
information about the motivations and tactics of attackers, which helps in understanding
emerging threats and improving cybersecurity measures.
4. Production Honeypots: Used within actual business environments to help protect
resources while also gathering intelligence on attacks targeting the organization.

Advantages of Honeypots
1. Threat Intelligence: Honeypots provide valuable insights into attack methods, helping
organizations to understand current threats and tailor their defenses accordingly.
2. Waste Attackers' Time: They can divert attackers from legitimate targets, wasting
their time and resources on systems that offer no value.

Page 12 of 13
 TY CO/CM/IF-NIS
3. Early Warning: Honeypots can serve as an early warning system for new types of
attacks or an increase in hacking activities targeting specific vulnerabilities.

Limitations and Challenges

1. Risk of Being Identified: Skilled attackers might recognize a honeypot and either
avoid it or use it to feed false information to the monitoring team.
2. Resource Intensive: High-interaction honeypots, in particular, require significant
resources to set up and maintain, as they need to be convincing enough to deceive
sophisticated attackers.
3. Legal and Ethical Considerations: There are potential legal and ethical issues
regarding the data collected from honeypots, especially concerning privacy laws and
regulations.

Page 13 of 13