Chapter 7

System Security
1. System architecture and overall system security :
System architecture :

Comprehensive security is vital to safeguard against

various threats, including communication and system
security. This approach ensures the adequate
protection of sensitive data and information.

Approaches to strengthen system security :

 Remove bugs and vulnerabilities in the operating system : New versions and security updates
 Add security features to the OS and microprocessor : Privilege levels, NX, ASLR
 Monitoring system security : Antivirus tools, Firewall, intrusion detection
 Virtualisation technology : Protecting processes by separating virtual machines
 Trusted calculation, e.g : Secure boot with UEFI, Secure hardware on the platform, e.g. TPM (Trusted Platform
Module)
I. Continuous Security update :
Continuous security updates are crucial to address new vulnerabilities, but patching can take time, leaving software
vulnerable. This is especially true for zero-day vulnerabilities that leave manufacturers with no time to fix before
being exploited.
II. Excecutable Files :
An executable file is a type of file that contains instructions that can be executed by the CPU. These instructions
(consists of “dead bytes”) are written in machine code, which is typically in binary form (if compiled). Executable files
are used to run both benign programs and malware on a system. They must follow a specific format depending on
the operating system. For instance, Windows uses the Portable Executable (PE) format for its executable files. There
are various tools available to inspect different aspects of executable files, such as strings, winhex, PEview, IDA, and
more. These tools can be used to examine the binary code, identify potential vulnerabilities, and reverse-engineer
the malware.
2. Processes and memory - buffer overflow :
I. Processes :

When an application or user starts an executable file, a new process is created. Part of the memory is allocated to
this process , and relevant parts of the executable file are loaded into memory. The instructions in the file are
executed in the CPU. Task manager provides information on processes running on a system, while Sysinternals suite
for Windows (10/11) provides more advanced tools to monitor processes. For example, Process Explorer provides
detailed information on running processes, and Process Monitor provides information on how different processes
communicate with the system. These tools are useful for analyzing and troubleshooting system performance issues
and identifying any suspicious processes or activities.

II. Virtual memory for a process :

Each process has its own virtual memory area that is translated by the OS into physical memory addresses before
being read or written to system memory. While all processes share the same virtual address space, they have
Page 1 of 4
separate physical address spaces. This approach ensures that a process can only access its own physical address
space through virtual to physical memory address translation and not the physical addresses of other processes. This
adds an extra layer of security and helps prevent processes from interfering with one another or accessing each
other's memory areas.

III. Buffer overflow :

A buffer overflow is a classic vulnerability that can allow an attacker to execute their own code on a system. This type
of vulnerability has been used in many high-profile attacks, such as the Morris worm and the SQL slammer. The
vulnerability works by overwriting memory, which can then cause the attacker's code to be executed with the same
privileges as the original application. One common way to achieve a buffer overflow is through a stack-based version.

When a function or method is called, a new

frame is created on the stack. Each frame
contains information such as the function's
arguments, local variables, and other data. It
also includes a frame register that contains
the address of where the frame begins, and a
return address that specifies the next
instruction to execute when the function is
finished. When a buffer overflow occurs, the
attacker can overwrite the return address with
a value that points to their own code,
elsewhere. When the function finishes
executing, the CPU jumps to the attacker's code. To do this, the attacker needs to know the
exact location of their code, which they often achieve by inserting a series of NOP
instructions before their code. This allows the attacker to execute their own code with the
same privileges as the original program.
IV. Buffer overflow - countermeasures :

Measures to prevent buffer overflow attacks include using the No Execute (NX) policy to prevent code from being run
from the stack, implementing stack canaries to detect overwriting of data, and using Address Space Layout
Randomization (ASLR) to make it harder for attackers to find the address of attack code in memory. Safer
programming languages, static analysis, and embedded security features in code are also helpful in preventing
buffer overflow attacks.

3. Level of privilege :
The OS privilege levels refer to different levels of access and control that the operating system has over system
resources. The X86 CPU architecture introduced hierarchical privilege levels in 1985, with four levels ranging from
level 0 (highest) to level 3 (lowest). Higher privilege levels have access to more system resources while lower levels
have restricted access. A new level, -1, was introduced around 2006. ‘’ Rings of protection ‘’ :

Principle for the use of the privilege levels :

It restricts processes from accessing data and software at higher privilege levels than they are authorized to access. A
process can only access and modify data and software at the same or lower privilege level as itself. A process running
Page 2 of 4
in kernel mode (level 0) has access to data and software at levels 0, 1, 2, and 3, but not at level -1. This hierarchical
structure is designed to prevent unauthorized access to sensitive system resources. However, attackers aim to gain
access to the kernel or hypervisor mode, often through exploits or malware that tricks users into installing them.

4. Virtualisation architectures :
A technique used to increase and improve the separation of processes to
enhance security. A virtual machine, which acts as a software emulation of
physical hardware, is created by a hypervisor. Virtualization has various
uses, such as in cloud computing where cloud providers operate server
farms and provide each customer with their own VM to increase/decrease
capacity. It is also useful in testing and software analysis, as potentially
harmful experiments can be safely carried out in an isolated environment.
Virtualization allows taking a snapshot of the current state of the operating
system and going back to that state at any time. Another important use of
virtualization is in malware analysis, where malware can be safely analyzed
in an isolated virtual environment without affecting the host system.

4.1.Type 1 virtualization (Native) :

The most direct and efficient way to run a virtual

machine. It provides a logical structure to the
privilege levels, where the hypervisor is more
privileged than the guest operating systems it
controls. Guest operating systems run with
privilege level 0, which they are designed for

4.2.Type 2 virtualization (Host-based) :

Easy to install and use because the

hypervisor application is installed as a
regular application on the host OS. The
hypervisor and guest OS run as normal level
3 applications, without using level -1.
However, the guest OS needs to perform
privileged functions that require level 0, and these functions must be called through the host OS, causing
additional processing delay.

4.3.Docker Engine and containers (host) :

A virtualization tool that allows
containerized applications to run on
any platform without needing a guest
OS. The container includes all
necessary dependencies, making the
application portable and easy to run
reliably and quickely.

⚠ The security objectives of virtualization include ensuring that guest systems cannot access or affect each other,
cannot affect the hypervisor, and cannot detect that they are virtualized.

5. Secure boot - UEFI and TPM :

Page 3 of 4
Modern operating systems have various security measures such as virtual memory and access control, but these only
protect the system after it has started. Malware can be launched before the OS, exploiting BIOS vulnerabilities in the
booting process to launch before the OS.. UEFI has replaced BIOS to mitigate such vulnerabilities. The combination of
UEFI and TPM 2.0 is used in newer versions of Windows to achieve a more secure and trusted boot process.

5.1. UEFI :
UEFI is a modern firmware interface that replaces BIOS in computers. It defines the interface between the operating
system and the hardware, addressing several limitations of BIOS, including security aspects. UEFI replaced EFI
(developed by Intel) in 2005.
Secure Boot with UEFI ensures the integrity of start-up software modules by requiring them to be digitally signed by
the computer supplier. UEFI code in ROM is unsigned but assumed to be correct and initiates the boot sequence.
Program modules loaded are checked for the correct digital signature with the Platform Key, and if an incorrect
signature is detected, the start-up sequence is aborted.

5.2.TPM :

The TPM (Trusted Platform Module) is a coprocessor that is typically built into modern computer systems. It is
designed to perform cryptographic operations in a secure environment, and can receive commands and return
responses. The TPM can increase trust in a computer system by ensuring that sensitive operations are performed in a
protected environment, and by securely storing cryptographic keys and other sensitive data. Windows 11, for
example, requires TPM 2.0 as a security measure to ensure that only trusted software is run on a computer.

Advantages of TPM : TPM technology is designed to provide hardware-based security functions, which can offer
numerous advantages in securing a computing system. The tamper-resistant design of a TPM chip makes it an ideal
place to store cryptographic keys, which can help prevent unauthorized access to sensitive data. The unique RSA key
that is burned into the chip can also be used for device authentication, which can help prevent unauthorized access
to a network. Additionally, the TPM can help ensure platform integrity by taking and storing security measurements
of the boot process, which can help detect and prevent malicious attacks.

Integrity Check : The TPM is commonly used for system integrity measurements and key creation. During the boot
process, the boot code is loaded and measured, along with firmware and operating system components, which are
recorded in the TPM. The integrity measurements provide evidence for how the system started and help ensure that
a TPM-based key is used only when the correct software is used to boot the system.

Measured Boot : Measured Boot is a process that utilizes the TPM to store and verify the integrity of the components
that are loaded during the boot process. This process ensures that the hash of each phase of the boot process cannot
be changed by the subsequent phases, thereby creating a chain of trust. Towards the end of the start-up process, the
TPM log is signed, and an attestation client is started in Windows. The attestation client verifies the log and other
security information using an external certifying officer. Based on this check, mitigation measures can be taken to
ensure the security state of the system.

Safe (and trusted) start-up vs measured start-up :

• Secure boot (by UEFI) :

o Trust chain based on certificate stored in firmware

o The next step starts only if the correct signature
o The device thus does not start if verification fails
o New components/update must be signed and certificate managed

• Measured start-up :

o Chain of trust stored in the TPM

o To be verified after the start-up steps are completed
o An independent third party can perform verification (external attestation).
Page 4 of 4