Exadata Storage Layout

To understand Exadata lay out it will be helpful if we divide the entire system mentally into two
parts, the storge layer and the database layer.

The database layer or database servers works same as any non-exa machines with ASM to access the
storage. We can have rac or non-rac database servers on the exa machines. ASM is required even if
the databases are not configured to use RAC.

Both the layers are connected via an an InfiniBand network. InfiniBand provides a low-latency, high-
throughput switched fabric communications link. Redundancy is provided through multiple switches
and links.

iDB is network-based protocol that is implemented using InfiniBand. iDB is used to send requests for
data along with metadata about the request (including predicates) to cellsrv. In certain
situations, cellsrv is able to use the metadata to process the data before sending results back to
the database layer. When cellsrv is able to do this, it is called a Smart Scan and generally results in
a significant decrease in the volume of data that needs to be transmitted back to the database
layer. When Smart Scans are not possible, cellsrv returns the entire Oracle block(s). Note that iDB
uses the RDS(Reliable Datagram Sockets) protocol, which is a low-latency, InfiniBand-specific
protocol. In certain cases, the Oracle software can set up remote direct memory access (RDMA) over
RDS, which bypasses doing system calls to accomplish low-latency, process-to-process
communication across the InfiniBand network.

Configuration Options
Exadata is delivered as a preconfigured, integrated system, there are few options available like X8-8
or X8-2. We can have a full, half or quarter rack configuration. This configuration mainly differs in the
servers and cores. Please refer oracle documentation for more details. This document is more from
layout perspective

Oracle Exadata X8M-2 System Model Specifications

 Exadata Software Architecture
As discussed above Exadata machine is mainly divided into two parts, database software and storge
software which are connect via Idb over InfiniBand . Below figure gives more systematic overview of
the architecture stack and how it spans multiple servers in both the database grid and the storage
grid.

From the storage point of view ASM is a key component, ASM is an object that cuts across, all the
communication lines between the two tiers. This is meant to indicate that ASM provides the
mapping between the files and the objects that the database knows about on the storage layer. ASM
does not actually sit between the storage and the database

The Major component in the database software is LIBCELL, which is a library that is linked with the
Oracle kernel. LIBCELL has the code that knows how to request data via iDB. This provides a very
nonintrusive mechanism to allow the Oracle kernel to talk to the storage tier via network-based calls
instead of operating system reads and writes. iDB is implemented on top of the RDS protocol.

Exadata Storage Server Architecture

The architecture on the storage servers is simple. There is one master process (cellsrv) which
provides many or majority of Exadata storage services like serving the database requests for disk
I/O , advanced SQL offload capabilities also manages the IORM functionality( I already written a post
about IORM https://www.linkedin.com/posts/amit-vashishth-644072111_exadata-iorm-activity-
7145349467624325120--gUb?utm_source=share&utm_medium=member_desktop )
 Management server (MS)
The Management Server implements a web service for cell management commands and runs
background monitoring threads. It works with the Exadata cell command line interface (CellCLI) and
processes most of the commands from CellCLI.The MS is the primary interface to administer,
manage and query the status of the Exadata cell.

Restart server (RS)

Restart Server (RS) ensures the ongoing functioning of the Exadata software and services. It
monitors the heartbeat with the Management server and the Cellsrv processes. If the processes fail
to respond within the allowable heartbeat period, RS Process will restart Cellsrv and Management
server processes. So the Purpose of RS is to make sure MS and CellSrv are up and running fine.

Also, from the architecture diagram we can see cellsrv uses an init.ora file and has an alert log. In
fact, the storage software bears a striking resemblance to an Oracle database. The cellinit.ora file
contains a set of parameters that are evaluated when cellsrv is started. The alert log is used to write
a record of notable events, much like an alert log on an Oracle database. Note also that Automatic
Diagnostic Repository (ADR) is included as part of the storage software for capturing and reporting
diagnostic information.
 Exadata Disk Architecture
In Normal oracle database disks are presented directly to ASM, but Exadata database machine we
have a slightly different configuration .

Mapping Physical Disks, LUNs, and Cell Disks on the Storage Servers

Physical Disk:
The Physical disk is the Actual device within the storage cell, these physical disks are connected to
the RAID controller.

Logical Unit Number (LUN)

The Logical Unit Number (LUN) is the Logical Resource Representing Underlying Disk with the Upper
Software Layers. LUN is the lowest level of storage abstraction visible to OS layer We cannot create
or remove LUN it exists because of the presence of physical disk

Every Physical Disk mapped to LUN with 1 on 1 Mapping. Out of 12 physical disks, only on the First 2
disks, there is some area around 30G reserve for OS Storage Area to store OS image, Exadata
software binaries, config files etc. Rest of the 10 disks, the Lun will be same size of Physical disk.

Cell Disk
Then Comes the Cell disk which is based on LUN. The First 2 Disks having System Area carved out and
Rest of the storage space on those 2 disks are mapped as Cell Disks. For the Other 10 Disks, there
will be 1 cell disk for each LUN which is of same size.

Grid Disks
Cell disks can be sub-divided into Grid Disks. Multiple Grid Disks created out of Cell disks which can
be presented to ASM. If ASM Disk Group is to be named as DATA, then the grid disk is created with
prefix DATA. If size is not specified it will take all the available space in the cell disk.