STORAGE ARCHITECTURE/

GETTING STARTED:
SAN SCHOOL 101
Marc Farley
President of Building Storage, Inc
Author, Building Storage Networks, Inc.

Agenda
Lesson 1: Basics of SANs
Lesson 2: The I/O path
Lesson 3: Storage subsystems
Lesson 4: RAID, volume management
and virtualization
Lesson 5: SAN network technology
Lesson 6: File systems

Lesson #1

Basics of storage networking

Connecting

Concentrator

Dish
Router

HBA or
NIC

Network Switch/hub
Bridge

Computer
System
VPN

Connecting
Networking or bus technology
Cables + connectors
System adapters + network device drivers
Network devices such as hubs, switches, routers
Virtual networking
Flow control
Network security

Storing

Host
Software

Storage
Protocol

Storage
Devices

Volume
Manager
Software

Storage
Device
Drivers

Mirroring
Software

Storage Command and Transfer Protocol

(wiring, network transmission frame)

Tape
Drives

Disk
Drives
RAID
Subsystem

Storing
Device (target) command and control
Drives, subsystems, device emulation

Block storage address space

manipulation (partition management)
Mirroring
RAID
Striping
Virtualization
Concatentation

Filing
C:\directory\file

Database
Object

User/Application View

User/Application View

Logical Block Mapping

(Storage)

Filing
Namespace presents data to end users and
applications as files and directories (folders)
Manages use of storage address spaces
Metadata for identifying data
file name
owner
dates

Connecting, storing and filing as a

complete storage system

Filing

Storing

Wiring
Connecting
HBA or
NIC

Cable

Cable

Network Switch/hub

Storing function in
an HBA driver
Computer System

Disk Drive

NAS and SAN analysis

NAS is filing over a network

SAN is storing over a network
NAS and SAN are independent technologies
They can be implemented independently
They can co-exist in the same environment
They can both operate and provide services to the same
users/applications

Protocol analysis for NAS and SAN

NAS

SAN
Network

Filing
Storing
Wiring
Connecting

Integrated SAN/NAS environment

NAS
Client

NAS Head
Server
NAS
+
Server
SAN Initiator
System
NAS
Head

SAN
SAN
Target
Storage

Filing
Filing

Storing
Storing

Wiring
Connecting

Wiring
Connecting

Common wiring with NAS and SAN

NAS
Client

NAS Head
Server
NAS
Head
System

Filing
Filing

SAN
Storage
Target

Storing
Storing
Wiring
Connecting

Lesson #2

The I/O path

Host hardware path components

Memory

Processor

Memory
Bus

System
I/O Bus

Storage
Adapter
(HBA)

Host software path components

Application
Operating Filing Cache Volume
System System Manager Manager

MultiPathing

Device
Driver

Network hardware path components

Cabling
Fiber optic
Copper

Switches, hubs, routers, bridges, gatways

Port buffers, processors
Backplane, bus, crossbar, mesh, memory

Network software path components

Access and
Security

Fabric
Services

Routing

Flow
Control

Virtual
Networking

Subsystem path components

Network
Ports

Access and
Security

Cache

Resource
Manager

Internal Bus
or Network

Device and media path components

Disk drives

Tape drives

Tape Media

Solid state devices

The end to end I/O path picture

App

Memory

Processor Operating
System

Cabling

Network Systems

Subsystem
Network Poirt

Cache Volume MultiFiling

System Manager Manager Pathing

Access and
Security

Access
and Security

Cache

Fabric
Services

Resource
Manager

Memory System
I/O Bus
Bus

Routing

Internal Bus
or Network

Flow
Control

Disk
drives

Device Storage
Driver Adapter
(HBA)

Virtual
Networking

Tape
drives

Lesson #3

Storage subsystems

Generic storage subsystem model

Controller (logic+processors)
Access control
Network
Ports

Resource manager

Cache Memory

Internal Bus
or Network

Storage
Resources

Power

Redundancy for high availability

Multiple hot swappable power supplies

Hot swappable cooling fans
Data redundancy via RAID
Multi-path support
Network ports to storage resources

Physical and virtual storage

Exported
storage

Exported
storage

Exported
storage

Exported
storage

Exported
storage

Exported
storage

Exported
storage

Exported
storage

Subsystem Controller
Resource Manager
(RAID, mirroring,
etc.)

Physical
storage
device

Physical
storage
device

Physical
storage
device

Physical
storage
device

Hot
Spare
Device

SCSI communications architectures

determine SAN operations
SCSI communications are independent of connectivity
SCSI initiators (HBAs) generate I/O activity
They communicate with targets
Targets have communications addresses
Targets can have many storage resources
Each resource is a single SCSI logical unit (LU) with a universal

unique ID (UUID) - sometimes referred to as a serial number

An LU can be represented by multiple logical unit numbers (LUNs)
Provisioning associates LUNs with LUs & subsystem ports

A storage resource is not a LUN, its an LU

Provisioning storage
LUN 0

SCSI LU
UUID A

LUN 1

Port S1
LUN 1

Port S2

Port S3

SCSI LU
UUID B

LUN 2
LUN 2
LUN 3

SCSI LU
UUID C

LUN 3

Port S4

LUN 0

Physical
storage
devices

SCSI LU
UUID D

Controller functions

Physical
storage
devices

Physical
storage
devices

Multipathing

LUN X

Path 1
SCSI LU
UUID A

MP SW
LUN X

Path 2

Caching
Exported
Volume

Exported
Volume

Exported
Volume

Exported
Volume

Controller Cache Manager

Read Caches

Write Caches

1. Recently Used
2. Read Ahead

1. Write Through (to disk)

2. Write Back (from cache)

Tape subsystems
Tape
Drive
Tape
Drive

Tape Subsystem Controller

Tape
Drive

Tape Slots

Robot

Tape
Drive

Subsystem management

Now
Now
with
with
SMIS
SMIS

Management station
browser-based
network mgmt software

Ethernet/TCP/IP
Out-of-band management port

In-band
management

Storage Subsystem

Exported
Storage
Resource

Data redundancy

2n

Duplication

Parity

n+1
Difference

-1
d(x) = f(x) f(x-1)

f(x-1)

Duplication redundancy with mirroring

Host-based

I/O Path

Within a subsystem

Mirroring
Operator
Terminate I/O & regenerate new I/Os
Error recovery/notification

I/O PathA
I/O PathB

Duplication redundancy with remote copy

Host
Uni-directional
(writes only)

Point-in-time snapshot

Subsystem Snapshot
Host

Lesson #4

RAID, volume management

and virtualization

RAID = parity redundancy

2n

Duplication

Parity

n+1
Difference

-1
d(x) = f(x) f(x-1)

f(x-1)

History of RAID
Late 1980s R&D project at UC Berkeley
David Patterson
Garth Gibson

(independent)
Redundant array of inexpensive disks

Striping without redundancy was not defined (RAID 0)

Original goals were to reduce the cost and

increase the capacity of large disk storage

Benefits of RAID
Capacity scaling

Combine multiple address spaces as a single virtual

address

Performance through parallelism

Spread I/Os over multiple disk spindles

Reliability/availability with redundancy

Disk mirroring (striping to 2 disks)

Parity RAID (striping to more than 2 disks)

Capacity scaling
Combined extents
1 - 12

Exported
RAID
disk
volume
(1 address)

RAID
Controller
(resource
manager)

Storage
extent 1

Storage
extent 2

Storage
extent 3

Storage
extent 4

Storage
extent 5

Storage
extent 6

Storage
extent 7

Storage
extent 8

Storage Storage
extent 9 extent10

Storage
extent11

Storage
extent12

Performance
RAID controller (microsecond performance)

Disk
drive

Disk
drive

Disk
drive

Disk
drive

Disk
drive

Disk drives (Millisecond performance)

from rotational latency and seek time

Disk
drive

Parity redundancy
RAID arrays use XOR for calculating parity
Operand 1
False
False
True
True

Operand 2
False
True
False
True

XOR Result
False
True
True
False

XOR is the inverse of itself

Apply XOR in the table above from right to left
Apply XOR to any two columns to get the third

Reduced mode operations

When a member is
missing, data that is
accessed must be
reconstructed with xor
An array that is
reconstructing data is said
to be operating in reduced
mode
System performance
during reduced mode
operations can be
significantly reduced

XOR {M1&M2&M3&P}

Parity rebuild

RAID Parity Rebuild

The process of recreating data on a replacement member

is called a parity rebuild
Parity rebuilds are often scheduled for non-production
hours because performance disruptions can be so severe

XOR {M1&M2&M3&P}

RAID 0+1, 10

Hybrid RAID:
0+1
RAID Controller

DiskDisk
drivedrive
1

DiskDisk
drive
drive
2

DiskDisk
drive
drive
3

DiskDisk
drive
drive

4
Mirrored pairs of striped members

DiskDisk
drive
drive
5

Volume management and virtualization

Storing level functions
Provide RAID-like functionality in host systems
and SAN network systems
Aggregation of storage resources for:
scalability
availability
cost / efficiency
manageability

OS kernel

File system

Volume management

RAID & partition

management
Device driver layer
between the kernel
and storage I/O
drivers

Volume Manager

HBA drivers
HBAs

Server system

Volume managers can use all

available connections and
resources and can span multiple
SANs as well as SCSI and SAN
resources

Virtual
Storage

Volume
manager

SCSI disk resource

SCSI HBA

HBA drivers
SAN HBA

SCSI Bus
SAN cable

SAN disk
resources
SAN Switch

SAN storage virtualization

RAID and partition management in SAN systems

Two architectures:

In-band virtualization (synchronous)

Out-of-band virtualization (asynchronous)

In-band virtualization
Exported virtual storage

SAN
virtualization
I/O Path
system

System(s),
switch or
router

Disk
subsystems

Out-of-band virtualization

Distributed volume
management

Virtualization
management
system

Virtualization
agents

Virtualization agents
are managed from a
central system in the
SAN

Disk
subsystems

Lesson #5

SAN networks

Fibre channel
The first major SAN networking technology
Very low latency
High reliability
Fiber optic cables
Copper cables
Extended distance
1, 2 or 4 Gb transmission speeds
Strongly typed

Fibre channel
A Fibre Channel fabric presents a consistent interface
and set of services across all switches in a network
Host and subsystems all 'see' the same resources

SAN
Storage
Target
Subsystem

SAN
Storage
Target
Subsystem

SAN
Storage
Target
Subsystem

Fibre channel port definitions

FC ports are defined by their network role

N-ports: end node ports connecting to fabrics

L-ports: end node ports connecting to loops

NL-ports: end node ports connecting to fabrics or loops

F-ports: switch ports connecting to N ports

FL-ports: switch ports connecting to N ports or NL ports in

a loop

E-ports: switch ports connecting to other switch ports

G ports: generic switch ports that can be F, FL or E ports

Ethernet / TCP / IP SAN technologies

Leveraging the install base of

Ethernet and TCP/IP networks
iSCSI native SAN over IP
FC/IP FC SAN extensions over IP

iSCSI
Native storage I/O over TCP/IP
New industry standard
Locally over Gigabit Ethernet
Remotely over ATM, SONET, 10Gb Ethernet

iSCSI
TCP
IP
MAC
PHY

iSCSI equipment

Storage NICs (HBAs)

SCSI drivers

Cables
Copper and fiber

Network systems
Switches/routers
Firewalls

FC/IP
Extending FC SANs over TCP/IP networks
FCIP gateways operate as virtual E-port connections
FCIP creates a single fabric where all resources appear
to be local

One fabric
FCIP
FCIP
Gateway
Gateway

E-port

TCP/IP
LAN, MAN
or WAN

FCIP
FCIP
Gateway
Gateway

E-port

SAN switching & fabrics

High-end SAN switches have latencies of 1 - 3

sec
Transaction processing requires lowest latency
Most other applications do not

Transaction processing requires non-blocking

switches
No internal delays preventing data transfers

Switches and directors

Switches
8 48 ports
Redundant power supplies
Single system supervisor

Directors
64+ ports
HA redundancy
Dual system supervisor
Live SW upgrades

SAN topologies
Star

Simplest
single hop

Dual star
Simple network
+ redundancy

Single hop
Independent or integrated
fabric(s)

SAN topologies
N-wide star

Scalable
Single hop
Independent or
integrated fabric(s)

Core - edge
Scalable
1 3 hops
integrated fabric

SAN topologies
Ring

Scalable
integrated fabric
1 to N2 hops

Ring + Star
Scalable
integrated fabric
1 to 3 hops

Lesson #6

File systems

File system functions

Name space
Access control
Metadata
Locking
Address space management

Filing
Filing

Storing
Storing

Think of the storage address space as a sequence

of storage locations (a flat address space)
1 2
10 11
19 20
28 29
37 38
46 .
55 .
64 .
73 .
82 83

12
21
.
.
.
.
.
.
84

13
.
.
.
.
.
.
.
85

6 7

14 15
. .
. .
. .
. .
. .
. .
. .
86 87

16
.
.
.
.
.
.
.
88

17
26
35
44
53
62
71
80
89

18
27
36
45
54
63
72
81
90

Superblocks
are known addresses used to find
Superblocks
file system roots (and mount the file system)

SB

SB

File systems must have a known and

Filing and Scaling
dependable address space
The fine print in scalability - How does the filing function
know about the new storing address space?

1
7
13
19
25
31
37

Filing
Filing

Storing
Storing
Storing
Storing

1
6
11
16
21

2
7
12
17
22

2
8
14
20
26
32
38

3
8
13
18
23

3
9
15
21
27
33
39

4
9
14
19
24

4
10
16
22
28
34
40

5
10
15
20
25

5
11
17
23
29
35
41

6
12
18
24
30
36
42