SAN Basic Overview

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.
SAN Overview - 1
The objectives for this module are shown here. Please take a moment to read them.
SAN Overview - 2
The objectives for this lesson are shown here. Please take a moment to read them.
SAN Overview - 3
Fibre Channel (FCP) is a serial data transfer interface that operates over copper wire and/or optical
fiber at data rates up to 8 Gbps (gigabits per second) and up to 10 Gbps when used as ISL (E_Ports) on
supported switches.
Networking and I/O protocols (such as SCSI commands) are mapped to Fibre Channel constructs, and
then encapsulated and transported within Fibre Channel frames. This process allows high-speed
transfer of multiple protocols over the same physical interface.
Fibre Channel systems are assembled from familiar types of components: adapters, hubs, switches and
storage devices.
Host bus adapters (HBAs) are installed in computers and servers in the same manner as a SCSI host
bus adapter or a network interface card (NIC).
Hubs link individual elements together to form a shared bandwidth loop.
Fibre Channel switches provide full bandwidth connections for highly scalable systems without a
practical limit to the number of connections supported (16 million addresses are possible).
The word fiber indicates the physical media. The word fibre indicates the Fibre Channel protocol and
standards.
SAN Overview - 4
The Fibre Channel standards define a layered protocol:
ULP (Upper level Protocol) is not actually part of Fibre Channel
FC-4 level of Fibre Channel is designed to hand off to another protocol such as SCSI.
Fundamentally, the commands at FC-4 for SCSI allow SCSI initiators and targets to communicate
over Fibre Channel
FC-3 Defines a set services to support advanced functions
FC-2 level serves as the transport mechanism of Fibre Channel. The transported data is transparent
to FC-2 and visible to FC-3 and above. Includes data framing, frame sequencing, flow control, and
class of service
FC-1 defines the transmission protocol including serial encoding and decoding, special characters
and error control. Information is encoded 8 bits at a time into a 10 bit transmission character and
transmitted over the cable
FC-0 is the lowest level and defines the physical link in the system, including the fiber, connectors,
optical and electrical components. This level covers a variety of media and the associated drivers
and receivers capable of operating at a wide range of speeds
SAN Overview - 5
Frames are the basic building blocks of a Fibre Channel connection. The frames contain the
information to transmit, the address of the source and destination ports, and link control information.
All information in Fibre Channel is passed in frames. The maximum amount of data carried in a frame
is 2112 bytes; the total frame size is 2148 bytes.
The header contains the Source and Destination Addresses, which allow the frame to be routed to the
correct port. The Type field interpretation is dependent on whether the frame is a link control or Fibre
Channel data frame. For example, if the frame is a data frame, a 08 in the Type field indicates SCSI
FCP information in the Data field.
SAN Overview - 6
Fibre Channel addresses are used to designate the source and destination of frames in the Fibre
Channel network. The Fibre Channel address field is 24 bits/3 bytes in length. Unlike Ethernet, these
addresses are not burned in, but are assigned when the node enters the loop or is connected to the
switch.
The Fibre Channel Address identifiers are three bytes in length. The Frame Header contains two three-
byte fields for address identifiers, the D_ID and S_ID fields. Each N_Port has a fabric-unique
identifier, the N_Port Identifier, by which it is known. The source and destination N_Port Identifiers
and alias address identifiers are used to route frames within the fabric.
The Physical Address is switch-specific and dynamically generated during the Fabric Login (FLOGI).
Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved. Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.
SAN Overview - 7
A Fabric is a virtual space in which all storage nodes communicate with each other over
distances. It can be created with a single switch or a group of switches connected together.
The primary function of the fabric is to receive data frames from a source N_Port and route them to the
destination N_Port whose address identifier is specified in the frames. Each N_Port is physically
attached through a link to the fabric.
Many switches, such as the Cisco MDS, support multiple fabrics within the same switch and also
within the same physical switch topology. When a device logs into a fabric, its information is
maintained in a database. At the same time, information required for it to access other devices or
changes to the topology is provided by another database.
SAN Overview - 8
The following are some of the common services found in a fabric:
Login Service: This Service (at address FFFFFE) is used by all nodes when they perform a Fabric
Login (FLOGI). For a node to communicate in a fabric, it has to register itself with this service.
When it does so, it sends a Source Identifier (S_ID) set to zero, that is, having the following hex
value 0x000000. The login service then returns a D_ID to the node with the with the Domain, Area,
and Port fields qualified. This gives the node (N_Port) a 24-bit address that identifies it uniquely in
the entire fabric. The fabric uses this unique address for routing services.
Name Service: This Service (at address FFFFFC) is used to store information about all devices
attached to the fabric. When a node performs a PLOGI, it registers itself with the name server. The
name server stores all these entries in a locally resident database on each switch. Each switch in the
fabric topology exchanges its Name Service information with other switches in the fabric to
maintain a synchronized, distributed view of the fabric.
Fabric Controller: This Service (at address FFFFFD) provides state change notification to all
registered nodes in the fabric. Devices that require a fabric controller are those devices that need to
keep track of changes in a fabric, for example scanning storage targets in a fabric. This is requested
by submitting a SCN (State Change Notification) frame to the controller which then responds back
with a RSCN (Registered State Change Notification) to the frame.
SAN Overview - 9
There are three types of login supported in Fibre Channel; Fabric, Port and Process. All node ports
must attempt to log in with the Fabric. This is typically done right after the link or the Loop has been
initialized.
When a device is physically connected to a fabric switch port, the Fibre Channel protocol establishes a
logical connection between the node and the fabric switch. This is called Link Initialization. Ordered
Sets are sent between the node and the switch to establish the link.
SAN Overview - 10
Once the physical link is established, the node sends a special frame (FLOGI) to the port to allow it to
communicate with the rest of the fabric through the Fabric Login or FLOGI. The FLOGI frame
contains the S_ID field filled in with its ALPA value (00 for a switched fabric). This frame is received
by the login server, which is located at address FFFFFE. The login server responds back with the D_ID
field filled with the domain ID and area location. In other words, the device now gets a 24-bit address
by which it is identified in the fabric. All ports except private NL_Ports have to go through this
sequence for them to be able to communicate with each other in the fabric.
Next, each node performs a Name Server registration. The Name Server is located at address FFFFFC
and obtains information from the node through the port login frame (PLOGI) and through subsequent
registration frames. Information in the Name Server is stored in the form of database objects. The node
may register values for all or some database objects depending on the requirement. The most
commonly registered objects are: 24-bit fabric address, 64-bit Port Name (WWPN), 64-bit Node Name
(WWNN), ULPs (SCSI, IP etc.), Classes of service supported and Port type. The Node also requests a
list of nodes that support the same FC-4 Upper Layer Protocols as itself. This list usually depends on
whether there are restrictions placed on which devices the node can talk to i.e. zoning.
SAN Overview - 11
The node then attempts a Port Login (PLOGI) to all nodes from the list it receives from the switchs
Name Server. It provides a specific set of operating characteristics associated with the destination
N_Port, Classes of Service are supported. It also initializes the destination end-to-end credit. The
process is repeated as other nodes are attached to other ports on the switch.
Finally, the node then sets up the environment between itself (originating N_Port) and the device its
communicating with (responding N_Port) by performing a Process Login. This environment is then
used to determine if there is a LUN present. This is the point at which storage connectivity is
established. A group of related processes is collectively known as an image pair. The processes
involved can be system processes, system images, control unit images or ULP processes. The use of
process login (PRLI) is required by a specific upper-level protocol such as SCSI-FCP mapping.
SAN Overview - 12
This table documents the various processes that occur when a Fibre Channel device is connected to an
FC-based SAN.
When a link light on the switch or on an HBA does not come on, it means that the link initialization
component in the FLOGI process is not complete. To troubleshoot this, one usually starts with the
hardware and then moves up to the HBA driver level.
SAN Overview - 13
The Internet Small Computer Systems Interface (iSCSI) protocol provides a means of transporting
SCSI packets over TCP/IP. iSCSI works by wrapping SCSI commands into TCP and transporting them
over an IP network. Since iSCSI is IP based traffic, it can be routed or switched on standard Ethernet
equipment.
SAN Overview - 14
Traditional Ethernet adapters (NIC) are designed to transfer packetized file level data among PCs,
servers, and storage devices such as NAS appliances. However, NICs do not traditionally transfer
block level data, which is handled by a storage host bus adapter, such as Fibre Channel or parallel
SCSI. In order for a NIC to process block level data, the data needs to be placed into a TCP/IP packet
before being sent over the IP network. Through the use of iSCSI drivers on the host or server, a NIC
can transmit packets of block level data over an IP network. When using a NIC, the server handles the
packet creation of block level data and performs all of the TCP/IP processing. This is extremely CPU
intensive and lowers the overall server performance.
The TCP/IP processing performance bottleneck has been the driving force behind the development of
TCP/IP offload engines (TOE) on adapter cards. A TOE moves the TCP/IP processing from the host
CPU to the TOE card. Thus, a TCP/IP offload storage NIC operates more like a storage HBA rather
than a standard NIC.
An iSCSI HBA is similar to a Fibre Channel HBA where all operations occur on the card. The iSCSI
HBA is required for EMC Boot from SAN environment.
SAN Overview - 15
Because a message is divided into a number of packets, each packet can, if necessary, be sent by a
different route across the network. Packets can arrive in a different order than the order they were sent
in. The Internet Protocol just delivers them. It's up to another protocol, the Transmission Control
Protocol (TCP) to put them back in the right order.
An iSCSI packet contains SCSI data and the iSCSI header, which is created by the iSCSI initiator, and
is then wrapped in other protocol layers to facilitate its transport.
The Ethernet Header is used to provide addressing for the physical layer while the IP Header provides
packet routing information used for moving the information across the network, and the TCP Header
contains the information needed to guarantee delivery to the target destination. The iSCSI Header
defines how to extract SCSI commands and data.
SAN Overview - 16
Native iSCSI allows for all communications using Ethernet. Initiators may be directly attached to
iSCSI Targets, or may be connected using standard Ethernet routers and switches. Bridging
architectures allow for the Initiators to exist in an Ethernet environment, while the storage remains in a
Fibre Channel SAN.
SAN Overview - 17
To ensure that data reaches all the users who need it, organizations are now looking for new ways to
transport data throughout the enterprise locally over the SAN as well as over much longer distances.
One of the best ways to achieve this goal is to interconnect geographically dispersed SANs through
reliable, high-speed links. This approach involves transporting Fibre Channel block data over existing
IP infrastructures currently used throughout the enterprise. The new FCIP protocol standard has rapidly
gained acceptance as a manageable, cost-effective way to blend the best of both worlds: Fibre Channel
block data storage and proven, widely deployed IP infrastructure. As a result, organizations now have
an excellent way to protect, store, and move their data while leveraging existing technology
investments.
SAN Overview - 18
The top layer of the FCIP protocol stack has SCSI Applications, which include the SCSI driver
program that executes read and write commands. These block, stream, and other command types are
grouped into a layer which also contains SCSI data and status information. Below the SCSI layer is
the Fibre Channel Protocol (FCP) layer, which is simply a fibre channel frame whose payload is SCSI.
The FCP layer rides on top of the Fibre Channel transport layer. This layer can then run natively within
a SAN fabric environment or be encapsulated into IP at the Fibre Channel over IP (FCIP) layer. TCP
and IP are then used to transport the encapsulated information across wired or wireless Ethernet, or
another transport that supports TCP/IP traffic.
Notice how FCIP creates a logical tunnel between two Fibre Channel protocol stacks. In reality, the
information flows through the IP network represented by the cloud.
SAN Overview - 19
FCIP entities are switches or other network adapters used during FCIP. The primary purpose of an
FCIP entity is to forward FC frames. Primitive signals, sequences and class 1 FC frames are not
transmitted through FCIP because they can't encode using FC Frame encapsulation. An IP network
sees the FCIP entities as peers, therefore require TCP/IP communication. FCIP entities contain 1+ TCP
endpoints in IP-based network. From a Fibre Channel perspective, the pairs of FCIP entities and their
FC entities, forward FC frames between FC fabric elements. The End nodes dont know an IP link
exists, therefore the path taken by the FC frames follow the normal routing procedure established by IP
network. FCIP doesnt participate in the FC frame routing. FCIP control and service may use QoS
features.
FCIP entities don't participate in the discovery of FC source and destination identifiers. Before a link is
established between two FCIP entities the IP address and TCP port of its pair must be configured with
the initiating entity.
SAN Overview - 20
FCIP can transport existing Fibre Channel services across the IP network such that two or more
interconnected SANs can appear as a single large SAN and can be managed by traditional SAN
management applications. In addition, FCIP enables SAN applications to support additional protocols
without modification. These applications might include disk mirroring between buildings in a campus
network or remote replication over the WAN. The type of applications utilized are based on the
distance the data must travel, the network bandwidth, and the QoS requirements and/or abilities of the
network connection.
While some implementations of FCIP are point-to-point tunnels, the protocol does not require that
the gateways support only point-to-point tunneling. The FCIP standard supports all Fibre Channel
services, including FSPF routing algorithms, such that multiple logical links created from a single
gateway can route Fibre Channel packets over the IP infrastructure. Not only is FCIP routable, but IP
networks do not need to know anything about the packets being routed.
The Fibre Channel services handle all routing between logical links, while the TCP protocol handles
the delivery of packets to the specific gateway device.
SAN Overview - 21
A Fibre Channel switch, acting as a FCIP Gateway, encapsulates FC frames into IP packets. Once
encapsulated into IP packets, they can be sent across the IP network through the FCIP link. The IP
network could be made up of Gigabit Ethernet, Fast Ethernet, IP switches and routers, etc. At the other
end of the IP network, the original Fibre Channel frames are recovered and sent through the SAN by
the receiving Fibre Channel Switch.
The FCIP connected FC SANs essentially form into a new unified fabric. The IP network is transparent
to the Fibre Channel fabric. Only edge devices need to be aware of FCIP encapsulation. This solution
can take advantage of existing Fiber Channel networks and tie them together using existing IP
networks.
SAN Overview - 22
SAN Overview - 23
The Port initialization takes place first. In this step the port determines the Port Type. For this
discussion, we will assume the Port has configured itself as an E_Port (Expansion Port). Once this port
is a recognized E_Port, Exchange Link Parameters (ELP) are sent. The ELP exchanges information
about the switch such as its WWN and Class of Service.
Next, the Fabric must elect a Principal Switch. This is accomplished by the Exchange Fabric
Parameters (EFP) and Principal Switch Selection (DIA) operations. Once the Principal Switch is
elected, Domain_IDs are assigned to the other member switches of the Fabric. Switches will attempt to
keep their previous Domain_ID.
The Final steps in configuring the Fabric are to build the Topology Database using Fibre Shortest Path
First (FSPF) and then create the Routing Tables.
Although not specified, the switches attempt to combine the existing Zone information into a single
Zone Set to be distributed throughout the new Fabric.
SAN Overview - 24
An E_Port is the point at which a Switch is connected to another Switch to create a multi-Switch
Fabric. An E_Port is the point at which a Switch is connected to a Bridge device. It normally functions
as a conduit between the Switches for frames destined for remote N_Ports and NL_Ports. An E_Port is
also used to carry frames between Switches for purposes of configuring and maintaining the Fabric.
An E_Port shall support the Class F service. An E_Port shall also be capable of routing one or more of
the following classes of service: Class 1 service, Class 2 service, Class 3 service. An E_Port shall not
admit to the Fabric any Primitive Sequences, or any Primitive Signals other than Idle, that the E_Port
receives on its inbound fibre.
An E_Port contains an FC-PH Transport element through which all frames are passed, and Primitives
are transferred across the Link to and from the other E_Port. Frames received from the other E_Port
are either directed to the Switch Construct via the Switch Transport element, or directed to the
Link_Control_Facility. The Link_Control_Facility receives frames related to Switch Fabric Internal
Link Services such as ELP, and transmits responses to those Link Service frames.
Frames received from the FC-PH Transport element that are destined for other ports are directed by the
Switch Transport to the Switch Construct for further routing. Frames received from the Switch
Construct by the Switch Transport are directed either to the FC-PH Transport for transmission to the
other E_Port, or to the Internal_Control_Facility. The Internal_Control_Facility receives frames related
to Switch Fabric Internal Link Services, and transmits responses to those Internal Link Services
frames.
Information is passed between the Internal_Control_Facility and the Link_Control_Facility to effect
the control and configuration of the Transport elements.
SAN Overview - 25
Switches in the fabric communicate through EFP (exchange fabric parameter) frames to determine the
principal switch as follows:
If one (and only one) director/switch is set to Principal, it becomes the principal switch. This
switch remains the principal switch as long as it is on line in the fabric, even if someone changes
another switchs Switch Priority to Principal.
If more than one director/switch is set to Principal, the director/switch with the lowest WWN
number among those set to Principal becomes the Principal switch. The Switch Priority setting for
every other switch/director (including any others that were previously set to Principal)
automatically changes to Default.
If no director/switch is set to Principal but one or more are set to Default, the director/switch with
the lowest WWN number among those set to Default becomes the principal switch. The Switch
Priority setting for every other switch/director remains unchanged.
If all directors/switches are set to Never Principal, no director/switch can be principal. In this case,
all of the ISLs will segment, with reason code 05 (no switch is capable of becoming a Principal
switch).
SAN Overview - 26
Each switch in a multiswitch fabric is identified by a unique Domain ID, in the range 1 through 239.
Domain IDs are used in 24-bit Fibre Channel addresses to identify source and destination ports in a
fabric.
When adding a new switch, assign it a Preferred Domain ID that is not in use. When the switch comes
on line, it requests that it be assigned this ID. If the ID is already in use, the principal switch assigns an
unused one.
To change a Domain ID, a switch must be taken off line. Some HBAs can handle a Domain ID change
without requiring device driver administration changes. In some operating environments, however
(currently HP-UX and AIX), drivers require device driver-level administrative changes when the
Domain ID is changed.
If two operational fabrics join, they determine if any Domain ID conflicts exist between the fabrics. If
a conflict exists, the interswitch link (ISL) E_Ports segment themselves from each other, preventing
the fabrics from joining. To avoid this problem, assign a unique Preferred Domain ID to each switch in
a data center. If there are plans to go data center to data center, apply this uniqueness across both data
centers. This will facilitate future fabric mergers, and is particularly important if zoning is
implemented through port number (and by default Domain ID) rather than WWN.
SAN Overview - 27
EMC recommends that you always attempt to give each switch a unique preferred Domain ID, whether
it is targeted for an existing fabric or a new fabric. (The exception to this rule would be if you were
swapping a new switch for a failed switch.) This provides the best opportunity to merge a switch or
fabric into another fabric with the least amount of manual intervention.
Since Domain IDs become part of an HBA or storage array port's fabric address, and data is transferred
to and from devices based on their address, Domain IDs cannot be changed without first bringing the
switch to an offline state. Every switch that is currently online will have been assigned a Domain ID
from the principal switch. If the fabric contains only one switch, that switch is the principal switch for
that fabric. Domain ID negotiation done during a fabric merge will not change the current Domain ID
when the switch is connected to the fabric while online.
During the exchange of parameters following a merger event, if the switches identify a domain
conflict, they will not merge and the ISLs connecting them will remain segmented. Taking a switch
that has segmented from the fabric due to a Domain ID conflict offline and then bringing it back online
allows the switch to renegotiate for a new unique Domain ID. It should be noted that taking a switch
offline terminates communication between the devices that are on that switch. EMC recommends that
if a conflict does exist that you change the preferred Domain ID of the switch yourself so that you can
ensure that it is unique in your entire environment and that it always attempts to use the same ID, even
after future merges and fabric changes
SAN Overview - 28
One switch in the fabric is responsible for the distribution of Domain IDs, and plays a role in the route creation
for fabric management traffic. This switch is known as the principal switch. Since this switch must communicate
with all other switches and is the basis for fabric traffic routing, this switch should be centrally located. This will
assist in the uniform delivery of information in the fabric, and provide a consistent response to fabric build
events.
Principal switches are selected during both the creation of the fabric and during fabric reconstruction events.
Two pieces of information located on the switch will determine the selection of the principal switch in the fabric.
The Domain ID negotiation is governed by the entity in the fabric called the principal switch. Principal switch
selection is made based on the switch with combination of the highest principal switch priority and then the
lowest World Wide Name.
Proper placement of the principal switch in the fabric can lead to short negotiation times, resulting in the fabric
returning to a normal state in an shorter amount of time. For these reasons, EMC recommends that the switch at
the logical center of the fabric be made the principal switch. This would normally mean the switch with both the
least amount of hops to the farthest extent of the fabric and/or the switch that has connections to the most other
switches in the fabric. These two placement strategies will help to ensure that your principal switch access times
are as quick as they can be.
The basic rules of Principle Switch selection are:
A standalone switch will declare itself the Principal Switch.
Each Fabric can have only one Principal Switch.
If both switches are ONLINE, a principal switch election will occur.
Switch Priority determines who becomes Principal Switch.
Domain IDs DO NOT determine who becomes Principal Switch.
SAN Overview - 29
The fabric-capable device drivers on Symmetrix-supported host bus adapters (HBAs) discover devices
by using a Name Server, which is a fabric-based function accessible to all nodes on the fabric via a
well-known fabric address.
When an N_Port logs onto the fabric, it registers its World Wide Port Name and FC-4 type (SCSI) with
the Name Server. A driver performing device discovery queries the Name Server for a list of all SCSI
devices that have registered with it. The driver uses this list to attempt a login to each of the registered
devices. In this manner, a fabric device driver discovers all logged-in Symmetrix ports attached to the
switch.
SAN Overview - 30
Connectrix Directors and switches use a number of algorithms to distribute the load over a set of ISLs.
These algorithms include:
Fibre Shortest Path First
Multiple Equal-Cost Paths
Load Assignment on World Wide Node Names
The ANSI standard for routing in a Fibre Channel fabric is Fibre Channel Shortest Path First (FSPF).
This standard measures routing cost in terms of the number of hops a frame must take from its source
to its destination.
A hop is a direct path between two switches with no other switches in between. A link between an end
device and a switch does not count as a hop.
The FSPF algorithm creates routes of minimum cost throughout the fabric. Every entry port into a
switch has a designated exit for each of the possible destinations. These exit assignments change only
if there is a reconfiguration event in the fabric. This single-static exit ensures in-order delivery of the
frames within an exchange. If an ISL or a new switch is added or removed, the assignments may
change. These assignments are done with logged-in N_Ports. No routing assignments are done for
F_Ports that have no logged-in ports.
When there are multiple equal-cost paths between a source and a destination, the routing algorithm
spreads the load over the multiple paths in a round robin fashion, sequentially assigning a set of entry
ports to a set of exit ports so that each exit port has the same number of assigned entry ports (or a
difference of only one).
SAN Overview - 31
In this standard, the term path selection indicates the discovery of the best path from source to
destination, and the term routing indicates the actual forwarding of frames to a specific destination.
FSPF performs hop-by-hop routing which means that a Switch only needs to know the next hop on the
best path to the destination. The replicated topology database insures that every Switch in the Fabric
has the same view of the Fabric itself, and therefore all Switches will make consistent routing
decisions.
Typically a Switch needs to know, for each destination domain in the Fabric, which path should be
used to route a frame to that domain. A routing table entry minimally consists of a destination
Domain_ID, and an E_Port to which frames are forwarded to the destination Switch.
SAN Overview - 32
The fewest hop path is always selected. If a new path with the least cost becomes available, frames
will immediately be switched to the newer path. If multiple minimum hop paths between switches
exist, switches will do load sharing
In this example, the cost for each link is equal:
There are two ISLs between switches Green and Yellow
Three HBAs login to switch Green with data destined to Switch Yellow
HBS 1 uses Path C (ISL 3)
HBA 2 uses Path D (ISL 4)
HBS 3 uses Path C (ISL 3)
Each link has a specific cost associated with it. The higher the throughput, the lower the cost. For
example a 1Gbps link has a cost of 1000 while a 2Gbps link costs 500. In this case, the 2Gbps link is
always used. You also can configure link costs for switches.
SAN Overview - 33
Zoning is a switch function that allows devices within the fabric to be logically segmented into groups
that can communicate with each other. When a device logs into a fabric, it is registered by the name
server. When a port logs into the fabric, it goes through a device discovery process with other devices
registered as SCSI FCP in the name server. The zoning function controls this process by only letting
ports in the same zone establish these link level services.
A collection of zones is called a zone set. The zone set can be active or inactive. An active zone set is
the collection of zones currently being used by the switched fabric to manage data traffic.
Single HBA zoning consists of a single HBA port and one or more storage ports. A port can reside in
multiple zones. This provides the ability to map a single storage port to multiple host ports. For
example, a Symmetrix FA port or a CLARiiON SP port can be mapped to multiple single HBA zones.
This allows multiple hosts to share a single storage port.
SAN Overview - 34
Switch Domain IDs must be unique
If two enabled switches are cabled together, they must have unique Domain Ids, or the Fabric is
segmented.
Ensuring unique Domain ID
If a new switch is cabled to the Fabric and then enabled, the master switch assigns a unique
Domain ID to the new switch.
Domain IDs can be configured by the administrator to ensure that uniqueness.
If you power cycle, the switch will receive a unique Domain ID.
Other configuration items that may affect cascade
Non-SCSI Tachyon Mode
Disable Device Probing
VC Encoded Address Mode
Per-frame Route Priority
SAN Overview - 35
SAN Overview - 36
There are several ways to monitor and manage Fibre Channel switches in a fabric:
If the switches in the fabric are contained in a cabinet with a Service Processor (SP), console software loaded
on the SP can be used to manage them.
Some switches also offer a console port, which is used for serial connection to the switch for initial
configuration using a Command Line Interface (CLI). This is typically used to set the management IP
address on the switch. Subsequently, all configuration and monitoring can be done via IP. Telnet or ssh may
be used to log into the switch over IP, and issue CLI commands to it. The primary purpose of the CLI is to
automate management of a large number of switches/directors with the use of scripts, although the CLI may
be used interactively, too. In addition, almost all models of switches support a browser-based graphical
interface for management.
There are vendor-specific tools and management suites that can be used to configure and monitor the entire
fabric. They include:
B-Series: Web Tools and Connectrix Manager
M-Models include Connectrix Manager Basic
MDS-Series: Cisco Fabric Manager
SAN Manager, an integral part of EMC ControlCenter, provides some management and monitoring capabilities
for devices from both vendors.
A final option is to deploy a third-party management framework such as Tivoli. Such frameworks can use SNMP
(Simple Network Management Protocol) to monitor all fabric elements.
SAN Overview - 37
Web Tools is an easy-to-use, browser-based application for switch management and is included with
all Connectrix B-Series products. Web Tools simplifies switch management by enabling administrators
to configure, monitor, and manage switch and fabric parameters from a single online access point. Web
Tools supports the use of aliases for easy identification of zone members. With Web Tools, firmware
upgrade is a one-step process. The Switch View allows you to check the status of a switch in the fabric.
The LED icon for the port reporting an issue changes color.
SAN Overview - 38
Connectrix Manager is a licensed software product widely used for the management of Connectrix B
switches and directors. It can be run locally on the Connectrix Service Processor, or remotely on any
network-attached workstation. Since this application is Java-based, IT administrators can run it from
virtually any type of client device.
SAN Overview - 39
Fabric Manager and Device Manager must be installed in a server and can contain several clients. This
Java-based tool simplifies management of the MDS Series through an integrated approach to fabric
administration, device discovery, topology mapping, and configuration functions for the switch, fabric,
and port.
Features of Fabric Manager include:
Fabric visualization: Automatic discovery, zone and path highlighting
Comprehensive configuration across multiple switches
Powerful configuration analysis including real-time monitoring, alerts, zone merge analysis, and
configuration checking
Network diagnostics: Probes network and switch health, enabling administrators to pinpoint
connectivity and performance issues
Comprehensive security: Storage Media Encryption Protection against unauthorized management
access with Simple Network Management Protocol Version 3 (SNMPV3), Secure Shell Protocol
(SSH), and role-based access control (RBAC)
SAN Overview - 40
SAN Manager provides a single interface to manage LUN Masking, switch zoning, device monitoring
and management. The integration of SAN Manager into ControlCenter provides a distributed
infrastructure allowing for remote management of a SAN. It offers reporting and monitoring features
such as threshold alarms, state change alerts and component failure notifications for devices in the
SAN.
SAN Manager has capabilities to automatically discover, map, and display the entire SAN topology at
a level of detail desired by the administrator. It can also display specific physical and logical
information about each object in the fabric. Administrators can view details on physical components
such as host bus adapters, Fibre Channel switches and storage arrays, as well as logical components
such as zones and LUN masking policies. SAN Manager offers support for non-EMC arrays such as
HDS Lightning, HP StorageWorks, and IBM Shark.
SAN Overview - 41
Command Line Interface is available for any Connectrix Switch. Each switch vendor implements its
own syntax for common switch and fabric functions.
SAN Overview - 42
SAN Overview - 43
You can install a switch as a stand-alone unit on a flat surface, in an Electronic Industries Association
(EIA) cabinet or the EC-1700 Cabinet using the Slide Rack Mount Kit included with the switch.
For this example, we review the configuration of a B-Series switch, The following modules cover the
details for each Connectrix series.
SAN Overview - 44
Floor loading is a badly understood issue in Data Centers. Not just in the place where the equipment eventually ends up but
also en-route from the loading bay to the final resting place.
Most data centers have a raised floor made of specialist tiles held up on struts and a tile mounting framework pushing down
on the underlying slab and fabric of the building. Each of these elements must be strong enough to withstand the weight of
the equipment to be installed. In addition, the floor must be strong enough to deal with carrying its design weight with a
number of the tiles removed for maintenance.
Typically a data center floor will have a loading capability measured in Kilo Pascals (kPa) aka Kilograms per square meter
(Kg/m2) or Pounds per square foot (lbs/ft2) depending on your preferred units of measure. For most EMC scenarios, the
floor tiles must be concrete filled or Heavy-duty tiles. Perforated and vented tiles as well as damaged tiles and tiles with
holes for cables (particularly on the edge of the tile rather than in the middle) are significantly weaker than complete tiles.
There are three kinds of load that we need to take into account:
Static Load - the load a stationary object places on the floor (kg/sq m)
Concentrated Load - The point load transferred directly to the floor from a wheel or foot support (kg/sq cm)
Rolling or Dynamic Load - The load placed on the floor as the object is pushed through the data center into its final
location (kg/Tile)
The underlying slab needs to be strong enough to support the gross load of the entire floor, including the struts, tiles,
equipment and M&E plant. If in doubt instruct a Structural Engineer or consult the initial design documents for your
building.
As an example, the installation of the MDS 9513 Director in a rack requires a mechanical lift to place the chassis in the
rack. A fully loaded 9513 can weigh about 300 pounds. A full rack with all components will be over 1000 pounds of static
load.
SAN Overview - 45
Verify the following requirements are met prior to product and management interface installation.
Ensure:
A site plan is prepared, configuration planning tasks are complete, planning considerations are
evaluated, and related planning checklists are complete.
Fibre Channel SAN design is complete.
Support is available for one of the following product management methods:
A browser-capable PC and Internet connectivity to support the product-resident Web Tools, or
A rack-mount service processor or browser-capable PC and LAN segment connectivity to
support operation of Connectrix Manager and Element Manager applications.
Support equipment and technical personnel are available for the installation.
The required number and type of fiber-optic jumper cables are delivered and available. Ensure
cables are the correct length and have the required connectors.
An EC-1700 cabinet or customer-supplied 19-inch equipment rack and associated hardware are
available (optional).
Remote workstations or simple network management protocol (SNMP) workstations are available
(optional). Workstations are customer-supplied and connected through a public or dedicated LAN
segment.
SAN Overview - 46
Take the following measures to ensure a successful installation and operation of the switch. In this
example the switch requires specific environmental conditions.
Install the switch with the non-port side facing the cool-air aisle to ensure proper cooling. This
prevents the fan assemblies from pulling in heated exhaust air.
Verify that the supply circuit, line fusing and wire size conform to the electrical rating of the switch.
Verify that the ambient air temperature at the non-port side of the switch does not exceed the listed
rating of the switch.
Verify air flow is available to the nonport vents.
SAN Overview - 47
The exact installation procedures vary among the different cabinet systems. Always review the
product documentation prior to installation.
Assemble the two side slide assemblies.
Install the inner slide rails on the switch chassis:
Position one of the inner slide rails with the flat side against the switch, oriented as needed for
slide removal.
Attach the rail using the three Phillips head 8-32 x 3/16 inch screws provided with the
installation kit.
Tighten the screws to a torque of 15 inch-pounds.
Repeat with the second inner slide rail on the other side of the switch.
Verify that two of the slide mount L-brackets are attached to each outer slide rail. These screws
should be at a torque of 15 inch-pounds.
Install the power cord clips onto the outer slide rails. The clips must be installed on the outer
surface of the slide rails so that they do not interfere with rail movement.
Insert the power cords into the power cord clips with the pronged end of the power cords toward
the cool air aisle for access to the power source.
Install the outer slide rails in the cabinet.
Install the switch in the cabinet.
Warning Warning: Using screws longer than the specified length can damage the switch.
SAN Overview - 48
Installation of the SFP transceivers is the final step:
Remove the shipping plugs from the ports to be used.
The transceiver is keyed so that it can be inserted into the port only with the correct orientation. If
it does not slide in easily, ensure it is correctly oriented.
Position a transceiver so that it is oriented correctly and insert into the port until it is firmly seated
and the latching mechanism clicks. For instructions specific to the type of transceiver, refer to the
transceiver manufacturers documentation.
Repeat for the remaining ports as required.
SAN Overview - 49
Connect the power cord to one or both of the power supplies and to a power outlet. Ensure that the
power cord has a minimum service loop of 6 inches available at the connection to the switch, and is
routed so that it is not exposed to stress. Turn on the power supply by flipping the AC switch to 1.
The switch is powered on and begins running the POST. Repeat these steps for the second power
supply.
For optimal redundancy, ensure that separate circuits are provided for each power supply.
When the switch is powered on, it automatically runs the POST to guarantee switch stability. Errors
that occur during POST are written to the system error log. Verify that the POST completes
successfully.
SAN Overview - 50
Connect the serial cable to the serial port on the switch and to an RS-232 serial port on the
workstation. If the serial port on the workstation is RJ-45 instead of RS-232, you can remove the
adapter on the end of the serial cable and insert the exposed RJ-45 connector into the RJ-45 serial port
on the workstation.
The serial cable must be a pass-through serial Cable. Null Modem cables establish a connection
but do not transfer data or commands. It is recommended that you use your laptop to configure
these switches through HyperTerminal.
SAN Overview - 51
Change the IP address, subnet mask, and gateway address as directed by the customers network
administrator.
Verify the address was correctly set by entering the ipAddrShow command at the prompt. Record the
IP address on the label provided for this purpose on the port side of the switch. Once the IP address is
verified as correct, log off of the serial console, remove the serial cable, and replace the shipping plug
in the serial port.
SAN Overview - 52
When configuring the switch using the telnet, Web Tools or other out-of-band management systems,
you must be aware of LAN configuration. Devices and applications such as routers, firewalls, proxy
servers, DNS, DHCP could interfere with network connectivity between the managements station and
the switch.
In the event of a failure to communicate, some simple tests should be performed. To determine if
connectivity is possible, begin with the basic TCP/IP test of PING <switch address>. If this is
successful you have an IP connection. For higher level applications to function you may next need to
test for Port availability and LAN timeouts. Commands such as TRACERT, NSLOOKUP and
NETSTAT are useful to determine the network configuration between station and switch.
SAN Overview - 53
Before you install Web Tools on your workstation, verify that your switches and workstation meet the
Web Tools requirements listed in the current product documents. Web Tools requires any browser that
conforms to HTML version 4.0, JavaScript version 1.0, and Java Plug-in 1.6.0 or higher.
If your fabric includes a switch with a limited switch license and you are opening Web Tools using that
switch, if the fabric exceeds the switch limit indicated in the license, Web Tools allows a 30-day grace
period in which you can still monitor the switch through Web Tools. However, Web Tools displays
warning messages periodically.
These messages warn you that your fabric size exceeds the supported switch configuration limit and
tells you how long you have before Web Tools will be disabled. After the 30-day grace period, you are
no longer able to open Web Tools from the switch with the limited switch license if that switch is still
exceeding the switch limit.
Web Tools is part of the Fabric OS of a switch. When you open Web Tools on a switch, you can
manage other switches in the fabric that have lower or higher firmware versions. It is important to note
that when accessing these switches you are opening the switchs version of Web Tools, and the
functionality available for those switches might vary.
SAN Overview - 54
To perform the initial login into a switch:
Verify that the switch is connected to your IP network through the RJ-45 Ethernet port to enable
connection through telnet.
Open a telnet connection to the switch. The login prompt is displayed if the telnet connection
successfully found the switch in the network.
Enter the user ID (usually user or admin) at the login prompt.
login: admin
Enter the default admin password. The default password is password. You are prompted to change
the password.
Enter the new password.
Verify that the login was successful.
A prompt is displayed showing the switch name and user ID to which you are logged.
SAN Overview - 55
It is important to have consistent system configuration settings since inconsistent parameters
amongst switches in the same fabric can cause fabric segmentation. To display and check system
configuration settings, perform the following procedure.
Fabric parameters include all the items listed in the configure command. Fabric parameters
must be identical for each switch across a fabric.
To save time when configuring the fabric parameters:
Configure one switch first (using the configure command.
Use the configUpload command to save the configuration information.
Use the configdownload command to download it onto each of the remaining switches.
The default Domain ID is 1. If the default domain ID is already in use when the switch is connected to
the fabric, the Domain ID for the new switch is automatically reset to a unique value. The Domain IDs
that are currently in use can be determined using the fabricshow command.
Domain IDs are assigned dynamically when a switch is enabled. However, the Domain ID can be set
manually in order to control the number, or to resolve a Domain ID conflict when merging fabrics.
Valid Domain IDs are 1 through 239.
SAN Overview - 56
Switch Date and Time - All switches maintain current date and time in non-volatile memory. Date and
time are used for logging events. Switch operation does not depend on the date and time; a switch with
an incorrect date and time value still functions properly.
Switch Status Policies - The policy parameter determines the number of failed or non-operational
units for each contributor that will trigger a status change in the switch.
Each parameter can be adjusted so that a specific threshold must be reached before that parameter
changes the overall status of a switch to MARGINAL or DOWN. For example, if the FaultyPorts
DOWN parameter is set to 3, the status of the switch will change if 3 ports fail.
Tracking Switch Changes - The Track Change feature allows you to keep record of specific changes
that may not be considered switch events, but may be useful. The output from the track changes feature
is dumped to the error log for the switch. Use the errdump command or errshow command to view the
error log. Items in the error log created from the Track changes feature are labeled TRACK. An
SNMP-TRAP mode can also be enabled.
Routing - In a stable fabric, frames are always delivered in order, even when the traffic between
switches is shared among multiple paths. However, when topology changes occur in the fabric (for
instance, a link goes down), traffic is rerouted around the failure. When topology changes occur, some
frames could be delivered out of order.
SAN Overview - 57
Switch Name - You can customize the switch names for the logical switches. If you chose to change
the default switch name, use a switch name that is unique and meaningful. Changing the switch name
causes a domain address format RSCN to be issued.
Switch names:
Can be up to 15-characters long
Must begin with an alpha character
Can consist of any combination of alphanumeric and underscore characters
SAN Overview - 58
Licensed features such as Fabric Watch and Performance Monitoring are already loaded onto
the switch firmware, but must be enabled with a license key. Once you have purchased these
features, you are provided with a key to unlock the features in the firmware.
You must log on to PowerLink or contact EMC Customer Support to convert the transaction key to a
license key.
Verifying License Activation
To verify that the required licenses are activated on the switch, perform the following steps:
Log into the command line interface as the Admin user.
Enter the licenseshow command at the command line prompt.
A list displays all of the licenses currently activated on the switch.
SAN Overview - 59
Configure the software features (such as Fabric Watch, Zoning, and Secure Fabric OS) for each switch.
To save time you may configure the software features on one switch, then save the configuration file,
and download it to the each of the remaining switches.
SAN Overview - 60
Power off all devices (to minimize PLOGIs) and connect them to the switch, according to your
topology. For devices that cannot be powered off, connect the devices, but use the portdisable
command to disable the port on the switch. When powering the devices back on, wait for each
device to complete the fabric login before powering on the next one.
EMC recommends that you create a blocking zone config and load that to the fabric before connecting
devices. More information on this is presented in the B-Series Zoning module.
To verify that you have fabric wide device connectivity, display the fabric wide device count. To
display the fabric-wide device count from a switch:
Log in to the switch as the admin user.
Enter the nsallshow command at the command line.
This command displays 24-bit fibre channel addresses of all devices in the fabric.
SAN Overview - 61
To verify that your switch is operating correctly, display information about the switch and port status.
To display information about the switch and port status:
Log in to the switch as the admin user.
Enter the switchshow command at the command line. This command displays a switch summary
and a port summary.
Verify that the device is connected to the switch by entering the nsshow [-r] command.
Use the -r option to replace the TTL attribute output with SCR (state change registration) information
in the display. SCR is the state change registration of a device. This value indicates what type of RSCN
a device registers to receive.
SAN Overview - 62
To verify that you have fabric-wide switch connectivity, display a summary of information about the
fabric.
To display a summary of information about the fabric:
1. Log in to the switch as the admin user.
2. Enter the fabricshow command at the command line. This command displays a summary of
all the switches in the fabric.
SAN Overview - 63
It is recommended that you make a hard copy backup of all key configuration data, including license key
information for every switch, and store it in a safe and secure place for emergency reference. Refer to the
following procedures.
Print out the information from the licenseshow command and store in a secure location.
Print out the information from the configUpload command (contains license and configuration information)
and store in a secure location.
Print out the information from the ipaddrshow command (select option 4 to display all configured
addresses) and store in a secure location.
Depending on the security procedures of your company, you may want to keep a record of the user levels and
passwords for all switches in the fabric. This is sensitive information and access to such information should be
limited. Save all key configuration data, including license key information for every switch and upload it to a
host for emergency reference.
The configuration file is written as three sections, and is broken up as follows:
The first section contains the switch boot parameters. It has variables such as the switch's name and IP
address. This section corresponds to the first few lines of output of the configshow command.
The second section contains general switch configuration variables, such as diagnostic settings, fabric
configuration settings, and SNMP settings. This section corresponds to the output of the configshow
command (after the first few lines), although there are more lines uploaded than shown by the command.
The third section contains zoning configuration parameters.
SAN Overview - 64
These are the key points covered in this module. Please take a moment to review them.

SAN Basic Overview

Uploaded by

Copyright:

Available Formats

You might also like

SAN Basic Overview

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SAN Basic Overview

Uploaded by

Copyright:

Available Formats

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

You might also like