FICON For CCIEstudy May PDF

FICON
101
CCIE SAN Study Group
By Ozden Karakok – May 2011
FICON © 2010 Cisco Systems, Inc. All rights reserved. 1

Objective & Agenda
At the end of this session, attendees will be able to answer what is FICON is all about!
  FICON Concepts
•  Terminology
•  Topologies
•  Addressing
•  Frames
  FICON Applications
  MDS Requirements
•  FICON Manager
•  FICON Control Unit Process (FCD)
  Configuring FICON
•  SAN-OS
•  Mainframe
  SAN-OS 3.0 Update
  Troubleshooting FICON
  Summary

FICON Terminology

IBM Mainframe Systems
  Current IBM mainframe systems:

S/390 G5 OS/390
VM
S/390 G6
VSE/ESA
zSeries 800 MVS
zSeries 890
zSeries 900 z/OS
z/VM
zSeries 990 Linux
z9 Ent. Class
z9 Business C.
z10 Ent. Class
z10 Business C.

Introduction to FICON
FICON Protocol Layer
•  FICON is Fibre
Channel FC-4 layer
Single Byte protocol
ULP SCSI – 3/4 Command Code
(Upper Level Protocol) Sets •  It is at the same
layer as FCP
protocol
SCSI— FC-SB2/FC-SB3
FC - 4 Command Set
•  Fibre Channel
Mapping (FCP) (FICON) –”Single Byte
Command Code”
FC - 3 mapping protocol
Common Services is described in FC-
SB3, FC-SB4
FC - 2 Framing Protocol/Flow control •  FC-SB2 is an older
version. FC-SB3-4
FC - 1 is superset.
Encode/Decode (8/10)
FC - 0 Ref: www.t11.org
Physical Variant
FICON Terminology
  A Channel is a physical entity/element of the processor, acting as
the interface between the CPU and the CU
–  A Channel Image instantiates multiple, logical channels behind a
physical channel
  A Channel Path includes the interface adapter, physical fiber pair,
and up to two intervening FICON switches/directors between the
channel and the CU
  A Channel Path Identifier (CHPID) is a single hexadecimal byte
value identifying each channel path to the system

FICON Terminology
  Outboard I/O processors called Control Units (CUs)
–  CUs can be stand-alone or integrated in peripheral devices
–  CUs are identified by a two-byte hexadecimal CUNUMBER
–  A Control Unit Image instantiates multiple, logical CUs behind
a physical CU
  A Device Address (aka Unit Address) is a decimal value
that selects a specific device on a given CU image

FICON Terminology
  A “Channel” is an entity
typically mapped to an N-port
in a host computer which
performs access to I/O devices
Channel
M IBM
A Control Unit   There can be multiple
Channel
I Image “Channel Images” behind one
Images N port, each of which represent
N a separate logical channel.
F
R
A
M N Port   A “Control Unit” is the interface
E to I/O devices such as printer,
disks etc.
N Port
Legend
Physical   There can be multiple “Control

Unit images” behind each
Logical Control Unit (CU) “Control Unit”

FICON Terminology
  A Logical Partition (LPAR) is a subset of mainframe

resources, dedicated to implementing a particular
programming environment
–  Each LPAR functions independently of other LPARs and is
protected against the actions of poorly-behaved LPARs
–  Resources such as CHPIDs (Channel Path Identifier) may be
shared between LPARs

MIF
LPARs can share channels
Multiple Image Facility (MIF)
MIF
LPAR 1
Channel N_Port CU
N_Port
LPAR 2
N_Port CU
LPAR n

FICON Terminology
  An Output Port is the FICON director or switch port that connects
to the CU
–  In a switched point-point environment, it can be a single byte
hexadecimal value identifying the director/switch port number
connected to the CU
–  FICON port numbering schemes are proprietary to the switch vendor/
model; typically different than what’s used for FC config
–  The port number may also be added with the switch domain ID to
form a two-byte Output Port value if supported by the CU
– In a cascaded environment, Output Port must be a two-byte value
derived from the non-Entry Switch

FICON Cascaded Topology
Output Ports
Entry Switch Non-entry switch
CH =
Initiator
CU = Target
Switch ID = Domain ID Portaddress = int fc X/Y

FICON Terminology
  The Input Output Control Program (IOCP) is a

mainframe application that defines the Input Output
Control Data Set (IOCDS), which the mainframe uses
to define & identify control units, devices, channel
paths, channels and their relationships to each other.
  I/O definition file (IODF)
  All of the terms we’ve just covered are required IOCP/
IOCDS parameters

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
FICON Topologies
  A Point-to-Point (direct connection)
  A Switched Point-Point Environment results when there is one
FICON director/switch in the channel path between the channel
and CU
  A Cascaded Environment results when two FICON directors/
switches are connected in series along the channel path
between the channel and CU
  An Entry Switch is the FICON director or switch connected to the
channel via the channel path
–  In a cascaded environment, the Entry Switch is first switch
encountered in the channel path from the channel to the CU
–  The Entry Switch is identified by a single hexadecimal byte equal
to it’s FC domain ID

FICON Topologies (Contd)
Point to Point
FICON   In Point to Point topology, channel
Channel and control unit are directly
connected. There is no switch
between them.
FICON
Control
Unit

Single Switch Fabric Path
FICON FICON Configuration
Channel Channel
  In Single Switch Fabric path
configuration, all channels and
control units are attached to a
single switch
  This is called ‘switched point-to-
point’ configuration.
FICON FICON FICON

Control Control Control
Unit Unit Unit

Two Switch Fabric Path
FICON FICON Configuration
Channel Channel
  In Two Switch Fabric path
configuration there may be more
than 2 switches in the fabric, but
only one hop between any pair of
channel and control units.
  Multiple ISLs between connected
switches are allowed. However, the
fabric should perform load
balancing on these ISLs.
FICON FICON FICON

Control Control Control
Unit Unit Unit

FICON Addressing
8 Static in Link Address
Link
FICON fabric   FICON defines concept of link
address (usually 0) address.
  Each channel and control unit have
one unique link address.
Domain Area Port   In older versions, link address used

to be single byte and could only be
used in single switch environment
Single Byte Link address to FCID mapping   FC-SB3 defines two byte link
addresses which allow more than
one switch in FICON path.
16 Static in   Link addresses are mapped to FC-
FICON fabric ID as depicted in figure.
Link address (usually 0)
Domain Area Port
Two Byte Link address to FCID mapping

(for Cascaded environments)
HCD – Point to Point

HCD – Cascaded

Using DM to identify Port Addresses
9513 - 16 ports per slot 9509 - 32 ports per slot
Notice FICON port

numbers are displayed

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
FICON Channel Initialization
Same Well
Known Fibre
CH to Switch Channel
Addresses
CH to CU

FICON Control Unit Initialization
Can register
Defined both SCSI and
In FC-FS FC-SB FC-4
types
port-wwn (vendor) :50:05:07:63:00:c2:04:2b (IBM)

node-wwn :50:05:07:63:00:c0:04:2b
Defined class :2,3
node-ip-addr :0.0.0.0
In FC-GS ipa :ff ff ff ff ff ff ff ff
fc4-types:fc4_features:scsi-fcp fcsb2-ch-cu fcsb2-cu-ch
[snip]

QSA – Query Security Attributes
  Used by Channel to verify Fabric Binding and Insistent
Domain IDs is configured on MDS
  Only used on Cascade environment
•  QSA = 0x7E 00 00 00
•  Revision field = 0x00 00
00 01
QSA Accept
  QSA Accept sent if Security Features enabled
Bit Meaning
0 Fabric Binding
1  Static DomainID
2  Soft Zoning
3  Hard Zoning
4  SLAP
5  SRP
6  MS CT
0000 0011 Bit 0 & 1 are on
Authentication
7654 3210 BIT numbers 7-31 Reserved

ESA – Exchange Security Attributes
  Used between Switches (ISL) to validate QSA information

  Passed during ISL link initialization
  Address by and to MDS Fabric Controller 0xFF FF FD
  ESA frame command code is 0x32 00 00 00
272.210534 ff.ff.fd -> ff.ff.fd 0xa 0xffff SW_ILS 0x32000000! ESA
272.210594 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1! Class F Ack
272.241828 ff.ff.fd -> ff.ff.fd 0xa 0xa SW_ILS SW_ACC (0x32000000)! ESA Accept
272.242168 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1!
Class F Ack

RNID – Request Node Id Data
•  Used to determine characteristics of node at other end

RNID Accept
  Accept is broken into two sections:
Common ID Field - shows

pWWN and nWWN.
Ignored by FICON
Specific ID Data – shows

SB-2, FCID, Name,
Manufacturer, Model…..
LIRR, RLIR, DRLIR
  LIRR (Link Incident Record Registration) used to
register for RLIR (Registered Link Incident Record) -
similar to SCR
  RLIR used by N-Ports to send link incidents to
registered ports - similar to RSCN
  DRLIR (Distribute Registered Link Incident Records)
used to send incidents to entire fabric
  Switch receives DRLIR and send RLIR to registered N-
Ports
  LIRR sent to all Switches in path and Control Unit
  Sent to 0xFF FF FA (MDS Management Server)

ELP – Establish Logical Path
LPE – Logical Path Established
•  ELP - Sent from

Channel to Control
Unit to verify optional
parameters and verify
logical path
• LPE – Sent from
Control Unit to
Channel as
verification of logical
path and pass
optional parameters
• LACK – Close
exchange

Frame Format
FC-SB-3 Frame Format
Bytes: 4 24 32 0 - 8160 4 4
S FC FICON E
Idles O Data Payload CRC O Idles
F Header Header F
Bit 0 7 8 15 16 23 24 31
Reserved CH_ID Reserved CU_ID

Device Address Reserved SB-3 Header
IUI DH_FLGS CCW Number
Reserved Token IU Header
Command / Data / Status / Control / Link Header
Reserved IU Count DIB Data Count DIB Header

LRC

FICON Read Operation
FICON FICON FCP FCP

Channel CU Initiator Target
Command DIB (READ) FCP_READ
Command DIB (RSP) FCP_DATA
Data DIB FCP_DATA
Data DIB FCP_RSP
Status DIB
FICON Read Operation FCP Read Operation

FICON Write Operation
FICON FICON FCP FCP

Channel CU Initiator Target
Command DIB (WRITE) FCP_WRITE
Command DIB (RSP) FCP_XFER_RDY
Data DIB FCP_DATA
Data DIB FCP_DATA
Status DIB FCP_RSP
FICON Write Operation FCP Write Operation

Flow Control
FICON uses standard FC-2 BB_Credit flow control

FICON also uses IU Pacing:
Similar to Fibre Channel EE_Credits
IU Pacing credit is sent to the channel by the CU
Indicates the maximum number of IUs a channel can send
before a command response IU is expected
IU Pacing
CRR CMR
Channel CU
Frames Frames
N_Port F_Port E_Port E_Port F_Port N_Port
R_RDYs R_RDYs
FC-2 BB_Credit FC-2 BB_Credit

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
Linux Support
IBM zSeries supports Linux LPARs:
Allows Linux to access ESCON/FICON storage
Linux also supports open systems FCP:
Same adapters used for FICON and FCP
Different firmware versions
Applications:
CPU-intensive
workloads FICON FC
Server/storage FICON FC
consolidation LINUX
NPIV (like MIF for FCP
FCP channels) FC
z/OS FICON FC FC
FICON FC
FC
FC
zSeries server
Disaster Recovery Applications
Data replication over FICON:

Backup, snapshot, and disaster tolerance
Local and remote replication
Globally Dispersed Parallel Sysplex (GDPS) w/
IBM XRC
IBM PtP VTS
GDPS w/ IBM PPRC (runs over FCP)
HDS HXRC

Disaster Recovery Applications
XRC - eXtended Remote Copy

Channel Extension
  Channel extension is different than distance extension:

Distance extension simply means extending FICON over a
WAN
Channel extension involves protocol translation
Channel extenders proxy some commands—e.g. writes—to
increase performance over distance
MDS does not do channel extension in SAN-OS 3.X

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
MDS FICON Requirements
Feature Explanation
Predictable and persistent Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to
FCID allocation be predictable and persistent.
FICON also requires uniform “port” byte (last byte in FC-ID) in FICON
fabric.
Support for new LIRR, RLIR, Link Incident Record (LIR) ELS reports link level events in the fabric
and DRLIR ELS to the channel N-port.
Examples of link incidents are loss of signal or synchronization,
primitive sequence timeout, Invalid primitive sequence, NOS
received
Concept similar to RSCN
Report Node Identification RNID is an ELS command, which can have common and specific
(RNID) ELS Node data.
FICON defines a specific node data field format.
Large Buffer to Buffer Credit FICON requires 60 B2B credit for 100 Km and 6 B2B credit for 10 Km.
Allocation MDS 9000 FICON supports up to 255 B2B Credits per Port +
extended buffer credits

MDS FICON Requirements
Feature Explanation
Insistent Domain ID Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to be
predictable and hence Domain ID to be static in cascaded FICON
switches.
In-order Delivery FICON requires in-order delivery for class 2 and class 3 frames.
Fabric Binding and EFMD ELS Fabric Binding feature prevents unauthorized switches joining the fabric
based on switch WWNs.
Exchange Fabric Membership Data (EFMD) is used to distribute this
information fabric wide.
Security Attributes (QSA and FICON requires Fabric Binding and Insistent Domain ids to be enabled
ESA) in the entire fabric.
Query Security Attribute (QSA) is used by Channels to ensure these
features are turned on.
Exchange Security Attribute (ESA) is used by switches to validate these
attributes during E port bringup.

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
What is FICON manager?
FICON Manager
Running process that manages functionality required
to support FICON in a VSAN, doing the necessary
checks for enabling FICON and holding all FICON
specific configuration information.

FICON Manager
  FICON Manager is responsible for:

-  Providing CMI interface for configuration
-  Serialization of FICON config
-  Maintain and apply FICON configuration files
-  Port Number Allocation and assignment
-  Port Address Allocation and assignment
-  Consistency Checks during enabling of FICON
-  Maintaining port numbers in FICON VSANs
-  Interact with ACL Manager for necessary ACLs during port bringup
-  Exchanging new ELS During E-port bringup

FICON Manager Role
CLI
FCD
Mainframe SNMP GUI
(CUP)
FICON
Manager
FC2 Others
Vsan Port
Manager Manager

FICON Configuration Files
  Up to 16 configuration files per FICON VSAN

- Restricted to 8 alphanumeric characters
  Shared lock for file access

  Proprietary IBM format IPL contains the following
attributes for each implemented port:
-  Port Address Name
-  Host Control
-  Block
-  Prohibit Dynamic Connectivity Mask
-  ‘show ficon vsan <x>’ will show saved IPL files

Port Number and Port Address
  Implemented/Unimplemented Port Number Slot 91xx 92xx 9506 9509 9513
Refers to any available port in the chassis
Reflected in the table to the right 1 0 - 39 0 - 31 0 - 31 0 - 31 0 - 15
  Installed/Uninstalled Port
Only ports participating in FICON VSANs are
2 32 - 63 32 - 63 32 - 63 16 - 31
considered ‘installed’
i.e. If port 4 is in a FICON VSAN on a 16 port line 3 64 - 95 64 - 95 32 - 47
card in a chassis, then ports 0-3, 5-31 are
considered ‘uninstalled’ 4 96 - 127 96 - 127 48 - 63
  Port number is the physical location on the switch
  Port address is:
5 -- -- 64 - 79
-  Port name
-  Block/Unblock state
6 -- -- 80 - 95
-  Connectivity attributes 7 128 - 159 --
-  Same as port number unless port “swapped”
  256 possible port number/addresses for 9509 (8 8 160 - 191 --
bits, some unimplemented)
  0xFE and 0xFF reserved
9 192 - 223 96 - 111
  An FCIP tunnel must be explicitly bound to a port 10 112 - 127
number for FICON
- Available port numbers depends on the slot the
IPS blade is located in 11 128 - 143
  A Port Channel must be explicitly bound to a port
number for FICON
12 144 - 159
-  Available port numbers depends on chassis ‘show
ficon first-available port-number’ reveals next port-
13 160 - 175
number free for allocation
Max
40 64 128 224 528
Physical*
Port
40-55 64-79 128-143 224-249 224 - 249
Channels

Using DM to identify Port Addresses
9513 - 16 ports per slot 9509 - 32 ports per slot
Notice FICON port

numbers are displayed

FICON States
  Various states for FICON
STATE Meaning
Enabled User has configured FICON on a given VSAN.
Disabled User configures a non-FICON vsan.
Up FICON enabled and all prerequisites met

FICON enabled, but one or more prerequisites not
Down satisfied

FICON Prerequisites
  Before FICON is ‘up’, FICON manager must check

prerequisite conditions:
  In-order delivery enabled
  Fabric Binding
  Insistent Domain ID
  No conflicting persistent FCIDs
-  Does not allow user configuration for reserved range:
D.0.0 thru D.MaxPa.0 where:
D is domain
MaxPa is max Port Addr implemented in FICON

FICON Port Initialization
  Blocked/Unblocked State
-  A blocked port is configured ‘down’ by Port manager, at request of FICON
manager
-  Blocked ports will continue to transmit an Off-Line State (OLS) primitive
sequence.
-  Similar to shut/no shut, as blocked port must be reinitialized
- Exception is TE port:
- Removal of the VSAN from the VSAN allowed list
- Port Channels
- Members get port channel configuration (block/unblock)
  Prohibit State
-  FICON manager programs ACLs to allow communication between certain
ports and prohibit others
-  Similar concept to zoning
-  Port remains ‘up’

FICON Port Initialization ELS
  Replies to Query Security Attributes (QSA)

-  For Fx, Responsible for sending:
-  Fabric Binding
-  Insistent Domain ID
-  In-order delivery
  Sends Exchange Security Attributes (ESA)

-  Upon bring up of (T)E Ports
  Sends Request Node Identification (RNID)

-  Sent after QSA reply

FICON Port Swap Support
  Port Swap feature

-  Must be supported for FICON directors
-  Enables swapping configuration on two physical ports
-  FICON manager swaps all config except Port #
-  Ports left in “shut” state following…’no shut’ required
FICON FICON
Configuration Configuration
Including Including
Port Address, Port Address,
Block State, etc. Block State, etc.
Physical Port A Physical Port B

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
FICON Control Unit Port (CUP)
  CUP protocol provides inband
management of switch from
mainframe
  In order to be classified as
“FICON director”, CUP protocol
needs to supported.
  CUP specification is proprietary
to IBM and is available only
through special license
agreement with them.
  CUP can configure a port to be
prohibited from another port,
blocked, etc.
  Complimentary to SNMP, CLI,
SANOS 3.0(2) View of FICON IPL File and is serialized through CMI
  FICON Manager holds all
configuration and FICON specific
information; FCD acts as a client

FICON Control Device (FCD)
  FCD process is responsible for:

-  Acquiring persistent pWWN for Control Device
-  Uses Key-Based Chassis Specific (KCS) WWNs
-  Uses VSAN based nWWN
-  Acquiring FCID from FPort server
-  0xFE is well defined address required
-  FCID will be <domain, 0xFE, 00> for Control Device
-  Device allegiance based Serialization of FICON config
-  Support for both SB-3 configuration and query commands

FICON Port Configuration
Block Block all communication from a port. In this state port

is down and will not send any information.
Prohibit Each port (identified by port address) can block any
other port. This is done via ACLs.
Port Address Each port (address) can have a 24 character port
Name address name.
Port Swap Reassigns port address, connectivity attributes and
port address name of ports by swapping.
Service This is set when FICON Director determines that some
Required service is required on the port. Port is kept offline.
Cleared when required service performed.
Port Statistics The MDS needs to maintain detailed statistics on each
port. The statistics are reported based on port
numbers and not port addresses

HCD - CUP

FCD Control Device
An example of an FCD Control Device on a FICON vsan:
switch2# show ficon control-device sb3 vsan 1977
Control Unit Image:0x80b9df4

VSAN:1977 CU:0x4dfe00 CUI:0 CUD:0 CURLP:(nil)
ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0
PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
Control Unit Image:0x80bcbec

VSAN:1978 CU:0x4efe00 CUI:0 CUD:0 CURLP:(nil)
ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0 Domain ID = 77 (0x4d)
PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0 Well known port = 0xFE
ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
pWWN assigned
switch2# show flogi database vsan 1977
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
sup-fc0 1977 0x4dfe00 22:00:00:05:30:00:46:e0 27:b9:00:05:30:00:46:df
Total number of flogi = 1.

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
CLI Configuration of FICON
An example of using the ‘setup ficon’ script:
switch1# setup ficon
--- Ficon Configuration Dialog ---
This setup utility will guide you through basic Ficon Configuraton on the system.
Press Enter if you want to skip any dialog. Use ctrl-c at anytime to skip all remaining dialogs.
Would you like to enter the basic configuration dialog (yes/no)[yes]:yes
Enter vsan [1-4093] :3
vsan 3 does not exist, create it? (yes/no)[yes]:yes
Enable ficon on this vsan? (yes/no)[yes]:yes
The following options required for ficon will be enabled:
In-order-delivery
Fabric binding
Configure domain-id for this ficon vsan (1-239) :3
Would you like to configure ficon in cascaded mode: (yes/no)[no]:yes
Configure peer wwn (hh:hh:hh:hh:hh:hh:hh:hh): 20:00:00:0d:ec:0e:5b:80
swwn from
Configure peer domain (1-239) :4 remote switch
Would you like to configure additional peers: (yes/no)[no]: # show wwn switch
Enable SNMP to modify port connectivity parameters? (yes/no)[yes]:
Disable Host from modifying port connectivity parameters? (yes/no)[no]:
Enable active=saved? (yes/no)[yes]:no
Would you like to configure additional ficon vsans (yes/no)[yes]:no

The following configuration will be applied:
vsan database
vsan 3
in-order-guarantee vsan 3
fcdomain domain 3 static vsan 3
fabric-binding enable
fabric-binding database vsan 3
swwn 20:00:00:0d:ec:0e:5b:80 domain 4
fabric-binding activate vsan 3
zone default-zone permit vsan 3
ficon vsan 3
no active equals saved

A FICON port-channel configuration example:
switch1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDS9506(config)# int port-channel 1
Remember that PortChannels need to be assigned FICON port numbers, so in

the next command we are determining what the next available FICON port number is.
switch1(config-if)# do show ficon first-available port

Port number 128(0x80) is available
Next step is to bind the FICON port number, set the port type to an ISL (E_Port),
and set it up so that only VSANs 1 and 10 can trunk across the PortChannel.
switch1(config-if)# ficon portnumber 0x80

switch1(config-if)# switchport mode e
switch1(config-if)# switchport trunk allowed vsan 1
switch1(config-if)# switchport trunk allowed vsan add 10
Next we move interfaces fc1/9 and fc2/1 into PortChannel 1
switch1(config-if)# int fc1/9

switch1(config-if)# channel-group 1 force
fc1/9 added to port-channel 1 and disabled
please do the same operation on the switch at the other end of the
port-channel,then do "no shutdown" at both ends to bring them up

port-channel,then do "no shutdown" at both ends to bring them up

A FICON port-channel configuration example continued:
Next we no shut interfaces fc1/9, fc2/1, and PortChannel 1.
switch1(config-if)# int fc1/9, fc2/1

switch1(config-if)# no shut
switch1(config)# int port-channel 1
Next we repeat the configuration steps on the switch2.
switch2(config)# int port-channel 1

switch2(config-if)# do show ficon first-available port-number
Port number 64(0x40) is available
swich2(config-if)# ficon portnumber 0x40
switch2(config-if)# switchport trunk allowed vsan 1
switch2(config-if)# switchport trunk allowed vsan add 10
swich2(config-if)# int fc1/1
port-channel, then do "no shutdown" at both ends to bring them up

port-channel,
then do "no shutdown" at both ends to bring them up
switch2(config)# int fc1/1, fc1/9

switch2(config)# interface port-channel 1
switch2(config-if)# end

CLI RLIR
MDS-1# show rlir ?
erl Show Established Registration List
history Show link incident history
internal Show RLIR internal
recent Show recent link incident
statistics Show RLIR Statistics
MDS-1# show rlir internal vsan 3

Internals for VSAN: 3
------------------------
VSAN state : Active

Ficon State : Up
Fabric Name : 20:03:00:05:30:00:51:1f
R_A_TOV : 10000 ms
Local Domain : 0x3(3)
Remote Domains : None
MDS-1# show rlir statistics vsan 3

Statistics for VSAN: 3
-------------------------
Number of LIRR received = 0

Number of LIRR ACC sent = 0
Number of LIRR RJT sent = 0
Number of RLIR sent = 0
Number of RLIR ACC received = 0
Number of RLIR RJT received = 0
Number of DRLIR received = 0
Number of DRLIR ACC sent = 0
Number of DRLIR RJT sent = 0
Number of DRLIR sent = 0
Number of DRLIR ACC received = 0
Number of DRLIR RJT received = 0

CLI RNID and Port attributes
MDS-1# show ficon vsan 5 portaddress 0x21

Port Address 33(0x21) is up in vsan 5
Port number is 33(0x21), Interface is fc1/3
Port name is
Port is not admin blocked
Prohibited port addresses are 250-253,255(0xfa-0xfd,0xff)
Admin port mode is FX
Port mode is F
Peer is type 002064 model 104 manufactured by IBM
MDS-1# show ficon vsan 5 portaddress brief installed

-----------------------------------------------------------------------------
Port Port Interface Admin Status Oper FCID
Address Number Blocked Mode
-----------------------------------------------------------------------------
0x40 0x40 port-channel 5 off up TE --
0xfe 0xfe -- off up -- --

CLI FICON File Configuration
switch1# show run
ficon vsan 2
file IPL
switch1(config)# ficon vsan 2

switch1(config-ficon)# ?
FICON Configuration:
active Configure automatic saving of ficon configuration
code-page Ficon configuration for codepage on a VSAN
do EXEC command
end Exit from configure mode
exit Exit from this submode
file Ficon file configuration on a VSAN
host Enable host control of ficon
no Negate a command or set its defaults
portaddress Ficon configuration for portaddress on a VSAN
snmp Enable snmp control of ficon
switch1(config-ficon)# portaddress 1
switch1(config-ficon-portaddr)# ?
FICON config for portaddress:
block Block a range of portaddress
do EXEC command
end Exit from configure mode
exit Exit from this submode
name Give name for a portaddress
no Negate a command or set its defaults
prohibit Prohibit communication with a range of portaddress

Creating FICON VSAN from DM

Creating FICON VSAN from FM

Creating/Modifying FICON Files from DM

FICON Configuration File (IPL)
Edit IPL file, Block – Prohibit – Name Ports

Creating Port Channel through FM
Now select the Port Channel Wizard

Determine the ISLs to add to the Port Channel

Creating Port Channel through FM
(cont’d)
Select a Port number for the logical PC
Fill in Port Channel Information Port Channel is now created

Showing Port Channel through DM
Select Interface->Port Channels
All Port channel information is displayed

Swapping FICON Ports through DM
Select Ports to swap Select FICON->Swap Selected Ports

Swapping FICON Ports through DM
(contd)
Notice the swapped ports

FICON VSAN Options
•  Offline SW = Enables ability for Host to disable FICON or offline MDS

•  Sync Time = Enables ability for Host to sync MDS time with Host
•  Port Control – Host = Enables ability for Host to Name, Prohibit, Block MDS
ports
•  Port Control – SNMP = Enables ability for Device Manager to Name, Prohibit,
Block MDS ports
•  Active=Saved = Enables ability for IPL (ficon running configuration) to be
saved automatically and immediately

FICON Load Balancing
  Packets of a flow (SID,DID) or an exchange (SID, DID,

OXID) are received in the order they were sent. Some
applications can not handle Out Of Order exchanges
  FICON is one such application. In fact FICON uses two
unique exchanges, one for the entire outbound exchange
(For example a write chain) with an OXID allocated by
the CH. The CU sends back a command response with a
different OXID representing the inbound exchange

FICON Load Balancing
  The FICON load balancing tool has been enhanced in

the Fabric Manager 3.0 release.

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
Configuring Devices on Mainframe
Consider the following example and IOCP definition

Configuring CHPID 89 online HCD
SDSF
TSO Command Input

Varying devices online
SDSF
TSO Command Input
HCD
Vary devices D40 through D74 online. I/O errors occur

because the volumes at these addresses are not
initialized.
V D40-D74,ONLINE
IEF503I UNIT 0D50 I/O ERROR
IEF503I UNIT 0D51 I/O ERROR
...
Following command automatically executed based on

the range that was varied on in previous command.
D U,,,0D40,00052,L=OZI-Z VARY RANGE DISPLAY

IEE457I 10.55.06 UNIT STATUS 198
UNIT TYPE STATUS VOLSER VOLSTATE
0D40 3390 O PPRD40 PRIV/RSDNT
0D41 3390 O PPRD41 PRIV/RSDNT
…
0D50 3390 OFFLINE /RSDNT

Displaying Devices on Mainframe
Notice the online state for HCD

configured CHPID 86
TSO Notice the availability of devices

configured in HCD
HCD
HCD
TSO

Displaying CHPID 86 and 89 Matrix
52 Devices
Unit = 3390
Begins at D40
CUP Device
Unit = 2032
at well-known
0xFE

Displaying Path Matrix for Device D40
CHPID with corresponding

Destination Link Address
Entry Link Address

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
New FICON Port Numbering scheme
  Port-numbers are user-configurable
  Duplicates allowed
-  Port-number 1 can be assigned to fc1/1 and fc2/1
  Assigned on per-slot or logical basis

ficon slot <x> assign port-numbers <range>
ficon logical-port assign port-numbers <range>
  Default configuration
-  No change for 1, 2, 6 and 9 slot chassis
-  13 slot chassis will have 11 slots w/ 16 port-numbers each
  HA compatibility

Configuring Port-Numbers
Consider the following 9509 with LC modules in slots 2, 3, 4, and 8
switch1# sh module
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
2 24 1/2/4 Gbps FC Module DS-X9124 ok
3 8 IP Storage Services Module DS-X9308-SMIP ok
4 16 1/2 Gbps FC Module DS-X9016 ok
5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active *
6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby
8 4 10 Gbps FC Module DS-X9704 ok
Finding out what is assigned – this example show the defaults for a 9509
switch1# show ficon port-numbers assign

ficon slot 1 assign port-numbers 0-31
ficon logical-port assign port-numbers 224-249

Unreserve FICON port-numbers from an unused slot, slots 1, 7, & 9 in this case
switch1# show ficon port-numbers assign

ficon logical-port assign port-numbers 224-249
switch(config)# no ficon slot 1 assign port-numbers 0-31

Reserve port-numbers for slot 2, a 24 port card
switch(config)# ficon slot 2 assign port-numbers 128-151

The output should reflect the current status for all the ports
switch(config)# do show ficon vsan 2 portaddress brief | inc fc2
0x80 0x80 fc2/1 off not installed -- --

0x8a 0x8a fc2/11 off not installed -- --
0x8b 0x8b fc2/12 off not installed -- --
0x8c 0x8c fc2/13 off not installed -- --
0x8d 0x8d fc2/14 off not installed -- --
0x8e 0x8e fc2/15 off not installed -- --
0x8f 0x8f fc2/16 off not installed -- --

Also the running config will show the newly reserved ports
Switch-9509(config)# do sh run | incl assign

no ficon slot 1 assign port-numbers
ficon slot 2 assign port-numbers 128-151 force

Assign logical port-numbers for FCIP interfaces and Port-channels
switch(config)# ficon logical-port assign port-numbers 16-31,224-249
Logical ports are shown in a show running config
Switch-9509(config)# do sh run | incl assign

no ficon logical-port assign port-numbers
ficon slot 2 assign port-numbers 128-151 force
ficon logical-port assign port-numbers 16-31, 224-249 force

Objective & Agenda
•  Terminology
•  Topologies
•  Addressing
•  Frames
•  SAN-OS
•  Mainframe
  Summary
Fabric Binding Troubleshooting
  First, verify that identical Fabric Binding Databases are active on both
switches
  Debug information:
debug fabric-binding all (collect from both switches)
Activate the fabric binding database on both switchs while the debug is running.
fabric-binding activate vsan X
Flap the port-channel or ISL interface (Only if customer can take the outage)
  Collect the following

show fabric-binding statis vsan X (x=vsan 500)
show fabric-binding efmd statistics vsan X
show fabric-binding database vsan X
show fabric-binding database act vsan X
show fabric-binding internal global
show fabric-binding internal info vsan X
show fabric-binding internal event
show fabric-binding internal efmd event interface fcx/x or port-channel x

CUP Troubleshooting
Collect the following:
show tech-support details
show ficon control-device sb3
show ficon control-device internal info
show ficon control-device internal event-history vsan x (x=ficon vsan)
show ficon control-device internal event-history errors
show ficon control-device internal event-history msgs
show ficon control-device internal event-history vsan <x>
sh ficon internal event-history errors
sh ficon internal event-history msgs
sh ficon internal event-history vsan <x>
From the Mainframe:

MVS SYSLOG before, during, and after the failure.
Output from D IOS,FICON

Basic Troubleshooting
  Troubleshoot port problems (performance, failing) like any other case. Assume the switch is the problem
and rule it out!
attach mod x
term len 0
show hardware internal debug-info interface fc x/x
clear asic-cnt list-all-devices
clear asic-cnt all
clear asic-cnt device-id 3
  From Supervisor
show flogi internal event interface fcx/y (for both CH and CU interfaces)
show port internal event-history interface fcx/y
show ficon internal event-history vsan x interface fcx/y.
show ficon internal event-history vsan 1
show ficon internal event-history msgs
show ficon internal event-history errors
show ficon internal info

Basic Troubleshooting
The list of commands that are useful from SAN-OS(later NX-OS) CLI:
Setup ficon
ficon slot X assign port-numbers 48-63
show ficon port-numbers assign slot 3
show ficon port-numbers assign logical-port
show ficon vsan X portaddress brief
show ficon vsan vsan-id file name filename portaddress
show ficon vsan 20 portaddress 8 counters
ficon swap portnumber old-port-number new-port-number

ficon swap interface old-interface new-interface
clear ficon vsan 1 allegiance
ficon vsan 2 apply file SampleFile

ficon vsan vsan-id offline
ficon vsan vsan-id online
show ficon control-device sb3 vsan X
show flogi database vsan X

Basic Troubleshooting continued
  Gather the following Mainframe displays before and

after any failures or recreates
D M=DEV(nnnn)
D M=CHP(mmm)
Collect the MVS SYSLOG before, during, and after
the failure.
Collect any messages that may appear on Hardware
Console, for example, "LINK LEVEL ERROR -
FICON PLOGI ELS ERROR",

Summary / Review
  FICON uses the Single-Byte Command Code Sets-3 (SB-3) and
Single-Byte Command Code Sets-4 (SB-4)
  Three types of FICON topologies: Point-to-point, Switched point-to-

point (via a FICON Director) ,Cascaded FICON Directors (through
two FICON Directors)
  For FICON, there is a maximum of 1 hop from CH to CU
  Allow for fabric-binding between FICON fabrics
  Utilize ‘setup ficon’ command
  Utilize GUI (FM - DM) for basic troubleshooting / configuration

References
  FICON Planning and Implementation Guide:
http://www.redbooks.ibm.com/abstracts/sg246497.html?Open
  MDS FICON Configuration Guide:
http://www.cisco.com/en/US/docs/switches/datacenter/
mds9000/sw/4_1/configuration/guides/cli_4_1/ficon.html
  MDS Fabric Manager Configuration Guide
http://www.cisco.com/en/US/docs/switches/datacenter/
mds9000/sw/4_1/configuration/guides/fm_4_1/ficon.html

Question and Answer…
Thanks for attending this session and Good Luck in your CCIE Journey 


FICON For CCIEstudy May PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FICON For CCIEstudy May PDF

Uploaded by

Copyright:

Available Formats

FICON

By Ozden Karakok – May 2011

FICON © 2010 Cisco Systems, Inc. All rights reserved. 1

FICON © 2010 Cisco Systems, Inc. All rights reserved. 2

FICON © 2010 Cisco Systems, Inc. All rights reserved. 3

 Current IBM mainframe systems:

FICON © 2010 Cisco Systems, Inc. All rights reserved. 4

FICON © 2010 Cisco Systems, Inc. All rights reserved. 6

FICON © 2010 Cisco Systems, Inc. All rights reserved. 7

Physical  There can be multiple “Control

FICON © 2010 Cisco Systems, Inc. All rights reserved. 8

 A Logical Partition (LPAR) is a subset of mainframe

FICON © 2010 Cisco Systems, Inc. All rights reserved. 9

FICON © 2010 Cisco Systems, Inc. All rights reserved. 10

FICON © 2010 Cisco Systems, Inc. All rights reserved. 11

Switch ID = Domain ID Portaddress = int fc X/Y

FICON © 2010 Cisco Systems, Inc. All rights reserved. 12

 The Input Output Control Program (IOCP) is a

FICON © 2010 Cisco Systems, Inc. All rights reserved. 13

FICON © 2010 Cisco Systems, Inc. All rights reserved. 15

FICON © 2010 Cisco Systems, Inc. All rights reserved. 16

FICON FICON FICON

FICON © 2010 Cisco Systems, Inc. All rights reserved. 17

FICON FICON FICON

FICON © 2010 Cisco Systems, Inc. All rights reserved. 18

Domain Area Port  In older versions, link address used

Domain Area Port

Two Byte Link address to FCID mapping

FICON © 2010 Cisco Systems, Inc. All rights reserved. 20

FICON © 2010 Cisco Systems, Inc. All rights reserved. 21

Notice FICON port

FICON © 2010 Cisco Systems, Inc. All rights reserved. 22

FICON © 2010 Cisco Systems, Inc. All rights reserved. 24

port-wwn (vendor) :50:05:07:63:00:c2:04:2b (IBM)

FICON © 2010 Cisco Systems, Inc. All rights reserved. 25

FICON © 2010 Cisco Systems, Inc. All rights reserved. 27

 Used between Switches (ISL) to validate QSA information

272.210534 ff.ff.fd -> ff.ff.fd 0xa 0xffff SW_ILS 0x32000000! ESA

FICON © 2010 Cisco Systems, Inc. All rights reserved. 28

FICON © 2010 Cisco Systems, Inc. All rights reserved. 29

 Accept is broken into two sections:

Common ID Field - shows

Specific ID Data – shows

FICON © 2010 Cisco Systems, Inc. All rights reserved. 31

• ELP - Sent from

FICON © 2010 Cisco Systems, Inc. All rights reserved. 32

FC-SB-3 Frame Format

Reserved CH_ID Reserved CU_ID

Command / Data / Status / Control / Link Header

Reserved IU Count DIB Data Count DIB Header

FICON © 2010 Cisco Systems, Inc. All rights reserved. 33

FICON FICON FCP FCP

Command DIB (READ) FCP_READ

Command DIB (RSP) FCP_DATA

Data DIB FCP_DATA

Data DIB FCP_RSP

FICON Read Operation FCP Read Operation

FICON © 2010 Cisco Systems, Inc. All rights reserved. 34

FICON FICON FCP FCP

Command DIB (WRITE) FCP_WRITE

Command DIB (RSP) FCP_XFER_RDY

Data DIB FCP_DATA

Data DIB FCP_DATA

  Current IBM mainframe systems:

Physical   There can be multiple “Control

  A Logical Partition (LPAR) is a subset of mainframe

  The Input Output Control Program (IOCP) is a

Domain Area Port   In older versions, link address used

  Used between Switches (ISL) to validate QSA information

  Accept is broken into two sections:

•  ELP - Sent from

  Channel extension is different than distance extension:

  FICON Manager is responsible for:

  Up to 16 configuration files per FICON VSAN

  Shared lock for file access

-  ‘show ficon vsan <x>’ will show saved IPL files

  Various states for FICON

  Before FICON is ‘up’, FICON manager must check

  Replies to Query Security Attributes (QSA)

  Sends Exchange Security Attributes (ESA)

  Sends Request Node Identification (RNID)

  Port Swap feature

  FCD process is responsible for: