Brkcom 3001

Troubleshooting the Cisco UCS Compute Deployment
BRKCOM-3001
www.ciscolivevirtual.com
Agenda
Overview
Physical Setup Software Setup Configuration Path Trace
Maintenance
Key Takeaways
Q&A
BRKCOM-3001 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Overview
UCS Building Blocks

UCS Manager
Embedded manages entire system
UCS Fabric Interconnect

Nexus Switch
UCS Fabric Extender

Remote line card
UCS Blade Server Chassis

Flexible bay configurations
UCS Blade Server

Industry-standard architecture
UCS Virtual Adapters

Choice of multiple adapters
BRKCOM-3001
2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCS Manager 2.0 Release

Hardware
UCS 6248UP Fabric Interconnect UCS 2208XP IOM UCS 1280 VIC
Features
Unified Ports 80 Gb Host connectivity Fabric port-channeling L2 disjoint upstream in EHM VM-FEX for Red Hat KVM iSCSI Boot
Physical Setup

Standalone or Clustered
Primary / Subordinate Data Management Engine (DME)
Virtual IP
Management Network
IP #A
IP #B
DB
DB
FI-A# Cluster links

FI-B#
IO Module HIF to NIF Pinning (2104XP)

HIF
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
NIF
Server slots pinned to uplink
F E X F E X F E X
BRKCOM-3001
1 link
Fabric Interconnect
Uplink: slots 1,2,3,4,5,6,7,8
2 links
Fabric Interconnect
NIF 4 links
Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8
Fabric Interconnect
NIF
2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Uplink 1: slots 1,5 Uplink 2: slots 2,6 Uplink 3: slots 3,7 Uplink 4: slots 4,8
9

Slot 1 Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8
1-4
FEX
5-8
9-12 13-16 17-20 21-24 25-28
Fabric Interconnect
29-32
10

1-4
FEX
5-8
9-12 13-16 17-20 21-24 25-28
Fabric Interconnect
29-32
11

1-4
FEX
5-8
9-12 13-16 17-20 21-24 25-28
Fabric Interconnect
29-32
12

1-4
FEX
5-8
9-12
Fabric Interconnect
13-16
17-20 21-24 25-28
29-32
13
IO Module and Failover

Lose IOM link 1
FI
4 links active
IOM 1
Blade1 Blade2 Blade3 Blade4 Blade5 Blade6 Blade7 Blade8
BRKCOM-3001
Lose connectivity on mezzanine port mapped to IOM 1 for blades 1 and 5

Cisco Public
If you re-acknowledge theuse chassis, Blades Note: Only 2,3,4,6,7,8 1 and a re-ack 5 links willwill still is Which of the 2 to we will fallto back aout continue have required, not use ato chassis all 3 of theconnectivity remaining 32 islink a topology, and blades 1,3,5,7 will links the discover mezzanine policy port change random decision. use one uplink, and blades 2,4,6,8 connected to IOM 2 will use a different uplink
14
Port-Channel Pinning
No slot based pinning No invalid link count for NIF ports
VIC1280 adaptor with DCE links in Port-Channel
IOM
Pinned to Po
Gen-1 adaptor with single 10G link

15
Chassis Discovery Policy

Discovery policy only defines the minimum number of links necessary before a chassis can be discovered and NOT how many links will be utilised
BRKCOM-3001
Cisco Public
16
Software Setup
Upgrade
Upgrade Considerations
Release Notes will cover pre-requisites and concerns in the upgrade process Schedule an outage window
FI and IOM will reboot during upgrade Make sure network and storage fabric are redundant
Highly recommended to backup UCSM configuration Upgrade process is not quick so be patient Follow the upgrade procedure for each version
19
BRKCOM-3001
Cisco Public
Upgrade Process
Backup UCS Config (Full & All Config) Download firmware
Update components (adapters, CIMC, IOMs)

Activate components in order of:
Adapter cards Set Startup Only
CIMC
UCSM IOM Set Startup Only Fabric Interconnect BIOS/Storage Controller/etc (through firmware package)
20
Fabric Interconnect Modes
FI Ethernet Modes: End Host

Spanning Tree LAN
No Spanning Tree Protocol

MAC Learning
6x00 A vEth 3 Fabric A

VLAN 10
Maintains MAC table for Servers only

Allows Multiple Active Uplinks per VLAN Prevents Loops by preventing Uplink-toUplink switching
Cisco Public
vEth 1
L2 Switching
MAC Learning
VNIC 0
VNIC 0
Server 2
BRKCOM-3001
Server 1
22
End Host Mode Unicast Forwarding

LAN
Server to server traffic on the same VLAN is locally switched Each server link is pinned to an uplink port / port-channel
Server 2
Uplink Ports
6x00 vEth 1
Deja-Vu
RPF
Network to server unicast traffic is forwarded to server only if it arrives on pinned uplink port. This is termed as the Reverse Path Forwarding(RPF) check Packet with source MAC belonging to a server received on an uplink port is dropped (Deja-Vu Check)
Unknown unicast traffic is dropped
BRKCOM-3001 2012 Cisco and/or its affiliates. All rights reserved.
VLAN 10
vEth 3
VNIC 0
VNIC 0
Server 2
Cisco Public
Server 1
23
End Host Mode Broadcast/Multicast Forwarding

LAN
Pre 2.0 - Broadcast traffic is pinned on exactly one uplink port (or port-channel) i.e., it is dropped when received on other uplinks All multicast groups are pinned to same uplink port Server to server multicast traffic is locally switched RPF and deja-vu check also applies for multicast traffic
B
Broadcast Listener All VLANs
Uplink Ports
6x00 vEth 1 vEth 3
VNIC 0
VNIC 0
Server 2
Cisco Public
Server 1
24
FI Ethernet Modes: Switch

Root
LAN
FI acts as L2 switch Server vNIC traffic follows VLAN forwarding
6x00 A vEth 3 Fabric A

VLAN 10
MAC Learning
Rapid PVST+ is used Configuration of STP parameters (bridge priority, Hello Timers etc) or VTP is not supported MAC learning/aging happens on both the server and uplink ports like in a typical Layer 2 switch
vEth 1
L2 Switching
VNIC 0
VNIC 0
Server 2
BRKCOM-3001
Server 1
25
Hybrid Topology with direct-attach and SAN

SAN Fabric
Fabric A Direct Attach Storage Arrays Fibre Channel Fabric B Ethernet Unified I/O FCoE
Core
FC Storage FCoE Storage

SAN Edge A SAN Edge B
Security via Zoneset Merge and Security via LUN Masking

UCS B-Series
26
Unsupported Storage Topologies

Direct connect without zoning (no upstream MDS/N5K)
Multi-Hop FCoE or FCoE northbound on Ethernet Uplinks

FC connections between the two Fabric Interconnects running in FC switch mode (ISL) Connecting any other L2 based object off of the Appliance Ports
27
Disjoint Layer 2
End Host Mode Disjoint L2
SW1
SW2
SW3
SW4
BRKCOM-3001
Cisco Public
29
Disjoint L2 Pre 2.0 Workarounds

Switch Mode
Production VLAN 10-20 Backup VLAN 30-40 Production VLAN 10-20
End Host Mode

Backup VLAN 30-40
Prune VLANs
Prune VLANs
L2 Switch -A
Fabric Interconnect - A
L2 Switch -B
Fabric Interconnect - B
Fabric Interconnect - A
Fabric Interconnect - B
Eth 0
Eth 1
Eth 0
Eth 1
BRKCOM-3001
Bare Metal OS
Cisco Public
Bare Metal OS
30
EHM Disjoint L2 Upstream >= 2.0

Hardware independent: 6100 & 6200 support this feature with UCS 2.0 and above The ability to selectively assign VLANs to uplinks vNIC pinning decision based on - VLAN membership on uplink ports Allocating a designated broadcast/multicast receiver on a per VLAN rather than global basis
Fabric Interconnect - A Fabric Interconnect - B
Production VLAN 10-20
Backup VLAN 30-40
Eth 0
Eth 1
Bare Metal OS
31
Disjoint L2 Do NOT
1. Configure different range of VLANs across uplinks of a disjoint network
10 -20 Production Production VLAN 10-20 10-20 VLAN Backup Backup VLAN 30-40 30-40 VLAN
2. Overlap VLANs across L2 networks

3. Configure vNIC with VLANs across multiple L2 networks. vNIC does not get pinned if the uplink vlan membership is partial, vNIC stays down
1
Fabric Interconnect - A End Host Mode
11,13,15,17
20,30-40
Fabric Interconnect - B End Host Mode
IOM-A
IOM-B
VLAN 10,30
Adapter
vNIC-A vNIC-B
Half Width Blade
Cisco Public
32
VLAN Designated Receiver

1.4 = Connect nxos : show platform software enm internal info global | grep -A 6 'Global Params
2.0 = Connect nxos : show platform software enm internal info vlandb id <vlanID>
BRKCOM-3001
Cisco Public
33
Disjoint L2 Verification Overview

From UCSM CLI NXOS Context:
UCS-CiscoLive(NXOS)# show interface trunk
Should expect to see your associated VLANs on the assigned interfaces only.
Port Vlans Allowed on Trunk ---------------------------------------------------------Eth1/16 1,5-7,1000,3000 Only Eth1/16 allows VLAN 5 Eth1/17 1,6-7,1000,3000 <snip>
34
Blade Issues
Finite State Machine (FSM)

FSM runs as a workflow (FSM-Stages) Workflows are predefined -stages can be skipped if
not needed (in HA if remote is down, no vHBA configuration) FSM Flags (shallow checkpoint vs deep checkpoint)
Each Stage is an interaction between DME Application Gateway -> End Point
BRKCOM-3001
Cisco Public
36
FSM Details
Error Description for that stage

BRKCOM-3001
Operation (workflow)
Stage Description 37
Blade Issues
Server Blade discovery failed KVM connectivity
Hardware problems
Adapter issues
BRKCOM-3001
Cisco Public
38
Configuration
Configuration Validation & Errors
BRKCOM-3001
Cisco Public
40
Configuration Boot issues
Boot from SAN
SAN Boot in UCS

Must be defined in the boot policy applied to the service profile
SAN boot and Local HDD are mutually exclusive in the boot policy
During SAN boot, the local HDDs are disabled Some O/S will not install if more than one path to LUN is seen check O/S installation guide
BRKCOM-3001
Cisco Public
43
SAN Boot Checklist

Verify the following
Zoning and zoneset active Target Login LunID masked to host NPV/NPIV configuration FC uplinks SAN Admin
SAN+UCS Admin
FC VSAN membership
UCS FC module login into FC switch Service profile configuration Boot policy configuration
BRKCOM-3001
UCS Admin
Cisco Public
44
SAN Boot Failure Checklist

SAN configuration errors Zoning
Host and Target WWPNs Service Profile fc0/fc1
SAN boot order configuration

Zones and zonesets Matching VSANs
Lun Masking
Check that the access control configuration allows the host WWPN to access the array WWPN
LunID value is correct

45
SAN Boot Policy
BRKCOM-3001
Cisco Public
46
SAN Failures Unable to locate LunID

Both adapters (0 and 1) cant find the configured LunID
ROM BIOS fails and reports to setup there is no SAN device to boot from
BRKCOM-3001
Cisco Public
47

In Option ROM confirm that adapter was enabled by boot policy
In Option ROM confirm that adapter Boot was enabled and check target + LunID
BRKCOM-3001
Cisco Public
48

In Option ROM scan the fabric for visible LUNs Notice how the LunID configured is not seen, but another is
BRKCOM-3001
Cisco Public
49
iSCSI Boot
iSCSI Boot Support

iSCSI boot configuration is added to UCSM 2.0(1) Allows stateless model with no FC or FCoE in the environment Limited adapter support
Cisco VIC and VIC-1280 (iBFT only, no iSCSI offloads) Broadcom 57711 M51KR-B (full offload, iSCSI HBA)
Operating System support

ESXi 4.1U1, 4.1U2, 5.0 Windows 2008 R2 and R2 SP1 RHEL 5.6, 6.0 and 6.1
iSCSI Hardware Offload is not a requirement to support booting, only supporting iSCSI Boot Firmware Table (iBFT) in the option ROM
51
iSCSI Setup Flow

Create iSCSI vNICs and iSCSI Boot policy
Provide UCSM with iSCSI boot information
Target IP, IQN
Initiator IP/Mask/Gateway, IQN
Broadcom only Modify adapter policy for install vMedia map the OS and drivers if required Adapter successfully initialises Install OS and Drivers (if required)
Broadcom only Modify adapter policy for boot

52
iSCSI Boot Service Policy iSCSI vNIC

Overlay vNIC iSCSI adapter policy Identify native VLAN MAC address:
M81KR none (do not set) M51KR-B must be set (manual or pool)
BRKCOM-3001
Cisco Public
53
iSCSI Boot Service Policy Boot Policy

Set the iSCSI parameters Select the first iSCSI boot devices (you can have multiple iSCSI initiators)
BRKCOM-3001
Cisco Public
54
iSCSI Boot Service Policy Boot Parameters

Initiator name (EUI or IQN formats). Initiator IP Address Target name (Query your array for the correct name, usually IQN format)
Target Port, IP and LUN ID

Initiator Auth
CHAP Authentication
Initiator Auth to iSCSI array Target Auth from iSCSI array
Target Auth
55
Cisco M81KR VIC Verification
BRKCOM-3001
Cisco Public
56
M51KR-B Verification
BRKCOM-3001
Cisco Public
57
Path Trace
Ethernet troubleshooting and tracing
UCS Networking
Blade cannot talk to outside network
Is vethernet & uplink interface up?
Is vlan tagging correct?

Is the mac address being learnt on the FI and upstream switches? Can it reach other devices in the same L2? Can it reach its default gateway?
60
UCS Frame Flow Decisions

Northbound
LAN
FI A FI B
Which port in the port channel? Port Channeling Algorithm (2-8 choices)
Which uplink/border port or port channel? (many Border choices) Port Pinning
Local or remote destination? L2 Switching in FIs (2 choices) Which Fabric Port? Fabric Port (4 choices)
FEX
UCS Blade Chassis
Blade Server
CNA
CNA Port 1 Eth0 MAC A
CNA Port 2 Eth1 MAC B OS NIC Teaming
FEX
Pinning
Which UCS CNA Port? Fabric (2 choices)
Failover
OS Routing Table or Which PCIe Interface? OS Ethernet NIC Teaming (1-58 choices depending on CNA)
Cisco Public
OS
BRKCOM-3001
61
UCS Frame Flow Decisions

Southbound
LAN
FI A FI B
Which downlinkSwitch or port channel? (Upstream Decides) Allow the frame inbound? Dj vu, RPF, border port (decision depends on switch mode vs. end host mode) pinning Which Fabric Extender Port? Fabric Port Pinning
FEX
UCS Blade Chassis
FEX
Blade Server
CNA
CNA Port 1 Eth0 MAC A
CNA Port 2 Eth1 MAC B OS NIC Teaming
Which Server Bay Port?on (MAC Learning (8 choices)
VNTag + Offset FIs)
OS
Which PCIe Device (vNIC)? VNTag Identifier (1-58 choices depending on CNA)
Dest. MAC and Pass frame to OS?

Cisco Public
Ethertype binding
62
BRKCOM-3001
Service Profile Interfaces

show service-profile circuit server <chassis#>/<slot#>
BRKCOM-3001
Cisco Public
63
VIF Identities
Show host-eth-if
BRKCOM-3001
Cisco Public
64
Port Information M81KR Adapter

MCP level : vnic
BRKCOM-3001
Cisco Public
65

If too many vNICs, can find correct interface with vnicfind
BRKCOM-3001
Cisco Public
66

MCP level : lifstats <lifid>
BRKCOM-3001
Cisco Public
67
FEX Port Information

Connect nxos : show fex <chassis#> detail
BRKCOM-3001
Cisco Public
68
FEX Ports Information

2.0 = show system internal fex info satport ethernet <chassis#>/<adapter#>/<slot#>
1.4 = show platform software fex info satport ethernet <chassis#>/<adapter#>/<slot#>

69
IOM Ports Information

Connect iom <chassis #> show platform software <redwood|woodside> sts
BRKCOM-3001
Cisco Public
70

show platform software <redwood|woodside> rate
BRKCOM-3001
Cisco Public
71

show platform software <redwood|woodside> rmon 0 <interface>
BRKCOM-3001
Cisco Public
72
VIF Information
Connect nxos : show interface brief
Show interface vethernet <id>
BRKCOM-3001
Cisco Public
73
MAC Information
show mac address-table
BRKCOM-3001
Cisco Public
74
VIF Pinning
show pinning border-interface
show pinning server-interfaces
BRKCOM-3001
Cisco Public
75
Port Channel Information

show port-channel summary
BRKCOM-3001
Cisco Public
76
Port Channel Information

show port-channel load-balance forwarding-path interface port-channel <#> ...
BRKCOM-3001
Cisco Public
77
VLAN Designated Receiver

2.0 = show platform software enm internal info vlandb id <vlanID>
1.4 = show platform software enm internal info global | grep -A 6 'Global Params
78
SAN troubleshooting and tracing
Service Profile Interfaces

show service-profile circuit server <chassis#>/<slot#>
BRKCOM-3001
Cisco Public
80
VIF Identities
Show host-fc-if
BRKCOM-3001
Cisco Public
81
FC Interfaces M81KR
Attaching to the fls of the adapter provides the ability to view information related to SAN boot attributes The vnic command provides us with numbering associated with vNIC interfaces
BRKCOM-3001
Cisco Public
82
FC Login Information M81KR

Login information (Target PWWN and FCID) for each port can be obtained with the login <vnicID> command:
BRKCOM-3001
Cisco Public
83
FC Login Information M81KR

The lunmap command will confirm if the interfaces have logged into the target or not
BRKCOM-3001
Cisco Public
84
UCS 2.0: FC Login Information M81KR

Lunlist <vnic ID>
BRKCOM-3001
Cisco Public
85
FC Statistics
BRKCOM-3001
Cisco Public
86
VIF Information
Connect nxos : show interface brief
Show interface vfc <id>
BRKCOM-3001
Cisco Public
87
VFC Membership and Login

show vsan membership
show npv flogi-table
BRKCOM-3001
Cisco Public
88
FC Switch Northbound
Show flogi database
Show zoneset active
BRKCOM-3001
Cisco Public
89
Maintenance
System Event Logs

Chassis
Server
BRKCOM-3001
Cisco Public
91
SEL Configuration
Users can define rules (policies) for backing up and clearing SEL across all servers in the UCS system,
BRKCOM-3001
Cisco Public
92
UCSM Faults and Events Lifecycle

Faults Lifecycle
Clear Condition
Events Lifecycle
Raise Fault
Clear Fault Create Event
Delete Fault
Delete Retention Interval or Acknowledged
Clear Action Delete Event

Retain
Log Full
BRKCOM-3001
Cisco Public
93
BRKCOM-3001
Cisco Public
94
Data Gathering for SupportUCSM
BRKCOM-3001
Cisco Public
95
Core Dumps
Once TFTP core Exporter is configured and enabled, dumps will be transferred Once transferred, select and move to trash can
BRKCOM-3001
Cisco Public
96
O/S Troubleshooting Tools

Ping, mping, arping Arp
Esxcfg-nics, esxcfg-vmknic, ifconfig, ipconfig

Ethtool or lsmod or lspci or vmkload_mod or modinfo Esxtop, htop Tcpdump, Wireshark O/S Logs (/var/log/messages or event viewer) O/S Core Files (BSOD or PSOD)
97
Key Takeaways
Narrowing Down the Problem

Define the problem
From which point to what other point is the problem? Do we see the problem in one direction or both?
Eliminate variables
Is the problem seen between traffic traversing the same fabric? Is the problem only happening on a specific fabric path?
List all the ports in the traffic path

VIFs, HIFs, NIFs, Server ports and Uplink ports
99
What to do next?
Ask questions on Cisco Support Community Review documents at the links provided in the appendix section Raise TAC cases if you encounter issues If interested, speak with your Account Team about participating in Beta programs. Alternatively, come speak to me afterwards or email me: mipetrin@cisco.com
100
Q&A
Complete Your Online Session Evaluation

Complete your session evaluation:
Directly from your mobile device by visiting www.ciscoliveaustralia.com/mobile and login by entering your username and password
Visit one of the Cisco Live internet stations located throughout the venue
Open a browser on your own computer to access the Cisco Live onsite portal
Dont forget to activate your Cisco Live Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.
Cisco Public
BRKCOM-3001
102
BRKCOM-3001
Cisco Public
103

Brkcom 3001

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkcom 3001

Uploaded by

Copyright:

Available Formats

Troubleshooting the Cisco UCS Compute Deployment

UCS Building Blocks

UCS Fabric Interconnect

UCS Fabric Extender

UCS Blade Server Chassis

UCS Blade Server

UCS Virtual Adapters

2012 Cisco and/or its affiliates. All rights reserved.

UCS Manager 2.0 Release

UCS Fabric Interconnect

FI-A# Cluster links

IO Module HIF to NIF Pinning (2104XP)

Server slots pinned to uplink

Uplink: slots 1,2,3,4,5,6,7,8

slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8

Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8

slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8

IO Module HIF to NIF Pinning (2208XP)

IO Module HIF to NIF Pinning (2208XP)

IO Module HIF to NIF Pinning (2208XP)

IO Module HIF to NIF Pinning (2208XP)

IO Module and Failover

Lose connectivity on mezzanine port mapped to IOM 1 for blades 1 and 5

Gen-1 adaptor with single 10G link

Chassis Discovery Policy

2012 Cisco and/or its affiliates. All rights reserved.

2012 Cisco and/or its affiliates. All rights reserved.

Update components (adapters, CIMC, IOMs)

Fabric Interconnect Modes

FI Ethernet Modes: End Host

No Spanning Tree Protocol

6x00 A vEth 3 Fabric A

Maintains MAC table for Servers only

End Host Mode Unicast Forwarding

End Host Mode Broadcast/Multicast Forwarding

6x00 vEth 1 vEth 3

FI Ethernet Modes: Switch

FI acts as L2 switch Server vNIC traffic follows VLAN forwarding

6x00 A vEth 3 Fabric A

Hybrid Topology with direct-attach and SAN

FC Storage FCoE Storage

UCS Fabric Interconnect

UCS Fabric Interconnect

Security via Zoneset Merge and Security via LUN Masking

Unsupported Storage Topologies

Multi-Hop FCoE or FCoE northbound on Ethernet Uplinks

End Host Mode Disjoint L2

2012 Cisco and/or its affiliates. All rights reserved.

Disjoint L2 Pre 2.0 Workarounds

End Host Mode

2012 Cisco and/or its affiliates. All rights reserved.

EHM Disjoint L2 Upstream >= 2.0

Production VLAN 10-20

Backup VLAN 30-40

2. Overlap VLANs across L2 networks

Fabric Interconnect - B End Host Mode

Half Width Blade

VLAN Designated Receiver

2012 Cisco and/or its affiliates. All rights reserved.

Disjoint L2 Verification Overview

Finite State Machine (FSM)

2012 Cisco and/or its affiliates. All rights reserved.

Error Description for that stage