VPC Nexus 7k

Cisco Nexus 7000 Series NX-OS Troubleshooting Guide
This article provides guidance for troubleshooting issues that may appear when using Cisco Nexus 7000 Series. This article
introduces tools and methodologies to recognize a problem, determine its cause, and find possible solutions. However, these
documentation helps only in basic troubleshooting.
We encourage users to review the Cisco Live Presentation for detailed troubleshooting for Nexus 7000.
Sections of this presentation covers, both platform independent, and platform specific step by step troubleshooting for most
common issues. Access to this presentation is available FREE. Follow the below instructions to access the presentation.
1. Visit https://www.ciscolivevirtual.com/
2. Register for free.
3. Click on "Cisco Live Virtual" link.
4. Click on the ?Sessions? Tab on top, and select ?2011 Sessions Catalog?
5. In the search box, type ?BRKCRS-3144? and Submit search.
6. Select the session. You can either View the Session (or) download the pdf
Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to improve Cisco product documentation. Note that you
cannot log in to DocWiki with Cisco.com "guest" account credentials.
See Terms of Use and About DocWiki for more information about Cisco DocWiki.
Select the "edit" tab to edit an article or select the "discussion" tab to submit questions or comments about the article.
Click here to return to the Cisco Nexus 7000 Series documentation on www.cisco.com.
Contents
• 1 Audience and Generating a PDF of This
Guide
• 2 Organization
Audience and Generating a PDF of This Guide

This article is for experienced network administrators who configure and maintain NX-OS devices.
Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Book PDF
Organization
This article is organized into the following sections:
Troubleshooting Overview
Troubleshooting Installs, Upgrades, and Reboots
Troubleshooting Licensing
Contents 1
Troubleshooting VDCs
Troubleshooting CFS
Troubleshooting Ports
Troubleshooting vPCs
Troubleshooting VLANs
Troubleshooting STP
Troubleshooting Routing
Troubleshooting Unicast Traffic
Troubleshooting Multicast Traffic
Troubleshooting WCCP
Troubleshooting Memory
Troubleshooting Packet Flow Issues
Troubleshooting Service Failures
Before Contacting Technical Support
Troubleshooting Tools and Methodology
This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when
configuring and using Cisco NX-OS.
Guide Contents
Troubleshooting Overview (this section)
Troubleshooting CFS
Troubleshooting STP
Troubleshooting FCoE
Organization 2
Contents
• 1 Overview of the Troubleshooting
Process
♦ 1.1 Gathering Information
♦ 1.2 Verifying Ports
♦ 1.3 Verifying Layer 2
Connectivity
♦ 1.4 Verifying Layer 3
Connectivity
• 2 Overview of Symptoms
• 3 System Messages
♦ 3.1 System Message Text
♦ 3.2 syslog Server
Implementation
• 4 Troubleshooting with Logs
• 5 Troubleshooting Modules
• 6 Viewing NVRAM logs
• 7 Contacting Customer Support
• 8 See Also
• 9 Further Reading
• 10 External Links
Overview of the Troubleshooting Process

To troubleshoot your network, follow these general steps:
1. Maintain a consistent Cisco NX-OS release across all your devices.

2. See the Cisco NX-OS release notes for your Cisco NX-OS release for the latest features, limitations, and caveats.
3. Enable system message logging. See System Messages.
4. Troubleshoot any new configuration changes after implementing the change.
5. Gather information that defines the specific symptoms. See Gathering Information.
6. Verify the physical connectivity between your device and end devices. See Verifying Ports.
7. Verify the Layer 2 connectivity. See Verifying Layer 2 Connectivity.
8. Verify the end-to-end connectivity and the routing configuration. See Verifying Layer 3 Connectivity.
9. After you have determined that your troubleshooting attempts have not resolved the problem, contact Cisco TAC or your
technical support representative.
Note: View the Cisco Nexus 7000 instructional videos for an overview of Cisco NX-OS.
Gathering Information
This section describes the tools that are commonly used to troubleshoot problems within your network. Specific troubleshooting
articles may include additional tools and commands specific to the symptoms and possible problems covered in that article.
Note: You should have an accurate topology of your network to isolate problem areas. Contact your network architect for this
information.
Use the following commands to gather general information on your device:
Contents 3
• show module
• show version
• show running-config
• show logging log
• show interfaces brief
• show vlan
• show spanning-tree
• show {ip | ipv6} routing
• show processes | include ER
• show accounting log
Verifying Ports
Answer the following questions to verify that your ports are connected correctly and are operational:
• Are you using the correct media (copper, optical, fiber type)?
• Is the media broken or damaged?
• Is the port LED green on the module?
• Is the interface in the correct VDC?
Use the show vdc membership command to check which VDC that the interface is a member of. You must log
into the device with the network-admin role to use this command.
• Is the interface operational?
Use the show interface brief command. The status should be up.
See Troubleshooting Ports for more troubleshooting tips for ports.
Verifying Layer 2 Connectivity

Use the following commands to verify Layer 2 connectivity:
• Use the show vlan all-ports command to verify that all the necessary interfaces are in the same VLAN. The status should
be active for the VLAN.
• Use the show port-channel compatibility-parameters command to verify that all the ports in a port channel are
configured the same for the speed, the duplex, and the trunk mode.
• Use the show running-config spanning-tree command to verify that the Spanning Tree Protocol (STP) is configured the
same on all devices in the network.
• Use the show processes | include ER command to verify that no essential Layer 2 processes are in the error state.
• Use the show spanning-tree blockedports command to display the ports that are blocked by STP.
• Use the show mac address-table dynamic vlan command to determine if learning or aging is occurring at each node.
See Troubleshooting VLANs and Troubleshooting STP for more information on troubleshooting Layer 2 issues.
Verifying Layer 3 Connectivity

Answer the following questions to verify Layer 3 connectivity:
• Have you configured a default gateway?

• Have you configured the same dynamic routing protocol parameters throughout your routing domain or configured static
routes?
• Are any IP access lists, filters, or route maps blocking route updates?
Use the following commands to verify your routing configuration:
• show arp
• show ip routing
• show platform forwarding
See Ping and Traceroute to verify Layer 3 connectivity. See Troubleshooting Routing for more information on troubleshooting
Layer 3 issues.
Overview of Symptoms
This article uses a symptom-based troubleshooting approach that allows you to diagnose and resolve your Cisco NX-OS problems
by comparing the symptoms that you observed in your network with the symptoms listed in each chapter.
By comparing the symptoms in this publication to the symptoms that you observe in your own network, you should be able to
diagnose and correct software configuration issues and inoperable hardware components so that the problems are resolved with
minimal disruption to the network. Those problems and corrective actions include the following:
• Identify key Cisco NX-OS troubleshooting tools.

• Obtain and analyze protocol traces using SPAN and RSPAN or Ethanalyzer on the CLI.
• Identify or rule out physical port issues.
• Identify or rule out switch module issues.
• Diagnose and correct Layer 2 issues.
• Diagnose and correct Layer 3 issues.
• Recover from switch upgrade failures.
• Obtain core dumps and other diagnostic data for use by Cisco TAC or your customer support representative.
System Messages
The system software sends syslog (system) messages to the console (and, optionally, to a logging server on another device). Not
all messages indicate a problem with your device. Some messages are purely informational, while others might help diagnose
problems with links, internal hardware, or the device software.
System Message Text

Message text is a text string that describes the condition. This portion of the message might contain detailed information about the
event, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memory
address space. Because the information in these variable fields changes from message to message, it is represented here by short
strings enclosed in square brackets ([ ]). A decimal number, for example, is represented as [dec ].
Verifying Layer 3 Connectivity 5

PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not supported.
Use this string to find the matching system message in the NX-OS System Messages Reference or in the Error Message Decoder.
Each system message is followed by an explanation and recommended action. The action may be as simple as "No action is
required." It may involve a fix or a recommendation to contact technical support as shown in the following example:
Error Message PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not

supported.
Explanation Transceiver (SFP) is not from an authorized vendor.
Recommended Action Enter the show interface transceiver CLI command or similar DCNM command to
determine the transceiver being used. Please contact your customer support representative for a list of authorized
transceiver vendors.
syslog Server Implementation

The syslog facility allows the Cisco NX-OS device to send a copy of the message log to a host for more permanent storage. This
feature allows you to examine the logs over a long period of time or if the Cisco NX-OS device is not accessible.
This example shows how to configure a Cisco NX-OS device to use the syslog facility on a Solaris platform. Although a Solaris
host is being used, the syslog configuration on all UNIX and Linux systems is very similar.
syslog uses the facility to determine how to handle a message on the syslog server (the Solaris system in this example) and the
message severity. Different message severities are handled differently by the syslog server. They could be logged to different files
or e-mailed to a particular user. Specifying a severity level on the syslog server determines that all messages of that level and
greater severity (lower number) will be acted upon as you configure the syslog server.
Note: You should configure the syslog server so that the Cisco NX-OS messages are logged to a different file from the
standard syslog file so that they cannot be confused with other non-Cisco syslog messages. Do not locate the logfile on
the / file system. You do not want log messages to fill up the / file system. This example uses the following values:
◊ syslog client: switch1

◊ syslog server: 172.22.36.211
◊ (Solaris) syslog facility: local1
◊ syslog severity: notifications (level 5, the default)
◊ File to log Cisco NX-OS messages to: /var/adm/nxos_logs
To configure the syslog feature on Cisco NX-OS, follow these steps:
1. switch1# config terminal

2. switch1(config)# logging server 192.0.2.1 6 facility local1
Use the show logging server command to verify the syslog configuration.
switch1# show logging server
System Message Text 6

Logging server: enabled
{172.22.36.211}
server severity: notifications
server facility: local1
server VRF: management
To configure a syslog server, follow these steps:
1. Modify /etc/syslog.conf to handle local1 messages. For Solaris, you must allow at least one tab between the facility.severity and
the action (/var/adm/nxos_logs).
local1.notice /var/adm/nxos_logs
2. Create the log file.
touch /var/adm/nxos_logs
3. Restart the syslog process.
/etc/init.d/syslog stop
/etc/init.d/syslog start
syslog service starting.
4. Verify that the syslog process has started.
ps -ef |grep syslogd
root 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd
Test the syslog server by creating an event in Cisco NX-OS. In this case, port e1/2 was shut down and reenabled and the following
was listed on the syslog server. The IP address of the switch is listed in brackets.
tail -f /var/adm/MDS_logs
Sep 17 11:07:41 [172.22.36.142.2.2] : 2004 Sep 17 11:17:29 pacific: PORT-5-IF_DOWN_INITIALIZING: %$VLAN 1%$ Interf
Sep 17 11:07:49 [172.22.36.142.2.2] : 2004 Sep 17 11:17:36 pacific: %PORT-5-IF_UP: %$VLAN 1%$ Interface e 1/2 is u
Sep 17 11:07:51 [172.22.36.142.2.2] : 2004 Sep 17 11:17:39 pacific: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configuring cons
Troubleshooting with Logs

Cisco NX-OS generates many types of system messages on the device and sends them to a syslog server. You can view these
messages to determine what events may have led up to the current problem condition that you are facing.
Use the following commands to access and view logs in Cisco NX-OS:
switch# show logging ?
console Show console logging configuration

info Show logging configuration
internal syslog syslog internal information
ip IP configuration
syslog Server Implementation 7

last Show last few lines of logfile
level Show facility logging configuration
logfile Show contents of logfile
loopback Show logging loopback configuration
module Show module logging configuration
monitor Show monitor logging configuration
nvram Show NVRAM log
onboard show logging onboard
pending server address pending configuration
pending-diff server address pending configuration diff
server Show server logging configuration
session Show logging session status
status Show logging status
timestamp Show logging timestamp configuration
This example shows the output of the show logging server command:
switch# show logging server
Logging server: enabled

{172.28.254.254}
server severity: notifications
server facility: local7
server VRF: management
Troubleshooting Modules
You can directly connect to a module console port to troubleshoot module bootup issues. Use the attach console module
command to connect to the module console port.
Viewing NVRAM logs

System messages that are priority 0, 1, or 2 are logged into NVRAM on the supervisor module. After a switch reboots, you can
display these syslog messages in NVRAM by using the show logging nvram command.
switch# show logging nvram
2008 Sep 10 15:51:58 switch %$ VDC-1 %$ %SYSMGR-2-NON_VOLATILE_DB_FULL: System n

on-volatile storage usage is unexpectedly high at 99%.
2008 Sep 10 15:52:13 switch %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual sys
tem restart from Command Line Interface
2008 Sep 10 15:57:49 switch %$ VDC-1 %$ %KERN-2-SYSTEM_MSG: Starting kernel... -
kernel
2008 Sep 10 15:58:00 switch %$ VDC-1 %$ %CARDCLIENT-2-REG: Sent
2008 Sep 10 15:58:01 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: R2D2: P1 SUP NO GMTL
FOR P1 SUP - r2d2
2008 Sep 10 15:58:01 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: R2D2: P1 SUP NO GMTL
FOR P1 SUP - r2d2
2008 Sep 10 15:58:05 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: R2D2: P1 SUP: Reset
Tx/Rx during QOS INIT - r2d2
2008 Sep 10 15:58:16 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: can't dlsym ssnmgr_i
s_session_command: please link this binary with ssnmgr.so! - svi
2008 Sep 10 15:58:16 switch %$ VDC-1 %$ %CARDCLIENT-2-SSE: LC_READY sent
2008 Sep 10 15:58:17 switch %$ VDC-1 %$ snmpd: load_mib_module :Error, while loa
ding the mib module /isan/lib/libpmsnmp_common.so (/isan/lib/libpmsnmp_common.so
: undefined symbol: sme_mib_get_if_info)
2008 Sep 10 15:58:17 switch %$ VDC-1 %$ %CARDCLIENT-2-SSE: MOD:6 SUP ONLINE
Troubleshooting with Logs 8

2008 Sep 10 15:58:17 switch %$ VDC-1 %$ %VDC_MGR-2-VDC_LIC_WARN: Service using g
race period will be shutdown in 9 day(s)
Contacting Customer Support

If you are unable to solve a problem after using the troubleshooting suggestions in these articles, contact a customer service
representative for assistance and further instructions. Before you call, have the following information ready to help your service
provider assist you as quickly as possible:
• Date that you received the switch

• Chassis serial number (located on a label on the right side of the rear panel of the chassis)
• Type of software and release number
• Maintenance agreement or warranty information
• Brief description of the problem
• Brief explanation of the steps that you have already taken to isolate and resolve the problem
For more information on steps to take before calling Technical Support, see Before Contacting Technical Support.
See Also
Further Reading
The following links contain further information on this topic from Cisco.com:
Cisco NX-OS System Messages Reference
External Links
External links contain content developed by external authors. Cisco does not review this content for accuracy.
Nexus: Hands on with NX-OS, Part#1
This article describes how to identify and resolve problems that might occur when upgrading or restarting.
Guide Contents
Troubleshooting Installs, Upgrades, and Reboots (this section)
Troubleshooting CFS
Troubleshooting STP
Viewing NVRAM logs 9

Contents
• 1 Information About Upgrades and Reboots
• 2 Upgrades and Reboot Checklist
• 3 Verifying Software Upgrades
• 4 Verifying a Nondisruptive Upgrade
♦ 4.1 Using ROM Monitor Mode
• 5 Troubleshooting Software Upgrades and Downgrades
♦ 5.1 Software Upgrade Ends with Error
♦ 5.2 Upgrading Cisco NX-OS Software
• 6 Troubleshooting Software System Reboots
♦ 6.1 Power-On or Switch Reboot Hangs
♦ 6.2 Corrupted Bootflash Recovery
♦ 6.3 Recovery from the loader> Prompt on Supervisor
Modules
♦ 6.4 Recovery from the loader> Prompt
♦ 6.5 Recovery from the switch(boot)# Prompt
♦ 6.6 Recovery for Systems with Dual Supervisor
Modules
◊ 6.6.1 Recovering One Supervisor Module
With Corrupted Bootflash
◊ 6.6.2 Recovering Both Supervisor Modules
with Corrupted Bootflash
♦ 6.7 System or Process Resets
♦ 6.8 Recoverable System Restarts
♦ 6.9 Unrecoverable System Restarts
♦ 6.10 Standby Supervisor Fails to Boot
♦ 6.11 Recovering the Administrator Password
• 7 See Also
Information About Upgrades and Reboots

Cisco NX-OS consists of two images--the kickstart image and the system image. In order to bring up the system, both images
should have the same image version.
Upgrades and reboots are ongoing network maintenance activities. You should try to minimize the risk of disrupting the network
when performing these operations in production environments and to know how to recover quickly when something does go
wrong.
Note: This publication used the term upgrade to refer to both Cisco NX-OS upgrades and downgrades.
External Links 10
Upgrades and Reboot Checklist

Use the following checklist to prepare for an upgrade:
Check
Checklist
off
Read the Release Notes for the release that you are upgrading or downgrading to.
Ensure that an FTP or TFTP server is available to download the software images.
Copy the new image onto your supervisor modules in bootflash: or slot0:.
Use the show install all impact command to verify that the new image is healthy and the impact that the new load will
have on any hardware with regards to compatibility. Check for compatibility.
Copy the startup-config file to a snapshot configuration in NVRAM. This step creates a backup copy of the
startup-config file (see the Rollback chapter in the Cisco NX-OS System Management Configuration Guide).
Save your running configuration to the startup configuration.
Back up a copy of your configuration to a remote TFTP server.
Schedule your upgrade during an appropriate maintenance window for your network.
After you have completed the checklist, you are ready to upgrade the systems in your network.
Note: It is normal for the active supervisor to become the standby supervisor during an upgrade.
Note: Log messages are not saved across system reboots. However, a maximum of 100 log messages with a severity level of
critical and below (levels 0, 1, and 2) are saved in NVRAM. You can view this log at any time by entering the show
logging nvram command.
Verifying Software Upgrades
You can use the show install all status command to watch the progress of your software upgrade or to view the ongoing install
all command or the log of the last installed install all command from a console, SSH, or Telnet session. This command shows the
install all output on both the active and standby supervisor module even if you are not connected to the console terminal.
switch# show install all status

There is an on-going installation... <---------------------- in progress installation
Enter Ctrl-C to go back to the prompt.
Verifying image bootflash:/b-4.0.0.104
-- SUCCESS
Verifying image bootflash:/i-4.0.0.104
-- SUCCESS
Extracting system version from image bootflash:/i-4.0.0.104.
-- SUCCESS
Extracting kickstart version from image bootflash:/b-4.0.0.104.
-- SUCCESS
Extracting loaderâ version from image bootflash:/b-4.0.0.104.
-- SUCCESS
switch# show install all status

This is the log of last installation. <----------------- log of last install
Verifying image bootflash:/b-4.0.0.104
-- SUCCESS
Upgrades and Reboot Checklist 11

Verifying image bootflash:/i-4.0.0.104
-- SUCCESS
Extracting system version from image bootflash:/i-4.0.0.104.
-- SUCCESS
Extracting kickstart version from image bootflash:/b-4.0.0.104.
-- SUCCESS
Extracting loader version from image bootflash:/b-4.0.0.104.
-- SUCCESS
Verifying a Nondisruptive Upgrade

When you initiate a nondisruptive upgrade, Cisco NX-OS notifies all services that an upgrade is about to start and finds out
whether or not the upgrade can proceed. If a service cannot allow the upgrade to proceed at this time, then the service aborts the
upgrade and you are prompted to enter the show install all failure-reason command to determine the reason why the upgrade
cannot proceed.
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.
Notifying services about the upgrade.
>[# ] 0% -- FAIL. Return code 0x401E0066 (request timed out).
Please issue "show install all failure-reason" to find the cause of the failure.<---prompt failure-reason
Install has failed. Return code 0x401E0066 (request timed out).
Please identify the cause of the failure, and try 'install all' again.
switch# show install all failure-reason

Service: "cfs" failed to respond within the given time period.
switch#
If a failure occurs for whatever reason (such as a save runtime state failure or module upgrade failure) after the upgrade is in
progress, then the device reboots disruptively because the changes cannot be rolled back. In such cases, the upgrade has failed.
If you need further assistance to determine why an upgrade is unsuccessful, you should collect the details from the show
tech-support command output and the console output from the installation, if available, before you contact your technical support
representative.
Using ROM Monitor Mode

If your device does not find a valid system image to load, the system will start in ROM monitor mode. ROM monitor mode can
also be accessed by interrupting the boot sequence during startup. From ROM monitor mode, you can boot the device or perform
diagnostic tests.
On most systems, you can enter ROM monitor mode by entering the reload EXEC command and then pressing the Break key on
your keyboard or by using the Break key-combination (the default Break key combination is Ctrl-C) during the first 60 seconds of
startup.
Troubleshooting Software Upgrades and Downgrades

This section describes how to troubleshoot a software installation upgrade or downgrade failure.
Verifying Software Upgrades 12

Software Upgrade Ends with Error
Problem Possible Cause Solution

The standby supervisor module
bootflash: file system does not have Use the delete command to remove unnecessary files from the file
sufficient space to accept the updated system.
image.
Check the output of the installation process for details on the
The specified system and kickstart
incompatibility. Possibly update the kickstart image before updating the
images are not compatible.
system image.
The install all command is entered on
Enter the command on the active supervisor module only.
the standby supervisor module.
The upgrade A module was inserted while the
ends with an Restart the installation.
upgrade was in progress.
error.
The system experienced a power
disruption while the upgrade was in Restart the installation.
progress.
An incorrect software image path was
Specify the entire path for the remote location accurately.
specified.
Verify the state of the system at every stage and restart the upgrade after
Another upgrade is already in 10 seconds. If you restart the upgrade within 10 seconds, the command is
progress. rejected. An error message displays, indicating that an upgrade is
currently in progress.
Restart the upgrade or use the install module command to upgrade the
Module failed to upgrade.
failed module.
Upgrading Cisco NX-OS Software

To perform an automated software upgrade on any system from the CLI, follow these steps:
1. Log into the system through the console, Telnet, or SSH port of the active supervisor.
2. Create a backup of your existing configuration file, if required.
3. Perform the upgrade by entering the install all command.
4. Exit the system console and open a new terminal session to view the upgraded supervisor module by using the show
module command.
Tip: Always carefully read the output of the install all compatibility check command. This compatibility check tells you exactly
what needs to be upgraded (such as the BIOS, loader, or firmware) and what modules will experience a disruptive upgrade. If
there are any questions or concerns about the results of the output, type n to stop the installation and contact the next level of
support.
The following example shows an upgrade using the install all command with the source images located on an SCP server.
switch# install all system scp://testuser@tftp-server1/tftpboot/rel/qa/4.0/final/m95 00-sf1ek9-mz.4.0.bin kickstart

scp://testuser@tftp-server1/tftpboot/rel/qa/4.0/fin al/n7000-s1-kickstart-mz.4.0.bin
For scp://testuser@tftp-server1, please enter password:
Software Upgrade Ends with Error 13

For scp://testuser@tftp-server1, please enter password:
Copying image from scp://testuser@pal/tftpboot/rel/qa/4.0/final/n7000-s1

-kickstart-mz.4.0.bin to bootflash:///n7000-s1-kickstart-mz.4.0.bin.
[####################] 100% -- SUCCESS
Copying image from scp://testuser@pal/tftpboot/rel/qa/4.0/final/n7000-s1

-mz.4.0.bin to bootflash:///n7000-s1-mz.4.0.bin.
[####################] 100% -- SUCCESS
Verifying image bootflash:///n7000-s1-kickstart-mz.4.0.bin

[####################] 100% -- SUCCESS
Verifying image bootflash:///n7000-s1-mz.4.0.bin

[####################] 100% -- SUCCESS
Extracting "slc" version from image bootflash:///n7000-s1-mz.4.0.bin.

[####################] 100% -- SUCCESS
Extracting "ips" version from image bootflash:///n7000-s1-mz.4.0.bin.

[####################] 100% -- SUCCESS
Extracting "svclc" version from image bootflash:///n7000-s1-mz.4.0.bin.

[####################] 100% -- SUCCESS
Extracting "system" version from image bootflash:///n7000-s1-mz.4.0.bin.

[####################] 100% -- SUCCESS
Extracting "kickstart" version from image bootflash:///n7000-s1-kickstart-mz

.4.0.bin.
[####################] 100% -- SUCCESS
Extracting "loader" version from image bootflash:///n7000-s1-kickstart-mz.2.

1.1a.bin.
[####################] 100% -- SUCCESS
Compatibility check is done:

Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive rolling
2 yes non-disruptive rolling
3 yes disruptive rolling Hitless upgrade is not supported
4 yes disruptive rolling Hitless upgrade is not supported
5 yes non-disruptive reset
6 yes non-disruptive reset
Images will be upgraded according to following table:

Module Image Running-Version New-Version Upg-Required
------ ---------- -------------------- -------------------- ------------
1 slc 2.0(2b) 2.1(1a) yes
1 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no
2 slc 2.0(2b) 2.1(1a) yes
2 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no
3 ips 2.0(2b) 2.1(1a) yes
3 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no
4 svclc 2.0(2b) 2.1(1a) yes
4 svcsb 1.3(5m) 1.3(5m) no
4 svcsb 1.3(5m) 1.3(5m) no
4 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no
5 system 2.0(2b) 2.1(1a) yes
5 kickstart 2.0(2b) 2.1(1a) yes
5 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no
5 loader 1.2(2) 1.2(2) no
6 system 2.0(2b) 2.1(1a) yes
6 kickstart 2.0(2b) 2.1(1a) yes
6 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no
6 loader 1.2(2) 1.2(2) no
Do you want to continue with the installation (y/n)? [n] '''y'''

Install is in progress, please wait.
Syncing image bootflash:///n7000-s1-kickstart-mz.4.0.bin to standby.

[####################] 100% -- SUCCESS
Syncing image bootflash:///n7000-s1-mz.4.0.bin to standby.

[####################] 100% -- SUCCESS
Setting boot variables.

[####################] 100% -- SUCCESS
Performing configuration copy.

[####################] 100% -- SUCCESS
Module 5: Waiting for module online.

2005 May 20 15:46:03 ca-9506 %KERN-2-SYSTEM_MSG: mts: HA communication with standby terminated. Please check the
-- SUCCESS
"Switching over onto standby".
If the configuration meets all guidelines when the install all command is used, all modules (supervisor and switching) are
upgraded.
Troubleshooting Software System Reboots

This section describes how to troubleshoot software reboots.
Power-On or Switch Reboot Hangs

The bootflash is
Use the Recovery for Systems with Dual Supervisor Modules procedure.
corrupted.
The BIOS is Replace this module. Contact your customer support representative to return the
corrupted. failed module.
Power cycle the switch if required and enter CTRL-C when the switch says
The kickstart
A power-on or switch reboot "Loading Boot Loader" to interrupt the boot process at the >loader prompt. Use
image is
hangs for a dual supervisor the Recovery from the loader> Prompt on Supervisor Modules procedure to
corrupted.
configuration. update the kickstart image.
Boot parameters
Verify and correct the boot parameters and reboot.
are incorrect.
Power cycle the switch if required and enter CTRL-] when the switch says
The system
"Checking all filesystems....r. done." to interrupt the boot process at the
image is
switch#boot prompt. Use the Recovery from the switch(boot)# Prompt
corrupted.
procedure to update the system image..
Corrupted Bootflash Recovery
All device configurations reside in the internal bootflash. If you have a corrupted internal bootflash, you could potentially lose
your configuration. Be sure to save and back up your configuration files periodically. The regular system boot goes through the
following sequence (see Figure 1):
Troubleshooting Software System Reboots 15

1. The basic input/output system (BIOS) loads the loader.

2. The loader loads the kickstart image into RAM and starts the kickstart image.
3. The kickstart image loads and starts the system image.
4. The system image reads the startup-configuration file.
Figure 1 Regular Boot Sequence
If the images on your system are corrupted and you cannot proceed (error state), you can interrupt the system boot sequence and
recover the image by entering the BIOS configuration utility described in the following section. Access this utility only when
needed to recover a corrupted internal disk.
Caution: The BIOS changes explained in this section are required only to recover a corrupted bootflash.
Recovery procedures require the regular sequence to be interrupted. The internal sequence goes through four phases between the
time that you turn on the system and the time that the system prompt appears on your terminal--BIOS, boot loader, kickstart, and
system.
Recovery Interruption
Recovery
Normal
Prompt--appears when
Prompt--appears at
Phase the system cannot Description
the end of each
progress to the next
phase.
phase.
The BIOS begins the power-on self test, memory test, and other
operating system applications. While the test is in progress, press
BIOS loader> No bootable device
Ctrl-C to enter the BIOS configuration utility and use the netboot
option.
Corrupted Bootflash Recovery 16

The boot loader uncompresses the loaded software to boot an image

Boot using its filename as a reference. These images are made available
Starting kickstart loader>
loader through bootflash. When the memory test is over, press Esc to enter
the boot loader prompt.
When the boot loader phase is over, press Ctrl-] (Control key plus
right bracket key) to enter the switch(boot)# prompt. Depending on
Uncompressing your Telnet client, these keys may be reserved, and you may need to
Kickstart switch(boot)#
system remap the keystroke. See the documentation provided by your Telnet
client. If the corruption causes the console to stop at this prompt, copy
the system image and reboot the system.
The system image loads the configuration file of the last saved
System Login: --
running configuration and returns a switch login prompt.
Figure 2 Regular and Recovery Sequence
Recovery from the loader> Prompt on Supervisor Modules

Caution: This procedure uses the init system command, which reformats the file system of the device. Be sure that you have
made a backup of the configuration files before you begin this procedure.
The loader> prompt is different from the regular switch# prompt. The CLI command completion feature does not work at the
loader> prompt and may result in undesired errors. You must type the command exactly as you want the command to appear.
Note: If you boot over TFTP from the loader> prompt, you must supply the full path to the image on the remote server.
Note: The TFTP boot method is available only as a backup for diagnostics and for repairing bootflash corruption. The TFTP
boot method is not intended to bring up the system to a fully operational state. Reloading the system is mandatory after
all diagnostics and repairs have been completed.
Use the help command at the loader> prompt to display a list of commands available at this prompt or to obtain more information
about a specific command in that list.
To recover a corrupted kickstart image (system error state) for a system with a single supervisor module, follow these steps:
Recovery from the loader> Prompt on Supervisor Modules 17

1. Enter the local IP address and subnet mask for the system at the loader> prompt, and press Enter.
loader> set ip 172.16.1.2 255.255.255.0
2. Specify the IP address of the default gateway.
loader> set gw 172.16.1.1
3. Boot the kickstart image file from the required server.
loader> boot tftp://172.16.10.100/tftpboot/n7000-s1-kickstart-4.0.bin
In this example, 172.16.10.100 is the IP address of the TFTP server, and n7000-s1-kickstart-4.0.bin is the name of the kickstart
image file that exists on that server.
The switch(boot)# prompt indicates that you have a usable Kickstart image.
4. Enter the init system command at theswitch(boot)# prompt.
switch(boot)# init system
Caution: Be sure that you have made a backup of the configuration files before you enter this command.
5. Follow the procedure specified in the Recovery from the switch(boot)# Prompt procedure.
Recovery from the loader> Prompt

Caution: This procedure uses the init system command, which reformats the file system of the device. Be sure that you have
made a backup of the configuration files before you begin this procedure.
Note: The loader>prompt is different from the regular switch# or switch(boot)# prompt. The CLI command completion
feature does not work at the loader> prompt and may result in undesired errors. You must type the command exactly as
you want the command to appear.
Tip: Use the help command at the loader> prompt to display a list of commands available at this prompt or to obtain more
information about a specific command in that list.
To recover a corrupted kickstart image (system error state) for a system with a single supervisor module, follow these steps:
1. Specify the local IP address and the subnet mask for the system.
loader> set ip 172.21.55.213 255.255.255.224

set ip 172.21.55.213 255.255.255.224
Correct - ip addr is 172.21.55.213, mask is 255.255.255.224
Found Intel 82546GB [2:9.0] at 0xe040, ROM address 0xf980
Probing...[Intel 82546GB]
Management interface
Link UP in 1000/full mode
Ethernet addr: 00:1B:54:C1:28:60
Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 0.0.0.0
Recovery from the loader> Prompt 18

Gateway: 172.21.55.193
loader> set gw 172.21.55.193

Correct gateway addr 172.21.55.193
Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 0.0.0.0
Gateway: 172.21.55.193
loader> loader> '''boot tftp://172.28.255.18/tftpboot/n7000-s1-kickstart.4.0.3.gbin '''

Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 172.28.255.18
Gateway: 172.21.55.193
Filesystem type is tftp, using whole disk
Booting: /tftpboot/n7000-s1-kickstart.4.0.3.gbin console=ttyS0,9600n8nn quiet loader
_ver="3.17.0"....
.............................................................................Im
age verification OK
Starting kernel...
INIT: version 2.85 booting
Checking all filesystems..r.r.r.. done.
Setting kernel variables: sysctlnet.ipv4.ip_forward = 0
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_no_pmtu_disc = 1
.
Setting the System Clock using the Hardware Clock as reference...System Clock set. Local time: Wed Oct 1
11:20:11 PST 2008
WARNING: image sync is going to be disabled after a loader netboot
Loading system software
No system image Unexporting directories for NFS kernel daemon...done.
INIT: Sending processes the KILL signal
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch(boot)#
The switch(boot)# prompt indicates that you have a usable kickstart image.
4. Enter the init system command at the switch(boot)# prompt.
switch(boot)# init system
Caution: Be sure that you have made a backup of the configuration files before you enter this command.
5. Follow the procedure specified in the Recovery from the switch(boot)# Prompt.
Recovery from the loader> Prompt 19

Recovery from the switch(boot)# Prompt

To recover a system image using the kickstart image for a system with a single supervisor module, follow these steps:
1. Change to configuration mode and configure the IP address of the mgmt0 interface.
switch(boot)# config t
switch(boot)(config)# interface mgmt0
2. Follow this step if you entered an init system command. Otherwise, skip to Step 3.
a. Enter the ip address command to configure the local IP address and the subnet mask for the system.
switch(boot)(config-mgmt0)# ip address 172.16.1.2 255.255.255.0
b. Enter the ip default-gateway command to configure the IP address of the default gateway.
switch(boot)(config-mgmt0)# ip default-gateway 172.16.1.1
3. Enter the no shutdown command to enable the mgmt0 interface on the system.
switch(boot)(config-mgmt0)# no shutdown
4. Enter end to exit to EXEC mode.
switch(boot)(config-mgmt0)# end
5. If you believe there are file system problems, enter the init system check-filesystem command. This command checks all
internal file systems and fixes any errors that are encountered. This command takes a few minutes to complete.
switch(boot)# init system check-filesytem
6. Copy the system image from the required TFTP server.
switch(boot)# copy tftp://172.16.10.100/system-image1 bootflash:system-image1
7. Copy the kickstart image from the required TFTP server.
switch(boot)# copy tftp://172.16.10.100/kickstart-image1 bootflash:kickstart-image1
8. Verify that the system and kickstart image files are copied to your bootflash: file system.
switch(boot)#''' dir bootflash: '''

12456448 Jul 30 23:05:28 1980 kickstart-image1
12288 Jun 23 14:58:44 1980 lost+found/
27602159 Jul 30 23:05:16 1980 system-image1
Usage for bootflash://sup-local

135404544 bytes used
49155072 bytes free
184559616 bytes total
9. Load the system image from the bootflash: files system.
switch(boot)# '''load bootflash:system-image1'''
Recovery from the switch(boot)# Prompt 20

Uncompressing system image: bootflash:/system-image1
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Would you like to enter the initial configuration mode? (yes/no): yes
Note: If you enter no, you will return to the switch# login prompt, and you must manually configure the system.
Recovery for Systems with Dual Supervisor Modules
This section describes how to recover when one or both supervisor modules in a dual supervisor system have corrupted bootflash.
Recovering One Supervisor Module With Corrupted Bootflash
If one supervisor module has a functioning bootflash and the other has a corrupted bootflash, follow these steps:
1. Boot the functioning supervisor module and log on to the system.

2. At the switch# prompt on the booted supervisor module, enter the reload module slot force-dnld command, where slot is
the slot number of the supervisor module with the corrupted bootflash.
The supervisor module with the corrupted bootflash performs a netboot and checks the bootflash for corruption. When the bootup
scripts discover that the bootflash is corrupted, it generates an init system command, which fixes the corrupt bootflash. The
supervisor boots as the HA Standby.
Caution: If your system has an active supervisor module currently running, you must enter the system standby manual-boot
command in EXEC mode on the active supervisor module before entering the init system command on the standby
supervisor module to avoid corrupting the internal bootflash:. After the init system command completes on the
standby supervisor module, enter the system no standby manual-boot command in EXEC mode on the active
supervisor module.
Recovering Both Supervisor Modules with Corrupted Bootflash
If both supervisor modules have corrupted bootflash, follow these steps:
1. Boot the system and press the Esc key after the BIOS memory test to interrupt the boot loader.
Note: Press Esc immediately after you see the following message: 00000589K Low Memory Passed00000000K Ext
Memory PassedHit ^C if you want to run SETUP....Wait.....If you wait too long, you will skip
the boot loader phase and enter the kickstart phase.
You see the loader> prompt.
Caution: The loader> prompt is different from the regular switch# or switch(boot)# prompt. The CLI command completion
feature does not work at the loader> prompt and may result in undesired errors. You must type the command exactly
as you want the command to appear.
Tip: Use the help command at the loader> prompt to display a list of commands available at this prompt or to obtain more
information about a specific command in that list.
2. Specify the local IP address and the subnet mask for the system.
loader> set ip 172.21.55.213 255.255.255.224

set ip 172.21.55.213 255.255.255.224
Correct - ip addr is 172.21.55.213, mask is 255.255.255.224
Found Intel 82546GB [2:9.0] at 0xe040, ROM address 0xf980
Probing...[Intel 82546GB]
Management interface
Recovery for Systems with Dual Supervisor Modules 21

Link UP in 1000/full mode
Ethernet addr: 00:1B:54:C1:28:60
Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 0.0.0.0
Gateway: 172.21.55.193
loader> set gw 172.21.55.193

Correct gateway addr 172.21.55.193
Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 0.0.0.0
Gateway: 172.21.55.193
loader> loader> '''boot tftp://172.28.255.18/tftpboot/n7000-s1-kickstart.4.0.3.gbin '''

Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 172.28.255.18
Gateway: 172.21.55.193
Filesystem type is tftp, using whole disk
Booting: /tftpboot/n7000-s1-kickstart.4.0.3.gbin console=ttyS0,9600n8nn quiet loader
_ver="3.17.0"....
.............................................................................Im
age verification OK
Starting kernel...
INIT: version 2.85 booting
Checking all filesystems..r.r.r.. done.
Setting kernel variables: sysctlnet.ipv4.ip_forward = 0
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_no_pmtu_disc = 1
.
Setting the System Clock using the Hardware Clock as reference...System Clock set. Local time: Wed Oct 1
11:20:11 PST 2008
WARNING: image sync is going to be disabled after a loader netboot
Loading system software
No system image Unexporting directories for NFS kernel daemon...done.
INIT: Sending processes the KILL signal
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch(boot)#
The switch(boot)# prompt indicates that you have a usable kickstart image.
5. Enter the init-system command to repartition and format the bootflash.
6. Perform the steps in the Recovery from the switch(boot)# Prompt procedure.
Recovering Both Supervisor Modules with Corrupted Bootflash 22

7. Perform the steps in the Recovering One Supervisor Module With Corrupted Bootflash procedure to recover the other
supervisor module.
Note: If you do not enter the reload module command when a boot failure has occurred, the active supervisor module
automatically reloads the standby supervisor module within 3 to 6 minutes after the failure.
System or Process Resets
When a recoverable or nonrecoverable error occurs, the system or a process on the system may reset. See Table 2-4 for possible
causes and solutions.

The system has automatically recovered from the problem. Use
A recoverable error occurred on the
the Recoverable System Restarts procedure and the System or
system or on a process in the system.
Process Resets procedure.
The system or a
process on the system A nonrecoverable error occurred on The system cannot recover automatically from the problem. Use
resets. the Recoverable System Restarts procedure to determine the
the system.
cause.
Verify that a clock module failed. Replace the failed clock module
A clock module failed.
during the next maintenance window.
Recoverable System Restarts
Every process restart generates a syslog message and a Call Home event. Even if the event does not affect service, you should
identify and resolve the condition immediately because future occurrences could cause a service interruption.
To respond to a recoverable system restart, follow these steps:
1. Check the syslog file to see which process restarted and why it restarted.
switch# show log logfile | include error
For information about the meaning of each message, see the Cisco NX-OS System Messages Reference. The system output looks
like the following example:
Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has finished with error
code SYSMGR_EXITCODE_SY.
switch# show logging logfile | include fail
Jan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, ad
dr 0.0.0.0, in_classd=0 flags=1 fails: Address already in use
Jan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/13 is down (Link failure
or not-connected)
Jan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/14 is down (Link failure
or not-connected)
Jan 28 00:55:12 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure o
r not-connected)
Jan 28 00:58:06 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating p
ort fc1/1 (VSAN 100)
System or Process Resets 23

Jan 29 19:01:34 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure o
r not-connected)
switch#
2. Identify the processes that are running and the status of each process.
switch# show processes
The following codes are used in the system output for the state (process state):
• D = uninterruptible sleep (usually I/O)

• R = runnable (on run queue)
• S = sleeping
• T = traced or stopped
• Z = defunct (zombie) process
• NR = notrunning
• ER = should be running but currently notrunning
Note: ER usually is the state that a process enters if it has been restarted too many times and has been detected as faulty by the
system and disabled.
The system output looks like the following example. (This output has been abbreviated to be more concise.)
PID State PC Start_cnt TTY Process

----- ----- -------- ----------- ---- -------------
1 S 2ab8e33e 1 - init
2 S 0 1 - keventd
3 S 0 1 - ksoftirqd_CPU0
4 S 0 1 - kswapd
5 S 0 1 - bdflush
6 S 0 1 - kupdated
71 S 0 1 - kjournald
431 S 2abe333e 1 - httpd
443 S 2abfd33e 1 - xinetd
446 S 2ac1e33e 1 - sysmgr
452 S 2abe91a2 1 - httpd
456 S 2ac73419 1 S0 vsh
3. Show the processes that have had abnormal exits and to if there is a stack-trace or core dump.
switch# show process log

Process PID Normal-exit Stack-trace Core Log-create-time
---------------- ------ ----------- ----------- ------- ---------------
ntp 919 N N N Jan 27 04:08
snsm 972 N Y N Jan 24 20:50
4. Show detailed information about a specific process that has restarted.
switch# show processes log pid 898

Service: idehsd
Description: ide hotswap handler Daemon
Started at Mon Sep 16 14:56:04 2002 (390923 us)
Stopped at Thu Sep 19 14:18:42 2002 (639239 us)
Uptime: 2 days 23 hours 22 minutes 22 seconds
Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGTERM (3)
Exit code: signal 15 (no core)
CWD: /var/sysmgr/work
Virtual Memory:
CODE 08048000 - 0804D660
DATA 0804E660 - 0804E824
BRK 0804E9A0 - 08050000
STACK 7FFFFD10
Register Set:
EBX 00000003 ECX 0804E994 EDX 00000008
ESI 00000005 EDI 7FFFFC9C EBP 7FFFFCAC
EAX 00000008 XDS 0000002B XES 0000002B
EAX 00000003 (orig) EIP 2ABF5EF4 XCS 00000023
EFL 00000246 ESP 7FFFFC5C XSS 0000002B
Stack: 128 bytes. ESP 7FFFFC5C, TOP 7FFFFD10
0x7FFFFC5C: 0804F990 0804C416 00000003 0804E994 ................
0x7FFFFC6C: 00000008 0804BF95 2AC451E0 2AAC24A4 .........Q.*.$.*
0x7FFFFC7C: 7FFFFD14 2AC2C581 0804E6BC 7FFFFCA8 .......*........
0x7FFFFC8C: 7FFFFC94 00000003 00000001 00000003 ................
0x7FFFFC9C: 00000001 00000000 00000068 00000000 ........h.......
0x7FFFFCAC: 7FFFFCE8 2AB4F819 00000001 7FFFFD14 .......*........
0x7FFFFCBC: 7FFFFD1C 0804C470 00000000 7FFFFCE8 ....p...........
0x7FFFFCCC: 2AB4F7E9 2AAC1F00 00000001 08048A2C ...*...*....,...
PID: 898
SAP: 0
UUID: 0
switch#
5. Determine if the restart recently occurred.
switch# show system uptime

Start Time: Fri Sep 13 12:38:39 2002
Up Time: 0 days, 1 hours, 16 minutes, 22 seconds
To determine if the restart is repetitive or a one-time occurrence, compare the length of time that the system has been up with the
time stamp of each restart.
6. View the core files.
switch# show cores

Module-num Process-name PID Core-create-time
---------- ------------ --- ----------------
5 fspf 1524 Jan 9 03:11
6 fcc 919 Jan 9 03:09
8 acltcam 285 Jan 9 03:09
8 fib 283 Jan 9 03:08
The output shows all cores that are presently available for upload from the active supervisor. The module-num column shows the
slot number on which the core was generated. In the previous example, an FSPF core was generated on the active supervisor
module in slot 5. An FCC core was generated on the standby supervisory module in slot 6. Core dumps generated on the module
in slot 8 include ACLTCAM and FIB.
Copy the FSPF core dump to a TFTP server with the IP address 1.1.1.1, as follows:
Recoverable System Restarts 25

switch# copy core://5/1524 tftp::/1.1.1.1/abcd
Display the file named zone_server_log.889 in the log directory as follows:
switch# '''show pro log pid 1473'''

======================================================
Service: ips
Description: IPS Manager
Started at Tue Jan 8 17:07:42 1980 (757583 us)

Stopped at Thu Jan 10 06:16:45 1980 (83451 us)
Uptime: 1 days 13 hours 9 minutes 9 seconds
Start type: SRV_OPTION_RESTART_STATELESS (23)

Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Exit code: signal 6 (core dumped)
CWD: /var/sysmgr/work
Virtual Memory:
CODE 08048000 - 080FB060

DATA 080FC060 - 080FCBA8
BRK 081795C0 - 081EC000
STACK 7FFFFCF0
TOTAL 20952 KB
Register Set:
EBX 000005C1 ECX 00000006 EDX 2AD721E0

ESI 2AD701A8 EDI 08109308 EBP 7FFFF2EC
EAX 00000000 XDS 0000002B XES 0000002B
EAX 00000025 (orig) EIP 2AC8CC71 XCS 00000023
EFL 00000207 ESP 7FFFF2C0 XSS 0000002B
Stack: 2608 bytes. ESP 7FFFF2C0, TOP 7FFFFCF0
0x7FFFF2C0: 2AC8C944 000005C1 00000006 2AC735E2 D..*.........5.*

0x7FFFF2D0: 2AC8C92C 2AD721E0 2AAB76F0 00000000 ,..*.!.*.v.*....
0x7FFFF2E0: 7FFFF320 2AC8C920 2AC513F8 7FFFF42C ... ..*...*,...
0x7FFFF2F0: 2AC8E0BB 00000006 7FFFF320 00000000 ...*.... .......
0x7FFFF300: 2AC8DFF8 2AD721E0 08109308 2AC65AFC ...*.!.*.....Z.*
0x7FFFF310: 00000393 2AC6A49C 2AC621CC 2AC513F8 .......*.!.*...*
0x7FFFF320: 00000020 00000000 00000000 00000000 ...............
0x7FFFF330: 00000000 00000000 00000000 00000000 ................
0x7FFFF340: 00000000 00000000 00000000 00000000 ................
0x7FFFF350: 00000000 00000000 00000000 00000000 ................
0x7FFFF360: 00000000 00000000 00000000 00000000 ................
0x7FFFF370: 00000000 00000000 00000000 00000000 ................
0x7FFFF380: 00000000 00000000 00000000 00000000 ................
0x7FFFF390: 00000000 00000000 00000000 00000000 ................
0x7FFFF3A0: 00000002 7FFFF3F4 2AAB752D 2AC5154C .
... output abbreviated ...
Stack: 128 bytes. ESP 7FFFF830, TOP 7FFFFCD0
7. Enter the system cores {[tftp:[//servername]/[path/] | scp:[//[username@]server][/path] | sftp:[//[username@]server][/path]] vrf
Recoverable System Restarts 26

{vrf-name | default | management} | slot0:[path]} command to configure the system to use TFTP to send the core dump to a TFTP
server.
This command causes the system to enable the automatic copy of core files to a TFTP server. For example, the following
command sends the core files to the TFTP server with the IP address 10.1.1.1:
switch(config)# system cores tftp://10.1.1.1/cores
The following conditions apply:
• The core files are copied every 4 minutes. This time interval is not configurable.
• The copy of a specific core file to a TFTP server can be manually triggered, by using the command copy
core://module#/pid# tftp://tftp_ip_address/file_name.
• The maximum number of times that a process can be restarted is part of the high-availability (HA) policy for any process.
(This parameter is not configurable.) If the process restarts more than the maximum number of times, the older core files
are overwritten.
• The maximum number of core files that can be saved for any process is part of the HA policy for any process. (This
parameter is not configurable, and it is set to three.)
Note:
When using the system cores command with the scp or sftp option, you also need to generate a passwordless SSH using the
following:
username admin keypair generate rsa force username admin keypair export bootflash:key_rsa rsa force copy
bootflash:key_rsa.pub scp://<userid>@<server>/<dir>/ vrf management
On the specified <server>, enter the following:
cat key_rsa.pub >> $HOME/.ssh/authorized_keys
8. Determine the cause and resolution for the restart condition by contacting your technical support representative and asking the
representative to review your core dump.
See the Cisco NX-OS High Availability and Redundancy Guide for more information on high-availability policies.
Unrecoverable System Restarts

An unrecoverable system restart might occur in the following cases:
• A critical process fails and is not restartable.

• A process restarts more times than is allowed by the system configuration.
• A process restarts more frequently than is allowed by the system configuration.
The effect of a process reset is determined by the policy configured for each process. An unrecoverable reset may cause
Unrecoverable System Restarts 27

functionality loss, the active supervisor to restart, a supervisor switchover, or the system to restart.
To respond to an unrecoverable reset, see the Troubleshooting Cisco NX-OS Software System Reboots procedure.
The show system reset-reason command displays the following information:
• The last four reset-reason codes for the supervisor modules are displayed. If either supervisor module is absent, the
reset-reason codes for that supervisor module are not displayed.
• The show system reset-reason module number command displays the last four reset-reason codes for a specific module
in a given slot. If a module is absent, then the reset-reason codes for that module are not displayed.
• The overall history of when and why expected and unexpected reloads occur
• The time stamp of when the reset or reload occurred
• The reason for the reset or reload of a module
• The service that caused the reset or reload (not always available)
• The software version that was running at the time of the reset or reload
switch# show system reset-reason module 6

----- reset reason for Supervisor-module 6 (from Supervisor in slot 6) ---
1) At 281000 usecs after Wed Jun 25 20:16:34 2008
Reason: Reset Requested by CLI command reload
Service:
Version: 4.0(2.45)
Service:
Version: 4.0(2.45)
Service:
Version: 4.0(2)
Service:
Version: 4.0(2)
Standby Supervisor Fails to Boot

The standby supervisor does not boot after an upgrade. You may see the following system message:
Error Message SYSMGR-2-STANDBY_BOOT_FAILED: Standby supervisor failed to boot up.
Explanation This message is printed if the standby supervisor doesn't complete its boot procedure (i.e. it doesn't reach the
login prompt on the local console) 3 to 6 minutes after the loader has been loaded by the BIOS. This message is usually
caused by boot variables not properly set for the standby supervisor. This message can also be caused by a user
intentionally interrupting the boot procedure at the loader prompt (by means of pressing ESC).
Recommended Action Connect to the local console of the standby supervisor. If the supervisor is at the loader prompt,
try to use the boot command to continue the boot procedure. Otherwise, issue a reload command for the standby
supervisor from a vsh session on the active supervisor, specifying the force-dnld option. Once the standby is online, fix
the problem by setting the boot variables appropriately.
Symptom Possible Cause Solution

Standby supervisor does not Active supervisor kickstart image booted from Reload the active supervisor from
boot. TFTP. bootflash:.
Recovering the Administrator Password
You can access the system if you forget the administrator password.
Problem Solution
You forgot the administrator password for Use the Password Recovery procedure to recover the password using a local
accessing. console connection.
See Also
Cisco NX-OS/IOS Configuration Fundamentals Comparison
Further Reading
Cisco Nexus 7000 Series Upgrade/Downgrade Guides
Cisco Nexus 7000 Series Release Notes
External Links
Nexus 7000 NX-OS Upgrade (walkthru example)
NX-OS Intro (part 1) (video)
This article describes how to troubleshoot licensing on a Cisco NX-OS device.
Guide Contents
Troubleshooting Licensing (this section)
Troubleshooting CFS
Troubleshooting STP
Recovering the Administrator Password 29

Contents
• 1 Information About Troubleshooting Licensing Issues
♦ 1.1 Chassis Serial Numbers
♦ 1.2 Swapping out a Chassis
♦ 1.3 Grace Period
• 2 Licensing Guidelines
• 3 Initial Troubleshooting Checklist
• 4 Displaying License Information Using the CLI
♦ 4.1 Example: Displays Information About Current
License Usage
♦ 4.2 Example: Displays the List of Features in a
Specified Package
♦ 4.3 Example: Displays the Host ID for the License
♦ 4.4 Example: Displays All Installed License Key
Files and Contents
• 5 Licensing Installation Issues
♦ 5.1 Serial Number Issues
♦ 5.2 RMA Chassis Errors or License Transfers
Between Systems
♦ 5.3 Receiving Grace Period Warnings After License
Installation
♦ 5.4 Grace Period Alerts
♦ 5.5 License Listed as Missing
• 6 See Also
Information About Troubleshooting Licensing Issues

Cisco NX-OS requires licenses select features. The icenses enable those features on your system. You must purchase a license for
each system that you want to enable the licensed features on.
Note: You can enable a feature without installing the license. Cisco NX-OS provides a grace period that allows you to try out
the feature before purchasing the license.
Chassis Serial Numbers

Licenses are created using the serial number of the chassis where the license file is to be installed. Once you order a license based
on a chassis serial number, you cannot use this license on any other system.
External Links 30
Swapping out a Chassis

If you swap out a chassis which included licenses, you must contact TAC to generate a new license. The old license was based on
the chassis serial number and will not work with the new chassis.
Grace Period
If you use a feature that requires a license but you have not installed a license for that feature, you are given a 120-grace period to
evaluate the feature. You must purchase and install the number of licenses required for that feature before the grace period ends or
Cisco NX-OS will disable the feature at the end of the grace period.
License packages can contain several features. If you disable a feature during the grace period and there are other features in that
license package that are still enabled, the clock does not stop for that license package. To suspend the grace period countdown for
a licensed feature, you must disable every feature in that license package. Use the show license usage command to determine
which features are enabled for a license package.
Licensing Guidelines
Follow these guidelines when dealing with licenses for Cisco NX-OS:
• Do not ignore the grace period expiration warnings. Allow 60 days before the grace period expires to allow time for
ordering, shipping, and installation for a new license purchase.
• Carefully determine the license(s) that you require based on the features that require a license.
• Order your license accurately, as follows:
♦ Enter the Product Authorization Key that appears in the Proof of Purchase document that comes with your
system.
♦ Enter the correct chassis serial number when ordering the license. The serial number must be for the same chassis
that you plan to install the license on. Use the show license host-id command to obtain your chassis serial
number.
♦ Enter serial numbers accurately. Do not use the letter "O" instead of a zero in the serial number.
♦ Order the license that is specific to your chassis.
• Back up the license file to a remote, secure place. Archiving your license files ensures that you will not lose the licenses in
the case of a failure on your system.
• Install the correct licenses on each system, using the licenses that were ordered using that system's serial number. Licenses
are serial-number specific and platform specific.
• Use the show license usage command to verify the license installation.
• Never modify a license file or attempt to use it on a system that it was not ordered for. If you return a chassis, contact your
customer support representative to order a replacement license for the new chassis.
Initial Troubleshooting Checklist

Begin troubleshooting license issues by checking the following issues first:
Check
Checklist
off
Swapping out a Chassis 31

Verify the chassis serial number for all licenses ordered.

Verify the platform or module type for all licenses ordered.
Verify that the Product Authorization Key that you used to order the licenses comes from the same chassis that you
retrieved the chassis serial number on.
Verify that you have installed all licenses on all systems that require the licenses for the features you enable.
Displaying License Information Using the CLI

Use the show license commands to display all license information configured on this system.
Example: Displays Information About Current License Usage
switch(config)# show license usage

Feature Ins Lic Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
LAN_ADVANCED_SERVICES_PKG No - In use Grace 102D 0H
LAN_ENTERPRISE_SERVICES_PKG No - In use Grace 103D 22H
------------------------------------------------------------------------------------------
Example: Displays the List of Features in a Specified Package
switch(config)# show license usage LAN_ENTERPRISE_SERVICES_PKG

Application
-----------
pbr
Tunnel
-----------
Example: Displays the Host ID for the License
switch# show license host-id

License hostid: VDH=FOX0646S017
Note: Use the entire ID that appears after the colon (:) . The VHD is the Vendor Host ID.
Example: Displays All Installed License Key Files and Contents
switch# show license

entp.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT LAN_ENTERPRISE_SERVICES_PKG cisco 1.0 permanent uncounted \
VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>N7K-LAN1K9=</SKU> \
HOSTID=VDH=TBC10412106 \ >
NOTICE="<LicFileID>20071025133322456</LicFileID>LicLineID>1/LicLineID>
\
Initial Troubleshooting Checklist 32

Licensing Installation Issues

Common problems with licenses usually occur from incorrectly ordering the license file, installing the license file on an incorrect
system, or not ordering the correct number of licenses for your fabric.
Serial Number Issues

Make sure that you use the correct chassis serial number when ordering your license. Use the show license host-id command to
obtain the correct chassis serial number for your system using the CLI.
If you use a license meant for another chassis, you may see the following system message:
Error Message: LICMGR-3-LOG_LIC_INVALID_HOSTID: Invalid license hostid VDH=[chars] for feature [chars].
Explanation: The feature has a license with an invalid license Host ID. This can happen if a supervisor module with licensed
features for one system is installed on another system.
Recommended Action: Reinstall the correct license for the chassis where the supervisor module is installed.
When entering the chassis serial number during the license ordering process, do not use the letter "O" instead of any zeros in the
serial number.
RMA Chassis Errors or License Transfers Between Systems

A license is specific to the system for which it is issued and is not valid on any other system. If you need to transfer a license from
one system to another, contact your technical support representative.
Receiving Grace Period Warnings After License Installation

If the license installation does not proceed correctly, or if you are using a feature that exists in a license package that you have not
installed, you will continue to get grace period warnings.

The license file is copied to
the system but is not Use the license install command to install the license.
You are receiving grace period
installed.
warnings after a license
installation. Check your logs for any system messages for a failed license
The license installation
installation. Use the show license usage command to determine
failed.
which feature is in use without a license.
Licensing Installation Issues 33

Grace Period Alerts

Cisco NX-OS gives you a 120-day grace period. This grace period starts or continues when you are evaluating a feature for which
you have not installed a license.
The grace period stops if you disable a feature that you are evaluating. If you enable that feature again without a valid license, the
grace period countdown continues where it left off.
The grace period operates across all features in a license package. License packages can contain several features. If you disable a
feature during the grace period and there are other features in that license package that are still enabled, the countdown does not
stop for that license package. To suspend the grace period countdown for a license package, you must disable every feature in that
license package.
The Cisco NX-OS license counter keeps track of all licenses on a system. If you are evaluating a feature and the grace period has
started, you will receive console messages, SNMP traps, system messages, and daily Call Home messages.
Beyond that, the frequency of these messages become hourly during the last seven days of the grace period. The following
example uses the VDC feature. On January 30th, you enabled the VDC feature, using the 120-day grace period. You will receive
grace period ending messages as follows:
• Daily alerts from January 30th to May 21st

• Hourly alerts from May 22nd to May 30th
On May 31st, the grace period ends, and the VDC feature is automatically disabled. You will not be allowed to use multiple VDCs
until you purchase a valid license.
Note: You cannot modify the frequency of the grace period messages.
Caution: After the final seven days of the grace period, the feature is turned off and your network traffic may be disrupted.
Any future upgrade will enforce license requirements and the 120-day grace period.
If you try to use an unlicensed feature, you may see one of the following system messages:
Error Message: LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature [chars].
Explanation: The unlicensed feature has exceeded its grace time period. Applications using this license will be shut down
immediately.
Recommended Action: Install the license file to continue using the feature.
Error Message: LICMGR-3-LOG_LICAPP_NO_LIC: Application [chars] running without [chars] license, shutdown in [dec]
days.
Grace Period Alerts 34

Explanation: The Application [chars1] has not been licensed. The application will work for a grace period of [dec] days after
which it will be shut down unless a license file for the feature is installed.
Recommended Action: Install the license to continue using the feature.
Error Message: LICMGR-3-LOG_LIC_LICENSE_EXPIRED: Evaluation license expired for feature [chars].
Explanation: The feature has exceeded its evaluation time period. The feature will be shut down after a grace period.
Error Message: LICMGR-3-LOG_LIC_NO_LIC: No license(s) present for feature [chars]. Application(s) shutdown in [dec]
days.
Explanation: The feature has not been licensed. The feature will work for a grace period, after which the application(s) using the
feature will be shut down.
Error Message: LICMGR-6-LOG_LICAPP_EXPIRY_WARNING: Application [chars] evaluation license [chars] expiry in [dec]
days.
Explanation: The application will exceed its evaluation time period in the listed number of days and will be shut down unless a
permanent license for the feature is installed.
Recommended Action: Install the license file to continue using the feature.
Use the show license usage command to display grace period information for a system.
switch(config)# show license usage

Feature Ins Lic Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
LAN_ADVANCED_SERVICES_PKG No - In use Grace 102D 0H
LAN_ENTERPRISE_SERVICES_PKG No - In use Grace 103D 22H
------------------------------------------------------------------------------------------
License Listed as Missing

After a license is installed and operating properly, it may show up as missing if you modify your system hardware or encounter a
bootflash: issue.
Symptom Possible Causes Solutions

The supervisor module was replaced
A license is listed as Use the Corrupted Bootflash Recovery procedure to recover from
after the license was installed.
missing. the corrupted bootflash:. Reinstall the license.
The supervisor bootflash: is corrupted
License Listed as Missing 35

See Also
Further Reading
Cisco Nexus 7000 Series Licensing Information
External Links
This article describes how to troubleshoot virtual device contexts (VDCs).
Guide Contents
Troubleshooting VDCs (this section)
Troubleshooting CFS
Troubleshooting STP
Contents
• 1 Information About Troubleshooting VDCs
• 3 VDC Issues
♦ 3.1 You Cannot Create a VDC
♦ 3.2 You Cannot Log into a Device
♦ 3.3 You Cannot Switch to a VDC
♦ 3.4 You Cannot Delete a VDC
♦ 3.5 You Cannot Allocate an Interface to a VDC
See Also 36
◊ 3.5.1 Table: Port Numbers for Cisco Nexus 7000 Series 32-port
10-Gbps Ethernet module
♦ 3.6 The VDC Does Not Reflect a Resource Template Change
♦ 3.7 The VDC Remains in a Failed State
♦ 3.8 You Cannot Copy the Running-Config File to the Startup-Config File in
a VDC
• 4 See Also
Information About Troubleshooting VDCs

Cisco NX-OS supports VDCs, which you can use to divide the physical NX-OS device into separate virtual devices. Each VDC
appears as a unique device to the connected users. A VDC runs as a separate logical entity within the physical NX-OS device,
maintains its own unique set of running software processes, has its own configuration, and can be managed by a separate
administrator.
VDC issues may not be directly related to VDC management. See the troubleshooting chapter that reflects your symptoms to find
other issues related to VDCs. For instance, if you configure a VDC template that limits the number of port channels in that VDC,
you may experience problems if you try to create more port channels than the VDC template allows.
VDC templates set limits on the following features:
• Port channels
• SPAN sessions
• IPv4 route map memory
• VLANs
• Virtual routing and forwarding instances (VRFs)
The minimum resource value configures the guaranteed limit for that feature. The maximum resource value represents
oversubscription for the feature and is available on a first-come,first-served basis.
Note: When you allocate an interface to a VDC, Cisco NX-OS removes all configuration for that interface.
See the Cisco NX-OS Virtual Device Context Configuration Guide for more information on VDCs or for details on any VDC
configuration changes recommended in this article.

Begin troubleshooting VDC issues by checking the following issues first:
Checklist Check off
Contents 37
Verify that you are logged into the device as network-admin if you are creating or modifying VDCs.
Verify that you are in the correct VDC. You must be in the default VDC to configure VDCs.
Verify that you have installed the Advanced Services license to configure VDCs.
Verify that you are not attempting to create more than three VDCs.
Use the following commands to display VDC information:
• show vdc membership - Displays information about which interfaces are assigned to a VDC.
• show vdc resource - Displays information about the resources assigned (Command is available only in the default VDC).
• show vdc current-vdc - Displays the VDC you are currently in.
VDC Issues
Problems with VDCs usually occur from logging into the incorrect VDC or misallocating resources for a VDC.
You Cannot Create a VDC

When you have a problem with creating a VDC, you may see one of the following system messages:
Error Message: VDC_MGR-2-VDC_BAD: vdc_mgr: There has been a failure at res_mgr
Explanation: You cannot create a VDC because not enough resources are available based on the template configuration. If no
template is used, a default template is applied.
Recommended Action: Verify that you have sufficient resources available to create this VDC by using the show vdc resources
[detail] or show vdc resource template command. Modify the template that you are using to create the VDC or create a new
template with resource limits that are currently available.
Error Message: VDC_MGR-2-VDC_BAD: vdc_mgr: : There has been a failure at sys_mgr
Explanation: Some services crashed or failed to come up because of insufficient system resources other than what can be
reserved using the resource templates. These dynamic resources are based on system utilization and may not be available to
support a new VDC.
Recommended Action: Use the show system internal sysmgr service running command to determine what caused the failure.

You are not logged in as
Log into the device with an account that has network-admin privileges.
network-admin.
You cannot You are not logged into
Use the switchto command to switch to the default VDC to allocate resources.
create a VDC. the default VDC.

Use the show vdc resources [detail] or show vdc resource template command to
There are not enough
determine your available resources. Modify your template or create a VDC with fewer
resources.
resources by using the limit-resource command in VDC configuration mode.
You Cannot Log into a Device

You may have a problem when logging into a device.

There is no account Log into the device as network-admin and use the switchto command to switch
You cannot log information for the VDC. to the VDC and configure the password and network connectivity for this VDC.
into a device. You are using an incorrect
Log into the device with the account created for that VDC.
VDC username.
You Cannot Switch to a VDC

You may have a problem when you switch to another VDC.

You cannot switch to a You are not logged in as network-admin or Log into the device with an account that has the
VDC. network-operator. correct privileges.
You Cannot Delete a VDC

When you have a problem with deleting a VDC, you may see one of the following system messages:
Error Message: VDC_MGR-2-VDC_UNGRACEFUL: vdc_mgr: Ungraceful cleanup request received for vdc [dec], restart
count for this vdc is [dec]
Explanation: Vdc_mgr has begun an ungraceful cleanup for a VDC.
Recommended Action: No action is required.
Error Message: VDC_MGR-2-VDC_OFFLINE: vdc [dec] is now offline
Explanation: Vdc_mgr has finished deleting a VDC.
Recommended Action: No action is required.
You Cannot Create a VDC 39

You attempted to delete the default VDC. You cannot delete the default VDC.
You cannot delete a
VDC. Unknown errors occurred when deleting Use the show tech-support VDC command to gather more
a VDC. information.
You Cannot Allocate an Interface to a VDC

When you have a problem with creating a VDC, you may see the following system message:
Error Message: VDC_MGR-2-VDC_BAD: vdc_mgr: There has been a failure at gim (port_affected_list).
Explanation: An interface allocation has failed.
Recommended Action: Use the show vdc membership status or show interface brief command to gather more information.

You are not logged in as network-admin. Log into the device with an account that has the correct privileges.
Use the switchto command to switch to the default VDC to allocate
You are not logged into the correct VDC.
resources.
Use the show interface capabilities command to determine if the
You cannot The interface is part of a dedicated port
port is dedicated. All ports in a dedicated port group must be in the
allocate an group.
same VDC.
interface to a
VDC. You must allocate all ports in a port group to the same VDC for
The interface is on the Cisco Nexus 7000
this module. For information about the port number to port group
Series 32-port 10-Gbps Ethernet module
mapping, see Table: Port Numbers for Cisco Nexus 7000 Series
(N7K-M132XP-12).
32-port 10-Gbps Ethernet module.
Use the show vdc membership [status] or show interface brief
The VDC allocation has failed.
command to gather more information.
Table: Port Numbers for Cisco Nexus 7000 Series 32-port 10-Gbps Ethernet module. shows the port allocation requirements for
the Cisco Nexus 7000 Series 32-port 10-Gbps Ethernet module (N7K-M132XP-12).
Table: Port Numbers for Cisco Nexus 7000 Series 32-port 10-Gbps Ethernet module
Port Group Port Numbers

1 1, 3, 5, 7
2 2, 4, 6, 8
3 9, 11, 13, 15
4 10, 12, 14, 16
5 17, 19, 21, 23
6 18, 20, 22, 24
7 25, 27, 29, 31
You Cannot Delete a VDC 40

8 26, 28, 30, 32
The VDC Does Not Reflect a Resource Template Change

You may have a problem when updating a resource template.

Use the show vdc resource template command to verify the template. Use the
The VDC does not The template has not
template command in VDC configuration mode to reapply the template to the
reflect a resource been reapplied to a VDC
VDC. You may have to use the reload command to reboot the device or force a
template change. after a template change.
stateful switchover to get the new resource limits.
The VDC Remains in a Failed State

You may have a problem when a VDC fails. You configure switchover and high availability (HA) policies for a VDC when you
create the VDC. These policies determine what happens when the VDC fails or when a stateful switchover occurs to the standby
supervisor.

Use the show vdc detail command to verify the HA policy for
The VDC failed and the HA policy was set
this VDC. Use the ha-policy command in VDC configuration
to bring down the VDCs.
The VDC remains mode to change the HA policy.
in failed state. A supervisor switchover has occurred and Use the no vdc command to delete the failed VDC. Recreate the
the switchover policy was set to bring VDC with a different switchover policy using the sw-policy
down the VDCs. keyword.
You Cannot Copy the Running-Config File to the Startup-Config File in a VDC
You may have a problem when trying to save the configuration in a VDC.

You must save the resource allocation from the default VDC before you can save
You cannot copy the The resource the configuration in a nondefault VDC. Log into the default VDC and use the copy
running-config file to the allocation was not running-config startup-config command to save the resource allocation. Log into
startup-config file in a saved in the the nondefault VDC and save the configuration or use the copy running-config
VDC. default VDC. startup-config vdc-all command in the default VDC to save the configuration in
all VDCs.
See Also
Table: Port Numbers for Cisco Nexus 7000 Series 32-port 10-Gbps Ethernet module 41
Further Reading
Cisco Nexus 7000 Series NX-OS Virtual Device Context Configuration Guide
Technical Overview of Virtual Device Contexts
Cisco Techwise TV: NXOS Virtual Devices (video)
External Links
Hands on with the Cisco Nexus, Part#2: Virtualization
NX-OS Intro - part 8 - VDCs (video)
NX-OS and VDCs
This article describes how to troubleshoot Cisco Fabric Services (CFS) problems on a Cisco NX-OS device.
Guide Contents
Troubleshooting CFS {this section}
Troubleshooting STP
Contents
• 1 Information About
Troubleshooting CFS
♦ 2.1 Verifying CFS Using
the CLI
Further Reading 42
• 3 Troubleshooting Merge Failures

• 4 Troubleshooting Lock Failures
• 5 Troubleshooting CFS Regions
♦ 5.1 Changing CFS Regions
• 6 See Also
Information About Troubleshooting CFS

Many features in Cisco NX-OS require configuration synchronization across multiple devices in the network. CFS provides a
common infrastructure for automatic configuration synchronization for an application in the network. It provides the transport
function as well as a rich set of common services to the applications. CFS can discover CFS-capable devices in the network as
well as their application capabilities.
Some of the applications that can be synchronized using CFS are as follows:
• Call Home
• RADIUS
• TACACS+
• User roles
Note: Do not enable CFS for an application that you manage using Cisco DCNM.
You can use CFS regions to limit the CFS configuration distribution to a subset of devices on the network.

Begin troubleshooting CFS issues by checking the following issues first:
Check
Checklist
off
Verify that CFS is enabled for the same applications on all affected devices.
Verify that CFS distribution is enabled for the same applications on all affected devices.
If you are using CFS regions, verify that the application is in the same region on all the affected devices.
Verify that there are no pending changes for an application and that a CFS commit was issued for any configuration
changes in a CFS-enabled application.
Verify that there are no unexpected CFS locked sessions. Clear any unexpected locked sessions.
Contents 43
Verifying CFS Using the CLI

To verify CFS using the CLI, follow these steps:
1. Verify that CFS is globally enabled on all devices in the network or CFS region.
switch(config)# show cfs status

Distribution : Enabled
Distribution over IP : Enabled - mode IPv4
IPv4 multicast address : 239.255.70.83
IPv6 multicast address : ff15::efff:4653
Distribution over Ethernet : Disabled
2. Verify that CFS is enabled for the application on all devices in the network or CFS region.
switch(config)# show cfs application

----------------------------------------------
Application Enabled Scope &
----------------------------------------------
ntp No Physical-fc-ip
stp Yes Physical-eth
vpc Yes Physical-eth
igmp Yes Physical-eth
l2fm Yes Physical-eth
role Yes Physical-fc-ip
radius Yes Physical-fc-ip
tacacs No Physical-fc-ip
callhome Yes Physical-fc-ip
Total number of entries = 9
The Physical-fc-ip scope means that CFS uses IP to apply the configuration for that application to all devices in the network or
region. The Physical-eth scope means that CFS uses Ethernet to apply the configuration for that application to all devices in the
network or region.
3. Verify that CFS distribution is enabled for the application on all devices in the network or CFS region.
switch(config)# show cfs application name radius

Enabled : Yes
Timeout : 20s
Merge Capable : Yes
Scope : Physical-fc-ip
Region : 99
4. If you configure CFS regions, verify that the application is in the same region on all applicable devices.
switch(config)# show cfs regions brief

---------------------------------------
Region Application Enabled
---------------------------------------
4 callhome yes
99 radius yes
Verifying CFS Using the CLI 44

5. Verify the set of devices that are registered with CFS for that application.
switch# show cfs peers name radius

--------------------------------------------------
Switch WWN IP Address
--------------------------------------------------
20:00:00:0e:d7:0e:bf:c0 192.0.2.51 [Local]
20:00:00:0e:d7:00:3c:9e 192.0.2.52
6. Compare the output of the show cfs merge status name application-name command and the show cfs peers name
application-name command to verify that the network is not partitioned.
switch# show cfs merge status name radius

Physical-fc-ip Merge Status: Success [ Mon Jan 5 11:59:36 2009 ]
Local Fabric
---------------------------------------------------------
---------------------------------------------------------
20:00:00:05:30:00:4a:de 192.0.2.51 [Merge Master]
20:00:00:0d:ec:0c:f1:40 192.0.2.204
Total number of switches = 2
switch# show cfs peers name radius

--------------------------------------------------
--------------------------------------------------
20:00:00:0d:ec:0c:f1:40 192.0.2.51 [Local]
20:00:00:05:30:00:4a:de 192.0.2.204
If the list of switch WWNs in the show cfs merge status name command output is shorter than the list of switch WWNs in
theshow cfs peers name command output, the network is partitioned into multiple CFS fabrics and the merge status may show
that the merge has failed, is pending, or is waiting.
7. Verify that a distribution is not in progress in the network for the application.
switch# show cfs lock

Application: callhome
-------------------------------------------------------------------------------
Switch WWN IP Address User Name User Type
-------------------------------------------------------------------------------
20:00:00:22:55:79:a4:c1 172.28.230.85 admin CLI/SNMP v3
switch
If the application does not show in the output, the distribution has completed.
8. Verify that there are no CFS sessions in progress for the application.
switch(config)# show radius session status
Verifying CFS Using the CLI 45

Last Action Time Stamp : Wed Dec 24 13:25:00 2008
Last Action : Commit
Last Action Result : Success
Last Action Failure Reason : none
Troubleshooting Merge Failures

During a merge, the merge managers in the merging networks exchange their configuration databases with each other. The
application on the merge master device merges the information, decides if the merge is successful, and informs all devices in the
combined network of the status of the merge. When a merge is successful, the merge master distributes the database to all devices
in the combined network and the combined network remains in a consistent state. A merge failure indicates that the merged
network contains inconsistent data that could not be merged.
If you add a new device to the network and the merge status for any application shows "In Progress" for a prolonged period of
time, then there may be an active session for that application in some other device. Use the show cfs lock command to check the
lock status for that application on all the devices. The merge will not proceed if there are any locks present for that application on
any device in the network or CFS region. Use the application-name commit command to commit the changes or use the clear
application-name session command to clear the session lock so that the merge can proceed.
To recover from a merge failure using the CLI, follow these steps:
1. Identify a device that shows a merge failure.
switch# show cfs merge status

-------------------------------------------------------------
Application Scope Vsan Status
-------------------------------------------------------------
role Physical-fc-ip - Success
radius Physical-fc-ip - Success
callhome Physical-fc-ip - Failed
2. Commit the application configuration to restore all peers in the fabric to the same configuration database.
switch(config)# callhome commit
Troubleshooting Lock Failures

In order to distribute a configuration in the network, CFS must first acquire a lock on all devices in the network or CFS region.
Once CFS acquires the locks, CFS issues a commit to distribute the configuration to all devices in the network or CFS region.
Under normal circumstances, CFS releases the lock after the commit.
When another application peer acquires a lock, you cannot commit new configuration changes. This is a normal operation and you
should postpone any changes to an application until the application peer releases the lock.
An inconsistent lock state also occur in the following scenarios:
• When locks are not held on all of the devices in the network or CFS region.
• When locks are held on all devices in the network or region, but a CFS session does not exist on the device that holds the
lock.
Note: Use the troubleshooting steps in this section only if you believe the lock has not been properly released.
To troubleshoot a lock failure, follow these steps:
1. Determine all the devices that participate in the CFS distribution for this application.
switch1# show cfs peers name radius

--------------------------------------------------
--------------------------------------------------
20:00:00:0d:ec:0c:f1:40 192.0.2.51 [Local]
20:00:00:05:30:00:4a:de 192.0.2.204
2. Check for a lock for this application on all CFS peer devices to determine the name of the administrator who owns the lock for
the application.
switch2# show cfs lock

Application: radius
--------------------------------------------------------------------------------
Switch WWN IP Address User Name User Type
--------------------------------------------------------------------------------
20:00:00:05:30:00:4a:de 192.0.2.204 admin CLI/SNMP v3
switch
You should check with that administrator before clearing the lock.
3. Connect to the device that owns the CFS lock.
4. Release the CFS lock on the device that owns the lock.
switch2# radius abort
5. If the device does not release the lock, clear the CFS session on the device that owns the lock.
switch2# clear radius session
Troubleshooting CFS Regions

The following rules apply to CFS regions:
• When using CFS regions, an application on a given device can only belong to one region at a time.
• An application in a CFS region ignores all CFS distributions in any other region (including the default region).
• All applications that you do not assign to a CFS region exist in the default region.
To resolve a configuration distribution failure to all devices in a CFS region, follow these steps:
Troubleshooting Lock Failures 47

1. Verify the list of devices in a region for the application.
switch(config)# show cfs region name radius

Region-ID : 4
Application: radius
-------------------------------------------------------------------------
-------------------------------------------------------------------------
20:00:00:22:55:79:a4:c1 172.28.230.85 [Local]
switch
2. Verify that the application distribution is enabled and is in the same region on all devices in the region.
switch2# show cfs application name radius

Enabled : Yes
Timeout : 20s
Merge Capable : Yes >>>>> Application is capable of being merged.
Region : 1 >>>>> Application is in Region 1.
switch2(config)# cfs region 4

switch2(config-cfs-region)# radius
Note: You must reassign an application to a region whenever you disable that application. CFS assigns new applications in the
default region.
Changing CFS Regions

If you move an application from one region to another, you may encounter a database mismatch when attempting a merge. Follow
the steps outlined in the Troubleshooting Merge Failures to identify and resolve the conflicts.
Note: When an application is moved from one region to another (including the default region), the application loses all CFS
history.
See Also
Further Reading
Configuring CFS
External Links
Troubleshooting CFS Regions 48

This article describes how to identify and resolve problems that can occur with ports in Cisco NX-OS.
Guide Contents
Troubleshooting CFS
Troubleshooting Ports (this section)
Troubleshooting STP
Contents
• 1 Information About Troubleshooting Ports
• 2 Port Guidelines
• 3 License Requirements
♦ 4.1 Viewing Port Information
• 5 Troubleshooting Port States from the CLI
♦ 5.1 Example: show interface Command Output
• 6 Port-Interface Issues
♦ 6.1 You Cannot See The Interface
♦ 6.2 The Interface Configuration Has
Disappeared
♦ 6.3 You Cannot Enable an Interface
♦ 6.4 You Cannot Configure a Dedicated Port
♦ 6.5 A Port Remains in a Link Failure or Not
Connected State
♦ 6.6 An Unexpected Link Flapping Occurs
♦ 6.7 A Port Is in the ErrDisabled State
◊ 6.7.1 Verifying the ErrDisable State
Using the CLI
• 7 See Also
Information About Troubleshooting Ports

Before a switch can relay frames from one data link to another, the characteristics of the interfaces through which the frames are
received and sent must be defined. The configured interfaces can be Ethernet interfaces, the management interface (mgmt0), or
External Links 49
VLAN interfaces (SVIs).
Each interface has an associated administrative configuration and operational status as follows:
• The administrative configuration does not change unless you modify it. This configuration has various attributes that you
can configure in administrative mode.
• The operational status represents the current status of a specified attribute like the interface speed. This status cannot be
changed and is read-only. Some values may not be valid when the interface is down (such as the operation speed).
For a complete description of port modes, administrative states, and operational states, see the Cisco NX-OS Interfaces
Configuration Guide.
Port Guidelines
Follow these guidelines when you configure a port interface:
• Before you begin configuring a switch, make sure that the modules in the chassis are functioning as designed. Use the
show module command to verify that a module is OK or active before continuing the configuration.
• When configuring dedicated ports in a port group, follow these port mode guidelines:
♦ You can configure only the one port in each four-port group in dedicated mode. The other three ports are not
usable and remain shut down.
♦ If any of the other three ports are enabled, you cannot configure the remaining port in dedicated mode. The other
three ports continue to remain enabled.
License Requirements
There are no licensing requirements for port configuration in Cisco NX-OS.

Begin troubleshooting the port configuration by checking the following issues:
Check
Checklist
off
Check the physical media to ensure that there are no damaged parts.
Verify that the SFP (small form-factor pluggable) devices in use are those authorized by Cisco and that they are not
faulty.
Verify that you have enabled the port by using the no shutdown command.
Use the show interface command to verify the state of the interface. See the Cisco NX-OS Interfaces Configuration
Guide for reasons why a port may be in a down operational state.
Verify that you have configured a port as dedicated and make sure that you have not connected to the other three ports in
the port group.
Information About Troubleshooting Ports 50

Viewing Port Information

You can use the show interface counters command to view port counters. Typically, you only observe counters while actively
troubleshooting, in which case you should first clear the counters to create a baseline. The values, even if they are high for certain
counters, can be meaningless for a port that has been active for an extended period. Clearing the counters provides a better idea of
the link behavior as you begin to troubleshoot.
Use one of the following commands to clear all port counters or the counters for specified interfaces:
• clear counters interface all

• clear counters interface range
The counters can identify synchronization problems by displaying a significant disparity between received and transmitted frames.
Use the following commands to gather more information about ports:
• show interface status

• show interfaces capabilities
• show udld
• show tech-support udld
Troubleshooting Port States from the CLI

To display complete information for an interface, use the show interface command. In addition to the state of the port, this
command displays the following:
• Speed
• Trunk VLAN status
• Number of frames sent and received
• Transmission errors, including discards, errors, and invalid frames
Example: show interface Command Output displays the show interface command output.
Example: show interface Command Output
switch(config)# show interface ethernet 2/45

Ethernet2/45 is down (Administratively down)
Hardware is 10/100/1000 Ethernet, address is 0019.076c.4dd8 (bia 0019.076c.4dd8)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
auto-duplex, auto-speed
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned on
Last clearing of "show interface" counters never
1 minute input rate 0 bytes/sec, 0 packets/sec
1 minute output rate 0 bytes/sec, 0 packets/sec
L3 Switched:
input: 0 pkts, 0 bytes - output: 0 pkts, 0 bytes
Rx
0 input packets 0 unicast packets 0 multicast packets
0 broadcast packets 0 jumbo packets 0 storm suppression packets
0 bytes
Tx
0 output packets 0 multicast packets
0 broadcast packets 0 jumbo packets
0 bytes
0 input error 0 short frame 0 watchdog
0 no buffer 0 runt 0 CRC 0 ecc
0 overrun 0 underrun 0 ignored 0 bad etype drop
0 bad proto drop 0 if down drop 0 input with dribble
0 output error 0 collision 0 deferred
0 late collision 0 lost carrier 0 no carrier
0 babble
0 Rx pause 0 Tx pause 0 reset
Receive data field Size is 2112
Port-Interface Issues
This section includes symptoms and solutions for troubleshooting ports.
You Cannot See The Interface

You may have a problem where an interface does not show up on your device because of the VDC configuration.

You cannot see the The interface has been allocated Log in as network-admin and use the show vdc membership command
interface. to a different VDC. to determine which VDC owns the interface.
The Interface Configuration Has Disappeared

You may have a problem where your interface configuration disappears.

Cisco NX-OS removes the interface configuration when you
The interface was reallocated to a
reallocate an interface to a different VDC. You must reconfigure
The interface different VDC.
the interface.
configuration has
disappeared. Cisco NX-OS removes the interface configuration when you switch
The interface mode has changed
between Layer 2 and Layer 3 port mode. You must reconfigure the
to or from the switchport mode.
interface.
Example: show interface Command Output 52

You Cannot Enable an Interface

You may have a problem when enabling an interface.

You cannot enable the other three ports in a port group if one port is dedicated.
The interface is part of a a
Use the show running-config interface CLI command to verify the rate mode
dedicated port group.
setting.
Use the show interface capabilities command on both ports to determine if
The interface configuration is
You cannot both ports have the same capabilities. Modify the configuration as needed to
incompatible with a remote port.
enable an make the ports compatible.
interface. The Layer 2 port is not Use the show interface brief command to see if the interface is configured in a
associated with an access VLAN. Use the show vlan brief command to determine the status of the
VLAN or the VLAN is VLAN. Use the state active command in VLAN configuration mode to
suspended. configure the VLAN as active.
An incorrect SFP is connected Use the show interface brief command to see if you are using an incorrect
to the port. transceiver. Replace with a Cisco-supported SFP.
You Cannot Configure a Dedicated Port

You may have a problem when trying to configure a port as dedicated.

The other three ports in the port group are not Use the shutdown command in interface configuration
shut down. mode to disable the other three ports in the port group.
You cannot configure One or more of the other three ports in the port Use the show vdc membership command to find out
a dedicated port. group are not configured in the same VDC. which ports are in a different VDC.
You can only set the first port in a port group to the
The port is not the first port in the port group.
dedicated mode.
A Port Remains in a Link Failure or Not Connected State

You may have a problem with ports or links becoming operational.

The port connection is bad. Use the show port internal info command to verify the port status is in
link-failure.
Verify the type of media in use. Is it copper or optical, single-mode (SM), or

A port remains multimode (MM)?
in a link-failure
state.
You Cannot Enable an Interface 53
Verify that the media is not broken or damaged. Is the LED on the switch
green?
Use the shutdown command followed by the no shutdown command to

disable and enable the port. If this problem persists, try moving the connection
to a different port on the same or another module.
When this problem occurs, the port stays in a transit port state and you see no
There is no signal because of a
signal. There is no synchronization at the MAC level. The problem may be
transit fault in the small
related to the port speed setting or autonegotiation. Verify that the SFP on the
form-factor pluggable (SFP) or
interface is seated properly. If reseating the SFP does not resolve the issue,
the SFP may be faulty.
replace the SFP or try another port on the switch.
Use the show logging command to check for a "Link Failure, Not Connected
system" message.
The link is stuck in the
initialization state or the link is
in a point-to-point state. Use the shutdown command followed by the no shutdown command to
disable and enable the port. If this problem persists, try moving the connection
to a different port on the same or another module.
An Unexpected Link Flapping Occurs

When a port is flapping, it cycles through the following states, in this order, and then starts over again:
1. Initializing-The link is initializing.

2. Offline-The port is offline.
3. Link failure or not connected-The physical layer is not operational and there is no active device connection.
When you are troubleshooting an unexpected link flapping, you should know the following information:
• Who initiated the link flap.

• The actual link down reason.

An unexpected The bit rate exceeds the threshold and
Use the shutdown command followed by the no shutdown command
link flapping puts the port into the errDisabled
to return the port to the normal state.
occurs. state.
A problem in the system triggers the Determine the link flap reason as indicated by the MAC driver. Use the
link flap action by the end device. debug facilities on the end device to troubleshoot the problem. An
Some of the causes are as follows: external device may choose to reinitialize the link when it encounters
the error. In such cases, the method of reinitializing the link varies by
• A packet drop in the switch device.
occurs, because of either a
hardware failure or an
intermittent hardware error
such as an X-bar sync loss.
A Port Remains in a Link Failure or Not Connected State 54

• A packet drop results from a

software error.
• A control frame is
erroneously sent to the device.
A Port Is in the ErrDisabled State

The ErrDisabled state indicates that the switch detected a problem with the port and disabled the port. This state could be caused
by a flapping port or a high amount of bad frames (CRC errors), which could indicate a problem with the media.

The port is flapping.
A port is in the Use the Verify the ErrDisable State Using the CLI
ErrDisabled state. The device detected a high amount of bad frames procedure to verify the SFP, cable, and connections.
(CRC errors), which might indicate a problem with
the media.
Verifying the ErrDisable State Using the CLI
To verify the ErrDisable state using the CLI, follow these steps:
1. Use the show interface command to verify that the switch detected a problem and disabled the port.
switch# show interface e1/14

e1/7 is down (errDisabled)
2. Check cables, SFPs, and optics.
3. View information about the internal state transitions of the port.
switch# show system internal ethpm event-history interface e1/7

>>>>FSM: <e1/7> has 86 logged transitions<<<<<
1) FSM:<e1/7> Transition at 647054 usecs after Tue Jan 1 22:44..
Previous state: [ETH_PORT_FSM_ST_NOT_INIT]
Triggered event: [ETH_PORT_FSM_EV_MODULE_INIT_DONE]
Next state: [ETH_PORT_FSM_ST_IF_INIT_EVAL]
2) FSM:<e1/7> Transition at 647114 usecs after Tue Jan 1 22:43..
Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
Triggered event: [ETH_PORT_FSM_EV_IE_ERR_DISABLED_CAP_MISMATCH]
Next state: [ETH_PORT_FSM_ST_IF_DOWN_STATE]
In this example, port ethernet 1/7 entered the ErrDisabled state because of a capability mismatch, or "CAP MISMATCH."
4. Display the switch log file and view a list of port state changes.
switch# show logging logfile

. . .
An Unexpected Link Flapping Occurs 55

Jan 4 06:54:04 switch %PORT_CHANNEL-5-CREATED: port-channel 7 created
Jan 4 06:54:24 switch %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel 7 is down (No operational
Jan 4 06:54:40 switch %PORT_CHANNEL-5-PORT_ADDED: e1/8 added to port-channel 7
Jan 4 06:54:56 switch %PORT-5-IF_DOWN_ADMIN_DOWN: Interface e1/7 is down (Admnistratively down)
Jan 4 06:54:59 switch %PORT_CHANNEL-3-COMPAT_CHECK_FAILURE: speed is not compatible
Jan 4 06:55:56 switch%PORT_CHANNEL-5-PORT_ADDED: e1/7 added to port-channel 7
In this example, an error was recorded when someone attempted to add port e1/7 to port channel 7. The port was not configured
identically to port channel 7, so the attempt failed.
See Also
Cisco NX-OS/IOS Interface Comparison
Further Reading
Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide
External Links
NX-OS Intro part 5 -Port channels (video)
packetlife.net - Errdisable autorecovery
This article describes how to do basic troubleshooting of virtual Port Channel(vPC) problems on a Cisco Nexus 7000 NX-OS
device.
Guide Contents
Troubleshooting CFS
Troubleshooting vPCs {this section}
Troubleshooting STP
Verifying the ErrDisable State Using the CLI 56


Contents
• 1 Information About Troubleshooting vPCs
• 3 Verifying vPCs Using the CLI
• 4 Received Type 1 Configuration Element Mismatch
♦ 4.1 Example: show vpc consistency-parameters
Command Output
• 5 Cannot Enable the vPC Feature
♦ 5.1 Example: show module Command Output
• 6 vPC in Blocking State
• 7 VLANs on a vPC moved to suspend state
• 8 Hosts with an HSRP Gateway Cannot Access Beyond Their
VLAN
• 9 Traffic Disrupted when the Primary vPC Device Goes Down
• 10 See Also
Information About Troubleshooting vPCs

A vPC allows links that are physically connected to two different Cisco Nexus 7000 Series devices to appear as a single port
channel by a third device.
See the Configuring vPC chapter in the Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide for more information on
vPCs.

Begin troubleshooting vPC issues by checking the following issues first:
Check
Checklist
off
Verify that all vPC interfaces in a vPC domain are configured in the same virtual device context (VDC).
Verify that you have a separate vPC peer-link and peer-keepalive link infrastructure for each VDC deployed.
Is the vPC keepalive link mapped to a separate vrf? If not, it will be mapped to the management vrf by default. In this
case, do you have a management switch connect to the management ports on both vPC peer devices?
Verify that the vPC peer-link is configured on a N7K-M132XP-12. It is recommended to have at least two
N7K-M132XP-12 for redundancy.
External Links 57
Verify that both the source and destination IP addresses used for the peer-keepalive messages are reachable from the
VRF associated with the vPC peer-keepalive link.
Verify that the peer-keepalive link is up or the vPC peer-link will not come up.
Verify that the vPC peer-link is configured as a Layer 2 Port Channel trunk which only allows vPC VLANs.
Verify that the vPC number that you assigned to the port channel that connects to the downstream device from the vPC
peer device is identical on both vPC peer devices.
If you manually configured the system priority, verify that you assigned the same priority value on both vPC peer
devices.
Check the show vpc consistency-parameters command to verify that both vPC peer devices have identical type-1
parameters.
Verify that the primary vPC is the primary STP root and the secondary vPC is the secondary STP root.
Verifying vPCs Using the CLI
To verify vPCs using the CLI, follow these steps:
1. Use the show running-config vpc command to verify the vPC configuration.
2. Use the show vpc command to check the status of vPC.
3. Use the show vpc peer-keepalive command to check the status of the vPC peer-keepalive link.
4. Use the show vpc consistency-parameters command to verify that both the vPC peers have the identical type-1 parameters.
5. Use the show port-channel summary command toverify the members in the port channel are mapped to the vPC.
6. Use the show cfs status commands to verify that distribution over Ethernet is enabled.
7. If you enable STP, use the show spanning-tree command on both sides of the vPC peer link to verify that the following STP
parameters are identical:
• BPDU Filter
• BPDU Guard
• Cost
• Link type
• Priority
• VLANs (PVRST+)
Received Type 1 Configuration Element Mismatch

You may have a problem where you cannot bring up a vPC link because of a type 1 configuration element mismatch.

Received a type 1 The vPC peer ports or membership Use the show vpc consistency-parameters interface
configuration element ports do not have identical command to determine where the configuration mismatch
mismatch. configurations. occurs.

Example: show vpc consistency-parameters Command Output
This example shows how to display the vPC consistency parameters on a port channel:
switch# show vpc consistency-parameters interface po 10

Legend:
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
STP Mode 1 Rapid-PVST Rapid-PVST
STP Disabled 1 None None
STP MST Region Name 1 "" ""
STP MST Region Revision 1 0 0
STP MST Region Instance to 1
VLAN Mapping
STP Loopguard 1 Disabled Disabled
STP Bridge Assurance 1 Enabled Enabled
STP Port Type 1 Normal Normal
STP MST Simulate PVST 1 Enabled Enabled
Allowed VLANs - 1-10,15-20,30,37,99 1-10,15-20,30,37,99
Cannot Enable the vPC Feature

You may receive an error when you enable the vPC feature.

The hardware is Use the show module command to determine the hardware version of each
Cannot enable the
incompatible with the N7K-M132XP-12 Ethernet module. The hardwere version must be 1.3 or later to
vPC feature.
vPC. enable the vPC feature.
Example: show module Command Output
This example shows how to display the module hardware version:
switch# show module

Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
2 32 10 Gbps Ethernet Module N7K-M132XP-12 ok
3 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok
5 0 Supervisor module-1X N7K-SUP1 active *
6 0 Supervisor module-1X N7K-SUP1 ha-standby
10 32 10 Gbps Ethernet Module N7K-M132XP-12 ok
Mod Sw Hw
--- -------------- ------
2 4.1(5) 1.2
3 4.1(5) 1.0 >>> Must be 1.3 or later.
vPC in Blocking State

vPC may be in the blocking state because of Bride Assurance (BA).
Example: show vpc consistency-parameters Command Output 59


vPC is in blocking BPDU only sends on a single link of a port-channel. If BA dispute is detected, the entire Do not enable
state. vPC will be in the blocking state. BA on vPC.
VLANs on a vPC moved to suspend state

VLANs on a vPC may move to the suspend state.

VLANs on a vPC VLANs allowed on the vPC have All VLANs allowed on a vPC must also be allowed on the vPC
moved to suspend not been allowed on the vPC peer-link. Also, it is recommended that only vPC VLANs are allowed
state. peer=link. on the vPC peer-link.
Hosts with an HSRP Gateway Cannot Access Beyond Their VLAN

When HSRP is enabled on both vPC peer devices on a VLAN and hosts on that VLAN set the HSRP as their gateway, they may
not able to reach anything outside their own VLAN.

Hosts with an If the host gateway mac-address is mapped Map the host gateway's mac-address to the HSRP MAC
HSRP gateway to the physical MAC address of any one of address and not the physical MAC address of any one of the
cannot access the vPC peer-devices, packets may get vPC peer-devices. Peer-gateway can be a workaround for this
beyond their dropped due to the loop prevention scenario. Please read the configuration guide for peer-gateway
VLAN. mechanism in vPC. for further information before implementing it.
Traffic Disrupted when the Primary vPC Device Goes Down

Traffic may remain disrupted when the N7K-M132XP-12 module on the primary vpc device goes down.

Enable object tracking. With object tracking enabled, all vPC on the
Traffic disrupted
All core facing interfaces and vPC primary will shut down. The vPC secondary will take over as the
when the primary
peer-links are configured on a operational primary and all the vPC on the secondary will stay up. As a
vPC device goes
single N7K-M132XP-12 module. result, traffic will still be flowing thru the secondary which became
down.
operational primary.
See Also
vPC in Blocking State 60

Further Reading
Configuring vPCs (Cisco Nexus 7000 Series Interfaces Configuration Guide)
Nexus 5000 Virtual PortChannel Quick Configuration Guide
External Links
The following links contain content developed by external authors. Cisco does not review this content for accuracy.
Our Nexus Data Center Network - To vPC or not to vPC
Nexus 7000 Virtual Portchannel Part 1
vPC (Virtual Port-Channel) and the Nexus 5000 Platform
Blog on Cisco Nexus Features (VLANs, vPCs)
This article describes how to troubleshoot VLANs.
Guide Contents
Troubleshooting CFS
Troubleshooting VLANs (this section)
Troubleshooting STP
Contents
• 1 Information About Troubleshooting
VLANs
Further Reading 61
• 3 VLAN Issues
♦ 3.1 You Cannot Create a
VLAN
♦ 3.2 You Cannot Create a
PVLAN
♦ 3.3 The VLAN Interface is
Down
• 4 See Also
Information About Troubleshooting VLANs

VLANs provide a method of isolating devices that are physically connected to the same network but are logically considered to be
part of different LANs that do not need to be aware of one another.
You should use only the following characters in a VLAN name:
• a through z or A through Z
• 0 through 9
• - (hyphen) or _ (underscore)
Follow these guidelines when configuring VLANs:
• Keep user traffic off the management VLAN; keep the management VLAN separate from user data.
• You can apply different Quality of Service (QoS) configurations to primary, isolated, and community VLANs.
• To apply output VACLs to all outgoing private VLAN traffic, map the secondary VLANs on the Layer 3 VLAN interface
of the primary VLAN and then configure the VACLs on the SVI of the primary VLAN.
• VACLs that apply to the Layer 3 VLAN interface of a primary VLAN automatically apply to the associated isolated and
community VLANs.
• If you do not map the secondary VLAN to the Layer 3 VLAN interface of the primary VLAN, you can have different
VACLs for primary and secondary VLANs.
• Because traffic in private VLANs flow in different directions, you can have different VACLs for ingressing traffic and
different VACLs for egressing traffic.
Note: We recommend that you keep the same VACLs for the primary VLAN and all secondary VLANs in the private VLAN.
• You can enable DHCP snooping on private VLANs. When you enable DHCP snooping on the primary VLAN, it is
propagated to the secondary VLANs. If you configure DHCP on a secondary VLAN, the configuration does not take
effect if the primary VLAN is already configured.
• You can configure IEEE 802.1X port-based authentication on a private VLAN port, but do not configure 802.1X with port
security or per-user ACL on private VLAN ports.
• 802.1X works with private VLANs, but the 802.1X dynamic VLAN assignment or the guest VLAN assignment does not
work with private VLANs.
• IGMP runs only on the primary VLAN and uses the configuration of the primary VLAN for all secondary VLANs.
• Any IGMP join request in the secondary VLAN is treated as if it is received in the primary VLAN.
• Private VLANs support these Switched Port Analyzer (SPAN) features:
Contents 62
♦ You can configure a private VLAN port as a SPAN source port.

♦ You can use VLAN-based SPAN (VSPAN) on primary, isolated, or community VLANs or use SPAN on only
one VLAN to separately monitor egress or ingress traffic.
• Do not configure a remote SPAN (RSPAN) VLAN as a private VLAN primary or secondary VLAN.
• A private VLAN host or promiscuous port cannot be a SPAN destination port. If you configure a SPAN destination port
as a private VLAN port, the port becomes inactive.
• A destination SPAN port cannot be an isolated port. (However, a source SPAN port can be an isolated port.)
• You can configure SPAN to span both primary and secondary VLANs or, alternatively, to span either one if the user is
interested only in ingress or egress traffic.
• A MAC address learned in a secondary VLAN is placed in the shared table of the primary VLAN. When the secondary
VLAN is associated to the primary VLAN, their MAC address tables are merged into one, shared MAC table.

Troubleshooting a VLAN problem involves gathering information about the configuration and connectivity of individual devices
and the entire network. Begin your troubleshooting VLAN issues by checking the following issues first:
Checklist Check off

Verify the physical connectivity for any problem ports or VLANs.
Verify that you have both end devices in the same VLAN.
The following CLI commands are used to display VLAN information:
• show vlan vlan-id

• show vlan private-vlan
• show vlan all-ports
• show vlan private-vlan
• show vlan private-vlan type
• show interface vlan vlan-id private-vlan mapping
• show tech-support vlan
VLAN Issues
This section includes symptoms and solutions for VLAN issues.
You Cannot Create a VLAN

You may have a problem when creating a VLAN.

There are not enough Use the show vdc resource vlan command to determine how many unused VLANs
You cannot resources in the virtual that you can configure. If this value is 0, log in as network-admin and use the
create a device context (VDC). limit-resource command in VDC configuration mode to add more VLAN resources
VLAN.
Information About Troubleshooting VLANs 63
to this VDC.
You are using a reserved VLANs 3968 to 4047 and 4094 are reserved for internal use in each VDC; you
VLAN ID. cannot change or use these reserved VLANs.
You Cannot Create a PVLAN

You may experience issues creating a private VLAN (PVLAN).

Use the feature pvlan command to enable the PVLAN
You cannot create a PVLAN. The PVLAN feature is not enabled.
feature.
The VLAN Interface is Down

You may have a problem when configuring VLAN interfaces.

Use the show vlan command to determine if the VLAN exists. Use the vlan
The VLAN does not exist.
command to create the VLAN.
Use the show vlan internal vlan-info command to check the operating state
No interfaces on the VLAN are
of the Spanning Tree Protocol (STP). Configure STP so that at least one
in the STP forwarding state.
The VLAN interface goes into the STP forwarding state.
interface is One or more services prevented Use the show vlan internal vlan-info command to determine the state of the
down. the VLAN interface from VLAN interface. If the state is oper-es, use the show tech-support interface
coming up. vlan command to gather more information.
The VLAN is a secondary Use the show vlan internal vlan-info command to determine the state of the
VLAN. VLAN interface. Change the VLAN to a primary or user VLAN.
The interface is in the wrong Use the show vrf interface command to determine the interface that the
VRF. VLAN interface is assigned to.
See Also
Further Reading
Cisco Nexus 7000 Series NX-OS Layer 2 Switching Configuration Guide
You Cannot Create a VLAN 64

External Links
This article describes how to identify and resolve problems that might occur when implementing the Spanning Tree Protocol
(STP).
Guide Contents
Troubleshooting CFS
Troubleshooting STP (this section)
Contents
• 1 Information About Troubleshooting STP
• 3 Troubleshooting STP Data Loops
• 4 Troubleshooting Excessive Packet
Flooding
• 5 Troubleshooting Convergence Time
Issues
• 6 Securing the Network Against Forwarding
Loops
• 7 See Also
External Links 65
Information About Troubleshooting STP

STP provides a loop-free network at the Layer 2 level. Layer 2 LAN ports send and receive STP frames at regular intervals.
Network devices do not forward these frames but use the frames to construct a loop-free path. See the Cisco NX-OS Layer 2
Switching Configuration Guide for more information on STP.
Follow these guidelines when configuring STP:
• If you are running private VLANs with multiple STP (MST), verify that all secondary VLANs belong to the same MST
instance as that of the primary VLANs.
• Disabling spanning tree on the native VLAN of an 802.1Q trunk when you are working in Rapid PVST+ spanning tree
mode can cause a spanning tree loop on that VLAN. We recommend that you leave spanning tree enabled on the native
VLAN of the 802.1Q trunks. Make sure that your network has no physical loops before you disable spanning tree.
• When you connect two Cisco switches through 802.1Q trunks, the switches exchange spanning tree bridge protocol data
units (BPDUs) on each VLAN allowed on the trunks. The BPDUs on the native VLAN of the trunk are sent untagged to
the reserved IEEE 802.1D spanning tree multicast MAC address (01-80-C2-00-00-00). The BPDUs on all other VLANs
on the trunk are sent tagged to the reserved Cisco Shared Spanning Tree (SSTP) multicast MAC address
(01-00-0c-cc-cc-cd).
• In STP, the port-channel bundle is considered as a single port. The port cost is the aggregation of all the configured port
costs that are assigned to that channel.
• When a secondary VLAN is associated with the primary VLAN, the STP parameters of the primary VLAN, such as
bridge priorities, are propagated to the secondary VLAN. However, STP parameters do not necessarily propagate to other
devices. You should manually check the STP configuration to ensure that the spanning tree topologies for the primary,
isolated, or community VLANs match exactly so that the VLANs can share the same forwarding database.
• For normal trunk ports, note the following:

♦ There is a separate instance of STP for each VLAN in the private VLAN.
♦ STP parameters for the primary and all secondary VLANs must match.
♦ The primary and all associated secondary VLANs should be in the same MST instance.
♦ The duplex configuration for both sides of the link should be set to full to prevent collisions under heavy traffic
conditions.
♦ In MST mode, a misconfiguration cannot be detected if you configure one end of a link in trunk mode and the
other end of the link in access mode. This misconfiguration will cause an STP loop.
• For nontrunking ports, note the following:

♦ STP is aware only of the primary VLAN for any private VLAN host port; STP does not run on secondary VLANs
on a host port.
• For Rapid PVST+ on private VLANs, note the following:

♦ On a trunk port, the primary and secondary private VLANs are two different logical ports and must have the exact
same STP topology.
♦ On access ports, STP sees only the primary VLAN.
Note: In some cases, the configuration is accepted with no error messages, but the commands have no effect.

Troubleshooting an STP problem involves gathering information about the configuration and connectivity of individual devices
and the entire network.
Begin troubleshooting STP issues by checking the following issues first:
Check
Checklist
off
Verify the type of spanning tree configured on your device.
Verify the network topology including all interconnected ports and switches. Identify all redundant paths on the
network and verify that the redundant paths are blocking.
Use the show spanning-tree summary totals command to verify that the total number of logical interfaces in the
Active state are less than the maximum allowed. See the Cisco NX-OS Layer 2 Switching Configuration Guide for
information on these limits.
Verify the primary and secondary root bridge and any configured Cisco extensions.
Use the following commands to view STP configuration and operational details:
• show running-config spanning-tree

• show spanning-tree summary
• show spanning-tree detail
• show spanning-tree bridge
• show spanning-tree mst
• show spanning-tree mst configuration
• show spanning-tree interface interface-type slot/port [detail]
• show tech-support stp
• show spanning-tree vlan
Use the show spanning-tree blockedports command to display the ports that are blocked by STP.
Use the show mac address-table dynamic vlan command to determine if learning or aging occurs at each node.
Troubleshooting STP Data Loops

Data loops are a common problem in STP networks. Some of the symptoms of a data loop are as follows:
• High link utilization, up to 100 percent

• High CPU and backplane traffic utilization
• Constant MAC address relearning and flapping
• Excessive output drops on an interface
To troubleshoot STP loops, follow these steps:
1. Identify the ports involved in the loop by looking at the interfaces with high link utilization.

switch# show interface ethernet 2/1 | include rate
1 minute input rate 19968 bits/sec, 0 packets/sec

1 minute output rate 3952023552 bits/sec, 957312 packets/sec
2. Shut down or disconnect the affected ports.
switch(config)# interface ethernet 2/1

switch(config-if)# shutdown
3. Locate every switch in the redundant paths using your network topology diagram.
4. Verify that the switch lists the same STP root bridge as the other nonaffected switches.
switch# show spanning-tree vlan 9
VLAN0009
Spanning tree enabled protocol rstp
Root ID Priority 32777''
Address 0018.bad7.db15''
Cost 4
...
5. Verify that the root port is correctly identified as the port with the lowest cost to the root bridge.
switch# show spanning-tree vlan 9
VLAN0009
Spanning tree enabled protocol rstp
Root ID Priority 32777
Address 0018.bad7.db15
Cost 4
Port 385 (Ethernet3/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
6. Verify that the root port and alternate ports are regularly receiving BPDUs.
switch# show spanning-tree interface ethernet 3/1 detail
Port 385 (Ethernet3/1) of VLAN0001 is root forwarding

Port path cost 4, Port priority 128, Port Identifier 128.385
Designated root has priority 32769, address 0018.bad7.db15
Designated bridge has priority 32769, address 0018.bad7.db15
Designated port id is 128.385, designated path cost 0
Timers: message age 16, forward delay 0, hold 0
Number of transitions to forwarding state: 1
The port type is network by default
Link type is point-to-point by default
BPDU: sent 1265, received 1269
7. If the received BPDU counter is not incremented, check if the BPDUs are received by the internal packet manager.
switch# show system internal pktmgr interface ethernet 3/1
Ethernet3/1, ordinal: 36
SUP-traffic statistics: (sent/received)
Packets: 120210 / 15812
Bytes: 8166401 / 1083056
Instant packet rate: 5 pps / 5 pps
Troubleshooting STP Data Loops 68

Average packet rates(1min/5min/15min/EWMA):
Packet statistics:
Tx: Unicast 0, Multicast 120210
Broadcast 0
Rx: Unicast 0, '' Multicast 15812''
Broadcast 0
switch# show system internal pktmgr client 303
Client uuid: 303, 2 filters

Filter 0: EthType 0x4242, Dmac 0180.c200.0000
Filter 0: EthType 0x010b, Snap 267, Dmac 0100.0ccc.cccd
Options: TO 0, Flags 0x1, AppId 0, Epid 0
Ctrl SAP: 171, Data SAP 177 (1)
Rx: 28356632, Drop: 0, Tx: 35498365, Drop: 0
8. If the BPDUs are not received by the packet manager, check the hardware packet statistic (error drop) counters.
switch# show interface counters errors
--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
mgmt0 -- -- -- -- -- --
Eth1/1 0 0 0 0 0 0
Eth1/2 0 0 0 0 0 0
Eth1/3 0 0 0 0 0 0
Eth1/4 0 0 0 0 0 0
Eth1/5 0 0 0 0 0 0
Eth1/6 0 0 0 0 0 0
Eth1/7 0 0 0 0 0 0
Eth1/8 0 0 0 0 0 0
9. Check that the designated port is regularly sending BPDUs.
switch# show spanning-tree interface ethernet 3/1 detail
Port 385 (Ethernet3/1) of VLAN0001 is root forwarding

Port path cost 4, Port priority 128, Port Identifier 128.385
Designated root has priority 32769, address 0018.bad7.db15
Designated bridge has priority 32769, address 0018.bad7.db15
Designated port id is 128.385, designated path cost 0
Timers: message age 16, forward delay 0, hold 0
Number of transitions to forwarding state: 1
The port type is network by default
Link type is point-to-point by default
BPDU: sent 1265, received 1269
10. If the BPDU send counter is incrementing, check if BPDUs are transmitted by the packet manager.
switch# show system internal pktmgr interface ethernet 3/1
Ethernet3/1, ordinal: 36
SUP-traffic statistics: (sent/received)
Packets: 120210 / 15812
Bytes: 8166401 / 1083056
Instant packet rate: 5 pps / 5 pps
Average packet rates(1min/5min/15min/EWMA):
Packet statistics:
Tx: Unicast 0, M'' ulticast 120210''
Broadcast 0
Rx: Unicast 0, Multicast 15812
Troubleshooting STP Data Loops 69

Broadcast 0
switch# show system internal pktmgr client 303
Client uuid: 303, 2 filters

Filter 0: EthType 0x4242, Dmac 0180.c200.0000
Filter 0: EthType 0x010b, Snap 267, Dmac 0100.0ccc.cccd
Options: TO 0, Flags 0x1, AppId 0, Epid 0
Ctrl SAP: 171, Data SAP 177 (1)
Rx: 28356632, Drop: 0, Tx: 35498365, Drop: 0
11. If the packet manager BPDU sent counters is incrementing, check the hardware packet statistic counters for a possible BPDU
error drop.
switch# show interface counters errors
--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
mgmt0 -- -- -- -- -- --
Eth1/1 0 0 0 0 0 0
Eth1/2 0 0 0 0 0 0
Eth1/3 0 0 0 0 0 0
Eth1/4 0 0 0 0 0 0
Eth1/5 0 0 0 0 0 0
Eth1/6 0 0 0 0 0 0
Eth1/7 0 0 0 0 0 0
Eth1/8 0 0 0 0 0 0
Troubleshooting Excessive Packet Flooding

Unstable STP topology changes can trigger excessive packet flooding in your STP network. With Rapid STP or Multiple STP
(MST), a change of the port's state to forwarding, as well as the role change from designated to root can trigger a topology change.
Rapid STP immediately flushes the Layer 2 forwarding table. 802.1D shortens the aging time. The immediate flushing of the
forwarding table restores connectivity faster but causes more flooding.
In a stable topology, a topology change should not trigger excessive flooding. Link flaps can cause a topology change, so
continuous link flaps can cause repetitive topology changes and flooding. Flooding slows the network performance and can cause
packet drops on an interface.
To troubleshoot excessive flooding, follow these steps:
1. Determine the source of the excessive topology change.
switch# show spanning-tree vlan 9 detail
VLAN0009 is executing the rstp compatible Spanning Tree protocol

Bridge Identifier has priority 32768, sysid 9, address 0018.bad8.27ad
Configured hello time 2, max age 20, forward delay 15
Current root has priority 32777, address 0018.bad7.db15
Root port is 385 (Ethernet3/1), cost of root path is 4
Topology change flag not set, detected flag not set
'' Number of topology changes 8 last change occurred 1:32:11 ago''
'' from Ethernet3/1''
Times: hold 1, topology change 35, notification 2
Troubleshooting Excessive Packet Flooding 70

...
2. Determine the interface where the topology change occurred.
switch# show spanning-tree vlan 9 detail
VLAN0009 is executing the rstp compatible Spanning Tree protocol

Bridge Identifier has priority 32768, sysid 9, address 0018.bad8.27ad
Configured hello time 2, max age 20, forward delay 15
Current root has priority 32777, address 0018.bad7.db15
Root port is 385 (Ethernet3/1), cost of root path is 4
Topology change flag not set, detected flag not set
Number of topology changes 8 last change occurred 1:32:11 ago
'' from Ethernet3/1''
Times: hold 1, topology change 35, notification 2
...
3. Repeat step 2 on devices connected to the interface until you can isolate the device that originated the topology change.
4. Check for link flaps on the interfaces on this device.
Troubleshooting Convergence Time Issues

STP convergence can take longer than expected or result in an unexpected final network topology.
To troubleshoot convergence issues, check the following issues:
• Errors in the documented network topology diagram.

• Misconfiguration-Check that the timers, diameter, Cisco extension features such as Bridge Assurance, Root Guard, BPDU
Guard, and so on are not misconfigured.
• Overloaded switch CPU during convergence that exceeds the recommended logical port (port-vlan) limit.
Note: The recommended scalability limits are system wide and not per VDC.
• Software defects that affect STP.
Securing the Network Against Forwarding Loops

To handle the inability of STP to deal correctly with certain failures, Cisco has developed a number of features and enhancements
to protect the networks against forwarding loops.
Troubleshooting STP helps to isolate and find the cause for a particular failure, while the implementation of these enhancements is
the only way to secure the network against forwarding loops.
To protect your network against forwarding loops, follow these steps:
1. Enable the Cisco-proprietary Unidirectional Link Detection (UDLD) protocol on all the switch-to-switch links. See the UDLD
Troubleshooting Convergence Time Issues 71

section in the Cisco NX-OS Interfaces Configuration Guide.
2. Set up the Bridge Assurance feature by configuring all the switch-to-switch links as the spanning tree network port type.
Note: You should enable the Bridge Assurance feature on both sides of the links or Cisco NX-OS will put the port in the
blocked state because of a Bridge Assurance inconsistency.
3. Set up all the end-station ports as a spanning-tree edge port type.
You must set up the STP edge port to limit the amount of topology change (TC) notices and subsequent flooding that can
affect the performance of the network. Use this command only with ports that connect to end stations. Otherwise, an
accidental topology loop can cause a data-packet loop and disrupt the device and network operation.
4. Enable the Link Aggregation Control Protocol (LACP) for port channels to avoid any port-channel misconfiguration issues. See
the LACP section in the Cisco NX-OS Interfaces Configuration Guide.
Do not disable autonegotiation on the switch-to-switch links. Autonegotiation mechanisms can convey remote fault
information, which is the quickest way to detect failures at the remote side. If failures are detected at the remote side, the
local side brings down the link even if the link is still receiving pulses.
Caution! Be careful when you change STP timers. STP timers are dependent on each other and changes can impact the
entire network.
5. (Optional) To prevent denial-of-service attacks, use the spanning-tree loopguard default command to secure the network STP
perimeter with Root Guard. Root Guard and BPDU Guard allow you to secure STP against influence from the outside.
6. Use the spanning-tree bpduguard enable command to enable BPDU Guard on STP edge ports to prevent STP from being
affected by unauthorized network devices (such as hubs, switches, and bridging routers) that are connected to the ports.
Root Guard prevents STP from outside influences. BPDU Guard shuts down the ports that are receiving any BPDUs (not
only superior BPDUs).
Note: Short-living loops are not prevented by Root Guard or BPDU Guard if two STP edge ports are connected directly
or through the hub.
7. Use the vlan command to configure separate VLANs and avoid user traffic on the management VLAN. The management
VLAN is contained to a building block, not the entire network.
8. Use the spanning-tree vlan vlan-range root primary command to configure a predictable STP root.
9. Use the spanning-tree vlan vlan-range root secondary command to configure a predictable backup STP root placement.
You must configure the STP root and backup STP root so that convergence occurs in a predictable way and builds optimal
topology in every scenario. Do not leave the STP priority at the default value.
See Also
Cisco NX-OS/IOS STP Comparison
Further Reading
Securing the Network Against Forwarding Loops 72

Cisco Nexus 7000 Series NX-OS Layer 2 Switching Configuration Guide
External Links
This article describes troubleshooting procedures for routing.
Guide Contents
Troubleshooting CFS
Troubleshooting STP
Troubleshooting Routing (this section)
Contents
• 1 Information about Troubleshooting
Routing Issues
• 3 Troubleshooting Routing
• 4 See Also
Further Reading 73
Information about Troubleshooting Routing Issues

Layer 3 routing involves determining optimal routing paths and packet switching. You can use routing algorithms to calculate the
optimal path from the router to a destination. This calculation depends on the algorithm selected, route metrics, and other
considerations such as load balancing and alternate path discovery.
Cisco NX-OS uses the virtual device contexts (VDCs) to provide separate management domains per VDC and software fault
isolation. Each VDC supports multiple Virtual Routing and Forwarding Instances (VRFs) and multiple routing information bases
(RIBs) to support multiple address domains. Each VRF is associated with a routing information base (RIB) and this information is
collected by the Forwarding Information Base (FIB).
See the Cisco NX-OS Unicast Routing Configuration Guide and the Cisco NX-OS Multicast Routing Configuration Guide for
more information on routing.

Begin troubleshooting routing issues by checking the following issues first:
Checklist Check off

Verify that the routing protocol is enabled.
Verify that the address family is configured if necessary.
Verify that you have configured the correct VRF for your routing protocol.
Use the following commands to display routing information:
• show ip arp
• show ip traffic
• show tcp statistics udp4
• show ip client
• show tcp client
• show ip fib
• show ip process
• show ip route
• show pktmgr interface
• show frame traffic
• show platform fib
• show platform forwarding
• show platform ip
• show vrf
• show vrf interface
Information about Troubleshooting Routing Issues 74

To troubleshoot basic routing issues, follow these steps:
1. Verify that the routing protocol is enabled.
switch(config)# show ospf

^
% invalid command detected at '^' marker.
If the feature is not enabled, Cisco NX-OS reports that the command is invalid. Use the feature command to enable the
routing protocol.
2. Verify the configuration for this routing protocol.
switch# show running-config eigrp all

version 4.0(1)
feature eigrp
router eigrp 99
log-neighbor-warnings
log-neighbor-changes
log-adjacency-changes
graceful-restart
nsf
timers nsf signal 20
distance 90 170
metric weights 0 1 0 1 0 0
metric maximum-hops 100
default-metric 100000 100 255 1 1500
maximum-paths 16
address-family ipv4 unicast
log-neighbor-warnings
log-neighbor-changes
log-adjacency-changes
graceful-restart
router-id 192.0.2.1
nsf
timers nsf signal 20
distance 90 170
metric weights 0 1 0 1 0 0
metric maximum-hops 100
default-metric 100000 100 255 1 1500
maximum-paths 16
3. Verify the VRF configuration for this routing protocol.
switch# show running-config eigrp

version 4.0(1)
feature eigrp
router eigrp 99
address-family ipv4 unicast
router-id 192.0.2.1
vrf red
stub
4. Check the memory utilization for this routing protocol.

switch# show processes memory | include isis
8913 9293824 bffff1d0/bffff0d0 isis
32243 8609792 bfffe0c0/bfffdfc0 isis
5. Verify that the routing protocol is receiving packets.
switch# show ip client pim

Client: pim, uuid: 284, pid: 3839, extended pid: 3839
Protocol: 103, client-index: 10, routing VRF id: 255
Data MTS-SAP: 1519
Data messages, send successful: 2135, failed: 0
6. Verify that the routing protocol is enabled on an interface.
switch# show ip interface loopback0

loopback0, Interface status: protocol-up/link-up/admin-up, iod: 36, Context:"default"
IP address: 1.0.0.1, IP subnet: 1.0.0.0/24
...
IP multicast groups locally joined:
224.0.0.2 224.0.0.1 224.0.0.13
...
7. Verify that the interface is in the correct VRF.
switch(config)# show vrf interface loopback 99

Interface VRF-Name VRF-ID
loopback99 default 1
8. Verify that the routing protocol is registered with the RIB.
switch(config)# show routing unicast clients

CLIENT: am
index mask: 0x00000002
epid: 3908 MTS SAP: 252 MRU cache hits/misses: 2/1
Routing Instances:
VRF: management table: base
Messages received:
Register : 1 Add-route : 2 Delete-route : 1
Messages sent:
Add-route-ack : 2 Delete-route-ack : 1
CLIENT: rpm
Messages received:
Register : 1
Messages sent:
...
CLIENT: eigrp-99
Routing Instances:
VRF: default table: base notifiers: self
Messages received:
Register : 1 Delete-all-routes : 1
Messages sent:
...
9. Verify that the RIB is interacting with the forwarding plane.
switch# show forwarding distribution multicast client

Number of Clients Registered: 3
Client-name Client-id Shared Memory Name
Troubleshooting Routing 76
igmp 1 N/A
mrib 2 /procket/shm/mrib-mfdm
m6rib 3 /procket/shm/m6rib-mfdm
See Also
Cisco NX-OS/IOS BGP (Basic) Comparison
Cisco NX-OS/IOS BGP (Advanced) Comparison
Cisco NX-OS/IOS EIGRP Comparison
Cisco NX-OS/IOS HSRP Comparison
Cisco NX-OS/IOS Layer-3 Virtualization Comparison
Cisco NX-OS/IOS OSPF Comparison
Further Reading
Cisco Nexus 7000 Series NX-OS Unicast Routing Configuration Guide
External Links
Packetlife Routing Cheatsheets
This article below provides only basic information on how to troubleshoot unicast packet flow traffic issues for the M1 Series
modules.
Troubleshooting L2/L3 unicast is covered in detail in Cisco-Live presentation. Sections of this presentation covers, both platform
independent, and platform specific step by step troubleshooting for unicast, among other things. Access to this presentation is
available FREE. Follow the below instructions to access the presentation
1. Visit https://www.ciscolivevirtual.com/
2. Register for free.
3. Click on "Cisco Live Virtual" link.
4. Click on the ?Sessions? Tab on top, and select ?2011 Sessions Catalog?
5. In the search box, type ?BRKCRS-3144? and Submit search.
See Also 77
6. Select the session. You can either View the Session (or) download the pdf.
7. Unicast troubleshooting is covered from slides 81 through 93.
Guide Contents
Troubleshooting Licensing'
Troubleshooting CFS
Troubleshooting STP
Troubleshooting Unicast Traffic(this section)
Contents
• 1 Packet is Received into Interface from Wire
• 2 Linksec Decryption Occurs, 1st stage Port QoS
• 3 Second Stage Port QoS Occurs
• 4 Layer 2 Source/Destination MAC Processing
• 5 Layer 3 Engine Processing
♦ 5.1 Layer 3 Engine Processes Layer 3 Features
♦ 5.2 Layer 3 forwarding for Routed Traffic
• 6 SFabric Processing Occurs (optional)
• 7 Layer 2 Engine Performs Source/Destination MAC
Processing
• 8 Egress Port QoS is Performed
• 9 Linksec Encryption Occurs
• 10 Packet is Transmitted
Packet is Received into Interface from Wire

During this step, the packet is received into the Nexus 7000 port. When troubleshooting this step, we want to look to ensure there
is transceiver interoperability, and validate whether we are seeing any errors on the interface. We do this via using the following
commands
• show interface interface

• show interface interface transceiver
PHX2-N7K-1# show interface e1/1
External Links 78
Ethernet1/1 is up
Hardware: 10000 Ethernet, address: 0024.986c.00b0 (bia 0024.986c.00b0)
Description: N7K-vdc-1 connecting to core 6506
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10g
Beacon is turned off
Auto-Negotiation is turned off
Input flow-control is off, output flow-control is off
Rate mode is shared
Switchport monitor is off
Last link flapped 7week(s) 4day(s)
Last clearing of "show interface" counters never
1 minute input rate 13056 bits/sec, 9 packets/sec
1 minute output rate 4608 bits/sec, 0 packets/sec
Rx
341190251 input packets 276211313 unicast packets 52112947 multicast packets
12865991 broadcast packets 0 jumbo packets 0 storm suppression packets
94295027129 bytes
Tx
462437316 output packets 85121 multicast packets
188251 broadcast packets 0 jumbo packets
648159081064 bytes
0 input error 0 short frame 0 watchdog
0 no buffer 0 runt 0 CRC 0 ecc
0 overrun 0 underrun 0 ignored 0 bad etype drop
0 bad proto drop 0 if down drop 0 input with dribble
0 input discard
0 output error 0 collision 0 deferred
0 late collision 0 lost carrier 0 no carrier
0 babble
0 Rx pause 0 Tx pause
1 interface resets
PHX2-N7K-1# show interface e1/1 transceiver details
Ethernet1/1
sfp is present
name is CISCO-AVAGO <<< If this says type is (unknown), it is not supported.
part number is SFBR-7700SDZ
revision is B4
serial number is AGD12434116
nominal bitrate is 10300 MBits/sec
Link length supported for 50/125um fiber is 82 m(s)
Link length supported for 62.5/125um fiber is 26 m(s)
cisco id is --
cisco extended id number is 4
SFP Detail Diagnostics Information (internal calibration)

----------------------------------------------------------------------------
Alarms Warnings
High Low High Low
----------------------------------------------------------------------------
Temperature 45.46 C 75.00 C -5.00 C 70.00 C 0.00 C
Voltage 3.28 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 6.92 mA 10.50 mA 2.50 mA 10.50 mA 2.50 mA
Tx Power -2.75 dBm 1.69 dBm -11.30 dBm -1.30 dBm -7.30 dBm
Rx Power -2.43 dBm 1.99 dBm -13.97 dBm -1.00 dBm -9.91 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Packet is Received into Interface from Wire 79

Linksec Decryption Occurs, 1st stage Port QoS

In step 2, Linksec decryption occurs as well as receive side stage 1 QoS.
It is important to step back and evaluate the difference between stage 1 and stage 2 QoS. The difference is that some ports can be
configured in shared mode, whereas some can be configured in dedicated mode, on the 10G modules. What this means, is that
there is 10g of bandwidth that can be dedicated to a port or shared amongst ports (4 ports, on the m132 module).
When running in shared mode, there exists a chance for contention accessing the 10g bandwidth through the 4:1 Mux. To alleviate
this, some QoS intelligence was passed down to the 4:1 Mux which aggregates the ports.
In dedicated mode, there is no QoS applied at the Mux, instead, all traffic is processed in phase 2 QoS. To summarize, in shared
mode, 1st stage QoS ensures fair access to the 10g of port bandwidth. In both shared and dedicated mode, 2nd stage QoS occurs to
provide ingress queuing to the system.
For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.
Use the show policy-map command to see per queue dropped packets.
The commands to troubleshoot Linksec and Port QoS are as follows:
• show cts interface [all | interface]

• show queuing interface interface
• show policy-map interface (for per queue drop)
switch# show cts interface all Working Example
CTS Information for Interface Ethernet2/24:

CTS is enabled, mode: CTS_MODE_DOT1X
IFC state: CTS_IFC_ST_CTS_OPEN_STATE
Authentication Status: CTS_AUTHC_SUCCESS
Peer Identity: india1
Peer is: CTS Capable
802.1X role: CTS_ROLE_AUTH
Last Re-Authentication:
Authorization Status: CTS_AUTHZ_SUCCESS
PEER SGT: 2
Peer SGT assignment: Trusted
Global policy fallback access list:
SAP Status: CTS_SAP_SUCCESS
Configured pairwise ciphers: GCM_ENCRYPT
Replay protection: Enabled
Replay protection mode: Strict
Selected cipher: GCM_ENCRYPT
Current receive SPI: sci:1b54c1fbff0000 an:0
Current transmit SPI: sci:1b54c1fc000000 an:0
PHX2-N7K-1# show cts interface eth 1/8 Broken Example
CTS Information for Interface Ethernet1/8:

CTS is enabled, mode: CTS_MODE_MANUAL
IFC state: Unknown
Authentication Status: CTS_AUTHC_INIT
Peer Identity:
Peer is: Not CTS Capable
802.1X role: CTS_ROLE_UNKNOWN
Last Re-Authentication:
Authorization Status: CTS_AUTHZ_INIT
PEER SGT: 0
Peer SGT assignment: Not Trusted
SAP Status: CTS_SAP_INIT
Configured pairwise ciphers:
Replay protection:
Replay protection mode:
Selected cipher:
Current receive SPI:
Current transmit SPI:
PHX2-N7K-1# show queuing int eth 1/1
Interface Ethernet1/1 TX Queuing strategy: Weighted Round-Robin

Port QoS is enabled
Queuing Mode in TX direction: mode-cos
Transmit queues [type = 1p7q4t]
Queue Id Scheduling Num of thresholds
_____________________________________________________________
1p7q4t-out-q-default WRR 04
1p7q4t-out-q2 WRR 04
1p7q4t-out-pq1 Priority 04
Configured WRR
WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 12[1p7q4t-out-q3]
12[1p7q4t-out-q4] 12[1p7q4t-out-q5] 12[1p7q4t-out-q6] 12[1p7q4t-out-q7]
WRR configuration read from HW
WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 11[1p7q4t-out-q3]
11[1p7q4t-out-q4] 11[1p7q4t-out-q5] 11[1p7q4t-out-q6] 11[1p7q4t-out-q7]
Configured queue-limit ratios

queue-limit ratios: 78[1p7q4t-out-q-default] 1[1p7q4t-out-q2] 1[1p7q4t-out-q3]
*1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1]
* means unused queue with mandatory minimum queue-limit
queue-limit ratios configuration read from HW

*1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1]
Thresholds:
COS Queue Threshold Type Min Max
__________________________________________________________________
0 1p7q4t-out-q-default DT 100 100
5 1p7q4t-out-pq1 DT 100 100
6 1p7q4t-out-pq1 DT 100 100
7 1p7q4t-out-pq1 DT 100 100
Interface Ethernet1/1 RX Queuing strategy: Weighted Round-Robin

Queuing Mode in RX direction: mode-cos
Receive queues [type = 8q2t]
Port Cos not configured
Linksec Decryption Occurs, 1st stage Port QoS 81

____________________________________________________________
8q2t-in-q-default WRR 02
8q2t-in-q2 WRR 02
8q2t-in-q3 WRR 02
8q2t-in-q4 WRR 02
8q2t-in-q5 WRR 02
8q2t-in-q6 WRR 02
8q2t-in-q7 WRR 02
8q2t-in-q1 WRR 02
Configured WRR
WRR bandwidth ratios: 20[8q2t-in-q-default] 0[8q2t-in-q2] 0[8q2t-in-q3]
0[8q2tin-q4] 0[8q2t-in-q5] 0[8q2t-in-q6] 0[8q2t-in-q7] 80[8q2t-in-q1]
WRR bandwidth ratios: 20[8q2t-in-q-default] 0[8q2t-in-q2] 0[8q2t-in-q3]
0[8q2t-in-q4] 0[8q2t-in-q5] 0[8q2t-in-q6] 0[8q2t-in-q7] 80[8q2t-in-q1]
No queue-limit ratios user configuration

________________________________________

queue-limit ratios: 100[8q2t-in-q-default] 100[8q2t-in-q2] 100[8q2t-in-q3]
100[8q2t-in-q4] 100[8q2t-in-q5] 100[8q2t-in-q6] 100[8q2t-in-q7] 100[8q2t-in-q1]
Thresholds:
__________________________________________________________________
0 8q2t-in-q-default DT 100 100
5 8q2t-in-q1 DT 100 100
6 8q2t-in-q1 DT 100 100
7 8q2t-in-q1 DT 100 100
PHX2-N7K-1# show policy-map interface eth 1/2
Global statistics status : enabled

Ethernet1/2
Service-policy (queuing) input: default-in-policy
policy statistics status: enabled
Class-map (queuing): in-q1 (match-any)

queue-limit percent 50
bandwidth percent 80
queue dropped pkts : 0
Class-map (queuing): in-q-default (match-any)

Service-policy (queuing) output: default-out-policy

Class-map (queuing): out-pq1 (match-any)

priority level 1
Linksec Decryption Occurs, 1st stage Port QoS 82

Class-map (queuing): out-q2 (match-any)

Class-map (queuing): out-q-default (match-any)

bandwidth remaining percent 25
Second Stage Port QoS Occurs

For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.
Use the show policy-map command to view queue drops .
The commands to troubleshoot Port QoS are

• show policy-map interface
Layer 2 Source/Destination MAC Processing

In this step, the ASIC submits the packet headers to theLayer 2 engine for lookup, and the Layer 2 engine performs
source/destination MAC processing.
To validate forwarding of the Layer 2 engine, we should first look at the centralized mac table aggregated on the supervisor to
validate whether the mac addresses are correlated as we expect them, and assigned to the ports where we expect the Mac?s to
reside.
Based off of this, we can then validate the hardware programming on the ingress linecard to validate that our mac address table is
properly programmed into the hardware based Layer 2 engine on the linecard.
We first will look at the mac address table, then we can ensure programming is properly occurring in the hardware table.
The commands used to accomplish this are as follows:
• show mac address-table

• show hardware mac address-table module interface interface
To drill down on a specific MAC address, we can use the grep function with these commands to validate the mac is associated
with a particular port, and that the hardware programming reflects that.
• show mac address-table | grep macaddress

• show hardware mac address-table module interface interface | grep macaddress
Note: When evaluating the Hardware mac table, if the Index is set to 0x00400, or the GM bit is set to ?1?, that traffic will be
routed. For example, you will see the index set to 0x00400 and GM bit set to 1 for traffic destined to the mac address
local to the device
PHX2-N7K-1# show mac address-table
Legend:
Second Stage Port QoS Occurs 83

* - primary entry, G - Gateway MAC, (R) - Routed MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+------+----------------
G - 0023.ac67.dd41 static - False False sup-eth1(R)
G 5 0023.ac67.dd41 static - False False sup-eth1(R)
* 5 0000.0c07.ac01 dynamic 0 False False Eth1/1
* 5 000c.2943.a67e dynamic 180 False False Eth1/1
* 5 000c.294b.c5ca dynamic 0 False False Eth1/1
* 5 000d.ece2.0640 dynamic 180 False False Eth1/1
* 5 0013.5f32.aa80 dynamic 0 False False Eth1/1
* 5 0018.8b45.41b7 dynamic 0 False False Eth1/1
* 5 0019.bb2f.4871 dynamic 0 False False Eth1/1
* 5 0019.bbe5.f3b8 dynamic 1230 False False Eth1/1
* 5 001a.4b33.ccdc dynamic 1080 False False Eth1/1
* 5 001a.4ba8.6a9c dynamic 1680 False False Eth1/1
* 5 001b.210a.87f9 dynamic 600 False False Eth1/1
* 5 001b.d46f.70e0 dynamic 60 False False Eth1/1
* 5 001c.c4e5.ac9a dynamic 150 False False Eth1/1
* 5 0023.ac64.6f7c dynamic 1230 False False Eth1/1
* 5 0024.986d.21c8 dynamic 270 False False Eth1/1
PHX2-N7K-1# show hardware mac address-table 1 int eth1/1
Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr | GM | Sec|

TR | NT | RM | RMA | Cap | Fld | Always | | | |
| ic | | fied| Byte| Sel | | ure| AP | FY | | | TURE| | Learn
-----+----+-------+---------------+--------+-----+----+-----+-----+-----+----+--
--+----+----+----+-----+-----+-----+--------
1 1 2 000c.294b.c5ca 0x00422 0 3 0 67 1 0
0 0 0 0 0 0 0 0
1 1 2 0050.567e.58e6 0x00422 0 3 0 68 1 0
0 0 0 0 0 0 0 0
1 1 2 0050.56aa.6067 0x00422 0 3 0 67 1 0
0 0 0 0 0 0 0 0
1 1 2 00c0.b72e.cfa0 0x00422 0 3 0 67 1 0
0 0 0 0 0 0 0 0
1 1 2 0018.8b45.41b7 0x00422 0 3 0 68 1 0
0 0 0 0 0 0 0 0
1 1 2 0013.5f32.aa80 0x00422 0 3 0 68 1 0
0 0 0 0 0 0 0 0
1 1 2 0050.56aa.75ca 0x00422 0 3 0 64 1 0
0 0 0 0 0 0 0 0
1 1 2 00a0.9811.a233 0x00422 0 3 0 39 1 0
0 0 0 0 0 0 0 0
PHX2-N7K-1# show mac address-table | grep 000c.294b.c5ca
* 5 000c.294b.c5ca dynamic 150 False False Eth1/1
PHX2-N7K-1# show hardware mac address-table 1 int eth 1/1 | grep 000c.294b.c5ca
1 1 2 000c.294b.c5ca 0x00422 0 3 0 67 1 0
0 0 0 0 0 0 0 0
Layer 2 Source/Destination MAC Processing 84

Layer 3 Engine Processing

After the Layer 2 engine is finished, it sends the header to the Layer 3 engine. The layer 3 engine applies layer 3 intelligent
features, to all packets, and layer 3 forwarding, to routed packets. As such, this section will divide the troubleshooting into two
components, the layer 3 features applied to all packets, and the layer 3 forwarding.
The layer 3 features which are applied to all packets include the below features :
1) ACL 2) QoS 3) Netflow 4) Hardware IPS
Following the evaluation of the features, we will evaluate the layer 3 forwarding troubleshooting.
Layer 3 Engine Processes Layer 3 Features

We?ll drill through each of the layer 3 features below, looking at one feature at a time.
The first feature we will look at is ACL. To troubleshoot ACL, we want to evaluate the configuration, and any relevant hit
counters. We then can identify if the hardware on the linecard is programming the ACL.
It is important to note, that if you wish to see per ACL counters, you must enable ?statistics per-entry? in the ACL.
The commands to troubleshoot ACL are as follows:
• show access-lists name

• show hardware access-lists module module
• show hardware access-lists resource-utilization module module
PHX2-N7K-1# show access-lists sample-86
IP access list sample-86

statistics per-entry
10 permit ospf any any [match=0]
20 permit pim any any [match=0]
30 permit tcp any any [match=0]
40 permit ip any any [match=0]
PHX2-N7K-1# show hardware access-list mod 1
VLAN 86 :
=========
No ingress policies
No Netflow profiles in ingress direction
Policies in egress direction:

Policy type Policy Id Policy name
------------------------------------------------------------
RACL 4 sample-86
No Netflow profiles in egress direction
Tcam 1 resource usage:

----------------------
Label_b = 0x800
Bank 0
------
IPv4 Class
Policies: RACL(sample-86) [Merged]
Layer 3 Engine Processing 85

1 tcam entries
1 l4 protocol cam entries

0 mac etype/proto cam entries
0 lous
0 tcp flags table entries
0 adjacency entries
VDC-1 Ethernet1/1 :
====================
Policies in ingress direction:
------------------------------------------------------------
QoS 1
Policies in egress direction:

------------------------------------------------------------
QoS 2
...
VDC-1 CoPP :
====================
Policies in ingress direction:
------------------------------------------------------------
QoS 3
Tcam 1 resource usage:

----------------------
Label_b = 0x1
Bank 1
------
IPv4 Class
Policies: QoS()
100 tcam entries
IPv6 Class
Policies: QoS()
73 tcam entries
3 l4 protocol cam entries

0 mac etype/proto cam entries
2 lous
0 tcp flags table entries
0 adjacency entries
No egress policies
PHX2-N7K-1# show hardware access-list resource utilization mod 1
ACL Hardware Resource Utilization (Module 1)

--------------------------------------------
Used Free Percent
Utilization
Layer 3 Engine Processes Layer 3 Features 86

-----------------------------------------------------
Tcam 0, Bank 0 1 16383 0.00
Tcam 0, Bank 1 2 16382 0.01
Tcam 1, Bank 0 6 16378 0.03
Tcam 1, Bank 1 176 16208 1.07
The next feature we will look at is the QoS troubleshooting for the Nexus. Note, we will have QoS applied, potentially, on both
ingress and egress. So we should interrogate both the ingress and egress QoS.
The commands to troubleshoot QoS are
• show policy-map interface interface

Ethernet1/2



Service-policy (qos) output: test-police-86

Class-map (qos): test-police-86 (match-all)

Match: dscp 18
police cir 100 mbps bc 200 ms


priority level 1



Netflow processing also has portions which occur in hardware. For netflow, we collect statistics in hardware on the linecards. We
then, can export them via software.
The commands to troubleshoot Netflow are
• show flow interface

• show flow record
• show flow monitor
• show hardware flow ip module module
PHX2-N7K-1# show flow interface vlan 86
Interface Vlan86:
Monitor: sample-86
Direction: Input
Monitor: sample-86
Direction: Output
PHX2-N7K-1# show flow record
Flow record netflow-original:

Description: Traditional IPv4 input NetFlow with origin ASs
No. of users: 2
Template ID: 256
Fields:
match ipv4 source address
match ipv4 destination address
match ip protocol
match ip tos
match transport source-port
match transport destination-port
match interface input
match interface output
match flow direction
collect routing source as
collect routing destination as
collect routing next-hop address ipv4
collect transport tcp flags
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last
PHX2-N7K-1# show flow monitor
Flow Monitor Solarwinds1:

Use count: 0
Flow Record: netflow-original
Flow Exporter: Solarwinds
Flow Monitor sample-86:
Use count: 2
Flow Record: netflow-original
switch(config)# show hardware flow ip module 8
D - Direction; L4 Info - Protocol:Source Port:Destination Port

IF - Interface: ()ethernet, (S)vi, (V)lan, (P)ortchannel, (T)unnel
TCP Flags: Ack, Flush, Push, Reset, Syn, Urgent
D IF SrcAddr DstAddr L4 Info PktCnt TCP Flags

-+-----+---------------+---------------+---------------+----------+-----------
I 8/26 007.002.000.002 007.001.000.002 000:00000:00000 0000421885 . . . . . .
I 8/25 007.001.000.002 007.002.000.002 000:00000:00000 0000421900 . . . . . .
O 8/25 007.002.000.002 007.001.000.002 000:00000:00000 0000422213 . . . . . .
O 8/26 007.001.000.002 007.002.000.002 000:00000:00000 0000422228 . . . . . .
Cisco NX-OS supports a hardware based intrusion detection system that checks for ip packet verification. These checks handle
well known, and unusable traffic types which can be witnessed during malicious activity, such as if the source is a broadcast
address, or if the destination is the 0.0.0.0 address. You can validate if any of these checks are dropping packets.
Note: It has been shown in the field, that frequently it is advantageous to disable IP fragment verification. This is done via the
below command
• no hardware ip verify fragment

To validate if you are seeing any drops because of ip packet verification, you can use the below command.
• show hardware forwarding ip verify

• show hardware rate-limit module module
PHX2-N7K-1# show hardware forwarding ip verify
IPv4 and v6 IDS Checks Status Packets Failed

-----------------------------+---------+------------------
address source broadcast Enabled 0
address source multicast Enabled 0
address destination zero Enabled 0
address identical Enabled 0
address source reserved Enabled 8
address class-e Disabled --
checksum Enabled 0
protocol Enabled 0
fragment Enabled 0
length minimum Enabled 0
length consistent Enabled 0
length maximum max-frag Enabled 0
length maximum udp Disabled --
length maximum max-tcp Enabled 0
tcp flags Disabled --
tcp tiny-frag Enabled 0
version Enabled 0
-----------------------------+---------+------------------
IPv6 IDS Checks Status Packets Failed
-----------------------------+---------+------------------
length consistent Enabled 0
length maximum max-frag Enabled 0
length maximum udp Disabled --
length maximum max-tcp Enabled 0
tcp tiny-frag Enabled 0
version Enabled 0
PHX2-N7K-1# show hardware rate-limit module 1
Units for Config: packets per second

Allowed, Dropped & Total: aggregated since last clear counters
Rate Limiter Class Parameters

------------------------------------------------------------
layer-3 mtu Config : 500
Allowed : 0
Dropped : 0
Total : 0
layer-3 ttl Config : 500
Allowed : 0
Dropped : 0
Total : 0
layer-3 control Config : 10000

Allowed : 0
Dropped : 0
Total : 0
layer-3 glean Config : 100

Allowed : 0
Dropped : 0
Total : 0
layer-3 multicast directly-connected Config : 3000

Allowed : 0
Dropped : 0
Total : 0
layer-3 multicast local-groups Config : 3000

Allowed : 0
Dropped : 0
Total : 0
layer-3 multicast rpf-leak Config : 500

Allowed : 0
Dropped : 0
Total : 0
layer-2 storm-control Config : Disabled
access-list-log Config : 100

Allowed : 0
Dropped : 0
Total : 0
copy Config : 30000

Allowed : 0
Dropped : 0
Total : 0
receive Config : 30000

Allowed : 20450903
Dropped : 0
Total : 20450903
layer-2 port-security Config : Disabled
layer-2 mcast-snooping Config : 10000

Allowed : 906
Dropped : 0
Total : 906
Layer 3 forwarding for Routed Traffic

The Layer 3 engine will only perform Layer 3 forwarding for traffic that is routed through the router. This is traffic which has
been sent to the MAC address of a valid routed interface, local to the router.
To troubleshoot the routed traffic, we need to perform the following tasks: 1. Ensure that the control plane routing is correct. 2.
Ensure that the hardware forwarding entries on the ingress module have the corresponding information.
Note: All routing of traffic is performed on the forwarding engine of the ingress module.
For our example, we will troubleshoot a route 86.86.87.0/24, which is set to a next hop of 86.86.86.1, and set to route out of
VLAN 86 (an SVI).
We first will look at the route, ensure it is set to the correct next hop (86.86.86.1), and set to route out of VLAN 86. We will then
want to ensure that we have a corresponding ARP entry associated with this next hop, and validate that the adjacency is in the
adjacency table.
As we can see below, 86.86.87.0/24 is set to route to 86.86.86.1, out VLAN 86. This next hop is associated with MAC address
0011.aabb.ccdd. We will use this information to investigate the hardware, next.
• show ip route (prefix)

• show ip arp (nexthop)
• show ip adjacency
PHX2-N7K-1# show ip route 86.86.87.0/24
IP Route Table for VRF "default"

86.86.87.0/24, 1 ucast next-hops, 0 mcast next-hops
*via 86.86.86.1, Vlan86, [1/0], 01:19:24, static
PHX2-N7K-1# show ip arp 86.86.86.1
IP ARP Table
Total number of entries: 1
Address Age MAC Address Interface
86.86.86.1 - 0011.aabb.ccdd Vlan86
PHX2-N7K-1# show ip adjacency
IP Adjacency Table for VRF default

Total number of entries: 1
Address Age MAC Address Pref Source Interface
86.86.86.1 00:00:37 0011.aabb.ccdd 1 Static Vlan86
The above example shows the control plane. Now that we know how things are supposed to work, we can interrogate the
hardware to ensure the hardware entries have propagated properly to the Layer 3 hardware engine. We can see that the IP FIB has
properly associated 86.86.87.0/24 to the next hop of 86.86.86.1. We can also see, in the hardware entry, that this is routed out
VLAN 86, that the RPF is valid if we have enabled RPF Checking), and that the route entry is correctly associated with the MAC
address of 0011.aabb.ccdd.
This demonstrates that the routing in the forwarding plane is programmed correctly and that the forwarding will follow the
information contained in the routing protocols.
• show ip fib route prefix module module

• show system internal forwarding route prefix detail module module
PHX2-N7K-1# sho ip fib route 86.86.87.0/24 mod 1
IPv4 routes for table default/base
------------------+------------------+---------------------
Prefix | Next-hop | Interface
------------------+------------------+---------------------
86.86.87.0/24 86.86.86.1 Vlan86
PHX2-N7K-1# show system internal forwarding route 86.86.87.1/24 detail mod 1
RPF Flags legend:

S - Directly attached route (S_Star)
V - RPF valid
M - SMAC IP check enabled
G - SGT valid
E - RPF External table valid
86.86.87.0/24 , Vlan86
Dev: 1 , Idx: 0x19001 , RPF Flags: V , DGT: 0 , VPN: 1
RPF_Intf_5: Vlan86 (0x55 )
AdjIdx: 0x43005, LIFB: 0 , LIF: Vlan86 (0x55 ), DI: 0x0
DMAC: 0011.aabb.ccdd SMAC: 0023.ac67.dd41
SFabric Processing Occurs (optional)

This step occurs if the packet needs to traverse the fabric.
In this step, we need to interrogate if the fabrics are functioning properly, and if their utilization is at an acceptable level. We can
view the fabric status and utilization using the following commands:
• show hardware fabric-utilization

• show module fabric
switch(config)# show hardware fabric-utilization
-----------------------------
Slot Direction Utilization
-----------------------------
2 ingress 3%
2 egress 3%
6 ingress 1%
6 egress 1%
PHX2-N7K-1# show module fabric
Xbar Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------
1 0 Fabric Module 1 N7K-C7010-FAB-1 ok
Xbar Sw Hw
--- -------------- ------
1 NA 1.0
2 NA 1.0
3 NA 1.0
Xbar MAC-Address(es) Serial-Num

--- -------------------------------------- ----------
1 NA JAF1252AHRB
2 NA JAF1251CABF
3 NA JAF1252AHBL
SFabric Processing Occurs (optional) 92

* this terminal session
Layer 2 Engine Performs Source/Destination MAC Processing

To validate forwarding of the Layer 2 engine, we should first look at the centralized MAC table aggregated on the supervisor to
validate whether the MAC addresses are correlated as we expect them, and assigned to the ports where we expect the MAC?s to
reside.
Based on this, we can then validate the hardware programming on the egress module to validate that our MAC address table is
properly programmed into the hardware based Layer 2 engine on the module.
• show mac address-table

• show hardware mac address-table module interface interface
The output from these commands are documented in steps 3-4 above.
Egress Port QoS is Performed

As the packet nears entry from the Cisco Nexus 7000, one of the final steps is the application of Port QoS. The port level QoS will
be able to buffer the traffic in times of congestion. To interrogate the egress QoS, we look at following QoS commands, paying
attention to the Transmit QoS:

• show policy-map interface
PHX2-N7K-1# show queuing int eth 1/1
Interface Ethernet1/1 TX Queuing strategy: Weighted Round-Robin

Port QoS is enabled
Queuing Mode in TX direction: mode-cos
Transmit queues [type = 1p7q4t]
_____________________________________________________________
1p7q4t-out-q-default WRR 04
1p7q4t-out-pq1 Priority 04
Configured WRR
WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 12[1p7q4t-out-q3] 12[1p7q4t-out-q4] 12[1p7
WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 11[1p7q4t-out-q3] 11[1p7q4t-out-q4] 11[1p7
Configured queue-limit ratios

*1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7]
16[1p7q4t-out-pq1]

*1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7]
16[1p7q4t-out-pq1]
Thresholds:
__________________________________________________________________
5 1p7q4t-out-pq1 DT 100 100
6 1p7q4t-out-pq1 DT 100 100
7 1p7q4t-out-pq1 DT 100 100
...

Ethernet1/2




priority level 1



Linksec Encryption Occurs

The command used to troubleshoot Linksec encryption is
Egress Port QoS is Performed 94

• show cts interface {all | interface}
The output from this command is documented in step 2 above.
Packet is Transmitted
The final step in the process is the transmission of the frame out of the physical egress port. Troubleshooting of the physical port,
is the same as in step 1, and includes the following commands:
• show interface interface

• show interface interface transceiver
The output from these commands are documented in step 1 above.
This article describes how to troubleshoot Web Cache Communication Protocol version 2 (WCCPv2) on Cisco NX-OS.
Guide Contents
Troubleshooting CFS
Troubleshooting STP
Troubleshooting WCCP (this section)
Contents
• 1 Information About Troubleshooting
WCCP
• 2 Problem Scenarios
♦ 2.1 Reasons For Service Group
Startup Failure
♦ 2.2 Client Loss
♦ 2.3 Packet Redirect Counts Not
Incrementing
♦ 2.4 Potential problems
Linksec Encryption Occurs 95

Information About Troubleshooting WCCP

The Web Cache Communication Protocol (WCCP) is a content-routing protocol that enables a Cisco NX-OS router to
transparently redirect packets to cache engines. It has built-in load balancing, scaling, fault tolerance, and service-assurance
(failsafe) mechanisms. WCCP version 2 (WCCPv2) is the only version supported on Cisco NX-OS devices.
See the Configuring WCCPv2 chapter in the Cisco Nexus 7000 Series NX-OS Unicast Routing Configuration Guide for more
information on WCCPv2.
Problem Scenarios
Reasons For Service Group Startup Failure
• WCCP Client fails to see ISU messages and is stuck in the NOT Usable state.
♦ Confirm by enabling the WCCP event debugging messages and looking for bad Receive ID messages.
♦ The reason for the failure would in general be a connectivity problem and may be mismatched speed or duplex
settings.
• The WCCP Client is requesting a capability which is not supported by the router probably because of platform limitations.
♦ This can be confirmed by enabling WCCP event debugging and looking for Capability Mismatch messages.
• The WCCP Client is requesting a capability which is not supported by the service group.
♦ This will occur if a service group is already formed and the WCCP Client configuration does not match the
existing service.
♦ This is a misconfiguration of the WCCP Client and can be confirmed by enabling WCCP event debugging and
looking for Capability Mismatch messages.
• HIA event messages may indicate other reasons why the router rejected an incoming "Here I Am" message.
• Some WCCP clients don't adhere to the configured forward/return methods and prefer to always default to "GRE"
forward/return. The Cisco Nexus 7000 requires L2 forward/return methods. ACNS, WAAS, etc. might need to be
configured with the assign-method-strict option. This type of failure can be seen with packet traces. The client does not
respond to the Cisco Nexus 7000 with a sent RXID but will keep sending HIA with a receive ID of 0.
Client Loss
A WCCP Client is removed form a service group when the router loses contact with the WCCP Client.
The reasons why this might occur include:

• The service group has been disabled on the WCCP Client.

♦ Check the WCCP Client configuration.
• The service definition has been changed on the WCCP Client
♦ Check the WCCP Client configuration.
• Loss of physical connectivity to the WCCP Client.
♦ Verify connectivity using the ping command.
• Message loss between the router and WCCP Client perhaps because of a heavily utilized link or a flapping interface.
♦ Enable WCCP packet debugging and confirm message exchange.
• WCCP Client loss from a service group is indicated by the console message:
%WCCP-1-SERVICELOST: Service Service ID lost on WCCP client IP address
Packet Redirect Counts Not Incrementing

• The WCCP service not present on the interface.
♦ Verify using the show running-config interface ifname command.
• The WCCP service is not active.
♦ Check the service state using the show ip wccp web-cache | service-number detail command.
♦ Verify using the show running-config wccp command.
• The WCCP service definition does not match the traffic to be redirected.
♦ Check the service definition using the show ip wccp service-number service command.
♦ WCCP service mismatch is indicated by the console message:
%WCCP-1-SERVICEMISMATCH: Service Service ID mismatched on WCCP client IP
address
• Matching traffic is excluded from redirection by the redirect ACL.
♦ Monitor the redirect ACL count using the show ip wccp service-number command.
• No traffic matching the WCCP service is traversing the interface.
♦ Define an extended IP access list to get an independent count of matching traffic.
• All traffic redirection is done by the platform hardware.
Potential problems
• Direct communication between WCCP client and host
♦ The Cisco Nexus 7000 requires that the host running the browser, the WCCP clients, be attached to different L3
interfaces (the hosts cannot be present in the same subnet).
• Service definition mismatch
♦ There is no mechanism on a Cisco WCCP Client to mark two services as complementary. This also appears to be
true for third party vendors. This has the consequence that the two services can drift apart over time. On service
startup there is usually no problem however, as WCCP clients leave and rejoin either of the services the
assignments change independently meaning that an outgoing connection and the corresponding incoming
connection may not be redirected to the same WCCP Client. If that happens the configuration is broken.
♦ Check for this condition by comparing the assignments shown with the show ip wccp [web-cache | service
number] detail command. The only way currently to correct the condition is to restart both services.
• Asymmetric routing
♦ As long as the incoming connection returns to any router in the complementary service group that will happen
automatically. Note that the connection does not have to go to the exact same router as the outgoing connection,
just the same service group. In any given network the routes to a particular destination may be numerous which
raises the possibility that traffic returning from an origin server may take a different route to the outgoing traffic
and fail to hit routers in the complementary service group. In that case the connection will not be redirected and
the configuration will be broken. There is no way round this other than to ensure that there is no asymmetric
routing taking place.
Client Loss 97
This article describes how to troubleshoot memory issues that may occur when configuring and using Cisco NX-OS.
Guide Contents
Troubleshooting CFS
Troubleshooting STP
Troubleshooting Memory (this section)
Contents
• 1 Overview
• 2 General/High Level Assessment of Platform Memory
Utilization
• 3 Detailed Assessment of Platform Memory Utilization
♦ 3.1 Page Cache
♦ 3.2 Kernel
♦ 3.3 User Processes
◊ 3.3.1 Figuring Out Which Process is Using
a Lot of Memory
◊ 3.3.2 Figuring Out How a Specific Process
is Using Memory
• 4 Built-in Platform Memory Monitoring
♦ 4.1 Memory Thresholds
♦ 4.2 Memory Alerts
Overview
Dynamic random access memory (DRAM) is a limited resource on all platforms and must be controlled/monitored to ensure
utilization is kept in check.
Cisco NX-OS uses memory in the following three ways:
Page cache
When you access files from persistent storage (CompactFlash), the kernel reads the data into the page cache, which means
that when you access the data in the future, you can avoid the slow access times that are associated with disk storage.
Cached pages can be released by the kernel if the memory is needed by other processes.
Some file systems (tmpfs) exist purely in the page cache (for example, /dev/sh, /var/sysmgr, /var/tmp), which means that
there is no persistent storage of this data and that when the data is removed from the page cache, it cannot be recovered.
tmpfs-cached files release page-cached pages only when they are deleted.
Overview 98
Kernel
The kernel needs memory to store its own text, data, and Kernel Loadable Modules (KLMs). KLMs are pieces of code
that are loaded into the kernel (as opposed to being a separate user process). An example of kernel memory usage is when
an inband port driver allocates memory to receive packets.
User processes
This memory is used by Cisco NX-OS/Linux processes that are not integrated in the kernel (such as text, stack, heap, and
so on).
When you are troubleshooting high memory utilization, you must first determine what type of utilization is high (process, page
cache, or kernel). Once you have identified the type of utilization, you can use additional troubleshooting commands to help you
figure out which component is causing this behavior.
General/High Level Assessment of Platform Memory

Utilization
You can assess the overall level of memory utilization on the platform by using two basic CLI commands: show system
resources and show processes memory.
Note: From these command outputs, you might be able to tell that platform utilization is higher than normal/expected, but you
will not be able to tell what type of memory usage is high.
The show system resources command displays platform memory statistics (not per VDC).
N7K# show system resources
Load average: 1 minute: 0.43 5 minutes: 0.30 15 minutes: 0.28
Processes : 884 total, 1 running
CPU states : 2.0% user, 1.5% kernel, 96.5% idle
Memory usage: 4135780K total, 3423272K used, 712508K free
0K buffers, 1739356K cache
Note: This output is derived from the Linux memory statistics in /proc/meminfo.
total - The amount of physical RAM on the platform
free - The amount of unused or available memory
used - The amount of allocated (permanent) and cached (temporary) memory
The cache and buffers are not relevant to customer monitoring.

This information provides a general representation of the platform utilization only. You need more information to troubleshoot
why memory utilization is high.
The show process memory command displays the memory allocation per process for the current VDC (the output will
contain non-VDC global processes also).
N7K# show processes memory
PID MemAlloc MemLimit MemUsed StackBase/Ptr Process
----- -------- ---------- ---------- ----------------- ----------------
4662 52756480 562929945 150167552 bfffdf00/bfffd970 netstack
While this output is more detailed, it is only useful for verifying process-level memory allocation within a specific VDC.
Detailed Assessment of Platform Memory Utilization

Use the show system internal kernel command or the show system internal memory-alerts-log command for a more detailed
representation of memory utilization in Cisco NX-OS.
N7K# show system internal kernel meminfo

MemTotal: 4135780 kB
MemFree: 578032 kB
Buffers: 5312 kB
Cached: 1926296 kB
RAMCached: 1803020 kB
Allowed: 1033945 Pages
Free: 144508 Pages
Available: 177993 Pages
SwapCached: 0 kB
Active: 1739400 kB
Inactive: 1637756 kB
HighTotal: 3287760 kB
HighFree: 640 kB
LowTotal: 848020 kB
LowFree: 577392 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 1903768 kB
Slab: 85392 kB
CommitLimit: 2067888 kB
Committed_AS: 3479912 kB
PageTables: 20860 kB
VmallocTotal: 131064 kB
VmallocUsed: 128216 kB
VmallocChunk: 2772 kB
In the output above, the most important fields are as follows:
MemTotal (kB)- Total amount of memory in the system (4 GB in the Cisco Nexus 7000 Series Sup1)
Cached (kB) - Amount of memory used by the page cache (includes files in tmpfs mounts and data cached from persistent
storage /bootflash)
RamCached (kB) - Amount of memory used by the page cache that cannot be released (data not backed by persistent
storage)
Available (Pages) - Amount of free memory in pages (includes the space that could be made available in the page cache
and free lists)
Mapped (Pages) - Memory mapped into page tables (data being used by nonkernel processes)
Slab (Pages) - Rough indication of kernel memory consumption
Note: One page of memory is equivalent to 4 kB of memory.

The show system internal kernel memory global command displays the memory usage for the page cache and kernel/process
memory.
N7K# show system internal kernel memory global

Total memory in system : 4129600KB
Total Free memory : 1345232KB
Total memory in use : 2784368KB
Kernel/App memory : 1759856KB
RAM FS memory : 1018616KB
Detailed Assessment of Platform Memory Utilization 100

Note: In Cisco NX-OS, the Linux kernel monitors the percentage of memory that is used (relative to the total RAM present)
and platform manager generates alerts as utilization passes default or configured thresholds. If an alert has occurred, it is
useful to review the logs captured by the platform manager against the current utilization. Additional information about
this monitoring is included later in this article.
By reviewing the output of these commands, you can determine if the utilization is high as a result of the page cache, processes
holding memory, or kernel.
For more detailed information, see the following topics:
Page Cache
Kernel
User Processes
Page Cache
If Cached or RAMCached is high, you should check the file system utilization and determine what kind of files are filling the page
cache.
The show system internal flash command displays the file system utilization (the output is similar to df -hT included in the
memory alerts log).
N7K# show system internal flash
Mount-on 1K-blocks Used Available Use% Filesystem
/ 409600 43008 367616 11 /dev/root
/proc 0 0 0 0 proc
/sys 0 0 0 0 none
/isan 409600 269312 140288 66 none
/var/tmp 307200 876 306324 1 none
/var/sysmgr 1048576 999424 49152 96 none
/var/sysmgr/ftp 307200 24576 282624 8 none
/dev/shm 1048576 412672 635904 40 none
/volatile 204800 0 204800 0 none
/debug 2048 16 2032 1 none
/dev/mqueue 0 0 0 0 none
/mnt/cfg/0 76099 5674 66496 8 /dev/hda5
/mnt/cfg/1 75605 5674 66027 8 /dev/hda6
/bootflash 1796768 629784 1075712 37 /dev/hda3
/var/sysmgr/startup-cfg 409600 27536 382064 7 none
/mnt/plog 56192 3064 53128 6 /dev/mtdblock2
/dev/pts 0 0 0 0 devpts
/mnt/pss 38554 6682 29882 19 /dev/hda4
/slot0 2026608 4 2026604 1 /dev/hdc1
/logflash 7997912 219408 7372232 3 /dev/hde1
/bootflash_sup-remote 1767480 1121784 555912 67 127.1.1.6:/mnt/bootflash/
/logflash_sup-remote 7953616 554976 6994608 8 127.1.1.6:/mnt/logflash/
Note: When reviewing this output, the value of none in the Filesystem column means that it is a tmpfs type.
In this example, utilization is high because the /var/sysmgr (or subfolders) is using a lot of space. /var/sysmgr is a tmpfs mount,
which means that the files exist in RAM only. You need to determine what type of files are filling the partition and where they
came from (cores/debugs/etc). Deleting the files will reduce utilization, but you should try to determine what type of files are
taking up the space and what process left them in tmpfs.
In Cisco NX-OS release 4.2(4) and later releases, use the following commands to display and delete the problem files from the
CLI:
The show system internal dir full directory path command lists all the files and sizes for the specified path (hidden
command).
The filesys delete full file path command deletes a specific file (hidden command).
Note: Use caution when using this command. You cannot recover a deleted file.
Note: If you are running a Cisco NX-OS release prior to Cisco NX-OS release 4.2(4), you should contact your customer
support representative.
You can also use the show hardware internal proc-info pcacheinfo command to determine how much space each file system is
using in the page cache (Cached). The command output may help you determine which persistent file systems are using the page
cache and how much memory they are using.
Kernel
Kernel issues are less common, but you can determine the problem by reviewing the slab utilization in the show system internal
meminfo command output. Generally, kernel troubleshooting requires Cisco customer support assistance to isolate why the
utilization is increasing.
If slab memory usage grows over time, use the following commands to gather more information:
The show system internal kernel malloc-stats command displays all the currently loaded KLMs, malloc, and free counts.
N7K# show system internal kernel malloc-stats
Kernel Module Memory Tracking
-------------------------------------------------------------
Module kmalloc kcalloc kfree diff
klm_usd 00318846 00000000 00318825 00000021
klm_eobcmon 08366981 00000000 08366981 00000000
klm_utaker 00001306 00000000 00001306 00000000
klm_sysmgr-hb 00000054 00000000 00000049 00000005
klm_idehs 00000001 00000000 00000000 00000001
klm_sup_ctrl_mc 00209580 00000000 00209580 00000000
klm_sup_config 00000003 00000000 00000000 00000003
klm_mts 03357731 00000000 03344979 00012752
klm_kadb 00000368 00000000 00000099 00000269
klm_aipc 00850300 00000000 00850272 00000028
klm_pss 04091048 00000000 04041260 00049788
klm_rwsem 00000001 00000000 00000000 00000001
klm_vdc 00000126 00000000 00000000 00000126
klm_modlock 00000016 00000000 00000016 00000000
klm_e1000 00000024 00000000 00000006 00000018
klm_dc_sprom 00000123 00000000 00000123 00000000
klm_sdwrap 00000024 00000000 00000000 00000024
klm_obfl 00000050 00000000 00000047 00000003
By comparing several iterations of this command, you can determine if some KLMs are allocating a lot of memory but are not
freeing/returning the memory back (the differential value will be very large compared to normal).
The show system internal kernel skb-stats command displays the consumption of SKBs (buffers used by KLMs to send and
receive packets).
N7K# show system internal kernel skb-stats
Kernel Module skbuff Tracking
-------------------------------------------------------------
Module alloc free diff
klm_shreth 00028632 00028625 00000007
klm_eobcmon 02798915 02798829 00000086
klm_mts 00420053 00420047 00000006
klm_aipc 00373467 00373450 00000017
klm_e1000 16055660 16051210 00004450
Compare the output of several iterations of this command to see if the differential value is growing or very high.
The show hardware internal proc-info slabinfo command dumps all of the slab information (memory structure used for
kernel management). The output can be large.
User Processes
If page cache and kernel issues have been ruled out, utilization might be high as a result of some user processes taking up too
much memory or a high number of running processes (due to the number of VDCs/features enabled).
Note: Cisco NX-OS defines memory limits for most processes (rlimit). If this rlimit is exceeded, sysmgr will crash the process
and a core file is usually generated. Processes close to their rlimit may not have a large impact on platform utilization
but could still become an issue if a crash occurs.
Figuring Out Which Process is Using a Lot of Memory
The following commands can help you identify if a specific process is using a lot of memory:
The show process memory command displays the memory allocation per process for the current VDC (the output will
contain non-VDC global processes also).
N7K# show processes memory
PID MemAlloc MemLimit MemUsed StackBase/Ptr Process
----- -------- ---------- ---------- ----------------- ----------------
4662 52756480 562929945 150167552 bfffdf00/bfffd970 netstack
Note: The output of the show process memory command might not provide a completely accurate picture of the current
utilization (allocated does not mean in use). This command is useful for determining if a process is approaching its
rlimit.
To determine how much memory the processes are really using, you should check the Resident Set Size (RSS). This value will
give you a rough indication of the amount of memory (in KB) that is being consumed by the processes. You can gather this
information by using the following command:
The show system internal processes memory command displays the process information in the memory alerts log (if the
event occurred).
N7K# show system internal processes memory
PID TTY STAT TIME MAJFLT TRS RSS VSZ %MEM COMMAND
4727 ? Ss 00:00:00 0 1549 123248 132832 2.9 /isan/bin/pixm
4728 ? Ssl 00:00:00 0 408 78388 143104 1.8 /isan/bin/routing-sw/mrib -m 4
6662 ? Ssl 00:00:05 0 2762 64024 144396 1.5 /isan/bin/routing-sw/netstack /isan/etc/routing-sw/pm.cfg
4271 ? Ssl 00:00:00 0 609 49812 61420 1.2 /isan/bin/routing-sw/urib
7879 ? Ssl 00:00:00 0 1909 44800 90508 1.0 /isan/bin/routing-sw/bgp -t 64000
5696 ? Ssl 00:00:17 0 337 44696 55252 1.0 /isan/bin/routing-sw/clis -cli /isan/etc/routing-sw/cli
6825 ? Ssl 00:00:00 0 1402 44576 84020 1.0 /isan/bin/routing-sw/pim -t
4268 ? Ssl 00:00:00 0 363 27132 38896 0.6 /isan/bin/routing-sw/u6rib
4732 ? Ssl 00:00:00 0 404 25220 65360 0.6 /isan/bin/routing-sw/m6rib
4726 ? S<s 00:00:00 0 144 25208 30188 0.6 /isan/bin/pixmc
remaining output omitted
If you see an increase in the utilization for a specific process over time, you should gather additional information about the process
utilization.
Figuring Out How a Specific Process is Using Memory

If you have determined that a process is using more memory than expected, it is helpful to investigate how the memory is being
used by the process.
The show system internal sysmgr service pid <PID in decimal> command dumps the service information running the
specified PID.
N7K# show system internal sysmgr service pid 4727
Service "pixm" ("pixm", 109):
UUID = 0x133, PID = 4727, SAP = 176
State: SRV_STATE_HANDSHAKED (entered at time Fri Nov 12 01:42:01 2010).
Restart count: 1
Time of last restart: Fri Nov 12 01:41:11 2010.
The service never crashed since the last reboot.
Tag = N/A
Plugin ID: 1
Convert the UUID from the above output to decimal and use in the next command.
Note: If troubleshooting in lab, you can use NX-OS hex/dec conversion using following hidden commands :
hex <dec to convert>

dec <hex to convert>
The show system internal kernel memory uuid <UUID in decimal> command displays the detailed process memory usage
including its libraries for a specific UUID in the system (convert UUID from the sysmgr service output).
N7K# show system internal kernel memory uuid 307
Note: output values in KiloBytes
Name rss shrd drt map heap ro dat bss stk misc
---- --- ---- --- --- ---- -- --- --- --- ----
/isan/bin/pixm 7816 5052 2764 1 0 0 0 0 52 0
/isan/plugin/1/isan/bin/pixm 115472 0 115472 0 109176 752 28 6268 0 24
/lib/ld-2.3.3.so 84 76 8 2 0 76 0 0 0 8
/usr/lib/libz.so.1.2.1.1 16 12 4 1 0 12 4 0 0 0
/usr/lib/libstdc++.so.6.0.3 296 272 24 1 0 272 20 4 0 0
/lib/libgcc_s.so.1 1824 12 1812 1 1808 12 4 0 0 0
/isan/plugin/1/isan/lib/libtmifdb.so.0 12 8 4 1 0 8 4 0 0 0
/isan/plugin/0/isan/lib/libtmifdb_stub 12 8 4 1 0 8 4 0 0 0
/dev/mts0 0 0 0 1 0 0 0 0 0 0
/isan/plugin/1/isan/lib/libpcm_sdb.so. 16 12 4 1 0 12 4 0 0 0
/isan/plugin/1/isan/lib/libethpm.so.0. 76 60 16 1 0 60 16 0 0 0
/isan/plugin/1/isan/lib/libsviifdb.so. 20 4 16 1 12 4 4 0 0 0
/usr/lib/libcrypto.so.0.9.7 272 192 80 1 0 192 76 4 0 0
/isan/plugin/0/isan/lib/libeureka_hash 8 4 4 1 0 4 4 0 0 0
This output helps you to determine if a process is holding memory in a specific library and can assist with memory leak
identification.
The show system internal <service> mem-stats detail command displays the detailed memory utilization including the
libraries for a specific service.
N7K# show system internal pixm mem-stats detail
Private Mem stats for UUID : Malloc track Library(103) Max types: 5
--------------------------------------------------------------------------------
TYPE NAME ALLOCS BYTES
CURR MAX CURR MAX
2 MT_MEM_mtrack_hdl 32 33 16448 16596
3 MT_MEM_mtrack_info 424 531 6784 8496
4 MT_MEM_mtrack_lib_name 636 743 30054 35112
--------------------------------------------------------------------------------
Total bytes: 53286 (52k)
--------------------------------------------------------------------------------
Private Mem stats for UUID : Non mtrack users(0) Max types: 105
--------------------------------------------------------------------------------
TYPE NAME ALLOCS BYTES
CURR MAX CURR MAX
4 [r-xp]/isan/plugin/0/isan/lib/libacfg.s 0 4 0 51337
9 [r-xp]/isan/plugin/0/isan/lib/libavl.so 79 81 1568 1608
25 [r-xp]/isan/plugin/0/isan/lib/libfsrv.s 6 6 34 34
32 [r-xp]/isan/plugin/0/isan/lib/libindxob 6 6 456 456
46 [r-xp]/isan/plugin/0/isan/lib/libmpmts. 0 2 0 100
48 [r-xp]/isan/plugin/0/isan/lib/libmts.so 7 10 816 972
51 [r-xp]/isan/plugin/0/isan/lib/libpfm_in 0 1 0 3490
53 [r-xp]/isan/plugin/0/isan/lib/libpss.so 169 196 27316 114880
57 [r-xp]/isan/plugin/0/isan/lib/libsdb.so 140 140 5632 5632
62 [r-xp]/isan/plugin/0/isan/lib/libsrg.so 0 1 0 3480
68 [r-xp]/isan/plugin/0/isan/lib/libsysmgr 3 3 2094 2094
79 [r-xp]/isan/plugin/0/isan/lib/libutils. 61 69 512 55389
84 [r-xp]/isan/plugin/1/isan/bin/pixm 238 240 532920 533440
88 [r-xp]/isan/plugin/1/isan/lib/libpixm.s 0 1 0 48
92 [r-xp]/lib/ld-2.3.3.so 21 26 3483 4233
94 [r-xp]/lib/tls/libc-2.3.3.so 286 287 8163 8490
100 [r-xp]/usr/lib/libglib-2.0.so.0.600.1 12 19 6328 6800
--------------------------------------------------------------------------------
Total bytes: 589322 (575k)
These outputs are usually requested by the Cisco customer support representative when investigating a potential memory leak in a
process or its libraries.
Built-in Platform Memory Monitoring

Cisco NX-OS has built-in kernel monitoring of memory usage to help avoid system hangs, process crashes, and other undesirable
behavior. The platform manager periodically checks the memory utilization (relative to the total RAM present) and automatically
generates an alert event if the utilization passes the configured threshold values. When an alert level is reached, the kernel
attempts to free memory by releasing pages that are no longer needed (for example, the page cache of persistent files that are no
longer being accessed), or if critical levels are reached, the kernel will kill the highest utilization process. Other Cisco NX-OS
components have introduced memory alert handling, such as BGP's graceful low memory handling, that allow processes to adjust
their behavior to keep memory utilization under control.
Note: While Cisco NX-OS implements VDCs, it is important to remember that a specific VDC's memory utilization is not
limited. Platform memory issues will impact all configured VDCs.
Memory Thresholds
Prior to Release 4.2(4), the default memory alert thresholds were as follows:
• 70% MINOR
• 80% SEVERE
• 90% CRITICAL
From Release 4.2(4) and later releases, the memory alert thresholds were changed to the following:
• 85% MINOR
• 90% SEVERE
• 95% CRITICAL
This change was introduced in part due to baseline memory requirements when many features/VDCs are deployed.
The thresholds are configurable, using the following command:
system memory-thresholds minor percentage severe percentage critical percentage
The show system internal memory-status command allows you to check the current memory alert status.
N7K# show system internal memory-status
MemStatus: OK
Memory Alerts
If a memory threshold has been passed (OK -> MINOR, MINOR -> SEVERE, SEVERE -> CRITICAL), the Cisco NX-OS
platform manager will capture a snapshot of memory utilization and log an alert to SYSLOG (as of Release 4.2(4), default VDC
only). This snapshot is useful in determining why memory utilization is high (process, page cache, or kernel). The log is generated
in the Linux root path (/) and copy is moved to OBFL (/mnt/plog) if possible. This log is very useful for determining if memory
utilization is high due to the memory that was consumed by the page cache, kernel, or Cisco NX-OS user processes.
The show system internal memory-alerts-log command displays the memory alerts log.
The memory alerts log consists of the following outputs:
Command Description
cat
Provides a log of timestamps when memory alerts occurred.
/proc/memory_events
Shows the overall memory statistics including the total RAM, memory consumed by the page cache,
cat /proc/meminfo
slabs (kernel heap), mapped memory, available free memory, and so on.
cat /proc/memtrack
Memory Thresholds 106

Displays the allocation/deallocation counts of the KLMs (Cisco NX-OS processes running in kernel
memory).
df -hT Displays file system utilization information (with type).
du --si -La /tmp Displays file information for everything located in /tmp (symbolic link to /var/tmp).
cat
Dumped a second time to help determine if utilization changed during data gathering.
/proc/memory_events
cat /proc/meminfo Dumped a second time to help determine if utilization changed during data gathering.
This article describes how to troubleshoot packet flow issues for Cisco NX-OS.
Guide Contents
Troubleshooting CFS
Troubleshooting STP
Troubleshooting Packet Flow Issues(this section)
Contents
• 1 Packet Flow Issues
♦ 1.1 Packets Dropped Because of
Rate Limits
♦ 1.2 Packets Dropped Because of a
QoS Policy
♦ 1.3 Packets Dropped in Hardware
♦ 1.4 show hardware internal
statistics rates
◊ 1.4.1 show hardware
internal statistics pktflow
all
Memory Alerts 107

Packet Flow Issues

Packets could be dropped for the following reasons :
• Software switched packets could be received from the interface, but dropped by the supervisor because of rate limits.
• Packets could be dropped because of a QoS policy.
• Hardware switched packets could be dropped by the hardware because of a bandwidth limitation.
Packets Dropped Because of Rate Limits

Use the show hardware rate-limit command to determine if packets are being dropped because of a rate limit.
dctl-n7010-7# show hardware rate-limit copy


------------------------------------------------------------
copy Config : 30000
Allowed : 13651778
Dropped : 228295 <-- caused by ICMP redirect or OSPF Hello
Total : 13880073
dctl-n7010-7(config)# show hardware rate-limit module 1


------------------------------------------------------------
layer-3 mtu Config : 500
Allowed : 0
Dropped : 0
Total : 0
layer-3 ttl Config : 500

Allowed : 0
Dropped : 0
Total : 0
layer-3 control Config : 10000

Allowed : 17020000
Dropped : 88790262 <---HSRP, OSPF hello
Total : 105810262
layer-3 glean Config : 100

Allowed : 0
Dropped : 0
Total : 0
layer-3 multicast directly-connected Config : 3000

Allowed : 0
Dropped : 0
Total : 0
layer-3 multicast local-groups Config : 3000

Allowed : 0
Dropped : 0
Total : 0
layer-3 multicast rpf-leak Config : 500

Allowed : 0
Dropped : 0
Total : 0
layer-2 storm-control Config : Disabled
access-list-log Config : 100

Allowed : 0
Dropped : 0
Total : 0
copy Config : 30000

Allowed : 552173
Dropped : 0
Total : 552173
receive Config : 30000

Allowed : 85134
Dropped : 0
Total : 85134
layer-2 port-security Config : Disabled
Packets Dropped Because of a QoS Policy

Use the show policy-map interface control-plane command to determine if packets are being dropped because of a QoS policy.
dctl-n7010-7# sh policy-map interface control-plane

class-map copp-system-class-exception (match-any)
match exception ip option
match exception ip icmp unreachable
police cir 360 kbps , bc 250 ms
module 1 :
conformed 0 bytes; action: transmit
violated 0 bytes; action: drop
module 2 :
module 3 :
module 4 :
module 10 :
Packets Dropped in Hardware

Use the following show hardware commands to determine if packets are being dropped by the hardware.
• show hardware internal statistics rates

• show hardware internal statistics pktflow all
Packets Dropped Because of Rate Limits 109

show hardware internal statistics rates

dctl-n7k-dist-1# sh hardware internal statistics rates
+ =============================
+ R2D2 Instance 0
+ =============================
|
|-- Ingress IN
| |--- Packets/sec
| | |--- sum: 0
| |
| |--- Bytes/sec
| | |--- sum: 0
|
|-- Ingress OUT
| |--- Packets/sec
| | |--- sum: 0
| |
|
|-- Egress IN
| |--- Packets/sec
| | |--- sum: 0
| |
|
|-- Egress OUT
| |--- Packets/sec
| | |--- sum: 0
| |
| |--- Bytes/sec
| | |--- sum: 0
|
|
|
+ =============================
+ Metropolis Instance 0
+ =============================
|
|-- Ingress IN
| |--- Packets/sec
| | |--- I1: 2
| | |--- I1: 0
| | |--- sum: 0
| |
|
|-- Ingress OUT
| |--- Packets/sec
| | |--- I1: 2
| | |--- I1: 0
| | |--- sum: 0
| |
| |--- Bytes/sec
| | |--- I1: 1166
| | |--- I1: 0
| | |--- sum: 0
|
|-- Egress IN
| |--- Packets/sec
| | |--- I1: 0
| | |--- I1: 0
| | |--- sum: 0
| |
| |--- Bytes/sec
| | |--- I1: 0
| | |--- I1: 0
| | |--- sum: 0
show hardware internal statistics rates 110

|
|-- Egress OUT
| |--- Packets/sec
| | |--- I1: 0
| | |--- sum: 0
| |
|
|
|
show hardware internal statistics pktflow all
This command displays per ASIC statistics, including packets into and out of the ASIC. This command helps to identify where
packet loss is occurring.
dctl-n7k-dist-1# show hardware internal statistics pktflow all

bhv_bitmask:0
|------------------------------------------------------------------------|
| Device:R2D2 Role:MAC |
| Packets
|------------------------------------------------------------------------|
Instance: 0 Ports:-
|----------|-------------------|------------------|
| | IN | OUT |
|----------|-------------------|------------------|
|Ingress | 00000000014a40c0 | 00000000014a40c0 |
|----------|-------------------|------------------|
|Egress | 000000000007e9dc | 000000000007e9dc |
|----------|-------------------|------------------|
|------------------------------------------------------------------------|
| Device:Metropolis Role:REWR |
| Packets
|------------------------------------------------------------------------|
Instance: 0 Ports:-
|----------|-------------------|------------------|
| | IN | OUT |
|----------|-------------------|------------------|
|Ingress | 00000000014a40c0 | 0000000001498ccc |
|----------|-------------------|------------------|
|Egress | 000000000007e9dc | 000000000007e9dc |
|----------|-------------------|------------------|
|------------------------------------------------------------------------|
| Device:Octopus Role:QUE |
| Packets
|------------------------------------------------------------------------|
Instance: 0 Ports:-
|----------|-------------------|------------------|
| | IN | OUT |
|----------|-------------------|------------------|
|Ingress | 0000000001498ccc | 0000000001498cc6 |
|----------|-------------------|------------------|
|Egress | 000000000007e9c5 | 000000000007e9dc |
|----------|-------------------|------------------|
*** Counters above represent packets combined into a larger one ***
show hardware internal statistics pktflow all 111

VPC Nexus 7k

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

VPC Nexus 7k

Uploaded by

Copyright:

Available Formats

Cisco Nexus 7000 Series NX-OS Troubleshooting Guide

2. Register for free.

3. Click on "Cisco Live Virtual" link.

5. In the search box, type ?BRKCRS-3144? and Submit search.

Audience and Generating a PDF of This Guide

Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Book PDF

Troubleshooting Installs, Upgrades, and Reboots

Troubleshooting Unicast Traffic

Troubleshooting Multicast Traffic

Troubleshooting Packet Flow Issues

Troubleshooting Service Failures

Before Contacting Technical Support

Troubleshooting Tools and Methodology

Overview of the Troubleshooting Process

1. Maintain a consistent Cisco NX-OS release across all your devices.

• Is the interface operational?

See Troubleshooting Ports for more troubleshooting tips for ports.

Verifying Layer 2 Connectivity

Verifying Layer 3 Connectivity

• Have you configured a default gateway?

Use the following commands to verify your routing configuration:

• Identify key Cisco NX-OS troubleshooting tools.

System Message Text

Verifying Layer 3 Connectivity 5

PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not supported.

Error Message PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not

Explanation Transceiver (SFP) is not from an authorized vendor.

syslog Server Implementation

◊ syslog client: switch1

To configure the syslog feature on Cisco NX-OS, follow these steps:

1. switch1# config terminal

switch1# show logging server

System Message Text 6

To configure a syslog server, follow these steps:

2. Create the log file.

3. Restart the syslog process.

syslog service starting.

4. Verify that the syslog process has started.

ps -ef |grep syslogd

root 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd

Troubleshooting with Logs

switch# show logging ?

console Show console logging configuration

syslog Server Implementation 7

switch# show logging server

Logging server: enabled

Viewing NVRAM logs

switch# show logging nvram

2008 Sep 10 15:51:58 switch %$ VDC-1 %$ %SYSMGR-2-NON_VOLATILE_DB_FULL: System n

Troubleshooting with Logs 8

Contacting Customer Support

• Date that you received the switch

Cisco NX-OS System Messages Reference

Nexus: Hands on with NX-OS, Part#1

Viewing NVRAM logs 9

Information About Upgrades and Reboots

Upgrades and Reboot Checklist

switch# show install all status

switch# show install all status

Upgrades and Reboot Checklist 11

Verifying a Nondisruptive Upgrade

Do you want to continue with the installation (y/n)? [n] y

switch# show install all failure-reason

Using ROM Monitor Mode