MDS Slow Drain Device Training: © 2009 Cisco Systems, Inc. All Rights Reserved. Cisco Confidential

You might also like

Download as pptx, pdf, or txt
Download as pptx, pdf, or txt
You are on page 1of 38

MDS Slow Drain

Device Training

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1


What’s the Issue?

Today almost all of data traffic between end devices is carried by FC Class 3
service that uses link level, per hop based, buffer to buffers flow control.
There is no End-to-End flow control with Class 3 services and in presence of
slow devices, that are not accepting frames at the originating device rate, the
FC networks are prone to ISL credit starvation and may choke the links.
Frames destined to slow devices have to be carefully isolated in separate
queues and switched over to egress queues without congesting the backplane.
Eventually decision has to be made if the frames are considered stuck and
when to drop them.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2


Fibre Channel Class of Service 3

Class 3 services is defined as connectionless service based on frame


switching. There is no acknowledgment of a frame delivery or rejection.
Frames could be held by the Fabric or by the end devices without any
notification.
Upper layer protocols must guarantee the transmission and handle the lost
frames efficiently.
It uses buffer-to-buffer flow control system that works only on per hop basis.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 3


Buffer to Buffer Flow Control
B2B flow control is a link-level flow control system that manages the rate of frame transmission
between two FC ports on a link.
Upon receiving (1) a B2B credit signal from the FC switch the server will transmit (2) a frame to the
switch.
The switch will hold the frame in its buffer (3) until it receives (4) a B2B credit signal from the
destination device. After a credit is received, switch then transmits (5) the frame to its destination
and will release (6/1) a B2B credit to the server so it can send another frame.
Devices that do not release the B2B credits in timely fashion are termed as slow drain devices.

FC Switch
1
Server Receive 4 Disk
B2B Credit Receive
6
B2B Credit

Transmit
Transmit Buffers
Frame
Buffers 3
2 Frame

5

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 4


ISL Blockage
B2B credits are on per hop basis and in a multi hop networks this limitation may create a problem.
Consider the example below where yellow disk is not releasing (1) a B2B credit while blue disk is
behaving fine.
The frames going from the yellow server to the yellow disk will be held (2) in the buffer at “FC switch
2” and eventually it will stop generating the B2B credit (3) signal to the adjacent “FC switch 1”,
attached via ISL.
This would cause blockage for all flows (yellow & blue in this case) going over that ISL. In other
words ISL would be blocked until the frames are released from the buffers and B2B credit is
released.

FC Switch 1 FC Switch 2

ISL
Slow

NO
X Drain

B2B Credit
Buffers
Full X Device
NO
3 2 B2B Credit
1
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 5
MDS Solution to ISL
Blockage Problem

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6


What’s the Solution?

The ISL blockage problem requires three steps solution.


First, all the frames related to slow drain devices should be held in separate
queues.
Second, frames destined to slow drain devices should not be switched over
the backplane as this would cause backplane congestion.
Third, if a stuck frame is held in the queue for more than “Drop_Timer” value
then it should be dropped.
The first two steps are done automatically in MDS architecture and the 3rd step
would require some manual fine tuning of the “Drop_Timer” value.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7


Step1: Isolate Slow Drain related flows
Cisco MDS architecture implements a Virtual Output Queuing (VoQ) architecture. Upon receiving a
frame the source index (SI), destination index (DI) and the QoS flag is checked and the frame is
placed in a unique queue.
To accommodate all possible permutations MDS VoQ architecture can dynamically provide up to
96,000 queues. (24src X 1024dst X 4QoS = 98,304 queues).
On average it maintains about 4,096 queues. Frames related to slow drain device are held in
separate ingress queues (S2-D1 & S3-D3) at each switch. This step is done automatically in MDS by
VoQ technology.

96,000 VoQ FC Switch 1 96,000 VoQ FC Switch 2

S1-D1
ISL
S1 D1 S3 S3-D2
D2

Slow
Drain
D3
S2 Buffers Buffers X Device
S3-D3
S2-D1
NO
B2B Credit

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 8


Step2: Avoid Backplane Congestion
Once the frame arrives in the VoQ it generates a request signal “R” to the central arbiter asking for
its permission to transmit that frame over the backplane to its destination.
If a buffer space is available in the destination queue a token signal “T” is generated to the arbiter
indicating that it can accept a frame. At this point arbiter sends the grant signal “G” to the VoQ
indicating it to transmit the frame to its destination.
This three way hand shake is done at line rate for every frame that is switched within MDS
architecture. This ensures that the backplane is not congested by stuck frames taking up buffers and
blocking other flows. Again this step is done automatically by MDS central arbiter.

FC Switch 1 FC Switch 2

G
G
T
ISL

R T
Slow
R Drain
Buffers Buffers X Device

NO
Frames Holdup Frames Holdup B2B Credit

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 9


Step3: Drop the Stuck Frames
If the slow drain device continues to hold the release of B2B credit the switch cannot hold frames
indefinitely.
Currently within MDS if a frame is held for more than 500ms it is flagged as stuck frame and is
dropped automatically.
This releases the buffer space and allows the switch to generate a B2B credit signal on the blocked
ISL, clearing the ISL blockage. This drop timer is called “Drop_Timer” and it is configurable by the
end user.

FC Switch 1 FC Switch 2

G
G
T
ISL

R T
Slow
R Drain
B2B Credit
Buffers Buffers X Device

NO
Drop Frames
B2B Credit
after “Twait” expires

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 10


Step3a: Identify the blocked port
Check if any ports have timeout drops:
 switch# show hardware internal packet-flow dropped

 module-x# show hardware internal packet-flow dropped

 module-x# show hardware internal fc-mac port <y> statistics


The above CLI provides the timestamp dropped frame count and also provides the excessive TX
credit transition to zero.

Check if any ports are experiencing credit loss and recovery


 module-x# show process creditmon statistics
The above CLI provides the credit loss/recovery count.

 module-x# show process creditmon event-history


The above CLI provides the credit loss/recovery timestamp.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 11


Step3a: Identify the blocked port continued
Check Request Timeouts
 module-x# show hardware internal up-xbar <0|1> asic-info
This CLI captures the DI/GI (Destination Index or Global Index) to which the requests are timing out.

Check Queued Frames
 module-x# show hardware internal up-xbar <0|1> queued-packet-info
This CLI captures the DI/GI (Destination Index or Global Index) to which the frames are destined to if
they are queued for an extended time.

GI/DI to Interface Conversion:


 switch# show system internal fcfwd idxmap port-to-interface
This CLI helps to find the interface associated with the DI/GI (Destination Index or Global Index). DI
and GI refer to the same index. FCIP links will not have the timestamp drops; take a look at the
queued frame count to find out if the FCIP link is congested.
If the timestamp drop or the credit recovery is seen on FC ISLs, follow that link to the next switch
and repeat the steps mentioned. These steps have to be followed till an end device or the end ISL
that is congested is found.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 12


Step3b: Reconfigure the timeout value
 Timeout values are configured on per port basis. The blocked port is identified by Step 3a. You
may have to try different values based on your network characteristics.

 Gen-2:
 module-x# debug fc-mac port <y> reg-wr 0x566 <to-val-1>
 module-x# debug fc-mac port <y> reg-wr 0x567 <to-val-1>

 module-x# debug fc-mac port <y> reg-wr 0x5a6 <to-val-2>

 Gen-3:
 module-x# debug fc-mac port <y> reg-wr 0x3cb <to-val-1>

 module-x# debug fc-mac port <y> reg-wr 0x3cc <to-val-1>

 module-x# debug fc-mac port <y> reg-wr 0x446 <to-val-2>

 Fabric Switches: 9222i:


 Same as Gen-2 above

 9124 and 9134:


 module-x# debug fc-mac port <y> reg-wr 0x505 <to-val-1>

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 13


New Features Available
in 4.2(7)

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14


Slow Drain Device

Credit issues at this Host


“Slow Drain Device”

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15


Detection:
Average Credit Non-Available Duration

Sys Log/Trap
(Switch/Interface)

show system internal snmp credit-not-available


show port-monitor
show logging logfile

 0 sec  credits

 10% (100
ms)
e.g. 10%

Enabled by default, threshold: 10% in 1s, action: syslog and trap


 1 sec

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 16


Detection:
Average Credit Non-Available Duration Configuration

Switch Level, per Port Mode (F/E)/ All Ports


Threshold: 10% - 100% , 10% increment, default: 10%
Interval: 1 s – 1 hour, default: 1 s
Events and actions include
Threshold configuration will
1. Syslog (default) be in % of credit non-available
2. Trap (default) duration in a specified
interval(s), default 10% (100
3. Error Disable or Flap the port
ms) in 1 second.
switch(config)# port-monitor name Cisco
switch(config-port-monitor)# counter tx-credit-not-available poll-
interval 1 delta rising-threshold 10 event 4 falling-threshold 0
event 4 portguard errordisable

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 17


Level 0 – No Recovery Attempt “Error Disable”

Sys Log, Trap


(Switch/Interface)

show system internal snmp credit-not-available


show port-monitor
show logging logfile
Show interface fcx/y

X
 0 sec  credits

 10% (100ms)

e.g.10%

Action (Error Disable) and Threshold depends on topology and customer


requirements and are customer driven configs. Error Disabled port(s) can
be recovered by shut/no shut.  1 sec

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18


Level 1 - Recovery “Congestion Drop Timeout”
Switch Level, per Port Mode
switch(config)# system timeout congestion-drop <value> mode (F) / (E)
switch(config)# system timeout congestion-drop default mode (F) / (E)

show logging onboard timeout-drops


Show logging onboard request-timeout
show process creditmon credit-loss-events

Frees up credits

e.g. drop timeout = 200 ms


 0 sec ---
 Check Timestamp
Timeout Value (100 ms - 1000 ms),  200 ms  of each frame
10 ms increment, default value – 500 ms

 Drop the frame


 from the Queue

Congestion Drop Timeout is enabled by default.  credit


Timeout Value change is dependent on topology and customer
requirements.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 19


Level 2 – Recovery “No-Credit Preemptive drops”
Switch Level, F mode only
switch(config)# system timeout no-credit-drop <value> mode F
switch(config)# system timeout no-credit-drop default mode F

show logging onboard timeout-drops


Show logging onboard request-timeout
show process creditmon credit-loss-events
Frees up credits

 0 sec ---
e.g. No-Credit Timeout = 300 ms
 300 ms  --  NO credit
(100 ms – 1000 ms), 100 ms increments  ------
 ------
 ----- 
 Frames
Start dropping
 -----
 the frames
 -----
* No-Credit Preemptive Drops is disabled by default.  at the line rate till

 Credits are received. ------
Timeout Value and Enable Decision is based on topology and  ------
 --------
customer requirements.  credit
 --------
 --------

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 20


Level 3 – Recovery “Credit Loss”  Link Reset
 Automatic – Enabled by Default
 1 sec for F Ports, 1.5 sec for E Ports
(Sys Log/Trap)
 These timers are not configurable (Switch/Interface)

show process creditmon credit-loss-events


show port-monitor
show logging logfile

 credits
 0 sec --
 LR
 shut/no shut  No Credits (Dead)

 1 sec --  LR
 No Response
 shut
The interface should run out of credits for 1 s (F) or 1.5 s (E) continuously
 no shut
for LR to kick-in.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 21


Level 4 - Recovery “Credit Loss”  Error Disable

(Sys Log/Trap)
(Switch/Interface)

show process creditmon credit-loss-events


show system internal snmp credit-not-available
show port-monitor
show logging logfile

X  credits
* Error Disable  0 sec --

 No Credits (Dead)

 1 sec --  LR

Error Disable X  No Response

anytime after 1 sec  Shut


 no shut
Error Disable action config is determined by the topology
and customer requirements. Error Disabled port(s) can be
recovered by shut/no shut.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 22


Level 4 - Recovery “Credit Loss”  Error Disable
Action Configuration

Switch Level, per Port Mode (F/E)/ All Ports


Events and actions include
1. Syslog (default)
2. Trap (default)
3. Error Disable or Flap the port
switch(config)# port-monitor name Cisco
switch(config-port-monitor)# counter credit-loss-reco poll-interval 60
delta rising-threshold 4 event 4 falling-threshold 1 event 4 portguard
errordisable

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 23


Debug Commands
switch# show logging onboard request-timeout [module <module number>]
This command will display the source interface(s) and destination interface pair for which the
requests are getting timed-out (requests take > 90 ms to be granted). Destination Interface displayed
in this command output is the slow interface.

switch# show logging onboard request-timeout


----------------------------
Module: 2
----------------------------
--------------------------------------------------------------------------------
| Dest | Source |Events| Timestamp | Timestamp |
| Intf | Intf | Count| Latest | Earliest |
--------------------------------------------------------------------------------
|fc2/1 |fc2/19, | 77 |Mon May 31 13:12:22 2010|Mon May 31 13:12:39 2010 |
--------------------------------------------------------------------------------
|fc2/19 |fc2/18, | 1 |Mon May 31 13:12:21 2010|Mon May 31 13:12:21 2010 |
--------------------------------------------------------------------------------
|fc2/1 |fc2/19, | 13 |Mon May 31 13:12:13 2010|Mon May 31 13:12:21 2010 |
--------------------------------------------------------------------------------
|fc2/19 |fc2/18, | 2 |Mon May 31 13:12:11 2010|Mon May 31 13:12:11 2010 |
--------------------------------------------------------------------------------
switch#

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 24


Debug Commands (contd…)
switch# show process creditmon credit-loss-events [module <module number>]
This command will display the interface on which the credit loss events are seen with the total number
of credit loss events and upto 10 latest timestamp of credit loss events for each interfaces.

switch# show process creditmon credit-loss-events module 2


Module: 02 Credit Loss Events: YES
-----------------------------------------------------------------
| Interface | Total | Timestamp |
| | Events | |
-----------------------------------------------------------------
| fc2/1 | 444748 | 1. Tue Jun 8 20:22:02 2010 |
| | | 2. Tue Jun 8 20:22:01 2010 |
| | | 3. Tue Jun 8 20:22:00 2010 |
| | | 4. Tue Jun 8 20:21:59 2010 |
----------------------------------------------------------------
| fc2/19 |2 | 1. Wed Jun 9 10:15:49 2010 |
| | | 2. Wed Jun 9 09:07:45 2010 |
-----------------------------------------------------------------
switch#

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 25


Debug Commands (contd…)
switch# show process creditmon credit-loss-event-history [module <module number> ]
This command will display the credit loss event-history for all the interfaces in a specified module.

switch# show process creditmon credit-loss-event-history module 2


1) Event:CREDITMON_EVENT_CREDIT_LOSS, length:4, at 854490 usecs after Wed Jun 9
10:15:49 2010
interface = fc2/19
2) Event:CREDITMON_EVENT_CREDIT_LOSS, length:4, at 205391 usecs after Wed Jun 9
09:07:45 2010
interface = fc2/19
3) Event:CREDITMON_EVENT_CREDIT_LOSS, length:4, at 730422 usecs after Tue Jun 8
20:22:02 2010
interface = fc2/1
4) Event:CREDITMON_EVENT_CREDIT_LOSS, length:4, at 630446 usecs after Tue Jun 8
20:22:01 2010
interface = fc2/1
switch#

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 26


Debug Commands (contd…)
switch# show logging onboard credit-loss [last mm minutes] [last hh hours] [last dd days]
[module <module number>]
This command will display the total credit loss events per interface and the timestamp in the
latest specified duration.

switch# show logging onboard credit-loss


----------------------------
Module: 2
----------------------------
-----------------------------------------------------------------------------------------------------
ERROR STATISTICS INFORMATION FOR DEVICE ID 118 DEVICE Aakash
------------------------------------------------------------------------------------------------------
Interface | | | Time Stamp
Range | Error Stat Counter Name | Count | MM/DD/YY HH:MM:SS
| | |
----------------------------------------------------------------------------------------- ------------
fc2/1 | AK_FCP_CNTR_CREDIT_LOSS | 647 | 05/25/10 23:53:34
fc2/7 | AK_FCP_CNTR_CREDIT_LOSS | 400 | 05/25/10 23:50:34
------------------------------------------------------------------------------------------------------
switch#

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 27


Debug Commands (contd…)
switch# show logging onboard timeout-drops [last mm minutes] [last hh hours] [last dd days]
[module <module number>]
This command will display the total timeout drops per interface and the timestamp in the
latest specified duration.

switch# show logging onboard timeout-drops


----------------------------
Module: 2
----------------------------
--------------------------------------------------------------------------------------------------------------------
ERROR STATISTICS INFORMATION FOR DEVICE ID 118 DEVICE Aakash
---------------------------------------------------------------------------------------------------------------------
Interface | | | Time Stamp
Range | Error Stat Counter Name | Count | MM/DD/YY HH:MM:SS
| | |
--------------------------------------------------------------------------------------------------------- ------------
fc2/1 | AK_FCP_CNTR_LAF_TOTAL_TIMEOUT_FR | 647 | 05/25/10 23:53:34
fc2/2 | AK_FCP_CNTR_LAF_TOTAL_TIMEOUT_FR | 200 | 05/25/10 20:53:34
-----------------------------------------------------------------------------------------------------------------------
switch#

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 28


Debug Commands (contd…)
switch# show system internal snmp credit-not-available [module <module number>]
This command will display the rising and falling thresholds, timestamp and the interval/% duration
of time credit not available per interface.

switch# show system internal snmp credit-not-available module 2

Module: 2 Number of events logged: 5


---------------------------------------------------------------------------------------------------------------------------------------
Port Threshold Rising Interval(s) Event Time Type Duration of time not available
--------------------------------------------------------------------------------------------------------------------------------------
fc2/1 20/10(%) 1 Tue Jun 1 16:27:24 2010 Falling 0%
fc2/13 20/10(%) 1 Tue Jun 1 16:27:24 2010 Falling 0%
fc2/19 20/10(%) 1 Tue Jun 1 16:27:24 2010 Falling 0%
fc2/13 20/10(%) 1 Tue Jun 1 16:27:43 2010 Rising 100%
fc2/13 20/10(%) 1 Tue Jun 1 16:27:44 2010 Falling 0%
switch#

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 29


Example command
output

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 30


CLI Commands
 How to check ISLs for lack of transmit credits.In the below example, fc2/1
has no tx credits available and fc3/1 has available tx credits, but it may be
below its full complement. This may indicate congestion in the fabric:
switch# sh int | i "fc|Belong|low pri" | ex "descript" | ex "Peer" | i "trunking" n 2 
fc2/1 is trunking Belongs to port­channel 20 
0 low priority transmit B2B credit remaining 
fc3/1 is trunking Belongs to port­channel 20 
185 low priority transmit B2B credit remaining 

 How to check for issues queuing frames to a port on the ingress port
switch# attach module < module­number> 
module­x# show hardware internal up­xbar 0 queued­packet­info 
... 
+­­­­­­­­­­+­­­­­­­­+­­­­­­­­+­­­­­­­­+­­­­­­­­+ 
| GI (Hex) | Prio 0 | Prio 1 | Prio 2 | Prio 3 | 
+­­­­­­­­­­+­­­­­­­­+­­­­­­­­+­­­­­­­­+­­­­­­­­+ 
| c  | 000000 | 000000 | 000000 | 001005 | 
| 1c | 000000 | 000000 | 000000 | 00200a |
   ^                                   ^ 
   |                                   | 
egress port (slow)               ingress port 

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 31


CLI Commands (continued)

 How to quickly check for transmit frame drops


switch# show hardware internal packet­flow drop 
Module: 03 Dropped Packets: YES 
+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+ 
| DEVICE NAME      |     PORTS        |           DROPPED COUNT             | 
|                  |                  |     RX (Hex)    |      TX (Hex)     | 
+­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­ |­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+ 
|    Aakash        |       3          |               0 |                1c | 
|    Aakash        |       4          |               0 |                10 |
       ^                   ^                                              ^ | 
       |                   |                                              | 
MAC device name        egress                                    port count 

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 32


CLI Commands (continued)

 How to check frame drop reason on egress port


switch# show hardware internal packet­dropped­reason           
Module: 02       Dropped Packets: YES    
+­­­­­­­­­­­+­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+    
|           |               |      DROPS        |                              |    
|   PORTS   |  DEVICE NAME  |­­­­­­­­­­­­­­­­­­­|          COUNTER NAME        |    
|           |               | Rx(Hex) | Tx(Hex) |                              |    
+­­­­­­­­­­ +­­­­­­­­­­­­­­­+­­­­­­­­­+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+    
|  2        |Stratosphere   |    ­    |  849    |FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD
|  ^                                      ^                  ^       
   |                                      |                  |    
 egress port index (slow)               count        timeout discard reason

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 33


CLI Commands (continued)

 How to check frame transmit timeout history on egress port


switch# show logging onboard error­stats 
­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 
Module: 1
­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 
­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 
ERROR STATISTICS INFORMATION FOR DEVICE ID 63 DEVICE Stratosphere
­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 
                                  |                |    Time Stamp    |In|Port 
    Error Stat Counter Name       | Count          |MM/DD/YY HH:MM:SS |st|Rang | 
                                  |                |                   Id|e 
­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 
FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD|ff     |02/17/10 16:26:03  |00|1 
FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD|2fd    |02/17/10 16:26:13  |00|1 
FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD|4fb    |02/17/10 16:26:23  |00|1 
FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD|6f9    |02/17/10 16:26:33  |00|1 
FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD|8f7    |02/17/10 16:26:43  |00|1 
FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD|af5    |02/17/10 16:26:53  |00|1 
FCP_CNTR_LAF_C3_TIMEOUT_FRAMES_DISCARD|10ef   |02/17/10 16:27:23  |00|1 

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 34


CLI Commands (continued)

 How to check frame transmit timeout history on egress port -


continued
switch# attach module < module­number> 
module­x# show hardware internal up­xbar 0 asic­info 
... 
     DI      priority    q­len   time 
­­­­­­­­­­­­ ­­­­­­­­­­ ­­­­­­­ ­­­­­­­­­ 
21 (0x015)      0          1     812028 usecs on Mon Oct 19 22:06:05 2010 
239 (0x0ef)     3          1     935355 usecs on Mon Oct 19 22:05:44 2010 
18 (0x012)      3          1     972427 usecs on Mon Oct 19 22:05:42 2010 
34 (0x022)      3          1     931536 usecs on Mon Oct 19 22:05:42 2010 
        ^                                        ^  
        |                                        | 
egress port index (slow)                timestamp of timeout 

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 35


CLI Commands (continued)

 How to check for credit loss on an egress port


switch# attach module < module_number> 
module­x# show process creditmon statistics 
***** Credit Monitor Info ***** 
Port    Mode    Monitor  VL  Cr Thresh  Cr Loss 
                             Exceeded     Event 
­­­­   ­­­­­­­ ­­­­­­­  ­­  ­­­­­­­­­   ­­­­­­­ 
   1      E       yes   NA       0         7        
   ^                                       ^ 
   |                                       | 
egress port                                count 

switch# attach module < module_number> 
module­x# show process creditmon event­history | include CREDIT_LOSS next 1 
Event:CREDITMON_EVENT_CREDIT_LOSS, length:1, at 818948 usecs after Sat Nov 21 06:33:33 2010 port = 18
    

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 36


CLI Commands (continued)

 Please refer to the following extensive “Slow Drain


Receiver Troubleshooting” document prepared by Tim
Greenland for a detailed explanation of troubleshooting
slow drain related problems.
 The previous command outputs were all taken from this
document.

 http://bock-bock/wiki/User:Smarkovi:Troubleshooting:Identify_Slow_Draining_Device

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 37


© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 38

You might also like