Professional Documents
Culture Documents
BRKRST 3143
BRKRST 3143
BRKRST 3143
BRKRST-3143
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3
Agenda
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4
Multicast Expansion
Fabric ASIC Table (MET)
Switch and
Fabric Replication
Engine
PFC
L2 L2 CAM
Engine
CPU Card
LC-RBUS
LC-DBUS
NetFlow Table
RP
Controller FIB TCAM
CPU
EOBC
Agenda
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6
Unwanted flooding
Do we learn MAC, are L2 tables in sync ?
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8
R1 DUT R2
Po1 Po2
Gig5/2 Gig7/2 Po1 Po2
Gig8/2 Gig7/3
Ten8/1
Gig8/1 Gig7/4
Gig8/3 Gig7/5 Ten8/3 Ten8/3
Gig8/4 Gig7/6
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11
Po1 Po2
Gig5/2 Gig7/2 Po1 Po2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Gig8/3 Gig7/5 Ten8/3 Ten8/3
Host1
Gig8/4 Gig7/6 Host2
R1 DUT R2
7.0.1.1 Vlan700 7.0.1.2
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14
Repeat same steps for finding links used in Po2, Po1 on DUT and Po2 on R2 in
both directions (to 7.0.1.2 and to 7.0.1.1)
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16
Po1 Po2
Gig5/2 Gig7/2 Po1 Po2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Gig8/3 Gig7/5 Ten8/3 Ten8/3
Host1
Gig8/4 Gig7/6 Host2
R1 DUT R2
7.0.1.1 Vlan700 7.0.1.2
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17
R1#remote command switch test etherchannel load-balance int po1 ip 9.0.1.2 Check which link between R1 and
Computed RBH: 0x7 DUT in 5 port etherchannel, based on
Would select Gi8/2 of Po1 etherchannel loadbalance
Traffic flow 8.0.1.1 -> 9.0.1.2 leaves R1 on Gi8/2 link, in vlan 705, to next
hop 7.5.1.2 for HW CEF switched packets; for SW CEF switched packets,
same link, but in vlan 701, to next hop 7.1.1.2
Repeat the same steps for finding L3 next hops and links on DUT, and R2,
in both directions
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19
Interface: Te8/3, Next Hop: 7.7.1.2, Vlan: 1090, Destination Mac: 000f.f8e4.d000
DUT#sh vlan internal usage | i 1090 Next hop for L3 interface is linked to
1090 TenGigabitEthernet8/3 internal vlan; check internal VLAN
matches physical interface
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20
Interface: Vl705, Next Hop: 7.5.1.1, Vlan: 705, Destination Mac: 0011.bc75.9c00
Traffic flow 9.0.1.2 -> 8.0.1.1 leaves DUT on Gi7/6 link, in vlan 705
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21
Interface: Te8/3, Next Hop: 7.7.1.1, Vlan: 4043, Destination Mac: 0050.f0f8.7400
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22
Po1 Po2
Gig5/2 Gig7/2
Gig8/2 Gig7/3
Gig8/1 Gig7/4 Ten8/1 Ten8/1
Gig8/3 Gig7/5
Ten8/3 Ten8/3
Gig8/4 Gig7/6
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25
Switch Fabric
WS-X6748
Fabric Layer 2 Layer 2 Fabric Module 7
Interface & Interface &
MET Engine Engine MET
Replication Replication
Engine L3/4 Engine
Port Port DFC3 Engine Port Port
ASIC ASIC ASIC ASIC
4 x 12xGE port asic
Gig7/4
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26
Port counters Port Port Port counters Port Port 4 x 1x10GE port asic
ASIC ASIC ASIC ASIC
WS-X6704
L2 Engine counters
Fabric Layer& 2Tables
Layer 2 Fabric Module 8
Interface & Interface &
MET Engine Engine MET
Replication Replication
Engine L3/4 Engine
DFC3 Engine
Fabric counters Channel0 Channel1 Fabric counters
Switch Fabric
Fabric counters
Channel1 WS-X6748
Fabric L2 Engine counters Fabric
Layer& 2 Layer 2
Tables Module 7
Interface & Interface &
MET Engine Engine MET
Replication Replication
Engine L3/4 Engine
Port counters Port Port DFC3 Engine Port Port
ASIC ASIC ASIC ASIC
4 x 12xGE port asic
Gig7/4
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28
Did a ping (2000 packets/100 bytes per packet) from 7.0.1.1 -> 7.0.1.2,
verify interface counters relevant to the path did move sufficiently !!
DUT#sh int gi 7/4 count
Port InOctets InUcastPkts InMcastPkts InBcastPkts
Gi7/4 249784 2000 8 40
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
Gi7/4 245614 2000 6 0
DUT#sh int ten 8/3 count
Ten8/3 4 x 10GE port asic Ten8/1
Port InOctets InUcastPkts InMcastPkts InBcastPkts
Te8/3 10590 18 28 0 ? ? Mod 8
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
L2 L2
Te8/3 246449 2000 10 0
DUT#sh int ten 8/1 count
L3/4
Port InOctets InUcastPkts InMcastPkts InBcastPkts Fabric
Te8/1 273441 2032 174 0
Mod 7
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts L2 L2
Te8/1 2890 0 11 0
L3/4
DUT#
Gig7/4
?
4 x 12xGE port asic
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29
. . . Increases per new learn (source mac lookup miss) Increases per flooded packet (destination mac lookup miss)
Src Mac misses = 0x000000000425D50C (69588236)
Dst Mac misses = 0x0000000005340140 (87294272)
line full encountered during New l = 0x0000000000000000 (0) Unable to learn because all hash buckets full
. . .
correctable errors in bank 0 = 0x0000000000000000 (0)
Correctable ECC errors upon reading entry in L2 table
uncorrectable errors in bank 0 = 0x0000000000000000 (0)
correctable errors in bank 1 = 0x0000000000000000 (0)
uncorrectable errors in bank 1 = 0x0000000000000000 (0) Uncorrectable ECC errors upon reading entry in L2 table .. HW
DBus Header Checksum errors = 0x0000000000000000 (0)
address of the line full = 0x00000204
L2 Engine sees bad CRC DBUS header
address of the last error in Bank0 = 0x00004022
address of the last error in Bank1 = 0x00002040
Superman 1 Forwarding statistics:
L2 Engine 1 on module 7
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37
Gig7/3 Gig7/6
Similar to L2 check port counters, relevant fabric channels, L2 Engine counters and
tables;
Additionally: check L3 Engine counters and tables
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38
show ip route
<ip address> Verify Layer 3 Verify Layer 2 rewrite
IOS Routing Table (RP) IOS ARP Cache Table (RP) show ip cef
adjacency
show ip cef < ip <interface>
address> <next hop ip
IOS FIB Table (RP) IOS Adjacency Table (RP) address>
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49
Checking VRF’s Check interface (vlan 701) is in the correct VRF (VPN
DUT#remote com sw sh mls vlan-ram 701 end 701 value 0: default routing table)
TYCHO Vlan RAM
Key: * => Set, - => Clear
vlan eom nf-vpn mpls mc-base siteid stats rpf vpn-num bgp-grp l2-metro rpf-pbr-ovr
----+---+------+----+-------+------+-----+---+-------+-------+--------+-----------
701 - - * 0 0 - - 0 0 - *
DUT(config)#int vlan 701
Illustration: move to different VRF, and how to
DUT(config-if)#ip vrf forwarding customer-1 check this got programmed in HW …
DUT#remote com sw sh mls vlan-ram 701 end 701 sometimes issues seen with interface staying
TYCHO Vlan RAM in default VRF; check this on each DFC/PFC !!
Key: * => Set, - => Clear
vlan eom nf-vpn mpls mc-base siteid stats rpf vpn-num bgp-grp l2-metro rpf-pbr-ovr
----+---+------+----+-------+------+-----+---+-------+-------+--------+-----------
701 - - * 0 0 - - 256 0 - *
DUT#show mls cef exact-route vrf ?
Use further same commands as with default
WORD VPN Routing/Forwarding instance name
FIB, specifying VPN with vrf key word
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50
PFC3/DFC3
L2 Engine L2 CAM
contains
V
L2 CAM (64K) MAC entries
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54
PFC3/DFC3
L2 Engine L2 CAM
contains
V
L2 CAM (64K) MAC entries
Verified already at FIB and adjacency tables, as well as L2 CAM table, ACL
TCAM/counters
Still to look at: Netflow Table
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60
“show tcam acl” commands on in/egress interface explain what traffic gets punted
to SW because of NAT (first packet(s), till SW installs Netflow entry), follow same
logic as in ACL TCAM when interpreting the output
Similar HW assisted features: Reflexive ACL, SLB, TCP intercept … look at the
SW installed Netflow entries, as well as the ACL TCAM content for the relevant
interfaces
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61
Are we running out of L2/L3 Engine Resources (FIB, ACL, Netflow TCAM full …) ?
show platform hardware capacity forwarding
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68
Unwanted flooding
Check we learn MAC, L2 tables are in sync
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69
Agenda
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73
Entry
OIF VLAN LTL
ID
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76
EARL-DBUS
EARL-RBUS
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79
Group
225.10.10.10 Gi9/1 Gi5/2
L3 Network VLAN 20 VLAN 10
Source Receiver
172.16.25.1 10.10.10.100
Gi9/4
VLAN 20
Receiver
20.20.20.100
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80
EARL-DBUS
Replication Engine/Fabric
EARL-RBUS ASIC (Supervisor): Transmits &
receives packets from the switch
fabric. Responsible for SPAN
and multicast replication.
PFC/Switching Engine:
Performs all MET lookups using
indices from L3 lookups. Does
Switch L2
Fabric
engine: L2 lookups
L3 engine: L3 FIB & Adj
packet rewrites for packets sent
lookups; NetFlow lookups;
across the fabric. Also serves as
RACL, VACL & QoS lookups
LC-DBUS
SP RP BUS ASIC
LC-RBUS
CPU CPU
Fabric ASIC L2 L3
Port
& Engine Engine
ASIC
Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81
VLAN B R
Module 1 VLAN O Module 2
R Port Port Port Port
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
EARL-DBUS
EARL-RBUS
7. L3 engine performs ACL,
5. Supervisor DBUS 6. L2 engine performs
VACL and QoS lookup in
ASIC receives DBUS L2 lookup in ingress
ingress VLAN and performs an
packet and accepts it VLAN and forwards
RPF check
and forwards to PFC headers to L3 engine Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82
VLAN B R
Module 1 VLAN O Module 2
R Port Port Port Port
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
EARL-DBUS
EARL-RBUS
9. L2 engine
sends final result
10. Fabric ASIC on over LC-RBUS to
Supervisor Fabric ASIC
forwards result Switch Fabric
onto E-RBUS
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN B R
Module 1 VLAN O Module 2
R Port Port Port Port
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
12. Fabric ASIC on ingress
card rewrites the packet
according to the result, builds
Replication Replication Replication Replication
Engine & Engine & a fabric packet containing theEngine & Engine &
Fabric ASIC Fabric ASIC rewritten packet and the result
Fabric ASIC Fabric ASIC 14. Fabric ASIC
and forwards to the fabric
on egress
module receives
packet from the
EARL-DBUS 13. Switch Fabric uses the
switch fabric and
FPOE in the fabric packet and
EARL-RBUS forwards to port
forwards only to channels that
ASIC
have receivers or mrouters in
the ingress VLAN (VLAN O)
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85
VLAN B R
Module 1 VLAN O Module 2
R Port Port Port Port
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
22. ASIC
Replication Replication originating the Replication Replication
Engine & Engine & DBUS packets Engine & Engine &
Fabric ASIC Fabric ASIC accepts the results, Fabric ASIC Fabric ASIC
all others discard
EARL-DBUS
EARL-RBUS
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 86
VLAN B R
Module 1 VLAN O Module 2
R Port Port Port Port
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 87
VLAN O
VLAN B R
Module 1 S
Module 2
R Port Port S Port Port
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
EARL-DBUS
EARL-RBUS
LC-DBUS
SP RP
LC-RBUS
CPU CPU 7. L3 engine performs
lookup using the
primary CEF entry.
L3 engine also does
VLAN G R
R Port Fabric ASIC L2 L3 ACL, VACL and QoS
& Engine Engine lookup in ingress
R ASIC
VLAN P R Replication PFC VLAN and RPF check
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88
EARL-DBUS
EARL-RBUS
9. L2 engine
sends final result
10. Fabric ASIC on over LC-RBUS to
Supervisor forwards Fabric ASIC
result onto E-RBUS Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
8. L3 engine
returns result to L2
engine. Result
VLAN G R
R Port Fabric ASIC L2 L3 contains LTL index
& Engine Engine for forwarding in
R ASIC
VLAN P R Replication PFC the ingress VLAN
as well as indices
Supervisor Engine for MET lookup
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 89
VLAN O
VLAN B R
Module 1 S
Module 2
R Port Port S Port Port
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
12. Fabric ASIC on ingress
card rewrites the packet
according to the result, builds a
Replication Replication Replication Replication
Engine & Engine & fabric packet containing the Engine & Engine &
Fabric ASIC Fabric ASIC rewritten packet and the result
Fabric ASIC Fabric ASIC 14. Fabric
and forwards to the fabric
ASIC on
egress
module
EARL-DBUS 13. Switch Fabric uses the
receives
FPOE in the fabric packet and
EARL-RBUS packet from
forwards only to channels that
the switch
have receivers or mrouters in
fabric and
the ingress VLAN (VLAN O)
forwards to
port ASIC
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 90
EARL-DBUS
EARL-RBUS
17. RE copies the original
packet onto VLAN G and 18. L2 engine performs
Fabric ASIC sends DBUS no lookup and
packet to the switching forwards appropriate
engine for egress headers to L3 engine
Switch Fabric
processing
LC-DBUS
SP RP
LC-RBUS
CPU CPU 19. L3 engine
receives appropriate
headers from the L2
engine and performs
VLAN G R
R Port Fabric ASIC L2 L3 ACL, VACL and QoS
& Engine Engine lookup for VLAN G.
R ASIC
VLAN P R Replication PFC Result is forwarded to
L2 engine
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 91
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 92
EARL-DBUS
EARL-RBUS
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 93
EARL-DBUS
EARL-RBUS
26. Result received by
fabric ASIC on the
module 1. All others
discard result
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 94
EARL-DBUS
EARL-RBUS
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 95
EARL-DBUS
EARL-RBUS
31. L2 engine
Note:
Note: Steps
Steps 3030 -- 32
32 are
are repeated
repeated for for each
each of
of the
the recognizes packet is
fabric
fabric ASICs
ASICs that
that received
received the the packet
packet on on the
the flagged for egress
internal replication VLAN. Each
internal replication VLAN. Each needs the needs the Switch Fabric
replication and
result
result of
of the
the CEF
CEF lookup
lookup (i.e.,
(i.e., the
the index
index for
for the
the forwards headers to
MET
MET lookup
lookup toto get
get thethe OIL
OIL forfor all
all the
the local
local L3 engine
receivers
receivers and
and mrouters)
mrouters) LC-DBUS
SP RP
LC-RBUS
CPU CPU
32. L3 engine performs
CEF lookup using
secondary entry.
VLAN G R
R Port Fabric ASIC L2 L3 Lookup yields MET
& Engine Engine index for replication to
R ASIC
VLAN P R Replication PFC all local OIFs. Result is
forwarded to the L2
Supervisor Engine engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 96
EARL-DBUS
EARL-RBUS
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 97
Switch Fabric
36. L2 engine
performs L2 lookup
in egress VLAN
(VLAN B) and LC-DBUS
SP RP forwards headers to
LC-RBUS
CPU CPU L3 engine 37. L3 engine
performs RACL,
VACL and QoS
lookups for
VLAN G R
R Port Fabric ASIC L2 L3 egress VLAN and
& Engine Engine forwards result to
R ASIC
VLAN P R Replication PFC L2 engine
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 98
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 99
Switch Fabric
43. L2 engine
performs L2 lookup
in egress VLAN
(VLAN P) and LC-DBUS
SP RP forwards headers to
LC-RBUS
CPU CPU L3 engine 44. L3 engine
performs RACL,
VACL and QoS
lookups for
VLAN G R
R Port Fabric ASIC L2 L3 egress VLAN and
& Engine Engine forwards result to
R ASIC
VLAN P R Replication PFC L2 engine
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 100
EARL-DBUS
EARL-RBUS
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 101
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 102
EARL-DBUS
EARL-RBUS
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 103
56. Packet
EARL-DBUS 55. RE on supervisor is forwarded
performs a MET lookup over the
EARL-RBUS using the MET index DBUS to the
from the result and forwarding
replicates packet onto engine for
VLAN P an egress
lookup.
57. L2 engineSwitch Fabric
performs L2 lookup
in egress VLAN
(VLAN P) and LC-DBUS
SP RP forwards headers to
LC-RBUS
CPU CPU L3 engine 58. L3 engine
performs RACL,
VACL and QoS
lookups for
VLAN G R
R Port Fabric ASIC L2 L3 egress VLAN and
& Engine Engine forwards result to
R ASIC
VLAN P R Replication PFC L2 engine
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 104
EARL-DBUS
EARL-RBUS
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 105
Verify L1/L2
Use…
Show interfaces
Show interfaces counters
Show interfaces counters errors
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 106
Group
225.10.10.10 Gi9/1 Gi5/2
L3 Network VLAN 20 VLAN 10
Source Receiver
172.16.25.1 10.10.10.100
Gi9/4
VLAN 20
Receiver
20.20.20.100
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 107
Note:
Note: TheThe output
output only
only shows
shows thethe last
last reporter,
reporter,
so
so aa given
given host
host may
may not
not show
show upup in
in the
the output
output ifif
there
there are
are other
other receivers
receivers on
on the
the same
same interface.
interface.
Make
Make sure
sure that
that the
the OIF
OIF shows
shows up up in
in the
the
interface
interface column.
column.
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 108
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 109
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 110
Group: 225.10.10.10, Source count: 1, Packets forwarded: 350, Packets received: 350
RP-tree: Forwarding: 0/0/0/0, Other: 0/0/0
Source: 172.16.25.1/32, Forwarding: 350/1/975/2, Other: 350/0/0
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 112
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 113
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 114
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 115
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 116
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 117
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 118
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 119
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 120
(172.16.25.1, 225.10.10.10)
IOSVPN:0 (1) PI:1 (1) CR:0 (1) Recirc:0 (1)
RPF VLAN
Vlan:20 AdjPtr:30 FibRpfNf:1 FibRpfDf:1 FibAddr:0x30080
rwvlans:20 rwindex:0x939 adjmac:001d.a29a.1f00 rdt:1 E:0 CAP1:0
fmt:Mcast l3rwvld:1 DM:0 mtu:1518 rwtype:L2&L3 met2:0x8A met3:0x8B
Egress VLAN
packets:0000000000049 bytes:000000000000005782
met2 block Starting Offset: 0x008A Packet and byte counts
V E L0 C:1015 I:0x0080B
should increment with MET index used to retrieve the
Starting Offset: 0x008B packets forwarded
met3 block egress replication VLAN and the
V E C: 10 I:0x0091B
LTL index used to forward a
single copy of the multicast
packet across the fabric in the
egress replication VLAN
Met3 lookup result.
Show egress VLAN 10 Met2 lookup result.
and LTL index 0x91B Shows egress
replication VLAN 1015
and LTL index 0x80B
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 121
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 122
***The slot number will be the slot with the ingress replication engine if looking at the
primary entry and the egress slot if looking at the non-primary entry
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 123
Cat6K#rem comm sw test mcast ltl index 91b contains only the port
on the egress module
index 0x91B contain ports 5/2 where the receiver in
VLAN 10 lives
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 124
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 125
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 126
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 127
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 128
R1 R2
8.0.1.1 DUT 9.0.1.2
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 129
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 130
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 131
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 132
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 133
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 134
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 135
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 137
Gig2/9/1
Po1 Po2
Gig5/9 Gig2/6/12
Ten1/3/2 Ten1/1
Gig4/16 Gig1/9/36
Gig2/2 Gig2/9/15 Ten2/2/7 Ten1/4
Gig5/2 Gig1/6/2
Gig2/4 Gig1/5/1 R2
8.0.1.1 R1 Trusted Port-channel2 DUT 9.0.1.2
Port channel 2 is the only port channel trusted for dual active detection
BFD direct connection is between Gig1/6/1 and Gig2/9/1
Both mechanisms can be on simultaneously
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 138
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 139
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 140
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 142
Configuration has been modified (sh/no sh), it needs to be saved before it will recover; if not and
configurations are possibly out of sync (any “conf t” has been issued without saving while the VSS was still
up), standby mode will be RPR+ until we manually save/sync the configuration and reset standby;
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 143
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 144
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 146
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 147
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 148
DUT learns 8.0.1.0/24 via ECMP on VLAN’s 701 to 705 over port
channel 2
DUT learns 9.0.1.0/24 via ECMP on L3 interfaces Ten1/3/2 and
Ten2/2/7
Launching ping from 8.0.1.1 to 9.0.1.2
Using similar commands/steps as in Unicast L3
troubleshooting to find out the path, only VSS specifics are
highlighted in next slides
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 149
Identify the physical interface flow to host 1 (out of Port-channel 2) will use
DUT#show etherchannel load-balance hash-result interface Port-channel 2 switch 2 ip 9.0.1.2 8.0.1.1
Computed RBH: 0x3
Would select Gi2/9/15 of Po2 Packet coming in on switch id 2, needing to go
out on Po2 will select Gi2/9/15
DUT#show etherchannel load-balance hash-result interface Port-channel 2 switch 1 ip 9.0.1.2 8.0.1.1
Computed RBH: 0x3
Would select Gi1/6/2 of Po2 Packet coming in on switch id 1, needing to go
out on Po2 will select Gi1/6/2
For MEC, load-balance should prefer physical interfaces local to the switch the packet was
received on
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 150
Looking at the HW table shows next hop directly attached to local switch is
preferred
DUT#show mls cef lookup 9.0.1.0 switch 1 mod 3
Codes: decap - Decapsulation, + - Push Label Packet coming in on switch 1 module 3, for 9.0.0.0/8
Index Prefix Adjacency prefers next hop attached to local switch id 1
108775 9.0.0.0/8 Te1/3/2 , 000f.35ed.7c00
DUT#show mls cef lookup 9.0.1.0 switch 2 mod 2
Codes: decap - Decapsulation, + - Push Label Packet coming in on switch 2 module 2, for 9.0.0.0/8
Index Prefix Adjacency prefers next hop attached to local switch id 2
108775 9.0.0.0/8 Te2/2/7 , 000f.35ed.7c00
DUT#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3
? ... show vlan internal usage | I 4064
Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00
DUT#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2
Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00
Further, use similar commands (enhanced with extra argument of switch id) as in
standalone switch
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 151
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 152
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 153
Troubleshooting VSS
Problems We’ve Looked at
VSS control plane issues: VSS doesn’t form, dual
active, dirty configuration …
Checked using VSS specific commands
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 154
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 155
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 156
Include:
1. Brief Description On RP
2. Bridge number terminal length 0
Catalyst 6500
3. Hostname and IP
show log Sup720 Native IOS
show clock
Supervisor show tech
Troubleshooting
Failover Show tech platform Procedure
On Route Processor (RP)
show scp accounting
show scp counters Determine Problem Type
show eobc
show ibc Routing
show ipc status Multicast
show ipc ports On RP On SP
show heartbeat Show platform tech unicast <..> show mls cef ip detail
show fabric errors show ip arp show mls cef inconsistency On RP
show fabric utilization show ip cef show platform tech-support ipmulticast <..>
show mls cef summary
show fabric channel show adjacency detail show tech ipmulticast
show ip route show ip mroute
On Switch Processor (SP) show ip ospf statistics Module show mls ip multi connected
show mls ip multi statistics
show scp accounting show ip ospf data data
show scp counters show ip ospf neigh show mls ip multi sum
show eobc On RP show mls ip multi group <ip> source <ip>
show ip bgp neighbor
show ibc show module <mod> show mls rp ip
show ip bgp summary
show earl status show idprom all detail
show ip eigrp neighbor
show earl statistics show power
show ip eigrp topology
show diagnostic result <mod>
show fabric errors traceroute <w.x.y.z> On SP
show fabric timeout show mls cef summary show mmls v g g
show ipc status show mls cef show mls cef ip multicast detail
show ipc ports show mls cef adjacency
show heartbeat
sh platform hard superman config
show platform hard tycho interrupt
BRKRST-3143
14664_05_2008_c2
Send data
© 2008 Cisco Systems, Inc. All rights reserved.
to Cisco TAC and attach to case
Cisco Public 157
Q and A
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 158
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 160
QoS troubleshooting
WS-SUP32P (PISA) troubleshooting
Modular IOS troubleshooting
Monitoring the health of the system (GOLD/EEM)
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 162
QoS troubleshooting
WS-SUP32P (PISA) troubleshooting
Modular IOS troubleshooting
Monitoring the health of the system (GOLD/EEM)
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 163
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 164
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 165
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 166
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 167
!
policy-map police-host-to-host interface GigabitEthernet9/1
class host-to-host switchport
police cir 9000000 bc 281250 be 281250 switchport access vlan 20
conform-action set-dscp-transmit cs5 switchport mode access
exceed-action drop violate-action drop mls qos vlan-based
!
interface Vlan20 Cat6K#show mls qos
ip address 20.20.20.1 255.255.255.0 QoS is enabled globally
ip pim sparse-dense-mode Policy marking depends on port_trust
load-interval 30 QoS ip packet dscp rewrite enabled globally
service-policy input police-host-to-host Input mode for GRE Tunnel is Pipe mode
Input mode for MPLS is Pipe mode
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 168
Note: May be difficult to catch the interesting traffic as this will show the last packet
switched in hardware.
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 169
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 170
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Full
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Destination-Source-Interface
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Source-only
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Destination
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Destination-Source
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 172
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 173
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 174
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 175
In this case
it’s NDE
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 176
Bandwidth
Priority
Queue-limit
Random-detect
Set qos-group
Service policy (nested policies are not supported)
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 177
Unsupported Features
***All are supported on OSM’s, FlexWAN and Enhanced FlexWAN. See QoS
configuration guide for OSM and FlexWAN modules for specific Caveats
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 178
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 179
QoS troubleshooting
WS-SUP32P (PISA) troubleshooting
Modular IOS troubleshooting
Monitoring the health of the system (GOLD/EEM)
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 180
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 181
WS-Sup32P – PISA
Programmable IP Services Accelerator (PISA)
Packet back to EARL for FIB
CDE redirects packets to NP or RP lookup (ingress case) or L2
lookup /VACL (egress case)
GE Uplinks
Supervisor Engine 32
Baseboard
DRAM
DRAM
512 SP
SP CPU
CPU 1 Gbps
512 MB
MB
Port counters L3/4 Engine
DRAM
DRAM 1 Gbps
RP
RP CPU
CPU Classification Port ASIC
11 GB
GB Classification
10G and
and Dispatch
Dispatch L2/L3 Engine counters
1-3G & tables
Network Engine
Engine PISA
PISA
NP counters PISA Channel
Process
DRAM
DRAM
Micro Engines
or CDE counters
L2 Engine
768
768 MB
MB PFC3B
Replication
32M
32M Daught
Engine
SRAM
SRAM PISA Daughter er
CPU Card
Card
Bus
Up to 3 Gbps internal
Network Processor EtherChannel interface
Accelerates NBAR and Incoming packet on bus gets redirected to
for PISA connection
FPM at up to 2 Gbps PISA based on PFC3B ACL redirect
(Po256)
BRKRST-3143 (ingress) or modified FIB entry (egress)
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 182
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 183
WS-Sup32P
Is the Internal Port Channel to PISA OK ?
DUT#sh int po 256 Truncated output
Port-channel256 is up, line protocol is up (connected)
input flow-control is on, output flow-control is on
Members in this channel: Gi6/8 Gi6/10
DUT#sh running-config Truncated output
interface Port-channel256
mtu 4160
Verify flowcontrol, MTU and pisa-
…
channel configuration on port channel
flowcontrol receive on interface, and its physical members;
flowcontrol send on
pisa-channel
interface GigabitEthernet6/8
mtu 4160
…
flowcontrol receive on
flowcontrol send on
no cdp enable
channel-group 256 mode on
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 185
WS-Sup32P
Do the Packets Get Punted to PISA ?
DUT#sh tcam interface vlan 701 acl in ip detail
Interface: 701 label: 1537 lookup_type: 0 Truncated output
protocol: IP packet-type: 0
+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+
|T|Index| Dest Ip Addr | Source Ip Addr| DPort | SPort | TCP-F |Pro|MRFM|X|TOS|TN|COD|F-P|
+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+
V 36250 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 1 0 -- C-- 0-0 <-
M 36251 0.0.0.0 0.0.0.0 0 0 0 ---- 1 0 <-
R rslt: REDIRECT_ADJACENCY (*) rtr_rslt: PERMIT_RESULT (*) indx: 0x7E03 hit_cnt=118 <-
DUT#show mls cef adjacency entry 0x7F803 detail Calculate redirect index: 0x7E03 – 0x7E00 + 0x7F800 = 0x7F803
Index: 522243 mtu: 65535, vlan: 0, dindex: 0x340, l3rw_vld: 1
format: RECIR, flags: 0xA0000001000E00
packets: 140, bytes: 8960 Gets redirected to internal port index 0x340,
matching the port channel 256 to PISA module
DUT#show table ltl module 6 start 0x340 end 0x340
LTL indexes from: 0x340 to 0x340 - slot: 6
Index Ports
Index 0x340 maps to interfaces 8 and 10 on module 6 (so
---------+----------------------------------------------------
Gi6/8 and Gi6/10), matches Po256 Members !! …. If not
0x00340 8,10 correct, packets won’t get punted to NP
DUT#show interface Port-channel 256 | i Members
Members in this channel: Gi6/8 Gi6/10
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 186
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 187
WS-Sup32P
Do the Packets Get Out of Port Channels to PISA?
DUT#show interface port-channel 256 counters
DUT#show interface port-channel 256 counters errors Check counters on internal port channel
DUT#show interface port-channel 256 are moving, any errors … ?
…
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max) Indication CDE is flow controlling towards the port
30 second input rate 266000 bits/sec, 211 packets/sec ASIC, because e.g. NP is too busy
30 second output rate 273000 bits/sec, 211 packets/sec
Gi6/10 on on on on 0 0
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 188
WS-Sup32P
Looking at the NP Counters ?
DUT#show platform hardware pisa np ?
ME ME Counters Truncated output
acl Access-list
all All
RX = what comes in from CDE, TX, what goes back to
fpm Flexible Packet Matching Info CDE
mqc Modular QoS CLI Info
nbar Network Based Application Recognition Info
rx Receive Engine Info
tx Transmit Engine Info
DUT#show platform hardware pisa np nbar counters
NBAR Statistics(ME2)
In our example, we did reclassification based on
NBAR for telnet
--------------------
NBAR Pkts Received : 325
NBAR Pkts Classified: 325
PD Pkts Received : 0
NBAR Pkts Out : 325
NBAR Debug 0 : 82
NBAR Debug 1 : 81
NBAR Debug 2 : 81
NBAR Debug 3 : 81
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 190
QoS troubleshooting
WS-SUP32P (PISA) troubleshooting
Modular IOS troubleshooting
Monitoring the health of the system (GOLD/EEM)
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 193
Process Crash
Memory Leak
High CPU utilization
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 194
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 195
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 196
Cat6K#dir disk0:
Directory of disk0:/
Both
filenames
Crashinfo
encode the
filename
process that
and location
crashed
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 197
Restarting a Process
To restart a process use the command process restart [process]
Cat6K#
03:47:08: %SYSMGR-6-RESPAWN: Process tcp.proc:1 has been respawned :
sysmgr.proc : (PID=20498, TID=14) : -Traceback=(s72033_rp-
ipservices_wan-57-dso-p.so+0x11364) ([36:0]+0x134FC) ([36:0]+0xB418)
([25:-9]1+0x167C) ([35:0]+0x39B4) ([35:0]+0x3F48) ([0:-
3]libc+0x252D4) ([7:0]+0x127AC)
Cat6K#
Restarting the process produces a log message stating that the process has been
respawned and a traceback
Use show processes detailed [process] to see that a process has been restarted
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 198
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 199
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 200
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 201
Show Memory
Cat6K#show clock
*01:39:31.399 UTC Wed Apr 9 2008
Cat6K#show memory
System Memory: 524288K total, 282464K used, 241824K free, 1000K kernel reserved
Lowest(b) : 233308160
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 202
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 204
* Use pipe option with exclude 0.0 to eliminate the irrelevant output
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 205
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 206
QoS troubleshooting
WS-SUP32P (PISA) troubleshooting
Modular IOS troubleshooting
Monitoring the health of the system (GOLD/EEM)
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 207
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 208
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 210
Diagnostics capabilities
built in hardware
Depending on hardware,
Gold can catch:
Port failure
Bent backplane connector
Bad fabric connection
Malfunctioning forwarding engines
Stuck control plane
Bad memory
—
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 211
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 213
1) TestTransceiverIntegrity:
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
Test results: (. = Pass, F = Fail, U = Untested)
U U . U . . U U . . U U . . U U U U U U U U U U
2) TestLoopback:
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
. . . . . . . . . . . . F . . . . . . . . . . .
3) TestScratchRegister -------------> .
4) TestSynchedFabChannel -----------> .
<snip>
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 216
TestSPRPInbandPing :
By default, this test is enabled as health-monitoring test.
The SP-RP Inband test catches most of the runtime software driver
and hardware issues on supervisors. This is done by using diagnostic
packet tests exercising the layer 2 forwarding engine, the L3-4
forwarding engine, and the replication engine along the path from
the Switch Processor to the Route Processor.
Packets are sent at an interval of 15 seconds and 10 consecutive
failures of the SP-RP Inband test result in failover to the
redundant supervisor (default).
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 217
Bootup diagnostics:
Set level to complete
On demand diagnostics:
Use as a pre-deployment tool: run complete diagnostics
before putting hardware into production environment
Use as a troubleshooting tool when suspecting
Si
hardware failure
Scheduled diagnostics:
Schedule key diagnostics tests periodically
Schedule all non-disruptive tests periodically
Health-monitoring diagnostics:
Key tests running by default
Enable additional non-disruptive tests for specific functionalities
enabled in your network: IPv6, MPLS, NAT
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 218