Professional Documents
Culture Documents
Cisco Live - Cisco Nexus 7000 Hardware Architecture PDF
Cisco Live - Cisco Nexus 7000 Hardware Architecture PDF
BRKARC-3470
Session Goal
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Agenda
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Nexus 7010 Chassis System status
ID LEDs on Front-to-
all FRUs back airflow
LEDs
Integrated cable
Air exhaust
management
with cover
Optional
locking front
System fan trays
doors
Power supplies
Integrated cable
management Optional front
door
Side-to-side
airflow
Supervisor Crossbar
slots (9-10) 25RU fabric
modules
Presentation_ID
Front N7K-C7018
2010 Cisco and/or its affiliates. All rights reserved. Cisco Public
Rear
5
Agenda
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Supervisor Engine
ID LED
AUX Port USB Ports CMP Ethernet
Management
Status Console Port Ethernet Compact Flash
Reset Button
LEDs Slots
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Management Interfaces
Management Ethernet
10/100/1000 interface used exclusively for system
management
Belongs to dedicated management VRF
Prevents data plane traffic from entering/exiting from mgmt0
interface
Cannot move mgmt0 interface to another VRF
Cannot assign other system ports to management VRF
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Supervisor Engine Architecture
To Modules To Fabrics To Modules
n * 23G
Switched Arbitration
Gigabit Fabric Path
Ethernet
ASIC
System Controller
CMP 266MHz
4GB
DRAM Security
Link Processor
Encryption 2MB Main
2GB CPU
NVRAM
Internal CF 1.66GHz
PHY PHY
OBFL Dual-Core
10/100/1000 Flash 10/100/1000
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Agenda
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
8-Port 10GE I/O Module
N7K-M108X2-12L
N7K-M108X2-12L
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
8-Port 10G XL I/O Module Architecture
N7K-M108X2-12L
EOBC To Fabric Modules To Central Arbiter
Fabric ASIC
LC
CPU
VOQs VOQs
Forwarding Forwarding
Engine Engine
Replication Replication
Engine Engine
Replication Replication
Engine Engine
10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC
1 2 3 4 5 6 7 8
Front Panel Ports
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
32-Port 10GE I/O Module
N7K-M132XP-12
rate-mode shared
10G
(default)
Shared mode
9 11 13 15
Four interfaces in port group
share 10G bandwidth
rate-mode dedicated
10G
Dedicated mode
9 11 13 15 First interface in port group
gets 10G bandwidth
Other three interfaces in port
group disabled
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
32-Port 10G I/O Module Architecture
N7K-M132XP-12
EOBC To Fabric Modules To Central Arbiter
Fabric ASIC
LC
CPU
VOQs VOQs
Forwarding Engine
Replication Replication
Engine Engine
Replication Replication
Engine Engine
10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC
4:1 Mux + 4:1 Mux + 4:1 Mux + 4:1 Mux + 4:1 Mux + 4:1 Mux + 4:1 Mux + 4:1 Mux +
Linksec Linksec Linksec Linksec Linksec Linksec Linksec Linksec
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
48-Port 1G I/O Modules
N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L
N7K-M148GS-11L
Release 5.0(2) and later
Fabric ASIC
LC VOQs
CPU
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Agenda
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Forwarding Engine Hardware
IPv4 and IPv6 multicast support Ingress and egress NetFlow (full
(SM, SSM, bidir) and sampled)
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Scalable Services License
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Forwarding Engine Architecture
Layer 3 Engine
IPv4/IPv6 Layer 3 lookups
ACL, QoS, NetFlow and other processing
Linear, pipelined architectureevery packet subjected to both
ingress and egress pipeline
Enabling features does not affect forwarding engine performance
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Forwarding Engine
Pipelined Architecture
FE Daughter
Card FIB TCAM and
adjacency table
Ingress Pipeline lookups for Layer
Ingress NetFlow 3 forwarding
collection Egress Pipeline
ECMP hashing
Multicast RPF
Layer 3 Egress policing check
Ingress ACL Engine
and QoS
classification Egress NetFlow Ingress policing
lookups collection
Unicast RPF
check Egress ACL and QoS
classification lookups
Ingress MAC
Layer 2
table lookups Egress MAC
IGMP snooping Engine
lookups
lookups IGMP snooping
IGMP snooping lookups
redirection
Packet Headers from Final lookup result to
I/O Module Replication Engine I/O Module Replication Engine
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Agenda
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Crossbar Switch Fabric Module
N7K-C7010-FAB-1
N7K-C7018-FAB-1
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Fabric Module Capacity
Fabric Modules
Crossbar 1
46Gbps/slot Fabric
ASICs
2 x 23G channels per
I/O module slot
Crossbar 2
46Gbps/slot Fabric
1 x 23G channel per ASICs
supervisor slot
Crossbar 3
46Gbps/slot Fabric
ASICs
230Gbps
46Gbps
184Gbps
138Gbps
92Gbps
per slot bandwidth
Crossbar 4
46Gbps/slot Fabric
ASICs
Crossbar 5
46Gbps/slot Fabric
ASICs
Nexus 7018
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
I/O Module Capacity
Fabric Modules
Crossbar 1
1G modules 46Gbps/slot Fabric
Require 1 fabric for full ASICs
bandwidth
Require 2 fabrics for N+1
redundancy Crossbar 2
46Gbps/slot Fabric
ASICs
230Gbps
46Gbps
184Gbps
138Gbps
92Gbps 46Gbps/slot
Crossbar
Fabric
3
per slot bandwidth ASICs
10G modules
Require 2 fabrics for full Crossbar 5
bandwidth 46Gbps/slot Fabric
ASICs
Require 3 fabrics for N+1
redundancy
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Access to Fabric Bandwidth
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
What Are VOQs?
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Benefits of Central Arbitration with VOQ
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Agenda
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Layer 2 Forwarding
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Layer 2 Forwarding Architecture
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Hardware Layer 2 Forwarding Process
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
IP Forwarding
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
IP Forwarding Architecture
Routing protocol processes learn routing
information from neighbors Supervisor Engine
IPv4 and IPv6 unicast RIBs calculate
routing/next-hop information
BGP OSPF ISIS RIP EIGRP
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
IPv4 FIB TCAM Lookup
Generate TCAM lookup key
(destination IP address)
Ingress
Generate
unicast IPv4
Lookup Key
packet header
Compare
10.1.1.10 lookup
key Flow Forwarding Engine
Data
10.1.1.2 Index, # next-hops Next-hop 1 (IF, MAC)
Load-Sharing
10.1.1.3 Index, # next-hops Hash Next-hop 2 (IF, MAC)
10.1.1.4 Index, # next-hops
10.10.0.10 Index, # next-hops
Offset
10.10.0.100 Index, # next-hops Next-hop 3 (IF, MAC)
10.10.0.33 Index, # next-hops
Return
10.1.1.xx
10.1.2.xx Index, # next-hops lookup
result
10.1.3.xx Index, # next-hops Next-hop 4 (IF, MAC)
# next-
10.10.100.xx Index, # next-hops
hops
Next-hop 5 (IF, MAC)
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Displaying Routing
and Forwarding Information
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Displaying Routing
and Forwarding Information (Cont.)
n7010# sh routing ipv4 10.100.7.0/24
------------------+------------------+---------------------
------------------+------------------+---------------------
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Identifying the ECMP Path for a Flow
show routing [ipv4|ipv6] hash <sip>
<dip> [<sport> <dport>] [vrf <vrf>]
n7010# sh routing hash 192.168.44.12 10.200.71.188
load-share type: 1
n7010#
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
IP Multicast Forwarding
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
IP Multicast Forwarding Architecture
Multicast routing processes learn routing Supervisor Engine
information from neighbors/hosts
PIM IGMP PIM6 ICMP6 BGP MSDP
IPv4 and IPv6 multicast RIBs calculate
multicast routing/RP/RPF/OIL information MRIB/M6RIB
Multicast Forwarding Distribution Manager
(MFDM) interfaces between MRIBs on MFDM
supervisor and IP FIB on
I/O modules
IP FIB process programs hardware:
FIB TCAM IP FIB IP FIB IP FIB
Adjacency table
Multicast Expansion Table (MET) Hardware Hardware Hardware
n7010# sh processes cpu | egrep pim|igmp|PID
I/O Module I/O Module I/O Module
PID Runtime(ms) Invoked uSecs 1Sec Process
3842 109 32911620 0 0 pim
3850 133 33279940 0 0 igmp
n7010# sh processes cpu | egrep m.?rib
3843 177 33436550 0 0 mrib Hardware MET
3847 115 47169180 0 0 m6rib
n7010# sh processes cpu | egrep mfdm FIB TCAM ADJ Table
3846 2442 743581240 0 0 mfdm
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Hardware Programming
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Multicast FIB
TCAM Lookup Generate TCAM
lookup key (source
and group IP address)
Ingress
multicast Generate
packet header Lookup Key
10.1.1.10, 239.1.1.1
Forwarding Engine
Compare
lookup
key 10.1.1.12, 239.1.1.1 RPF, ADJ Index MET Index
10.1.1.10, 232.1.2.3 RPF, ADJ Index MET Index
10.4.7.10, 225.8.8.8 RPF, ADJ Index MET Index
HIT! 10.1.1.10, 239.1.1.1 RPF, ADJ Index MET Index
Adj Index Result
10.6.6.10, 239.44.2.1 RPF, ADJ Index MET Index
FIB TCAM FIB DRAM Adjacency Table
Hit in FIB Return
Identifies multicast
returns result lookup
adjacency entry
in FIB DRAM result
Replication
Engine OIFs
OIFs
Replication for OIFs
each OIF in Replicate
MET block OIFs MET index used
to find OIFs for
MET replication
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Displaying Multicast Routing
and Forwarding Information
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Displaying Multicast Routing and
Forwarding Information (Cont)
n7010# sh routing multicast 10.1.1.2 239.1.1.1
IP Multicast Routing Table for VRF "default"
n7010# sh forwarding ipv4 multicast route group 239.1.1.1 source 10.1.1.2 module 9
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Egress Replication
IIF
Local
OIF
Distributes multicast replication load
among replication engines of all I/O Module 1
Replication
modules with OIFs Engine
MET
the replication
2 3 4
For OIFs on other modules, Fabric ASIC Fabric ASIC Fabric ASIC
ingress replication engine
replicates a single copy
of packet over fabric to Replication
Engine
MET
Replication
Engine
MET
Replication
Engine
MET
those egress modules
Each egress forwarding engine
performs lookup to drive replication
Replication engine on egress Local Local Local
OIFs OIFs OIFs
module performs replication for
local OIFs
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
HDR = Packet Headers DATA = Packet Data
Multicast traffic
Locally arbitrated
Load balanced over available fabric channels using hash
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Agenda
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Security ACLs
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Security ACL
ip access-list example
xxxxxxx| 10.1.2.100
xxxxxxx | 10.2.2.2 | |xx
xx| xxx
| xxx| xxx
| xxx Permit
xxxxxxx | 10.1.68.44 | xx | xxx | xxx Deny
xxxxxxx | 10.33.2.25 | xx | xxx | xxx Deny
xxxxxxx xxxxxxx| |06
xxxxxxx | xxxxxxx 06| |xxx
xxx| 0050
| 0016 Permit
X=Mask
xxxxxxx | xxxxxxx | 06 | xxx | 0017 Deny
xxxxxxx | xxxxxxx | 11 | xxx | 0202 Deny
Result affects
HIT! xxxxxxx | xxxxxxx | 06 | xxx | 0050 Permit final packet
Result handling
xxxxxxx | xxxxxxx | 11 | xxx | 00A1 Permit
CL TCAM Hit in CL TCAM CL SRAM
returns result in Return
CL SRAM lookup
result
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Displaying Classification Resources
show hardware access-list resource utilization module <mod>
n7010# sh hardware access-list resource utilization module 9
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Quality of Service
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
QoS Architecture
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Port QoS 8-Port 10G Module
Buffers
96MB ingress per port
80MB egress per port
Queue Structure
8q2t ingress
1p7q4t egress
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
8-Port 10G Module Buffering
Replication
Ingress Engine
1
Egress
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Port QoS 32-Port 10G Module
Buffers
Ingress (two-stage ingress buffering)
Dedicated mode: 1MB per port + 65MB per port
Shared mode: 1MB per port + 65MB per port group
Egress
Dedicated mode: 80MB per port
Shared mode: 80MB per port-group
Queue Structure
8q2t + 2q1t ingress
1p7q4t egress
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
32-Port 10G Module Buffering Shared Mode
Replication
Engine
Ingress
(Fixed) 12 12345678
Ports 1,3,5,7 Port 1,3,5,7
65MB 80MB
2q1t Port ASIC 1p7q4t
Port 3
1MB
Port 1
12345678 1MB
4:1 Mux
8q2t Port 7
1MB
Port 5
1MB
1,3,5,7
Egress
Port Group
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
32-Port 10G Module Buffering Dedicated
Mode
Replication
Engine
Ingress
(Fixed) 12 12345678
Port 1 Port 1
65MB 80MB
2q1t Port ASIC 1p7q4t
Port 1
12345678 1MB
4:1 Mux
8q2t
1,3,5,7
Egress
Port Group
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Port QoS 48-Port 1G Modules
Buffers
7.56MB ingress per port
6.15MB egress per port
Queue Structure
2q4t ingress
1p3q4t egress
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
48-Port 1G Modules Buffering
Ingress
Replication
Port 10
Engine 6.2MB
Port 10 PortPort
7 4
7.6MB 6.2MB
6.2MB
PortPort
7 4 Port 1
6.2MB Port 11
7.6MB
7.6MB 6.2MB
1234
Port 1
12 7.6MB Port 11 PortPort
8 5
7.6MB 6.2MB
Port 2
6.2MB 1p3q4t
2q4t PortPort
7.6MB
8 5
Port ASIC 6.2MB Port 12
7.6MB 6.2MB
Port 2 PortPort
9 6
7.6MB Port 12
7.6MB 6.2MB
6.2MB
PortPort
9 6 Port 3
1-12 6.2MB
7.6MB
7.6MB
Port 3
7.6MB
Egress
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Marking and Policing
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
QoS Classification ACLs
ip access-list police
10.1.1.xx | xxxxxxx | 11
06 || xxx
xxx|| xxx
xxx Remark DSCP 32
Result affects
HIT! 10.1.1.xx | xxxxxxx | 06 | xxx | xxx Remark DSCP 40 final packet
Result handling
xxxxxxx | 10.5.5.xx| 06 | xxx | 0017 Remark IP Prec 3
CL TCAM CL SRAM
Hit in CL TCAM Return
returns result in lookup
CL SRAM result
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Monitoring QoS Service Policies
show policy-map interface [[<interface>] [type
qos|queuing]]|brief]
n7010# show policy-map interface e9/1
Global statistics status : enabled
Ethernet9/1
Chassis Architecture
Supervisor Engine Architecture
I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
Layer 2 Forwarding
IP Forwarding
IP Multicast Forwarding
ACLs
QoS
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
NetFlow
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
NetFlow Architecture
Supervisor Engine
NetFlow manager
CLI XML
receives configuration via
CLI/XML NetFlow Manager
NetFlow manager
distributes configuration
to NetFlow-Clients on NF-C NF-C NF-C
I/O modules Hardware Hardware Hardware
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Viewing NetFlow Records
show hardware flow ip [detail] module <mod>
n7010# sh hardware flow ip interface e9/1 module 9
D - Direction; IF - Intf/VLAN; L4 Info - Protocol:Source Port:Destination Port
TCP Flags: Ack, Flush, Push, Reset, Syn, Urgent
To NetFlow Collector
I/O Module
Supervisor LC NetFlow
Fabric Engine CPU Table
Hardware
ASIC Aged Flows Flow Creation
Forwarding
via Inband
Engine
VOQs
I/O Module
LC NetFlow
CPU Table
Main Switched Hardware
CPU EOBC Aged Flows Flow Creation
Forwarding
Engine
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Q and A
Presentation_ID 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
Complete Your Online
Session Evaluation