BRKDCT 1890

Network Visibility using Advanced
Analytics in Nexus Switches

BRKDCT-1890
Karishma Gupta (kargupta@cisco.com)

@karismagupta
Technical Marketing Engineer
BRKDCT-1890 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Abstract
• Session ID : BRKDCT – 1890
• Title : Network visibility using advanced Analytics in Nexus switches
• Abstract : Learn how to get the most visibility from your Nexus-based network
with new monitoring capabilities and advanced enhancements to traditional
features like SPAN, ERSPAN and Netflow to quickly pinpoint trouble spots in the
network
Session Goal
• Create awareness of the Analytics and Monitoring tools
available in the Nexus family (3k, 5K, 7K, 9K)
• Provide the ability to choose the right tool to analyze, which

helps in timely resolution of the problem
• It will NOT focus on other management aspects like SNMP,

Syslog, RMON, troubleshooting, QOS, architecture and
packet flows
 Reference Slide
Agenda
• Introduction
• Quick Product Overview
• Advanced Visibility
• SPAN/ ERSPAN
• Flexible Netflow
• Conclusion
Network Network Bandwidth Explosion
4G Mobile
IP Traffic Cloud Video M2M
Adoption
Trillions of new
Global IP traffic Global cloud 4G will account
By 2017, the “connected
will grow 3X to traffic will grow for 45% of
world will reach events” will
1.4 zettabytes 6X by 2016 global mobile
3 trillion Internet occur over IP
annually by data traffic
video minutes networks
2017 per month throughout the
next decade
If not handled well..
Degrading performance
Difficulty to troubleshoot
Improper planning of
resources
Studying past historical
What is Analytics? data to research potential
trends
Discovery and The systematic

communication of computational analysis
meaningful patterns in of data or statistics
data statistics.
Advanced Analytics on Nexus Switches
• Collection of various features and enhancements to the traditional monitoring

tools
• Latency Monitoring, Buffer Monitoring, SPAN-on-drop, Exception SPAN, SPAN

filters, Microburst Monitoring and a LOT MORE!
• Advantages: Microbursts, Congestion, find malicious source, filter SPAN

packets etc..
Agenda
• Introduction
• Quick Product Overview
• SPAN / ERSPAN
• Conclusion
Nexus Family NEW!
Application Centric
Infrastructure (ACI)
Nexus 9000
Nexus 6000 APIC*
Nexus 2300 Nexus 7000 Nexus 7700 ACI
Nexus 5600
Nexus 3100
Nexus
1000V Nexus 2000
Nexus 3000
Nexus 5000
Nexus 3500
Agenda
• Introduction
• Quick Products Overview
• SPAN / ERSPAN
• Conclusion
Latency monitoring
Why do we need to correct latency problems?
Many applications can get impacted because

of high latency –
Website download
Video streaming
Video conferencing
Online gaming
Banking
Airline reservation
Stock Market
Web hosting
How does Latency Monitoring work?
Packet
Packet Time T1 INGRESS TIMESTAMPING
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12 17 18 19 20
N5K-C56-72UP
ID
5 6 7 8 13 14 15 16 21 22 23 24
STAT
EGRESS TIMESTAMPING Packet Time T2
Packet
Latency Monitoring Feature measure: T2 – T1 in ns

Modes of Latency monitoring
Instantaneous - Enabled by default on all pairs of ports. No configuration
switch# show hardware profile latency monitor interface e1/23 interface e1/22
---------------------------------------------------------------------------
Ingress Port | Egress Port | Minimum | Maximum | Average
---------------------------------------------------------------------------
Ethernet1/22 | Ethernet1/23 | 856 | 1208 | 901 |
----------------------------------------------------------------------------
Modes of Latency Monitoring
Custom histogram – Counts packets in defined range. Needs below configurations
switch(config)# interface e1/3

switch(config-if)# packet latency interface e1/1 mode custom low-latency 800 high-
latency 10000
switch# show hardware profile latency monitor interface e1/3 interface e1/1
--------------------------------------------------------------------------------
| Egress Interface : Ethernet1/3 Ingress Interface : Ethernet1/1 |
--------------------------------------------------------------------------------
| Range | 800 <= Latency < 10000 | Outside the first range |
-------------------------------------------------------------------------------
| counter | 3542903 | 56792 |
Microburst monitoring
Microburst – A Concern
• Spike of high activity
• Passes under the radar of traditional load-monitoring tools
• Traffic spike that causes that system to saturate
• How short and how high? – Capacity of worst system in N/W
Challenge: It’s Very Hard to see Microbursts
5672# show interface ethernet 1/2
Ethernet1/2 is up
[…]
Last clearing of "show interface" counters 00:00:58
0 interface resets
30 seconds input rate 96315720 bits/sec, 1331 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 77.00 Mbps, 1.05 Kpps; output rate 0 bps, 0 pps
RX
200000 unicast packets 0 multicast packets 0 broadcast packets
200000 input packets 1800000000 bytes
200000 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
[…]
Solution: Burst Monitoring
Configure your own burst filter per port per direction
Provides counters and syslog messages
For interface burst counters
burst threshold {ingress | egress} {limit percent | size max_bytes} interval interval_time
For Syslogs
[no] burst maximum {ingress | egress} burst-count max-burst
How to see bursts?
switch# show interface burst-counters
--------------------------------------------------------------------
| Interface | Ingress Bursts | Egress Bursts | Total Bursts |
--------------------------------------------------------------------
| Ethernet1/1 | 10 | N/A | 10 |
| Ethernet2/1 | 15 | 0 | 15 |
Buffer monitoring
Why do we need to monitor buffers?
Can I add a new server?
Will the performance be impacted?
Why are the drops happening?
Is my network congested?
What is Buffer monitoring on Nexus?
• Buffer utilization is on a per port basis
• Buffer utilization shows buffer for unicast traffic in ingress and unicast and multicast in
egress directions
• Histogram mode – slow (1sec) or fast (250ms) sampling
Configuration
• Buffer utilization histogram must be enabled on interface.
[no] hardware profile buffer monitor
• Fast sampling must be enabled in global configuration mode

[no] hardware profile buffer monitor sampling fast
• To see buffer utilization histogram, the next command must be executed

show hardware profile buffer monitor { interface <ifid> | all } history {
brief | detail }
• To clear buffer utilization history use

clear hardware profile buffer monitor [ interface <ifid> ]
Output of Buffer Monitoring tool
switch# show hardware profile buffer monitor interface ethernet 1/21 history brief
--------------------------------------------------------------------------------
Interface : Eth1/21
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Sampling Mode : Slow (1 second)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Ingress Buffer Utilization Detected(in KB)
Per asic Ingress Total Usage (15.628800MB)
--------------------------------------------------------------------------------
1 sec | 5 sec | 1 min | 5 min | 1 hour |
--------------------------------------------------------------------------------
0.6| 0.6| 0.6| 0.6| 0.6|
--------------------------------------------------------------------------------
Egress Buffer Utilization Detected(Unicast|Multicast)(in KB)
Per asic Egress Total Usage (8.611850MB)
--------------------------------------------------------------------------------
1 sec | 5 sec | 1 min | 5 min | 1 hour |
--------------------------------------------------------------------------------
112.6| 0.0| 177.2| 0.0| 158.0| 0.0| 164.1| 0.0| 164.3| 0.0|
--------------------------------------------------------------------------------
Real World Example Impacted
Application not responding application
WAN
Identify Packet Drops
switch# show interface ethernet 1/1
<snip>
30 seconds input rate 96315720 bits/sec, 1331 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 5.20 Gbps; output rate 0 bps, 0 pps
<snip>
RX
<snip>
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 235847488 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
Real World Example
Application not responding
Problem Zone
Use burst monitoring to detect Micro bursts in leaf switches
2014 Dec 5 01:13:23 %US switch %$ VDC-1 %$ R-2-SYSTEM_MSG: Micro Burst

has been detected on ingress side on Ethernet1/1 - bigsurusd
Agenda
• Introduction
• SPAN / ERSPAN
• Conclusion
“You can not correct what you do not see!”

Switch Port Analyzer (SPAN)
 A SPAN session is an association of SPAN all the packets
source ports/vlans to one or more ingressing e1/1
destination ports
 Once the traffic is identified for Host A
e2/1 Host B
replication, switch copies the matching
e1/1 e3/1
traffic to the SPAN destination port(s)
 The SPAN (copied) packets are created
in hardware without overloading the SPAN Source
CPU Spanned
(copied)
SPAN Destination traffic
Sniffer Device
Encapsulated Remote SPAN (ERSPAN)
 ERSPAN supports source
and destinations on different
Packets are replicated and At ERSPAN Destination device,
switches GRE encapsulated at GRE packet is decapsulated
ERSPAN source device
 It uses a GRE tunnel

Sniffer Device
to carry traffic
Layer 3
Network
 Packets replicated in
hardware
ERSPAN Source ERSPAN Destination
ERSPAN with IEEE1588 timestamp – Find Network Latency
GPS
PTP messages
Data
ERSPAN type III PTP grandmaster
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12 17 18 19 20
N5K-C56-72UP
ID
STAT 5 6 7 8 13 14 15 16 21 22 23 24
Switch A
Latency from Switch A

to Switch N = T2 – T1
N5K-C56-72UP
ID
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12 17 18 19 20
Switch N
5 6 7 8 13 14 15 16 21 22 23 24
STAT
Nexus 5600/6000 SPAN Features
Packet drops !!
5672(config) # show internet ethernet
1/5
Ethernet1/3 is UP
<snipped>
RX
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
0 watchdog 0 bad etype drop 0 bad

1 2 3 4 9 10 11 12 17 18 19 20
N5K-C56-72UP
ID
proto drop 0 if down drop
5 6 7 8 13 14 15 16 21 22 23 24
STAT
0 input with dribble 2675837 input

discard
0 Rx pause
TX
0 unicast packets 0 multicast packets
0 broadcast packets
What packets
are dropped?
SPAN-on-Drop
Port 3 is
Tail-Drop
Ingress congested
Data Buffer
Destination
SPAN-On-Drop
Monitoring
SPAN Buffer Station
Nexus 5600
SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014
SPAN-on-Drop
• Works for unicast packets only
• Supports both local SPAN and ERSPAN
• One SPAN-on-Drop session is supported
• Can have multiple source ports, and multiple destination ports
• Source port(s) can be a part of a SPAN-on-Drop session, and a local SPAN session
simultaneously
SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014
SPAN-on-Drop
The source interface is the ingress port for which we want to monitor drops.
switch(config)# monitor session 1 type span-on-drop

Always Rx : Ingress
switch(config-span-on-drop)# source interface e1/1 “rx” interface – Packets
dropped at ingress
switch(config-span-on-drop)# source interface e1/2 “rx”
switch(config-span-on-drop)# destination interface e1/4
switch(config)# monitor session 2 type span-on-drop-erspan

switch(config-span-on-drop-erspan)# source interface e1/1 “rx”
switch(config-span-on-drop-erspan)# destination ip 100.1.1.2
SPAN-on-Latency – Identify delayed flows
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12 17 18 19 20
N5K-C56-72UP
ID
What took so
STAT 5 6 7 8 13 14 15 16 21 22 23 24
long?
Connected to
SPINE
SPAN-on-Latency
Latency Monitoring:
Gives port-port latency
1 T1-T0
33
2 T0 34
3 35
4 36
T3-T2
5 37 If Latency Threshold > 10 usec:
T2 SPAN to 1/64
6 38
… …
Monitoring
32 64 Station
Timestamp Nexus 5600

Packet
SPAN-on-Latency Configuration
monitor session 1 type span-on-latency Always Tx: packets egressing on 1/3

(any source) with latency >10us will
source interface Ethernet1/35 tx be replicated to the SPAN dest 1/4
destination interface Ethernet1/64
interface Ethernet1/35
packet latency threshold 10001
interface Ethernet1/64
switchport mode monitor
SPAN-on-Latency
• Support for one SPAN-on-latency session
• Multiple sources can be configured – latency threshold is per SPAN-on-drop TX

source port
• A SPAN-on-Latency source port cannot be in another SPAN session
Real World Example
Problem - Slow Download Rate
Monitored - Errors on interface and CPU usage
Eth1/7 Eth1/14
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12 17 18 19 20
N5K-C56-72UP
ID
5 6 7 8 13 14 15 16 21 22 23 24
STAT
Analytics :
May be congestion? Buffer monitoring ✗
SVI 572
10.5.72.1/24
0000.0c9f.f23c
10.5.72.72 10.5.72.155
547f.ee36.e841 547f.ee35.e001
Find which application is impacted - SPAN-on-
Latency ✓
monitor session 1 type span-on-latency
source interface Ethernet1/14 tx Set Latency threshold
destination interface Ethernet1/21
Real World Example
Root cause -
In SPAN, found server subnet mask
incorrectly set to /25 instead of /24
Eth6/7 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12
Eth6/14
17 18 19 20
N5K-C56-72UP
ID
5 6 7 8 13 14 15 16 21 22 23 24
STAT
Fix -
[root@Car ~]# ifconfig eth0 | grep "inet addr"
Update server subnet mask inet addr:10.5.72.72 Bcast:10.5.72.127 Mask:255.255.255.128
Server Client
SVI 572
10.5.72.1/24
0000.0c9f.f23c
10.5.72.72
547f.ee36.e841 10.5.72.155
547f.ee35.e001
These hosts are in the same VLAN yet the Server (10.5.72.72) is sending traffic destined
to the gateway’s MAC address
SPAN with ACL filter
• Selectively monitor traffic in a SPAN session using Access-control list (ACL)
• SPAN session ignores any permit/deny actions specified in the ACL
• SPANs packets that match the ACL filter criteria
switch(config-acl)# sh access-lists
IPV4 ACL acl-ip-01

10 permit ip 10.1.1.1/24 any
switch(config)# monitor session 1

switch(config-monitor)# source interface ethernet 3/1
switch(config-monitor)# destination interface ethernet 3/2
switch(config-monitor)# filter access-group acl-ip-01
switch(config-monitor)# no shut
Nexus 7000/9000 SPAN Features
SPAN VLAN Filters
VLAN filters allow monitoring subset of VLANs on trunk ports
Filter specifies list of VLANs to capture
Traffic for other VLANs not sent to SPAN destination
n7010(config)# monitor session 1

n7010(config-monitor)# source int e 2/4
n7010(config-monitor)# dest int e2/1
n7010(config-monitor)# filter vlan 55,56
SPAN filtering
ACL FILTERS
switch(config)# ip access-list match_11_pkts

switch(config-acl)# permit ip 11.0.0.0 0.255.255.255 any
switch(config)# vlan access-map span_filter 5
switch(config-access-map)# match ip address match_11_pkts
switch(config-access-map)# action forward
switch(config-erspan-src)# filter access_group span_filter
ACL Capture
Selectively monitor traffic on an interface or VLAN
Packets that match ACL rule are permitted or denied and/or sent to an alternate
destination
switch(config)# monitor session 1 type acl-capture
switch(config)# destination interface ethernet 1/1
ACL Capture
Enable a capture session for an ACL's access control entries (ACEs) and then apply the ACL to an interface:
switch(config)# ip access-list acl1
switch(config-acl)# permit tcp any any capture session 1
switch(config)interface ethernet 7/1
switch(config-if)# ip access-group acl1 in
Enable a capture session for the whole ACL and then apply the ACL to an interface:
switch(config)# ip access-list acl1
switch(config-acl)# capture session 1
switch(config)# interface ethernet 7/1
switch(config-if)# ip access-group acl1 in
Inband SPAN – Monitor control traffic
• Supervisor CPU sends/receives traffic via dedicated interface to Fabric using
INBAND interface

switch(config-monitor)# source interface sup-eth 0
• Monitoring direction is from perspective of switch fabric, not CPU

Tx SPAN monitors traffic from switch fabric to CPU
Rx SPAN monitors traffic from CPU to switch fabric
INBAND SPAN capture
Real world Example
High CPU – Use INBAND SPAN to find out!
N7k# show processes cpu sort

CPU utilization for five seconds: 100%/100%; one minute: 99%; five minutes:98%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
6131 11367100 1497150 7 78.02% 77.12% 76.35% - X
5615 44622720 3059816 14 15.121% 14.13% 14.59% - Y
Rule Based SPAN
Filter applied selectively on a session results in desired subset of traffic
Filter by L2/L3/L4 fields
• MAC address
• Ether type
• VLAN
• IP address
• L4 protocol
• ToS
• CoS/VL
• Frame type (IPv4, IPv6, FCoE, ARP/RARP)….
Simple Filter Configuration
Configure a filter within the session configuration mode
monitor session 1
All fields
source interface e1/1 AND’ed
destination interface e2/1
filter vlan 10, 20
filter frame-type ipv4 src-ip 10.1.1.1/24 dest-ip 20.1.1.1/24
no shut
FAQ: So many filters for SPAN! Which should I use?
GOAL IS SAME i.e SPAN WHAT I WANT TO ..
Nexus 5600/6000 :
ACL filters for SPAN (Use Access lists to filter SPAN)
Nexus 7000/7700 :
VLAN filters (Filter by VLAN)
Rule based SPAN (Filter by L2/L3/L4 fields)
Nexus 3100/9300/9500 :
VLAN filters (Filter by VLAN)
ACL filters (Use Access lists to filter SPAN)
SPAN rate limiting
• Limits the number of SPAN copies made on ingress
• In manual mode, the rate limit will be in 1-100 range, i.e., 1%, 2%, 3% …100% of 10G
SPAN rate
• In auto mode, the rate limit will automatically calculated as follows:
Rate limit = Dest. Bandwidth / Source Bandwidth
Command
N7k(config-monitor-local)# [no] rate-limiter [auto | manual [1..100]]
(Local SPAN + ERSPAN supported on F-series LCs)
Exception SPAN
• Exception SPAN enables you to span exception packets. Packets that have failed an intrusion detection
system (IDS) & Layer 3 IP verification
• Rate limiters, MTU truncation, and sampling are supported in the exception SPAN session
• Each VDC supports one exception SPAN session
• Exception SPAN is supported in the Tx direction only
switch# configure terminal

switch(config-monitor)# source exception all
switch(config-monitor)# destination interface ethernet 2/5
switch(config-monitor)# no shut
Exception which lead to SPAN
Exceptions Brief explanation
No route in hardware This is seen when adjacency is not yet formed
Unicast/Multicast route error (incoming/outgoing This is seen when the outgoing interface is not
interface) available (say, when the LC is reloaded)
Multicast DF failure Seen when the designated forwarder is not available.
SMAC IP check failure Incorrect SMAC / DMAC combinations, like multicast

SRC MAC or SRC.IP = DST.IP or SRC.IP is a
broadcast address or DST.IP is all zeros
Protocol field failure Incorrect IP protocol specified in the IP header

FCS / CRC errors Errors related to icnorrect FCS or CRC
Exception which lead to SPAN
Exceptions Brief explanation
TTL expiry When the number of hops in the header exceeds TTL
configured
SPAN replication before L2/L3 ACL deny If the copy is made before the decision engine takes
a decision, it is Ingress replication.
IPV6 scope check fail Seen when there are multiple link-local addresses
tied to an interface and the route does not exist for
the packet through either one of them,.
MTU fail When pkt size exceeds the link MTU
Stale adjacency When the adjacency does not exist / is not updated
for a long time / fails refresh
CoPP violations Any packets that violated CoPP rate-limits
Exception SPAN – Verify CLI
Nexus7000(config)# show hardware ip verify
IPv4 IDS Checks Status Packets Failed
-----------------------------+---------+------------------
address source broadcast Enabled 65536
address source multicast Enabled 65536
address destination zero Enabled 65536
address identical Enabled 65536
checksum Enabled 768
protocol Enabled 0
fragment Enabled 0
length minimum Enabled 0
length consistent Enabled 0
length maximum max-frag Enabled 0
length maximum max-tcp Enabled 0
tcp flags Enabled 0
tcp tiny-frag Enabled 0
version Enabled 0
Real world Example
CRC errors
Use Exception SPAN –

Find reason for drop and
what was dropped!
Packets coming
into interface were Packet dropped in hardware–
mishandled by Packet which came in didn't
TRANSCEIVER
leading to CRC make it to the egress
errors
 Receive
packet
from wire
Nexus 5000-7000 SPAN
SPAN Features Nexus 5600 Nexus 7000 Nexus 7700
ERSPAN destination session Yes All except F1(F3*) All LC’s(F3*)
Prioritize data over SPAN Yes Yes (F2E/F3/M1/M2) Yes (F2E/F3)
Line-rate SPAN throughput Yes No No
ERSPAN with 1588 PTP timestamp Yes * M2/F2/F2E/F3 F2E/F3
Multi-destination SPAN Yes All M series LC’s N/A
Number of SPAN destinations 16 32 N/A
SPAN with MTU truncation Yes Yes (Except M1) Yes
Virtual SPAN Yes Yes Yes
ACL filters Yes Rule based SPAN Rule based SPAN
SPAN source as VLAN Receive only Bidirectional Bidirectional

*Software roadmap feature
Nexus 3000/9000 SPAN
SPAN Features Nexus 3100 Nexus 9300 Nexus 9500
SPAN source as VLAN Receive only Receive only Receive only
ERSPAN destination session Yes No No
ERSPAN with V2 header Yes No No
Prioritize data over SPAN Yes Yes Yes
Line-rate SPAN throughput Yes No No
ERSPAN with 1588 PTP
No Yes * No
timestamp
Multi-destination SPAN Yes No No
Number of SPAN destinations

1 1 1
per session
ACL filters for SPAN Yes Yes Yes
BRKDCT-1890 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 100 *Software roadmap feature
Agenda
• Introduction
• SPAN / ERSPAN
• Conclusion
“Flexible” NetFlow ( Nexus 5k – 7k )
• Enhanced network anomaly
• Customized user configurable flow
• Monitor a wider range of packet information
NetFlow = Visibility
A single NetFlow Record provides a wealth of information
switch# show flow monitor MONITOR-1 cache
…
IPV4 SOURCE ADDRESS: 192.168.100.100
IPV4 DESTINATION ADDRESS: 192.168.20.6
TRNS SOURCE PORT: 47321
TRNS DESTINATION PORT: 443
INTERFACE INPUT: E1/1
IP TOS: 0x00
IP PROTOCOL: 6
ipv4 next hop address: 192.168.20.6
tcp flags: 0x1A
interface output: Gi0/1.20
counter bytes: 1482
counter packets: 23
timestamp first: 12:33:53.358
timestamp last: 12:33:53.370
ip dscp: 0x00
ip ttl min: 127
ip ttl max: 127
application name: nbar secure-http
…
Seven Steps of Flow Creation
1 Packet I/O Module
2 Extract relevant fields 3 Flow

Flow 4
Statistics
Statistics
DMAC SMAC VLAN Ethertype
Flow Statistics
001E:A12D:128
000A:ABCD:00EF
7
16 0x86DD Flow Statistics
Flow Statistics
6 Formatted
into
I/O module collects
the flows and their statistics
5
NetFlow once the flow ages out
Export
Collector
7
Full vs. Sampled NetFlow
• NetFlow collects full or sampled flow data
• Full NetFlow: Accounts for every packet of every flow on interface
– Available on M-Series modules only on Nexus 7000
– Flow data collection up to capacity of hardware NetFlow table
• Sampled NetFlow: Accounts for M in N packets on interface
– Available on both M2 (ingress/egress) and F2E/F3 (ingress only) in Nexus 7000
and Nexus 5600
– M2: Flow data collection up to capacity of hardware NetFlow table
– F3: Flow data collection for up to ~1000/3000pps per module
– F3 (future): Increased per-module sampling rate leveraging on-board Fabric Services
Accelerator (FSA) complex
– Nexus 5600: Flow data collection for up to ~120kpps per chassis
NetFlow - Traffic statistics
Configuration
flow record SAMPLE-FLOW flow exporter SAMPLE-EXPORT-1
match ipv4 source address description SAMPLE FnF v9 Exporter
match ipv4 destination address destination 11.1.1.1 use-vrf management
KEY
match transport source-port source Loopback0
match transport destination-port transport udp 2055
collect counter bytes flow exporter SAMPLE-EXPORT-2
collect counter packets description SAMPLE FnF v9 Exporter
NON-KEY collect timestamp sys-uptime first destination 12.1.1.1 use-vrf management
collect timestamp sys-uptime last transport udp 2055
Steps:
flow monitor SAMPLE-MONITOR 1. Create Flow Record
description SAMPLE FnF v9 Monitor 2. Create Flow Exporter
record SAMPLE-FLOW
exporter SAMPLE-EXPORT-1 3. Associate Record and
exporter SAMPLE-EXPORT-2 Exporter to a Flow
Monitor
4. Apply to the interfaces
interface eth 1/1 interface eth 2/1
ip address 172.16.0.1 255.255.255.0 ip address 172.16.1.1 255.255.0
ip flow monitor SAMPLE-MONITOR input ip flow monitor SAMPLE-MONITOR input
ip flow monitor SAMPLE-MONITOR output ip flow monitor SAMPLE-MONITOR output
Use Case – Resolving High CPU using FnF
Nexus# show processes cpu sort
CPU utilization for five seconds: 65%/8%; one minute: 63%; five minutes: 61%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
310 30544 189234 81 47.12% 45.11% 45.23% 0 IP Input High CPU due to process “IP Input”
Nexus(config)#flow RECORD FnF-Receive-record

Nexus(config-flow-record)#match ipv4 source address Building a FnF record, matching L3 and L4
Nexus(config-flow-record)#match ipv4 destination address parameters (key fields) and collecting details on
Nexus(config-flow-record)#match transport source-port Input interface and packet count (non-key fields)
Nexus(config-flow-record)#match transport destination-port
Nexus(config-flow-record)#collect counter packets
Nexus(config-flow-record)#exit
Nexus(config)#flow MONITOR FnF-Receive Associating the FnF record to a monitor.

Nexus(config-flow-monitor)#record FnF-Receive-record Here, you can add an option (not enabled
here) to export the data to the collector
Nexus(config-flow-monitor)#exit
Nexus(config)#control-plane Applying to the control-plane

Nexus(config-cp)#ip flow monitor FnF-Receive interface
Nexus(config-cp)#exit
Use Case - Monitoring Control-Plane traffic using
FnF
Nexus(config)# show flow monitor FnF-Receive cache detailed
Source: 10.1.1.1, Destination: 20.1.1.1

Transport source port: 0, Transport
First flowdestination port:
with high number of 0
Input interface: 0x0, Output packets
interface: 0x1a109000
hitting the CPU
Packet count: 122542, Byte count : 105336
Start time: 576346e0, End time : 5763a888
protocol : 0x1, TOS : 0x0, TCP flags: 0x0
Active timer expire in 1766 (secs)
Idle timer expire in 8 (secs)
After few seconds…

Nexus(config)# show flow monitor FnF-Receive cache detailed
Source: 10.1.1.1, Destination: 20.1.1.1

Counters are
Once the flow is identified, further action Transport source port: 0, Transport
increasing destination port: 0
could be (1) blocking the flow with an Input interface: 0x0, Output interface: 0x1a109000
Access List (ACL) or (2) rate-limiting it Packet count: 124791, Byte count : 105336
Start time: 576346e0, End time : 5763a888
using Control Plane Policing (CoPP) protocol : 0x1, TOS : 0x0, TCP flags: 0x0
depending on the criticality of the flow to Active timer expire in 1766 (secs)
the production. Idle timer expire in 8 (secs)
NetFlow
NetFlow collects flow data for packets traversing the switch
M2 (N7000) F3 (Nexus 7x00) Nexus 5600
Per-interface NetFlow Yes Yes Yes
NetFlow direction Ingress/Egress Ingress only Ingress only
Full NetFlow Yes No No
Sampled NetFlow Yes Yes Yes
FSA Assist for Sampled NetFlow No F3 only (future) No
Bridged NetFlow Yes Yes Yes
Hardware Cache Yes No No
Software Cache No Yes Yes

512K entries per
Hardware Cache Size N/A N/A
forwarding engine
NDE (v5/v9) Yes Yes Yes
Agenda
• Introduction
• SPAN / ERSPAN
• Conclusion
Tools designed with you in mind
Netflow • Advanced feature rich analytics tools
SPAN • Visibility into the products helping to validate the path-of-the-packet
ERSPAN • Analytics tools can help in isolating problems we see in Datacenters

today
Microburst
monitoring • Reduce the time to resolution of network issues
SPAN-on-drop
Latency mon
ACL Capture
Related Sessions
Session Id Session Name
BRKDCT-2333 Data Center Network Failure Detection
BRKARC-3470 Cisco Nexus 7000/7700 Switch Architecture
BRKARC-2222 Cisco Nexus 9000 Architecture
End-to-End QoS Implementation and Operation with Cisco

BRKDCT-3346
Nexus
Cisco Nexus 5600 and 6000 with Fabric Extender 2000

BRKARC-3452
Switch Architecture
Call to Action
• Visit the World of Solutions for
– Cisco Campus Walk in Labs
– Technical Solution Clinics
• Meet the Engineer
• Lunch time Table Topics
• DevNet zone related labs and sessions
• Recommended Reading: for reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2015
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.
• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations

BRKDCT 1890

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 1890

Uploaded by

Copyright:

Available Formats

Network Visibility using Advanced

Analytics in Nexus Switches

Karishma Gupta (kargupta@cisco.com)

• Session ID : BRKDCT – 1890

• Title : Network visibility using advanced Analytics in Nexus switches

• Provide the ability to choose the right tool to analyze, which

• It will NOT focus on other management aspects like SNMP,

Discovery and The systematic

• Collection of various features and enhancements to the traditional monitoring

• Latency Monitoring, Buffer Monitoring, SPAN-on-drop, Exception SPAN, SPAN

• Advantages: Microbursts, Congestion, find malicious source, filter SPAN

Many applications can get impacted because

Packet Time T1 INGRESS TIMESTAMPING

EGRESS TIMESTAMPING Packet Time T2

Latency Monitoring Feature measure: T2 – T1 in ns

Custom histogram – Counts packets in defined range. Needs below configurations

switch(config)# interface e1/3

• Spike of high activity

• Passes under the radar of traditional load-monitoring tools

• Traffic spike that causes that system to saturate

• How short and how high? – Capacity of worst system in N/W

Provides counters and syslog messages

For interface burst counters

[no] burst maximum {ingress | egress} burst-count max-burst

| Interface | Ingress Bursts | Egress Bursts | Total Bursts |

Will the performance be impacted?

Why are the drops happening?

• Histogram mode – slow (1sec) or fast (250ms) sampling

• Fast sampling must be enabled in global configuration mode

• To see buffer utilization histogram, the next command must be executed

• To clear buffer utilization history use

Use burst monitoring to detect Micro bursts in leaf switches

2014 Dec 5 01:13:23 %US switch %$ VDC-1 %$ R-2-SYSTEM_MSG: Micro Burst

“You can not correct what you do not see!”

 It uses a GRE tunnel

ERSPAN Source ERSPAN Destination

ERSPAN type III PTP grandmaster

Latency from Switch A

0 watchdog 0 bad etype drop 0 bad

0 input with dribble 2675837 input

switch(config)# monitor session 1 type span-on-drop

switch(config)# monitor session 2 type span-on-drop-erspan

Timestamp Nexus 5600

monitor session 1 type span-on-latency Always Tx: packets egressing on 1/3

• Multiple sources can be configured – latency threshold is per SPAN-on-drop TX

• A SPAN-on-Latency source port cannot be in another SPAN session

source interface Ethernet1/14 tx Set Latency threshold

destination interface Ethernet1/21

• SPAN session ignores any permit/deny actions specified in the ACL

• SPANs packets that match the ACL filter criteria

IPV4 ACL acl-ip-01

switch(config)# monitor session 1

Filter specifies list of VLANs to capture

Traffic for other VLANs not sent to SPAN destination

n7010(config)# monitor session 1

switch(config)# ip access-list match_11_pkts

switch(config)# monitor session 1 type acl-capture

switch(config)# destination interface ethernet 1/1

switch(config)# monitor session 1

• Monitoring direction is from perspective of switch fabric, not CPU

N7k# show processes cpu sort

Filter by L2/L3/L4 fields

(Local SPAN + ERSPAN supported on F-series LCs)