Brkarc 2025 PDF

Network State
Awareness and
Troubleshooting
Aamer Akhter / aa@cisco.com

BRKARC-2025
Agenda
• Troubleshooting Methodology
• Packet Forwarding Review
• Control Plane
• Topology
• Logging
• Routing Protocol Stability
• Data Plane
• Active Monitoring
• Passive Flow Monitoring
• QoS
• Getting Started
Keeping Focused: What This Session is About
• This session is about basic network troubleshooting,

focusing on fault detection & isolation
• Some non-Cisco specifics
• For context, we will cover some basic methodologies and

functional elements of network behavior
• This session is NOT about
• Architectures of specific platforms
• Data Center technologies
• This is the 90 min tour. ;-)
BRKARC-2025 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
The Big Picture Internet’s
down.
It’s the
network
Somebody's
downloading
It’s not something.
Application Operator
the (?)
Can’t
network Network Operator
ping it.
Is it Pings
Monday? fine!
Not Server
happy network
Client
Some More (network) Detail
• A lot of stuff going on

• Multiple networks Enterprise
DC
• Multiple applications
Server B
• Multiple layered services Enterprise

WAN Server A
• Mis-information / inconsistency ISP A
Not DNS
happy LAN
Internet
DNS
DHCP
Client
802.1x
… and it keeps on going
• Redundant paths / ECMP / LAG

• Overlays Enterprise
DC
• Load balancers
Server B
• Firewalls Enterprise
WAN Server A
• NATs ISP B
ISP A
Not DNS
happy LAN
Internet
DNS
DHCP
Client
802.1x
Network state awareness?
• What is it:
• View of network, what it is doing, and why
• Monitoring of data network performance,
in comparison with previous working states
• Quick detection of hard failures
• Early warning for
• soft failures
• performance issues
• and tomorrows’ problems
• Faster problem resolution

• Greater confidence in network by users and application operators
Think Like a Network Detective
Find the Suspects Question Suspects Improve

Be Prepared
Control Plane & Data Plane
• Control Plane Gossip from

other routers
Admin Edict
• Processes variety of information

sources and policies, creates Routing show ip bgp
forwarding information base (FIB) Protocol(s)
APIs Statics PfR
show ip ospf
• Best known intention w/o actual

packet in hand Control Plane show ip route
show ip policy
• Data Plane
Int B show ip cef
• The actual forwarding process packet
Int A
Data Plane show mpls forwarding…
show mac address-table
(might be SW or HW based) Int C
• Granted some decision flexibility show policy-map int…

• Driven by arriving packet details, Passive Measurements show interface
traffic conditions etc. show flow monitor
ifmib CbQoS *Flow
Data Plane Decision Flexibility
• Control plane: condenses options driven by policies and (relatively) slower

moving (ms to secs), aggregated information, eg. prefix reachability,
interface state
• Data plane responds to packet conditions
• Destination prefix to egress interface matching
• Multi-path (ECMP / LAG) member selection
• Interface congestion
• QoS class state
• Access Lists
• Packet processing fields (TTL expire, etc)
• IPv4 fragmentation, etc
Network as a System: Independent Decisions
• Each network device makes an independent forwarding decision

• Explicit Local / domain policies
• Device perspective might not be symmetric
• Data plane flexibility
• Asymmetric routing: forward and reverse path are different
• Caused by traffic engineering policies, popular at WAN-edge and admin boundaries
Congested link
R5 is doing
ECMP hash
R3
R1 R2 R6
R5
A B
R4
your network You don’t control
Data Plane and Control Plane Changes
• Change is normal, but some

changes are more interesting:
• Single change that causes loss
of reachability or suboptimal
performance
• Instability: high rate of change
• 3Ws: When, where, and what
Control Plane
3Ws: When, where, and what

What do I have?
• Establish inventory baseline
• Device names, IPs, configuration
<owner/dept>
• Modular HW configuration <device-name>
• Serial # (for support & replacement) <IP address>
• History (where has it been placed) <Contact>
Example device label
• Clearly label devices, ownership

and contact info
• Establish standards for location, <current-location> to
device/port names <destination-location>
<circuit src/dst id>
• Check for changes periodically
Example cable label
(tooling)
How is it wired together?
• Establish network topology baseline • Visual inspection 

• Be prepared to be surprised! • show cdp neighbor
show lldp neighbor
• CDP / LLDP for Layer-2
neighborships
• Traverse spanning-tree blocked, but
not L3
R1 SW1 R2
• Monitor for non-leaf changes
R1#show cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater
Device ID Local Intrfce Holdtme Capability Platform Port ID

SW1 Eth 0 157 T S WS-C3524-XFas 0/0/0
BRKARC-2025 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tools for Topology & Inventory
Management
• Most NMS tools have some element
of inventory and topology
awareness
• Cisco APIC-EM
• Cisco Prime Infrastructure APIC-EM Topology
• NetBrain
• (open source) NetDisco
http://www.netdisco.org
• (open source) Netdot

https://osl.uoregon.edu/redmine/projects/netdot
APIC-EM Topology (Layer 2 filtering)

Logging
• Centrally: for ease of analysis and search
• Cisco Prime Infrastructure & Cisco EPNM– full featured tool for inventory, and monitoring
• Moogsoft - automates early detection of service failures, collaboration & knowledge base
• syslog-ng – preprocessing, relay and store(file/db)
• Logstash(ELK), fluentd – multisource collection, storage and analysis
• Locally: in case logs can’t get home
service timestamps log datetime msec show-timezone

!
logging host <ipaddr>
logging trap 6
logging source interface Loopback 0
!
logging buffered <size> 6
logging presistant url disk0:/syslog size <TotalLogsSize> filesize <OneFileSize>
State of the Routing Table
• Be familiar with normal behavior of important service prefixes
• Establish quickly if problem is control plane or data plane
• show ip route / ipRouteTable MIB / show ip traffic (Drop stats)
• Nagios: check_snmp_iproute.pl
• Track objects and EEM
(config)
track 100 ip route 0.0.0.0 0.0.0.0 reachability
event manager applet TrackRoute_0.0.0.0
event track 100 state any
action 1.0 syslog msg "route is $_track_state“
#
01:09:21: %HA_EM-6-LOG: TrackRoute_0.0.0.0: route is down
blog.ipsapce.net
#show ip route 192.168.2.2

Routing entry for 192.168.2.2/32
Known via "ospf 1", distance 110, metric 11, type intra area
Last update from 10.0.0.2 on FastEthernet0/0, 00:00:13 ago
Routing Descriptor Blocks:
* 10.0.0.2, from 2.2.2.2, 00:00:13 ago, via FastEthernet0/0
Route metric is 11, traffic share count is 1 blog.ipspace.net
# show ip ospf
Routing Process "ospf 1" with ID 192.168.0.1
Start time: 00:01:46.195, Time elapsed: 00:48:27.308
Supports only single TOS(TOS0) routes
OSPF Area / AS-Wide Supports opaque LSA

Supports Link-local Signaling (LLS)
Supports area transit capability
Supports NSSA (compatible with RFC 3101)
Supports Database Exchange Summary List Optimization (RFC 5243)
Event-log enabled, Maximum number of events: 1000, Mode: cyclic
Router is not originating router-LSAs with maximum metric
Initial SPF schedule delay 5000 msecs
Minimum hold time between two consecutive SPFs 10000 msecs
• Remember that OSPF data in area should be Maximum wait time between two consecutive SPFs 10000 msecs
consistent Incremental-SPF disabled
Minimum LSA interval 5 secs
Minimum LSA arrival 1000 msecs
• Understand ‘normal’ rate of changes LSA group pacing timer 240 secs
Interface flood pacing timer 33 msecs
• LSA refresh /30-min unless a change Retransmission pacing timer 66 msecs
• Track SPF runs over time Number of external LSA 0. Checksum Sum 0x000000
• show ip ospf stat detail Number of opaque AS LSA 0. Checksum Sum 0x000000
Number of DCbitless external and opaque AS LSA 0
• number of LSAs expected Number of DoNotAge external and opaque AS LSA 0
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
• OSPF-MIB: OspfSpfRuns, ospfAreaLSACount Number of areas transit capable is 0
External flood list length 0
• Route missing? IETF NSF helper support enabled
Cisco NSF helper support enabled
• Where is the network supposed to be attached? Is it still? Reference bandwidth unit is 100 mbps
Area BACKBONE(0)
• show interface (on advertising router) Number of interfaces in this area is 4 (1 loopback)
• show ip ospf database … Area has no authentication
SPF algorithm last executed 00:47:05.379 ago
SPF algorithm executed 4 times
• Of course, all lines have a purpose: BRKRST-3310 Area ranges are
Number of LSA 16. Checksum Sum 0x078460
Number of opaque link LSA 0. Checksum Sum 0x000000
Number of DCbitless LSA 0
Number of indication LSA 0
Number of DoNotAge LSA 0
Flood list length 0
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
OSPF Neighborships
• neighbor adjacencies
• log-adjacency-changes [detail] (on by default, detail optional)
• show ip ospf neighbor detail (OSPF-MIB: ospfNbrState, ospfNbrEvents, ospfNbrLSRetransQLen)
(config) router ospf <id>
(config-router) log-adjacency-changes [detail]
%OSPF-5-ADJCHG: Process 12, Nbr 172.25.25.1 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer
expired Oct 14 09:57:43: %OSPF-5-ADJCHG: Process 12, Nbr 172.25.25.1 on ...
# show ip ospf neighbor detail

Neighbor 192.168.0.7, interface address 10.0.0.3
In the area 0 via interface GigabitEthernet0/1
Neighbor priority is 1, State is FULL, 6 state changes
DR is 10.0.0.3 BDR is 10.0.0.4
Options is 0x12 in Hello (E-bit, L-bit)
Options is 0x52 in DBD (E-bit, L-bit, O-bit)
LLS Options is 0x1 (LR)
Dead timer due in 00:00:39
Neighbor is up for 00:33:10
Index 2/2/2, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0)/0x0(0) Next 0x0(0)/0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Neighbors
Show IP EIGRP Neighbors
Outstanding Packets
Last Reliable Packet Sent
RtrA#show ip eigrp neighbors
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
2 10.1.1.1 Et0 12 6d16h 20 200 0 233
1 10.1.4.3 Et1 13 2w2d 87 522 0 452
0 10.1.4.2 Et1 10 2w2d 85 510 0 3
Seconds Remaining Before Declaring Neighbor Down
How Long Since the Last Time Neighbor Was Discovered
How Long It Takes for This Neighbor to Respond to Reliable Packets
How Long We’ll Wait Before Retransmitting if No Acknowledgement

Neighbors
Log-Neighbor-Changes Messages
• So this tells us why the neighbor is
bouncing—but what do they mean?
• eg: peer restarted means you have
to ask the peer; he’s the one that
restarted the session
Neighbor 10.1.1.1 (Ethernet0) is down: peer restarted

Neighbor 10.1.1.1 (Ethernet0) is up: new adjacency
Neighbor 10.1.1.1 (Ethernet0) is down: holding time expired
Neighbor 10.1.1.1 (Ethernet0) is down: retry limit exceeded Others, but not often
BGP Monitoring Protocol (BMP) Overview
Collecting Pre-Policy BGP Messages
BMP collector
BMP message
Adj-RIB-in (pre-inbound-filter)
BGP Monitor Protocol update
Loc-RIB (post-inbound-filter) Adj-RIB-in (pre-inbound-filter)

iBGP update eBGP update
BGP peer
(internal) BMP client
Inbound BGP peer’s (external)
filtering
policing
BGP Monitoring Protocol
• IETF draft-ietf-grow-bmp
• BMP client (router) provides pre-policy view of the ADJ-RIB-IN of a peer
• Update messages from peer sent to BMP receiver
• Example uses:
• Realtime visualizer of BGP state
• Traffic engineering analytics
• BGP policy exploration
OpenBMP
http://www.openbmp.org
Historical record of prefix withdraws
Current route views and peer status
Data Plane
3Ws: When, where, and what

User / Agent Checks
• Treat network as a black box: are your synthetic tests working?
• Synthetic service check (HTTP, DNS, etc.)
• Ping (not all remotes will respond)
• Data plane is exercised and tested
• Variety = better coverage (multiple IP addresses / L4 ports per location)
• Validate similar treatment (QoS) as real user traffic
• Uptime and performance (loss, latency) metrics
• Look for patterns, changes from normal. All down vs some down.
• Capture and validate real user (human) incidents. What got missed?
• Use wisely: network and server resources consumed
R3
R1 R2 R6
R5
A B
IPSLA and Relatives
• IPSLA on router/switch – makes use of deployed network infra
• May not be true check of data plane (shadow router)
• Resource contention (CPU) – group scheduling
• Simplistic service checks
• User end-system based agent software (Nagios agents…)

• Uses host stack (OS, browser) on PC
• End to end (could include WiFi)
• Includes end system resource view
• BYOD deployment challenges
• Dedicated Agent (Cisco NAM, RIPE Atlas probes…)

• Mixture of benefits from end-system and network
• Matching real user end-system stack can be challenging
IP SLA: Synthetic Traffic Measurements
Uses
Multiprotocol
Service Level
Network Label
VoIP Agreement Network Trouble
Availability Performance Switching
Monitoring (SLA) Assessment Shooting
Monitoring (MPLS)
Monitoring
Monitoring
Measurement Metrics
Packet Network Dist. of
Latency Connectivity
Loss Jitter Stats
Operations
Jitter FTP DNS DHCP DLSW ICMP UDP TCP HTTP LDP H.323 SIP RTP
Cisco IOS
Software
IP SLA Destination
Source MIB Data Active Generated Traffic to Measure the
Cisco IOS Network
Cisco IOS IP SLA
Software Software
IP SLA Responder
Reference
IPSLA Multicast Support
• IPSLA Multicast
One Way Delay (NTP req)
One Way Jitter
Packet Loss
• Configuration is on IP SLA Sender

• Have to specify each responder explicitly in endpoint-list
• Responder becomes mcast receiver, IGMPv3 (G) and (S,G) behavior
Unicast control
• ISRG2, ISR4451X, ASR1k, CSR1000v, cat4k(sup7/6), c7600 Multicast traffic
SLAsender(config)#ip sla endpoint-list type ip mylist

ip-address 172.16.1.2,172.17.1.2 port 3800
SLAsender(config)#ip sla 1
udp-jitter 224.1.1.1 4000 endpoint-list mylist source-ip 172.16.1.1 source-port 4500 num-packets 100 interval 25
iperf3
• Active measurement tool to discover available path capacity
• worst link and worst host configurations
• Test can be in either direction (only static NAT works)
• TCP (retransmissions, rate, cwd), SCTP and UDP (loss, jitter, out of order) tests
TCP/5201
sender receiver
Test traffic: TCP,
SCTP, UDP
bwctl
• bwctl client coordinates active measurement tests
• Authentication – IP subnets, AES key/username
• Scheduling/reserving
• result gathering – gathered from both server and client systems
• Does not have to be on bwctl server (3rd party)
• bwctl server hosts the test resources (iperf3, ping, traceroute/path, owamp)
• Allows for multi-admin domain (along path) active tests
• bwtraceroute: wrapper for bwctl and traceroute
• bwctl distributed with Ubuntu, may need to be installed (yum, apt-get, compiled) for other UNIXes
$ bwtraceroute -s 205.186.62.54
bwtraceroute: Using tool: traceroute Local machine
bwtraceroute: 17 seconds until test results available
SENDER START
traceroute to 152.22.242.103 (152.22.242.103), 30 hops max, 60 byte packets
1 205-186-62-53.generic.c-light.net (205.186.62.53) 0.104 ms 0.098 ms 0.102 ms
2 xe-1-1-1-816-t01-sox.culr.net (205.186.63.2) 2.932 ms 2.934 ms 2.929 ms
…
9 152.22.242.103 (152.22.242.103) 12.188 ms 12.180 ms 12.144 ms
SENDER END
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
$ bwctl -T iperf3 -t 30 -O 4 -s "56m-ps-4x10.sox.net:4823"
bwctl: Using tool: iperf3
Iperf3 bwctl: 40 seconds until test results available
SENDER START
examples Connecting to host 152.22.242.103, port 5160
[ 15] local 143.215.194.123 port 45609 connected to 152.22.242.103 port 5160
Client to server [ ID] Interval Transfer Bandwidth Retr Cwnd
(local to remote) [ 15] 0.00-1.00 sec 107 MBytes 898 Mbits/sec 0 3.06 MBytes (omitted)
[ 15] 1.00-2.00 sec 112 MBytes 944 Mbits/sec 0 3.06 MBytes (omitted)
…
[ 15] 29.00-30.00 sec 112 MBytes 944 Mbits/sec 0 3.06 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Throw away stats [ ID] Interval Transfer Bandwidth Retr
from first 4 sec [ 15] 0.00-30.00 sec 3.29 GBytes 942 Mbits/sec 0 sender
[ 15] 0.00-30.00 sec 3.29 GBytes 943 Mbits/sec receiver
iperf Done.
Run for 30 sec
SENDER END
Use –P for parallel
streams
$ $ bwctl -T iperf3 -t 30 -O 4 -c "56m-ps-4x10.sox.net:4823"
bwctl: Using tool: iperf3
bwctl: 39 seconds until test results available
SENDER START
~940 mbps (remote
Connecting to host 143.215.194.123, port 5327 to local)
[ 15] local 152.22.242.103 port 44855 connected to 143.215.194.123 port 5327
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 15] 0.00-1.00 sec 5.14 MBytes 43.1 Mbits/sec 411 25.5 KBytes (omitted)
[ 15] 1.00-2.00 sec 2.26 MBytes 19.0 Mbits/sec 15 19.8 KBytes (omitted) retransmissions
…
[ 15] 28.00-29.00 sec 2.26 MBytes 18.9 Mbits/sec 16 25.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 15] 0.00-30.00 sec 59.8 MBytes 16.7 Mbits/sec 539 sender ~19mbps (local to
[ 15] 0.00-30.00 sec 60.7 MBytes 17.0 Mbits/sec receiver
remote)
iperf Done.
SENDER END ∫∫∫∫∫∫∫

> netperf -t TCP_STREAM -H 162.209.79.211 -i 30,10 -I 95,5 -j -l 60
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 162.209.79.211 ()
port 0 AF_INET : +/-2.500% @ 95% conf. : demo
netperf !!! WARNING

!!! Desired confidence was not achieved within the specified iterations.
!!! This implies that there was variability in the test environment that
!!! must be investigated before going further.
!!! Confidence intervals: Throughput : 8.965%
!!! Local CPU util : 0.000%
!!! Remote CPU util : 0.000%
• Similar to iperf3 but: Recv Send Send
Socket Socket Message Elapsed
• Works bidirectionally in a Size Size Size Time Throughput
NAT environment bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 60.52 13.91
• additional connection/per
second and
tracnsaction/per second
tests download
• statistical confidence
intervals (-I)
∫∫∫∫∫∫∫
owamp
• One way delay/jitter to/from end systems
• Checks for loss, order
> owping -c 10000 -i 0.01 2hd32g-2.cenic.org:861
• NTP needed (check is done) Approximately 103.5 seconds until results available
--- owping statistics from [152.22.242.103]:9525 to [2hd32g-
2.cenic.org]:9105 ---
SID: 89a41e75da6e5be4ad003a66630c3668
first: 2016-02-16T21:39:34.059
last: 2016-02-16T21:41:13.152
10000 sent, 1 lost (0.010%), 0 duplicates
one-way delay min/median/max = 52.7/54.5/58.5 ms, (err=1.6 ms)
one-way jitter = 1.3 ms (P95-P50)
55ms(to) vs Hops = 10 (consistently)
no reordering
12ms(from)
--- owping statistics from [2hd32g-2.cenic.org]:9207 to
[152.22.242.103]:9111 ---
SID: 9816f267da6e5be4b0980a5547a7e2f0
first: 2016-02-16T21:39:34.046
last: 2016-02-16T21:41:13.438
10000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 10.2/11.9/16 ms, (err=1.6 ms)
one-way jitter = 1.4 ms (P95-P50)
Hops = 10 (consistently)
no reordering
∫∫∫∫∫∫∫
perfsonar
• Scheduling, execution
and visualization for
various tests across
servers
• Registry of public
servers
∫∫∫∫∫∫∫ BRKARC-2025 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Diagnostic Tools Hosting Platforms Along the Path
IOS XR
Support RPM package installation directly to the system.
Nexus OS
Support for 3rd party LXC containers. Support for Guest Shell LXC. Future support for
Docker containers.
IOS XE
Open to any 3rd party or custom KVM application on routing platforms.
Ultimate flexibility with UCS-E module.
bwctl/oaping/iperf3 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
ISR 4400 Series Service Containers
• Data Plane to Container: 200Mbps

• Container to Data Plane: 1Gbp
• Service container is on
independent CPU
• Router features (QoS, NetFlow,
etc.) are applied to container traffic
Internet: aka the
traceroute TCP/80 network
Widest dispersion
• Understand the limitations against possibilities.
• Sends 3 packets (default) at each TTL Difficult to
understand though.
• Implementations
• Linux/Cisco: UDP (ICMP and TCP-SYN are Linux optional)
• UDP DST port # used to keep track of packets, increments per packet. Initial= 33434 (default)
• SRC port #: randomized (linux), incrementing per packet (IOS) Narrower
• Linux (GNU inetutils-traceroute) dispersion.
• UDP DST port# increments per TTL (not per packet) Story might be
• SRC port is random but fixed per entire run misleading.
• Windows: ICMP Echo request
ICMP blocked
• IOS ICMP responses limited to 1 per 500ms frequently 
• Configurable via: ip icmp rate-limit unreachable <ms>
Reference
Unix traceroute
1 AAA
• Multiple path options 2 BBB
3 CCC
• Topology ‘shortcuts’ (same router seen at diff hop) 4 DDD
5 EEE
6 FGF
• Ultimately all paths result in similar e2e delay 7 HII
8 JKK +10ms (unsustained)
$ traceroute 62.2.88.172
traceroute to 62.2.88.172 (62.2.88.172), 30 hops max, 60 byte packets 9 JLJ
1 152.22.242.65 (152.22.242.65) 1.044 ms 1.371 ms 1.585 ms 10 LLM +120ms (sustained)
2 152.22.240.8 (152.22.240.8) 0.219 ms 0.328 ms 0.327 ms 11 NNM
3 128.109.70.9 (128.109.70.9) 1.066 ms 1.059 ms 1.168 ms 12 NNO
4 rtp7600-gw-to-dep7600-gw2.ncren.net (128.109.70.137) 1.634 ms 1.628 ms 1.736 ms 13 PPP
5 rlasr-gw-link1-to-rtp7600-gw.ncren.net (128.109.9.17) 5.354 ms 5.446 ms 5.557 ms
6 128.109.9.117 (128.109.9.117) 5.671 ms 128.109.9.170 14 QQQ
Multiple paths(128.109.9.170) 7.141 ms 128.109.9.117 (128.109.9.117)
15 ***
7 wscrs-gw-to-ws-a1a-ip-asr-gw-sec.ncren.net (128.109.1.105) 9.174 ms 128.109.1.209 (128.109.1.209)
5.433 ms
8.256 ms 6.397 ms
+120ms Atlantic
8 dcp-brdr-03.inet.qwest.net (205.171.251.110) 18.414 ms chr-edge-03.inet.qwest.net 16 RRR
(65.114.0.205) ~268ms
27.353 (all ms
ms 27.438 three)
9 dcp-brdr-03.inet.qwest.net (205.171.251.110) 21.739 ms 63-235-40-106.dia.static.qwest.net (63.235.40.106) 17.750 ms
dcp-brdr-03.inet.qwest.net (205.171.251.110) 22.450 ms
crossing
10 63-235-40-106.dia.static.qwest.net (63.235.40.106) 22.531 ms 22.516 ms 84-116-130-173.aorta.net (84.116.130.173) 140.738 ms
11 nl-ams02a-rd1-te0-2-0-2.aorta.net (84.116.130.65) 140.831 ms 140.816 ms 84-116-130-173.aorta.net (84.116.130.173) 144.819 ms
12 nl-ams02a-rd1-te0-2-0-2.aorta.net (84.116.130.65) 144.074 ms 144.761 ms 84-116-130-58.aorta.net (84.116.130.58) 138.455 ms
13 84-116-130-58.aorta.net
filter (84.116.130.58) 141.844 ms 141.924 ms 142.459 ms
14 84.116.204.234 (84.116.204.234) 145.603 ms 145.891 ms 145.987 ms + > 100 ms delay
15 * * *
16 62-2-88-172.static.cablecom.ch (62.2.88.172) 268.281 ms 268.245 ms 268.176 ms
Reference
Unix inetutils traceroute
• Narrower view (no alternate paths directly seen)
• Repeating nodes suggests multipath, or (unlikely) routing issue
$ inetutils-traceroute --resolve-hostname 62.2.88.172

traceroute to 62.2.88.172 (62.2.88.172), 64 hops max Packets for hop 9,12 took a
1 152.22.242.65 (152.22.242.65) 0.783ms 0.727ms 0.798ms
2 152.22.240.8 (152.22.240.8) 0.226ms 0.228ms 0.221ms ‘shortcut’ and packets for
3 128.109.70.9 (128.109.70.9) 0.967ms 0.980ms 0.962ms
4 128.109.70.137 (rtp7600-gw-to-dep7600-gw2.ncren.net) 1.576ms 1.598ms 1.567ms hop 10,13 went long way
5 128.109.9.17 (rlasr-gw-link1-to-rtp7600-gw.ncren.net) 5.149ms 5.140ms 5.126ms
6 128.109.9.166 (128.109.9.166) 7.113ms 7.098ms 7.306ms
7 128.109.1.209 (128.109.1.209) 7.835ms 8.326ms 7.958ms
8 65.114.0.205 (chr-edge-03.inet.qwest.net) 19.944ms 9.299ms 40.372ms
9 63.235.40.106 (63-235-40-106.dia.static.qwest.net) 18.442ms 18.412ms 18.432ms
10 63.235.40.106 (63-235-40-106.dia.static.qwest.net) 22.424ms 22.391ms 75.960ms
11 84.116.130.173 (84-116-130-173.aorta.net) 145.434ms 146.301ms 145.445ms
12 84.116.130.58 (84-116-130-58.aorta.net) 137.583ms 137.556ms 137.661ms
13 84.116.130.58 (84-116-130-58.aorta.net) 142.476ms 141.886ms 141.819ms
14 84.116.204.234 (84.116.204.234) 144.841ms 145.034ms 144.964ms
15 * * *
16 62.2.88.172 (62-2-88-172.static.cablecom.ch) 287.318ms 176.670ms 254.237ms
Reference
LFT
• lft ‘layer 4 traceroute’ dynamically adjusts to responses
• Firewall detection, whois and AS lookup integrated
• Narrower packet changes, so narrower multi-path
$ sudo lft -ENA 62.2.88.172
Tracing ________________________________________________________________. Used tcp/80 SYN
TTL LFT trace to 62-2-88-172.static.cablecom.ch (62.2.88.172):80/tcp
1 [AS81] [NCREN-B22] 152.22.242.65 20.1/17.2ms
2 [AS81] [NCREN-B22] 152.22.240.8 20.1/20.1ms
3 [AS81] [CONCERT] 128.109.70.9 20.1/20.1ms
4 [AS81] [CONCERT] rtp7600-gw-to-dep7600-gw2.ncren.net (128.109.70.137) 20.1/20.1ms
5 [AS81] [CONCERT] rlasr-gw-link1-to-rtp7600-gw.ncren.net (128.109.9.17) 20.1/20.1ms
6 [AS81] [CONCERT] 128.109.9.117 20.1/20.1ms
7 [AS209] [unknown] chr-edge-03.inet.qwest.net (65.121.156.209) 20.1/19.5ms
8 [AS209] [QWEST-INET-35] dcp-brdr-03.inet.qwest.net (205.171.251.110) 20.1/18.4ms
9 [AS209] [QWEST-INET-17] 63-235-40-106.dia.static.qwest.net (63.235.40.106) 20.1/60.3ms
10 [AS6830] [84-RIPE/LGI-Infrastructure] 84-116-130-173.aorta.net (84.116.130.173) 160.7/160.7ms
11 [AS6830] [84-RIPE/LGI-Infrastructure] nl-ams02a-rd1-te0-2-0-2.aorta.net (84.116.130.65) 160.7/160.7ms
12 [AS6830] [84-RIPE/LGI-Infrastructure] 84-116-130-58.aorta.net (84.116.130.58) 140.6/140.6ms
** [firewall] the next gateway may statefully inspect packets
13 [AS6830] [84-RIPE/LGI-Infrastructure] 84.116.204.234 160.7/160.6ms
** [neglected] no reply packets received from TTL 14
15 * [AS6830] [RIPE-C3/CC-HO841-NET] [target] 62-2-88-172.static.cablecom.ch (62.2.88.172):80 160.7ms
Reference
MTR
• Interactive combined traceroute and ping
• Gives a sense of health of path (loss, delay Standard Deviation)
Just local noise, no
• Narrow path view carry over to later hops Sustained loss.
aakhter-nlr-ubuntu-01 (0.0.0.0) Sat May 30 18:57:09 2015 Likely something
Keys: Help Display mode Restart statistics Order of fields quit wrong 12->13, or
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev way back
1. 152.22.242.65 0.0% 145 0.8 0.9 0.7 10.0 0.8
2. 152.22.240.8 0.0% 145 0.3 0.2 0.2 0.3 0.0
3. 128.109.70.9 0.0% 145 1.0 3.3 1.0 182.3 17.2
4. rtp7600-gw-to-dep7600-gw2.ncren.net 1.0% 145 9.2 4.1 1.6 203.4 18.6
5. rlasr-gw-link1-to-rtp7600-gw.ncren.net 0.0% 145 5.3 5.3 5.1 6.8 0.2
6. 128.109.9.166 0.0% 145 7.1 7.3 7.1 16.1 0.8
7. wscrs-gw-to-ws-a1a-ip-asr-gw-sec.ncren.net 0.0% 145 6.8 8.3 6.2 10.6 1.0
8. chr-edge-03.inet.qwest.net 0.0% 145 9.4 12.3 9.3 62.1 9.5 Note
9. dcp-brdr-03.inet.qwest.net 0.0% 145 21.8 22.8 21.7 70.7 5.5 variability,
10. 63-235-40-106.dia.static.qwest.net 0.0% 145 21.8 24.5 21.7 86.1 10.6
11. 84-116-130-173.aorta.net 0.0% 145 144.8 145.0 144.7 152.9 1.0 probably just
12. nl-ams02a-rd1-te0-2-0-2.aorta.net 0.0% 145 144.1 145.5 144.0 165.4 3.7 the end
13. 84-116-130-58.aorta.net 5.0% 144 142.9 142.3 142.0 145.6 0.4
14. 84.116.204.234 5.0% 144 145.1 145.1 144.9 145.3 0.0 system
15. 217-168-62-150.static.cablecom.ch 5.0% 144 145.9 146.1 145.2 164.3 1.9
16. 62-2-88-172.static.cablecom.ch 5.0% 144 313.0 260.3 152.6 508.0 80.0
Show interface # show interface
GigabitEthernet1 is up, line protocol is up
Hardware is CSR vNIC, address is 000c.291a.7f97 (bia
000c.291a.7f97)
Internet address is 192.168.225.130/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is RJ45
output flow-control is unsupported, input flow-control is
unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:05:35, output 00:09:58, output hang never
Last clearing of "show interface" counters never
• Classic command Input queue: 0/375/0/0 (size/max/drops/flushes); Total output
drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
• Check ‘up’ status 5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
25349 packets input, 2381158 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
• Stability: log event or ‘show ip route’ 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
3958 packets output, 312408 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
• Monitor in/out bit/packet changes 56 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
snmp ifmib ifindex persist

snmp ifmib trap throttle
interface <intf>
[no] logging event link-status
[no] no snmp trap link-status
load-interval 30
Follow the Flow with NetFlow
• Per-Node: Data plane observations and decisions captured
• Src/dst mac/IP/port#s, DSCP values, in/out interfaces, etc.
• Network view: flows centrally analyzed- NetFlow collector/analyzer
• Biggest value: strategically placed partial views
(eg WAN edge)
NetFlow Collector
LiveAction
R3
R1 R2 R6
R5
A B
R4
NetFlow—What Is It?
• Developed and patented at Cisco

Systems in 1996
• NetFlow is the de facto standard for
acquiring IP operational data
• Standardized in IETF via IPFIX
• Provides network and security
monitoring, network planning, traffic
analysis, and IP accounting
• Packet capture is like a wire tap
• NetFlow is like a phone bill Network World Article—NetFlow Adoption on the Rise
http://www.networkworld.com/newsletters/nsm/2005/0314nsm1.html
Flexible NetFlow
Multiple Monitors with Unique Key Fields
Flow Flow
Monitor 1 Monitor 2
Key Fields Packet 1 Non-Key Fields Key Fields Packet 1 Non-Key Fields
Packets Source IP 3.3.3.3 Packets
Source IP 3.3.3.3
Bytes Dest IP 2.2.2.2 Timestamps
Destination IP 2.2.2.2
Timestamps Input Interface Ethernet 0
Source Port 23
Next Hop Address SYN Flag 0
Destination Port 22078
Layer 3 Protocol TCP - 6
TOS Byte 0
Security Analysis Cache
Input Interface Ethernet 0
Source IP Dest. IP Input I/F Flag … Pkts
Traffic Analysis Cache 3.3.3.3 2.2.2.2 E0 0 … 11000

Src. Dest. Source Dest. TO Input
Protocol … Pkts
IP IP Port Port S I/F
3.3.3.3 2.2.2.2 23 22078 6 0 E0 … 1100

NetFlow Forwarding Status
& Drop Count Fields
• Flexible NetFlow Forwarding Status field

captures forwarding (and drop reason) for flow.
• Drop Count increments on any explicit drop by
router
Network Performance Monitor
Network nodes are able to discover & validate RTP, TCP and IP-CBR traffic on hop by hop
basis
À la carte metric (loss, latency, jitter etc.) selections, applied on operator selected sets of traffic
Allows for fault isolation and network span validation
Per-application threshold and altering.
Performance Monitor Information Elements
Media Monitoring Application Response Time Other Metrics
• RTP SSRC • CND - Client Network Delay (min/max/sum) • L3 counter (bytes/packets)
• RTP Jitter (min/max/mean) • SND – Server Network Delay (min/max/sum) • Flow event
• Transport Counter (expected/loss) • ND – Network Delay (min/max/sum) • Flow direction
• Media Counter • AD – Application Delay (min/max/sum) • Client and server address
(bytes/packets/rate)
• Total Response Time (min/max/sum) • Source and destination address
• Media Event
• Total Transaction Time (min/max/sum) • Transport information
• Collection interval
• Number of New Connections • Input and output interfaces
• TCP MSS
• Number of Late Responses • L3 information (TTL, DSCP,
• TCP round-trip time TOS, etc.)
• Number of Responses by Response Time (7-
bucket histogram) • Application information (from
• Number of Retransmissions NBAR2)
• Monitoring class hierarchy
• Number of Transactions
• Client/Server Bytes
• Client/Server Packets
NetFlow QoS Analysis
How is my flow being classified?

Did this QoS class drop traffic?
Cisco Prime Infra
LiveAction
flow 5-tuple DPI/NBAR QoS processing DSCP
NetFlow QoS Flow exporter:
option c3pl-class-table timeout <timeout>
option c3pl-policy-table timeout <timeout>
QoS Queue performance:
flow record type performance monitor qos-record
match policy qos queue index
collect policy qos queue drops
(or)
flow record qos-record
• QoS queue performance Flow to QoS Association:
flow record type performance-monitor A
(drops) match connection client ipv4 address
match connection server ipv4 address
match connection server transport port
• QoS class structure class-map collect policy qos class hierarchy
collect policy qos queue id
…
and policy map names (or)
flow record qos-class-record
match ipv4 source address
match ipv4 destination address
collect policy qos classification hierarchy
collect policy qos queue index
…
Enhanced NetFlow CLI Example
R1#show flow monitor qos-flow-monitor cache
IP FORWARDING STATUS: Forward
IPV4 SOURCE ADDRESS: 192.168.32.128
platform qos performance-monitor IPV4 DESTINATION ADDRESS: 224.0.0.5
! INTERFACE INPUT: Null
flow record qos-class-record INTERFACE OUTPUT: Gi2 0x30 = CS6: in
match routing forwarding-status FLOW DIRECTION: Output ‘control’ class
match ipv4 dscp IP DSCP: 0x30
match ipv4 source address policy qos class hierarchy: WAN-EDGE-4-CLASS: CONTROL
match ipv4 destination address policy qos queue index: 1073741827
match interface input IP FORWARDING STATUS: Consume
match interface output IPV4 SOURCE ADDRESS: 192.168.225.128
match flow direction IPV4 DESTINATION ADDRESS: 192.168.225.130
My VTY
collect policy qos classification hierarchy INTERFACE INPUT: Gi1 session
collect policy qos queue index INTERFACE OUTPUT: Null
! FLOW DIRECTION: Input
flow monitor qos-flow-monitor IP DSCP: 0x04
record qos-class-record policy qos class hierarchy: WAN-EDGE-4-CLASS: class-default
! policy qos queue index: 0
interface GigabitEthernet1 Data
IP FORWARDING STATUS: Forward
ip flow monitor qos-flow-monitor input IPV4 SOURCE ADDRESS: 192.168.225.128 traffic
! IPV4 DESTINATION ADDRESS: 5.5.5.5
interface GigabitEthernet2 INTERFACE INPUT: Gi1
ip flow monitor qos-flow-monitor output INTERFACE OUTPUT: Gi2
service-policy output WAN-EDGE-4-CLASS FLOW DIRECTION: Output
IP DSCP: 0x00
policy qos class hierarchy: WAN-EDGE-4-CLASS: class-default
policy qos queue index: 1073741829
CBQoS MIB
• IOS QoS collects vital information regarding
health of QoS classes
• Pre and Post bytes, drops, etc
• Same class names from different routers
can be compared
• For flow level analysis, use NetFlow QoS
reporting
• ‘snmp mib persist CBQoS’ (IOS 12.4(4)T)
Adventnet © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dedicated Protocol Analyzers
• Wireshark, Cisco NAM and other protocol analyzers are great
• Detailed analysis for variety of protocols at deep level
• Dedicated probes are expensive to deploy pervasively
• Operator has to make difficult judgment calls on where the problem is going to be– before it
happens
• Can be challenging after the fact- need on-site trained personnel.
Embedded Packet Capture & Analyze
• Capture packets locally to buffer on router
• Store to flash, USB, FTP, TFTP for analysis in protocol analyzer
• IOS XE Cat 4k Sup 7E & Sup 7L-E (XE 3.3.0 SG) include built in Wireshark decode capability
• Capture does not add traffic to network
LY-2851-8#monitor capture buffer pcap-buffer1 size 10000 max-size 1550

LY-2851-8#monitor capture point ip cef pcap-point1 g0/0 both
LY-2851-8#monitor capture point associate pcap-point1 pcap-buffer1
LY-2851-8#monitor capture point start pcap-point1
LY-2851-8#monitor capture point stop pcap-point1
LY-2851-8#monitor capture buffer pcap-buffer1 export ftp://10.17.0.252/images/test.cap
Gig0/0
APIC-EM Flow Path Analysis
5 tuple Input via Use Interface
Required Information Optional Information
SRC and DEST IP Address SRC and DEST L4 Port Numbers;

[End-Host or L3 Interface] L4 Protocol (TCP or UDP)
Note: L4 Port and Protocol information is optional but highly recommended for accurate path calculation
Flow Path Analysis
Enhanced Application Flow Visibility
CAPWAP
tunnel
visualization Link source
information
Accuracy value
Ingress/Egress
Interface
∫
Flow Path Analysis
Enhanced Application Flow Visibility – Key Statistics
Area of Interest
Interface and
QoS Queue
Stats
∫
Getting Started
Be Prepared!
• Be prepared and have data collection systems enabled
• Enable passive monitoring on endpoints and network
• Enable active tests
• Helpdesk
• Interview Script => establish & maintain checklists
• Multi-group access to tools, logs, etc.
• Firefighters run drills, so should your teams!

• Be familiar with the tools and how they respond on your network
• Red phone: Cross-domain teams (applications, UC, security, servers)
Expanding your Toolbox and Knowledge
• Commercial and open source tools to look at

• Network topology & IP address management: APIC-EM, netdot, GestióIP
• Performance tests: iperf3
• Service checks: AppDynamics, Nagios Core, Zenoss Community
• NetFlow / Log analysis: Cisco Prime Infra, Lancope, logstash, fluentd
• Template driven config generation: Cisco Prime Infra, ansible
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you
Backup Slides
Performance Monitor Configuration
Flow
Flow Where to send data?
Exporter
Record (optional)
Policy-map
Applied inbound or
Flow Monitor outbound
Class-map
What metrics to collect? Interface

What traffic to monitor?
Example Configuration – Flow Record
flow record type performance-monitor default-rtp-pt-name
match ipv4 protocol
Flow Record defines what metrics to match ipv4 source address
collect and how to collect them (just like in match transport source-port
match transport destination-port
Flexible NetFlow configuration) match transport rtp ssrc
match policy performance-monitor classification hierarchy
collect routing forwarding-status
collect ipv4 dscp
Performance monitor introduces collect ipv4 ttl
collect transport packets expected counter
flow record type performance-monitor collect transport packets lost counter
collect transport packets lost rate
collect transport event packet-loss counter
Match field types perform aggregation collect transport rtp jitter mean
collect transport rtp jitter minimum
towards that field. collect transport rtp jitter maximum
collect interface input
collect interface output
collect counter bytes
Ie collect counter packets
collect counter bytes rate
match ipv4 source address collect timestamp interval
collect application name
match ipv4 destination address collect application media bytes counter
collect application media bytes rate
collect application media packets counter
collect application media packets rate
will create a unique entry per src-dst collect application media event
collect monitor event
combinations !
collect transport rtp payload-type
Example Configuration – monitor
flow exporter mn-campus-samplicator
destination 10.1.160.37
source Loopback0
flow monitor pulls together the flow record, transport udp 2055
exporter, and specific cache management template data timeout 60
option c3pl-class-table
configurations (just like Flexible NetFlow) option c3pl-policy-table
option interface-table
option application-table
Special type of flow monitor option sub-application-table
flow monitor type performance-monitor !
flow monitor type performance-monitor default-rtp-pt-name
record default-rtp-pt-name
(optional) Flow exporter configures how the exporter mn-campus-samplicator
NetFlow exporting is done cache timeout synchronized 10 export-spread 5
history size 10
!
Policy map specifies which traffic to monitor policy-map type performance-monitor rtp-traffic-name
class VOIP
(via class-map), how to monitor (via monitor), flow monitor default-rtp-pt-name
and any per-class threshold crossing actions react 1 transport-packets-lost-rate
threshold value ge 1.00
alarm severity error
Typed policy-map (performance monitor) action syslog
class VIDEO-CONF
flow monitor default-rtp-pt-name
Example Configuration – Interface attachment
• Finally, policy map is applied to interface
• Note typed policy is used
• Direction of monitoring (input|output) selectable for some platforms
interface gigabitEthernet 0/1

service-policy type performance-monitor input rtp-traffic-name
Audio Quality Metrics (AQM) on CUBE
• AQM provides deeper insight

into the media flows that are
processed by the CUBE /
SIP/media
Voice gateways
ISRG2, c8xx 15.3(3)M

PRI
• Available via MIB, CDR and

performance monitor
Example Configuration –
AQM performance monitor
‘media monitoring’ configuration under voice service voip
media monitoring [num] persist
! num is number of channels used to monitor
‘voice service voip’ or dial-peer media statistics
! delay calc, MOS etc
Controls generation of metrics on OR

CUBE/VG
dial-peer voice [tag] voip
media monitoring
To export via NetFlow, regular !
performance monitor configuration – flow record type performance-monitor aqm
just include the AQM fields match ipv4 destination address
match transport source-port
collect application voice number called
MIB collect application voice number calling
…
CISCO-VOICE-DIAL-CONTROL-MIB Regular performance monitoring configuration continues
Video Quality Metrics (VQM) on ISR G2
VQM deeper insight into the

video flows (H.264) that are
crossing routers
ISRG2, c8xx 15.3(3)M
Available via performance
monitor
Example Configuration –
VQM performance monitor
video monitoring
maximum-sessions 10
no shutdown
‘no shut’ under ‘video flow record type performance-monitoring vqm-rec

match ipv4 protocol
monitoring’ global config. match ipv4 destination address
match transport source-port
match transport rtp ssrc
To export via NetFlow, collect application video resolution [ width | height ] last
collect application video frame rate
collect application video payload bitrate [ average | fluctuation ]
regular performance collect application video frame [ I | STR | LTR | super-P | NR ] counter
frames
monitor configuration – just collect application video frame [ I | STR | LTR | super-P | NR ] counter
packets [lost]
collect application video frame [ I | STR | LTR | super-P | NR ] counter bytes
include the AQM fields collect application video frame [ I | STR | LTR | super-P | NR ] slice-
quantization-level
collect application video eMOS compression [ network | bitstream ]
collect application video eMOS packet-loss [ network | bitstream ]
collect application video frame percentage damaged
collect application video scene-complexity
collect application video level-of-motion
collect transport rtpsequence-number [ last ]
show commands 1861-AA0213#show performance monitor history
Load for five secs: 20%/16%; one minute: 8%; five minutes: 4%
Time source is NTP, 01:52:12.052 EST Fri Oct 29 2010
Codes: * - field is not configurable under flow record

NA - field is not applicable for configured parameters
Match: ipv4 src addr = 10.1.160.19, ipv4 dst addr = 10.1.3.5, ipv4 prot = udp, trns src
port = 32760, trns dst port = 22802, SSRC = 1717646439
Policy: all-apps, Class: telepresence-CS4, Interface: FastEthernet0/0, Direction: input
start time 01:51:31

============
*history bucket number : 1
*counter flow : 1
counter bytes : 162329
counter bytes rate (Bps) : 5410
*counter bytes rate per flow (Bps) : 5410
Individual monitor intervals: *counter bytes rate per flow min

*counter bytes rate per flow max
counter packets
*counter packets rate per flow
counter packets dropped
(Bps)
(Bps)
:
:
:
:
:
5410
5410
773
25
0
routing forwarding-status reason : Unknown
• interface input
interface output
:
:
Fa0/0
Vl1000
monitor event : false
ipv4 dscp : 32
ipv4 ttl : 58
application media bytes counter : 146869
Aggregation over all stored application media packets counter

application media bytes rate
*application media bytes rate per flow
*application media bytes rate per flow min
(Bps)
(Bps)
(Bps)
:
:
:
:
773
4895
4895
4895
*application media bytes rate per flow max (Bps) : 4895
intervals: application media packets rate

application media event
*transport rtp flow count
transport rtp jitter mean
(pps)
(usec)
:
:
:
:
25
Normal
1
476
transport rtp jitter minimum (usec) : 1
• transport rtp jitter maximum
*transport rtp payload type
transport event packet-loss counter
(usec) :
:
:
1997
96
0
*transport event packet-loss counter min : 0
*transport event packet-loss counter max : 0
transport packets expected counter : 773
transport packets lost counter : 0
*transport packets lost counter minimum : 0
*transport packets lost counter maximum : 0
transport packets lost rate ( % ) : 0.00
*transport packets lost rate min ( % ) : 0.00
for reference *transport packets lost rate max ( % ) : 0.00
Service Planning
FNF Configuration - Example
1. Configure the Exporter
Router(config)# flow exporter my-exporter
Where do I want my data sent?
Router(config-flow-exporter)# destination 1.1.1.1
2. Configure the Flow Record

Router(config)# flow record my-record
Router(config-flow-record)#
What data do Imatch
want toipv4 destination address
meter?
Router(config-flow-record)# match ipv4 source address
Router(config-flow-record)# collect counter bytes
3. Configure the Flow Monitor
Router(config)# flow monitor my-monitor
How do I want to cacheexporter
Router(config-flow-monitor)# information?
my-exporter
Router(config-flow-monitor)# record my-record
4. Apply to an Interface
Router(config)# interface s3/0
Which interface do I want to monitor?
Router(config-if)# ip flow monitor my-monitor input
NetFlow QoS Reporting
Flow exporter:
option c3pl-class-table timeout <timeout>
option c3pl-policy-table timeout <timeout>
• How is my flow being QoS Queue performance:
flow record type performance monitor qos-record
classified? match policy qos queue index
(or)
• Did this class drop traffic? flow record qos-record
• QoS queue performance Flow to QoS Association:
flow record type performance-monitor A
(drops) match connection client ipv4 address
match connection server ipv4 address
match connection server transport port
collect policy qos class hierarchy
• QoS class structure class-map collect policy qos queue id
…
and policy map names (or)
flow record qos-class-record
collect policy qos classification hierarchy
collect policy qos queue index
…
R1#show ip traffic [interface <interface>]
IP statistics:
Rcvd: 1117 total, 1116 local destination
0 format errors, 0 checksum errors, 0 bad hop count
show ip traffic
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 fragments, 0 couldn't fragment
Bcast: 58 received, 0 sent
Mcast: 442 received, 221 sent
Sent: 842 generated, 1195 forwarded
Drop: 1 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
0 physical broadcast
Reinj: 0 in input feature path, 0 in output feature path
ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
0 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
0 time exceeded, 0 info replies
Sent: 0 redirects, 0 unreachable, 0 echo, 0 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 0 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
UDP statistics:
Rcvd: 58 total, 0 checksum errors, 58 no port 0 finput
Sent: 0 total, 0 forwarded broadcasts
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh
TCP statistics:
Rcvd: 1471 total, 0 checksum errors, 85 no port
Sent: 597 total
..
OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 460 total, 0 checksum errors
414 hello, 8 database desc, 3 link state req
22 link state updates, 13 link state acks
Sent: 245 total
199 hello, 12 database desc, 2 link state req
21 link state updates, 12 link state acks

Brkarc 2025 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkarc 2025 PDF

Uploaded by

Copyright:

Available Formats

Network State

Aamer Akhter / aa@cisco.com

• This session is about basic network troubleshooting,

• For context, we will cover some basic methodologies and

• This is the 90 min tour. ;-)

• A lot of stuff going on

• Multiple layered services Enterprise

• Mis-information / inconsistency ISP A

• Redundant paths / ECMP / LAG

• Faster problem resolution

Find the Suspects Question Suspects Improve

• Control Plane Gossip from

• Processes variety of information

• Best known intention w/o actual

• Granted some decision flexibility show policy-map int…

ifmib CbQoS *Flow

• Control plane: condenses options driven by policies and (relatively) slower

• Each network device makes an independent forwarding decision

• Change is normal, but some

3Ws: When, where, and what

• Clearly label devices, ownership

• Establish network topology baseline • Visual inspection 

R1#show cdp neighbors

Device ID Local Intrfce Holdtme Capability Platform Port ID

• (open source) Netdot

APIC-EM Topology (Layer 2 filtering)

• Locally: in case logs can’t get home

service timestamps log datetime msec show-timezone

#show ip route 192.168.2.2

OSPF Area / AS-Wide Supports opaque LSA

# show ip ospf neighbor detail

Seconds Remaining Before Declaring Neighbor Down

How Long Since the Last Time Neighbor Was Discovered

How Long It Takes for This Neighbor to Respond to Reliable Packets

How Long We’ll Wait Before Retransmitting if No Acknowledgement

Neighbor 10.1.1.1 (Ethernet0) is down: peer restarted

Loc-RIB (post-inbound-filter) Adj-RIB-in (pre-inbound-filter)

Historical record of prefix withdraws

Current route views and peer status

3Ws: When, where, and what

• User end-system based agent software (Nagios agents…)

• Dedicated Agent (Cisco NAM, RIPE Atlas probes…)

• Configuration is on IP SLA Sender

• ISRG2, ISR4451X, ASR1k, CSR1000v, cat4k(sup7/6), c7600 Multicast traffic

SLAsender(config)#ip sla endpoint-list type ip mylist

SENDER END ∫∫∫∫∫∫∫

netperf !!! WARNING

• Data Plane to Container: 200Mbps

$ inetutils-traceroute --resolve-hostname 62.2.88.172

snmp ifmib ifindex persist

• Developed and patented at Cisco

Traffic Analysis Cache 3.3.3.3 2.2.2.2 E0 0 … 11000

3.3.3.3 2.2.2.2 23 22078 6 0 E0 … 1100

• Flexible NetFlow Forwarding Status field

How is my flow being classified?

Cisco Prime Infra

LY-2851-8#monitor capture buffer pcap-buffer1 size 10000 max-size 1550

Required Information Optional Information

SRC and DEST IP Address SRC and DEST L4 Port Numbers;

• Firefighters run drills, so should your teams!

• Commercial and open source tools to look at

Don’t forget: Cisco Live sessions will be

What metrics to collect? Interface