5 ACI Monitoring and Tshoot PDF

Accelerator
Operations Planning & Best Practices: Cisco ACI

Ruben Del Monte
CSS DC - SP EMEAR
Cisco Customer Experience
March 2022
What you’ll learn Monitoring
today to help you on

your Cisco ACI Day 2 Troubleshooting
operations journey Infrastructure and

services
How can you get more value

from Cisco ACI Day 2
Operations?
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public ATX: Prepare to Implement Cisco ACI
1 10-minute daily routine
2 Navigation shortcuts
Monitor 3 Health Scores
4 Faults, audits, and events
First 10 Minutes – Daily Routine
• Alert List
• Health Scores
• System
• Node
• APIC Controller
• Faults
Alert List
Shows critical warnings and
error information to inform
user to take action
Alert List (Continued)
• Alert to detect if OSPF connectivity is

down (MPoD) configuration
• Alert to detect process crash and

acknowledge old crashes
Check Health Score
Looks like we
Looks like
hadwean issue!
had an issue!
Health Score
Health Scores are based on Faults and events
Check APIC Cluster Status
APIC Cluster Status

“Fully Fit” – All APICs are in sync
Check Faults
Faults
Faults are indications of mis-config or any
issues on ACI Fabric
※ This is a lab setup. Try to clear all Faults whenever a
new one is raised in production.
First 10 Minutes – Daily Routine (contd)
• Fabric inventory
• Topology Summary
• Physical switches
• Global EndPoints
• Interfaces’ status
• Policies-to-interfaces association
• Duplicated Ips
• Disabled interfaces
• DOM values
Topology Summary: Physical Switches
Global Endpoints
EPG Level
Interface
Interface and Policies
Duplicate IP/Disabled Interfaces
Duplicate IPs
Disabled interfaces on faric
Understand DOM values (inventory > leaf > physical interfaces)
• dBm – logarithmic scale
• value between low and high warnings
Notice:
• Optics used must support DOM
• Must create Fabric Node Controls policy with DOM enabled first, to be associated with switch profiles:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/troubleshooting/b_APIC_Troubleshooting/m_troubleshooting_tools_and_methodology.html
First 10 Minutes – Daily Routine (contd)
• Tenant statistics
• Flows stats
• Drops (operational tab)
Stats
Operations: Packets
1 Health score overview
2 Health score impact
Health Scores 3 Drill downs
4 Evaluation policy
Health Score overview
• Health Score provides a quick overview of the health of the system/module
• It is based on the Faults generated in the Fabric
• Range: 0 to 100 (100 is perfect Health Score)
• Each Fault reduces the Health Score based on the severity of the Fault
• Health Score is propagated to container and related MOs
• Health Score policies can control the penalty values, propagation, healthRecords
Health Score Views
• System — Aggregation of system-wide health, including Pod Health Scores, Tenant Health
Scores, system Fault counts by domain and type, and the APIC Cluster health state
• Pod — Aggregation of Health Scores for a Pod (a group of Spine and Leaf switches), and Pod-
wide Fault counts by domain and type
• Tenant — Aggregation of Health Scores for a Tenant, including performance data for objects
such as applications and EPGs that are specific to a Tenant, and Tenant-wide Fault counts by
domain and type
• Managed Object — Health Score policies for managed objects (MOs), which includes their
dependent and related MOs. These policies can be customized by an administrator
Health Score: Impact Example
• In this example, a hardware Fault impacts the
Health Score of an application component
• The Health Score is propagated to the
following MOs:
• Parent MO
• MOs that have a relation pointing to the
current MO
Health Score
Drill Down
Health Score Evaluation Policy
• Currently can modify the penalty of the Health Score at the Fault severity level or ignore
acknowledged Faults
1 Statistics overview
2 GUI counters
Statistics
3 Capacity dashboard
Statistics
• Helps in quantifying the data with respected to application traffic
• Statistics contain counters
• Sampled into various granularities (5min, 15min, 1hr, etc.)
• History retained for each granularity
• Statistics are related to observable overlay objects
• Tenants/VRF/BD/EPG
Counter values in GUI
Stats tab to see stats counter values

Traffic is aggregated by bytes
Capacity Dashboard – Fabric Capacity
Capacity Dashboard – Leaf Capacity
1 Faults
Faults/Events/Audit 2 Events
logs
3 Audit logs
Faults
• Faults, events and audit logs are essential tools for monitoring the administrative
and operational state of an ACI Fabric as well as troubleshooting current and past
issues
• They are the first thing to check when something is not behaving as expected
Faults
• When a Fault occurs in an MO, a Fault instance MO is created
under the MO (Fault:Inst)
• It can be queried by DN and class
• For every Fault, a Fault record object (Fault:Record) is created in

the Fault log
• Severity:
• Critical, Major, Minor, Warning, Info, Cleared
• Types:
• Generic, Equipment, Configuration, Connectivity,
Environmental, Management, Network
Fault Types
Type Description
Generic The system has detected a generic issue
Equipment The system has detected that a physical component is inoperable or has another
functional issue
Configuration The system is unable to successfully configure a component
Connectivity The system has detected a connectivity issue, such as an unreachable adapter
Environmental The system has detected has detected a power issue, thermal issue, voltage issue,
or a loss of CMPS settings
Management The system has detected a serious management issue, such as one of the
following:
• Critical services could not be started
• Components in the instance include incompatible firmware versions
Network The system has detected a network issue, such as a link down
Operational The system has detected an operational issue, such as a log capacity limit or a
failed component discovery
Fault Triggers
• Four types of Fault triggers:
1. Specific conditions described in the model by Fault rules
2. Counters crossing thresholds specified in user-programmable policies
3. Task or FSM failures
4. Object resolution failures
• Faults are raised and managed on the node (switch or controller) where the condition is
detected
• Faults are raised and cleared automatically by the system
• User cannot define new Faults
Fault Lifecycle and Acknowledgement
• Faults can be “acknowledged”, meaning “mark as
viewed”
• Acknowledging a Fault in “retaining” state causes
the Fault to be deleted immediately (without
waiting for expiration or retention timer)
• Faults in Acknowledged state will not affect Health
Score
Fault Policies
Sample fault
Faults
• Documentation
• https://<APIC IP>/doc/html/
• Or go directly to a Fault code:
• https://<APIC IP>/doc/html/FAULT-<Fault Code>.html
• Cisco APIC Faults, Events, and System Messages Management Guide

• https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-
x/Faults/guide/b_APIC_Faults_Errors/b_IFC_Faults_Errors_preface_00.html
• ACI System Messages:
• https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/2-
x/syslog/guide/b_ACI_System_Messages_Guide.html
Events
• An event is a specific condition that occurs at a certain point in time (for example “link went
from down to up”)
• As they are part of the normal system workflow, they do not necessarily require user
attention
• Useful for monitoring and debugging issues
• Similar to an entry in a log file: once created, they are never modified
• Deleted once the maximum number specified in retention is reached
• Events are triggered by “event rules”
Events
• Configuration or state change creates an event
• Logs:
• Audit log: user-initiated actions
• Health Score log: changes in Health Scores
• Event log: other system-generated events
• Viewing
• Via GUI (“History” tab in-context)
• Via CLI, API, Syslog, SNMP, Cisco Call
Home, subscriptions
Audit Logs
• A mechanism to track user-initiated configuration changes
• When a user creates/modifies/deletes an MO, we create an “audit record” containing affected MO DN,
user name, timestamp and change details
• System also creates logs for log-in/log-out to controllers and nodes
• Similar to an entry in a log file: once created, the aaaModLR are never modified
• Configuration change logs are MOs of class modification log record
• Login/logout logs are MOs of class aaaSessionLR
• Accounting logs get deleted only when a maximum number specified in a retention policy is hit
Audit Logs
All fields have filters and searchable
Audit Logs Search - Date
Audit Logs Search - User
Audit Logs Search - Action
Faults with Event Correlation
Show relevant
events before
Fault
BRKACI-1001 48
1 SPAN
2 Endpoint Tracker
Troubleshooting
(“operations” tab)
3 Troubleshooting Wizard
4 CLI commands
ACI SPAN Feature Overview
Tenant SPAN Fabric SPAN Access SPAN
Source EPG Fabric Links (leaf or spine) Leaf downlinks
Destination ERSPAN ERSPAN ERSPAN, Leaf downlink port
Filter None Bridge Domain, VRF EPG, Routed Outside (L3out)
SPAN S10
• ACI allows for SPAN of EPG EP Learnt

L1 L2
• ERSPAN Destination must be an IP EP Learnt ERSPAN
in ACI
10.10.10.10
• EP Can run Wireshark or Tshark Leaf101# show monitor session all
session 1
---------------
description : Span session 1
type : erspan
version : 2
EPG 100 oper version : 1
state : up (active)
erspan-id : 1
granularity :
vrf-name : CiscoLive:VRF1
acl-name :
ip-ttl : 64
ip-dscp : ip-dscp not
specified
SPAN Source SPAN Destination 51 destination-ip
origin-ip
: 10.10.10.10/32
: 1.1.1.1
mode : access
source VLANs :
EPG ERSPAN rx : 100
tx : 100
both : 100
Port ERPSAN/Local Port filter VLANs
specified
: filter not
Access SPAN Overview
• Access SPAN in ACI is used to configure local SPAN on leaf switches
• Access SPAN is the only SPAN option that supports SPAN destination to physical port. Access
SPAN also supports ERSPAN destination
• SPAN can be ingress, egress, or both directions
• SPAN sources can be physical port, port-channels, VPC port-channels, or VPC component
ports
• Access SPAN currently supports EPG filters. An EPG filter is translated to a VLAN filter when
the SPAN session is programmed on the leaf
SPAN Enhancements
• In ACI 4.1, we are introducing 3 key functionalities for SPAN
• SPAN on drop
• SPAN based on filter (5 tuple)
• SPAN with destination as Port-channel
Traffic Map (“operations” tab > “visualization”)
• Find out quickly whether your

network is dropping traffic and
where
• Find out quickly whether you
have any hot spot
• Find out latency between
switches
5
EP Tracker
“We had a
problem at
14:21!”
Attach/Detach
events are logged
for each EP
Was IP Moving?
Troubleshooting Wizard - Faults
Shows Faults
in the Path
Builds Topology of Flow
Troubleshooting Wizard – Drop Stats
Shows Drops on Every Hop. Green

Arrows portray no Drops
NOTE: Some Drops are expected.
Look for Drops like “Buffer” and “Error”
Recommended Content! – Understanding Drop Faults in ACI

http://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-
infrastructure-controller-apic/210539-Explanations-of-Packet-Drop-Faults-in-AC.html
Troubleshooting Wizard - Contracts
Shows Contracts for Flows
Implicit Deny Allow SSH
Troubleshooting Wizard – Atomic Counters
No Drops!
Troubleshooting Wizard – SPAN
Ability to SPAN to APIC or other devices
attached to the Fabric
User can select which ports to SPAN
iPing
• iPing works by sending an ICMP frame to the destination endpoint, from the source leaf on behalf of the
source endpoint
• The reply is relayed back to the source leaf over the infra VRF to ensure the source leaf recognizes the
response without disrupting existing traffic
• Recommend to set the source IP address for troubleshooting
Leaf101 # show vrf

VRF-Name VRF-ID State Reason
black-hole 3 Up --
management 2 Up --
overlay-1 4 Up --
rcdn:vrf 5 Up --
leaf101 # iping -h
iping: option requires an argument -- 'h'
usage: iping [-V vrf] [-c count] [-i wait] [-p pattern] [-s packetsize] [-t timeout] [-S source
ip/interface] host
CLI Commands
show version
acidiag avread (show controller)
acidiag fnvread (show switch)
moquery (CLI base MO browser)
show audit
show event
show faults
moquery
show all tenants and children objects
moquery -c fvTenant -x 'query-target=subtree’
find EPG with pcTag between 1 and 16389

moquery -c fvAEPg -f 'fv.AEPg.pcTag~"(1,16389)”’
find active faults

moquery -c faultInst | egrep -e "^descr" | sort | uniq -c
find out changes made by user on date

moquery -c aaaModLR -f 'aaa.ModLR.created>"2020-01-01" and
aaa.ModLR.user=="admin"'
switch CLI commands
fabric <nodeID> <command> - run cmd from APIC
show lldp neighbor

show interface
show port-channel summary
show vpc extended
show vlan extended
show endpoint (apic and switch)
show zoning-rule
CURL commands
• use Postman to generate code
OUT=$(curl -s -X POST -k https://$APIC/api/aaaLogin.json -

d '{ "aaaUser" : { "attributes" : { "name" : "admin" ,
"pwd" : ”password" } } }' -c cookie.txt EOF)
curl -b cookie.txt -X GET -k
https://10.201.35.169/api/node/class/fvTenant.json | jq
Automating Operations
• Cover in Automation Accelerators
• Automation Tools
• Save/Edit/Post XML/JSON
• API inspector
• Visore
• Curl
• Postman
• Postman Runner
• Ansible
• Terraform
• Python
• UCS Director
• CloudCenter
Other Operational Support Products
• Cisco Business Critical Service – Insights (AS service)

• Cisco Network Assurance Engine
• ACI Apps
• Network Insight – Resource
• Network Insight – Adviser
• Cisco NAE explorer
• UCS Integration App (provision FI vlans automatically) – 4.1
Troubleshooting Review Questions:
• We notice a slight decrease in System health, how do we find out what happened?
• How to check if Fabric is running out of resources?
• Server team is reporting connectivity issues between two servers. How do I check
if Fabric is in good shape on datapath between two endpoints?
• Server team just connected a new server, gave me only IP and asking if I see their
server on the network?
• I see a new server, but can I ping it from the leaf?
1 Backups
Infrastructure and
2 SNMP
Services
3 Syslog
*NTP, DNS and management already

covered in previous sessions
Infrastructure and Services
Backups – Configuration Export
• The current Fabric configuration/policy in JSON/XML
• Best practice for DISASTER RECOVERY
Enabled -> Encrypted password (best practice-must)

Disabled -> No password
Backups - Snapshots
Creates a Config Backup that is stored on the APIC by default

Run on a Per Fabric or Tenant Basis
Backups - Snapshots
• Rollback feature allows config

rollback between 2 snapshots
Object
• Can also compare differences
between a previous snapshot
Changed To
Changed From
Changed From
Changed To
SNMP Support
• SNMP
(read-only)
• http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/aci/apic/sw/1-
x/mib/list/mib-support.html
SNMP Configuration
Open Question: SNMP or REST API?
• SNMP support for any information needs to be explicitly implemented
• Therefore SNMP will never cover 100% of all info available
• The REST API covers 100% of all info available
• It is very easy figuring out which REST call to make to poll a specific counter
• Most tools out there support REST-based polling (mostly indirectly through the help of additional scripts):
Cacti, Nagios, Graphite, etc.
Syslog Integration
• Forwards to syslog server
• Can forward events, audit
and faults
Syslog Integration
• Verify SYSLOG configuration
• apic1# logit severity 1 dest-grp SyslogServer server 10.18.188.110 “THIS IS A TEST“

• [root@syslogserver]# tail -f /var/log/messages
Aug 11 09:19:10 Aug 11 11:28:32.743 apic2 %LOG_-6-SYSTEM_MSG [login,session][info][subj-
[uni/userext/user-admin]/sess-8590507914] From-10.99.71.20-client-type-ssh-Success
Aug 11 09:19:17 Aug 11 11:28:39.572 apic1 %LOG_-1-SYSTEM_MSG [E4210472][transition][info][sys] sent
user message to syslog group:SyslogServer:THIS IS A TEST
Syslog configuration – Why 3 locations?
Questions
Interlock and communications
Next Steps
Additional services
Additional Accelerators and Ask the

Expert sessions
Accelerator close
Resources
ACI Troubleshooting Guide
ACI Upgrade/Downgrade Matrix Continue the conversation in
our ACI community
https://community.cisco.com/t
5/application-centric/bd-
p/12206936-discussions-aci

5 ACI Monitoring and Tshoot PDF

Uploaded by

Copyright:

Available Formats

You might also like

5 ACI Monitoring and Tshoot PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

5 ACI Monitoring and Tshoot PDF

Uploaded by

Copyright:

Available Formats

Accelerator

Operations Planning & Best Practices: Cisco ACI

today to help you on

operations journey Infrastructure and

How can you get more value

Monitor 3 Health Scores

4 Faults, audits, and events

• Alert to detect if OSPF connectivity is

• Alert to detect process crash and

APIC Cluster Status

Disabled interfaces on faric

2 Health score impact

Health Scores 3 Drill downs

Stats tab to see stats counter values

• For every Fault, a Fault record object (Fault:Record) is created in

Generic The system has detected a generic issue

Configuration The system is unable to successfully configure a component

• Faults are raised and cleared automatically by the system

• User cannot define new Faults

• Cisco APIC Faults, Events, and System Messages Management Guide

Tenant SPAN Fabric SPAN Access SPAN

Source EPG Fabric Links (leaf or spine) Leaf downlinks

Destination ERSPAN ERSPAN ERSPAN, Leaf downlink port

Filter None Bridge Domain, VRF EPG, Routed Outside (L3out)

• ACI allows for SPAN of EPG EP Learnt

• Find out quickly whether your

Builds Topology of Flow

Shows Drops on Every Hop. Green

Recommended Content! – Understanding Drop Faults in ACI

Shows Contracts for Flows

Implicit Deny Allow SSH

User can select which ports to SPAN

Leaf101 # show vrf

find EPG with pcTag between 1 and 16389

find active faults

find out changes made by user on date

show lldp neighbor

OUT=$(curl -s -X POST -k https://$APIC/api/aaaLogin.json -

• Cisco Business Critical Service – Insights (AS service)

*NTP, DNS and management already

Enabled -> Encrypted password (best practice-must)

Creates a Config Backup that is stored on the APIC by default

• Rollback feature allows config

• apic1# logit severity 1 dest-grp SyslogServer server 10.18.188.110 “THIS IS A TEST“

Interlock and communications

Additional Accelerators and Ask the

You might also like