#ArubaAirheads

ArubaOS-CX
Troubleshooting
Steve Elbaz– Aruba Engineer Resolution Team

November 14, 2018

#ArubaAirheads
Agenda

•Why do we need troubleshooting Tools?

•ArubaOS
•ArubaOS-CX
•Troubleshooting Tools :
•Support-files Troubleshooting
•Core-dump
•Diag
•Debug
•VSX Troubleshooting Demonstration
#ArubaAirheads 3
Why do we need good troubleshooting Tools?

• How was it done in the past?

• Troubleshooting tools were limited.
• Per case debug build might be required to collect the required tools.
• It required R&D resources and time
• The result was:
• It took longer to troubleshoot and
• it did not scale.
• How is it today?
• Both ArubaOS and ArubaOS-CX has extensive troubleshooting tool box
• Speed up the time it takes to collect the required information
• Result is you are faster at root cause and able to provide solution.
• The faster you fix it, your customer is more happy.

#ArubaAirheads 4
ArubaOS
Extensive Troubleshooting tools
• Debug – configure debug logging for each protocol
Examples:
5400R
− debug ip packet
3810M
− debug security radius-server
• Logging 2930M

• Core-dump
• Mirriorig
2930F

2540

2530

Campus, branch and SMB networks

#ArubaAirheads 5
 ArubaOS-CX
• Programmable next-gen OS for switching
• Open Source leverage
• Analytics to automate troubleshooting

Don’t reinvent the wheel! Use existing tools such as tcpdump.

Aruba Network Analytics Engine

Aruba 8400
Native analytics & visibility to automate
troubleshooting and streamline operations
Aruba 8320

Campus core and aggregation solutions

#ArubaAirheads 6
ArubaOS-CX Troubleshooting tools

Extensive Troubleshooting tools

• Debug
Example:
- debug mstp packet
- debug bgp all
• Logging
• Core-dump
• Mirroring

#ArubaAirheads 7
ArubaOS-CX Troubleshooting tools improvements

ArubaOS-CX 10.1

• Repeat command
• Pipe (|) Support
• Show History

Note: Even though these CLI commands were not available in version 10.0 in CLI, ERT
had alternatives in start-shell.

ArubaOS-CX 10.2
• ERSPAN – tunneling the mirror traffic to a remote switch.

#ArubaAirheads 8
ArubaOS-CX Troubleshooting tools

Support files
Copy support files command could be used to copy all relevant information while debugging an issue.
We support the following types of copy support-files:

Type Purpose
copy support-files Copy generic system related information
copy support-files all Copy all the support files related information
copy support-files feature Copy feature specific information
copy support-files standby Copy standby module specific information
copy support-files module Copy Line module specific information
copy support-files previous-boot Copy information available about previous boot
diag ovsdb-dump Display or Copy OVSDB Tables/Columns

#ArubaAirheads 9
ArubaOS-CX Troubleshooting tools

Copy Support files (Generic Data Collection)

• OS release information • Last 10000 system wide journal logs

• Disk free space • Last 10000 specific journal log
• Current memory utilization • /var/log/messages
• Process information (top) • Fastlog
• Systemctl status • OVSDB dump.
• Event Logs • Kernel information
• Full Show tech • HW information
• Proc information

#ArubaAirheads 10
ArubaOS-CX Troubleshooting tools
Copy Support files all (Generic Data Collection)
• OS release information
• Disk free space
• Current memory utilization
• Process information (top)
• Systemctl status
• Last 40000 system wide journal logs
• Kernel core-dump (if available)
• Kernel information
• Proc information
• Eevent and shutdown journal files
• Standby related(.tar) information
• Line modules related(.tar) information
• Systemctl status
• Event Logs
• Show tech output
• Audit log information (if available)
• Event log information
• Auth log information(if available)
• OVSDB dump
• Feature folder:
Latest current boot core-dump(if any) of
daemons part of that feature.
Diagnostic information of that feature.
Fast log of daemons part of that feature.
Last 10000 journal logs specific to daemon
#ArubaAirheads 11
ArubaOS-CX Troubleshooting tools

Copy Support files all – Files list in the support package

#ArubaAirheads 12
ArubaOS-CX Troubleshooting tools

Types of files present in support package

File Types Example Extraction File Types Example Extraction
SupportLog ops-switchd.supportlog fastlogParser Textfiles events.json Text editor/viewer
hpe-cardd.supportlog system_status.txt
hpe-powerd.supportlog hw_status.txt, dmesg.txt
Journalfiles system.journal journalctl --file showtech.txt,
hpe-cardd.journal processinfo.txt
hpe-mgmtmd.journal memoryinfo.txt, interrupts,
hpe-powerd.journal cpuinfo, diskinfo.txt ,
hpe-rdntmgmtd.journal sf.log, os-release.txt,
ops-switchd.journal version,cmdline
OVSDB Html Port.table.html Browser
Redundant_Management.t
able.html
Interface.table.html
Daemon.table.html
Subsystem.table.html
System.table.html

#ArubaAirheads 13
ArubaOS-CX Troubleshooting tools

Syntax
8400X# copy support-files
REMOTE_URL URL of syntax {tftp://|sftp://USER@}{IP|HOST}[:PORT][;blocksize=VAL]/FILE
STORAGE_URL URL of syntax {usb|flash}:/FILE
feature Capture feature specific information
module Module slot number
previous-boot Capture previous-boot information
all Copy all support files information
standby Copy standby module support files information
8400X# diag ovsdb-dump
REMOTE_URL URL of syntax {tftp://|sftp://USER@}{IP|HOST}[:PORT][;blocksize=VAL]/FILE
STORAGE_URL URL of syntax {usb|flash}:/FILE
columns Specify OVSDB Table columns to be captured (Default:all columns)
format Specify OVSDB data format (Default: table for display, html for copy)
table Specify OVSDB table to be captured (Default:all tables)
<cr>

#ArubaAirheads 14
ArubaOS-CX Troubleshooting tools

Example
copy support-files all sftp://steve@10.10.114.253//csf_20181114.tar.gz vrf mgmt.

copy support-files usb://8400-1-AMM-support-files.tar.gz

copy support-files all usb://8400-1-AMM-support-files-all.tar.gz
copy support-files previous usb://8400-1-LASTBOOT-support-files.tar.gz
copy support-files module 1/1 usb://8400-1-slot01-support-files.tar.gz
copy support-files module 1/2 usb://8400-1-slot02-support-files.tar.gz
copy support-files module 1/10 usb://8400-1-slot10-support-files.tar.gz

#ArubaAirheads 15
ArubaOS-CX Troubleshooting tools

Core dump

8400-1# show core-dump all

==================================================================================
Daemon Name | Instance ID | Present | Timestamp | Build ID
==================================================================================
Active management module
==================================================================================
hpe-mclagd 1326 Yes 2018-09-13 14:30:18 0db079f
==================================================================================
Total number of core dumps : 1
==================================================================================
8400-1# copy core-dump daemon hpe-mclagd:1326 usb://8400-1-hpe-mclagd.cor

#ArubaAirheads 16
ArubaOS-CX Troubleshooting tools

Troubleshooting Tools
• 8320# diagnostics <- Enable diag CLI command options
• 8320# diag
• diag Diagnostic commands
• diag-dump Display diagnostic dump
• diagnostics Change diagnostic commands availability
• 8320# diag
• crash-notification Configure crash console log • show Show running system information
• fan Show system fan status • utilities Linux tool utilities
• interface Interface information
• ledtest Diagnostic level of LED testing
• ospf OSPF information
• ospfv3 OSPF for IPv6

#ArubaAirheads 17
ArubaOS-CX Troubleshooting tools

Example

8320# diagnostic
8320# diag ovsdb-dump table Subsystem columns "name alias state"
Subsystem table
alias name state
----- ----- -----
"LC1" "1/1" ready
"MM1" "1/1" ready
base base ready

#ArubaAirheads 18
ArubaOS-CX Troubleshooting tools

Troubleshooting Tools
• 8320-3# diag-dump
• aaa Authentication Authorization& Accounting • mstp Multiple Spanning Tree Protocol
• acl Access control list • mvrp Multiple VLAN Registration Protocol
• arp ARP Manager Daemon • ntp Network Time Protocol
• bfd Bidirectional Fordwarding Detection • pim Protocol-Independent Multicast
• dhcp-relay Dynamic Host Configuration Relay • policy Classifier policy
• igmp Internet Group Management Protocol • qos Quality of Service
• interface Physical interfaces • rpvst Per VLAN Spanning Tree Protocol
• ipsec IP Security • sflow Sampled Flow Protocol
• l2mac Layer2 MAC • sys System Daemon and Mgmt Info
• l3 L3 Managers information • udld Unidirectional Link Detection Protocol
• l3port Layer 3 Port • vrrp Virtual Router Redundancy Protocol
• lag Link Aggregation Protocol • vsx Multi-Chassis Link Aggregation Group
• lldp Link Layer Discovery Protocol
• loop-protect L2 Loop protect

#ArubaAirheads 19
ArubaOS-CX Troubleshooting tools

Ecample
8320-3-P# diag-dump vsx basic
=========================================================================
[Start] Feature vsx Time : Tue Oct 30 01:26:29 2018
=========================================================================
-------------------------------------------------------------------------
[Start] Daemon hpe-vsxd
-------------------------------------------------------------------------
ISL statistics
===========================================
VSX protocol state :IN_SYNC
Lower edge of sender's window :8212
Upper edge of sender's window :8212
Maximum window size :250……
…
#ArubaAirheads 20
ArubaOS-CX Troubleshooting tools

Troubleshooting Tools
• 8320-3# debug
• acl Enable ACL debug logs • lag Enable LAG operations debug logs
• all Enable all debug modules • lldp Enable LLDP debug logs
• arp Enable ARP debug logs • loopprotect Enable Loop Protect debug logs
• bfd Enable BFD operations debug logs • mstp Enable MSTP debug logs
• bgp Enable BGP debug logs • mvrp Enable MVRP debug logs
• classifier Enable Classifier operations debug logs • ospfv2 Enable ospfv2 debug logs
• counters Enable Counters operations debug logs • ospfv3 Enable ospfv3 debug logs
• destination Configure the debug destination • pim Enable PIM debug logs
• dhcprelay Enable DHCP Relay debug logs • vlan Enable VLAN debug logs
• ipsec Enable IPsec debug log • vrf Enable VRF debug logs
• l2mac Enable L2 MAC feature debug logs • vrrp Enable VRRP debug logs
• l3 Enable L3 debug logs • vsx Enable VSX debug logs
• #ArubaAirheads 21
ArubaOS-CX Troubleshooting tools

Troubleshooting Tools
8320-3# debug lldp all
8320-3-P# show debug
-----------------------------------------------------------------------------------------------------------------------------------------------
module sub_module severity vlan port ip mac
instance vrf
-----------------------------------------------------------------------------------------------------------------------------------------------
lldp all debug ----- ----- ----- ----- -----

8320-3-P#
2018-10-30T01:38:23.063+00:00 lldpd[1275]: debug|LOG_DEBUG|AMM|-|LLDP|LLDP_CONFIG|No Port cfg changes
2018-10-30T01:38:23.063+00:00 lldpd[1275]: debug|LOG_DEBUG|AMM|-|LLDP|LLDP_EVENT|lldpd_stats_run entered at
time 6985888
2018-10-30T01:38:23.064+00:00 lldpd[1275]: debug|LOG_DEBUG|AMM|-|LLDP|LLDP_STATE|lldpd_ovsdb_nbrs_run
timestamps: cur=5bd7b60f last_update=5bd7b5f9 port_ttl=120 (sec)

#ArubaAirheads 22
 VSX Troubleshooting Demonstration
3810-2

VLAN20:
20.10.1.3/24
1 TRK1 2
VLAN20:
20.10.1.1/24 VLAN20:
Active gateway: 20.10.1.254 LAG 2 20.10.1.2/24
Active gateway: 20.10.1.254
Keepalive
1/1/4 1/1/6 1/1/6 1/1/4

VSX-Primary 1/1/53 1/1/53 VSX-Secondary

1/1/54 1/1/54 1/1/3

VLAN10: 1/1/3 VLAN10:
10.10.1.1/24 LAG10 10.10.1.2/24
Active gateway: 10.10.1.254 Active gateway: 10.10.1.254
LAG1

1 TRK1 2
1/1/1 1/1/2 1/1/1 1/1/2
IXIA 1/3/3 1/3/4 IXIA 1/3/1 1/3/2
3810-1 VLAN10:
10.10.1.4/24

#ArubaAirheads 23
VSX Troubleshooting Demonstration

ISL out-of-sync
–You can check isl status by executing the command show BSX status
– Check software version same on both the VSX switches
– Check platform name is same on both the VSX switches
– Check the link status of ISL link
– Check hello packet statistics to confirm packets are sent and received.
– Check vlan membership of ISL port by command show vsx config-consistency

#ArubaAirheads 24
VSX Troubleshooting Demonstration
8320-3-P# show vsx status
VSX Operational State
---------------------
ISL channel : In-Sync
ISL mgmt channel : operational
Config Sync Status : configuration_sync_missing_reference
NAE : peer_reachable
HTTPS Server : peer_reachable

Attribute Local Peer

------------ -------- --------
ISL link lag10 lag10
ISL version 2 2
System MAC d0:67:26:e2:77:f6 d0:67:26:e2:27:c2
Platform 8320 8320
Software Version TL.10.01.0040 TL.10.01.0040
Device Role primary secondary

#ArubaAirheads 25
VSX Troubleshooting Demonstration
8320-3-P# show vsx brief
ISL State : In-Sync
Keepalive State : Keepalive-Established
Device Role : primary
Number of Multi-chassis LAG interfaces : 2

8320-3-P# show vsx status config-sync

Admin state : Enabled
Operational State : Operational
Error State : Missing reference error
Recommended remediation : A. Execute 'show running-config vsx-sync peer-diff'
to determine which lines did not sync correctly on secondary
B. Identify the configuration that is missing from
secondary and manually fix via CLI.
Current time : Tue Oct 30 02:24:52 2018
Last sync time : Not available

#ArubaAirheads 26
VSX Troubleshooting Demonstration
8320-3-P# show running-config vsx-sync peer-diff
--- /tmp/running-config-vsx.5b20 2018-10-30 02:26:37.006262566 +0000
+++ /tmp/peer-running-config-vsx.5b20 2018-10-30 02:26:37.001262566 +0000
@@ -5,15 +5,14 @@
interface lag 1 multi-chassis
vsx-sync vlans
no shutdown
- description 8320-P_1/1/3____3800-1_1/1 VLAN10 LAG1
no routing
vlan trunk native 1
- vlan trunk allowed 10
+ vlan trunk allowed all
lacp mode active
+ lacp rate fast
interface lag 2 multi-chassis
vsx-sync vlans
no shutdown
- description 8320-P_1/1/4____3800-2_1/1 VLAN20 LAG2
no routing
vlan trunk native 1
vlan trunk allowed 20

#ArubaAirheads 27
VSX Troubleshooting Demonstration
8320-3-P# show vsx config-consistency
Configurations Local Peer
------------------------------------------------------------------------------------------
software version TL.10.01.0040 TL.10.01.0040
ISL hello interval 1 1
ISL dead interval 20 20
ISL hold interval 0 0
Keepalive hello interval 1 1
Keepalive dead interval 3 3
Keepalive UDP port 7678 7678

ISL VLAN list

1 10 20 30 40
Peer ISL VLAN list
1 20 30 40
Local VSX active-forwarding enabled interface-vlans
None
Peer VSX active-forwarding enabled interface-vlans
None

#ArubaAirheads 28
VSX Troubleshooting Demonstration
8320-3-P# show vsx config-consistency
Configurations Local Peer
------------------------------------------------------------------------------------------
software version TL.10.01.0040 TL.10.01.0040
ISL hello interval 1 1
ISL dead interval 20 20
ISL hold interval 0 0
Keepalive hello interval 1 1
Keepalive dead interval 3 3
Keepalive UDP port 7678 7678

ISL VLAN list

1 10 20 30 40
Peer ISL VLAN list
1 20 30 40
Local VSX active-forwarding enabled interface-vlans
None
Peer VSX active-forwarding enabled interface-vlans
None

#ArubaAirheads 29
VSX Troubleshooting Demonstration
8320-S# show vsx status
VSX Operational State
---------------------
ISL channel : In-Sync
ISL mgmt channel : operational
Config Sync Status : in-sync
NAE : peer_reachable
HTTPS Server : peer_reachable

Attribute Local Peer

------------ -------- --------
ISL link lag10 lag10
ISL version 2 2
System MAC d0:67:26:e2:27:c2 d0:67:26:e2:77:f6
Platform 8320 8320
Software Version TL.10.01.0040 TL.10.01.0040
Device Role secondary primary

#ArubaAirheads 30
Thank You