AX Troubleshooting

Section 9

1
Section objectives

Learn troubleshooting techniques and tools

Fix broken configuration

2
Troubleshooting philosophy

Bottom-up approach
Basic Network – L1 / L2 / L3
Server Status
TCP / UDP - L4
HTTP ( HTTPS ) - L7
Persistence L4/7
Performance

3
Problem identification

Description of the issue

Type of service
Identify the VIP / V-Port that has the issue
Identify affected features
Frequency
Determine frequency of problem occurrence
Scale
Check whether it happens to every client or only certain clients
Find the possible layer of issue & start there

4
Logs

Before starting the investigation look at the logs

AX logs many informational, warning, and error messages. The log is the first place to check
when experiencing any issues.
Port/Interface up/down messages
L2 loop detection warnings
Unicast/Multicast/Broadcast packet limit warnings
MAC address movement warnings
Duplicate IP warnings
Server & service port up/down messages
Application-specific error messages: SLB, PBSLB, HTTP, HA, and so on

5
Physical layer

Typical Symptoms
Slow Network ( High Latency )
Collisions
CRC Errors
Half Duplex
Speed
Interface Up & Down
Interface flapping
Any indication from show log
Configuration / Status on next hop switch / router
Any specific configuration on next hop’s interface

6
L2/3

Is VIP reachable via ping?

Look up the configured routes, learned MAC & ARP
Is IP checksum correct on every packet?
Check the access list
Is Ping always working?
Any flapping in MAC – interface
Any flapping in ARP – MAC
What’s the MAC / ARP learned on next hop?
Is it outbound traffic initiated by backend server?
Check the access list
Check the IP NAT configuration & usage

7
Server status

Health Check Results?

Health-check type, health-check up / down reason
Show log (log entries on up / down events)

8
TCP

Test TCP access

Use telnet [VIP] [v-port] to try establishing connection
Packet trace from client side
After sending SYN to VIP, any SYN-ACK / RST from VIP?
Is the connection terminated prematurely?
Responses from server & responses from VIP?
Do all the packets have proper IPs?
Is the TCP checksum correct?
Packet trace from server side (or AX)
Is AX able to establish connection to server?
Use axdebug

9
UDP

Packet trace from both client side & server side

What are the source and destination protocol ports?
Is server able to receive client’s packet?
Is client able to receive server’s packet?
Do all the packets have proper IPs?
Is the UDP checksum correct?
Use axdebug

10
HTTP

What are the L7 features enabled

Try without the advanced features first (compression, connection reuse, and so on)
Packet trace from client side & server side
Is server receiving the request sent by the AX?
Any standard HTTP header missing? (host, method, … and so on)
Do all of the HTTP headers have desired values?
Response Code from server’s response?
Size of request / response payload?
Is it taking a long time to process the request?
What are the cookies?
Use axdebug

11
HTTPS

Enabled Features?
Client SSL?
Server SSL?
Packet trace from client side & server side
Is client able to finish SSL Handshake with VIP?
Is AX able to finish SSL Handshake with server?
Any issues pertaining to redirect?
Decrypted trace
Are there any absolute links in Javascripts / Links / Images (http://xxx)?
Use axdebug

12
Performance

When?
When most of the traffic is working fine and you are trying to grab a quick picture of
everything
Performance Issue
Latency

13
Useful commands

Network
show int eth 1
show arp
show mac
show access-list
show ip route
show health stat
SLB
show slb […]
Performance
show memory system
show cpu

14
AXdebug

AXdebug
Captured files are in pcap format (Wireshark / tcpdump)
Able to see every detail of the packets the AX receives & sends
AXdebug is session based
If one pkt matches filter, dump all the following pkts in the same session

15
AXdebug: filters

Prepare capture
Capture file name
Filters
Multiple filters are “OR”-ed
AX# axdebug
AX(axdebug)# filter 1
AX(axdebug-filter:1)# ip 1.2.3.4 /32
AX(axdebug)# capture save file_name

Stop AXdebug trace

AX# no axdebug
Export AXdebug trace

16
Lab

Use session-control and packet-level CLI tools

17
Course map

Section 0: Course Introduction

Section 1: Load Balancing Concepts
Section 2: AX Management
Section 3: FTP
Section 4: HTTP
Section 5: HTTPS
Section 6: AX Acceleration
Section 7: AX Security
Section 8: High Availability
Section 9: AX Troubleshooting
Section 10: aFleX

18