Troubleshooting Wireless

LANs

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public 1
Troubleshooting Wireless LANs

Supportability
Software and Support Model
Troubleshooting Basics
The Client Debug
WLC Config Analyzer (WLCCA)
Additional Troubleshooting

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 2

Supportability

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 3

Supportability

WLC Supportability
Methods of Management
Using the GUI
Important Show Commands (CLI)
Important Debugs (CLI)
Best Practices

AP Supportability
Methods of Accessing theAP
Important Show Commands

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 4

WLC Supportability

Using the GUI

Monitor
AP/Radio Statistics
WLC Statistics
Client Details
Trap Log

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 5

WLC Supportability

Using the GUI

Wireless > All APs
AP list shows AP Physical UP Time
APs are sorted by Controller Associated Time
Check bottom of AP list for any recent AP disruptions
Select AP to see Controller Associated Time (duration)

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 6

WLC Supportability

Using the GUI

 Management
SNMP Config
Logs
Tech Support

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... 7

WLC Supportability

Important Show Commands (CLI)

Show run-config
Must have! No exceptions!
“show run-config commands” (like IOS show running-config)
“show run-config no-ap” (no AP information added)

Show tech-support
CLI Tip
Log all output
Config Paging Disable

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 8

WLC Supportability

Important Debugs (CLI)

Debug client <client mac address>
Client Involved? Must Have! No Exceptions

Debug capwap <event/error/detail/info> enable

CLI Tips
Log all output
Debugs are session based, they end when session ends
“Config session timeout 60”, sets 60 minute idle timeout
Debug mac addr <mac address>
Used to filter debugs on specific Mac Address
Debug disable-all (Disables all debugs)

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 9

WLC Supportability

Best Practices
Change default SNMP Parameters
Configure Syslog for WLC and AP
Enable Coredump for WLC and AP
Configure NTP Server for Date/Time

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 10

AP Supportability

Methods of Accessing theAP

Default Mode
Console (E)=Enabled (D)=Disabled
Telnet (D) / SSH (D)
No GUI support
AP Remote Commands

Enabling Telnet/SSH
WLC CLI: config ap [telnet/ssh] enable <ap name>
WLC GUI: Wireless > All APs > Select AP > Advanced
Select [telnet/ssh] > Apply

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 11

AP Supportability

AP Remote Commands (WLC CLI)

Debug AP enable <AP name>
Enables AP Remote Debug
AP Must be associated to WLC
Redirects AP Console output to WLC session

Debug AP command “<command>” <AP name>

Output is redirected to WLC session
AP runs IOS, numerous generic IOS commands available

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 12

AP Supportability

Show Commands (AP CLI or WLC Remote Cmd)

Show controller Do[0/1] (or Show Tech)
Must have! Before/During/After event

Show log
WLC: show ap eventlog <ap name>
Show capwap client <?>
CLI Tips
Debug capwap console client
Debug capwap client no-reload

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 13

Troubleshooting Basics

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 14

The Client Debug

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 15

The Client Debug
debug client <mac address>
A multi-debug macro
(Cisco Controller) >debug client 00:16:EA:B2:04:36
(Cisco Controller) >show debug
MAC address ................................ 00:16:ea:b2:04:36
Debug Flags Enabled:
dhcp packet enabled
dot11 mobile enabled
dot11 state enabled
dot1x events enabled
dot1x states enabled
pem events enabled
pem state enabled
CCKM client debug enabled

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 16

 Steps to Building an 802.11 Connection
802.11
1. Listen for Beacons
State 1:
Unauthenticated, 2. Probe Request
Unassociated
3. Probe Response AP

4. Authentication Request
5. Authentication Response
State 2:
Authenticated, 6. Association Request
Unassociated
7. Association Response
WLC

State 3:
8. (Optional: EAPOLAuthentication)
Authenticated,
Associated 9. (Optional: Encrypt Data)
10. Move User Data
© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 17
 Understanding the Client State
Name Description
8021X_REQD 802.1x (L2) Authentication Pending
DHCP_REQD IP Learning State
WEBAUTH_REQD Web (L3) Authentication Pending
RUN Client Traffic Forwarding

(Cisco Controller) >show client detail 00:16:ea:b2:04:36

Client MAC Address............................... 00:16:ea:b2:04:36
…..
Policy Manager State............................. WEBAUTH_REQD

00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 18

The Client Debug - Walkthrough
Association (Start)
L2 Authentication (8021X_REQD)
Client Address Learning (DHCP_REQD)
L3 Authentication (WEBAUTH_REQD)
Client Fully Connected (RUN)
Deauth/Disassoc
Tips and Tricks

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 19

Client Debug - Association

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 20

 Association
(Cisco Controller) >debug client 00:16:EA:B2:04:36
(Cisco Controller) >
(Cisco Controller) >
Association received from mobile on AP 00:26:cb:94:44:c0
0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1621)
Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface '3'
Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3‘

STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0

Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36

0.0.0.0 START (0) Initializing policy

0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)
0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)
0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client
0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1
apfMsAssoStateInc
apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle toAssociated
Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds

Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 21

 Association
Association received from mobile on AP 00:26:cb:94:44:c0
0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1621)
Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface '3'
Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3'

Association received
Association Request, client did not “Roam” (Reassociate)
AP Base Radio = 00:26:cb:94:44:c0

vapId 1, site 'default-group', interface '3‘

vapId = WLAN # (Wlan 1)
site = AP Group (default-group)
Interface = Dynamic Interface name (3)

vlan 3
Vlan = Vlan # of Dynamic Interface

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 22

 Association
STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0
Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36

STA - rates
Madatory Rates (>128) = (#-128)/2
Supported Rates (<128) = #/2
1m,2m,5.5m,11m,6s,9s,12s,18s,24s,36s,48s,54s

Processing RSN IE type 48

WPA2-AES
Processing WPA IE type 221 = WPA-TKIP

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 23

 Association
0.0.0.0 START (0) Initializing policy
0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)
0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)
0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client
0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1
apfMsAssoStateInc
apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle toAssociated
Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds

0.0.0.0 START
0.0.0.0 = IP we know for client (In this case nothing)

Change state to 8021X_REQD

Passed association, moving client to next state: 8021X_REQD

Scheduling deletion
Session Time on WLAN (1800 seconds in this case)

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 24

 Association
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

Slot 0 = B/G(2.4) Radio

Slot 1 = A(5) Radio

Sending Assoc Response Status 0 = Success

Anything other than Status 0 is Failure

Common Assoc Response Failures:

1 – Unknown Reason – Anything not matching defined reason codes
12 – Unknown or Disabled SSID
17 – AP cannot handle any more associations
18 – Client is using a datarate that is not allowed
35 – WLAN requires the use of WMM and client does not support it
201 – Voice client attempting to connect to a non-platinum WLAN
202 – Not enough available bandwidth to handle a new voice call (CAC Rejection)

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 25

 Association - FSR
Processing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36
CCKM: Mobile is using CCKM
CCKM: Processing REASSOC REQ IE
Including CCKM Response IE (length 62) in Assoc Resp to mobile
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) Vap Id 6 Slot 1
OR
Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36
Received RSN IE with 1 PMKIDs from mobile 00:16:ea:b2:04:36
Received PMKID: (16)
[0000] cb bc 27 82 88 14 92 fd 3b 88 de 6a eb 49 be c8
Found an entry in the global PMK cache for station
Computed a valid PMKID from global PMK cache for mobile

FSR aIOS CUWN

CCKM - WPA yes yes

CCKM - WPA2 yes yes

WPA2 PKC no yes

WPA2 "Sticky" yes no

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 26
Association - Takeaway
Association vs. Reassociation
Debug shows
AP, Slot, AP-Group, WLAN ID, Interface, Data Rates,
Encryption type
Association Response
Confirms if Client is associated
Defines reason if denied

Further troubleshooting
May require Wireless Sniffer or capture at AP Switchport
If not sending Assoc Request, must know why from Client
Trying disabling WLAN features to “dumb it down”

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 27

Client Debug –
L2 Authentication

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 28

 802.1X Authentication
Supplicant
Authenticator Server

EAPOL-START

EAP-ID-Request

EAP-ID-Response
RADIUS (EAP-ID_Response)

Rest of the EAP Conversation

Radius-Access-Accept
EAP-Success
(Key)
The Supplicant Derives the
Session Key from User Password or Session
Certificate and Authentication Key
Exchange
© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 29
WPA2-AES-802.1X
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

Station 00:16:ea:b2:04:36 setting dot1x reauth timeout = 1800

dot1x - moving mobile 00:16:ea:b2:04:36 into Connecting state
Sending EAP-Request/Identity to mobile 00:16:ea:b2:04:36 (EAP Id 1)
Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36
Username entry (cisco) created for mobile
Received Identity Response (count=1) from mobile 00:16:ea:b2:04:36
EAP State update from Connecting to Authenticating for mobile 00:16:ea:b2:04:36
dot1x - moving mobile 00:16:ea:b2:04:36 into Authenticating state
…………………..
Entering Backend Auth Req state (id=3) for mobile 00:16:ea:b2:04:36
Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3)
Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36
Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25)
...........................
Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 10, EAP Type 25)
Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36
Processing Access-Challenge for mobile 00:16:ea:b2:04:36
Entering Backend Auth Req state (id=11) for mobile 00:16:ea:b2:04:36
Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 11)
Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36
Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 11, EAP Type 25)
Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36
Processing Access-Accept for mobile 00:16:ea:b2:04:36

***OR***
Processing Access-Reject for mobile 00:16:ea:b2:04:36

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 30

 Common EAP Types
 1 – Identity

 2 – Notification

 3 – NAK Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3)

Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36
 4 – MD5 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type
25)
 5 – OTP

 6 – Generic Token

 13 – EAP TLS

 17 – LEAP

 18 – EAP SIM

 21 – EAP TTLS

 25 – PEAP

 43 – EAP-FAST

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 31

 802.1X (Cont.) (WPA2-AES-PSK)
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0
Creating a PKC PMKID Cache entry for station 00:16:ea:b2:04:36 (RSN 2)
Adding BSSID 00:26:cb:94:44:c0 to PMKID cache for station 00:16:ea:b2:04:36
New PMKID: (16)
[0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd
Initiating RSN PSK to mobile 00:16:ea:b2:04:36
dot1x - moving mobile 00:16:ea:b2:04:36 into Force Auth state
Skipping EAP-Success to mobile 00:16:ea:b2:04:36
Including PMKID in M1 (16)
[0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd
Starting key exchange to mobile 00:16:ea:b2:04:36, data packets will be dropped

Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36

state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00
Received EAPOL-Key from mobile 00:16:ea:b2:04:36
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36
Received EAPOL-key in PTK_START state (message 2) from mobile 00:16:ea:b2:04:36
Stopping retransmission timer for mobile 00:16:ea:b2:04:36
Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36
state PTKINITNEGOTIATING (message 3), replay counter 00.00.00.00.00.00.00.01
Received EAPOL-Key from mobile 00:16:ea:b2:04:36
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36
Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile 00:16:ea:b2:04:36
apfMs1xStateInc
0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state L2AUTHCOMPLETE (4)

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 32

 WPA2-AES-PSK - Failed
Starting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be dropped

Sending EAPOL-Key Message to mobile 00:1e:8c:0f:a4:57

state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00
Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57
802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57
Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57
Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57
Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57
Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57
802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57
Retransmit 2 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57
…………………
802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57
Retransmit failure for EAPOL-Key M1 to mobile 00:1e:8c:0f:a4:57,
retransmit count 3, mscb deauth count 3
Blacklisting (if enabled) mobile 00:1e:8c:0f:a4:57
apfBlacklistMobileStationEntry2 (apf_ms.c:4192) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:16:9c:4b:c4:c0 from Associated to Exclusion-list (1)

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 33

EAP Timers
Show advanced eap

Config advanced eap <timer name> <value>

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 34

L2 Authentication - Takeaway
8021X_REQD means L2 Authentication pending
Authentication/Encryption has not be established

PSK is 802.1X, key is derived from PSK not AAA

If “Processing Access-Reject”
AAA/RADIUS Rejected the user (not the WLC)
If “Processing Access-Accept”
AAA/Radius Accepted the user
M1-M4 should follow

Further Troubleshooting
Debug aaa [all/event/detail/packet] enable
Debug dot1x [aaa/packet] enable

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 35

Client Debug –
IP Learning State

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 36

 Client DHCP
00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state
00:16:ea:b2:04:36 apfMs1xStateInc
00:16:ea:b2:04:36 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4)
00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 3 apVapId 3for this client
00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 3 apVapId 3
00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7)
00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4755, Adding TMP rule
00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
00:16:ea:b2:04:36 Stopping retransmission timer for mobile 00:16:ea:b2:04:36
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0
...................
00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 29, encap 0xec03)
...................
00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)
...................
00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 37

 Client DHCP

Client State =
Client is in DHCP_REQD state “DHCP_REQD“

Proxy Enabled: DHCP Proxy Enabled DHCP Proxy Disabled

DHCP Relay/Proxy
Between WLC and Server Client DHCP Discover
Unicast to DHCP Client DHCP Discover Is
Bridged to DS
Required for Internal DHCP Servers

Proxy Disabled: DHCP Offer from Server

Between Client and Server
DHCP is broadcast out VLAN Client DHCP Request

IP helper or other means required

DHCP ACK from Server

IP Address Learned

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 38

 DHCP Proxy Enabled – DHCP Discover
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

32.151: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)
32.151: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings:
dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,
dhcpGateway: 0.0.0.0, dhcpRelay: 0.0.0.0 VLAN: 0
32.151: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.1
(local address 10.10.1.4, gateway 10.10.1.1, VLAN 0, port 29)
32.151: 00:16:ea:b2:04:36 DHCP transmitting DHCP DISCOVER (1)
32.151: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 1
32.151: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0
32.152: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
32.152: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
32.152: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.4
32.152: 00:16:ea:b2:04:36 DHCP requested ip: 10.99.76.147
32.152: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.1 (len 346, port 29, vlan 0)
32.152: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings:
dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,
dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 0
32.152: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 39

 DHCP Proxy Enabled – DHCP Offer
166.: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)
34.166: 00:16:ea:b2:04:36 DHCP setting server from OFFER (server 10.10.1.3, yiaddr 10.10.1.103)
34.167: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)
167. : 00:16:ea:b2:04:36 DHCP transmitting DHCP OFFER (2)
34.167: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0
34.167: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0
34.167: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
34.167: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.1.103
34.167: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
34.168: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.3

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 40

 DHCP Proxy Enabled – DHCP Request
38.169: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)
38.169: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings:
dhcpServer: 10.10.1.3, dhcpNetmask: 0.0.0.0,
dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 0
38.169: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.3
(local address 10.10.1.4, gateway 10.10.1.3, VLAN 0, port 29)
38.169: 00:16:ea:b2:04:36 DHCP transmitting DHCP REQUEST (3)
38.169: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 1
38.170: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0
38.170: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
38.170: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
38.170: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.4
38.170: 00:16:ea:b2:04:36 DHCP requested ip: 10.10.1.103
38.170: 00:16:ea:b2:04:36 DHCP server id: 10.10.1.3 rcvd server id: 1.1.1.1
38.170: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.3 (len 354, port 29, vlan 0)
38.170: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings:
dhcpServer: 10.10.1.3, dhcpNetmask: 0.0.0.0,
dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 0
38.171: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 41

 DHCP Proxy Enabled – DHCP Ack
172.: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)
38.173: 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Reached PLUMBFASTPATH: from line 5273
38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Replacing Fast Path rule
38.173: 00:16:ea:b2:04:36 Assigning Address 10.10.1.103 to mobile
173.: 00:16:ea:b2:04:36 DHCP success event for client. Clearing dhcp failure count for interface managemen
38.174: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)
174. : 00:16:ea:b2:04:36 DHCP transmitting DHCP ACK (5)
38.174: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0
38.174: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0
38.174: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
38.174: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.1.103
38.174: 00:16:ea:b2:04:36 DHCP siaddr: 10.10.1.30, giaddr: 0.0.0.0
38.174: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.3
38.179: 00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 42

 DHCP Proxy Disabled – Discover/Offer

*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)
*00:16:ea:b2:04:36 DHCP processing DHCP DISCOVER (1)
*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0
*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0
*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86
*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS

*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)
*00:16:ea:b2:04:36 DHCP processing DHCP OFFER (2)
*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0
*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 0, flags: 0
*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.3.86
*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3
*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 43

 DHCP Proxy Disabled – Request/Ack
*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)
*00:16:ea:b2:04:36 DHCP processing DHCP REQUEST (3)
*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0
*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0
*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86
*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3
*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS

*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)
*00:16:ea:b2:04:36 DHCP processing DHCP ACK (5)
*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0
*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 0, flags: 0
*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36
*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.3.86
*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3
*00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
*00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile
*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA
*00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 44

 Learning IP without DHCP
*Orphan Packet from 10.99.76.147 on mobile
*0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
*Installing Orphan Pkt IP address 10.99.76.147 for station
*10.99.76.147 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

Client IP can be learned by ways other than DHCP

Client sends gratuitous ARP or ARP Request (Static Client)
Client sends IP packet (Orphan Packet), we learn IP
DS sends packet to client, we learn IP from DS

Seen with mobile devices that talk before validating DHCP

Up to client to realize their address is not valid for the subnet
DHCP Required on WLAN for prevent this

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 45

Client DHCP - Takeway
DHCP_REQD means Learning IP State
Only “Required” if enabled on WLC

If Proxy is enabled

Confirm DHCP Server on Interface (or Wlan) is correct
DHCP Server may not respond to WLC Proxy (Firewalls?)

If Proxy is disabled, DHCP is similar to wired client

Further Troubleshooting
Check DHCP Server for what it believes is happening
If WLC does not show a BOOTREQUEST, confirm the client
request arrives to the WLC and leaves in the configured way
If still believed to be on WLC: debug dhcp message enable

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 46

Client Debug –
L3 Authentication

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 47

 Webauth
*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)
……………………………...
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to
WEBAUTH_REQD (8) last state WEBAUTH_REQD (8)
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) pemAdvanceState2 5170, Adding TMP rule
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Successfully plumbed mobile rule (ACL ID 255)
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 Assigning Address 10.10.3.86 tomobile
*pemReceiveTask: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 2, dtlFlags 0x0
*pemReceiveTask: 00:16:ea:b2:04:36 Sent an XID frame
*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile
*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile
*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile
………………………………
*emWeb: 00:16:ea:b2:04:36 Username entry (cisco) created for mobile
*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14) last state
WEBAUTH_NOL3SEC (14)
*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last
state RUN (20)
*emWeb: 00:16:ea:b2:04:36 Session Timeout is 1800 - starting session timer for the mobile
*emWeb: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063
*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID=
5006 IPv6 Vlan = 3, IPv6 intf id = 8
*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Successfully plumbed mobile rule (ACL ID 255)
*pemReceiveTask: May 17 22:25:16.578: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1,
dtlFlags 0x0

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 48

 Webauth Redirect Webauth

Client State =
Client in WEBAUTH_REQD state “WEBAUTH_REQD“

ARP and DNS must be functional ARP and DNS Function

Client attempts to browse internet 3-Way Handshake HTTP

HTTP GET
WLC “Hijacks” the handshake 200 Response

Client redirects to Virtual Interface 3-Way Handshake

HTTP(S) GET
Certificate negotiation if applicable
Webauth Page Displayed
Webauth page is displayed
Client authenticates Successful Authentication

Client State = “RUN“

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 49

 ARP and DNS Function

Confirm ARP and DNS Function

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public 50

 3-Way Handshake
HTTP GET
200 Response
3-Way Handshake
HTTP(S) GET

Capture from Wireless Adapter Webauth Page Displayed

Webauth Redirect
WLC Responding
with SYN, ACK
Redirect to Virtual
Interface Comes
from Here
WLC Responding
with SYN, ACK

Client Is Talking to
Webauth….

Address for Client

to Redirect to
(Virtual IP/Name)
BRKEWN-3011 2019 Cisco
© 2011 Cisco and/or
and/or its
its affiliates.
affiliates. All
Trainer-Amuya....
rights reserved. Cisco Public 60
Webauth - Takeaway
If WEBAUTH_REQD, then not authenticated
Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv6*

If not redirected, can client browse to virtual IP?

Cert issue? Consider disabling HTTPS for HTTP webauth
Most common scenario involves ARP/DNS failure
Must confirm that client actually sends TCP SYN (http) to IP

If proven that TCP SYN is sent and WLC does not SYN
ACK, then there may be a WLC side problem
Debug webauth enable <client ip address>
debug client <MAC Address>
debug pm ssh-appgw enable
debug pm ssh-tcp enable
BRKEWN-3011 2019 Cisco and/or its affiliates. All
© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Client Debug - Run

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Run State
10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
10.10.3.82 RUN (20) Reached PLUMBFASTPATH: from line 5273
10.10.3.82 Added NPU entry of type 1, dtlFlags 0x0

OR

10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14)

10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20)
Session Timeout is 1800 - starting session timer for the mobile
10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063
10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0

RUN State is the Client Traffic Forwarding State

Client is Connected and should be functional

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Client Debug – Deauth/
Disassoc

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Deauthenticated Client
Idle Timeout
Occurs after no traffic received from Client
Default Duration is 300 seconds
Received Idle-Timeout from AP 00:26:cb:94:44:c0, slot 0 for STA 00:1e:8c:0f:a4:57
apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4
Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

Session Timeout
Occurs at scheduled duration (default 1800 seconds)
Will force WEBAUTH user to WEBAUTH again
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:26:cb:94:44:c0 from Associated to Disassociated
Scheduling deletion of Mobile Station: (callerId: 45) in 10 seconds
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Deauthenticated Client
WLAN Change
Modifying a WLAN in anyway Disables and Renables WLAN
apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile
00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

Manual Deauth
From GUI: Remove Client
From CLI: config client deauthenticate <mac address>
apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1
Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Deauthenticated Client

Authentication Timeout
Auth or Key Exchange max-retransmissions reached
Retransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3,
mscb deauth count 0
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)

AP Radio Reset (Power/Channel)

AP disasassociates clients but WLC does not delete entry
Cleaning up state for STA 00:1e:8c:0f:a4:57 due to event for AP 00:26:cb:94:44:c0(0)
apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile
00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Deauthentication - Takeaway

Client can be removed for numerous reasons

WLAN change, AP change, configured interval

Start with Client Debug to see if there is a reason

for a client’s deauthentication
Further Troubleshooting
Client debug should give some indication of what kind of
deauth is happening
Packet capture or client logs may be require to see exact
reason

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Client Debug – Tips and Tricks

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Tips and Tricks

Collect a client debug for an extended duration

Several roams, deauths, failures, etc…

Use an enhanced text editor with filter or “find all”

I use Notepad++

Find All
“Association Received” (will also pull reassociations)
“Assoc Resp”
“Access-Reject”
“timeoutEvt”

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Tips and Tricks

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Tips and Tricks

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Client Debug – Summary

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Client Connectivity

Unified Wireless Network: Troubleshoot Client

Issues Document ID: 107585
Configuration Issues
SSID Mismatch
Security Mismatch
Disabled WLAN
Unsupported Data-Rates
Disabled Clients
Radio Preambles

Cisco Features - Issues with Third Party Clients

Aironet IE
MFP
BRKEWN-3011 2019 Cisco and/or its affiliates. All
© 2011 Trainer-Amuya....
rights reserved. Cisco Public
802.11n Speeds

Troubleshoot 802.11n Speeds Document ID: 112055

Configuration Issues
11n Support Enabled
WMM is Allowed or Required
Open or WPA2-AES
5Ghz Channel Width
2.4Ghz does not support 40-Mhz Channels

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
WLC Config Analyzer (WLCCA)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 What Is the WLCCA?
It is a Post Sales tool
Main objective: Save time while analyzing configuration files
from WLCs
Secondary objective: Carry out RF analysis
It is NOT a management or monitoring tool
Focused to work off-line to the WLC
Not TAC supported
“Pet project”: no official Cisco product.

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Where?
Support Forums DOC-1373

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Functionality Overview - Checks
Audit Checks
More than 100 config detail verifications
Based on TAC/Escalation cases experience
Some obvious, some hard to catch
No “change this” messages, some need “contextualization”

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Functionality Overview
Audit Checks

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Audit Check Example
Symptom: APs are Layer 2 to WLC, but disconnect from
WLC when a client associates. Packets start going to GW.
Client Vlan Information
Client Gateway: 10.225.18.129 (/25)
AP Information
AP IP: 10 .255 17.205 (/25)
AP Gateway: 10 .
.225 17.129 (/25)
WLCCA Audit Checker now warns of this for everyAP
. with
incorrect/invalid IP information

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Functionality Overview
Config View

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
WLCCA – High RF Index APs

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Reducing CCI

Turn off excess 2.4 radios. May want to do this

gradually, e.g. turn off 20% of radios per attempt
After turning off excess radios, could set DCA
sensitivity to high
Let DCA/power settings settle down overnight.
See how things look in the morning
Repeat till you see the desired coverage in 2.4GHz

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 2.4GHz – Target Coverage

Most all 2.4GHz radios are at power 2 - 5 (don't want 7

or 8)
In all locations, you have coverage that looks like this
(take these as guidelines, not gospel):
Hottest channel's AP is at least -67dBm
Next hottest AP on that channel is at least 19 dB below the hottest
Next hottest channel's AP is at least -67dBm
OK if next hottest AP on that channel is less than 19 dB below the hottest

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
5 GHz – Target Coverage
Most all 5GHz radios are at power 1 – 3 (at
least 14dBm)
Consider the RRM min power setting in 6.0
Consider a radically high tx-power-threshold, like -55
dBm
8 – 12 channels in use (20 seem to be too
many for the 792x to scan)
In all locations, seek this:
Hottest channel's AP is at least -67dBm
Next hottest AP on that channel is at least 19 dB below the hottest
Next hottest channel's AP is at least -67dBm
OK if next hottest AP on that channel is less than 19 dB below the hottest

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Additional Troubleshooting

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Additional Troubleshooting

Wireshark Tutorial
Clean Air SE-Connect / AP Sniffer Mode
AP Join
RRM
Multicast/Broadcast
Mobility
VoWiFi

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Wireshark Tutorial

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Wireshark Tutorial
Default Wireshark view might look like this:

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Wireshark Tutorial

Newer versions of Wireshark have a feature for

“Apply as Column”
This will take any decodable parameter and make a column

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Wireshark Tutorial
Within seconds your wireshark can also have:

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Wireshark Tutorial - CAPWAP
User data is encapsulated in CAPWAP

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Wireshark Tutorial
Wireshark can also de-encapsulate CAPWAP DATA
Edit > Preference > Protocols > CAPWAP

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Wireshark Tutorial
With CAPWAP de-encapsulated you can see all the
packets to/from client (between AP and WLC)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
SE-Connect – Clean Air

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
SE-Connect

Clean Air APs can be used in lieu of Spectrum Card

for Spectrum Analysis
AP can be placed in SE-Connect mode for full functionality
AP in local mode can be used now for Spectrum Analysis of
current channel

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Spectrum Expert with Clean Air

Obtain Spectrum Key

Connect to Remote Sensor

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Spectrum Expert with Clean Air

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
AP Discover/Join

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
AP Discover/Join
AP Runs Hunting
Algorithm to Find
Candidate Controllers
to Join

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
AP - Discover Process
Discovery Req sent to known and learned WLCs
Broadcast
Reaches WLCs with MGMT Interface in local subnet of AP
Use “ip helper-address <ip>” with “ip forward-protocol udp”

Dynamic
DNS: cisco-capwap-controller.domain
DHCP: Option 43

Configured (nvram)
High Availability WLCs – Pri/Sec/Ter/Backup
Last WLC
All WLCs in same mobility group as last WLC
Manual from AP - “capwap ap controller ip address <ip>”
BRKEWN-3011 2019 Cisco and/or its affiliates. All
© 2011 Trainer-Amuya....
rights reserved. Cisco Public
AP - Discover Process

broadcast
X

Discover Request sent to all methods the AP knows

Discover Response sent from all WLCs that
received the Discovery Request

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 AP – WLC Selection/Join
WLCs send Discovery Response back to AP
Name, Capacity, AP Count, Master?, AP-MGR, Load perAP-
MGR
AP selects the single best WLC candidate from
High Availability Config: Primary/Secondary/Tertiary/Backup
Master Controller
Greatest available capacity
Ratio of total capacity to available capacity

AP sends single Join Request to best candidate

WLC responds with Join Response
AP joins and receives config (or downloads image if not correct)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Troubleshooting AP Discovery/Join

“Lightweight AP (LAP) Registration to a Wireless LAN

Controller (WLC)”, Document ID 70333
Make sure time on WLC is accurate!
From AP:
Debug ip udp
Debug capwap client events

From WLC
Debug mac addr <AP ethernet mac>
Debug capwap [event/error/packet] enable
Debug pm pki enable

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Client Mobility

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Mobility—Intra-Controller

Client roams between two APs on the same controller

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Mobility—Inter-Controller (Layer 2)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Mobility—Layer 3

Layer 3 roaming (a.k.a. anchor/foreign)

New WLC does not have an interface on the subnet the client is on
New WLC will tell the old WLC to forward all client traffic to the
new WLC

Asymmetric
traffic path
established
(deprecated)
Symmetric
traffic path

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Mobility—Messaging Flow

When a client connects to a WLC for the first time,

the following happens:
New WLC sends MOBILE_ANNOUNCE to all controllers in
the mobility group when client connects
Old WLC sends HANDOFF_REQUEST
New WLC sends HANDOFF_REPLY

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Debug Client <Mac Address>
Mobility— L2 Inter WLC Debug Mobility Handoff Enable

WLC 1 (From) WLC 2 (To)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Debug Client <Mac Address>
Mobility— L3 Inter WLC Debug Mobility Handoff Enable

WLC 1 (From) WLC 2 (To)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Debug Client <Mac Address>
Mobility— L3 Inter WLC Debug Mobility Handoff Enable

WLC 1 (From) WLC 2 (To)

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Mobility— L3 Handoff Ignored
WLC 1 (From)
*mmListen: Mobility packet received from:
*mmListen: 10.4.22.55, port 16666
*mmListen: type: 3(MobileAnnounce) subtype: 0 version: 1 xid: 783 seq: 1453 len 116 flags 0
*mmListen: group id: e42cb3a9 87f62b45 57c0f8a3 92747b23
*mmListen: mobile MAC: 00:23:33:41:71:10, IP: 0.0.0.0, instance: 0
*mmListen: VLAN IP: 10.4.23.97, netmask: 255.255.255.0
*mmListen: Switch IP: 10.4.22.55
*mmListen: Handoff Virtual IP Mismatch, Local = 1010101, Request = 1020304
**** Handoff Request Ignored
*apfReceiveTask: 10.4.122.127 RUN (20) State Update from Mobility-Complete to Mobility-Incomplete
*apfReceiveTask: Mobile 00:23:33:41:71:10 associated with another AP elsewhere, delete mobile
*apfReceiveTask: 10.4.122.127 RUN (20) mobility role update request from Local to Handoff
Peer = 0.0.0.0, Old Anchor = 10.4.130.70, New Anchor = 0.0.0.0
*apfReceiveTask: Clearing Address 10.4.122.127 on mobile
*apfReceiveTask: apfMsRunStateDec
*apfReceiveTask: 10.4.122.127 RUN (20) Change state to DHCP_REQD (7) last state RUN (20)
*apfReceiveTask: apfMmProcessDeleteMobile (apf_mm.c:548) Expiring Mobile!
*apfReceiveTask: Mobility Response: IP 0.0.0.0 code Handoff Indication (2), reason Client handoff successful -
anchor retained (0), PEM State DHCP_REQD, Role Handoff(6)
*apfReceiveTask: apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:23:33:41:71:10 on
AP 10:8c:cf:eb:69:80 from Associated to Disassociated
*apfReceiveTask: Deleting mobile on AP 10:8c:cf:eb:69:80(1)
*pemReceiveTask: 0.0.0.0 Removed NPU entry.

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Mobility Group vs. Mobility Domain
Mobility Group - WLCs with the same group name
L2/L3 Handoff
Auto Anchoring
Fast Secure Roaming
APs get all of these as a Discover candidate

Mobility Domain - WLCs in the mobility list

L2/L3 Handoff
Auto Anchoring

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
 Mobility Data/Control Path
Sent between all WLCs, by member with lowest MAC
Control Path = UDP 16666 (30 Seconds)
Data Path = EoIP Protocol 97 (10 Seconds)
debug mobility keep-alive enable <IP Address>

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
Voice over WiFi

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
VoWiFi

Wireless IP Phone Deployment Guide

http://www.cisco.com/en/US/docs/voice_ip_comm/cuipph/
7925g/7_0/english/deployment/guide/7925dply.pdf

Best Practices
-67 dBm signal with 20-30% cell overlap
802.11A
CCKM for Fastest Roaming
Avoid designs where AP is seen at superb signal, but drops
off instantly

BRKEWN-3011 2019 Cisco and/or its affiliates. All

© 2011 Trainer-Amuya....
rights reserved. Cisco Public
VoWiFi - Troubleshooting

Must know if problem occurs during roaming events

or when no association change takes place
If no change in connection
Interference
Coverage loss with no other candidate
End to End QOS missing/problem

If during roaming event

How long did the roam take?
Does the client associate to another AP again within
seconds?
Does the client associate to the same AP again?
Is the phone roaming to the designed next candidate?
BRKEWN-3011 2019 Cisco and/or its affiliates. All
© 2011 Trainer-Amuya....
rights reserved. Cisco Public
VoWiFi - Troubleshooting
Define a reproducible area where you believe you
have perfect voice coverage but have problems
Place phone in Neighbor List Mode (On a call)
Real Time current AP RSSI and candidate list
Confirm AP as next best candidate is realistically a good
candidate
Confirm devices roams to correct candidate where the
intended design specifies

Watch out for sudden drops in coverage

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public

VoWiFi - Debugs
Phone can Trace (debug) to file or syslog
Recommend USB Connection and SYSLOG
Configured via GUI
Enable Debug level for Kernel, WLAN MGR, WLAN Driver
WLC Debugs
Debug client <mac>
Debug cac all enable

Wireless Packet Captures

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public

Summary

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public

Summary
Client
WLC - show run-config, debug client <mac>, debug dhcp message enable,
debug dot1x <?> enable, debug aaa <?> enable,
AP - Show tech, show controller D<0/1>
Data - Driver/Supplicant Logs, Wireless Capture, AAA Logs, DHCP Logs
Webauth
WLC - (Client debugs), debug webauth enable <IP>, debug pm ssh-appgw enable,
debug pm ssh-tcp enable
Client - local capture
Mobility
WLC - debug mobility handoff enable, debug mobility keepalive enable <IP>
Data - Wired capture
AP Join
WLC - debug capwap [events/error/packet] enable
AP - debug capwap client events, debug ip udp
Data - Wired capture
RRM
WLC - show run-config, debug airewave-director <?>
AP - debug capwap rm measurements, debug capwap rm rogue
Multicast/Broadcast
AP - show capwap mcast, show capwap mcast mgid all
Data - Infrastructure Configuration
Voice
WLC - (Client debugs), debug cac all enable
Data – Wireless capture, Phone traces
© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public
Summary
Links:
Understanding Debug Client on Wireless LAN Controllers
(WLCs) Document ID: 100260

Unified Wireless Network: Troubleshoot Client Issues

Document ID: 107585

Troubleshoot 802.11n Speeds Document ID: 112055

Troubleshoot a Lightweight Access Point Not Joining a

Wireless LAN Controller Document ID: 99948

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public

Join Cisco Support Communities!
 Free for anyone with Cisco.com
registration
 Get timely answers to your technical
questions
 Find relevant technical
documentation Documents Blogs
 Engage with over 200,000 top
technical experts Ask the Expert Video

 Seamless transition from discussion

to TAC Service Request (Cisco Mobile Discussions
customers and partners only)
 Visit the Cisco Support Community
booth in the World of Solutions for
more information

The Cisco Support Community is your one-stop

supportforums.cisco.com community destination from Cisco for sharing
supportforums.cisco.mobi current, real-world technical support knowledge
with peers and experts.

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public

Complete Your Online
Session Evaluation

Receive 25 Cisco Preferred Access points for each session

evaluation you complete.
Give us your feedback and you could win fabulous prizes.
Points are calculated on a daily basis. Winners will be notified
by email after July 22nd.
Complete your session evaluation online now (open a browser
through our wireless network to access our portal) or visit one
of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live and Networkers Virtual
account for access to all session materials, communities, and
on-demand and live activities throughout the year. Activate
your account at any internet station or visit
www.ciscolivevirtual.com.

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public

 Visit the Cisco Store for
Related Titles
http://theciscostores.com

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public
Thank you.

© 2019 Cisco and/or its affiliates. Trainer-Amuya.... Cisco Public