Advanced Wireless Troubleshooting: Tim Smith, Technical Consulting Engineer

Advanced Wireless
Troubleshooting
Tim Smith, Technical Consulting Engineer
DGTL-BRKEWN-3011
#CiscoLive
Agenda
• Section 1 • Section 4
➢ Where do we start? ➢ Device Troubleshooting
• Section 2 ➢ AP Join Troubleshooting
➢ Client On-Boarding • Section 5

• Section 3 ➢ Understanding Multicast
➢ Client Run State ➢ Bonjour Troubleshooting
➢ Mobility Troubleshooting
➢ FlexConnect Clients
➢ Client Roaming
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Advanced Wireless
Troubleshooting – Section 1
DGTL-BRKEWN-3011
#CiscoLive
Where do we Start?
Where is my problem?
EAP
IP RADIUS ISE
CAPWAP
EOIP
IP
802.11 IP
CAPWAP WLC
DHCP
802.11 Management
Where is my problem? New!
ISE
CAPWP
In the APs?
EOIP
• Different client types impacted
802.11 • AP reload fixes the issue
Radio resets
• APWAP WLCreported
• SSID not heard DHCP
• One traffic QoS works and not
others
• Stops working after X days
ISE
CAPWP
EOIP
In the Controllers?
802.11
APWAP WLC
• Multiple different clients impacted across
different Aps DHCP
• Same client type across different Aps
• Authentication issues
• Ping works, but TCP/UDP not
ISE
CAPWP
EOIP
In the Clients?
802.11
Same client type across different Aps,WLC

APWAP
• others work
• Legacy client talking to recent Aps DHCP
• Recent OS update
• No/slow roaming
• New HW, old drivers
• Single device auth issues
Troubleshooting Basics New!
▪ Troubleshooting 101 Problem

Definition
• Clearly define the problem
• Understand any possible triggers Questions
• Know the expected behavior
• Reproducibility Tests
• Do not jump to conclusions
Analysis
Solution(s)
Reducing Scope New!
▪ From very broad, to

narrow point
▪ Helps on workaround,
prevention Wifi does
not work
▪ Speeds up case
resolution
Reducing Scope New!
Connected, no
No DNS traffic
Wifi does
not work
No connection
Can ping GW
Can ping local net

Reducing Scope New!
AAA/Dot1x
Nothing Pending
Client State
No Connection
WebAuth
Required
DHCP
Required
Association
• Wrong Password/PSK
Reducing Scope •
•
Radius Config
Shared Secret New!
• EAP method
• Wrong Client Config
• Unsupported IE
AAA/Dot1x
Nothing Pending
• Exhausted pool
Permanent • DHCP server reachability
No • DHCP required and static IP
Connection
WebAuth • Queue stuck
• Certificate issues DHCP • VLAN config
Required
• No DNS Required
• No GW
• URL
• Preauth ACL
Association
• Rates
• Unsupported IE
• Queue Stuck
• AP Memory leaks
Troubleshooting Basics
▪ Troubleshooting is an art with no right or wrong procedure, but best with a logical
methodology.
▪ Step 1: Define the problem
• Bad description: “Client slow to connect”
• Good description: “Windows 10 clients associations with card 8260 are rejected
with Status17 several times before they associate successfully.”
• Reduce Scope!
• Isolate multiple possible problems over same setup
▪ Step 2: Understand any possible triggers

• If something previously worked but no longer works, there should be an
identifiable trigger
• Understanding any and all configuration or environmental changes could help
pinpoint a trigger
• Finding a pattern may help with root cause isolation
▪ Step 3: Know the expected behavior
• If you know the order of expected behavior that is failing, defining where the
behavior breaks down (Problem Description) is better than defining the end result.
• Example: “One way audio between Phone A and B, because Phone A does not get
an ARP Response for Phone B”
▪ Step 4: Reproducibility
• Any problem that has a known procedure to reproduce (or frequently randomly
occurs) should be easier to diagnose
• Being able to easily validate or disprove a potential solution saves time by being
able to quickly move on to the next theory
• If the problem can be reproduced, it makes things much easier to work with
development, test the fix and deliver with lower impact to the end customer
• Tests will be conducted to isolate the root cause
▪ Step 5: Fix
• Validate Root Cause Analysis
• Local Reproduction
• Develop Fix
• Validate Fix
Advanced Wireless
DGTL-BRKEWN-3011
#CiscoLive
Client On-Boarding
What is Client On-Boarding?
The process for a Wireless device to find and connect to a Network,
including the following:
• Scanning the RF channels by sending Probe messages looking for the
target SSID
• Once the best AP servicing that SSID is found, send the initial
Authentication (typically this will be an Open Authentication message)
• Association request sent to the target AP
• Perform any optional 802.1x Authentication
• Encryption Key handshake
• IP addressing to be used (either static IP or DHCP)
• Perform any optional Layer 3 Authentication (Webauth)
What State are We in? Unlocking the key…
What is the Client state?
START
Auth Failure
AUTHCHECK L2AUTHCOMPLETE
8021X_REQD
DHCP_REQD
DHCP
Failure
Auth Failure
WEBAUTH_REQD
FastPath
Failure
DHCP_NOL3SEC
Auth Failure
RUN
Understanding the Client State
Name Description
8021X_REQD 802.1x (L2) Authentication Pending
DHCP_REQD IP Learning State
WEBAUTH_REQD Web (L3) Authentication Pending
RUN Client Traffic Forwarding
(Cisco Controller) >show client detail 00:16:ea:b2:04:36

Client MAC Address............................... 00:16:ea:b2:04:36
…..
Policy Manager State............................. WEBAUTH_REQD
The Split MAC Architecture (Local Mode AP)
1. Transmitting Beacons
2. Probe Requests
State 1:
Unauthenticated, AP
Unassociated 3. Probe Responses
4. Authentication Request
5. Authentication Response
State 2:
Authenticated,
6. Association Request
Unassociated
7. Association Response WLC
State 3: 8. (Optional: EAPOL Authentication)
Authenticated,
Associated 9. (Optional: Encrypt Data)
10. Forward User Data
Understanding Debug Commands
• There are many debug commands available for use, key will be to
know which to use for what issue.
• For most client troubleshooting use the “debug client <MAC>”
• Debug commands will remain active until either the session is
closed or “debug disable-all”
• Debug may be filtered via MAC address (up to 10)
• Be aware that running many debugs on busy systems may affect
WLC overall performance (Debug client has been tested on fully loaded
systems without any adverse affects)
Common Debugs (Local Mode APs)
Client Troubleshooting: Radius:
(wlc3504) >debug client 11:22:33:44:55:66 (wlc3504) >debug aaa ?
MAC Addr 1.................................. 11:22:33:44:55:66 all Configures debug of all AAA messages.
detail Configures debug of AAA detailed events.
SGT debugging .............................. Disabled events Configures debug of AAA events.
Flex-AP Client Debugging .............. Disabled ldap Configures debug of AAA LDAP events.
Flex-Group Client Debugging ......... Disabled local-auth Configures debug of AAA Local Authentication.
packet Configures debug of AAA packets.
Debug Flags Enabled: tacacs Configures debug of AAA TACACS+ events.
dhcp packet enabled.
Client Event enabled.
dot11 mobile enabled.
dot11 state enabled Web Authentication:
dot1x events enabled. (wlc3504) >debug web-auth redirect enable mac <MAC>
dot1x states enabled.
mobility client handoff enabled. Bonjour:
pem events enabled. (wlc3504) >debug mdns all enable
pem state enabled.
802.11r event debug enabled. AP Join:
802.11w event debug enabled. (wlc3504) >debug capwap errors enable
CCKM client debug enabled.
(wlc3504) >debug capwap events enable
Open Authentication, No Encryption
START
Auth Failure
AUTHCHECK L2AUTHCOMPLETE
8021X_REQD
DHCP_REQD
DHCP
Failure
Auth Failure
WEBAUTH_REQD
FastPath
Failure
DHCP_NOL3SEC
Auth Failure
RUN
Client Association
▪ Received management frame ASSOCIATION REQUEST on BSSID 70:70:8b:a3:2e:2f destination addr 70:70:8b:a3:2e:2f
▪ Processing assoc-req station:38:71:de:4e:43:8b AP:70:70:8b:a3:2e:20-01 ssid : CiscoLive
▪ Adding mobile on LWAPP AP 70:70:8b:a3:2e:20(1)
▪ Association received from mobile on BSSID 70:70:8b:a3:2e:2f AP CAP2802
▪ Station: 38:71:DE:4E:43:8B trying to join WLAN with RSSI -42. Checking for XOR roam conditions on AP:
70:70:8B:A3:2E:20 Slot: 1
▪ Station: 38:71:DE:4E:43:8B is associating to AP 70:70:8B:A3:2E:20 which is not XOR roam capable
▪ Allocate AID 1 slot 1 on AP CAP2802 #clients on this slot 1
▪ 0.0.0.0 START (0) Initializing policy
▪ 0.0.0.0 START (0) Change state to AUTHCHECK (2) last state START (0)
▪ AuthenticationRequired = 0 [NOTE: 0 means no further authentication such as 802.1x, WebAuth is needed]
▪ 0.0.0.0 AUTHCHECK (2) Change state to L2AUTHCOMPLETE (4) last state AUTHCHECK (2)
Client Association
▪ 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 70:70:8b:a3:2e:20 vapId 1 apVapId 1
▪ 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7) last state L2AUTHCOMPLETE (4)
▪ Changing state for mobile 38:71:de:4e:43:8b on AP 70:70:8b:a3:2e:20 from Idle to Associated
▪ session timeout for station 38:71:de:4e:43:8b - Session Tout 1800, apfMsTimeOut '1800' and
sessionTimerRunning flag is 0
▪ Sending Assoc Response (status: '0') to station on AP CAP2802 on BSSID 70:70:8b:a3:2e:2f ApVapId 1 Slot 1,
mobility role 0
Since 11k was enabled
▪ Mobility query, PEM State: DHCP_REQD on WLAN, Client is
▪ Mobile Announce sent to 1 members of the local group. requesting Neighbor
Report
▪ Successful transmission of LWAPP Add-Mobile to AP 70:70:8b:a3:2e:20
▪ Received management frame ACTION on BSSID 70:70:8b:a3:2e:2f destination addr 70:70:8b:a3:2e:2f
▪ Station: 38:71:DE:4E:43:8B sent 802.11K neighbor request to AP 70:70:8B:A3:2E:20
Client Association - Common Problems
There can be a number of reasons why a client association request fails, some examples:
• MAC Filtering – Access Reject received from Radius
Access-Reject received from RADIUS server 10.100.76.10 for mobile bc:9f:ef:1b:89:ef
Sending assoc-resp with status 1 station:bc:9f:ef:1b:89:ef AP:70:70:8b:a3:2e:20-01
Sending Assoc Response (status: 'unspecified failure') to station on AP CAP2802 on BSSID
70:70:8b:a3:2e:2f
• Blacklisted Client (either through repeated Authentication failures or manually excluded)

00:40:96:b5:db:d7 Ignoring assoc request due to mobile in exclusion list or marked for
deletion
• Data rate mismatch in the client association request

STA - rates (0): 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
suppRates statusCode is 18 and gotSuppRatesElement is 0
STA - rates (6): 152 36 48 72 96 108 0 0 0 0 0 0 0 0 0 0
extSuppRates statusCode is 18 and gotExtSuppRatesElement is 0
Sending Assoc Response to station on BSSID 0c:85:25:9e:62:b0 (status 18) ApVapId 5 Slot 0
Client Association - Common Problems
Be mindful of enabling Cisco specific features that may cause Client Association Failures
• Aironet Extensions
• Blocking WiFi Direct :

Association received from mobile on BSSID 00:e1:6d:29:09:a1 AP ap700
WFD Client rejected on BSSID 00:e1:6d:29:09:a0 slot 0
Sent Deauthenticate to mobile on BSSID 00:e1:6d:29:09:a0 slot 0
Wifi Direct P2P IE processing Failure
• Clients probing but not connecting:

“Debug dot11 probe event enable”
Client DHCP
Client State = “DHCP_REQD“
Client is in DHCP_REQD state DHCP Proxy Enabled DHCP Proxy Disabled
• Proxy Enabled:
Client DHCP Discover Client DHCP Discover Is
DHCP Relay/Proxy Unicast to DHCP Servers Bridged to DS
Between WLC and Server

DHCP Offer from Server
Required for Internal DHCP
• Proxy Disabled: Client DHCP Request
Between Client and Server

DHCP ACK from Server
DHCP is forwarded as a broadcast on VLAN
IP helper or other means required IP Address Learned
Client DHCP (IP Learn)
▪ DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 1, encap 0xec03, xid 0x268b23e)
▪ DHCP dropping packet due to ongoing mobility handshake exchange, (siaddr 0.0.0.0, mobility state =
'apfMsMmQueryRequested'
▪ 0.0.0.0 DHCP_REQD (7) mobility role update request from Unassociated to Local Peer = 0.0.0.0, Old Anchor =
0.0.0.0, New Anchor = 192.168.158.80
▪ 0.0.0.0 DHCP_REQD (7) State Update from Mobility-Incomplete to Mobility-Complete, mobility role=Local, client
state=APF_MS_STATE_ASSOCIATED
▪ 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0
NPU type 9 is IP Learn
Client DHCP
DHCP processing DHCP DISCOVER (1)
DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0
DHCP xid: 0x6e43ffb3 (1849950131), secs: 0, flags: 0
DHCP chaddr: 10:4a:7d:b1:a8:e1
Discover
DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
DHCP successfully bridged packet to DS
..
DHCP received op BOOTREPLY (2) (len 308,vlan 10, port 1, encap 0xec00, xid 0x6e43ffb3) Offer
DHCP processing DHCP OFFER (2)
..
DHCP received op BOOTREQUEST (1) (len 343,vlan 0, port 1, encap 0xec03, xid 0x6e43ffb3)
DHCP processing DHCP REQUEST (3)
.. Request
DHCP received op BOOTREPLY (2) (len 308,vlan 10, port 1, encap 0xec00, xid 0x6e43ffb3
DHCP processing DHCP ACK (5)
DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 1
DHCP xid: 0x6e43ffb3 (1849950131), secs: 0, flags: 0
DHCP chaddr: 10:4a:7d:b1:a8:e1
DHCP ciaddr: 0.0.0.0, yiaddr: 172.18.254.130 ACK
DHCP siaddr: 0.0.0.0, giaddr: 172.18.254.1
DHCP server id: 172.18.108.43 rcvd server id: 172.18.108.43
Learning IP without DHCP
Orphan Packet from 10.99.76.147 on mobile
Installing Orphan Pkt IP address 10.99.76.147 for station10.99.76.147 DHCP_REQD
(7) Change state to RUN (20) last state RUN (20)
▪ Multiple mechanisms to learn Client IP address:

➢ Mobility
➢ ARP/GARP from client
➢ Traffic from/to client
➢ DHCP
▪ Non-DHCP: Seen with mobile devices that attempt to send data before validating DHCP
▪ Up to client to realize their address is not valid for the subnet
▪ DHCP Required enabled on WLAN mitigates this client behavior
Client DHCP - Common Problems
• Internal DHCP -> DHCP proxy must be enabled!
• Internal DHCP -> point DHCP server to management
• Bridge mode: check for DHCP server or relay agent on vlan
• DHCP pool exhaustion
• Dirty Server -> Interface Groups.
• Static IP with DHCP required
• DHCP required with RFC compliant clients
Dirty Server
Pre-Shared Key Encryption
Client Encryption Key Exchange Packet Flow
AP WLC Radius
Probe Request
Probe Response
Auth Request
Auth Response
Association Request
Association Response
EAPoL 4 way Exchange
DATA
Client Encryption Key Exchange
• Change state to AUTHCHECK (2) last state START (0)
• AuthenticationRequired = 1
• 0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state AUTHCHECK (2)
• Starting key exchange to mobile bc:9f:ef:1b:89:ef, data packets will be dropped
• Sending EAPOL-Key Message to mobile bc:9f:ef:1b:89:ef state INITPMK (message 1), replay counter
00.00.00.00.00.00.00.00
• Allocating EAP Pkt for retransmission to mobile bc:9f:ef:1b:89:ef
• validating eapol pkt: key version = 2
• Received EAPOL-Key from mobile bc:9f:ef:1b:89:ef
• key Desc Version FT – 0
• Received EAPOL-key in PTK_START state (message 2) from mobile bc:9f:ef:1b:89:ef
Client Encryption Key Exchange
• Encryption Policy: 4, PTK Key Length: 48
• Successfully computed PTK from PMK!!!
• Received valid MIC in EAPOL Key Message M2!!!!!
• Compare RSN IE in association and EAPOL-M2 frame(Skip pmkIdLen:0,and grpMgmtCipherLen:0)
• Sending EAPOL-Key Message to mobile bc:9f:ef:1b:89:ef state PTKINITNEGOTIATING (message 3), replay
counter 00.00.00.00.00.00.00.01
• validating eapol pkt: key version = 2
• Received EAPOL-Key from mobile bc:9f:ef:1b:89:ef
• key Desc Version FT - 0
• Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile bc:9f:ef:1b:89:ef
• 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state 8021X_REQD (3)
• Mobility query, PEM State: L2AUTHCOMPLETE
Encryption Key Exchange - Common Problems
• Incorrect Pre-Shared Key configured on the client

Received EAPOL-key in PTK_START state (message 2) from mobile bc:9f:ef:1b:89:ef
Received EAPOL-key M2 with invalid MIC from mobile bc:9f:ef:1b:89:ef version 2
Retransmit failure for EAPOL-Key M1 to mobile bc:9f:ef:1b:89:ef, retransmit count 3, mscb deauth count 0
• Client taking a long time or not responding at all to respond to M1 key sent from
WLC
➢ “show advanced EAP” can be used to check your EAP timers, some clients need more time to
respond
Sending EAPOL-Key Message to mobile 00:44:19:6e:33:11
802.1x 'timeoutEvt' Timer expired for station 00:44:19:6e:33:11 and for message = M2
Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:44:19:6e:33:11
EAP Authentication (802.1x)
802.1x EAP Authentication Packet Flow
Probe Request AP WLC Radius
Probe Response
Auth Request
Auth Response
Association Request
EAP Start
EAP ID Request
EAP ID Response
EAP Method
Between 4 and
20+ frames EAP Success
DATA
802.1x EAP Authentication
• Sending EAP-Request/Identity to mobile 70:d4:f2:c9:bb:ae (EAP Id 1)
• Received Identity Response (count=1) from mobile 70:d4:f2:c9:bb:ae
• EAP State update from Connecting to Authenticating for mobile 70:d4:f2:c9:bb:ae
• Entering Backend Auth Response state for mobile 70:d4:f2:c9:bb:ae
• Processing Access-Challenge for mobile 70:d4:f2:c9:bb:ae
• Entering Backend Auth Req state (id=2) for mobile 70:d4:f2:c9:bb:ae
• Sending EAP Request from AAA to mobile 70:d4:f2:c9:bb:ae (EAP Id 2)
• Received EAP Response from mobile 70:d4:f2:c9:bb:ae (EAP Id 2, EAP Type 25)
• Processing Access-Challenge for mobile 70:d4:f2:c9:bb:ae
• Entering Backend Auth Req state (id=3) for mobile 70:d4:f2:c9:bb:ae
• Sending EAP Request from AAA to mobile 70:d4:f2:c9:bb:ae (EAP Id 3)
802.1x EAP Authentication
• Received EAP Response from mobile 70:d4:f2:c9:bb:ae (EAP Id 3, EAP Type 25)
• Processing Access-Accept for mobile 70:d4:f2:c9:bb:ae
• Username entry (CiscoLive) created for mobile, length = 253
• Username entry (CiscoLive) created in mscb for mobile, length = 253
• Sending EAP-Success to mobile 70:d4:f2:c9:bb:ae (EAP Id 3)
802.1x EAP Authentication - Common Problems
• Radius sending an Access Reject for the client

Processing Access-Reject for mobile 64:80:99:dc:2f:e9
Sending EAP-Failure to mobile 64:80:99:dc:2f:e9 (EAP Id 145)
• Radius Server is not responding

➢ Check “show radius auth statistics”, look for any timeout or drops
Timeout Requests................................... 9
Consecutive Drops ................................. 0
Unknowntype Msgs................................. 0
Other Drops............................................. 0
• Certificate Trust issues between the Client and Radius Server

• Radius Server and Client are unable to negotiate a common EAP type
➢ For example, your client is configured for EAP-Fast, but your Radius Server does not support
it.
Local Web Authentication
Local Web Authentication Packet Flow
Client Controller Radius
Association
1
2
DHCP
3
HTTP Request
4
Redirect URL to Captive Portal
HTTP Request
5
Local Web Authentication Packet Flow
Client Controller Radius
Sign up page (web user)
User Form Submit

6 Authenticate
Username/pass
7
Access Accept
Redirect to splash page

8
Network Access
9
• 172.18.254.134 DHCP_REQD (7) Change state to WEBAUTH_REQD (8) last state DHCP_REQD (7)
• 172.18.254.134 Added NPU entry of type 2, dtlFlags 0x0
• captive-bypass detection disabled, Not checking for wispr in HTTP GET, client mac=38:71:de:4e:43:8b
• Preparing redirect URL according to configured Web-Auth type
• unable to get the hostName for virtual IP, using virtual IP =1.1.1.1
• Checking custom-web config for WLAN ID:3
• Global status is enabled, checking on web-auth type
• Web-auth type Internal, no further redirection needed. Presenting default login page to user
• http_response_msg_body1 is <HTML><HEAD><TITLE> Web Authentication Redirect</TITLE><META http-

equiv="Cache-control" content="no-cache"><META http-equiv="Pragma" content="http_response_msg_body2
is "></HEAD></HTML>
• parser host is captive.apple.com
• parser path is /hotspot-detect.html
• added redirect=, URL is now https://1.1.1.1/login.html?
• str1 is now https://1.1.1.1/login.html?redirect=captive.apple.com/hotspot-detect.html
• SSL Connection created for MAC:38:71:de:4e:43:8b
• Message to be sent is HTTP/1.1 200 OK Location:

https://1.1.1.1/login.html?redirect=captive.apple.com/hotspot-detect.html
• ewaURLHook: Entering:url=/login.html, virtIp = 1.1.1.1, ssl_connection=1, secureweb=1
• WLC received client 38:71:de:4e:43:8b request for Web-Auth page /login.html
• Username entry (CiscoLive) created for mobile, length = 9
• Access-Accept received from RADIUS server 172.18.123.43 (qid:11) with port:1812, pktId:4
• 72.18.254.134 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state WEBAUTH_NOL3SEC (14)
Local Web Authentication - Common Problems
• Client unable to get an IP address

• Client unable to resolve DNS
• Client sending the HTTP packet with a destination port other than 80
➢ Typically seen when client is using a proxy
• HTTP Get too large to redirect due to cookie size
➢ WLC has a max buffer size (2k), this must include the HTTP Get request and any cookies also
sent. (try webauth with your browser in private browsing mode, so cookies don’t get sent)
• Certificate Trust issues,
Local Web Authentication - Common Problems
• WLAN Session timeout too low

• Clients going idle:
Received DELETE mobile, reasonCode MN_IDLE_TIMEOUT, deleteReason 4 from AP 70:70:8b:a3:2e:20,
slot 1
To help with this issue, use the Sleeping client feature on the WLAN
Check for sleep client support vapId =3, apfMmRole=1, sleepClientSupport=1
Create sleep client entry username=CiscoLive,password=********,sleepClientTimeout=15
Use the command “show custom-web sleep-client summary” to display the list of any clients in this state
sleepLogin: sending AAA auth for sleep client 38:71:de:4e:43:8b

sleepLogin: deleted sleep client entry 38:71:de:4e:43:8b
38:71:de:4e:43:8b Access-Accept received from RADIUS server 172.18.123.43
172.18.254.134 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state WEBAUTH_NOL3SEC (14)
Central Web Authentication using ISE
Central Web Authentication Packet Flow
Client Controller ISE Radius
Association
1
MAC Authentication
2
Access Accept with Redirect URL and ACL returned
3 DHCP
HTTP Request
4
Redirect URL from ISE
HTTP Request
5
HTTP Response with ISE guest Portal page
Central Web Authentication Packet Flow
Client Controller ISE Radius
6 User Form Submit
Change of Authorization causing

7 client Re-authentication
8 MAC Authentication
Access Accept with Client Redirect

URL Removed and ACL Updated
Network Access
Central Web Authentication
• Association received from mobile on BSSID 70:70:8b:a3:2e:2e AP CAP2802
• Applying Interface(vlan10) policy on Mobile, role Unassociated. Ms NAC State 0 Quarantine Vlan 0 Access Vlan 0
• WLAN CiscoLive-CWA has ISE-NAC security policy, using external RADIUS only for MacAuth-Request
(Client MAC Authentication)
• AAA Override Url-Redirect

'https://172.18.123.43:8443/portal/gateway?sessionId=c0a89e500000001f5ad1031d&portal=f079c670-7159-
11e7-a355-005056aba474&action=cwa&token=e47b1
• Redirect URL received for client from RADIUS. Client will be moved to WebAuth_Reqd state to facilitate
redirection. Skip web-auth Flag = 0
• AAA Override Url-Redirect-Acl 'CWA_redirect' mapped to ACL ID 0 and Flexconnect ACL ID 65535
• Sending Assoc Response (status: '0') to station on AP CAP2802 on BSSID 70:70:8b:a3:2e:2e
(Client Obtains an IP Address via DHCP)
• 172.18.254.81 DHCP_REQD (7) Change state to WEBAUTH_REQD (8) last state DHCP_REQD (7)
• Preparing redirect URL according to configured Web-Auth type
• Client configured with AAA overridden redirect URL
https://172.18.123.43:8443/portal/gateway?sessionId=c0a89e50000000025adf905e&portal=f079c670-7159-
11e7-a355-005056aba4
(Client Web Auth Authentication Success)
• Received a 'CoA-Request' from 172.18.123.43 port 24780
• handleCoARequest (radiusCoAsupport.c:1672) Changing state for mobile 38:71:de:4e:43:8b on AP

70:70:8b:a3:2e:20 from Associated to AAA Pending
• AAA redirect is NULL. Skipping Web-auth for Radius NAC enabled WLAN
• WLAN CiscoLive-CWA has ISE-NAC security policy, using external RADIUS only for MacAuth-Request
• CoA Request on MAC-Filter enabled security, initiate mac-auth for the client 38:71:de:4e:43:8b
• 38:71:de:4e:43:8b Send Radius Auth Request with pktId:8 into qid:11 of server at index:0
• Sent a 'CoA-Ack' to 172.18.123.43 (port:24780)
• 8:71:de:4e:43:8b Access-Accept received from RADIUS server 172.18.123.43 (qid:11) with port:1812, pktId:8
• 38:71:de:4e:43:8b Resetting web IPv4 acl from 0 to 255
• 38:71:de:4e:43:8b Username entry (ciscolive) created for mobile, length = 253
• 8:71:de:4e:43:8b Applying new AAA override for station 38:71:de:4e:43:8b
• 38:71:de:4e:43:8b Applying Interface(vlan10) policy on Mobile, role Local. Ms NAC State 2 Quarantine Vlan 0
Access Vlan 10
• 38:71:de:4e:43:8b Re-applying interface policy for client
• 172.18.254.158 WEBAUTH_REQD (8) Change state to START (0) last state WEBAUTH_REQD (8)
• 72.18.254.158 START (0) Change state to AUTHCHECK (2) last state START (0)
• 38:71:de:4e:43:8b AuthenticationRequired = 0
• AUTHCHECK (2) Change state to L2AUTHCOMPLETE (4) last state AUTHCHECK (2)
• L2AUTHCOMPLETE (4) Change state to STATICIP_NOL3SEC (12) last state L2AUTHCOMPLETE (4)
• STATICIP_NOL3SEC (12) Change state to RUN (20) last state STATICIP_NOL3SEC (12)
• 38:71:de:4e:43:8b 172.18.254.158 Added NPU entry of type 1, dtlFlags 0x0
Central Web Authentication - Common Problems
• Verify the “Allow AAA override” is checked and “NAC State” set to “ISE NAC”
• AAA override passing down an ACL name that does not exist on the WLC (case matters)
AAA Override Url-Redirect-Acl 'CWA_redirect' mapped to ACL ID 0 and Flexconnect ACL ID 65535
• Verify the packet flow for the ACL used. If the ACL is wrong, you will begin to loop:
Preparing redirect URL according to configured Web-Auth type
Client configured with AAA overridden redirect URL
https://172.18.123.43:8443/portal/gateway?sessionId=c0a89e50000000215ad105a7&portal=f079c670-7159-
11e7-a355-005056aba4
[Looping…]
Preparing redirect URL according to configured Web-Auth type

Client configured with AAA overridden redirect URL
https://172.18.123.43:8443/portal/gateway?sessionId=c0a89e50000000215ad105a7&portal=f079c670-7159-
11e7-a355-005056aba4
Central Web Authentication - Common Problems
• Verify that all authentication steps occur on ISE:

1. The MAC authentication should occur first, to which CWA attributes are returned.
2. The portal login authentication occurs.
3. The dynamic authorization occurs.
4. The final authentication is a MAC authentication that shows the portal username on the ISE, to which the final
authorization results are returned (such as the final VLAN and ACL).
• Verify that you have “Support for CoA” set to “Enable” on your Radius Authentication Server
entry on the WLC
• Central Web Authentication on the WLC and ISE Configuration Example:
https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/115732-central-web-auth-
00.html
Advanced Wireless
DGTL-BRKEWN-3011
#CiscoLive
Client Run State
RUN State
• RUN means: client has completed all required policy states
• “NPU entry of Type 1” is the goal
172.18.254.81 DHCP_REQD (7) Change state to RUN (20) last state DHCP_REQD (7)
172.18.254.81 RUN (20) Reached PLUMBFASTPATH: from line 7656, null
RUN (20) Replacing Fast Path rule
type = Airespace AP Client
on AP 70:70:8b:a3:2e:20, slot 1, interface = 1, QOS = 0
IPv4 ACL ID = 255, IPv6 ACL ID
172.18.254.81 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID = 64206, IntfId = 10 Local
Bridging Vlan = 10, Local Bridging intf id = 10
172.18.254.81 RUN (20) Successfully plumbed mobile rule (IPv4 ACL ID 255, IPv6 ACL ID 255, L2 ACL ID
255,URL ACL ID 255,URL ACL Action 0)
172.18.254.81 RUN (20) No 11v BTM
172.18.254.81 RUN (20) NO release MSCB
172.18.254.81 Added NPU entry of type 1, dtlFlags 0x0
RUN State - AP Radio Events
• Random Disconnections – Radio Reset
00:1a:70:35:84:d6 Cleaning up state for STA 00:1a:70:35:84:d6 due to event for AP
04:da:d2:4f:f0:50(0)
00:1a:70:35:84:d6 Scheduling deletion of Mobile Station: (callerId: 45) in 10 seconds
• Watch out for Abnormal Radio Rests

AP2802# show history interface dot11Radio all reset
Timestamp Slot Client count Reset reason
Fri Apr 20 21:26:18 2018 0 0 2 (Radio firmware crashed)
• If you are using DFS Channels, be aware of Radar Events:

AP2802# show history interface dot11Radio all radar detected
Timestamp Slot Client count Channel
Fri Apr 20 21:32:47 2018 1 0 52
Trap: Radar signals have been detected on channel 56 by 802.11a radio with MAC:70:70:8b:a3:2e:20
and slot 1
RUN State - High RF Channel Utilization
AP2802# show controllers dot11Radio 0 | b QBSS

QBSS Load: cca_load: 0x6b(42%), rx_load: 0x0(0%), tx_load: 0x0(0%)
%DOT11-3-NO_BEACONING: Error on Dot11Radio0 - Not Beaconing for too long - Current 2887074 Last
2887074
%LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
%LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
%LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
RUN State - High RF Channel Utilization
Run State – High RF Duty Cycle
RUN State - RF Analysis WLCCA
WLCCA
• Tool for quick RF analysis
• RF Health - > simplified quick
view on RF, per Band, AP, AP
Group, Flex Group
Download:
https://developer.cisco.com/docs
/wireless-troubleshooting-
tools/#!wireless-config-analyzer
RUN State - RF Analysis WLCCA
Deauthenticated Client
Idle Timeout (300 sec)
38:71:de:4e:43:8b Received DELETE mobile, reasonCode MN_IDLE_TIMEOUT, deleteReason 4 from AP
38:71:de:4e:43:8b apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4,
reasonCode 4
38:71:de:4e:43:8b Sent Disassociate to mobile on AP 70:70:8b:a3:2e:20-1 on BSSID
70:70:8b:a3:2e:2d (reason 4)
Session Timeout (1800 secs)

38:71:de:4e:43:8b Expiring PMK cache of 38:71:de:4e:43:8b
38:71:de:4e:43:8b Initiating 802.1x due to PMK Timeout Event for STA
38:71:de:4e:43:8b Sent 1x reauth initiate message to multi thread task for mobile
38:71:de:4e:43:8b
WLAN Change
Modifying a WLAN in anyway Disables and Re-enables WLAN
38:71:de:4e:43:8b Successfully freed AID 1, slot 1 on AP 70:70:8b:a3:2e:20, #client on this
slot 0
Changing state for mobile 38:71:de:4e:43:8b on AP 70:70:8b:a3:2e:20 from Associated to
Disassociated
Sent Disassociate to mobile on AP 70:70:8b:a3:2e:20-1 on BSSID 70:70:8b:a3:2e:2f(reason 1)
AP Radio Reset (Power/Channel)

38:71:de:4e:43:8b Cleaning up state for STA 38:71:de:4e:43:8b due to event for AP
70:70:8b:a3:2e:20(1)
Manual Deauthentication
Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds
Changing state for mobile 38:71:de:4e:43:8b on AP 70:70:8b:a3:2e:20 from Associated to
Disassociated
Broadcast Key Rotation Timeout

Updated broadcast key sent to mobile 10:4A:7D:B1:A8:E1
802.1x 'timeoutEvt' Timer expired for station 10:4a:7d:b1:a8:e1 and for message = M5
Retransmit 1 of EAPOL-Key M5 (length 131) for mobile 10:4a:7d:b1:a8:e1

Retransmit 2 of EAPOL-Key M5 (length 131) for mobile 10:4a:7d:b1:a8:e1

Retransmit failure for EAPOL-Key M5 to mobile 10:4a:7d:b1:a8:e1, retransmit count 3, mscb

deauth count 0
Sent Deauthenticate to mobile on BSSID 70:70:8b:a3:2e:2f slot 1
Cloud tools - WLC Debug Analyzer
Tool which Automates debug analysis from debug client
https://cway.cisco.com/tools/WirelessDebugAnalyzer/
Cloud tools – WLC Debug Analyzer
Cloud tools - WCAE
https://cway.cisco.com/tools/WirelessAnalyzer/
https://developer.cisco.com/docs/wireless-troubleshooting-tools
Cloud tools - WCAE
Run State – Key Ideas
▪ Client can be removed for numerous reasons
✓ WLAN change, AP channel change, configured interval (session timeout)
▪ Start with Client Debug to see if there is a reason for a client’s

deauthentication
▪ Further Troubleshooting
✓ Client debug should give some indication of what kind of deauth is
happening
✓ Packet capture or client logs may be required to see the exact reason
✓ Never forget Radio status and RF conditions
Flexconnect Clients
The Split MAC Architecture
(FlexMode AP, Central Auth, Central Switching)
State 1: 2. Probe Requests
Unauthenticated,
AP
State 2:
Authenticated,
Unassociated
7. Association Response
Authenticated,
Associated 9. (Optional: Encrypt Data) WLC
The Split MAC Architecture
(FlexMode AP, Local Auth, Local Switching)
State 1: 2. Probe Requests
Unauthenticated,
State 2:
AP
Authenticated,
Unassociated
7. Association Response
Authenticated,
Associated 9. (Optional: Encrypt Data)
WLC

COS (Cisco OS) based AP’s
include all 802.11ac Wave2
and above:
• Example COS based AP:
Flexconnect ✓ 1800 series
✓ 2800 series
Debugging for COS ✓ 3800 series
AP’s ✓ 9100 series
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Show commands on COS Flex APs
Ap2802# show flexconnect ?
calea Calea Information

cckm CCKM Cache Entry Information
client Client Information
dhcp DHCP Information
dot11r 802.11r Cache Entry Information
mcast Multicast Information
mobexp Mobility Express Configuration
oeap Flexconnect OEAP Information
pmk OKC/PMK Cache Entry Information
status Standalone status
url-acl URL for DNS ACL Status of all flex clients
vlan-acl VLAN ACL mapping
currently active on this AP
vlan-name Vlan name ID mapping
wlan WLAN Configuration
Ap2802# show flexconnect client

Flexconnect Clients:
mac radio vap aid state encr aaa-vlan aaa-acl aaa-ipv6-acl assoc auth switching …..
44:61:32:E7:5E:DE 0 1 9 FWD AES_CCM128 none none none Local Central Local ……
00:04:20:FA:DA:00 0 1 5 FWD AES_CCM128 none none none Local Central Local …..
Ap2802# show flexconnect wlan
Status of my
Flexconnect WLANs:
Flex wlans,
Radio Vap SSID State Auth Assoc Switching showing client
switching mode
0 0 DOWN Central Local Central
0 1 CiscoLiveB UP Central Local Local
0 2 GuestNet UP Central Local Local
Ap2802# show flexconnect client aaa-override Status of any AAA

override attributes being
applied to flex clients
Ap2802# show dot11 clients onboarding dot11 0 wlan 1
Client on-boarding
status per
radio/VAP
Ap2802# show flexconnect mcast
IPv4 Multicast Groups:
BG-ID MC-Address Sub-If Count Rpt-Age Status of Flex

Multicast groups for
1 224.0.0.251 apr0v1 1 0 AP & multicast
clients
2 224.0.0.251 apr0v2 1 60
IPv4 Multicast Clients:
BG-ID Address Port Dot11 VAP Vlan-ID
1 00:04:20:FA:DA:00 3 true 0 1
2 64:52:99:B0:46:5A 4 true 0 5
Ap2802# show dot11 clients
Total dot11 clients: 2
Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate WGB
1C:F2:9A:51:75:6E 1 1 2 CiscoLiveA -57 MCS91SS No
44:61:32:E7:5E:DE 0 2 9 CiscoLiveB -54 M7 No

MCS9 with 1
Spatial Stream
Ap2802# show datapath client ip-table
id vap port node tunnel mac seen_ip
00:04:20:FA:DA:00 1 apr0v1 6.185.128.24 - 00:04:20:FA:DA:00 192.168.1.240
Ap2802# show dot11 clients deauthenticated
timestamp mac vap reason_code
Thu Aug 20 12:57:27 2020 96:D3:A5:83:85:C7 0 2
Give me a list of the last

Ap2802# show flexconnect client exclusion-list 100 client deauths
Client Exclusion List:
Excluded Mac life Type AP Client

Exclusion List
96:D3:A5:83:85:C7 0 static
“wlan” here is
really the VAP
Ap2802# show dot11 clients onboarding dot11radio 1 wlan 0
Ap2802# show flexconnect wlan vlan
Native Vlan: 1
Flexconnect WLAN-VLAN mapping: Shows the VAP to

vlan mappings
vap_id vlan_id ap_specific
0 1 false
1 1 false
2 5 false
3 1 false
COS AP – Debug Client
Ap2802# debug client 96:d3:a5:83:85:c7
Ap2802# show debug

client:
Client 96:d3:a5:83:85:c7 debugging enabled for critical, errors, events, info, arp, dhcp, eapol, access-lists
Client Trace Status : Started

Client Trace ALL Clients : disable
Client Trace Address : 96:d3:a5:83:85:c7
Remote/Dump Client Trace Address : none Debugs can vary based on AP
Client Trace Filter : auth
Client Trace Filter : assoc
chipset vendor, Typically “debug
Client Trace Filter : eap client” works best, but for 1800
Client Trace Filter : dhcp series, use “debug dot11 client”
Client Trace Filter : dhcpv6
Client Trace Filter : icmp
Client Trace Filter : icmpv6
Client Trace Filter : ndp
Client Trace Filter : arp
Client Trace Output : eventbuf
Client Trace Output : console-log
COS AP – Debug Client Walk Through
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [U:W] DOT11_AUTHENTICATION : (.)
CLSM[96:D3:A5:83:85:C7]: US Auth(b0) seq 2611 IF 29 slot 1 vap 0 len 64 state NULL
CLSM[96:D3:A5:83:85:C7]: DS Auth len 64 slot 1 vap 0
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] DOT11_AUTHENTICATION : (.)
CLSM[96:D3:A5:83:85:C7]: Driver send mgmt frame success Radio 1 Vap 0
CLSM[96:D3:A5:83:85:C7]: client moved from UNASSOC to AUTH
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [U:W] DOT11_ASSOC_REQUEST : (.)
CLSM[96:D3:A5:83:85:C7]: US Assoc Req(0) seq 2612 IF 29 slot 1 vap 0 len 237 state AUTH
CLSM[96:D3:A5:83:85:C7]: DS Assoc Resp(10) IF 16 slot 1 vap 0 state AUTH, generated by AP
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] DOT11_ASSOC_RESPONSE : (.)
CLSM[96:D3:A5:83:85:C7]: Driver send mgmt frame success Radio 1 Vap 0
CLSM[96:D3:A5:83:85:C7]: client moved from AUTH to ASSOC
CLSM[96:D3:A5:83:85:C7]: DS Assoc Resp(10) IF 16 slot 1 vap 0 state ASSOC, generated by WLC
CLSM[96:D3:A5:83:85:C7]: ADD_MOBILE AID 8
CLSM[96:D3:A5:83:85:C7]: Driver Add Client success AID 8 Radio 1 Vap 0 Enc 1
CLSM[96:D3:A5:83:85:C7]: Added to ClientIPTable on apr1v0
CLSM[96:D3:A5:83:85:C7]: client moved from ASSOC to 8021X
CLSM[96:D3:A5:83:85:C7]: Added to WCP client table AID 8 Radio 1 Vap 0 Enc 1
CLSM[96:D3:A5:83:85:C7]: Decoding TLV_CLIENTCAPABILITYPAYLOAD: capbaility: 6 Apple Client: Yes
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] EAPOL_KEY.M1 : DescType 0x02 KeyInfo 0x008b
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] EAPOL_KEY.M2 : DescType 0x02 KeyInfo 0x010b
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] EAPOL_KEY.M3 : DescType 0x02 KeyInfo 0x13cb
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] EAPOL_KEY.M4 : DescType 0x02 KeyInfo 0x030b
CLSM[96:D3:A5:83:85:C7]: ADD_MOBILE AID 8
CLSM[96:D3:A5:83:85:C7]: Client ADD Encrypt Key success AID 8 Radio 1 Enc 4 Key Len 16 Key idx 0 Key 36 7d
CLSM[96:D3:A5:83:85:C7]: client moved from 8021X to IPLEARN_PENDING
CLSM[96:D3:A5:83:85:C7]: ADD_CENTRAL_AUTH_INFO_MOBILE Payload
CLSM[96:D3:A5:83:85:C7]: TLV_FLEX_CENTRAL_AUTH_STA_PAYLOAD
CLSM[96:D3:A5:83:85:C7]: Decoding TLV_CLIENTCAP
ABILITYPAYLOAD: capbaility: 6 Apple Client: Yes
Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] ARP_QUERY : Sender 192.168.1.196 TargIp 192.168.1.1
[Ap2802-Office] [96:d3:a5:83:85:c7] <wired0> [U:E] ARP_QUERY : Sender 192.168.1.196 TargIp 192.168.1.1 **[U:E=APtoSw]
[Ap2802-Office] [96:d3:a5:83:85:c7] <wired0> [D:E] ARP_REPLY : Sender 192.168.1.1 HwAddr e0:cb:bc:29:e2:50 **Sw to AP
** We are learning IP via the ARP request the client sent us
CLSM[96:D3:A5:83:85:C7]: client moved from IPLEARN_PENDING to FWD
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] DHCP_REQUEST : TransId 0xb78335bc
[Ap2802-Office] [96:d3:a5:83:85:c7] <wired0> [U:E] DHCP_REQUEST : TransId 0xb78335bc
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] ARP_REPLY : Sender 192.168.1.1 HwAddr e0:cb:bc:29:e2:50
[Ap2802-Office] [96:d3:a5:83:85:c7] <wired0> [D:E] DHCP_ACK : TransId 0xb78335bc
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] ICMPV6_RS : Src ff02:0:0:0:0:0:0:2 Dst ff02:0:0:0:0:0:0:2
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] DOT11_ACTION : (.)
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] DHCP_ACK : TransId 0xb78335bc
COS AP – Debug Client – Client Disassociation
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [U:W] DOT11_DISASSOC : (.)
CLSM[96:D3:A5:83:85:C7]: US Disassoc(a0) seq 3313 IF 29 slot 1 vap 0 len 47 state FWD
CLSM[96:D3:A5:83:85:C7]: US deauth/disassoc from client to WLC

Client requests
CLSM[96:D3:A5:83:85:C7]: DELETE_MOBILE payload reason 262 to Deauth from
AP
CLSM[96:D3:A5:83:85:C7]: Received DELETE_MOBILE payload - Marked for deletion
CLSM[96:D3:A5:83:85:C7]: Client delete initiated with timeout of 10 seconds
CLSM[96:D3:A5:83:85:C7]: Remove success from ClientIPTable on apr1v0
CLSM[96:D3:A5:83:85:C7]: client moved from FWD to DELETE_PENDING
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [D:W] DOT11_DEAUTHENTICATION : (.)
Client Roaming
Client Roaming - Definition
• Set of steps taken by client to move from one AP to another
• It will include different actions:

• Scan of channels to find possible new parent (active/passive probing)
• Selection of adequate new parent
• Auth request
• Association/Reassociation
• Optional: 802.1x Authentication
• Optional: DHCP
Client Roaming
• Main rule: Roaming is a client-side decision
• There is a price to pay: some packet loss will happen
• Infrastructure can influence it

✓ Power levels
✓ RF design
✓ Data rates
✓ RX-SOP
✓ Optimized Roaming
Roaming Triggers
• Possible triggers
✓ RSSI
✓ Beacon loss
✓ Packet errors
✓ 802.11v BSS transition request
• Client implementation can be very different
• There are timers associated
Roaming Patterns
Roaming – Avoiding micro cells
Client Roaming - Security
• Any new connection must comply with the security policies
• Authentication on each roam can be slow (802.1x)
• Key caching comes to the rescue:

✓ CCKM
✓ PMKID
✓ 802.11r (FT)
802.1x EAP Authentication Packet Flow
Probe Response
Auth Request
Auth Response
Association Request
EAP Start
EAP ID Request
EAP ID Response
EAP Method
Between 4 and
20+ frames EAP Success
DATA
CCKM Roaming Packet Flow
Probe Response
Auth Request
Auth Response
Re Association Request
Re Association Response
DATA
Between 4 and
20+ frames
FT Roaming Packet Flow – Over Air
Old AP
Probe Request New AP WLC
Probe Response
Auth Request
Auth Response
DATA
FT Roaming Packet Flow – Over DS
Old AP
New AP WLC
Probe Request
Probe Response
FT Request
FT Response
Auth Request
Auth Response
DATA
FT Roam – Over air
• *apfOpenDtlSocket: Received management frame AUTH on BSSID a8:0c:0d:db:d4:1d
destination addr a8:0c:0d:db:d4:1d
• *apfMsConnTask_6: Doing preauth for this client over the Air
• *apfMsConnTask_6: Doing local roaming for destination address a8:0c:0d:db:d4:1d
• *apfMsConnTask_6: Got 1 AKMs in RSNIE
• *apfMsConnTask_6: RSNIE AKM matches with PMK cache entry :0x3
• *apfMsConnTask_6: pmkRoName derived successfully
• *apfMsConnTask_6: Validate FTIE for R0KH-ID, Store SNonce passed
• *apfMsConnTask_6: FT Auth over the ds. Generate the Anonce
FT Roam – Over air
• *Dot1x_NW_MsgTask_4: 11r roamed client, ft force auth with pskMode : 0
• *Dot1x_NW_MsgTask_4: Finishing FT roaming for mobile 00:80:48:65:09:34
• *Dot1x_NW_MsgTask_4: EAP-PARAM Debug - eap-params for Wlan-Id :4 is disabled -

applying Global eap timers and retries
• *Dot1x_NW_MsgTask_4: Disable re-auth, use PMK lifetime.
• *Dot1x_NW_MsgTask_4: dot1x - moving mobile 00:80:48:65:09:34 into Force Auth state
• *Dot1x_NW_MsgTask_4 Skipping EAP-Success to mobile 00:80:48:65:09:34

(encryptBit:0)
• **Dot1x_NW_MsgTask_4: Apr 30 15:44:58.264: 00:80:48:65:09:34 192.168.50.77
8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state 8021X_REQD (3)
Advanced Wireless
DGTL-BRKEWN-3011
#CiscoLive
Device
Troubleshooting
WLC Serviceability
WLC maintains a rolling log of events call the msglog:

(wlc3504) > show msglog
Message Log Severity Level ...................... INFORMATIONAL
*Dot1x_NW_MsgTask_3: Apr 21 20:12:13.681: %APF-6-RADIUS_OVERRIDE_DISABLED: Radius overrides

disabled, ignoring source 4
*Dot1x_NW_MsgTask_3: Apr 21 20:12:13.681: %APF-6-USER_NAME_CREATED: Username entry (CiscoLive)
with length (253) created for mobile 38:71:de:4e:43:8b
*Dot1x_NW_MsgTask_3: Apr 21 20:12:13.680: %APF-6-RADIUS_OVERRIDE_DISABLED: Radius overrides
disabled, ignoring source 2
*apfReceiveTask: Apr 21 20:11:52.191: %APF-6-USER_NAME_DELETED: Username entry (CiscoLive) is
deleted for mobile 38:71:de:4e:43:8b
*Dot1x_NW_MsgTask_3: Apr 21 20:11:22.437: %DOT1X-4-MAX_EAP_RETRIES: Max EAP identity request
retries (3) exceeded for client 38:71:de:4e:43:8b
WLC Serviceability
(wlc3504) > show traplog
Number of Traps Since Last Reset ……............ 95
Number of Traps Since Log Last Displayed .... 1
Log System Time Trap

--- ------------------------ -------------------------------------------------
0 Sat Apr 21 20:31:44 2018 Rogue AP: 28:34:a2:e8:53:a1 detected on Base Radio MAC:
70:70:8b:a3:2e:20 Interface no: 0(802.11a/b/g/n) Channel: 1 RSSI: -73 SNR: 2 Classification:
unclassified, State: Alert, RuleClassified : N,Severity Score: 0, RuleName: N.A.,Classified APMAC:
00:00:00:00:00:00 ,Classified RSSI: 0
1 Sat Apr 21 20:30:31 2018 RF Manager updated TxPower for AP CAP2802 Base Radio MAC:
70:70:8b:a3:2e:20 and Radio Type: 802.11a New Tx Power is: 8 ,
Reason: DTPC
WLC Serviceability
(wlc) > show logging last-reset
!!!Message and Trap Logs from Previous Reset
------------------------- Last Reboot MsgLog & Traplog ------------------------------------------

Sys Name: wlc3504-timsmith
Model: AIR-CT3504-K9
Version: 8.5.124.34
Primary Boot Image: 8.5.124.43 (default)
Backup Boot Image: 8.5.124.34 (active)
LastReset Reason: Planned Reset
Timestamp: Sun Apr 15 18:01:43 2018
SystemUpTime: 13 days 0 hrs 5 mins 3 secs
-------------------------- MsgLog Dump --------------------------------------------------------

*spamApTask6: Apr 15 14:29:00.584: %CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c:7080
08:96:ad:fd:6c:60: DTLS connection closed forAP 192:168:158:102 (5272), Controller:
192:168:158:80 (5246) Echo Timer Expiry
-------------------------- TrapLog Dump --------------------------------------------------------
Tue Apr 24 15:34:37 2018 Rogue AP : 78:ba:f9:8c:72:ef removed from Base Radio MAC :
70:70:8b:a3:2e:20 Interface no:1(802.11ac)
WLC Serviceability
WLC inter-process communication is handled via Queues
(wlc3504) > show queue-info
Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details

--------------------------------------------------------------------------------------------
1 PRINTF-Q 256 80 0 0 0 N
2 GRE Queue 100 8 0 2 0 N
3 dtlqueue 4096 2 0 6 0 N
4 dtlarpqueue 4096 24 0 30 0 Y
5 NIM-Q 96 12 0 3 0 N
6 SIM-Q 96 1392 0 4 0 N
7 DHCP Client Queue 8 16 0 1 0 N
AP Serviceability
IOS APs have a “flight recorder”
CAP2702# dir
Directory of flash:/
2 -rwx 337 Jan 1 1970 00:05:17 +00:00 info
3 -rwx 64 Apr 18 2018 15:40:45 +00:00 sensord_CSPRNG0
25 drwx 2368 Apr 15 2018 18:07:36 +00:00 ap3g2-k9w8-mx.v153_3_jf.20
4 -rwx 368 Apr 18 2018 15:43:05 +00:00 capwap-saved-config
9 -rwx 965 Dec 20 2017 22:00:13 +00:00 lwapp_mm_mwar_hash.cfg
19 -rwx 326 Apr 18 2018 15:43:05 +00:00 env_vars
69 -rwx 75228 Apr 18 2018 15:40:49 +00:00 event.log
70 drwx 704 Mar 1 1993 00:00:42 +00:00 configs
7 -rwx 64 Apr 18 2018 15:40:45 +00:00 sensord_CSPRNG1
6 -rwx 60266 Apr 2 2018 17:56:17 +00:00 event.capwap
11 -rwx 280 Apr 15 2018 18:10:14 +00:00 lwapp_officeextend.cfg
14 -rwx 368 Apr 21 2018 18:08:15 +00:00 capwap-saved-config-bak
23 -rwx 95008 Apr 18 2018 15:40:34 +00:00 lwapp_reap.cfg
18 -rwx 0 Aug 4 2016 16:40:07 +00:00 config.txt
AP Serviceability
COS APs have various Flash Directories to view syslog, crash and core dumps
CAP2802# show flash syslogs
Directory of /storage/syslogs/
total 332
-rw-r--r-- 1 root root 855 Apr 12 16:50 12
-rw-r--r-- 1 root root 20404 Mar 27 23:21 12.0
-rw-r--r-- 1 root root 4809 Apr 12 16:50 12.last_write
CAP2802# show flash cores

Directory of /storage/cores/
total 1772
-rw-r--r-- 1 root root 1810521 Apr 20 21:26 CAP2802_core-radio0FW-8.5.124.43.2018-04-20-21-
26-18.tgz
CAP2802# show flash crash

No AP crashfile found
AP Serviceability
Any of the Syslog Files on COS AP’s may be reviewed using the “more” command,
these syslog files are a rolling log so older information may also be viewed
CAP2802# more syslogs 16
Apr 21 18:28:32 kernel: [*04/21/2018 18:28:32.9447] Stopped Radio 1
Apr 21 18:28:32 kernel: [*04/21/2018 18:28:32.9619] DOT11_DRV[1]: set_channel Channel set to 56
Apr 21 18:28:34 kernel: [*04/21/2018 18:28:34.2099] 1:change to DFS channel 56, CAC for 60 seconds.
Apr 21 18:28:34 kernel: [*04/21/2018 18:28:34.2897] Started Radio 1
Apr 21 18:28:48 NCI: I1: openSensor(slot=1)
Apr 21 18:28:50 NCI: I1: SensorApp=1.15.4
Apr 21 18:28:50 NCI: I1: SensorHdw=1.2.3.0
Apr 21 18:28:50 NCI: I1: Hardware Radio Band = [4890, 5935] MHz, BW=150625
Apr 21 18:28:50 NCI: slot=1 mode=0 chanCnt=1 cw=1
Apr 21 18:28:50 NCI: chans: 56 0 0 0 0 0 0 0 0 0 0
Apr 21 18:28:50 NCI: I1: channel map channels: in=1 cloned=1
AP Serviceability
Methods of Accessing the AP
• Console
• Telnet / SSH
• No GUI support
• AP Remote Commands
Enabling Telnet/SSH
• WLC CLI: config ap [telnet/ssh] enable <ap name>
• WLC GUI: Wireless > All APs > Select AP > Advanced > Select [telnet/ssh] > Apply
• (No telnet on AP-COS)
AP Serviceability
▪ Show controller Do[0/1] (or Show Tech)
Must have! Before/During/After event
AP# show capwap client rcb
▪ Show log AdminState : ADMIN_ENABLED
OperationState : UP
Name : CAP2802
▪ WLC: show ap eventlog <ap name> SwVer : 8.5.124.43
HwVer : 1.0.0.0
▪ Show capwap client <?> MwarApMgrIp : 192.168.158.80
MwarName : wlc3504
▪ CLI Tips MwarHwVer : 0.0.0.0
Location : default location
Debug capwap console cli (IOS AP only) ApMode : Local
ApSubMode : Not Configured
Debug capwap client no-reload CAPWAP Path MTU : 1485
CAPWAP UDP-Lite : Enabled
IP Prefer-mode : IPv4
AP Link DTLS Encryption : OFF
AP Wired Capture
(New for 8.10 MR1) New!
• Sniffer trace at AP port

• Useful for remote captures, DTLS debugging, performance issues
SS-I-1#deb traffic wired ip capture
% Writing packets to "/tmp/pcap/SS-I-1_capture.pcap0"
SS-I-1#reading from file /dev/click_wired_log, link-type EN10MB (Ethernet)
SS-I-1#no deb traffic wired ip capture
SS-I-1#Killed
SS-I-1#copy pcap SS-I-1_capture.pcap0 tftp: 192.168.0.45
######################################################################## 100.0%
AP Wired Capture New!
(New for 8.10 MR1)
AP Wireless Dump
(New for 8.10MR1) New!
• “Wireless” dump at driver level, compatible with wireshark

• Up to 5 clients
• Filtering is possible, using config client-trace commands
SS-I-1#deb client dump 00:1E:E5:E2:35:CF
[*01/14/2020 14:58:07.5350] Time:535994us Dir:Rx Rate:a15.1-0 Rssi:-46 Ch:40 Fc:188 Dur:2c

70:69:5a:78:77:8f 00:1e:e5:e2:35:cf f8:72:ea:b7:3a:40 Seq:7e6(2022) Info:ICMP Retry:0 Len:161
Typesub:28 Tid:q0
[*01/14/2020 14:58:07.5350] 0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00
[*01/14/2020 14:58:07.5350] 0010 00 00 00 00 00 00 00 11 00 00 00 00 00 00 00 00
[*01/14/2020 14:58:07.5350] 00d0 ff
AP Wireless Dump
• Export the data directly to

Wireshark
• Data is sent to PC IP
address running sniffer
SS-I-1#config ap client-trace output
remote enable 192.168.0.17 5555
SS-I-1#config ap client-trace start
Only VIP packets will be seen in remote

if VIP is enabled
SS-I-1#config ap client-trace stop
Warning: SS-I-1#undeb all
All possible debugging has been turned

off
AP Wireless Remote Dump
AP Join
Troubleshooting
AP Join Process
▪ WLC Discovery
▪ DTLS/Join
▪ Image Download
▪ Configuration Check
▪ REG
L3 WLC Discovery
AP tries to send discover messages to all the WLC addresses that its hunting process
has turned up
Discover
AP Discover/Join
▪ AP Discovery Request sent to known and learned WLCs

▪ Broadcast
• Reaches WLCs with MGMT Interface in local subnet of AP
• Use “ip helper-address <ip>” with “ip forward-protocol udp 5246”
▪ Dynamic ▪ Configured (nvram)
• DNS: cisco-capwap-controller • High Availability WLCs – Pri/Sec/Ter/Backup
• DHCP: Option 43 • Last WLC
• All WLCs in same mobility group as last WLC
• Manual from AP - “capwap ap controller ip
address <ip>”
AP Join Debugging
Use MAC Filtering when running CAPWAP debugs on WLC

• debug mac addr <MAC ADDR>
• debug capwap events enable
• Debug capwap errors enable
• Use radio mac for mac-addr filters
Since AP’s are always sending packets to the WLC, be careful to make sure you have
the debug mac filter enabled before turning on the other debugs
AP Discover/Join – AP Side
%CAPWAP-3-EVENTLOG: Starting Discovery. Initializing discovery latency in discovery

responses.
%CAPWAP-3-EVENTLOG: CAPWAP State: Discovery.
CAPWAP Control mesg Sent to 192.168.70.10, Port 5246
Msg Type : CAPWAP_DISCOVERY_REQUEST
CAPWAP Control mesg Recd from 192.168.5.54, Port 5246
HLEN 2, Radio ID 0, WBID 1
Msg Type : CAPWAP_DISCOVERY_RESPONSE
CAPWAP Control mesg Recd from 192.168.5.55, Port 5246
HLEN 2, Radio ID 0, WBID 1
Msg Type : CAPWAP_DISCOVERY_RESPONSE
AP Discover/Join – AP Side
%CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
%CAPWAP-3-ERRORLOG: Selected MWAR '5500-5'(index 0).
%CAPWAP-3-EVENTLOG: Selected MWAR '5500-5' (index 2).
%CAPWAP-3-EVENTLOG: Ap mgr count=1
%CAPWAP-3-ERRORLOG: Go join a capwap controller
%CAPWAP-3-EVENTLOG: Adding Ipv4 AP manager 192.168.5.55 to least load

%CAPWAP-3-EVENTLOG: Choosing AP Mgr with index 0, IP = 192.168.5.55, load = 3..
%CAPWAP-3-EVENTLOG: Synchronizing time with AC time.

%CAPWAP-3-EVENTLOG: Setting time to 15:41:52 UTC Jan 2 2014
%CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 192.168.5.55

peer_port: 5246
AP Discover/Join – WLC Side
04:da:d2:4f:f0:50 Discovery Request from 192.168.5.156:7411
04:da:d2:4f:f0:50 ApModel: AIR-CAP2602I-E-K9
apModel: AIR-CAP2602I-E-K9
apType = 27 apModel: AIR-CAP2602I-E-K9
apType: Ox1b bundleApImageVer: 8.3.141.0
version:8 release:3 maint:141 build:0
04:da:d2:4f:f0:50 Discovery Response sent to 192.168.5.156 port 7411
44:03:a7:f1:cf:1c DTLS keys for Control Plane are plumbed successfully for AP
192.168.5.156. Index 7
44:03:a7:f1:cf:1c DTLS Session established server (192.168.5.55:5246), client
(192.168.5.156:7411)
44:03:a7:f1:cf:1c Starting wait join timer for AP: 192.168.5.156:7411

04:da:d2:4f:f0:50 Join Request from 192.168.5.156:7411
04:da:d2:4f:f0:50 Join Response sent to 192.168.5.156:7411
04:da:d2:4f:f0:50 CAPWAP State: Join

AP Join – Country Mismatch – AP Side
%CAPWAP-3-ERRORLOG: Selected MWAR '5500-4'(index 0).
%CAPWAP-3-ERRORLOG: Go join a capwap controller
%CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.5.54 peer_port:

5246
%CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 192.168.5.54
peer_port: 5246
%CAPWAP-5-SENDJOIN: sending Join Request to 192.168.5.54
%CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
%CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.

%CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
%CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 192.168.5.54
AP Join – Country Mismatch – WLC Side
#CAPWAP-3-POST_DECODE_ERR: capwap_ac_sm.c:5660 Post decode processing failed for

Config status from AP 04:da:d2:28:94:c0
#LWAPP-3-RD_ERR4: capwap_ac_sm.c:3085 The system detects an invalid regulatory domain

802.11bg:-A 802.11a:-A for AP 04:da:d2:28:94:c0
#LOG-3-Q_IND: spam_lrad.c:10946 Country code (ES ) not configured for AP

04:da:d2:28:94:c0[...It occurred 2 times.!]
*** The key point is to make sure you look at both sides of an AP Join issue.
Lightweight AP Joins – Key Ideas
➢ Make sure the AP is getting an address from DHCP (check the DHCP server leases
for the AP’s MAC address)
➢ If the AP’s address is statically set, ensure it is correctly configured. The AP will
switch back to DHCP if a number of join failures occur.
➢ Try pinging from AP to controller and vice versa
➢ If pings are successful, ensure the AP has at least one method to discover the WLC
➢ Console or telnet/ssh into the controller to run debugs
➢ If you do not have access to APs, use “show cdp neighbors port <x/y> detail” on
connected switch to verify if the AP has an IP address
Advanced Wireless
DGTL-BRKEWN-3011
#CiscoLive
Understanding
Multicast
Multicast Transport
• Problem: how to replicate multicast traffic to all Aps
Multicast Transport
• Solution 1: Unicast replication
• Simple, but costly
Client to AP: IP SRC: Client IP DST: Multicast Payload
AP to WLC: IP SRC: AP IP DST: WLC CAPWAP IP SRC: Client IP DST: Multicast Payload
WLC to Aps: IP SRC: WLC IP DST: AP1 CAPWAP IP SRC: Client IP DST: Multicast Payload
IP SRC: WLC IP DST: AP2 CAPWAP IP SRC: Client IP DST: Multicast Payload
IP SRC: WLC IP DST: APN CAPWAP IP SRC: Client IP DST: Multicast Payload
Multicast Transport
• Solution 2: Multicast replication
• Efficient but needs network infrastructure support
Client to AP: IP SRC: Client IP DST: Multicast Payload
AP to WLC: IP SRC: AP IP DST: WLC CAPWAP IP SRC: Client IP DST: Multicast Payload
WLC to Aps: IP SRC: WLC IP DST: MCAST CAPWAP IP SRC: Client IP DST: Multicast Payload
Multicast Transport
• Most platforms support Multicast Mode, preferred over Unicast Mode
• Your network between WLC and AP must support multicast routing
• WLC will construct “MGID” to represent the multicast traffic relation between AP and
WLCs, against VLANs
• Common problems:
• Traffic is lost from WLC to AP due to multicast drop in network
• Using wrong multicast destination address
• Duplicate address between WLCs
• Why is this needed: IPv6, Multicast applications, ARP replication
• Bonjour may not need it, if using mDNS feature
Multicast Verification
• WLC L2 MGID
(3500-1) >show network multicast mgid summary
Layer2 MGID Mapping:
-------------------
Interface Name vlanId MGID
-------------------------------- ------ ----
management 15 0
vlan2 2 11
vlan50 50 10
• WLC L3 MGID
(3500-1) >show network multicast mgid summary
..
Layer3 MGID Mapping:
-------------------
Number of Layer3 MGIDs........................... 5
Group address VLAN MGID IGMP/MLD

--------------------------------------- ---- ---- --------
234.5.6.7 50 12356 IGMP
• AP L2 MGID
ap1700-sw1mgig-1-0-18#sh capwap mcast mgid all
L2 MGID Information:
L2 MGID = 1 WLAN bit map (all slots) = 0x0005 VLAN ID = 0
Slot map/tx-cnt: R0:0x0005/8 R1:0x0005/11 R2:0x0000/0
L2 MGID = 10 WLAN bit map (all slots) = 0x0005 VLAN ID = 50

L3 MGID Information
• AP L3 MGID
L3 MGID Information
L3 MGID = 12356 WLAN bitmap = 0x0001
Clients per Wlan
Wlan : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Clients R0:
Clients R1: 1
Clients R2:
• At Switch
r3-sw1#sh ip igmp groups
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
Group Accounted
239.255.255.240 Vlan30 00:43:28 00:02:43 192.168.30.115
239.0.85.86 Vlan5 2w1d 00:02:35 192.168.5.186
234.5.6.7 Vlan50 00:00:23 00:02:36 192.168.50.15
234.5.6.9 Vlan50 00:00:23 00:02:36 192.168.50.15
234.5.6.8 Vlan50 00:00:23 00:02:36 192.168.50.15
234.5.6.11 Vlan50 00:00:23 00:02:36 192.168.50.15
Bonjour
Troubleshooting
mDNS – Multicast DNS
• mDNS is typically see used for both Apple and Google/Android devices
• Enabling the mDNS service (mDNS Global Snooping checkbox) on the WLC will enable the
caching of mDNS service advertisements.
• Verify that Multicast has been enabled on the WLC. If you are using the Multicast/Multicast AP
multicast mode verify that you have full multicast routing enabled on the wired infrastructure.
• If your mDNS services and clients are on the same VLAN, then the mDNS server is not needed,
but the WLC multicast must be enabled.
• Each query or advertisement is sent to the Bonjour multicast address for delivery to all clients on
the subnet. Apple’s Bonjour protocol relies on Multicast DNS (mDNS) operating at UDP port
5353 and sends to these reserved group addresses:
➢ IPv4 Group Address - 224.0.0.251 debug mdns error enable
➢ IPv6 Group Address - FF02::FB debug mdns message enable
debug mdns detail enable
debug mdns all enable
mDNS – Default Services
The mDNS default services on the WLC
(wlc)> show mdns service summary
Number of Services.............................. 10
Mobility learning status ........................ Enabled
Service-Name LSS Origin No SP Service-string
-----------------------------------------------------------------
AirTunes No All 0 _raop._tcp.local.
Airplay No All 0 _airplay._tcp.local.
HP_Photosmart_Printer_1 No All 0 _universal._sub._ipp._tcp.local.
HP_Photosmart_Printer_2 No All 0 _cups._sub._ipp._tcp.local.
HomeSharing No All 0 _home-sharing._tcp.local.
Printer-IPP No All 0 _ipp._tcp.local.
Printer-IPPS No All 0 _ipps._tcp.local.
Printer-LPD No All 0 _printer._tcp.local.
Printer-SOCKET No All 0 _pdl-datastream._tcp.local.
iTuneWirelessDeviceSharing 2 No All 0 _apple-mobdev2._tcp.local.
mDNS – Custom Service Strings
(wlc) > show mdns service detailed GoogleCast
Service Name...................................... GoogleCast

Service String...................................... _googlecast._tcp.local.
Service Id............................................ 3
Service query status............................ Enabled
Service LSS status............................... Disabled
Service learn origin.............................. Wireless
Number of Profiles............................... 1
Profile.................................................. default-mdns-profile
Number of Service Providers .............. 0

Number of priority MAC addresses ..... 0
mDNS – Services Learned
WLC> show mdns service detailed GoogleCast
Service Name..................................... GoogleCast
Service String..................................... _googlecast._tcp.local.
Service Id............................................ 4
Service query status........................... Enabled
Service LSS status.............................. Enabled
Service learn origin............................. Wireless
Number of Profiles.............................. 1
Profile.................................................. default-mdns-profile
Number of Service Providers ............... 1

Number of priority MAC addresses ...... 0
ServiceProvider MAC Address AP Radio MAC Vlan Id Type TTL Time left
-----------------------------------------------------------------------------------------------
39b124._googlecast._tcp.local. 54:60:09:B5:A2:60 00:A6:CA:F1:4B:00 279 Wireless 4500 4361
mDNS - Query and Response - Success
38:71:de:4e:43:8b Query Service Name: _airplay._tcp.local., RR-Type: TYPE_DOMAIN_NAME_PTR , Class: 1
sendBonjourPkt : 3881 sendBonjourPkt msg-type = BONJOUR_AGGREGATED_QUERY_RESPONSE toSend =

Wireless
processBonjourPacket : 1017 qNameStr : _airplay._tcp.local., bonjServiceNameStr : _airplay._tcp.local.,

bonjSpNameStr : _airplay._tcp.local.
Building bonjour packet; mtu = 1412
Service Name : AppleTV Service String : _airplay._tcp.local. is supported in MSAL-DB
buildBonjourPacket : 2936 allVlan = 0 , vlanId = 279

buildBonjourPacket : 2948 simInterfaceMacAddrGet( vlan279 ) = F4:7F:35:B6:B9:EF
SRV-TYPE = AppleTV SP = RR-TYPE = TYPE_DOMAIN_NAME_PTR VLAN = 279

bgSendQueryResponseMsg : 4164 Sending BONJOUR_AGGREGATED_QUERY_RESPONSE SRV :AppleTV , MsgId
:0, dst-Mac :38:71:de:4e:43:8b , dst-IP :192.168.179.104
mDNS - Query and Response - Failure
Watch out for applications that use sub-types in the mDNS requests (examples are Netflix, HBO
GO, etc…), while the WLC has support for Apple sub-types (see CSCue05421), not all sub-types
may be supported with all service strings. (see CSCvc99976)
*Bonjour_Msg_Task: processBonjourPacket : 1017 qNameStr : _CA5E8412._sub._googlecast._tcp.local.,

bonjServiceNameStr : _CA5E8412._sub._googlecast._tcp.local., bonjSpNameStr :
*Bonjour_Msg_Task: processBonjourPacket : 1067 Queried service-string :

_CA5E8412._sub._googlecast._tcp.local. is not configured in MSAL-DB
Mobility
Troubleshooting
What is Mobility
• Protocol between AireOS controllers, to allow smooth client roaming

• Two “flavors”:
• Legacy Mobility: used by AireOS, flat, simple. Data goes over Ethernet over IP
• Legacy Encrypted: same, but using CAPWAP transport (Secure Mobility, starting in 8.5)
** Note: Catalyst 9800 series Wireless Controllers

• Data goes over CAPWAP and is always encrypted for Mobility Control and optional for Mobility Data
• Intra Controller roaming between AireOS (WLC) and 9800 Series is possible starting 8.8.111.0 and above
• If you need to use 8.5, you will need a special 8.5.164.0 (IRCM) version.
• See: Cisco Catalyst 9800 Wireless Controller-Aireos IRCM Deployment Guide
Mobility Group vs. Mobility Domain
▪ Mobility Group - WLCs with the same group name

• L2/L3 Handoff
• Auto Anchoring
• Fast Secure Roaming
• APs get all of these as a Discover candidate
▪ Mobility Domain - WLCs in the mobility list

• L2/L3 Handoff
• Auto Anchoring
Layer 2 roaming
Client
▪ Same VLAN on both WLCs Entry
Data
▪ Only one Client entry Old WLC
remains
Association
▪ Data termination point is
moved to new WLC (local)
New
WLC
Mobile Announce Entry
Mobile Handoff
Data
Layer 3 roaming
Client
▪ VLAN does not match Entry
Data
▪ Two Client entries Old WLC
▪ L3 data termination point Association
remains at old WLC
(Anchor)
New
▪ New WLC is L2 termination WLC
(Foreign) Mobile Announce Entry
Mobile Handoff
Data Mobile Confirm
EthoIP
Thank you
#CiscoLive

Advanced Wireless Troubleshooting: Tim Smith, Technical Consulting Engineer

Uploaded by

Copyright:

Available Formats

You might also like

Advanced Wireless Troubleshooting: Tim Smith, Technical Consulting Engineer

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced Wireless Troubleshooting: Tim Smith, Technical Consulting Engineer

Uploaded by

Copyright:

Available Formats

Advanced Wireless

➢ Client On-Boarding • Section 5

Same client type across different Aps,WLC

▪ Troubleshooting 101 Problem

▪ From very broad, to

Can ping local net

▪ Step 2: Understand any possible triggers

(Cisco Controller) >show client detail 00:16:ea:b2:04:36

▪ Processing assoc-req station:38:71:de:4e:43:8b AP:70:70:8b:a3:2e:20-01 ssid : CiscoLive

▪ Adding mobile on LWAPP AP 70:70:8b:a3:2e:20(1)

▪ Association received from mobile on BSSID 70:70:8b:a3:2e:2f AP CAP2802

▪ Station: 38:71:DE:4E:43:8B is associating to AP 70:70:8B:A3:2E:20 which is not XOR roam capable

▪ Allocate AID 1 slot 1 on AP CAP2802 #clients on this slot 1

▪ 0.0.0.0 START (0) Initializing policy

▪ AuthenticationRequired = 0 [NOTE: 0 means no further authentication such as 802.1x, WebAuth is needed]

▪ Changing state for mobile 38:71:de:4e:43:8b on AP 70:70:8b:a3:2e:20 from Idle to Associated

▪ Received management frame ACTION on BSSID 70:70:8b:a3:2e:2f destination addr 70:70:8b:a3:2e:2f

▪ Station: 38:71:DE:4E:43:8B sent 802.11K neighbor request to AP 70:70:8B:A3:2E:20

• Blacklisted Client (either through repeated Authentication failures or manually excluded)

• Data rate mismatch in the client association request

• Blocking WiFi Direct :

• Clients probing but not connecting:

Client is in DHCP_REQD state DHCP Proxy Enabled DHCP Proxy Disabled

Between WLC and Server

Between Client and Server

▪ 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

NPU type 9 is IP Learn

▪ Multiple mechanisms to learn Client IP address:

EAPoL 4 way Exchange

• Starting key exchange to mobile bc:9f:ef:1b:89:ef, data packets will be dropped

• Allocating EAP Pkt for retransmission to mobile bc:9f:ef:1b:89:ef

• validating eapol pkt: key version = 2

• Received EAPOL-Key from mobile bc:9f:ef:1b:89:ef

• key Desc Version FT – 0

• Received EAPOL-key in PTK_START state (message 2) from mobile bc:9f:ef:1b:89:ef

• Successfully computed PTK from PMK!!!

• Received valid MIC in EAPOL Key Message M2!!!!!

• Compare RSN IE in association and EAPOL-M2 frame(Skip pmkIdLen:0,and grpMgmtCipherLen:0)

• validating eapol pkt: key version = 2

• Received EAPOL-Key from mobile bc:9f:ef:1b:89:ef

• key Desc Version FT - 0

• Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile bc:9f:ef:1b:89:ef

• Mobility query, PEM State: L2AUTHCOMPLETE

• Incorrect Pre-Shared Key configured on the client

• Received Identity Response (count=1) from mobile 70:d4:f2:c9:bb:ae

• EAP State update from Connecting to Authenticating for mobile 70:d4:f2:c9:bb:ae

• Entering Backend Auth Response state for mobile 70:d4:f2:c9:bb:ae

• Processing Access-Challenge for mobile 70:d4:f2:c9:bb:ae

• Entering Backend Auth Req state (id=2) for mobile 70:d4:f2:c9:bb:ae

• Sending EAP Request from AAA to mobile 70:d4:f2:c9:bb:ae (EAP Id 2)

• Entering Backend Auth Response state for mobile 70:d4:f2:c9:bb:ae

• Processing Access-Challenge for mobile 70:d4:f2:c9:bb:ae

• Entering Backend Auth Req state (id=3) for mobile 70:d4:f2:c9:bb:ae

• Sending EAP Request from AAA to mobile 70:d4:f2:c9:bb:ae (EAP Id 3)

• Entering Backend Auth Response state for mobile 70:d4:f2:c9:bb:ae

• Processing Access-Accept for mobile 70:d4:f2:c9:bb:ae

• Username entry (CiscoLive) created for mobile, length = 253

• Username entry (CiscoLive) created in mscb for mobile, length = 253