Professional Documents
Culture Documents
WAAS TS Guide PDF
WAAS TS Guide PDF
WAAS TS Guide PDF
3 and Later
Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to collaborate on Cisco product documentation. You do not
need to log in to read the text. However, you must log in to edit the text. Select the "edit" tab to edit an article or select the "discussion" tab to submit questions
or comments about documentation content.
See Terms of Use and About DocWiki for more information about Cisco DocWiki.
Contents
• 1 Audience
• 2 Organization
• 3 Creating a PDF of the WAAS
Troubleshooting Wiki
• 4 Related Documentation
This guide provides a systematic approach to identifying and remedying problems that may arise as you use your WAAS system
over a period of time. This guide is not intended to replace configuration best practices or to be an all-inclusive guide for every
application. Rather, it is an attempt to provide you with the knowledge and skills necessary to correct the most common issues that
you may encounter.
Audience
This article is intended for trained network administrators who have experience with the configuration and maintenance of the
WAAS system.
Organization
This article consists of the following major sections:
Troubleshooting Optimization
Contents 1
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting vWAAS
Related Documentation
• Release Notes for Cisco Wide Area Application Services
• Cisco Wide Area Application Services Quick Configuration Guide
• Cisco Wide Area Application Services Configuration Guide
• Cisco Wide Area Application Services vWAAS Installation and Configuration Guide
• Cisco Wide Area Application Services Command Reference
• Cisco Wide Area Application Services Upgrade Guide
• Cisco Wide Area Application Services API Reference
• Cisco Wide Area Application Services Monitoring Guide
• Using the Print Utilities to Troubleshoot and Fix Samba Driver Installation Problems
• Cisco WAAS Installation and Configuration Guide for Windows on a Virtual Blade
• Cisco WAAS Installation and Configuration Guide for ACNS on a Virtual Blade
• Cisco WAAS on Service Modules for Cisco Access Routers
• Cisco SRE Service Module Configuration and Installation Guide
• Configuring Cisco WAAS Network Modules for Cisco Access Routers
This article describes the WAAS architecture and how data flows into, gets processed, and flows out of a WAAS device. It
provides a basic understanding of these concepts to assist you in troubleshooting the WAAS system.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Organization 2
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 Understanding the WAAS
Architecture
♦ 1.1 AOs
♦ 1.2 WoW and Virtual
Blades
♦ 1.3 Configuration
Management System
♦ 1.4 DRE with Scheduler
♦ 1.5 Storage
♦ 1.6 Network I/O
♦ 1.7 Interception & Flow
Management
◊ 1.7.1
Auto-Discovery
◊ 1.7.2 Policy
Engine
◊ 1.7.3
Filter-Bypass
• 2 WAAS Traffic Flow
Related Documentation 3
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
The WAAS system architecture is divided into a series of functional areas or services as shown in Figure 1.
AOs
AOs (Application Optimizers, also known as Application Accelerators) are the application-specific software that optimizes certain
protocols at Layer 7 (beyond the generic Layer 4 optimizations). AOs may be viewed as the "applications" in the WAE system (in
an OS analogy). The generic AO acts as a catch-all for all traffic that has no protocol-specific AO and also functions as a delegate
if a protocol-specific AO decides to not apply optimization.
Storage
The storage system manages the system disks and the logical RAID volumes on systems that have multiple disks. Disk storage is
used for system software, the DRE cache, the CIFS cache, and virtual blade storage.
Network I/O
The network input/output component is responsible for all aspects that are related to handling data communication coming into or
going out from a WAE, including WAE to WAE communication and WAE to client/server communication.
Auto-Discovery
Auto-discovery allows peer devices to discover each other dynamically and does not require that you preconfigure WAE pairs.
Auto-discovery is a multi-WAE end-to-end mechanism that defines a protocol between the WAEs that discovers a pair of peer
WAEs for a given connection.
WAE devices automatically discover one another during the TCP three-way-handshake that occurs when two nodes establish a
TCP connection. This discovery is accomplished by adding a small amount of data to the TCP options field (0x21) in the SYN,
SYN/ACK, and ACK messages. This TCP option allows WAE devices to understand which WAE is on the other end of the link
and allows the two to describe which optimization policies they would like to employ for the flow. If intermediate WAEs exist in
the network path, they simply pass through flows that are being optimized by other WAEs. At the end of the auto-discovery
process, the WAEs shift the sequence numbers in the TCP packets between the participating WAEs by increasing them to more
than 2 billion, to mark the optimized segment of the connection.
Policy Engine
The policy engine module determines if traffic needs to be optimized, which AO to direct it to, and the level of data-reduction
(DRE) if any, that should be applied to it. The policy engine classifies traffic beyond connection establishment (for example,
based on payload information) and changes the flow of a connection dynamically from unoptimized to optimized.
• Application definition: A logical grouping of traffic to help report statistics on the type of traffic.
• Traffic classifier: Access control list (ACL) that helps choose connections based on IP addresses, ports, and so on.
• Policy Map: Binds the application and the classifier with an action, which specifies the type of optimization, if any, to be
applied. There are two kinds of policy maps:
♦ Static policy map: Configured on the device through the CLI or GUI (or installed by default) and is persistent
unless removed.
♦ Dynamic policy map: Automatically configured by the WAE and has a life span just long enough to accept a new
connection.
The following configuration example shows a policy engine application definition (Web) that includes a classifier (HTTP) and an
action (optimize full accelerate http):
Filter-Bypass
After interception, the filter-bypass module acts as the mediator between the policy engine and auto discovery. The filter-bypass
module tracks all optimized connections in a filtering table for the life of the connection. Additionally, it tracks pass-through
connections, but pass-through table entries are timed out after 3 seconds.
Figure 2 shows the filter-bypass flow establishment as a packet enters the system.
1. A SYN packet on a flow enters the system. This packet is routed to the filter-bypass module.
2. The filter-bypass module consults the policy engine on how the flow should be handled.
2a. The policy engine consults the configured and dynamically added policies, and based on the current operational status of the
AOs and SO-DRE, decides what the WAE could do for this flow: pass through, locally terminate, or optimize.
2b. The packet and the decision from the policy engine are then returned to the filter-bypass module.
3. The filter-bypass module acts on the policy engine decision in one of the following ways:
If the filter-bypass module chooses option 3c, the packet is sent to the auto-discovery module. The auto-discovery module
Policy Engine 6
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
determines what optimizations can be done, based on the availability of a peer WAE and its enabled features. A peer WAE is
discovered through the use of TCP options added during the TCP handshake to the remote node. If the auto-discovery module
determines that a peer WAE is available, the connection is handed off for further processing once the TCP three-way-handshake
completes. If a peer WAE is discovered for the first time, the WAEs additionally negotiate about AO versions and capabilities.
This information is used to decide the AO level capabilities for the connection.
4. The connection is finally admitted into the system with specific L4 and L7 optimizations and handed off to the appropriate L4
(DRE) and L7 (AO) acceleration modules. For connections that are later discovered to be not optimize-able by protocol-specific
AOs (HTTP, MAPI, and so on), the connection is handled by the generic AO, with or without DRE optimization (as negotiated
during connection establishment).
This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when
you configure and use your WAAS system.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 Overview of the WAAS Troubleshooting Process
• 2 Verifying the WAAS Image
• 3 Enabling WAAS Logging
• 4 Running Diagnostics
• 5 Verifying the Physical Connectivity Between Peer WAAS Devices and to Application
Servers
• 6 Checking CPU Load
• 7 Gathering WAAS Troubleshooting Information
♦ 7.1 Rebooting the WAAS Device
♦ 7.2 Using show Commands
1. Maintain a consistent and recommended software version across all your WAAS devices. If versions must differ, the
Central Manager must be running the highest version. See the "Verifying the WAAS Image" section to determine the
version in use.
2. See the WAAS release notes for your software version for the latest features, operating considerations, caveats, and CLI
command changes.
3. Before you introduce configuration changes on the WAAS Central Manager, use the CMS backup feature to save your
configuration. If you run into problems with the new configuration, you can restore the previous configuration. See the
Backing Up and Restoring your WAAS System section in the Cisco Wide Area Application Services Configuration Guide.
Troubleshoot any problems with new configuration changes immediately after making them.
4. Verify that your configuration is correct for your network application. Make any required changes to the running-config
file, and then test the configuration. If it is satisfactory, save it to the startup-config file using the copy running-config
startup-config command.
5. Enable system message logging. See the "Enabling WAAS Logging" section.
6. Run the diagnostic tool to verify device functionality and connectivity. See the "Running Diagnostics" section.
7. Verify the physical connectivity between WAAS peers and to the application servers. See the "Verifying the Physical
Connectivity Between Peer WAAS Devices and to Application Servers" section.
8. Gather information that defines the specific symptoms. See the "Gathering WAAS Troubleshooting Information" section.
9. Refer to one of the other articles in this WAAS Troubleshooting Guide for information on troubleshooting specific
problems:
♦ If the system appears to be having hardware or disk problems, see the article Troubleshooting Disk and Hardware
Problems.
♦ If the system is having trouble receiving traffic, see the article Troubleshooting WCCP. This problem also could
be due to a firewall issue.
♦ If the system is passing through traffic instead of optimizing it or is having problems optimizing specific kinds of
application traffic (HTTP, MAPI, SSL, and so on), see the articles Troubleshooting Optimization and
Troubleshooting Application Acceleration.
♦ If the system is passing through more traffic than expected instead of optimizing it, see the article
Troubleshooting Overload Conditions.
10. After you have determined that your troubleshooting attempts have not resolved the problem, contact the Cisco Technical
Assistance Center (TAC) or your technical support representative. See the "Contacting Cisco Technical Support" section.
Contents 8
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
• Device model (the numbers in the first part of the Version string encode the device model number; a WAE-7341 is shown
here.)
• WAE uptime
To verify that there is no pending software upgrade (waiting for a device reboot), enter the following command:
To enable logging to the console, enter the following global configuration command:
NOTE: Setting the logging priority to a level lower than notice can be CPU intensive and can generate a large amount of output.
Use it judiciously and sparingly in a production environment.
You can use the following file system navigation commands to navigate and view the log files:
• cd
• pwd
• dir
• type-tail filename lines [| | follow]
• find-pattern
Running Diagnostics
The WAAS Central Manager includes a built-in diagnostic tool that can help you troubleshoot many device problems, including
the following:
• Network configuration
• Interface configuration
• Connectivity to hosts
• WCCP configuration
• Inline configuration
• TFO configuration
• WAFS configuration
We recommend that you run the diagnostic tool first before taking other troubleshooting actions. The tool reports on the status and
configuration of many system functions.
To run the diagnostic tool from the Central Manager, follow these steps:
1. From the WAAS Central Manager GUI navigation pane, choose My WAN > Manage Devices (or Manage Device
Groups).
2. Click the Edit icon next to the name of the device (or device group) for which you want to perform diagnostic tests.
3. In the navigation pane, choose Troubleshoot > Diagnostics Tests. The Diagnostic Tool window appears.
4. Check the check box next to each diagnostic test that you want to run, or check the top check box to run all tests.
5. Click Run.
6. View the test results in the lower part of the window. You may have to scroll the window to see all results.
For tests that fail, error messages describe the problem and provide recommended solutions. You can find error message
descriptions in the test command in the Cisco Wide Area Application Services Command Reference.
You can run the same diagnostic tests again and refresh the results by clicking the Refresh icon in the taskbar.
To run the diagnostic tests from the CLI, use the test EXEC command.
To verify the physical connectivity of the peer WAAS device, follow these steps:
1. Check all cable connections on the switch or router that may impact the WAAS device.
2. Use the ping command to send an ICMP Echo request to the peer WAE.
If a device is one hop away and you are unable to reach the device, then ping the intermediary gateway. If the gateway is not
reachable, enter the show ip routes command and check to make sure that the correct route is displayed. For example, enter:
You can use a similar ping command to verify connectivity between the WAAS data center device and the application server
hosts.
Note that firewalls might block ICMP traffic and ICMP traffic does not follow the WCCP redirection path, so using the ping
command does not verify redirection or acceleration. As an alternative you could use a third party tool that performs a TCP-based
ping.
1. From the WAAS Central Manager GUI navigation pane, choose My WAN > Manage Devices.
2. Click the Edit icon next to the name of the device on which you want to check the CPU load.
3. In the navigation pane, choose Monitor > Platform > CPU Statistics.
Verifying the Physical Connectivity Between Peer WAAS Devices andto Application Servers 11
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
You may want to adjust the time period of the chart, since the default is Last Hour. To adjust the time period, click the Settings
icon in the task bar and choose a different Time Frame such as Last Day or Last Week.
It is common for a WAAS device to show spikes or even longer durations of high CPU utilization during high user activity
periods. When the CPU remains at a high CPU level for significantly long durations, further troubleshooting or resizing of the
device may be indicated.
copy tech-support {disk filename | ftp {hostname | ip-address} remotedirectory remotefilename | tftp {hostname | ip-address}
remotefilename}
For example, to copy the output of the command to a disk file on the local system, specify the command as follows:
• For WCCP deployments, use the following commands on the router or switch (for each service group, where applicable):
♦ show ip wccp
♦ show ip wccp interfaces detail
♦ show ip wccp service
♦ show ip wccp service detail
• For WCCP deployments, use the following commands on the router or switch when hashing is used:
♦ show tcam counts
♦ show mls stat
♦ show mls netflow table detail
♦ show mls netflow ip count
♦ show mls netflow ip sw-installed count
♦ show mls netflow ip sw-installed detail
♦ show fm inteface interface_name
• For WCCP deployments, use the following commands on the router or switch when masking is used:
♦ show ip wccp service mask
♦ show ip wccp service merge
♦ show tcam interface interface_name acl {in | out} ip
♦ show tcam interface interface_name acl {in | out} ip detail
Before generating a system report, use the test command to run the diagnostic tests so that this information is included in the
system report. When generating a system report on a Central Manager (or standby Central Manager), you should first make a
database backup by using the cms database backup command.
To generate a sysreport and store it to an FTP server, use this form of the command: copy sysreport ftp server-ip
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
remote-directory remote-file-name
For example:
When generating a system report, do not use any command options that limit the report to a specific time period, as this could
cause information even within that time period not to be included.
Two packet capture utilities are available: tcpdump and tethereal. These commands require admin privileges.
By default, these commands capture only the first 64 bytes of each packet. We recommend that you use the -s 1600 option to
capture full packet data.
If you will be taking large traces, use tcpdump to create rolling packet captures in multiple files. (The -C option sets the
maximum size of each captured file in KB and the -M option sets the maximum number of log files to create.)
If you need to filter the packets captured, use tethereal with the -R read filter option. You can use tcpdump to create a large
packet capture, then use tethereal against the captured file to perform filtering.
Be careful when using tcpdump in a WCCP environment because tcpdump filters do not look within the GRE wrapper. You will
need to use tethereal if you need to do that.
With both commands, use the -i any option to capture all interfaces, or separate telnet sessions to capture on separate interfaces.
Use ^c (CTRL+c) to stop the packet capture.
There are several packet analysis tools that you can use to analyze packet capture files after you have captured them:
• Wireshark: A free packet analysis tool with extensive capabilities (recommended over Ethereal).
• Ethereal: Another free packet analysis tool with extensive capabilities.
• Microsoft Netmon: Included with Windows server software.
• Sniffer Pro
Using tcpdump
For the full tcpdump syntax, see tcpdump in the Cisco Wide Area Application Services Command Reference.
• -i interface : The interface where you want to capture packets, for example:
♦ lo : localhost
♦ eth0 : GigabitEthernet 1/0
♦ eth1 : GigabitEthernet 2/0
♦ eth2 : InlinePort 1/1/wan
♦ eth3 : InlinePort 1/1/lan
Using tethereal
For the full tethereal syntax, see tethereal in the Cisco Wide Area Application Services Command Reference.
• -R read_filter: Filtering can be very useful. Use the same filtering syntax as you would use with Ethereal or Wireshark, so
you can use one of those tools to help you compose a filter. tethereal is also useful for file conversion and filtering of a
packet capture file that has already been captured (for example, from tcpdump).
• -F output_filetype: The default filetype is a libpcap file; however, the following options are available:
♦ libpcap - libpcap (tcpdump, Ethereal, etc.)
♦ rh6_1libpcap - RedHat Linux 6.1 libpcap (tcpdump)
♦ suse6_3libpcap - SuSE Linux 6.3 libpcap (tcpdump)
♦ modlibpcap - modified libpcap (tcpdump)
♦ nokialibpcap - Nokia libpcap (tcpdump)
♦ lanalyzer - Novell LANalyzer
♦ ngsniffer - Network Associates Sniffer (DOS-based)
♦ snoop - Sun snoop
♦ netmon1 - Microsoft Network Monitor 1.x
♦ netmon2 - Microsoft Network Monitor 2.x
♦ ngwsniffer_1_1 - Network Associates Sniffer (Windows-based) 1.1
♦ ngwsniffer_2_0 - Network Associates Sniffer (Windows-based) 2.00x
♦ nettl - HP-UX nettl trace
♦ visual - Visual Networks traffic capture
♦ 5views - Accellent 5Views capture
♦ niobserverv9 - Network Instruments Observer version 9
The following examples show various options used for filtering and conversion:
To convert from one file format to another, use a command similar to the following:
To use a read filter for the SYN flag, use a command similar to the following:
Note: The tethereal command has some usage caveats that you should be aware of:
• A filter defined using the -R option is ignored when it is combined with the -w option (writing to a file) in WAAS 4.1.1
and 4.1.3. To filter captured traffic and write to a disk file, use -f option to specify a capture filter. This issue is resolved in
version 4.1.5.
• When using the -a option to print heavy traffic to the screen, it can take significantly longer than the autostop duration to
display the information on the screen. Wait for the command to finish. Displaying output to the console can take
significantly longer than through telnet or SSH, so console display is not recommended.
• When using the -f option with the "host" or "not host" filter expression, the wrong traffic may be captured with WCCP
GRE encapsulated or VLAN traffic. With WCCP GRE traffic, tethereal sees only the outermost IP address, not the
original IP address inside the encapsulated packets. Add the "proto 47" keyword into the -f filter expression to capture the
correct traffic. Additionally, for VLAN traffic, add the "vlan" keyword into the -f filter expression to let the command
parse VLAN traffic correctly.
• When using the -a filesize option together with the -R option, tethereal may stop unexpectedly and print the message
"Memory limit is reached" before reaching the specified autostop file size. In this case, the maximum memory limit for
the command was reached before the autostop file size limit.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 TFO
Troubleshooting
• 2 DRE
Troubleshooting
Basic WAAS optimizations include TCP flow optimization (TFO), data redundancy elimination (DRE), and persistent
Lempel-Ziv (LZ) compression.
TFO Troubleshooting
The number of TCP connections, their status, and disposition can give an indication of the health of the WAAS system in a
specific location. A healthy system will show a large number of connections, with a significantly large percentage of these closed
normally. The show statistics tfo detail command provides an indication of the volume, status, and disposition of connections
between a particular WAAS device and other devices in the network.
You can view global TFO statistics by using the show statistics tfo detail command as follows:
The No. of active connections field reports the number of connections that are currently being optimized.
In the Policy Engine Statistics section of the output, the Rejected Connection Counts section show various reasons why
connections have been rejected. The Connection Limit counter reports the number of times that a connection has been rejected
because the maximum number of optimized connections has been exceeded. If you see a high number here, you should look into
overload conditions. See the article Troubleshooting Overload Conditions for more information.
Additionally, TFO optimization for connections that are pushed down from other AOs because they cannot optimize the traffic is
handled by the generic AO, which is covered in the article Troubleshooting the Generic AO.
TFO Troubleshooting 18
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
You can view TFO connection statistics by using the show statistics connection command. For details on using this command,
see the section "Checking the Optimized TCP Connections" in the Troubleshooting Overload Conditions article.
DRE Troubleshooting
When application acceleration is expected but not being observed, verify that the appropriate optimizations are being applied to
the traffic flow and that the DRE cache is reducing the size of the optimized traffic appropriately.
Policy engine maps for DRE and LZ optimization include the following:
Various conditions could cause DRE and/or LZ not to be applied to a connection, even though it is configured:
Note: In all of the above conditions, the show statistics connection command will report Acceleration of "TDL" for connections
where this was the negotiated policy. Looking at the amount of DRE or LZ bypass traffic will tell you whether DRE or LZ
optimizations were actually applied. Use the show statistics connection conn-id command, as described later, and look at the
DRE encode numbers to see if the DRE or LZ ratio is near 0% and most of the traffic is bypassed. The first three conditions will
be reported by the "Encode bypass due to" field and the last three conditions result from the traffic data pattern and are accounted
for in the reported DRE and LZ ratios.
You can view the statistics for a specific connection to determine what basic optimizations were configured, negotiated with the
peer, and applied by using the show statistics connection conn-id command. First you will need to determine the connection ID
for a particular connection by using the show statistics connection command, as follows:
You will find the connection IDs for each connection listed at the end of the output. To view the statistics for a specific
connection, use the show statistics connection conn-id command, as follows:
The Application Name and Classifier Name fields tell you the application and classifier applied to this connection.
The optimization policies are listed in the Policy Details section. If the Configured and Applied policies do not match, it means
that you configured one policy for this type of connection but a different policy was applied. This could result from the peer being
down, misconfigured, or overloaded. Check the peer WAE and its configuration.
The following section of output shows DRE encode/decode-related statistics including the number of messages, how many had
DRE applied, LZ applied, or bypassed DRE and LZ:
. . .
DRE: 353
The following statistics are highlighted in the above example for both encoding and decoding:
• Overall ratio - the overall compression ratio for the data including both DRE and LZ
DRE Troubleshooting 20
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
• DRE ratio - the compression ratio due to DRE alone
• DRE Bypass - the number of messages and bytes that bypassed DRE
• LZ ratio - the compression ratio due to LZ alone
• LZ Bypass - the number of messages and bytes that bypassed LZ
• Avg latency - the average latency for the encode or decode operation
If you see a large amount of bypass traffic, the DRE compression ratio will be smaller than expected. It could be due to encrypted
traffic, small messages, or otherwise uncompressible data. Consider contacting TAC for further troubleshooting help.
If you see a large amount of LZ Bypass traffic, this could be due to a large amount of encrypted traffic, which is not generally
compressible.
The Average latency numbers can be useful for debugging a throughput issue. Depending on the platform, both the encode and
decode average latency are typically in the single digits of ms. If users experience low throughput and one or both of these
numbers are higher, it indicates an issue with encoding or decoding, generally on the side with the higher latency.
It may be useful to look at the DRE statistics data such as the oldest usable data, cache size, percent of cache used, hash table
RAM used, and so on by using the show statistics dre detail command, as follows:
Cache:
Status: Usable, Oldest Data (age): 10h <-----Cache age
Total usable disk size: 311295 MB, Used: 0.32% <-----Percent cache used
Hash table RAM size: 1204 MB, Used: 0.00% <-----Output from here is in 4.3.3 and earlier o
. . .
If you are not seeing significant DRE compression, it could be because the DRE cache is not populated with enough data. Check if
the cache age is short and less than 100 percent of the cache is used, which would indicate this situation. The compression ratio
should improve as the cache fills with more data. If 100% of the cache is used and the cache age is short, it indicates that the WAE
may be undersized and not able to handle the traffic volume.
If you are not seeing significant DRE compression, look at the Nack/R-tx counters in the following section of command output:
Connection details:
Chunks: encoded 398832, decoded 269475, anchor(forced) 43917(9407) <-----In 4.3.3 and earlier only
Total number of processed messges: 28229 <-----In 4.3.3 and earlier only
num_used_block per msg: 0.053597 <-----In 4.3.3 and earlier only
Ack: msg 18088, size 92509 B <-----In 4.3.3 and earlier only
Encode bypass due to: <-----Encode bypass reasons
remote cache initialization: messages: 1, size: 120 B
last partial chunk: chunks: 482, size: 97011 B
skipped frame header: messages: 5692, size: 703 KB
Nacks: total 0 <-----Nacks
R-tx: total 0 <-----Retransmits
Encode LZ latency: 0.133 ms per msg
Decode LZ latency: 0.096 ms per msg
. . .
The Nacks and R-tx counters should generally be low relative to the traffic volume. For example, about 1 per 100 MB of original
(unoptimized) traffic. If you see significantly higher counts, it could indicate a DRE cache synchronization problem. Use the clear
cache dre command to clear the DRE cache on all devices, or contact TAC.
The encode bypass reasons counters report the number of bytes bypassed due to various reasons. This can help you determine
what is causing bypass traffic (other than an unoptimizable data pattern).
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
It is sometimes helpful to identify the connected and active peer WAEs and look at peer statistics, which you can do with the
show statistics peer dre command as follows:
Cache: Used disk: 544 MB, Age: 14d23h <-----Peer cache details in 4.3.3 and earlie
Cache: Used disk: 544 MB <-----Peer cache details in 4.4.1 and later
Peer version: 0.4 <-----
Ack-queue size: 38867 KB |
Buffer surge control: |<---In 4.3.3 and earlier only
Delay: avg-size 0 B, conn: 0, flush: 0 |
Agg-ft: avg-size 20902 B, conn: 388, flush: 0 |
remote low-buff: 0, received flush: 0 <-----
Other output from this command shows the encode and decode statistics similar to an individual connection.
This article describes a general troubleshooting methodology for all application accelerators.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
DRE Troubleshooting 22
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Contents
• 1 General Troubleshooting Methodology
for AOs
WAAS provides several application accelerators (also known as AOs) that accelerate various TCP protocols such as CIFS, HTTP,
NFS, EPM, MAPI, SSL, and live streaming video (RTSP). These application accelerators do not accelerate specific applications,
but rather they accelerate any application that uses the specified protocol.
To verify the configuration and operational status of all AOs, use the show accelerator and show license commands as shown in
Figure 1.
Contents 23
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
For specific information on troubleshooting individual AOs, see the following articles:
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Contents
• 1 CIFS AO Troubleshooting
♦ 1.1 CIFS AO Logging
♦ 1.2 Windows Print Accelerator
Troubleshooting
CIFS AO Troubleshooting
The CIFS accelerator transparently optimizes CIFS traffic on ports 139 and 445.
You can verify the general AO configuration and status with the show accelerator and show license commands, as shown in
Figure 1. The Enterprise license is required for CIFS accelerator operation.
Contents 25
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Next, verify the status that is specific to the CIFS AO by using the show accelerator cifs command, as shown in Figure 2. You
want to see that the CIFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State
is Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Use the show running-config command to verify that the CIFS traffic policy is properly configured. You want to see accelerate
cifs for the WAFS application action and you want to see appropriate match conditions listed for the CIFS classifier, as follows:
classifier CIFS
name WAFS classifier CIFS action optimize full accelerate cifs
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
WAE674# sh run | begin CIFS
...skipping
classifier CIFS
match dst port eq 139
match dst port eq 445
exit
Use the show statistics connection optimized cifs command to check that the WAAS device is establishing optimized CIFS
connections. Verify that "TCDL" appears in the Accel column for a connection. A "C" indicates that the CIFS AO was used.
D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
If you see "TDL" in the Accel column, the connection was optimized by transport optimizations only and was not inspected by the
CIFS AO. This situation can happen if the CIFS AO is disabled, the Enterprise license is not configured, or if the maximum
connection limit is reached.
If you see a "G" instead of a "C" in the Accel column, then the connection was pushed down from the CIFS AO to the generic AO
and was optimized with transport optimizations only. This situation can happen if the connection requires SMB2 or a digital
signature and an error message is logged for it.
In version 4.1.3, the syslog has the following error message for digitally signed connections:
2009 Apr 25 13:42:08 wae java: %WAAS-CIFSAO-4-131230: (146708) Connection to test1.example.com will be handled by
generic optimization only, since test1.example.com requires digital signing.
In version 4.1.5 and later, check the CIFS internal error logs to see the reason why the connection was pushed down to the generic
AO. In the cifs_err.log, look for this message for SMB2 connections:
In the cifs_err.log, look for this message for digitally signed connections:
To view similar information from the Central Manager, choose the WAE device, then choose Monitor > Optimization >
Connections Statistics.
CIFS AO Troubleshooting 27
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
You can view the CIFS connection statistics by using the show statistics connection optimized cifs detail command as follows:
Original Optimized
-------------------- --------------------
Bytes Read: 189314 10352510
Bytes Written: 91649704 28512
. . .
Connection details:
Chunks: encoded 3, decoded 49922, anchor(forced) 0(1)
Total number of processed messges: 1820
num_used_block per msg: 0.140659
Ack: msg 1609, size 7066 B
Encode bypass due to:
last partial chunk: chunks: 1, size: 142 B
skipped frame header: messages: 138, size: 27202 B
Nacks: total 0
R-tx: total 0
Encode LZ latency: 0.060 ms per msg
Decode LZ latency: 0.071 ms per msg
Aggregation encode: Retransmissions: 0 <-----Packets lost between peers
level 0: chunks: 3 hits: 0 miss: 3
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
level 1: chunks: 0 hits: 0 miss: 0
level 2: chunks: 0 hits: 0 miss: 0
level 3: chunks: 0 hits: 0 miss: 0
Aggregation decode: Collisions: 0
level 0: chunks: 174093 hits: 128716 miss: 0
level 1: chunks: 0 hits: 0 miss: 0
level 2: chunks: 0 hits: 0 miss: 0
level 3: chunks: 0 hits: 0 miss: 0
Aggregation stack memory usage: Sender: 452 B Receiver: 9119 B
Noise filter: Chunks: 0, Bytes: 0 B
. . .
If the Retransmissions counter increases, it means that packets are getting lost in the middle, between the two peer WAEs. This
situation will result in lower throughput. You should investigate possible causes for packet lose in the network between the two
peer WAEs.
You can view the CIFS request statistics by using the show statistics cifs requests command as follows:
CIFS AO Logging
The following log files are available for troubleshooting CIFS AO issues:
For easier debugging, you should first set up an ACL to restrict packets to one host.
CIFS AO Logging 29
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
WAE674(config)# ip access-list extended 150 permit tcp host 10.10.10.10 any
WAE674(config)# ip access-list extended 150 permit tcp any host 10.10.10.10
You can view the end of a transaction log file by using the type-tail command as follows:
To set up and enable debug logging of the CIFS AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
You can enable debug logging for CIFS connections and then display the end of the debug error log as follows:
Troubleshooting the Windows print accelerator is similar to troubleshooting the CIFS AO. You can verify the general AO
configuration and status with the show accelerator and show license commands, as shown in Figure 1. The CIFS accelerator
must be enabled and the Enterprise license is required. Next, verify the status specific to the CIFS AO by using the show
accelerator cifs command.
Use the show statistics windows-print requests command and verify that the "Documents spooled" and "Pages spooled"
counters are incrementing, as follows:
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 HTTP Accelerator Troubleshooting
♦ 1.1 Viewing HTTP Statistics
♦ 1.2 Viewing HTTPS
Statistics
♦ 1.3 Viewing the HTTP
Metadata Cache
♦ 1.4 Viewing the HTTPS
Metadata Cache
♦ 1.5 Metadata Cache
Cache-Control Behavior
♦ 1.6 Metadata Caching
Exceptions
• 2 HTTP AO Logging
Contents 31
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
• TCP connection reuse across the WAN. Avoids a connection setup penalty for subsequent HTTP connections requested
by the same client. (Does not apply to HTTPS traffic.)
• HTTP metadata caching. Certain HTTP responses are cached, along with their URLs and metadata information, so that
the edge WAE can respond locally to subsequent requests for the same URL. (Available only in version 4.2.1 and later.)
The three types of cached responses are as follows:
♦ 301 Permanently-Redirected
♦ 304 Not-Modified
♦ 401 Authorization-Required
• HTTPS metadata caching. Certain HTTPS responses are cached, along with their URLs and metadata information, so
that the edge WAE can respond locally to subsequent requests for the same URL. (Available only in version 4.3.1 and
later.)
• HTTP suppress server encoding. Removes the Accept-Encoding header from the HTTP and HTTPS requests,
preventing the server from sending compressed data towards the WAN. This allows the WAE to apply its own
compression, typically resulting in a better compression ratio. (Available only in version 4.2.1 and later.)
• DRE hints. Provides specific hints to the DRE module to better compress the HTTP and HTTPS traffic based on the
additional knowledge on the HTTP protocol provided by parsing the layer 7 payload:
♦ Skip header: Instructs the DRE module to not compress HTTP/HTTPS headers resulting in a better compression
of the object.
♦ Flush: Instructs the DRE module to start compressing as soon as an HTTP/HTTPS transaction is fully processed.
♦ Skip LZ: Instructs the DRE module to not apply LZ compression to all objects already compressed by the original
server, thus reducing the CPU overhead.
The HTTP metadata caching, suppress server encoding, and DRE hinting features can be configured separately. The TCP
connection reuse feature is always active when the HTTP AO is enabled and applies only to HTTP traffic.
You can verify the general AO configuration and status with the show accelerator and show license commands, as described in
the Troubleshooting Application Acceleration article. The Enterprise license is required for HTTP accelerator operation.
Next, verify the status that is specific to the HTTP AO by using the show accelerator http command, as shown in Figure 1. You
want to see that the HTTP AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State
is Enabled but the Operational State is Shutdown, it indicates a licensing problem. For each of the HTTP features the current mode
is shown (User/Default) along with the value (Enabled, Disabled or configured value). The Suppress Server Encoding and
Metadatacache items were added in version 4.2.1, and the DRE Hints and HTTPS Metadatacache items were added in version
4.3.1.
For HTTPS traffic to be optimized by both the SSL and HTTP AOs, ensure that one of these optional features is enabled: HTTPS
metadata caching, suppress-server-encoding or DRE hints.
Use the show running-config command to verify that the HTTP/HTTPS traffic policy is properly configured and which of the
features is enabled. You want to see accelerate http for the Web application action and you want to see appropriate match
conditions listed for the HTTP classifier, as follows:
classifier HTTP
classifier HTTPS
name Web classifier HTTP action optimize full accelerate http <----- HTTP acceleration
name Web classifier HTTPS action optimize DRE no compression none <----- HTTPS static policy applies to t
matching any SSL accelerated-ser
• How much time is being saved by the HTTP AO. You can see the overall Time Saved by the entire HTTP AO or the Time
Saved by each of the features:
♦ Time Saved by fast connection reuse
♦ Time Saved by the three metadata caches
• Number of cache hits/misses for the metadata caches
• Number of times suppress server encoding is applied to HTTP requests
• Number of times DRE hints are provided based on the content of the HTTP headers
• Number of HTTP transactions (request+response) processed
• Number of errors in the HTTP header processing
• Number of cache revalidations
HTTP:
Global Statistics
-----------------
Time Accelerator was started: Tue Apr 6 06:04:06 2010
Time Statistics were Last Reset/Cleared: Tue Apr 6 06:04:06 2010
Total Handled Connections: 3743984
Total Optimized Connections: 3743984
Total Connections Handed-off with Compression Policies Unchanged: 0
Total Dropped Connections: 0
Current Active Connections: 48
Current Pending Connections: 0
Maximum Active Connections: 176
Total Time Saved (ms): 35584437 <-----Should be incrementing
Current Active Connections Free For Fast Connection Use: 2
Total Connections Handed-off: 0
Total Connections Handed-off with Compression Policies Disabled: 0
Total Connections Handed-off to SSL: 0
Total Connection Hand-off Failures: 0
Total Fast Connection Successes: 3617244 <-----Should be incrementing
Total Fast Connection Failures: 0
Maximum Fast Connections on a Single Connection: 100
Total CONNECT Requests with Incomplete Message: 0
Percentage of Connection Time Saved: 37
Total Round Trip Time For All Connections (ms): 4922767377
Total Fast Connections Initiated by Peer: 0
Total SYN Timeouts: 0
Total Time for Metadata Cache Miss (ms): 2 <-----Output from here is in 4.2.1
RTT saved by Redirect Metadata Cache (ms): 5988 <-----Should be incrementing
RTT saved by Authorization Redirect Metadata Cache (ms): 345 <-----Should be incrementing
RTT saved by Content Refresh Check Metadata Cache (ms): 44987 <-----Should be incrementing
Total Time Saved by Fast Connection Use (ms): 456
Total Locally Served Redirect Responses: 453 <-----Should be incrementing
Total Locally Served Unauthorized Responses: 56 <-----Should be incrementing
Total Locally Served Conditional Responses: 4932 <-----Should be incrementing
Total Remotely Served Redirect Responses: 0
Total Remotely Served Unauthorized Responses: 0
Total Remotely Served Conditional Responses: 1
Total Requests with URL Longer than 255 Characters: 0
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Total Requests with HTTP Pipelining: 0
Total Transactions Handled: 2 <-----Total number of HTTP transac
Total Server Compression Suppression: 1 <-----Total number of Accept-Encod
Total Requests Requiring Server Content-Revalidation: 0
Total Responses not to be Cached: 0
Total Connections Expecting Authentication: 0
Total Connections with Unsupported HTTP Requests: 0
Total Connections with Unsupported HTTP Responses: 0
Total Hints Sent to DRE Layer to Flush Data: 2
Total Hints Sent to DRE Layer to Skip LZ: 0
Total Hints Sent to DRE Layer to Skip Header Information: 1
If the Total Time Saved counter in the output above is not incrementing or is quite small, it indicates that the HTTP AO is not
providing much benefit. If the Total Time Saved by one of the three metadata caches is not incrementing or is quite small, it
indicates that the corresponding metadata cache is not providing much benefit.
The Total Server Compression Suppression counter indicates how many times the Accept-Encoding header has been removed, in
an attempt to provide a better compression by the WAE device. The Total Hints Sent to DRE Layer counters indicate how many
times each of the DRE hints (Flush Data, Skip LZ, Skip Header) has been issued to the DRE module, in an attempt to better
compress the data.
To view similar information from the Central Manager in version 4.2.1 and later, choose the WAE device, then choose Monitor >
Acceleration > HTTP Acceleration Report and choose the Details tab to see the following charts:
• HTTP Response Time Savings (fast connection reuse, redirect, conditional, and unauthorized cached)
• HTTP Optimization Count (number of times each of the above optimizations has been applied)
• HTTP Optimization Techniques (for all HTTP optimizations, including metadata caches, connection reuse, DRE hints and
suppress-server-encoding)
To see debugging information on the HTTP header parsing and error conditions, use the show statistics accelerator http debug
command (in 4.3.1 and later) to determine the following:
Use the show statistics connection optimized http command to check that the WAAS device is establishing optimized HTTP
connections. Verify that an "H" appears in the Accel column for HTTP connections, which indicates that the HTTP AO was used,
as follows:
You can check connection statistics for closed connections by using the show statistics connection closed http command.
In the Connection Statistics report, the globe icon in the Applied Policy column shows that the HTTP AO was used for a
connection. (Place your cursor over an icon to see its meaning.)
You can view the HTTP connection statistics by using the show statistics connection optimized http detail command. Look for
the "Fast connections" counter in the output. A positive value for this counter means that the HTTP AO benefits clients by reusing
persistent connections, which reduces latency.
Original Optimized
-------------------- --------------------
Bytes Read: 266 139160
Bytes Written: 82686 128
. . .
HTTP : 1496
Use the show statistics accelerator http https command to see the following statistics:
• How much time is being saved by the HTTP AO for HTTPS traffic. You can see the overall Time Saved by the entire
HTTPS metadata cache or the Time Saved by each of the three metadata caches
• Number of cache hits/misses for the metadata caches
• Number of times suppress server encoding is applied to HTTPS requests
• Number of times DRE hints are provided based on the content of the HTTPS headers
• Number of HTTPS transactions (request+response) processed
• Number of errors in the HTTPS header processing
• Number of cache revalidations
HTTPS Statistics
-----------------
Total Optimized HTTPS Connections: 10 <-----Should be incrementing
Total Handled HTTPS Connections: 10 <-----Should be incrementing
Total Active HTTPS Connections: 2
Total Proxy-Connect HTTPS Connections: 0
Total Proxy-Connect HTTPS Insert Failures: 0
RTT saved by HTTPS Content Refresh Check Metadata Cache - (ms): 44 <-----Should be incrementing
RTT saved by HTTPS Redirect Metadata Cache - (ms): 10 <-----Should be incrementing
RTT saved by HTTPS Authorization Required Metadata Cache - (ms): 5 <-----Should be incrementing
Total Locally Served HTTPS Conditional Responses: 44 <-----Should be incrementing
Total Locally Served HTTPS Redirect Responses: 10 <-----Should be incrementing
Total Locally Served HTTPS Unauthorized Responses: 5 <-----Should be incrementing
Total Remotely Served HTTPS Conditional Responses: 32
Total Remotely Served HTTPS Redirect Responses: 2
Total Remotely Served HTTPS Unauthorized Responses: 1
Total Hints Sent to DRE Layer to Skip Header Information - HTTPS: 121
Total Hints Sent to DRE Layer to Flush Data - HTTPS: 121
If the Total Time Saved counter in the output above is not incrementing or is quite small, it indicates that the HTTP AO is not
providing much benefit to the HTTPS traffic. If the Total Time Saved by one of the three metadata caches is not incrementing or
is quite small, it indicates that the corresponding metadata cache is not providing much benefit.
The Total Server Compression Suppression counter indicates how many times the Accept-Encoding header has been removed
from HTTPS requests, in an attempt to provide a better compression by the WAE device. The Total Hints Sent to DRE Layer
counters indicate how many times each of the DRE hints (Flush Data, Skip LZ, Skip Header) has been issued to the DRE module,
in an attempt to better compress the data.
To view similar information from the Central Manager in version 4.3.1 and later, choose the WAE device, then choose Monitor >
Acceleration > HTTPS Acceleration Report and choose the Details tab to see the following charts:
To see debugging information on the HTTPS header parsing and error conditions, use the show statistics accelerator http debug
command to determine the following:
Use the show statistics connection optimized http command to check that the WAAS device is establishing optimized HTTPS
connections. Verify that both an "H" and an "S" appear in the Accel column for HTTPS connections, which indicates that both the
HTTP and SSL AOs were used, as follows:
You can check connection statistics for closed connections by using the show statistics connection closed http or show statistics
connection closed ssl commands.
To view similar information from the Central Manager, choose the WAE device, then choose Monitor > Optimization >
Connections Statistics.
In the Connection Statistics report, the globe icon in the Applied Policy column shows that the HTTP AO was used for a
connection and the lock icon indicates that the SSL AO was applied. (Place your cursor over an icon to see its meaning.)
You can view the HTTPS connection statistics by using the show statistics connection optimized http detail and show statistics
connection optimized ssl detail commands.
Original Optimized
-------------------- --------------------
Bytes Read: 5162 21874
Bytes Written: 1977819 5108
HTTP : 34
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
SSL : 34
Hostname in HTTP CONNECT: <------ the last three counters apply only to
IP Address in HTTP CONNECT: Proxy Connect type of HTTPS connections
TCP Port in HTTP CONNECT:
Redirect Cache
Active entries: 1, Max Entries: 1500
URL: www.abcnews.com/, Expiration (sec): 3206
Conditional Cache
Active entries: 6, Max Entries: 10500
URL: www.cisco.com/web/fw/i/quicklinks-rnd-corners.gif, Expiration (sec): 3594
URL: www.cisco.com/web/fw/i/hp-sprites.gif, Expiration (sec): 3594
URL: www.cisco.com/en/US/home/images/ba-actsGreen-logo.jpg, Expiration (sec): 3594
URL: www.cisco.com/en/US/home/images/fp-eos3.jpg, Expiration (sec): 3594
URL: www.cisco.com/en/US/home/images/fp-AP541n.jpg, Expiration (sec): 3594
URL: www.cisco.com/web/fw/c/home.min.css, Expiration (sec): 3592
Unauthorized Cache
Active entries: 1, Max Entries: 3000
URL: l.yimg.com/index.html, Expiration (sec): 86393
You can clear the content of the three caches with the clear cache http-metadatacache all command.
If you want to clear the content of each cache separately, you can use the following commands:
If you want to specify a URL to be deleted you can use the following command:
You can clear the content of the three caches with the clear cache http-metadatacache https command.
If you want to clear the content of each cache separately, you can use the following commands:
Understand that disabling the cache control checks might increase the benefits of the metadata-cache, because some browsers or
web servers might have a default option to include one cache control header in all responses in order to force revalidation of the
object through the original server. This would make the metadata cache ineffective for 304 responses.
The option can be independently controlled for HTTP/S requests (cache lookups) and responses (cache insertions).
To disable cache control checks on HTTP/S 304 requests, use the following command:
This command forces the metadatacache to disregard all Cache-Control directives in HTTP/S 304 requests. (The default [no] form
of this command forces the metadatacache to honor all Cache-Control directives in HTTP/S 304 requests.)
To disable cache control checks on HTTP/S 304 responses, use the following command:
This command forces the metadatacache to disregard all Cache-Control directives in HTTP/S 304 responses. (The default [no]
form of this command forces the metadatacache to honor all Cache-Control directives in HTTP/S 304 responses.)
The metadata cache honors Cache-Control headers for 301 and 401 responses. If the response has any of the Cache-Control
headers (no-cache, no-store, private, must-revalidate, proxy-revalidate, max-age=0, Pragma: no-cache), it is not cached.
• Non-RFC complaint requests and responses: malformed/invalid headers, repeated headers, missing headers, unexpected
body, unexpected chunked encoding
• URL size is more than 255 characters
• HTTP pipelined transactions
• WebDav methods
• HEAD method
• 301/401 responses with cookie headers
• 301 responses with a total header length of more than 768 bytes
• 401 responses with a total header length of more than 384 bytes
• 401 responses with a chunked body
• 401 responses with unsupported authentication method (supported methods include: Basic, NTLM, Negotiate, Kerberos,
Digest, Oauth)
• Partial HTTP header (header split) available for processing
HTTP AO Logging
The following log files are available for troubleshooting HTTP AO issues:
For easier debugging, you should first set up an ACL to restrict packets to one host.
You can view the end of a transaction log file by using the type-tail command as follows:
To set up and enable debug logging of the HTTP AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
The options for HTTP AO debugging (on 4.2.1 and later) are as follows:
You can enable debug logging for HTTP connections and then display the end of the debug error log as follows:
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
HTTP AO Logging 43
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 EPM Accelerator
Troubleshooting
• 2 EPM AO Logging
You can verify the general AO configuration and status with the show accelerator and show license commands, as described in
the Troubleshooting Application Acceleration article. The Enterprise license is required for EPM accelerator operation.
Next, verify the status that is specific to the EPM AO by using the show accelerator epm command, as shown in Figure 1. You
want to see that the EPM AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is
Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Contents 44
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Use the show running-config command to verify that the EPM traffic policy is properly configured. You want to see adaptor
EPM for the applications or UUIDs that are configured to use the EPM AO, as follows:
Use the show policy-engine application dynamic command to verify the dynamic policy engine match conditions as follows:
Use the show statistics connection optimized epm command to check that the WAAS device is establishing optimized EPM
connections. Verify that "TE" or "TDLE" appears in the Accel column for EPM connections, which indicates that the EPM AO
was used, as follows:
D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
You can check connection statistics for closed connections by using the show statistics connection closed epm command.
To view similar information from the Central Manager, choose the WAE device, then choose Monitor > Optimization >
Connections Statistics.
You can view the EPM connection specific statistics by using the show statistics connection optimized epm detail command as
follows:
Original Optimized
-------------------- --------------------
Bytes Read: 5220 5076
Bytes Written: 5076 5220
. . .
EPM AO Logging
The following log files are available for troubleshooting EPM AO issues:
For easier debugging, first set up an ACL to restrict packets to one host.
You can view the end of a transaction log file by using the type-tail command as follows:
To set up and enable debug logging of the EPM AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
You can enable debug logging for connections in the ACL as follows:
EPM AO Logging 47
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
all enable all EPM accelerator debugs
shell enable EPM shell debugs
You can enable debug logging for EPM connections and then display the end of the debug error log as follows:
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 MAPI Accelerator
• 2 Encrypted MAPI Acceleration
♦ 2.1 Summary
♦ 2.2 Feature Information
♦ 2.3 Troubleshooting Methodology
◊ 2.3.1 Step 1 - Verify Encryption Service Identity configuration and key retrieval success
◊ 2.3.2 Step 2 - In 5.0.3 a new diagnostic command was introduced to check some of the required settings.
◊ 2.3.3 Step 3- Manually verify the WAE settings that are not checked by the diagnostic command above.
♦ 2.4 Data Analysis
♦ 2.5 Common Problems
◊ 2.5.1 Problem 1: The Encryption service identity configured on the Core WAE does not have the correct
permissions in AD.
◊ 2.5.2 Resolution 1: Consult the configuration guide and verify the object in AD has the correct
permissions. "Replicating Directory Changes" and "Replicating Directory Changes All" must both be set
to allow.
Contents 48
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
◊ 2.5.3 Problem 2: There is a time skew between the Core WAE and the KDC it attempts to retrieve the
key from
◊ 2.5.4 Resolution 2: Use ntpdate on all WAEs (especially the Core) to sync the clock with the KDC. Then
point to the enterprise NTP server (prefereably the same as the KDC).
◊ 2.5.5 Problem 3: The domain you defined for your Encryption service does not match the domain your
Exchange server is in.
◊ 2.5.6 Resolution 3: If your Core WAE services multiple Exchange servers in different domains you must
configure an Encryption service Identity for each domain the Exchange servers reside in.
◊ 2.5.7 Problem 4: If WANSecure fails your connections can drop to TG
◊ 2.5.8 Resolution 4: Remove peer cert verify configuration from both WAEs and restart the encrytpion
service on the Core WAE(s).
◊ 2.5.9 Problem 5: If NTLM is used by the Outlook client the connection will be pushed down to Generic
AO.
◊ 2.5.10 Resolution 5: The customer must enable / require Kerberos authentication in their Exchange
environment. NTLM is NOT supported (as of 5.1)
• 3 MAPI AO Logging
MAPI Accelerator
The MAPI accelerator optimizes Microsoft Outlook Exchange e-mail traffic. Exchange uses the EMSMDB protocol, which is
layered on MS-RPC, which in turn uses either TCP or HTTP (unsupported) as the low level transport.
The MAPI AO supports Microsoft Outlook 2000 through 2007 clients for both cached and noncached mode traffic. Secure
connections that use message authentication (signing) or encryption are not accelerated by the MAPI AO. Such connections and
connections from older clients are handed off to the generic AO for TFO optimizations. Additionally, Outlook Web Access
(OWA) and Exchange-Exchange connections are not supported.
Note: Microsoft Outlook 2007 has encryption enabled by default. You must disable encryption to benefit from the MAPI
application accelerator. In Outlook, choose Tools > E-mail Accounts, choose View or Change Existing E-mail Accounts, and
then click Next. Choose the Exchange account, and then click Change. Click More Settings, and then click the Security tab.
Uncheck the Encrypt data between Microsoft Office Outlook and Microsoft Exchange Server check box, as shown in Figure
1.
Alternatively, you can disable encryption for all users of an Exchange Server by using a Group Policy.
MAPI Accelerator 49
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
The Outlook client and server interact in a session over a group of TCP connections called an association group. Within an
association group, object accesses can span across any connection and connections are dynamically created and released as
needed. A client can have more than one association group open at the same time to different servers or the same server. (Public
folders are deployed on different servers from the mail store.)
It is essential that all MAPI connections within an association group go through the same pair of WAEs in the branch and data
center. If some connections within an association group do not go through the MAPI AO on these WAEs, the MAPI AO would
not see the transactions performed on those connections and the connections are said to "escape" the association group. For this
reason, the MAPI AO should not be deployed on serially clustered inline WAEs that form a high availability group.
The symptoms of MAPI connections that escape their WAE association group are Outlook error symptoms such as duplicate
messages or Outlook stops responding.
During a TFO overload condition, new connections for an existing association group would be passed through and escape the
MAPI AO, so the MAPI AO reserves a number of connection resources in advance to minimize the impact of an overload
condition. For more details about reserved MAPI connections and their impact on device overload, see the section "MAPI
Application Accelerator Reserved Connections Impact on Overload" in the Troubleshooting Overload Conditions article.
Verify the general AO configuration and status with the show accelerator and show license commands, as described in the
MAPI Accelerator 50
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Troubleshooting Application Acceleration article. The Enterprise license is required for MAPI accelerator operation and the EPM
application accelerator must be enabled.
Next, verify the status specific to the MAPI AO by using the show accelerator mapi command, as shown in Figure 2. You want
to see that the MAPI AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is
Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Use the show statistics accelerator epm command to verify that the EPM AO is functional. Check that the Total Handled
Connections, Total Requests Successfully Parsed, and Total Responses Successfully Parsed counters increase when a client is
started.
Use the show running-config command to verify that the MAPI and EPM traffic policies are properly configured. You want to
see accelerate mapi for the Email-and-Messaging application action and you want to see the MS-EndPortMapper classifier and
traffic policy defined, as follows:
Use the show policy-engine application dynamic command to verify that dynamic match rules exist, as follows:
• Look for a rule with User ID: EPM and Map Name: uuida4f1db00-ca47-1067-b31f-00dd010662da.
• The Flows field indicates the total number of active connections to the Exchange service.
• For each MAPI client you should see a separate entry with the User ID: MAPI.
Use the show statistics connection optimized mapi command to check that the WAAS device is establishing optimized MAPI
connections. Verify that "M" appears in the Accel column for MAPI connections, which indicates that the MAPI AO was used, as
follows:
Note: In version 4.1.5, the Current Reserved Flows counter was added in the output. This counter refers to the number of reserved
MAPI connection resources on the WAE that are currently unused but set aside for future MAPI connections. For more details
about reserved MAPI connections and their impact on device overload, see the section "MAPI Application Accelerator Reserved
Connections Impact on Overload" in the Troubleshooting Overload Conditions article.
If you observe connections with "TGDL" in the Accel column, these connections were pushed down to the generic AO and
optimized with transport optimizations only. If these are connections that you expected to be handled by the MAPI AO, it may be
because they are encrypted MAPI connections. To check on the number of encrypted MAPI connections that have been requested,
use the show statistics accelerator mapi command as follows:
MAPI:
Global Statistics
-----------------
Time Accelerator was started: Thu Nov 5 19:45:19 2009
Time Statistics were Last Reset/Cleared: Thu Nov 5 19:45:19 2009
Total Handled Connections: 8615
Total Optimized Connections: 8614
Total Connections Handed-off with Compression Policies Unchanged: 0
Total Dropped Connections: 1
Current Active Connections: 20
Current Pending Connections: 0
Maximum Active Connections: 512
Number of Synch Get Buffer Requests: 1052
Minimum Synch Get Buffer Size (bytes): 31680
Maximum Synch Get Buffer Size (bytes): 31680
Average Synch Get Buffer Size (bytes): 31680
Number of Read Stream Requests: 3844
Minimum Read Stream Buffer Size (bytes): 19
Maximum Read Stream Buffer Size (bytes): 31744
Average Read Stream Buffer Size (bytes): 14556
Minimum Accumulated Read Ahead Data Size (bytes): 0
MAPI Accelerator 52
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Maximum Accumulated Read Ahead Data Size (bytes): 1172480
Average Accumulated Read Ahead Data Size (bytes): 594385
Local Response Count: 20827
Average Local Response Time (usec): 250895
Remote Response Count: 70486
Average Remote Response Time (usec): 277036
Current 2000 Accelerated Sessions: 0
Current 2003 Accelerated Sessions: 1
Current 2007 Accelerated Sessions: 0
Secured Connections: 1 <-----Encrypted connections
Lower than 2000 Sessions: 0
Higher than 2007 Sessions: 0
You can find the IP addresses of clients requesting encrypted MAPI connections in the syslog by searching for messages like the
following:
2009 Jan 5 13:11:54 WAE512 mapi_ao: %WAAS-MAPIAO-3-132104: (929480) Encrypted connection. Client ip: 10.36.14.82
You can view the MAPI connection statistics by using the show statistics connection optimized mapi detail command as
follows:
Local and remote response counts and average response times are shown in this output:
. . .
MAPI : 1830
MAPI Accelerator 53
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Minimum Synch Get Buffer Size (bytes): 0
Maximum Synch Get Buffer Size (bytes): 0
Average Synch Get Buffer Size (bytes): 0
Number of Read Stream Requests: 0
Minimum Read Stream Buffer Size (bytes): 0
Maximum Read Stream Buffer Size (bytes): 0
Average Read Stream Buffer Size (bytes): 0
Minimum Accumulated Read Ahead Data Size (bytes): 0
Maximum Accumulated Read Ahead Data Size (bytes): 0
Average Accumulated Read Ahead Data Size (bytes): 0
Local Response Count: 0 <----------
Average Local Response Time (usec): 0 <----------
Remote Response Count: 19 <----------
Average Remote Response Time (usec): 89005 <----------
. . .
Feature Information
eMAPI will be enabled by default in 5.0.3 and will require the following to successfully accelerate encrypted traffic.
1) CMS secure store must be initialized and open on all Core WAEs
2) The WAEs must be able to resolve the FQDN of the Exchange server(s) and Kerberos KDC (Active Directory Controller)
4) SSL acclerator, WAN Secure, and eMAPI must be enabled on all WAEs in the path from Outlook to Exchange
5) The WAEs in the path must have the correct policy-map configuration
6) The Core WAE(s) must have one or more Encrypted Services Domain Identies configured (User or Machine account)
8) Then with either the Machine or User account use case, those objects in Active directory need to be given specific permissions.
"Replicating Directory Changes" and "Replicating Directory Changes All" must both be set to allow.
The recommended way to do this is via a Universal Security group (e.g. assign the permissions to the group and then add the
WAAS devices and/or usernames specified in the Encryption service to this group). See the attached guide for screenshots of AD
configuration and WAAS CM GUI.
Troubleshooting Methodology
Step 1 - Verify Encryption Service Identity configuration and key retrieval success
While the diagnostics command (step 2 below) verifies the existence of an Encryption service it does not verify whether key
retrieval will be successful. Hence we do not know by just running that diagnostic command if the proper permissions were given
to the object in Active Directory (either Machine or User account).
Summary of what needs to be done to configure and verify Encryption service will succeed key retrieval
User account :
1. create AD user
2. create AD group and set "Replicating Directory Changes" and "Replicating Directory Changes All" to ALLOW
Note that you should use the actual/real exchange server name configured on the server and not a NLB/VIP type FQDN which
might resolves to multiple exchange servers.
Example of success:
Machine account
2.create AD group and set "Replicating Directory Changes" and "Replicating Directory Changes All" to ALLOW
5. Give sometime to get the Group Policy to be applied to the joined machine or force the application of group policy from the
AD. gpupdate /force.
Step 1 - Verify Encryption Service Identity configuration and key retrieval success 55
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Note that you should use the actual/real exchange server name configured on the server and not a NLB/VIP type FQDN which
might resolves to multiple exchange servers.
For more details and screen shots on Encrytpion service and AD config see the attached guide.
Step 2 - In 5.0.3 a new diagnostic command was introduced to check some of the required settings.
1.CLI does various validity checks. It output is summary of ability to accelerate encrypted MAPI traffic as edge or core.
2.Checks the various components? status/config attributes for Encryption Service to work properly.
3.When a configuration issue is found it will output what is missing and the CLI or actions to fix it.
4.It give the summary out as Edge device and Core device. Device which can be both edge and core should have EMAPI
operational for both edge and core.
Step 2 - In 5.0.3 a new diagnostic command was introduced to check some of the required settings. 56
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
---------------------------------
Status: FAILED
Accelerator Config Item Mode Value
----------------------- ---- ------
WanSecure Mode User Not Applicable
>>Mapi wan-secure setting should be auto/always
Verifying NTP State
--------------------
Status: FAILED
>>NTP status should be enabled and configured
Summary [EDGE]:
===============
Device has to be properly configured for one or more components
[CORE:]
Verifying encryption-service State
----------------------------------
Status: FAILED
Service Config State Operational State
----------- ------------ -----------------
Encryption-service Disabled Shutdown
>>Encryption Service should be Enabled
>>Encryption Service status should be in 'Running' state
[CORE:]
Step 2 - In 5.0.3 a new diagnostic command was introduced to check some of the required settings. 57
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Step 3- Manually verify the WAE settings that are not checked by the diagnostic command above.
1) The above command while it chekcs for the existence of NTP configured, it does not actually verify the times are in sync
between the WAE and KDC. It is very important the times are in sync between the Core and KDC for key retrieval to be
successful.
If the manual check reveals they are out of sync a simple way to force the clock of the WAE to be in sync would be the ntpdate
command (ntpdate <KDC ip>). Then point the WAEs to the enterprise NTP server.
2) Verify dnslookup succeeds on all WAEs for the Exchange servers' FQDN and the KDCs' FQDN
3) Verify the class-map and policy-map is configured correctly on all WAEs in the path.
4) Verify the CMS secure store is open and initialized on all WAEs "show cms secure store"
Data Analysis
Besides analyzing the output of the diagnostic command and the manual show commands you may need to review the sysreport.
Specifically you'll want to review the mapiao-errorlog, sr-errorlog (core WAE only), and wsao-errorlog files.
Step 3- Manually verify the WAE settings that are not checked by the diagnostic command above. 58
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
There will be hints in each log depending on the scenario which will lead you to the reason connections drop to Generic AO.
This output is from sr-errorlog and shows validation of Machine Account Encryption Service Identity
Note: This only confirms the Core WAE joined the domain and the machine account exists.
07/03/2012 19:12:07.278(Local)(6249 1.5) NTCE (278902) Adding Identity MacchineAcctWAAS to map active list in SRMa
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE (279018) Adding identity(MacchineAcctWAAS) to Map [SRDiIdMgr.cpp:562
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE (279282) Activate Id: MacchineAcctWAAS [SRMain.cpp:260]
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE (279306) Identity MacchineAcctWAAS found in the Map [SRDiIdMgr.cpp:7
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE (279321) Authentication for ID: MacchineAcctWAAS [SRDiIdMgr.cpp:398]
07/03/2012 19:12:07.330(Local)(6249 1.5) NTCE (330581) Authentication success, tkt validity starttime 1341342727 e
07/03/2012 19:12:07.330(Local)(6249 1.5) NTCE (330599)
ID_TAG :MacchineAcctWAAS
Name : pdi-7541-dc
Domain : PDIDC.CISCO.COM
Realm : PDIDC.CISCO.COM
CLI_GUID :
SITE_GUID :
CONF_GUID :
Status:ENABLED
Black_Listed:NO
AUTH_STATUS: SUCCESS
ACCT_TYPE:Machine [SRIdentityObject.cpp:85]
07/03/2012 19:12:07.331(Local)(6249 1.5) NTCE (331685) DN Info found for domain PDIDC.CISCO.COM [SRIdentityObject.
07/03/2012 19:12:07.347(Local)(6249 1.5) NTCE (347680) Import cred successfull for pn: pdi-7541-dc@PDIDC.CISCO.COM
This output is from the Core sr-errorlog again and shows successful key retrieval from KDC.
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673766) Key Not Found in cache, initiating retrieval for spn:exchan
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673811) Queued InitiateKeyRetrieval task [SRServer.cpp:264]10/23/20
Key retrieval is in Progress [SRServer.cpp:322]
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673818) Initiating key retrieval [SRServer.cpp:271]
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673827) initiating key retrieval in progress [SRDataServer.cpp:441]
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673834) Sending ack for result 2, item name /cfg/gl/sr/sr_get_key/p
[SRDataServer.cpp:444]
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673922) Match found for DN: pdidc.cisco.com is ID:MacchineAcctWAAS
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673937) Identity MacchineAcctWAAS found in the Map [SRDiIdMgr.cpp:7
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673950) DN Info found for domain pdidc.cisco.com [SRIdentityObject.
10/23/2012 15:46:55.674(Local)(3780 0.0) NTCE (674011) DRS_SPN: E3514235-4B06-11D1-AB04-00C04FC2DCD2/e4c83c51-0b59
PDI-7541-DC@PDIDC.CISCO.COM [GssCli.cpp:51]
10/23/2012 15:46:55.674(Local)(3780 0.0) NTCE (674020) CREATED srkr obj(0x50aa00) for spn (exchangeMDB/pdidc-excha
10/23/2012 15:46:55.674(Local)(3780 1.3) NTCE (674421) Import cred successfull for pn: PDI-7541-DC@PDIDC.CISCO.COM
10/23/2012 15:46:55.676(Local)(3780 1.3) NTCE (676280) session(0x50aa00) Complete TGT stage of GSS Successful, Ini
10/23/2012 15:46:55.676(Local)(3780 0.1) NTCE (676415) SRKR: Success in posting connect to service <ip:0e:6e:03:a3
10/23/2012 15:46:55.676(Local)(3780 0.0) NTCE (676607) Connected to server. [IoOperation.cpp:389]
10/23/2012 15:46:55.677(Local)(3780 0.0) NTCE (677736) SRKR: Success in posting connect to service <ip:0e:6e:03:a3
10/23/2012 15:46:55.678(Local)(3780 0.1) NTCE (678001) Connected to server. [IoOperation.cpp:389]
10/23/2012 15:46:55.679(Local)(3780 0.1) NTCE (679500) Cleaning up credential cache for PDI-7541-DC@PDIDC.CISCO.CO
10/23/2012 15:46:55.680(Local)(3780 0.1) NTCE (680011) Parsing DRSBIND Response [AppApiDrsBind.cpp:222]
10/23/2012 15:46:55.680(Local)(3780 0.1) NTCE (680030) DRSBind Success, Status:00000000 [AppApiDrsBind.cpp:359]
10/23/2012 15:46:55.685(Local)(3780 0.1) NTCE (685502) session(0x50aa00) Successful in Key Retrieval from AD for S
[SRKeyRetriever.cpp:269]
10/23/2012 15:46:55.685(Local)(3780 0.1) NTCE (685583) Send Key response to the Client for spn: exchangeMDB/pdidc-
[SRKeyMgr.cpp:296]
10/23/2012 15:46:55.685(Local)(3780 0.1) NTCE (685594) Deleting spn: exchangeMDB/pdidc-exchange1.pdidc.cisco.com e
Data Analysis 59
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
This output is from the mapiao-errorlog file on the Edge WAE for a successful eMAPI connection
'''10/23/2012 17:56:23.080(Local)(8311 0.1) NTCE (80175) (fl=2433) Edge TCP connection initiated (-1409268656), Co
Flavor: 0 [EdgeTcpConnectionDceRpcLayer.cpp:43]
10/23/2012 17:56:23.080(Local)(8311 0.1) NTCE (80199) Edge TCP connection initiated (-1409268656), Conn: [14.110.3
[EdgeTcpConnectionDceRpcLayer.cpp:48]
10/23/2012 17:56:23.108(Local)(8311 0.0) NTCE (108825) (fl=2433) Bind Request from client with AGID 0x0, callId 2,
AuthType: SPNEGO AuthCtxId: 0 WsPlumb:1
[EdgeTcpConnectionDceRpcLayer.cpp:1277]'''
10/23/2012 17:56:23.109(Local)(8311 0.0) NTCE (109935) CheckAndDoAoshReplumbing perform replumbing wsPlumbState 1
10/23/2012 17:56:23.109(Local)(8311 0.0) NTCE (109949) (fl=2433) AOSH Replumbing was performed returned Status 0 [
10/23/2012 17:56:23.109(Local)(8311 0.0) NTCE (109956) CheckAndPlumb WanSecure(14) ret:= [1,0] WsPlumb:4 fd[client
10/23/2012 17:56:23.312(Local)(8311 0.1) NTCE (312687) (fl=2433) Connection multiplexing enabled by server on the
10/23/2012 17:56:23.312(Local)(8311 0.1) NTCE (312700) (fl=2433) Header signing enabled by server on the connectio
10/23/2012 17:56:23.312(Local)(8311 0.1) NTCE (312719) (fl=2433) OnNewConnection - Client IP 14.110.3.117 (0xe6e03
nAssociationGroup=0x11de4,conn_fd=26,
bWasConnectionFromReservedPool=0, bIsNewMapiSession=1 [ConnectionReservationManager.cpp:255]
'''10/23/2012 17:56:23.366(Local)(8311 0.1) NTCE (366789) (fl=2433) Received security context from core with auth
10/23/2012 17:56:23.367(Local)(8311 0.1) NTCE (367157) (fl=2433) Security Layer moved to ESTB state [FlowSecurityL
10/23/2012 17:56:23.368(Local)(8311 0.1) NTCE (368029) (fl=2433) Informational:: Send APC set to WS: asking for Ci
10/23/2012 17:56:23.368(Local)(8311 0.1) NTCE (368041) (fl=2433) Sec-Params [CtxId, AL, AT, ACT, DCT, [Hs, ConnMpl
[FlowIOBuffers.cpp:477]
10/23/2012 17:56:23.369(Local)(8311 0.0) NTCE (369128) (fl=2433) CEdgeTcpConnectionEmsMdbLayer::ConnectRequestComm
Product Minor:0, Build Major:6117,
Build Minor:5001 Client ip 14.110.3.117 Client port 58352 Dest ip 14.110.3.99 Dest port 27744 [EdgeTcpConnectionEm
10/23/2012 17:56:23.868(Local)(8311 0.1) ERRO (868390) (fl=2433) ContextHandle.IsNull() [EdgeTcpConnectionEmsMdbLa
10/23/2012 17:56:23.890(Local)(8311 0.0) NTCE (890891) (fl=2433) CEdgeTcpConnectionEmsMdbLayer::ConnectRequestComm
Product Minor:0, Build Major:6117,
Build Minor:5001 Client ip 14.110.3.117 Client port 58352 Dest ip 14.110.3.99 Dest port 27744 [EdgeTcpConnectionEm
Here is the corresponding Core WAE output from mapiao-errorlog for the same TCP conneciton
'''10/23/2012 17:56:54.092(Local)(6408 0.0) NTCE (92814) (fl=21) Core TCP connection initiated (11892640), Conn: [
lavor: 0 [CoreTcpConnectionDceRpcLayer.cpp:99]
10/23/2012 17:56:54.092(Local)(6408 0.0) NTCE (92832) Core TCP connection initiated (11892640), Conn: [14.110.3.11
[CoreTcpConnectionDceRpcLayer.cpp:104]'''
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175035) SrlibCache Cache eviction starting: static void srlib::CSrl
id*, aosh_work*) [SrlibCache.cpp:453]
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175068) last_cleanup_time (1344411860), evict_in_progress(1) handle
cpp:464]
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175121) SendNextCmd isDuringSend 0, WriteQueue sz 1, isDuringclose
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175132) SendNextCmd: Sending request: exchangeMDB/PDIDC-EXCHANGE1.p
[bClose 0] [SrlibClientTransport.cpp:168]
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185576) OnReadComplete len 4 status 0 isDuringRead 1, isDuringHeade
cpp:127]
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185587) Parse header, msg body len 152 [SrlibTransport.cpp:111]
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185592) ReadNextMsg isDuringRead 0, isDuringHeaderRead 1, isDuringc
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185623) OnReadComplete len 148 status 0 isDuringRead 1, isDuringHea
t.cpp:127]
'''10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185688) Insert new KrbKey: exchangeMDB/PDIDC-EXCHANGE1.pdidc.cis
rlibCache.cpp:735]
'''10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185747) ReadNextMsg isDuringRead 0, isDuringHeaderRead 0, isDuri
'''10/23/2012 17:56:54.261(Local)(6408 0.1) NTCE (261575) (fl=21) Successfully created memory keytab with name: ME
.com0nxrPblND [GssServer.cpp:468]
10/23/2012 17:56:54.261(Local)(6408 0.1) NTCE (261613) (fl=21) Successfully added entry in memory keytab. [GssServ
10/23/2012 17:56:54.261(Local)(6408 0.1) NTCE (261858) (fl=21) Successfully acquired credentials. [GssServer.cpp:1
Data Analysis 60
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Common Problems
Below are some common reasons which results in eMAPI connection Hand-off to Generic AO (TG).
Problem 1: The Encryption service identity configured on the Core WAE does not have the correct
permissions in AD.
09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147570) session(0x517fa0) Failed to Retrieve Key from AD for SPN:ex
'''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147592) Key retrieval failed with Status 16 [SRKeyMgr.cpp:157]
''''''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147623) Identity "WAASMacAct" has been blacklisted [SRDiIdMgr
''''''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147631) Key retrieval failed due to permission issue [SRKeyMg
'''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147636) Identity: WAASMacAct will be black listed. [SRKeyMgr.cpp
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147657) Calling KrbKeyResponse key handler in srlib [SRServer.cpp:
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147722) Queued send reponse buffer to client task [SrlibServerTrans
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147730) KrbKeyResponse, sent to client session object [SrlibServer.
09/25/2012 18:47:54.147(Local)(9063 0.0) NTCE (147733) SendNextCmd isDuringSend 0, WriteQueue size 1 isDuringClose
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147740) Send Key response to the Client
Resolution 1: Consult the configuration guide and verify the object in AD has the correct permissions.
"Replicating Directory Changes" and "Replicating Directory Changes All" must both be set to allow.
http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v511/configuration/guide/policy.html#wp1256547
Problem 2: There is a time skew between the Core WAE and the KDC it attempts to retrieve the key from
Resolution 2: Use ntpdate on all WAEs (especially the Core) to sync the clock with the KDC. Then point to the
enterprise NTP server (prefereably the same as the KDC).
Problem 3: The domain you defined for your Encryption service does not match the domain your Exchange
server is in.
Resolution 3: If your Core WAE services multiple Exchange servers in different domains you must configure
an Encryption service Identity for each domain the Exchange servers reside in.
Note, there is NO support for sub-domain include at this time. So if you have myexchange.sub-domain.domain.com , the
Encryption service Identity must be in sub-domain.domain.com; it CAN NOT be in the parent domain.
eMAPI connections can be handed over to Generic AO because WAN secure plumb failed. WAN Secure plumb failed because
cert verify failed. Peer cert verify will failed because the default self-signed peer cert is being used or the cert has legitimately
failed OCSP check.
peer-cert-verify
exit
WAN Secure:
Resolution 2: Use ntpdate on all WAEs (especially the Core) to sync the clock with the KDC. Then point to the enterprise
62 NTP
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
SSL AO User enabled
Secure store User enabled
Peer SSL version User default
Peer cipher list User default
Peer cert User default
Peer cert verify User enabled
The hint here is the first highlighted line "disconnected more than four consecutive times"
'''10/08/2012 20:02:15.025(Local)(24333 0.0) NTCE (25621) (fl=267542) Client 10.16.1.201 disconnected more than fo
[EdgeTcpConnectionDceRpcLayer.cpp:1443]
'''10/08/2012 20:02:15.025(Local)(24333 0.0) NTCE (25634) (fl=267542) CEdgeIOBuffers:: StartHandOverProcessSingleC
[EdgeIOBuffers.cpp:826]
10/08/2012 20:02:15.025(Local)(24333 0.0) NTCE (25644) (fl=267542) CEdgeIOBuffers::CheckSendHandOverRequestToCoreA
fragment of call id 0, current call id is 2 [EdgeIOBuffers.cpp:324]
10/08/2012 20:02:15.048(Local)(24333 0.1) NTCE (48753) (fl=267542) Connection multiplexing enabled by server on th
10/08/2012 20:02:15.048(Local)(24333 0.1) NTCE (48771) (fl=267542) Header signing enabled by server on the connect
10/08/2012 20:02:15.048(Local)(24333 0.1) NTCE (48779) (fl=267542) CEdgeIOBuffers:: StartHandOverProcessSingleConn
'''10/08/2012 20:04:34.430(Local)(5939 4.0) ERRO (430001) certificate verification failed 'self signed certificate
'''10/08/2012 20:04:34.430(Local)(5939 4.0) ERRO (430047) ssl_read failed: 'SSL_ERROR_SSL' [open_ssl.cpp:1217]
10/08/2012 20:04:34.430(Local)(5939 4.0) ERRO (430055) openssl errors: error:14090086: SSL routines: SSL3_GET_SERV
[open_ssl.cpp:1220]
Resolution 4: Remove peer cert verify configuration from both WAEs and restart the encrytpion service on the
Core WAE(s).
pdi-7541-dc(config-ssl-peering)#no peer-cert-verify
Problem 5: If NTLM is used by the Outlook client the connection will be pushed down to Generic AO.
You will see the following in the mapiao-errorlog on the client side WAE:
Be aware there is a Microsoft tech brief that calls out a fall back to NTLM when a CAS is used.
The scenario where Kerberos does not function is specific to Exchange 2010, and is in the following scenario:
In the scenario above, Kerberos does not work - and clients will fall-back to NTLM by defult. I believe this is due to the fact that
clients have to AUTH to the CAS server vs. the Mailbox server, as they did in previous Exchange releases.
In Exchange 2010 RTM, there is no fix for this! Kerberos in the above scenario will never function pre-Exchange 2010-SP1.
In SP1, Kerberos can be enabled in these environments, but it's a manual process. See the article here:
http://technet.microsoft.com/en-us/library/ff808313.aspx
MAPI AO Logging
• The following log files are available for troubleshooting MAPI AO issues:
• Transaction log files: /local1/logs/tfo/working.log (and /local1/logs/tfo/tfo_log_*.txt)
For easier debugging, you should first set up an ACL to restrict packets to one host.
You can view the end of a transaction log file by using the type-tail command as follows:
To set up and enable debug logging of the MAPI AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
You can enable detailed logging to the disk as follows:
You can enable debug logging for connections in the ACL as follows:
You can enable debug logging for MAPI connections and then display the end of the debug error log as follows:
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
MAPI AO Logging 65
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Contents
• 1 NFS Accelerator
Troubleshooting
• 2 NFS AO Logging
You can verify the general AO configuration and status with the show accelerator and show license commands, as described in
the Troubleshooting Application Acceleration article. The Enterprise license is required for NFS accelerator operation.
Next, verify the status specific to the NFS AO by using the show accelerator nfs command, as shown in Figure 1. You want to
see that the NFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is
Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Use the show running-config command to verify that the NFS traffic policy is properly configured. You want to see accelerate
nfs for the File-System application classifier NFS action and you want to see appropriate match conditions listed for the NFS
classifier, as follows:
Contents 66
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
name File-System classifier NFS action optimize full accelerate nfs <-------------
Use the show statistics connection optimized nfs command to check that the WAAS device is establishing optimized NFS
connections. Verify that "N" appears in the Accel column for NFS connections, which indicates that the NFS AO was used.
Use the show statistics accelerator nfs command to verify the following:
• The NFS traffic is NFSv3. Look at the Total RPC Calls per NFS Version field. The output of that field is an array of 5
values, and you want to see mostly NFSv3 traffic, which is reported in the 4th counter. High numbers in other array
positions signify other NFS versions.
• NFS traffic is not encrypted. Look at the Total RPC Calls per Authentication Flavor field. The output of that field is an
array of 4 values, and you want to see mostly unencrypted traffic, which corresponds to the first 3 counters. A high
number in the last counter signifies encrypted NFS traffic. Also check the Total RPC Calls with Unknown Authentication
Flavor field, where you want to see 0 or a small number because these connections are not optimized.
• The NFS connection is asynchronous. Verify that the Percentage of Requests Served Locally field is nonzero.
NFS:
Global Statistics
-----------------
Time Accelerator was started: Fri Oct 23
16:40:06 2009
Time Statistics were Last Reset/Cleared: Fri Oct 23
16:40:06 2009
Total Handled Connections: 170
Total Optimized Connections: 170
Total Connections Handed-off with Compression Policies Unchanged: 0
Total Dropped Connections: 0
Current Active Connections: 0
Current Pending Connections: 0
Maximum Active Connections: 13
Total RPC Calls per Authentication Flavor: 65
298544 0 0 <----Should see 0 or few in last fi
Total RPC Calls with Unknown Authentication Flavor: 0 <----Should see 0 or few
Total RPC Calls per NFS Version: 0
0 0 298609 0 <----Should see 0 or few in first t
Total RPC Calls with Unknown NFS Version: 0 <----Should see 0 or few
Total Requests: 298609
Total Local Replies: 191713
Percentage of Requests Served Locally: 64 <----Should be nonzero
Percentage of Requests Served Remotely: 36
Average Time to Generate Local READ Reply (ms): 15
Average Time to Generate Local WRITE Reply (ms): 0
Average Time to Generate Local GETATTR Reply (ms): 0
Average Time to Generate Local Reply (ms): 0
Average Time to Receive Remote Reply (ms): 10
Meta-Data Cache Access Count: 206017
Meta-Data Cache Hit Count: 191673
You can view the NFS connection statistics by using the show statistics connection optimized nfs detail command as follows:
NFS : 1916
NFS AO Logging
The following log files are available for troubleshooting NFS AO issues:
For easier debugging, you should first set up an ACL to restrict packets to one host.
You can view the end of a transaction log file by using the type-tail command.
To set up and enable debug logging of the NFS AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
You can enable debug logging for connections in the ACL as follows:
You can enable debug logging for NFS connections and then display the end of the debug error log as follows:
NFS AO Logging 69
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
WAE674# debug accelerator nfs all
WAE674# type-tail errorlog/nfsao-errorlog.current follow
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 SSL Accelerator Overview
• 2 Troubleshooting the SSL AO
♦ 2.1 Troubleshooting HTTP AO to SSL AO Handoff
Connections
♦ 2.2 Troubleshooting Server Certificate Verification
♦ 2.3 Troubleshooting Client Certificate Verification
♦ 2.4 Troubleshooting Peer WAE Certificate Verification
♦ 2.5 Troubleshooting OCSP Revocation Checking
♦ 2.6 Troubleshooting DNS Configuration
♦ 2.7 Troubleshooting HTTP to SSL AO Chaining
♦ 2.8 SSL AO Logging
♦ 2.9 Troubleshooting Certificate Expiry Alarms on NME and
SRE Modules
Contents 70
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
In a WAAS network, the data center WAE acts as a trusted intermediary node for SSL requests by the client. The private key and
server certificate are stored on the data center WAE. The data center WAE participates in the SSL handshake to derive the session
key, which it distributes securely in-band to the branch WAE, allowing the branch WAE to decrypt client traffic, optimize it,
reencrypt it, and send it over the WAN to the data center WAE. The data center WAE maintains a separate SSL session with the
origin server.
• Accelerated Service ? A configuration entity that describes acceleration characteristics to be applied for a SSL server or
set of servers. Specifies the certificate and private key to be used while posing as a trusted intermediary, ciphers to be
used, SSL version allowed, and certificate verification settings.
• Peering Service ? A configuration entity that describes acceleration characteristics to be applied for in-band SSL
connections between branch and data-center WAEs. This service is used for transferring session key information from
data-center to branch WAEs for optimizing SSL connections.
• Central Manager Admin Service ? Not used directly by the SSL accelerator, but to be used by an administrator for the
configuration management of SSL accelerated services. Also used to upload certificates and private keys to be used in
SSL accelerated services.
• Central Manager Management Service ? Not used directly by the SSL accelerator, but used for communication between
application accelerator devices and the Central Manager. This service is used for configuration management, secure store
encryption key retrieval, and device status updates.
The Central Manager secure store is essential for the SSL AO to operate because it stores secure encryption keys for all WAEs.
After each Central Manager reload, the administrator needs to reopen the secure store by providing the passphrase with the cms
secure-store open command. A WAE automatically retrieves its secure store encryption key from the Central Manager whenever
the WAE reboots, so no action is required on the WAE after a reload.
If clients are using an HTTP proxy solution, the initial connection is handled by the HTTP AO, which recognizes it as an SSL
tunnel request to port 443. The HTTP AO looks for a matching SSL accelerated service defined on the data center WAE and when
it finds a match, hands off the connection to the SSL AO. However, the traffic that the HTTP AO hands off to the SSL AO for an
HTTPS proxy gets reported as part of the web application statistics, not in the SSL application. If the HTTP AO does not find a
match, the connection is optimized as per static HTTPS (SSL) policy configuration.
The SSL AO can use self-signed certificates rather than CA-signed certificates, which can be helpful in deploying proof of
concept (POC) systems and in troubleshooting SSL issues. By using self-signed certificates, you can quickly deploy a WAAS
system without having to import the origin server certificates, and you can eliminate certificates as a potential source of problems.
You can configure a self-signed certificate in the Central Manager when creating an SSL Accelerated Service. However, when
you use a self-signed certificate, the client browser will display a security alert that the certificate is untrusted (because it is not
signed by a well-known CA). To avoid this security warning, install the certificate in the Trusted Root Certification Authorities
store on the client browser. (On Internet Explorer, on the security warning, click View Certificate, then on the Certificate dialog
click Install Certificate and complete the Certificate Import Wizard.)
Configuring the SSL Management Services is optional, and allows you to change the SSL version and cipher list used for Central
Manager communications to WAEs and to the browser (for administrative access). If you configure ciphers that are not supported
by your browser, you will lose the connection to the Central Manager. In this case, use the crypto ssl management-service
configuration command from the CLI to set the SSL management service settings back to the default.
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Next, verify the status that is specific to the SSL AO on both the data center and branch WAEs by using the show accelerator ssl
command, as shown in Figure 1. You want to see that the SSL AO is Enabled, Running, and Registered, and that the connection
limit is displayed. If the Config State is Enabled but the Operational State is Shutdown, it indicates a licensing problem. If the
Operational State is Disabled, it may be because the WAE cannot retrieve the SSL keys from the Central Manager secure store,
either because the secure store is not open or the Central Manager is unreachable. Use the show cms info and ping commands to
confirm that the Central Manager is reachable.
If you see an Operational State of Gen Crypto Params, wait until the status becomes Running, which may take a few minutes
following a reboot. If you see a state of Retrieving Keys from CM for more than a few minutes, it could indicate that the CMS
service on the Central Manager is not running, that there is no network connectivity to the Central Manager, that the WAAS
versions on the WAE and Central Manager are incompatible, or that the Central Manager secure store is not open.
You can verify that the Central Manager secure store is initialized and open by using the show cms secure-store command as
follows:
If the secure store is not initialized or open, you will see critical alarms such as mstore_key_failure and secure-store. You can
open the secure store with the cms secure-store open command or from the Central Manager, choose Admin > Secure Store.
Tip: Document the secure store password to avoid having to reset the secure store if you forget the password.
If there is a problem with the disk encryption on a WAE, that can also prevent the SSL AO from operating. Use the show disk
details command to verify that disk encryption is enabled and check if the CONTENT and SPOOL partitions are mounted. If
these partitions are mounted, it indicates that the disk encryption keys were successfully retrieved from the Central Manager and
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
encrypted data can be written and read from the disks. If the show disk details command shows "System is initializing," that
indicates the encryption keys have not yet been retrieved from the Central Manager and the disks have not yet been mounted. The
WAE will not provide acceleration services in this state. If the WAE is unable to retrieve disk encryption keys from the Central
Manager, it will raise an alarm.
You can verify that the SSL accelerated service is configured and its status is "Enabled" on the data center WAE (in the Central
Manager, choose the device, then choose Configure > Acceleration > SSL Accelerated Services ). A configured and enabled
accelerated service may be rendered inactive by the SSL accelerator due to the following conditions:
• The certificate configured in the accelerated service been deleted from the WAE. Use the show running-config command
to determine the certificate being used in the accelerated service, then use the show crypto certificates and show crypto
certificate-details commands to confirm that the certificate is present secure store. If the certificate is missing, reimport
the certificate.
• The accelerated service certificate has expired. Use the show crypto certificates and show crypto certificate-details
commands to check the certificate expiry date.
• The accelerated service certificate has a valid date starting in the future. Use the show crypto certificates and show
crypto certificate-details commands and check the validity section of the command output. Also, ensure that the WAE
clock and timezone information is accurate.
You can verify that SSL connections have the correct policy applied, that is, they have full optimization with SSL acceleration, as
shown in Figure 2. In the Central Manager, choose the WAE device, then choose Monitor > Optimization > Connections
Statistics.
Use the show running-config command to verify that the HTTPS traffic policy is properly configured. You want to see optimize
DRE no compression none for the SSL application action and you want to see appropriate match conditions listed for the HTTPS
classifier, as follows:
An active accelerated service inserts dynamic policies corresponding to the server IP:port, server name:port, or server domain:port
configured within the accelerated service. These policies can be inspected using the show policy-engine application dynamic
command. The Dst field in each displayed policy indicates the server IP and port matching the accelerated service. For the
wildcard domain (for example, server-domain *.webex.com port 443), the Dst field will be 'Any:443'. For the server-name
configuration, forward DNS lookup is performed when the accelerated service is activated and all the IP addresses returned in the
DNS response will be inserted in the policy engine. This command is useful to catch situations where an accelerated service is
marked "inservice" but the accelerated service is rendered inactive because of some other error. For example, all accelerated
services are dependent on the peering service, and if the peering service is inactive because of a missing/deleted certificate, then
an accelerated service will also be marked as inactive although it appears to be "inservice" in the show running-config output. You
can verify that the SSL dynamic policy is active on the data center WAE by using the show policy-engine application dynamic
command. You can verify the peering service status by using the show crypto ssl services host-service peering command.
An SSL AO accelerated service configuration can have four types of server entries:
Once the connection is received by the SSL AO, it decides which accelerated service should be used for optimization. The static
IP configuration is given the highest preference, followed by server name, server domain, and then the server ip any. If none of the
configured and activated accelerated services match with the server IP for the connection, the connection is pushed down to the
generic AO. The cookie inserted into the policy engine by the SSL AO is used to determine which accelerated service and what
type of server entry is matched for a particular connection. This policy engine cookie is a 32-bit number and is meaningful only to
the SSL AO. The higher bits are used to indicate different server entry types and the lower bits indicate the accelerated service
index, as follows:
Server Entry
Cookie Value Comments
Type
Server IP
0x8xxxxxxx Static IP address configuration
address
Server Data center WAE performs a forward DNS lookup for the hostname and it adds the IP addresses
0x4xxxxxxx
hostname that are returned into the dynamic policy configuration. Refreshed every 10 minutes by default.
Data center WAE performs a reverse DNS lookup on the destination host IP address to determine
Server
0x2FFFFFFF if it matches with the domain. If it matches, then SSL traffic is accelerated, and if it does not
domain name
match, the traffic is handled according to the static HTTPS policy.
0x1xxxxxxx Server Any All SSL connections are accelerated using this accelerated service configuration
This configuration allows easy deployment for optimization of enterprise SSL applications. It is adaptable to DNS configuration
changes and reduces IT administrative tasks.
This configuration allows WAAS devices to configure a single wildcard domain that avoids the need to know IP addresses for all
the servers. The data center WAE uses reverse DNS (rDNS) to match traffic belonging to the configured domain. Configuring a
wildcard domain avoids configuring multiple IP addresses, making the solution scalable and applicable for SaaS architecture.
This configuration provides a catch-all mechanism. When an accelerated service with server-ip any port 443 is made active, it
allows all connections on port 443 to be optimized by the SSL AO. This configuration can be used during POCs to optimize all
traffic on a particular port.
You can verify the ciphers being used with the show statistics crypto ssl ciphers commands, as shown in Figure 3.
You can verify that these ciphers match those configured on the origin server. Note: Ciphers that include DHE are not supported
by Microsoft IIS servers.
On an Apache server, you can verify the SSL version and cipher details in the httpd.conf file. These fields may also be in a
separate file (sslmod.conf) referenced from httpd.conf. Look for the SSLProtocol and SSLCipherSuite fields as follows:
To verify the certificate issuer on an Apache server, use the openssl command to read the certificate as follows:
In the browser, you can view a certificate and its details to determine the certificate chain, version, encryption key type, issuer
common name (CN), and subject/site CN. In Internet Explorer, click the padlock icon, click View Certificate, and then look at the
Details and Certification Path tabs for this information.
Most browsers require that client certificates be in the PKCS12 format rather than the X509 PEM format. To export the X509
PEM format to PKCS12 format, use the openssl command as follows on an Apache server:
If the private keys are encrypted, the passphrase is required for export. The export password is used again for importing
credentials to the WAAS device.
Use the show statistics accelerator ssl command to see the SSL AO statistics.
Global Statistics
-----------------
Time Accelerator was started: Mon Nov 10 15:28:47 2008
Time Statistics were Last Reset/Cleared: Mon Nov 10 15:28:47 2008
Total Handled Connections: 17 <----------------
Total Optimized Connections: 17 <----------------
Total Connections Handed-off with Compression Policies Unchanged: 0 <----------------
Total Dropped Connections: 0 <----------------
Current Active Connections: 0
Current Pending Connections: 0
Maximum Active Connections: 3
Total LAN Bytes Read: 25277124 <----------------
Total Reads on LAN: 5798 <----------------
Total LAN Bytes Written: 6398 <----------------
Total Writes on LAN: 51 <----------------
Total WAN Bytes Read: 43989 <----------------
Total Reads on WAN: 2533 <----------------
Total WAN Bytes Written: 10829055 <----------------
Total Writes on WAN: 3072 <----------------
. . .
Failed sessions and certificate verifications statistics can be useful for troubleshooting and are more easily retrieved by using the
following filter on the show statistics accelerator ssl command:
DNS related statistics can be useful for troubleshooting server name and wildcard domain configuration. To retrieve these
statistics use the show statistics accelerator ssl command, as follows:
SSL rehandshake related statistics can be useful for troubleshooting and can be retrieved using the following filter on the show
statistics accelerator ssl command:
Use the show statistics connection optimized ssl command to check that the WAAS device is establishing optimized SSL
connections. Verify that "TDLS" appears in the Accel column for a connection. "S" indicates that the SSL AO was used as
follows:
D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
You can check connection statistics for closed connections by using the show statistics connection closed ssl command.
If the connections are not getting optimized, check if WCCP/PBR is properly configured and working, and check for asymmetric
routing.
You can view the SSL connection statistics by using the show statistics connection optimized ssl detail command, where you
will see the dynamic policy that results from the configured SSL accelerated service. Note: The configured policy is TFO
optimization only, but full optimization is applied as a result of the configured SSL service.
Original Optimized
-------------------- --------------------
Bytes Read: 1318 584
Bytes Written: 208 1950
. . .
Later in this output, extended SSL session level details are shown as follows:
. . .
SSL : 1633
Check the following things when troubleshooting issues with handed-off connections:
• Check the output of the show statistics accelerator http command to confirm that a connection was handled by the
HTTP AO and then handed off to the SSL AO. Look at the Total Handled Connections and Total Connections Handed-off
to SSL counters. If there are any issues, verify the following:
♦ The HTTP AO is enabled and in the running state on the peer WAEs.
♦ The SSL accelerated service is configured with the port used by the client in the CONNECT URL (or implied port
443 if HTTPS is being used). Often the proxy port is different from the CONNECT URL port and this proxy port
should not be configured in the SSL accelerated service. However, the proxy port should be included in the traffic
classifier that is mapped to the HTTP AO.
• Check the output of the show statistics accelerator http command to confirm that this connection was handled and
optimized by the SSL AO. Look at the Total Handled Connections and Total Optimized Connections counters. If the
statistics counters are not correct, perform basic SSL troubleshooting as discussed in the previous section.
• On the data center WAE, verify that the show statistics connection optimized detail command output shows the actual
SSL server?s hostname, IP address, and TCP port. If these fields are not set correctly, check the following:
♦ Verify that the client browser proxy settings are correct.
♦ Verify that the DNS server is configured on the data center WAE and is reachable. You can configure a DNS
server on the WAE with the ip name-server A.B.C.D command.
1. Inspect the server certificate and retrieve the Issuer name. This Issuer name within the server certificate must match the subject
name within the matching CA certificate. If you have PEM encoded certificates, you can use the following openssl command on a
server with openssl installed:
2. Ensure that the matching crypto pki ca configuration exists on the data center WAE by using show running-config command.
For a CA certificate to be used by the WAE in the verification process, a crypto pki ca configuration item is required for each CA
certificate imported. For example, if a CA certificate company1.ca is imported, then the following configuration must be made on
the data center WAE:
Note: If a CA certificate is imported using the Central Manager GUI, the Central Manager automatically adds the above crypto
pki ca configuration to include the imported CA certificate. However, if the CA certificate is imported via the CLI, then you will
need to manually add the above configuration.
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
3. If the certificate being verified includes a certificate chain, then ensure that the certificate chain is coherent, and the topmost
issuer?s CA certificate is imported on the WAE. Use the openssl verify command to verify the certificate separately first.
4. If verification still fails, then examine the SSL accelerator debug log. Use the following commands to enable debug logging:
wae# config
wae(config)# logging disk priority debug
wae(config)# logging disk enable
wae(config)# exit
wae# undebug all
wae# debug accelerator ssl verify
wae# debug tfo connection all
5. Initiate a test connection and then examine the /local/local1/errorlog/sslao-errorlog.current log file. This file should indicate the
issuer name that was included in the server certificate. Ensure that this issuer name exactly matches the subject name of the CA
certificate.
If there are any other internal errors in the logs, it may be helpful to enable additional debug options.
6. Even if the Issuer name and Subject names match, the CA certificate may not be the correct one. In such cases, if the server
certificate is issued by a well-known CA, then a browser can be used to directly (without WAAS) reach the server. When the
browser sets up the connection, the certificate can be examined by clicking the Lock icon that appears on the bottom right of the
browser window or within the browser?s address bar. The certificate details may indicate the appropriate CA certificate matching
this server certificate. Check the Serial Number field within the CA certificate. This serial number should match the serial number
of the certificate that is being imported on the data center WAE.
7. If you have OCSP revocation checking enabled, disable it and check that certificate verification by itself works. For help
troubleshooting OCSP settings see the "Troubleshooting OCSP Revocation Checking" section.
If client certificate verification on the data center WAE is not working, it is likely because the CA certificate matching the client
certificate is not imported on the data center WAE. See the "Troubleshooting Server Certificate Verification" section for
instructions how to check if you have the correct CA certificate imported on the WAE.
1. Verify that the certificate being verified is a CA signed certificate. A self signed certificate by one WAE is not verifiable by
another WAE. WAEs by default are loaded with self signed certificates. A self signed certificate must be configured using the
crypto ssl services global-settings machine-cert-key command.
2. Verify that the correct CA certificate is loaded on the device that is verifying the certificate. For example, if peer-cert-verify is
configured on the data center WAE, then it is essential for the branch WAE certificate to be CA-signed and the same signing
CA?s certificate should be imported on the data center WAE. Do not forget to create a CA using the crypto pki ca command to
use the imported certificate, if you are importing the certificate manually through the CLI. When imported by the Central Manager
GUI, the Central Manager automatically creates a matching crypto pki ca configuration.
3. If verification of the peer WAE still fails, check the debug logs as described in the "SSL AO Logging" section.
1. Ensure that the OCSP responder service is running on the responder server.
2. Ensure good connectivity between the WAE and the responder. Use the ping and telnet commands (to the appropriate
port) from the WAE to check.
3. Confirm that the certificate being validated is indeed valid. The expiry date and correct responder URL are typically areas
where there are issues.
4. Verify that the certificate for OCSP responses is imported on the WAE. Responses from an OCSP responder are also
signed and the CA certificate matching the OCSP responses must reside on the WAE.
5. Check the show statistics accelerator ssl command output to check for OCSP statistics and check the counters
corresponding to OCSP failures.
6. If the OCSP HTTP connection is going through an HTTP proxy, try disabling the proxy to see if it helps. If it does help,
then check that the proxy configuration is not causing the connection failure. If the proxy configuration is fine, then there
may be some HTTP header peculiarity which may be causing some incompatibility with the proxy. Capture a packet trace
for further investigation.
7. If all else fails, you may have to capture a packet trace of the outgoing OCSP request for further debugging. You can use
the tcpdump or tethereal commands as described in the section "Capturing and Analyzing Packets" in the Preliminary
WAAS Troubleshooting article.
The URL used by the data center WAE to reach an OCSP responder is derived in one of two ways:
• The static OCSP URL configured by the crypto pki global-settings configuration command
• The OCSP URL specified in the certificate being checked
If the URL is derived from the certificate being checked, then it is essential to ensure that the URL is reachable. Enable the SSL
accelerator OCSP debug logs to determine the URL and then check for connectivity to the responder. See the next section for
details on using debug logs.
1. Ensure that the DNS server configured on the WAE is reachable and can resolve names. Use the following command to check
the configured DNS server:
Try to perform DNS or reverse DNS lookup on the WAE using the following commands:
This response indicates the name cannot be resolved by the configured name servers.
Try ping/traceoute for the configured name servers to check their reachability and the round trip time.
2. If the DNS server is reachable and it can resolve names and still the SSL connections are not getting optimized, make sure the
accelerated service configuring the specified domain or hostname is active and there are no alarms for the SSL AO. Use following
commands:
Major Alarms:
-------------
None
Minor Alarms:
-------------
None
The presence of the "accl_svc_inactive" alarm is an indication that there is some discrepancy in the accelerated service
configuration and there might be one or more accelerated services having overlapping configuration for server entries. Check the
accelerated service configuration and make sure the configuration is correct. Use the following command to verify the
configuration:
To check details about a particular accelerated service use the following command:
One reason that the operational state of the accelerated service might be INACTIVE is a DNS failure. For example, if there is a
server hostname in the accelerated service configuration and the WAE cannot resolve the server IP address, then it cannot
configure the appropriate dynamic policy.
3. If the statistics counter for ?Pipe-through due to non-matching domain name? is increasing, it is an indication that the SSL
connection is for a server that is configured for optimization. Check the policy engine entries using following command:
Check the connection status using the show statistics connection command. The first connection should show an Accelerator of
TSGDL and the subsequent connections, until the lifetime of the TIME_DENY policy entry, should be TDL.
4. If the DNS server is across the WAN with respect to the data center WAE, or if the reverse DNS response time is too long, then
some connections may be dropped. This depends on the client timeout and the rDNS response time. In this case, the counter for
?Number of reverse DNS lookups cancelled? increases and the connection is dropped. This situation is an indication that the DNS
server is not responsive or very slow and/or NSCD on WAAS is not working. The NSCD status can be checked using the show
alarms command. The probability of this happening is very low since in most deployments, the DNS server is expected to be on
the same LAN as the data center WAE.
Chaining allows an AO to insert another AO at any time during the lifetime of a flow and both AOs can apply their AO-specific
optimization independently on the flow. AO chaining is different from the AO handoff feature provided by WAAS in pre-4.3.1
releases because with AO chaining the first AO continues to optimize the flow.
• Byte-0 SSL: The SSL AO receives the connection first and completes the SSL handshake. It parses the initial part of the
payload to check for an HTTP method. If the payload indicates HTTP, it inserts the HTTP AO; if not, it applies the
regular TSDL optimization.
• Proxy CONNECT: The HTTP AO receives the connection first. It identifies the CONNECT header method in the client?s
request and inserts the SSL AO after the proxy confirms with a 200 OK message.
The SSL AO uses a lightweight HTTP parser that detects the following HTTP methods: GET, HEAD, POST, PUT, OPTIONS,
TRACE, COPY, LOCK, POLL, BCOPY, BMOVE, MKCOL, DELETE, SEARCH, UNLOCK, BDELETE, PROPFIND,
BPROPFIND, PROPPATCH, SUBSCRIBE, BPROPPATCH, UNSUBSCRIBE, and X__MS_ENUMATTS. You can use the
debug accelerator ssl parser command to debug issues related to the parser. You can use the show stat accel ssl payload
http/other command to view statistics of traffic classified based on the payload type.
Troubleshooting tips:
1. Make sure the HTTPS feature is enabled in the HTTP AO configuration as this is owned by the HTTP AO. For details,
see the Troubleshooting the HTTP AO article.
2. Check the connection state using the show stat connection command. If correctly optimized, it should show THSDL
indicating TCP, HTTP, SSL and DRE-LZ optimization. If any of these optimizations are missing, debug further on that
optimizer (SSL, HTTP, and so forth). For example, if the connection state shows THDL, it means SSL optimization was
not applied on the connection. Details on debugging issues related to the SSL AO follow.
3. Make sure the SSL AO is enabled and is in the running state (see the section "Troubleshooting the SSL AO").
4. Make sure there are no alarms by using the show alarms command.
5. If SSL traffic is not being optimized, make sure the server IP address, host-name, or domain-name and port number is
added as part of the accelerated service.
6. Make sure the accelerated service is in the ACTIVE state by using the show crypto ssl services accelerated-service
ASVC-name command (see the "Troubleshooting DNS Configuration" section).
7. Make sure the policy engine has an entry for this server and port by using the show policy-engine application dynamic
command.
8. If the destination server is using SSL on a non-default port (the default is 443), make sure this is reflected in the policy
engine configuration. The Central Manager relies on this information for reporting SSL traffic data.
9. Make sure the configured host-name resolves to a valid IP address by using the show crypto ssl services
accelerated-service ASVC-name command. If no IP address is found, check if the name server is configured correctly.
Also check the output of the dnslookup IP-address command.
SSL AO Logging
The following log files are available for troubleshooting SSL AO issues:
SSL AO Logging 86
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
For easier debugging, you should first set up an ACL to restrict packets to one host.
You can view the end of a transaction log file by using the type-tail command as follows:
To set up and enable debug logging of the SSL AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
You can enable debug logging for connections in the ACL as follows:
SSL AO Logging 87
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
synchronization enable synchronization debugs
verify enable certificate verification debugs
waas-to-waas enable waas-to-waas datapath debugs
You can enable debug logging for SSL connections and then display the end of the debug error log as follows:
The clock in all WAAS NME and SRE modules is set to January 1, 2006 during first startup, even though the NME or SRE
module is more recent. This causes the self-signed certificate to expire on January 1, 2011 and the device generates certificate
expiration alarms.
If you are not using the default factory certificate as the global certificate, and instead are using a custom certificate for the SSL
AO, you will not experience this unexpected expiration and you can update the custom certificate whenever it expires. Also, if you
have updated the NME or SME module with a new software image and have synchronized the clock to a more recent date, you
may not experience this issue.
The symptom of certificate expiration is one of the following alarms (shown here in the output of the show alarms command):
Major Alarms:
-------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 cert_near_expiration sslao/SGS/gsetting cert_near_expiration
or
The Central Manager GUI reports the following alarm: "Certificate__waas-self__.p12 is near expiration it is configured as
machine cert in global settings"
You can use one of the following solutions to resolve this problem:
• Update the self-signed factory certificate with a later expiration date. This solution requires a script that you can obtain by
contacting Cisco TAC.
NOTE: This issue is fixed by the resolution of caveat CSCte05426, released in WAAS software versions 4.1.7b, 4.2.3c, and 4.3.3.
The certification expiration date is changed to 2037.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 Video Accelerator
Troubleshooting
• 2 Video AO Logging
You can verify the general AO configuration and status with the show accelerator and show license commands, as described in
the Troubleshooting Application Acceleration article. The Video and Enterprise licenses are required for Video accelerator
operation.
Next, verify the status that is specific to the video AO by using the show accelerator video command, as shown in Figure 1. You
want to see that the video AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State
is Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Contents 89
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Figure 1. Verifying the Video Accelerator Status
Use the show statistics accelerator video command to see the Video AO statistics. The following output shows that one
incoming video stream from the WAN was split to 10 clients, which removed 9 video streams from the WAN.
Video Connections
==================================================================
To examine the reasons why the video AO is not accelerating video connections, use the show statistics accelerator video detail
command. In the example below, the video is not a live broadcast stream but is a video-on-demand (VoD), which is not
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
accelerated.
If videos are not being accelerated as expected, it is often because they are not marked with the live broadcast cache-control
header, x-wms-stream-type="broadcast". VoD streams lack this header. Figure 2 shows where to find the cache-control header in
the Windows Media Server response to the player, using Wireshark.
The URLs for video streams are case sensitive to the video AO, so if a video stream is not being optimized or not playing,
carefully check the URL case and verify that the video is still played. Also verify that the video can be played directly from the
video server, without using WAAS in the network path, to ensure that the video is playable.
Use the show statistics connection optimized video command to check that the WAAS device is establishing optimized video
connections. Verify that "V" appears in the Accel column for video connections, which indicates that the video AO was used as
follows:
You can see in the connections above that DRE and LZ optimizations are not used with video, but the primary server connection
is TFO optimized. All subsequent connections for the same video stream show a reduction of 100% because they are completely
removed from the WAN and instead are split from the primary stream at the branch WAE.
To view similar information from the Central Manager, choose the WAE device, then choose Monitor > Optimization >
Connections Statistics.
The show statistics connection optimized video windows-media command is useful to show the status of all inbound video
streams, including the requesting URL. The show statistics connection optimized video detail command is useful to list all the
inbound and outbound video streams being handled by the video AO.
Video AO Logging
The following log files are available for troubleshooting video AO issues:
You can view the end of a transaction log file by using the type-tail command.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
To set up and enable debug logging of the video AO, enable detailed logging to the disk:
You can enable debug logging for video connections and then display the end of the debug error log as follows:
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 Generic Accelerator
Troubleshooting
• 2 Generic AO Logging
Contents 94
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
• Failure case: An AO determines that it cannot handle the connection after sensing that the data is incomprehensible to it.
For example, if the CIFS AO senses encrypted data or unauthenticated content, it will not be able to handle it and will
push down the connection to the generic AO.
• Multiple protocol handling: For instance, the video AO could accept all connections that are related to multiple protocols
like WMT, RTSP, and so on. However, the video AO currently provides only RTSP optimization, so it will not handle the
connections that are related to other protocols and it will push down these connections to the generic AO.
Common scenarios in which connections are pushed down to the generic AO include the following conditions where there is
connectivity that the AO does not understand or cannot optimize:
• Unauthenticated CIFS
• SMB-signed CIFS
• Encrypted MAPI
• Non-RTSP video
One way to check if the generic AO is being used is to look at statistics from the other AOs. For example, the CIFS AO reports
connections that are pushed down to the generic AO as follows:
CIFS:
Global Statistics
-----------------
Time Accelerator was started: Tue Jul 14
11:55:09 2009
Time Statistics were Last Reset/Cleared: Thu Jul 16
04:16:35 2009
Total Handled Connections: 32
Total Optimized Connections: 1
Total Connections Handed-off with Compression Policies Unchanged: 24 <-----Pushed down to generic
Total Dropped Connections: 0
Current Active Connections: 0
Current Pending Connections: 0
Maximum Active Connections: 4
Number of local reply generating requests: 3388
Number of remote reply generating requests: 415
The Average time to generate a local reply (msec): 25
Average time to receive remote reply (ms): 2147
You can also check connection statistics to see what optimizations are being applied to connections. In the show statistics
connection output, a "G" indicates the connection was handled by the generic AO as follows:
D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
If you take a closer look at the connection above, you will see that CIFS was configured but the generic AO was applied as
follows:
To view similar information from the Central Manager, choose the WAE device, then choose Monitor > Optimization >
Connections Statistics. Connections handled by the generic AO look as follows:
You can use the show statistics accelerator generic detail command to see more details about connections being handled by the
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
generic AO as follows:
Generic:
-------
If you are seeing a large Total number of connections handled, some kind of configuration or communication error might be
causing a large number of connections to be pushed down.
Generic AO Logging
The following log files are available for troubleshooting generic AO issues:
For easier debugging, you should first set up an ACL to restrict packets to one host.
Generic AO Logging 97
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
To set up and enable debug logging of the generic AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
You can enable debug logging for connections in the ACL as follows:
You can enable debug logging for generic AO connections and then display the end of the debug error log as follows:
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Generic AO Logging 98
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Contents
• 1 Overview
• 2 How to Monitor TFO Flows and Overload Conditions
♦ 2.1 Checking the TCP Connection Limit
♦ 2.2 Checking the Optimized TCP Connections
• 3 MAPI Application Accelerator Reserved Connections Impact on
Overload
• 4 Solutions for Overload Conditions
Overview
The Cisco WAAS network would have been designed to optimize a certain number of TCP connections, based on customer
requirements. Depending on the model of the WAE, there could be additional connection limitations for the SSL and CIFS
application accelerators. When either the overall connection limit or a specific application accelerator connection limit is
exceeded, the device is overloaded. In this situation, more traffic is entering the device than it can handle and so traffic may not be
optimized as expected (overloaded traffic is passed through unoptimized).
The device also logs a syslog error message similar to the following: Sysmon: %WAAS-SYSMON-3-445015: Fault detected: The
TFO accelerator is overloaded (connection limit)
You can use various show commands at the CLI to determine the number of allowed and actual connections and gather more
information.
Contents 99
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
The Connection Limit value tells you that this WAAS device can support 12000 TFO optimized connections.
The Effective Limit may be lower than the Connection Limit if the MAPI AO has reserved some connections. The reserved
connections are subtracted from the Connection Limit to get the Effective Limit.
D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
From the first line in the output (Current Active Optimized Flows), you can see that the device currently has five active optimized
flows. From the second counter (Current Active Optimized TCP Plus Flows), you can see that they are all being handled with
TFO/DRE/LZ optimization (TFO Plus means that DRE and/or LZ optimization is being used in addition to TFO). The third
counter (Current Active Optimized TCP Only Flows) shows flows that are optimized by TFO only.
Another useful counter is the Current Active Auto-discovery Flows, which displays flows that have not been fully set up to
become optimized flows or pass-through flows. To become fully set up, the connection must see the SYN, SYN ACK, ACK
handshake, which is useful to note when dealing with an overload condition. The Current Active Pass-Through Flows counter
shows connections that the device has determined to be pass-through or where the device did not see the SYN, SYN ACK, ACK
setup. These flows will not be counted as optimized flows. For pass-through flows, a device should be able to handle up to 10
times the number of optimized flows for which it is rated.
The Current Reserved Flows counter shows the number of connections reserved for the MAPI accelerator. For more details about
reserved MAPI connections and their impact on device overload, see the section MAPI Application Accelerator Reserved
Connections Impact on Overload.
The sum of the following three counters tells you how close the WAE device is to its connection limit:
If this sum is equal to or greater than the connection limit, the device is in an overload condition.
Details about the five optimized flows are displayed in the table below the counters.
Another command that you can use to see the number of TFO flows currently on a device is the show statistics tfo detail
command. Two of the most useful counters in the output are "No. of active connections" and under the Policy Engine Statistics the
"Active connections", as follows:
Auto-Discovery Statistics
-------------------------
Total Connections queued for accept: 22907
Connections queuing failures: 0
Socket pairs queued for accept: 0
Socket pairs queuing failures: 0
In some cases, the two counters will differ and the reason is that the ?No. of active connections? displays all the current flows that
being optimized by TFO, TFO/DRE, TFO/DRE/LZ, and TFO/DRE/LZ and an application accelerator. The ?Active Connections?
under the policy engine statistics includes all the flows in the state above plus the connections that are optimized only by TFO and
an application accelerator. This situation means that a TCP flow has come in and matched an application accelerator classifier but
the SYN, SYN ACK, ACK handshake has not been completed.
In many TFO overload cases, if the problem is still occurring, you can look at these commands and determine if the number of
optimized flows is around the rated number of optimized TCP connections for the hardware. If it is, then you can view the flow
details and see what is using up all the flows to determine if this traffic is legitimate and overloading the device or is a virus,
security scanner, or something else occurring on the network.
The "Connection Limit" counter under the policy engine statistics reports the number of connections rejected and passed through
because the WAE has exceeded its rated number of optimized TCP connections. If this counter is high, it means the WAE is
frequently getting more connections than it can handle.
If the number of optimized connections is not close to the rated number of optimized TCP connections and you are still getting an
overload alarm, then you should look at the Current active auto-discovery flows from the show statistics connection command or
the ?Active Connections? under Policy Engine Statistics from the show statistics tfo detail command. In some cases, the number
of optimized connections can be very low but the Active Connections under the Policy Engine Statistics are roughly equal to the
rated number of optimized flows for the hardware. This situation means that there are many flows that match a classifier but they
are not fully established. When a TCP SYN matches a classifier, it will reserve an optimized connection. This connection will not
appear in the optimized TCP connections count until the TCP handshake is finished and optimization starts. If the device
determines that the flow should not be optimized, it will be removed from the count of active connections under the Policy Engine
Statistics.
To further troubleshoot cases where TFO overload is occurring and the Policy Engine Statistics Active Connections seem to be
using up all the optimized TCP connections on the device, use the show statistics accelerator detail command. In the output of
this command, look at the Active Connections under the Policy Engine Statistics for each application accelerator to determine
which application accelerator is receiving these connections that are not fully established. Next, look at what state these flows
might be in by using the show statistics filtering command, which provides you with the number of filtering tuples on the device,
as follows:
The number of filtering tuples is the number of flows on the device that are optimized, in pass-through, in FIN WAIT state, in
setup state, and so on. Each established flow appears as two tuples, one for each side of the flow, so the number that you see in
this output may be much larger than the number of flows that you are seeing in the other commands.
To get more information on the flows in the filtering list, you can use the show filtering list command as follows:
If the show statistics accelerator all command shows you which application accelerator is using up all the optimized TFO
connections, you can filter on that port or traffic. For example, if you want to filter on port 80 traffic, use the show filtering list |
I :80 command.
Look at the legend in the State column. In the case where the flows are in the SYN state, you may see a lot of flows with a state of
S. If the WAE has sent back the SYN ACK with options set you may see the state SAsO. This indication may help you determine
the state of the flow and from there, you can determine if there is a routing problem, virus, or a problem with the WAE not
releasing connections. You may need traces to determine exactly what is happening to the flows but the commands above should
give you an idea of what to look for.
The MAPI application accelerator reserves TFO connections to ensure that it will have enough connections available to it to
accelerate all current and future connections that the clients will make to the Exchange servers. It is normal for a MAPI client to
make multiple connections. If a client makes the initial connection through the MAPI application accelerator, but the subsequent
connections fail in the MAPI application accelerator, there is a risk that the client's connection might fail.
• Before any client connections begin, it reserves 10 connections for itself, as a buffer for anticipated new connections.
• For each client connection to the server, it reserves three TFO connections for that client-server pair and one of the three
is used as an active connection for this first connection. If the same client makes a second or third connection to the same
server, those are handled out of the reserved connection pool. If a client only ever makes a single connection to the server,
those two reserved connections will be unused and remain in the reserved pool. If the client makes a connection to a
different server, three new connections are again reserved for that client-server pair.
All of these reserved connections are designed to improve performance and to reduce the possibility of a client connection failing
because of the inability to make additional connections through the MAPI application accelerator.
Overload occurs when Current Active Optimized Flows + Current Active Auto-Discovery Flows + Current Reserved Flows is
greater than the device's fixed Connection Limit. In general, new connections would then be passed through. But some new MAPI
connections may still be optimized. When the device is at the overload point, if a client makes an additional request to a MAPI
server it already has connected to, then reserved connections are used. But if there are not enough reserved connections (for
example, if a client makes a fourth connection to the same MAPI server and the WAE is already in overload) then an escaped
connection condition might occur, and this could lead to erroneous behavior such as a client receiving many duplicate copies of
the same single mail message.
If the system did not forward the connection to the MAPI application accelerator, you should see "PT Rjct Resources" or "PT in
progress", depending on whether there is activity on the connection. If the connection was forwarded to the MAPI application
accelerator and then the reservation failed, the connection will be marked with a "G" for the Accelerator, instead of an "M" (in the
show statistics connection optimized mapi command output). For an example of this command, see the article Troubleshooting
the MAPI AO.
If you are experiencing frequent overload conditions, it is important to understand how the Outlook clients are making
connections (how many connections to how many Exchange servers). With Outlook running on a client, hold the Ctrl key while
you right-click on the Outlook icon in the system tray on the task bar. Choose Connection Status to display the list of servers to
which the Outlook client has connected. From that you can see how many connections the client is making and to how many
different Exchange servers. If the client is making connections to several different servers, it would be helpful to investigate ways
to consolidate mail so a user only opens MAPI connections to a single Exchange server, and makes use of multiple connections to
that server.
It is also useful to investigate whether there are any other applications that might be making MAPI connections.
In cases where the connections are legitimate, the WAE deployed in the location is undersized and may need to be upgraded, or an
additional WAE can be deployed to increase scalability within that site.
Guide Contents
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 Troubleshooting WCCP on the Router
♦ 1.1 Troubleshooting WCCP on the Catalyst 6500 Series Switches and the ISR and 3700
Series Routers
♦ 1.2 Troubleshooting WCCP on the ASR 1000 Series Routers
• 2 Troubleshooting WCCP on the WAE
• 3 Troubleshooting Configurable Service IDs and Variable Timeouts in Version 4.4.1
WCCP issues can result from problems with the router (or redirecting device) or from the WAE device. It is necessary to look at
the WCCP configuration both on the router and on the WAE device. First we will look at the WCCP configuration on the router,
then we will check the WCCP configuration on the WAE.
Contents 105
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
• Catalyst 6500 Series Switches and the ISR and 3700 Series Routers
• ASR 1000 Series Routers
Troubleshooting WCCP on the Catalyst 6500 Series Switches and the ISR and 3700
Series Routers
Begin troubleshooting by verifying WCCPv2 interception on the switch or router by using the show ip wccp IOS command as
follows:
Service Identifier: 61
Number of Service Group Clients: 1 <-----Client = WAE
Number of Service Group Routers: 1
Total Packets s/w Redirected: 68755 <-----Increments for software-based redirection
Process: 2 <-----
Fast: 0 <-----
CEF: 68753 <-----
Service mode: Open
Service access-list: -none-
Total Packets Dropped Closed: 0
Redirect access-list: -none-
Total Packets Denied Redirect: 0 <-----Match service group but not redirect list
Total Packets Unassigned: 0
Group access-list: -none-
Total Messages Denied to Group: 0
Total Authentication failures: 0 <-----Packets have incorrect service group password
Total Bypassed Packets Received: 0
--More--
On platforms that use software-based redirection, verify that the Total Packets s/w Redirected counters are incrementing in the
above command output. On platforms that use hardware-based redirection, these counters should not be incrementing much. If
you are seeing these counters increment significantly on hardware-based platforms, WCCP could be misconfigured on the router
(WCCP GRE is processed in software by default), or the router could be falling back to software redirection due to hardware
resources issues such as running out of TCAM resources. More investigation is required if you see these counters incrementing on
a hardware-based platform, which could lead to high CPU usage.
The Total Packets Denied Redirect counter increments for packets that match the service group but do not match the redirect list.
The Total Authentication failures counter increments for packets that are received with the incorrect service group password.
On routers where WCCP redirection is performed in the software, continue by verifying WCCPv2 interception on the router by
using the show ip wccp 61 detail IOS command as follows:
Verify that the WAE state in the service group 61 is Usable. Verify that hash buckets are assigned to the WAE in the Hash
Allotment field. The percentage tells you how many of the total hash buckets are handled by this WAE. The amount of time that
the WAE has been in the service group is reported in the Connect Time field. The hash assignment method should be used with
software-based redirection.
You can determine which WAE in the farm will handle a particular request by using the show ip wccp service hash dst-ip src-ip
dst-port src-port hidden IOS command on the router as follows:
On routers where WCCP redirection is performed in the hardware, continue by verifying WCCPv2 interception on the router by
using the show ip wccp 61 detail IOS command as follows:
You want to see the Mask assignment method for routers that are capable of hardware redirection.
In order to save TCAM resources on the router, consider altering the default WCCP mask to suit your network environment.
Consider these recommendations:
• Use the smallest number of mask bits possible when using WCCP redirect ACL. A smaller number of mask bits when
used in conjunction with Redirect ACL results in lower TCAM utilization. If there are 1-2 WCCP clients in a cluster, use
one bit. If there are 3-4 WCCP clients, use 2 bits. If there are 5-8 WCCP clients, then use 3 bits and so on.
• We do not recommend using the WAAS default mask (0x1741). For data center deployments, the goal is to load balance
the branch sites into the data center rather than clients or hosts. The right mask minimizes data center WAE peering and
Troubleshooting WCCP on the Catalyst 6500 Series Switches and the ISR and 3700Series Routers 107
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
hence scales storage. For example, use 0x100 to 0x7F00 for retail data centers that have /24 branch networks. For large
enterprises with a /16 per business, use 0x10000 to 0x7F0000 to load balance the businesses into the enterprise data
center. In the branch office, the goal is to balance the clients that obtain their IP addresses via DHCP. DHCP generally
issues client IP addresses incrementing from the lowest IP address in the subnet. To best balance DHCP assigned IP
addresses with mask, use 0x1 to 0x7F to only consider the lowest order bits of the client IP address to achieve the best
distribution.
The TCAM resources consumed by a WCCP redirect access-list is a product of the content of that ACL multiplied against the
configured WCCP bit mask. Therefore, there is contention between the number of WCCP buckets (which are created based on the
mask) and the number of entries in the redirect ACL. For example, a mask of 0xF (4 bits) and a 200 line redirect permit ACL may
result in 3200 (2^4 x 200) TCAM entries. Reducing the mask to 0x7 (3 bits) reduces the TCAM usage by 50% (2^3 x 200 =
1600).
Catalyst 6500 series and Cisco 7600 series platforms are capable of handling WCCP redirection in both the software and
hardware. If packets are inadvertently being redirected in software, when you expect hardware redirection, it could result in overly
high router CPU use.
You can inspect the TCAM information to determine if redirection is being handled in the software or the hardware. Use the show
tcam IOS command as follows:
"Punt" matches represent requests not handled in the hardware. This situation could be caused by the following errors:
In the following example, the policy-route entries show that the router is doing full hardware redirection:
The Here I Am (HIA) from the WAE must enter the same interface that the WAE MAC is known through. We recommend that
you use a loopback interface and not a directly connected interface in the WAE router list.
To display route processor WCCP information, use the show platform software wccp rp active commands as follows:
The following example shows additional commands that you can use to examine forwarding processor information:
To display redirected packet statistics for each interface, use the show platform software wccp interface counters command as
follows:
Use the show platform software wccp web-cache counters command to display WCCP cache information as follows:
For more information, see the white paper "Deploying and Troubleshooting Web Cache Control Protocol Version 2 on Cisco ASR
1000 Series Aggregation Services Routers"
Next check the WCCP status by using the show wccp status command. You want to see that WCCP version 2 is enabled and
active as follows:
Look at the WCCP farm information by using the show wccp wide-area-engine command. This command shows the number of
WAEs in the farm, their IP addresses, which one is the lead WAE, routers that can see the WAEs, and other information, as
follows:
Look at the router information by using the show wccp routers command. Verify that there is bidirectional communication with
WCCP-enabled routers and all routers show the same KeyIP and KeyCN (change number), as follows:
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
WAE-612# show wccp routers
In cases where the WAE is not Layer 2-adjacent to the router, or a loopback address is used, either static routes or a default
gateway is required to support WCCP.
To examine the hash bucket distribution in the service group, use the show wccp flows tcp-promiscuous command as follows:
Alternatively, you can use the summary version of the command to see similar information, as well as bypass flow information:
BYP = 0
AWAY = 0
Use the show wccp gre command to display GRE packet statistics as follows:
If WCCP redirection is working, either of the first two counters should be incrementing.
The Transparent non-GRE packets received counter increments for packets that are redirected using the WCCP Layer 2 redirect
forwarding method.
The Transparent non-GRE non-WCCP packets received counter increments for packets that are redirected by a non-WCCP
interception method (such as ACE or PBR).
The Total packets accepted counter indicates packets that are accepted for optimization because auto-discovery found a peer
WAE.
The GRE packets sent to router (not bypass) counter indicates packets that were handled using the WCCP negotiated return egress
method.
The packets sent to another WAE counter indicates that flow protection is occurring when another WAE is added to the service
group and begins handling a bucket assignment that was previously being handled by another WAE.
Verify that the egress methods that are being used are the expected ones by using the show egress-methods command as follows:
TCP Promiscuous 61 :
WCCP negotiated return method : WCCP GRE
TCP Promiscuous 62 :
WCCP negotiated return method : WCCP GRE
• The negotiated return egress method is configured, but WCCP negotiates the Layer 2 return method and only GRE return
is supported by WAAS.
• The generic GRE egress method is configured, but the interception method is Layer 2 and only WCCP GRE is supported
as the interception method when generic GRE egress is configured.
In either of these cases, a minor alarm is raised and is cleared when the mismatch is resolved by changing the egress method or the
WCCP configuration. Until the alarm is cleared, the default IP forwarding egress method is used.
The following example shows the command output when a mismatch exists:
WARNING: WCCP has negotiated WCCP L2 as the intercept method for <-----Warning if mismatch occurs
which generic GRE is not supported as an egress method
in this release. This device uses IP forwarding as the
egress method instead of the configured generic GRE
egress method.
TCP Promiscuous 62 :
WARNING: WCCP has negotiated WCCP L2 as the intercept method for <-----Warning if mismatch occurs
which generic GRE is not supported as an egress method
in this release. This device uses IP forwarding as the
egress method instead of the configured generic GRE
egress method.
For Catalyst 6500 Sup720 or Sup32 routers, we recommend using the generic GRE egress method, which is processed in
hardware. Additionally, we recommend using one multipoint tunnel for ease of configuration, instead of one point-to-point tunnel
for each WAE. For tunnel configuration details, refer to the section Configuring a GRE Tunnel Interface on a Router in the Cisco
To view the GRE tunnel statistics for each intercepting router, use the show statistics generic-gre command as follows:
Failure to ensure that egress packets from a WAE are not reintercepted can lead to a redirection loop. If a WAE detects its own ID
returned in the TCP options field, a redirection loop has occurred and results in the following syslog message:
%WAAS-SYS-3-900000: 137.34.79.11:1192 - 137.34.77.196:139 - opt_syn_rcv: Routing Loop detected - Packet has our ow
You can search the syslog.txt file for instances of this error by using the find command as follows:
This error also shows up in the TFO flow statistics available in the show statistics filtering command as follows:
If you are doing outbound redirection on the router, as traffic leaves the router it will get redirected back to the WAE, which will
reroute the packet out the router, causing a routing loop. If the data center WAE and servers are on different VLANs and the
branch WAE and the clients are on different VLANS, you can avoid a routing loop by using the following router configuration on
the WAE VLAN:
If the WAE shares the same VLAN with its adjacent clients or servers, you can avoid routing loops by using the negotiated return
method, or generic GRE return for platforms where WCCP redirection is performed in the hardware. When using generic GRE
return, the WAE uses a GRE tunnel to return traffic to the router.
All WAEs in a WCCP farm must use the same pair of WCCP service IDs (the default is 61 and 62), and these IDs must match all
routers that are supporting the farm. A WAE with different WCCP service IDs than those configured on the routers is not allowed
to join the farm and the existing "Router Unreachable" alarm is raised. Likewise, all WAEs in a farm must use the same value for
the failure detection timeout. A WAE raises an alarm if you configure it with a mismatching value.
If you see an alarm that a WAE is not able to join a WCCP farm, check that the WCCP service IDs configured on the WAE and
the routers in the farm match. On the WAEs, use the show wccp wide-area-engine command to check the configured service IDs.
On the routers, you can use the show ip wccp IOS command.
Troubleshooting Configurable Service IDs and Variable Timeouts in Version 4.4.1 114
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
To check if the WAE has connectivity to the router, use the show wccp services detail and show wccp router detail commands.
Additionally, you can enable WCCP debug output on the WAE by using the debug ip wccp event or debug ip wccp packet
commands.
If you see a "Router Unusable" minor alarm for a WAE, it could mean that the variable failure detection timeout value set on the
WAE is not supported by the router. Use the show alarm minor detail command to check if the reason for the alarm is "Timer
interval mismatch with router":
Minor Alarms:
-------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 rtr_unusable WCCP/svc051/rtr2.192.9.161
On the router, check if the IOS version supports variable failure detection timeout. If so, you can check the configured setting by
using the show ip wccp xx detail command, where xx is the WCCP service ID. There are three possible results:
• WAE is using default failure detection timeout of 30 seconds and router is configured the same or does not support
variable timeout: The router output shows no details about the timeout setting. This configuration operates fine.
• WAE is using non-default failure detection timeout of 9 or 15 seconds and router does not support variable timeout: State
field shows "NOT Usable" and the WAE cannot use the router. Change the WAE failure detection timeout to the default
value of 30 seconds by using the wccp tcp failure-detection 30 global configuration command.
• WAE is using non-default failure detection timeout of 9 or 15 seconds and router supports variable timeout: Client
timeout field shows the configured failure detection timeout, which matches the WAE. This configuration operates fine.
If the WCCP farm is unstable due to link flapping, it could be because the WCCP failure detection timeout is too low.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 AppNav Troubleshooting
♦ 1.1 In-Path (Inline) Interception
♦ 1.2 Off-Path (WCCP) Interception
◊ 1.2.1 Configuring and Verifying WCCP
Interception on the Router
◊ 1.2.2 Additional Information
♦ 1.3 Network Connectivity Troubleshooting
◊ 1.3.1 Passing Through Specific Traffic
◊ 1.3.2 Disabling an Inline ANC
◊ 1.3.3 Disabling an Off-Path ANC
♦ 1.4 AppNav Cluster Troubleshooting
◊ 1.4.1 AppNav Alarms
◊ 1.4.2 Central Manager Monitoring
◊ 1.4.3 AppNav CLI Commands for Monitoring
Cluster and Device Status
◊ 1.4.4 AppNav CLI Commands for Monitoring
Flow Distribution Statistics
◊ 1.4.5 AppNav CLI Commands for Debugging
Connections
◊ 1.4.6 Connection Tracing
◊ 1.4.7 AppNav Debug Logging
♦ 1.5 AppNav Packet Capture
Contents 116
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
AppNav Troubleshooting
Cisco WAAS AppNav simplifies network integration of WAN optimization and greatly reduces dependency on the intercepting
switch or router by using AppNav Controllers (ANCs) to distribute traffic among WAAS Nodes (WNs) for optimization using a
powerful class and policy mechanism. You can use WAAS nodes (WNs) to optimize traffic based on sites and/or applications.
This article describes how to troubleshoot AppNav.
NOTE: The AppNav feature was introduced in WAAS version 5.0.1. This section is not applicable to earlier WAAS versions.
Note: Bridge interfaces do not block bridge protocol data unit (BPDU) packets and in the case of redundant interfaces that create
loops, one of the interfaces is blocked by the Spanning Tree Protocol.
• Verify correct inline placement of the ANC by checking the network design. If necessary, use basic tools like ping and
traceroute, or Layer 7 tools or applications to confirm that the network traffic path is as expected. Check the physical
cabling of the ANC.
• Verify that the ANC is set to inline interception mode.
• Verify that the bridge-group interface is configured correctly.
The last two steps can be performed either in Central Manager or at the command line, though the Central Manager is the
preferred method and is described first.
In the Central Manager, choose Devices > AppNavController, then choose Configure > Interception > Interception
Configuration. Verify that the Interception Method is set to Inline.
In the same window, verify that a bridge interface is configured. If a bridge interface is needed, click Create Bridge to create it.
You can assign up to two member interfaces to the bridge group. You can use the VLAN Calculator to define the VLAN entries
based on include or exclude operations. Note that the bridge interface is not assigned an IP address.
Use the Alarm panel or the show alarm exec command to check if any bridge related alarms are raised on the device. A
bridge_down alarm indicates that one or more member interfaces in the bridge are down.
wave# config
wave(config)# interception-method inline
You can use the show bridge exec command to verify the bridge interface operational status and see statistics for the bridge.
Interception Statistics:
GigabitEthernet 1/0 GigabitEthernet 1/1
Operation State : Down Down(lsp) <<< Down due to LSP
Input Packets Forwarded/Bridged : 16188 7845
Input Packets Redirected : 5068 0
Input Packets Punted : 1208 605
Input Packets Dropped : 0 0
Output Packets Forwarded/Bridged : 7843 21256
Output Packets Injected : 301 301
Output Packets Dropped : 2 0
In the example above, the Gig 1/0 interface is down and the Gig 1/1 interface is also down due to link state propagation (LSP).
You might also see Down(flow sync), which means that the ANC is joining a cluster and synchronizing flow information with
other ANCs in the cluster. It keeps the interception path (bridge interface) shut for about two minutes until all ANCs are
synchronized so that existing flows can be distributed correctly.
The lower part of the output shows traffic statistics for the member interfaces.
In the interface configuration for an off-path deployment, the interception and distribution roles can share the same interfaces on
the Cisco AppNav Controller Interface Module, but it is not required.
• Verify correct placement of the WCCP routers to ensure they are in the path of traffic going to and from the optimized
hosts. You can use the show run or show wccp commands to verify that these are the same routers that are configured for
WCCP. If necessary, use basic tools like ping and traceroute, or Layer 7 tools or applications to confirm that all traffic
needing optimization passes through the WCCP routers.
• Verify the WCCP configuration on the WAAS ANCs, using either the Central Manager (preferred) or the CLI.
• Verify the WCCP configuration on the redirecting routers, using the router CLI.
To verify the WCCP configuration on the ANCs, in the Central Manager, choose Devices > AppNavController, then choose
Configure > Interception > Interception Configuration.
Check for any WCCP related alarms on the ANCs that are part of the router WCCP farm. On the Central Manager, click the
Alarms panel at the bottom of the screen or use the show alarm command on each device to view alarms. Correct any alarm
conditions by changing the configuration on the ANC or router, as needed.
2. Configure the WCCP router list, which contains the IP addresses of the routers participating in the WCCP farm.
3. Configure the WCCP service ID. A single service ID is preferred for AppNav, though two service IDs are supported.
wave(config-wccp-service)# router-list-num 1
5. Configure the WCCP assignment method (only the mask method is supported on an ANC). If you do not specify the
dst-ip-mask or src-ip-mask options, the default source IP mask is set to f and the destination IP mask is set to 0.
6. Configure the WCCP redirect method (the egress and return methods are set automatically to match the redirect method and are
not configurable for an ANC). You can choose L2 (the default) or GRE. L2 requires that the ANC has a Layer 2 connection with
the router and the router is also configured for Layer 2 redirection.
wave(config-wccp-service)# enable
Verify WCCP interception on each ANC by using the show running-config command. The two examples below show the
running config output for L2 redirect and GRE redirect.
Verify the WCCP status on each ANC by using the show wccp status command.
Verify the routers that have responded to keep-alive messages in the WCCP farm by using the show wccp routers command.
Verify each ANCs' view of the other ANCs in the WCCP farm and the routers reachable by each of them by using the show wccp
clients command.
IP address = 10.10.10.32 Lead WAE = YES Weight = 0 <<< YES indicates ANC is serving as the lead
Routers seeing this Wide Area Engine(2)
192.168.1.1 <<< List of routers seeing this ANC
192.168.1.2
Verify that packets are being received by each ANC from the routers in the farm by using the show statistics wccp command.
Statistics for traffic that is received from, passed through, and sent to each router are shown. Cumulative statistics for all routers in
the farm are shown at the bottom. A similar command is show wccp statistics. Note that "OE" refers to the ANC devices here.
To configure WCCP interception on each router in the WCCP farm, follow these steps.
1. Configure the WCCP service on the router by using the ip wccp router command.
2. Configure WCCP interception on the router LAN and WAN interfaces. You can configure the same service ID on both
interfaces if you are using a single service ID on the ANCs.
3. (Optional) Configure a tunnel interface if you are using generic GRE egress (only if you chose GRE for the ANC WCCP
redirect method).
Verify the WCCP configuration on each router in the farm by using the show wccp command.
Additional Information
• WCCP Network Integration with Cisco Catalyst 6500: Best Practice Recommendations for Successful Deployments
• Cisco Wide Area Application Services Web Cache Communication Protocol Redirection: Cisco Router Platform Support
• Configuring Advanced WCCP Features on Routers, from the Cisco Wide Area Application Services Configuration Guide
• Configuring WCCP on WAEs, from the Cisco Wide Area Application Services Configuration Guide
Before testing Layer 3 connectivity, verify that the AppNav Controller Interface Module is connected to proper switch ports. If the
connected switch supports and has Cisco Discovery Protocol (CDP) enabled, run the command show cdp neighbors detail to
verify proper connectivity to the network switch.
Disabling WAAS may not be applicable in all cases. If some traffic is being optimized and some is not, it may be unacceptable to
disable WAAS, thereby disrupting the traffic that is being optimized successfully. In such a case, the interception ACL or the
AppNav policy can be used to pass through the specific type of traffic that is experiencing problems. For details, see the section
Passing Through Specific Traffic.
To disable WAAS, different steps are performed for inline mode than for off-path mode:
• Inline mode requires putting the interception bridge in the pass-through state. For details, see the section Disabling an
Inline ANC.
• Off-path mode requires disabling the WCCP protocol. For details, see the section Disabling an Off-Path ANC.
In AppNav environments, only the ANCs need to be disabled. WNs need not be disabled, since they do not participate in
interception.
1. Reenable the WAAS functionality on the WAAS ANCs and, if applicable, the WCCP routers.
2. If you have determined that there is a WAAS-related problem, enable each AppNav cluster, and/or ANC individually, to isolate
it as a potential cause of the observed problem.
3. As each ANC is enabled, perform the same basic network connectivity tests as in earlier steps and note whether this specific
ANC seems to be operating correctly. Do not be concerned with individual WNs at this stage. The goal at this stage is to
determine which clusters, and which specific ANCs, are experiencing desired or undesired behavior.
4. As each ANC is enabled and tested, disable it again so that the next one can be enabled. Enabling and testing each ANC in turn
allows you to determine which ones require further troubleshooting.
This troubleshooting technique is most applicable in situations where the WAAS configuration appears to be not only failing to
optimize, but also causing problems with normal network connectivity.
You can pass through specific traffic either by using an interception ACL or by configuring the AppNav policy for pass through.
• Create an ACL that denies the specific traffic to be passed through and permits everything else. In this example, we want
to pass through HTTP traffic (dest port 80). Set the ANC interception access list to the defined ACL. Connections
destined for port 80 are passed through. You can use the show statistics pass-through type appnav command to verify
that pass-through is happening by checking that the PT Intercept ACL counters are incrementing.
anc# config
anc(config)# ip access-list extended pt_http
anc(config-ext-nacl)# deny tcp any any eq 80
anc(config-ext-nacl)# permit ip any any
anc(config-ext-nacl)# exit
anc(config)# interception appnav-controller access-list pt_http
• Configure the ANC policy to pass through traffic matching specific classes.
There are several ways to disable an inline ANC by putting it in pass-through state:
• Set the interception bridge VLAN list to none. In the Central Manager, choose an ANC device, then choose Configure >
Interception > Interception Configuration. Select the bridge interface and click the Edit taskbar icon. Set the VLANs
field to the value "none".
• Disable the service context containing the ANC. In the Central Manager, choose a cluster, then click the AppNav
Controllers tab, select an ANC, and click the Disable taskbar icon.
• Apply an interception ACL with "deny ALL" criteria. This method is preferred. (The first two methods disrupt existing
optimized connections.) Define an ACL with the deny ALL criteria. In the Central Manager, choose an ANC device, then
choose Configure > Interception > Interception Access List, and choose the deny ALL access list in the AppNav
Controller Interception Access List drop-down list.
To disable interception with an ACL from the CLI, use the following commands:
anc# config
anc(config)# ip access-list standard deny
anc(config-std-nacl)# deny any
anc(config-std-nacl)# exit
anc(config)# interception appnav-controller access-list deny
To disable an ANC that is running in off-path mode, disable the WCCP protocol for the ANC. You can do this action on the ANC
or on the redirecting router or both. On the ANC, you can disable or delete the WCCP services, or you can remove the
interception method or change it from WCCP to another method.
To disable WCCP interception, in the Central Manager, choose an ANC device, then choose Configure > Interception >
Interception Configuration. Uncheck the Enable WCCP Service check box or click the Remove Settings taskbar icon to remove
WCCP interception settings completely (they will be lost).
To disable WCCP interception from the CLI, use the following commands:
anc# config
anc(config)# wccp tcp-promiscuous service-pair 61
anc(config-wccp-service)# no enable
In some cases, there may be multiple ANCs receiving redirected traffic from the same router. For convenience, you may choose to
disable WCCP at the router, rather than the ANCs. The advantage is that you can remove multiple ANCs from a WCCP farm in a
single step. The disadvantage is that you cannot do this from the WAAS Central Manager.
RTR1(config)# no ip wccp 61
RTR1(config)# no ip wccp 62 <<< Only needed if you are using two WCCP service IDs
RTR1(config)# ip wccp 61
RTR1(config)# ip wccp 62 <<< Only needed if you are using two WCCP service IDs
At each WCCP router, verify that the ANCs you chose to disable are not showing up as WCCP clients. The following output is
displayed when the WCCP services have been deleted on the router.
• AppNav Alarms
• Central Manager Monitoring
• AppNav CLI Commands for Monitoring Cluster and Device Status
• AppNav CLI Commands for Monitoring Flow Distribution Statistics
• Connection Tracing
• AppNav Debug Logging
AppNav Alarms
The Cluster Membership Manager (CMM) raises the following the alarms due to error conditions:
• Degraded Cluster (Critical)--Partial visibility among ANCs. ANC will pass through new connections.
• Convergence Failed (Critical)--ANC failed to converge on a stable view of ANCs and WNs. ANC will pass through new
connections.
• ANC Join Failed (Critical)--ANC failed to join an existing cluster due to potential degradation of the cluster with the
ANC in it.
• ANC Mixed Farm (Minor)--ANCs in the cluster are running different but compatible versions of the cluster protocol.
• ANC Unreachable (Major)--A configured ANC is unreachable.
• WN Unreachable (Major)--A configured WN is unreachable. This WN is not used for traffic redirection.
• WN Excluded (Major)--A configured WN is reachable but excluded because one or more other ANCs cannot see it. This
WN is not used for traffic redirection (new connections).
You can see alarms in the Central Manager Alarms panel or by using the show alarms EXEC command on a device.
Note: The CMM is an internal AppNav component that manages the grouping of ANCs and WNs into an AppNav cluster
associated with a service context.
You can use the Central Manager to verify, monitor, and troubleshoot AppNav clusters. The Central Manager has a global view of
all registered WAAS devices in your network and can quickly help you locate most AppNav issues.
From the Central Manager menu, choose AppNav Clusters > cluster-name. The cluster home window displays the cluster
topology (including WCCP and gateway routers), overall cluster status, device status, device group status, and link status.
Note that the ANC and WN icons shown in this diagram have the same device name because they reside on the same device. On
an ANC that is also optimizing traffic as a WN, these two functions are shown as separate icons on the topology diagram.
An orange triangle warning indicator is shown on any device for which the Central Manager may not have current information
because the device has not responded within the last 30 seconds (the device could be offline or unreachable).
You can get a detailed 360 degree status view of any ANC or WN device by hovering your cursor over the device icon. The first
tab displays alarms on the device. You should resolve any alarms that are inhibiting proper cluster operation.
Click the Interception tab to verify the device interception method on each ANC.
File:Waast-an-interceptiontab.png
Click the Cluster Control tab to see the IP address and status of each device in the cluster that this ANC can see. Each ANC in the
cluster should have the same list of devices. If not, it indicates a configuration or network issue.
File:Waast-an-clustercontrol.png
If all ANCs cannot see each other, the cluster is nonoperational and all traffic is passed through due to the cluster's inability to
synchronize flows.
If all ANCs are connected but have different views of the WNs, the cluster is in a degraded state. Traffic is still distributed, but
only to the WNs that are seen by all ANCs.
Click the Interfaces tab to verify the state of the physical and logical interfaces on the ANC.
Look at the 360 degree view on each WN in the cluster and verify the green status of all accelerators in the Optimization tab. A
yellow status for an accelerator means that the accelerator is running but is unable to service new connections, for example
because it is overloaded or because its license has been removed. A red status indicates that the accelerator is not running. If any
accelerators are yellow or red, you must separately troubleshoot those accelerators. If the Enterprise license is missing, the
description reads System license has been revoked. Install the Enterprise license in the Admin > History > License Management
device page.
A split cluster results from connectivity issues between ANCs in the cluster. If the Central Manager can communicate with all
ANCs, it can detect a split cluster, however, if it cannot communicate with some ANCs, it cannot detect the split. The
"Management status is offline" alarm is raised if the Central Manager loses connectivity with any device and the device is shown
as offline in the Central Manager.
It is best to separate the management interfaces from the data interfaces to maintain management connectivity even if a data link is
down.
In a split cluster, each subcluster of ANCs independently distributes flows to the WNGs that it can see, but since flows between
the subclusters are not coordinated, it can cause reset connections and the overall cluster performance is degraded.
Check the Cluster Control tab of each ANC to see if one or more ANCs are unreachable. The "Service controller is unreachable"
alarm is raised if two ANCs that once could communicate with each other lose connectivity between themselves but this situation
is not the only cause of a split cluster so it is best to check the Cluster Control tab of each ANC.
If an ANC has a gray status light, it might be disabled. Check that all ANCs are enabled by clicking the AppNav Controllers tab
below the topology diagram. If an ANC is not enabled, its Enabled status is No. You can click the Enable taskbar icon to enable
an ANC.
Check the AppNav policy on each ANC that has anything other than a green status light. If you hover your cursor over the status
light on a device, a tool tip tells you the status or problem, if one is detected.
To check the defined policies, from the Central Manager menu, choose Configure > AppNav Policies and then click the Manage
button.
There should generally be a single policy assigned to all ANCs in the cluster. The default policy is named appnav_default. Select
the radio button next to a policy and click the Edit taskbar icon. The AppNav Policy pane shows you the ANCs to which the
selected policy is applied. If all ANCs are not shown with a checkmark, click the checkbox next to each unchecked ANC to assign
After verifying the policy assignments, you can verify the policy rules in the AppNav Policies page that remains displayed. Select
any policy rule and click the Edit taskbar icon to change its definition.
An ANC could have a yellow or red status light if one or more policies are overloaded. Check the Overloaded Policies tab of the
360 degree device view to see a list of monitored policies that are overloaded.
If an ANC is joining the cluster, it is shown with a yellow status light and joining status.
The Interception tab of the 360 degree device view shows that the interception path is down due to the joining state. Interception is
held down until the ANC has synchronized its flow tables with the other ANCs and is ready to accept traffic. This process
typically takes no more than two minutes.
If you remove an ANC from a cluster, it is still shown for a few minutes in the topology diagram and as alive in the Cluster
Control tab, until all ANCs agree on the new cluster topology. It does not receive any new flows in this state.
You can use the show service-insertion service-context command on an ANC to see the service context status and stable view of
the devices in the cluster:
If the Device Interception State field (above) shows Shutdown, it means that the CMM has shut down interception due to this
ANC not being ready to receive traffic flows. For example, the ANC could still be in the joining process and the cluster has not
yet synchronized flows.
The Stable View fields (above) list the IP addresses of the ANCs and WNs seen by this ANC device in its last converged view of
the cluster. This is the view used for distribution operations. The Current View fields list the devices advertised by this ANC in its
heartbeat messages.
You can use the show service-insertion appnav-controller-group command on an ANC to see the status of each ANC in the
ANC group:
AppNav Controller : 10.1.1.2 (local) <<< local indicates this is the local A
AppNav Controller ID : 1
Current status of AppNav Controller : Alive
Time current status was reached : Wed Jul 11 02:05:23 2012
Joining status of AppNav Controller : Joined
Secondary IP address : 10.1.1.2
Cluster protocol ICIMP version : 1.1
Cluster protocol Incarnation Number : 2
Cluster protocol Last Sent Sequence Number : 0
Cluster protocol Last Received Sequence Number: 0
For a list of possible ANC statuses and joining statuses, see the show service-insertion command in the Cisco Wide Area
Application Services Command Reference.
You can use the show service-insertion service-node command on an ANC to see the status of a particular WN in the cluster:
Service Node ID : 1
Current status of Service Node : Alive <<< WN is visible
Time current status was reached : Sun May 6 11:58:11 2011
Cluster protocol DMP version : 1.1
Cluster protocol incarnation number : 1
Cluster protocol last sent sequence number : 1692060441
Cluster protocol last received sequence number: 1441393061
AO state
--------
AO State For
-- ----- ---
tfo GREEN 3d 22h 11m 17s <<< Overall/TFO state reported by WN
AppNav CLI Commands for Monitoring Cluster and Device Status 139
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
epm GREEN 3d 22h 11m 17s <<< AO states reported by WN
cifs GREEN 3d 22h 11m 17s
mapi GREEN 3d 22h 11m 17s
http RED 3d 22h 14m 3s
video RED 11d 2h 2m 54s
nfs GREEN 3d 22h 11m 17s
ssl YELLOW 3d 22h 11m 17s
ica GREEN 3d 22h 11m 17s
You can use the show service-insertion service-node-group command on an ANC to see the status of a particular WNG in the
cluster:
AO state
--------
AO State For
-- ----- ---
tfo GREEN 3d 22h 12m 52s
epm GREEN 3d 22h 12m 52s
cifs GREEN 3d 22h 12m 52s
mapi GREEN 3d 22h 12m 52s
http RED 3d 22h 15m 38s
video RED 11d 2h 4m 29s
nfs GREEN 3d 22h 12m 52s
ssl YELLOW 3d 22h 12m 52s
ica GREEN 3d 22h 12m 52s
AO state
--------
AO State For
-- ----- ---
tfo GREEN 3d 22h 12m 52s
epm GREEN 3d 22h 12m 52s
cifs GREEN 3d 22h 12m 52s
mapi GREEN 3d 22h 12m 52s
http RED 3d 22h 15m 38s
video RED 11d 2h 4m 29s
nfs GREEN 3d 22h 12m 52s
ssl YELLOW 3d 22h 12m 52s
ica GREEN 3d 22h 12m 52s
AppNav CLI Commands for Monitoring Cluster and Device Status 140
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
SNG Availability per AO <<< AO status for entire WNG
-----------------------
AO Available Since
-- --------- -----
tfo Yes 3d 22h 12m 52s
epm Yes 3d 22h 12m 52s
cifs Yes 3d 22h 12m 52s
mapi Yes 3d 22h 12m 52s
http No 3d 22h 15m 38s
video No 11d 2h 4m 29s
nfs Yes 3d 22h 12m 52s
ssl No 11d 2h 4m 29s
ica Yes 3d 22h 12m 52s
The first WN in the example above has a status of Excluded, which means that the WN is visible to the ANC but it is excluded
from the cluster because one or more other ANCs cannot see it.
The SNG Availability per AO table shows if each AO is able to service new connections. An AO is available if at least one WN in
the WNG has a GREEN status for the AO.
You can use the show service-insertion service-node command on a WN to see the status of the WN:
AppNav CLI Commands for Monitoring Cluster and Device Status 141
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Current State: GREEN
Time in current state: 43d 7h 44m 41s
MAPI
Current State: GREEN
Time in current state: 43d 7h 44m 43s
HTTP
Current State: GREEN
Time in current state: 43d 7h 44m 45s
VIDEO
Current State: GREEN
Time in current state: 43d 7h 44m 41s
NFS
Current State: GREEN
Time in current state: 43d 7h 44m 44s
SSL
Current State: RED
Time in current state: 43d 7h 44m 21s
Reason:
AO is not configured
ICA
Current State: GREEN
Time in current state: 43d 7h 44m 40s
The monitored state of an accelerator is its actual state but the reported state can differ because it is the lower of the system state
or the accelerator state.
For more information about troubleshooting optimization on a WN, see the Troubleshooting Optimization and Troubleshooting
Application Acceleration articles.
Several CLI commands are useful for troubleshooting policies and flow distribution on an ANC:
• show policy-map type appnav policymap-name -- Shows the policy rules and hit counts for each class in the policy map.
• show class-map type appnav class-name -- Shows the match criteria and hit counts for each match condition in the class
map.
• show policy-sub-class type appnav level1-class-name level2-class-name -- Shows the match criteria and hit counts for
each match condition in a class map in a nested AppNav policy map.
• show statistics class-map type appnav class-name -- Shows traffic interception and distribution statistics for a class
map.
• show statistics policy-sub-class type appnav level1-class-name level2-class-name -- Shows traffic interception and
distribution statistics for a class map in a nested AppNav policy map.
• show statistics pass-through type appnav -- Shows AppNav traffic statistics for each pass-through reason.
• show appnav-controller flow-distribution -- Shows how a specific hypothetical flow would be classified and distributed
by an ANC, based on the defined policy and dynamic load conditions. This command can be useful to verify how a
particular flow will be handled on an ANC and what class it belongs to.
• show statistics service-insertion service-node ip-address -- Shows statistics for accelerators and traffic distributed to the
WN.
• show statistics service-insertion service-node-group name group-name -- Shows statistics for accelerators and traffic
distributed to the WNG.
You can use the show statistics class-map type appnav class-name command on an ANC to troubleshoot flow distribution, for
example to determine why traffic might be slow for a particular class. This could be an application class map such as HTTP or, if
all traffic to a branch seems slow, it could be a branch affinity class map. Here is an example for the HTTP class:
Connections
-----------
Intercepted by ANC 4 <<< Are connections being intercepted?
Passed through by ANC 0 <<< Passed-through connections
Redirected by ANC 4 <<< Are connections being distributed to W
Accepted by SN 4 <<< Connections accepted by WNs
Passed through by SN (on-Syn) 0 <<< Connections might be passed through by
Passed through by SN (post-Syn) 0 <<< Connections might be passed through by
Passthrough Reasons Packets Bytes <<< Why is ANC passing through connections
------------------- ------- -----
Collected by ANC:
PT Flow Learn Failure 0 0 <<< Asymmetric connection; interception pr
PT Cluster Degraded 0 0 <<< ANCs cannot communicate
PT SNG Overload 0 0 <<< All WNs in the WNG are overloaded
PT AppNav Policy 0 0 <<< Connection policy is pass-through
PT Unknown 0 0 <<< Unknown passthrough
The WN pass-through reasons in the Indicated by SN section increment only if pass-through offload is configured on a WN.
Otherwise, the ANC does not know that the WN is passing through a connection and does not count it.
If the Connections: Intercepted by ANC counter is not incrementing, there is an interception problem. You can use the WAAS
TcpTraceroute utility to troubleshoot the placement of the ANC in the network, find asymmetric paths, and determine the policy
applied to a connection. For details, see the section Connection Tracing.
To debug an individual connection or set of connections on an ANC, you can use the show statistics appnav-controller
connection command to display the active connection list.
Passthrough Flows:
You can filter the list by specifying the client or server IP address and/or port options and you can show detailed statistics about
connections by specifying the detail keyword.
Optimized Flows
--------------------------
Client: 2.30.5.10:55330
Server: 2.30.1.10:5001
AppNav Controller Owned: Yes <<< This ANC is seeing activity on this connection
Service Node IP:2.30.1.5 <<< Connection is distributed to this SN
Classifier Name: se_policy:p5001 <<< Name of matched class map
Flow association: 2T:No,3T:No <<< Connection is associated with dynamic app or session (MAPI and ICA only)?
Application-ID: 0 <<< AO that is optimizing the connection
Peer-ID: 00:14:5e:84:41:31 <<< ID of the optimizing peer
Client: 2.30.5.10:55331
Server: 2.30.1.10:5001
AppNav Controller Owned: Yes
Service Node IP:2.30.1.5
Classifier Name: se_policy:p5001
Flow association: 2T:No,3T:No
Application-ID: 0
Peer-ID: 00:14:5e:84:41:31
...
You can specify the summary option to display the number of active distributed and pass-through connections.
Connection Tracing
To assist in troubleshooting AppNav flows, you can use the Connection Trace tool in the Central Manager. This tool shows the
following information for a particular connection:
1. From the Central Manager menu, choose AppNav Clusters > cluster-name, then choose Monitor > Tools > Connection
Trace.
2. Choose the ANC, the peer WAAS device, and specify connection match criteria.
The WAAS TCP Traceroute is another tool not specific to AppNav that can help you troubleshoot network and connection issues,
including asymmetric paths. You can use it to find a list of WAAS nodes between the client and server, and the configured and
applied optimization policies for a connection. From the Central Manager, you can choose any device in your WAAS network
from which to run the traceroute. To use the WAAS Central Manager TCP Traceroute tool, follow these steps:
1. From the WAAS Central Manager menu, choose Monitor > Troubleshoot > WAAS Tcptraceroute. Alternatively, you can
choose a device first and then choose this menu item to run the traceroute from that device.
2. From the WAAS Node drop-down list, choose a WAAS device from which to run the traceroute. (This item does not appear if
you are in the device context.)
3. In the Destination IP and Destination Port fields, enter the IP address and port of the destination to which you want to run the
traceroute
WAAS nodes in the traced path are displayed in the table below the fields. You can also run this utility from the CLI with the
waas-tcptrace command.
The following log file is available for troubleshooting AppNav cluster manager issues:
To set up and enable debug logging of the AppNav cluster manager, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
The options for cluster manager debugging (on 5.0.1 and later) are as follows:
You can enable debug logging for the cluster manager and then display the end of the debug error log as follows:
You can also enable debug logging for the flow distribution manager (FDM) or the flow distribution agent (FDA) with these
commands:
Note: Either packet capture or debug capture can be active, but not both simultaneously.
Data packets sent between ANCs and WNs are encapsulated, as shown in the following diagram.
If you capture packets at points 1 or 4 in the diagram, they are unencapsulated. If you capture packets at points 2 or 3, they are
encapsulated.
• A packet-capture ACL is always applied to inner IP packet for WCCP-GRE and SIA encapsulated packets.
• Packet capture is done on all ANC interfaces if the ANC interface for the packet capture is not provided.
This article describes how to troubleshoot disk, RAID, and hardware issues.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Contents
• 1 Checking the Disk Health
• 2 Disk Failures
• 3 RAID-5 Rebuilding and Synchronization
• 4 Firmware Upgrade for WAE-7341/7371/674
• 5 Boot Sequence Issue on WAE-7341/7371/674
• 6 Serial Port Disabled Issue on WAE-7341/7371/674
• 7 Boot Status Display
• 8 Replacing Disks on the WAE-612 with WAAS Versions 4.0.11 and Earlier
Releases
♦ 8.1 Disk01 Fails
♦ 8.2 Disk00 Fails and Disk01 has Problematic Status and is Marked Bad
♦ 8.3 Disk00 Fails and Disk01 is Not Marked Bad
From the command line, you can use the show disks details command as follows:
Contents 148
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
It is also useful to check the Predictive Failure Analysis (PFA) flag for RAID-5 disks by using the show disks tech-support
command. You will find the PFA flag at the end of the output. If the PFA flag is set to Yes, it indicates a predicted drive failure
and you should replace the disk. A critical alarm is also raised on the WAE.
Disk Failures
Disk failures are automatically detected by the system. Failed disks are automatically removed from service.
You can also shut down a disk for scheduled replacement by using the following commands:
After you replace a disk on a RAID-5 system, the system rebuilds the logical RAID drive automatically.
WAE7326# config
WAE7326(config)# disk disk-name disk01 shutdown
Device maybe busy while going offline ... please wait!
mdadm: set /dev/sdb1 faulty in /dev/md0
mdadm: set /dev/sdb2 faulty in /dev/md1
. . .
After you replace the disk on a RAID-1 system, use the following command to reenable the disk:
WAE7326# config
WAE7326(config)# no disk disk-name disk01 shutdown
If you do remove a disk during the RAID build process, reinsert the disk and wait up to 6 hours for the RAID build process to
complete.
There are slight differences in the RAID rebuild and synchronization, as follows:
• Rebuild: Occurs after hard disk replacement. A disk and RAID failure alarm is raised on the device. The hard disk LEDs
blink rapidly and the LED on the replaced hard disk remains amber until the rebuild process is complete. The show disks
detail command shows the RAID physical disk that was replaced in the "Rebuilding" state and the RAID-5 logical disk in
the "Critical" state.
• Synchronization: Occurs after system installation from the CD or RAID recreation. A RAID failure alarm is raised on the
device. The hard disk LEDs blink rapidly until the rebuild process is complete. The show disks detail command shows all
the RAID physical disks in the "Online" state and the RAID-5 logical disk in the "Impacted" state.
Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller Status : Okay
Channel description : SAS/SATA
Controller Model : IBM ServeRAID 8k
Controller Serial Number : 40453F0
Physical Slot : 0
Installed memory : 256 MB
Copyback : Disabled
Data scrubbing : Disabled
Defunct disk drive count : 0
Logical drives/Offline/Critical : 1/0/0
---------------------------------------------------
Controller Version Information
---------------------------------------------------
BIOS : 5.2-0 (15418)
Firmware : 5.2-0 (15418) <-----Firmware version
Driver : 1.1-5 (2449)
Boot Flash : 5.1-0 (15418)
---------------------------------------------------
. . .
If your RAID controller firmware needs to be updated, obtain the recommended version from the Cisco software download
website (registered customers only) and upgrade the firmware as described in the documentation that accompanies the firmware.
If you encounter this situation, change the BIOS back to boot from the compact flash to allow proper booting. For details on how
to change the startup sequence, see the chapter Using the Configuration/Setup Utility Program in the Cisco Wide Area Application
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Engine 7341, 7371, and 674 Hardware Installation Guide. You can choose the option Load Default Settings to restore the
correct default settings, which include booting from the internal compact flash storage device.
If you encounter this situation, you should reenable the serial port. For details, see the chapter Using the Configuration/Setup
Utility Programin the Cisco Wide Area Application Engine 7341, 7371, and 674 Hardware Installation Guide. You can choose the
option Load Default Settings to restore the correct default settings, which include enabling the serial port.
Cisco WAE and WAVE appliances have video connectors that should not be used in normal operation. The video output is for
troubleshooting purposes only during the BIOS boot and stops displaying output as soon as the serial port becomes active.
If you are monitoring the video output, it may appear that the device has stopped booting when the output stops, but it is normal
for the video output to stop while the device continues booting.
• Disk01 Fails
• Disk00 Fails and Disk01 has Problematic Status and is Marked Bad
• Disk00 Fails and Disk01 is Not Marked Bad
If you are running WAAS version 4.0.13 or a later release, see the Performing Disk Maintenance for RAID-1 Systems section in
the Cisco Wide Area Application Services Configuration Guide for the hot-swap disk replacement procedure.
NOTE: On a WAE-612 that is running any WAAS version from 4.0.13 through 4.0.19, which supports the hot-swap replacement
of drives, a problem may occur while replacing the drives while the unit is running. Occasionally, after a drive hot-swap
procedure, the WAE-612 may stop operating and require a reboot. To avoid this problem, upgrade your WAAS software to
version 4.0.19 or a later release.
Disk01 Fails
If the disk only in slot 01 (right slot) fails and disk00 is good, use the following procedures to replace the disk, depending on the
WAAS version on the device.
Disk00 Fails and Disk01 has Problematic Status and is Marked Bad
If disk00 fails and disk01 shows a status of Problematic, with an asterisk (*) next to the status (the asterisk means the disk is
marked bad), it means that disk00 has failed but disk01 is misclassified as bad and its partition table has been removed. In this
situation, all data will be lost after the disk replacement.
Use the following procedures to replace the disk, depending on the WAAS version on the device.
Use the following procedures to replace the disk, depending on the WAAS version on the device.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 Checking for Connectivity Between the Serial
Peers
• 2 Verifying that the Serial Peers are Configured
Correctly
• 3 Verifying that a Serial Inline Cluster is
Operational
• 4 Detecting Serial Peer Configuration Mismatch
• 5 Troubleshooting MAPI Acceleration
♦ 5.1 Check EPM and MAPI Dynamic
Policies
NOTE: Serial inline clustering between non-optimizing peers and interception ACLs were introduced in WAAS version 4.2.1.
This section is not applicable to earlier WAAS versions.
If the serial peers are separated by one or more switches, the peer will not show up in the output above.
Run this command on both the peers and make sure that each device shows up on the other correctly.
Use the show device-id command to check the device ID, as follows:
WAE#show device-id
System Device ID is: 00:21:5e:57:e9:d4
or
Normally, optimization should take place between the outermost WAEs, that is, BR-WAE and DC-WAE1, or BR-WAE1 and
DC-WAE1. To ensure this, verify the device IDs on the connections by using the show statistics connection command. The
PeerID on BR-WAE should indicate that it is optimizing with DC-WAE1 and the PeerID on DC-WAE1 should indicate that it is
optimizing with BR-WAE.
If DC-WAE1 fails or goes into overload, new connections should be optimized between BR-WAE1 and DC-WAE2. You can
verify this by using the show statistics connection optimized command on DC-WAE2. Optimized connections should be seen on
DC-WAE2, with the peer ID of BR-WAE1 as the peer device.
If BR-WAE1 fails or goes into overload, there should not be optimization between DC-WAE2 and DC-WAE1. All connections
should be in the "PT Non-optimizing Peer" state on DC-WAE1 and "PT No Peer" on DC-WAE2. An example of the expected
show statistics connection command output follows:
You can also use the Central Manager Connection Statistics report (Device > Monitor > Optimization > Connections Statistics)
to display device connection statistics in a table, as shown in Figure 1. The Peer IDs are indicated by device name.
To detect a serial peer configuration mismatch, you can also look for syslog messages such as the following:
The following issues can occur with MAPI acceleration on serial inline clusters:
Number: 2 Type: Any->Host (6) User Id: EPM (3) <------ EPM Policy
Src: ANY:ANY Dst: 10.56.45.68:1025
Map Name: uuidf5cc5a18-4264-101a-8c59-08002b2f8426
Flags: TIME_LMT REPLACE FLOW_CNT
Seconds: 1200 Remaining: 10 DM Index: 32766
Hits: 1 Flows: 0 Cookie: 0x00000000
DM Ref Index: -None- DM Ref Cnt: 0
As always, disk logging needs to be enabled and the logging level for disk has to be set to debug.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
1. The interface may be down. If it is an inline interface, all traffic will be bypassed in hardware. Use the following command to
check the interface status:
2. If the interface is up, check the state of the connections, and if they are in pass-through, check the reason using the following
command:
3. If the reason appears as "PT Interception ACL", then it is due to the interception ACL denying the SYN packets.
You can look at the following output to drill down into the ACL to see which condition matched:
WAE#show ip access-list
Space available:
49 access lists
499 access list conditions
Standard IP access list test
1 permit any (1296 matches)
(implicit deny any: 0 matches)
total invocations: 1296
Interface access list references:
None Configured
Application access list references:
INTERCEPTION Standard test
Any IP Protocol
WAE#show ip access-list
Space available:
49 access lists
499 access list conditions
Standard IP access list test
1 permit any (1296 matches)
(implicit deny any: 0 matches)
total invocations: 1296
Interface access list references:
None Configured
Application access list references:
INTERCEPTION Standard test
Any IP Protocol
Check the hit counts from the above output to see if they are incrementing as expected.
As always, disk logging needs to be enabled and the logging level for disk has to be set to debug.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
• 1 Identifying a vWAAS Device
• 2 Troubleshooting vWAAS Device
Registration
• 3 Verifying vWAAS Virtual Interfaces
• 4 Troubleshooting vWAAS Networking
• 5 Troubleshooting VPATH Interception
• 6 Troubleshooting Undersized Alarm
Virtual WAAS (vWAAS) implements a virtual WAAS appliance in VMware ESXi on a host server such as Cisco UCS.
NOTE: vWAAS was introduced in WAAS version 4.3.1. This section is not applicable to earlier WAAS versions.
The model of the vWAAS device is determined from the number of CPUs and Maximum TCP Connections shown in the Device
Dashboard window when you select the device from the Manage Devices page. These two fields are displayed only for vWAAS
devices.
For vCM devices, you can use the show hardware command to determine the number of CPUs, which tells you which model of
vCM is installed.
Note: The vWAAS device shows 2 disks installed. The first, disk00, is 4 GB and emulates the flash storage in a physical WAAS
device. The second, disk 01, emulates the hard disk in a physical WAAS device and varies in size depending on the vWAAS
model.
The show tfo detail command also displays the maximum TCP connection limit:
Critical Alarms:
----------------
None
Major Alarms:
-------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 notregistered vwaas/model <------Not registered ala
. . .
To register the vWAAS device with the Central Manager, use the cms enable global configuration command on the vWAAS
device:
vWAAS# config
vWAAS(config)# cms enable
Registering WAAS Application Engine...
Sending device registration request to Central Manager with address 2.75.16.100
Please wait, initializing CMS tables
Successfully initialized CMS tables
. . .
management services enabled
You can verify the registration with the show cms info command:
vWAAS device registration and deregistration is logged in the system message log with a line that begins with "vWAAS:". You
can view the system message log in the Central Manager by choosing Admin > Logs > System Messages.
In the Central Manager device > Configure > Network > Network Interfaces page, the vWAAS interface type appears as Virtual
(Port Channel, Standby, Inline, and GigabitEthernet are not applicable), which is similar to the GigabitEthernet . Some of the
GigabitEthernet interface options, such as Port Channel, autosense, speed, mode, and standby, do not apply to virtual interfaces.
You can also see the virtual interfaces with the show running-config command:
Additional details are available with the show interface virtual 1/0 or show interface virtual 2/0 commands.
To make interface configuration changes, you can use the Central Manager Network Interfaces page or the interface, ip, and
primary-interface configuration commands, as follows:
vWAAS# config
vWAAS(config)# interface virtual 1/0
vWAAS(config-if)# ip addr 10.10.10.15 255.255.255.0
vWAAS(config-if)# end
vWAAS# config
vWAAS(config)# ip default-gateway 10.10.10.1
vWAAS(config)# primary-interface virtual 1/0
vWAAS(config)# end
Using the vSphere Client, you can trace vWAAS network connectivity from the device page. Identify which network label the
network adapter is connected to, determine the virtual switch that this network is connected to, and determine the physical NIC
that is a member of this virtual switch. Verify that the configuration is correct.
Also make sure the virtual switch VLAN settings are correctly configured to reach the network.
Verify the configured IP address, netmask, default gateway, and primary interface on the vWAAS device. For details, see the
previous section, "Verifying vWAAS Virtual Interfaces".
From the vWAAS device, ping the default gateway and Central Manager to make sure they are reachable.
You can use the vn-service vpath global configuration command to enable or disable VPATH interception.
From the vWAAS device CLI, you can view VPATH status and statistics with the show statistics vn-service vpath command:
To determine if VPATH is sending ARP requests, use the tcpdump arp command.
To display VPATH MAC address information for TCP flows, use the show statistics connection egress-methods command:
Critical Alarms:
----------------
None
Major Alarms:
-------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 undersized vwaas/model memory <-----Undersized alarm
. . .
You should never see this alarm if you are using valid OVA files to deploy vWAAS. If you see this alarm, delete the vWAAS VM
and redeploy it using a valid OVA file.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Contents
• 1 Verifying WAAS Express Image Version
• 2 Verifying WAAS Express License
• 3 Verifying WAAS Enabled Interfaces
• 4 Verifying WAAS Optimized Connections
• 5 Verifying WAAS Optimized Data
• 6 Verifying WAAS Express Alarms
• 7 Verifying WAAS Express Peers
• 8 Offline Alarms
• 9 Verifying WAAS Express HTTPS Configuration
• 10 WAAS-Express - WAE - WAAS CM Compatibility
♦ 10.1 WAAS-Express Version 1.0,1.5
◊ 10.1.1 Known Issues
♦ 10.2 WAAS-Express Version 2.0.0
◊ 10.2.1 Known Issues
• 11 Unexpected WAAS-Express License Expiration
• 12 WAAS-Express and WAAS CM interaction issues
♦ 12.1 Symptom: WAAS-Express fail to register with the WAAS CM
◊ 12.1.1 Possible Cause #1: Connectivity issue
♦ 12.2 Symptom: WAAS CM shows WAAS-Express goes off-line after successful registration
◊ 12.2.1 Possible Cause #1: WAAS-Express device certificate changes
◊ 12.2.2 Possible Cause #2: Incorrect certificates or trustpoints are used
◊ 12.2.3 Possible Cause #3: Device authentication problem
◊ 12.2.4 Debug Information
♦ 12.3 Symtom: Mismtach Statistic between WAAS CM and WAAS-Express
◊ 12.3.1 Possible Cause #1: Clocks are not synchronized
• 13 Connections are not getting optimized
♦ 13.1 Symptom: Connections are getting pass-through
◊ 13.1.1 What could cause asymetric routing or dropped packets in the network
◊ 13.1.2 Information to be provided to development team:
• 14 Connections are not getting the desired optimization level
♦ 14.1 Symtom: Established connections do not get the desired or configured policy to use CIFS, SSL, or
HTTP-Express AO
♦ 14.2 Symtom: Expected connection optimization is THDL, but established connection has TDL
♦ 14.3 Symtom: Expected connection optimization is TCDL, but established connection has TDL
♦ 14.4 Symtom: Expected connection optimization is TSDL, but established connection has TDL
♦ 14.5 Expected connection optimization is TSHDL, but established connection has only TSDL or THDL
• 15 Symptom: Unexpected Connection Reset
♦ 15.1 Steps to troubleshoot
♦ 15.2 Information to be provided to the development team:
• 16 Router crash/tracebacks
♦ 16.1 Information to be provided to the development team:
• 17 Slow connection/degraded performance
♦ 17.1 Step to troubleshoot
• 18 Hung connections
♦ 18.1 Step to troubleshoot and collect information
• 19 SSL-Express Accelerator issues:
♦ 19.1 Having issues with SSL-Express Accelerator enable or disable
• 20 Moving WAAS-Express device between Device-Groups on CM
• 21 Other useful information
♦ 21.1 Statistics mismatch on WAAS-Express and WCM/WAE:
◊ 21.1.1 Information in addition to debugs and show commands, that needs to be provided to the
development team:
♦ 21.2 Troubleshooting router crash
♦ 21.3 Capturing packets on router
WAAS Express is WAAS functionality built into IOS running on a device such as a router. The WAAS Central Manager can
manage a WAAS Express device along with other WAAS devices in the WAAS network. This article describes how to
troubleshoot WAAS Express device operation.
Note: WAAS Express Central Manager support was introduced in WAAS version 4.3.1. This section is not applicable to
earlier WAAS versions.
Contents 168
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Note: WAAS should be enabled on WAN interfaces only. If connections, to be optimized, are routed over multiple WAN
interfaces, then, WAAS should be applied on all those WAN interfaces.
Note: If WAAS is enabled on a logical or virtual interface it need not be implemented on the corresponding physical
interface.
To view similar information from the Central Manager, choose the WAAS Express device, then choose Monitor > Optimization
> Connections Statistics to see the Connections Summary Table.
Number of applications : 1
Application: waas-default
TCP Data Volumes
Connection Type Inbound Outbound
Opt TCP Plus 53001765483 41674120
Orig TCP Plus 0 87948683030
Opt TCP Only 1165 863
Orig TCP Only 60 0
Internal Client 0 0
Internal Server 0 0
To view alarms for all devices from the Central Manager, choose My WAN > Alerts. In addition to the alarms listed above, an
alarm is raised if the clocks of the WAAS Express and WAAS Central Manager devices are not synchronized.
Local AO statistics
Local AO: TFO
Total number of incompatible connections: 0
Version: 0.11
Registered: Yes
Local AO: HTTP
Total number of incompatible connections: 0
Version: 1.1
Registered: Yes
Local AO: SSL
Total number of incompatible connections: 0
Version: 1.0
Registered: Yes
Cache:
Cache in storage 0 B
Age 00:00:00
AckQ:
AckQ in storage 0 B
WaitQ:
WaitQ in storage 0 B
WaitQ size 0 B
Sync-clock:
Local-head 0 ms
Local-tail 0 ms
Remote-head 18609143000 ms
Curr-sync-clock 24215235228 ms
Encode Statistics
DRE msgs: 1
R-tx total: 0
R-tx chunk-miss: 0
R-tx collision: 0
Bytes in: 0
Bytes out: 0
Bypass bytes: 178
Compression gain: 0%
Decode Statistics
DRE msgs: 4
Bytes in: 299
Bytes out: 277
Bypass bytes: 51
Compression gain: 0%
To view similar information from the Central Manager, choose Monitor > Topology.
Offline Alarms
The WAAS Express device may go to an offline state in the Central Manager because of the following issues:
Credentials are not configured for this WAAS Express device in the Central Manager. The WAAS Central Manager needs
the WAAS Express username and password to communicate with the WAAS Express device. You can configure
credentials in the Central Manager by choosing My WAN (or a WAAS Express device or device group) > Admin >
WAAS Express Credentials.
The Central Manager is not able to communicate with the WAAS Express because wrong credentials are configured. You
can configure credentials in the Central Manager by choosing My WAN (or a WAAS Express device or device group) >
Admin > WAAS Express Credentials.
The WAAS Express device certificate is changed and the same certificate is not imported for this device in the Central
Manager. To reimport the WAAS Express device certificate, choose the WAAS Express device, then choose Admin >
Certificate.
The Central Manager is not able to reach the WAAS Express Device. Configure the correct WAAS Express management
IP address by choosing the WAAS Express device, then choosing DeviceName > Activation.
The HTTPS server port configured on the WAAS Express device is not the same as the port shown in the Central
Manager DeviceName > Activation page. Configure the correct WAAS Express HTTPS server port in this page.
The WAAS Express device is downgraded to an IOS image version with no WAAS support. Install an IOS image with
WAAS support.
The WAAS Express device is taking more than 30 seconds to respond to the Central Manager. It could be because the
WAAS Express device is overloaded or the network is slow.
The Evaluation license on the WAAS Express device is expired. Install a Permanent license by using the WAAS Express
license install command.
• SSL connection closed incorrectly while communicating with WAAS Express device.
The WAAS Express device and Central Manager are using the cipher rc4-128-md5 for SSL communication. Sometimes
the Central Manager fails to decrypt the SSL data sent by the WAAS Express. Configure the ciphers 3des-ede-cbc-sha,
des-cbc-sha, and rc4-128 by using the WAAS Express command ip http secure-ciphersuite 3des-ede-cbc-sha
des-cbc-sha rc4-128-sha.
The Central Manager is not receiving configuration status from the WAAS Express device. Contact Cisco TAC for
assistance troubleshooting.
If you see this error message, contact Cisco TAC for assistance troubleshooting.
Known Issues
Known Issues
Solution: Upgrade to a recommended WAAS-Express Version 2 image - 15.2(4)M1 or install a permanent license.
Troubleshoot steps: Verify that WAAS CM is ping?able from the router. In addition, if WAAS-Express router is behind
NAT and/or firewall, a static NAT entry and/or firewall permit rule are required to allow WAAS CM to connect to
WAAS-Express HTTPS server. To manage WAAS-Express devices behind NAT/Firewall, WAAS CM allows user to
manually change/specify address of WAAS-Express device for WAAS CM to use. User can change the address from the
device activation page.
Solution: Check route and network topology to make sure WAAS CM is reachable from the router and vice versa, please
enable the following debugs on WAAS-Express device.
If required, check following debugs to figure out if SSL handshake during registration is failing:
Verify this by comparing the WAAS-Express router certificate expiration date stored on the WAAS CM. Navigate to this
page from the WAAS-Express device page, Admin->Certificate. Compare the certificate information with the output of
show crypto pki certificate output on the WAAS-Express router. If there is any mismatch, it is very likely the certificate
ia re-generated.
• Verify this by comparing the WAAS-Express router certificate expiration date stored on the WAAS CM. Navigate to this
page from the WAAS-Express device page, Admin->Certificate. Compare the certificate information with the output of
show crypto pki certificate output on the WAAS-Express router. If there is any mismatch, it is very likely the certificate
ia re-generated.
Issue show run | include crypto pki trustpoint. Non-persistent trustpoint naming is in the format of
TP-self-signed-xxxxxxxxxx.
• There are serveral instances where the certificate could be re-generated but the main reason is trustpoing is created as
non-persistent. If you enable SSL Express AO with 15.2(3)T, you could also potentially hit CSCtz85134.
Solution: Upgrade to 15.2(4)M1 and re-create persistent trustpoint. Delete the certificate from WAAS CM and
re-register.
In 15.2(3)T, there is a mandatory config within the crypto pki trustpoint, which requires rsa-keypair to be configured. If
this config does not present before upgrade, this could potentially cause the router not be able to detect the trustpoint. This
will cause HTTPS connectivity to fail. This problem is documented in CSCty04359.
Solution: Remove the trustpoint and re-create. Delete the certificate from WAAS CM and re-register.
During WAAS CM registration, WAAS-Express router selects the trustpoint which it uses for sending certificate to
WAAS CM. This may be different trustpoint from what the local HTTPS server on the WAAS-Express router uses.
Solution: Verify that the same trustpoing is configured in ip http secure-trustpoint <trustpoint_name> and ip http-client
secure-trustpoint <trustpoint_name>
• Is authentication failing?
Verify that you can login to the WAAS-Express router, by directing your browser to WAAS-Express router using HTTPS
and attempt the authentication manually.
Debug Information
If you believe you are running into certificate related issuses, please provide below information to support team.
WAAS CM and WAAS-Express clock need to be in sync and hence configuring NTP server to sync clocks is highly
recommended.
Solution: Configure NTP and verify that all devices' clock are synchronized. Follow the workaround in the
DDTS mentioned above, or upgrade to the latest 15.2(4)M1 or later.
Use the following command to check the reason '''show waas statistics auto-discovery'''
• If the counter for Interface Application Configincrements, it is likely your policy is configured to pass-through this
particulate connection. Check your WAAS policy on both WAAS-Express and its peer.
Solution: Check and validate your optimization policy. Use below debug to discover if traffic is marked as pass-through
in the policy.
• If the counter for Interface Global Config increments, this could be caused by asymetrical routing in your network. This
is the case where WAAS-Express or its peer does not see both directions of the TCP traffic. This could be caused by true
asymetrical routing in the network, or could be caused by some packets are getting dropped by devices in the traffic path
(ACL, firewall, etc.)
Solution: Check for asymetric routing of dropped packets in the network.See what could cause asymetric routing or
dropped packets in the network below.
• Connections could also be pass-through if the peers are not compatible with each other. This may happen if you run the
non-compatible version between WAAS-Express and WAE. Check the compatibility table above for recommedned
software releases.
Solution #1: Check if the peer is incompatible using show waas statistics aoim
Solution #2: If you believe you have asymetrical routing scenario in your network, check the following.
• Multiple WAN links in either the WAAS-Express router or the peer. Note that WAAS-Express it not supported on
active/active or active/standby routers because both traffic leaving and entering the WAN need to be on the same
WAAS-Express router. If there are multiple WAN links, make sure all the WAN links have config waas enable. Make
sure that all the WAN links and routers on the peer routers have config to redirect traffic to WAAS.
• Control packets (SYN, SYN-ACK, ACK) are not tagged with WAAS option.This could happen if the traffic is not
redirected to WAAS on the peer side. Check your WCCP ACL.
Network topology
IOS version
Configuration
Note: Pass-through connections are not counted in the per-platform connection limit. WAAS-Express does not track
pass-through connections, hence there are no statistics related to pass-through flows. There, however, are counters that
indicate how many flows were put into pass-through and why.
Symtom: Established connections do not get the desired or configured policy to use
CIFS, SSL, or HTTP-Express AO
• Verify that CIFS, SSL, or HTTP-Express AO is enabled globally
Maximum Flows : 75
Total Active connections : 4
Total optimized connections : 4
What could cause asymetric routing or dropped packets in the network 180
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Solution #1: Check if the core WAAS device is compatible. This check can be done using show waas statistics aoim
Solution #2: Check if HTTP-Express Accelerator is getting negotiated during auto-discovery using auto-discovery
debugs. This may be because the accelerator is disabled globally (note that HTTP accelerator is not enabled by default), or
HTTP class is missing ?accelerate http? in the action.
class HTTP
optimize tfo dre lz application Web accelerate http-express
• Check Configured, Derived and Applied Accelerator fields under show waas connection detail
class CIFS
optimize tfo dre lz application CIFS accelerate cifs-express
• The connection got pipe-through?ed and shows up as TG. As shown above, check reason in show waas statistics
accelerator ssl
• If the connection shows up as TSDL could be due to one of the following
♦ HTTP-Express Accelerator is disabled.
♦ HTTP-Express Accelerator is not compatible with the HTTP AO on core WAAS device.
◊ At least 3 optimization features of HTTP-Express Accelerator are not enabled.
♦ The first data packet does not contain HTTP content.
Aug 18 03:02:52.861: %WAAS-3-WAAS_TFO_DEC_FRAME_FAILED: IOS-WAAS failed to decode TFO frame for connection 100.2.0
Symtom: Expected connection optimization is TSDL, but established connection hasTDL 182
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
Steps to troubleshoot
• Turn on error debugs, depending on the module, debug waas <module_name> error.
• Check End-Reason in show waas connection detail
• Check show waas statistics error for possible reasons.
• Is a core-dump generated on core WAE when connection resets are seen?
♦ Malformed TCP headers sent by WAAS-Express resulted in core-dumps on WAE.
♦ DDTSs capturing this issue: CSCto59459, CSCua61097. Search for these DDTS and check whether the issue
seen is similar to the one outlined by them.
• If this is an SSL-Express Accelerator connection is the reset being caused by W2W handshake failure?
Debug logs Show command logs show-tech show-running config Network topology Client and server details, along with the
application (and version, e.g. IE6) being used for connection.
Router crash/tracebacks
Router crashes and tracebacks may have been seen during testing. Search of previous cases and DDTSs for similar known issues.
In addition we also need to isolate what feature is resulting in the crash. If an IOS feature other than ios-waas or layer4-forwarding
is resulting in a crash/traceback, then that particular feature development team/ router TAC should be contacted accordingly.
Step to troubleshoot
Note: Per-packet load-sharing is not a supported deployment. This is not a default load sharing mode.
Hung connections
There are no known issues with hung connections, please provide the following information to the development team to help RCA
the problem.
• Search the flow in WAAS-Express connection table using show waas connection.
• From the first column, collect the L4F flow id and use the information to get the detail L4F connection information.
• The output of show l4f flow detail <flow_id> show the two TCP TCBs. Use the TCB information in show tcp tdb
<tcb_info>
SRTT: 6687 ms, RTTO: 59312 ms, RTV: 52625 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 2857348 ms, ACK hold: 200 ms
Status Flags: passive open, Timestamp echo present
Option Flags: keepalive running, SACK option permitted, non-blocking reads
non-blocking writes, win-scale, 0x200000, 0x1000000, 0x10000000
0x20000000
IP Precedence value : 0
• The following command output can be useful in debugging the WAAS-Express AO.
• Check if you have an NPE image (this image does not support SSL-Express Accelerator)
• Enable ssl, aoim and infra debugs during enable/disable operation and provide debug logs.
• Check alarms:
• Check configuration on edge and core devices. Check they are in-sync with respect to cipher-list, SSL version, and
certificate verification and revocation checks.
• If self-signed certificates are being used, revocation-check and certificate verification should be disabled.
• Turn on debug waas accelerator ssl error
Use the following steps when moving the device between device-groups:
* Go to the Policy Definitions page of that device and select the new device-group and click on Submit.
OR
* Go to device-group-1 -> Assign Devices page and unassign the device from this DG.
* Go to device-group-2 -> Assign Devices page and assign the device to this DG.
* Go to device-group-2 -> Policy Definitions page and click on 'Force DG settings' button.
Information in addition to debugs and show commands, that needs to be provided to the development team:
show tech-support
show ip interface
show ip virtual-reassembly
show ip route
show ip cef detail
show ip cef internal
show ip cef switching statistics
show process cpu history
For details on IOS packet capture, see the document: IP Traffic Export.
interface Serial0/0/0
ip virtual-reassembly out
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later
encapsulation frame-relay
ip traffic-export apply waas_wan size 20000000
frame-relay map ip 10.0.0.2 557 broadcast
no frame-relay inverse-arp
frame-relay local-dlci 557
Use following commands to start, stop, copy and clear the buffer:
This article describes how to troubleshoot the integration of Network Analysis Module (NAM) into the WAAS Central Manager.
Guide Contents
Main Article
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Contents
•1
Connectivity
Issues
• 2 Rendering
Issues
• 3 Chart Data
Issues
The Cisco Network Analysis Module (NAM) is a standalone network analysis product that you can access through the WAAS
Central Manager if you have a NAM server installed in your network. This article describes how to troubleshoot the integration of
NAM into the WAAS Central Manager.
Connectivity Issues
If you are unable to connect to NAM from the WAAS Central Manager, do the following:
• In the Central Manager GUI, choose Configure > Network Analysis Module (Beta) > Basics > Setup and ensure that
NAM server address and credentials are entered correctly, then click Test Connectivity/Credentials. All of the address,
user, and password fields should show a green check mark to indicate success. Any fields that show a red X are
configured incorrectly.
• Ensure that the NAM HTTP or HTTPS server is enabled.
• Ensure that the NAM server IP address and port are reachable both from the Central Manager and from the client
computer on which you are running the browser to access the Central Manager. You should be able to use the same IP
address and port to access the NAM server from both machines, regardless of whether NAT is involved.
• Check your browser settings. In Internet Explorer, you may need to revert to the default settings in the Security and
Privacy tabs of Tools > Internet Options, then change the Privacy setting to Low.
• If you are using the NAM HTTPS server, the NAM self-signed certificate may not be installed in the browser. Manually
add the NAM self-signed certificate into your browser. Alternatively, you can launch the NAM user interface in a separate
browser tab or window and accept and install the certificate, which allows you to view the integrated charts.
If you are unable to directly telnet into the NAM server, use the exsession on command on the NAM console
Rendering Issues
If the NAM page does not render properly or some action buttons on NAM do not work, it is likely due to an incompatible
browser version. The NAM server has the following browser requirements:
If you get the message, "Navigation to the webpage was canceled" when you click on a NAM report, log out of the Central
Manager and log in again. When the browser asks if you want to view only the content delivered securely, click "No" to show all
content. The Central Manager data is based on HTTPS and NAM access uses HTTP by default, so the browser must be able to
show both secure (HTTPS) and unsecure (HTTP) content.
Why Is There a Difference in Statistics When Different Data Sources Are Selected?
The netflow data source reflects data through the netflow export device (router/switch) and the FA data source reflects data
intercepted by WAAS.