Ace Ts Guide

Cisco Application Control Engine (ACE) Troubleshooting Guide
Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to collaborate on Cisco product
documentation. You do not need to log in to read the text. However, you must log in to edit the text. Select the "edit" tab to edit
an article or select the "discussion" tab to submit questions or comments about documentation content.
See Terms of Use and About DocWiki for more information about Cisco DocWiki.
Click here to go to the Cisco ACE Module documentation on www.cisco.com.
Click here to go to the Cisco ACE Appliance documentation on www.cisco.com.
Contents
• 1 Audience
• 2 Organization
• 3 Creating a PDF of the ACE
Troubleshooting Wiki
• 4 Related Documentation
♦ 4.1 ACE Module Documentation
♦ 4.2 ACE Appliance Documentation
This article provides a systematic approach to identifying and remedying problems that may arise as you use your ACE over a
period of time. This guide is not intended to replace configuration best practices or to be an all-inclusive guide for every
application. Rather, it is an attempt to provide you with the knowledge and skills necessary to correct the most common issues that
you may encounter.
Audience
This article is intended for all trained network administrators who have experience with the configuration and maintenance of the
ACE.
Organization
This article consists of the following major sections:
Overview of ACE Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Troubleshooting Connectivity
Troubleshooting Ethernet Ports (ACE appliance)
Troubleshooting Remote Access
Contents 1
Troubleshooting Access Control Lists
Troubleshooting Network Address Translation
Troubleshooting ACE Health Monitoring
Troubleshooting Layer 4 Load Balancing
Troubleshooting Redundancy
Troubleshooting SSL
Troubleshooting Compression
Troubleshooting Performance Issues
ACE Resource Limits
Managing Resources
Show Counter Reference
Creating a PDF of the ACE Troubleshooting Wiki

You can create a PDF of one or more articles in this wiki, including the entire ACE Troubleshooting Wiki. For details, see the
Creating a PDF article.
Related Documentation
ACE Module Documentation
• Customer Documentation for the Cisco Application Control Engine (ACE) Module
• Cisco Application Control Engine (ACE) Configuration Examples on DocWiki
ACE Appliance Documentation

• Hardware Installation Guides for the Cisco ACE 4710 Appliance
• Release Notes for the Cisco 4700 Series Application Control Engine Appliance
• Command Reference for the Cisco 4700 Series Application Control Engine Appliance
• Configuration Guides for the Cisco 4700 Series Application Control Engine Appliance
• Cisco CSS-to-ACE Conversion Tool User Guide
• Cisco Application Control Engine (ACE) Configuration Examples on DocWiki
This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when
you configure and use your ACE.
Organization 2
Guide Contents
Main Article
Troubleshooting ACE Appliance Ethernet Ports
Troubleshooting SSL
ACE Resource Limits
Managing ACE Resources
Contents
• 1 Overview of the ACE Troubleshooting Process
• 2 Verifying the ACE Image
• 3 Enabling ACE Logging
• 4 Gathering ACE Troubleshooting Information
♦ 4.1 Rebooting the ACE
♦ 4.2 Using show Commands
♦ 4.3 Capturing Packets in Real Time
♦ 4.4 Copying Core Dumps
♦ 4.5 After Gathering Troubleshooting Information
• 5 Verifying the Physical Connectivity Between the ACE and the End
Hosts
• 6 Verifying the ACE Layer 2 Connectivity
• 7 Verifying the ACE Layer 3 Connectivity
• 8 Contacting Cisco Technical Support
ACE Appliance Documentation 3

Overview of the ACE Troubleshooting Process

To troubleshoot your ACE, follow these general guidelines:
1. Maintain a consistent and recommended software version across all your ACEs. See the "Verifying the ACE Image"
section.
2. See the ACE module release notes for your software version for the latest features, operating considerations, caveats, and
CLI command changes.
3. Before you introduce configuration changes, use the ACE checkpoint feature to bookmark a known good configuration
and save your configuration. If you run into problems with the new configuration, you can roll back the new configuration
to the known good configuration. See the Cisco Application Control Engine Module Administration Guide. Troubleshoot
new configuration changes immediately after adding them.
4. Verify that your configuration is correct for your network application. Make any required changes to the running-config
file, and then test the configuration. If it is satisfactory, save it to the startup-config file using the copy running-config
startup-config command for a particular virtual context or the write memory command from the Admin context to copy
all running-config files in every virtual context to their respective startup-config files.
5. Enable system message logging. See the "Enabling ACE Logging" section.
6. Gather information that defines the specific symptoms. See the "Gathering ACE Troubleshooting Information" section.
7. Verify the physical connectivity between your device and end devices. See the "Verifying the Physical Connectivity
Between the ACE and the End Hosts" section.
8. Verify the ACE Layer 2 connectivity. See the "Verifying the ACE Layer 2 Connectivity" section.
9. Verify the ACE end-to-end connectivity and the routing configuration. See the "Verifying the ACE Layer 3 Connectivity"
section.
10. After you have determined that your troubleshooting attempts have not resolved the problem, contact the Cisco Technical
Assistance Center (TAC) or your technical support representative. See the "Contacting Cisco Technical Support" section.
Verifying the ACE Image

To display the version of the software image and the image filename that is currently running in your ACE, enter the following
command:
ACE_module5/Admin# show version

Cisco Application Control Software (ACSW)
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.
Software
loader: Version 12.2[121]
system: Version A2(2.0) [build 3.0(0)A2(2.0)] <--------
system image file: [LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin <--------
installed license: no feature license is installed
Hardware
Cisco ACE (slot: 5)
cpu info:
number of cpu(s): 2
cpu type: SiByte
cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz
memory info:
total: 955396 kB, free: 289704 kB
shared: 0 kB, buffers: 2336 kB, cached 0 kB
cf info:
filesystem: /dev/cf
total: 1000000 kB, used: 494912 kB, available: 505088 kB
last boot reason: NP 1 Failed : NP ME Hung
configuration register: 0x1

ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)
This command provides other useful information, for example:
• Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5)
• Available control plane memory
• Last boot reason
• Configuration register (confreg) value (0x0 boot to rommon, 0x1 boot using boot string)
• ACE uptime
Enabling ACE Logging

To enable logging on the ACE module and to send system logging (syslog) messages to the monitor, enter the following
commands:
ACE_module5/Admin(config)# logging enable

ACE_module5/Admin(config)# logging monitor 7
ACE_module5/Admin(config)# exit
ACE_module5/Admin# terminal monitor
Note: Use the terminal no monitor command to stop viewing log messages in your remote session.
For more information about logging, see the "Troubleshooting with ACE Logging" section.
Gathering ACE Troubleshooting Information

The following sections recommend ways to gather information that is relevant to the problem that is occurring.
Rebooting the ACE

Do not reboot the ACE unless it is absolutely necessary. Some information that is important to troubleshooting your problem may
not survive a reboot. Try to gather as much information as possible before rebooting.
Using show Commands

You can use a number of show commands in Exec mode to gather information specific to the symptoms you are observing in your
ACE. In most cases, you can gather the information you need to troubleshoot the ACE by entering the show tech-support
command. This command runs many show commands that are useful for troubleshooting the ACE. You can redirect the output of
the show tech-support command to one the following destinations:
ACE_module5/Admin# show tech-support > ?

<File> Name of file to redirect stdout.
disk0: Enter the URI to redirect the output.
ftp: Enter the URI to redirect the output.
sftp: Enter the URI to redirect the output.
tftp: Enter the URI to redirect the output.
volatile: Enter the URI to redirect the output.
Verifying the ACE Image 5

Capturing Packets in Real Time

Capturing packets (sometimes referred to as a "TCP dump") is a useful aid in troubleshooting connectivity problems with the ACE
or for monitoring suspicious activity. The ACE can track packet information for network traffic that passes through the ACE. The
attributes of the packet are defined by an ACL. The ACE buffers the captured packets, and you can copy the buffered contents to a
file in Flash memory on the ACE or to a remote server. You can also display the captured packet information on your console or
terminal.
The ACE captures packets subject to the following guidelines:
• One capture session is used per context

• Capture is triggered at flow setup
• Capture is configured on the client interface where the flow is received
Note: Probe traffic will not hit a security ACL, so ACLs cannot control the capture of those packets. Therefore, probe traffic
cannot be captured by the packet capture utility.
If possible, you should capture packets using the ACE packet capturing utility before and after symptoms appear. Save the packet
captures to a file.
To capture packets in real time, follow these steps:
1. Create an ACL for packet capturing or use an existing ACL if it meets the packet capture requirements by entering the
following command:
ACE_module5/Admin(config)# access-list FILTER line 10 extended permit tcp any any eq www
ACE_module5/Admin# exit
2. Enter the capture command, for example:
ACE_module5/Admin# capture CAPTURE1 interface vlan 200 access-list FILTER
Note: Ensure that the ACL you specify in the capture command is for an input interface. If you configure the packet capture
on the output interface, the ACE will fail to match any packets.
3. Display the capture status to determine the capture status and the buffer size by entering the following command:
ACE_module5/Admin# show capture CAPTURE1 status
Capture session : TEST

Buffer size : 64 K
Circular : no
Buffer usage : 0.00%
Status : stopped
Notice that the capture has not started yet. The default buffer size is 64 KB. You can specify a maximum of buffer size of 5000
KB and you can specify a circular buffer.
4. Start the packet capture on the ACE by entering the following command:
ACE_module5/Admin# capture CAPTURE1 start

11:56:15.354930 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 62: 209.165.201.10.4144 > 172.16.1.100.80: S [bad tcp cksum 2ae
11:56:15.355257 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 62: 209.165.201.10.4144 > 192.168.1.11.80: S [bad tcp cksum c6
11:56:15.355669 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 58: 192.168.1.11.80 > 209.165.201.10.4144: S [tcp sum ok] 118
11:56:15.355979 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 58: 172.16.1.100.80 > 209.165.201.10.4144: S [bad tcp cksum 641
11:56:15.356442 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 1
11:56:15.356839 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9b
11:56:15.357203 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 494: 209.165.201.10.4144 > 172.16.1.100.80: P [tcp sum ok] 1:44
11:56:15.357918 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 494: 209.165.201.10.4144 > 192.168.1.11.80: P [bad tcp cksum 9
11:56:15.358436 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack
11:56:15.358582 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641
11:56:15.358822 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 272: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 1:
11:56:15.359106 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 272: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 64
11:56:15.359391 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 407: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 21
11:56:15.359751 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 407: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 64
11:56:15.360101 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: F [tcp sum ok] 572
11:56:15.360238 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: F [bad tcp cksum 641
11:56:15.360973 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: F [tcp sum ok] 441:4
11:56:15.361130 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: F [bad tcp cksum 9b
11:56:15.361290 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack
11:56:15.361436 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641
ACE_module5/Admin# capture CAPTURE1 stop
5. Copy the packet capture to disk0: by entering the following command:
ACE_module5/Admin# copy capture CAPTURE1 disk0:CAPTURE1
You can also copy the packet capture to an FTP, SFTP, or TFTP server.
6. Display the messages and connections within a packet capture by entering the following command:
ACE_module5/Admin# show capture CAPTURE1

0001: msg_type: ACE_HIT ace_id: 637 action_flag: 0x3
0002: msg_type: CON_SETUP con_id: 1308623156 out_con_id: 167772463
0003: msg_type: PKT_RCV con_id: 1308623156 other_con_id: 0
0004: msg_type: PKT_XMT con_id: 167772463 other_con_id: 0
<snip>
0027: msg_type: CON_CLOSE con_id: 167772463 reason: 0
0028: msg_type: CON_CLOSE con_id: 1308623156 reason: 0
7. Display the details of each packet within a capture by entering the following command:
ACE_module5/Admin# show capture CAPTURE1 detail

0001: msg_type: ACE_HIT
ace_id: 637 action_flag: 0x3
src_addr: 209.165.201.10 src_port: 4144
dst_addr: 172.16.1.100 dst_port: 80
l3_protocol: 0 l4_protocol: 6
message_hex_dump:
0x0000: 0006 0104 0000 027d 0000 0000 d1a5 c90a .......}........
0x0010: ac10 0164 0609 0013 1030 0050 0000 0000 ...d.....0.P....
0x0020: 0052 0000 05b4 0000 0000 027d 0300 0000 .R.........}....
0x0030: 0000 0040 0000 0000 0000 0000 0000 0000 ...@............
0x0040: 0000 0000 0000 0001 ........
0002: msg_type: CON_SETUP

con_id: 1308623156 out_con_id: 167772463
src_addr: 209.165.201.10 src_port: 4144
dst_addr: 172.16.1.100 dst_port: 80
Capturing Packets in Real Time 7

l3_protocol: 0 l4_protocol: 6
message_hex_dump:
0x0000: 0006 0101 4e00 0134 0a00 012f 0000 0000 ....N..4.../....
0x0010: d1a5 c90a ac10 0164 06e9 0013 1030 0050 .......d.....0.P
0x0020: e5b3 f25e 0012 0000 05b4 0100 0a00 012f ...^.........../
0x0030: 0000 0000 0018 0480 2445 0000 0000 0001 ........$E......
0x0040: 0000 0030 faf0 0010 05b4 0000 3008 e685 ...0........0...
0x0050: 0000 0000 e1ad 6b69 0000 0000 0000 027d ......ki.......}
0x0060: 0000 0000 e1ad 6b69 0000 0000 0000 0000 ......ki........
0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0080: c0a8 010b d1a5 c90a 06a1 0018 0050 1030 .............P.0
0x0090: 1a4c 0da2 0055 0000 05b4 0100 4e00 0134 .L...U......N..4
0x00a0: 0000 0000 0018 0480 2445 0022 0000 0000 ........$E."....
0x00b0: 0000 0000 0000 0000 05b4 0000 0000 0000 ................
0x00c0: 4a54 f427 e1ad 6b6b 0000 0000 0000 027d JT.'..kk.......}
0x00d0: 0000 0000 e1ad 6b6b 0000 0000 0000 0000 ......kk........
0x00e0: 0000 0000 0000 0000 0000 0000 ............
0003: msg_type: PKT_RCV

con_id: 1308623156 other_con_id: 0
message_hex_dump:
0x0000: 0500 0050 0050 8034 0008 0014 0010 1488 ...P.P.4........
0x0010: 0020 000b fcfe 1b01 001c f909 1800 0800 ................
0x0020: 4500 0030 0933 4000 7f06 aa70 d1a5 c90a E..0.3@....p....
0x0030: ac10 0164 1030 0050 3008 e684 e5b3 f25e ...d.0.P0......^
0x0040: 7002 faf0 18fd 0000 0204 05b4 0101 0101 p...............
0004: msg_type: PKT_XMT

message_hex_dump:
0x0000: 4010 0050 0050 8034 0000 0028 0000 0088 @..P.P.4...(....
0x0010: 0004 000c 29f3 cde6 000b fcfe 1b01 0800 ....)...........
0x0020: 4500 0030 0933 4000 7f06 aa70 d1a5 c90a E..0.3@....p....
0x0030: c0a8 010b 1030 0050 4a54 f426 0000 0000 .....0.PJT.&....
0x0040: 7002 faf0 18fd 0000 0204 05b4 0101 0101 p...............

message_hex_dump:
0x0000: 0500 004c 0050 8034 0008 0028 0010 2888 ...L.P.4...(..(.
0x0010: 0020 0018 b9a6 890d 000c 29f3 cde6 0800 ..........).....
0x0020: 4500 002c 0000 4000 4006 de68 c0a8 010b E..,..@.@..h....
0x0030: d1a5 c90a 0050 1030 46ca 2127 4a54 f427 .....P.0F.!'JT.'
0x0040: 6012 16d0 6df5 0000 0204 05b4 `...m.......

message_hex_dump:
0x0000: 4010 004c 0050 8034 0000 0014 0000 0088 @..L.P.4........
0x0010: 0004 001c f909 1800 000b fcfe 1b01 0800 ................
0x0020: 4500 002c 0000 4000 4006 de68 ac10 0164 E..,..@.@..h...d
0x0030: d1a5 c90a 0050 1030 2c7e 1385 3008 e685 .....P.0,~..0...
0x0040: 6012 16d0 6df5 0000 0204 05b4 `...m.......

message_hex_dump:
0x0000: 0500 004a 0050 8034 0008 0014 0010 1488 ...J.P.4........
0x0010: 0020 000b fcfe 1b01 001c f909 1800 0800 ................
0x0020: 4500 0028 0934 4000 7f06 aa77 d1a5 c90a E..(.4@....w....
0x0030: ac10 0164 1030 0050 3008 e685 2c7e 1386 ...d.0.P0...,~..
0x0040: 5010 faf0 05ad 0000 0000 P.........

message_hex_dump:
Capturing Packets in Real Time 8

0x0000: 4010 004a 0050 8034 0000 0028 0000 0088 @..J.P.4...(....
0x0010: 0004 000c 29f3 cde6 000b fcfe 1b01 0800 ....)...........
0x0020: 4500 0028 0934 4000 7f06 aa77 d1a5 c90a E..(.4@....w....
0x0030: c0a8 010b 1030 0050 4a54 f427 46ca 2128 .....0.PJT.'F.!(
0x0040: 5010 faf0 05ad 0000 0000 P.........

message_hex_dump:
0x0000: 0500 0200 0050 8034 0008 0014 0010 1488 .....P.4........
0x0010: 0020 000b fcfe 1b01 001c f909 1800 0800 ................
0x0020: 4500 01e0 0935 4000 7f06 a8be d1a5 c90a E....5@.........
0x0030: ac10 0164 1030 0050 3008 e685 2c7e 1386 ...d.0.P0...,~..
0x0040: 5018 faf0 a0bb 0000 4745 5420 2f73 6d61 P.......GET./sma
0x0050: 6c6c 2e68 746d 6c20 4854 5450 2f31 2e31 ll.html.HTTP/1.1
0x0060: 0d0a 486f 7374 3a20 3137 322e 3136 2e31 ..Host:.172.16.1
0x0070: 2e31 3030 0d0a 5573 6572 2d41 6765 6e74 .100..User-Agent

message_hex_dump:
0x0000: 4010 0200 0050 8034 0000 0028 0000 0088 @....P.4...(....
0x0010: 0004 000c 29f3 cde6 000b fcfe 1b01 0800 ....)...........
0x0020: 4500 01e0 0935 4000 7f06 a8be d1a5 c90a E....5@.........
0x0030: c0a8 010b 1030 0050 4a54 f427 46ca 2128 .....0.PJT.'F.!(
0x0040: 5018 faf0 a0bb 0000 4745 5420 2f73 6d61 P.......GET./sma
0x0050: 6c6c 2e68 746d 6c20 4854 5450 2f31 2e31 ll.html.HTTP/1.1
0x0060: 0d0a 486f 7374 3a20 3137 322e 3136 2e31 ..Host:.172.16.1
0x0070: 2e31 3030 0d0a 5573 6572 2d41 6765 6e74 .100..User-Agent
<snip>
Note: If you view the ACE capture file in a third-party sniffer (for example, Wireshark), you will notice only the messages or
type PKT_RCV and PKT_XMT are displayed. This situation is expected because the sniffer is not aware of the ACE's
internal messaging.
Copying Core Dumps
If the ACE fails with a core dump, the core dump files may contain useful information. The core dump files reside in the core:
directory. To view the contents of the core: directory, enter the following command:
ACE_module5/Admin# dir core:
123589 Feb 22 00:34:20 2009 qnx_1_mecore_log.999.tar.gz

30361 Feb 22 00:34:22 2009 ixp1_crash.txt
Usage for core: filesystem

153950 bytes total used
202943138 bytes free
203097088 total bytes
You can copy the contents of the core: directory to several locations by using the copy core: command. The syntax of this
command is as follows:
copy {core:filename | disk0:[path/]filename | running-config | startup-config} {ftp://server/path[/filename] |

sftp://[username@]server/path[/filename] | tftp://server[:port]/path[/filename]}
The ACE provides core dumps for both the control plane and the data plane. Each core dump file contains the following
information:
• Version
• Time of failure
Copying Core Dumps 9

• Number of CPUs
• Current CPU
• BKL status
• IRQ lock status
• Buffers
After Gathering Troubleshooting Information

After you have gathered all the above information, be prepared to send the information to your customer service representative or
TAC. You can send the information in the following ways:
• FTP
• SFTP
• TFTP
Verifying the Physical Connectivity Between the ACE and the End
Hosts
To verify the physical connectivity of the ACE, follow these steps:
1. Check all cable connections on the Catalyst 6500 series switch or Cisco 7600 series router that may impact the ACE.
2. Use the extended ping command to send an ICMP Echo request to the end devices.
ACE_module5/Admin# ping
Target IP address: 10.1.1.2
Repeat count [5]: 4
Datagram size [100]: 200
Timeout in seconds [2]: 10
Extended commands [n]: 4
Pinging 10.1.1.2 with timeout = 10, count = 4, size = 200 ....
Response from 10.1.1.2 : seq 1 time 0.494 ms

4 packet sent, 4 responses received, 0% packet loss
If a host is one hop away and you are unable to reach the host, then ping the intermediary gateway. If the gateway is not reachable,
enter the show ip route command and check to make sure that the correct route is displayed. For example, enter:
ACE_module5/Admin# show ip route
Routing Table for Context Admin (RouteId 0)
Codes: H - host, I - interface

S - static, N - nat
A - need arp resolve, E - ecmp
Destination Gateway Interface Flags

------------------------------------------------------------------------
0.0.0.0 10.2.2.1 vlan130 S [0xc]
10.2.2.0/24 0.0.0.0 vlan130 IA [0x30]
172.27.15.0/24 0.0.0.0 vlan100 IA [0x30]
172.27.16.0/24 0.0.0.0 vlan200 IA [0x30]
172.19.110.0/26 0.0.0.0 vlan55 IA [0x30]
172.27.16.16/29 0.0.0.0 vlan200 N [0x280]
172.27.16.33/32 0.0.0.0 vlan100 N [0x280]
Total route entries = 7
If necessary, enter a static route for the gateway.
Verifying the ACE Layer 2 Connectivity

To verify the Layer 2 connectivity of the ACE, follow these steps:
1. Verify that the ARP table is populated with the IP addresses and corresponding MAC addresses of the ACE, the gateway, the
local interface, and other IPs that the ACE has learned.
switch/Admin# show arp
Context Admin
================================================================================
IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status
================================================================================
10.86.215.208 00.02.7e.39.51.9c vlan130 LEARNED 5 2350 sec up
10.86.215.228 00.e0.81.22.78.ff vlan130 LEARNED 6 6379 sec up
10.86.215.234 00.1a.a1.48.f3.44 vlan130 LEARNED 4 1114 sec up
10.86.215.1 00.00.0c.07.ac.00 vlan130 GATEWAY 2 153 sec up
10.86.215.2 00.11.5d.e1.2f.fc vlan130 LEARNED 3 12054 sec up
10.86.215.134 00.18.b9.a6.91.15 vlan130 INTERFACE LOCAL _ up
================================================================================
Total arp entries 6
2. Verify that the ACE is connected to the switch fabric of the Catalyst 6500 series switch or the Cisco 7600 series router. The
ACE uses a 10-Gigabit Ethernet switch fabric interface (SFI) to connect to the chassis backplane as opposed to the CSM, which
uses a port channel. The ACE uses the following format for this interface:
Te<slot>/1
For example, if the ACE is in slot 5, you can see the status of the backplane connection by entering the following command on the
Catalyst 6500 series switch or the Cisco 7600 series router:
cat6k# show interface te5/1 status
Port Name Status Vlan Duplex Speed Type

Te5/1 connected trunk full 10G MultiService Module
If there is no output from this command, then either the ACE is not installed properly or the ACE is powered down.
3. Verify the association of the ACE MAC entries with the allocated VLAN interfaces. Enter the following command at the
Supervisor CLI:
cat6k# show mac-address-table dynamic

Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports

------+----------------+--------+-----+----------+--------------------------
.
.
.
* 130 0018.b9a6.9115 dynamic Yes 40 Te5/1 <------- MAC address should be in the range displayed by
.
.
.
Verifying the Physical Connectivity Between the ACE and the EndHosts 11
cat6k# show module 5
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S
Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------
5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- MAC address range assigned to
Mod Online Diag Status

---- -------------------
5 Pass
4. Check the status of the Te5/1 port to ensure that it is in the forwarding state by entering the following command:
cat6k# show spanning-tree vlan 130
MST0
Spanning tree enabled protocol mstp
Root ID Priority 32768
Address 0001.632f.2c17
Cost 200019
Port 642 (GigabitEthernet6/2)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)

Address 0011.bc06.f800
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------
Gi2/14 Desg FWD 20000 128.142 P2p
Gi2/37 Desg FWD 200000 128.165 P2p
Te3/1 Desg FWD 2000 128.257 Edge P2p
Gi6/2 Root FWD 200000 128.642 Shr Bound(STP)
Verifying the ACE Layer 3 Connectivity

Use the traceroute command to check the route between the ACE and the end devices.
ACE_module5/Admin# traceroute 10.20.12.153

traceroute to 10.20.12.153 (10.20.12.153), 30 hops max, 40 byte packets
1 10.20.215.2 (10.20.215.2) 0.532 ms 0.436 ms 0.362 ms
2 10.20.239.161 (10.20.239.161) 0.421 ms 0.488 ms 0.404 ms
3 10.20.238.93 (10.20.238.93) 0.471 ms 0.422 ms 0.413 ms
4 172.27.16.177 (172.27.16.177) 0.488 ms 0.435 ms 0.430 ms
5 172.27.16.226 (172.27.16.226) 0.474 ms 0.363 ms 0.368 ms
6 192.168.0.134 (192.168.0.134) 0.624 ms 0.510 ms 0.494 ms
7 10.20.12.153 (10.20.12.153) 23.982 ms 24.702 ms 25.976 ms
Contacting Cisco Technical Support

If you are unable to resolve a problem after using the troubleshooting suggestions in the articles in this wiki, contact the Cisco
Technical Assistance Center (TAC) for assistance and further instructions. Before you call, have the following information ready
to help your TAC engineer assist you as quickly as possible:
Verifying the ACE Layer 2 Connectivity 12

• Date that you received the ACE

• Chassis serial number (located on a label on the right side of the rear panel of the chassis)
• Type of software and release number (if possible, enter the show version command)
• Maintenance agreement or warranty information
• Brief description of the problem
• Brief explanation of the steps that you have already taken to isolate and resolve the problem
For information on steps to take before calling Technical Support, see the "Gathering ACE Troubleshooting Information" section.
You can reach TAC in several ways as follows:
• Create a service request online
• Call the TAC at the telephone numbers on this page.
• Contact the Cisco Small Business Support Center
This article describes the ACE architecture and how data flows into, gets processed, and flows out of the ACE. It provides a basic
understanding of these concepts to assist you in troubleshooting the ACE.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Understanding the ACE Architecture
♦ 1.1 Overview of the ACE
Hardware Architecture
♦ 1.2 Control Plane
Contacting Cisco Technical Support 13

♦ 1.3 Data Plane

◊ 1.3.1 Classification and
Distribution Engine
◊ 1.3.2 Network Processors
◊ 1.3.3 SSL Crypto Module
• 2 Understanding the ACE Traffic Flow
♦ 2.1 To-the-ACE Traffic
♦ 2.2 Through-the-ACE Traffic
Understanding the ACE Architecture

Having a basic understanding of the ACE architecture and data flow can help to make troubleshooting the ACE easier. This
section describes the major functional areas of the ACE and how they work together.
Overview of the ACE Hardware Architecture

The ACE hardware architecture is divided into a series of functional areas or subsystems that are defined by processors or groups
of processors and interfaces as shown in Figure 1.
Figure 1. ACE Module Architecture
Contents 14
A console connection allows direct access to the ACE control plane (CP) for initial configuration, management, and
troubleshooting. The supervisor engine connection allows you to determine the status of the ACE, to load images into the ACE, to
reboot the ACE, and to provide remote access to the ACE from the Catalyst 6500 series switch or Cisco 7600 series router when
you use the session command. Because the ACE has no external ports, packets enter the ACE through the Switch Fabric Interface
(SFI) connected to the Catalyst 6500 series switch or Cisco 7600 series router back plane. The two major functional areas of the
ACE are as follows:
• Control plane
• Data plane
Control Plane
The control plane (CP) is used to configure the ACE and for management traffic, syslogs, ARP, DHCP, and so on. You can access
the CP directly by using the console port. For remote management, you must configure a management interface and enable remote
access using a management policy to permit Telnet or SSH access, for example. The CP is responsible for the following ACE
functions:
• Device management and control

• Configuration management (CLI or XML interface)
• Server health monitoring
• syslogs
• SNMP
• Address Resolution Protocol (ARP)
• DHCP relay
• Redundancy (also known as high availability or fault tolerance)
• Access control list (ACL) compilation
Overview of the ACE Hardware Architecture 15

Data Plane
The data plane (DP) is responsible for distributing and processing packets and connections that do not match a management
policy.
In the ACE, the CP and the DP are separated and run on different processors for maximum performance. See Figure 2.
Figure 2. ACE Data Plane
The DP is responsible for the following ACE functions:
• Access control lists (ACLs)

• Connection management
• TCP termination
• Network address translation (NAT)
• SSL processing (termination, initiation, encryption, and decryption)
• Regular expression matching
• Load balancing and forwarding
• Application protocol inspection
The DP consists of the following functional areas:
• Classification distribution engine (CDE)
Data Plane 16
• Network processors (NPs)
• SSL Crypto Module
• Daughter card interfaces (for future feature expansion)
Classification and Distribution Engine
The Classification and Distribution Engine (CDE) is the traffic controller for the ACE. Its main purpose is to forward packets that
it receives from the SFI to the two network processors (NPs). It also acts as the central point of contact among all the major
subsystems within the ACE. The CDE computes, and if necessary, adjusts the IP, TCP, and UDP checksums of every packet that
it receives.
The CDE appends a special header known as the IMPH header to each packet before sending it to the fast path. The IMPH header
is 18 bytes long and contains information from the DBUS header (the header sent to the ACE Module by the Catalyst 6500 series
switch or Cisco 7600 series router) as well as special messaging directly understood by the fast path. Fields in the IMPH header
can include notification of a checksum error, Layer 3 or Layer 4 offsets, source and destination ports of the CDE, the VLAN for
determining the interface that the fast path will use, and so on.
Network Processors
The ACE has two network processors (NP1 and NP2) that perform most of the packet processing in the ACE. All traffic entering
the ACE must traverse one or both NPs after being forwarded by the CDE. Each NP contains a CPU (XScale) and several
components called microengines (MEs). See Figure 3.
Figure 3. Network Processor Micorengines
Each microengine can handle eight simultaneous threads or processes and performs a specific function for the NP as follows:
• Receive - One ME for receiving incoming packets

• Fast Path - Four MEs for the hardware accelerated data path that is used for MAC rewrite, NAT, TCP normalization, and
so on (essentially all operations performed on a per-packet basis)
• ICM - One ME for the inbound connection manager
• OCM - One ME for the outbound connection manager
• CCM - One ME for the connection close manager
• TCP - Two MEs for TCP termination with a full TCP stack
• HTTP - Two MEs for HTTP parsing
• Unused (future expansion) - One ME
• SSL Record Layer - One ME for the SSL record layer
• IP fragmentation timers - One ME for IP fragmentation reassembly and timer management
• DNS and ICMP Inspection - One ME for DNS and ICMP packet inspection
The XScale microprocessor is programmed to handle the following features:
• Load-balancing algorithms
• SSL handshake
• FTP and Real-Time Streaming Protocol (RTSP) inspection
• HTTP inspection (although a considerable part is performed by the microengines)
• High-availability heartbeat generation
• Returned statistics for most connection-related commands
Each network processor has RDRAM memory to store ACL entries, routing table entries, ARP entries, and inspection policies.
Additional SRAM memory provides faster access times and is used to store regular expressions and statistics on a per?virtual
system basis, among other things.
SSL Crypto Module
The SSL Crypto Module is responsible for SSL record layer processing. This processing includes encrypting and decrypting data
for SSL flows.
Understanding the ACE Traffic Flow

Because the ACE has no native ports, it relies on the switch fabric in the Catalyst 6500 series switch or the Cisco 7600 series
router back plane to send and receive packets to and from the network. Packets that are marked with a destination VLAN and
Layer 2 information enter the ACE through the SFI on the 10 Gbps Ethernet link. Packets entering or leaving the ACE traverse
this link using VLAN tagging. The switch fabric interface (SFI) forwards to the CDE all packets that are destined to the ACE. See
Figure 1.
The CDE classifies the packets based on the configured traffic policies, fills out the IMPH header information, and forwards the
traffic to one of the NPs. To determine which NP to forward the traffic to, the CDE hashes incoming packets based on the traffic
type as follows:
• TCP/UDP - Hash of source/destination port

• Non-TCP/UDP IP - Hash of source/destination IP address
• Non-IP - Hash of source/destination Layer 2 MAC address
The NPs process the traffic and forward it back through the CDE to either the control plane or the SSL Crypto Module for further
processing.
To-the-ACE Traffic
To-the-ACE traffic is traffic that is destined to an interface VLAN IP address on the ACE. This traffic must match a class map of
type management, which is associated with a policy map and applied as a service policy on an interface VLAN. The management
class map supports the following protocols:
• Hypertext Transfer Protocol (HTTP)

• Hypertext Transfer Protocol Secure (HTTPS)
• Internet Control Message Protocol (ICMP)
• Keepalive Application Protocol (KAL-AP)
• User Datagram Protocol (UDP)
• Simple Network Management Protocol (SNMP)
• Secure Shell (SSH) Protocol
• Telnet
This management traffic is called control plane traffic because it is destined to the CP. Because of the separation of the CP traffic
from the data plane traffic on different processors, the control plane traffic will never interfere with data plane traffic, even if the
control plane is oversubscribed.
Through-the-ACE Traffic
The CDE sends traffic that requires load balancing, forwarding, routing, or other processing by the ACE to one of the NPs.
The NPs comprise two parallel forwarding paths that maintain their own connection state information and forward traffic
independently.
This article describes some basic troubleshooting steps that you can perform to rule out some of the simpler issues before delving
deeper into the troubleshooting process.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
To-the-ACE Traffic 19
Contents
• 1 Preliminary ACE Troubleshooting Steps
• 2 Checking the ACE Status from the Supervisor Engine
• 3 Verifying the MSFC VLAN Configuration
• 4 Establishing a Session with the ACE from the Supervisor
Engine
• 5 Verifying the ACE is Receiving VLAN Allocations from the
MSFC
• 6 Verifying the ACE Image
• 7 Verifying Your ACE Licenses
• 8 Configuring an ACL to Permit Input Traffic to the ACE
• 9 Verifying that the ACE is Sending and Receiving Traffic
• 10 Verifying to-the-ACE Traffic
Preliminary ACE Troubleshooting Steps

1. Check the status of the ACE module from the Catalyst 6500 series switch or Cisco 7600 series router. See the "Checking
the ACE Status from the Supervisor Engine" section.
2. Verify that you have allocated the correct VLANs to the ACE in the Multilayer Switch Feature Card (MSFC) VLAN
configuration. See the "Verifying the MSFC VLAN Configuration" section.
3. Verify that you can establish a session to the ACE from the supervisor engine. See the "Establishing a Session with the
ACE from the Supervisor Engine" section.
4. Verify that the ACE is receiving VLAN allocations from the MSFC. See the Verifying the ACE is Receiving VLAN
Allocations from the MSFC" section.
5. Verify your ACE bandwidth, SSL, and virtualization licenses. See the "Verifying Your ACE Licenses" section.
6. Verify that you have configured an access control list (ACL) to permit traffic on the interfaces on which you wish the
ACE to receive traffic. If you do not configure an ACL to permit traffic on an interface, all traffic destined to that
interface will be blocked by the ACE. See the "Configuring an ACL to Permit Input Traffic to the ACE" section.
7. Verify that the ACE is sending and receiving traffic. See the "Verifying that the ACE is Sending and Receiving Traffic"
section.
8. Verify the management traffic to the control plane. See the "Verifying to-the-ACE Traffic"section.
Checking the ACE Status from the Supervisor Engine

Before you begin to troubleshoot your ACE, Telnet to the Catalyst 6500 series switch or Cisco 7600 series router supervisor
engine, log in, and check the status of the ACE.
Contents 20
telnet 10.1.1.2
User Access Verification
Password:
cat6k> enable
Password:
cat6k# show module
--- ----- -------------------------------------- ------------------ -----------
.
.
5 1 Application Control Engine Module ACE20-6500-K9 SAD1031044S
.
.

--- ---------------------------------- ------ ------------ ------------ -------
.
.
5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- ACE status
.
.
Mod Sub-Module Model Serial Hw Status

--- --------------------------- ------------------ ----------- ------- -------
6 Policy Feature Card 3 WS-F6K-PFC3A SAL09094NUB 2.5 Ok
6 MSFC3 Daughterboard WS-SUP720 SAL09094N33 2.5 Ok

--- -------------------
.
.
5 Pass
.
.
Verifying the MSFC VLAN Configuration

To verify that the VLANs that you intend to use in your ACE have been configured and allocated to the ACE in the MSFC, follow
these steps:
1. Check the VLANs configured and allocated to the ACE by entering the following command from the supervisor
engine:
cat6k# show run | include svclc
svclc module 5 vlan-group 123,130,133
2. Ensure that the VLAN groups that you intend to use for your ACE are allocated properly in the MSFC configuration by
entering the following commands:
cat6k# show svclc module 5 vlan-group

Module Vlan-groups
------ -----------
05 123,130,133
cat6k# show svclc vlan-group

Display vlan-groups created by both ACE module and FWSM commands
Checking the ACE Status from the Supervisor Engine 21

Group Created by vlans

----- ---------- -----
123 ACE 103,105,107,111-112,119,134,160,171,200,203,205,207,211-212,226,253,260
130 ACE 130
133 ACE 100,194,221,256-257
3. Verify that the VLANs you intend to use in your ACE are configured in the MSFC by entering the following command:
cat6k# show interface te5/1 trunk
Port Mode Encapsulation Status Native vlan

Te5/1 on 802.1q trunking 1
Port Vlans allowed on trunk

Te5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257,260
Port Vlans allowed and active in management domain

Te5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257
Port Vlans in spanning tree forwarding state and not pruned

Te5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257
4. Ensure that traffic is routed to two ACEs in the same chassis when both client- and server-side VLANs are configured
as switched virtual interfaces (SVIs) on the MSFC in routed mode by entering the following command:
cat6k# show svclc multiple-vlan-interfaces

Multiple ACE vlan interfaces feature is enabled
Establishing a Session with the ACE from the Supervisor Engine

To verify that you can establish a session with the ACE from the supervisor engine in the Catalyst 65000 series switch or the
Cisco 7600 series router, enter the following command:
cat6k# session slot 5 processor 0

The default escape character is Ctrl-^, then x.
You can also type 'exit' at the remote prompt to end the session
Trying 127.0.0.50 ... Open
ACE_module5 login:
Verifying the ACE is Receiving VLAN Allocations from the MSFC

Ensure that the VLAN that you intend to use on ACE are allocated properly in the MSFC configuration by entering the following
command:
ACE-1/Admin# show vlans

Vlans configured on SUP for this module
vlan123 vlan130 vlan133
If interface VLANs are already assigned on the ACE you can use the show interface vlan <num> command to verify the
interface is properly assigned on the MSFC and up on the MSFC:
Verifying the MSFC VLAN Configuration 22

ACE-1/Admin# show interface vlan 123
vlan10 is up
Hardware type is VLAN
MAC address is 00:18:b9:a6:89:0d
Mode : routed
IP address is 10.10.10.1 netmask is 255.255.255.0
FT status is non-redundant
Description:not set
MTU: 1500 bytes
Last cleared: never
Alias IP address not set
Peer IP address not set
Assigned from the Supervisor, up on Supervisor
7101679 unicast packets input, 878043707 bytes
0 multicast, 0 broadcast
0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
6387914 unicast packets output, 1541924399 bytes
0 output errors, 0 ignored
Verifying the ACE Image

To display the version of the software image and the image filename that is currently running in your ACE, enter the following
command:
ACE_module5/Admin# show version

Software
loader: Version 12.2[121]
system: Version A2(2.0) [build 3.0(0)A2(2.0)] <--------
system image file: [LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin <--------
installed license: no feature license is installed
Hardware
Cisco ACE (slot: 5)
cpu info:
number of cpu(s): 2
cpu type: SiByte
memory info:
total: 955396 kB, free: 289704 kB
shared: 0 kB, buffers: 2336 kB, cached 0 kB
cf info:
filesystem: /dev/cf
total: 1000000 kB, used: 494912 kB, available: 505088 kB
last boot reason: NP 1 Failed : NP ME Hung
configuration register: 0x1

ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)
This command provides other useful information, for example:
Verifying the ACE is Receiving VLAN Allocations from the MSFC 23

• Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5)
• Available memory
• Last boot reason
• Configuration register (confreg) value
• ACE uptime
Verifying Your ACE Licenses

Log in to your ACE and enter the following command to display the SSL, virtualization, and bandwidth licenses that are currently
installed and in use in your ACE:
ACE_module5/Admin# show license usage

License Ins Lic Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
ACE-08G-LIC No - Unused -
ACE-16G-LIC No - Unused -
ACE-UPG1-LIC No - Unused -
ACE-UPG2-LIC No - Unused -
ACE-VIRT-020 No - Unused -
ACE-VIRT-250 Yes 1 In use never -
ACE-VIRT-UP1 No - Unused -
ACE10-16G-LIC No - Unused -
ACE-SEC-LIC-K9 No - Unused -
ACE-SSL-05K-K9 No - Unused -
ACE-SSL-UP1-K9 No - Unused -
ACE_module5/Admin# show license status

Licensed Feature Count
------------------------------ -----
SSL transactions per second 1000
Virtualized contexts 250
Module bandwidth in Gbps 4
You can also see the licenses that reside on the Flash disk by entering the following command:
ACE_module5/Admin# dir disk0:

236 Oct 17 09:18:26 2006 ACE-SSL-05K-K9.lic <--------
235 Oct 17 09:16:58 2006 ACE-VIRT-250.lic <--------
1024 Sep 28 19:11:11 2006 cv/
1654606 Oct 26 12:56:16 2006 dplug
Usage for disk0: filesystem

8405120 bytes free
In the above example, there is an SSL 5K TPS license on the Flash disk that has not yet been installed in the ACE.
To install the license, enter the following command:
ACE_module5/Admin# license install disk0:ACE-SSL-05K-K9.lic

Installing license... done
ACE_module5/Admin#
Configuring an ACL to Permit Input Traffic to the ACE

You must configure an ACL to allow the ACE to receive traffic. All traffic to the ACE is blocked until you do so. For example, to
configure an ACL that permits all IP trafffic except from the 10.1.1.0 network, enter the following commands:
ACE_module5/Admin(config)# access-list ACL1 extended deny ip 10.1.1.0 255.255.255.0 any

ACE_module5/Admin(config)# access-list ACL1 extended permit ip any any
ACE_module5/Admin(config)# interface vlan 100
ACE_module5/Admin(config-if)# access-group input ACL1
Verifying that the ACE is Sending and Receiving Traffic

You can tell if traffic is reaching the ACE by using the show svclc module number traffic command on the Catalyst 6500 series
switch or Cisco 7600 series router. This command displays counters (packets input and packets output) that increase when the
switch or router sends packets to or receives packets from the ACE.
cat6k# show svclc module 5 traffic

ACE module 5:
Specified interface is up line protocol is up (connected)

Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s
input flow-control is on, output flow-control is unsupported
Last input never, output never, output hang never
Last clearing of "show interface" counters 4d02h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
528888 packets input, 41329093 bytes, 0 no buffer <-------
Received 469945 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
7776 packets output, 746361 bytes, 0 underruns <-------
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
A trace of the Te?/1 (10-Gbps switch fabric interface, where ? = the module number) interface will show you whether packets are
arriving at the switch fabric interface (SFI).
Another useful command is the show cde health command on the ACE. This command shows the current state of the
Classification Distribution Engine (CDE). The network processors (NP1 and NP2) are represented by IXP0 and IXP1,
respectively. You should not observe any drops, errors, or flow control issues in the output of this command. If the Packets
Received or the Packets Transmitted counters of the CDE Hyperion Interface are not increasing, then packets are not coming into
or going out of the ACE.
ACE_module5/Admin# show cde health
CDE BRCM INTERFACE

======================
Packets received 4933
Packets transmitted 2437922
Broadcom interface CRC error count 0
BRCM VOQ status [empty] [not full]
BRCM pull status [pulling]
CDE HYPERION INTERFACE

======================
Packets received 29913371 <-------
Packets transmitted 8034 <-------
Short packets drop count 0
Fifo Full drop count 0
Protocol error drop count 0
FCS error drop count 0
CRC error drop count 0
Num times flow control triggered on hyp interface 0
Num self generated multicast packets filtered 1880
HYP IXP0 VOQ status [empty] [not full]
HYP SLOW VOQ status [empty] [not full]
HYP tx pull status [pulling]
CDE IXP0 INTERFACE

======================
Num bad pkts recvd on fast spi channel0 0
Num bad pkts recvd on slow spi channel8 0
IXP0 Fast VOQ status [empty] [not full]
IXP0 BRCM VOQ status [empty] [not full]
IXP0 pull status [pulling]
IXP0 spi src status [healthy]
IXP0 spi snk status [healthy]
CDE1 SWITCH1 INTERFACE

======================
Packets received (hyp, ixp0) 4415
Packets received (bcm) 1656608
Packets received (daughter card 0) 0
Packets received (daughter card 1) 0
Packets Errors received (hyp, ixp0) 0
Packets Errors received (bcm) 0
Packets Errors received (daughter card 0) 0
Packets Errors received (daughter card 1) 0
Packets transmitted (ixp1) 2089360
Packets transmitted (nitrox) 0
Packets Errors transmitted (ixp1) 0
Packets Errors transmitted (nitrox) 0
CDE2 SWITCH2 INTERFACE

======================
Packets received (ixp1) 2089360
Packets received (nitrox) 0
Packets Errors received (ixp1) 0
Packets Errors received (nitrox) 0
Packets transmitted (hyp, ixp0) 4415
Packets transmitted (broadcom) 1656608
Packets transmitted (daughter card 0) 0
Packets transmitted (daughter card 1) 0
Packets Errors transmitted (ixp1) 0
Packets Errors transmitted (nitrox) 0
Packets Errors transmitted (daughter card 0) 0
Packets Errors transmitted (daughter card 1) 0
CDE IXP1 INTERFACE

======================
IXP1 Fast VOQ status [empty] [not full]
IXP1 BRCM VOQ status [empty] [not full]
IXP1 pull status [pulling]
IXP1 spi src status [healthy]
IXP1 spi snk status [healthy]
CDE NITROX INTERFACE

======================
Packets received 0
NTX Fast VOQ status [empty] [not full]
NTX BRCM VOQ status [empty] [not full]
NTX pull status [pulling]
NTX spi src status [healthy]
NTX spi snk status [healthy]
== Backplane ==
ITASCA_SYS_CNTL1 0x300 data 0x61f0000
ITASCA_SYS_CNTL2 0x304 data 0x80c30000
You can also use the show interface command on the ACE to display traffic that is sent and received on the interface for each
VLAN that is configured on the ACE.
ACE_module5/Admin# show interface
bvi2 is administratively down

Hardware type is BVI
MAC address is 00:18:b9:a6:91:15
Mode : unknown
Description:not set
MTU: 1500 bytes
Last cleared: never
vlan100 is administratively down
Verifying that the ACE is Sending and Receiving Traffic 27

Mode : unknown
Description:not set
MTU: 1500 bytes
Last cleared: never
vlan130 is up
Mode : routed
Description:not set
MTU: 1500 bytes
Last cleared: never
59858 unicast packets input, 41711169 bytes <-------
193118 multicast, 280789 broadcast <-------
0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops <-------
6260 unicast packets output, 785167 bytes <-------
0 multicast, 1892 broadcast <-------
0 output errors, 0 ignored <-------
Verifying to-the-ACE Traffic

Traffic that is destined to the ACE itself arrives at the control plane in one of the following ways:
• Directly from the console connection

• Directly from the supervisor engine connection
• Traffic from the SFI that is forwarded by the CDE in the data plane
Use the following commands to verify that traffic is going to and coming from the control plane.
ACE_module5/Admin# show netio stats
High Priority (Control) Normal Priority (Data)

----------------------- ----------------------
Net Rx Packets : 224 Net Rx Packets : 2521794
Net Rx Bytes : 17528 Net Rx Bytes : 196704169
Net Rx Unsupported L2 : 0 Net Rx Unsupported L2 : 0
Net Rx Lock Errors : 0 Net Rx Lock Errors : 0
Net Rx Interface Miss : 0 Net Rx Interface Miss : 2326290
Net Rx No Arp Client : 0 Net Rx No Arp Client : 0
Net Rx Alias Drops : 0 Net Rx Alias Drops : 0
Net Rx Repl. Errors : 0 Net Rx Repl. Errors : 0
Net Rx Repl. If Err : 0 Net Rx Repl. If Errs : 0
Net Rx Internal Errs : 0 Net Rx Internal Errs : 0
Net Tx Packets : 0 Net Tx Packets : 5213
Verifying to-the-ACE Traffic 28

Net Tx Bytes : 0 Net Tx Bytes : 414073
Net Tx Lock Errors : 0 Net Tx Lock Errors : 0
Net Tx Bad Context ID : 0 Net Tx Bad Context ID : 0
Net Tx No Route Found : 0 Net Tx No Route Found : 0
Net Tx No Adjacency : 0 Net Tx No Adjacency : 1
Net Tx Invalid If ID : 0 Net Tx Invalid If ID : 0
Net Tx If Down : 0 Net Tx If Down : 0
Net Tx No Src Addr : 0 Net Tx No Src Addr : 0
Net Tx No Encap : 0 Net Tx No Encap : 0
Net Tx FIFO Errors : 0 Net Tx Fifo Errors : 0
Net Tx No VMAC Errors : 0 Net Tx No VMAC Errors : 0
IPC Tx Packets : 76 IPC Tx Packets : 0

IPC Tx Bytes : 17638 IPC Tx Bytes : 0
IPC Tx Fifo Errors : 0 IPC Tx Fifo Errors : 0
Client Rx Queue Full : 0 Client Rx Queue Full : 0
Pseudo Rx Queue Full : 0 Pseudo Rx Queue Full : 0
ACE_module5/Admin# show fifo stats

----------------------- ----------------------
Rx Packets : 224 Rx Packets : 2524886
Rx Bytes : 17528 Rx Bytes : 196952927
Rx DMA Errors : 0 Rx DMA Errors : 0
Rx Drop Events : 0 Rx Drop Events : 0
Rx Descr Errors : 0 Rx Descr Errors : 0
Rx Bad Descrs : 0 Rx Bad Descrs : 0
Rx Length Errors : 0 Rx Length Errors : 0
Tx Packets : 76 Tx Packets : 5241

Tx Bytes : 17682 Tx Bytes : 464991
Tx Drops : 0 Tx Drops : 0
Tx DMA Errors : 0 Tx DMA Errors : 0
Tx SOP Errors : 0 Tx SOP Errors : 0
Global Errors
-------------
Rx Underflows : 0
Rx Overflows : 0
Tx Underflows : 0
Tx Overflows : 0
Resets : 0
Zbuff alloc fail : 0
Interrupt Stats
---------------
Total Interrupt count : 2529603
Rx Interrupt count : 2524302
Tx interrupt count : 5310
This article describes how to troubleshoot basic ACE boot issues.
Guide Contents
Verifying to-the-ACE Traffic 29

Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Understanding ROMMON Mode and the ACE Boot
Configuration
♦ 1.1 Setting the Boot Method from the Configuration
Register
♦ 1.2 Booting the ACE from the ROMMON Prompt
♦ 1.3 Setting the BOOT Environment Variable
♦ 1.4 Displaying the ACE Boot Configuration
• 2 Restarting the ACE
♦ 2.1 Restarting the ACE from the ACE CLI
♦ 2.2 Restarting the ACE from the Supervisor Engine
• 3 Establishing a Console Connection to the ACE
• 4 Troubleshooting ACE Boot Problems
Contents 30
Understanding ROMMON Mode and the ACE Boot Configuration

You can control how the ACE performs its boot process through either the ACE configuration mode or ROM Monitor
(ROMMON) mode. ROMMON is the ROM-resident code that starts executing at power up, reset, or when a fatal exception
occurs.
Two user-configurable parameters determine how the ACE boots:
• Boot field in the configuration register (confreg)

• BOOT environment variable
The ACE enters ROMMON mode if it does not find a valid system image, if the Flash memory configuration is corrupted, or if
the configuration register is set to enter ROMMON mode.
Note: You can manually enter ROMMON mode by restarting the ACE and then pressing the Break key during the first 60
seconds of startup. If you are connected to the ACE through a terminal server, you can escape to the Telnet prompt and
then enter the send break command to enter the ROMMON mode.
Setting the Boot Method from the Configuration Register
To change the configuration register settings and how the ACE boots from the CLI, use the following configuration mode
command:
config-register value
The value argument-supported entries are as follows:
• 0?ACE boots to the ROMMON prompt. The ACE remains in ROMMON mode at startup.
• 1?ACE boots from the system image identified in the BOOT environment variable. If the ACE encounters an error or if
the image is not valid, it will try the second image (if one is specified). If the second image also fails to boot, the ACE
returns to ROMMON mode.
For example, to set configuration register to boot the system image identified in the BOOT environment variable, enter the
following command:
ACE_module5/Admin(config)# config-register 1
Booting the ACE from the ROMMON Prompt

If you specify a value of 0 for the config-register command, this configuration register setting forces the ACE to enter the
ROMMON mode upon a reload or power cycle of the ACE. The ACE remains in ROMMON mode until you identify the location
of an image file to boot.
The ACE supports two methods of booting the module from the ROMMON prompt:
• To manually change the configuration register setting in ROMMON mode, use the confreg command followed by a value
of 0 or 1.
• To change the boot characteristics using onscreen prompts, use the confreg command without a value.
To instruct the ACE to manually boot from a particular system image, use the confreg command and specify a configuration
register value of 1. Identify the name of the system image file that the ACE uses to boot.
Understanding ROMMON Mode and the ACE Boot Configuration 31

For example, to use the confreg command at the ROMMON prompt to instruct the ACE to boot from the
c6ace-t1k9-mzg.3.0.0_A2_2_0.bin system image, enter the following command:
rommon 1 > confreg 1

rommon 2 > BOOT=disk0:c6ace-t1k9-mzg.3.0.0_A2_2_0.bin
rommon 3 > sync
To instruct the ACE to automatically boot from the image specified in the BOOT variable, use the confreg command without
specifying a configuration register value to launch the Configuration Summary menu-based utility. You can then instruct the ACE
to boot from the system image identified in the BOOT environment variable. See the "Setting the BOOT Environment Variable"
section.
For example, to use the confreg command to display the onscreen prompts for changing the boot characteristics of the ACE and
change the configuration register to boot from an image on disk0:, enter the following command:
rommon 4 > confreg
Configuration Summary
(Virtual Configuration Register: 0x2000)
enabled are:
ignore system config info
console baud: 9600
boot: the ROM monitor
do you wish to change the configuration? y/n [n]: y

disable "ignore system config info"? y/n [n]:
change the boot characteristics? y/n [n]: y
enter to boot:
0 = ROM Monitor
1 = boot file specified in BOOT variable
[0]: 1
Configuration Summary
(Virtual Configuration Register: 0x2001)
enabled are:
ignore system config info
console baud: 9600
boot: the file specified in BOOT variable
do you wish to change the configuration? y/n [n]:

You must reset/power cycle for new config to take effect
rommon 7 > dir disk0:
Directory of disk0:
23951 31071143 -rw- c6ace-t1k9-mzg.A2_2_0.bin

2 74448896 -rw- TN-CONFIG
4546 32505856 -rw- TN-CERTKEY-STORAGE
6530 11534336 -rw- TN-LOGFILE
7234 11534336 -rw- TN-HOME
7938 209715200 -rw- TN-COREFILE
20738 1048576 -rw- lkcddump
22689 250 -rw- scripted_hm.txt
24584 30337516 -rw- c6ace-t1k9-mz.A2_1_1.bin
29540 1048640 -rw- ACE_FUR_BOOT_ROM.img.rel.2008Apr01_ver121
29605 1048640 -rw- ACE_BOOT_ROM.img.rel.2008Apr01_ver121
rommon 8 > BOOT=disk0:c6ace-t1k9-mzg.A2_2_0.bin
variable name contains illegal (non-printable) characters
rommon 9 > sync
Setting the BOOT Environment Variable

The BOOT environment variable specifies a list of image files from which the ACE can boot at startup. To set the BOOT
environment variable, use the boot system image: command. The syntax of this command is as follows:
boot system image:image_name
The image_name argument specifies the name of the system image file. If the file does not exist (for example, if you entered the
wrong filename), then the filename is appended to the bootstring, and the "Warning: File not found but still added in the
bootstring" message appears. If the file does exist, but is not a valid image, the file is not added to the bootstring, and the
"Warning: file found but it is not a valid boot image" message appears.
For example, to set the BOOT environment variable, enter the following command:
ACE_module5/Admin(config)# boot system image:c6ace-t1k9-mzg.3.0.0_A2_2.0.bin
Displaying the ACE Boot Configuration

To display the current BOOT environment variable and configuration register setting, use the show bootvar command in Exec
mode. For example, to display the BOOT environment variable settings, enter the following command:
ACE_module5/Admin# show bootvar
BOOT variable = "disk0:c6ace-t1k9-mzg.3.0.0_A2_2_0.bin"

Configuration register is 0x1
Restarting the ACE

You can reload the ACE directly from its CLI or reboot it by using the supervisor engine CLI. You may need to reboot the ACE
from the supervisor engine if you cannot reach the ACE through an external Telnet session or a console connection (for example,
the ACE is remote).
Restarting the ACE from the ACE CLI

To reboot the ACE directly from its CLI and reload the configuration, use the reload command in Exec mode. The reload
command reboots the ACE and performs a full power cycle of both the hardware and software. The reset process can take several
minutes. Any open connections with the ACE are dropped after you enter the reload command.
Caution: Configuration changes that are not written to the Flash partition are lost after a reload. Before rebooting, enter the copy
running-config startup-config command in Exec mode to store the current configuration in Flash memory. If you fail to save
your configuration changes, the ACE reverts to its previous settings upon restarting.
When you enter the reload command, the ACE prompts you for confirmation and performs a cold restart of the ACE:
ACE_module5/Admin# reload
This command will reboot the system

Save configurations for all the contexts. Save? [yes/no]: [yes]
Generating configuration....
running config of context Admin saved
Perform system reload. [yes/no]: [yes]
Setting the BOOT Environment Variable 33

Restarting the ACE from the Supervisor Engine

To restart the ACE from the supervisor engine CLI, use the hw-module command. The syntax of this command is as follows:
hw-module module mod_num reset
For example, to use the supervisor engine CLI to reset the ACE located in slot 5 of the chassis, enter the following command:
cat6k# hw-module module 5 reset

Proceed with reload of module?[confirm]
% reset issued for module 5
Establishing a Console Connection to the ACE

In case the ACE becomes unresponsive or you cannot boot the ACE using the reload command from the Admin context, you can
establish a direct serial connection between your terminal (laptop) and the ACE by making a serial connection to the console port
on the front of the ACE. The console port is an asynchronous RS-232 serial port with an RJ-45 connector. Any device connected
to this port must be capable of asynchronous transmission. Connection requires a terminal configured as 9600 baud, 8 data bits, 1
stop bit, no parity.
Note: Only the Admin context is accessible through the console port; all other contexts can be reached through Telnet or SSH
sessions.
After you connect the terminal to the console port, use any terminal communications application to access the ACE CLI. The
following procedure uses HyperTerminal for Windows.
To access the ACE by using a direct serial connection, follow these steps:
1. Launch HyperTerminal. The Connection Description window appears.
2. Enter a name for your session in the Name field.
3. Click OK. The Connect To window appears.
4. From the drop-down list, select the COM port to which the device is connected.
5. Click OK. The Port Properties window appears.
6. Set the following port properties:
• Baud Rate = 9600

• Data Bits = 8
• Flow Control = none
• Parity = none
• Stop Bits = 1
7. Click OK to connect.
8. Press Enter to access the ACE login prompt.
switch login:
9. If the ACE does not find a valid software image on disk0: or if the ACE is configured to enter ROMMON mode upon booting
up, the ROMMON prompt appears.
Restarting the ACE from the Supervisor Engine 34

rommon 1>
Troubleshooting ACE Boot Problems

The ACE module receives power from the chassis back plane and boots up automatically when you insert the module into the
chassis. If your ACE does not boot up when you insert it into the chassis or when you enter the reload Exec mode command from
the Admin context, you cannot Telnet to the ACE or establish a session from the supervisor engine. In these cases, use the
following steps to troubleshoot the issue and boot the ACE:
1. Log in to the Catalyst 6500 series switch or the Cisco 7600 series router and check the status of the ACE by entering the
following command:
User Access Verification
Password:
cat6k>enable
Password:
cat6k# show module 5
--- ----- -------------------------------------- ------------------ -----------
5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S <------- Module is receiving power

--- ---------------------------------- ------ ------------ ------------ -------
5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 Unknown Unknown Other <------- Firmware and software image

---- -------------------
5 Unknown <------- Diagnostics status is Unknown
The first row of information is populated, so you know that the ACE is powered up. The firmware and software versions are
Unknown and the Status is Other. At this point, you cannot session into the ACE from the supervisor engine.
2. Power cycle the ACE from the supervisor engine to attempt to boot the ACE by entering the following commands:
cat6k# config t
Enter configuration commands, one per line. End with CNTL/Z.
cat6k(config)# no power enable module 5
cat6k(config)# power enable module 5
Wait long enough for the ACE to boot up. Try to Telnet or session to the ACE. If you still cannot Telnet or session to the ACE,
continue with Step 3.
3. Establish a console connection to the ACE. For details about establishing a console connection to the ACE, see the
"Establishing a Console Connection to the ACE" section.
rommon 1>
4. Check the ACE configuration register (confreg) by entering the following command:
rommon 2> confreg

Configuration Summary (Virtual Configuration Register: 0x1)
enabled are: console
baud: 9600
boot: the file specified in BOOT variable
A value of 0x1 instructs the ACE to boot from the image in disk0:. A value 0x0 instructs the ACE to boot to the ROMMON
prompt. If the image specified in the BOOT variable is not in disk0:, then the ACE boots to the ROMMON prompt as shown in
this example issue.
5. Check the BOOT variable by entering the following command:
rommon 3> set

PS1=rommon ! >
RELOAD_REASON=reload command by admin
BOOT=disk0:c6ace-t1k9-mz.3.0.0_A2_2_0.bin
ARGV0=quiet
?=0
6. Ensure that the software image specified in the BOOT variable is present in disk0: by entering the following command:
rommon 4> dir disk0:
31071143 Dec 1 17:01:06 2008 c6ace-t1k9-mzg.A2_2_0.bin

250 Feb 8 20:04:44 2008 scripted_hm.txt
30337516 Jul 31 05:47:42 2008 c6ace-t1k9-mz.A2_1_3.bin
1048640 Aug 8 11:45:06 2008 ACE_FUR_BOOT_ROM.img.rel.2008Apr01_ver121
1048640 Aug 8 13:27:32 2008 ACE_BOOT_ROM.img.rel.2008Apr01_ver121
Usage for image: filesystem

517210112 bytes free
7. If the specified image is not in disk0:, then you can boot from another image in disk0: by entering the following command:
rommon 5> boot system disk0:image_name
8. If there is no image on the ACE disk0: to boot from, you can still boot from the supervisor engine. Copy the image to the
supervisor engine's disk0: or disk1:, and then from the supervisor CLI, enter the following command:
cat6k(config)# boot device module slot_number disk[0 | 1]:image_name
The ACE boots and stops at the ROMMON prompt.
9. At the ROMMON prompt on the ACE console, enter the following command to boot the ACE from the Ethernet Out-of-Band
Channel (EOBC) between the ACE and the Catalyst 6500 series switch or the Cisco 7600 series router:
rommon 6> boot eobc:
10. If the ACE is not local or you cannot establish a console connection for any other reason, use the following procedure to finish
booting the ACE from the supervisor engine with the boot eobc: command:
cat6k# remote login switch

Trying Switch ...
Entering CONSOLE for Switch
cat6k-sp# svclc console 5

Entering svclc ROMMON of slot 5 ...
Type "end" to end the session.
rommon 7> boot eobc:
This article describes the ACE system logging facility, how to enable logging, and how to use system messages as troubleshooting
tools.
Troubleshooting ACE Boot Problems 36

Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE System
Logging
• 2 Enabling ACE Logging
• 3 Logging Severity Levels
• 4 Adding Information to
syslogs
• 5 Troubleshooting ACE
Logging
♦ 5.1 Displaying Logging
Statistics
♦ 5.2 Displaying the
Logging History
Messages
Persistence
♦ 5.5 Displaying the
Logging Rate Limit
Contents 37
Overview of ACE System Logging

The ACE provides a system logging (syslog) facility that collects and saves system messages and outputs them to destinations that
you specify as follows:
• Buffer
• Console
• Flash memory
• Host (remote syslog server)
• Monitor
• Standby
• Supervisor
Each virtual context generates logs independently from the other virtual contexts. The admin virtual context does not log on behalf
of the other contexts in the ACE.
The ACE logs the following connection setup and teardown messages in the CP at the connection speed: 106023, 302022,
302023, 302024, and 302025. These setup and teardown syslogs are directly forwarded to a remote server. Because of the
potentially large number of these messages or if you require high-rate system logging of connection setup and teardown messages,
use the logging fastpath command. This command disables CP syslogs and enables logging of these messages through the DP
using a slightly different format and the following syslog IDs: 106028, 302028, 302029, 302030, and 302031, respectively. You
can log these messages through the fast path only to an external syslog server. All other enabled logging destinations are disabled
by the logging fastpath command.
You can limit the rate at which the ACE generates syslog messages by using the logging rate-limit command. This command
allows you to rate limit syslogs based on one of the following criteria:
• Time interval
• Logging level
• Message ID
Besides logging system messages, the ACE logs access control list (ACL) deny entries.
Note: Remember to enable logging on the standby ACE in a redundant configuration by entering the logging standby
command. To log failover information on the standby ACE, you need to set the logging level to 4.
For details about ACE system message logging, see the Cisco Application Control Engine Module System Message Guide
(Software Version A2(1.0)).
Enabling ACE Logging

To enable logging on the ACE module and send syslogs to the monitor, enter the following commands:
ACE_module5/Admin(config)# logging enable

ACE_module5/Admin(config)# logging monitor 7
ACE_module5/Admin(config)# logging trap 7
Overview of ACE System Logging 38

ACE_module5/Admin(config)# no logging message 111008
ACE_module5/Admin(config)# no logging message 111009
ACE_module5/Admin(config)# logging timestamp
ACE_module5/Admin(config)# do terminal monitor
Use the logging monitor severity_level command only when you are troubleshooting problems on the ACE or when there is
minimal load on the network. Using this command at other times when the ACE is active may degrade performance.
Note: logging trap defines the severity sent to the syslog server.
Note: If you do not see syslog messages on the console after enabling logging with the logging enable and logging monitor 7
commands, log out of the ACE and then log in again.
To enable logging to a syslog server, use the following command syntax:
logging host ip_address [tcp | udp [/port#]] | [default-udp] | [format emblem]
Note: If you specify the default-udp option and TCP logging fails, the ACE sends logging messages over UDP.
You can verify that the ACE defaults to UDP by entering the following command:
ACE_module5/Admin# show logging
Syslog logging: enabled

Facility: 20
History logging: disabled
Trap logging: enabled (level - debugging)
Timestamp logging: disabled
Fastpath logging: disabled
Persist logging: disabled
Standby logging: disabled
Rate-limit logging: disabled (min - 0 max 100000 msgs/sec)
Console logging: disabled
Monitor logging: disabled
Logging to 5.1.0.40 tcp/514 default-udp
(sending on UDP)
Device ID: disabled
Message logging: none
Buffered logging: enabled (level - debugging) maximum size 681984
Buffer info: current size - 681984 global pool - 1048576 used pool - 1048576
min - 0 max - 681984
cur ptr = 42894 wrapped - yes
Use the logging supervisor command to allow the aggregation of critical syslogs from multiple virtual devices to the Catalyst
6500 series switch or to the Cisco 7600 series router syslog. For example, enter the following command:
ACE_module5/Context(config)# logging supervisor ?

<0-7> 0-emerg;1-alert;2-crit;3-err;4-warn; 5-notif;6-inform;7-debug
cat6k# show logging

.
.
.
cat6k#17w3d: %TRINITY-7-TRINITY_SYSLOG_DEBUG: %ACE-7-111009: User 'admin' executed cmd: show running.
Enabling ACE Logging 39

Logging Severity Levels

The severity_level argument specifies the maximum level for system log messages sent to the console. The severity level that you
specify indicates that you want syslog messages at that level and messages less than the level. For example, if the specified level is
3, the syslog displays level 3, 2, 1, and 0 messages. We recommend that you use a lower severity level, such as 3, since logging at
a high rate may impact the performance of the ACE.
Allowable entries are as follows:
• 0?emergencies (System unusable messages)

• 1?alerts (Take immediate action)
• 2?critical (Critical condition)
• 3?errors (Error message)
• 4?warnings (Warning message)
• 5?notifications (Normal but significant condition)
• 6?informational (Information message)
• 7?debugging (Debug messages)
Adding Information to syslogs

After you have enabled system message logging and have specified a destination for the system messages, you can add more
information to the system messages that may be helpful in troubleshooting issues with your ACE module. For example, you can
do the following:
• Add a timestamp
• Identify the messages sent to a syslog server
• Identify the ACE device ID in messages that are sent to a syslog server
To add a timestamp to syslog messages, enter the following command:
ACE_module5/Admin(config)# logging timestamp
To identify messages that are sent to a syslog server by severity level, enter the following command:
ACE_module5/Admin(config)# logging trap severity_level
For example, to identify the ACE device ID in messages that are sent to a syslog server, use the following command syntax:
ACE_module5/Admin(config)# logging device-id {context-name | hostname | ipaddress interface_name | string text}
Troubleshooting ACE Logging

The commands in the following sections are useful for troubleshooting the system message logging facility.
Displaying Logging Statistics

ACE_module5/Admin# show logging statistics
Syslog statistics: sent 349 discarded 64
Messages sent:
console 0
buffer 348
persistent 0
supervisor 1
history 0
monitor 0
host 0
misc 0
Messages discarded:
cfg rate-limit 0
hard rate-limit 0
server down 5
queue full 59
errors 0
SNMP-related counters:
notifications sent 0
history table flushed 0
messages ignored 0
NP-related counters:
to-CP dropped 0
fastpath sent 0
fastpath dropped 0
ACE_module5/Admin# show logging queue
Logging Queue length limit : 80 msg(s), 59 msg(s) discarded.

Current 0 msg on queue, 80 msgs most on queue
CP messages received: 426 , 59 msg(s) discarded.

IXP messages received: 82
Xscale messages received: 0
System Max Queue size: 20080

System Free Queue size for allocation: 19920
In the above example, the ACE has discarded 59 control plane (CP) messages. By default, the syslog message queue can hold 80
messages. You can increase the size of the syslog message queue by using the logging queue command in configuration mode.
Set the queue size before you start collecting syslog messages. When traffic is heavy, messages may be discarded if the queue size
is too small. The maximum number of messages that the queue can hold is 8192.
Displaying the Logging History

To display the ACE logging history, enter the following command from the console:
ACE_module5/Admin# show logging history

syslog_trinity_show_history for context 0:
1(Mar 24 2009 16:39:36): from "KERN" ACE-5-111008:User 'admin' executed the 'logging history 7' command.
2(Mar 24 2009 16:39:48): from "KERN" ACE-5-111008:User 'admin' executed the 'logging console 7' command.
3(Mar 24 2009 16:39:56): from "KERN" ACE-7-111009:User 'admin' executed cmd: do sho logging history
4(Mar 24 2009 16:49:50): from "KERN" ACE-7-111009:User 'admin' executed cmd: do show logging message
5(Mar 24 2009 16:51:35): from "KERN" ACE-4-405001:Received ARP REQUEST collision from 10.1.1.240 00.0c.29.
6(Mar 24 2009 16:51:36): from "KERN" ACE-4-405001:Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc
7(Mar 24 2009 16:51:40): from "KERN" ACE-4-405001:Received ARP REQUEST collision from 10.1.1.240 00.0c.29.
8(Mar 24 2009 16:51:41): from "KERN" ACE-4-405001:Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc
9(Mar 24 2009 16:54:46): from "KERN" ACE-7-111009:User 'admin' executed cmd: telnet 10.1.1.130
Displaying Logging Messages

If a particular system message does not appear in the syslog history, check that the message is enabled and that the logging level is
correct for that message. To display the default logging level and the current status of a system message, all system messages, or
Displaying Logging Statistics 41

disabled system messages, enter the following command:
ACE_module5/Admin# show logging message message_id | all | disabled
For example, to display all disabled system messages in the ACE, enter the following command:
ACE_module5/Admin# show logging message disabled

Message logging:
message 111008: default-level 5 (disabled)
message 111009: default-level 7 (disabled)
Displaying Logging Persistence

The logging persistence command enables logging to disk0: in Flash memory. The messages are stored in a subdirectory of
disk0: called /messages.
To display logging persistence, enter the following command:
ACE_module5/Admin# show logging persistent

Persist info: current size - 6626 global pool - 1048576 used pool - 6626
min - 0 max - 189582
cur ptr = 6638 wrapped - no
Mar 24 2009 09:51:31 Admin: %ACE-4-405001: Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on int
Mar 24 2009 09:51:32 Admin: %ACE-4-405001: Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on in
Mar 24 2009 09:51:36 Admin: %ACE-4-405001: Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on int
Mar 24 2009 09:51:37 Admin: %ACE-4-405001: Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on in
Displaying the Logging Rate Limit

To display the logging rate limit, enter the following command:
ACE_module5/Admin# show logging rate-limit

Rate-limit logging: (min - 0 max 100000 msgs/sec)
100000 messages 1 seconds level 7
This article describes how the ACE establishes connections and how to troubleshoot connectivity issues with your ACE.
Guide Contents
Main Article
Displaying Logging Messages 42

Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE Connection Handling
• 2 Internal Mapping of ACE TCP and UDP
Flows
• 3 ACE Connection Table Entries
• 4 Tracking Connections Through the ACE
• 5 Troubleshooting Connections
Overview of ACE Connection Handling

This article describes how the ACE handles connections at Layer 4 (L4) and Layer 7 (L7). For L4 connections, the ACE receives
a TCP packet from a client and load balances the connection to a server on the first packet (see Figure 1). The SYN-ACK from the
server matches an existing flow and the rest of the connection is handled in the fast path (hardware accelerated path in the network
processors), which is represented here as "shortcut." The ACE completes the TCP handshake . This process applies to the
following functions:
• Basic load balancing

• Source IP sticky
• TCP/IP normalization
Displaying the Logging Rate Limit 43

Figure 1. Layer 4 Flow Setup
For L7 flows (for example, L7 load balancing, URL parsing, and generic TCP payload parsing), the ACE acts as a proxy (spoofs
the server), intercepts the client's VIP request that matches an L7 rule, and terminates the TCP connection. See Figure 2. The ACE
sends a SYN-ACK to the client in response to the client's TCP SYN. The client responds with an ACK to complete the TCP
handshake and an L7 request method (for example, HTTP GET or POST).
Figure 2. Layer 7 Flow Setup -- Client Connection
Overview of ACE Connection Handling 44

After the ACE receives the L7 information (for example, HTTP GET), it sets up the back-end connection to the real server based
on the load-balancing method and other criteria. See Figure 3.
Figure 3. Layer 7 Flow Setup -- Server Connection

Finally, the ACE unproxies the connection with the client and splices it together with the back-end connection to the server. For
the life of the HTTP flow, the client communicates directly with the server through the fast path (hardware-accelerated path in the
network processors), which is depicted in the figures as "Shortcut." See Figures 4.
Figure 4. Layer 7 Flow Setup -- Splicing the Flows Together

Figure 5 shows how the ACE adjusts the sequence numbers and ACK numbers when it splices the two flows together.
Figure 5. Layer 7 Flow Setup -- Adjusting the Sequence and ACK Numbers

With the persistence rebalance (connection keepalive) command configured, the ACE reproxies and parses subsequent HTTP 1.1
requests over the same TCP connection. In this case, the ACE again spoofs the server and ACKs the HTTP GET as shown in
Figure 6. The sequence shown in Figure 2 through Figure 5 repeats for each new HTTP 1.1 request over the same TCP
connection.
Figure 6. Layer 7 Flow Setup -- Reproxy

For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the ACE fully terminates the
client TCP connection. This connection remains fully proxied because the ACE is acting on behalf of the real server. For SSL
termination, the ACE completes an SSL handshake after it establishes the TCP connection with the server. See Figure 7.
Figure 7. SSL Handshake

For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the client and server
connections are completely independent and flows are handled in the software, not in the fast path. See Figure 8.
Figure 8. Layer 7 Flow Setup -- Full Proxy

Internal Mapping of ACE TCP and UDP Flows

The ACE maps TCP and UDP flows as two halves of the same flow: one input flow and one output flow. You can display the
current connections in the ACE by entering the show connections command. See Figure 8.
Figure 8. Internal Flow Mapping
Internal Mapping of ACE TCP and UDP Flows 51

ACE Connection Table Entries

Understanding ACE?s Conn Table Entries During:
• L4 TCP Connection Setup (3 Way Handshake)

♦ Normalisation Enabled
♦ Normalisation Disabled
• L7 TCP Connection Setup (3 Way Handshake)
• TCP Connection Teardown
♦ 3 Way Handshake
♦ 4 Way Handshake
♦ Reset
ACE Connection Table Entries 52















Tracking Connections Through the ACE

You can display the IDs for the request and response connections in the ACE by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-c 9"

Connection ID:seq: 9[0x9].6 <------- Request and response connection ID
Other ConnID : 3[0x3].4
Proxy ConnID : 0[0x0].0
Next Q : 1124073484[0x4300000c]
192.168.12.15:1985 -> 10.1.1.2:1985 [RX-NextHop: Drop] [TX-NextHop: TX]

Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No
L3 Protocol : IPv4 L4 Protocol : 17
Inbound Flag : 0
Interface Match : Yes
Interface MatchID: 0
EncapsID:ver : 0:0 TCP ACK delta : 0x0
MSS : 0 TOS Stamp : 0
Repeat mode : No ARP Lookup : No
TOS Stamp : No TCP Window Check: No
ACE ID : 152 NAT Policy ID : 0
Post NAT hop : 0
Packet Count : 0 Byte Count : 0
Tracking Connections Through the ACE 67

TCP Information: (State = 0)
Window size : 0 Window scale : 0
FIN seen : No FIN/ACK seen : No
FIN/ACK exp : No Close initiator : No
FIN/ACK expval: 0 Last seq : 0
timestamp_delta: 0 Last ack : 1ce862
No Trigger : 0 Trigger Status : 0
Timestamp : 26ebc96d
TCP options negotiated:
Sack:Allow TS:Allow Windowscale: Allow
Reserved: Allow Exceed MSS: Allow Window var: Allow
You can display both the front-end and the back-end connection statistics by entering the "-v" (verbose) option of the show np
command as follows:
ACE_module5/Admin# show np 1 me-stats "-c 9 -v"

Connection ID:seq: 9[0x9].2
Next Q : 0[0x0]
10.1.1.5:23 -> 172.27.16.143:4837 [RX-NextHop: TX] [TX-NextHop: CP]

Inbound Flag : 0
Post NAT hop : 4
FIN/ACK expval: 5b40000 Last seq : 53768a51
timestamp_delta: 0 Last ack : 658c1f72
Timestamp : 459f781e
Sack:Clear TS:Clear Windowscale: Clear
Reserved: Allow Exceed MSS: Deny Window var: Allow
Flags: debug: 0 TCP Normalize: Yes
Syslog: No Reproxy Request: No Policying Reqd: No
Inbound IPSec: No Replicated: No Data Channel: No
L7: No Fin Detect: Yes FP Timeout: No
Standby: No ConnState: 2
ACA Method: 0 ReqTS: 00000000 RspTS: 00000000
Raw Connection Entry

0000 0x00000000 0x0a56d786 0xa12c438f 0x06210007
0010 0x001712e5 0x00000000 0x00030000 0x04ec1004
0020 0x4e000007 0x00000000 0x00080480 0x24450000
0030 0x0000015b 0x00005f9c 0x16d00030 0x05b40000
0040 0x53768a51 0x658c1f72 0x459f781e 0x00000000
0050 0x00000094 0x00000000 0x45729985 0x00000000
0060 0x00000000 0x00000000 0x00000000 0x00000000
Doing verbose output for proxy id: 0
Tracking Connections Through the ACE 68

No valid proxy entry.
No valid TCB proxy entry.
No valid HTTP proxy entry.
No valid SSL proxy entry.
No valid AI proxy entry.
Connection ID:seq: 7[0x7].14
Next Q : 0[0x0]
172.27.16.143:4837 -> 10.1.1.5:23 [RX-NextHop: CP] [TX-NextHop: TX]

Inbound Flag : 1
Post NAT hop : 0
FIN/ACK expval: 5b40000 Last seq : 658c1f72
timestamp_delta: 0 Last ack : 53768a51
Timestamp : 459f781e
Sack:Clear TS:Clear Windowscale: Clear
Reserved: Allow Exceed MSS: Deny Window var: Allow
Flags: debug: 0 TCP Normalize: Yes
Syslog: No Reproxy Request: No Policying Reqd: No
Inbound IPSec: No Replicated: No Data Channel: No
L7: No Fin Detect: Yes FP Timeout: No
Standby: No ConnState: 2
ACA Method: 0 ReqTS: 00000000 RspTS: 00000000
Raw Connection Entry

0000 0x00000000 0xa12c438f 0x0a56d786 0x06e90007
0010 0x12e50017 0x00000000 0x00030000 0x05b41000
0020 0x02000009 0x00000000 0x00080481 0x24450000
0030 0x000001e6 0x00004d62 0xff5b0030 0x05b40000
0040 0x658c1f72 0x53768a51 0x459f781e 0x00000000
0050 0x00000094 0x00000000 0x45729985 0x00000000
0060 0x00000000 0x00000000 0x00000000 0x00000000
Doing verbose output for proxy id: 0
No valid proxy entry.

No valid TCB proxy entry.
No valid HTTP proxy entry.
No valid SSL proxy entry.
No valid AI proxy entry.
Troubleshooting Connections
To troubleshoot suspected connectivity issues, follow these steps:
1. Check the ACL hit count by entering the show access-list acl_name command. If the hit count is increasing, go to Step 2.
Otherwise, verify that the access list is configured properly to permit traffic.
ACE_module5/Admin# show access-list anyone detail

access-list:anyone, elements: 1, status: ACTIVE
remark :
access-list anyone line 8 extended permit ip any any (hitcount=3438) [0x44c2baf1] <------- Hit count
2. Check the service policy hit count by entering the show service-policy detail command. If the hit count is 0, verify that the
service policy is active (show service-policy command) and the server farm is up (show server-farm detail command). If the
service policy is large, use the show service-policy policy_name summary command for more information as follows:
ACE_module5/Admin# show service-policy VIP summary
service-policy: VIP
Class VIP Prot Port VLAN
State Curr Conns Hit Count Conns Drop
VIP 192.168.12.192 tcp eq 443 100
IN-SRVC 0 0 0
192.168.12.192 tcp eq 80 100
IN-SRVC 0 0 0
192.168.12.193 tcp eq 80 100
IN-SRVC 0 0 0
192.168.12.193 tcp eq 443 100
IN-SRVC 0 0 0
VIP2 192.168.12.194 tcp eq 80 100
IN-SRVC 0 0 0
192.168.12.194 tcp eq 443 100
IN-SRVC 0 0 0
3. Check the load-balancing statistics by entering the show stats loadbalance command. If the Layer 4 or Layer 7 rejections or
the Layer 4 or Layer 7 policy misses are increasing, check the configured class maps for any misconfiguration.
ACE_module5/Admin# show stats loadbalance
+------------------------------------------+
+------- Loadbalance statistics -----------+
+------------------------------------------+
Total version mismatch : 0
Total Layer4 decisions : 0
Total Layer4 rejections : 3 <-------|
Total Layer7 decisions : 0 |------- Failed connections due to traffic not matching the conf
Total Layer7 rejections : 7 <-------|
Total Layer4 LB policy misses : 0
Total times rserver was unavailable : 10 <------- Failed connections due to no real server'
Total ACL denied : 0
Total IDMap Lookup Failures : 0
To clear the load-balancing statistical information stored in the ACE buffer, enter the clear stats loadbalance command.
4. If none of the error statistics is increasing, check the connection record by entering the show conn detail command and
checking the connections for the affected VIP.
ACE_module5/Admin# show conn detail
total current connections : 6
conn-id np dir proto vlan source destination state

----------+--+---+-----+----+---------------------+---------------------+------+
7 1 in TCP 130 10.1.1.2:1171 10.1.1.134:23 ESTAB
[ idle time : 00:00:00, byte count : 60055 ]
[ elapsed time: 04:15:29, packet count: 1473 ]
9 1 out TCP 130 10.1.1.134:23 10.1.2.74:1171 ESTAB
[ conn in reuse pool : FALSE]
[ idle time : 00:00:00, byte count : 64880 ]
[ elapsed time: 04:15:29, packet count: 1086 ]
5. Display existing ACE connection statistics by entering the following command:
ACE_module5/Admin# show stats connection
+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created : 628950
Total Connections Current : 7
Total Connections Destroyed: 389
Total Connections Timed-out: 3958
Total Connections Failed : 624596 <------- Server did not reply to a SYN within the pending timeout period or i
The Total Connection Failed counter increases when the ACE cannot set up the back-end connection with the server. To clear the
statistical information stored in the ACE buffer, enter the clear stats connection command.
6. Display service policy statistics by entering the following command:
ACE/Context# show service-policy client-vips detail

Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 211
service-policy: client-vips
class: VIP-HTTPS
VIP Address: Protocol: Port:
172.16.11.190 tcp eq 443 <------- Shows the VIP address, port, and protocol
loadbalance:
L7 loadbalance policy: HTTPS-POLICY
VIP Route Metric : 77
VIP Route Advertise : DISABLED
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP State: INSERVICE <------- Service is INSERVICE
curr conns : 22 , hit count : 22
dropped conns : 0
client pkt count : 0 , client byte count: 0
server pkt count : 0 , server byte count: 0
max-conn-limit : 0 , drop-count : 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : HTTPS-POLICY
class/match : class-default
LB action :
primary serverfarm: backend-ssl
backup serverfarm : -
hit count : 22 <------- Shows the hit count
dropped conns : 0
7. Display server farm connection statistics by entering the following command:
ACE/Context# show serverfarm HTTPS-FARM detail

serverfarm : HTTPS-FARM, type: HOST
total rservers : 4
active rservers: 4
Troubleshooting Connections 71
description : -
state : ACTIVE
predictor : ROUNDROBIN <------- Shows the load-balancing predictor that was used
failaction : -
back-inservice : 0
partial-threshold : 0
num times failover : 0
num times back inservice : 0
total conn-dropcount : 0
---------------------------------
----------connections-----------
real weight state current total failures
---+---------------------+--------+---------------------+-----------+----------+---------
rserver: linux-1
192.168.1.11:0 8 OPERATIONAL 0 0 0 <------- Shows connection s
each real server
max-conns : - , out-of-rotation count : -
min-conns : -
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
retcode out-of-rotation count : -
The Connections Failures counter for a real server in a server farm may increment for one of the following reasons:
• SYN timeout (the three-way handshake fails to complete)

• RST received (a client sends an RST to the server)
• Internal exception (internal software issue)
8. Display the statistics for a connection parameter map by entering the following command:
ACE_module5/Admin# show parameter-map CONN_PARAMMAP
Number of parameter-maps : 1
Parameter-map : CONN_PARAMMAP
Type : connection
nagle : disabled
slow start : disabled
buffer-share size : 32768
inactivity timeout (seconds) : TCP: 3600, UDP: 120, ICMP: 2
embryonic timeout (seconds) : 5
ack-delay (milliseconds) : 200
WAN Optimization RTT (milliseconds): 65535
half-closed timeout (seconds) : 3600
TOS rewrite : disabled
syn retry count : 4
TCP MSS min : 0
TCP MSS max : 1460
tcp-options drop range : 0-0
tcp-options allow range : 0-0
tcp-options clear range : 1-255
selective-ack : clear
timestamp : clear
window-scale : clear
window-scale factor : 0
reserved-bits : allow
random-seq-num : enabled
SYN data : drop
exceed-mss : drop
urgent-flag : allow
conn-rate-limit : disabled
bandwidth-rate-limit : disabled
Troubleshooting Connections 72
9. Reset the ACE connection statistics by entering the following commands:
• clear conn [all | flow {icmp | tcp | udp} | rserver server_name]

• clear stats conn
• clear tcp statistics
• clear udp statistics
This article describes how to troubleshoot issues involving ACE remote access.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE Remote Access
• 2 Configuring a Management Policy for Remote
Access
• 3 Troubleshooting Remote Access
♦ 3.1 Troubleshooting Telnet
♦ 3.2 Troubleshooting SSH
♦ 3.3 Troubleshooting KAL-AP
Contents 73
Overview of ACE Remote Access

You can access the ACE remotely using several different protocols as follows:
• HTTP
• HTTPS
• ICMP
• KALAP-UDP
• SSH
• SNMP
• Telnet
These protocols require that you configure a management traffic policy on the ACE and associate that policy with the interface
that you intend to use for management traffic. A management policy allows the classification and distribution engine (CDE) to
classify (match) incoming management traffic to the management policy and to forward that traffic to the control plane (CP). For
complete details about remote access, see the Cisco Application Control Engine Module Administration Guide (Software Version
A2(1.0)).
Configuring a Management Policy for Remote Access

To configure a management policy that allows remote access to the ACE using ICMP, SSH, or Telnet, enter the following
commands:

ACE_module5/Admin(config)# class-map type management match-any MGMT_CLASS
ACE_module5/Admin(config-cmap-mgmt)# 2 match protocol icmp any
ACE_module5/Admin(config-cmap-mgmt)# 3 match protocol ssh any
ACE_module5/Admin(config-cmap-mgmt)# 4 match protocol telnet any
ACE_module5/Admin(config-cmap-mgmt)# exit
ACE_module5/Admin(config)# policy-map type management first-match MGMT_POLICY
ACE_module5/Admin(config-pmap-mgmt)# class MGMT_CLASS
ACE_module5/Admin(config-pmap-mgmt-c)# permit
ACE_module5/Admin(config-pmap-mgmt-c)# exit
ACE_module5/Admin(config-pmap-mgmt)# exit
ACE_module5/Admin(config-if)# ip address 192.168.12.15 255.255.255.0
ACE_module5/Admin(config-if)# service-policy input MGMT_POLICY
ACE_module5/Admin(config-if)# access-group input ACL1

If you cannot access the ACE module remotely, follow these steps:
Overview of ACE Remote Access 74

1. Beginning with ACE software release A2(1.1), by default, the ACE CLI is only locally accessible either using the ACE console
port or through the supervisor by entering the session command. Remote access to the ACE (for example, Telnet, SSH, and so on)
is disabled until you change the admin user account password from the default. Access to the XML API is also disabled until you
change the www user account password from the default. The ACE will display these warnings each time you access the CLI
using the the console port or the supervisor until you change these passwords.
cat6k#session slot 5 processor 0

The default escape character is Ctrl-^, then x.
You can also type 'exit' at the remote prompt to end the session
Trying 127.0.0.20 ... Open
ACE_module5 login: admin

Password:
Please change the password for admin user.
Admin user is allowed to login only from supervisor until the password is changed.
User 'www' is disabled. Please change the password to enable the user.
Use the following commands to change the passwords of the admin and www user accounts:
ACE_module5/Admin# config
ACE_module5/Admin(config)# username admin password 0 cisco123 role Admin domain default-domain
ACE_module5/Admin(config)# username www password 0 cisco123 role Admin domain default-domain
Note that, although the passwords were entered in clear text above, they will be stored in the ACE configuration in an encrypted
format:
ACE_module5/Admin# show run | i username

username admin password 5 $1$M7gtcvBC$9ca78Q.ZH5jZpqDVuLnkN0 role Admin domain default-domain
username www password 5 $1$ulc7KHL5$2HlgNTEez03.ElmbiWKyY/ role Admin domain default-domain
2. Ensure that the remote access method protocol (for example, Telnet or SSH) that you are trying to use is configured in the
management class map and that the management class has been permitted in the management policy. If necessary, correct your
ACE configuration. To display your management policy configuration elements, enter the following Exec mode commands:
ACE_module5/Admin# show running-config class-map

class-map type management match-any MGMT_CLASS

2 match protocol icmp any
3 match protocol ssh any
4 match protocol telnet any
ACE_module5/Admin# show running-config policy-map MGMT_POLICY

policy-map type management first-match MGMT_POLICY

class MGMT_CLASS
permit
class class-default
Troubleshooting Remote Access 75

3. Ensure that the management policy is applied to the correct interface and that you are using the correct IP address for that
interface. If necessary, correct your configuration. Enter the following command:
ACE_module5/Admin# show running-config interface
interface vlan 100

ip address 192.168.12.15 255.255.255.0
access-group input ACL1
access-group output ACL1
service-policy input MGMT_POLICY
no shutdown
4. Check the status of the management interface by entering the following command:
ACE_module5/Admin# show interface vlan 100
vlan100 is up
Mode : routed
Description:not set
MTU: 1500 bytes
Last cleared: never
5. If the interface is down, ensure that the no shutdown command is configured on the interface to enable it. If necessary, correct
your configuration. Enter the following command:
6. Ensure that you have not exceeded the allocated resources for management connections or maximum management bandwidth
by entering the following commands:
ACE_module5/Admin# show resource usage resource mgmt-connections

Allocation
Resource Current Peak Min Max Denied
-------------------------------------------------------------------------------
Context: Admin
mgmt-connections 2 10 0 100000 0
Context: C1
ACE_module5/Admin# show resource usage resource rate mgmt-traffic

Allocation
-------------------------------------------------------------------------------
Context: Admin
mgmt-traffic rate 78 3588 0 125000000 0
Context: C1
7. If necessary, allocate more resources to management connections by entering the following command:

ACE_module5/Admin(config-resource)# limit-resource mgmt-connections minimum number maximum number
8. Ensure that traffic is reaching the CP by entering the following command:
ACE_module5/Admin# show netio stats

----------------------- ----------------------
Net Rx Packets : 680 Net Rx Packets : 10799640 <-------
Net Rx Bytes : 52902 Net Rx Bytes : 842376636 <-------
Net Rx Unsupported L2 : 0 Net Rx Unsupported L2 : 0
Net Rx Lock Errors : 0 Net Rx Lock Errors : 0
Net Rx Interface Miss : 0 Net Rx Interface Miss : 10470926
Net Rx No Arp Client : 0 Net Rx No Arp Client : 0
Net Rx Alias Drops : 0 Net Rx Alias Drops : 0
Net Rx Repl. Errors : 0 Net Rx Repl. Errors : 0
Net Rx Repl. If Err : 0 Net Rx Repl. If Errs : 0
Net Rx Internal Errs : 0 Net Rx Internal Errs : 0
Net Tx Packets : 0 Net Tx Packets : 10469 <-------

Net Tx Bytes : 0 Net Tx Bytes : 1194879 <-------
Net Tx Lock Errors : 0 Net Tx Lock Errors : 0
Net Tx Bad Context ID : 0 Net Tx Bad Context ID : 0
Net Tx No Route Found : 0 Net Tx No Route Found : 0
Net Tx No Adjacency : 0 Net Tx No Adjacency : 0
Net Tx Invalid If ID : 0 Net Tx Invalid If ID : 0
Net Tx If Down : 0 Net Tx If Down : 0
Net Tx No Src Addr : 0 Net Tx No Src Addr : 0
Net Tx No Encap : 0 Net Tx No Encap : 0
Net Tx FIFO Errors : 0 Net Tx Fifo Errors : 0
Net Tx No VMAC Errors : 0 Net Tx No VMAC Errors : 0
IPC Tx Packets : 78 IPC Tx Packets : 0

IPC Tx Bytes : 17766 IPC Tx Bytes : 0
IPC Tx Fifo Errors : 0 IPC Tx Fifo Errors : 0
Client Rx Queue Full : 0 Client Rx Queue Full : 0
Pseudo Rx Queue Full : 0 Pseudo Rx Queue Full : 0
Management traffic is considered to-the-ACE traffic or CP traffic. If traffic is reaching the CP, the Normal Priority (Data) Net Rx
Packets, Net Rx Bytes, Net TX packets, and Net TX bytes counters should be increasing. If not, contact TAC.
9. If traffic is not arriving at the CP, ensure that traffic is reaching the classification and distribution engine (CDE) from the SFI by
entering the following command:
ACE_module5/Admin# show cde health
CDE BRCM INTERFACE

======================
Broadcom interface CRC error count 0
BRCM VOQ status [empty] [not full]
BRCM pull status [pulling]
CDE HYPERION INTERFACE

======================
Packets received 12312951 <-------
Packets transmitted 17361 <-------
Short packets drop count 0
Protocol error drop count 0

FCS error drop count 0
CRC error drop count 0
Num times flow control triggered on hyp interface 0
Num self generated multicast packets filtered 4618
HYP SLOW VOQ status [empty] [not full]
HYP tx pull status [pulling]
<snip>
If traffic is reaching the CDE, the Packets received and the CDE Hyperion Interface Packets transmitted counters should be
increasing. If not, contact TAC.
10. If packets are not reaching the CDE, ensure that the MSFC in the Catalyst 6500 series switch or the Cisco 7600 series router is
sending packets to the switch fabric interface (SFI) by entering the following command on the supervisor engine:
cat6k# show svclc module 5 traffic

ACE module 5:
Specified interface is up line protocol is up (connected)

Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s
input flow-control is on, output flow-control is unsupported
Last input never, output never, output hang never
Last clearing of "show interface" counters 1w2d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
912150 packets input, 74727962 bytes, 0 no buffer
Received 796374 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
17390 packets output, 2145844 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
If the MSFC is sending traffic to the SFI, the packets input and the packets output counters should be increasing. If not, contact
TAC.
Troubleshooting Telnet
If you cannot Telnet to the ACE, ensure that you have not reached the maximum connection limit for Telnet by entering the
following commands:
ACE_module5/Admin# show telnet
Session ID Remote Host Active Time

4254 127.0.0.51 :41985 0: 8:13
The show telnet command output shows only one Telnet session. A maximum of 15 more users can potentially Telnet to the
Admin context.
Troubleshooting Telnet 78
To display the maximum number of users allowed to Telnet to a particular context, enter the following command:
ACE_module5/Admin# show telnet maxsessions
Maximum Sessions Allowed is 16
Troubleshooting SSH
If you attempt to connect to the ACE using SSH and receive the following error, follow these steps:
[linux]$ ssh admin@192.168.0.210

ssh_exchange_identification: Connection closed by remote host [linux]$
1. Ensure that SSH is enabled in the management policy by entering the following command:

2 match protocol http any
3 match protocol https any
6 match protocol ssh any <------- SSH is enabled
8 match protocol snmp any
switch/Admin# show running-config policy-map MGMT_POLICY


class MGMT_CLASS
permit <------- All protocols in the MGMT_CLASS class-map are permitted including SSH
2. Ensure that the SSH key has been generated by entering the following command:
switch/Admin# show ssh key

**************************************
could not retrieve rsa1 key information
**************************************
could not retrieve rsa key information
**************************************
could not retrieve dsa key information
**************************************
no ssh keys present. you will have to generate them
**************************************
The show ssh key command output shows that no SSH key has been generated.
3. Generate an SSH key based upon your security requirements by entering the following commands:
ACE_module5/Admin(config)# ssh key rsa 4096
generating rsa key(4096 bits).....
.........................
generated rsa key
switch/Admin# show ssh key
dsa rsa rsa1
ACE_module5/Admin# show ssh key
**************************************
could not retrieve rsa1 key information
**************************************
rsa Keys generated:Tue Apr 7 15:55:20 2009
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAgEAr3z0MO0knoS6YwntSxUkWDWjHZfFE7Y6nLd8qQVcPMu0
XpvkabDswLwoEdC9nOWM4v4g4PDpUz+tk+2WHJ4MMgRVeomVbK/2+Zx0Eds1p2XlhiV9KPcV
pflpNNt63Mr01oLHoHpjxJ8ubfJJ+gPhMoBmQyGKedQOk5tVlbOpyxS2f7yWGqzF26AzXTFFdS0xYEcN4GrtziduBlh4TYRxk99JR13
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+9C0i
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+uowK
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+gHaa
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+ryoo
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+iKsh
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+/cut
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+7OvN
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+w6ig
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+msEF
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+XvRo
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+w34s
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mbRk
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+e9p2
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+PEAQ
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+D+Ty
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+GsLz
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+EaKa
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+MNQm
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+VVkV
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+4d21
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+x6VH
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+fEGT
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+64zM
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+Hf/a
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+e9Tu
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+W4nD
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+7SUP
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+leL+
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+Tmwt
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+8ymk
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+20L0
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+ym2A
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mfe5
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+A+Py
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+k/Dx
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+NeCF
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mSK1
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+V747
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+M0xG
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+QiEP
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+CCTj
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+8lkb
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+GXOo
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+C094
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+eDPM
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+F+xp
ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+eNj
zsFzc1023QU8xwlrhgvL0xXKxUmITV4y2VaSmEc/0RH/c4XAinNy955X6w9ejqXG9aYFjljLBtCHKXgyAZbQ48E4UPtPwPTCC3R1pFapfNAJMQ55kf
/6x/6XVFPBw4okLcz6tVDLqn7dGiTEjzYgfQBwKXiIPrGg9EmBgwmQKWuJpOde+jbMD9kU1WmUoUWvTvPtXQ0=
bitcount:4096
fingerprint:
13:96:fe:f0:f7:d7:5f:9c:d7:2a:da:72:8a:93:53:a6
**************************************
could not retrieve dsa key information
**************************************
Now SSH should work.

4. Try connecting to the ACE via SSH again by entering the following command:
[linux]$ ssh admin@192.168.0.210

Warning: Permanently added '192.168.0.210' (RSA) to the list of known hosts.
Password:
Cisco Application Control Software (ACSW) TAC support: http://www.cisco.com/tac Copyright (c) 1985-2009 by Cisco S
The copyrights to certain works contained herein are owned by other third parties and are used and distributed und
Some parts of this software are covered under the GNU Public License. A copy of the license is available at http:/
ACE_module5/Admin#
5. Confirm the SSH session from the ACE CLI by entering the following command:
ACE_module5/Admin# show ssh session-info
Session ID Remote Host Active Time

6986 10.76.248.6 :42116 0: 0:46
Troubleshooting KAL-AP
To troubleshoot KAL-AP related issues, follow these steps:
1. Make sure that KAL-AP is enabled under the management policy by entering the following commands:


5 match protocol kalap-udp any <------- KAL-AP is enabled
ACE_module5/Admin# show running-config policy-map MGMT_POLICY


class MGMT_CLASS
permit <------- All protocols in the MGMT_CLASS class-map are permitted including SSH
2. Verify that traffic from the Cisco Global Site Selector (GSS) is reaching the ACE module. KAL-AP statistics should get
incremented.
ACE_module5/Admin# sh stats kalap
+-----------------------------------------------------+
+---------------- KAL-AP(UDP) statistics -------------+
+-----------------------------------------------------+
Total bytes received : 243956

Total bytes sent : 184884
Total requests received : 5100
Total responses sent : 5100
Total requests successfully received : 5100
Total responses successfully sent : 5100
Total secure requests received : 0
Total secure responses sent : 0
Total requests with errors : 0
Troubleshooting KAL-AP 81
Total requests with parse errors : 0
Total response transfer errors : 0
-----------------------------------------------------
3. Allow secure KAL-AP requests, and add the GSS IP address and the shared secret to the ACE by entering the following
commands:
ACE_module5/Admin(config)# kalap udp
ACE_module5/Admin(config-kalap-udp)# ip address 192.168.10.52 encryption md5 cisco
(GSS IP)
4. Display information about the load VIP by entering the following command:
ACE_module5/Admin# show kalap udp load vip 10.1.1.1
Error: Vip object not found!

ACE_module5/Admin#
If the VIP object is not found while displaying the load value as shown above, check whether the VIP got downloaded in the
configuration manager internal table by entering the following command:
ACE_module5/Admin# show cfgmgr internal table vip
VIP-Id VIP-Addr Ctx-Id Flags

---------------------------------------------------------------------------
1 10.1.1.1 1 DATA_VALID,
L3Rule_list :-->: 41: 42
Load Value: 255 Load Time stamp: Wed Apr 8 05:10:20 2009
This article describes security access control lists (ACLs) in the ACE, how to configure them, and troubleshooting steps to follow
if you encounter problems with ACLs.
Guide Contents
Main Article
Troubleshooting SSL
Troubleshooting KAL-AP 82
ACE Resource Limits

Contents
• 1 Overview of Security Access
Control Lists
♦ 1.1 ACL Types and Uses
♦ 1.2 ACL Configuration
Guidelines
◊ 1.2.1 ACL Entry
Order
◊ 1.2.2 ACL Implicit
Deny
◊ 1.2.3 Maximum
Number of ACL
Entries
• 2 Configuring ACLs
• 3 ACL-Related syslogs
• 4 Troubleshooting ACLs
Overview of Security Access Control Lists

An ACL consists of a series of statements called ACL entries that define the network traffic profile. Each entry permits or denies
network traffic (inbound and outbound) from and to the parts of your network specified in the entry. Each entry also contains a
filter element that is based on criteria such as the source address, the destination address, the protocol, and protocol-specific
parameters such as ports and so on.
An implicit deny-all entry exists at the end of each ACL, so you must configure an ACL on each interface that you want to permit
connections. Otherwise, the ACE denies all traffic on the interface.
ACLs allow you to control network connection setups rather than processing each packet. Such ACLs are commonly referred to as
security ACLs. You can configure ACLs as parts of other features (for example, security, Network Address Translation (NAT),
Contents 83
server load balancing (SLB), and so on). The ACE merges these individual ACLs into one large ACL called a merged ACL. The
ACL compiler then parses the merged ACL and generates the ACL lookup mechanisms. A match on this merged ACL can result
in multiple actions.
Note: You can apply only one extended ACL to each direction (inbound or outbound) of an interface. You can also apply the
same ACL on multiple interfaces. You can apply EtherType ACLs only in the inbound direction and only on Layer 2
interfaces.
ACL Types and Uses
You can configure the following two types of ACLs in the ACE:
• Extended?Control network access for IP traffic (Layer 3 and Layer 4)

• EtherType?Control network access for non-IP traffic on Layer 2 interfaces
The ACE does not explicitly support standard ACLs. To configure a standard ACL, specify the destination address as any and do
not specify ports in an extended ACL. For details about configuring an extended ACL, see the ?Configuring an Extended ACL?
section.
ACL Configuration Guidelines

This section describes the guidelines to observe when you configure and use ACLs in your network. It contains the following
topics:
• ACL Entry Order

• ACL Implicit Deny
• Maximum Number of ACL Entries
ACL Entry Order
An ACL consists of one or more entries. Depending on the ACL type, you can specify the source and destination addresses, the
protocol, the ports (for TCP or UDP), the ICMP type, the ICMP code, or the EtherType as the match criteria. By default, the ACE
appends each ACL entry at the end of the ACL. You can also indicate the location of each entry within an ACL by specifying a
line number.
The order of the entries is important. When the ACE decides whether to accept or refuse a connection, the ACE tests the packet
against each ACL entry in the order in which the entries are listed. After it finds a match, the ACE does not check any more
entries. For example, if you create an entry at the beginning of an ACL that explicitly permits all traffic, the ACE does not check
any other statements in the ACL.
Note: If there is a deny statement for packets coming to the Control Plane (CP), then the ACE skips to the next ACL entry.
ACL Implicit Deny
All ACLs have an implicit deny entry at the end of the ACL, so, unless you explicitly permit it, traffic cannot pass. For example, if
you want to allow all users to access a network through the ACE except for those users with particular IP addresses, then you must
deny the particular IP addresses in one entry and permit all other IP addresses in another entry.
Maximum Number of ACL Entries
ACLs are used in ACE as conventional access-groups, and also for use in class maps. The ACLs are converted into trees of
different data structures called nodes, which are merged into a single tree instance that consists of nodes of one type called Policy
Action Nodes (PANs). This merged tree is then transferred to the dataplane. A tree is created for each instance, and an instance is
defined as a VLAN interface in either the input or output direction. Therefore, all ACLs that are applied to a given VLAN and a
given direction contribute to the node usage for that instance.
The ACE supports a maximum of 256,000 Policy Action Nodes (PANs) entries. Some ACLs use more memory than others, such
as an ACL that uses large port number ranges or overlapping networks (for example, one entry specifies 10.0.0.0/8 and another
entry specifies 10.1.1.0/24). Depending on the type of ACL, the actual limit that the ACE can support may be less than 256,000
PANs entries.
If you use object groups in ACL entries, you enter fewer actual ACL entries, but the same number of expanded ACL entries as
you did when you entered entries without object groups. Expanded ACL entries count toward the system limit. To view the
number of expanded ACL entries in an ACL, use the show access-list name command.
If you exceed the memory limitations of the ACE, it generates a syslog message and increments the Download Failures counter in
the output of the show interface vlan number command. The configuration remains in the running-config file and the interface
stays enabled. The ACL entries stay the same as they were before the failing configuration was attempted.
For example, if you add a new ACL with ten entries, but the addition of the sixth entry fails because the ACE runs out of memory,
the ACE removes the five entries that you successfully entered.
Note: You must allocate sufficient ACL memory resources for each virtual context in the ACE. The ACE does not generate a
syslog if you exceed the maximum number of ACL entries.
Configuring ACLs
You can configure ACLs in one of two ways:
• Using the access-list command in configuration mode

• Using the match access-list command in a Layer 3 and Layer 4 class map
You can permit or deny network connections based on the IP protocol, source and destination IP addresses, and TCP or UDP
ports. To configure a non-ICMP extended ACL, enter the following command:
access-list name [line number] extended {deny | permit} {protocol {any | host src_ip_address | src_ip_address netmask |
object-group net_obj_grp_name} [operator port1 [port2]] {any | host dest_ip_address | dest_ip_address netmask | object-group
net_obj_grp_name} [operator port3 [port4]]} | {object-group service_obj_grp_name} {any | host src_ip_address |
src_ip_address netmask | object-group net_obj_grp_name} {any | host dest_ip_address | dest_ip_address netmask |
object-group net_obj_grp_name}
You can also permit or deny network connections based on the ICMP type (for example, echo, echo-reply, unreachable, and so
on). To configure an ICMP extended ACL, enter the following command:
access-list name [line number] extended {deny | permit} {icmp {any | host src_ip_address | src_ip_address netmask |
object-group net_obj_grp_name} {any | host dest_ip_address | dest_ip_address netmask | object-group net_obj_grp_name}
[icmp-type code [operator code1 [code2]]]} | {object-group service_obj_grp_name} {any | host src_ip_address | src_ip_address
netmask | object-group net_obj_grp_name} {any | host dest_ip_address | dest_ip_address netmask | object-group
net_obj_grp_name}
You can configure an ACL that controls traffic based on its EtherType. An EtherType is a subprotocol identifier. EtherType ACLs
support Ethernet V2 frames; they do not do not support 802.3-formatted frames. To configure an Ethertype ACL, enter the
following command:
access-list name ethertype {deny | permit} {any | bpdu | ipv6 | mpls}
Note: You can configure an EtherType ACL on a Layer 2 interface in the inbound direction only. If you are operating the
ACE in bridge mode, be sure to configure an ACL on all interfaces that permit BPDUs. Otherwise, a bridge loop may
result.
For example, to configure an extended ACL to permit all IP traffic from any source IP address and that is destined to any IP
address on interface VLAN 200, enter the following commands:

ACE_module5/Admin(config-if)# ip address 192.168.1.1 255.255.255.0
ACE_module5/Admin(config)# access-group input ACL1
You can apply an ACL to all interfaces in a context at once, subject to the following conditions:
• No interface in the context has an ACL applied to it.

• You can globally apply one Layer 2 and one Layer 3 ACL in the inbound direction only.
• On Layer 2 bridged-group virtual interfaces (BVIs), you can apply both Layer 3 and Layer 2 ACLs.
• On Layer 3 virtual LAN (VLAN) interfaces, you can apply only Layer 3 ACLs.
• In a redundant configuration, the ACE does not apply a global ACL to the FT VLAN.
For example, to apply ACL1 to all interfaces in the Admin context, enter the following command in configuration mode:
ACE_module5/Admin(config)# access-group input ACL1
The syntax of the match access-list command is as follows:
match access-list acl_name
To configure an ACL match statement in a class map, enter the following commands:
ACE_module5/Admin(config)# class-map match-any L4_CLASS

ACE_module5/Admin(config-cmap)# match access-list ACL1
ACE_module5/Admin(config-cmap)# exit
ACE_module5/Admin(config)# policy-map multi-match L4_POLICY
ACE_module5/Admin(config-pmap)# class L4_CLASS
ACE_module5/Admin(config-pmap-c)#
For more details about ACLs and how to configure them, see the Cisco Application Control Engine Module Security
Configuration Guide.
ACL-Related syslogs
When a packet matches an ACL entry, a syslog message is generated based on the following rules:
• All ACL deny entries generate a syslog message unless logging is explicitly disabled using the no logging enable
command in configuration mode.
• An ACL permit entry generates a syslog message only if logging is enabled using the logging enable command in
configuration mode.
• All implicit deny entries generate the default deny syslog (%ACE-4-106023).
To minimize syslog message generation, the ACE uses the flow cache as follows:
1. For the first packet hit on an ACL entry, the ACE generates a syslog and caches the flow (5-tuple) in the connection table.
2. For subsequent hits on the same ACL entry, the ACE checks the cache. If it finds the flow in the cache, the ACE
increments a hit counter for this entry in the cache and does not generate a syslog.
3. After some time (the default is 300 seconds, which is configurable in the ACL entry definition in the CLI as the
interval_secs option), the ACE generates a syslog and sets the hit count to 0.
4. However, if at the expiry of the above time, the hit count is 0, the ACE deletes the cache entry silently. So by default, a
cache entry is aged out 600 seconds after the last hit.
Troubleshooting ACLs
Many ACL issues manifest themselves by all traffic or only certain traffic being denied or permitted access to the ACE or out of
the ACE. Remember that, initially, all traffic to the ACE is denied until you permit traffic using an ACL. Every ACL contains an
implicit deny at the end of it, so only traffic that you explicitly permit will have access to the ACE. To troubleshoot ACLs, follow
these steps:
1. Verify that your ACL configuration is correct for your network application. Make any required changes to the running-config
file, and then test the configuration. If it is satisfactory, save it to the startup-config file using the copy runnning-config
startup-config command.
For example, to display the ACLs that you have configured in your ACE, enter the following command:
ACE_module5/Admin# show running-config access-list

access-list ACL1 remark This ACL permits any IP traffic from any source going to any destination except for ICMP t
192.168.12.15 255.255.255.192.
access-list ACL1 line 8 extended permit ip any any
access-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=0) [0x
access-list ANYONE line 8 extended permit ip any any
To verify that the configured ACLs are applied to the correct interfaces and in the right directions (input or output), enter the
following command:

interface vlan 100

ip address 10.2.1.1 255.255.255.0
access-group input ANYONE
access-group output ANYONE
no shutdown
interface vlan 200
ip address 192.168.1.1 255.255.255.0
service-policy input MGMT_POLICY
no shutdown
2. Verify that you have allocated sufficient resources for ACLs. To display the allocated resources in your ACE, enter the
following command:
ACE_module5/Admin# show resource usage

Allocation
-------------------------------------------------------------------------------
Context: Admin
conc-connections 10 18 0 8000000 0
proxy-connections 584 590 0 1048574 0
xlates 0 0 0 1048574 0
bandwidth 880 16194 0 625000000 0
throughput 880 12606 0 500000000 0
connection rate 1 21 0 1000000 0
ssl-connections rate 0 0 0 5000 0
mac-miss rate 0 16 0 2000 0
inspect-conn rate 0 0 0 6000 0
acl-memory 33448 33448 7858944 70749384 0 <------- ACL memory resource allocat
sticky 0 0 0 0 0
regexp 0 0 0 1048576 0
syslog buffer 188416 188416 0 4194304 0
syslog rate 0 9 0 100000 0
Context: C1
xlates 0 0 0 1048574 0
bandwidth 0 0 0 625000000 0
throughput 0 0 0 500000000 0
acl-memory 0 0 7858944 70749384 0
sticky 0 0 0 0 0
regexp 0 0 0 1048576 0
syslog buffer 0 0 0 4194304 0
syslog rate 0 0 0 100000 0
For example, to allocate a 10 percent minimum and a maximum of unlimited resources for ACL memory in the Admin virtual
context, enter the following commands:
ACE_module5/Admin(config)# resource myclass

ACE_module5/Admin(config-resource)# limit-resource acl-memory minimum 10 maximum unlimited
ACE_module5/Admin(config-resource)# exit
ACE_module5/Admin(config)# context Admin
ACE_module5/Admin(config-context)# member myclass
3. Display the details of an individual ACL by using the show access-list acl_name detail command. This command displays
every entry in the specified ACL, the hit counts for each entry, and a 32-bit hexadecimal MD5-hash value that the ACE computes
from the access-list command immediately when you configure an ACL. The ACE includes this hash value in deny syslog
messages (106023) to help you identify the ACL entry that caused the deny syslog. For example to display the details of the ACL1
access control list, enter the following command:
ACE_module5/Admin# show access-list ACL1 detail
access-list:ACL1, elements: 2, status: ACTIVE

remark : This ACL permits any IP traffic from any source going to any destination except for ICMP traffic origin
access-list ACL1 line 8 extended permit ip any any (hitcount=9) [0x894c1008] <------- 32-bit hexadecimal MD5-hash
access-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=15) [0
The format of the deny syslog message is as follows:
%ACE-4-106023: Deny protocol number | name src incoming-interface:src-ip dst outgoing-interface:dst-ip by access-g
An IP packet was denied by the ACL.
Explanation: This message displays even if you do not have the log option enabled for an ACL. If a packet hits an input
ACL, the outgoing interface will not be known. In this case, the ACE prints the outgoing interface as undetermined. The
source IP and destination IP addresses are the unmapped and mapped addresses for the input and output ACLs,
respectively, when used with NAT.
Troubleshooting ACLs 88
Recommended Action: If messages persist from the same source address, messages may indicate a foot-printing or
port-scanning attempt. Contact the remote host administrators.
An ACL merged list is a large ACL that the CP compiles from multiple security ACL entries and policies. When the ACE
executes an ACL merged list, it performs multiple actions on a flow that matches the merged list.
4. Display the actions that the ACE will perform on a flow by entering the show acl-merge merged-list command. For example,
to display the merged list for VLAN 100, enter the following command:
ACE_module5/Admin# show acl-merge merged-list vlan 100 in non-redundant
All ACEs in merged list 2 Total:18 Non-redundant:12
Priority:1000, Lineno:0, ACE-id:211 Action:PERMIT, Path-id:0x81/0x0/0x0:6/0[6/0]

[6/0]
Pmap:0x5, Log:FALSE/FALSE[FALSE][FALSE], Interval:0/0[0][0]
Hash1:0x0 Hash2:0x0
Generated:TRUE, need-to-add-in-comp:NO_ACT_NEEDED, redundant:FALSE
Parent:: feature:SECURITY ace-lineno:5 ACL priority:0[G:0,P:0,C:0,ACL:0]
Parent:: feature:TO CP ace-lineno:2 ACL priority:16779265[G:0,P:1,C:8,ACL:1]
Feature:SECURITY Policy:1[1][1] sec-level:0x0 Intratype:SKIP
Feature:TO CP Policy:1[1][1] sec-level:0x0 Intratype:TERMINATE
Intertype:TERMINATE
IP address SRC:161.44.0.0/255.255.0.0 DST:10.86.215.134/255.255.255.255
Ports SRC:RANGE 0 65535 DST:RANGE 22 22
Protocol:6
Hit Count:0 Active:TRUE Timerange:0
.
.
.
Feature:SECURITY Policy:0[0] sec-level:0x0 Intratype:TERMINATE
.
.
.
Feature:SLB Policy:14[14] sec-level:0x0 Intratype:TERMINATE
.
.
.
Feature:SRC NAT Policy:2[2] sec-level:0x0 Intratype:TERMINATE
.
.
.
5. If the acl-memory Denied counter in the output of the show resource usage command is incrementing and the Peak (ACL)
memory counter has not exceeded the Max Allocated ACL memory counter, the problem may lie with one of the nodes in the
ACL merge tree. The ACL merge tree contains several different kinds of nodes (see the example output below), each of a different
size and each with a maximum limit. If you allocate a minimum of 10 percent of the ACE resources to ACL memory, the ACE
will guarantee 10% of the maximum number of each node. If your configuration causes the ACE to exceed the maximum value of
one of these nodes, the ACL resource allocation will fail and the acl-memory Denied counter will increment.
To monitor the ACL merge tree node usage in the ACE, enter the following command:
ACE_module5/Admin# show np 1 access-list resource
ACL Tree Statistics for Context ID: Admin

=========================================
ACL memory max-limit: None
ACL memory guarantee: 10.00 %
MTrie nodes(used/guaranteed/max-limit):
43 / 26214 / 262143 (compressed) <-------|
3 / 2199 / 21999 (uncompressed) <--------|
Leaf Head nodes (used/guaranteed/max-limit): |
39 / 26214 / 262143 <--------------------|---- Maximum number of available nodes in the ACE
Leaf Parameter nodes (used/guaranteed/max-limit):|
594 / 52428 / 524288 <-------------------|
Policy action nodes used: 153
memory consumed: 23776 bytes resource-limited 4896 bytes other 28672 bytes total
.
min-guarantee: 7861043 bytes total, 0 % consumed.
max-limit: 78610432 bytes total, 0 % consumed.
ACL Tree Statistics for the linecard

====================================
MTrie nodes(used): 43 (compressed) 3 (uncompressed) <--------------|
(shared): 235929 (compressed) 19800 (uncompressed) |
Leaf Head nodes (used/shared): 39 / 235929 <-----------------------|---- Number of used nodes in the ACE
Leaf Parameter nodes (used/shared): 594 / 471860 <-----------------|
Policy action nodes (used/shared): 153 / 261990 <------------------|
You can calculate the percentage of use for each node type by dividing the used nodes value by the maximum number of nodes
and multiplying the result by 100. If any of these percentages exceeds the maximum value of allocated ACL memory for the
context, increase the max value of allocated ACL memory using the limit-resource acl-memory command in resource class
configuration mode so that that value is greater than or equal to the highest used nodes percentage that you calculated.
Alternatively, if you are approaching the limits of ACL resource capacity, you may consider consolidating your ACL
configuration.
If the ACL nodes are depleted while the ACE is downloading ACL configurations for an interface, the complete ACL merged list
for that interface is deleted and no traffic flows through that interface. The ACE increments the download failure counter in the
output of the show interface command and the ACE logs a system message from the configuration manager.
6. To trace a packet through a specific ACL, enter the following command:
ACE_module5/Admin# show np 1 access-list trace vlan 130 in protocol 1 source 172.27.16.23 2000 destination 192.168
Root 0x2c01b00
Src Mtrie (0) offset 1 curr 0x2c01b00 child 0x0 leaf 0x10a840
Dst Mtrie (0) offset 2 curr 0x10a840 child 0x0 leaf 0x3c01330
proto ICMP head node 0x4004880
proto node 0x4004880
src op range port 0/65535
dst op range port 0/65535 lineno 112000
inner match line#:112000
inner match line#: 112000
packet matched priority 112000
action node 0x4c02460

Action Leaf-node
version+aceid 0x99 (version 0 ace_id 153 dirty no)
action_flag 0x10 (permit no log no punt_to_cp no capture no bridge yes)
path ID 0x0
src nat 0x0 dst nat 0x0 vserver 0x0 fixup 0x0
TCP conn 0x0 AAA 0x0 Websense 0x0 QOS Policer 0x0
Syslog Info 0
Hitcount 130426
Syslog info:
idx:[153:0] name_idx:[0:0] hash1:0x0 hash2:0x0 name_len:0 invalid
Number of DRAM access: 6 (2 mtrie 4 non-mtrie)
Troubleshooting ACLs 90
This article describes ACE network address translation (NAT), how to configure it, and how to troubleshoot issues with NAT that
you may encounter.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE Network Address
Translation
• 2 NAT Configuration Guidelines and
Restrictions
• 3 Configuring Dynamic NAT and PAT
• 4 Configuring Server-Farm Based Dynamic
NAT
• 5 Configuring Static NAT and Port Redirection
• 6 Configuring SNAT with Cookie and Load
Balancing
• 7 Troubleshooting ACE NAT and PAT
Contents 91
Overview of ACE Network Address Translation

You can configure the ACE to translate a client source IP address to a routable address in the server's network. This process is
called source NAT (SNAT). If you want to preserve the client source IP address, do not configure SNAT.
You can also configure the ACE to translate the private address of a server to a global IP address that is accessible to clients. This
process is called destination NAT (DNAT) and protects the server by hiding its real IP address from the Internet.
Besides translating IP addresses, you can configure the ACE to translate TCP and UDP ports. This process is called port address
translation (PAT).
The ACE provides the following types of NAT and PAT:
• Interface-based dynamic NAT

• Interface-based dynamic PAT
• Server farm-based dynamic NAT
• Static NAT
• Static port redirection
NAT Configuration Guidelines and Restrictions

When you configure NAT and PAT on your ACE, keep in mind the following NAT and PAT guidelines and restrictions:
• If a packet egresses an interface that you have not configured for NAT, the ACE transmits the packet untranslated.
• You can configure dynamic NAT or static NAT as an input service policy only; you cannot configure it as an output
service policy.
• When you remove a traffic policy from the last VLAN interface on which you applied the service policy, the ACE
automatically resets the associated service-policy statistics. The ACE performs this action to provide a new starting point
for the service-policy statistics the next time that you attach a traffic policy to a specific VLAN interface.
Configuring Dynamic NAT and PAT

Dynamic NAT is typically used for SNAT. When you configure dynamic NAT and PAT, be sure to configure an interface for the
client-side VLAN and an interface for the server-side VLAN.
The following SNAT configuration example shows the commands that you use to configure dynamic NAT and PAT on your
ACE. In this SNAT example, packets that ingress the ACE from the 192.168.12.0 network are translated to one of the IP
addresses in the NAT pool defined on VLAN 200 by the nat-pool command. The pat keyword indicates that ports higher than
1024 are also translated.
Note: If you are operating the ACE in one-arm mode, omit the client-side interface VLAN 100 and configure the service
policy on interface VLAN 200.
access-list NAT_ACCESS line 10 extended permit tcp 192.168.12.0 255.255.255.0 172.27.16.0 255.255.255.0 eq http
Overview of ACE Network Address Translation 92

class-map match-any NAT_CLASS
match access-list NAT_ACCESS
policy-map multi-match NAT_POLICY

class NAT_CLASS
nat dynamic 1 vlan 200
interface vlan 100

mtu 1500
ip address 192.168.1.100 255.255.255.0
service-policy input NAT_POLICY
no shutdown
interface vlan 200

mtu 1500
ip address 172.27.16.2 255.255.255.0
nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 pat
no shutdown
Configuring Server-Farm Based Dynamic NAT

The following SNAT configuration example shows the commands that you use to configure server farm-based dynamic NAT on
your ACE. In this SNAT example, real server addresses on the 172.27.16.0 network are translated to one of the IP addresses in the
NAT pool defined on VLAN 200 by the nat-pool command.
Note: If you are operating the ACE in one-arm mode, omit interface VLAN 100 and configure the service policy on interface
VLAN 200.
access-list NAT_ACCESS line 10 extended permit tcp 192.168.12.0 255.255.255.0 172.27.16.0 255.255.255.0 eq http
rserver SERVER1
ip address 172.27.16.3
inservice
rserver SERVER2
inservice
serverfarm SFARM1
rserver SERVER1
inservice
rserver SERVER2
inservice
class-map type http loadbalance match-any L7_CLASS
match http content .*cisco.com
match access-list NAT_ACCESS
policy-map type loadbalance http first-match L7_POLICY

class L7_CLASS
serverfarm SFARM1
nat dynamic 1 vlan 200 serverfarm primary
class NAT_CLASS
loadbalance policy L7_POLICY
loadbalance vip inservice
interface vlan 100

mtu 1500
ip address 192.168.1.100 255.255.255.0
no shutdown
interface vlan 200

mtu 1500
Configuring Dynamic NAT and PAT 93

ip address 172.27.16.2 255.255.255.0
nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0
no shutdown
Configuring Static NAT and Port Redirection

The following DNAT configuration example shows those sections of the running configuration that are related to the commands
necessary to configure static NAT and port redirection on your ACE. Typically, this configuration is used for DNAT, where
HTTP packets that are destined to 192.0.0.0/8 and ingress the ACE on VLAN 101 are translated to 10.0.0.0/8 and port 8080. In
this example, the servers are hosting HTTP on custom port 8080.
access-list acl1 line 10 extended permit tcp 10.0.0.0 255.0.0.0 eq 8080 any

match access-list acl1

class NAT_CLASS
nat static 192.0.0.0 255.0.0.0 80 vlan 101
interface vlan 100

mtu 1500
ip address 192.168.1.100 255.255.255.0
no shutdown
interface vlan 101

mtu 1500
ip address 172.27.16.100 255.255.255.0
no shutdown
Configuring SNAT with Cookie and Load Balancing

The following configuration example shows those commands necessary to configure SNAT (dynamic NAT) with cookie load
balancing. Any source host that sends traffic to the VIP 20.11.0.100 is translated to one of the free addresses in the NAT pool in
the range 30.11.100.1 to 30.11.200.1, inclusive. If you want to use PAT instead of NAT, replace nat dynamic 1 vlan 2021 with
nat dynamic 2 vlan 2021 in the L7SLBCookie policy map.
server host http

inservice
serverfarm host httpsf
rserver http
inservice
class-map match-any vip4

2 match virtual-address 20.11.0.100 tcp eq www
class-map type http loadbalance match-any L7SLB_Cookie
3 match http cookie JG cookie-value ?.*?
policy-map type loadbalance first-match L7SLB_Cookie

class L7SLB_Cookie
serverfarm httpsf
policy-map multi-match L7SLBCookie
class vip4
loadbalance L7SLB_Cookie
interface vlan 2020

ip address 20.11.0.2 255.255.0.0
alias 20.11.0.1 255.255.0.0
peer ip address 20.11.0.3 255.255.0.0
service-policy input L7SLBCookie
no shutdown
interface vlan 2021
ip address 30.11.0.2 255.255.0.0
alias 30.11.0.1 255.255.0.0
peer ip address 30.11.0.3 255.255.0.0
fragment min-mtu 68
nat-pool 3 30.11.202.1 30.11.202.3 netmask 255.255.255.255
nat-pool 1 30.11.100.1 30.11.200.1 netmask 255.255.255.255
no shutdown
Troubleshooting ACE NAT and PAT

To verify your NAT and PAT configurations and make any necessary corrections, follow these steps:
1. Display your NAT and PAT configurations by entering the following commands:
class-map match-any L4_CLASS

2 match access-list ACL1
ACE_module5/Admin# show running-config policy-map

class NAT_CLASS
ACE_module5/Admin# show service-policy NAT_POLICY
Status : ACTIVE
-----------------------------------------
Interface: vlan 100
service-policy: NAT_POLICY
class: NAT_CLASS
nat:
dropped conns : 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
interface vlan 100

ip Address 192.168.12.2
mtu 1500
no shutdown
interface vlan 200
ip address 172.27.16.2 255.255.255.0
mtu 1500
access-group input acl1
no shutdown
Configuring SNAT with Cookie and Load Balancing 95

2. Use the show xlate command to verify that dynamic NAT and PAT, and static NAT and port redirection, are taking place
properly.
Dynamic NAT Example
The following example output of the show xlate command shows dynamic NAT (SNAT in this example). When you use
Telnet from IP address 172.27.16.5 in VLAN 2020, the ACE translates it to IP address 192.168.100.1 in VLAN 2021.
host1/Admin# show xlate global 192.168.100.1 192.168.100.10
NAT from vlan2020:172.27.16.5 to vlan2021:192.168.100.1 count:1
Dynamic PAT Example
The following example shows dynamic PAT. When you use Telnet from IP address 172.27.16.5 port 38097 in VLAN
2020, the ACE translates it to IP address 192.168.201.1 port 1025 in VLAN 2021.
host1/Admin# show xlate
TCP PAT from vlan2020:172.27.16.5/38097 to vlan2021:192.168.201.1/1025
Static NAT Example
The following example shows static NAT. The ACE maps real IP address 172.27.16.5 to IP address 192.168.210.1.
NAT from vlan2020:172.27.16.5 to vlan2021:192.168.210.1 count:1
host1/Admin# show conn
conn-id dir prot vlan source destination state

----------+---+----+----+----------------+----------------+----------+
7 in TCP 2020 172.27.16.5 192.168.100.1 ESTAB
6 out TCP 2021 192.168.100.1 192.168.210.1 ESTAB
Static Port Redirection (Static PAT) Example
The following example shows static port redirection (DNAT in this example). A host at IP address 192.168.0.10:37766
uses Telnet to connect to IP address 192.168.211.1:3030 on VLAN 2021 on the ACE. The ACE maps IP address
172.27.0.5:23 on VLAN 2020 to IP address 192.168.211.1:3030 on VLAN 2021.
TCP PAT from vlan2020:172.27.0.5/23 to vlan2021:192.168.211.1/3030

Mar 24 2006 20:05:41 : %ACE-7-111009: User 'admin' executed cmd: show xlate
host1/Admin# show conn

conn-id dir prot vlan source destination state

----------+---+----+----+------------------+------------------+------+
6 in TCP 2021 192.168.0.10:37766 192.168.211.1:3030 ESTAB
7 out TCP 2020 172.27.0.5:23 192.168.0.10:1025 ESTAB
3. To display the NAT policy and pool information for the current context, enter the show nat-fabric command. The syntax of
this command is as follows:
show nat-fabric {policies | src-nat policy_id mapped_if | dst-nat static_xlate_id | nat-pools | implicit-pat| global-static}
policies -- Displays the NAT policies.
src-nat policy_id mapped_if -- Displays the specified source NAT policy information. To obtain the values for the
policy_id and mapped_if arguments, view the policy_id and mapped_if fields displayed by the show nat-fabric policies
command.
dst-nat static_xlate_id -- Displays the static address translation for the specified static XLATE ID. To obtain the value for
the static_xlate_id argument, view the static_xlate_id field displayed by the show nat-fabric policies command.
nat-pools -- Displays NAT pool information for a dynamic NAT policy.
implicit-pat -- Displays the implicit PAT policies.
global-static -- Displays global static NAT information when the static command in global configuration mode is
configured.
ACE_module5/Admin# show nat-fabric policies
Nat objects:
NAT object Hash Bucket: 9

NAT object ID:2 mapped_if:8 policy_id:1 type:DYNAMIC nat_pool_id:4
Pool ID:4 PAT:1 pool_id:1 mapped_if:8 Ref_count:1 ixp_binding:in all IXPs
lower:172.27.16.15 upper:172.27.16.24 Bitmap-ID:40
List of NAT object IDs: 2
This article describes ACE health monitoring (probes), how to configure it, and how to troubleshoot issues with probes that you
may encounter.
Guide Contents
Main Article
Troubleshooting ACE NAT and PAT 97


Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE Health
Monitoring
♦ 1.1 Configuring Probes
♦ 1.2 Example of a Probe
Configuration
• 2 Troubleshooting ACE Health
Monitoring
♦ 2.1 Troubleshooting an
HTTP Probe Error
HTTPS Probe Error
SNMP Probe Issue
♦ 2.4 Using the Last Status
Code Field
Overview of ACE Health Monitoring

This section describes health monitoring on the ACE. The ACE uses probes that you configure to track the state of a server. By
default, no probes are configured in the ACE. Also referred to as out-of-band (OOB) health monitoring, the ACE verifies the
server response to a probe or checks for any network problems that can prevent a client from reaching a server. Based on the
Contents 98
server response, the ACE can place the server in or out of service and can make reliable load-balancing decisions.
You can also use health monitoring to detect failures for a gateway or a host in high-availability (redundant) configurations. For
more information, see the Cisco Application Control Engine Module Administration Guide.
The ACE evaluates the health of a server by marking the probes as follows:
• Passed?The server returns a valid response.
• Failed?The server fails to provide a valid response to the ACE and the ACE is unable to reach a server for a specified
number of retries.
By configuring the ACE for health monitoring, the ACE sends active probes periodically to determine the server state. The ACE
supports 4096 unique probe configurations, which includes ICMP, TCP, HTTP, and other predefined health probes. The ACE can
execute only up to 200 concurrent scripted probes at a time. The ACE also allows the opening of 2048 sockets simultaneously.
You can associate the same probe with multiple real servers or server farms. Each time that you use the same probe again, the
ACE counts it as another probe instance. You can allocate a maximum of 16,000 probe instances.
Configuring Probes
You can configure health probes on the ACE to actively make connections and explicitly send traffic to servers. The probes
determine whether the health status of a server passes or fails by the server's response.
Configuring active probes is a three-step process:
1. Configure the health probe with a name, type, and attributes.
2. Associate the probe with one of the following:
◊ A real server.
◊ A real server and then associate the real server with a server farm. You can associate a single probe or multiple
probes with real servers within a server farm.
◊ A server farm. All servers in the server farm receive probes of the associated probe types.
3. Place the real server or server farm in service.
Example of a Probe Configuration

The following example shows a running configuration that load balances DNS traffic across multiple real servers and transmits
and receives UDP data that spans multiple packets. The configuration uses a UDP health probe.
probe udp UDP

interval 5
passdetect interval 10
description THIS PROBE IS INTENDED FOR LOAD BALANCING DNS TRAFFIC
port 53
send-data UDP_TEST
rserver host SERVER1

ip address 192.168.10.45
Overview of ACE Health Monitoring 99

inservice
ip address 192.168.10.46
inservice
ip address 192.168.10.47
inservice
serverfarm host SFARM1

probe UDP
rserver SERVER1
inservice
rserver SERVER2
inservice
rserver SERVER3
inservice
class-map match-all L4UDP-VIP_114:UDP_CLASS

2 match virtual-address 192.168.120.114 udp eq 53
policy-map type loadbalance first-match L7PLBSF_UDP_POLICY

class class-default
serverfarm SFARM1
policy-map multi-match L4SH-Gold-VIPs_POLICY
class L4UDP-VIP_114:UDP_CLASS
loadbalance policy L7PLBSF_UDP_POLICY
loadbalance vip icmp-reply
connection advanced-options 1SECOND-IDLE
interface vlan 120

description Upstream VLAN_120 - Clients and VIPs
ip address 192.168.120.1 255.255.255.0
fragment chain 20
fragment min-mtu 68
service-policy input L4SH-Gold-VIPs_POLICY
no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254

This section describes how to troubleshoot two common probe configuration issues.
Troubleshooting an HTTP Probe Error

In this first scenario, you have configured an HTTP probe, but the real server's health status is displayed as FAILED and the Last
disconnect err field indicates that an invalid status code was received as displayed in the show probe detail command output. You
have checked your server and it is up and running. A packet capture on the server also shows that everything is fine. Where is the
issue?
1. Display the probe status details by entering the following command:
ACE_module5/Admin# show probe detail
probe : HTTP_PROBE
type : HTTP
state : ACTIVE
description :
----------------------------------------------
port : 80 address : 0.0.0.0 addr type : -
interval : 10 pass intvl : 10 pass count : 3
fail count: 3 recv timeout: 10
http method : GET
http url : /
conn termination : GRACEFUL
expect offset : 0 , open timeout : 1
expect regex : -
send data : -
------------------ probe results ------------------
associations ip-address port porttype probes failed passed health
------------ ---------------+-----+--------+--------+--------+--------+------
rserver : SERVER1
192.168.10.45 80 -- 2 2 0 FAILED
Socket state : CLOSED

No. Passed states : 0 No. Failed states : 1
No. Probes skipped : 0 Last status code : 200 <------- Last status code from server
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : Received invalid status code <-------
Last probe time : Tue Apr 7 16:17:26 2009
Last fail time : Tue Apr 7 16:17:16 2009
Last active time : Never
The Last disconnect err field indicates that the ACE received an invalid status code. This error means that you have has not
configured the expect status command for the probe.
2. Confirm this finding by entering the following command:
ACE_module5/Admin# show running-config probe

probe http HTTP_PROBE

interval 10
open 1
3. Correct the problem by entering the following commands:
ACE_module5/Admin(config)# probe http HTTP_PROBE
ACE_module5/Admin(config-probe-http)# expect status 200 200 <------- 200 indicates the 200 OK message from the ser
ACE_module5/Admin(config-probe-http)# end
4. Confirm the configuration by entering the following command:

probe http HTTP_PROBE

interval 10
expect status 200 200
open 1
5. Display the probe status details again and observe that the server health status value is SUCCESS by entering the following
command:
Troubleshooting an HTTP Probe Error 101

ACE_module5/Admin# show probe HTTP_PROBE detail
probe : HTTP_PROBE
type : HTTP
state : ACTIVE
description :
----------------------------------------------
http method : GET
http url : /
conn termination : GRACEFUL
expect offset : 0 , open timeout : 1
expect regex : -
send data : -
------------------ probe results ------------------
------------ ---------------+-----+--------+--------+--------+--------+------
rserver : SERVER1
192.168.10.45 80 -- 24 15 9 SUCCESS

No. Probes skipped : 0 Last status code : 200
Last disconnect err : - <------- No error indicated now. The probe is successful.
Last probe time : Tue Apr 7 16:21:05 2009
Last active time : Tue Apr 7 16:20:05 2009
Troubleshooting an HTTPS Probe Error

In addition to the methods for Troubleshooting an HTTP Probe Error, use SSL statistics to troubleshoot HTTPS probe failures.
HTTPS probe traffic runs in the Admin virtual context so view the output of the show stats crypto client command in that context.
Troubleshooting an SNMP Probe Issue

In this scenario, you have configured an SNMP probe, but the Last disconnect err field indicates that the sum of the weights does
not add up to the maximum weight value as displayed in the output of the show probe detail command.
1. Display the probe status details by entering the following command:
ACE_module5/test# show probe detail
probe : SNMP_PROBE
type : SNMP
state : ACTIVE
description : snmp probe
----------------------------------------------
version : 2c community : test_comm
oid string #1 : .1.3.6.1.2.1.4.3.0
type : ABSOLUTE max value : 1000000000
weight : 10000 threshold : 1000000000
------------------ probe results ------------------
------------ ---------------+-----+--------+--------+--------+--------+------
serverfarm : least-loaded, predictor least-loaded
real : SERVER1[0]
Troubleshooting an HTTPS Probe Error 102

192.168.10.45 161 -- 0 0 0 INIT

Last disconnect err : Sum of weights don't add up to max weight value <------- Error condition
Last probe time : Never
Last fail time : Never
Last active time : Never
Server load : 16000 <------- Note the server load value
The reason for this error is that the weight command needs to be configured when you have multiple OIDs configured for a single
probe and from those OIDs if you want to give priority to a specific OID.
The sum of the weights should equal 16000 (see the Server Load field). For a single OID, the weight command does not have any
significance.
2. Display the probe configuration by entering the following command:
probe snmp SNMP_PROBE

description snmp probe
port 161
interval 15
version 2c
community TEST_COMM
oid .1.3.6.1.2.1.4.3.0
type absolute max 1000000000
weight 10000 <-------
In the above configuration, the weight is configured as 10000 for a single OID. The ACE is expecting another OID to be
configured in the probe and the sum of both weights should equal 16000.
The configuration is not complete and the ACE is expecting additional parameters in the probe configuration. Because there is not
another OID in the configuration, the ACE is not able to calculate the load and that is why the "Sum of weights don't add up to
max weight value" error message appears.
3. Resolve the issue by modifying the probe configuration as follows:
probe snmp SNMP_PROBE

description test
port 161
interval 15
version 2c
community test_comm
oid .1.3.6.1.2.1.4.3.0
weight 10000
oid .1.3.6.1.2.1.4.10.0
weight 6000 <------- 10000 + 6000 = 16000
4. Display the probe status details again by entering the following command:
ACE_module5/test# show probe SNMP_PROBE detail
probe : snmp1
Troubleshooting an SNMP Probe Issue 103

type : SNMP
state : ACTIVE
description : snmp probe
----------------------------------------------
version : 2c community : test_comm
oid string #1 : .1.3.6.1.2.1.4.3.0

oid string #2 : .1.3.6.1.2.1.4.10.0

------------------ probe results ------------------
------------ ---------------+-----+--------+--------+--------+--------+------
serverfarm : least-loaded, predictor least-loaded
real : SERVER1[0]
192.168.10.45 161 -- 4143 0 4143 SUCCESS

Last disconnect err : - <------- No error indicated now. The probe is successful.
Last probe time : Mon Apr 6 09:12:54 2009
Last fail time : Never
Last active time : Sun Apr 5 15:57:28 2009
Server load : 0
Using the Last Status Code Field

Details regarding the last status code field can be provided for nontrivial probes. For example, in the case of scripted probe
PROBENOTICE_PROBE, the status code 30001 means that the probe is successful and the value 30002 indicates an error in
probe arguments. The last disconnect error for status code 30002 displays "Did not receive correct response from the server," but
the actual issue is related to arguments in the probe configuration, which can be checked by looking at the script for the probe.
ACE_module5/Admin# show probe TEST detail
probe : TEST
type : SCRIPTED
state : ACTIVE
description :
----------------------------------------------
script filename : PROBENOTICE_PROBE
--------------------- probe results --------------------
probe association probed-address probes failed passed health
------------------- ---------------+----------+----------+----------+-------
serverfarm : sf1
real : rs2[0]
23.0.0.5 4082 54 4028 SUCCESS
Socket state : RESET

No. Probes skipped : 8 Last status code : 30001 <------- Indicates success
Last disconnect err : -
Using the Last Status Code Field 104

Last probe time : Wed Apr 8 04:44:41 2009
Last active time : Tue Apr 7 12:03:45 2009
This article describes how to troubleshoot Layer 4 (L4) load balancing on the ACE.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE L4 Load Balancing
♦ 1.1 Classifying L4 Traffic for Server
Load Balancing
♦ 1.2 Example of a Layer 4 Load-Balancing
Configuration
• 2 Troubleshooting L4 Load Balancing on the
ACE
Contents 105
Overview of ACE L4 Load Balancing

Load balancing at L4 involves selecting a server in a server farm to service a client request based on the VIP address and protocol
in the request. You configure a class map to classify (match) interesting traffic arriving at the ACE and associate the class map
with a policy map to perform an action on the traffic based on the classification. With L4 load balancing, the ACE selects a server
based on the first packet it receives in a particular flow. See the "Overview of ACE Connection Handling" section in the
Troubleshooting Connectivity article.
For detailed information about ACE load balancing, see the Cisco Application Control Engine Module Server Load Balancing
Classifying L4 Traffic for Server Load Balancing

You classify inbound network traffic destined to or passing through the ACE based on a series of flow match criteria specified by
a class map. Each class map defines a traffic classification, which is network traffic that is of interest to you. A policy map defines
a series of actions (functions) that you want applied to a set of classified inbound or outbound traffic.
ACE L3 and L4 traffic policies support the following server load-balancing (SLB) traffic attributes:
• Source or destination IP address

• Source or destination port
• Virtual IP (VIP) address
• IP protocol
The three major steps in the traffic classification process are as follows:
1. Create a class map using the class-map command and the associated match commands, which comprise a set of match
criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocol classifications.
2. Create a policy map using the policy-map command, which refers to the class maps and identifies a series of actions to
perform based on the traffic match criteria.
3. Activate the policy map by associating it with a specific VLAN interface or globally with all VLAN interfaces using the
service-policy command to filter the traffic received by the ACE.
Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7 policies that the
ACE uses for SLB. The figure also shows how you associate the various components of the SLB policy configuration with each
other.
Overview of ACE L4 Load Balancing 106

Figure 1. SLB Flow Diagram
Example of a Layer 4 Load-Balancing Configuration

The following example shows a L4 load-balancing configuration:

ip address 192.168.252.245
inservice

ip address 192.168.252.246
inservice

ip address 192.168.252.247
inservice
Classifying L4 Traffic for Server Load Balancing 107


ip address 192.168.252.248
inservice

ip address 192.168.252.249
inservice

ip address 192.168.252.250
inservice

probe TCP_PROBE
predictor roundrobin
rserver SERVER1
weight 10
inservice
rserver SERVER2
weight 20
inservice
rserver SERVER3
weight 30
inservice

probe TCP_PROBE
predictor roundrobin
rserver SERVER4
weight 10
inservice
rserver SERVER5
weight 20
inservice
rserver SERVER6
weight 30
inservice
class-map match-all L4WEB_CLASS

policy-map type loadbalance first-match LB_WEB_POLICY

class class-default
serverfarm SFARM1 backup SFARM2
policy-map multi-match L4WEB_POLICY

class L4WEB_CLASS
loadbalance policy LB_WEB_POLICY
loadbalance vip icmp-reply active
nat dynamic 1 VLAN 120
interface vlan 100

ip address 192.168.120.1 255.255.255.0
fragment chain 20
fragment min-mtu 68
service-policy input L4WEB_POLICY
no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Example of a Layer 4 Load-Balancing Configuration 108

Troubleshooting L4 Load Balancing on the ACE

To troubleshoot L4 load-balancing issues, follow these steps:
1. Ensure that your load-balancing configuration is correct and that the following conditions exist:
◊ Real servers have valid IP addresses and are in service.

◊ Servers are associated with server farms of the same type.
◊ A load-balancing policy exists with an associated server farm and is associated with a L4 multimatch policy.
◊ An L4 class map contains a valid match virtual-address command and is associated with the L4 multimatch
policy map.
◊ The L4 policy is applied to the appropriate active interface using a service policy.
◊ A static route is configured for the server network.
Use the following show commands:
◊ show running-config rserver

◊ show running-config serverfarm
◊ show running-config policy-map
◊ show running-config class-map
◊ show running-config interface
◊ show ip route
2. Check the ACE connectivity. See the "Troubleshooting Connectivity" section.
3. Verify that the L4 VIP class map is referenced in a L4 policy by entering the following command. Also, check the following
fields:
◊ VIP address and port

◊ VIP state
◊ Hit count
◊ Dropped connections
ACE_module5/Admin# show service-policy L4WEB_POLICY detail
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 100
service-policy: L4WEB_POLICY <------- L4 multimatch policy map
class: L4WEB_CLASS <------- L4 VIP class map
192.168.120.112 tcp eq 80 <------- VIP address, protocol, and port
loadbalance:
L7 loadbalance policy: LB_WEB_POLICY
VIP ICMP Reply : ENABLED
VIP State: INSERVICE <------- VIP state should be INSERVICE
dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded
L7 Loadbalance policy : LB_WEB_POLICY
LB action :
serverfarm: SFARM1
hit count : 0 <-------|-- Check these counters to see if they are increasing
Troubleshooting L4 Load Balancing on the ACE 109

dropped conns : 0 <-------|
The dropped conns counter under a VIP in the output of the show service policy detail command is incremented whenever the
ACE discards a connection request destined to that VIP. There are several reasons why the ACE discards such connection
requests. For example:
• If all the real servers in the server farm associated with the VIP go down, then the VIP will go down. So, all the incoming
connections to that VIP are discarded.
• If the URL in a connection request to the VIP is unknown, then the connection request is discarded.
• If the server to which the ACE load balances the connection does not respond to the request, then, after the maximum
number of retries, the ACE discards the connection.
The dropped conns counter is cumulative and the value may comprise entries from any of the following show command counters:
• show stats loadbalance
- Total Layer4 rejections

- Total Layer7 rejections
- Total Layer4 LB policy misses
- Total Layer7 LB policy misses
- Total times rserver was unavailable
• show stats connection
- Total Connections Timed-out

- Total Connections Failed
• The failures counter of the show serverfarm serverfarm_name command
• The Total drop decisions counter of the show stats inspect command
4. Verify that the L4 policy is applied as a service policy to an active interface by entering the following command:

interface vlan 100

ip address 192.168.120.1 255.255.255.0
access-group output anyone
no shutdown
.
.
.
5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also, check the IP
address, state, and the connection statistics for each real server that is configured in the server farm.
ACE_module5/Admin# show serverfarm SFARM1 detail
serverfarm : SFARM1, type: HOST

total rservers : 3
active rservers: 3
description : -
state : ACTIVE <------- Current state of the server farm
predictor : ROUNDROBIN <------- Load-balancing method
weight : -
autoadjust : MAXLOAD
failaction : -
back-inservice : 40
total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded
---------------------------------
----------connections-----------
---+---------------------+------+------------+----------+----------+---------
rserver: SERVER1
192.168.252.245:0 10 INSERVICE 0 0 0 <------- Real server IP address, state, an
max-conns : 4000000 , out-of-rotation count : 0
min-conns : 4000000
load value : 0
rserver: SERVER2
192.168.252.246:0 20 INSERVICE 0 0 0
min-conns : 4000000
load value : 0
rserver: SERVER3
192.168.252.247:0 30 INSERVICE 0 0 0
min-conns : 4000000
load value : 0
6. Check the L4 load-balance statistics by entering the following command:
+------------------------------------------+
+------------------------------------------+
Total Layer4 rejections : 0
Total times rserver was unavailable : 0
Note: The ID Map is used to map real servers and server farms between the local and the remote peers in a redundant
configuration. The Total IDMap Lookup Failures field increments if the local ACE fails to find the local ACE to peer
ACE ID mapping. A failure can occur if the peer ACE did not send a proper remote ID for the local ACE to look up and
so the local ACE could not perform a mapping or if the ID Map table was not created.
Troubleshooting L4 Load Balancing on the ACE 111

This article describes how to diagnose and troubleshoot ACE L7 load-balancing issues.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE Layer 7 Load Balancing
♦ 1.1 Load-Balancing Predictors
♦ 1.2 Classifying L7 Traffic for Server
Load Balancing
• 2 Example of a L7 Load-Balancing
Configuration
• 3 Troubleshooting Layer 7 Load Balancing
Contents 112
Overview of ACE Layer 7 Load Balancing

Layer 7 server load balancing (SLB) is the process that the load-balancing device uses to decide which server should fulfill a
client request for an L7 service. For example, a client request may consist of a HyperText Transport Protocol (HTTP) GET for a
web page or a File Transfer Protocol (FTP) GET to download a file. The job of the load balancer is to select the server that can
successfully fulfill the client request and do so in the shortest amount of time without overloading either the server or the server
farm as a whole.
The ACE supports the load balancing of the following protocols:
• Generic protocols
• HTTP
• Remote Authentication Dial-In User Service (RADIUS)
• Reliable Datagram Protocol (RDP)
• Real-Time Streaming Protocol (RTSP)
• Session Initiation Protocol (SIP)
Depending on the load-balancing algorithm?or predictor?that you configure, the ACE performs a series of checks and calculations
to determine which server can best service each client request. The ACE bases server selection on several factors including the
source or destination address, cookies, URLs, HTTP headers, or the server with the fewest connections with respect to load.
For detailed information about ACE load balancing, see the Cisco Application Control Engine Module Server Load Balancing
Load-Balancing Predictors
The ACE uses the following predictors to select the best server to fulfill a client request:
• Application response?Selects the server with the lowest average response time for the specified response-time
measurement based on the current connection count and server weight (if configured).
• Hash address?Selects the server using a hash value based on either the source or destination IP address or both. Use these
predictors for firewall load balancing (FWLB). For more information about FWLB, see Configuring Firewall Load
Balancing in the Cisco Application Control Engine Module Server Load-Balancing Configuration Guide (Software
Version A2(1.0)).
• Hash content?Selects the server using a hash value based on a content string in the Trusted Third Parties (TTP) packet
body.
• Hash cookie?Selects the server using a hash value based on a cookie name.
• Hash header?Selects the server using a hash value based on the HTTP header name.
• Hash URL?Selects the server using a hash value based on the requested URL. You can specify a beginning pattern and an
ending pattern to match in the URL. Use this predictor method to load balance cache servers.
Overview of ACE Layer 7 Load Balancing 113

• Least bandwidth?Selects the server that processed the least amount of network traffic based on the average bandwidth that
the server used over a specified number of samples.
• Least connections?Selects the server with the fewest number of active connections based on the server weight. For the
least-connections predictor, you can configure a slow-start mechanism to avoid sending a high rate of new connections to
servers that you have just put into service.
• Least loaded?Selects the server with the lowest load based on information obtained from Simple Network Management
Protocol (SNMP) probes. To use this predictor, you must associate an SNMP probe with it.
• Round-robin?Selects the next server in the list of real servers based on the server weight (weighted round-robin). Servers
with a higher weight value receive a higher percentage of the connections. This is the default predictor.
Note: The hash predictor methods do not recognize the weight value that you configure for real servers. The ACE uses the
weight that you assign to real servers only in the least-connections, application-response, and round-robin predictor
methods.
Classifying L7 Traffic for Server Load Balancing
You classify inbound network traffic destined to or passing through the ACE based on a series of flow match criteria specified by
a class map. Each class map defines a traffic classification, which is network traffic that is of interest to you. A policy map defines
a series of actions (functions) that you want applied to a set of classified inbound or outbound traffic.
ACE traffic policies support the following server load-balancing (SLB) traffic attributes:
• Layer 3 and Layer 4 connection information?Source or destination IP address, source or destination port, virtual IP
address, and IP protocol
• Layer 7 protocol information?Hypertext Transfer Protocol (HTTP) cookie, HTTP URL, HTTP header, Remote
Authentication Dial-In User Service (RADIUS), Remote Desktop Protocol (RDP), Real-Time Streaming Protocol
(RTSP), Session Initiation Protocol (SIP), and Secure Sockets Layer (SSL)
The three major steps in the traffic classification process are as follows:
1. Create a class map using the class-map command and the associated match commands, which comprise a set of match
criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocol classifications.
2. Create a policy map using the policy-map command, which refers to the class maps and identifies a series of actions to
perform based on the traffic match criteria.
3. Activate the policy map by associating it with a specific VLAN interface or globally with all VLAN interfaces using the
service-policy command to filter the traffic received by the ACE.
Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7 policies that the
ACE uses for SLB. The figure also shows how to associate the various components of the SLB policy configuration with each
other.
Figure 1. SLB Flow Diagram
Load-Balancing Predictors 114

Example of a L7 Load-Balancing Configuration

The following example shows a running configuration that includes multiple class maps and policy maps that define a traffic
policy for SLB. In this configuration, when a server farm is chosen for a connection, the connection is sent to a real server based
on one of several load-balancing predictors. The leastconns predictor method load balances connections to the server that has the
lowest number of open connections.
probe tcp TCP

interval 5
faildetect 2
open 3
parameter-map type http PERSIST-REBALANCE

persistence-rebalance <---------------------------- Enabled by default in the ACE appliance. Enabled by
parameter-map type connection PRED-CONNS-UDP_CONN
set timeout inactivity 300
rserver SERVER1
Classifying L7 Traffic for Server Load Balancing 115

ip address 10.1.0.2
inservice
rserver SERVER2
ip address 10.1.0.3
inservice
rserver SERVER3
ip address 10.1.0.4
inservice
rserver SERVER4
ip address 10.1.0.5
inservice
rserver SERVER5
ip address 10.1.0.6
inservice
rserver SERVER6
ip address 10.1.0.7
inservice
rserver SERVER7
ip address 10.1.0.8
inservice
rserver SERVER8
ip address 10.1.0.9
inservice
serverfarm host PRED-CONNS

predictor leastconns
rserver SERVER1
inservice
rserver SERVER2
inservice
rserver SERVER3
inservice
rserver SERVER4
inservice
rserver SERVER5
inservice
rserver SERVER6
inservice
rserver SERVER7
inservice
rserver SERVER8
inservice
serverfarm host PRED-CONNS-UDP

failaction purge
predictor leastconns
rserver SERVER1
inservice
rserver SERVER2
inservice
rserver SERVER3
probe ICMP
inservice
rserver SERVER5
inservice
rserver SERVER6
inservice
rserver SERVER7
inservice
serverfarm host PREDICTOR

probe TCP
rserver SERVER1
inservice
rserver SERVER2
inservice
Example of a L7 Load-Balancing Configuration 116

rserver SERVER6
inservice
rserver SERVER7
inservice
sticky http-cookie COOKIE_TEST STKY-GRP-43

cookie offset 1 length 999
timeout 30
replicate sticky
serverfarm PREDICTOR
class-map match-all L4PRED-CONNS-UDP-VIP_128:2222_CLASS

2 match virtual-address 192.168.120.128 udp eq 0
class-map match-all L4PRED-CONNS-VIP_128:80_CLASS
class-map match-all L4PREDICTOR_117:80_CLASS
policy-map type loadbalance first-match L7PLBSF_PRED-CONNS_POLICY

class class-default
serverfarm PRED-CONNS
policy-map type loadbalance first-match L7PLBSF_PRED-CONNS-UDP_POLICY
class class-default
serverfarm PRED-CONNS-UDP
policy-map type loadbalance first-match L7PLBSF_PREDICTOR_POLICY
class class-default
sticky-serverfarm STKY-GRP-43
policy-map multi-match L4SH-Gold-VIPs_POLICY
class L4PREDICTOR_117:80_CLASS
loadbalance policy L7PLBSF_PREDICTOR_POLICY
appl-parameter http advanced-options PERSIST-REBALANCE
class L4PRED-CONNS-VIP_128:80_CLASS
loadbalance policy L7PLBSF_PRED-CONNS_POLICY
class L4PRED-CONNS-UDP-VIP_128:2222_CLASS
loadbalance policy L7PLBSF_PRED-CONNS-UDP_POLICY
connection advanced-options PRED-CONNS-UDP_CONN
interface vlan 120

ip address 192.168.120.1 255.255.255.0
fragment chain 20
fragment min-mtu 68
service-policy input L4SH-Gold-VIPs_POLICY
no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254

To troubleshoot L7 load-balancing issues, use the following steps:
Troubleshooting Layer 7 Load Balancing 117

1. Ensure that your load-balancing configuration is correct and that the following conditions exist:
◊ Real servers have valid IP addresses and are in service

◊ Servers are associated with server farms of the same type
◊ L7 load-balancing policy exists with an associated server farm and that the L7 load-balancing policy is associated
with a L4 multimatch policy
◊ An L4 class map contains a valid match virtual-address command and is associated with the L4 multimatch
policy map
◊ The L4 policy is applied to the appropriate active interface using a service policy
◊ A static route is configured for the server network
Use the following show commands to verify your load-balancing configuration:
◊ show running-config rserver

◊ show running-config serverfarm
◊ show running-config policy-map
◊ show running-config class-map
◊ show running-config interface
◊ show ip route
2. Check the ACE connectivity. See the Troubleshooting Connectivity section.
3. Verify that the L7 load-balancing policy is referenced in the L4 policy by entering the following command. Also, check the
following fields:
◊ VIP address, protocol, and port

◊ VIP state
◊ Hit count
◊ Dropped connections
ACE_module5/Admin# show service-policy L4WEB_POLICY detail
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 100
service-policy: L4WEB_POLICY
class: L4WEB_CLASS
192.168.120.112 tcp eq 80 <------- VIP address, protocol, and port
loadbalance:
L7 loadbalance policy: LB_WEB_POLICY <-------L7 load-balancing policy referenced in the L4 multimatch poli
VIP ICMP Reply : ENABLED
VIP State: INSERVICE <------- VIP state should be INSERVICE
dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded
L7 Loadbalance policy : LB_WEB_POLICY <------- L7 policy statistics
LB action :
serverfarm: SFARM1
hit count : 0 <-------|-- Check these counters to see if they are increasing
dropped conns : 0 <-------|
4. Verify that the L4 policy is applied as a service policy to an active interface by entering the following command:
interface vlan 100

ip address 192.168.120.1 255.255.255.0
access-group output anyone
no shutdown
.
.
.
5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also, check the IP
address, state, and the connection statistics for each real server that is configured in the server farm.
ACE_module5/Admin# show serverfarm SFARM1 detail
serverfarm : SFARM1, type: HOST

total rservers : 3
active rservers: 3
description : -
state : ACTIVE <------- Current state of the server farm
predictor : ROUNDROBIN <------- Load-balancing method
weight : -
autoadjust : MAXLOAD
failaction : -
back-inservice : 40
total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded
---------------------------------
----------connections-----------
---+---------------------+------+------------+----------+----------+---------
rserver: SERVER1
192.168.252.245:0 10 INSERVICE 0 0 0 <------- Real server IP address, state, an
min-conns : 4000000
load value : 0
rserver: SERVER2
192.168.252.246:0 20 INSERVICE 0 0 0
min-conns : 4000000
load value : 0
rserver: SERVER3
192.168.252.247:0 30 INSERVICE 0 0 0
min-conns : 4000000
load value : 0
Note:

The connection failures counter increments only if the ACE attempts to load balance a connection and the ACE does not
receive a SYN-ACK from the real server in response to a SYN or if the real server responds to the SYN with a RST.
6. Check the L7 load-balance statistics by entering the following command:
+------------------------------------------+
+------------------------------------------+
Total times rserver was unavailable : 0
Note: The ID Map is used to map real servers and server farms between the local and the remote peers in a redundant
configuration. The Total IDMap Lookup Failures field increments if the local ACE fails to find the local ACE to peer
ACE ID mapping. A failure can occur if the peer ACE did not send a proper remote ID for the local ACE to look up and
so the local ACE could not perform a mapping or if the ID Map table was not created.
7. If you are having problems with HTTP, check the HTTP statistics and error counters by entering the following command:
ACE_module5/Admin# show stats http
+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 0 , TCP data msgs sent : 0
Inspect parse result msgs : 0 , SSL data msgs sent : 0
sent
TCP fin/rst msgs sent : 0 , Bounced fin/rst msgs sent: 0
SSL fin/rst msgs sent : 0 , Unproxy msgs sent : 0
Drain msgs sent : 0 , Particles read : 0
Reuse msgs sent : 0 , HTTP requests : 0
Reproxied requests : 0 , Headers removed : 0
Headers inserted : 0 , HTTP redirects : 0
HTTP chunks : 0 , Pipelined requests : 0
HTTP unproxy conns : 0 , Pipeline flushes : 0
Whitespace appends : 0 , Second pass parsing : 0
Response entries recycled : 0 , Analysis errors : 0
Header insert errors : 0 , Max parselen errors : 0
Static parse errors : 0 , Resource errors : 0
Invalid path errors : 0 , Bad HTTP version errors : 0
Headers rewritten : 0 , Header rewrite errors : 0
8. If you suspect a probe issue, for example, a TCP probe, check the probe statistics and error counters by entering the following
command:
ACE_module5/Admin# show stats probe type tcp
+------------------------------------------+
+----------- Probe statistics -------------+
+------------------------------------------+
----- tcp probe ----
Total probes sent : 0 Total send failures : 0
Total probes passed : 0 Total probes failed : 0
Total connect errors : 0 Total conns refused : 0
Total RST received : 0 Total open timeouts : 0

Total receive timeout : 0 Total active sockets : 0
9. Check the parameter map statistics for an HTTP parameter map by entering the following command:
ACE_module5/Admin# show parameter-map HTTP_PMAP
Number of parameter-maps : 1
Parameter-map : HTTP_PMAP
Type : http
server-side connection reuse : disabled
case-insensitive parsing : disabled
persistence-rebalance : disabled <---------------------------- Enabled by default in the ACE
header modify per-request : disabled
header-maxparse-length : 4096
content-maxparse-length : 4096
parse length-exceed action : drop
urlcookie-delimiters : /&#+
10. Clear the L7 load-balancing statistics by entering the following commands:
◊ clear stats loadbalance [radius | rdp]

◊ clear service-policy policy_name
◊ clear stats http
◊ clear rserver server_name
◊ clear serverfarm serverfarm_name
This article describes the procedures for troubleshooting redundancy issues with your ACE.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits

Contents
• 1 Overview of ACE Redundancy
♦ 1.1 Redundancy Protocol
♦ 1.2 FT VLAN
♦ 1.3 Configuration Requirements and
Restrictions
♦ 1.4 Example of a Redundancy Configuration
• 2 Troubleshooting ACE Redundancy
• 3 FT Peer and Group Status Details
♦ 3.1 FT Group Status Conditions
◊ 3.1.1 STANDBY_COLD
◊ 3.1.2 STANDBY_CONFIG
♦ 3.2 FT Peer Status Conditions
◊ 3.2.1 PEER_DOWN
◊ 3.2.2 TL_ERROR
◊ 3.2.3 FT_VLAN_DOWN
◊ 3.2.4 FSM_PEER_STATE_ERROR
♦ 3.3 About WARM_COMPATIBLE and
STANDBY_WARM
Overview of ACE Redundancy

Redundancy (or fault tolerance) allows your network to remain operational even if one of the ACEs becomes unresponsive.
Redundancy ensures that your network services and applications are always available.
Redundancy provides seamless switchover of flows if an ACE becomes unresponsive or a critical host, interface, or HSRP group
(ACE module only) fails. Redundancy supports the following network applications that require fault tolerance:
• Mission-critical enterprise applications

• Banking and financial services
• E-commerce
• Long-lived flows such as FTP and HTTP file transfers
Redundancy Protocol
You can configure a maximum of two ACE modules (peers) in the same Catalyst 6500 series switch or in different chassis for
redundancy. You can also configure a maximum of two ACE 4710 appliances for redundancy. Each peer ACE can contain one or
more fault-tolerant (FT) groups. Each FT group consists of two members: one active context and one standby context. For more
information about contexts, see the Cisco Application Control Engine Module Virtualization Configuration Guide or the Cisco
4700 Series Application Control Engine Appliance Administration Guide (Software Version A3(2.4)). An FT group has a unique
group ID that you assign.
Both ACEs can be active at the same time, processing traffic for distinct virtual devices and backing up each other (stateful
redundancy). An Active-Active configuration requires two FT groups and two virtual contexts on each ACE. See Figure 1.
Figure 1. Example of an Active-Active Configuration
Contents 122
The ACE uses the redundancy protocol to communicate between the redundant peers. The election of the active member within
each FT group is based on a priority scheme. The member configured with the higher priority is elected as the active member. If a
member with a higher priority is found after the other member becomes active, the new member becomes active because it has a
higher priority. This behavior is known as preemption and is enabled by default.
One virtual MAC address (VMAC) is associated with each FT group. The format of the VMAC is: 00-0b-fc-fe-1b-groupID.
Because a VMAC does not change upon a switchover, the client and server ARP tables does not require updating. The ACE
selects a VMAC from a pool of virtual MACs available to it. You can specify the pool of MAC addresses that the local ACE and
the peer ACE use by configuring the shared-vlan-hostid command and the peer shared-vlan-hostid command, respectively. To
avoid MAC address conflicts, be sure that the two pools are different on the two ACEs. For more information about VMACs and
MAC address pools, see the Cisco Application Control Engine Module Routing and Bridging Configuration Guide.
Each FT group acts as an independent redundancy instance. When a switchover occurs, the active member in the FT group
becomes the standby member and the original standby member becomes the active member. A switchover can occur for the
following reasons:
• The active member becomes unresponsive.

• A tracked host, interface, or HSRP group fails.
• You enter the ft switchover command to force a switchover.
FT VLAN
Redundancy uses a dedicated FT VLAN between redundant ACEs to transmit flow-state information and the redundancy
heartbeat. You must configure this same VLAN on both peer ACEs. You also must configure a different IP address within the
same subnet on each ACE for the FT VLAN. Cisco recommends two port-channeled 1-Gigabit Ethernet links for the FT VLAN.
For the appliance, when you configure the ft-port-vlan command, the ACE modifies the associated Ethernet port or port-channel
interface to a trunk port, so you have to put your switch port in trunk mode or tag the FT VLAN, it's very important.
Note: Do not use the FT VLAN for any other network traffic, including HSRP traffic and data.
The two redundant ACEs constantly communicate over the FT VLAN to determine the operating status of each ACE. The standby
member uses the heartbeat packet to monitor the health of the active member. The active member uses the heartbeat packet to
monitor the health of the standby member. Communications over the switchover link include the following data:
• Redundancy protocol packets

• State information replication data
• Configuration synchronization information
Redundancy Protocol 123

• Heartbeat packets
For multiple contexts, the FT VLAN resides in the system configuration file. Each FT VLAN on the ACE has one unique MAC
address associated with it. The ACE uses these device MAC addresses as the source or destination MACs for sending or receiving
redundancy protocol state and configuration replication packets.
Note: The IP address and the MAC address of the FT VLAN do not change at switchover.
Configuration Requirements and Restrictions
Follow these requirements and restrictions when configuring the redundancy feature:
• Redundancy is not supported between an ACE module and an ACE appliance operating as peers. Redundancy must be of
the same ACE device type and software release.
• In bridged mode (Layer 2), two contexts cannot share the same VLAN.
• To achieve active-active redundancy, a minimum of two contexts and two FT groups are required on each ACE.
• When you configure redundancy, the ACE keeps all interfaces that do not have an IP address in the Down state. The IP
address and the peer IP address that you assign to a VLAN interface should be in the same subnet but should be different
IP addresses. For more information about configuring VLAN interfaces, see the Cisco Application Control Engine
Module Routing and Bridging Configuration Guide or the Cisco ACE 4700 Series Appliance Routing and Bridging
Configuration Guide
• FT Interfaces are put into an automatic trunk status and, for the module, the Catalyst 6500 series switch needs to be set
to trunk the specific VLAN you are using for the FT interface.
Example of a Redundancy Configuration

The following example shows a running-configuration file that defines fault tolerance (FT) for a single ACE operating in a
redundancy configuration. You must configure a maximum of two ACEs (peers) for redundancy to fail over from the active ACE
to the standby ACE.
Note: All FT parameters are configured in the Admin context.

This configuration addresses the following redundancy components:
• A dedicated FT VLAN for communication between the members of an FT group. You must configure this same VLAN
on both peers.
• An FT peer definition.
• An FT group that is associated with the Admin context.
• A critical tracking and failure detection process for an interface.
class-map type management match-any L4_REMOTE-MGT_CLASS

policy-map type management first-match L4_REMOTE-MGT_POLICY

class L4_REMOTE-MGT_CLASS
permit
interface vlan 100

ip address 192.168.83.219 255.255.255.0
peer ip address 192.168.83.230 255.255.255.0
alias 192.168.83.200 255.255.255.0
FT VLAN 124
service-policy input L4_REMOTE-MGT_POLICY
no shutdown
ft interface vlan 200

ip address 192.168.1.1 255.255.255.0
peer ip address 192.168.1.2 255.255.255.0
no shutdown
ft peer 1
ft-interface vlan 200
heartbeat interval 300
heartbeat count 10
ft group 1
peer 1
priority 200
associate-context Admin
inservice
ft track interface TRACK_VLAN100

track-interface vlan 100
peer track-interface vlan 200
priority 50
peer priority 5
ip route 0.0.0.0 0.0.0.0 192.168.83.1
Troubleshooting ACE Redundancy

This section describes the methods and CLI commands that you can use to troubleshoot redundancy issues in your ACE.
1. Ensure that the software versions and licenses installed in the two ACEs are identical. A software or license mismatch may
generate the following syslog message:
%ACE-1-727006: HA: Peer is incompatible due to error str. Cannot be Redundant.
To verify the software (SRG) and license compatibility of the FT peer, enter the following command:
ACE_5/Admin# show ft peer status
Peer Id : 1
State : FSM_PEER_STATE_MY_IPADDR
Maintenance mode : MAINT_MODE_OFF
SRG Compatibility : COMPATIBLE
License Compatibility : COMPATIBLE
FT Groups : 1
If the software or license is incompatible, install the appropriate software image or license in the peer to correct the problem.
2. Ensure that any SSL certificates (certs) and keys that exist in the active ACE are also configured in the standby ACE. SSL certs
and keys are not synchronized automatically from the active to the standby. Use the crypto export and crypto import commands
to accomplish this task. This requirement also applies to scripts and scripted probes. Failure to keep the active and standby
configurations identical will cause configuration synchronization to fail and may cause the standby ACE to enter the
STANDBY_COLD state.
3. The ACE sends heartbeat packets via UDP over the FT VLAN between peers. When heartbeats are not received during the
specified interval (the interval and count are configurable), the ACE notifies the HA processor on the CP by sending a Peer_Down
interprocess communication protocol (IPCP) message. If a peer is down or unreachable, you may receive one of the following
Example of a Redundancy Configuration 125

syslog messages:
%ACE-1-727001: HA: Peer IP address is not reachable. Error: error str
%ACE-1-727002: HA: FT interface interface name to reach peer IP address is down. Error: error str
4. Verify connectivity between the peers over the FT VLAN. If a peer device is physically up but connectivity is the problem, you
may end up with two active devices. If connectivity is lost due to the peer going down, reboot the peer to restore redundancy
between the two devices.
5. Display heartbeat statistics, including missed heartbeats, by entering the following command:
ACE_5/Admin# show ft stats

HA Heartbeat Statistics
------------------------
Number of Heartbeats Sent : 0

Number of Heartbeats Received : 0
Number of Heartbeats Missed : 0
Number of Unidirectional HB's Received : 0
Number of HB Timeout Mismatches : 0
Num of Peer Up Events Sent : 0
Num of Peer Down Events Sent : 0
Successive HB's miss Intervals counter : 0
Successive Uni HB's recv counter : 0
6. Provide an alternate path for the ACE to check the peer's status in case of missed heartbeats and configure a query interface
using the followng commands:
ACE_5/Admin# config
ACE_5/Admin(config)# ft peer 1
ACE_5/Admin(config-ft-peer)# query-interface vlan 100
If the query interface is configured, upon receiving a PEER_DOWN message from the heartbeat process, the ACE data plane
attempts to ping the peer using the Query VLAN. If the ping fails, the standby transitions to the ACTIVE state. If the ping is
successful, the standby transitions to the STANDBY_COLD state. To recover from the STANDBY_COLD state, reboot the
standby.
7. Each peer uses a VMAC that is dependent on the FT group number. If you are using multiple ACE modules in the same chassis,
be careful when you configure the same FT groups in more than one module.
Display the VMAC for an FT group by entering the following command:
ACE_5/Admin# show interface internal iftable vlan100

vlan100
--------
ifid: 6
Context: 0
ifIndex: 16777316
physid: 100
rmode: 0 (unknown)
iftype: 0 (vlan)
bvi_bgid: 0
MTU: 1500
MAC: 00:18:b9:a6:91:15
VMAC: 00:00:00:00:00:00 <------- Virtual MAC Address
Flags: 0x8a000800 (valid, down, admin-down, Non-redundant, tracked)
ACL In: 0
ACL Out: 0
Troubleshooting ACE Redundancy 126

Route ID: 0
FTgroupID: 0
Zone ID: 6
Sec Level: 0
L2 ACL: bpdu DENY, ipv6 DENY, mpls DENY, all DENY
LastChange: 0 (Thu Jan 1 00:00:00 1970)

iflookup index: 100
vlan-vmac index:0
Next Shared IF: 0
Lock: Unlocked, seq 5
Lock errors: 0
Unlock errors: 0
No. of times locked: 5
No. of times unlocked: 5
Current/last owner: 0x40a7fc
8. If the members of an FT group are unable to reach the ACTIVE or the STANDBY_HOT state, there may be a context name
mismatch for the same FT group. You may receive the following syslog message:
%ACE-1-727003: HA: Mismatch in context names detected for FT group FTgroupID. Cannot be redundant.
Be sure that the context names within the same FT group are identical on both ACEs.
9. Check the FT group configuration on both devices. Make sure that both devices are associated with the same context. Enter the
following command:
ACE_5/Admin# show running-config ft
10. Verify the FT peer status and configuration by entering the following command:
ACE_5/Admin# show ft peer detail
Peer Id : 1
State : FSM_PEER_STATE_COMPATIBLE
FT Vlan : 100
FT Vlan IF State : DOWN
My IP Addr : 10.1.1.1
Peer IP Addr : 10.1.1.2
Query Vlan : 110
Query Vlan IF State : DOWN
Peer Query IP Addr : 172.25.91.202
Heartbeat Interval : 300
Heartbeat Count : 20
Tx Packets : 318573
Tx Bytes : 66301061
Rx Packets : 318540
Rx Bytes : 66272840
Rx Error Bytes : 0
Tx Keepalive Packets : 318480
Rx Keepalive Packets : 318480
TL_CLOSE count : 0
FT_VLAN_DOWN count : 0
PEER_DOWN count : 0
SRG Compatibility : COMPATIBLE
License Compatibility : COMPATIBLE
FT Groups : 3
11. Verify the FT group status and configuration by entering the following command:
ACE_5/Admin# show ft group detail
Troubleshooting ACE Redundancy 127

FT Group : 1
No. of Contexts : 1
Configured Status : in-service
My State : FSM_FT_STATE_ACTIVE
My Config Priority : 110
My Net Priority : 110
My Preempt : Enabled
Peer State : FSM_FT_STATE_STANDBY
Peer Config Priority : 100
Peer Net Priority : 100
Peer Preempt : Enabled
Peer Id : 1
Last State Change time : Thu Apr 2 00:00:00 2009
Running cfg sync enabled : Enabled
Running cfg sync status : Running configuration sync has completed
Startup cfg sync enabled : Enabled
Startup cfg sync status : Running configuration sync has completed
Bulk sync done for ARP: 0
Bulk sync done for LB: 0
Bulk sync done for ICM: 0
For information on troubleshooting the FT group status, see the "FT Group Status Conditions"
FT Peer and Group Status Details

This section describes how to diagnose unexpected status conditions for the FT group and FT peer. This information may enable
you to troubleshoot an issue directly or help you to provide additional information to your Cisco support representative.
FT Group Status Conditions

This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT group status.
STANDBY_COLD
An FT group status of STANDBY_COLD may appear when:
• Config sync fails (including, incr-sync and bulk-sync), or

• FT VLAN is down while the query interface is up
Config Sync Failure
In configuration synchronization fails, the peers are not correctly exchanging configuration information. This failure can be
identified as follows:
1. Output of the show ft peer detail command shows that the peer state is "Compatible".
2. Entering show ft group detail shows that the FT group is in "Standby Cold" mode and entering cfg sync status shows the
reason for the failure. For incr-sync failure, the output shows exactly which command resulted in an execution error on the
standby. For a bulk-sync failure, the reason is "Error on Standby device when applying configuration file replicated from
active".
3. To further investigate a bulk-sync failure, perform these steps on the standby device:
◊ For software version A2(2.0) and earlier and version A2(1.3) and earlier releases, from the Admin context, enter
show ft history cfg_cntlr and grep for "error:" to find any CLI commands that caused execution errors.
◊ For later releases, enter show ft config-error ctx_name to view the failed CLI commands.
FT Peer and Group Status Details 128

To work around a bulk sync failure, perform these steps to remove the CLI commands that triggered the error (as identified from
the preceding analysis) and then retrigger the bulk sync operation, as follows:
1. Retrigger bulk sync by disabling config sync with the no ft auto-sync running command in configuration mode in the
affected context.
2. Re-enable config sync with ft auto-sync running.
If the problem persists, repeat the above sequence until you eliminate the CLI command that triggered the problem.
FT VLAN Down with Query Interface Up
This condition can be identified by:
1. Entering show ft peer detail, which shows a peer state of FT_VLAN_DOWN.

2. Entering show ft stats, which shows that heartbeats are being missed.
In this case, check the physical connectivity of the device. It might be a physical port or cable issue.
STANDBY_CONFIG
If a device appears to be stuck in the STANDBY_CONFIG state:
1. Run show ft history cfg_cntlr to determine whether the peer devices successfully exchanged notifications regarding
configuration synchronization.
2. Grep for the keywords MTS_OPC_REQ_CFG_DNLD_STATUS and MTS_OPC_CFG_DNLD_STATUS.
If one or both of the messages are missing, an error occurred in the synchronization exchange process.
Note that once it is stuck in the STANDBY_CONFIG state, configuration mode will be disabled on both the active and standby
devices. It can be stuck in this state for up to 4 hours, after which a timeout period expires.
FT Peer Status Conditions

This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT peer.
PEER_DOWN
If the peer status shows PEER_DOWN:
1. Check whether IP addresses for the local and peer device are configured correctly on both.
2. Verify that pinging or telneting to the peer IP address works. If ping fails, check whether the interface is up (show
interface). If so, the interface VLAN is probably not allocated to the ACE module on the supervisor (which suggests a
configuration issue on the supervisor).
3. Enter show arp to see if the FT peer IP address is resolved. (If ARP is not resolved and ping/telnet also failed, it might be
an encapsulation issue requiring support).
4. Enter show conn on both sides to see if HA connections have been set up. If connections have not been set up, check the
HA DP manager log (show ft history ha_dp_mgr). Setup may have failed for various reasons. If this is the case, contact
Cisco technical support.
5. Enter show ft stats on both devices to see if heartbeats are being sent or received. If the Number of Heartbeats Missed
counter is incrementing, the heartbeat packets could be getting dropped. Enter show np 1 me-stats -sfp to see if heartbeat
packets are being received and forwarded to X-Scale, as indicated by the counter Packets forward to XScale. If this
counter is not incrementing, provide the information to Cisco technical support.
STANDBY_COLD 129
TL_ERROR
This state may occur when the telnet connection used to exchange configuration information between the peers cannot be
established but heartbeat packets are exchanged successfully. To identify this issue:
1. Verify that heartbeats are flowing by checking the statistics, show ft stats.
2. Attempt to connect by telnet or to ping the FT peer. The telnet connection attempt will likely fail.
3. Run show arp to see if the FT peer IP address can be resolved.
If show arp indicates that the address is not resolvable and the ping or telnet connect attempts fail, it is likely an encapsulation
issue on the ACE.
FT_VLAN_DOWN
This state typically occurs when the FT VLAN goes down while the query interface is up. If the heartbeat exchange fails and the
query interface is determined to be up based on an ICMP message check, the status is FT_VLAN_DOWN.
To verify, attempt to connect to the FT VLAN Peer IP address by ping or telnet.
If running show ft stats indicates that heartbeats are being missed, it is likely a physical connectivity issue, such as the physical
port or cable failure.
FSM_PEER_STATE_ERROR
This indicates a Software Relationship Graph (SRG) version inconsistency between the peers. See the relationship graph table in
the following section.
About WARM_COMPATIBLE and STANDBY_WARM

While peers in a redundant configuration are designed to operate with identical versions of the software, when you are upgrading
or downgrading the software in the ACEs, it is possible for the peers to temporarily employ different software versions. The
WARM_COMPATIBLE and STANDBY_WARM redundancy states help minimize the operational impact of CLI compatibility
issues between the peers, and allow failovers to occur on a best-effort basis during such transitions.
When you upgrade or downgrade the ACE software in a redundant configuration with different software versions, the
STANDBY_WARM and WARM_COMPATIBLE states allow the configuration and state synchronization process between the
peers to continue on a best-effort basis. This basis allows the active ACE to synchronize configuration and state information with
the standby even though the standby may not recognize or understand the CLI commands or state information.
In the STANDBY_WARM state, as with the STANDBY_HOT state, configuration mode is disabled on the standby ACE and
configuration and state synchronization continues. A failover from the active to the standby based on priorities and preempt can
still occur while the standby is in the STANDBY_WARM state. However, while stateful failover is possible for a WARM
standby, it is not guaranteed. In general, modules should be allowed to remain in this state only for a short period of time.
When redundancy peers run different software versions, the SRG compatibility field shown by the show ft peer status command
output displays WARM_COMPATIBLE instead of COMPATIBLE. When the peer is in the WARM_COMPATIBLE state, the
FT groups in the standby transition to the STANDBY_WARM state instead of the STANDBY_HOT state.
The following software version combinations tables indicate whether the SRG compatibility field will display
WARM_COMPATIBLE (WC) or COMPATIBLE (C):
ACE Module: C = COMPATIBLE / WC = WARM_COMPATIBLE

Active(Column)/Standby(Row) A2(1.5) A2(1.6) A2(2.0) A2(2.1) A2(2.2) A2(3.0) A2(2.3)
TL_ERROR 130
A2(1.5) C WC C C WC WC WC
A2(1.6) WC C C WC WC WC WC
A2(2.0) C C C C C C C
A2(2.1) C WC C C WC WC WC
A2(2.2) WC WC C WC C WC WC
A2(3.0) WC WC C WC WC C WC
A2(2.3) WC WC C WC WC WC C
ACE Appliance: C = COMPATIBLE / WC = WARM_COMPATIBLE
Active(Column)/Standby(Row) A3(1.0) A3(2.0) A3(2.1) A3(2.2) A3(2.3) A3(2.4)
A3(1.0) C C C C WC WC
A3(2.3) WC WC WC WC C WC
A3(2.4) WC WC WC WC WC C
This article describes the process and CLI commands for troubleshooting SSL in the ACE.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE SSL Troubleshooting
♦ 1.1 Example of an SSL
Termination Configuration
♦ 1.2 Example of an SSL Initiation
Configuration
• 2 Troubleshooting ACE SSL
About WARM_COMPATIBLE and STANDBY_WARM 131

Overview of ACE SSL Troubleshooting

Secure Sockets Layer (SSL) runs over TCP. After the TCP three-way handshake completes and the ACE has proxied the
connection, the SSL handshake takes place. For information about proxied connections, see the Troubleshooting Connectivity
article. See Figure 1 for an illustration of the SSL handshake.
Figure 1. SSL Handshake
Overview of ACE SSL Troubleshooting 132

The ACE supports the following SSL configurations (see Figure 2):
• SSL termination (ACE acts as an SSL server)

• SSL initiation (ACE acts as a client)
• End-to-end SSL (SSL termination plus SSL initiation)
Figure 2. SSL Configurations

Before you begin to troubleshoot potential SSL issues, be sure that the following conditions exist:
• You have configured basic SLB and SSL on your ACE. For details about configuring SLB, see the Cisco Application
Control Engine Module Server Load-Balancing Configuration Guide or the Cisco ACE 4700 Series Appliance Server
Load-Balancing Configuration Guide. For details about configuring SSL, see the Cisco Application Control Engine
Module SSL Configuration Guide or the Cisco ACE 4700 Series Appliance SSL Configuration Guide.
• If you are running multiple ACEs in a redundant configuration, be sure that you have copied the SSL certificates (certs)
and keys to the standby ACE. Certs and keys are not replicated in a redundant configuration from the active ACE to the
standby ACE. Also, ensure that the configurations on the active and the standby are identical, including the same licenses
and software versions.
• Be sure that the certs and keys are no larger than 4096 bits and that they are of an RSA type supported by the ACE. For
details about configuring SSL, see the Cisco Application Control Engine Module SSL Configuration Guide or the Cisco
ACE 4700 Series Appliance SSL Configuration Guide. The ACE supports the following RSA key pair sizes:
◊ 512 (least security)
◊ 768 (normal security)
◊ 1024 (high security, level 1)
◊ 4096 (high security, level 4) - For software release A2(2.4) and later in the ACE module and software release
A3(2.6) and later in the ACE appliance, you can use 4096-bit SSL certificates in chaingroups and authgroups.
You can also import public certificates and keys that are 4096 bits in length.
• Server certs are valid, installed, and have not expired

Example of an SSL Termination Configuration

The following example shows a running-configuration file of the ACE acting as an SSL proxy server; terminating SSL or TLS
connections from a client and then establishing a TCP connection to an HTTP server. When the ACE terminates the SSL or TLS
connection, it decrypts the cipher text from the client and transmits the data as clear text to the HTTP server.
probe http GEN-HTTP

port 80
interval 50
faildetect 5
rserver SERVER1
inservice
rserver SERVER2
inservice
rserver SERVER3
inservice
rserver SERVER4
inservice
rserver SERVER5
inservice
rserver SERVER6
inservice
rserver SERVER7
inservice
rserver SERVER8
inservice

description SERVER FARM 1 FOR SSL TERMINATION
probe GEN_HTTP
rserver SERVER1 80
inservice
rserver SERVER2 80
inservice
rserver SERVER3 80
inservice
rserver SERVER4 80
inservice

probe GEN_HTTP
rserver SERVER5 80
inservice
rserver SERVER6 80
inservice
rserver SERVER7 80
inservice
rserver SERVER8 80
inservice
parameter-map type ssl PARAMMAP_SSL_TERMINATION

cipher RSA_WITH_3DES_EDE_CBC_SHA
cipher RSA_WITH_AES_128_CBC_SHA priority 2
cipher RSA_WITH_AES_256_CBC_SHA priority 3
version all
parameter-map type connection TCP_PARAM
syn-data drop
exceed-mss allow
ssl-proxy service SSL_PSERVICE_SERVER

ssl advanced-options PARAMMAP_SSL_TERMINATION
key MYKEY.PEM
cert MYCERT.PEM
class-map type http loadbalance match-all L7_SERVER_CLASS

description Sticky for SSL Testing
2 match http url .*.jpg
3 match source-address 192.168.130.0 255.255.255.0
class-map type http loadbalance match-all L7_SLB-HTTP_CLASS
2 match http url .*
class-map match-all L4_SSL-TERM_CLASS
description SSL Termination VIP
2 match virtual-address 192.168.130.11 tcp eq https
policy-map type loadbalance first-match L7_SSL-TERM_POLICY

class L7_SERVER_CLASS
serverfarm SFARM1
insert-http I_AM header-value "SSL_TERM"
insert-http SRC_Port header-value "%ps"
insert-http DEST_IP header-value "%id"
insert-http DEST_Port header-value "%pd"
insert-http SRC_IP header-value "is"
class L7_SLB-HTTP_CLASS
serverfarm SFARM1
insert-http I_AM header-value "SSL_TERM"
insert-http SRC_IP header-value "is"
policy-map multi-match L4_SSL-VIP_POLICY

class L4_SSL-TERM_CLASS
loadbalance policy L7_SSL-TERM_POLICY
loadbalance vip icmp-reply
ssl-proxy server SSL_PSERVICE_SERVER
connection advanced-options TCP_PARAM
interface vlan 120

ip address 192.168.120.1 255.255.255.0
fragment chain 20
fragment min-mtu 68
service-policy input L4_SSL-VIP_POLICY
no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Example of an SSL Initiation Configuration

The following example shows a running-configuration file of the ACE acting as an SSL proxy client, initiating and maintaining an
SSL connection between itself and an SSL server. The ACE receives clear text from an HTTP client, and then encrypts and
transmits the data as cipher text to the SSL server. On the reverse side, the ACE decrypts the cipher text that it receives from the
SSL server and sends the data to the client as clear text.
probe http GEN-HTTP

port 80
interval 50
faildetect 5
rserver SERVER1
inservice
rserver SERVER2
inservice
rserver SERVER3
inservice
rserver SERVER4
inservice
rserver SERVER5
inservice
rserver SERVER6
inservice
rserver SERVER7
inservice
rserver SERVER8
inservice

description SERVER FARM 1 FOR SSL INITIATION
probe GEN_HTTP
rserver SERVER1 443
inservice
rserver SERVER2 443
inservice
rserver SERVER3 443
inservice
rserver SERVER4 443
inservice

probe GEN_HTTP
rserver SERVER5 443
inservice
rserver SERVER6 443
inservice
rserver SERVER7 443
inservice
rserver SERVER8 443
inservice
parameter-map type http PARAMMAP_HTTP

server-conn reuse
case-insensitive
persistence-rebalance
parameter-map type ssl PARAMMAP_SSL_INITIATION
Example of an SSL Initiation Configuration 137

cipher RSA_WITH_RC4_128_MD5
cipher RSA_WITH_RC4_128_SHA
cipher RSA_WITH_DES_CBC_SHA
cipher RSA_WITH_3DES_EDE_CBC_SHA
cipher RSA_WITH_AES_128_CBC_SHA
cipher RSA_WITH_AES_256_CBC_SHA
cipher RSA_EXPORT_WITH_RC4_40_MD5
cipher RSA_EXPORT1024_WITH_RC4_56_MD5
cipher RSA_EXPORT_WITH_DES40_CBC_SHA
cipher RSA_EXPORT1024_WITH_DES_CBC_SHA
cipher RSA_EXPORT1024_WITH_RC4_56_SHA
version all
parameter-map type connection TCP_PARAM
syn-data drop
exceed-mss allow
ssl-proxy service SSL_PSRVICE_CLIENT

ssl advanced-options PARAMMAP_SSL_INITIATION
class-map type http loadbalance match-all L7_SERVER_CLASS

description Sticky for SSL Testing
2 match http url .*.jpg
class-map type http loadbalance match-all L7_SLB-HTTP_CLASS
2 match http url .*
class-map match-all L4_SSL-INIT_CLASS
description SSL Initiation VIP
policy-map type loadbalance first-match L7_SSL-INIT_POLICY

class L7_SERVER_CLASS
serverfarm SFARM1
insert-http SRC_IP header-value "%is"
insert-http I_AM header-value "SSL_INIT"
ssl-proxy client SSL_PSERVICE_CLIENT
class L7_SLB-HTTP_CLASS
serverfarm SFARM2
insert-http SRC_IP header-value "%is"
insert-http I_AM header-value "SSL_INIT"
ssl-proxy client SSL_PSERVICE_CLIENT
policy-map multi-match L4_SSL-VIP_POLICY
class L4_SSL-INIT_CLASS
loadbalance policy L7_SSL-INIT_POLICY
appl-parameter http advanced-options PARAMMAP_HTTP
connection advanced-options TCP_PARAM
interface vlan 120

ip address 192.168.120.1 255.255.255.0
fragment chain 20
fragment min-mtu 68
service-policy input L4_SSL-VIP_POLICY
no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Example of an SSL Initiation Configuration 138

Troubleshooting ACE SSL

To troubleshoot SSL issues, follow these steps:
1. For the ACE module, check the health of the Nitrox-II (crypto module) and ensure that it has not become unresponsive. Stop all
traffic, and then enter the following command:
ACE_module5/Admin# show crypto hardware
Figure 3. Example of the Show Crypto Hardware Command Output for an Unresponsive Crypto Module
Troubleshooting ACE SSL 139

STX1 is a count of the number of packets transmitted by the Nitrox-II and IMX1 is the number of packets received by the
Nitrox-II. On a normal system, these values should be the same once traffic has stopped. If the values are not the same, the
Nitrox-II has become unresponsive.
The Nitrox-II uses 0x500 TX buffers to transmit packets and 0x200 RX buffers to receive packets. If the [TR]X Buffers used
count ever exceeds the amount available, the Nitrox-II has become unresponsive.
The available cores field shows which of the 22 cores of the Nitrox-II are active. When no traffic is flowing, there should be no
numbers following the Using: statement. If there are, as in the sample output above, then that core (0 in this case) is hung, and the
Nitrox-II has become unresponsive.
For the POM count, there are two numbers, A(B). The "A" value is the number of outstanding packets to the Packet Order
Manager, while the "B" value, counts the number of packets that have been processed in the last second. When no traffic is
flowing, both of these values should be 0. If no traffic is flowing, and the value of "A" is nonzero as shown above, then there are
outstanding requests to the POM that are not being processed, because the Nitrox-II has become unresponsive.
2. Ensure that appropriate ports are designated for PAT in an SSL termination configuration. By default, connections to the real
server from the ACE will inherit the destination port from the client to VIP connection so that a connection to port 443 on the VIP
will go to port 443 on the real server, unless otherwise specified in the server farm configuration. This will cause problems if you
are using ACE to offload SSL between the client and the VIP and send clear-text traffic to the real servers. The following example
demonstrates a port definition in a server farm configuration:
serverfarm host sf1

probe HTTP_PROBE
rserver rs1 80
inservice
rserver rs2 80
inservice
3. Verify that the SSL certificate and key are correct by entering the following command:
ACE_module5/Admin# crypto verify key cert
4. Verify that a certificate revocation list (CRL) has been downloaded, enter the following command:
ACE_module5/Admin# show crypto crl test1
test1:
URL: http://192.168.12.23/test.crl
Last Downloaded: not downloaded yet
Total Number Of Download Attempts: 0
Failed Download Attempts: 0
5. Verify the contents of an authgroup by entering the following command:
ACE_module5/Admin# show crypto authgroup authgroup_name
6. Display client SSL statistics by entering the the following command:
ACE_module5/Admin# show stats crypto client
+----------------------------------------------+
+---- Crypto client termination statistics ----+
+----------------------------------------------+
SSLv3 negotiated protocol: 0
TLSv1 negotiated protocol: 0
SSLv3 full handshakes: 0
SSLv3 resumed handshakes: 0
SSLv3 rehandshakes: 0
TLSv1 full handshakes: 0
TLSv1 resumed handshakes: 0
TLSv1 rehandshakes: 0
SSLv3 handshake failures: 0
SSLv3 failures during data phase: 0
TLSv1 handshake failures: 0
TLSv1 failures during data phase: 0
Handshake Timeouts: 0
total transactions: 0
SSLv3 active connections: 0
SSLv3 connections in handshake phase: 0
SSLv3 conns in renegotiation phase: 0
SSLv3 connections in data phase: 0
TLSv1 active connections: 0
TLSv1 connections in handshake phase: 0
TLSv1 conns in renegotiation phase: 0
TLSv1 connections in data phase: 0
+----------------------------------------------+
+------- Crypto client alert statistics -------+
+----------------------------------------------+
SSL alert CLOSE_NOTIFY rcvd: 0
SSL alert UNEXPECTED_MSG rcvd: 0
SSL alert BAD_RECORD_MAC rcvd: 0
SSL alert DECRYPTION_FAILED rcvd: 0
SSL alert RECORD_OVERFLOW rcvd: 0
SSL alert DECOMPRESSION_FAILED rcvd: 0
SSL alert HANDSHAKE_FAILED rcvd: 0
SSL alert NO_CERTIFICATE rcvd: 0
SSL alert BAD_CERTIFICATE rcvd: 0
SSL alert UNSUPPORTED_CERTIFICATE rcvd: 0
SSL alert CERTIFICATE_REVOKED rcvd: 0
SSL alert CERTIFICATE_EXPIRED rcvd: 0
SSL alert CERTIFICATE_UNKNOWN rcvd: 0
SSL alert ILLEGAL_PARAMETER rcvd: 0
SSL alert UNKNOWN_CA rcvd: 0
SSL alert ACCESS_DENIED rcvd: 0
SSL alert DECODE_ERROR rcvd: 0
SSL alert DECRYPT_ERROR rcvd: 0
SSL alert EXPORT_RESTRICTION rcvd: 0
SSL alert PROTOCOL_VERSION rcvd: 0
SSL alert INSUFFICIENT_SECURITY rcvd: 0
SSL alert INTERNAL_ERROR rcvd: 0
SSL alert USER_CANCELED rcvd: 0
SSL alert NO_RENEGOTIATION rcvd: 0
SSL alert CLOSE_NOTIFY sent: 0
SSL alert UNEXPECTED_MSG sent: 0
SSL alert BAD_RECORD_MAC sent: 0
SSL alert DECRYPTION_FAILED sent: 0
SSL alert RECORD_OVERFLOW sent: 0
SSL alert DECOMPRESSION_FAILED sent: 0
SSL alert HANDSHAKE_FAILED sent: 0
SSL alert NO_CERTIFICATE sent: 0
SSL alert BAD_CERTIFICATE sent: 0
SSL alert UNSUPPORTED_CERTIFICATE sent: 0
SSL alert CERTIFICATE_REVOKED sent: 0
SSL alert CERTIFICATE_EXPIRED sent: 0
SSL alert CERTIFICATE_UNKNOWN sent: 0
SSL alert ILLEGAL_PARAMETER sent: 0
SSL alert UNKNOWN_CA sent: 0
SSL alert ACCESS_DENIED sent: 0
SSL alert DECODE_ERROR sent: 0
SSL alert DECRYPT_ERROR sent: 0
SSL alert EXPORT_RESTRICTION sent: 0
SSL alert PROTOCOL_VERSION sent: 0
SSL alert INSUFFICIENT_SECURITY sent: 0
SSL alert INTERNAL_ERROR sent: 0
SSL alert USER_CANCELED sent: 0
SSL alert NO_RENEGOTIATION sent: 0
+-----------------------------------------------+
+--- Crypto client authentication statistics ---+
+-----------------------------------------------+
Total SSL client authentications: 0
Failed SSL client authentications: 0
SSL client authentication cache hits: 0

SSL static CRL lookups: 0
SSL best effort CRL lookups: 0
SSL CRL lookup cache hits: 0
SSL revoked certificates: 0
Total SSL server authentications: 0
Failed SSL server authentications: 0
+-----------------------------------------------+
+------- Crypto client cipher statistics -------+
+-----------------------------------------------+
Cipher sslv3_rsa_rc4_128_md5: 0
Cipher sslv3_rsa_rc4_128_sha: 0
Cipher sslv3_rsa_des_cbc_sha: 0
Cipher sslv3_rsa_3des_ede_cbc_sha: 0
Cipher sslv3_rsa_exp_rc4_40_md5: 0
Cipher sslv3_rsa_exp_des40_cbc_sha: 0
Cipher sslv3_rsa_exp1024_rc4_56_md5: 0
Cipher sslv3_rsa_exp1024_des_cbc_sha: 0
Cipher sslv3_rsa_exp1024_rc4_56_sha: 0
Cipher sslv3_rsa_aes_128_cbc_sha: 0
Cipher tlsv1_rsa_rc4_128_md5: 0
Cipher tlsv1_rsa_rc4_128_sha: 0
Cipher tlsv1_rsa_des_cbc_sha: 0
Cipher tlsv1_rsa_3des_ede_cbc_sha: 0
Cipher tlsv1_rsa_exp_rc4_40_md5: 0
Cipher tlsv1_rsa_exp_des40_cbc_sha: 0
Cipher tlsv1_rsa_exp1024_rc4_56_md5: 0
Cipher tlsv1_rsa_exp1024_des_cbc_sha: 0
Cipher tlsv1_rsa_exp1024_rc4_56_sha: 0
Cipher tlsv1_rsa_aes_128_cbc_sha: 0
7. Display SSL server statistics by entering the following command:
ACE_module5/Admin# show stats crypto server
+----------------------------------------------+
+---- Crypto server termination statistics ----+
+----------------------------------------------+
SSLv3 negotiated protocol: 0
TLSv1 negotiated protocol: 0
SSLv3 full handshakes: 0
SSLv3 resumed handshakes: 0
SSLv3 rehandshakes: 0
TLSv1 full handshakes: 0
TLSv1 resumed handshakes: 0
TLSv1 rehandshakes: 0
SSLv3 handshake failures: 0
SSLv3 failures during data phase: 0
TLSv1 handshake failures: 0
TLSv1 failures during data phase: 0
Handshake Timeouts: 0
total transactions: 0
SSLv3 active connections: 0
SSLv3 connections in handshake phase: 0
SSLv3 conns in renegotiation phase: 0
SSLv3 connections in data phase: 0
TLSv1 active connections: 0
TLSv1 connections in handshake phase: 0
TLSv1 conns in renegotiation phase: 0
TLSv1 connections in data phase: 0

+----------------------------------------------+
+------- Crypto server alert statistics -------+
+----------------------------------------------+
SSL alert CLOSE_NOTIFY rcvd: 0
SSL alert UNEXPECTED_MSG rcvd: 0
SSL alert BAD_RECORD_MAC rcvd: 0
SSL alert DECRYPTION_FAILED rcvd: 0
SSL alert RECORD_OVERFLOW rcvd: 0
SSL alert DECOMPRESSION_FAILED rcvd: 0
SSL alert HANDSHAKE_FAILED rcvd: 0
SSL alert NO_CERTIFICATE rcvd: 0
SSL alert BAD_CERTIFICATE rcvd: 0
SSL alert UNSUPPORTED_CERTIFICATE rcvd: 0
SSL alert CERTIFICATE_REVOKED rcvd: 0
SSL alert CERTIFICATE_EXPIRED rcvd: 0
SSL alert CERTIFICATE_UNKNOWN rcvd: 0
SSL alert ILLEGAL_PARAMETER rcvd: 0
SSL alert UNKNOWN_CA rcvd: 0
SSL alert ACCESS_DENIED rcvd: 0
SSL alert DECODE_ERROR rcvd: 0
SSL alert DECRYPT_ERROR rcvd: 0
SSL alert EXPORT_RESTRICTION rcvd: 0
SSL alert PROTOCOL_VERSION rcvd: 0
SSL alert INSUFFICIENT_SECURITY rcvd: 0
SSL alert INTERNAL_ERROR rcvd: 0
SSL alert USER_CANCELED rcvd: 0
SSL alert NO_RENEGOTIATION rcvd: 0
SSL alert CLOSE_NOTIFY sent: 0
SSL alert UNEXPECTED_MSG sent: 0
SSL alert BAD_RECORD_MAC sent: 0
SSL alert DECRYPTION_FAILED sent: 0
SSL alert RECORD_OVERFLOW sent: 0
SSL alert DECOMPRESSION_FAILED sent: 0
SSL alert HANDSHAKE_FAILED sent: 0
SSL alert NO_CERTIFICATE sent: 0
SSL alert BAD_CERTIFICATE sent: 0
SSL alert UNSUPPORTED_CERTIFICATE sent: 0
SSL alert CERTIFICATE_REVOKED sent: 0
SSL alert CERTIFICATE_EXPIRED sent: 0
SSL alert CERTIFICATE_UNKNOWN sent: 0
SSL alert ILLEGAL_PARAMETER sent: 0
SSL alert UNKNOWN_CA sent: 0
SSL alert ACCESS_DENIED sent: 0
SSL alert DECODE_ERROR sent: 0
SSL alert DECRYPT_ERROR sent: 0
SSL alert EXPORT_RESTRICTION sent: 0
SSL alert PROTOCOL_VERSION sent: 0
SSL alert INSUFFICIENT_SECURITY sent: 0
SSL alert INTERNAL_ERROR sent: 0
SSL alert USER_CANCELED sent: 0
SSL alert NO_RENEGOTIATION sent: 0
+-----------------------------------------------+
+--- Crypto server authentication statistics ---+
+-----------------------------------------------+
Total SSL client authentications: 0
Failed SSL client authentications: 0
SSL client authentication cache hits: 0
SSL static CRL lookups: 0
SSL best effort CRL lookups: 0
SSL CRL lookup cache hits: 0
SSL revoked certificates: 0
Total SSL server authentications: 0
Failed SSL server authentications: 0

+-----------------------------------------------+
+------- Crypto server cipher statistics -------+
+-----------------------------------------------+
Cipher sslv3_rsa_rc4_128_md5: 0
Cipher sslv3_rsa_rc4_128_sha: 0
Cipher sslv3_rsa_des_cbc_sha: 0
Cipher sslv3_rsa_3des_ede_cbc_sha: 0
Cipher sslv3_rsa_exp_rc4_40_md5: 0
Cipher sslv3_rsa_exp_des40_cbc_sha: 0
Cipher sslv3_rsa_exp1024_rc4_56_md5: 0
Cipher sslv3_rsa_exp1024_des_cbc_sha: 0
Cipher sslv3_rsa_exp1024_rc4_56_sha: 0
Cipher tlsv1_rsa_rc4_128_md5: 0
Cipher tlsv1_rsa_rc4_128_sha: 0
Cipher tlsv1_rsa_des_cbc_sha: 0
Cipher tlsv1_rsa_3des_ede_cbc_sha: 0
Cipher tlsv1_rsa_exp_rc4_40_md5: 0
Cipher tlsv1_rsa_exp_des40_cbc_sha: 0
Cipher tlsv1_rsa_exp1024_rc4_56_md5: 0
Cipher tlsv1_rsa_exp1024_des_cbc_sha: 0
Cipher tlsv1_rsa_exp1024_rc4_56_sha: 0
8. Display the number of SSL data messages sent and SSL FIN/RST messages sent by entering the following command:
ACE_module5/Admin# show stats http
+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 0 , TCP data msgs sent : 0
Inspect parse result msgs : 0 , SSL data msgs sent : 0 <-------
sent
TCP fin/rst msgs sent : 0 , Bounced fin/rst msgs sent: 0
SSL fin/rst msgs sent : 0 , Unproxy msgs sent : 0 <-------
Drain msgs sent : 0 , Particles read : 0
Reuse msgs sent : 0 , HTTP requests : 0
Reproxied requests : 0 , Headers removed : 0
Headers inserted : 0 , HTTP redirects : 0
HTTP chunks : 0 , Pipelined requests : 0
HTTP unproxy conns : 0 , Pipeline flushes : 0
Whitespace appends : 0 , Second pass parsing : 0
Response entries recycled : 0 , Analysis errors : 0
Header insert errors : 0 , Max parselen errors : 0
Static parse errors : 0 , Resource errors : 0
Invalid path errors : 0 , Bad HTTP version errors : 0
Headers rewritten : 0 , Header rewrite errors : 0
9. Display session cache statistics for the current context by entering the following command:
switch/Admin# show crypto session

SSL Session Cache Stats for Context
------------------
Number of Client Sessions: 0
Number of Server Sessions: 0
This article describes how to troubleshoot performance issues with your ACE.

Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of Troubleshooting
Performance Issues
• 2 Troubleshooting Performance Issues
Contents 146
Overview of Troubleshooting Performance Issues

Before you begin to troubleshoot ACE performance issues, check and record the following items:
1. Be sure that the correct licenses are installed in your ACE.
2. Record the number of flows that you are sending to the ACE.
3. Record the performance of a single flow.
4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on)
5. Identify the ACE context that is receiving the traffic.
6. Enter the following Exec mode commands and save the output to a file:
• clear stats all

• show clock
• show tech-support
• show clock
7. Be familiar with your application setup.

To troubleshoot performance issues with your ACE, follow these steps:
1. Display the resources allocated to each resource class in the ACE by entering the following command:
ACE_module5/Admin# show resource allocation

---------------------------------------------------------------------------
Parameter Min Max Class
---------------------------------------------------------------------------
acl-memory 0.00% 100.00% default

0.00% 100.00% RC1
syslog buffer 0.00% 100.00% default

0.00% 100.00% RC1
conc-connections 0.00% 100.00% default

0.00% 100.00% RC1
mgmt-connections 0.00% 100.00% default

0.00% 100.00% RC1
proxy-connections 0.00% 100.00% default

0.00% 100.00% RC1
bandwidth 0.00% 100.00% default

0.00% 100.00% RC1
connection rate 0.00% 100.00% default

0.00% 100.00% RC1
Overview of Troubleshooting Performance Issues 147

inspect-conn rate 0.00% 100.00% default
0.00% 100.00% RC1
syslog rate 0.00% 100.00% default

0.00% 100.00% RC1
regexp 0.00% 100.00% default

0.00% 100.00% RC1
sticky 0.00% 100.00% default

5.00% 5.00% RC1
xlates 0.00% 100.00% default

0.00% 100.00% RC1
ssl-connections rate 0.00% 100.00% default

0.00% 100.00% RC1
mgmt-traffic rate 0.00% 100.00% default

0.00% 100.00% RC1
mac-miss rate 0.00% 100.00% default

0.00% 100.00% RC1
throughput 0.00% 100.00% default

0.00% 100.00% RC1
2. Display the resources allocated to the context in question by entering the following command:
ACE_module5/Admin# show resource usage context C1

Allocation
-------------------------------------------------------------------------------
Context: C1
xlates 0 0 0 1048574 0
bandwidth 0 0 0 625000000 0
throughput 0 0 0 500000000 0
mgmt-traffic rate 0 0 0 125000000 0 <------- 1 GBps bandwidth reserved f
acl-memory 0 0 0 78610432 0
sticky 0 0 209714 0 0
regexp 0 0 0 1048576 0
syslog rate 0 0 0 100000 0
Note: All bandwidth values are in units of bytes per second. To convert to bits per second (bps), multiply the displayed
bandwidth value by eight. The ACE reserves 1 Gbps of bandwidth for management (to-the-ACE) traffic.
3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:
cat6k# show fabric status

slot channel speed module fabric
status status
2 0 8G OK OK
3 0 8G OK OK
4 0 8G OK OK
5 0 8G OK OK <-------Shows 8 Gbps connectivity to the chassis back p
6 0 20G OK OK
8 0 8G OK OK
Troubleshooting Performance Issues 148

4. Check the fabric utilization by entering the following command:
cat6k# show fabric utilization

slot channel speed Ingress % Egress %
2 0 8G 3 2
3 0 8G 0 0
4 0 8G 0 0
5 0 8G 0 0
6 0 20G 0 0
8 0 8G 2 3
5. Display the load of the network processors (NPs) in terms of packets and connection processing for each microengine (ME) by
entering the following command:
ACE_module5/Admin# show np 1 me-stats -cpu

0 proxies open.
ME Utilization Statistics
--------------
RECEIVE: 7
FASTPATH: 44
SLOWTX: 0
TCP_RX: 0
HTTP: 0
IH_RX 0
SSL_ME: 0
CM_CLOSE: 36
X_TO_ME: 0
FIXUP: 0
REASSEMBLY: 0
OCM: 0
TCP_TX: 0
ICM: 39
ACE/Admin# show np 2 me-stats -cpu

0 proxies open.
ME Utilization Statistics
--------------
RECEIVE: 9
FASTPATH: 46
SLOWTX: 2
TCP_RX: 0
HTTP: 0
IH_RX 0
SSL_ME: 0
CM_CLOSE: 43
X_TO_ME: 0
FIXUP: 0
REASSEMBLY: 0
OCM: 0
TCP_TX: 0
ICM: 46
Note: All show np commands must be entered for both NP1 and NP2 to obtain the total combined results. NPs operate safely
at any percentage of utilization. As ME functions within the NPs approach 100 percent, the traffic load is stressing the
system close to its architectural limits. Any ME function that reaches 100 percent utilization can cause back pressure
and lead to dropped packets or dropped connections.
6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering the following
command:
ACE_module5/Admin# show cde health | include Fifo


Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally. Queues that can be
affected and create backpressure are as follows:
• FIFOs for the CDE, NPs, and the Crypto Module

• Internal queues for each ME
It is possible that some packets that are received by the ACE could be dropped internally if backpressure is applied.
7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQ Transmit Backpressure,
the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are not incrementing by entering the following
command:
ACE_module5/Admin# show np 1 me-stats "-s fp" | include Backpressure

FastQ Transmit Backpressure: 0
SlowQ Transmit Backpressure: 0
Drop: Transmit Backpressure: 0
ACE/Admin# show np 1 me-stats "-s fp" | include queue

Drop: Next-Hop queue full: 0
8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpath queue full, Drops
due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Drops due to Fixup queue full are not
incrementing by entering the following command. If TCP receives backpressure, it can drop packets, fail to ACK packets, and fail
to properly track the next packet in the TCP connection.
ACE/Admin# show np 1 me-stats "-s tcp" | include queue

Drop reproxy msg queue full: 0
Drops due to FastTX queue full: 0
Drops due to Fastpath queue full: 0
Drops due to HTTP queue full: 0
Drops due to SSL queue full: 0
Drops due to AI queue full: 0
Drops due to Fixup queue full: 0
The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP, probes, and so
on) and handles configuration management to parse the CLI for syntactical errors and enforce configuration dependencies and
requirements before pushing the configuration to the data plane.
9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by entering the following
command:
ACE_module5/Admin# show processes cpu | inc util

CPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%
The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections (such as TCP,
UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing or inspection, and SSL
connection when using SSL acceleration). The ACE can support the maximum bidirectional concurrent connection limit
regardless of the features enabled.
Table 1. Concurrent Connection Support
Connection Type ACE Module Limit

Layer 4 4,000,000
Layer 7 512,000

The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained with distinct connection
objects.
10. Display the connection table by entering the following command:
ACE_module5/Admin# show conn
conn-id np dir proto vlan source destination state

----------+--+---+-----+----+---------------------+---------------------+------+
1 1 in TCP 130 161.44.67.242:2856 10.86.215.134:23 ESTAB
2 1 out TCP 130 10.86.215.134:23 161.44.67.242:2856 ESTAB
4 1 in TCP 130 161.44.67.242:2837 10.86.215.134:23 ESTAB
3 1 out TCP 130 10.86.215.134:23 161.44.67.242:2837 ESTAB
4 2 in TCP 130 161.44.67.242:2857 10.86.215.134:23 ESTAB
3 2 out TCP 130 10.86.215.134:23 161.44.67.242:2857 ESTAB
Note: You can add the detail command option to provide the following additional fields: connection idle time, elapsed time of
the connection, byte count, and packet count for each connection object.
The total current connections counter is also maintained in the output of the following command:
switch/Admin# show stats connection
+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created : 124
Total Connections Current : 6
Total Connections Destroyed: 62
Total Connections Timed-out: 58
Total Connections Failed : 0
Note: The Total Connections Current counter counts the number of used connection objects, not the number of TCP flows.
The number of TCP flows can be roughly determined as half the number of connection objects minus any UDP
connections. The Total Connections Current counter is always up to date and the maximum value can be 8,000,000.
Because of the Cisco ACE Module?s architecture, with distinct paths for new and established connections, the number of existing
concurrent connections does not heavily impact the rate at which new connections can be set up. Nevertheless, a very large
number of concurrent connections will eventually affect the performance of the system in setting up new connections.
11. Use the command "tcp wan-optimization rtt 0" for slow connections.
The ACE module architecture includes a mechanism where connections can be moved to the fastpath in order to increase
performance for a given connection. The LB decision is made in the software (proxy) and then moved to the fastpath (unproxy). In
a persistence rebalance scenario, the proxy/unproxy can occur Many times on a given connection. It is possible that if a packet
enters the system during the transition Between the proxy and unproxy states, a packet may not be forwarded as expected and a
retransmission may be relied upon. This can affect performance. As a workaround, it is possible to configure the ACE such that
fastpath forwarding is prohibited This can be accomplished by configuring a parameter map with the following:
"tcp wan-optimization rtt 0"
This article describes the ACE system limits and performance numbers for various resources and configuration objects.

Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 ACE Performance Numbers and Resource
Limits
♦ 1.1 ACE Appliance Data Sheet
♦ 1.2 ACE Module Data Sheets
♦ 1.3 SLB-Related Limits
♦ 1.4 Security-Related Limits
♦ 1.5 Management-Related Limits
Contents 152
ACE Performance Numbers and Resource Limits

For the most current performance numbers for the ACE products, always refer to the data sheets for the ACE appliance and the
ACE module.
ACE Appliance Data Sheet

ACE appliance data sheet
ACE Module Data Sheets

ACE10/ACE20 module data sheet
ACE30 module data sheet
If you have any questions or concerns related to ACE performance, please contact your Cisco account team for guidance.
SLB-Related Limits
Scalability Numbers The scalability numbers provided here are intended to provide guidelines related to configuration
scalability. The scalability numbers, however, are based on basic configurations. In order to obtain scalability numbers specific to
your deployment, testing with your feature combination is strongly recommended. If there are any questions or concerns related to
ACE performance, please contact your Cisco account team for guidance.
ACE Module ACE Module ACE Appliance

SLB-Related Object Additional Information
System Limit Context Limit Limit
ARP Entries 32,768 32,768 32,768
A few are reserved for L2 interafces,
Bridge Table Entries 32,768 32,768 32,768
redundancy, and so on.
Bridge-Group Virtual
4096 2048 512
Interfaces (BVIs)
Concurrent Conns L4
4,000,000 4,000,000 1,000,000
(Unproxied)
Concurrent Connections
512,000 512,000 128,000
L7 (Proxied)
Domains 2,500 10 (9) 10 (9 per context) One is used for the default domain.
Any object within the virtual partition
Domain Objects None None None
can be added to a domain.
Logical Interfaces 8,192 8,192 8,192
Resource Classes 100 (99) 1 100 (99) One is used for the default class.
Roles 4,000 16 (8) 16 (8) per context Eight are predefined.
Sticky Groups 4,096 4,096 4,096
Sticky Table Entries 4,000,000 4,000,000 800,000
21 (1 Admin
Virtual Contexts 251 N/A 250 user contexts + 1 Admin context
context)
VLANs 4,000 (2-4094) 4,000 (2-4094) 4,000 (2-4094)
ACE Performance Numbers and Resource Limits 153

Security-Related Limits
Scalability Numbers The scalability numbers provided here are meant to provide guidelines related to configuration scalability.
The scalability numbers, however, are based on basic configurations. In order to obtain scalability numbers specific to a particular
customer, testing with that customer?s feature combination is strongly recommended before any commitment on ACE
performance is made to the customer. If there are any questions or concerns related to ACE performance, please contact your
Cisco account team for guidance.
Security Related ACE Module ACE Module

ACE Appliance Limit Additional Information
Object System Limit Context Limit
Static NAT Policies 4096 4096 4096
Dynamic NAT
4096 4096 4096
Policies
Maximum of
addresses in a NAT 64 64 32
pool
Maximum of
addresses in a PAT 63k 63k 63l
pool
PAT Entries 4,000,000 4,000,000 1,000,000
Total NAT Pools 8,192 8,192 8,192
Xlates 1,000,000 1,000,000 64,000
Concurrent SSL
100,000 100,000 100,000 Subset of L7 (proxied) connections
Conns
Supported: 512, 786, 1536, 1024, 2048,
RSA key size up to 4096 bits up to 4096 bits up to 4096 bits and 4096 (imported public keys only)
bits
SSL Certs/Key files 3800/3800 3800/3800 3800/3800 (A3(1.x) and This number is strictly enforced in A220,
(A2(3.x) and (A2(3.x) and earlier) A214, and A322
earlier) earlier)
4096/4096 (A3(2.x) and
4096/4096 4096/4096 later, incl. A4(1.0))
Security-Related Limits 154

(A4(1.0) and later) (A4(1.0) and later)
Management-Related Limits
Scalability Numbers The scalability numbers provided here are meant to provide guidelines related to configuration scalability.
The scalability numbers, however, are based on basic configurations. In order to obtain scalability numbers specific to a particular
customer, testing with that customer?s feature combination is strongly recommended before any commitment on ACE
performance is made to the customer. If there are any questions or concerns related to ACE performance, please contact your
Cisco account team for guidance.
Management-Related ACE Module ACE Module

ACE Appliance Additional Information
Object System Limit Context Limit
AAA LDAP Servers 6,144 8 (24 total) 8
AAA RADIUS Servers 2K (256*8) 8 (24 total) 8
AAA TACACS+ Servers 6K (256*24) 8 (24 total) 8
One domain is used for the
Domains 2500 64 (63) 64 (63)
default-domain and cannot be removed
30 (Admin 31 (including admin,
Local Users 7500
context: 28) www, and dm)
Any object within the virtual partition
Objects within a Domain No limit No limit
can be added to a domain
Resource-classes 252 Not applicable 100
Eight are predefined and cannot be
Roles 4000 16 (8) 16 (8) altered, leaving eight for you to
customize
SNMP Hosts No Limit 10
SSH Sessions 256 4 4
Syslog buffer size 4 MB 4 MB 1 MB
5,000 per
Syslog CP rate 5,000 per seconds 3,000 per seconds
seconds
Management-Related Limits 155

350,000 per 350,000 per

Syslog DP rate 100,000 per second
second second
Syslog history table size 256 x 500 500
Syslog Hosts 256 2 2
Syslog internal queue size 10 MB 10 MB 8,192 messages
Syslog persistence size 1M 1M
Syslog rate limit table 10,000 messages per
256 x 100 100
size sec
Telnet Sessions 256 4 4
This article describes how to manage and control the ACE system resources.
Guide Contents
Main Article
Troubleshooting SSL
ACE Resource Limits
Contents
• 1 Overview of ACE Resources
• 2 Managing ACE Resources
♦ 2.1 ACE Resource Planning
♦ 2.2 Creating a Resource Class for Resource
Management
♦ 2.3 Allocating Resources Within a
Resource Class
♦ 2.4 Changing the Resource Allocation of a
Resource Class
♦ 2.5 Displaying the ACE Resource
Allocation and Usage
Contents 156
Overview of ACE Resources

Resource classes allow you to manage context access to ACE resources, such as concurrent connections or bandwidth rate. The
ACE is preconfigured with a default resource class that it applies to the Admin context and any user context upon creation. The
default resource class is configured to allow a context to operate within a range that can vary from no resource access (0 percent)
to complete resource access (100 percent).
When you use the default resource class with multiple contexts, you run the risk of oversubscribing ACE resources because the
ACE permits all contexts to have full access to all of the resources on a first-come, first-served basis. When a resource is utilized
to its maximum limit, the ACE denies additional requests made by any context for that resource.
To avoid oversubscribing resources and to help guarantee access to a resource by any context, the ACE allows you to create
customized resource classes that you associate with one or more contexts. A context becomes a member of the resource class
when you make the association. Creating a resource class allows you to set limits on the minimum and maximum amounts of each
ACE resource that a member context is entitled to use. You define the minimum and maximum values as percentages of all
resources. For example, you can create a resource class that allows its member contexts access to no less that 25 percent of the
total number of SSL connections that the ACE supports.
You can limit and manage the allocation of the following ACE resources:
• ACL memory
• Buffers for syslog messages and TCP out-of-order (OOO) segments
• Concurrent connections (through-the-ACE traffic)
• Management connections (to-the-ACE traffic)
• Proxy connections
• Set resource limit as a rate (number per second)
• Regular expression (regexp) memory
• SSL connections
• Sticky entries
• Static or dynamic network address translations (Xlates)
By default, when you create a context, the ACE associates the context with the default resource class. The default resource class
provides resources of a minimum of 0 and a maximum of unlimited for all resources except sticky entries. For stickiness to work
properly, you must explicitly configure a minimum resource limit for sticky entries by using the limit-resource command.
Overview of ACE Resources 157

For more information about managing ACE resources, see the Cisco Application Control Engine Module Virtualization
Configuration Guide (Software Version A2(1.0)).

You can allocate system resources to multiple contexts by creating and defining one or more resource classes and then associating
the contexts with a resource class. This section contains the following topics:
• ACE Resource Planning

• Creating a Resource Class for Resource Management
• Allocating Resources within a Resource Class
• Changing the Resource Allocation of a Resource Class
ACE Resource Planning

When you plan the initial resource allocations for the virtual contexts in your configuration, allocate only the minimum required
or estimated resources. The ACE protects resources that are in use, so to decrease a context's resources, those resources must be
unused. Although it is possible to decrease the resource allocations in real time, it may require additional management overhead to
clear any used resources before reducing them. Therefore, it is considered a best practice to initially keep as many resources in
reserve as possible and allocate the unused reserved resources as needed.
To address scaling and capacity planning, we recommend that new installations do not exceed 60 to 80 percent of the ACE's total
capacity. To accomplish this goal, create a reserved resource class with a guarantee of 20 to 40 percent of all the ACE resources
and configure a virtual context dedicated solely to ensuring that these resources are reserved. Then, you can efficiently distribute
such reserved resources to contexts as capacity demands for handling client traffic increase over time.
Creating a Resource Class for Resource Management

You can create a resource class to allocate and manage system resources by one or more contexts. The ACE supports a maximum
of 100 resource classes. After you create and configure the resource class, use the member command in context configuration
mode to assign a resource class to the context (see the "Associating a Context with a Resource Class" section). To create a
resource class, use the resource-class command in configuration mode. The syntax of the command is as follows:
resource-class name
For the name argument, enter an unquoted text string with no spaces and a maximum of 64 alphanumeric characters.
For example, to create the RC1 resource class, enter the following command:
ACE_module5/Admin(config)# resource-class RC1

ACE_module5/Admin(config-resource)
To remove the resource class from the configuration, enter the following command:
host1/Admin(config)# no resource-class RC1
When you remove a resource class from the ACE, any contexts that were members of that resource class automatically become
members of the default resource class. The default resource class allocates a minimum of 0.00 percent to a maximum of 100.00
percent of all ACE resources to each context. You cannot modify the default resource class.
Managing ACE Resources 158

Allocating Resources Within a Resource Class

You can allocate all resources or individual resources to all member contexts of a resource class. For example, you can allocate
only concurrent connections or sticky table memory or management traffic. To allocate system resources to all members
(contexts) of a resource class, use the limit-resource command in resource-class configuration mode.
The syntax of this command is as follows:
limit-resource {acl-memory | all | buffer {syslog} | conc-connections | mgmt-connections | proxy-connections | rate

{bandwidth | connections | inspect-conn | mac-miss | mgmt-traffic | ssl-bandwidth | syslog} | regexp | sticky | xlates}
{minimum number} {maximum {equal-to-min | unlimited}}
Note: The limit that you set for individual resources when you use the limit-resource command overrides the limit that you
set for all resources when you use the limit-resource all command.
If you lower the limits for one context (context A) in order to increase the limits of another context (context B), you may
experience a delay in the configuration change because the ACE will not lower the limits of context A until the resources are no
longer being used by the context.
For example, to allocate 20 percent of all resources (minimum and maximum) to all member contexts of the resource class, enter
the following command:
(config-resource)# limit-resource all minimum 20% maximum equal-to-min
To restore resource allocation to the default values of 0 percent minimum and 100 percent maximum for all resources to all
member contexts, enter the following command:
(config-resource)# no limit-resource all
Table 1 lists the managed system resources of the ACE. You can limit these resources per context or for all contexts associated
with the resource class by using the limit-resource command. See the "Allocating Resources within a Resource Class" section.
Table 1. System Resource Maximum Values
Resource Maximum Value

ACL Memory 78,610,432 bytes
Buffer Memory (Syslog) 4,000,000 bytes
4,000,000 connections
(Layer 4)
200,000
(SSL)
Management Connections 100,000 connections
Proxy Connections (Layer 7) 524,286 connections
SSL Proxy Connections 200,000
Rate
---Bandwidth 4 gigabits per second (Gbps)
You can upgrade the ACE maximum bandwidth to 8 Gbps or 16 Gbps by purchasing a separate
license from Cisco. For more information, see the Cisco Application Control Engine Module
Administration Guide (Software Version A2(1.0)).
---Connections (any kind) 325,000 connections per second (CPS)
Allocating Resources Within a Resource Class 159

---MAC miss 2000 packets per second (PPS)

---Management traffic 1 Gbps
---SSL transactions 1000 transactions per second (TPS), upgradeable to 15000 TPS with a separate license. For more
information, see the Cisco Application Control Engine Module Administration Guide (Software
Version A2(1.0)).
---Syslog For traffic going to the ACE (control plane), 5000 messages per second
For traffic going through the ACE (data plane), 350,000 messages per second
Regular Expression Memory 1,048,576 bytes
Sticky Entries 4,194,304 entries
Xlates (network and port 524,286 translations
address translation entries)
Changing the Resource Allocation of a Resource Class

If you (as the global Admin) need to change the resource allocation in a resource class of which two or more user contexts are
members, you may do so at any time by entering the appropriate CLI commands. (For details about allocating resources, see the
"Allocating Resources Within a Resource Class" section.) However, the shift in resources between the contexts does not take
place immediately unless the appropriate resources are available to accommodate the change. In most cases, to effect a change in
resource allocation, you must inform the context administrators involved to ensure that the new resource allocation is possible.
Changing the Resource Allocation of a Resource Class 160

For example, suppose that context A is using 100 percent of the available resources of the class and you want to allocate 50
percent of the resources to context A and 50 percent of the resources to context B. Although the CLI accepts your resource
allocation commands, context B cannot allocate 50 percent of the resources until context A deallocates 50 percent of its resources.
In this case, you must perform the following:
• Inform the Context A administrator to start deallocating resources
• Inform the Context B administrator to start allocating resources after the Context A administrator releases the resources
Note: As resources are released from other contexts, the ACE assigns the resources to resource-starved contexts (contexts
where the resource-class minimum allocations have not been met).
Displaying the ACE Resource Allocation and Usage
To view the current resource allocation in your ACE, enter the following command:
ACE_mdule5/Admin# show resource allocation

---------------------------------------------------------------------------
Parameter Min Max Class
---------------------------------------------------------------------------
acl-memory 0.00% 100.00% default
syslog buffer 0.00% 100.00% default
conc-connections 0.00% 100.00% default
mgmt-connections 0.00% 100.00% default
proxy-connections 0.00% 100.00% default
bandwidth 0.00% 100.00% default
connection rate 0.00% 100.00% default
inspect-conn rate 0.00% 100.00% default
syslog rate 0.00% 100.00% default
regexp 0.00% 100.00% default
sticky 0.00% 100.00% default
xlates 0.00% 100.00% default
ssl-connections rate 0.00% 100.00% default
mgmt-traffic rate 0.00% 100.00% default
mac-miss rate 0.00% 100.00% default
throughput 0.00% 100.00% default
To view the current resource usage, enter the following command:
ACE_mdule5/Admin# show resource usage

Allocation
-------------------------------------------------------------------------------
Context: Admin
xlates 0 0 0 1048574 0
bandwidth 1094 80192 0 625000000 0
throughput 938 75902 0 500000000 0
acl-memory 23776 28616 0 78610432 0
sticky 0 0 0 0 0
regexp 0 0 0 1048576 0
syslog rate 0 0 0 100000 0
Note: All bandwidth values are in bytes per second. To convert to bits per second (bps), multiply the values by eight. The
ACE guarantees 1 Gbps of bandwidth for management traffic. So, the total bandwidth for a 4-Gbps ACE license is
actually 5 Gbps. Throughput is still 4 Gbps.
To display the data plane resource allocation and usage and to cross-check the output of the above two commands, enter the
following command:
ACE_module5/Admin# show np 1 me-stats -L0

Resource limts for context : 0
Rate Configured Counters
Policer Name Min Max min-toks max-toks peak-toks deny
bandwidth: 0 ee6b280 0 ee6b0fa d8a4 0
throughput: 0 ee6b280 0 ee6b280 d8a4 0
mgmt-traffic rate: 0 3b9aca0 0 3b9aca0 a0e 0
connection rate: 0 7a120 0 7a120 11 0
ssl-connections rate: 0 9c4 0 9c4 0 0
mac-miss rate: 0 3e8 0 3e8 0 0
inspect-conn rate: 0 bb8 0 bb8 0 0
Resource Configured Counters

Policer Name Min Max Min Max peak deny
conc-connections: 0 3d0900 0 0 0 0
mgmt-connections: 0 c350 0 0 4 0
proxy-connections: 0 7ffff 0 0 0 0
ip-reassemble buffer: 0 0 0 0 0 0
tcp-ooo buffer: 0 0 0 0 0 0
regexp: 0 0 0 0 0 0
xlates: 0 7ffff 0 0 0 0
The Admin context has a context ID of 0. To display the resource allocation and and usage statistics for another context, change
the "0" in the "-L<context_id>" parameter to the context ID of another context.
Displaying the ACE Resource Allocation and Usage 162

Ace Ts Guide

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ace Ts Guide

Uploaded by

Copyright:

Available Formats

Cisco Application Control Engine (ACE) Troubleshooting Guide

Click here to go to the Cisco ACE Module documentation on www.cisco.com.

Click here to go to the Cisco ACE Appliance documentation on www.cisco.com.

Overview of ACE Troubleshooting

Understanding the ACE Module Architecture and Traffic Flow

Preliminary ACE Troubleshooting

Troubleshooting ACE Boot Issues

Troubleshooting with ACE Logging

Troubleshooting Ethernet Ports (ACE appliance)

Troubleshooting Remote Access

Troubleshooting Access Control Lists

Troubleshooting Network Address Translation

Troubleshooting ACE Health Monitoring

Troubleshooting Layer 4 Load Balancing

Troubleshooting Layer 7 Load Balancing

Troubleshooting Performance Issues

ACE Resource Limits

Show Counter Reference

Creating a PDF of the ACE Troubleshooting Wiki

ACE Appliance Documentation

• Cisco CSS-to-ACE Conversion Tool User Guide

• Cisco Application Control Engine (ACE) Configuration Examples on DocWiki

ACE Appliance Documentation 3

Overview of the ACE Troubleshooting Process

Verifying the ACE Image

ACE_module5/Admin# show version

last boot reason: NP 1 Failed : NP ME Hung

configuration register: 0x1

This command provides other useful information, for example:

Enabling ACE Logging

ACE_module5/Admin(config)# logging enable

Gathering ACE Troubleshooting Information

Rebooting the ACE

Using show Commands

ACE_module5/Admin# show tech-support > ?

Verifying the ACE Image 5

Capturing Packets in Real Time

The ACE captures packets subject to the following guidelines:

• One capture session is used per context

To capture packets in real time, follow these steps:

2. Enter the capture command, for example:

ACE_module5/Admin# capture CAPTURE1 interface vlan 200 access-list FILTER

ACE_module5/Admin# show capture CAPTURE1 status

Capture session : TEST

ACE_module5/Admin# capture CAPTURE1 start

ACE_module5/Admin# capture CAPTURE1 stop

5. Copy the packet capture to disk0: by entering the following command:

ACE_module5/Admin# copy capture CAPTURE1 disk0:CAPTURE1

ACE_module5/Admin# show capture CAPTURE1

ACE_module5/Admin# show capture CAPTURE1 detail

0002: msg_type: CON_SETUP

Capturing Packets in Real Time 7

0003: msg_type: PKT_RCV

0004: msg_type: PKT_XMT

0005: msg_type: PKT_RCV

0006: msg_type: PKT_XMT

0007: msg_type: PKT_RCV

0008: msg_type: PKT_XMT

Capturing Packets in Real Time 8

0009: msg_type: PKT_RCV

0010: msg_type: PKT_XMT

ACE_module5/Admin# dir core:

123589 Feb 22 00:34:20 2009 qnx_1_mecore_log.999.tar.gz

Usage for core: filesystem