Lab Guide

Advanced Network Automation Solutions using Cisco IOS EEM

LABNMS-2001

Arie Vayner, avayner@cisco.com

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Advanced Network Automation Solutions using Cisco IOS EEM LABNMS-2001

Contents
Advanced Network Automation Solutions using Cisco IOS EEM .............................. 2 LABNMS-2001 ...................................................................................................... 2 Session Abstract ................................................................................................... 2 Introduction to EEM ............................................................................................... 3 EEM References ................................................................................................... 3 EEM Debugging Commands ................................................................................. 4 Lab Structure ......................................................................................................... 5 Task 1 Block a CLI Command ............................................................................ 6 Task 2 Control CLI Command Execution Rate ................................................... 8 Task 3 Scheduling Events .................................................................................. 9 Task 4 Manually Triggered EEM Scripts ........................................................... 10 Task 5 Monitor Interface Parameters ................................................................ 11 Task 6 Switch between Primary and Backup Paths .......................................... 14 Task 7 Consolidated Custom Status Command ............................................... 19 Task 8 Secure Automatic Provisioning ............................................................. 22 Appendix I ............................................................................................................... 28

Session Abstract
In this session we will review advanced automation and manageability solutions based on Cisco IOS Embedded Event Manager (EEM) functionality. The session will allow delegates to gain hands-on experience of the implementation of advanced solutions including high availability, network performance optimization, network monitoring and efficient automation. The session would provide a relevant tool set for enhancing network operations within networks built on Cisco IOS based routers and switches.

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Introduction to EEM
Cisco IOS Embedded Event Manager (EEM) is a powerful and flexible subsystem that provides real-time network event detection and onboard automation. It gives you the ability to adapt the behaviour of your network devices to align with your business needs. Your business can benefit from the capabilities of IOS Embedded Event Manager without upgrading to a new version of Cisco IOS Software. It is available on a wide range of Cisco platforms.
Figure 1
Syslog OIR GOLD IP SLA SNMP None
SNMP Proxy
SNMP Object

Event Detectors
Watchdog ERM XML RPC
Neighbor Discovery

Counter EOT Routing Identity

CLI RF Netflow MAC

IOS Embedded Event Manager supports more than 20 event detectors that are highly integrated with different Cisco IOS Software components to trigger actions in response to network events.
Figure 2 EEM Architecture
EEM Server Subsystem EEM Event Detector EEM Policies

Your business logic can be injected into network operations using IOS Embedded Event Manager policies. These policies are programmed using either simple command-line interface (CLI) or using a scripting language called Tool Command Language (Tcl). Harnessing the significant intelligence within Cisco devices, IOS Embedded Event Manager helps enable creative solutions, including automated troubleshooting, fault detection, and device configuration.

EEM References
Embedded Event Manager Overview http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_eem_overvie w_ps10591_TSD_Products_Configuration_Guide_Chapter.html Writing Embedded Event Manager Policies Using Cisco IOS CLI Configuration Guide http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_eem_policy_ cli.html
LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Embedded Event Manager (EEM) Scripting Community (Cisco Beyond) http://forums.cisco.com/eforum/servlet/EEM?page=main Relevant Command Reference Guides http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_01.html http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_02.html http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_05.html http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.html

EEM Debugging Commands

The following commands can be used to debug and display the operations of the different scripts used in this lab: debug event manager action cli debug event manager detector <event detector type> show event manager detector <event detector type> detailed show event manager policy registered show track <id>

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Lab Structure
The different tasks in this lab provide introduction to different elements of Embedded Event Manager applet programming. Each task would present a problem that may solve some operational challenge, providing an example using an EEM CLI applet. Each applet would present new programming elements and tools available in EEM. It is recommended to try and configure the different examples in the lab, and then try executing them (see the output examples). It is also highly recommended to turn on relevant debugging commands (see the above debugging reference) and use the different show commands provided above. Feel free to experiment and modify the applets to create more advanced solutions.

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 1 Block a CLI Command

Goal: Use an EEM applet to block the show beep ... command Script Logic: Use the event cli pattern event detector to catch any relevant command by matching a regular expression. The syslog action allows publishing a customer syslog event The puts actions writes a string to the active terminal Setting the _exit_status variable to 0 would block the executed command (1 would allow the original command to run after script execution) Introduced EEM Elements: event cli action syslog action puts Using _exit_status Example: ! event manager applet BLOCK-CLI-SHOW-BEEP event cli pattern "^show beep" mode "exec" enter action 1.0 syslog msg "BEEP!" action 1.1 puts "BEEP!" action 2.0 set _exit_status "0" !

Output Example: Router#show beep BEEP! Router# *Nov 24 20:58:18.424: %HA_EM-6-LOG: BLOCK-CLI: BEEP!

Tip: If the cli pattern is in another mode than exec it is possible to identify the mode by running debug event manager all and execute the required command:
LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Router#debug event manager all Router#show beep *Nov 24 20:56:29.856: check_eem_cli_key: line=show beep mode=exec

Tip 2: Using _exit_status=0 allows creating new CLI commands. Matching for an undefined CLI string can trigger a script which can perform custom tasks. A common example is combining the partial (using | include) output of multiple show commands creating an overview show status command (see Error! Reference source not found.)

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 2 Control CLI Command Execution Rate

Goal: Use an EEM applet to control the rate allowed for any show command Script Logic: Adding the occurs 3 period 10 to the event cli detector allows triggering the script only if it occurs at least 3 times in a period of 10 seconds Introduced EEM Elements: event ... occurs ... period puts nonewline Example:
! event manager applet TOO-FAST-SHOW event cli pattern "show" sync yes occurs 3 period 10 action 1.0 puts nonewline "Too fast show commands, please slow down" action 2.0 set _exit_status "0" !

Output Example: Router#show clock *21:14:07.560 CET Tue Nov 24 2009 Router#show clock *21:14:09.720 CET Tue Nov 24 2009 Router#show clock Too fast show commands, please slow down Tip: The occurs X period Y option is available on other event detectors, including syslog events, track objects etc. It can be used to detect different kinds of repeating events allowing detection of events such as link flaps, routing update flapping etc.

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 3 Scheduling Events

Goal: Use an EEM applet to execute scripts at specific time of day. The example script would perform a daily configuration backup task. Script Logic: Script is executed every day, Monday to Friday, at 23:55 enable has to be executed as the script is executed in a separate VTY, and starts in low privilege level Introduced EEM Elements: event timer cron Example:
! event manager applet PERIODIC-CONFIG-SAVE event timer cron name CONFIG-SAVE-TIMER cron-entry "55 23 * * 1-5" action 1.0 cli command "enable" action 2.0 cli command "copy running-config startup-config" !

Tips: The cron scheduler syntax is: Minute, Hour, Day, Month, Day of Week (similar to unix cron syntax) Some shortcuts are available (@hourly, @weekly etc) Full details can be reviewed at http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.ht ml#wp1157622

Note

EEM scripts do not pass authentication. If the script is in the configuration, it means that the person who configured it had full config rights CLI commands executed by an EEM script (action cli command) can get authorized (for example in case of per-command authorization with TACACS). The username used for authorization is set using the event manager session cli username <username> command

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 4 Manually Triggered EEM Scripts

Goal: Use an EEM applet to create a single line command to perform clear counters hiding the [confirm] prompt Script Logic: event none allows manual execution of the script from a CLI command using event manager run <script-name> The pattern keyword allows catching a string written to the VTY terminal Using the alias command will allow easy execution of the new command Introduced EEM Elements: event none and manually running EEM applets Matching CLI prompts (pattern) Using a CLI alias to run EEM applets Example:
! event manager applet CLEAR-COUNTERS event none action 1.0 cli command "enable" action 2.0 cli command "clear counters" pattern "\[confirm\]" action 3.0 cli command "y" ! alias exec cc event manager run CLEAR-COUNTERS !

Tips: debug event manager action cli would allow seeing the operation of the scripts Note the escape sequence \[ instead of just using a [ character. The [ (and ]) have a special meaning in regular expressions, and have to be escaped using \. Output Example: Router#cc Router# *Nov 25 09:02:09.517: %CLEAR-5-COUNTERS: Clear counter on all interfaces by on vty0 (EEM:CLEAR-COUNTERS)

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 5 Monitor Interface Parameters

Goal: Use an EEM applet to react to crossing a counter threshold on an interface. This script would monitor the input bit per second (BPS) and packet per second (PPS) counters of an interface, and if the exceed a given threshold, an alert would be generated. Script Logic: Event tags define different events that may be correlated in order to have a combined trigger event for the applet. The trigger statement defines the correlation between the 2 different events. We would be monitoring two different events: o Ethernet0/0 receive BPS rate crossing 128000 bps o Ethernet0/1 receive PPS rate crossing 100 pps If the above thresholds have been crossed, we also want to know when the issue has been resolved, so if BPS rate on E0/0 drops below 32000 bps, or PPS rate on E1/0 drops below 50, an exit event would be triggered The exit-event true statement in the events would trigger the script also when the lower threshold has been reached We use the boolean system variable $_interface_exit_event to detect if the event is an entry or exit event (high or low threshold). This is done using an if action. For reporting the event, we use some other system variables which are populated automatically when an interface event is triggered. Introduced EEM Elements: event interface name multiple events and event correlation triggers using event detector specific system variables using the if/else conditional syntax Example:
! event manager applet MONITOR-INTERFACES event tag ETH0-0-RX-BPS interface name Ethernet0/0 parameter receive_rate_bps entry-op gt entry-val 128000 entry-type value exit-op lt exit-val 32000 exit-type value exit-event true poll-interval 1 event tag ETH0-1-RX-PPS interface name Ethernet0/1 parameter receive_rate_pps entry-op gt entry-val 100 entry-type value exit-op lt exit-val 50 exit-type value exit-event true poll-interval 1 trigger correlate event ETH0-0-RX-BPS or event ETH0-1-RX-PPS action 1.0 if $_interface_exit_event eq 0 action 1.1.1 syslog priority alerts msg "RED ALERT: $_interface_name $_interface_parameter is $_interface_value" action 1.2 else

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

action 1.3.1 syslog priority informational msg "GREEN ALERT: $_interface_name $_interface_parameter is $_interface_value" action 1.4 ! end

Output Example: *Jul 29 08:34:32.979: %HA_EM-1-LOG: MONITOR-INTERFACES: RED ALERT: Ethernet0/0 receive_rate_pps is 320 How to Test: In order to generate traffic on the links being monitored by the script, it is possible to log in into the opposite router on the other side of the link, and use the ping command to generate traffic. In order to generate a high rate of packets, it is recommended to use an extended ping command using a high count of ping packets and a timeout of 0. This will make the opposite router to generate a high rate of packets. In order to get a high BPS rate, it is also possible to increase the ping packet size.
R102#ping Protocol [ip]: Target IP address: 10.1.5.100 Repeat count [5]: 10000 Datagram size [100]: 1000 Timeout in seconds [2]: 0 Extended commands [n]: Sweep range of sizes [n]: Type escape sequence to abort. Sending 10000, 1000-byte ICMP Echos to 10.1.5.100, timeout is 0 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

If the ping operation is taking too long to complete (due to a high repeat count), it is possible to break it using the break sequence CTRL-SHIFT-6. Tips: In order to see which system variables are available for each of the different event detectors it is possible to use the show event manager detector <name> detailed Router#show event manager detector interface detailed .... Applet Built-in Environment Variables: $_event_id $_event_type $_event_type_string $_event_pub_time $_event_pub_sec
LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

$_event_pub_msec $_event_severity $_interface_name $_interface_parameter $_interface_is_increment $_interface_value $_interface_delta_value $_interface_exit_event It is possible to use the elseif action for more conditions in the if structure.

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 6 Switch between Primary and Backup Paths

Goal: Use an EEM applet to react on primary link failure and restoration. When the primary path fails, the backup path should be activated When the primary path is restored, the backup path should be deactivated, but only after verifying the primary path has fully converged. Script Logic:
Figure 3 Logical Lab Topology

Primary
R100
Lo0: 10.10.10.100/32 2001::100/128

E0/0-E0/3 10.1.1-4.100/24

R102
10.1.1-4.101/24 E0/0-E0/3 Lo0: 10.10.10.101/32 2001::101/128

R101

Backup
On the spoke router, R102, a pair of tunnels are used to provide connectivity to the remote hub sites. Tunnel100 is the primary tunnel, and should be enabled at all times. Tunnel101 is the backup tunnel, and is kept shutdown as long as the primary path is active. One script should detect the primary path failing, and enable (no shut) Tunnel 101 As soon as the primary path is restored, we should probe the primary path and make sure it is restored, and only then shut down Tunnel101 again. This is done by another EEM script. Track object 10 tracks the line protocol of Tunnel 100 (note that Tunnel 100 is configured with Keepalive enabled). Track 10 going down would trigger ENABLE-BACKUP-PATH, which would perform no shut to interface Tunnel101 When Tunnel100 recovers, the directly connected subnet 10.20.20.0/24 would be added into the routing table, triggering the event routing in RECOVER PRIMARY-PATH. The script would use a while loop to ping the remote side of Tunnel100, and only after it becomes available would perform shut for interface Tunnel101. The script uses a regexp action to match the !!!! outpu t of the ping command.

Introduced EEM Elements:

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

track objects and the track event detector event routing event detector using the regexp action using while loops using the _cli_result system variable

Example: ! track 10 interface Tunnel100 line-protocol ! event manager applet ENABLE-BACKUP-PATH event track 10 state down action 001 cli command "enable" action 002 cli command "conf t" action 003 cli command "int tunnel 101" action 004 cli command "no shut" action 005 syslog priority alerts msg "PRIMARY LINK IS DOWN. BACKUP LINK ACTIVATED" ! event manager app RECOVER-PRIMARY-PATH event routing network 10.20.20.0/24 type add maxrun 30 action 001 cli command "enable" action 002 set done 0 action 003 while $done ne 1 action 004 wait 5 action 005 cli command "ping 2001:20:20::100" action 006 regexp "!!!!!" "$_cli_result" action 007 if $_regexp_result eq 1 action 008 cli command "config t" action 009 cli command "int Tunnel101" action 010 cli command "shut" action 011 cli command "end" action 012 set done 1 action 013 end action 014 end action 015 syslog priority alerts msg "PRIMARY LINK IS RESTORED. BACKUP LINK DEACTIVATED" ! Tips: The routing event detector can detect not only specific prefixes, but any prefix which falls inside a predefined subnet range. For the complete syntax please refer to: http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.ht ml#wp1156862 The track object can be enhanced to delay any down or up event from propagating into the triggered event by configuring the delay up/down command under the track object configuration. This can allow more advanced policies which trigger events only after a certain condition is stable for a while. The show track <id> command would show how much time left for the state delay to expire.

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html# wp1163288 Stub tracking objects can be configured to be used through the tr ack set/read EEM actions. These track objects can maintain states inside or between different runs of EEM applets using the EEM actions track set and track read. http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_21.ht ml#wp1098882 Track objects can also track ip routes and IP SLA probes (http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html# wp1163396) (http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html# wp1163503) Multiple track objects can be combined to create complex logical conditions and trigger EEM applets with the track list command: http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html# wp1163622 In order to get faster response from track object events the track timer command should be used. http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html# wp1158894

Output Example:
R102#debug event manager action cli Debug EEM action cli debugging is on R102#debug event manager detector routing Debug EEM Routing Event Detector debugging is on (Perfromed shut on interface Tunnel100 on R100) R102# R102# *Nov 24 09:14:45.438: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel100, changed state to down *Nov 24 09:14:45.438: %TRACKING-5-STATE: 10 interface Tu100 line-protocol Up->Down *Nov 24 09:14:45.438: %DUAL-5-NBRCHANGE: EIGRP-IPv6 1: Neighbor FE80::A8BB:CCFF:FE00:6400 (Tunnel100) is down: interface down *Nov 24 09:14:45.446: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.20.20.100 (Tunnel100) is down: interface down ... *Nov 24 09:14:45.466: cli_open called. *Nov 24 09:14:45.478: R102> *Nov 24 09:14:45.478: R102>enable *Nov 24 09:14:45.598: R102# *Nov 24 09:14:45.598: R102#conf t %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : CTL : %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT : %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN :

%HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT : %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN :

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

*Nov 24 09:14:45.738: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z. *Nov 24 09:14:45.738: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT : R102(config)# *Nov 24 09:14:45.738: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN : R102(config)#int tunnel 101 *Nov 24 09:14:45.990: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT : R102(config-if)# *Nov 24 09:14:45.990: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN : R102(config-if)#no shut *Nov 24 09:14:46.126: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT : R102(config-if)# *Nov 24 09:14:46.126: %HA_EM-1-LOG: ENABLE-BACKUP-PATH: PRIMARY LINK IS DOWN. BACKUP LINK ACTIVATED *Nov 24 09:14:46.126: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : CTL : cli_close called. R102# *Nov 24 09:14:46.130: %SYS-5-CONFIG_I: Configured from console by vty0 R102# *Nov 24 09:14:48.054: %LINK-3-UPDOWN: Interface Tunnel101, changed state to up *Nov 24 09:14:48.070: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel101, changed state to up R102# R102# R102# R102# (Perfromed no shut on interface Tunnel100 on R100) R102# *Nov 24 09:15:00.546: changed state to up *Nov 24 09:15:00.546: *Nov 24 09:15:00.550: mask=255.255.255.0 *Nov 24 09:15:00.550: pattern network/len/, ... R102# *Nov 24 09:15:00.562: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : cli_open called. *Nov 24 09:15:00.586: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : R102> *Nov 24 09:15:00.586: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : R102>enable *Nov 24 09:15:00.722: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : R102# R102# *Nov 24 09:15:03.302: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.20.20.100 (Tunnel100) is up: new adjacency ... *Nov 24 09:15:04.434: %DUAL-5-NBRCHANGE: EIGRP-IPv6 1: Neighbor FE80::A8BB:CCFF:FE00:6400 (Tunnel100) is up: new adjacency R102# *Nov 24 09:15:05.754: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) R102#ping 2001:20:20::100 *Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) Type escape sequence to abort. *Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) Sending 5, 100-byte ICMP Echos to 2001:20:20::100, timeout is 2 seconds: *Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) !!!!! *Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) Success rate is 100 percent (5/5), round-trip min/avg/max = 0/0/4 ms *Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) R102# LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved CTL : OUT : IN : %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel100, %TRACKING-5-STATE: 10 interface Tu100 line-protocol Down->Up EEM routing ED: event to match: type=add, network=10.20.20.0, EEM routing ED: network/mask: 10.20.20.0/255.255.255.0 matched ge/le/ne: 10.20.20.0/24, 0/0/0

OUT :

: : IN

: : OUT : : : OUT : : : OUT : : : OUT : : : OUT :

*Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN : R102#config t *Nov 24 09:15:06.134: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z. *Nov 24 09:15:06.134: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT : R102(config)# *Nov 24 09:15:06.134: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN : R102(config)#int Tunnel101 *Nov 24 09:15:06.262: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT : R102(config-if)# *Nov 24 09:15:06.262: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN : R102(config-if)#shut *Nov 24 09:15:06.310: %DUAL-5-NBRCHANGE: EIGRP-IPv6 1: Neighbor FE80::A8BB:CCFF:FE00:6500 (Tunnel101) is down: interface down *Nov 24 09:15:06.318: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.20.21.101 (Tunnel101) is down: interface down *Nov 24 09:15:06.390: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT : R102(config-if)# *Nov 24 09:15:06.390: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN : R102(config-if)#end *Nov 24 09:15:06.422: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:RECOVER-PRIMARY-PATH) R102# *Nov 24 09:15:06.522: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT : R102# *Nov 24 09:15:06.522: %HA_EM-1-LOG: RECOVER-PRIMARY-PATH: PRIMARY LINK IS RESTORED. BACKUP LINK DEACTIVATED R102# *Nov 24 09:15:06.522: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : CTL : cli_close called. R102# *Nov 24 09:15:08.310: %LINK-5-CHANGED: Interface Tunnel101, changed state to administratively down *Nov 24 09:15:08.330: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel101, changed state to down R102# *Nov 24 09:15:09.894: EEM routing ED: RIB update: event=2, proc=2048, table=0, *Nov 24 09:15:09.894: EEM routing ED: event to match: type=remove, network=10.20.21.0, mask=255.255.255.0 *Nov 24 09:15:09.894: EEM Routing ED: num_matches = 0

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 7 Consolidated Custom Status Command

Goal: Use an EEM applet to create a custom consolidated show status command Script Logic: Add a new CLI command (show status) by using the cli event detector The event detector would match a non-existing command, and would execute the required actions The command matching uses a regular expression that would allow partial command matching (for example sh stat or sho statu) We use a global environment variable (_SERVICE_INF_LIST) which is defined in the router configuration. This variable holds a list of interfaces grouped into logical groups. The group names are marked with <<GROUP NAME>>. A foreach loop iterates through all the values in _SERVICE_INF_LIST. o Each time a group name is found (<<xxx>>) a group header is printed. The rest of the operations inside the loop instance are skipped using the continue action. o Each time a regular interface is found, the interface operational status is extracted, and printed in a custom format Introduced EEM Elements: Matching an unknown command pattern using a regular expression Using environment variables for global script parameters foreach loop to iterate through a list of values continue keyword inside a loop regexp matching and extraction of sub-strings Example:
! event manager environment _SERVICE_INF_LIST <<Core Interfaces>>,Eth0/0,Eth0/1,Eth0/2,Eth0/3,<<Spoke Interfaces>>,Eth1/0,<<Spoke Tunnels>>,Tun100, ! event manager applet SHOW_STATUS event cli pattern "(sh|sho|show)\s+(stat|statu|status)" mode "exec" enter action 001.1 cli command "ena" action 003.1 foreach _inf "$_SERVICE_INF_LIST" "," action action action action action action 004.1 004.2 004.3 004.4 004.5 004.6 regexp "<<(.*)>>" $_inf _match _str1 if $_regexp_result eq 1 puts "\n$_str1" puts "------------------------" continue end cli command "show interface $_inf" foreach _line $_cli_result "\n" regexp "^(.* is .*, line protocol is .*)\r$" $_line if $_regexp_result eq 1 puts $_str1 LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

action 005.1 action 006.1 action 007.1 _match _str1 action 007.2 action 007.3

action 007.4 action action action action action action action action action action action action 008.1 008.2 008.3 008.4 009.1 009.2 009.3 009.4 010.1 010.2 010.3 010.4

end regexp "^(.*Description.*)\r$" $_line _match if $_regexp_result eq 1 puts $_match end regexp "^(.* input rate .*)\r$" $_line _match if $_regexp_result eq 1 puts $_match end regexp "^(.* output rate .*)\r$" $_line _match if $_regexp_result eq 1 puts $_match end

action 099.1 end action 099.2 end !

Output Example:
R100#show stat Core Interfaces -----------------------Ethernet0/0 is up, line protocol is up Description: TO HUB2-R101 30 second input rate 0 bits/sec, 0 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec Ethernet0/1 is up, line protocol is up Description: TO HUB2-R101 30 second input rate 0 bits/sec, 0 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec Ethernet0/2 is up, line protocol is up Description: TO HUB2-R101 30 second input rate 0 bits/sec, 0 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec Ethernet0/3 is up, line protocol is up Description: TO HUB2-R101 30 second input rate 0 bits/sec, 0 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec Spoke Interfaces -----------------------Ethernet1/0 is up, line protocol is up 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 1000 bits/sec, 2 packets/sec Spoke Tunnels -----------------------Tunnel100 is up, line protocol is up 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec

Tips:

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

The regexp action matches a pattern (1st parameter) in the string that follows it (2nd parameter). The 3rd parameter would be populated with whatever part of the string was matched by the whole pattern, allowing content extraction from the matched string portion: o Any other parameters (4th and on) on the regexp command would be populated by extracted parts of the string (marked with ( and ) in the pattern) o The $_regexp_result system variable would hold a Boolean (0/1) result of the last regexp match. o http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm _01.html#wp1139025 The different regular expression strings used in the example include the following shortcuts: o \s match any white space o \r carriage return o \n new line (note that a show output end of line is matched with \r\n) o .* would match any (also zero) number of any characters o ^ - beginning of line o $ - end of line o _ - space (used in CLI | include syntax) note the | inc rate_ A more complete reference to Cisco regular expression support can be found at http://www.cisco.com/en/US/docs/ios/12_2/termserv/configuration/guide/tcfaa pre_ps1835_TSD_Products_Configuration_Guide_Chapter.html

The foreach loop action would iterate through all the fields in the provided string list, separated by the provided field separator o The field separator used in the foreach action could be \n. This would allow iterating through a list of lines (for example all the lines in the output of a show command or as it is used later in task 4) o http://www.cisco.com/en/US/docs/ios-xml/ios/eem/command/eem-cra1.html#GUID-A9FDB5DC-ED8F-422B-BE4B-B59DE5A1D0B4 The continue action would skip over later command, and would make the current loop skip to the next iteration. http://www.cisco.com/en/US/docs/ios-xml/ios/eem/command/eem-cra1.html#GUID-5A6D5C59-2EEF-44FE-B336-2F66FDC38757 You can also use the while loop to match for a loop condition http://www.cisco.com/en/US/docs/ios-xml/ios/eem/command/eem-cra2.html#GUID-5010385D-272C-48F7-BDE6-F413A44F3523

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Task 8 Secure Automatic Provisioning

Goal: Provision link configuration only when a specific neighbor is detected Script Logic: Use the neighbor-discovery event detector to detect new or expired CDP neighbors. Correlate either one of the two different events: o CDP add neighbor o CDP delete neighbor If a new neighbor is detected on the uplink interfaces, the remote hostname is checked, and only if it matches the expected hostname, the interface is provisioned If an unexpected hostname is detected, the interface is disabled as a security measure, and the script is aborted If a neighbor is deleted (for example if interface is down), the IP configuration is deleted Introduced EEM Elements: Neighbor-discovery event detector Event correlation (using event tag, and trigger) if/else/elseif logical operations Example:

Note

This example should be deployed on router R102 in the topology

event manager applet AUTO_PROVISION_LINKS event tag CDP-ADD neighbor-discovery interface regexp (Ethernet0/0|Ethernet1/0) cdp add event tag CDP-DEL neighbor-discovery interface regexp (Ethernet0/0|Ethernet1/0) cdp delete trigger correlate event CDP-ADD or event CDP-DEL action 001 cli command "ena" action 002 cli command "conf t" action 003 cli command "interface $_nd_local_intf_name" action 004 if $_nd_notification eq "cdp-delete" action 005 cli command "interface $_nd_local_intf_name" action 006 cli command "no ip address" action 007 syslog priority alerts msg "Uplink device $_nd_cdp_entry_name on interface $_nd_local_intf_name is no longer detected. IP Address configuration removed." action 008 elseif $_nd_notification eq cdp-add action 009 action 010 action 011 if $_nd_local_intf_name eq "Ethernet0/0" if $_nd_cdp_entry_name eq "R101" set ip_addr "10.1.6.102" LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

action 012 set ip_mask "255.255.255.0" action 013 else action 014 syslog priority alerts msg "Unexpected device detected on interface $_nd_local_intf_name ($_nd_cdp_entry_name). Interface is disabled." action 015 cli command "shut" action 016 exit action 017 end action 018 action action action action 019 020 021 022 elseif $_nd_local_intf_name eq "Ethernet1/0" if $_nd_cdp_entry_name eq "R100" set ip_addr "10.1.5.102" set ip_mask "255.255.255.0" else

action 023 syslog priority alerts msg "Unexpected device detected on interface $_nd_local_intf_name ($_nd_cdp_entry_name). Interface is disabled." action 024 cli command "shut" action 025 exit action 026 action 027 end end

action 028 cli command "ip address $ip_addr $ip_mask" action 029 syslog priority informational msg "Uplink device $_nd_cdp_entry_name on interface $_nd_local_intf_name detected. IP Address configuration applied." action 030 end

Tips: All system variables used by a specific event detector can be seen in IOS by using the following command:
router#show event manager detector neighbor-discovery detailed No. Name Version Node Type 1 neighbor-discovery 01.00 node0/0 RP <skipped> Applet Built-in Environment Variables: $_event_id $_job_id $_event_type $_event_type_string $_event_pub_time $_event_pub_sec $_event_pub_msec $_event_severity COMMON VARIABLES: $_nd_notification $_nd_intf_linkstatus $_nd_intf_linestatus $_nd_local_intf_name $_nd_short_local_intf_name $_nd_port_id CDP EVENT VARIABLES: $_nd_protocol $_nd_proto_notif $_nd_proto_new_entry $_nd_cdp_entry_name $_nd_cdp_hold_time $_nd_cdp_mgmt_domain $_nd_cdp_platform $_nd_cdp_version LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

$_nd_cdp_capabilities_string $_nd_cdp_capabilities_bits $_nd_cdp_capabilities_bits_[0-31] LLDP EVENT VARIABLES: $_nd_protocol $_nd_proto_notif $_nd_proto_new_entry $_nd_lldp_chassis_id $_nd_lldp_system_name $_nd_lldp_system_description $_nd_lldp_ttl $_nd_lldp_port_description $_nd_lldp_system_capabilities_string $_nd_lldp_enabled_capabilities_string $_nd_lldp_system_capabilities_bits $_nd_lldp_enabled_capabilities_bits $_nd_lldp_capabilities_bits $_nd_lldp_capabilities_bit_[0-31]

In order to figure out how all the different parameters used by the neighbordiscovery event detector, the following test script can be used:

event manager applet TEST-ND event tag CDP-ADD neighbor-discovery interface regexp .* cdp add event tag CDP-DELETE neighbor-discovery interface regexp .* cdp delete trigger correlate event CDP-ADD or event CDP-DELETE action 100 puts "_nd_notification=$_nd_notification" action 101 puts "_nd_intf_linkstatus=$_nd_intf_linkstatus" action 102 puts "_nd_intf_linestatus=$_nd_intf_linestatus" action 103 puts "_nd_local_intf_name=$_nd_local_intf_name" action 104 puts "_nd_short_local_intf_name=$_nd_short_local_intf_name" action 105 puts "_nd_port_id=$_nd_port_id" action 110 puts "_nd_protocol=$_nd_protocol" action 111 puts "_nd_proto_notif=$_nd_proto_notif" action 112 puts "_nd_proto_new_entry=$_nd_proto_new_entry" action 113 puts "_nd_cdp_entry_name=$_nd_cdp_entry_name" action 114 puts "_nd_cdp_hold_time=$_nd_cdp_hold_time" action 115 puts "_nd_cdp_mgmt_domain=$_nd_cdp_mgmt_domain" action 116 puts "_nd_cdp_platform=$_nd_cdp_platform" action 117 puts "_nd_cdp_version=$_nd_cdp_version" action 118 puts "_nd_cdp_capabilities_string=$_nd_cdp_capabilities_string" action 119 puts "_nd_cdp_capabilities_bits=$_nd_cdp_capabilities_bits"

Output Example:
R102#conf t Enter configuration commands, one per line. End with CNTL/Z. R102(config)#int e0/0 R102(config-if)#no shut R102(config-if)# *May 11 21:10:54.364: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up *May 11 21:10:55.372: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up R102(config-if)# R102(config-if)# *May 11 21:11:26.356: %HA_EM-6-LOG: AUTO_PROVISION_LINKS: Uplink device R101 on interface Ethernet0/0 detected. IP Address configuration applied. R102(config-if)#
LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

R102(config-if)#do show run int e0/0 Building configuration... Current configuration : 87 bytes ! interface Ethernet0/0 description TO R101 ip address 10.1.6.102 255.255.255.0 end R102(config-if)#int e0/0 R102(config-if)#shut R102(config-if)# *May 11 21:12:04.956: %HA_EM-1-LOG: AUTO_PROVISION_LINKS: Uplink device R101 on interface Ethernet0/0 is no longer detected. IP Address configuration removed. R102(config-if)# *May 11 21:12:06.052: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down *May 11 21:12:07.052: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to down R102(config-if)# R102(config-if)# R102(config-if)#do show run int e0/0 Building configuration... Current configuration : 75 bytes ! interface Ethernet0/0 description TO R101 no ip address shutdown end

-------------------------------------------------------------------------

R102#show cdp neighbors Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone, D - Remote, C - CVTA, M - Two-port Mac Relay Device ID R101 R100 Local Intrfce Eth 0/0 Eth 1/0 Holdtme 168 140 Capability R R Platform Port ID Solaris U Eth 1/0 Solaris U Eth 1/0

R102#show run int e1/0 Building configuration... Current configuration : 105 bytes ! interface Ethernet1/0 description TO R100 ip address 10.1.5.102 255.255.255.0 load-interval 30 end

R100(config)#hostname NOT_R100
LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

R102# *May 11 21:21:40.127: %HA_EM-1-LOG: AUTO_PROVISION_LINKS: Unexpected device detected on interface Ethernet1/0 (NOT_R100). Interface is disabled. R102#show run int e1/0 Building configuration... Current configuration : 93 bytes ! interface Ethernet1/0 description TO R100 no ip address load-interval 30 shutdown end

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Appendix I
Figure 4 provides a reference to the different EEM event detectors available in the different Cisco software trains and versions:
Figure 4 Event Detector Support Matrix

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved

Figure 5 shows the mapping between the EEM version and the different IOS version trains available:
Figure 5 IOS to EEM Version Mapping

The following command provides the EEM version on an IOS device: Router#show event manager version Embedded Event Manager Version 3.10 Component Versions: eem: (v310_throttle)4.1.10 eem-gold: (v310_throttle)1.0.7 eem-call-home: (v310_throttle)1.0.6 Event Detectors: Name Version Node application 01.00 node0/0 syslog 01.00 node0/0 track 01.00 node0/0 resource 01.00 node0/0 routing 02.00 node0/0 ...

Type RP RP RP RP RP

LABNMS-2001 2013 Cisco Systems, Inc. All rights reserved