Embedded Diagnostics and Management in Cisco IOS: TCL, EEM Scripting and Service Diagnostics

BRKDEV-1191

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

1

BRKDEV-1191
Session ID: BRKDEV-1191 Title: Embedded Diagnostic and Mgmt in IOS: TCL, EEM Scripting and Service Diagnostics Abstract: In this session, we discuss new embedded management capabilities in IOS including Service Diagnostics, EEM (Embedded Event Manager), EMM (Embedded Menu Manager), and TCL scripting. The session presents use-cases for autonomic diagnostic solutions using Cisco provided scripts. The session concludes with an overview of the new EEM event detectors and how they can be used to build your own custom embedded solutions. Owner: Speakers: Rick Williams, rwill@cisco.com
Product Manager OS Infrastructure and Services Software Group Cisco Systems, Inc.
Cisco Public

Level:
BRKDEV-1191 14596_04_2008_c2

Introductory
3

© 2008 Cisco Systems, Inc. All rights reserved.

What Do We Mean By “Embedded Diagnostics and Management”?
Embedded = On the box (Router or Switch) Diagnostics and Management
Two applications for on-board programming

The ability to develop programs within Cisco IOS can enhance other applications Partners and ISVs may benefit from on-board programming End customers also benefit from automation and customization In this session we’ll introduce Cisco IOS Embedded Event Manager, Embedded Menu Manager, and Service Diagnostics
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

4

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

2

Today’s Network Management Environment
Centralized, server-based Data gathering, polling, event correlation, root cause analysis Multiple, network management application specific systems
Trend for past 5-8 years has been toward off the shelf products Lately, incorporating Linux and Open-Source tools Managed Service Providers: Tighter integration Enterprises: Loosely integrated;

Issues with integration, scale, accuracy, customization, control, SPEED

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

The Future of Network Management
Networks are getting larger and larger—Increased dependence High dependence means higher availability requirements Reaction times are shorter Continuous cost pressures More “intelligence” in the net All driving need for automation and distributed/cooperative network management
Manager of Managers
Si Si Si

Context or Domain Network Management

Si

Si

Onboard Device Management

Embedded Management
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Devices play an active role in management
6

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Management Throughout

3

Cisco IOS Embedded Event Manager (EEM)

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

7

EEM—What and Why
What in the world is an embedded event manager?
An extremely flexible and very powerful subsystem within Cisco IOS Software Using EEM you can: Write programs that run on the router or switch

OK, so what? Why do I care?
It’s about automation—onboard automation Automate troubleshooting Automate commands Program automatic actions based on events Only limited by your imagination Really just the tip of the iceberg—as we’ll soon see
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

8

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

4

EEM Basic Architecture
* All within Cisco IOS

Policies (Scripts)
Applets and Tcl-based

EEM Server
The “brains” of the system

Event Detectors
“watch for events of interest”

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

9

EEM Event Detectors and EEM Policies
All of this is internal to Cisco IOS

Think of a policy as an action registered to an event
Applet-based policies Defined via CLI Simpler

ED notifies EEM Server; which triggers interested policies
Tcl-based policies Programmed in Tcl As complex as you want

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

5

Let’s Make an EEM Policy
Let’s start with a very simple applet policy Let’s write a special syslog message when we see a particular syslog message For example:
When someone leaves config mode, this message is seen:

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

11

Configure an EEM Applet Policy

We can make an Applet policy to match the on the syslog message Uses a REGEXP match We will register our policy…

Syslog Event

… Syslog Event Detector will notify the EEM Server when the message string match occurs … Our policy action will be invoked
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Syslog Action

12

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

6

Applet Configuration Detail
event manager applet CFGMSG
Define applet named CFGMSG

Event type will be “syslog”

iin-rtr1(config-applet)#event syslog

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

Syslog REGEXP Match “Pattern”

iin-rtr1(config-applet)#event syslog pattern "%SYS-5-CONFIG_I:"
BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

14

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

7

Add the Action—Syslog Action

iin-rtr1(config-applet)#action 1.0 syslog

Label—used to sort actions Alphabetic sort on the label

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

15

Complete Applet Policy Definition
event manager applet CFGMSG event syslog pattern "{%SYS-5-CONFIG_I:}" action 1.0 syslog priority warnings msg "Configuration event occurred"

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

16

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

8

Embedded Event Manager v2
Event Detectors
*Not all available in all releases
Interface Counter—Policies can be triggered based on the specific interface counter. Includes thresholds. Online Insertion and Removal—Triggers policies based on hardware installation and removal activity. Object Tracking—Triggers policies based on routing protocol events. Simple Network Management Protocol— Triggers policies based on the associated SNMP MIB variable. Includes MIB variable threshold setting. Syslog—Triggers policies based on the regular expression match of a local Syslog message. Resource Thresholding—Triggers policies based on certain internal resource usage and conditions. Interface to Embedded Resource Manager. Generic Online Diagnostics—Triggers policies based on diagnostic results “None” ED—Triggers policies by command
17

Cisco IOS CLI—Triggers policies based on commands entered via the CLI. Cisco IOS Counter—Policies can be triggered based on a change of the designated Cisco IOS counter. Cisco IOS Redundancy Facility—Provides for detection of hardware and software failures related to the Stateful Switchover service. This ED will trigger policies based on the RF state change. It is also used to initiate switchovers as a result of a policy action. Cisco IOS Timer Services—Policies can be scheduled to occur at the designated time or interval. Cisco IOS Watchdog/System Monitor— Triggers policies based on certain conditions relative to a certain Cisco IOS process or subsystem’s activity. EEM Application Specific—Application specific events can be detected or set by a Cisco IOS subsystem or a policy script. This provides the ability for one policy to trigger another policy.
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Embedded Event Manager Built-in Actions
An Embedded Event Manager Policy Can:
Execute an IOS CLI command and receive the result Send a CNS event Increment or decrement an EEM counter Force a switchover to the standby in a redundant configuration Request system information Send an e-mail Cause another EEM policy to be executed Publish an application specific EEM event Reload the box Send an SNMP trap with custom data Log a message to Syslog
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

18

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

9

Environment Variables
Variables that are referenced within policies that can be set in the config Can be used to customize policies Example:
Router#config t Router(config)#event manager environment _email_server email.cisco.com Router(config)#event manager environment _email_from soandso@somecompany.com

Note: environment variable names that start with the underscore character are reserved for Cisco use only

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

19

Environment Variables (Built-In)
Environment Variables Available for All Events $_event_type The event type that triggered the event. The time at which the event type was published. $_event_pub_time Environment Variables Available for SNMP Events $_snmp_oid The Simple Network Management Protocol (SNMP) object ID that caused the event to be published. The SNMP object ID value when the event was published. $_snmp_oid_val Environment Variables Available for Syslog Events $_syslog_msg The syslog message that caused the event to be published.

When a policy is entered… Cisco defined read-only environment variables called built-in variables are pre-set with the characteristics of the event that triggered the policy to run These environment variables can be used in ‘msg’ text
Will be replaced with the relevant text

Can be checked by policies There are a bunch of these—refer to Cisco documentation
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

20

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

10

Tcl Policies or Scripts
Tcl V8.3.4+ script support
This is the same support available in IOS for tclsh, ESM (Embedded Syslog Manager), and IVR There are several Tcl language extension dialects and some differences among them

EEM keyword extensions as documented in the “Guide To Writing EEM Policies” documentation implement event specification, system information requests, and built-in actions EEM Tcl library support is available for some common functions such as CLI, SMTP, and Tcl global variable state checkpointing. EEM uses the ::cisco::eem Tcl namespace
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

21

Getting Started: Create Policy Directory
ashcroft#mkdir ABCCoTclPol Create directory filename [ABCCoTclPol]? Created dir disk0:ABCCoTclPol ashcroft#dir Directory of disk0:/ 1 drw1 Oct 26 2003 13:37:42 +00:00 sys 6 drw1 Oct 30 2003 12:56:04 +00:00 ABCCoTclPol 47843328 bytes total (29356032 bytes free) ashcroft#conf t Enter configuration commands, one per line. End with CNTL/Z. ashcroft(config)#event manager directory user policy disk0:/ABCCoTclPol ashcroft(config)#^Z ashcroft#ashcroft#sh event man dir user policy disk0:/ABCCoTclPol ashcroft#
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

22

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

11

Getting Started: Copy Tcl Policy to Router
ashcroft#copy tftp disk0: Address or name of remote host []? 88.1.88.9 Source filename []? sl_cfgSaveRemT.tcl Destination filename [sl_cfgSaveRemT.tcl]? ABCCoTclPol/sl_cfgSaveRemT.tcl Accessing tftp://88.1.88.9/sl_cfgSaveRemT.tcl...! 1232 bytes copied in 0.620 secs (1987 bytes/sec) ashcroft#dir Directory of disk0:/ 1 drw1 Oct 26 2003 13:37:42 +00:00 sys 6 drw1 Oct 30 2003 12:56:04 +00:00 ABCCoTclPol 47843328 bytes total (29351936 bytes free) ashcroft#cd ABCCoTclPol ashcroft#dir Directory of disk0:/ABCCoTclPol/ 8 -rw1232 Oct 30 2003 14:14:58 +00:00 sl_cfgSaveRemT.tcl 47843328 bytes total (29351936 bytes free) ashcroft#

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

Getting Started: Register the Policy
ashcroft#conf t Enter configuration commands, one per line. End with CNTL/Z. ashcroft(config)#event manager policy sl_cfgSaveRemT.tcl type user ashcroft(config)# ashcroft#sh event manager policy registered No. Type Event Type Trap Time Registered Name 1 user syslog Off Thu Oct30 14:54:17 2003 sl_cfgSaveRem.tcl occurs 1 pattern {%SYS-5-CONFIG_I: Configured} nice 0 priority normal maxrun 90.000

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

24

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

12

Security of the Embedded Event Manager
Cisco scripts run in full Tcl mode User scripts run in Safe-Tcl mode Safe-Tcl allows Cisco to disable or customize individual Tcl commands
Certain commands are not permitted

User scripting can be completely disabled by omitting the “event manager directory user…” command
Remember, you don’t have to use it!

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

25

Security of the Embedded Event Manager
All config commands are privileged global config mode commands If users modify Cisco system scripts, they need to be run from the user directory in user mode When EEM policies execute CLI commands, the “event manager session cli username <username>” command specifies the username that is sent to TACACS+ for command authorization

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

26

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

13

Writing Your Own EEM Tcl-Based Policies
Start By Reading the Manual :-)
See “Writing Embedded Event Manager Policies”
EEM v2.2 Doc: http://www.cisco.com/en/US/docs/ios/netm gmt/configuration/guide/12_4t/nm_12_4t_ book.html EEM v2.1.5 Doc: http://www.cisco.com/en/US/docs/ios/ 12_2sx/sw_modularity/configuration guide/evnt_mgr.html

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

27

EEM Tcl Policy Structure
Begin with EEM Event Register keyword
Required

Next is any input variables or required environment variables to control the script Names space imports Entry criteria for the policy

Required

Body (logic of the script) Exit status

Required

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

28

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

14

Example of Tcl Policy Structure
Registration command (Tcl extension) Tcl namespace (namespace import)
::cisco::eem This namespace includes all Tcl commands closely related to Embedded Event Manager ::cisco::lib This namespace includes auxiliary library commands that are not necessarily specific to the Embedded Event Manager

Body

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

RegExp Tcl Command
http://www.tcl.tk/man/tcl8.3/TclCmd/regexp.htm

regexp ?switches? exp string ?matchVar? ?subMatchVar subMatchVar ...?
remote-pe#sh int fa5/0 | inc put, 2901 packets input, 1605860 bytes 3165 packets output, 189978 bytes, 0 underruns remote-pe#

Suppose we want the in and out ‘numbers’ from the above output
tclsh % set t "sh int fa5/0 | inc put, 2901 packets input, 1605860 bytes 3165 packets output, 189978 bytes, 0 underruns remote-pe#" % regexp {(\d+) packets input.+?(\d+) bytes.+?(\d+) packets output.+?(\d+) bytes} $t match pin bin pout bout 1 % set pout % set pin 3165 2901 % set bout % set bin 189978 1605860
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

30

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

15

Cisco IOS Embedded Event Manager Examples

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

Automatic Write Running Config to Flash Disk
Simple applet to keep a copy of running-config on flash disk whenever “write memory” is issued
event manager applet backup-config event cli pattern "write memory" sync yes action 1.0 cli command "enable" action 2.0 cli command "config t" action 3.0 cli command "file prompt quiet" action 4.0 cli command "end" action 5.0 cli command "copy running disk0:running-config" action 6.0 cli command "config t" action 7.0 cli command "no file prompt quiet" action 8.0 cli command "end"

Dealing with interaction:
event manager applet test2 *cli ’pattern’ option added by DDTS CSCsc96567 event none maxrun 20 action 10 cli command "enable" action 20 cli command "clear counters" pattern "\[confirm\]" action 25 cli command "y" action 30 cli command "disable" action 40 syslog msg "test2 ran"
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

32

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

16

Automatic Write Running Config to Flash Disk (Alternative)
Simple applet to keep a copy of running-config on flash disk whenever “write memory” is issued If you truly want the nvram "write memory" to come after the copy to flash you can continue to use "sync yes" and set a value for _exit_status that is greater than 0
event manager applet wrmem event cli pattern "write memory" sync yes action 1.0 cli command "enable" action 2.0 cli command "config t" action 3.0 cli command "file prompt quiet" action 4.0 cli command "end" action 5.0 cli command "copy running disk0:running-config" action 6.0 cli command "config t" action 7.0 cli command "no file prompt quiet" action 8.0 cli command "end" set 9 _exit_status 1

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

33

Using Environment Variables

Configuration
remote-pe#conf t Enter con igura ion com mands, one perl ne End w th CNTL/Z f t i . i . remote-pe(conf )#event manager env ronment u_c ig i fgSave_on 1

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

34

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

17

Cli Command Procedure
# # Local procedure for CLI interface # Pass a list of cli commands and it returns a list of outputs # # proc CLICmdProc {cmds} { global errorInfo lappend clicmd "show clock" if [catch {cli_open} result] { lappend clicmd "show auto discovery qos" error $result $errorInfo set cliout [CLICmdProc $clicmd] } else { # array set cli1 $result } if [catch {cli_exec $cli1(fd) "enable"} result] { error $result $errorInfo } if [catch {cli_exec $cli1(fd) "term len 0"} result] { error $result $errorInfo } foreach a_cmd $cmds { if [catch {cli_exec $cli1(fd) $a_cmd} result] { error $result $errorInfo } else { lappend cmd_output $result } } if [catch {cli_close $cli1(fd) $cli1(tty_id)} result] { error $result $errorInfo } return $cmd_output }
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

35

Cli Configuration Command Procedure
proc CLICfgProc {cmds} { global errorInfo lappend cmdlist "interface $intf" if [catch {cli_open} result] { lappend cmdlist "no auto discovery qos" error $result $errorInfo CLICfgProc $cmdlist } else { array set cli1 $result } if [catch {cli_exec $cli1(fd) "enable"} result] { error $result $errorInfo } if [catch {cli_exec $cli1(fd) "config t"} result] { error $result $errorInfo } foreach a_cmd $cmds { if [catch {cli_exec $cli1(fd) $a_cmd} result] { error $result $errorInfo } else { set cmd_output $result } } if [catch {cli_exec $cli1(fd) "end"} result] { error $result $errorInfo } if [catch {cli_exec $cli1(fd) "write mem"} result] { error $result $errorInfo } if [catch {cli_close $cli1(fd) $cli1(tty_id)} result] { error $result $errorInfo } } BRKDEV-1191

Note “enable” command

14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

36

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

18

Context Save/Retrieve Example
# Globals are saved in context global bw_counter # #First time through, we don't have a context, so catch that error # if [catch {context_retrieve ctx1 bw_counter} bw_counter] { set bw_counter 0 } Do some stuff if { $bool_val } { incr bw_counter # # Code for sending a syslog message would go here.... # context_save ctx1 bw_counter return }

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

37

File I/O Example
tenet#sh run | inc ofilename event manager environment _ofilename disk2:/eem/output.dat tenet#

# # write to a file # # if [file exists $_ofilename] { puts "file $_ofilename being overwritten" } set myfileid [open $_ofilename w+] foreach outs $cliout { puts $myfileid $outs } close $myfileid
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

38

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

19

Application Event Example
Pol icy A # Reg te fo an app ica ion event fo sub_sys is r r l t r tem ID 798 and type 10 event_reg te is r_app sub_sys l tem 798 type 10 # Query the eventi nfo ar ray se a r_e fo [ t r in event_req fo in ] i {$_cer != 0 { f rno } se resu t[ rmat "component=%s; subsys er t l fo r=%s; pos x er i r=%s; \n%s" \ $_cer r_sub_num $_cer r_sub_er $_cer r r_pos ix_er $_cer t ] r r_s r e ro $ r r resu t l } # log a message set msg [ rmat "app ica ion event a fo l t : rg1: %s arg2 %s arg3 %s arg4 %s" \ $ar in (da a1) $ar in (da r_e fo t r_e fo ta2) $ar in (da r_e fo ta3) $ar in (da r_e fo ta4) ] act ion_sys log pr rt in msg $msg io i y fo Pol icy B # Reg te fo a t is r r imer event tha tiggers once every 60 secs tr # pub ish an event wi sub_sys l th tem ID o 798 and type 10 f event_reg te t is r_ imer watchdog name tmer1 t i ime 60 .0 event_pub ish sub_sys l tem 798 type 10 a “ isi 1 a rg1 th s ” rg2 “ isi 2 th s ” a “ isi 3 a rg3 th s ” rg4 “ isi 4 th s ” act ion_sys log pr rt in msg “event t io i y fo ype 10 pub ished” l
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

39

EEM Feature/Product Support Matrix
5/5/08 8:55 AM Legend Shipping In EFT EC Planning N/A

CISCO ACCESS ROUTERS EEM Version
1.0 2.0 2.1 2.1.5 2.2 2.3 2.4 3.0

Cisco 800 Series

Cisco 1800 Series
12.3(11)T 12.3(14)T1

Cisco 2800 Series
12.3(11)T 12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T 12.4(pi9)T

Cisco 3800 Series
12.3(11)T 12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T 12.4(pi9)T

Cisco 1700 Series
12.3(4)T 12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T Planning

Cisco 2600 Series
12.3(4)T 12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T Planning

Cisco 2600XM Series
12.3(4)T 12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T Planning

Cisco 2691 Series

Cisco 3600 Series
12.3(4)T

Cisco 3700 Series

12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T Planning

12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T Planning

12.3(14)T1 12.4(2)T 12.4(11)T 12.4(20)T Planning

12.4(2)T 12.4(11)T 12.4(20)T 12.4(pi9)T

12.4(2)T 12.4(11)T 12.4(20)T 12.4(pi9)T

CISCO 5000 SERIES & UP EEM Cisco 7200 Version Series
1.0 2.0 2.1 2.1.5 2.2 2.3 2.4 3.0 12.4(2)T 12.4(11)T 12.4(20)T 12.4(pi9)T 12.3(14)T1

Cisco 7301

Cisco 7304
12.2(27)SBC

Cisco 7600 Series

Cisco 10000

Cisco 12000 Series
12.0(26)S

Cisco XR 12000
See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr Planning

Cisco CRS-1
See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr See IOS-XR Fault Mgr Planning

Cisco 7500 Series
12.0(26)S

Cisco 5000 Series

12.3(14)T1

12.2(28)SB

12.2(18)SXF5

12.2(28)SB

12.4M

12.4(2)T1 12.2(33)SB 12.2(SR) Eagle Planning 12.2(33)SB 12.2(SR) Eagle Planning 12.2(33)SRB 12.2(SR) Eagle Planning 12.2(33)SB 12.2SR Planning

12.4(11)T Planning Planning

CISCO CATALYST SWITCHES EEM Cisco 3750 Cisco 4500 Version Switches Switches
1.0 2.0 2.1 2.1.5 2.2 2.3 2.4 3.0 12.2(40)SE Winter 09 Planning 12.2(40)SE 12.2(44)SE Planning

Cisco 6500 Switches
IOS w/o Modularity 12.2(18)SXF5 w/ Modularity 12.2(18)SXF4

12.2(33)SXH

Includes futures, subject to change; no commitment implied
Cisco Public

12.2(33)SXI)
Halfdome

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

40

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

20

Embedded Event Manager Event Detectors

1.0
BRKDEV-1191 14596_04_2008_c2

2.0

2.1
Cisco Public

2.2

2.1+

2.4

* Modular IOS only
41

© 2008 Cisco Systems, Inc. All rights reserved.

What’s Coming in EEMv2.4?
Two new EEM Event Detectors
Event detectors are key pieces of infrastructure software that drive the event engine

EEM RPC ED
Remote procedure call event detector—allows for SOA-type interface where EEM policies can be invoked from outside the box

SNMP Proxy ED
Allows for an event to be generated when a specific trap is received at the router or switch—allows for local action based on external triggers from other equipment—A UPS system, for example, might tell a switch it is on battery back-up and local action could be taken by an EEM policy (shut interfaces, reroute traffic, gracefully shutdown modules)
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

42

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

21

What Else Is Coming in EEMv2.4?
Introduce three new capabilities:
Multiple event support Bytecode support Registration substitution enhancement

Introduce three new CLI commands:
Script policy refresh A new command to display the supported event detectors Add clear command to kill a Tcl script And other ease of use enhancements

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

What Other Improvements Are Coming in EEMv2.4?
Improve two Event Detectors:
Interface Counter ED—rate based trigger SNMP ED enhancement—delta value

ait, But w ore! ’s m There

When and Where?
Cisco IOS Embedded Event Manager v2.4 will be available with the Cisco IOS Software 12.2SXI—Whitney 2 release for the Cisco Catalyst 6500 Series switches, Cisco IOS Software Release 12.4(20)T for the Cisco ISRs, 7600 Series SR with ‘Eagle’ release, Already in 3K metro Ethernet 12.2(40)SE, more lower-end switch support to be announced (3k, 4k) Other releases will gain support as well—publish dates to be determined
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

44

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

22

Multiple Event Support
Provides multiple event correlation Trigger on user specified events that happen within specified time window Support for Boolean functions Event set triggers action or another unique event

New
Event Correlation Capabilities

Benefit
More flexibility, provides unique troubleshooting capability Goes beyond today’s one event per policy
BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

45

EEM v2.4 RPC Event Detector
Allows and outside entity to make an XML RPC request TO the router or switch and INVOKE a defined EEM policy or script Uses SOAP message format over SSHv2 transport Filters on interface and port for receipt of xml-rpc message and parameters Policy might handle cli, config, anything, and return formatted (XML) reply Opens door to more flexible solutions and automation driven from outside the box
SOAP over SSH

Define your own SOA Services
Application Web Server services

EEM

EEM

Make XML RPC calls to invoke EEM script from outside the box
46

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

23

Some Configuration Needed to Use the EEM RPC Event Detector
1 2 3 4
Need to configure for SSH, if you haven’t already Need to prepare your EEM policy Need to register the EEM policy for external invocation Need to create your external program interface

1 4
External Program
Secu re SS

2
EEM

re S S Secu

Hv2

3

1
Hv2

2
EEM

3

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

47

Create Your External Program Interface and Send an RPC Request
Use Perl of some other programming language to send the SOAP RPC request OpenSSH and XML can be used The request specifies the EEM policy— “script_name” that has been registered using the RPC event detector You can pass arguments, too Sample programs on the way
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

<?xml version="1.0"?> Request Syntax <SOAP:Envelope xmlns:SOAP="http://www.cisco.com/e em.xsd"> <SOAP:Body> <run_emscript> <script_name> name of script </script_name> <argc> argc value </argc> <arglist> <l> argv1 value </l> <l> argv2 value </l> … <l> argvn value </l> </arglist> </run_Eemscript> </SOAP:Body> </SOAP:Envelope> ]]>]]> Don’t forget “end of message”
48

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

24

The Response
The “output string” comes from your ‘onthe-router’ EEM policy You define it’s contents and structure Your program can use the result for whatever intent you have
<?xml version="1.0"?> Reply Syntax <SOAP:Envelope xmlns:SOAP="http://www.cisco.com/e em.xsd"> <SOAP:Body> <run_Eemscript_response> <return_code> rc </return_code> <output> output string </output> </run_eemscript_response> </SOAP:Body> </SOAP:Envelope>

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

EEM v2.4 SNMP Proxy Event Detector
Router or switch can RECEIVE an SNMP trap EEM event upon trap receipt Execute (trigger) EEM script to take local action Script sees varbind info Example:
UPS on battery backup Shut non-critical POE ports to conserve power Only 5 minutes remaining Shutdown service modules gracefully
SNMP alert

Yikes! Better shut interfaces, begin graceful shutdown procedures

EEM

EEM

Uninterruptible Power Supply

SNMP alert—on battery! 5 Minutes Remaining!
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

50

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

25

Script Policy Refresh
Network/Device-based script management Create central repositories and let the devices update from there “Pull” model rather than “push” model Using the power of EEM, update periodically, by command, or as directed
Please give me any script updates!
EEM Repo

EEM

EEM

EEM

EEM

EEM EEM
BRKDEV-1191 14596_04_2008_c2

EEM

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

51

Cisco IOS Embedded Event Manager v3.0
Theme: 3.0—EEM for Power Users This development will enhance the performance, increase feature integration, add new capabilities, and increase the flexibility so that EEM can be used in new and exciting ways Major new customer facing function:
Increased performance—up to 150 events per second* New Event Detectors Routing ED, Flexible NetFlow ED, IP SLA ED, Enhanced CLI Command ED User interface enhancements SNMP library extensions Get, Set and Notify for local and remote hosts Ease of use items … and More * Target to Be Verified and May Be Platform Specific
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

52

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

26

Cisco IOS Service Diagnostics

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

53

Cisco IOS Service Diagnostics
Designed to help solve diagnostics issues using an event-based embedded network management approach
- Focused on “service” domain experience - Scenario’s seeded by Cisco’s TAC for OSPF, QOS , BGP & Resource Diagnostics

Provide infrastructure to facilitate the deployment of customized diagnostics functionality Provide a set of scripts addressing specific diagnostics scenarios Scripts available for editing and use by customers on the Cisco Beyond – Product Extension Community
http://cisco.com/go/ciscobeyond
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

54

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

27

Service Diagnostics Key Features
9 scripts for OSPF, BGP and QoS 3 scripts for monitoring CPU, Memory, Buffer Ability to configure thresholds Ability to run scripts proactively and in reaction to network events A rich set of general reusable libraries Script management made easy with utility scripts Scripts are platform independent; can run on all Cisco devices supporting TCL, EEM Digitally Signed TCL scripts Email and Syslog notification of diagnostic results

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

55

Example: OSPF Diagnosis
Problem: OSPF neighbor state goes from FULL to DOWN Trigger: Syslog message indicating “OSPF neighbor state going from FULL to DOWN” triggers OSPF diagnostic script Action: Diagnostic script checks OSPF and other configuration to identify the root cause Problem could be a. b. c. d. e. Mismatch area ids Incorrectly configured passive interfaces Incorrectly configured “network” command Incorrectly configured interfaces (link status is DOWN) Blocking access lists

f.…. A detailed message with the problem root cause is sent out via email or a syslog message

How important it is to have a log of diagnostic activities in persistent storage?
Email / Syslog Ethernet 0/0

X

Ethernet 0/1
56

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

28

Service Diagnostics Benefits
Cost savings (Reduced MTTR) Increased network uptime Automatically identify the most common root causes for the most common failure scenarios related to BGP, OSPF, QoS Send automatic alerts on resource monitoring when configured thresholds are crossed Automatically collect additional context information that is relevant to diagnosing a problem, to accelerate problem resolution Infrastructure to customize and add additional diagnostics Enhanced programmable platform capabilities of Cisco IOS software

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

Cisco Beyond—Product Extension Community: EEM Scripting Community
Now on Cisco.com /go/ciscobeyond Open source scripts, share, upload, download, learn by example Categories include: Ntwk mgmt, routing, QoS, High availability, User interface, etc Comments, ratings, community managed forum RSS feed notification Script URL http://cisco.com/go/ciscobeyond
http://forums.cisco.com/eforum/servlet/EEM?page=main

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

58

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

29

Cisco IOS Embedded Menu Manager

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

Embedded Menu Manager (EMM)
Programmable menu interface built on top of tclsh Uses an XML-based Menu Definition File (MDF) to generate menus Useful to provide a custom menu-driven application for operators Available in Cisco IOS version 12.4(20)T Ready-to-use menus available for Cisco IOS Service Diagnostics scripts on the Cisco Beyond – Product Extension Community

http://cisco.com/go/ciscobeyond

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

60

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

30

EMM Features
Menu item action can
Launch a optional wizard (one or more question/response pairs) Run an IOS exec-mode command Run an IOS config-mode command Run an embedded TCL script Launch a sub-menu

Query prompts can be static or dynamic User input can be
Free form (with optional maximum length) Discrete choices Range

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

EMM Sample Menu – Resource Diagnostics
====================================================================== Resource Diagnostics Enter ? for help or ?# for item help ---------------------------------------------------------------------1. Install Diagnostic Scripts 2. Set Global Variables (email parameters) 3. Deploy CPU Diagnostic Script 4. Deploy Memory Diagnostic Script 5. Deploy Buffer Diagnostic Script 6. Display Diagnostic Policy Configuration 7. Remove Diagnostic Policies 8. Exit Enter selection [8]:

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

62

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

31

Embedded Menu Manager Benefits
XML MDF file very flexible and file based
Definitions can be centrally stored on network servers Menu elements can be made more dynamic with Tcl

Built-in customizable context-sensitive help Wizard mode
Steps users through menu application

Built-in input validation Ability to record and play back menu sessions

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

Embedded Diagnostics and Management: Summary and Benefits
Hierarchical Management
Right data, right analysis, right place

Security
Fault and Network Correlation Mgmt Network

Performance and Capacity Planning

Faster reaction time
Local action with notification Multi-level actions

Better (More Accurate) Data
Node perspective vs.. inference by symptom Increased collection rates More scalable with distribution

Egress networks

Onboard Event Analysis and Intelligence
Ingress networks

For Highly Available, High Performance Computing
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

64

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

32

Q and A

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

65

Recommended Reading

Available Onsite at the Cisco Company Store
BRKDEV-1191 14596_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

66

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

33

More Recommendations

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

67

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

68

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

34

BRKDEV-1191 14596_04_2008_c2

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

69

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

35