Advanced Zenoss Training

//
Zenoss Advanced Administration Training
Copyright 2009 Zenoss, Inc. All rights reserved. This work is licensed under a Creative Commons Attribution Share Alike 3.0 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/; or send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California, 94105, USA.
// Section 1: Preparing the Environment

What this section will cover:
Connectivity Preparations Managed Device Preparation Preparation Testing
// Connectivity Preparation
From Zenoss Server to internet
Outbound http/s (80/443) to update zenoss packages Outbound ssh (22) if zenoss interactive support is desired
Connectivity and Topology

Inbound from infrastructure to zenoss server
Syslog (UDP 514), SNMP Traps (UDP 162)
Outbound from zenoss server to infrastructure

ICMP echo request inbound echo reply TCP ports for all monitored IPServices (smtp,http,etc) DCOM & RPC = Dynamic Port allocation SNMP (UDP 161) and SSH (TCP 22)
// Managed Device Preparation

Unix Servers
SNMP agent with community and ACL Net-SNMP recommended Syslog.conf that points back to Zenoss server (*.info @zenoss) Optional public key configured with Zenoss server Optional install of zenplugins for external monitoring
Network Devices (and other SNMP devices)

SNMP agent with community string and ACL Optional SNMP trap that points back to Zenoss server
For instance APC and other network devices
Optional syslog that points back to Zenoss server

Recommended for Cisco devices
// Managed Device Preparation

Windows Servers
Windows SNMP Agent installed with community and ACL Login account for WMI access (Domain or Local Administrator) Optional SNMP-Informant DLL (for CPU, Memory, Disk I/O)
http://www.wtcs.org/informant/download.htm
Dell Servers
Dell Open Manage installed
HP/Compaq Servers
Insight Manager installed
// Preparation Testing
Testing SNMP connections from Zenoss server
snmpwalk -v1 -c COMMUNITY HOST ip Should return listing of IP addresses
Testing WMI from a windows System

Run wbemtest Click Connect In Namespace field enter \\HOST\root\cimv2 Enter login information in User and Password fields Click Query button Enter select * from win32_service You should get a dialog with list of services on box
// Section 2: Administration
Detailed Architecture (Review) Controlling Zenoss Daemons Minimal Zenoss for Debugging Backup and Restore Troubleshooting Daemons Escalating Issues and Contacting Support
Detailed Architecture
//
// Controlling Zenoss Daemons

status - check the run state of a daemon start - start daemon in background stop - stop a running daemon restart - reload a daemons configuration run - run the daemon in the foreground for debugging help - list all daemon options
Option list valid for configuration file
// Minimal Zenoss for Debugging
ZEO
Zope
MySQL
10
// Backup and Restore
ZenRRD
zenbackup
ZEO
MySQL
backup_19680101.tgz
11
// Escalating Issues and Contacting Support

When escalating an issue please provide:
Steps to reproduce the problem Look for stack traces Gather version information (Zenoss, OS, etc) Escalate the Problem
Email: support@zenoss.com AIM: zenosssupport Phone: 410-990-0274
12
// Administration Activities
13
// Section 3: Inventory and Configuration

Modeling Daemons (Review) Modeling Process Controlling Modeling Plugin Selection Controlling SNMP Connections Inventory and Configuration Activities
14
// Modeling Daemons (Review)
ZenDisc Auto-Discovery
ZenModel
Unified Configuration ZEO Model
ICMP
ZenModeler Detailed Modeling
SNMP/SSH/Telnet
ZenWinModeler Windows Modeling
WMI
15
// Modeling Process
ZenModeler
Collector Plug-ins
ZenModel
Track Changes
interface
service Transports
disk
Normalize Data
SNMP
SSH
Telnet
WMI
Collect Data
80
Web server apache zope eth0
Database server mysql eth0 /dev/sda /dev/sdb
3601
8080
switch
1 2 3 4 5 6 7 8
16
// Controlling Modeling
Plugin Names
zenoss.snmp.InterfaceMap, zenoss.cmd.ifconfig_an Stored in $ZENHOME/Products/DataCollector/plugins
Plugin Selection
zCollectorCollectPlugins - Regular expression that explicitly selects plugins to be run. zCollectorIgnorePlugins - Regular expression that excludes a plugin zTransportPreference - if two plugins perform the same function which one should be used
17
// Controlling SNMP Connections

zSnmpCommunities
List of SNMP Communities to try when first discovering a device
zSnmpCommunity
Community that will be used to connect to a device
zSnmpPort
Port to use when connecting to a device
zSnmpTimeout
Timeout in seconds before retransmit of an SNMP command. Should never be over 15 seconds
zSnmpTries
Number of times to retry a command before marking it as failed. On a poor quality link this value should be increased. High end is 4 - 5
zSnmpVer
SNMP version v1 or v2c. V2c must be used if 64-bit counters are being collected. Also v2c allows for bulk walks which improves modeling performance significantly
18
// Inventory and Configuration Activities
19
// Section 4: TALES Expressions

What is a TALES expression Where are they Used Using zLinks Custom User Commands TALES Activities
20
// What is a TALES Expression

A string substitution mechanism Evaluated in the context of an object Common variables
here - object on which the expression is evaluated evt - object pointing to the current event dev - object pointing to the current device
Syntax for substitution

${here/id} when evaluated on a device will be the device name ${here/getHWProductName} can call attribute or method ${here/cMyCustomField} or a custom defined field
21
// Where are they Used

User Interface Commands
Evaluated in device context e.g. here == device
zLinks
Any HTML markup evaluated in device context
Performance Templates
Command field evaluated in device context
Event Commands
Evaluated in an event context with device evt == the event dev == the device
22
// Using zLinks
zProperty like any other zProperty in /Devices Uses TALES expressions Any HTML text can be added Used for many things
Create link to external management system Insert graph on to front page of device
23
// Custom User Commands

Commands that are run by user through web UI Defined on any level of DMD hierarchy
Highest level in /Settings/Manage
Uses TALES expressions
24
// TALES Activities
25
// Section 5: Event Management

Event Architecture Begin / End Correlation (Review) Classification Process Event Class Mapping Event Commands SNMP Traps and Transformations Event Activities
26
// Event Architecture
Ajax Web Console

Event List
Process Events ZenSyslog ZenTrap

Syslog
SNMP Traps
ZenActions
Email and Pager Alerts Command Execution Event Aging
Events DB MySQL
ZenEventlog
WMI
27
// Begin/End Correlation
Ping Down
Severity = Critical Class = /Status/Ping Device = build.zenoss.loc

Clear
Status
Severity = Critical Class = /Status/Ping Device = build.zenoss.loc Severity = Clear Class = /Status/Ping Device = build.zenoss.loc History
Ping Up
28
// Classification Process
New Event
Parse Event
Tokenize Syslog
Find Class
Key + Regex or Expression
Apply Context
Severity = Critical
Insert Event
Status
29
// Event Class Mapping
Parsed Event
eventClassKey
Class List Class List Class Maps Regex or Test
Transform
Apply Context
30
// Event Commands
Event Command
Where: class = /Status/Ping
Ping Down
Command: echo ${evt/summary} Ping Up
echo device 1.2.3.4 is down
Clear Command: echo ${evt/summary}
echo device 1.2.3.4 is up
31
// SNMP Traps and Transformations

PowerNet-MIB DEFINITIONS ::= BEGIN
DMD OID Index

upsAdvBatteryCapacity -> .1.3.6.1.4.1.318.1.3.1.0 upsAdvBatteryTemperature -> .1.3.6.1.4.1.318.1.6.1.0 . . .
apc OBJECT IDENTIFIER ::= { enterprises 318 } products OBJECT IDENTIFIER ::= { apc 1 } apcmgmt OBJECT IDENTIFIER ::= { apc 2 }
zenmib load
upsAdvBatteryCapacity OBJECT-TYPE SYNTAX Gauge ACCESS read-only STATUS mandatory DESCRIPTION "The remaining battery capacity expressed in percent of full capacity." ::= { upsAdvBattery 1 }
SNMP Trap
ZenTrap
Resolve
Classify & Transform
Event Database
32
// Event Monitoring Activities
33
// Section 6: ZenCommands
ZenCommand Templates Running a ZenCommand ZenCommand API Using ZenPlugins
34
// ZenCommand Templates
Data Source
Type = COMMAND Component = unique name on device Event Class = /Cmd/Fail SSH if desired
Data Point
One per monitored output value Make sure that data type is correct
35
// Running ZenCommand
ZenCommand
exec /bin/df avail:34554
Remote Box
SSH
exec /bin/df avail:35654
36
// ZenPlugins
ZenCommand Plugins for Remote Monitoring Must be Installed on Remote Machine Executed over SSH Used for Devices
Outside the Firewall Without proper SNMP Agent
37
// ZenCommand Activities
38
// Section 7: SNMP Performance Monitoring

What is SNMP Performance Daemons Template Binding (Review) Templates and Inheritance SNMP Templates Two SNMP Performance Collectors SNMP Performance Monitoring Activities
39
// What is SNMP?
Simple Network Management Protocol
As in light weight not easy to understand :)
First Standard Management Protocol Widely implemented on network devices Servers implemented primarily at OS layer
Windows - Pretty OS support. Missing CPU, Memory, Disk I/O Net-SNMP - Good OS support. Varies on platform. Native OS agents vary widely.
40
// SNMP Tree
A tree structure like DNS Described by . delimited number sequence Each number in the OID has a corresponding name Schema defined in Management Information Base (MIB) Naming Conventions
RAW OID - .1.3.6.1.2.2.2.1.2 Full Name - iso.org.dod.internet.mib-2.interfaces.ifTable.ifEntry.ifDescr Short hand - SNMPv2-MIB::ifDescr
Worthless numbers -> iso(1) org(3) dod(6) internet(1)

41
// SNMP Single Values and Tables

Single Values
OID points directly to a value Usually, but not always, ends with .0 Zero must be added SNMPv2-MIB::sysDescr.0
Tables
OID points to a table or table column OID ends with the row index Row indexes are often single numbers Row indexes are sometimes a series of numbers
An IP Address - RFC1213-MIB::ipAdEntAddr.172.16.56.1 An ASCII String - SNMPv2-SMI::enterprises.9600.1.1.5.1.2.6.95.84.111.116.97.108
42
// SNMP Tree
mib-2(1)
system(1)
interfaces(2)
sysDescr.0 = linux
sysUpTime.0 = 3 days
ifTable(2)
ifEntry(1) ifDescr.1 = eth0 ifDescr.2 = eth1

43
// Important Parts of the Tree

Standard MIB-2
Things like IP Addresses, Interfaces, Routes, and Services
Host Resources
Things like Hardware, File Systems, Processes, and Software
Enterprises
Values defined by companies or other non-standard MIBs Company numbers assigned by IANA
http://www.iana.org/assignments/enterprise-numbers
Common Numbers
IBM = 2, Cisco = 9, HP = 11, Dell = 674, Net-SNMP = 2021 SNMP-Informant = 9600, Zenoss = 14296
44
// SNMP Commands
SNMP Get
Pull a single value from an agent Ends with index which often 0 for single item or row for table
SNMP Walk
Steps through the values published by an agent Used to investigate or trouble shoot agent
SNMP Trap
Message sent to management station Has unique OID based identifier Has a series of key value pares
45
// SNMP Command Examples

Walk basic system MIB
snmpwalk -v1 -cpublic build.zenoss.loc system
Walk interface descriptions

snmpwalk -v1 -cpublic build.zenoss.loc ifDescr
Get a single value

snmpget -v1 -cpublic build.zenoss.loc ifDescr.2
Detailed description of OID value

snmptranslate -Td RFC1213-MIB::ifDescr
Convert name to raw OID

snmptranslate -On RFC1213-MIB::ifDescr
Convert raw OID to short name

snmptranslate -OS .1.3.6.1.2.1.2.2.1.2
46
// Performance Daemons
ZenPerfSNMP SNMP Collection

ZenModel ZenRRD
Unified Configuration RRDTool Model
SNMP
ZenCommand Nagios/Cacti
Exec / SSH
ZenPerfXMLRPC XML-RPC Collection
XML-RPC
47
// Template Binding (Review)

Templates
Device
Device: build
FileSystem
FileSystem: /
ethernetCsmacd
ethernetCsmacd: eth0
48
// Templates and Inheritance

/Devices
FileSystem
SNMP usedBlocks
Device
SNMP sysUpTime
/Server/Linux
build.zenoss.loc
Device
SNMP sysUpTime SNMP ssCpuRawSystem SNMP memAvailReal
tilde.zenoss.loc
/Server/Linux/Remote
dev.zenoss.org
FileSystem
SSH disk_usedBlocks
Device
SSH uptime_sysUpTime SSH cpu_ssCpuRawSystem SSH mem_memAvailReal
www.zenoss.com
49
// Performance Templates
Data Source
How to collect information: SNMP, COMMAND, XMLRPC
Data Point
Parse Data Source output and assign type information Custom rrd creation command allowed
Threshold
Min / Max applied to Data Points Threshold Event Information
Graph
Combine Data Points into a graph Many graph attributes can be selected If default format doesnt work custom RRD command can be used
50
// Two Performance Monitors

DMD
ZenPerfSNMP localhost zenperfsnmp.conf
win2003.zenoss.loc tilde.zenoss.loc
localhost
gate.zenoss.loc
bin/zenperfsnmp
remote1
build.zenoss.loc esx.zenoss.loc dev.zenoss.loc
ZenPerfSNMP localhost2 zenperfsnmp2.conf bin/zenperfsnmp2
51
// SNMP Monitoring Activities
52
// Section 8: Programming Zenoss

Customizing the User Interface External APIs Device API Using ZenDMD Creating ZenCommands Custom Reports
53
// Customizing the User Interface

Zenoss uses Zope Skins Zenoss pages are build using ZPT Skins are defined in file system
Can be viewed in /zport/portal_skins/manage
Any Zenoss ZPT file can be overridden

Find file in portal_skins Click Custom button Navigate to custom folder and make modifications
Images can be overridden in same way

54
// External Zenoss API

Zope publishes objects and their methods Objects are navigated to by paths like a file system
/zport/dmd/Devices/build.zenoss.loc
Methods called on objects

/zport/dmd/Devices/build.zenoss.loc/setLocation
Parameters are URL arguments

/build.zenoss.loc/setLocation?newlocation=/Maryland
Methods can also be called using XML-RPC
55
// Device API
Device Add
/zport/dmd/DeviceLoader?deviceName=build.zenoss.loc&devicePath=/Server
Device Edit
Need to have the path to the device /zport/dmd/Devices/Server/devices/build.zenoss.loc?setLocation=/Maryland
Device Delete
/zport/dmd/Devices/Server/devices/build.zenoss.loc/deleteDevice
56
// Using ZenDMD
$ zendmd Welcome to zenoss dmd command shell! use zhelp() to list commands >>> d = find(build.zenoss.loc) >>> d.os.interfaces.objectIds() ['eth0', 'eth1', 'lo', 'sit0', 'vmnet1', 'vmnet8'] >>> for d in dmd.Devices.getSubDevices(): >>> print d.id, d.getManageIp()
DMD
57
// Report Plugins
Build any report possible with information in DMD Consists of two parts
Python code to collect information ZPT to display information
Debug collection code using zendmd
58
// Creating ZenCommands
ZenCommands implement Nagios and Cacti APIs Input are command line arguments Output is exit code and one line of standard out Standard output line
Nagios = message|key=value[UOM];[warn];[crit];[min];[max] Cacti = key:value key2:value
Full API defined here

http://nagiosplug.sourceforge.net/developer-guidelines.html
59
// Programming Activities
60

Advanced Zenoss Training

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced Zenoss Training

Uploaded by

Copyright:

Available Formats

//

Zenoss Advanced Administration Training

// Section 1: Preparing the Environment

Connectivity and Topology

Outbound from zenoss server to infrastructure

// Managed Device Preparation

Network Devices (and other SNMP devices)

Optional syslog that points back to Zenoss server

// Managed Device Preparation

Testing WMI from a windows System

// Controlling Zenoss Daemons

// Minimal Zenoss for Debugging

// Backup and Restore

// Escalating Issues and Contacting Support

// Section 3: Inventory and Configuration

// Modeling Daemons (Review)

ZenModeler Detailed Modeling

ZenWinModeler Windows Modeling

Web server apache zope eth0

Database server mysql eth0 /dev/sda /dev/sdb

// Controlling SNMP Connections

// Inventory and Configuration Activities

// Section 4: TALES Expressions

// What is a TALES Expression

Syntax for substitution

// Where are they Used

// Custom User Commands

Uses TALES expressions

// Section 5: Event Management

Ajax Web Console

Process Events ZenSyslog ZenTrap

Severity = Critical Class = /Status/Ping Device = build.zenoss.loc

Key + Regex or Expression

// Event Class Mapping

Class List Class List Class Maps Regex or Test

Command: echo ${evt/summary} Ping Up

echo device 1.2.3.4 is down

Clear Command: echo ${evt/summary}

echo device 1.2.3.4 is up

// SNMP Traps and Transformations

DMD OID Index

Classify & Transform

// Event Monitoring Activities

exec /bin/df avail:35654

// Section 7: SNMP Performance Monitoring

Worthless numbers -> iso(1) org(3) dod(6) internet(1)

// SNMP Single Values and Tables

ifEntry(1) ifDescr.1 = eth0 ifDescr.2 = eth1

// Important Parts of the Tree

// SNMP Command Examples

Walk interface descriptions

Get a single value

Detailed description of OID value

Convert name to raw OID

Convert raw OID to short name

ZenPerfSNMP SNMP Collection

ZenPerfXMLRPC XML-RPC Collection

// Template Binding (Review)

// Templates and Inheritance

// Two Performance Monitors

ZenPerfSNMP localhost2 zenperfsnmp2.conf bin/zenperfsnmp2

// SNMP Monitoring Activities

// SNMP Single Values and Tables