Zenoss Core4 Event Management Paper

Event Management
for Zenoss Core 4

January 2013
Jane Curry
Skills 1st Ltd
www.skills-1st.co.uk
JaneCurry
Skills1stLtd
2CedarChase
Taplow
Maidenhead
SL60EU
01628782565
jane.curry@skills1st.co.uk
www.skills1st.co.uk
Synopsis
ThispaperisintendedasanintermediateleveldiscussionoftheZenosseventsystemin
ZenossCore4.TheeventarchitecturehaschangeddramaticallyinZenoss4from
previousversions.
ItisassumedthatthereaderisalreadyfamiliarwiththeZenossEventConsoleand
withbasicnavigationaroundtheZenossGraphicalUserInterface(GUI).Itlooksin
somedetailatthearchitecturebehindtheZenosseventsystemthedaemonsandhow
theyareinterrelatedanditlooksatthestructureofaZenosseventandtheeventlife
cycle.
ZenosscanreceiveeventsfrommanysourcesinadditiontoZenossitself.Eventsfrom
Windows,UnixsyslogsandSimpleNetworksManagementProtocol(SNMP)TRAPsare
allexaminedindetail.
TheprocessbywhichanincomingeventisconvertedintoaparticularZenosseventis
knownaseventmappingandthereareanumberofdifferentpossibletechniquesfor
performingthatconversion.Thesewillallbeexploredalongwiththecreationofnew
eventclasses.
Onceaneventhasbeenreceived,classifiedandstoredbyZenoss,automationmaybe
required.Alertingtousersbyemailandpageisdiscussed,asarebackgroundactionsto
runcommandsorgenerateTRAPs.
LogginganddebuggingtechniquesarediscussedinsomedetailsasistheJSONAPIfor
extractingdataoutofZenoss.
ThispaperwaswrittenusingZenossCore4.2.3
ThepaperisacompaniontexttotheZenoss4EventManagementWorkshop.
Notations
Throughoutthispaper,texttobytyped,filenamesandmenuoptionstobeselected,are
highlightedbyitalics;importantpointstotakenoteofareshowninbold.
Pointsofparticularnotearehighlightedbyanicon.
EventManagementforZenossCore4Skills1stLtd
1February2013
Table of Contents
1Introduction..........................................................................................................................6
2Zenosseventarchitecture....................................................................................................6
2.1EventConsole...............................................................................................................6
2.2EventManagersettings.............................................................................................10
2.3Eventdatabasetables...............................................................................................11
2.3.1Zenoss2.xand3.x...............................................................................................11
2.3.2Zenoss4................................................................................................................14
2.4Neweventdaemons....................................................................................................20
2.4.1RabbitMQ.............................................................................................................20
2.4.2zeneventserver.....................................................................................................22
2.4.3zeneventd.............................................................................................................22
2.4.4zenactiond...........................................................................................................23
2.4.5memcached...........................................................................................................23
2.5OtherdatabaserelatedchangesinZenoss4............................................................24
2.6Eventlifecycle............................................................................................................25
2.6.1Eventgeneration.................................................................................................27
2.6.2Applicationofdevicecontext..............................................................................29
2.6.3Eventclassmapping...........................................................................................29
2.6.4Applicationofeventcontext...............................................................................30
2.6.5Eventtransforms.................................................................................................30
2.6.6Databaseinsertionsanddeduplication............................................................31
2.6.7Resolution............................................................................................................32
2.6.8Ageingandarchiving..........................................................................................34
3EventsgeneratedbyZenoss..............................................................................................34
3.1zenping........................................................................................................................35
3.2zenstatus.....................................................................................................................36
3.3zenprocess...................................................................................................................36
3.4zenwin.........................................................................................................................37
3.5zenwinperf...................................................................................................................37
3.6zenperfsnmp................................................................................................................37
3.7zencommand...............................................................................................................38
4Syslogevents......................................................................................................................38
4.1Configuringsyslog.conf.............................................................................................39
4.2Zenossprocessingofsyslogmessages.......................................................................40
5ZenossprocessingofWindowseventlogs.........................................................................48
5.1ManagementusingtheWMIprotocol.......................................................................48
5.2ManagementofWindowssystemsusingsyslog.......................................................51
6EventMapping...................................................................................................................51
6.1Workingwitheventclassesandeventmappings....................................................52
6.1.1Generatingtestevents........................................................................................54
6.2Regexineventmappings...........................................................................................55
1February2013
6.3Rulesineventmappings............................................................................................57
6.4Otherelementsofeventmappings...........................................................................58
7Eventtransforms...............................................................................................................58
7.1Differentwaystoapplytransforms...........................................................................59
7.2Understandingfieldsavailableforeventprocessing...............................................60
7.2.1EventProxies.......................................................................................................63
7.2.2EventDetails.......................................................................................................66
7.3Transformexamples...................................................................................................68
7.3.1CombininguserdefinedfieldsfromRegexwithtransform.............................68
7.3.2Applyingeventanddevicecontextinrelationtotransforms..........................69
8Testinganddebuggingaids..............................................................................................71
8.1Logfiles.......................................................................................................................71
8.1.1zeneventd.log.......................................................................................................71
8.1.2zeneventserver.log...............................................................................................72
8.1.3Otherlogfiles......................................................................................................75
8.2UsingzendmdtorunPythoncommands..................................................................75
8.2.1ReferencinganexistingZenosseventforuseinzendmd.................................75
8.2.2UsingzendmdtounderstandattributesforanEventSummaryProxy...........79
8.3UsingthePythondebuggerintransforms................................................................83
9ZenossandSNMP..............................................................................................................87
9.1SNMPintroduction.....................................................................................................87
9.2SNMPonLinuxsystems............................................................................................88
9.3ZenossSNMParchitecture........................................................................................91
9.3.1Thezentrapdaemon............................................................................................91
9.4InterpretingMIBs......................................................................................................93
9.4.1zenmibexample...................................................................................................94
9.4.2AfewcommentsonimportingMIBswithZenoss.............................................99
9.5TheMIBBrowserZenPack......................................................................................100
9.5.1ModifyingZenossCore4.2tomaketheMIBBrowserZenPackwork..........102
9.6MappingSNMPevents............................................................................................103
9.6.1SNMPeventmappingexample........................................................................103
10EventTriggersandNotifications.................................................................................108
10.1ZenosspriortoV4...................................................................................................108
10.2Zenoss4architecture.............................................................................................109
10.3Triggers...................................................................................................................110
10.4Notifications............................................................................................................111
10.4.1emailNotifications..........................................................................................113
10.4.2PageNotifications...........................................................................................118
10.4.3CommandNotifications..................................................................................118
10.4.4TRAPNotifications.........................................................................................120
10.5NotificationSchedules............................................................................................122
10.6Usingzenactiond.log..............................................................................................123
10.7TheeffectofdeviceProductionState....................................................................125
11AccessingeventswiththeJSONAPI...........................................................................126
4
1February2013
11.1Definitions...............................................................................................................126
11.2UnderstandingtheJSONAPI...............................................................................127
11.3UsingtheJSONAPI..............................................................................................130
11.3.1Bashexamples.................................................................................................130
11.3.2Pythonexamples.............................................................................................134
12Conclusions.....................................................................................................................139
13AppendixA.....................................................................................................................143
13.1getevents.py............................................................................................................143
13.2zensendevent..........................................................................................................148
14References.......................................................................................................................152
1February2013
1 Introduction
ZenossisanOpenSource,multifunctionsystemsandnetworkmanagementtool.There
isafree,Coreoffering(whichhasmostthingsyouneed),andachargeableoffering,
ZenossResourceManager,whichhasextraaddongoodiessuchashighavailability
configurations,distributedmanagementservers,servicemanagementandevent
correlation;italsoincludesasupportcontract.
Zenossoffersconfigurationdiscovery,includinglayer3topologymaps,availability
monitoring,problemmanagementandperformancemanagement.Itisdesignedaround
theITILconceptofaConfigurationManagementDatabase(CMDB),theZenoss
StandardModel.ZenossisbuiltusingthePythonbasedZopewebapplicationserver
andusestheobjectorientedZopeObjectDatabase(ZODB)astheCMDB,usedtostore
Pythonobjectsandtheirstates.Zenoss3usedZEO,asalayerbetweenZopeandthe
ZODB;inZenoss4theZODBdataisstoredinaMySQLdatabase.
TherelationalMySQLdatabaseisalsousedtoholdcurrentandhistoricalevents.
PerformancedataisheldinRoundRobinDatabase(RRD)files.
ThedefaultprotocolsformonitoringaretypicallyagentlesstheSimpleNetwork
Managementprotocol(SNMP),WindowsManagementInstrumentation(WMI)and
collectingeventsfromsyslogs.Itisalsopossibletomonitordevicesusingtelnet,sshand
touseNagiosplugins.
Zenossprovidesdocumentationat
http://community.zenoss.org/community/documentation.Thereisalsoawealthof
informationontheZenosswebsiteinvariousforums,FAQs,andtheWiki.Auseful
bookisavailablefromPACKTPublishing,ZenossCore3.xNetworkandSystem
MonitoringbyMichaelBadger,whichprovidesmuchofthesameinformationasthe
ZenossAdministrationGuidebutinamuchclearerformatwithplentyofscreenshots.
AlthoughthisisaZenoss3text,itstillprovidesgoodbasicinformation.
ThispaperisanattempttoexpandontheeventinformationintheZenossCore4
AdministrationGuidebydrawingonmyownexperienceandthecollectedwisdomof
severalZenossemployeesandcontributorsfromthecommunity.
2 Zenoss event architecture

2.1 Event Console
WhenaneventarrivesatZenoss,itisparsed,associatedwithaneventclassification
andthentypically(butnotalways),itisinsertedintotheevent_summarytableofthe
zenoss_zepdatabase.EventscanthenbeviewedbyusersusingtheEventConsoleof
theZenossGraphicalUserInterface(GUI).
1February2013
ThereareanumberwaystoaccesstheEventConsole.ThemainEventConsoleis
reachedfromthetopEVENTS>EventConsolemenu.Thedefaultistoshowevents
withaseverityofInfoorhigher,sortedfirstbyseverityandthenbytime(mostrecent
first).Eventsareassigneddifferentseverities:
Name
Number
Colour
Critical
Red
Error
Orange
Warning
Yellow
Info
Blue
Debug
Grey
Cleared
Green
AlleventsalsohaveaneventStatefield.Zenoss3eventStatehadthreepossiblevalues
New,AcknowledgedandSuppressed.Zenoss4hasenhancedthesedefinitionssowe
nowhave:
Name
Number
Description
New
Neweventnoprevioussimilarevent
Acknowledged
Acknowledgedbyuserorrule
Suppressed
Typicallyfrombeyondasinglepointof
failure
Closed
Closedbyauser
Cleared
Closedbyarule
Dropped
Discardednotsavedinthedatabase
Aged
Autoclosedduetoage/severity
NotethatClosed,ClearedandAgedeventsallhavethesamestatusiconintheEvent
Console.
Bydefault,NewandAcknowledgedeventsareshownintheEventConsole.Anyevent
whichhasbeenAcknowledgedhasatickinitsstatuscolumn.ASuppressedeventis
notshownbydefaultbutcanbefilteredinifdesired;ithasasnowflakeicon.Zenoss
buildsaninternaltopologyofthenetworkitismanaging(usingnmap).Ifaneventis
receivedforadevicethatthetopologymapknowsisunreachable,theeventis
automaticallysuppressed.ThusZenosshasabuiltinmechanismforpinpointingfailure
devicesandsuppressingthefloodofeventsfrombehindsuchfailurepoints.
Eventscanbesortedbyclickingonadesiredcolumnheader;clickingagainsortsinthe
reverseorder.Tochangetheorderofcolumns,simplydragacolumnheader.
1February2013
Thereisafilterboxaboveeachcolumnheadertohelpselectrelevantevents.Most
filtersareamatchforapartialtextstring(youdon'tneedtosupplywildcards).Date
fieldsprovideacalendaricontoselectanearliestdate.Thecountfieldpermitsyouto
enterarange,forexampletoshoweventswithcount>10,use10:(ifyoutype
somethingillegalinthecountfilteritwillsupplyhelpfortherequiredsyntax).
Toselectfieldstodisplay,hoverthemouseattheendofaheadertoseethedownarrow
forsorting;thethirdoptiononthedropdownmenuistoconfigurethefieldstodisplay.
Figure1:ZenossEventConsole
FromtheEventConsole,oneormoreeventscanbeselectedbyclickingonthelinebe
carefulnottoclicksomethingthatisalink(likethedevicenameoreventclass).The
iconsatthetopleftcanbeusedtoAcknowledge,Close,MaptoanEventClass,
UnacknowledgeorReOpen.The+iconattheendofthisrowoficonscanbeusedto
generatetestevents.
Doubleclickaneventtoshowthedetailsofanevent.Thisshowsbothstandardfields
andanyuserdefinedfieldsorganisedunderseveralgroupingswhichcanbeexpanded
andcontracted.AnyAcknowledge,CloseorReOpenwillbeshownatthebottom,
includingwhoperformedtheaction.Freeformnotescanalsobeloggedhere.
1February2013
Figure2:EventdetailsshowingAcknowledgementandaddednote
Thesummaryandmessagefieldsarefreeformtextfields.Thesummaryfieldallowsup
to255characters;themessagefieldallowsupto4096characters.Thesefieldsusually
containsimilardata.Fordetailsofotherfields,seesection7.1.2oftheZenossCore4
Administrationguide.
Bydefault,theEventConsoleisrefreshedeveryminute.Thedropdownbesidethe
Refreshbuttonallowsyoutochangetheintervalortorefreshmanually.
1February2013
EventConsolesarealsoavailableatvariousplacesintheGUIwhichhavefilters
alreadyapplied:
Fromadevice'sdetailpage,selectEventsinthelefthandmenu
Foradeviceclass,clicktheDETAILSlinkandthenEventsinthelefthand
menu
ForaLocation,GrouporSystem,clicktheDETAILSlinkandthenEventsin
thelefthandmenu
FromanEventClass,selectEventsinthelefthandmenu
PriortoV4,ZenosseventswereeitherOpenorClosed.Openeventswerestoredin
theMySQLeventsdatabaseinthestatustable.Whenaneventwasclosed,itwas
movedtothehistorytableoftheeventsdatabase.
WithZenoss4thereisasignificantchange.TheMySQLdatabaseforeventsiscalled
zenoss_zepandithasfarmoretables,includingevent_summaryand
event_archive.Openeventswillbestoredintheevents_summarytable.Beaware
thattheevents_summarytablewillalsoholdclosed,clearedandagedeventsthis
catchesoutmanypeoplemigratingfromolderversionsofZenosstoZenoss4.Checkthe
StatusfilterintheEventConsoletoshowClosed,ClearedandAgedevents(theyall
havethesamestatusicon).Closed,ClearedandAgedeventsmaybeautomatically
movedtotheevent_archivetablebasedonage(after3days,bydefault).
2.2 Event Manager settings

FromtheADVANCED>Settingsmenu,chooseEventsinthelefthandmenutosetup
variousparametersthatcontroltheeventssubsystem,includinghoweventsareaged
andfinallypurged.
Figure3onpage11showslargelydefaultsettings.EventsofseverityWarningand
belowwillbeAgedafter240minutes(4hours).After4320minutes(3days)eventswith
statusofClosed,ClearedorAgedwillbeArchived(movedtotheevents_archivetable).
After7daysArchivedeventswillbedeletedentirely(notethislastsettingis90daysby
defaultandcanresultinaverylargedatabase).
Seechapter7oftheZenossCore4AdministratorsGuideformoreinformation.
10
1February2013
Figure3:EventManagerparametersforageingandarchiving
2.3 Event database tables

2.3.1 Zenoss 2.x and 3.x
Theeventsarchitecturewasthesameforversions2and3andwasrelativelysimple.
Eventsweregeneratedfromsomewhere.Thezenhubdaemonprocessedthemand
usuallythensavedthemeitherinthestatustableoftheMySQLeventsdatabaseor
couldsendthemtothehistorytable.
Thedatabasefieldsofthestatusandhistorytablesmatchedthedetailsseeninan
EventConsoleandifyouwroterulesandtransformstoprocessevents,theywerebased
onthesesamefieldnames.
TheeventsdatabaseiscreatedautomaticallywhenZenossisinstalledandcantypically
beaccessedbythezenossuserwithapasswordofzenossseeFigure4.
1February2013
11
Figure4:ZenosseventsdatabasepriortoZenoss4
TheformatofeachofthesetablesandthevalidfieldsforaZenosseventcanbeseenby
examiningtheZenossdatabasesetupfilein
$ZENHOME/Products/ZenEvents/db/zenevents.sql,where$ZENHOMEwillbe
/opt/zenossforaCore4.2ZenossonRedHat/CentOS(theonlycurrentlysupported
platform).
12
1February2013
Figure5:Definitionofstatuseventfieldsinzenevents.sqlpriortoZenoss4
zenevents.sqlalsodefinesthehistorytableinasimilarfashion.
Afurtherfourtablesaredefinedforheartbeat,alert_state,loganddetail.Thedetail
tablecanbeusedtoextendthedefaulteventfieldstoincludeanyinformationthatthe
Zenossadministratorrequiresforanevent.
1February2013
13
Figure6:zenevents.sqlshowingheartbeat,alert_state,loganddetailtableszenoss2and3only
IfyouareusingZenosspriortoversion4,gettheolderversionofthisZenossEvent
Managementpaperfromhttp://www.skills
1st.co.uk/papers/jane/zenoss_event_management_paper.pdf.
2.3.2 Zenoss 4
WithZenoss4eventsarestillheldinaMySQLdatabasewhichisnowcalled
zenoss_zepanditiscreatedwhenZenossisinstalled.Aswithearlierversions,the
zenossusercanaccessthisdatabasewithapasswordofzenoss.
NotethatwithZenoss4.2.3,ifinstalledwiththecoreautodeployscript,thenthe
passwordfortheMySQLzenossuserischangedtoarobust,randompasswordthatis
thensavedin$ZENHOME/etc/global.conf.Permissionsfor$ZENHOME/etcandits
contentsareallsettofullaccessforthezenossuserandnoaccessforanyoneelse.
14
1February2013
Figure7:AccessingMySQLdatabaseswithZenoss4
Inpassing,notethatinadditiontothezenoss_zepdatabase,theirisalsoazodbanda
zodb_sessiondatabase.TheZopedatabase(ZODB)thatstoresalltheobjects(devices,
deviceclasses,processes,networks,etc)isnowinMySQL.
Examiningthetablesofthezenoss_zepdatabaseiswherethingsdivergesignificantly
frompreviousversions.
1February2013
15
Figure8:TablesintheZenoss4zenoss_zep
database
Themaintablesarenowevent_summaryandevent_archivebutthestructureis
morecomplicated.Someofthedataisheldinseparatetableswithpointerstothem
fromthemaintables.Theseinclude:
16
agent
event_class
event_class_key
event_group
event_key
monitor
1February2013
Thedetailsoftheevent_summarytableisshownbelow.Theeventarchivetableisvery
similarwithjustthetwofingerprint_hashfieldsomitted.
Figure9:Fieldsintheevent_summarytableinZenoss4
1February2013
17
Theeagleeyedwillalsospotthatsomeofthefieldnameshavechangedfromthosein
Figure5.eventClassintheoldversionbecomesevent_classinV4;firstTimeinFigure5
becomesfirst_seeninthelaterversionandthereareanumberofothersimilar,subtle
changes.
Asmentionedabove,someofthedataisheldinseparatetablessoagent_id,
event_class_id,event_class_key_id,event_group_id,event_key_idandmonitor_keyare
linkstoseparatetableswiththecorrespondingdata.
Somedatahaschangedfairlysubtly:
Old

New
evid
uuid
eventState
status_id
eventClassMapping
event_class_mapping_uuid
severity
severity_id
stateChange
status_change
firstTime
first_seen
lastTime
last_seen
count
event_count
facility
syslog_facility
priority
syslog_priority
ntevid
nt_event_code
ownerid
current_user_uuid/current_user_name
clearid
clear_fingerprint_hash/cleared_by_event_uuid
Allreferencestothedevicehavechangedsignificantly.deviceisreplacedbythefour
fields,element_uuid,element_type_id,elementidentifierandelement_title
whilstthecomponentfieldisreplacedbyelement_sub_uuid,
element_sub_type_id,element_sub_identifierandelement_sub_title.
dedupidhasbecomefingerprintandfingerprint_hash.
OtherfieldswithdevicecontextsuchasprodState,DeviceClass,Location,Systems,
DeviceGroups,ipAddress,monitorandDevicePrioritywillnowbefoundfromthe
tags_jsonfield;theyarealsoavailableintheeventdetails.
PriortoZenoss4therewasaseparatelogtablewhoseroleisnowtakenbythe
notes_jsonfieldoftheevent_summarytable.
Eventdetailsratherthanbeinginaseparatetable,arenowreachedfromdetails_json.
update_timehasbeenaddedthelasttimeaneventwasupdated.
18
1February2013
suppid(whichwasneverused)hasdisappearedintheZenoss4schema.managerhas
alsodisappearedfromZenoss4.
Thesetablesarecreatedbythefilesin$ZENHOME/share/zeneventserver/sql/mysql.
Figure10:Partofthe001.sqlfilethatdefinesMySQLtablesinthezenoss_zepdatabaseforZenoss4
Someoftheseeventfieldsareparticularlypertinentdependingonhowtheeventwas
generated:
Syslogeventspopulatethefacilityandpriorityfields
Windowseventspopulatethentevidfield
SNMPTRAPspopulateatleastcommunityandoidfieldsintheeventdetail.
TheyalsousetheeventdetailtoprovideanyvariablespassedbyanSNMP
TRAP.
TheagentfielddenoteswhichZenossdaemongeneratedorprocessedthe
incomingevent;forexample,zentrap,zeneventlog,zenping.
1February2013
19
FundamentallyZenossadministratorsshouldnotbeaccessingthezenoss_zepdatabase
directly.Zenosshaveprovidedaninternaleventmappingsothat,largely,
administratorscancontinuetousethesameeventattributenamesashavebeenused
previously.Thiseventproxymappingwillbediscussedinmoredetaillater.In
general,thispaperwillusetheoldnamesunlessexplicitlystatedotherwise.
Ifyoudoneedtoaccesseventdatainthedatabasetables,perhapsforreportingon
events,itispossiblewiththeJSONAPI(alsomoreonthislater).
2.4 New event daemons

PriortoZenoss4mostoftheworkofprocessinganeventwasperformedbythezenhub
daemonwhichalsohaslotsofotherrolestofulfil.Eventprocessingcouldbecomea
severebottleneck.Zenoss4hasintroducedseveralnewsubsystemsanddaemonsto
dramaticallyimprovethethroughputofeventprocessing.
2.4.1 RabbitMQ
AMessageQueueingarchitecturehasbeenimplementedtospeedupprocessingandto
offeranAPIsothatZenossandotherapplicationproviderscaninteractwithevents.It
isalsousedbythenewJobarchitecture.ItusestheAdvancedMessageQueueing
Protocol(AMQP)standard,andtheopensourceRabbitMQimplementationin
particular,fortheeventpipeline.
WhenZenossisinstalledtheRabbitMQsubsystemisalsoinstalledandconfiguredwith
avhostofzenoss,userzenoss,passwordzenoss.Therabbitmqctlutilitycanprovide
informationaboutthestateoftheMQenvironment;notethatrabbitmqctlcommands
mustberunbytherootuser.
Figure11:Usingtherabbitmqctlutilitytoshowqueuesforthe/zenossvhost
Aneasywaytoseequeuesbuildingupistotemporarilystopzeneventdandthe
raweventsqueuewillthenbuildrapidly.
20
1February2013
rabbitmqctlonitsownorwithinsufficientargumentsprovidestheusagehelp.
rabbitmqctlreportgivesagoodoverallviewofthesubsystem.
IftheZenossserverisrenamedthenyoumustclearandrebuildqueuesbeforethe
zenhubandzenjobsdaemonswillrestart.Toresolvethis,issuethefollowing
commandsastherootuser(althoughanydataqueuedatrestarttimewillbelost):
export VHOST="/zenoss"
export USER="zenoss"
export PASS="zenoss"
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl add_vhost "$VHOST"
rabbitmqctl add_user "$USER" "$PASS"
rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'
Seesection14.8oftheZenossCore4AdministratorsGuideforthisinformation.
NotethatwithZenossCore4.2.3installedusingtheautodeployscript,orifthe
secure_zenoss.shscripthasbeenrunstandalone,thenthepasswordinthethirdline
abovewillhavebeenchanged.Examine$ZENHOME/etc/global.confforthe
amqppasswordandsubstituethatvalue,ratherthanusingzenossasthepassword.
ProvidedtheRabbitMQsubsystemisrunning,anymissingqueuewillautomaticallybe
recreatedwhenZenossisrestarted.
Tosimplyhavethequeuesrecreated,startasthezenossuser:
zenossstop
su(tobecomerootuser)
rabbitmqctldelete_vhost/zenoss
rabbitmqctladd_vhost/zenoss
rabbitmqctladd_userzenosszenoss#mightcreateanerror
zenossrabbitmqctlset_permissionsp/zenosszenoss'.*''.*''.*'
rabbitmqctllist_vhosts
(shouldhavezenossagain)
rabbitmqctlp/zenosslist_queues(shouldbenone)
exit
(backtozenossuser)
zenossstart
su
rabbitmqctlp/zenosslist_queues(shouldbeseveral)
Thereisafurtherscriptavailableatgist,writtenbycluther,toresetRabbitMQ
https://gist.github.com/4192854.
TwoutilitiesareavailableforthezenossusertogetRabbitMQinformation:
zenqdump <queue name>
dumpstheeventsinaqueue,convertingthebinaryblobs(whichishowtheeventsare
actuallystored)intohumanreadabletext.
Notethatthezenqdumputilityhasparametersforuserandpasswordfor
authentication,thatdefaulttozenoss/zenoss(youcanfindthiscodein
$ZENHOME/lib/python/zenoss/protocols/amqpconfig.py).InZenoss4.2.3,passwords
arelikelytohavebeenimprovedoninstallationsothesimplecommandshownabove
1February2013
21
willfail.Examine$ZENHOME/etc/global.conffortheparametersamqpuserand
amqppasswordandsupplythosevalues.Forexample:
zenqdumpuzenosspuy+680bEubHgdPow8Tfhzenoss.queues.zep.rawevents
Thezenqutilityhasthreedifferentoptionstomanageaqueue:
zenq count <queue name>
zenq purge <queue name>
zenq delete <queue name>
Thecountparametergivesacontinualoutputoftimestampandqueuelength.
Thepurgeparameterpurgeseventsfromaqueue.ThiscommandissafewhenZenossis
running.
ThedeleteparameterdeletesthequeueandshouldnotbeusedwhenZenossisrunning.
zenqdoesnothaveauthenticationparameters.
2.4.2 zeneventserver
AnewJavadaemon,zeneventserver(alsoknownaszep),hasbeencreated.Itsroleisto
presenteventstotheuserinterfaceandotherclients,andtomanagetheflowofdata
betweentheRabbitMQqueuesandtheMySQLdatabase.Dataispresentedtoclients
viaJSONcalls.
2.4.3 zeneventd
zeneventdisanewPythondaemonwhoseresponsibilityistotakedatafromthe
incomingraweventqueue,classifyit(iftheeventdoesnotalreadyhaveaclass),add
devicecontextandeventcontext,andperformanytransforms.Itthenoutputstothe
zeneventsqueuesothatthezeneventserverdaemoncanmanageitsprogresstothe
MySQLdatabase,totheuserinterfaceandforalertingaction.
22
1February2013
Figure12:Zenoss4eventarchitecture
2.4.4 zenactiond
zenactiondhasbeencompletelyrewrittenforZenoss4.Itisresponsibleforexecuting
actionsassociatedwithnotificationssuchaspaging,email,executingbackground
commandsandraisingnotificationTRAPs.zenactiondwillperiodicallyinspectthe
signalqueueforsignalmessages,dumpthemintoitsshareofmemcachedand
subsequentlyactonthemessagesasinstructedintheassociatednotification.
2.4.5 memcached
PriortoZenoss4eachofthedaemonshaditsowncache.Thiscouldbeawasteful
allocationofmemory.WithZenoss4,amemcachedsubsystemisintroducedwhich
providessharedL2memorycacheforalldaemons,offeringmuchbetterperformance.
memcachedisconfiguredin/etc/sysconfig/memcached.Thedefaultistoconfigure
64Mbformemcached(whichisnotpreallocated;itisonlyusedasnecessary).This
shouldbeincreasedtoatleast1Gbonproductionsystemswithmorethan100devices
(andrun/etc/init.d/memcachedrestart).Alsoensurethatmemcachedisenabledin
$ZENHOME/etc/zope.conf.
1February2013
23
2.5 Other database-related changes in Zenoss 4

Notdirectlyrelatedtotheeventssubsystem,buttheZopedatabase(ZODB)thatusedto
beheldin$ZENHOME/var/Data.fsandaccessedbythezeoctldaemon,isnowstored
inthesameMySQLinstanceaszenoss_zep(andZEOhasgone).
ThezodbdatabaseisthemainZopedatabaseandthereisalsoazodb_session
databasewhichholdsuserpreferencesthinkofzodb_sessionasanexpandedsetof
user'scookies;ifnecessary,itcanbedeletedanditwillberecreatedautomatically.
ZODBiswherealltheobjectdataisstoredrelatingtodevices,components,processes,
services,networks,MIBs,etc.Theeventprocessingdaemonsneedaccesstothezodb
databasetoenricheventswithdeviceandcomponentinformation.
Zopeobjectsareknownaspickles,typicallyastringrepresentationofencodeddata(a
blob)inotherwords,treattheZODBdatabaseasablackbox(justasData.fswas).
AJSONinterfaceisprovidedtoaccessdataintheZODBandthezendmdtoolstill
worksinexactlythesamewayasinpreviousversionsofZenoss,despitetheZODBnow
beinginMySQL.
24
1February2013
Figure13:ComparisonofoldandnewtechnologiestoholdZopeZODBdatabase
ToprovideaccesstothethezodbMySQLdatabase,aRelStoragesubsystemisusedas
ahighperformancebackendtoZODB.RelStoragemayalsousememcachedtofurther
enhanceperformance.
TheolderversionsofZenossdidnotdomuchbywayofindexingtheeventsdatabase.
WithZenoss4holdingZODBdataaswellaseventsdatainMySQL,aneffective
indexingmechanismwasrequiredsotheLucenepackageisusedfromApache.Lucene
isahighperformance,fullfeaturedtextsearchenginelibrarywrittenentirelyinJava.
Itisusedtoholdindexesforbothzodbandzenoss_zep.
2.6 Event life cycle

Thelifecycleofaneventhaseightphases:
Eventgeneration
Devicecontextadditionalinformationaboutthedevicethatgeneratedtheevent
Eventclassmappingtodistinguishonetype(class)ofeventfromanother
Eventcontextadditionalinformationpertinenttoaclassofevent
1February2013
25
Eventtransformmanipulationofeventfields
Databaseinsertionanddeduplication
Resolution
Ageingandarchiving
Figure14:Eventlifecycle,generationtodatabaseinsertion
Processingofaneventdependsontheeventclassthataneventisassignedtothe
valueofitseventClassfield.Adescriptionofeachofthesephaseswillbegivenhere:
subsequentsectionsofthepaperprovidemoredetailsofsomeareas.
InFigure14,thefirstsixphasesoftheeventlifecycleareshown.Theblue,dashed
pathshowstheprogressofaninternallygeneratedZenossevent,whichdoesnotpass
throughaneventmappingphase.AneventClassfieldisproducedbythedaemonthat
generatedtheevent.Itsonlywaytoapplyatransformisasaclasstransform.
ThepurplepathshowstheprogressofaneventthatisgeneratedexternallytoZenoss.
TheinitialparsingdaemonmustprovideaneventClassKeyfieldwhichisthenused,
alongwithotherfields,inaneventclassmappingRuleand/orRegex,whichinturn
providesaneventClassfield.Aftermapping,theeventmaypassthroughbothan
eventclasstransformandaneventmappingtransform.
26
1February2013
AnareathathaschangedfairlysignificantlyinZenoss4isthemechanismforresolving
andageingevents.PriortoVersion4,aneventwasfundamentallyopen(whichalso
encompassedeventStateofAcknowledgedandSuppressedaswellasNew)andsuchan
eventresidedinthestatustableoftheeventsdatabase;alternatively,aneventwas
Closed,inwhichcaseitwasmovedtothehistorytableoftheeventsdatabase.
WithZenoss4,thepossiblevaluesofeventStatehavebeenexpandedtoinclude:
Name
Number
Description
New
Anewevent
Acknowledged
Acknowledgedbyuserortransform
Suppressed
Eventtypicallybeyondasinglepointoffailure
Closed
Eventresolvedbyauser
Cleared
Eventresolvedbyanautomaticrule
Dropped
WouldneverreachtheMySQLdatabase
Aged
Eventautomaticallyclosedaccordingtothe
severityandlastseentimeoftheevent.
Thesearewelldescribedinchapter7oftheZenossCore4AdministrationGuide.The
hugedifferencehereisthatthenewevent_summarytableintheMySQLdatabasewill
probablyhaveClosed/Cleared/Agedeventsinit.Theevent_archivetablehasevents
thathavebeenautomaticallyagedoutbasedontheirseverityandage.
2.6.1 Event generation

Fundamentally,eventswilleitherbegeneratedbyZenossitselfintheprocessof
discovery,availabilityandperformancechecking,oreventswillbegeneratedoutside
ZenossandcapturedbyspecialisedZenossdaemons.
1February2013
27
Zenossdaemon
Exampleofwheneventgenerated
zenping
pingfailureoninterface
zendisc
newdevicediscovered
zenstatus
TCP/UDPserviceunavailable
zenprocess
processunavailable
zenwin
Windowsservicefailed
zenwinperf
WMIperformancedatacollectionfailure/threshold
zencommand
sshperformancedatacollectionfailure/threshold
zenperfsnmp
SNMPperformancedatacollectionfailure/threshold
zenmodeler
Configurationdatachangedonzenmodelerpoll
Table2.1.:EventsgeneratedbyZenossitself
Zenossdaemon
Exampleofwheneventgenerated
zensyslog
processessyslogeventsreceivedonUDP/514(default)
zeneventlog
processesWindowseventsreceivedusingWMI
zentrap
processesSNMPTRAPsreceivedonUDP/162
Table2.2.:ExternaleventscapturedbyspecialisedZenossdaemons
EventsgeneratedinternallybyZenossneednofurtherprocessingtointerprettheevent.
Thedaemonthatgeneratestheeventparsesthenativeinformationandassignsavalue
totheeventClassfieldandanyotherrelevantfieldssuchascomponent,summary,
messageandagent.TypicallytheeventClassKeyfieldwillbeblank.SomeZenoss
daemonspopulatetheeventKeyfield(forexampleanInterfacediscoveryeventwill
populatetheeventKeyfieldwiththeIPaddressofthediscoveredinterface).
EventsthatareinitiallygeneratedoutsideZenossarecapturedbyzensyslog,
zeneventlogorzentrap.Thesedaemonseachhaveaparsingmechanismtointerpret
thenativeeventintotheZenosseventformat.ThePythoncodeforthezensyslogand
zentrapparsingisin$ZENHOME/Products/ZenEvents.(Bydefault,$ZENHOMEwill
be/opt/zenoss).SyslogProcessing.pydecodessyslogevents;zentrap.pydecodesSNMP
TRAPs.
ThedaemonsforprocessingWindowsWMIdatausedtobeastandardpartoftheCore
codebutwithZenoss4thishasmovedtoaZenosssuppliedZenPack
ZenPacks.zenoss.WindowsMonitor.zenwin,zenwinperfandzeneventlogcanallbe
foundunderthatZenPack'sbasedirectory.
Typically,theexternaleventparsingmechanismsdonotdeliveravalueforeventClass;
rathertheydeliveravaluefortheeventClassKeyfield,alongwithvaluesforsome
28
1February2013
otherfieldssuchascomponent,summary,messageandagent.Itisthenthejobofthe
eventmappingphasetodistinguishtheeventclass.
2.6.2 Application of device context

Earlyintheeventprocessinglifecycle,thezeneventddaemonappliesdevicecontext
totheevent.Thismeansthatsevenfieldsoftheeventarepopulatedbydeterminingthe
devicethatgeneratedtheeventandthenlookingupthefollowingvaluesforthedevice
intheZODBdatabase:
prodState
DevicePriority
Location
DeviceClass
DeviceGroups
Systems
ipAddress(mayhavealreadybeenassigned)
2.6.3 Event class mapping

Eventclassmappingtendsonlytobeapplicabletoeventsthatoriginateoutsidethe
Zenosssystem.Itistheprocessbywhichaneventisassignedavalueforits
eventClassfieldand,potentially,otherfields.
Typically,theeventgenerationphasewilldeliveraneventwithafewfieldspopulated;
generallythisdoesnotincludetheeventClassfieldbutdoesincludetheeventClassKey
field.OftentheZenossparsingdaemon(suchaszensyslog),willusethesame
eventClassKeyforseveraldifferentnativeevents.Forexample,aneventClassKeyof
dropbearisusedforseveralloginsecurityevents.Thecomponent,summary,message
andagentfieldsmayalsobepopulated.
Theeventclassmappingphaseexaminestheevent(suchasitis,sofar)andthenusesa
numberofteststodeterminetheeventClasstoassigntothisevent:
1. AneventClassKeyfieldmustexistformappingtobesuccessful.
2. APythonRulecanbewrittentotestanyavailablefieldoftheeventorany
availableattributeofthedevicefromwhichtheeventcame.Suchrulescanbe
complexPythonexpressions,includinglogicalANDsandORs.Iftheruleis
satisfied,theincomingevent'seventClassfieldwillbegiventheclassassociated
withthatmapping.Iftheruleisnotsatisfied,thismappingisdiscarded,the
classisnotassociated,andthenextmappingwillbetestedforamatch.ARule
doesnothavetoexistinamappinginstance.
3. IftheRuleissatisfied(ordoesnotexist),themappingcanthenuseaRegex
Pythonregularexpressiontoparsetheevent'ssummaryfield,checkingfor
particularstrings.TheRegexcanalsoassignpartsofthesummaryfieldtonew,
1February2013
29
userdefineddetailfieldsoftheevent.IfaRuleexistsandissatisfied,theclass
mappingwillapply,eveniftheRegexisnotsatisfied;anyuserdefinedfieldsin
theRegexwillnotbecreatediftheRegexdoesnotmatch.IfaRuledoesnot
existthentheRegexmustbesatisfiedforthemapping(andanytransform)to
apply.
4. TheGUIdialoguethatdefinesthemappingspecifiestheeventClassKey,theRule,
theRegexandanyTransform.Asequencenumberisalsoavailablesothatif
multipleincomingeventshavethesameeventClassKeythenthesequence
numberdefinestheorderinwhichthevariousmappingswillbeapplied,lowest
numberfirst.ThefirstRule/Regexmappingcombinationthatmatcheswillbe
applied.
Eventclassmappingisexecutedbythezeneventddaemon.
2.6.4 Application of event context

EventcontextisdefinedbytheConfigurationProperties(zProperties)ofanevent.
Eventcontextcanbedefinedattheeventclasslevel,foraneventsubclass,oratthe
eventmappinglevel.Aswithallobjectorientedattributes,thevaluesareinheritedby
childobjectssoapplyingeventcontexttoaclassautomaticallysetsitforanysubclasses
andsubclassmappings.Thethreeeventcontextattributesare:
zEventAction
status|history|dropdefaultisstatus
zEventClearClasses
bydefaultthisisanemptyPythonlistofstrings
zEventSeverity
Originalbydefault
Eventcontextisappliedintheeventlifecycle,afterRuleandRegexprocessingbut
beforeanyeventtransforms.Thus,thezEventActionzPropertycanspecifyhistorybut
aneventtransformcouldoverridethatactionbysettingtheevt._actionvalueto
status.
NotethatthestatusandhistoryvaluesreflecttheolddatabasetablespriortoZenoss4.
statusnowmapstoaneventStateofNewandhistorymapstoaneventStateofClosed;
bothwillbestoredintheevent_summarydatabasetable.
Eventcontextisappliedbythezeneventddaemon.
2.6.5 Event transforms

Eventtransformscanbespecifiedforaneventclassmappingorforaneventclass(or
subclass).AtransformiswritteninPythonandcanbeusedtomodifyanyavailable
fieldsofeithertheeventorthedevicethatgeneratedtheevent.Itcanalsocreateuser
definedfields.
FromZenoss2.4,cascadingeventtransformsmeanthatclasstransformsareapplied
fromeverylevelintheappropriateclasshierarchy,followedbyanytransformforan
30
1February2013
appliedeventmapping.PriortoZenoss2.4,eitheramappingtransformwasapplied,
oraclasstransform,butnotboth.Classtransformswereonlyappliedtotheexact
class,notfromtheeventclasshierarchy.
AtransforminaneventmappingwillonlybeexecutedoncetheeventClassKeyhasbeen
matched,andtheRulehasbeensatisfied(ifitexists).IfaRuledoesnotexist,any
Regexhastobesatisfiedforthetransformtobeexecuted.
Eventtransformsareexecutedbythezeneventddaemon.
2.6.6 Database insertions and de-duplication

ZenosseventsarenowstoredinaMySQLdatabasecalledzenoss_zep(usedtobe
events).Themaintablesfortheeventlifecyclearetheevent_summarytablefor
recentevents,theevent_archivetableforoldevents.
Somefieldsoftheeventareonlyassignedatdatabaseinsertiontimetheyarenot
availableateventmappingoreventtransformtime.Theseinclude:
count
eventState
evid
stateChange
dedupid
eventClassMapping
firstTime
lastTime
ItistheJavazeneventserverdaemonthatisresponsibleforgettingeventsintothedatabase.
Zenossautomaticallyappliesaduplicationdetectionrulesothatifaduplicateevent
arrives,thentherepeatcountofanexistingeventwillbeincremented.duplicateis
definedashavingthefollowingfieldsthesame:
device
component
eventClass
eventKey
severity
IftheeventdoesnotpopulatetheeventKeyfield,thenthesummaryfieldmustalso
match.Thededupidfieldiscreatedbyconcatenatingtheabovefieldstogether,
separatedbythepipe(verticalbar)symbol.Thusanexamplededupidmightbe:
zenoss.skills-1st.co.uk|su|/Security/Su||5|FAILED SU (to root)jane on /dev/pts/1
wherethedeviceiszenoss.skills1st.co.uk,componentissu,eventClassis/Security/Su,
theeventKeyisunset,severityis5(Critical),andthesummaryisFAILEDSU(toroot)
janeon/dev/pts/1.
InZenoss4,thededupidfieldisalsoknownasthefingerprint.
1February2013
31
Whenaneweventisreceivedbythesystem,thededupidisconstructedbythe
zeneventddaemon.Transformsmaymodifyeithercomponentfieldsofthefingerprintor
maydirectlymodifythededupidfield.
Whenzeneventservercomestoinserttheeventinthedatabase,ifitmatchesthe
dedupidforanyactiveevent,theexistingeventisupdatedwithpropertiesofthenew
eventoccurrence,theevent'scountisincrementedbyone,andthelastTimefieldis
updatedtobethecreatedtimeoftheneweventoccurrence.
NotethatthisisasubtlebutsignificantchangefrompriorversionsofZenossasthe
existingeventisupdatedwithpropertiesofthenewevent;olderversionsofZenoss
simplyupdatedthecountandlastTimefields.Forexample,ifthefingerprintincludes
aneventKeysodoesnotincludethesummary,theresultingeventwillnowshowthe
summaryofthelatestreceivedduplicateevent.
Iftheincomingeventdoesnotmatchthededupidofanyactiveevents,thenitisinserted
intotheactiveeventtablewithacountof1,andthefirstTimeandlastTimefieldsare
settothecreatedtimeofthenewevent.
2.6.7 Resolution
Resolutionofaproblemrepresentedbyaneventcanhappeninseveralways:
Auserclosestheevent(eventState=Closed)
TheeventcontextzEventActionzPropertyforaneventclassisdrop(theeventis
discarded).Forexample,eventclass/Ignore.
TheeventcontextzEventActionzPropertyforaneventclassishistory
(eventState=Closed).Forexample,eventclass/Archive.
Atransformsetsevt._actionto'drop'(theeventisdiscarded)
Atransformsetsevt._actionto'history'(eventState=Closed)
Anotherclearingeventarrivesthatclearstheinitialevent(eventState=Cleared)
TheEventManagersettingshaveseverityandlastSeenparametersthatdenote
whicheventswillbeautomaticallyaged(eventState=Aged)
Alltheaboveeventswillstillbeintheevent_summarytableoftheMySQLdatabase.
TheEventManagerparameterforEventArchiveThresholdistheonlyautomaticaction
thatmoveseventsfromevent_summarytoevent_archiveanditwillmoveallevents
witheventStateofClosed,ClearedandAged.
Themoreinterestingformsofeventresolutioninvolvecorrelationofevents;thereare
twodifferentmechanisms.Thebasicprincipleisthatgoodnewsclearsbadnews.
ThefirstclearingmechanismisthatanyeventwithaseverityofClearwillsearchthe
event_summarytableforsimilaractiveeventsandsettheireventStatetoCleared
(notClosed).
TheZenossCore4AdministratorsGuidedefinesthisautoclearfingerprintas:
32
1February2013
IfcomponentUUIDexists:
componentUUID
eventClass
eventKey(canbeblank)
IfcomponentUUIDdoesnotexist:
device
component(canbeblank)
eventClass
eventKey(canbeblank)
Thiscanbealittleconfusing.TheEventConsoleshowsacomponentfield.Itdoesnot
showacomponentUUIDfield.StrictlythecomponentfieldintheEventConsoleshows
theelement_sub_identifierfieldfromtheMySQLdatabasetablethenameofthe
component.SomeeventsgenerateacomponentUUID(UniversallyUniqueIdentifier)
andsomedonot.InspectingtheeventinthedatabaseorusingtheJSONinterfaceis
theonlywaytodeterminewhetherthisuniquecomponentidfieldexistsornot.Ifit
doesexistthenitshouldalso,byimplication,denotethedevicethatthecomponent
belongsto,hencethedevicefieldisunnecessary.(VersionsofZenosspriorto4didnot
haveacomponentUUID;similarwasdefinedashavingthesameeventClass,device
andcomponentfields.)
EitherwayinCore4,theeventClassandtheeventKeyfieldsaresignificant.Ifthe
componentUUIDdoesnotexistthenitistheelement_sub_identifier(componentname)
thatmustmatch,alongwiththedevicename(element_identifierintheMySQLtable).
Thesecondautomaticclearingmechanismextendstheautoclearfingerprintdefinition
ofeventClass.TheeventcontextofaneventclassincludeszEventClearClasseswhichis
alistofothereventclassesthatthisgoodnewseventwillclear,inadditiontoitsown
class.Theotherconditionsoftheautoclearfingerprintremainthesame.
Notethatthesameeffectcanbeachievedinatransformbyassigningalistofclass
namestoevt._clearClasses.
Alleventswiththesameautoclearfingerprintarecleared,notjustthemostrecent.
TheclearingeventwillautomaticallyhaveitseventStatesettoClosed,providedit
matchesoneormorebadnewsevents.Ifitdoesnotmatchanyeventsthenthe
clearingeventisdroppedandwillnotbepersistedtothezenoss_zepdatabase.Thisis
toavoidfillingupthedatabasewithredundantgoodnewsevents.
Whencorrelationtakesplacesomeoftheexistingbadnewseventfieldsareupdated;
stateChangebecomesthetimewhentheeventwasresolved;clearidispopulated
withtheevidfieldoftheclearing,goodnewsevent.
Thisautomaticresolutionofeventsisperformedbythezeneventserverdaemon.
1February2013
33
2.6.8 Ageing and archiving

Maintenanceisrequiredonthetablesofthezenoss_zepdatabaseorthediskwillsimply
fillupeventually.ThreemechanismsareprovidedbytheEventManager:
Bydefault,eventswithseveritylessthanErrorwillbeAgedafteranEvent
AgeingThresholdof4hours;thatis,theeventStatewillbesettoAged(strictly
thevalue6).
Bydefault,theEventArchiveThresholdis4320minutes(3days).Thismeans
anyeventwitheventStateofClosed,ClearedorAgedwillbemovedfromthe
event_summarytabletotheevent_archivetableofthezenoss_zepdatabase.
TheDeleteArchivedEventsOlderThan(days)parameteris90bydefault.Thisis
theonlyparameterthatautomaticallydeletesdata.Itisnotpossibletofinetune
thistodelete,say,lowerseverityeventsafterdifferentintervals.
Zenosspriortoversion4providedautility,
$ZENHOME/Products/ZenUtils/ZenDeleteHistory.py
whichcoulddeleteeventsselectivelybasedonageandseverity.Thisutilityisnot
shippedwithZenoss4andcurrentlyhasnoequivalentfunction.
DeletingdatafromtheoldhistorytableinZenoss3usedtobeveryslow.InZenoss4,
theevent_archivetableispartitioned,byday,ratherthanbeingonehugefile.This
meansthatdeletingdataissimplyamatterofdroppingpartitionfiles.Thiscanbeseen
fromthemysqlinterfacewith:
showcreatetableevent_archive;
3 Events generated by Zenoss

Inthecourseofdiscovery,availabilitymonitoringandperformancemonitoring,Zenoss
maygenerateeventstorepresentachangeinthecurrentstatus.Althoughmanyevents
arebadnewsitshouldberecognisedthateventscanalsobegoodnewsInterface
Up,Thresholdnolongerbreached,etc.
EventsgeneratedbyZenossaredependentonthevariouspollingintervalsconfigured.
Toexaminethedefaultparameters,usetheADVANCED>Collectorsmenu.Clickon
localhost(thecollectorontheZenosssystem).NotethatearlyversionsofZenossused
thetermandmenuoptionMonitorsratherthanCollectors.
34
1February2013
Figure15:DefaultparametersforlocalhostCollector
Parameterstonoteparticularlyare:
SNMPPerformanceCycleInterval
300secs(5mins)
ProcessCycleInterval
180secs(3mins)
StatusCycleInterval
60secs(1min)
WindowsServiceCycleInterval
60secs(1min)
PingCycleTime
60secs(1min)
ModelerCycleInterval
420mins(12hours)
3.1 zenping
Themostbasiclevelofavailabilitycheckingistopingpoll.Thezenpingdaemonwill,
bydefault,pingpolleachinterface,everyminute.Aninterfacedowneventisgenerated
whenthepingfailstogetaresponse.Thiseventisautomaticallyclearedwhena
similarpingissuccessful;meantime,whileaninterfaceremainsdown,thecountfieldof
theeventisincreased.
Thezenpingdaemoncandetectwhenthenetworkpathtoadeviceisbroken,for
exampleifasinglepointoffailurerouterisdown.WithZenoss4thisisachievedusing
nmap;withearlierversions,Zenossbuiltaninternaltopologybasedonquerying
routingtableswithSNMP.
Ifaneventisreceivedforanisolatedelement,aneventisgeneratedwithaneventState
fieldofSuppressedandthesummaryfieldreportsnotonlytheinterfaceforwhichthe
pingfailed,butalsothecausaldevice;forexample:
ip10.191.101.1isdown,failedatbino.skills1st.co.uk
1February2013
35
Allotherdeviceavailabilitymonitoringisdependentonpingaccess.Onceapinghas
failed,SNMP,process,TCP/UDPserviceandwindowsservicemonitoringwillallbe
suspendeduntilpingaccessisrestored.Thecountfieldofthehigherlevelmonitoring
eventswillnotincreaseuntilpingaccessisresumed.
Alsonotethatifthereisnopingaccess,noperformanceinformationwillbecollected.If
adevicereallydoesnotsupportping,perhapsbecauseoffirewallrestrictions,then
ensurethatthezPropertyzPingMonitorIgnoreissettoTrue;thiswillpermitSNMPand
sshavailabilitymonitoringandperformancedatacollection.
Thelogfileforzenpingiszenping.login$ZENHOME/log.
3.2 zenstatus
ThezenstatusdaemoncanbeconfiguredtocheckforaccesstovariousTCPand/orUDP
portsonbothWindowsandUnixarchitectures.Bydefault,itcheckseveryminute.
Zenosscomeswithahugenumberofservicespreconfigured;thesecanbeexamined
fromtheINFRASTRUCTURE>IpServicesmenu.Bydefault,theonlyservice
monitorsthatareactiveareforsmtpandhttp;therestaresetwithmonitoringdisabled.
Aswithpingpolling,agoodnewsserviceeventforadeviceautomaticallyclearsa
similarbadnewseventandthecountfieldoftheeventincreaseswhilsttheservice
remainsdown.
Thelogfileforzenstatusiszenstatus.login$ZENHOME/log.
3.3 zenprocess
zenprocessmonitorsWindowsandUnixsystemsforthepresenceofprocesses.Ina
Unixcontext,thiswouldbewhethertheprocessappearsinapseflisting;inaWindows
context,theprocessmustappearintheWindowsTaskManager(andnotethatthis
checkiscasesensitiveonbotharchitectures).Monitoringisevery3minutes,bydefault.
Configurationofprocessmonitoringforadeviceissimilarasforservicesthe
INFRASTRUCTURE>Processesmenuprovidesawaytoconfigureprocessestobe
monitored.Zenoss4comeswithdefinitionspreconfiguredforalltheZenossprocesses.
ProcessmonitoringisactuallyachievedusingtheHostResourcesManagement
InformationBase(MIB)ofSNMP,byretrievingthehrSWRuntable.Thismeansthat
ifSNMPaccesstoadeviceisbroken,therewillbenoprocessinformation.
Aswiththeotheravailabilitydaemons,goodnewseventsclearbadnewseventsand
thecountfieldincreasesonsubsequentfailedpolls.
Thelogfileforzenprocessiszenprocess.login$ZENHOME/log.
36
1February2013
3.4 zenwin
ThezenwindaemonshipswiththeZenPacks.zenoss.WindowsMonitorZenPackwith
Zenoss4(itwasastandardpartoftheCorecodeinearlierversions).Itmonitors
Windowsservices(notTCP/UDPservices).Thesecanbeexaminedfromthe
INFRASTRUCTURE>WindowsServices.Bydefault,noneofthesemonitorsare
active.
zenwinusestheWindowsManagementInstrumentation(WMI)interfacetoaccess
servicesontheremotesystemeveryminute,bydefault.ThezPropertiesforadevice(or
deviceclass)mustbeconfiguredtoallowaccesstoWMIbeforewindowsservicepolling
canbesuccessful.
Aswithpingpolling,agoodnewswindowsserviceeventforadeviceautomatically
clearsasimilarbadnewseventandthecountfieldincreasesonsubsequentfailed
polls.
Thelogfileforzenwiniszenwin.login$ZENHOME/log.
3.5 zenwinperf
zenwinperfisanewdaemonforZenoss4whichisalsopartofthe
ZenPacks.zenoss.WindowsMonitorZenPack.WithearlierversionsofZenoss,many
usersdeployedtheexcellentcommunityWMIDataSourceandWMIWindows
PerformanceZenPackstoachievesomethingverysimilartothisnewdaemon.
zenwinperfprovidesperformancemonitoringofinterfaces,filesystems,memory,CPU
andpagingusingtheWMIprotocol.Defaultthresholdsareconfiguredforsomemetrics
whichthengenerateeventswhenexceeded.Itcanbeextendedbytheusertomonitor
otherperfmonmetricsusingtheWMIprotocol.
Dataisgatheredevery5minutes.
Thelogfileforzenwinperfiszenwinperf.login$ZENHOME/log.
3.6 zenperfsnmp
zenperfsnmppollseachdeviceevery5minutes,bydefault.ItcancollectbothSNMP
performanceinformationandstatusinformationforprocesses.EvenifSNMP
performancemonitoringisnotconfigured,zenperfsnmpchecksthattheSNMPagentis
available.
Within5minutesofanSNMPpollfailure,ansnmpagentdowneventshouldbe
generated.Withinafurther3minutesthereshouldbeanUnabletoreadprocesseson
device..event,ifprocessmonitoringisconfigured.Notealsothatthecountfieldfor
individualmissingprocesseventsshouldstopincreasing.WhileSNMPaccesstothe
deviceremainsbroken,thecountfieldfortheUnabletoreadprocessesondevice..
eventwillincreaseevery3minutes.
1February2013
37
Thelogfileforzenperfsnmpiszenperfsnmp.login$ZENHOME/log.
3.7 zencommand
Thezencommanddaemonperformsmonitoringbasedonrunningcommands,typically
overansshconnection.Likezenperfsnmpandzenwinperfitusesperformance
templatestomonitormetricsandcangenerateaneventifathresholdisbreached.
Thelogfileforzencommandiszencommand.login$ZENHOME/log.
4 Syslog events
TheUnixsyslogmechanismispervasivethroughoutallversionsofUnix/Linux
althoughslightlydifferentversionsandformatsexist.Therearealsoopensource
implementationsofsyslogforWindowssystemsandmanynetworkingdevicesalso
supportthesyslogconcept.
Typicallysystemmessagesareoutputtooneormorelogfilessuchas
/var/log/messages.Thesyslogsubsystemcanalsobeconfiguredtosendsyslog
messagestoacentralsyslogratherthanholdingfilesoneachsystem.Thewellknown
defaultportforforwardingsyslogmessagesisUDP/514.
Astandardsyslogsystemisconfiguredbythesyslog.conffile,typicallyin/etc.Anewer
versionofsyslogisimplementedonsomesystems,syslogng,whichhasgreaterfiltering
capabilities.Thesyslogngconfigurationfileistypically/etc/syslogng/syslogng.conf.
AnothervariationisrsyslogdwhichistypicallyshippedwithnewerRedHat/CentOS
SuSEsystems,configuredthrough/etc/rsyslog.conf.
Asyslogmessageincludesapriorityandafacility.Theprioritiesare:
0
emerg
1
alert
2
crit
3
err
4
warning
5
notice
6
info
7
debug
Facilitiesinclude:
38
auth (4)
authpriv(10)
cron (9)
daemon(3)
ftp(11)
kern(0)
lpr(6)
mail(2)
1February2013
news (7)
syslog(5)
user (1)
uucp(8)
Thesedefinitionscanbefoundinsyslog.h(typicallyin/usr/include/sys).Bothpriority
andfacilityareencodedinasingle32bitintegerwherethebottom3bitsrepresent
priorityandtheremaining28bitsareusedtorepresentfacilities.
Forexample,ifthefacility/prioritytagis<22>,thiswouldbe00010110inbinary,where
thebottom110representsapriorityof6(info)andthetop00010representsafacilityof
2=mail.
4.1 Configuring syslog.conf

AnydevicethatisgoingtoreportsyslogeventstoZenossmusthaveitssyslog.conffile
configuredwiththedestinationaddressoftheZenosssystem.Theoriginalsyslog.conf
permitsfilteringbasedonpriorityandfacilityso,acatchallstatementtosendall
eventstotheZenosssystem,wouldbe:
*.debug
@<IP address of your Zenoss system>
Thisalsoworksforrsyslogd.SeeFigure16foranrsyslog/syslogexamplethatforwards
tozen42.class.example.orgallfacilitieswithpriorityofnoticeandabovebutallcron
messagesarefilteredout;authprivmessageswillbeforwardedwithseverityinfoand
above.
Figure16ConfigurationfileforrsyslogsendingselectedeventstoZenossserver
1February2013
39
syslogng.confrequiresatleastasource,adestinationandalogstatement.syslogng
offerssuperiorfilteringovertheoriginalsyslogsooneormorefilterstatementsmay
alsobepresent.
Figure17:syslogng.conftosendalleventstoZenosssystemat10.0.0.131(nofilteringactive)
4.2 Zenoss processing of syslog messages

TocollectsyslogmessageswithZenoss,thezensyslogprocessautomaticallystartson
portUDP/514andcollectsanysyslogmessagesdirectedfromothersystems.zensyslog
thenparsesthesemessagesintoZenossevents.Youmustensurethatthesyslog.conf
fileontheZenosssystemdoesnotenablecollectingremotesyslogsorthesyslogdand
zensyslogprocesseswillclashoverwhogetsUDP/514(itispossibletoreconfigureeither
daemon,ifrequired).
40
1February2013
Toexaminetheincomingsyslogmessagesandtheparsingthatzensyslogperforms,the
levelofzensyslogloggingcanbeincreased.
1. UsetheINFRASTRUCTURE>Settings>Daemonsmenu.
2. Clicktheeditconfiglinkforthezensyslogdaemon.
3. ChangethefollowingparametersandclickSave:
logorig
selectthis
logseverity
Debug
4. Inspecttheunderlyingconfigurationfilein$ZENHOME/etc/zensyslog.conf.
5. Thelogoriglinesaystologtheoriginalincomingsyslogmessage;itwillbein
$ZENHOME/log/origsyslog.log.Notethatthisparameterisuniquetozensyslog
andisusefulfordebugging.
6. ThelogseveritylineisagenericZenossdaemonparameter;avalueof10isthe
maximumDebuglevel.
7. Don'tforgettoSavethischange
8. UsetheRestartlinktorecyclezensyslog.Alternatively,asthezenossuser,issue
thecommand:
zensyslog restart
9. Examinethezensysloglogfilein$ZENHOME/log/zensyslog.log
10. Anewincomingeventstartswithalineshowinghostnameandipaddress,eg.
host=zen241.class.example.org, ip=172.16.222.241
11. Thenext2linesshowtherawmessageandthedecodingforfacilityandpriority.
12. Linesstartingwithtagshowthezensyslogparsingprocessasitteststhe
incominglineagainstvariousPythonregularexpressions,hopefullyendingwith
atagmatchline.
13. Ifamatchissuccessful,aneventClassKeymaybedetermined
14. ThelastlineforaparsedeventshouldbeaQueueingevent.
1February2013
41
Figure18:zensyslog.logshowingparsingprocess
Wheneverdifferentnativeeventlogsystemsareintegratedthereisalmostinevitablya
mismatchofseverities.Thefollowingtabledemonstratesthis.
Zenoss
syslogpriority
Windows
Critical(red)(5)
emerg(0)
Error(1)
Error(orange)(4)
alert(1)
Warning(2)
Warning(yellow)(3)
crit(2)
Informational(3)
Info(blue)(2)
err(3)
Securityauditsuccess(4)
Debug(grey)(1)
warning(4)
Securityauditfailure(5)
Clear(green)(0)
notice(5)
info(6)
debug(7)
Table4.1.:EventseveritiesforZenoss,syslogandWindows
NotethatthenumericvalueofZenosseventseveritydecreasesaseventsgetless
criticalbutthatthepriorityofsyslogeventsincreasesaseventsgetlesscritical.
DefaultmappingfromsyslogprioritytoZenosseventseverity,isperformedby
$ZENHOME/Products/ZenEvents/SyslogProcessing.pysearchfordefaultSeverityMap
aroundline187inCore4.2.Theresultisthat:
42
syslogpriority<3(emerg,alert,crit)maptoZenossseverity5(Critical)
syslogpriority3(err)mapstoZenossseverity4(Error)
syslogpriority4(warning)mapstoZenossseverity3(Warning)
syslogpriority5or6(notice,info)maptoZenossseverity2(Info)
1February2013
Outofthebox,allsyslogeventsmaptotheZenosseventclassof/Unknown.
SyslogProcessing.pyisthecodethatparsesanyincomingsyslogmessageandgenerates
aZenossevent.
ThefirstsectionhasaseriesofPythonregularexpressionstomatchagainstthe
incomingsyslogline.Eachexpressionischeckedinturnuntilamatchisfound.Ifno
matchisfoundthenanentrygoesto$ZENHOME/log/zensyslog.logwithparseTag
failed.
Figure19:SyslogProcessing.pyregularexpressionstomatchsyslogtags
ThemainbodyofSyslogProcessing.pystartsbyassigningvaluesfromtheincoming
eventtoZenosseventclassfields,asfollows:
1February2013
43
def process(self, msg, ipaddr, host, rtime):

evt = dict(device=host,
ipAddress=ipaddr,
firstTime=rtime,
lastTime=rtime,
eventGroup='syslog')
Atthisstage,noaccountofduplicatesistakensothefirstTimeandlastTimefieldsare
bothsettothetimestampontheincomingevent.NotethattheZenosseventGroupfield
ishardcodedatthisstagetosyslog.
Figure20:SyslogProcessing.pyprocessmainroutine
parsePRIisthePythonfunctioncalledtoparseoutthesyslogpriorityandfacility.
ThedefaultSeverityMapfunctioniscalledfromwithintheparsePRIfunctiontosetthe
severityfieldoftheZenossevent.
44
1February2013
Figure21:SyslogProcessing.pyparsingofpriority,facilityandseverity
Next,theparseHEADERfunctioniscalledtoextractthetimestampandhostnamefrom
theincomingevent.ThedeviceandipAddressfieldsoftheZenosseventaresetatthe
endofthisfunction.
1February2013
45
Figure22:SyslogProcessing.pyprocessingtheheaderinformation
TheparseTagfunctioniscalledtoparseoutthesyslogtag,usingtheregexexpressions
atthebeginningofthefile.IfnomatchexiststhenaparseTagfailedmessageislogged.
TheendofthefunctionreturnstheremainderoftheincomingmessageintheZenoss
eventsummaryfield.
46
1February2013
Figure23:SyslogProcessing.pyparsingthesyslogtag
ThecruxofeventprocessinginZenossistoderiveaneventClassKeythisisdone
withthebuildEventClassKeyfunction.
1February2013
47
Figure24:SyslogProcessing.pydeterminingtheEventClassKey
Notethatiftheeventhasthecomponentfieldpopulatedthenthatisusedasthe
eventClassKeyaftercheckingforapreexistingeventClassKeyandforanntevidfield.
5 Zenoss processing of Windows event logs

5.1 Management using the WMI protocol
Zenosspriortoversion4shippedWindowsmonitoringaspartoftheCorecode.Zenoss
4shipsWindowssupportwiththeZenPacks.zenoss.WindowsMonitorZenPackwhich
hasaprerequisiteofZenPacks.zenoss.PySamba.TheseareZenossprovidedCore
ZenPacks.
IfaWindowsdevicesupportsSNMPthenitisperfectlypossibletousethatprotocol,
especiallyasmostWindowsSNMPagentsalsosupporttheHostResourcesMIBsosome
systeminformationisavailableinadditiontothestandardMIB2networktype
information.
TheZenossWindowsZenPacksintroducethe/Server/Windows/WMIdeviceclasswhich
hasbothWMImodelerpluginsandWMIperformancetemplatesassociatedwithit.
Targetdevicesshouldbeaddedtothisclassorsubclassesthereof.Thisallows
monitoringusingtheWindowsManagementInstrumentation(WMI)protocol.Auserid
andpasswordneedtobeconfiguredontargethoststopermitWMIaccessfromthe
48
1February2013
Zenossserver;italsomeansthatfirewallsbothontheWindowsdevicesandany
interveningnetworkfirewalls,mustbeconfiguredtopermitWMIaccess.TheZenoss
ServermustthenbeconfiguredwithmatchingWindowszProperties(zWinUserand
zWinPassword)forthetargetdevices/deviceclasses.ThereareafewotherWindows
specificConfigurationPropertiesseeFigure25.ThesezPropertiescanbechangedfor
adeviceclassorforaspecificdevice.
Figure25zPropertiesforWindowstargets
ZenPacks.zenoss.WindowsMonitorprovidesthreenewdaemons:
zenwin
monitorswindowsservicesusingWMI
zenwinperf
collectsperformancedatausingtheWMIprotocol
zeneventlog
retrievesWindowseventloginformationusingWMI
ThethreezWinPerf...zPropertiesfinetunetheconfigurationofthezenwinperfdaemon;
thezWinEventlogparametermustbeTruetocollectWindowseventsfromatarget
device.
ThezWinEventlogMinSeveritypropertydefinestheleastseriousseverityeventsthat
willbeforwardedfromWindowstoZenoss.Notethatthenumericdenotationof
windowseventseveritiesandtheirnamesandsupportcurrency,havechangedoverthe
lifeofZenoss.SeeTable4.1onpage42forcurrentvalidseverities.Alsonotethatifyou
changethisparameteryouarepresentedwithalistofZenossseverities,notWindows
styleseverities;againrefertotheearliertableforatranslation.Ifyouwanttoinclude
allWindowsseverities,includingsecurityauditfailure(5),youneedtoselecttheClear
severityinthedropdownmenuwhenchangingzWinEventlogMinSeverity.
ThezWinEventlogClausewasintroducedduringthelifetimeofZenoss3tohelpfilter
eventsfromWindowsdevices.ConsulttheZenossCore4AdministratorsGuide,chapter
1February2013
49
6.6.6fordocumentationandexamples.Thisparameterisratherobtuse.Fundamentally
aWindowsQueryLanguage(WQL)queryisconstructedtoberunbyzeneventlog:
SELECT*FROM__InstanceCreationEvent
WHERETargetInstanceISA'Win32_NTLogEvent'
ANDTargetInstance.EventType<=zWinEventlogMinSeverity
AnyzWinEventlogClauseislogicallyAND'edwiththisWQL;thusifyouwanttoONLY
seeeventswitheventidof528and529(SuccessfullogonandLogonfailure),configure
zWinEventlogClausetobe:
(TargetInstance.EventCode=529orTargetInstance.EventCode=528)
Strictly,thezeneventlogdaemonpollstargetWindowssystemsforeventsandparses
themintoZenossstyleevents.Typically,theSourcefieldontheWindowseventmapsto
thecomponentfieldintheZenossevent;theZenosseventClassKeyiscomposedofthe
Windows<Source>_<EventID>(eg.Perflib_2003);theZenosseventGroupbecomesthe
Windowslogfilename(Application,Security,etc)andtheWindowsEventIDismapped
totheZenossntevidfield.
Toseetheworkingsofzeneventlog,changetheloggingleveltoDebug(10),restartthe
daemonandinspect$ZENHOME/log/zeneventlog.log.
AgoodwaytoseetheWQLstatementbeingusedistorunzeneventlogasaoneoff
commandintheforeground:
zeneventlogrunv10dwin2003.class.example.org
Figure26Partialoutputfromzeneventlogrunv10dwin2003.class.example.orgshowingWQLstatement
50
1February2013
ManyWindowseventlogeventsareautomaticallymappedtoeventclassesbutthey
mayhavealowseverity(suchasDebug)andtheymayhavetheirzEventActionevent
zPropertysettohistorysothattheydonotappearinthestatustableoftheevents
database.
5.2 Management of Windows systems using syslog

ThereisalsoasyslogutilityavailableforWindowssystemsfromDatagramConsulting
athttp://syslogserver.com.TheclientutilityisSyslogAgentandismadeavailable
undertheGNUlicense.SyslogserverutilitiesforWindowsarealsoavailableas
chargeableproducts.ThismeansthatWindowseventlogscanalsobecollectedwith
thezensyslogdaemon.
NotethattheSyslogagentiscapableofbeingconfiguredtomonitorWindows
applicationlogfiles,inadditiontothestandardWindowseventlogs.Whenmonitoring
thestandardeventlogs,therearebetterfilteringcapabilitieswithSyslogthenwith
zeneventlog.
6 Event Mapping
ZenosseventsarecategorisedintoahierarchyofeventClasses,manyofwhichare
definedoutoftheboxbutwhichcaneasilybemodifiedoraugmented.Theprocessof
EventClassMappingisaboutassociatinganincomingeventwithaparticularZenoss
EventClass(settingitseventClassfield)and,potentially,modifyingotherfieldsofthat
eventbyusinganeventtransform.
Eventclassesandsubclassesaretreatedidenticallyfromthepointofviewofeventclass
mapping.Theclasshierarchycanbeusefulinthateventcontext,asimplementedby
eventzProperties(zEventSeverity,zEventAction,zEventClearClasses),followsthe
normalrulesforobjectinheritanceifzEventActionissettodropontheevent
class/Ignore,thenanysubclassesof/Ignorewillalsoinheritthatproperty.
NotableoutoftheboxeventzPropertiesarethat/Ignoreclassesandsubclassesdrop
incomingeventstotally;/Archiveclassesandsubclassesautomaticallysetthe
eventStatefieldtoClosed.
Mosteventclasseshaveoneormoremappingsassociatedwiththemtheseareknown
asinstances.Notethataneventdoesnothavetohaveanymappingsassociated,in
whichcaseaneventofthatclasswillonlyappearinanEventConsoleifthedaemon
thatgeneratestheevent,assignstheeventclassatthattime(/Perfeventsmaywell
comeintothiscategory,forexample).Outoftheboxeventclassmappingsaredefinedin
$ZENHOME/Products/ZenModel/data/events.xml.Theycanbeinspectedfromthe
ZenossGUIbyselectingtheEVENTS>EventClassesmenu.
1February2013
51
MostoutoftheboxeventclassmappingssimplymatchontheeventClassKeyfield
whichispopulatedbythenativeeventparsingmechanism(suchaszensyslog,
zeneventlog,zentrap).Thesemechanismsmaygenerateseveraldifferenteventswith
thesameeventClassKeyfield;thusothertechniquesareneededtodistinguishbetween
sucheventsandpotentiallytoseparatethemintodifferenteventclasses.
Thesequencenumberinaneventmappinggivestheorderinwhichmappingsaretested
againsttheincomingeventlowestnumbersaretestedfirst.Dependingonwhich
mappingactuallymatches(ifany)willdeterminetheresultingeventClassoftheevent.
6.1 Working with event classes and event mappings

Eventsareorganisedinanobjectorientedhierarchy;thusattributesassignedtoa
parenteventclassareinheritedbyachildeventsubclass.
Neweventclassescanbedefinedbynavigatingtoaneventclassandusingthe
dropdownmenualongsideSubClassestoAddNewOrganizer.Thenamesuppliedisthe
nameoftheneweventclass.Forexample,drilldowntothe/Securityeventclassand
createanewsubclasscalledSu.
Anyeventwhichdoesnotmaptoaneventclassisthegiventheclassof/Unknown.The
simplestwaytomapsuchaneventistostartfromanexistingeventintheEvent
Console.Thefollowingscenarioexplainsthis,creatinganeweventclassmappingcalled
suwhichmapsanincomingeventtotheeventclass/Security/Su.
1. GenerateasyslogauthenticationfailureeventattheZenosssystem.
2. OpenanEventConsolethatshowstheeventandinspectitsdetails.
3. SelecttheeventandusetheReclassifyEventiconatthetopoftheconsole.Select
yournew/Security/Suclassfromthedropdownlist.Youshouldbeshownthe
eventclassmappingpanel.ClickthelefthandEditmenu.
4. Youshouldfindthatthenameoftheneweventclassmappingissettosuand
theEventClassKeyissettosu(notelowercasesinbothcases).The
eventClassKeyfieldisactuallyderivedfromthecomponentfieldoftheincoming
eventinSyslogProcessing.py(aroundline289).Thesummaryfieldoftheevent
shouldhavebeencopiedintothemappingExamplebox.
5. AddatextstringtotheExplanationboxsuchasAutoaddedbyeventmapping.
6. AddatextstringtotheResolutionboxsuchasThisisadummyresolution.
7. OpenaZenossGUIwindowthatshowsallSuevents(youmayfinditusefulto
haveseveralbrowsertabsopentofocusondifferentaspectsoftheZenossGUI).
SelectalltheSueventsandClosethem.
8. GenerateanewSuevent.
9. CheckthedetailsoftheneweventintheEventConsole.Theeventshouldhave
mappedtoeventClass/Security/Su.TheseverityshouldbeInfo(blue).The
52
1February2013
detailsoftheeventshouldshowtheeventClassMappingfieldsetto
/Security/Su/su.
Anyexistingeventmappingcanbemodifiedinasimilarfashion.
Figure27:Editdialogueforeventclassmapping
Wheneveryouchangeaneventmapping,itisadvisabletoclearanyexistingeventsof
thatcategorybeforetestingthenewconfiguration.
Whenyouareworkingwitheventmappings,don'tforgettheEventmenuwhichfilters
anEventConsolebyEventClass.
Itisusefultorefertoeventclassesusingthebreadcrumbpathseenatthetopofa
page,suchas/Events/Security/Su.
1February2013
53
6.1.1 Generating test events

TesteventscanbecreatedfromtheEventConsoleusingthe+icon.
Figure28:Dialoguetocreateatestevent
Alternatively,thecommandlinezensendeventcanbeused(youshouldensureyouare
thezenossuser).Thistakesparameters:
d
device
p
component
k
eventClassKey
s
severity
c
eventClass
y
eventKey
i
IPaddress
h
help
o
<field>=<value>(foranyotherattribute;canhavemultipleo)
monitor
collectorthiseventcamefrom
port=PORT
defaultis8081
server=SERVER
defaultislocalhost
auth=AUTH
defaultisadmin:zenoss
Theremainderofthelineaftertheseoptionsisusedforthesummaryfield
(strictlytheMessagefieldintheGUIdialoguepopulatestheeventsummaryfield)
ThecoreautodeployscriptdeliveredwithZenoss4.2.3hasnewfunctionalitytoincrease
securityonaZenossinstallation.FormanyyearstheZenossuserofadminwitha
passwordofzenosshasbeenconfiguredasstandard.Thenewinstallationscript
changesthis,generatingarobustpasswordwhichisstoredinseveralconfigurationfiles
in$ZENHOME/etc,includingglobal.confandhubpasswd.
54
1February2013
zensendeventisastandalonePythonutilityin$ZENHOME/binthatcommunicates
withthezenhubdaemon.Noteintheusagedescriptionabove,thatthedefaultauth
parametervalueisadmin:zenoss;typicallythismeansthatzensendeventcommandswill
failwithanUnauthorizedmessageunlesstheauthparameterisaddedwiththe
correctuserandpassword,foundin$ZENHOME/etc/hubpasswd.
Adiscussiononmodifyingzensendeventtoautomaticallylookupthecorrect
authenticationparameters,canbefoundontheZenosswikiat
http://wiki.zenoss.org/Zensendevent_in_Zenoss_4.2.3
ThecodeissuppliedinAppendixA.
6.2 Regex in event mappings

TheRegexelementofaneventclassmappingcanbeusedtoparsethesummaryfieldof
theincomingevent,whichispresentedbytheparsingdaemon(zensyslog,zeneventlog,
zentrap).TheRegexelementusesthePythonformatforregularexpressionsandcan
usethePythonnamedgroupsyntaxtonotonlycheckforliteralstringsbutalsoto
defineregularexpressionsforvariablepartsofastring,andassociatethatvariablepart
withaname.VariablepartsofthestringarecapturedintoPythonnamedgroups
thismeansthat:
Youcanhaveoneexpressionmatchlotsofsimilarbutdifferentincomingevents
Thevariablepart(typicallybetweenthe(?Pand\S+))canbepassedtotherest
oftheeventprocessingmechanismasanamedfieldoftheevent.
Thus,intheproductshippeddropbeareventmappingfor/Security/Login/Fail,
theRegexisasfollows:
exitbeforeauth$user'(?P<eventKey>\S+)',(?P<failures>\S+)fails$:Maxauthtriesreached
(?P<eventKey>\S+)willparsethecharactersafteruser'uptothenext
singlequoteandplacethatstringintotheeventKeyfieldoftheevent.
Similarly(?P<failures>\S+)willparsethestringthatfollowsacommaand
spaceandisendedbyspaceandfails,intoaneweventattributecalled
failures.
Matchingtheliteralstringrepresentingabracketrequiresthebackslash
escapeorthebracketwillbeinterpretedasametacharacter.
TherestoftheeventsummarymustmatchtheliteraltextintheRegex;
however,othertextcanappearbeyondtheendaftertriesreached.
TheExampleboxshouldshowsasampleeventsummarythatismatched
bytheregularexpressionintheRegexbox.IfyouattempttoSavearegex
thatdoesnotmatchtheexample,theregexfieldwillbeshowninred.
FormoreinformationonPythonregularexpressions,see
http://docs.python.org/2/library/re.html.
1February2013
55
SeeFigure29foranexampleofamorespecificmapping,su_root,fortheeventclass
/Security/Su.Theregexisusedtoensurethatthesummaryhasthestring
pam_unix(su:auth):authenticationfailure;followedbysomefixedandsomevariable
elements.
pam_unix$su:auth$:authenticationfailure;logname=(?P<logonUser>\S+)
uid=(?P<uuid>\d+)euid=(?P<euid>\d+)tty=(?P<tty>\S+)ruser=(?
P<fromUser>\S+)rhost=\s+user=(?P<toUser>\S+)
Figure29:EventmappingdialoguewithRegexforauthenticationfailure
Theeventsummaryfieldcanbeparsedtogeneratenew,userdefinedfieldsfortheevent
whichwillbeshowninthedetailsoftheeventandcanbeusedinanysubsequentevent
transforms.
Additionally,theConfigurationPropertyofzEventSeverityhasbeensettoWarningfor
thismapping.
Figure30Eventdetailsforauthenticationfailureeventshowingneweventfieldscreatedbytheregex
56
1February2013
TheRegexelementisonlyusedifboththeeventClassKeyandtheRule(ifany)are
satisfied.IftheRulefails,theRegexwillnotbetested,norwillanynamedgroup,user
definedfieldsbegenerated.IfaRuledoesnotexistandtheRegexdoesnotmatch,the
userdefinedfieldswillnotbegeneratedandtheeventclassmappingtothiseventclass
willfail.Noeventtransformswilltakeplace.IfaRuledoesexistandissatisfiedbut
theRegexfailsthenanyuserdefinedfieldswillnotbegeneratedbuttheeventclass
mappingwillbesuccessfulandanymappingtransformwilltakeplace.
6.3 Rules in event mappings

TheRuleelementofaneventclassmappingusesPythonexpressionstotestany
instantiatedfieldoftheincomingeventagainstavalue.Expressionscanbecomplex
includingPythonmethodcallsandlogicalANDsandORs.Thedefaulteventfieldsthat
aredefined,aregiveninAppendixD3oftheZenossCore4AdministrationGuide.Note
thatsomeofthesefieldsarenotactuallyavailableateventmappingtimenotably
evid,stateChange,count,dedupid,firstTime,lastTimeand
eventClassMapping.
Figure31:Eventmappinglinetest,showingcomplexRuletestingeventanddeviceattributes
TheRuleelementcanalsousePythonexpressionstotestforvaluesofattributesofthe
devicethatgeneratedtheevent.Someofthemethodsandattributesthatare
availablefordevicesaredocumentedinAppendixD2oftheZenossCore4
AdministrationGuide,underthesectiononTALESexpressions(TemplateAttribute
1February2013
57
LanguageExpressionSyntaxispartofZope.ZopeistheapplicationserverthatZenoss
isbuilton).
TheRuleelementwillonlybeusediftheeventClassKeyfieldinthemappinghas
achievedamatchwiththeincomingevent.Afterthat,ifaRuleexists,itmustbe
satisfiedbeforethismapping(andhenceclass)isapplied.
6.4 Other elements of event mappings

TheExampleelementofaneventclassmappingisasamplestringthatisusefulwhen
constructingaRegex.TheRegexwillturnrediftheRegexdoesnotmatchtheExample
stringwhentheSavebuttonisused.
TheExplanationandResolutionelementsofaneventclassmappingarestringsthat
canbeconfiguredtoprovidefurtherinformationtoZenossusers.Theyappearinthe
eventdetail.Notethattheseelementscanonlybeliteralstrings;theycannotuse
eitherstandardoruserdefinedfieldsfromtheevent.
ThecombinationofeventClassKey,RuleandRegexdeterminetheeventclassthatwill
beassociatedwiththeincomingeventandwhattransforms(ifany)willtakeplace.
Theremaystillbemultiplecombinationsofthesethatsatisfyanygivenincomingevent.
Ifso,theSequencemenuisusedtodecidetheprecedenceofevaluationofmatching
eventmappings.Themappingswillbetestedfromthelowesttothehighestsequence
number.Onceamatchisfound,anysubsequentmappings(withhighersequence
numbers)willbeignored.Generally,amappingwithmorespecificmatchingcriteria
willhavealowersequencenumber.
Intheexamplesaboveforthe/Security/Suclass,thegenericsumappinghassequence
number1andthemorespecificsu_rootmappinghassequence0.
Aparticularexampleofeventmappingsthatusesequencenumbers,istheeventclass
mappingcalleddefaultmappingwhichmusthaveaneventClassKeyof
defaultmapping.Thereareatleast6mappings,allcalleddefaultmapping,outofthe
box.Eachmapstoadifferentclass.Adefaultmappingisaspecialcasethatisusedby
theeventmappingprocessifnomatchcanbefoundfortheeventClassKeyfield(note
thatiftheeventClassKeyfielddoesnotexistthennomappingatallwillbeapplied).In
thecasewhereaneventClassKeymatchisnotfound,themappingprocessreevaluates
lookingforamatchwiththespecialeventClassKeyofdefaultmapping.Itispossibleto
createnewmappings,eitherwiththenameofdefaultmappingor,indeed,witha
differentname,providedtheeventClassKeyisdefaultmapping.Thesequencenumbers
ofallsuchdefaultmappingsshouldbeadjustedtoprioritisethesedefaultmappings.
7 Event transforms
Transformscanbeusedtomodifyfieldsofanevent,createnew,userdefinedfieldsor
fieldscanberetrievedfromeventsalreadyintheMySQLdatabase.
58
1February2013
7.1 Different ways to apply transforms

YoucanhavesimpleassignmentsoffieldvaluesorsetthembasedoncomplexPython
programs.Thetransformmechanismcanbeappliedintwoways:
eventclasstransforms
eventclassmappingtransforms
PriortoZenoss2.4,aneventclasstransformwasonlyusedforeventsinserteddirectly
tothatexacteventclassbytheparsingmechanism(zenping,zenperfsnmp,
zencommand,AddEventwithEventClassspecified,etc).Ifatransformexistedinan
eventclassmappingthatwasused,theeventclasstransformwasnotused.
Zenoss2.4introducedcascadingeventtransforms.Thischangedthingsintwoways.
Givenaneventclass/Toptestwithasubclassof/T1,ifaneventarrivesthatalready
hasclass/Toptest/T1,thentheToptesttransformwillbeapplied,followedbytheT1
transform.Ifaneventarrivesthatdoesnothaveapreallocatedclassbutwhoseevent
classisdeterminedtobe/Toptest/T1,bytheRule/Regexoftheeventclassmapping,
t1,thentransformswillbeappliedintheorder:
Toptestclass>T1class>t1eventclassmapping
Itisperfectlypossibleforatransformtouseuserdefinedeventfieldsinstantiatedby
earliertransforms;however,beveryawarethatifanystatementinatransformfails
(perhapsbecauseafielddoesn'texist),thentheprocessingofthattransformwillstopat
thatpointandnofurtherstatementswillbeexecuted.Anyfurthertransformswillbe
executed(atleastuntilanerrorisreached).
AlltransformsareexecutedoncetheRuleandRegexelementsofamappinghavebeen
successfullytestedandafterdeviceandeventcontexthavebeenapplied.Thus,at
transformtime,mostofthestandardeventfieldsareavailable,exceptthosepopulated
atdatabaseinsertionstime(evid,stateChange,eventState,dedupid,count,
eventClassMapping,firstTimeandlastTime).Anyuserdefinedfieldscreatedbythe
Regexarealsoavailable.
Eventclasstransformscanbeusefulonthe/Unknownclasstoselectivelychangethe
classforeventsthatwouldotherwisebe/Unknown.
Notethatifatransformtriestoreferenceafieldofaneventthatdoesnotyetexist
(likecount)thenthatlineofthetransformandanysubsequentlineswillbeignored.
Suchanerrorwillnottriggeranyerrormessagesinthetransformdialogue.
Transformsareimplementedbythezeneventddaemonsoinspecttheendof
$ZENHOME/log/zeneventd.logtoseetheerrormessagereportingtheabsenceofthe
attribute.
AclasstransformisconfiguredfromtheActioniconatthebottomofthelefthandmenu
foraneventclass.
1February2013
59
Amappingtransformisspecifiedaspartofthesameeventmappingdialoguethat
definestheRuleandRegexfields.Ineachcase,ifthePythonsyntaxisincorrect,when
youusetheSavebutton,thenthetransformisalldisplayedinredtext,indicatingan
error.
Figure31onpage57showedaneventmappingcalledlinetestwhichincludesa
transformtocreateseveraluserdefinedeventfields,somebasedonvaluesfromthe
eventandsomewithvaluesfromthedevicethatgeneratedtheevent.Theevent
summaryfieldissettoastringconstructedfromliteraltext,standardeventfieldsand
userdefinedfields.
evt.myDevId=device.id
evt.mySnmpSysLoc=device.snmpLocation
evt.mySnmpSysContact=device.snmpContact
evt.mySnmpStatus=device.getSnmpStatusString()
evt.summary="Problemis%sondevice%s.Pleasecall%s"%(evt.summary,
evt.myDevId,evt.mySnmpSysContact)
Mostoftheuserdefinedfieldsareassignedtosimpleattributesofeithertheeventor
thedevice;forexample,device.snmpContact.Thelinebeforetheenddemonstrates
usingaPythonmethodtogetvalues;forexampledevice.getSnmpStatusString()(note
the()attheendthisisthecluethatitisamethodratherthananattribute).
7.2 Understanding fields available for event processing

Sohowdoesoneworkoutwhatattributesandmethodsareavailable?TheZenoss
Core4AdministrationGuidedocumentstheTALESEventAttributesinAppendixD3
butthisisonlyastartingpoint.
Similarly,AppendixD2documentsTALESDeviceAttributesandmethodsbutthis
informationisveryincomplete.
Whenzeneventdisprocessinganevent,strictlyitisworkingonanumberofPython
dictionariesthatmakeupaZepRawEventProxyobjectclass.Rememberfromthe
architecturesectionthatzeneventdtakeselementsfromtheraweventsqueue,processes
themandoutputstheresulttothezeneventsqueuetobefurtherprocessedbythe
zeneventserverdaemon(Figure12,Zenoss4eventarchitecture).Themessagesonthe
raweventqueue(likeallotherqueuemessages)areblobsofbinarydata.
Thereareanumberofmodulesin$ZENHOME/lib/python/zenoss/protocolsthat
manipulatethismessagedatausingGoogleprotobufsasadatainterchangeformatfor
thestructuredqueuemessagedata.
$ZENHOME/Products/ZenEvents/events2containsthreePythonfilesthatarecrucial
forunderstandingthedetailsofhowzeneventdprocessestherawevent:
60
processing.py
fields.py
proxy.py
1February2013
$ZENHOME/Products/ZenEvents/zeneventd.pyhasanumberofpipelinesthatan
eventpassesthrough.Theireffectcanbeseenbeanalysingzeneventd.logiftheDebug
logginglevelisturnedon.
Figure32EventPipelineProcessorobjectclassinzeneventd.py
processing.pycontainsthecodetoimplementeachofthepipelinestagesexecutedby
zeneventd.Therearemethodstoprocessesarawevent,adddeviceandeventcontext,
processruleandregextoestablishaneventclass,andtoperformtransforms.Thereis
alsoamethodtogeneratethefingerprintfield.
1February2013
61
Figure33EventFieldobjectclassin$ZENHOME/Products/ZenEvents/events2/fields.py
62
1February2013
$ZENHOME/Products/ZenEvents/events2/fields.pycontainsobjectclassdefinitions
for:
EventField
TheEventFieldattributesmatchupwiththebaseMySQLdatabasefieldsin
zenoss_zep.
TheActor,DetailandTagfieldsaredefinedassubclassesoftheobject
EventSummaryField
Hastheadditionalfieldsthatarepopulatedwhentheeventisinsertedinto
thezenoss_zepdatabaseevent_summarytable.
Figure34EventSummaryFieldandZepRawEventFielddefinitions
ZepRawEventField
HasthesamefieldsasEventFieldbutalsohasclear_event_classasthatis
neededbythezeneventdprocessingpipelinesasitispartoftheeventcontext.
Notethatthedefinitionsinfields.pyarenothelpfulwhendecidingwhatattributesare
availabletotransforms;thesearethefieldsonefindsinthezenoss_zepdatabase.
7.2.1 Event Proxies

$ZENHOME/Products/ZenEvents/events2/proxy.pyisthekeytounderstandingwhat
attributesareavailablewhenwritingrulesandtransforms.proxy.pyprovides
1February2013
63
translationsbetweenencodedformatsofeventsandahumanreadableJSON
(JavaScriptObjectNotation)format.
Asfaraspossible,theattributespresentedbyaproxyarethesameinZenoss4asthey
wereinpreviousversions.
Figure35EventProxydefinitionin$ZENHOME/Products/ZenEvents/events2/proxy.py
64
1February2013
AnEventProxyisseveralPythondictionaries:
Themainbodyoftheeventisadictionarycalled_event
Adetailsdictionary
An_tagsdictionary
Adictionaryfor_clearClasses
Adictionaryfor_readOnlyattributes
TherearealargenumberofPython@propertydecoratorconstructswhosepurposeisto
presentanattributeusingamethod,forexample:
@property
defdevice(self):
returnself._event.actor.element_identifier
definesanattributecalleddevicewhichisdeliveredbyamethodthatreturnsthe
valueoftheevent'sactor'selement_identifier.deviceisthefieldthatwehave(have
alwayshad)tomanipulateintransforms.
The@propertydefinitionsattheendofFigure35showsimplerdefinitionsthatreturn
thevalueofabasicfieldofanevent(usingtheEventFielddefinitionsdefinedin
fields.py).
WhenauserviewseventdetailsusingtheZenossGUIoraccessesdatafromfromthe
event_summarytableofthezenoss_zepdatabaseusingtheJSONAPI,theeventdata
presentedisanEventSummaryProxy,whichisaJSONformat.The
EventSummaryProxyinheritsfromtheEventProxybutalsohasattributesthatare
addedondatabaseinsertion:
evid
stateChange
clearid
firstTime
lastTime
count
ownerid
eventState
TheEventSummaryProxywasoriginallydesignedwithanideaofkeepingallevent
data,treatingduplicatesasmultipleoccurrenceswithintheEventSummaryProxy;
howeverthescalabilitywasnotfeasibleso,inpractisethefieldsofaneventareinthe
zero'thelementofanEventSummaryoccurrencelist.
1February2013
65
Figure36EventSummaryProxyobjectclass
proxy.pyalsodefinesaclassforZepRawEventProxywhichinheritsfromEventProxy.
TheadditionalpropertiesforZepRawEventProxyarefor_ClearClasses,_actionand
eventClassMapping.
Itistheattributesdefinedinproxy.pyfortheZepRawEventProxyobjectclassthatare
availableforuseinrulesandtransforms.
7.2.2 Event Details

Sowhathappenstoauserdefinedeventattributegenerated,say,bythevarbindsthat
comeinonanSNMPTRAP?
RememberthattheEventProxyhasanumberofdictionaries,includingadetails
dictionary.ExaminationoftheEventProxyobjectclassinproxy.pyshowsthatany
66
1February2013
fieldsthatdon'tmatchthestandardfieldsareputin<name>,<value>pairsinthe
event'sdetailsdictionary.
Figure37Processingeventdetailsinproxy.py
Theevt.detailsdictionaryisavailableasanEventDetailProxyobject(alsodefinedin
proxy.py).
1February2013
67
Figure38EventDetailProxyobjectclassinproxy.py
Toaccessthesedetailsinaruleortransformtheycanbereferredtoasevt.<nameof
detailfield>ifthenamedoesnotincludea.(dot);otherwisetousethesedetailsina
ruleortransform,theyneedtobeaccessedthroughthe_mapdictionary.
7.3 Transform examples

7.3.1 Combining user defined fields from Regex with transform
Inthisexample,wewillreturntothe/Security/Susubclassofeventsandcombine
regularexpressionsandtransforms.Theobjectiveis,forimportantdevices,toescalate
theeventseverityifausertriestosutorootbuttodecreasetheseverityifthesuevent
comeseitherfromanunimportantdeviceorifthesuistoaparticularuserid(student
inthiscase).ImportantdevicesaredeterminedbytheeventfieldDevicePriority(note
twocapitallettersinthisfieldname).Thedevicepriorityforadevicecanbechanged
fromtheOverviewmenuonadevice'sdetailspage.
ThisexampleisthesameasshowninFigure29butatransformhasbeenadded.
68
1February2013
Figure39:su_rooteventmappingwithtransform
NotethattheStatusmenuofamappinglosesanyPythonindentationsyouhave
carefullycreated!Thetransformshouldbeenteredas:
ifevt.toUser=='root'andevt.DevicePriority>2:
evt.severity=5
elifevt.toUser=='student'orevt.DevicePriority<3:
evt.severity=1
evt._action='history'
TheuserdefinedfieldtoUser,createdbytheRegex,istestedagainsttheliteralstring
'root'.TheresultislogicallyAND'edwithatestofthestandardeventfield
DevicePriorityfor>2.IftheresultisTruethenthestandardeventfieldseverityisset
to5(Critical).Rememberthatthedefaultseverityforthesu_rootmappingwassetto
WarningbythezEventSeverityeventcontextzProperty.
Intheelifstatement,ifthisconditionisTruethentheevent'sseverityissetto1(Debug)
andthezPropertyzEventActionisoverriddenbysettingevt._action='history'inthe
transform.Inthiscase,theevent'seventStateissettoClosed.
NotewithanyPythontestthatincludesmultipleclauses,thetestfailsassoonasa
conditionfailssointheifstatementifevt.toUserisnot'root'thenevt.DevicePrioritywill
notbeevaluated.Performancecanbeimprovedbycarefulconsiderationandorderingof
suchtests.
7.3.2 Applying event and device context in relation to transforms

Eventcontext(zEventSeverity,zEventAction,zEventClearClasses)isappliedthrough
theConfigurationPropertiesmenuofaneventclassoreventclassmapping.
DevicecontextcomprisestheeventfieldsprodState,Location,DeviceClass,
DeviceGroups,SystemsandDevicePriority.ipAddressandthemonitor
(collector)responsiblefortheeventalsotendtobebracketedwiththedevicecontextbut
theselatterfieldsareinformationreceivedontheincomingevent,ratherthanthe
devicecontextdatathatislookedupintheZopedatabase.
1February2013
69
Thefollowingdevice_contexteventmappingexampledemonstratestheorderinwhich
devicecontext,eventcontextandthemappingtransformareapplied.
Createaneweventsubclass,Device_context,underthe/Skillsclass.
Createamapping,device_context,forthisneweventclass.Ensurethatthe
eventClassKeyisdevice_context.SettheRegextotheliteralstring:
Thisisadevicecontextevent
SetaRuleasfollows(allononeline):
getattr(evt,'Location','')=="/Kandersteg"andgetattr(evt,'_action','')
=="status"and'/Skills'notinevt._clearClassesandgetattr(evt,
'severity','')>4andnotevt.component
UsingtheConfigurationPropertiesmenuforthemapping,setthezEventSeverityevent
contextvaluetoError(4),zEventActiontohistoryandzEventClearClassesto/Skills.
Testthemappingwithazensendevent(allononeline):
zensendeventdgroup100r1.class.example.orgsCriticalk
device_contextThisisadevicecontextevent1
Thetesteventsetthedevicefieldtogroup100r1.class.example.orgwhichisincludedin
theLocationcalled/Kandersteg.TheeventClassKeyshouldbesettodevice_context,the
componentfieldshouldbeblankandtheeventClassshouldbeblank.
Figure40:CombiningaRule,contextandatransformforthedevice_contexteventmapping
TheRuledemonstratesthePythongetattrfunctiontotest:
70
Theevt.Locationfieldsetbydevicecontext,whichshouldevaluateTRUEatRule
timeie.devicecontexthasbeenapplied
Theevt._actionfieldthatissetbyeventcontexttohistory.Thetestshownabove
actuallyevaluatesTRUEshowingthateventcontexthasnotbeenappliedat
Ruletime.
1February2013
Similarly,theevt._clearClassesfieldtestevaluatesTRUEshowingthatevent
contexthasnotbeenapplied.ThePythonsyntaxforcheckingevt._clearClassesis
alittledifferentasthisattributeisdefinedasaPythonlistratherthanastring.
Theevt.severitystartsat5inthegeneratedeventandeventcontextsetsitto4.
ThistestevaluatesTRUEconfirmingthateventcontexthasnotbeenapplied.
Theevt.componentmustbeblank(thenullstringevaluatestothebooleanFalse)
Notethatthesyntaxforthelastfieldofthegetattristwosinglequotestosupply
anulldefault
Insummary,theRuleandRegexshouldevaluatesuccessfullyandthetransformwillbe
applied.
Thetransformdemonstrates:
Changingtheevt.severityfieldagainitwouldhavebeenmodifiedfromthe
originalvalue(5)downto(4)whentheeventcontextwasappliedafterRuleand
Regexprocessing.Thetransformchangesitto3.
Changingtheevt.componentfieldisinteresting.Rememberthatthefingerprint
dedupidfieldincludesthecomponent.Althoughtheraweventdidnotincludea
componentfield,thefingerprintisgeneratedafterthetransformasthededupid
intheeventdoescontainthecomponent.
Severaluserdefinedvariablesarecreated.Theevt.myClearClassesline
demonstratesthatalluserdefinedfieldsappeartobeoftypestringbut
evt._clearClassesisdefinedasaPythonlist.Youcannotassign
evt.myClearClassestosomethingoftypelistunlessyouusethejoinfunctionto
sticktogetherthelistelementsbackintoastringtype.
Theuserdefinedfieldsdemonstratethatbothdevicecontextandeventcontext
havebeenappliedbytransformtime
8 Testing and debugging aids

8.1 Log files
8.1.1 zeneventd.log
Devicecontext,eventcontext,rule,regexandtransformsareallappliedbythe
zeneventddaemon.Italsoconstructsthededupidfingerprintfield.Seetheevent
processingpipelinecodeforzeneventdinFigure32onpage61.
Turningupthedebugflagin$ZENHOME/etc/zeneventd.confto10(Debug)providesan
opportunitytotracktheprogressofeachofthestagesinthispipelinein
$ZENHOME/log/zeneventd.log,notingthattheeventgainsmorefieldsasprocessing
continues.
1February2013
71
zeneventd.logisalsotheplacetolookforproblemswitheventprocessing.Evenwiththe
usualdebuglevelof20(Info),rule,regexandtransformproblemsarehighlighted.
SearchforWARNINGinthelog.
Thefollowingextractshowsatransformattemptingtochangeevt.Location(which
appearsnottobeallowed).Notethatalthoughthemessageisdefinitelyhelpful,its
ideasaboutlinenumbersarewayout!
2012122010:02:01,923WARNINGzen.Events:Errorprocessing
transform/mappingonEventClass
/Skills/Device_context/instances/device_context
Problemonline475:AttributeError:can'tsetattribute
Transform:
0evt.Location='/Taplow'
1evt.severity=3
2evt.myProdState=evt.prodState
3evt.myDeviceClass=evt.DeviceClass
4evt.myDeviceGroups=evt.DeviceGroups
5evt.mySystems=evt.Systems
6evt.myAction=evt._action
7evt.myClearClasses=''.join(evt._clearClasses)
WithZenoss4,youwillalsoreceiveaneventfromtheZenossserverwithsimilar
information(andequallycreativelinenumbers!).WithversionsofZenosspriorto4
therewasnowarningeventandalltheeventprocessingwasperformedbyzenhubso
zenhub.logwastheplacetosearchforerrors.
8.1.2 zeneventserver.log
ThezeneventserverdaemoniswritteninJava.Thismeansthaterrormessagesare
difficulttocomprehendin$ZENHOME/log/zeneventserver.logwithoutanintimate
knowledgeoftheJavacode.
Whatisusefultohelpunderstandingofthearchitectureistoinspectthislogaround
daemonstartup.
72
1February2013
Figure41:zeneventserver.logshowingdaemonstartup
InFigure41lineshighlightedinyellowshowEventManagerconfigurationparameters
thatcanbecheckagainsttheADVANCED>Settings>Eventsmenu.
Maximumarchivedays:1000
Startingeventageingatinterval:60000milliseconds(s)
Startingeventarchivingatinterval:60000milliseconds(s)
Startingdatabasetableoptimizationatinterval:60minutes(s)
LineshighlightedingreenshowoperationsassociatedwiththeMySQLdatabaseand
theassociatedLuceneindexes.
1February2013
73
Figure42:EventManagerparametersthatmatchwithzeneventserver.logstartuplog
LineshighlightedinredareinteractingwithRabbitMQAMQPsystem.Thefirst
sectionshowszeneventserverconnectingtotheMQsubsystem;ifthisisunsuccessful
thenmanyoftheZenossdaemonswillfail.
Thesecondsectionshowsthethreadsstartinguptoconsumethevariousqueuesthat
zeneventserverprocesses.
zenoss.queues.zep.zenevents
zenoss.queues.zep.modelchange
zenoss.queues.zep.heartbeats
zenoss.queues.zep.migrated.summary
zenoss.queues.zep.migrated.archive
Notethatyouwouldnotexpecttoseezeneventserverworkingon
zenoss.queues.zep.raweventstheconsumerofthatqueueisthezeneventddaemon.
Lineshighlightedinlightbluearesubsequent,periodicoperationsbyzeneventserver
performingmaintenanceontheMySQLdatabase.Thelogshowsaneventtable
partitionbeingprunedeveryhourandanewonebeingcreated,asasectionofevents
areagedintotheevent_archivetable.
74
1February2013
8.1.3 Other log files

Otherlogfilesthatmayhaveabearingoneventsare:
zenhub.log
interactionsbetweendaemons )moreusefulprior
event.log
problemsseenbyevent.log
zenperfsnmp.log
issueswithperformancedataandthresholdevents
zenwinperf.log
zencommand.log
zensyslog.log
daemonthatreceivessyslogevents
zeneventlog.log
daemonthatreceivesWindowsevents
zentrap.log
daemonthatreceivesSNMPTRAPs
)toV4foreventissues
8.2 Using zendmd to run Python commands

ZenossprovidesaPythoncommandlineinterface,zendmd,wherecodefortransforms
canbetestedoutandtheattributesandmethodsavailablecanbeexplored.
Notecarefullytheindentationofstatements.Pythonisveryparticularabout
indentationtointerpretstructuresuchasforloops.Itdoesn'tmatterhowmanyspaces
youindentthebodyoftheforloopbutitmustbeindentedfromtheforlineandeach
lineinthemainbodyofthatforloopmusthavethesameindentation.Thebodyofafor
loop,insideaforloop,wouldindentfurtherandsoon.
Youshouldrunzendmdasthezenossuser.ThissectionisnotsupposedtobeaPython
tutorial;thatsaid,hereareacoupleoftrickswithzendmd.
Notethatthesetechniquesforaccessingeventshavechangedsubstantiallybetween
previousversionsofZenossandZenoss4.
8.2.1 Referencing an existing Zenoss event for use in zendmd

Ifyouwanttoexploretheattributesandmethodsavailableforaneventorthedevice
thatgeneratedtheevent,usingzendmd,youneedawaytoreferenceanevent.When
executingatransform,theseobjectsaremadeavailabletoyouautomaticallyastheevt
variableandthedevicevariablebutinazendmdtestenvironmentyouneedtosupply
these.
WithearlierversionsofZenosstherewasamethodontheZenEventManager,
getEventDetailFromStatusOrHistory,whichtookasaparameterthestringvalueofa
uniqueevidanddeliveredanEventDetailobject(seeFigure43).
Tofindtheevid,simplydisplayanappropriateeventintheEventConsole,bringupthe
detaileddata,andcutandpastetheevidvalueintothestatementinzendmd.
1February2013
75
Figure43:UsingzendmdtosettheevtvariabletoanexistingZenosseventZenosspriortoV4
WithZenoss4,itisalittlemorecomplex.Wereallyneedtogetbacktothe
ZepRawEventProxyformattotesttransformcode,butthatisnolongeravailablethe
datafromtheraweventqueueisgone.
WhatwedohaveaccesstoistheeventintheMySQLdatabase;howeverwedon'twant
itwithdatabasestyleattributes,wewantEventProxyattributes.
76
1February2013
Figure44:UsingzendmdtoretrieveaneventfromtheMySQLdatabase,converttoan
EventSummaryProxyandextractvariousfields
$ZENHOME/Products/Zuul/facades/zepfacade.pyprovidesanumberofutilitiesto
accessdatafromthezenoss_zepdatabaseandmanipulateit,typicallyprovidingJSON
formatdata.
Figure44demonstratesusingzendmdtoaccesseventsintheMySQLdatabase,convert
themtoEventSummaryProxyformatandthenprintoutvariousfields.
zep=getFacade('zep')
providesaccesstothezenoss_zepdatabase
evt=zep.getEventSummary('000c29d9f87b94fb11e2494936a92109')
RetrievestheeventwiththespecifieduuidtheresultisinJSON
rawevt=EventSummaryProxy(from_dict(EventSummary,evt))
TheEventSummaryProxyclasstakesaprotobufstyleeventasparameter,
nottheJSONstyleeventwecurrentlyhave.Usefrom_dicttoconvert
fromJSONtoprotobuf
rawevt.device
standardattribute
rawevt.myLineNum
attributefromdetails
REMEMBERthatthisisanEventSummaryProxy,notaZepRawEventProxysoyou
haveaccesstofieldsthatarenotavailableattransformtime(likecount,
firstTime,...)
evt
theJSONformatevent(dictionary)
TheJSONstyleeventsareveryhardtoreadasshownabove.zendmdunderstandsthe
pprintmethodtoprettyprintcomplexstructures.Itcanbeusefultocapturetheoutput
ofpprint(evt)intoafileandthenusethevieditor%techniquetohelpmatchopening
andclosingbrackets.
1February2013
77
Figure45Firstpartofzendmdpprint(evt)commanddisplayingsummaryeventinJSONformat
RememberthatFigure45andFigure46areshowingtheJSONstyleevent,notthe
EventSummaryProxythatdeliverssuitableattributesfortransformmanipulation.
78
1February2013
Figure46Secondpartofzendmdpprint(evt)commanddisplayingsummaryeventinJSONformat
8.2.2 Using zendmd to understand attributes for an EventSummaryProxy

AnEventSummaryProxyisanobjectclassrepresentingaZenosseventitisaPython
dictionarydatatypeadatastructureof<key>,<value>pairs.Toseewhatkeys
(attributes)areavailable,usethemethodshowninthefollowingfigure.Builtin
methodsstartingwithadoubleunderscorearedeliberatelyexcluded.
1February2013
79
Figure47:Usingzendmdtoprinteventattribute<key><value>pairs(partiallisting)
Thesearetheprimaryeventfieldsthatareavailabletouseinatransform
(rememberingtoalsoexcludethosethatdon'texistatraweventtimeeg.count,
firstTime,eventState,...).
Notethatsomeofthedictionaryelementsarethemselvesdictionarieseg.details.To
findoutwhatthedetailsattributesare,seeFigure48.RememberfromFigure38,that
theEventDetailProxyclasshasan_mapdictionarywithname,valuepairsinit.
80
1February2013
Figure48zendmdtodisplayeventdetailsdictionaryname,valuepairs
ThegetmethodofEventDetailProxydeliversvalueswhenthatitemisasingle,scalar
value.Iftheitemhasmultiplevalues,alistforexample,thenthegetmethodbreaksas
shownaboveonthezenoss.device.systemsattribute.Notethatitgetsawaywiththe
zenoss.device.groupsattributebecause,althoughadevicemaybeinmultiplegroups,in
thiscasethedeviceisonlyinasinglegroup,whereasitisamemberoftwoSystems.
ThisisalsoechoedintheEventDetailsoftheZenossGUI.
1February2013
81
Figure49ZenossGUIEventDetailsshowingoneinstanceofzenoss.device.groupsand2instancesof
zenoss.device.systems
Ifaneventdetailsattributeisnotascalar,usethegetAllmethodratherthantheget
method.Forexample:
>>>printlist(rawevt.details.getAll('zenoss.device.systems'))
[u'/Test',u'/Real']
>>>
AlsonoteinFigure48thatuserdefineddetailattributescansimplybereferredtoas
rawevt.mySummaryorrawevt.mySnmpSysLocbutyoucannotrefertodetailfieldsthat
containa.(dot)inthiswaythusexcludingthedefaultdetailsattributes(thosestarting
withzenoss.)andexcludingSNMPTRAPvarbindfieldsthattypicallycontainadot;use
thegetandgetAllmethodstoaccesssuchdetailfields.
82
1February2013
8.3 Using the Python debugger in transforms

AverypowerfulaidwhendebugginganyPythonistousethePythonDebugger,pdb.
Seehttp://docs.python.org/2/library/pdb.htmlfordetaileddocumentation.
pdballowsyoutobreakexecution,displaythestateofobjectsandtheirvaluesandstep
throughthecode.Whenusedintransforms,thismeansrunningzeneventdinthe
foregroundindebugmode(sodefinitelynotatechniqueforuseinproduction).
Whenusingpdbtoexaminetransforms,itisnoteasytostepthroughthetransformcode
(usingstostepornfornext)asyouendupnestedmanylayersdeepinthemethodsof
thezeneventdcode;howeveritisveryusefultoexaminethestateoftheevent(evt)and
alsoexplorethedevice(device).
Ifyouaredoingthis,youmaywishtoreducetheZenosssystemtoaminimumsetof
daemonstoavoideventsfromlotsofothersources.
If$ZENHOME/etc/DAEMONS_TXT_ONLYexiststhentheonlyZenossdaemonsthat
willbemanipulatedbyazenossstart/zenossstop/zenossstatuswillbethoselistedin
$ZENHOME/etc/daemons.txt.Aminimumsetofdaemonswouldbe:
zeneventserver
zopectl
zeneventd
zenhub
zenjobs
zenactiond
WhenyouhaverestartedZenoss,gotoADVANCED>Settings>Events,scrolltothe
bottomofthepageandclickClear.Thispreventstheheartbeatfromperiodically
checkingallthosedaemonsthatarenowdownandgeneratingheartbeatevents..
Toputabreakpointatthestartofatransform,addthefollowingline:
importpdb;pdb.set_trace()
Stopthezeneventddaemonandstartitintheforegroundindebugmode:
zeneventdstop
zeneventdrunv10
Generateaneventthatwilltriggerthetransform;forexample:
zensendeventdzen42.class.example.orgsErrorklinetestplinetesttestline24
Inthezeneventdforegroundwindowyoushouldseeapdbprompt.Youshouldnowhave
accessto:
evt
aZepRawEventProxyobject
device
aDeviceobject
1February2013
83
Figure50:pdbdialogueinzeneventdforegroundgeneratedbypdb.set_trace()intransform
Figure50demonstratesexploringsomeoftheattributesofbothevtanddevice.Note
thatenteringasimplecarriagereturnrepeatsthepreviouspdbcommand.
cinpdbcontinuesexecution.
ToseelegalattributesandmethodsfortheDeviceobject,examinetheDeviceclass
definitionin$ZENHOME/Products/ZenModel/Device.py.
pdbdoesnothavethepprintmethodseeninzendmdbutitdoeshaveanequivalentpp
utility.Forexample,toprintallprimaryeventfields,excludingbuiltinmethods,use:
pp[xforxindir(evt)ifnotx.startswith('__')]
84
1February2013
Figure51:Usingpdbtoprettyprintallprimaryeventfields
Toshowdetailfields:
ppevt.details._map.keys()
1February2013
85
Figure52:Usingpdbtodisplaydetaileventfields
Toprintascalarvalueforadetaileventfield,try:
(Pdb)printevt.details.get('zenoss.device.device_class')
/Server/Linux
(Pdb)printevt.details.get('mySummary')
ThisisNOTagoodnews/badnewseventtestline31
(Pdb)
Toprintanonscalar(alistforexample):
(Pdb)printlist(evt.details.getAll('zenoss.device.systems'))
[u'/Test',u'/Real']
Anattempttoprintalldetailfieldnamesandvaluesmightbe:
pp[(v,evt.details.get(v))forvinevt.details._map.keys()]
***Exception:Exception(u'Detailzenoss.device.systemshasmorethanone
valuebuttheoldeventsystemexpectsonlyone:
<google.protobuf.internal.cpp_message.RepeatedScalarContainerobjectat
0x5e01ef0>',)
(Pdb)
Thiscomesupagainsttheproblemdescribedinthezendmdsectionwheretheget
methodfailswithnonscalarvalues.Apartialcircumvention,giventheknowledgethat
noneoftheuserdefinedvariablesarenonscalar,wouldbe:
(Pdb)pp[(v,evt.details.get(v))forvinevt.details._map.keys()ifnot
v.startswith('zenoss.device')]
[('mySummary',u'ThisisNOTagoodnews/badnewseventtestline31'),
('eventClassMapping',u'/Skills/linetest'),
('line_num',u'31')]
(Pdb)
PerhapsabettersolutionistoacceptallvaluesaslistsandusethegetAllmethod,
whichthenworksforalleventdetailsname,valuepairs.
(Pdb)pp[(v,list(evt.details.getAll(v)))forvinevt.details._map.keys()]
[('mySummary',[u'ThisisNOTagoodnews/badnewseventtestline31']),
('eventClassMapping',[u'/Skills/linetest']),
('zenoss.device.location',[u'/Taplow']),
('line_num',[u'31']),
(u'zenoss.device.ip_address',[u'192.168.10.42']),
('zenoss.device.groups',[u'/Skills1st']),
('zenoss.device.device_class',[u'/Server/Linux']),
86
1February2013
('zenoss.device.production_state',[u'1000']),
('zenoss.device.priority',[u'3']),
('zenoss.device.systems',[u'/Test',u'/Real'])]
(Pdb)
9 Zenoss and SNMP

9.1 SNMP introduction
TheSimpleNetworkManagementProtocol(SNMP)definesManagementInformation
Base(MIB)variablesthatcanbepolledtoprovideperformanceandconfiguration
information.TheSNMPstandardalsoprovidesforagentstosendeventstoamanager.
Version1ofSNMPdefinestheseasTRAPs;versions2and3ofthestandardcallsthem
NOTIFICATIONs(ZenosssupportsallthreeversionsofSNMP).BothMIBvariables
andTRAPs/NOTIFICATIONsuseObjectIdentifiers(OIDs)todenotedifferent
variablesandevents.
SNMPTRAPsaredistinguishedbytheirEnterpriseObjectId(OID),thegenericTRAP
numberandthespecificTRAPnumber.
Natively,OIDsaredefinedasstringsofdotteddecimalsthatrepresentapaththrougha
treebasedhierarchy,wheretherootofthetreeis1andrepresentstheISO
organisation;ithasasubbranch,3,whichrepresentsorganisations(org);ithasasub
branch,6,whichrepresentstheUSDepartmentofDefense(dod);ithasasubbranch,1,
whichrepresentsinternet,andsoon.Thus,allOIDsstartwith1.3.6.1.
Thereisastandard,MIB2,whichdefinesanumberofvariablesthateverySNMP
capabledevicemustsupport;thesearelargelysimple,networkrelatedvariables,such
asinterfaceInOctets.InadditiontoMIB2,therearealargenumberofstandardised
MIBsdefinedinRequestForComment(RFC)documents;anexamplewouldbeRFC
1493definingthebridgeMIB.
ThethirdcategoryofMIBsareknownasEnterpriseSpecific,whicharespecifictoa
particularvendor'sparticularagentforexample,theCiscoFirewallMIB.Enterprise
specificMIBsoftenincludedefinitionsofEnterpriseSpecificTRAPs,inadditiontoMIB
variables.
MIBsourcefilestranslatedotteddecimalOIDsintomoremeaningfultext.MIBfiles
areavailableformanystandards(liketheHOSTRESOURCESMIB)and,typically,any
supplierwhogeneratestheirownenterprisespecificMIBvariablesandTRAPs,should
makeavailableasourceMIBfiletoaidthistranslation.
SNMPagentstypicallycomeaspartofthebaseOperatingSystem(Windows,Unix,
Linux,CiscoIOS);howevertheymaynotbeactivatedautomaticallyandwillrequire
someconfiguration.SomeagentssupportlittlemorethanMIB2;otherssupportawide
rangeofstandardMIBsandenterprisespecificMIBs.
TheSNMPcommunicationprotocolvariesdependingontheversionofSNMP.Versions
1and2(strictly2c)useacommunitynamestringasanauthenticationmechanism
1February2013
87
betweenSNMPmanagerandagent.Managersmustbeconfiguredwiththecorrect
communitynamestouseforanagent;SNMPagentsmustbeconfiguredforwhich
manager(s)areallowedaccesstothem,andwhichSNMPmanager(s)tosendTRAPsto.
SNMPV3ismorecomplextoconfigurebutprovidesfacilitiesforstrongauthentication
onSNMPpacketsandforencryptionofdataifsodesired.
InadditiontorequestingMIB2variables,ZenosswilltrytoaccessthestandardHost
ResourcesMIBtogetprocessinformationforservermachines.Itwillalsoattemptto
accesstheWindowsInformantMIBforallWindowsserversystems,inordertogetCPU
andfilesysteminformation.TheInformantMIBisafreeextensionsubagentandMIB
availablefromInformantathttp://www.wtcs.org/informant/index.htm.Notethatthe
baseWindowsSNMPagentshouldbeinstalledandconfiguredbeforeinstallingthe
Informantextension.
OnceSNMPagentsareconfiguredwithcommunitynameandTRAPdestination,a
simplewaytotestthemissimplytorecycletheSNMPagent(indeedtheywillneed
recyclingafteranyconfigurationchanges).OnaWindowssystem,usetheServices
utilitytostopandstartSNMP;onaLinuxsystem,/etc/init.d/snmpdrestartwill
usuallysuffice.IneithercaseyoushouldeitherseeacoldstartTRAP(genericTRAP
0)orawarmstartTRAP(genericTRAP1)intheZenossEventConsole.Theevent
detailsshouldshowthecommunitynamefromtheTRAPpacket.
AnothergoodwayofgeneratingTRAPsistoforceanauthenticationTRAP(generic
TRAP4).Aneasywaytodothisistousethesnmpwalkcommandwithabad
communityname.Ifthecommunityispublic,forahostsystemcalledzenoss,try:
snmpwalk -v 1 -c public zenoss system
snmpwalk -v 1 -c fred zenoss system
test with good community

to generate several TRAP 4's
9.2 SNMP on Linux systems

MostLinuxsystemscomewithsomeflavourofthenetsnmpagent(formerlytheUCD
agent).ManyLinuxdefaultconfigurationsforthisagentprovideverylimitedSNMP
access.Thesnmpagentconfigurationistypicallycalledsnmpd.conf;thelocationofthis
filevariesbetweendifferentLinuximplementationsbut/etc/snmpisacommonchoice.
YouwillneedrootauthoritytomanipulatetheSNMPconfigurationanddaemon.
88
1February2013
Figure53:snmpd.conffornetsnmpagent
Figure53showsansnmpd.confthatconfiguresforSNMPV1andSNMPV2c,providing
accesstotheentireMIB(theallview).TRAPs,includingAuthenticationTRAPs,are
senttothezen42host.ThesysContactandsysLocationvariablesareset(theseare
retrievedasstandardbyaZenossmodelerpoll).
Thesnmpdagentshouldbestoppedandrestartedafteranychangestosnmpd.conf.
/etc/init.d/snmpdstop
/etc/init.d/snmpdstart
1February2013
89
AsimplewaytotestthatTRAPsareconfiguredistogenerateanAuthenticationTRAP.
snmpwalkv1cpubliczen42system
snmpwalkv1cfredzen42system
testwithgoodcommunity
togenerateseveralTRAP4's
Whereavailable,theV3oftheSNMPstandardshouldreallybeusedasitprovides
strongauthentication(notjustacommunitynamethatpassesoverthenetworkinclear)
anditalsoprovidesdataencryptionifdesired.Althoughslightlyhardertosetup,itis
nottooonerous.Ontheagent,auseridmustbegeneratedwithparametersfor
authenticationandencryption(privacy),specifyingtheencryptionalgorithmandthe
encryptionpasswordtobeused.
#ForSNMPV3
#Uncommentnext5lines
com2secsnmpv3testlocalhostdummycontext
com2secsnmpv3testzen42dummycontext
groupsnmpv3groupusmsnmpv3test
#accesssnmpv3group""usmauthexactallallall
accesssnmpv3group""usmprivexactallallall
rwuserjane
#
#rwuserjanecreatedbySTOPPINGSNMPDandrunning
#netsnmpconfigcreatesnmpv3userafraclmyeaxfraclmyexXDESAMD5jane
#/var/lib/netsnmp/snmpd.confismodifiedwith(hidden)encryptionkeyand
#rwuserjaneisaddedtothisfile(/etc/snmp/snmpd.conf)
#testwithfollowingifnoprivacy(dataencryption)
#snmpwalkv3aMD5AfraclmyealauthNoPrivujanezen42system
#or,withencryption
#snmpwalkv3aMD5AfraclmyeaXfraclmyexlauthPrivujanezen42system
#
#Restartthesnmpddaemon
#NotethatonCentOSnetsnmpdevelmustbeinstalledtoprovide
#netsnmpconfig
ZenossmustalsobeconfiguredtohavematchingSNMPV3parametersforthisagent.
Figure54:ConfigurationPropertiesforagentwithSNMPV3
90
1February2013
NotethatthestandardsnmpwalkcommandfromtheCommandicondoesnotworkfor
SNMPV3butitisrelativelyeasytocreateanewcommandfromADVANCED>
Settings>CommandswhichrunsanappropriatesnmpwalkwiththeSNMPV3
parameterssubstituted.
Figure55:CreatinganewCommandoptiontorunsnmpwalkV3
NotethatdifferentimplementationsofnetsnmpondifferentOperatingSystemsmay
workslightlydifferently.Forexample,OpenSuSEdoesnotneedthenetsnmpdevel
packageandtherwuseriscreatedinaseparatesnmpd.confunder/usr/share/snmp
(whichiscreatedautomaticallyifitdoesn'texist).
9.3 Zenoss SNMP architecture

9.3.1 The zentrap daemon
zentrapistheZenossdaemonthatprocessesincomingSNMPTRAPs.Bydefault,
zentrapwillsitonthewellknowSNMPTRAPportofUDP/162thiscanbe
reconfigured,ifrequired.BothSNMPversion1TRAPsandSNMPversion2
NOTIFICATIONsaresupported.
zentrapprocessingisimplementedbythePythonprogram
$ZENHOME/Products/ZenEvents/zentrap.py.
1February2013
91
Figure56:zentrap.pypart1checkingforextra0andprocessingofgenericTRAPs
zentrap.pyparsestheincomingSNMPProtocolDataUnit(PDU)toretrievethe
enterpriseOID,thegenericTRAPnumberandthespecificTRAPnumber.
ThealgorithmforinterpretingincomingTRAPEnterprisefieldshaschangedseveral
timesovertheyearsbecausesomeagentshaveanextra0definedintheirMIBwhich
theydonotsendonanactualTRAP(seethecommentsinthecodeinFigure56).In
Zenoss4.2,thealgorithmfirsttriestofindaMIBintheZODBdatabasethat
correspondswiththeincomingTRAP,withtheextra0;ifthisfails,thenapartial
matchisattemptedwithouttheextra0(notethatthecommentinthecodeis
inaccurate).Eitherway,theoidfieldoftheeventissettotheconcatenationofthe
enterpriseandthespecifictrapnumber,withorwithoutthe0inthemiddle,depending
ontheoutcomeoftheoid2namelookupfunction.
ThegenericTRAPs(0through5)aretranslatedtostringssuchassnmp_coldStart.
usingtheeventTypedictionary.ForspecificTRAPs(genericTRAP6),eventTypedelivers
theconcatenationoftheenterpriseOIDandthespecificTRAPnumber;forexample,
1.3.6.1.4.1.123istheenterprise,thespecifictrapnumberis1234,soeventTypedelivers
92
1February2013
1.3.6.1.4.1.123.1234.AnyvariablesoftheTRAP(varbinds)arealsoparsedoutintoOID
/valuepairsiftheMIBprovidesthistranslation.
Theoid2namefunctionlooksupintheZODBdatabasetoseeiftranslationsare
availablefortheenterpriseOID,thespecificTRAPnumberandthevarbindidentifiers,
totranslatefromdotteddecimalnotationtotextualstrings.
Figure57:zentrap.pypart2eventfieldsettings
Thefollowingeventfieldsarethenset:
component
leftblank
eventClassKey
settoeventType
eventGroup
trap
severity
summary
snmptrapfollowedbyeventType
community
settocommunitynamestring(thisisauserdefinedfield)
firstTime
settotimestamp
lastTime
settotimestamp
monitor
settoCollectorthatreceivedtheTRAP
9.4 Interpreting MIBs

TohelpdecodeSNMPTRAPenterpriseOIDsfromdotteddecimal(suchas.
1.3.6.1.4.1.8072.4.0.2)intoslightlymoremeaningfultext(likensNotifyShutdown)the
zenmibcommandcanbeusedtoimportbothstandardMIBsourcefiles(suchas
SNMPv2SMIwhichdefinesstandardOIDs)andvendorspecificMIBs.Thebase
directoryforMIBsinlaterversionsofZenossis$ZENHOME/share/mibs.
1February2013
93
ThezenmibcommandwithoutparameterswilltrytoimportallMIBfilesthatarein
$ZENHOME/share/mibs/site.AspecificMIBfilecanbeprovidedasaparameter;the
commandshouldeitherberunfromthe$ZENHOME/share/mibs/sitedirectory(in
whichcaseafullpathnameisnotrequiredandthefileisexpectedtobeinthat
directory)orafullyqualifiedpathnamecanbespecified.
9.4.1 zenmib example

Tohelpunderstandthezenmibcommand,hereisaworkedexample.Itusestheagent
fornetsnmpwhichistheagenttypicallyshippedwithaLinuxsystem.Theenterprise
OIDfornetsnmpis.1.3.6.1.4.1.8072.
1. Recycleanetsnmpagentwith/etc/init.d/snmpdrestart.Inadditiontothe
genericcoldstartTRAP,youshouldalsoseeTRAP.1.3.6.1.4.1.8072.4.2.This
comesfromthenetsnmpenterprise(.1.3.6.1.4.1.8072).
2. TheactualTRAPisdefinedinthefileNETSNMPAGENTMIB.txtwhichshould
beshippedaspartoftheOperatingSystemnetsnmppackage.Typicallythis
MIBfilecanbefoundunder/usr/share/snmp/mibs.FindandexamineNET
SNMPAGENTMIB.txt.Strictly,theMIBfileisdefiningSNMPV2
NOTIFICATIONs,ratherthanSNMPV1TRAPssearchinthefileforthe
stringNOTIFItofindtherelevantlines.AlsonotetheIMPORTSsectionatthe
topoftheMIBfile,especiallytheimportfromNETSNMPMIB.Thisindicates
thatNETSNMPAGENTMIBisdependentonalsoloadingNETSNMPMIBin
additiontosomestandardSNMPv2MIBs.
Figure58:MIBfileforNET_SNMP_AGENTMIBshowingIMPORTSsection
94
1February2013
Figure59:MIBfileforNETSNMPAGENTMIBshowingnotifications
3. InspecttheNETSNMPMIB.txtfileandsearchforthestringNotifications.You
shouldseethatthenetSnmpNotificationPrefixisdefinedasbranch4beneath
netSnmpandthatnetSnmpNotificationsisbranch0under
netSnmpNotificationPrefix.
Figure60:MIBfileforNETSNMPMIBshowingOIDsfornotificationhierarchy
4. AtthetopofthefileyoushouldfindthelinesthatdefinetheenterpriseOIDfor
netSnmp.
1February2013
95
Figure61:MIBfileforNETSNMPMIBshowingOIDfornetSnmp
5. Betweenthem,thesefilesgiveus(almost)theOIDfortheunknownTRAPwe
received1.3.6.1.4.1.8072.4.0.2.
1.3.6.1.4.1isthestandardiso.org.dod.internet.private.enterprisesOID
whichisdefinedintheIMPORTfromSNMPv2SMI
netSnmpis{enterprises8072}
netSnmpNotificationPrefixisbranch4undernetSnmp
netSnmpNotificationsisbranch0undernetSnmpNotificationPrefix
nsNotifyShutdownisNOTIFICATION2undernetSnmpNotifications
6. NotethatsomeSNMPagents(includingthenetsnmpagent)areknowntoomit
the0fromtheTRAPthattheyactuallygenerate,whichiswhytheoidfieldinthe
detailsoftheeventdoesnotquitematchtheOIDspecifiedintheMIBfile.
7. $ZENHOME/share/mibscontainsfivesubdirectoriesfourofwhichcontain
sourceMIBfilesprovidedwithZenoss(iana,ietf,irtf,tubs).Thefifthdirectory,
site,iswhereotherMIBstobeimported,shouldbeplaced.
8. ThesitedirectoryshouldcontainZENOSSMIB.txtwhichisprovidedasstandard
todefineTRAPsthataresentbytheNotificationfunction(thiswillbediscussed
later).
9. CopyNETSNMPAGENTMIB.txttothesitedirectory.Atthispointdonotcopy
NETSNMPMIB.txt;wewilldemonstratetheerrormessagewhencorequisite
MIBsarenotavailable.
96
1February2013
10. ToimportintoZenossuse:
zenmib run -v10
NET-SNMP-AGENT-MIB.txt
11. YoushouldseethattheNETSNMPAGENTMIB.txtfileisimportedbutwith
errors;thereshouldbeaWARNINGmessagesayingtheNETSNMPMIBcould
notbefound.
Figure62:ImportingNETSNMPAGENTbeforeprerequisitesinplace
12. NoteintheRunningsmidumplinethatthestandardSNMPv2prerequisitefiles
thatwerelistedasIMPORTsinFigure58haveautomaticallybeenlocatedin
$ZENHOME/share/mibs/ietf;howeverultimately0nodesand0notifications
wereloaded.
13. FromtheZenossGUI,usetheADVANCED>MIBsmenu.TheNETSNMP
AGENTMIBislistedbut,assuggested,ithasnoOIDMappingsandnoTRAPs.
1February2013
97
Figure63:MIBGUIwithimportedNETSNMPAGENTMIBbutnoOIDsorTRAPs
14. CopyNETSNMPMIB.txtto$ZENHOME/share/mibs/siteandrerunthe
zenmibcommand.
Figure64:SuccessfulimportofNETSNMPAGENTgivencorrectprerequisites
15. ThereisaDEBUGlinenotingthattheNETSNMPAGENTMIBisalready
imported;thisisnotanissue.ThisimportwilloverwriteanyexistingMIBofthat
name.
98
1February2013
16. NotethattheRunningsmidumplinealsolooksinthesitedirectoryandfindsthe
prerequisiteNETSNMPMIB.txtinadditiontofindingthestandardSNMPv2
MIBsintheietfdirectory.45nodesand3notificationshavebeenloaded.
17. ReturntotheZenossGUIandrefreshtheMIBsmenu.ClickingontheNET
SNMPAGENTMIBshouldnowdisplay45OIDMappingsandthreeTRAPs,
includingnsNotifyShutdown.
18. RestartthesnmpagentontheZenosssystemwith/etc/init.d/snmpdrestart.You
shouldseeaneventintheEventConsolethatnowcontainssnmptrap
nsNotifyShutdowninthesummaryfield,ratherthansnmptrap
1.3.6.1.4.1.8072.4.2.Ifthisdoesnotwork,youmayneedtorecyclethezentrap
daemon.YoucandothiswiththeGUIfromtheADVANCED>Settings>
Daemonsmenuor,asthezenossuserfromacommandline,usezentraprestart.
19. ZenosshasimplementedanumberofchangesinthewayMIBsareinterpreted.
RememberfromFigure60thatnetSnmpNotificationsisbranch0under
netSnmpNotificationPrefix;however,someagentsomitthis0whentheyactually
generateTRAPs.Zenoss4.2hasprocessingin
$ZENHOME/Products/ZenEvent/zentrap.pytotryandinterpretactualTRAPs
bothwithandwithouttheextra0.TheeventconsoleshowedaneventwithOID
1.3.6.1.4.1.8072.4.2fortheoriginalevent;comparetheeventdetailsofthe
originaleventwiththenewonethatcontainsnsNotifyShutdowninthesummary
field.Youshouldfindthattheneweventhasanoidfieldof1.3.6.1.4.1.8072.4.0.2.
20. Examine$ZENHOME/Products/ZenEvent/zentrap.py(aroundline580inZenoss
Core4.2)toseethecodethathandlesthisextra0digitprocessing.
9.4.2 A few comments on importing MIBs with Zenoss

ThereareafewquirkstodowithimportingMIBsintoZenossandthequirkshave
changedsubtlyoverseveralversionsofZenoss.
NotethatMIBsimportedintoZenossareonlyusedforinterpretingSNMPV1
TRAPsandSNMPV2NOTIFICATIONsforuseintheEventsubsystem.
AlthoughtheOIDsareimportedfromMIBs,theycannotbeusedforMIB
browsingorwhenworkingwithOIDsforperformancesampling,thresholdingand
graphing.
AlwaysensureyoudoMIBworkasthezenossuser.
Bydefault.zenmibrunv10willtryandimporteverythingunder
$ZENHOME/share/mibs/site.Thev10simplyaddsmoreverboseoutput.
zenmibshouldcheckintheotherdirectoriesforprerequisites.
WheneveryouhaveimportedaMIB,checkattheGUIontheMIBspage.You
shouldseethenameoftheMIBandyoushouldusuallyseenonzerocounts
undertheOIDMappingsandTRAPdropdownmenus.
1February2013
99
TherearesomeMIBsthatwillresultinzerocounts,forexampleiftheMIB
sourcefileonlydefinesSNMPstructureanddoesnotincludethedefinitionfor
anyOIDsorTRAPs.
Checktheoutputofthezenmibcommandcarefullyforerrormessages.
IfOIDtranslationsdonotappeartobeworkingineventsafterimportingaMIB,
recyclethezentrapdaemonfromtheADVANCED>Settings>Daemonsmenu
or,asthezenossuser,runzentraprestart.
IfeventmappingsandtransformsarebuiltassumingthataMIBhasbeen
imported,forexample,testingtheeventClassKeyfieldforenterprises.8072.4.2,
andthatMIBisthenremovedfromtheZopedatabase,thenthemappingand/or
transformwillfail.EspecialcareshouldbetakenwithanyZenPackthatimports
MIBsastheremovaloftheZenPackislikelytoremovethoseMIBs.
Zenoss4.2(and3.2.1)appeartohaveatimingbugthataffectssomeinstallations.
Thesymptomisthatzenmibapparentlysatisfiesitschecksbutthenreports
Loaded0MIBfile(s).TheonlysolutionIhavefound(whichappearstowork
perfectly)istouseazenmib.pyfromaZenoss3.1installation.Thisfilebelongsin
$ZENHOME/Products/ZenModel.
Figure65:OccasionaltimingbugwithZenoss4.2.Replacezenmib.pywithaZenoss3.1version.
9.5 The MIB Browser ZenPack

ThereisanexcellentcommunityZenPackavailabletoperformMIBBrowsing.Thisis
notdirectlyrelevanttoTRAP/NOTIFICATIONprocessing,butitisusefulfor
investigatingMIBswithaviewtobuildingSNMPperformancetemplates.
Itcanbedownloadedfromhttp://wiki.zenoss.org/ZenPack:MIB_Browser.Unfortunately
thisZenPackkeepsgettingbrokenbynewversionsofZenoss.Ifyoufollowthelinkto
100
1February2013
DownloadforZenossCore3.1,thisdoesindeedworkforCore3.1;thisversionshouldbe
downloadedandmodifiedforCore3.2;forZenoss4.2,followtheDownloadforZenoss
Core4.2linkandperformthesamemodificationswhicharedocumentedinthe
commentsifyoufollowthedocumentationlinkhttp://community.zenoss.org/docs/DOC
10321.BasicallyyourevertthelaterCorefilesbacktothe3.1levelofcode.
ItprovidesaMIBbrowsertoexploreanyOIDthathasbeenloadedintoZenoss,along
withatestfacilitytosnmpwalkaconfigurabledevicetoretrievevaluesforanyselected
partoftheMIBtree.NotethatitonlysupportsSNMPV1.
TheMIBBrowserZenPackchangestheADVANCED>MIBsmenuandcreatesaMIB
Browserlefthandmenu.SelectingtheMIBBrowsermenuoffersasimilarlayouttothe
OverviewmenubutitintroducesnewiconsalongsidethenameofaMIB.Clickingthe
iconstartstheMIBBrowseragainsttheselectedMIB.
Figure66:StartingtheMIBBrowserclickagainstthemagnifiericonforagivenMIB
Inordertoperformansnmpwalk,youneedtoprovideatargetdeviceandanSNMPv1
communitynameundertheTestSettingstab.Arighthandmouseclickthenprovides
thesnmpwalkmenuagainsttheleveloftheMIBtreethatyouarepositionedon.
TheOIDDetailswindowgivesthesameinformationyouwouldseeifyouinspectedthe
MIBsourcefile.UsethiswindowtocutandpasteintoOIDfieldsinperformance
templates.
1February2013
101
Figure67:UsingtheMIBBrowserZenPack
9.5.1 Modifying Zenoss Core 4.2 to make the MIB Browser ZenPack work
1. Downloadtheeggfileandinstallinthenormalway.Itshouldinstallwithno
errors.
zenpackinstallZenPacks.community.mib_browser1.2py2.7.egg
zenhubrestart
zopectlrestart
2. Changeto$ZENHOME/Products/ZenUI3/browser.Backupbackcompat.py,
navigation.zcmlandbackcompat.zcml.
3. Inbackcompat.py,commentoutthelinesattheenddefiningMibClass.Ifthere
arealsosimilarlinesforMibNodeandMibNotification,commentthemouttoo.
#defMibClass(ob):
#id='/'.join(ob.getPhysicalPath())
#return'/zport/dmd/mibs#mibtree:'+id
4. Innavigation.zcml,aroundline233,changetheurllinetobe
url="/zport/dmd/Mibs/mibOrganizerOverview".Notecarefullythecasesensitivity
onmibs/Mibs.
url="/zport/dmd/mibs"
+url="/zport/dmd/Mibs/mibOrganizerOverview"
5. Inbackcompat.zcml,aroundline260commentoutlinesfortheadapterfor
Products.ZenModel.MibOrganizer.MibOrganizer.Ifadapterstanzasalsoexistfor
MibNode,MibNotificationandMibModule,commentthemouttoo.
6. Changedirectoryto$ZENHOME/Products/ZenModel/skins/zenmodeland
backupviewMibModule.pt.
102
1February2013
7. ModifyviewMibModule.py.Changethetemplateinthefirstline.
<tal:blockmetal:usemacro="here/templates/macros/page2">
+<tal:blockmetal:usemacro="here/page_macros/oldnew">
8. YouwillneedtocompletelyrestartZenossandmakesureyourbrowsercache
iscleared.
9.6 Mapping SNMP events

ZenossprovidessomeeventmappingsforSNMPTRAPsoutofthebox.Asdiscussedin
anearliersection,thefile$ZENHOME/Products/ZenModel/data/events.xml
configuresallthestandardmappingssosearchingthisfileforSNMPprovidesinsight
fordefaultcustomisation.
MostSNMPTRAPsmaptotheZenoss/Unknowneventclass.Thereareoneortwo
exceptionsforsomegenericTRAPssuchasLinkUp(3),LinkDown(2)andthe
AuthenticationTRAP(4).Eventfieldsthatareautomaticallypopulatedbythezentrap
processingincludesummary,eventClassKeyandagent.Theeventdetailsshowsthe
communityandoidName/Valuepairs.Notethatthevalueoftheoidfieldisalways
innumericformat,nottranslatedthroughanimportedMIB.
Thismeansthat,typically,theeventonlymapsontheEventClassKey,whichis
interpretedbyzentrap.pyasenterprises.<enterprisenumber>.<specifictrap>ifthe
SNMPv2SMIhasbeenimportedor1.3.6.1.4.1.<enterprisenumber>.<specifictrap>
otherwise.Thesummaryfieldwillbesnmptrap<enterpriseOID><specifictrap>and
theagentfieldwillbesettozentrap.Thesetranslationsassumethattheenterprise
specificMIBhasnotbeenimported.
TRAPsandNOTIFICATIONsmayhaveoneormoreTRAPvariables(varbinds).These
varbindsappearintheeventdetailswherethefieldnameisthevarbindOID(possibly
translatedthroughaMIBlookup)andthecorrespondingfieldvalueisthevalueofthat
varbind.
EventclassmappingscanbedevisedwithvariousRule,RegexandTransformelements,
toparseouttheintelligencefromSNMPTRAPsandeithercreatenewuserdefined
eventfieldsormodifyexistingfields(suchasevt.summary).
NotethateventmappingsthatparseoutSNMPOIDsandvarbindsmustbeawareof
whethertherelevantMIBshavebeenimported,ornot.IfaMIBisimported,OID
mappingbasedonmatchingdotteddecimalnotationwillfailastheMIBOID
translationshappenbeforeeventmapping.
9.6.1 SNMP event mapping example

InordertointerpretenterprisespecificTRAPs,mappingsareusuallyrequired.Often
anactionormodificationisrequired,effectivelybasedonwhatenterprisetheTRAP
camefrom(Cisco,netsnmp,...),soasubclassofeventsarerequiredthatinheritsome
1February2013
103
commoncharacteristicsbutsomeeventdetailsvarydependingontheexactenterprise
specificTRAPnumber.
ManyenterpriseTRAPsalsoincludeseveralvarbindsthatneedtobeinterpretedand
processed.
Inthemappingexampleshownhere,threesmallscriptsareusedtogenerateTRAPs
fromthe1.3.6.1.4.1.123enterpriseoneforeachofspecificTRAPs1234,1235and1236.
ThefirsttwohaveasinglevarbindwhosestringtypevalueisHelloworld4,wherethe
endnumberis4or5;thethirdscriptgeneratesaTRAPwith2varbinds.Notethateach
ofthevarbindsexhibittheextra0behaviour,ie.thevarbindfieldwillbe
1.3.6.1.4.1.123.0.1234.
#!/bin/bash
#
#Generateasampletrap
#Sendtrapusingthesnmptrapsuppliedwithnetsnmp
#TraphereisEnterprise1.3.6.1.4.1.123,trap1236
#EnsureyouchangethelineforMANAGERtobeyourZenossServer
#
#Uncommentnextlineforextradebugging
#setx
MANAGER=zen42.class.example.org
HOST=zen42.class.example.org
ENTERPRISE=.1.3.6.1.4.1.123
GENTRAP=6
SPECTRAP=1236
TRAPVAR1=.1.3.6.1.4.1.123.0.12361
TRAPVAR2=.1.3.6.1.4.1.123.0.12362
VARBIND1="Helloworldvarbind161"
VARBIND2="Helloworldvarbind262"
TIMESTAMP=1
#
/usr/bin/snmptrapv1cpublic$MANAGER$ENTERPRISE$HOST$GENTRAP
$SPECTRAP$TIMESTAMP\
$TRAPVAR1s"$VARBIND1"\
$TRAPVAR2s"$VARBIND2"
#
1. Withoutanymapping,whengen_mytrap_1234.shisrun,itwillmaptothe
/Unknowneventclass.
2. CreateaneweventsubclassSnmpundertheclass/Skills.
3. Mapthe1234eventbyselectingitandusingtheReclassifyanEventicon.
Choose/Skills/Snmpfromthedropdownselectionbox.Leavetherestofthe
EventClassMappingparametersasdefaultsfornow.Thismeansthattheevent
onlymapsontheeventClassKey,whichtranslatesto<enterpriseOID>.<specific
trap>.Themappingnameisautomaticallyassignedthenameofthe
eventClassKey(1.3.6.1.4.1.123.1234ifSNMPv2SMIisnotimported;
enterprises.123.1234ifitis).Referbacktothesnippetofthezentrapcodein
Figure57formoreinformationontheparsingoftheTRAPintoeventfields.
Checkthatyoureventclassmappingworks.
104
1February2013
Fromhere,ensurethattheSNMPv2SMIMIBisimported;thusanyTRAPenterprise
field(andhenceeventClassKey)willstartwithenterprises,not1.3.6.1.4.1.Inmost
cases,thesamewillapplytothenamefieldofaTRAPvarbind.
Thenextstepistointerpretthevarbind.EachoftheTRAPsgeneratedbythetest
scriptscomefromtheEnterprise1.3.6.1.4.1.123andthenameofeachofthevarbinds
alsostartswith1.3.6.1.4.1.123thus,inthedetailoftheinterpretedevent,thevarbind
namefieldswillstartwithenterprises.AtransformwillextractthatpartoftheOID
afterenterprises.Itwillalsosubstitutethevalueofthevarbindintotheevent
summary.
Attransformtime,strictlytheeventisaZepRawEventProxyobject,whichhasadetails
dictionary(anEventDetailProxyobject)aspartofit(referbacktoFigure35,Figure37
andFigure38).Alsorememberthatalthoughonecanrefertodetaileventfieldsby
name(eg.evt.line_num)iftheyaresimplenames,youcannotusethismethodifthe
detailnamehasadotinit.
Ifoneisinterestedinthevaluesofsuchfields,thegetorgetAllmethodsareneeded.
Sincethegetmethodfailswithanattributeerrorifthevalueisnonscalar,itissaferto
assumethatallvaluesmaybenonscalarandusethegetAllmethod.
InversionsofZenosspriorto4,atransformtointerpretTRAPvarbindswouldlooklike
this:
for attr in dir(evt):
if attr.startswith('enterprises.123.'):
evt.myRestOfOID=attr.replace('enterprises.123.','')
evt.myFieldValue=getattr(evt,attr)
evt.summary=(evt.summary + + evt.myFieldValue)
ThiswillfailwithZenoss4astheneweventstructuredoesnotdeliverdetailevent
fieldsasaresultofdir(evt).AZenoss4versionwouldbe:
forattrinevt.details._map.keys():
ifattr.startswith('enterprises.123'):
evt.myFieldValue=''.join(list(evt.details.getAll(attr)))
evt.summary=(evt.summary++evt.myFieldValue)
1. Thefirstlinecyclesthroughtheeventdetailsattributenames.
2. Thestartswithlineensuresthattransformsonlytakeplaceforattributesthat
startwithenterprises.123ie.varbindattributefields.
3. NotethatthereplacelineisreplacingtheOIDspecified,withthenullstring
thesyntaxafterthecommaissinglequotesinglequote.Therestoftheattribute
(ie.the0.1234bit)iskeptandbecomesthevalueoftheuserfieldmyRestOfOID.
4. Theevt.myFieldValuelineusesthegetAllmethodincasethevarbindvalueis
nonscalar.Toconcatenatetheresultinglistwiththeevt.summarystring,the
listisconvertedintoastringwiththejoinfunction.
1February2013
105
5. Runningthescripttogeneratea1234TRAPshouldnowgenerateanevent
with:
Theeventmappedtothe/Skills/Snmpclass
Thesummaryfieldshouldsaysnmptrapenterprises.123.1234Helloworld
4.
TheEventDetailsshouldshowvaluesforcommunity,oid,myFieldValue
andmyRestOfOID,inadditiontothedefaultvarbindname/valuepairof
enterprises.123.0.1234/Helloworld4
6. Runningthescripttogeneratea1235TRAPwillstillgenerateaneventwith
the/UnknownclassastheeventclassmappingisbasedontheeventClassKeyof
enterprises.123.1234.
Sofar,weareonlymatchingasingleSNMPTRAPwiththeeventClassKeyfield.The
objectiveistomapalleventsfromtheenterprise1.3.6.1.4.1.123.WithSNMP,you
oftenwanttoapplyatransformtoseveralsimilareventswhichareonlydistinguished
bythelaterpartsoftheOIDfield.Thetestscriptsallgenerateeventswhose
eventClassKeystartwith1.3.6.1.4.1.123.buttheydifferinthelastnumber.
ARulewillbeusedtomatchallappropriateevents.However,aRuleisonlyinspected
iftheeventClassKeyhasalreadymatchedsuccessfullyandwehavenocontroloverthe
eventClassKeythatissetbyzentrap.py.Thus,thedefaultmappingconceptwillbe
used.
1. ClearallSNMPeventsforyourZenosssystem.
2. Edittheenterprises.123.1234mapping.
IntheRuleboxputevt.eventClassKey.startswith('enterprises.123.')
ChangetheNameofthemappingtoenterprises.123
IntheTransformboxput:
evt.summary=evt.summary+"defaultmapping"+evt.myFieldValue
Savethemappingaway
3. Runthegen_mytrap_1234.shscriptandthegen_mytrap_1235.shscript.
4. ChecktheeventsintheEventConsole
5. Youshouldfindthatthe1234TRAPmapssuccessfullybutthe1235TRAP
doesn't.Thisisbecausetheinitialtestforeventclassmappingchecksthe
eventClassKeythatisstillsettoenterprises.123.1234sotheprocessingnever
evengetsasfarascheckingourRule!Notethatwehavenocontroloverhowthe
eventClassKeyfieldispopulatedbytheeventprocessingmechanismitisparsed
outforusbyzentrap.py(seeFigure57again).
106
1February2013
6. ThisiswherethemagicstringofdefaultmappingcanbeusedintheEventClass
Keyfield.SettheEventClassKeytodefaultmapping(Noteitmustbealllower
case).IftheprocessofmappinganeventcannotfindamatchfortheEventClassKey
thenitwillrerunthemappingprocesswithanEventClassKeyofdefaultmapping.
7. Savethemapping.
8. ChecktheSequencemenu.ThereareseveralmappingsthatallmaponanEvent
ClassKeyofdefaultmapping.Chooseasuitablesequencenumberforthenew
defaultmapping.Savethemapping.
9. Clearexistingevents.Rerunbothscripts.Checkthatbotheventsnowmap
correctly.
Figure68:MappingforSNMPTRAPwithrule,transformandeventClassKeyofdefaultmapping
Thetesteventsusedsofar,onlyhaveonevarbind.WhatifyourTRAPhasseveral
varbindsandyouwanttouseinformationfromeachofthem?Thescript
gen_mytrap_1236.shgeneratesaspecificTRAP1236,withtwovarbinds:
varbind1
1.3.6.1.4.1.123.0.12361
Helloworldvarbind161
varbind2
1.3.6.1.4.1.123.0.12362
Helloworldvarbind162
Runningthescriptgen_mytrap_1236.shshouldresultinaneventthatmapstothe
/Skills/Snmpclass,withthemyFieldValueandmyRestOfOIDfieldsmatchingthedatain
thelastvarbindthatwasprocessed,andthesummaryreflectingthedatafromallvarbinds.
Toprovideamoreeleganttransformsolutionwhereyoudonotknowifadetailvalueis
scalarornot,thePythontry/exceptconstructcouldbeused:
try:
evt.myFieldValue=evt.details.get(attr)
except:
evt.summary=evt.summary+"defaultmapping"+evt.myFieldValue
Checktheendof$ZENHOME/log/zeneventd.logfordebugginghelp.
1February2013
107
10 Event Triggers and Notifications

10.1 Zenoss prior to V4
PriortoZenoss4,thereweretwowaysofautomatingresponsestoevents.
UserAlertingRules
Emailtousers
Pagingtousers
EventCommands
Scriptsruninthebackground
Theuseractionswereconfiguredonaperuserorperusergroupbasis.Thismeantthat
similaremails/pagesformanyusersorgroupshadtobecreatedindividually;therewas
noeasywaytocopyanAlertingRulefromoneusertoanother.
EventCommandsusedaverysimilarmethodtodefinewhenacommandshouldbe
automaticallyruninthebackground.
AlertingRulesandEventCommandswereexecutedbythezenactionddaemonwhich
processedanyrequestsevery60seconds.Duplicateeventsdidnotcreatemultiple
actionsandthiswashandledbythealert_statetableoftheMySQLeventsdatabase.
ThisisprobablytheareathathaschangedmostforusersofZenoss4.
108
1February2013
10.2 Zenoss 4 architecture

Zenoss4hascompletelychangedthearchitectureoftheMySQLeventsdatabase.There
isnoalert_statetableinthezenoss_zepdatabase.zenactiondisstillresponsiblefor
executingactionsbutithasbeencompletelyrewrittenandtakesinputfromaRabbitMQ
queuecalledsignalwhichisfedbythezeneventserverdaemon.Thismakesalerting
muchmoreresponsive.
Figure69:Zenosseventarchitectureactionprocessinginbottomright
AlertingRuleshavegoneinZenoss4andarereplacedbytheconceptsof:
Triggers
Notifications
Triggersdefinewhatcausesaresponse.ANotificationistheresponse.Thisisbetter
inseveralways.Bothmechanismsaredecoupledfromusersandfromeachother.
Notificationsnowincludeeventcommandsaswellasthetraditionalemailandpaging,
andSNMPTRAPshavealsobeenaddedasanotificationaction.
TriggerandNotificationSubscriptionsobjectsaredefinedintheZopedatabase(though
theTriggerisastubobjectthatisusedformanagingpermissionsanddoesnotcontain
theactualtriggerrules).
ThereisanewEVENTS>TriggersmenufordefiningbothTriggersandNotifications.
1February2013
109
10.3 Triggers
Triggersdefineunderwhatconditionssomeactionshouldtakeplace.Theyaredefined
fromtheEVENTS>Triggersmenu.Usethe+icontoaddanewtrigger;doubleclick
anexistingtriggertomodifyit.
Figure70:CreatinganewTrigger
Notethatbydefault,anewtriggeriscreatedasEnabledbutwithanillegalrule!
DevicePriorityequalswithoutavaluewillcauselotsoferrorsinzeneventserver.log.
WhencreatingtheTriggerrule,combinationsorlogicalANDsandORscanbeused(the
allandanyoptions).Usethe+icontoaddfurtherconditions.Allthestandardevent
attributesareavailabletoselectfromthedropdownboxes.Userdefinedeventfieldsare
notavailableherealthoughitispossibleinZenPackstoprovideforuserdefinedevent
fields.
UnlikeearlierversionsofZenoss,itisalsopossibletonestcriteriatobuildupthe
overallrule.Usetherightmosticontoaddanestedclause.
Figure71:ATriggerrulewithnestedclause
TheUserstaboftheTriggerdefinitionistocontrolwhocanmanipulatethisTrigger.
Bothglobalandspecificrolescanbeallocated.Userswhohaveeithertheglobal
ManagerorZenManagerrolewillautomaticallyhavemanageaccesstotriggers,aswill
thetriggerowner(creator).
110
1February2013
Figure72:TriggerUserstabforglobalanduserspecificroles
NotethatthisUserstabhasnoeffectonwhoreceivesanyrelatedNotifications.
10.4 Notifications
NotificationsarecreatedfromthesamemenupathasTriggers.Anameanda
notificationtypearetheinitialrequirements.
NotethatacarefulnamingconventionforTriggersandNotificationsmakesthe
environmentmucheasiertoworkwith.
Figure73:CreatinganemailNotification
TheNotificationiscreatednotEnabledbydefault.Youcanchoosewhethertosend
goodnewsClearnotificationsandwhethertodelayaNotification(usefulforless
criticaleventsthatmayselfclear).Eventscanbesentrepeatedlyoronlyontheinitial
occurrence.
1February2013
111
Figure74:Notificationdetails
AkeyfieldforaNotificationistheTriggerthatcausestheNotification.Configured
Triggerswillbeofferedinthedropdownbox.MakesureyouselectaTriggerandclick
AddifyousimplyselecttheTriggerandthenSUBMITtheentireNotification,the
Triggerwillnotbesaved.
DependingontheNotificationtypeselectedwhentheNotificationiscreated,the
Contenttabwillvary;theothersremainthesame,thoughforCommandandTrap
notificationstheSubscribertabisnotrelevanttowhethertheactiontakesplaceas
thesearebackgroundactionsnotuserrelatedactions.
ThedifferentNotificationactionsareencodedin
$ZENHOME/Products/ZenModel/actions.py.
112
1February2013
Figure75:$ZENHOME/Products/ZenModel/actions.pyimplementsNotificationactions
10.4.1 email Notifications

TheContenttabforemailallowsyoutocustomisetheemailsubjectandbody,using
standardfieldsfromtheevent,usingTALESexpressions(TemplateAttribute
LanguageExpressionSyntax,fromZope)toreferencefieldsoftheevent,evt.See
AppendixDoftheZenossAdministrationGuideformoredetails.Notethatyoumust
useTALEStheevt.<eventfield>syntaxusedinmappingrulesandtransformsdoesnot
workineventcommands.TALESsyntaxtakestheform:
${evt/<event field>}
Alsoseesection2.6oftheZenossCore4AdministratorsGuide.
1February2013
113
Figure76:TheContenttabofaNotificationpart1
AlsonotethatpreviousversionsofZenossprovidedaccesstothedevvariabletoaccess
attributesofthedevicethatcausedtheevent.Thedevvariableisnolongerlegalforuse
inNotificationcontent.
SeparatedefinitionscanbeprovidedfortheproblemandclearingNotifications.
ThebottomoftheNotificationconfigurationpanelallowsyoutooverridedefault
configurationsformailhostparameters.
114
1February2013
Figure77:NotificationContentwithmailserverparameters
TheseparametersarespecifiedgloballyfromtheADVANCED>Settings>Settings
menu.
1February2013
115
Figure78:Defaultsettingsformailserverandpaging
DoensurethattheFromAddressforEmailssettingsarelegalformailservers.A
difficultscenariotodebugiswhereemailnotificationsneverarrivebecausetheyare
discardedbyamailserverbecauseoftheFromaddress.
Thethirdtab,Subscribers,ontheNotificationdefinitionpaneldefineswhoreceivesthe
notification.Inaddition,thispanelalsoserversasimilarpurposetotheUserstabfor
TriggersinthatitdefineswhoisallowedtomanagetheNotificationdefinition.Unlike
Triggers,ifnosubscriber(userorusergroup)isspecified(andexplicitlyAdded)thenno
emailwillbereceived.Itisnotnecessarytospecifyanymanagementrolesthough.
116
1February2013
Figure79:SubscriberstoNotifications
1February2013
117
10.4.2 Page Notifications
Figure80:Usersettingsshowingemailandpageparameters
APagenotificationisverysimilartoemail,simplyprovidingaContenttabtospecifya
MessageformatandaClearMessageformat.Aswithemail,theevtvariableisavailable
forparametersubstitution.Thecommandusedtosendpagemessagesisthatspecified
globallyfromADVANCED>Settings>Settings(seeFigure78).Theindividual
recipientcomesfromthoseusers/groupsspecifiedintheSubscriberstabwhomusthave
theirpagerdetailsconfiguredonthatusershomepage(thisisalsowhereauser'semail
addressisspecified).
10.4.3 Command Notifications

TheContenttabforaCommandNotificationspecifiesabadnewsandagoodnews
command,atimeparameterforhowlongthecommandmayrununtilitisdeemedto
havefailed,andenvironmentvariablescanalsobespecifiedas<variable>=<value>.
ThelatterisusefulasinpastversionsofZenossacommonissuewastocreateanEvent
Commandbutforgettosourceanynecessaryenvironmentvariablesinthescript.Since
thescriptisrunbyzenactiond,ithasverylittledefaultcontextinwhichtorunsothings
like$ZENHOME,$PATHwerenotautomaticallyset.
118
1February2013
Figure81:ACommandNotification
Notethattousetheseenvironmentvariablesinascriptyouneedtoescapethedollar
withadollareg.$$ZENHOME.Multipleenvironmentvariablearesemicolonseparated
andyoudonotincludethedollarwhenyouspecifythenameoftheenvironment
variable.
Alsonotethat,althoughasubscriberisnottypicallyrequiredastheCommand
notificationisabackgroundscript,duetoabugInCore4.2,environmentvariableswill
beignoredunlessthereisasubscriber.Itisnotoneroustosetupadummyuser
subscriberasacircumventiontothisissue.
CommandNotificationsmaybesimplebuiltinshellcommandsasshownaboveorthey
canbecomplexscriptsinotherlanguages,providedtheycanbeexecutedfromashell
environment.Again,standardfieldsfromtheeventcanbesubstitutedusingTALES
expressions.Noteinthefigureabovetheuseofbackticsaroundthedatecommandto
runthedatecommandbeforeaddingtheoutputoftheenvironmentvariablesandthe
goodnews/badnewsmessage.
1February2013
119
10.4.4 TRAP Notifications

SNMPTRAPnotificationsarenewwithZenoss4.ItwaspossibletocreateTRAP
forwardingscenariosusingEventCommandsinthepastbutthisabilityisnow
standard.TheContenttabinthiscaseconfigurestrapdestination.
Figure82:Trapnotification
ThetrapdestinationmayeitherbearesolvablenameoranIPaddress.
NotethatwithZenossCore4.2thereisabugthatmeansselectingSNMPv1resultsin
noTRAPbeingissued,eventhoughzenactiond.logreportsthataTRAPhasbeen
successfullysent.
TheTRAPisdefinedin$ZENHOME/share/mibs/site/ZENOSSMIB.txt.Itisasingle
TRAPwithmanyvarbindsthatarepopulatedwiththefieldsoftheoriginalevent.It
wouldbegoodpractisetoimportthisMIBintoaZenossserverthatisreceivingsuch
notificationTRAPs.
120
1February2013
Figure83:TrapresultingfromaNotificationTRAPwithouttheZENOSSMIB.txtimported
Thusthevarbindnameswillbetranslatedtosomethingmorehelpful.
Figure84:TrapresultingfromaNotificationTRAPwiththeZenossMIBimported
CarefulinspectionoftheTRAPwiththeZenossMIBimportedrevealsanomissionin
theMIB;varbind8forthemessagefieldisnotdefinedsoitshowsintheeventdetails
withthenamezenTrapDef.8.
1February2013
121
NotethattheversionofZENOSSMIB.txtshippedwithCore4.2.3hasbeenmodified
fromthe4.2versioninsuchawaythatitdoesnotimportcleanly(therearenon
printingcharactersinthefile).Foradescriptionoftheproblemandaworkingfile,see
http://jira.zenoss.com/jira/browse/ZEN5060.
10.5 Notification Schedules

AnyNotificationtypemayhaveoneormoreschedulesassociatedwithit.Theseare
effectivelyMaintenanceWindows(andareindeedimplementedbythesamecodeas
MaintenanceWindows).Theyallowdifferentresponsestotakeplaceatdifferenttimes.
IfnoNotificationScheduleexiststhentheNotificationisalwaysactive.
Figure85:Notificationschedule
ThescheduleiscreatedasnotEnabledbydefault.Typicallytheschedulewillrepeat
overcertainperiodsseeFigure85.
Withdebugloggingturnedonforthezenactionddaemon,thestartofaNotification
schedulecanbeclearlyseen.
AnInfoseverityeventiscreatedwhenanyMaintenanceWindowstartsanditiscleared
bytheClearseverityeventgeneratedwhentheMaintenanceWindowends.
122
1February2013
Figure86:zenactiond.logshowingthestartofaNotificationSchedule
Figure87:EventsforMaintenanceWindowsstarting/stopping
10.6 Using zenactiond.log

AllNotificationsareprocessedbythezenactionddaemon.Todebugissuesandalsoasa
learningaid,itishelpfultosetthedebuggingleveltoDebug(logseverity10),
rememberingtorecyclezenactiond.
Inspectingzenactiond.logprovidesagoodinsightintohowzenactiondprocessesevents
fromtheRabbitMQsignalqueueandthenteststhemagainsttheconfigured
Notifications.
1February2013
123
TheTriggersareprocessedbythezeneventserverdaemontodecidewhattoplaceonthe
signalqueue.ThereareobviouslydifferentsignalsforeachNotificationtype.
Aprocessingcyclestartswithaprocessingmessageentry(highlightedingreen)in
Figure88.
Notificationsarecheckedastowhethertheyareenabledornot(highlightedinblue).
Figure88:zenactiond.logprocessingasignalagainstvariousNotifications
Theeventthatgeneratedthissignalwasa/Security/Sueventandshouldtriggerboth
thezen42_email_traps_suNotificationandthezen42_trapNotification.InFigure88the
logshowszen42_email_traps_subeingdiscarded(highlightedinyellow);thisisbecause
thesignalmessageiskeyedtoaTRAPNotificationtype,notanemailone
(unfortunatelyzenactiond.logdoesnotshowthisdetail).
Thematchwithzen42_trapishighlightedinredwherethecheckingforanotification
schedulewindowcanalsobeseen.Thestartofthenotificationactiontogeneratethe
TRAPisalsohighlighted.
Oncetheactioniscompleted,zenactiond.logshowssimilariterationsthroughthe
Notificationslistwithaseparatesignalmessage,wherethezen42_email_traps_su
Notificationisselectedandactionedandthezen42_trapNotificationisdiscarded.
124
1February2013
10.7 The effect of device Production State

TheProductionStateofadevicecanbeusedtocontroldifferentmanagementaspects
ofasystem.ProductionStateforadeviceisconfiguredonthedevice'shomepage
OverviewandmaybemodifiedbyMaintenanceWindowsconfiguredforadevice,device
class,Group,SystemorLocation.
WhenconfiguringaMaintenanceWindow,theproductionstateisdefinedbothfor
duringthewindowandthestatetoreturnto,wherethelatteristypicallyOriginal.
Figure89:MaintenanceWindowfordeviceclass/Server/LinuxforfirstSundayinthemonth
Chapter8oftheZenossCore4AdministrationGuidedescribesthedifferentProduction
Statesandtheeffectthatthesehave.Threedifferenttypesofmanagementare
defined:
Monitoring
pingpollingandeventgeneration
Alerting
generatingalerts(emails,pagers,commands,traps)
Dashboard
whethertoincludeintheDeviceIssuesportlet
Inpractise,anythingtodowithNotificationsiscontrolledbythefiltersintheTrigger.
IfnoProductionStatefilterisconfiguredthentheNotificationwillrun,bydefault.
AdeviceProductionStateofProductionwillresultineventscontributingtotheDevice
IssuesportletoftheZenossDashboardandallmonitoringwilltakeplace.
AProductionStateofDecommissionedshouldresultinallmonitoringceasing;hence,all
eventsgeneratedbyZenosswillceaseandnorelatedNotificationswillbegenerated;
however,externallygeneratedevents(fromsyslog,externalTRAPs,Windowsevent
logs)willcontinuetobereceivedandrelatedNotificationswillbegeneratedunlessa
triggerfilterexcludingonProductionStateexists.Thedevicewillnotberecordedinthe
DashboardDeviceIssuesportlet.NotethattheoverallStatusicononadevice'sStatus
pagewillturngreen!
1February2013
125
AnyProductionStateotherthanProductionwillresultinthedevicenotbeingincluded
ontheDashboardDeviceIssuesportlet.
TheonlyProductionStatethatautomaticallystopsallmonitoringisDecommissioned;
however,thezPropertyofzProdStateThresholdcanbesetaspartofthe
ConfigurationPropertiesofadeviceordeviceclass.Thisvariablecontrolsthe
ProductionStatevaluebeneathwhichallmonitoringceases.Bydefaultthisvalueis
300whichmeansthatsettingaProductionStateofMaintenancedoesnotpreventping
andsnmpmonitoring.IfyouwanttopreventallmonitoringforMaintenancestate
devices,changethezProdStateThresholdvalueatthetopdeviceclasslevelto301.
11 Accessing events with the JSON API

DuringthelifeofZenoss3,theJSONAPIwasintroducedasameansofaccessingdata
withinZenoss.Insomeways,itissimilartousingthezendmdPythonenvironmentand
inmanycasesitreflectsthesamecallsavailableinzendmd,butagreatadvantageof
theJSONAPIisthatitcanbeusedremotelyfromtheZenossserveranditrequiresno
intimateknowledgeofPython.
11.1 Definitions
Forthosewhoarenotfromadevelopmentbackground(andpossiblywithapologiesto
thosewhoare),herearesomedefinitions.
AnApplicationProgrammingInterface(API)isawayofaccessingstuff.
StuffinthecontextofZenossmeansobjectsthatrepresentrealthings.Forexample,
Pythonobjectsthatrepresentdevices,networkinterfaces,filesystems,processesand
users;databaseobjectsintheMySQLdatabasethatrepresentevents.
JavaScriptObjectNotation(JSON)isalightweightdatainterchangeformat.Itiseasy
forhumanstoreadandwritebeingatextformatthatiscompletelylanguage
independentbutusesconventionsthatarefamiliartoprogrammersoftheCfamilyof
languages,includingC,C++,C#,Java,JavaScript,Perl,Python,andmanyothers.
ThustheJSONAPIprovidesadocumentedwayofaccessingdifferentsortsofdata
withinZenoss,usingacommoninterface.Whateverstuffisbeingaccessed,wepresent
requestsinatextformatandtheresultsaretranslatedbackintotextformatforus.
Inordertopresentourrequestsfordata,aURLisrequiredplusauseridandpassword
thathasauthoritytoaccesstheZenossdatarequested.Asusers,wecanconstruct
requestsinexactlythesamewayastheZenossGUIdoes;theZenossGUIitselfusesthe
JSONAPItopresentdatatous.
AnotherbenefitofusingtheJSONAPIratherthanusingPythondirectly,isthatZenoss
DevelopmentmaychangetheunderlyingPythonintheZenossCorecodebut,provided
theymaintaintheJSONAPIinterface,anyaccessfunctionalitybuiltontopoftheAPI
126
1February2013
canremainunchanged.ForthisreasonthereisarecommendationthattheAPIbeused
inpreferencetowritingPythoncodetoaccessdatadirectly.
11.2 Understanding the JSON API

TheJSONAPIisshippedasstandardwithZenossCore.Thedocumentationcanbe
foundat
http://community.zenoss.org/community/documentation/official_documentation/api;this
isactuallyazippedbundlecontainingdocumentationinhtmlformat,apdfguideand
bothPythonandJavasamplesforusingtheAPI.
TherearealsosomesamplesofusingtheJSONAPIwithbashandcurlat
TheJSONAPIexposesthemethodsthatcanbefoundintheZenosscodeunder
$ZENHOME/Products/Zuul/routers.
Theeasiestwaytoviewthedocumentationistodownloadthezipbundle,unzipitand
pointabrowserattheapidoc/html/index.htmlfile.
Figure90:JSONAPIdocumentationinhtmlformat
Thelefthandmenusshowthemodules,effectivelythefilesthatcanbefoundunder
$ZENHOME/Zuul/Products/routers.Typicallythesefileseachdefineoneclassthough
thenetworkfilehasaclassforeachofNetworkRouterandNetwork6Router.
Clickonamoduletoseeanoverviewofwhatitcontains.NotetheAvailableatlinethat
helpsindicatetheurlthatreachesthisdata.
ClickonthelinktotheClass,EventsRouter,toseeallthemethodsforthisclass.
1February2013
127
Figure91:JSONAPIdetailsofthezepmodule
Figure92:JSONAPImethodsfortheEventsRouterclass
Clickonamethodtogetamoredetailedoverviewwithdescriptionsoftheinput
parametersandthevaluesreturned.
128
1February2013
Figure93:JSONAPIdetailsforthequerymethodintheEventsRouterclass
Atalllevelsofthedocumentationtherearelinkstothesourcecode.Thisshouldbevery
closetothecodethatyouseeifyouinspectthefile$ZENHOME/Products/Zuul/routers
thoughthelinenumbersmaynotmatchexactlydependingontheexactlevelofcodeyou
arerunning.
Figure94:JSONAPIsourcecodeforthequerymethod
1February2013
129
Ifyouinspectthe__init__methodsourcecodefortheEventsRouterclass,youcansee
thatthezepattributeissetto:
self.zep = Zuul.getFacade('zep', context)
Eachofthefilesin$ZENHOME/Products/Zuul/routershasmethodsthatcallthe
matchingfacadefoundunder$ZENHOME/Products/Zuul/facades.
Thinkoftheroutersasawaytoreachtherightbasicareaofdatadevice,mibs,
triggers,zepwithsometoplevelmethodslikequery,_buildFilter;andthinkofthe
facadesasmoredetailedaccessmethods;so,havinggainedaccesstotheevents
throughthezeprouter,thefacadeprovidescreateEventFilter,getEventSummaries,
acknowledgeEventSummaries,andsoon.
11.3 Using the JSON API

ThedocumentationbundleincludessamplecodeforusingtheJSONAPIfromPython
programsandJavaprograms.Furthersamplesareavailableat
https://gist.github.com/1901884/thatdemonstrateabashshellharnessfordrivingthe
APIusingthecurlutility.
NotethatthePythonsamplesbothrequireslightbugfixestodevice.pyandzep.py
respectivelyin$ZENHOME/Products/Zuul/routersforthebaseZenossCore4.2code
seeadiscussionandsolutionsontheZenossUser'sforumat
http://community.zenoss.org/message/70052#70052.Theseissuesappeartobefixed
withCore4.2.3.
11.3.1 Bash examples

Getthebashexamplesfromhttps://gist.github.com/1901884/(usetheDownloadGist
link)andunpackthebundletogetzenoss_curlExamples.sh.Editthisfiletoreflectyour
Zenossserverparameters,ifrequired,thoughthecodealreadyhasadefaultserverof
localhost,port8080,userofadminandpasswordofzenosssoitwillprobablyworkasis
ifyouhavenotchangedinstalldefaults.
AllthecodetodowithservicesreferstotheenterpriseZenossResourceManager
chargeableproductsotheycanberemoved.Tocutthefiledowntoabasicsamplethat
justaddsadevice,n7k1,tothe/Network/Router/Ciscodeviceclass,alsoremovethe
helperfunctionsforUCSandVCSobjectssothatyouendupwithashellscriptasshown
inFigure95.Notethatthedeviceclasshasalsobeenchangedfromtheoriginalscript
astheclassmustexist.
Thesingleremainingbodylineofthescriptis:
zenoss_add_devicen7k1"/Network/Router/Cisco"
callingthehelperfunction:
zenoss_add_device()
130
1February2013
Figure95:Modifiedzenoss_curlExample.shtoaddasingle/Network/Router/Ciscodevice
Thisfunctiontakes2parameterswhere$1isthehostnameand$2isthedeviceclass.It
thencallsthezenoss_apifunction:
zenoss_apidevice_routerDeviceRouteraddDevice
"{\"deviceName\":\"$DEVICE_HOSTNAME\",\"deviceClass\":\"$DEVICE_CLASS\",\"c
ollector\":\"localhost\",\"model\":true,\"title\":\"\",\"productionState\":
\"1000\",\"priority\":\"3\",\"snmpCommunity\":\"\",\"snmpPort\":161,\"tag\"
:\"\",\"rackSlot\":\"\",\"serialNumber\":\"\",\"hwManufacturer\":\"\",\"hwP
roductName\":\"\",\"osManufacturer\":\"\",\"osProductName\":\"\",\"comments
\":\"\"}"
zenoss_apirequiresfourparameters:
zenoss_api(){
ROUTER_ENDPOINT=$1
ROUTER_ACTION=$2
ROUTER_METHOD=$3
DATA=$4
wheretheROUTER_ENDPOINTvalueofdevice_routerisfoundfromtheJSONAPI
documentationbylookingattheAvailableat:/zport/dmd/device_routerlineforthe
moduleProducts.Zuul.routers.device.TheROUTER_ACTIONisDeviceRouterthe
Classshowninthedocumentation;theROUTER_METHODisaddDevicethemethod
foundbyexploringtheDeviceRouterclass;andtheDATAparametercontains
1February2013
131
<parametername>:<parametervalue>stringpairs,commaseparated,withdouble
quotescarefullyescapedbybackslashes.
Figure96:addDevicemethodfortheDeviceRouterclassdetailinginputparameters
Ensurethattheshellscriptisexecutableandrunit.Checkthatthedeviceisadded.
Thesetxlineatthetopofthescriptcanbeuncommentedtoprovidedebugging.
Hereisasecondexamplethatexploresthecapabilitiesofthetriggersinterface.
ExploringthetriggersmodulewiththeAPIdocumentationshowsthatsomemethods
needadataparameterandsomedon't.Thisiswhytherearetwohelperfunctionsin
Figure97.
132
1February2013
Figure97:zenoss_JSONAPI_curl_triggers.shpart1showing2helperfunctions
Figure98:zenoss_JSONAPI_curl_triggers.shpart2callingthehelperfunctionswithdifferentmethods
Themainbodyofzenoss_JSONAPI_curl_triggers.shhastwocallstozenoss_api_triggers
(withnodataparameter)toproducealistoftriggersandthedetailforeachtrigger,
respectively;thethirdcallusesthesecondhelperfunctionwiththegetTriggermethod
andprovidesauuidparametertojustgetthedetailofaspecifictrigger.Theuuidwas
1February2013
133
determinedfromthegetTriggerListoutputandthenhardcodedbackintothescriptasan
example.
OutputlookslikeFigure99.
Figure99:Outputfromzenoss_JSONAPI_curl_triggers.sh
Notethatusingthebash/curlinterfacewiththeEventsRouterclassinthezeprouter
module,ismuchharderasmanyofthemethodsrequireadictionaryasaninput
parameter.Forthisreason,itiseasiertodrivetheeventspartoftheJSONAPIfroma
Pythonharness.
11.3.2 Python examples

TheJSONAPIdocumentationbundledeliversapythonsubdirectorywithexamples.Be
suretocheckhttp://community.zenoss.org/message/70052#70052ifyouareseeing
unexplainableerrors.
api_example.pyprovidesagenericclass,ZenossAPIExample(),whichconnectstothe
Zenossserver.
134
1February2013
Figure100:api_example.pypart1withconnectionlogicandroutersdefined
Theclasshasa_router_requestmethodthathasparametersfortherouterclassto
connectto,themethodtoexecuteandadatalistthatpassesparameterstothemethod,
performingthetranslationbetweenPythonobjectsandJSON,asrequired.
Fourhelperfunctionsarealsoprovidedinapi_example.py,eachofwhichutilisesthe
_router_requestmethod.
1February2013
135
Figure101:api_example.pypart2with_router_requestmethod
defget_devices(self,deviceClass='/zport/dmd/Devices'):
defget_events(self,device=None,component=None,eventClass=None):
defadd_device(self,deviceName,deviceClass):
defcreate_event_on_device(self,device,severity,summary):
Figure102:api_example.pypart3withhelpermethodstoaccessdeviceandeventsobjects
136
1February2013
event_curses.pyisanexamplescriptthatimportsapi_exampleandusestheget_events
methodtoaccesseventsintheMySQLdatabase.Theonlyotherdependencyisthe
importoftexttablewhichisalsoincludedinthesamedirectory(see
JSONAPIQuickstart.txtinthetopleveldirectoryofthedocumentation).
Figure103:event_curses.pyhighlightingcallstotheapi_examplefunctionality
Whenevent_curses.pyisrunwithpythonevent_curses.py,alistofeventsisoutputtothe
screenwithDevice,Component,SummaryandEventClassfields,eachlinebeing
colourcodedbyseverity.Asshipped,allNewandAcknowledgedstatuseventsof
severity5,4,3and2,areretrievedfromtheMySQLdatabase.
1February2013
137
Figure104:Outputofpythonevent_curses.py
Notethatifevent_curses.pydoesnotrunthenopenanewcommandterminalwitha
defaultscreensizeandtryagain.
Tobemoreselectiveontheeventcursesoutput,lookcloselyatthecommentedout
rawEvents=lineinFigure103.Thelinerestrictsoutputtojusteventsfrom
zen42.class.example.org.
ForanextensionofusingthequerymethodoftheEventsRouterclass,seeget_events.py
inAppendixA.Ittakesparameterstoselectthefiltercriteriaforactiveeventsandthen
outputsalargenumberoffields.pythonget_events.pyhelpprovidestheusage.
138
1February2013
Figure105:get_events.pyoutputtoselectactiveeventsandoutputtotheconsole
12 Conclusions
ZenosshasanextensiveeventsystemcapableofreceivingeventsfromWindows,syslogs
andSNMPTRAPs,inadditiontoreceivingtheeventsgeneratedinternallybyZenoss's
owndiscovery,availabilityandperformancemonitoring.
AlargenumberofeventclassesaredefinedandconfiguredwhenZenossisinstalled.
Thesecanbemodified,removedoraddedto.
Aneventfollowsafairlycomplexeventlifecycleprocesswherebyitismappedtoan
eventclassandthen,optionally,itistransformedsuchthatdefaultfieldsoftheevent
canbechangedanduserdefinedfieldscanbecreated.
EventmappingforeventsfromWindows,syslogsorSNMP,dependsontheinitial
ZenossparsingdaemondeliveringaneventClassKeyfieldwhichmustcorrespondtoa
definedmapping.Subsequently,aPythonRuleand/oraPythonRegexcanbeusedto
furtherdistinguishbetweenincomingeventsandmaptodifferenteventclasses.
1February2013
139
Figure106:Eventattributesthroughtheeventlifecycle(part1)
DevicecontextisappliedtoanincomingeventfromtheZODBdatabase;devicecontext
includestheprodState,DevicePriority,Location,DeviceClass,DeviceGroupsand
Systemsfieldvalues.Devicecontextprovidestheabilityfortransformstotakeaccount
ofthedeviceordeviceclasshierarchy.
AneventclassincludeseventcontextzEventAction,zEventSeverityand
zEventClearClasseswhichcanbeappliedtoindividualsubclassesofeventsortoclass
hierarchies.Thismeanstransformscanbeaffectedbyeventtype.
Eventtransformscanbesimpleassignmentofeventfieldsorcanincludecomplex
Pythonprograms.AgoodenvironmentfortestingPythonisthezendmdcommandline
utility.Transformsand/ortheeventcontextcanbeusedtohelpcleareventsthathave
beenresolved.AnyeventwithaseverityofClearedwillautomaticallyclearother
similarevents;zEventClearClassescanbeusedtolistextraclassesthatareclearedin
addition.
140
1February2013
EventsaresavedintheMySQLzenoss_zepdatabaseintheevent_summarytable.
EventscanbeClosedbyusersorClearedbyotherevents;theycanalsobeAgedbased
onseverityandlengthoftimethattheeventhaspersisted.Afteraconfigurable
interval,nonactiveevents(witheventStateofClosed,ClearedandAged)aremovedto
theevent_archivetableofthedatabase.Eventually,archivedeventscanbedeleted.
1February2013
141
Wheneventsoccur,actionscanbegeneratedeithertoalertusersbyusingemailora
pagingsystem;alternatively,backgroundactionscanbeconfiguredtorunacommand
ontheZenossserverortogenerateanSNMPTRAP.
TheJSONAPIprovidesagenericinterfaceforaccessingdataintheZenosssystem.
Aswithanyenterprisemanagementsystem,Zenosshasthetoolstoconfigurealmost
anyresponsetoanyevent.
142
1February2013
13 Appendix A
13.1 getevents.py
get_events.pytoselectactiveevents.
#Zenoss4.xJSONAPIExample(python)
#
#Toquicklyexplore,execute'pythoniget_events.py
#
#>>>z=getEventsWithJSON()
#>>>events=z.get_events()
#etc.
importjson
importurllib
importurllib2
fromoptparseimportOptionParser
importpprint
#ZENOSS_INSTANCE='http://ZENOSSSERVER:8080'
#Changethenextline(s)tosuityourenvironment
#
ZENOSS_INSTANCE='http://zen42.class.example.org:8080'
ZENOSS_USERNAME='admin'
ZENOSS_PASSWORD='zenoss'
ROUTERS={'MessagingRouter':'messaging',
'EventsRouter':'evconsole',
'ProcessRouter':'process',
'ServiceRouter':'service',
'DeviceRouter':'device',
'NetworkRouter':'network',
'TemplateRouter':'template',
'DetailNavRouter':'detailnav',
'ReportRouter':'report',
'MibRouter':'mib',
'ZenPackRouter':'zenpack'}
classgetEventsWithJSON():
def__init__(self,debug=False):
"""
InitializetheAPIconnection,login,andstoreauthentication
cookie
"""
#UsetheHTTPCookieProcessorasurllib2doesnotsavecookiesby
default
self.urlOpener=
urllib2.build_opener(urllib2.HTTPCookieProcessor())
ifdebug:
self.urlOpener.add_handler(urllib2.HTTPHandler(debuglevel=1))
self.reqCount=1
#ConstructPOSTparamsandsubmitlogin.
loginParams=urllib.urlencode(dict(
__ac_name=ZENOSS_USERNAME,
__ac_password=ZENOSS_PASSWORD,
1February2013
143
submitted='true',
came_from=ZENOSS_INSTANCE+'/zport/dmd'))
self.urlOpener.open(ZENOSS_INSTANCE+
'/zport/acl_users/cookieAuthHelper/login',
loginParams)
def_router_request(self,router,method,data=[]):
ifrouternotinROUTERS:
raiseException('Router"'+router+'"notavailable.')
#ConstructastandardURLrequestforAPIcalls
req=urllib2.Request(ZENOSS_INSTANCE+'/zport/dmd/'+
ROUTERS[router]+'_router')
#NOTE:ContenttypeMUSTbesetto'application/json'forthese
requests
req.add_header('Contenttype','application/json;charset=utf8')
#ConverttherequestparametersintoJSON
reqData=json.dumps([dict(
action=router,
method=method,
data=data,
type='rpc',
tid=self.reqCount)])
#Incrementtherequestcount('tid').Moreimportantifsending
multiple
#callsinasinglerequest
self.reqCount+=1
#SubmittherequestandconvertthereturnedJSONtoobjects
returnjson.loads(self.urlOpener.open(req,reqData).read())
defget_events(self,filter={},sort='severity',dir='DESC'):
"""UseEventsRouteraction(Class)andquerymethodfound
inJSONAPIdocsonZenosswebsite:
query(self,limit=0,start=0,sort='lastTime',dir='desc',
params=None,
archive=False,uid=None,detailFormat=False)
Parameters:
limit(integer)(optional)Maxindexofeventstoretrieve
(default:0)
start(integer)(optional)Minindexofeventstoretrieve
(default:0)
sort(string)(optional)Keyonwhichtosortthereturnresults
(default:'lastTime')
dir(string)(optional)Sortorder;canbeeither'ASC'or'DESC'
(default:'DESC')
params(dictionary)(optional)Keyvaluepairoffiltersforthis
search.(default:None)
paramsarethefilterstothequerymethodandcanbefoundinthe
_buildFiltermethod.
severity=params.get('severity'),
status=[iforiinparams.get('eventState',[])],
event_class=filter(None,[params.get('eventClass')]),
144
1February2013
Notethatthetimevaluescanberangeswhereavalidrange
wouldbe
'2012090707:57:33/2012112217:57:33'
first_seen=params.get('firstTime')and
self._timeRange(params.get('firstTime')),
last_seen=params.get('lastTime')and
self._timeRange(params.get('lastTime')),
status_change=params.get('stateChange')and
self._timeRange(params.get('stateChange')),
uuid=filterEventUuids,
count_range=params.get('count'),
element_title=params.get('device'),
element_sub_title=params.get('component'),
event_summary=params.get('summary'),
current_user_name=params.get('ownerid'),
agent=params.get('agent'),
monitor=params.get('monitor'),
fingerprint=params.get('dedupid'),
tags=params.get('tags'),
details=details,
archive(boolean)(optional)Truetosearchtheeventhistory
tableinsteadofactiveevents(default:False)
uid(string)(optional)Contextforthequery(default:None)
Returns:dictionary
Properties:
events:([dictionary])Listofobjectsrepresentingevents
totalCount:(integer)Totalcountofeventsreturned
asof:(float)Currenttime
"""
data=dict(start=0,limit=1000)
ifsort:data['sort']=sort
ifdir:data['dir']=dir
data['params']=filter
#print'data[params]is%s\n'%(data['params'])
#print'datais%s\n'%(data)
returnself._router_request('EventsRouter','query',[data])
['result']
if__name__=="__main__":
usage='python%progseverity=severityeventState=eventState
device=deviceeventClass=eventClasscomponent=componentagent=agent
monitor=monitorcount=countlastTime=lastTimefirstTime=firstTime
stateChange=stateChangesort=lastTimedir=DESC'
parser=OptionParser(usage)
parser.add_option("severity",dest='severity',
help='severitycommaseparatednumericvalueseg.
severity=5,4forCriticalandError')
parser.add_option("eventState",dest='eventState',default='0,1',
help='eventStatecommaseparatednumericvalues
eg.eventState=0,1forNewandAck')
parser.add_option("device",dest='device',
help='eg.device=\'zen42.class.example.org\'')
parser.add_option("eventClass",dest='eventClass',
1February2013
145
help='eg.eventClass=\'/Skills\'')
parser.add_option("component",dest='component',
help='eg.component=\'TestComponent\'')
parser.add_option("agent",dest='agent',
help='eg.agent=\'zensyslog\'')
parser.add_option("monitor",dest='monitor',
help='eg.monitor=\'localhost\'')
parser.add_option("count",dest='count',
help='numericvalueeg.count=3orrangecount
3,30')
parser.add_option("lastTime",dest='lastTime',
help='eg.forarangeseparatestart&endwith/
lastTime=\'2012090707:57:33/2012112217:57:33\'')
parser.add_option("firstTime",dest='firstTime',
help='eg.firstTime=\'2012112217:57:33\'')
parser.add_option("stateChange",dest='stateChange',
help='eg.stateChange=\'2012112217:57:33\'')
parser.add_option("sort",dest='sort',default='lastTime',
help='thekeytosortoneg.sort=\'lastTime\'')
parser.add_option("dir",dest='dir',default='DESC',
help='thedirectiontosorteg.dir=\'ASC\'or
dir=\'DESC\'')
(options,args)=parser.parse_args()
#optionsisanobjectwewantthedictionaryvalueofit
#Someoftheoptionsneedalittlemunging...
option_dict=vars(options)
ifoption_dict['severity']:
option_dict['severity']=option_dict['severity'].split(',')
ifoption_dict['eventState']:
option_dict['eventState']=option_dict['eventState'].split(',')
#countcaneitherbeanumberorarange(ineitherlistortuple
format)
#(see$ZENHOME/Products/Zuul/facades/zepfacade.py
createEventFiltermethod)
#butifthismethodgetsalistitassumesthereare2elementsto
thelist.
#Wemaygetalistwithasinglevaluesoconvertittoanumber
andthe
#createEventFiltermethodcancope
ifoption_dict['count']:
option_dict['count']=option_dict['count'].split(',')
iflen(option_dict['count'])==1:
option_dict['count']=int(option_dict['count'][0])
#option_dictincludesthesortanddirkeys(aswehavedefaultedthem
inoptparse)
#Thesearenotpartofthefilterstringsoweneedtopopthemoutof
thedictionary
#touseseparately.
s=option_dict.pop('sort')
d=option_dict.pop('dir')
#Needtocheckthesekeysforsanity
#andprovidesensibledefaultsotherwise
dirlist=['ASC','DESC']
ifnotdindirlist:
d='DESC'
146
1February2013
sortlist=['severity','eventState','eventClass','firstTime',
'lastTime',
'stateChange','count','device','component','agent',
'monitor']
ifnotsinsortlist:
s='lastTime'
#print'optionsis%s\n'%(options)
#print'option_dictis%s\n'%(option_dict)
events=getEventsWithJSON()
#filter['evid']='000c29d9f87b838911e2347cddf7a720'
pp=pprint.PrettyPrinter(indent=4)
fields=['eventState','DeviceClass','count','device','Location',
'Systems','severity','firstTime','lastTime','summary']
#fields=['eventState','DeviceClass','count','device','Location',
'severity','firstTime','lastTime','summary']
print'eventState,DeviceClass,count,device,Location,Systems,
severity,firstTime,lastTime,summary'
#print'eventState,DeviceClass,count,device,Location,severity,
firstTime,lastTime,summary'
out=events.get_events(filter=option_dict,sort=s,dir=d)
foreinout['events']:
#pp.pprint(e)
outState=e['eventState']
ife['DeviceClass']:
outDeviceClass=e['DeviceClass'][0]['name']
else:outDeviceClass=[]
outcount=e['count']
outdevice=e['device']['text']
ife['Location']:
outLocation=e['Location'][0]['name']
else:outLocation=[]
outSystems=[]
forpos,valinenumerate(e['Systems']):
sy=str(e['Systems'][pos]['name'])
outSystems.append(sy)
outseverity=e['severity']
outfirstTime=e['firstTime']
outlastTime=e['lastTime']
outsummary=e['summary']
print'%s,%s,%s,%s,%s,%s,%s,%s,%s,%s'%(outState,
outDeviceClass,outcount,outdevice,outLocation,outSystems,outseverity,
outfirstTime,outlastTime,outsummary)
#print'%s,%s,%s,%s,%s,%s,%s,%s,%s'%(outState,
outDeviceClass,outcount,outdevice,outLocation,outseverity,
outfirstTime,outlastTime,outsummary)
#print'\ntotalCountis%dandasofis%s'%(out['totalCount'],
out['asof'])
1February2013
147
13.2 zensendevent
Modifiedzensendeventtoautomaticallyretrievelocalauthenticationparameters.
ZenossCore4.2.3changedsomesecuritypoliciesatinstallationtimewhichresultsin
zensendeventfailingunlessauthparametersaredeterminedandsuppliedexplicitly.
#!/opt/zenoss/bin/python
__doc__="""zensendevent
SendeventsonacommandlineviaXMLRPCorfromaXMLfile.
ThiscommandcanbeputonanymachinewithPythoninstalled,and
doesnotneedZopeorZenoss.
"""
importsocket
fromxmlrpclibimportServerProxy
fromoptparseimportOptionParser
fromxml.saximportmake_parser,saxutils
fromxml.sax.handlerimportContentHandler
XML_RPC_PORT=8081
sevconvert={
"critical":5,
"error":4,
"warn":3,
"info":2,
"debug":1,
"clear":0
}
classImportEventXML(ContentHandler):
ignoredElements=set([
'ZenossEvents','url','SourceComponent',
'ReporterComponent','EventId',
'clearid','eventClassMapping',
'eventState','lastTime','firstTime','prodState',
'EventSpecific','stateChange',
])
evt={}
property=''
value=''
def__init__(self,serv):
ContentHandler.__init__(self)
self.sent=0
self.total=0
self.serv=serv
defstartElement(self,name,attrs):
self.value=''
ifname=='ZenossEvent':
self.evt={}
elifname=='property':
148
1February2013
self.property=attrs['name']
defcharacters(self,content):
self.value+=content
defendElement(self,name):
name=str(name)
value=str(self.value)
ifnameinself.ignoredElements:
return
elifname=='property'andvalueandvalue!='|':
self.evt[self.property]=value
elifnamein['Systems','DeviceGroups']:
ifvalueandvalue!='|':
self.evt[name]=value
elifnamein['eventClassKey','eventKey']:
ifvalue:
elifname=='severity':
self.evt[name]=int(value)
elifname=='ZenossEvent':
self.total+=1
try:
self.serv.sendEvent(self.evt)
self.sent+=1
exceptException,ex:
printstr(ex)
printevt
elifvalue:
defsendXMLEvents(serv,xmlfile):
infile=open(xmlfile)
parser=make_parser()
CH=ImportEventXML(serv)
parser.setContentHandler(CH)
try:
parser.parse(infile)
finally:
infile.close()
print"Sent%sof%sevents"%(CH.sent,CH.total)
device=socket.getfqdn()
ifdevice.endswith('.'):device=device[:1]
parser=OptionParser(usage="usage:%prog[options]summary")
parser.add_option("d","device",dest="device",default=device,
help="devicefromwhichthiseventissent,default:%default")
parser.add_option("i","ipAddress",dest="ipAddress",default="",
help="Ipfromwhichthiseventwassent,default:%default")
parser.add_option("y","eventkey",dest="eventkey",default="",
help="eventKeytobeused,default:%default")
parser.add_option("p","component",dest="component",default="",
help="componentfromwhichthiseventissent,default:''")
1February2013
149
parser.add_option("k","eventclasskey",dest="eventClassKey",
default="",
help="eventClassKeyforthisevent,default:''")
parser.add_option("s","severity",dest="severity",default="Warn",
help="severityofthisevent:Critical,Error,Warn,Info,Debug,
Clear")
parser.add_option("c","eventclass",dest="eventClass",default=None,
help="eventclassforthisevent,default:''")
parser.add_option("monitor",dest="monitor",default="localhost",
help="monitorfromwhichthiseventcame")
parser.add_option("port",dest="port",default=XML_RPC_PORT,
help="xmlrpcserverport,default:%default")
parser.add_option("server",dest="server",default="localhost",
help="xmlrpcserver,default:%default")
parser.add_option("auth",dest="auth",default="admin:zenoss",
help="xmlrpcserverauth,default:%default")
parser.add_option("o","other",dest="other",default=[],
action='append',
help="Specifyotherevent_field=valuearguments.Canbespecified"
"morethanonce.")
parser.add_option('f',"file",dest="input_file",default="",
help="ImporteventsfromXMLfile.")
parser.add_option('v',dest="show_event",default=False,
action='store_true',
help="ShowtheeventdatasenttoZenoss.")
opts,args=parser.parse_args()
#HackbyJCtogethubpasswdauthenticationintoauthoption
#Passwordisheldin$ZENHOME/etc/hubpasswdin(almost)correctformat
<user>:<password>\n
importos
#ifauthisthedefault
ifopts.auth=='admin:zenoss':
zenhome=os.environ['ZENHOME']
#Trytoaccess$ZENHOME/etc/hubpasswdandstriptrailingnewline
try:
pwfile=open(os.path.join(zenhome,'etc','hubpasswd'),'r')
opts.auth=pwfile.read().rstrip()
pwfile.close()
print'Extractingnecessaryuser:passwordautomatically\n'
#Ifthisfailsthenfallbacktodefaultandprintmessage
except:
print'Attempttodetecthubpasswdfailed\n'
#EndofJChack
url="http://%s@%s:%s"%(opts.auth,opts.server,opts.port)
serv=ServerProxy(url)
ifopts.input_file:
sendXMLEvents(serv,opts.input_file)
importsys
sys.exit(0)
evt={}
ifopts.severity.lower()insevconvert:
evt['severity']=sevconvert[opts.severity.lower()]
else:
150
1February2013
parser.error('Unknownseverity')
evt['summary']="".join(args)
ifnotevt['summary']:
parser.error('nosummarysupplied')
evt['device']=opts.device
evt['component']=opts.component
evt['ipAddress']=opts.ipAddress
ifopts.eventkey:
evt['eventKey']=opts.eventkey
ifopts.eventClassKey:
evt['eventClassKey']=opts.eventClassKey
ifopts.eventClass:
evt['eventClass']=opts.eventClass
evt['monitor']=opts.monitor
forlineinopts.other:
try:
field,value=line.split('=',1)
evt[field]=value
except:
pass
ifopts.show_event:
frompprintimportpprint
pprint(evt)
serv.sendEvent(evt)
1February2013
151
14 References
1. ZenossCommunitysitehttp://community.zenoss.org
2. Zenossnetwork,systemsandapplicationmonitoringcommercialsite
http://www.zenoss.com/
3. Zenossdocumentationmainpage
http://community.zenoss.org/community/documentation
4. ZenossCore4AdministrationGuide
http://community.zenoss.org/community/documentation/official_documentation/ze
nossguide
5. ZenossDeveloper'sGuide
http://community.zenoss.org/community/documentation/official_documentation/ze
nossdevguide
6. Zenoss4.2JSONAPIdocumentation
http://community.zenoss.org/community/documentation/official_documentation/api
7. SamplesofusingtheJSONAPIwithbashandcurlcanbefoundat
8. InformationonRelStorageandmemcachedhttp://wiki.zenoss.org/RelStorage
9. InformationonRabbitMQhttp://wiki.zenoss.org/Working_with_Queues
10. ScripttoresetRabbitMQhttps://gist.github.com/4192854
11. InformationonAMQPhttp://www.amqp.org/
12. InformationonLuceneindexinghttp://lucene.apache.org/core/
13. InformationonJSONhttp://www.json.org/
14. DiscussiononmodifyingzensendeventutilityonZenosswiki
http://wiki.zenoss.org/Zensendevent_in_Zenoss_4.2.3
15. ReferenceforWin32_NTLogEventclasseventlogseverities
http://msdn.microsoft.com/en
gb/library/windows/desktop/aa394226%28v=vs.85%29.aspx
16. InformationonPythonregularexpressions
http://docs.python.org/2/library/re.html, http://www.python.org/doc/2.5.2/lib/re-syntax.html
and http://docs.python.org/dev/howto/regex.html
17. Informationonprotobufshttp://code.google.com/p/protobuf/
18. InformationonthePythondebugger(pdb)
http://docs.python.org/2/library/pdb.html
19. AsageneralPythonreference,tryLearningPythonbyMarkLutz,publishedby
O'Reilly
152
1February2013
20. TheMIBBrowserZenPack.Documentationandcommentsat
http://community.zenoss.org/docs/DOC10321;codefrom
http://wiki.zenoss.org/ZenPack:MIB_Browser.
21. SNMPRequestsForComment(RFCs)http://www.ietf.org/rfc.html
V1RFCs1155,1157,1212,1213,1215
V2RFCs2578,2579,2580,3416,3417,3418
V3RFCs25782580,341618,3411,3412,3413,3414,3415
22. SNMPHostResourcesMIB,RFCs1514and2790http://www.ietf.org/rfc.html
23. FortheextensionSNMPMIBfromInformant,goto
http://www.wtcs.org/informant/index.htm
24. ForinformationonZopeTALESexpressions,see
http://docs.zope.org/zope2/zope2book/AppendixC.html
25. DatagramSyslogClienthttp://syslogserver.comforsyslogWindowssystems.
26. Raddlenetworkemulationopensourcepackagehttp://raddle.sourceforge.net/
27. Zenoss4EventManagementWorkshopavailablefromSkills1stLtd,
http://www.skills1st.co.uk/products/courses/
1February2013
153
Acknowledgements
AnumberofpeoplehavecontributedinformationandadvicetothisprojectandIwould
liketothankthem.
GeorgesReichsfortheoriginalamazingarchitecturedesigndiagram
ChetLutherforhisawesomeknowledgeofZenossandhiswillingnesstoshare
thatknowledge
AndrewKirchforinitialproofreadingandsomeusefulcomments
AndrewFindlayofSkills1stforhelpwithtypesetting
Abouttheauthor
JaneCurryhasbeenanetworkandsystemsmanagementtechnicalconsultantand
trainerfor25years.Duringher11yearsworkingforIBMshefulfilledbothpresales
andconsultancyrolesspanningthefullrangeofIBM'sSystemViewproductspriorto
1996andthen,whenIBMboughtTivoli,shespecialisedinthesystemsmanagement
productsofDistributedMonitoring&IBMTivoliMonitoring(ITM),thenetwork
managementproduct,TivoliNetViewandtheproblemmanagementproductTivoli
EnterpriseConsole(TEC).AllarebasedaroundtheTivoliFrameworkarchitecture.
Since1997Janehasbeenanindependentbusinesswomanworkingwithmany
companies,bothlargeandsmall,commercialandpublicsector,deliveringTivoli
consultancyandtraining.Overthelast5yearsherworkhasbeenmoreinvolvedwith
OpenSourceofferings,especiallyZenoss.
ShehasdevelopedanumberofZenPackaddonstoZenossCoreandhasalargenumber
oflocalandremoteconsultancyclientsforZenosscustomisationanddevelopment.She
hasalsocreatedseveralworkshopofferingstoaugmentZenoss'sowneducational
offerings.SheisafrequentcontributortotheZenossforumsandIRCchat
conversationsandwasmadeaZenossMasterbyZenossinFebruary2009
154
1February2013

Zenoss Core4 Event Management Paper

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Zenoss Core4 Event Management Paper

Uploaded by

Copyright:

Available Formats

Event Management

for Zenoss Core 4

2 Zenoss event architecture

2.2 Event Manager settings

2.3 Event database tables

2.4 New event daemons

2.5 Other database-related changes in Zenoss 4

2.6 Event life cycle

2.6.1 Event generation

2.6.2 Application of device context

2.6.3 Event class mapping

2.6.4 Application of event context

2.6.5 Event transforms

2.6.6 Database insertions and de-duplication

2.6.8 Ageing and archiving

3 Events generated by Zenoss

4.1 Configuring syslog.conf

@<IP address of your Zenoss system>

4.2 Zenoss processing of syslog messages

def process(self, msg, ipaddr, host, rtime):

5 Zenoss processing of Windows event logs

5.2 Management of Windows systems using syslog

6.1 Working with event classes and event mappings

6.1.1 Generating test events

6.2 Regex in event mappings

6.3 Rules in event mappings

6.4 Other elements of event mappings

7.1 Different ways to apply transforms

7.2 Understanding fields available for event processing

7.2.1 Event Proxies

7.2.2 Event Details

7.3 Transform examples

7.3.2 Applying event and device context in relation to transforms

8 Testing and debugging aids

8.1.3 Other log files

8.2 Using zendmd to run Python commands

8.2.1 Referencing an existing Zenoss event for use in zendmd

8.2.2 Using zendmd to understand attributes for an EventSummaryProxy

8.3 Using the Python debugger in transforms

9 Zenoss and SNMP

test with good community

9.2 SNMP on Linux systems

9.3 Zenoss SNMP architecture

9.4 Interpreting MIBs

9.4.1 zenmib example

9.4.2 A few comments on importing MIBs with Zenoss

9.5 The MIB Browser ZenPack

9.6 Mapping SNMP events

9.6.1 SNMP event mapping example

10 Event Triggers and Notifications

10.2 Zenoss 4 architecture

10.4.1 email Notifications

10.4.2 Page Notifications

10.4.3 Command Notifications

10.4.4 TRAP Notifications

10.5 Notification Schedules

10.6 Using zenactiond.log

10.7 The effect of device Production State

11 Accessing events with the JSON API

11.2 Understanding the JSON API

11.3 Using the JSON API

11.3.1 Bash examples

11.3.2 Python examples

You might also like