Professional Documents
Culture Documents
Troubleshooting coda 1
Version: A.01.00 February-2007
Document History
Troubleshooting coda 2
Preface
Intended Audience:
This Troubleshooting Guide is written primarily for Response Center and Competency
Center engineers who answer calls from the customers experiencing problems with
the OpenView performance suite of products. The information presented here
presumes a working knowledge of the products and familiarity with customer-
exposed features and documentation.
Troubleshooting coda 3
Table of Contents:
Troubleshooting coda 4
12.5. Configure Log File Size and Log File Number Limitation ......................... 39
13. Configuring coda in single port and multi port communication (OVPM,
OVPA, coda, OVO agent)................................................................................. 40
13.1. Single port communication ................................................................ 40
13.2. Multi port configuration ..................................................................... 40
13.3. Tools for verification of Settings ......................................................... 41
14. Known problem and workarounds .......................................................... 42
15. Latest Coda Patches ............................................................................. 50
Troubleshooting coda 5
1. OverView of CODA
CODA is a daemon that handles data communication (both HTTP and HTTPS) when
delivered as part of HP OpenView Performance Agent OVPA. Coda also handles
communication with perfalarm, the alarm management daemon. CODA, when
delivered as part of OV Operations Agent provides lightweight system performance
collection and SPI support.
1.1. Features
§ Performs a set of functions that the host OpenView product can take
advantage of:
§ Collects ~150 system performance metrics & stores data in its own
proprietary data store
§ Data viewable by OpenView Operations for Windows (embedded Graphing
component), OpenViewPerformance Manager, OpenView Reporter
§ More secure support for firewalls using BBC
Note: That's assuming that when the system was installed, the appropriate
information was included in /etc/issue. However, this may not be standard (or
typical) on Linux systems.
Troubleshooting coda 6
Red Hat Linux release 6.2 (Zoot)
Kernel 2.2.17-14 on an i686
* Unix:
- coda instruments and collects metrics.
- codautil Starts and stops collections; reports status/state of Coda (running
or not); and via the -support option, reports the values for all of the golden
metrics for the last interval (this provides an easy validation to determine if
Coda is collecting and logging the metrics).
* Windows:
- coda.exe
- codautil.exe
- coda_access.dll
* Unix:
- coda instruments and collects metrics.
- ovcodautil reports status/state of Coda (running or not); and via support
option, reports the values for all of the golden metrics for the last interval
(this provides an easy validation to determine if Coda is collecting and logging
the metrics).
* Windows:
- coda.exe
- ovcodautil.exe
- OvCoda.dll
Troubleshooting coda 7
2. Consumers of Coda, OVPM, OV Reporter, OVO and SPIs
From OVO for Windows, you can design custom graphs using metrics from either
OVOA-EPC or OVPA version 4.
From OVPM, the ’CODA’ data source now gives access to both OVPA 4 scope and DSI
logs and OVOA-EPC logs.
From Reporter, if metrics are gathered from OVPA’s scope log, the duplicate metrics
in Coda log files are not collected; SPI data in Coda log files will still be collected.
OVPA local export only works with scope and DSI logs; Coda log file data is not
locally exportable
Troubleshooting coda 8
3. Data Flow for Various Consumers of Coda
OpenView SPIs require an operations agent. Therefore, this “OVPA 4.x only” slide
with the SPI isn’t exactly true. This data flow is accurate if an older operations
agent, which does not have Coda or DDF installed.
Troubleshooting coda 9
3.2. SPIs and DSI2DDF
DSI2DDF
On HTTPS Agent node, the dsi2ddf determines whether to use DSI or DDF. The
dsi2ddf code asks a checks the system to determine the interface (DSI or DDF) and
the performance agent (OVPA or CODA) to use. The first step is to determine the
default interface.
If the default interface is DSI and the nocoda.opt file is empty è switch to
DDF.
The last step is to look for a performance agent that supports the selected interface.
If we cannot find the preferred performance agent, look for any performance agent.
No è ERROR
Details
The dsi2ddf code uses the XPL library to locate files on the HTTPS Agent node. To
determine if the default interface is overridden, the code looks at the contents of the
"nocoda.opt" file. The "nocoda.opt" file is located in the 'conf\dsi2ddf' subdirectory of
the OV Data Directory.
v On Unix platforms, the OV Data directory is, /var/opt/OV.
v On Windows, the OVData directory is “C:\Program Files\HP OpenView\data”.\
For Example:
/var/opt/OV/conf/dsi2ddf/nocoda.opt
C:\Program Files\HP OpenView\data\conf\dsi2ddf\nocoda.opt
Troubleshooting coda 10
To determine if CODA is installed, look for the CODA executable. The CODA
executable is located in the lbin\perf subdirectory of the OV Install directory. On Unix
platforms, the OV Install directory is typically, /opt/OV.
On Windows, the OV Install directory is typically “C:\Program Files\HP OpenView”.
For Example,
/opt/OV/lbin/perf/coda
C:\Program Files\HP OpenView\lbin\perf\coda.exe
To determine if OVPA is installed, the code looks for the sdlcomp executable
and the OVPA datasources file.
On DCE Agent node, how dsi2ddf determines whether to use DSI or CODA. The
dsi2ddf code asks a series of questions and examines the system to determine if it
should use DSI or CODA
To determine if OVPA4.x is installed, the code looks for the datasources file and the
sdlcomp executable. To determine if CODA is installed, the code looks for the CODA
executable in platform dependent locations.
To determine if the datasource is overridden, the code first looks at the contents of
the nocoda.opt file. If the nocoda.opt file does not exist, the code looks at the
location of the ddfcomp executable and the code determines if OVPA is installed.
Troubleshooting coda 11
Details
To determine if CODA is installed, the code looks for the CODA executable in
platform dependent locations.
- On AIX, the code looks for /usr/lpp/OV/bin/coda.
- On Tru64, the code looks for /usr/opt/OV/bin/coda.
- On all other Unix platforms, the code looks for /opt/OV/bin/coda.
- On Windows, the code uses the NT Registry key
[HKEY_LOCAL_MACHINE\SOFTWARE\Hewlett-Packard\HP
OpenView\AgentInstallDir]. The code adds the bin directory to the registry
key value.
For example:
C:\Program Files\HP OpenView\Installed Packages\{790C06B4-844E-11D2-
972B-080009EF8C2A}\bin\coda.exe
To determine if the datasource is overridden, the code looks at the contents of the
nocoda.opt file. The location of this file is platform dependent.
The code uses platform dependent methods to determine if OVPA 3.x is installed.
On Unix platforms, the code looks for sdlcomp and /var/opt/perf/perflbd.rc. If the
files exist, MWA is installed.
- On AIX, the code looks for /usr/lpp/perf/bin/sdlcomp.
- On Tru64, the code looks for /usr/opt/perf/bin/sdlcomp.
- On all other Unix platforms, the code looks for /opt/perf/bin/sdlcomp.
- On Windows, the code uses the NT Registry key
[HKEY_LOCAL_MACHINE\SOFTWARE\Hewlett-Packard\MeasureWare
Agent\CurrentVersion\CommonDataPath]. If the NT Registry key exists, then
MWA is installed.
Troubleshooting coda 12
4. How is Coda shared between OVPA and OVO?
When OVPA 4.5 is installed, the ‘llbserver’ broker is replaced by the ovbbccb
communication broker. The coda subagent which is previously registered with the
opcagt control agent will instead be registered with the ovc control agent, which gets
installed with OVPA 4.5. Coda10 will be registered with the new ovc control agent
and the two control agents will work together to ensure local and remote
management of the coda component. Post installation of OVPA 4.5, all incoming http
requests for performance data will come to the ovbbccb, which will then re-direct the
request to either scope or coda prospector.
Most customers will not notice the lightweight Coda data logging. However, some
customers of OVPA may want to disable Coda data logging as the OVPA metric set is
almost a superset of the Coda metric set with only a few exceptions.
1. Coda starts:
In order for Coda to start, a host product must be installed
2. Coda starts logging data:
If OVOA is installed, Coda references coda.db to find out which Coda log files to
open. If coda.db does not exist and the /databases/ dir exists, it creates coda.db and
starts a new mesa log file.
If OVPA is installed, Coda reads the datasources config file to determine what scope
and DSI logs are available.
3. Coda registers the port number with llbserver
4. Coda sets the prospector timer, which schedules it to collect data every 5 minutes
(only if Coda logs are created and prospector is not disabled).
5. Once startup is completed, Coda does one of two things; waits for incoming data
requests, waits for incoming DDF requests, or logs prospector data.
The concept of configuration is built into the both of these usage patterns as a DDF
will describe the new objects and metrics being provided and the presentation users
will need the ability to select specific metrics and to enable or disable specific
observation behaviors.
Troubleshooting coda 13
Coda
Troubleshooting coda 14
5. Supported Version of Coda
http://ovweb.deu.hp.com/set/hp/documentation/supp_matrices.htm
Troubleshooting coda 15
6. Checking Version of BBC Libraries
AIX:
/usr/lpp/OV/bin/bbcutil –version
Ex:
cscaix1:/opt/OV/bin > ./bbcutil -version
Windows:
> bbcutil -version
C:\Documents and Settings\Administrator>bbcutil -version
Troubleshooting coda 16
7. Checking coda version
Eg:
/opt/perf/bin/perfstat -v | grep -i coda
OVPA coda executables in the directories /opt/OV/bin /opt/OV/lbin/perf
ovcodautil 10.00.220 Solaris 7
coda 10.00.131 Solaris 7
OVPA coda/BBC libraries in the directory /opt/OV/lib
libOvCoda.so 10.00.220 Solaris 7
AIX:
/usr/lpp/perf/bin/perfstat –v | grep –i coda
Troubleshooting coda 17
8. Data collection
Coda can do that since it does not gather the detailed metrics that OVPA does. On
the other Unix platforms, the data is gathered primarily from /proc and as needed,
from kmem, and other OS sources. On Windows, most of the data comes from
Perflib; while the remaining data comes from the Win32 APIs.
No. There is no export option for the coda log files. This is true irrespective of the
products installed. Your options in this area are:
run “ovcodautil –support” – this gets you the last entries for the 30 “golden”
metrics (golden metrics are common across all platforms).
run “ovcodautil –dumpds <datasource>” -- this gets you the last entries
logged for this data source.
or, use OVPM, which in addition to graphing, also has CSV and TSV output
options from the command line.
Troubleshooting coda 18
9. Dependencies
Coda depends on the host OpenView product for the following functionalities:
§ deployment
§ install & uninstall
§ startup and shutdown
§ version control
§ packaging BBC files
Troubleshooting coda 19
10. Troubleshooting tools
For Unix/Linux
/var/opt/OV/log/coda.txt
/var/opt/OV/log/System.txt
For Windows
<Install Directory>\HP OpenView\data\log\coda.txt
<Install Directory>\HP OpenView\data\log\System.txt
XPL Tracing
Tracing is used by the application to record information during program execution
that may be helpful in
analyzing process flow and behavior or debugging.
Tracing Tools
ovtrccfg (<OV Install Dir>/support/ )
ovtrcd (<OV Install Dir>/lbin/xpl/trc/ )
ovtrcgui (<OV Install Dir>/support/ )
ovtrcadm ( <OV Install Dir>/support/ )
ovtrcmon (<OV Install Dir>/support/ )
Types of Tracing
The application to be traced can be configured through the following ways:
§ Static tracing
§ Dynamic tracing
Activating Tracing
Tracing is activated using a trace configuration file. The trace configuration file is the
one which tells the tracing system to enable tracing for the specific applications and
trace areas. The configuration file can be placed in a specific location on the local
system for static tracing or communicated through a Trace Server for dynamic
tracing.
Troubleshooting coda 20
Sink SINK: File "Output-name" "force=[1];maxfiles=[1..100];maxsize=[0..1000];"
SINK: Socket “node name” “node=<node name>;"
Trace TRACE: "Component-name" "Category-name" <keyword list>
The second argument is the trace category name and it should also be in double
quotes. If you are using one of the standard categories in the code, it will be mapped
to a string value (which you supply here). ). For the exact mapping of standard
category constants to string values, see the language-appropriate documentation
(C++, Java).
Troubleshooting coda 21
require access to the source code to be interpreted. Stack Include call stack
information in the trace output if possible. The stack trace message are not targeted
to the front line support engineers, since the trace messages require access to the
source code to be interpreted.
Static Tracing:
To activate static tracing, place a local trace configuration file named ‘OVTrace.tcf’ in
the same directory, where the application-to-be-traced resides, or define an
environment variable named ‘TRACE_CONFIG_FILE’ that points to the trace
configuration file, and then (re)start the application. The application will
automatically find and load the configuration file when the tracing subsystem inside
the application initializes.
When the static tracing method is used, trace settings cannot be dynamically
changed while the application is running and it does not provide for remote
configuration. During Static Tracing, the Trace configuration file search order is as
follows:
Dynamic Tracing:
Dynamic tracing can be configured by using the ovtrccfg tool. The ovtrcmon tool
communicates with the application through a Trace Server to monitor/view the
traces. In this case, the Trace Server must be running on the same machine where
the application runs. Note that the trace server must be running when the
application instantiates its first trace object (typically during the application startup).
If the Trace Server needs to restarted after the application has instantiated its first
trace object, the connection between the application and the trace server will be lost,
and tracing from this application will not work till this application is also restarted.
Troubleshooting coda 22
Dynamic Tracing allows the trace settings to be dynamically modified as the
application is running. This modification can be done remotely as well as using the
ovtrccfg tool.
1) Execute the following command to verify that the trace server process is running:
ps -ef | grep ovtrcd
2) If the trace server is not running, start the trace server manually using the
command
<InstallDir>/lbin/xpl/trc/ovtrcd
If trace server is not running currently, before startingyou must restart your
application which has to betraced as well.
3) Create trace configuration file with required sink type as shown above, but do not
place it in the locations mentioned above.
4) Run the command ovtrcadm –a < list of clients> e.g: ovtrcadm –a
15.76.19.234
The ovtrcadm is an administration tool to enable/disable the clients to monitor/view
traces as well as change the trace configuration; the clients here refer to nodes that
should be allowed/denied access for the same.
Note that the list of enabled clients persists for the lifetime of the trace server
process only, and hence, the above command has to be used to enable clients every
time the trace server is (re)started.
Best Practice: Disable the clients using ovtrcadm –d <list of clients> when you
have finished tracing from those clients.
5
7) If the SINK in your trace configuration file is FILE then the trace files are
generated in the specified location (on the machine where the application is running)
as mentioned in trace configuration file.
8) Tracing can be stopped or disabled by running ovtrccfg –server
<server_name> off
Best Practice: It is recommended that you disable tracing if you do not want to
trace any more, but do not stop the trace server. Else, you will have to restart your
application (after starting the trace server) when you need to trace again.
Essentially, it means that you must always run your trace server; you can disable it
when it is not required.
1) Open the Services window (or run “net start”) and verify that the state of the
service named HP OpenView Shared Trace Service has started.
Troubleshooting coda 23
2) Start the Trace Wizard and select the option to load a configuration file using the
following steps.
• Start the ovtrcgui tool
• From File menu, click Trace Wizard and then select Next.
• Select the Configure local applications by loading a saved
configuration option.
• From the Open dialog, locate and select the trace configuration file.
3) By default, a Trace Monitor window is displayed after the configuration is
completed. To open a new Trace Monitor window, use the following steps.
• From File menu, click New and then select Trace Monitor.
• Select the system you want to monitor trace messages and then click
OK. This starts a new trace monitor window.
4) Stop tracing using the following steps:
• Select the configuration window associated with the tracing.
• From the File menu, click Close.
Note: the trace server can be stopped using the command ovtrcadm -
srvshutdown in Unix and through service control manager in Windows. However,
you should not forcefully kill the trace server, e.g., with kill -9 on Unix. If the trace
server is forcefully killed, there might be problems in starting the trace server again.
XPL Logging:
XPL logging is used to report the run time status during program execution in their
respective locales. XPL logging is configured through log configuration file log.cfg
under <Data Dir>/conf/xpl/log/. The configuration cannot be changed during run
time.
Logging Utilities
XPL provide a tool ovlogdump which converts the locale neutral binary file and
displays in locale specific format. This tool is located under <OV Install Dir>/bin/.
For additional information on XPL Tracing and Logging the following resources are
available:
http://lcore.india.hp.com:9090/XPLTL.htm
http://ccntweb.bbn.hp.com/ccda/m10i/Components/L-Core/XPL-TL/2.5/
http://ovweb.external.hp.com/lpe/doc_serv/
Troubleshooting coda 24
10.3. ovc – Utility for stopping/starting CODA and other registered
processes
10.4. ovcodautil
Object Model
NumDataSources = 3
SCOPE
CODA
ELGIE
NumObjects = 14
SCOPE GLOBAL
SCOPE APPLICATION
SCOPE PROCESS
SCOPE TRANSACTION
Troubleshooting coda 25
SCOPE DISK
SCOPE NETIF
SCOPE CPU
SCOPE FILESYSTEM
SCOPE CONFIGURATION
CODA GLOBAL
CODA CPU
CODA NETIF
CODA FILESYSTEM
CODA DISK
Troubleshooting coda 26
10.7. perfstat
The perfstat utility checks the status of the performance products. It ships with
OVPA, PerfView and Glance. The descriptions below focus primarily on how it is used
to troubleshoot PerfView.
• -?
Lists all perfstat command line options
• –c
Shows system configuration information
• –e
Greps for all WARN and ERR lines in the status files.
• –f
Lists all status files and their sizes
• –p
Lists all active performance product processes.
• –t
This option tells perfstat to list just the last few lines (the tail) of each of the
performance product status files. This is useful when you are troubleshooting a
problem, because the most current information in the status files is always at the
end.
• –v
Lists the version information for all the performance products on the system. Unless
a patch has been installed, the version should match the list in the current Release
Notes.
• –z
Executes all perfstat commands and sends them to a file called perfstat.out. You can
also use this option to create a .tar file that contains the perfstat.out content. If
OVPA is also installed on the system, the OVPA raw logfile set is also included.
10.8. truss/trace/tusc
You can use truss, trace or tusc on any Unix system to trace procedures that are
being called during glance, xglance/gpm or scopeux or coda startup. Note that these
tools can be used to trace any program.
• To get a glance procedure and direct the output into the file /tmp/truss.out:
Eg:
#truss -f -a -e -d -o /tmp/coda_truss_out -p {pid of coda}
On HP-UX, truss doesn't belong to the standard toolset that is installed by default. It
needs to be installed on the test system first. Download it from one of these
locations:
HP-UX Porting and Archive Centre (http://hpux.cs.utah.edu/)
tusc - http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/tusc-7.3/
Troubleshooting coda 27
trace - http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/trace-1.6/
10.9. file core
When you have a core dump, you can use the file command to find out what
executable created the core file.
Example 1:
# file core
core: core file from 'alarmgen'
Example 2:
# what core
core:
$Header: nsswitch.conf,v 1.1 2001/07/18 14:46:57 twhite
Exp $
B.11.11_LR
$ PATCH_11.11/PHCO_25568 Dec 19 2001 06:21:26 $
Pthread Interfaces
$Revision: libpthread.1: @(#) depot-32pa
...
Example 3:
# file core
core: ELF 32-bit MSB core file SPARC Version 1, from
'scopeux'
# what core
core:
scopeux C.03.45.00 09/17/01 SunOS 5.6+ =*=
parm C.03.45.00 09/17/01 =*=
lv3.0.3.so C.03.45.00 09/17/01 SunOS 5.6+ =*=
Example 4:
# file core
core: core file from 'perflbd' - received SIGABRT
Example 5:
# file core
10.10. pstack/pfiles
pstack
Print a hex+symbolic stack trace for each lwp in each process.
Eg:
pstack <pid> > <output file>
pfiles
Report fstat(2) and fcntl(2) information for all open files in each process.
Eg: pfiles <pid of coda>
Troubleshooting coda 28
10.11. opcagt
opcagt is available through the OpenView operations agent (OVOA). In the case of
OVOA-EPC, ’coda’ runs as a child process of ‘opcagt’. Running “opcagt –status”
checks that ‘coda’ is running as a child process. It is possible that ‘coda’ may be
running, but ‘opcagt’ is not aware because it is not a child process.
When dealing with any problem-process, including ‘coda’, it’s best to stop and start
the entire agent, not just the individual process.
Troubleshooting coda 29
11. Troubleshooting coda
Check the hardware and software requirements. Check the release notes of the
product consuming Coda for known problems & workarounds. Check coda.log file for
error messages: <OvAgentDataDir>/log/coda.log
In this example a Windows based agent managed from an OVO Unix system is used.
The example applies as well for other managed platform operating systems like e.g.
HP-UX.
With OVO 7.x the file <OvAgentInstallDir>/lib/default.txt should contain the line
"DISABLE_PROSPECTOR=true" in the [coda] section.
Troubleshooting coda 30
Further checks:
C:\>ovcodautil -obj | more
Object Model
NumDataSources = 2
SCOPE
CODA
NumObjects = 14
SCOPE GLOBAL
SCOPE APPLICATION
SCOPE PROCESS
SCOPE TRANSACTION
SCOPE DISK
SCOPE NETIF
...
...
CONFIGURATION NON UTF8 GBL_THRESHOLD_NOKILLED
CONFIGURATION NON R32 GBL_THRESHOLD_PROCMEM
Unable to get requested raw data. ( CODA data collection is disabled - 12).
EPC, or Embedded Performance Component, is a feature shipped with OVOA 7.x and
later. EPC provides lightweight system metric collection and an internal interface to
allow OpenView SPIs to log data, and sends this data to the management server via
BBC.
Coda is a technology. It is:
§ a process that can perform several functions including system data
logging and processing incoming requests for metric data
§ a proprietary data store for system metric logging and application (SPI)
logging via an internal-only interface (dynamic data feed, DDF)
§ an interface to the BBC datacomm
Troubleshooting coda 31
Today, the overlap between EPC and Coda is 100% so it is easy to interchange the
two words. However, in the future EPC may be expanded to include other features
not provided by Coda.
Nothing. The SPI will continue to log to EPC/Coda. Changing the SPI logging from
EPC/Coda to OVPA/DSI is a manual process.
When I look for the ‘coda’ process on Linux, I see multiple entries - Why is
that ?
‘coda’ is a multi-threaded process. Currently, Linux shows a list of threads, not just
processes. Newest Linux distributions should just show the process list and omit the
thread details. Even when OVOA and OVPA are installed on the same machine, there
is only one ‘coda’ process.
perfstat is a command that has shipped with our performance products for a long
time (OVPA, OVPM, Reporter). perfstat checks the status of all critical process for
the performance products installed, including ‘coda’. opcagt is available via the
operations agent (OVOA). In the case of OVOA-EPC, ’coda’ runs as a child process of
‘opcagt’. Running “opcagt –status” checks that ‘coda’ is running as a child process.
It is possible that ‘coda’ may be running, but ‘opcagt’ isn’t aware because it’s not a
child process.
codautil is available to products that ship Coda (OVOA-EPC, OVPA 4). Running
“codautil –status” does a ping of sorts to the ‘coda’ process to make sure it is up and
running. When checking status, remember:
It’s important to check the status of all critical processes at the agent level.
This includes ‘coda’, among many other processes.
Keep in mind the differences between “opcagt –status” and “codautil –status”.
If “opcagt –status” shows ‘coda’ not running, check “codautil –status” before drawing
any conclusions.
Troubleshooting coda 32
OVPA 4.x “ovpa stop” OVPA starts coda if it is not running already
“ovpa start"
Can a customer get a copy of coda.db (i.e. want to know how metrics are
calculated)?
The ‘coda.db’ file is a cookbook of sorts used by the ‘coda’ process to determine how
to calculate metrics from raw data. This file is not a readable file. If a customer
needs to understand what the metrics mean, they should refer to the metric guide
For example, from coda.log: “CODA (pid 1072) Received sig 15 – Terminating”
This message can show up if OVOA is installed.
opcagt limits the amount of time it will wait for its monitored processes and
components to terminate after it has issued a stop command. If the time period
expires prior to the successful shutdown of the process or component, opcagt issues
a kill command to force the immediate termination. This is why you will periodically
see critical messages in the Message Browser that Coda was killed with a SIGTERM
(15). Coda catches the signal 15 and shuts down cleanly.
Ran "codautil -support" and it looks like got data from OVPA’s scope?! Why?
If the Coda log files do not exist, or if Coda system data logging is disabled, Coda will
actually return data from OVPA’s scope log if present.
Troubleshooting coda 33
4 on the other platforms (not shipping yet) will use datasources, and will likely keep
perflbd.rc around to preserve already defined data sources when customers upgrade.
The new datasources file will link or copy in what’s in perflbd.rc and make it available
to Coda.
2. If fixed port is not mandatory, then make the system automatically assign a port
with:
# ovconfchg -ns coda.comm -set SERVER_PORT 0
The "ovcodautil -ping" output shows coda is not available at the registered port.
The perfstat output also shows coda is not registered.
Due to invalid certificate requests were pending coda is not able to come up
and keeps giving following error message.
OvCoreId set : OK
Private key installed : FAILED
Certificate installed : FAILED
Certificate valid : FAILED
Trusted certificates installed : FAILED
Trusted certificates valid : FAILED
Troubleshooting coda 34
remove 3 config parameters -->
CERT_INSTALLED=TRUE
LAST_CERT_UPDATE=Fri Jul 14 16:42:27 2006 LAST_TRUSTED_CERT_UPDATE=Fri
Jul 14 16:42:27 2006
Or
After installing OVPA 4.5 on windows 2000/ 2003 nodes with OVO 7.30 agent
installed. The coda agent (sub agent 12) will be unregistered.
why the coda agent (version 7.5) is unregistered after restarting the ovo/ovpa
agent? This happens because OVPA 4.5 ships the latest CODA (CODA 10.X)
This is the expected behavior.
If you have one coda (version 10) running, will it work well for the OVO
agent/SPIs? Since some current SPIs are using coda to store data, Can the
SPIs work with coda 10 provided by OVPA 4.5? Or shoule we need version
coda 7.5 running at the same time for the SPIs?
- Yes, OVO agent SPIs will work with new CODA (version 10).
During OVPA 4.5 installation, CODA 7.X database will be migrated to CODA 10.
Therefore, only CODA 10.X needs to be running after installation of OVPA 4.5
If the agents are installed from a Windows management server then you may see
CODA 7.x still running after the installation.
If you still see that CODA 7.x is running after installing OVPA 4.5 please follow the
steps in "Workaround Notes" of the above defect to deregister CODA 7.x from OVO.
Troubleshooting coda 35
Policies deployed, metrics
ddflog - Tracing:
Add TRACE to ddflbd.rc (ddflbd.mwc) file
Eg:
DATASOURCE=ORADB_DUKAT_OPENVIEW
LOGFILE=/var/opt/OV/dbspi/dsi/oracle/openview.log TRACE=/tmp/openview.trc
Turning on Tracing
The datasource file shows datasource locations. The OVPA’s datasource file is created
when OVPA is installed, it is expected to already exist by ddfcomp. CODA’s
datasource file is created automatically. The ddfcomp utility generates this file if it
does not exist and inserts an entry in the file for the datasource name from the log
set name. The ddflog utility can then locate the datasource, based on this entry.
An additional use for this file is to turn on tracing. To enable debug tracing for
ddflog, insert the keyword TRACE in the file, optionally followed by a trace filename.
If you omit the trace filename, stderr is used. Any filename containing spaces must
be enclosed in quotes.
1.2.1. Syntax
datasource=datasource_name logfile=logfile_set [ trace [=tracefile] ]
Specific to : OVO 8 agents, OVPA 4.5 agent and OVPM 5.0 with a firewall between.
But how is communication between OVO / OVPM and OVO agent, Coda or
OVPA organized?
- When OVPM wants to contact coda/OVPA, it will first ask the Communication Broker
the port on which coda is running. By default the CB returns any random address. To
change this define a fixed address with the ovconfchg command.
"/opt/OV/bin/ovconfchg -ns coda.comm -set SERVER_PORT <port-number>"
changes the port on which coda is running.
If <port-number> is set to 0, a random port will get selected (which is default).
Now consider two addresses in firewall configuration. One for the Communication
Broker (Default 383, but also can change it with an "ovconfchg" command) and
another for Coda/OVPA.
Troubleshooting coda 36
- There is another configurable item called SERVER_BIND_ADDRESS. Coda server,
when it starts, can either bind to localhost or INADDR_ANY.
If coda binds to INADDR_ANY (which is the default), the CB returns coda's port to
the application and it can directly talk to coda, through coda's port, without going
through the CB. In this case, 2 ports are open through the firewall.
If coda's bind address is localhost, then the CB returns its own port to OVPM.
Therefore, OVPM has to route all its messages to coda through the CB. In this case,
only one port is open through firewall. This is the command to change the bind
address: "ovconfchg -ns coda.comm -set SERVER_BIND_ADDR localhost". This
ensures that only one port is open within the firewall.
Note: You should have the latest coda version to make this happen. Older versions
have defects in this area.
Troubleshooting coda 37
12. Log file management
On initial startup, one 5 MB Coda log file is created in the databases directory
eg Windows coda.log entry:
Initialization CODA 07/18/02 13:10:03.564
Starting The CODA(A.07.00.05:9:381)
Create File (c:\<OvAgentDataDir>\databases\coda00000)
Waiting for requests...
In the second week, you’ll see this in coda.log:
Starting The CODA(A.07.00.05:9:381)
Open File (c:\<install directory>\databases\coda00000)
Create File (c:\<install directory>\databases\coda00001)
Waiting for requests...
Coda continues to create a new log each week. When the 6th week is over, it
creates a 7th log and deletes the oldest.
Coda maintains a minimum of 5 full weeks of data, 1 log file per week.
At 6 weeks of data, a new log file is created and the oldest is deleted, always
leaving 5 full weeks of data.
Rollover is not configurable, and is triggered by time, not size
There is also no way to do any local export or archive to save the data
there is log file corruption (may show up as coda not starting, or error
message in coda.log)
log files too big and want to regain space
if you switch your SPI from logging to Coda (Mesa) to OVPA (DSI). It’s
important that the SPI datasource is not defined twice (once in Coda, once as
Troubleshooting coda 38
a DSI source) – OVPM and other consumers will not be able to distinguish
between the two and you won’t be able to tell which one the consumer is
getting data from. See SPI troubleshooting for more info. Be sure to also
delete ‘coda.db’ file and ‘coda.log’
CAUTION: Do not delete the log files if a SPI is logging data – it removes the SPI
data source in Coda and subsequent logging will fail
12.5. Configure Log File Size and Log File Number Limitation
Basically CODA uses XPL APIs to log its messages. The rollover mentioned here
happens based on the settings mentioned in the following file:
"/var/opt/OV/conf/xpl/log/log.cfg"
There will be two entries in the above file which are related to Text logs:
TextSizeLimit=1000000
TextFileLimit=10
Once the log file(coda.txt) reaches "TextSizeLimit", it gets rolled over automatically
and a new file gets created.
The max number of files are determined by the setting "TextFileLimit" (10 in this
case)
Troubleshooting coda 39
13. Configuring coda in single port and multi port
communication (OVPM, OVPA, coda, OVO agent)
How to verify?
ovcodautil -n <hostname> -ping
Eg: ovcodautil –n ovphpt4 -ping
Ping of 'OvBbcCb' at: 'http://ovphpt4:383/Hewlett-
Packard/OpenView/BBC/ping' successful
Ping of 'Coda' at: 'http://ovphpt4:62581/Hewlett-Packard/OpenView/Coda/'
successful
Note the different port numbers in two outputs
Troubleshooting coda 40
How to verify?
ovcodautil -n <hostname> -ping
Eg: ovocdautil –n ovphpt4 -ping
Ping of 'OvBbcCb' at: 'http://ovphpt4:383/Hewlett-
Packard/OpenView/BBC/ping' successful
Ping of 'Coda' at: 'http://ovphpt4:1025/Hewlett-Packard/OpenView/Coda/'
successful
Troubleshooting coda 41
14. Known problem and workarounds
PROBLEM:
This document explains some specials a user might experience, if on a
System with an OVO Agent version A.07.x the OV Performance Agent
(OVPA) in version A.04.50 (or higher) is applied.
As an example the user might encounter the error of an Aborted CODA Agent:
# perfstat
**********************************************************
*** perfstat for mysystem on Tue Apr 18 15:56:47 DFT 2006
*** AIX mysystem 2 5 004D326A4C00
**********************************************************
OVPA status:
Running scopeux (OVPA data collector) pid 7632
Running midaemon (Measurement Interface daemon) pid 10426
Running ttd (ARM registration daemon) pid 16244
# ovcodautil -status
OvXplMsg::LogicError_t
CONFIGURATION
Operating System - HP-UX, Sun Solaris
Version - 7.x
Troubleshooting coda 42
Product - operations for UNIX
RESOLUTION:
The root cause might be a wrong order of starting up the processes. The new OVPA
A.04.50 uses some new technology that is commonly shared in an OVO A.08.x
environment. The technology was not available at the time OVO A.07.x was present.
The strange behavior in the above perfstat output is that in one line the coda agent
is reported to be aborted and later on it is reported as running:
coda OV Performance Core AGENT,CODA Aborted
Subagent 12:
Performance Agent /usr/lpp/OV/bin/coda -redirect (5062) is running
So what is correct?
The ovcodautil -status command fails, so this indicates that there is a problem and
neither the OVPA nor the EPC of the OVO Agent is collecting performance data. To
resolve this, the user needs to stop the OVO Agent and OVPA Agent entirely using
the following commands:
# opcagt -kill
# ovc -kill
# ovpa -stop
# /opt/perf/bin/ttd -k
# /opt/perf/bin/midaemon -T
Check with the command "perfstat" that all processes are stopped.
Then start first the OVPA agent and then the OVO Agent:
# ovpa -start
# opcagt -start
Eg:
# ovpa -start
# perfstat
**********************************************************
*** perfstat for mysystem on Tue Apr 18 14:32:44 DFT 2006
*** AIX mysystem 2 5 004D326A4C00
**********************************************************
Troubleshooting coda 43
OVPA Status:
Running scopeux (OVPA data collector) pid 15144
Running midaemon (Measurement Interface daemon) pid 19932
Running ttd (ARM registration daemon) pid 13436
# opcagt -start
# perfstat
**********************************************************
*** perfstat for mysystem on Tue Apr 18 14:33:36 DFT 2006
*** AIX mysystem 2 5 004D326A4C00
**********************************************************
OVPA status:
Running scopeux (OVPA data collector) pid 15144
Running midaemon (Measurement Interface daemon) pid 19932
Running ttd (ARM registration daemon) pid 13436
Troubleshooting coda 44
-------------------------
Control Agent /usr/lpp/OV/OpC/opcctla (10380) is running
Message Agent /usr/lpp/OV/OpC/opcmsga (7570) is running
Subagent 1:
Action Agent /usr/lpp/OV/OpC/opcacta (19012) is running
Message Interceptor /usr/lpp/OV/OpC/opcmsgi (18128) is running
Subagent 12:
Performance Agent /usr/lpp/OV/bin/coda -redirect isn't running
# Note that in the perfstat output in the OVO Agent section still the Subagent 12
(which is coda) is reported NOT to be running, while above in the section "OV
Operations Agent Status" coda is reported as running.
The three processes ovcd, ocbbccb and coda in this section are part of the new
shared technology. Before applying the OVPA A.04.50 software, these processes did
not show up in a perfstat listing at all.
The important fact is that the line coda OV Performance Core AGENT,CODA (12704)
Running is correct and the line Performance Agent /usr/lpp/OV/bin/coda -redirect
isn't running should be ignored.
To check that both agents correctly log data, the user can use the ovcodautil
command.
Eg:
# ovcodautil -status
Coda daemon running
# ovcodautil -support
Data source CODA enabled.
=== 04/18/06 14:25:00
Instance :0
GBL_COLLECTOR : SCOPE/UX C.04.50(00)
GBL_INTERVAL : 215
GBL_OSNAME : AIX
GBL_OSVERSION :5
GBL_OSRELEASE : 5.2.0
GBL_RUN_QUEUE : 0.10
Note: In some special environments the data collection of the EPC might be
disabled. Then you would not see in the above command the first section named
"Data source CODA enabled." but only the second one "Data source SCOPE enabled.
Troubleshooting coda 45
PROBLEM:
OVO Agent is collecting coda performance data on one server. There is no
network connection from this server to the server where OV Performance
Manager is installed.
How can the coda perfomance data be analyzed?
CONFIGURATION
Operating System - HP-UX
Version - C.05.x
Product - performance manager
RESOLUTION:
This procedure is tested where the OVO Agents involved are running on the same
platform (HP-UX in the example) and are of the same version.
Troubleshooting coda 46
PROBLEM:
Is the default port for OVPA the same as Coda (381)?
CONFIGURATION
Operating System - AIX, HP-UX, Linux, NCR, SNI, Sun Solaris, Tru64 UNIX, Windows
Version - C.03.x, C.04.x
Product - performance agent
RESOLUTION
The answer really depends on what version of OVPA is being referred to. The 3.x
versions of OVPA agents used RPC-based technology and as a result, require multiple
ports to be opened up. Many of these ports are dynamically allocated and the port
range is configurable.
With the 4.0 and 4.1 versions of OVPA (Linux), a change was made to http based
communication to enable easier firewall configuration. These agents use the Coda
subcomponent internally, which in turn uses the BBC component for http(s)
communication. The BBC's location broker runs on port 383 and Coda binds to port
381 in the OVPA 4.0 and 4.1 Linux versions.
In the forthcoming OVPA 4.5 version (all platforms), we are switching over to Coda
version 10 (which in turn uses BBC5) for https communication. In this version, there
is only the BBC broker's port 383 which is fixed. Coda will dynamically assign its
ports, which clients like OVPI do not have to know. All communication including Coda
communication is brokered through BBC's location broker.
The legacy DCE/RPC datacomms will continue to be supported in the upcoming 4.5
versions, but at any point of time only one mode of communication can operate;
either DCE/RPC or BBC http based, not both.
PROBLEM
The SPI for Unix Administrator's Reference Guide 02.00 states on pages 65
and 68 respectively on Measureware/Glanceplus and CODA monitor
template naming conventions:
Monitor Name A string that must begin with one of the following prefixes:
OSSPI-MWA_ for OV Performance monitors
OSSPI-GP_ for GlancePlus monitors
Monitor Name A string that must begin with one of the following prefixes:
OSSPI-CODA_ for all Coda monitors
Is there a reason for these restrictions?
CONFIGURATION
Operating System - HP-UX, Sun Solaris
Version - A.01.xx
Product - smart plug-in for UNIX os
RESOLUTION:
There is indeed a reason for these restrictions, but this is only limited to MWA and
GP monitor templates:
After distribution of the templates to the managed node, an OSSPI script will run to
check all monitor templates that are distributed:
OSSPI-alarmdef_write_1
The script will decode the templates on the node and use the information to create a
new alarmdef (MWA) or a new syntax_gp (GP) file. However, it will need to filter out
all non-relevant templates first to accomplish this. This filtering is based on the
Troubleshooting coda 47
names of the templates, hence they need to start with OSSPI-MWA for Measureware
templates and OSSPI-GP_ for Glanceplus templates.
For OSSPI CODA monitor templates there is actually no reason for such a naming
restriction, save to keep the templates manageable more easily. Outside of that the
CODA templates can carry any possible name.
PROBLEM
In an environment with OVOU 8.x agents, the full OV Performance Agent (OVPA) is
installed to get more performance logging functionality. Only the data from OVPA will
be used for analysis and reporting.
Is there a way to disable the coda agent, running as a subagent of the OVO Agent?
CONFIGURATION
Operating System - HP-UX, Sun Solaris
Version - 8.x
Product - operations for UNIX
RESOLUTION
Yes, there is. The following command can be used to disable the coda agent:
ovcreg -del coda
The following commands can be used to enable the coda agent again.
ovcreg -add /opt/OV/newconfig/DataDir/conf/perf/coda.xml
ovc -start coda
The disable/enable procedures do not delete coda data which is gathered already,
but if the data is more than 5 weeks old, when the agent is enabled again, it will be
deleted as a result of the normal coda data maintenance process.
(The same procedure can be used for the other subagents)
CR#: QXCR1000282671
submitter text notes "opcagt -stop" stops coda on ovpa4.5/OVO8.x node Fix Text
PROBLEM
When trying to add a node to OVPI running an OVO https agent, a_discovery
gathers 0 files.
When testing the connection with ovcodautil, the following error message appears:
ovcodautil -ping -n admhost. Ping of 'OvBbcCb' at: 'http://admhost:383/Hewlett-
Packard/OpenView/BBC/ping' successful
Exception caught calling OvBbcCb::LocateSherver for node: 'admhost' and URL:
'/Hewlett-Packard/OpenView/Coda/' Exception='(sec.core-113) SSL certificate
verification error (The presented peer certificate is not trusted. The certificate
verification chain could not be built.).
Everything seems to run successfully from the OVO side.
Does a separate certificate for OVPI need to be installed?
Troubleshooting coda 48
CONFIGURATION
Operating System -
Version - 5.0, 5.1
Product - performance insight
RESOLUTION:
The output of "ovconfget" run on the OVO node shows that the section coda.comm
does not have the SERVER_PORT entry registered to port 381. A firewall between the
OVPI server and the OVO node has been setup to allow only communication on port
381.
Using the ovconfchg command, to change the port entry to 381 on the OVO node,
allows the node to be discovered correctly and the Coda data can be collected.
Port 381 is the default port.
Troubleshooting coda 49
15. Latest Coda Patches
Troubleshooting coda 50