You are on page 1of 38

automating emc ionix controlcenter

host agent deployment and


maintenance process
EMC Proven Professional Knowledge Sharing 2011
Mehmet Ozdemir
Program Delivery Team Lead, Turkey
Technology Solutions Group
EMC
mehmet.ozdemir@emc.com
EMC Proven Professional Knowledge Sharing 2

Table of Contents
Introduction ..................................................................................................................... 4
What Is ECC? ................................................................................................................. 5
How ECC works .............................................................................................................. 6
How ECC discovers hosts ............................................................................................... 8
What makes host discovery process and maintenance difficult? ..................................... 8
What is the solution? ....................................................................................................... 9
emc_checker script ....................................................................................................... 11
How emc_checker works .............................................................................................. 13
Conclusion .................................................................................................................... 17
Appendix A .................................................................................................................... 18
Source Code ................................................................................................................. 18
Appendix B .................................................................................................................... 36
Sample script outputs (emc_checker.sh) ....................................................................... 36
Appendix C ................................................................................................................... 38
Sample E-mail Report ................................................................................................... 38

EMC Proven Professional Knowledge Sharing 3

Table of Figures


Figure 1 - Communication between a single infrastructure host and ECC host agents /
SYMAPI server components ........................................................................................... 7
Figure 2 - emc_checker flowchart diagram .................................................................... 12
Figure 3 - NFS directory structure ................................................................................. 14
Figure 4 - Sample INQ Output ....................................................................................... 16





Disclaimer: The views, processes, or methodologies published in this article are those
of the author. They do not necessarily reflect EMC Corporations views, processes, or
methodologies.
Introduction
As strange as it may be to state that you can be successful where there is no
information, it is equally so to mention that one is bound to failure where information is
vast and available. Information should be regarded as a living fact and it should be
recalled that information is valuable and useful as long as it is managed properly and is
kept up-to-date. The sole existence of information is not sufficient. Incorrect, missing,
and most importantly, out-of-date information may cause loss of labor, time, and money.

EMC Ionix

ControlCenter (ECC) is one of EMCs core products that contain a vast
amount of information and is well worth its relatively high cost. As is well known, ECC is
used to manage a large SAN cloud, as well as scale and report. The process of keeping
the ECC information up-to-date and available are achieved through the implementation
and proper maintenance of software agents on the SAN environment by the storage
administrators. The storage administrators are also responsible for using ECC as a tool
that simplifies their daily life.

It is generally understood that new sales campaigns are introduced frequently due to stiff
competition, especially at large telcos. New applications and servers are introduced to
the environment all the time. In such dynamic environments, storage administrators may
become aware that new servers are added to the ECC environment, that the new
servers are not yet defined in the ECC, or that some of the older host agents may not
work anymore. Storage administrators have to track down the responsible server
administrator, question why the agents stopped working, or request that new agents are
installed on the servers from the already busy server administrators. Even though the
storage administrators may eventually share routine installation procedures with the
server administrators, they may end up having to wait hours or, sometimes, days. In
some environments, administrators may routinely change jobs or go on vacation, ending
up in situations where the root password may not be even known. Just as the situation is
resolved, this cycle of new and updated servers continues. Soon enough, a wonderful
product such as ECC that could make life easier may seem like an unattainable Utopia
and a part of an annoying infinite cycle by the users.

Turkcell, one of the largest GSM operators in Turkey, has experienced this problem for
many years now. A lot of effort has been wasted in an upstream battle. As the onsite
EMC Proven Professional Knowledge Sharing 5

EMC engineer, I devised a flexible and easily adaptable solution using UNIX shell
scripting that addresses this problem and keeps the information provided by ECC up-to-
date, by combining native tools from UNIX and EMC.

In summary, this script is copied remotely to all servers over the HP OpenView
framework. It detects all UNIX servers connected to the SAN, enabling ECC discovery
and remotely installs and configures all necessary software depending on the type of
operating system, starts up the services. If the applications are already installed, the
script checks the services and starts only the required ones. When errors are
encountered, the log files are collected and copied over to a centrally managed location,
accessible by the storage administrators, while keeping track of all activity in a daily log
on a separate remote server. If a host is connected to EMC storage, the script collects
an emcgrab (EMCs utility to analyze host side issues and interoperability) and
routinely emails its activities.

Using this script, storage administrators at Turkcell no longer need to worry about
installing and maintaining EMC host agents when they allocate storage from an EMC or
non-EMC storage to a host. They can focus on more important tasks rather than
worrying whether or not the server will appear on the EMC console. The later sections of
this article will contain details about the methods utilized and the value it contributed to
the ECC environment at Turkcell.

What Is ECC?
ECC is a family of products that provide customers with an integrated approach to
managing their multi-vendor storage environment and automating many of their common
storage management tasks. It is a comprehensive storage resource management (SRM)
application that delivers end-to-end visibility and control for heterogeneous customer
environments.

There are many components that can be part of a working ECC environment. Open
integration components (OIC), which complement licensed applications and packages
based on the implementation, are included in the ECC base product.


EMC Proven Professional Knowledge Sharing 6

The sub-components of OIC are:
Infrastructure
o Repository
o Key Management Server
o StorageScope Repository
o ECC Server
o Store
o Master Agent
Agents
o Common agents
o Storage agents
o Host agents
Host agent for platform (Windows, Solaris, HP-UX, AIX, Linux,
etc.)
SYMAPI Server
o Database agents
o Combination agents
o Connectivity agents
o Backup agents
o Other agents
User interfaces
Common services

How ECC works
In a typical ECC environment, the objects are discovered via agents. An agent is a
process that runs on a host and collects information from various objects. Once the
agent starts and initializes, it is constantly active, monitoring for configuration changes
and various conditions, and is always waiting for transactions (pieces of work) to arrive.
All supported components building up a SAN environmentincluding storage,
connectivity devices, databases, and hostscan be discovered via agents working with
various purposes. In addition, the agent publishes data to the ECC Server or Store
periodically, or as needed.

EMC Proven Professional Knowledge Sharing 7

The Store is a process that populates the Repository with persistent data from the
agents. It provides a store-and-retrieve interface between the agents and the Repository,
which contain a relational database. Installation and maintenance of the database is fully
embedded into ECC.

ECC provides host agents for all supported Windows and UNIX platforms apart from
TRU64 and Solaris X86-64 hosts. Proxy discovery is the preferred way to discover
TRU64 and Solaris x86-64 hosts. The Proxy discovery method uses SYMAPI Server
daemon of Solutions Enabler software which is installed on TRU64 or Solaris x86-64 to
be discovered to communicate with the common mapping agent (CMA) already installed
on a supported host. Consequently, Solutions Enabler software must be deployed and
SYMAPI daemon must be running on related TRU64 and Solaris x86-64 hosts to be
discovered. The rest of the supported UNIX and Windows platforms can be discovered
via Master and host agents.

Figure 1 illustrates the communication between a single infrastructure host and ECC
host agents / SYMAPI server components that reside in each host.


Figure 1 - Communication between a single infrastructure host and ECC host agents /
SYMAPI server components

EMC Proven Professional Knowledge Sharing 8

How ECC discovers hosts
ECC provides host agent software for discovering most open systems including
Windows, Solaris (SPARC), HP-UX, and Linux. There is no host agent software
developed for TRU64 and Solaris X86 hosts.

For Windows, Solaris (SPARC), HP-UX, and Linux hosts:
Master and host agent should be installed and master agent service should be started.
After starting master agent service, host agent service is triggered by master agent and
host appears on ECC console. To ensure up-to-date host information, master and host
agent must be running and communicating with ECC server all the time.

For TRU64 and Solaris X86 hosts:
Solutions Enabler software must be installed on the host with SYMAPI Server license.
SYMAPI Server daemon must be started and the host must be discovered via assisted
discovery feature on the ECC console. To ensure up-to-date host information, SYMAPI
Server daemon must be running and communicating with ECC server all the time.

What makes host discovery process and maintenance difficult?
Typically, hosts are the most difficult objects discovered by ECC. In large SAN
environments, storage administrators generally do not have access to hosts even with a
non-root user.

Turkcell has a large UNIX host environment used for the most business-critical
operations. As of todays numbers, there are over 3,000 UNIX and Windows hosts, with
hundreds of them connected to the SAN environment. As expected from a telco
company, these numbers change frequently. Turkcell uses the ECC environment for
reporting, management, and monitoring purposes. Special attention is given to the UNIX
hosts since the most critical applications run on them. As a company strategy, storage
administrators aim to discover a newly arrived UNIX host via ECC before connecting it to
SAN for the first time. However, since this involves the human factor, I have witnessed
many hosts connected to SAN but not discovered by ECC.

It is difficult to track master and host agents or Solutions Enabler SYMAPI Server
daemon that may be running since storage administrators usually do not have direct
EMC Proven Professional Knowledge Sharing 9

access to hosts apart from extreme conditions. When a service is down for any reason,
e.g. file system full, killed by the administrator, other conflicting applications, and so
forth, it must first be noticed by the storage administrator while checking reports, and
then communicated to the hosts UNIX administrator. At that time, UNIX administrators
may maintain the agent or postpone this activity since they may be dealing with a more
important host issue.

Similar scenarios can be played out numerous times, but this example is sufficient to
prove that human relations is the key factor affecting host information streaming to an
ECC environment. This is the issue of not only ECC but also all other ECC-like products
which collect information from objects via agents.

What is the solution?
There are two possible solutions for this issue.
(1) The design of ECC must be changed so that it can discover hosts without using
an agent, eliminating the need for host administrator action for agent deployment
and maintenance.
(2) A host connected to SAN must be discovered as soon as possible, and master
and host agent or Solutions Enabler (SYMAPI) services should be kept up and
running. All of these activities should be reported to the storage administrators so
that appropriate actions can be taken when needed.

The first solution is not yet an option, since ECC was designed so that it collects its
information from the hosts via agents just as similar products do. This option may
become available in the future, but not today.

However, we had the chance to implement the second solution at Turkcell using native
tools of EMC and the industry.

Lets look at the picture from the top. I considered sending only one script to the UNIX
administrators and expected them to run it; thats all. After running the script, we should
be sure that if that host is connected to any storage arrays, it will appear on the ECC
console as a discovered host.

EMC Proven Professional Knowledge Sharing 10

However, running a script and installing necessary agent software was not enough. We
needed to create a central tracking mechanism to catch problematic hosts where agent
software could not be installed, services could not be started somehow, and so on. Also,
it would be ideal to save necessary software installation logs from hosts into a central
location if any errors occur.

We had to make sure that no install binaries are copied to hosts local file systems. This
requires searching for file systems where necessary free space exists for copying
installation binaries. Also, network administrators might complain about suffering
network resources. For these reasons, I chose Network File System (NFS) as a central
point for not only storing software binaries but also script and software log files.

Turkcell has a rapidly-changing environment. On occasion, the UNIX administrators
approach storage administrators with interoperability queries for EMC storage arrays.
Sometimes storage administrators want to investigate whether or not certain hosts are
supported in case their configuration might be changed after initial provisioning. All these
processes require emcgrab script to be run and output to be collected from the affected
hosts.

When opening a service request with EMC support, one of the first things requested is
emcgrab output. Emcgrab is a utility developed by EMC for gathering detailed
information regarding a current system configuration. The data collected assists the
EMC support analyst with fault isolation. The output is a compressed file which may be
analyzed through tools such as HEAT (Host Environment Analysis Tool) that is
accessible thru Powerlink.

For these reasons, I decided to not only take care of the host discovery process but to
also collect emcgrab from the hosts which are connected to any EMC storage arrays. In
other words, I decided to take advantage of not only maintaining ECC discovery but also
collecting host information which may be used for any reason.

To accomplish all these goals, we had powerful technologies such as UNIX shell,
scripting, NFS sharing, INQ utility, and so on.

EMC Proven Professional Knowledge Sharing 11

The Inquiry utility (INQ) is a command-line troubleshooting utility that displays
information on storage devices. I parsed INQ output to determine if that host is
connected to any storage arrays. If so, then the script proceeds with either installing
master and host agent or Solutions Enabler SYMAPI daemon. It is used also for filtering
hosts connected to EMC storage arrays from which emcgrab output will be collected.

Any SNMP-based management and alerting tool, such as HP OpenView, can be used to
deploy our script to hosts without user interaction. For reporting purposes, NFS protocol
is used to create log files at a central location where storage administrators have access.
Reporting was done via native UNIX mailx command.

emc_checker script
emc_checker is the name of this scripting solution. It has several modules. After
assembling all parts the following flowchart appeared.

EMC Proven Professional Knowledge Sharing 12


Figure 2 - emc_checker flowchart diagram

EMC Proven Professional Knowledge Sharing 13


How emc_checker works
A UNIX host where storage administrators have root user access is selected and NFS
server is configured on that host. NFS server configuration steps can be found from the
UNIX vendors documentation. I will not go into particulars here.

The following NFS directory structure (depicted in Figure 3) must be formed.

EMC Proven Professional Knowledge Sharing 14


Figure 3 - NFS directory structure
EMC Proven Professional Knowledge Sharing 15

When emc_checker is run on the UNIX host, the first thing it checks is whether that host
can access the NFS share. The IP address and NFS share directory are defined as
variables at the beginning of the code and can be modified easily when needed. If the
script cannot access that NFS share, it exits with an error message. For example, this
most likely occurs if the host on which the script runs is behind a firewall. If this is the
case, network administrators should take action to change firewall rules. If emc_checker
can access NFS share successfully, it mounts it and the script starts working on the
host.

I preferred storing the actual functions and doing the job in a file on the NFS share and
including them when the emc_checker is run on the host in order to avoid re-sending the
script to the hosts if I modify the code. sh.library is the name of the library where actual
functions are called from emc_checker.

After including sh.library from emc_checker, a host directory is created on the NFS
share with the current date, e.g. /nfs_share/logs/20110103/my_host. All user output is
also reflected to the script log file on NFS share so that storage administrators can check
if they need to do so.

emc_checker checks if the host is connected to any storage arrays via INQ script. Figure
4 illustrates a sample INQ output.

EMC Proven Professional Knowledge Sharing 16


Figure 4 Sample INQ Output

The clue to figure out if this host is connected to any storage arrays are the :VEND and
:PROD columns. Basically, I parsed this output and searched for :SYMMETRIX, :DGC,
:DF600, :DF700, :OPEN for EMC Symmetrix

, EMC CLARiiON

, HDS 9500V, HDS


TagmaStore AMS, and HP/HDS high-end arrays, respectively. This control mechanism
may be altered easily in sh.library.

Then, the script makes a decision. If the host is either TRU64 or Solaris x86-64, the
script carries on with Solutions Enabler SYMAPI daemon section. If the host is AIX, HP-
UX, Linux, or Solaris, it goes on with ECC Master and Host agent section. As explained
previously, this is because ECC does not have agents for TRU64 or Solaris x86-64
platforms. These hosts are discovered via Solutions Enabler SYMAPI daemon.

Appendix A contains all of the source code for those who may be interested. Appendix B
contains sample script outputs taken from various UNIX hosts.

In summary, the script does the following for TRU64 or Solaris x86-64 hosts:
It installs Solutions Enabler software in unattended mode and SYMAPI daemon
license, and starts the daemon.
If the daemon is not running, it attempts to start.
It logs every single step to the hosts log file.
EMC Proven Professional Knowledge Sharing 17


In summary, the script does the following for AIX, HP-UX, Linux, and Solaris hosts:
It adds ECC server information to /etc/hosts file to guarantee name resolution.
If needed, it installs ECC Master agent in unattended mode and starts the
service. It waits for host agent to be triggered by master agent.
If the master agent is not running, it starts it, logging every single step to hosts
log file.
If any step fails, it copies the ECC installations logs and ECC installation folder to
the hosts folder in NFS share directory.

After finishing the steps above, the script does the following for all host types:
It checks if the host is connected to any EMC storage arrays.
It collects emcgrab output in unattended mode and saves the output to the logs
directory in the NFS share folder.
Ultimately, it delivers a daily report e-mail explaining the activities done with the
name of the hosts. A sample e-mail report can be found in Appendix C.

Conclusion
emc_checker helped Turkcell storage administrators deploy and maintain the ECC host
discovery process cycle. It minimized the interaction and time spent for ECC agent
maintenance between storage administrators and UNIX administrators, consequently
allowing people to spend their time on on-going projects and actual business. The script
also helped to keep host information in the ECC database up-to-date as much as
possible. This was reflected in StorageScope

reports and queries, which resulted in the


managers confidence of reaching true and valuable information all the time.

Comment of Erhan Savas, Storage Group Manager, Turkcell:
Using emc_checker helped me and my team to reach the most up-to-date host
information in ECC environment. We highly trust the accuracy of ECC and
StorageScope reports with the help of emc_checker tool.

EMC Proven Professional Knowledge Sharing 18

Appendix A
Source Code

emc_checker.sh:

#!/bin/ksh

#
# File name : emc_checker.sh
# EMC CHECKER v2.0
# Last revised on 2010, 16 Oct
# Mehmet OZDEMIR (ozdemir_mehmet@emc.com)
# EMC
#

#
# Global variables
#
NFS_SERVER_IP=xxx.xxx.xxx.xxx
NFS_SHARE_DIR=/nfs_share
LOCAL_MOUNT_POINT=/EMC_ECCINSTALL
SCRIPTS_LOCATION=scripts
NULL=/dev/null
START_TIME=$(perl -e 'print time')
umask 000
#
# Main
#
clear
umount -f $LOCAL_MOUNT_POINT 1>$NULL 2>$NULL
cd /
mkdir $LOCAL_MOUNT_POINT 1>$NULL 2>$NULL
mount $NFS_SERVER_IP:$NFS_SHARE_DIR $LOCAL_MOUNT_POINT 1>$NULL 2>$NULL

if [[ $? != 0 ]]; then
print "Could not connect to NFS Server --> $NFS_Server_IP"
ping -c 1 $NFS_SERVER_IP|grep icmp_seq
else
ECHO=on

. $LOCAL_MOUNT_POINT/$SCRIPTS_LOCATION/sh.library

print "### $PROGRAM_NAME $PROGRAM_VERSION started. ###"

cd $LOCAL_MOUNT_POINT/$SCRIPTS_LOCATION/

CREATE_HOST_LOGS_FOLDER

WRITE_TO_LOG $ECC_LOG_FILE "Local mount point : $LOCAL_MOUNT_POINT"
WRITE_TO_LOG $ECC_LOG_FILE "$NFS_SERVER_IP:$NFS_SHARE_DIR mounted."
WRITE_TO_LOG $ECC_LOG_FILE "Checking if $HOST_NAME is supported or not."

OS_TYPE=$(DETERMINE_OS_TYPE)

if [[ $OS_TYPE = "unsupported" ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "'uname -a' output of $HOST_NAME -->
$(uname -a)"
WRITE_TO_LOG $ECC_LOG_FILE "$HOST_NAME is unsupported."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE "unsupported"
else
EMC Proven Professional Knowledge Sharing 19

WRITE_TO_LOG $ECC_LOG_FILE "$HOST_NAME is $OS_TYPE host which is
supported."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE "supported"
WRITE_TO_LOG $ECC_LOG_FILE "Checking if $HOST_NAME is connected to
any storage arrays."
if [[ $(IS_HOST_CONNECTED_TO_STORAGE_ARRAYS $OS_TYPE) = $FALSE &&
$OS_TYPE != "solaris.i386" ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "$HOST_NAME is not connected to
any storage arrays."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"not_connected_to_storage_arrays"
else
WRITE_TO_LOG $ECC_LOG_FILE "$HOST_NAME is connected to
storage array(s)."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"connected_to_storage_arrays"
if [[ $OS_TYPE = "tru64" || $OS_TYPE = "solaris.i386" ]];
then
if [[ $OS_TYPE = "tru64" ]]; then
TRU64_CLUSTER_NODE=$(IS_TRU64_CLUSTER_NODE)
fi
if [[ $OS_TYPE = "tru64" && $TRU64_CLUSTER_NODE !=
$FALSE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "$HOST_NAME is a
TRU64 cluster node. Solutions Enabler should be installed manually."
WRITE_TO_LOG $ECC_LOG_FILE "Cluster
information : $TRU64_CLUSTER_NODE"
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"tru64_cluster_node"
else
WRITE_TO_LOG $ECC_LOG_FILE "Checking if
Solutions Enabler is already installed or not."

SOLUTIONS_ENABLER_VERSION=$(GET_SOLUTIONS_ENABLER_VERSION)
if [[ $SOLUTIONS_ENABLER_VERSION = $FALSE ]];
then
WRITE_TO_LOG $ECC_LOG_FILE "Could not
find any installed Solutions Enabler."
WRITE_TO_LOG $ECC_LOG_FILE "Installing
Solutions Enabler."
if [[ $(INSTALL_SOLUTIONS_ENABLER
$OS_TYPE) = $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE
"Solutions Enabler installed."
WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "solutions_enabler_installed"
else
WRITE_TO_LOG $ECC_LOG_FILE
"Solutions Enabler could not be installed."
WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "solutions_enabler_could_not_be_installed"
fi
else
WRITE_TO_LOG $ECC_LOG_FILE "Solutions
Enabler $SOLUTIONS_ENABLER_VERSION already exists."
WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "solutions_enabler_exists"
fi

if [[ $(GET_SOLUTIONS_ENABLER_VERSION) !=
$FALSE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "Checking if
SYMAPI daemon license is installed or not."
EMC Proven Professional Knowledge Sharing 20

if [[
$(IS_SYMAPI_DAEMON_LICENSE_INSTALLED) = $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE
"SYMAPI daemon license is already installed."
else
WRITE_TO_LOG $ECC_LOG_FILE
"SYMAPI daemon license could not be found. Now installing the license."
if [[
$(INSTALL_SYMAPI_DAEMON_LICENSE) = $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE
"SYMAPI daemon license installed."
else
WRITE_TO_LOG $ECC_LOG_FILE
"SYMAPI daemon license could not be installed."
fi
fi

if [[
$(IS_SYMAPI_DAEMON_LICENSE_INSTALLED) = $TRUE ]]; then

WRITE_TO_LOG $ECC_LOG_FILE
"Checking if SYMAPI daemon is running or not."
if [[ $(IS_SYMAPI_DAEMON_RUNNING)
= $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE
"SYMAPI daemon is already running."
WRITE_TO_REPORTS_FILE
$HOST_NAME $OS_TYPE "symapi_already_running"
else
COUNTER=1
until [[
$(IS_SYMAPI_DAEMON_RUNNING) = $TRUE || $COUNTER > 3 ]] ; do
WRITE_TO_LOG
$ECC_LOG_FILE "SYMAPI daemon is not running. Now trying to start.
Attempt#$COUNTER ..."
if [[
$(START_SYMAPI_DAEMON) = $TRUE ]]; then
WRITE_TO_LOG
$ECC_LOG_FILE "SYMAPI daemon started after $COUNTER attempt(s)."

WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE "symapi_started"
else
WRITE_TO_LOG
$ECC_LOG_FILE "SYMAPI daemon could not be started. Waiting 10 seconds."
sleep 10
fi
let
COUNTER=$COUNTER+1
done
if [[
$(IS_SYMAPI_DAEMON_RUNNING) = $FALSE ]]; then

WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE "symapi_could_not_be_started"
fi
fi

if [[ $(IS_SYMAPI_DAEMON_RUNNING)
= $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE
"Checking if SYMAPI options file should be edited."
if [[
$(SYMAPI_OPTIONS_FILE_SHOULD_BE_EDITED) = $TRUE ]]; then
EMC Proven Professional Knowledge Sharing 21

WRITE_TO_LOG
$ECC_LOG_FILE "SYMAPI options file should be edited."


COUNTER=1
until [[
$(IS_SYMAPI_DAEMON_RUNNING) = $FALSE || $COUNTER > 3 ]] ; do
WRITE_TO_LOG
$ECC_LOG_FILE "SYMAPI daemon is running. Now trying to stop. Attempt#$COUNTER
..."
if [[
$(STOP_SYMAPI_DAEMON) = $TRUE ]]; then

WRITE_TO_LOG $ECC_LOG_FILE "SYMAPI daemon stopped after $COUNTER
attempt(s)."
else

WRITE_TO_LOG $ECC_LOG_FILE "SYMAPI daemon could not be stopped. Waiting
10 seconds."

WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"symapi_daemon_could_not_be_stopped_before_editing_options_file"
sleep
10
fi
let
COUNTER=$COUNTER+1
done

if [[
$(IS_SYMAPI_DAEMON_RUNNING) = $FALSE ]]; then
WRITE_TO_LOG
$ECC_LOG_FILE "Editing SYMAPI options file."
if [[
$(EDIT_SYMAPI_OPTIONS_FILE) = $TRUE ]]; then

WRITE_TO_LOG $ECC_LOG_FILE "SYMAPI options file edited successfully."

WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE "symapi_options_file_edited"
else

WRITE_TO_LOG $ECC_LOG_FILE "SYMAPI options file could not be edited."

WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"symapi_options_file_could_not_be_edited"
fi

COUNTER=1
until [[
$(IS_SYMAPI_DAEMON_RUNNING) = $TRUE || $COUNTER > 3 ]] ; do

WRITE_TO_LOG $ECC_LOG_FILE "SYMAPI daemon is not running. Now trying to
start. Attempt#$COUNTER ..."
if [[
$(START_SYMAPI_DAEMON) = $TRUE ]]; then

WRITE_TO_LOG $ECC_LOG_FILE "SYMAPI daemon started after $COUNTER
attempt(s)."
else

WRITE_TO_LOG $ECC_LOG_FILE "SYMAPI daemon could not be started. Waiting
10 seconds."
EMC Proven Professional Knowledge Sharing 22


sleep 10
fi
let
COUNTER=$COUNTER+1
done
if [[
$(IS_SYMAPI_DAEMON_RUNNING) = $FALSE ]]; then

WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"symapi_could_not_be_started_after_editing_symapi_options"
fi
fi
else
WRITE_TO_LOG
$ECC_LOG_FILE "There is no need to edit SYMAPI options file."

WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"no_need_to_edit_symapi_options"
fi
fi
fi
fi
fi
else
WRITE_TO_LOG $ECC_LOG_FILE "Checking if ECC server
info is in /etc/hosts file."
if [[ $(IS_ECC_SERVER_IN_HOSTS_FILE) = $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "ECC server info is
already in /etc/hosts file."
else
WRITE_TO_LOG $ECC_LOG_FILE "Could not find ECC
server info in /etc/hosts file."
WRITE_TO_LOG $ECC_LOG_FILE "Adding ECC server
info to /etc/hosts file."
if [[ $(ADD_ECC_SERVER_TO_HOSTS_FILE) = $TRUE
]]; then
WRITE_TO_LOG $ECC_LOG_FILE "Added ECC
server info to /etc/hosts file."
else
WRITE_TO_LOG $ECC_LOG_FILE "Unable to
add ECC server info to /etc/hosts file."
fi
fi
WRITE_TO_LOG $ECC_LOG_FILE "Checking if ECC master
agent exists or not."
ECC_MASTER_AGENT_INSTALLED=$FALSE
if [[ $(IS_ECC_MASTER_AGENT_INSTALLED) = $TRUE ]];
then
WRITE_TO_LOG $ECC_LOG_FILE "ECC master agent
already exists."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"master_agent_already_exists"
ECC_MASTER_AGENT_INSTALLED=$TRUE
else
WRITE_TO_LOG $ECC_LOG_FILE "ECC master agent
does not exist."
WRITE_TO_LOG $ECC_LOG_FILE "Installing ECC
master agent."
if [[ $(INSTALL_ECC_MASTER_AGENT $OS_TYPE) =
$TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "ECC master
agent installed."
EMC Proven Professional Knowledge Sharing 23

WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "master_agent_installed"
ECC_MASTER_AGENT_INSTALLED=$TRUE
else
WRITE_TO_LOG $ECC_LOG_FILE "ECC master
agent could not be installed."
WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "master_agent_could_not_be_installed"
WRITE_TO_LOG $ECC_LOG_FILE "Copying ECC
master agent installation log to host's log directory."
if [[
$(COPY_ECC_MASTER_AGENT_INSTALLATION_LOG_TO_HOST_FOLDER) = $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "ECC
master agent installation log was copied to host's log directory."
else
WRITE_TO_LOG $ECC_LOG_FILE "ECC
master agent installation log could not be copied to host's log directory."
fi
fi
fi

if [[ $ECC_MASTER_AGENT_INSTALLED = $TRUE ]]; then

WRITE_TO_LOG $ECC_LOG_FILE "Checking if ECC
master agent service is running or not."
if [[ $(IS_ECC_MASTER_AGENT_SERVICE_RUNNING) =
$TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "ECC master
agent service is already running."
WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "master_agent_service_already_running"
else
COUNTER=1
until [[
$(IS_ECC_MASTER_AGENT_SERVICE_RUNNING) = $TRUE || $COUNTER > 3 ]] ; do
WRITE_TO_LOG $ECC_LOG_FILE "ECC
master agent is not running. Now trying to start. Attempt#$COUNTER ..."
if [[
$(START_ECC_MASTER_AGENT_SERVICE) = $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE
"ECC master agent service started after $COUNTER attempt(s)."
WRITE_TO_LOG $ECC_LOG_FILE
"Waiting 30 seconds for host agent to be triggered by master agent."
WRITE_TO_REPORTS_FILE
$HOST_NAME $OS_TYPE "master_agent_service_started"
sleep 30
else
WRITE_TO_LOG $ECC_LOG_FILE
"ECC master agent service could not be started. Waiting 10 seconds."
sleep 10
fi
let COUNTER=$COUNTER+1
done
if [[
$(IS_ECC_MASTER_AGENT_SERVICE_RUNNING) = $FALSE ]]; then
WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "master_agent_service_could_not_be_started"
fi
fi
WRITE_TO_LOG $ECC_LOG_FILE "Checking if ECC
host agent service is running or not."
if [[ $(IS_ECC_HOST_AGENT_SERVICE_RUNNING) =
$FALSE ]]; then
EMC Proven Professional Knowledge Sharing 24

WRITE_TO_LOG $ECC_LOG_FILE "ECC host
agent is not running."
WRITE_TO_REPORTS_FILE $HOST_NAME
$OS_TYPE "host_agent_service_not_running"
else
WRITE_TO_LOG $ECC_LOG_FILE "ECC host
agent is already running."
fi


if [[ $(IS_ECC_MASTER_AGENT_SERVICE_RUNNING) =
$FALSE || $(IS_ECC_HOST_AGENT_SERVICE_RUNNING) = $FALSE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "Copying ECC
installation folder to host's log directory."
if [[
$(COPY_ECC_INSTALLATION_FOLDER_TO_HOST_FOLDER) = $TRUE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "ECC
installation folder was copied to host's log directory."
else
WRITE_TO_LOG $ECC_LOG_FILE "ECC
installation folder could not be copied to host's log directory."
fi
fi
fi
fi

WRITE_TO_LOG $ECC_LOG_FILE "Checking if $HOST_NAME is
connected to any EMC arrays."

if [[ $(IS_HOST_CONNECTED_TO_EMC_ARRAYS $OS_TYPE) = $FALSE
]]; then
WRITE_TO_LOG $ECC_LOG_FILE "$HOST_NAME is not
connected to any EMC storage arrays. Skipping emcgrab script."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"not_connected_to_emc_arrays"
else
WRITE_TO_LOG $ECC_LOG_FILE "$HOST_NAME is connected
to EMC storage array(s)."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"connected_to_emc_arrays"
WRITE_TO_LOG $ECC_LOG_FILE "Executing EMCGRAB
script."
if [[ $(EXECUTE_EMCGRAB $OS_TYPE) = $FALSE ]]; then
WRITE_TO_LOG $ECC_LOG_FILE "EMCGRAB script
failed."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"emcgrab_could_not_be_collected"
else
WRITE_TO_LOG $ECC_LOG_FILE "EMCGRAB script
executed successfully."
WRITE_TO_REPORTS_FILE $HOST_NAME $OS_TYPE
"emcgrab_collected"
fi
fi
fi
fi
fi

unset ECHO

print "### $PROGRAM_NAME $PROGRAM_VERSION finished. ###"
END_TIME=$(perl -e 'print time')
DURATION=`expr $END_TIME - $START_TIME`
EMC Proven Professional Knowledge Sharing 25


print "Duration: $DURATION seconds. Exiting."

cd 1>$NULL 2>$NULL
umount $LOCAL_MOUNT_POINT 1>$NULL 2>$NULL
rmdir $LOCAL_MOUNT_POINT 1>$NULL 2>$NULL


#
# EOF
#
EMC Proven Professional Knowledge Sharing 26

sh.library:

#!/bin/ksh

#
# File name : sh.library
# EMC CHECKER LIBRARY
# Last revised on 2009, 16 Oct
# Mehmet OZDEMIR (ozdemir_mehmet@emc.com)
# EMC
#

#
# Global variables
#
PROGRAM_NAME="EMC_CHECKER"
PROGRAM_VERSION="v1.0"
HOST_NAME=$(hostname)
YEAR=$(date +%Y)
MONTH=$(date +%m)
DAY=$(date +%d)
LOGS_ROOT_DIRECTORY=$LOCAL_MOUNT_POINT/logs
HOST_LOGS_DIRECTORY=$LOGS_ROOT_DIRECTORY/$YEAR$MONTH$DAY/$HOST_NAME
ECC_LOG_FILE=$HOST_LOGS_DIRECTORY/emc_checker.sh.log
INQ_UTILITY_LOCATION=$LOCAL_MOUNT_POINT/software/inq
SOLUTIONS_ENABLER_LOCATION=$LOCAL_MOUNT_POINT/software/se
EMCGRAB_LOCATION=$LOCAL_MOUNT_POINT/software/emcgrab
EMCGRAB_OUTPUT_DAILY_LOCATION=$LOGS_ROOT_DIRECTORY/$YEAR$MONTH$DAY/_emcgrab_out
puts_
ECC_MASTER_AGENT_LOCATION=$LOCAL_MOUNT_POINT/software/ecc
ECC_MASTER_AGENT_SILENT_INSTALL_FILE=$ECC_MASTER_AGENT_LOCATION/silent_install.
txt
REPORTS_FILE_DAILY_LOCATION=$LOGS_ROOT_DIRECTORY/$YEAR$MONTH$DAY/_daily_reports
_
ECC_SERVER_NAME=myECCServer
ECC_SERVER_IP=xxx.xxx.xxx.xxx
NULL=/dev/null
TRUE=0
FALSE=1

#
# Functions
#

#
# Checks uname -a output and returns OS type.
# Params: None.
#
DETERMINE_OS_TYPE(){
typeset -l UNAME_A=$(uname -a)
case $UNAME_A in
*sunos*)
if [[ $UNAME_A = *sparc* ]]; then
print "solaris.sparc"
elif [[ $UNAME_A = *i386* ]]
then
print "solaris.i386"
else
print "unsupported"
fi
;;
*linux*)
#if [[ -a "/etc/redhat-release" ]]; then
EMC Proven Professional Knowledge Sharing 27

print "linux"
#else
# print "unsupported"
#fi
;;
*aix*)
print "aix"
;;
*hp-ux*)
print "hpux"
;;
*osf1*)
print "tru64"
;;
*)
print "unsupported"
;;
esac
}

#
# Runs the appropriate inq script. if host is connected to storage arrays TRUE
is returned, if not FALSE is returned.
# Params: OS_TYPE.
#
IS_HOST_CONNECTED_TO_STORAGE_ARRAYS(){
OS_TYPE=$1
INQ_OUTPUT=$($INQ_UTILITY_LOCATION/inq.$OS_TYPE -no_dots -skipread)
if [[ $INQ_OUTPUT = *:SYMMETRIX* || $INQ_OUTPUT = *:DGC* || $INQ_OUTPUT
= *:DF600* || $INQ_OUTPUT = *:DF700* || $INQ_OUTPUT = *:OPEN-* ]]; then
print $TRUE
else
print $FALSE
fi
}

#
# Appends text to log file. Current date is also added.
# Params: Log file name, text.
#
WRITE_TO_LOG(){
LOG_FILE=$1
TEXT=$2
print "$(date) \t $TEXT" >> $LOG_FILE
if [[ $ECHO = on ]]; then
print "$TEXT"
fi
}

#
# Creates log folder and file for host.
# Params: None.
#
CREATE_HOST_LOGS_FOLDER(){
if [ ! -d $HOST_LOGS_DIRECTORY ]; then
mkdir -p $HOST_LOGS_DIRECTORY 1>$NULL 2>$NULL
touch $ECC_LOG_FILE 1>$NULL 2>$NULL
WRITE_TO_LOG $ECC_LOG_FILE "Created $ECC_LOG_FILE."
else
WRITE_TO_LOG $ECC_LOG_FILE "Found existing $ECC_LOG_FILE file."
fi
}

EMC Proven Professional Knowledge Sharing 28

#
# Returns Solutions Enabler version if already installed. If not returns FALSE.
# Params: None.
#
GET_SOLUTIONS_ENABLER_VERSION(){
if [[ -d "/usr/symcli/bin" ]]; then
SYMCLI_VERSION=$(/usr/symcli/bin/symcli | awk '{print $7}' | grep
V)
print $SYMCLI_VERSION
else
print $FALSE
fi
}

#
# Installs Solutions Enabler. If installation is successful returns TRUE, if
not returns FALSE.
# Params: OS type.
#
INSTALL_SOLUTIONS_ENABLER(){
OS_TYPE=$1
TMP=$($SOLUTIONS_ENABLER_LOCATION/$OS_TYPE/se*_install.sh -install -
silent)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# Checks if SYMCLI base and server licenses are installed or not. If installed
returns TRUE, if not returns FALSE.
# Params: None.
#
IS_SYMAPI_DAEMON_LICENSE_INSTALLED(){
if [[ ! -a "/var/symapi/config/symapi_licenses.dat" ]]; then
print $FALSE
else
BASE_LICENSE=$(cat /var/symapi/config/symapi_licenses.dat | grep
7FBE-9150-E0BC-9CCD)
SERVER_LICENSE=$(cat /var/symapi/config/symapi_licenses.dat | grep
3A61-6044-C190-0196)
if [[ $BASE_LICENSE = "License Key: 7FBE-9150-E0BC-9CCD SYMAPI
Feature: BASE / Symmetrix" && $SERVER_LICENSE = "License Key: 3A61-6044-C190-
0196 SYMAPI Feature: SERVER / Symmetrix" ]]; then
print $TRUE
else
print $FALSE
fi
fi
}

#
# Installs Solutions Enabler base and server license. Returns TRUE if it is
successful, if not returns FALSE.
# Params: None.
#
INSTALL_SYMAPI_DAEMON_LICENSE(){
BASE_LICENSE=$(/usr/symcli/bin/symlmf 7FBE-9150-E0BC-9CCD)
BASE_LICENSE_INSTALLED=$?
SERVER_LICENSE=$(/usr/symcli/bin/symlmf 3A61-6044-C190-0196)
SERVER_LICENSE_INSTALLED=$?
EMC Proven Professional Knowledge Sharing 29

if [[ $BASE_LICENSE_INSTALLED = 0 && $SERVER_LICENSE_INSTALLED = 0 ]];
then
print $TRUE
else
print $FALSE
fi
}

#
# Checks if SYMAPI daemon is running or not. If it is running returns TRUE, if
not returns FALSE.
# Params: None.
#
IS_SYMAPI_DAEMON_RUNNING(){
TMP=$(/usr/symcli/bin/stordaemon list | grep storsrvd | awk '{print $1}')
if [[ $TMP = "[*]" ]]; then
print $TRUE
else
print $FALSE
fi
}

#
# Starts SYMAPI daemon. Returns TRUE if it is started, if not returns FALSE.
# Params: None.
#
START_SYMAPI_DAEMON(){
TMP=$(/usr/symcli/bin/stordaemon start storsrvd)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# Stops SYMAPI daemon. Returns TRUE if it is stopped, if not returns FALSE.
# Params: None.
#
STOP_SYMAPI_DAEMON(){
TMP=$(/usr/symcli/bin/stordaemon shutdown storsrvd)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# Checks if host is a TRU64 cluster node. Returns TRUE if is TRU64 cluster
node, if not returns FALSE.
# Params: None.
#
IS_TRU64_CLUSTER_NODE(){
TMP=$(/usr/sbin/clu_get_info)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TMP
fi
}

#
EMC Proven Professional Knowledge Sharing 30

# Checks if ECC master agent is already installed or not. Returns TRUE if
already installed, if not returns FALSE.
# Params: None.
#
IS_ECC_MASTER_AGENT_INSTALLED(){
if [[ -d "/usr/ecc/exec" ]]; then
print $TRUE
else
print $FALSE
fi
}

#
# Checks if ECC server info is in /etc/hosts file. Returns TRUE if it is
already there, if not returns FALSE.
# Params: None.
#
IS_ECC_SERVER_IN_HOSTS_FILE(){
CMD="grep $ECC_SERVER_NAME /etc/hosts | grep $ECC_SERVER_IP"
OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# /var/symapi/config/options file must be edited for TRU64 and Solaris X86
hosts to be discovered by ECC.
# There must not be a line starting with #SYMAPI_SERVER_SECURITY_LEVEL
# Params: None.
#
SYMAPI_OPTIONS_FILE_SHOULD_BE_EDITED(){
CMD="grep \"#SYMAPI_SERVER_SECURITY_LEVEL\" /var/symapi/config/options"
OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# /var/symapi/config/options file must be edited for TRU64 and Solaris X86
hosts to be discovered by ECC.
# Line starting with "#SYMAPI_SERVER_SECURITY_LEVEL" is replaced with
"SYMAPI_SERVER_SECURITY_LEVEL = NONSECURE"
# Params: None.
#
EDIT_SYMAPI_OPTIONS_FILE(){
SOURCE_FILE=/var/symapi/config/options
TEMP_FILE=/var/symapi/config/options.emc
ORIG_STRING="\#SYMAPI_SERVER_SECURITY_LEVEL"
NEW_STRING="SYMAPI_SERVER_SECURITY_LEVEL = NONSECURE"
CMD="sed 's/$ORIG_STRING.*/$NEW_STRING/g' $SOURCE_FILE > $TEMP_FILE && mv
-f $TEMP_FILE $SOURCE_FILE"
OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
EMC Proven Professional Knowledge Sharing 31

}

#
# Adds ECC server info to /etc/hosts file. Returns TRUE if it adds, if not
returns FALSE.
# Params: None.
#
ADD_ECC_SERVER_TO_HOSTS_FILE(){
CMD="print \"$ECC_SERVER_IP\t$ECC_SERVER_NAME\" >> /etc/hosts"
OUTPUT=$($CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# Installs ECC master agent software. If it installs returns TRUE,
# Params: OS type.
#
INSTALL_ECC_MASTER_AGENT(){
OS_TYPE=$1
CMD="cd $ECC_MASTER_AGENT_LOCATION/$OS_TYPE && ./install_native.sh
$ECC_MASTER_AGENT_LOCATION/$OS_TYPE -s $ECC_MASTER_AGENT_SILENT_INSTALL_FILE"
OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# If either of ECC master and host agent service cannot be started then ECC
installation
# directory is copied to host's log folder.
# Params: None.
#
COPY_ECC_INSTALLATION_FOLDER_TO_HOST_FOLDER(){
CMD="tar cvf $HOST_LOGS_DIRECTORY/USR_ECC_EXEC.tar /usr/ecc/exec"
OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# Checks if ECC master agent is running or not. Returns TRUE if it is running,
if not returns FALSE.
# Params: None.
#
IS_ECC_MASTER_AGENT_SERVICE_RUNNING(){
TMP=$(ps -ef | grep /usr/ecc/exec/mstragent | grep -v grep)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

EMC Proven Professional Knowledge Sharing 32

#
# If an error occurs during installation of ECC master agent then
# the installation log created by master agent installer is copied to host's
log folder.
# Param: None.
#
COPY_ECC_MASTER_AGENT_INSTALLATION_LOG_TO_HOST_FOLDER(){
if [[ ! -a "/usr/ecc/install_native.log" ]]; then
print $FALSE
else
CMD="cp /usr/ecc/install_native.log $HOST_LOGS_DIRECTORY"
OUTPUT=$($CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
fi
}

#
# Checks if ECC host agent is running or not. Returns TRUE if it is running, if
not returns FALSE.
# Params: None.
#
IS_ECC_HOST_AGENT_SERVICE_RUNNING(){
# converts the argument to lower case and assigns to UNAME_A variable.
typeset -l UNAME_A=$(uname -a)
case $UNAME_A in
*sunos*)
AGENT="MSR"
;;
*linux*)
AGENT="MLR"
;;
*aix*)
AGENT="MAR"
;;
*hp-ux*)
AGENT="MHR"
;;
*)
AGENT="unsupported"
;;
esac
CMD="ps -ef | grep /usr/ecc/exec/$AGENT | grep -v grep"
PS_EF_OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $PS_EF_OUTPUT
fi
}

#
# Starts ECC master agent service. Returns TRUE if it is started, if not
returns FALSE.
# Params: None.
#
START_ECC_MASTER_AGENT_SERVICE(){
if [[ ! -a "/usr/ecc/exec/eccmad" ]]; then
print $FALSE
else
EMC Proven Professional Knowledge Sharing 33

TMP=$(/usr/ecc/exec/eccmad start)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
fi
}

#
# Stops ECC master agent service. Returns TRUE if it is stopped, if not returns
FALSE.
# Params: None.
#
STOP_ECC_MASTER_AGENT_SERVICE(){
if [[ ! -a "/usr/ecc/exec/eccmad" ]]; then
print $FALSE
else
TMP=$(/usr/ecc/exec/eccmad stopall)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
fi
}

#
# Runs EMCGRAB script. Returns TRUE if it ends successfully, if not returns
FALSE.
# Params: OS type.
#
EXECUTE_EMCGRAB(){
OS_TYPE=$1
if [ ! -d $EMCGRAB_OUTPUT_DAILY_LOCATION ]; then
mkdir -p $EMCGRAB_OUTPUT_DAILY_LOCATION
fi
CMD="cd $EMCGRAB_LOCATION/$OS_TYPE && /bin/ksh ./emcgrab.sh -OUTDir
$EMCGRAB_OUTPUT_DAILY_LOCATION -autoexec -legal -lite 1>$NULL 2>$NULL"
OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print $FALSE
else
print $TRUE
fi
}

#
# Runs the appropriate inq script. if host is connected to EMC arrays TRUE is
returned, if not FALSE is returned.
# Params: OS_TYPE.
#
IS_HOST_CONNECTED_TO_EMC_ARRAYS(){
OS_TYPE=$1
INQ_OUTPUT=$($INQ_UTILITY_LOCATION/inq.$OS_TYPE -no_dots -skipread)
if [[ $INQ_OUTPUT = *:EMC* || $INQ_OUTPUT = *:SYMMETRIX* || $INQ_OUTPUT
= *:DGC* ]]; then
print $TRUE
else
print $FALSE
fi
}

EMC Proven Professional Knowledge Sharing 34

#
# Appends text to reports file. This file is used for generating report e-mail.
# Params: Host name, OS type, report file name.
#
WRITE_TO_REPORTS_FILE(){
HOSTNAME=$1
OS_TYPE=$2
FILE_NAME=$3
if [ ! -d $REPORTS_FILE_DAILY_LOCATION ]; then
mkdir $REPORTS_FILE_DAILY_LOCATION
fi
if [ -a $REPORTS_FILE_DAILY_LOCATION/$FILE_NAME ]; then
CMD="cat $REPORTS_FILE_DAILY_LOCATION/$FILE_NAME | grep
$HOSTNAME\t$OS_TYPE"
OUTPUT=$(eval $CMD)
if [[ $? != 0 ]]; then
print "$HOSTNAME\t$OS_TYPE" >>
$REPORTS_FILE_DAILY_LOCATION/$FILE_NAME
fi
else
print "$HOSTNAME\t$OS_TYPE" >>
$REPORTS_FILE_DAILY_LOCATION/$FILE_NAME
fi
}

#
# EOF
#

EMC Proven Professional Knowledge Sharing 35

emc_checker_reporter_module.sh:

#!/bin/ksh

#
# File name : emc_checker_reporter_module.sh
# EMC CHECKER v1.0
# Last revised on 2009, 16 Oct
# Mehmet OZDEMIR (ozdemir_mehmet@emc.com)
# EMC
#

#
# Global variables
#
YEAR=$(date +%Y)
MONTH=$(date +%m)
DAY=$(date +%d)
REPORTS_FILE_LOCATION="../logs/$YEAR$MONTH$DAY/_daily_reports_"
FROM="EMC_CHECKER@turkcell.com.tr"
TO="mehmet.ozdemir@consultant.turkcell.com.tr"
SUBJECT="EMC Checker Daily Report ($DAY.$MONTH.$YEAR)"
#
# Do not change below
#


LINES(){
print "$(awk ' END { print NR } ' $1)"
}


if [ ! -d $REPORTS_FILE_LOCATION ]; then
print "Unable to find reports folder."
return 1
fi

BODY="Sent by EMC_CHECKER tool developed by EMC (ozdemir_mehmet@emc.com).\r\r"
SUMMARY="########### SUMMARY ###########\r\n"
DETAILS="########### DETAILS ###########\r\n"


for FILE in $(ls -l $REPORTS_FILE_LOCATION | grep -v total | awk '{print
$9}'); do
FILE_CONTENT=""
while read LINE
do
FILE_CONTENT="$FILE_CONTENT\t$LINE\n"
done < $REPORTS_FILE_LOCATION/$FILE
NUMBER_OF_LINES=$(wc -l $REPORTS_FILE_LOCATION/$FILE|awk '{print $1}')
DASHES="--------------------------------------------"
DETAILS="${DETAILS}${FILE}
(${NUMBER_OF_LINES})\r${DASHES}\r${FILE_CONTENT}\r\r\r"
SUMMARY="$SUMMARY$FILE --> ${NUMBER_OF_LINES}\r\r"
done


BODY="$BODY\r\r$SUMMARY\r\r$DETAILS"

#print "E-mail sent to $TO" && (print $BODY | mailx -s "$SUBJECT" -r "$FROM"
"$TO")
print $BODY | mailx -s "$SUBJECT" -r "$FROM" "$TO"
EMC Proven Professional Knowledge Sharing 36

Appendix B
Sample script outputs (emc_checker.sh)

root@my_solaris_host # ./emc_checker.sh
Created /EMC_ECCINSTALL/logs/20101231/my_solaris_host/emc_checker.sh.log.
Local mount point : /EMC_ECCINSTALL
my_nfs_server:/nfs_share mounted.
Checking if my_solaris_host is supported or not.
my_solaris_host is solaris.sparc host which is supported.
Checking if my_solaris_host is connected to any storage arrays.
my_solaris_host is connected to storage array(s).
Checking if ECC server info is in /etc/hosts file.
ECC server info is already in /etc/hosts file.
Checking if ECC master agent exists or not.
ECC master agent does not exist.
Installing ECC master agent.
ECC master agent installed.
Checking if ECC master agent service is running or not.
ECC master agent is not running. Now trying to start. Attempt#1 ...
ECC master agent service started after 1 attempt(s).
Waiting 30 seconds for host agent to be triggered by master agent.
Checking if ECC host agent service is running or not.
ECC host agent is running.
Checking if my_solaris_host is connected to any EMC arrays.
my_solaris_host is connected to EMC storage array(s).
Executing EMCGRAB script.
EMCGRAB script executed successfully.
root@my_solaris_host #


root@my_aix_host # ./emc_checker.sh
Created /EMC_ECCINSTALL/logs/20101231/my_aix_host/emc_checker.sh.log.
Local mount point : /EMC_ECCINSTALL
my_nfs_server:/nfs_share mounted.
Checking if my_aix_host is supported or not.
my_aix_host is solaris.sparc host which is supported.
Checking if my_aix_host is connected to any storage arrays.
my_aix_host is connected to storage array(s).
Checking if ECC server info is in /etc/hosts file.
Could not find ECC server info in /etc/hosts file.
Adding ECC server info to /etc/hosts file.
Added ECC server info to /etc/hosts file.
Checking if ECC master agent exists or not.
ECC master agent does not exist.
Installing ECC master agent.
ECC master agent could not be installed.
Copying ECC master agent installation log to host's log directory.
ECC master agent installation log was copied to host's log directory.
Checking if my_aix_host is connected to any EMC arrays.
my_aix_host is connected to EMC storage array(s).
Executing EMCGRAB script.
EMCGRAB script executed successfully.
root@my_aix_host #



root@my_tru64_host # ./emc_checker.sh
Created /EMC_ECCINSTALL/logs/20101231/my_tru64_host/emc_checker.sh.log.
Local mount point : /EMC_ECCINSTALL
my_nfs_server:/nfs_share mounted.
Checking if my_tru64_host is supported or not.
EMC Proven Professional Knowledge Sharing 37

my_tru64_host is tru64 host which is supported.
Checking if my_tru64_host is connected to any storage arrays.
my_tru64_host is connected to storage array(s).
Checking if Solutions Enabler is already installed or not.
Could not find any installed Solutions Enabler.
Installing Solutions Enabler.
Solutions Enabler installed.
Checking if SYMAPI daemon license is installed or not.
SYMAPI daemon license could not be found. Now installing the license.
SYMAPI daemon license installed.
Checking if SYMAPI daemon is running or not.
SYMAPI daemon is not running. Now trying to start. Attempt#1 ...
SYMAPI daemon started after 1 attempt(s).
Checking if SYMAPI options file should be edited.
SYMAPI options file should be edited.
SYMAPI daemon is running. Now trying to stop. Attempt#1 ...
SYMAPI daemon stopped after 1 attempt(s).
Editing SYMAPI options file.
SYMAPI options file edited successfully.
SYMAPI daemon is not running. Now trying to start. Attempt#1 ...
SYMAPI daemon started after 1 attempt(s).
Checking if my_tru64_host is connected to any EMC arrays.
my_tru64_host is not connected to any EMC storage arrays. Skipping emcgrab
script.
root@my_tru64_host #
EMC Proven Professional Knowledge Sharing 38

Appendix C
Sample E-mail Report

-----Original Message-----
From: EMC_CHECKER@turkcell.com.tr
Sent: Friday, December 11, 2010 11:59 PM
To: mehmet.ozdemir@consultant.turkcell.com.tr
Subject: EMC Checker Daily Report

Sent by EMC_CHECKER tool developed by EMC (ozdemir_mehmet@emc.com).

########### SUMMARY ###########
connected_to_emc_arrays --> 106

connected_to_storage_arrays --> 217

emcgrab_collected --> 138

host_agent_service_not_running --> 38

master_agent_already_exists --> 195

master_agent_could_not_be_installed --> 13

master_agent_installed --> 1

master_agent_service_already_running --> 213

master_agent_service_could_not_be_started --> 1

master_agent_service_started --> 3

no_need_to_edit_symapi_options --> 75

not_connected_to_emc_arrays --> 254

not_connected_to_storage_arrays --> 357

solutions_enabler_could_not_be_installed --> 1

solutions_enabler_exists --> 52

solutions_enabler_installed --> 5

supported --> 409

symapi_already_running --> 63

symapi_options_file_edited --> 5

symapi_started --> 5


########### DETAILS ###########
connected_to_emc_arrays (106)
--------------------------------------------
host_1 solaris.sparc
host_2 solaris.sparc
host_3 solaris.sparc
host_4 solaris.sparc