Centera Monitoring and Reporting in CentraStar 3.1-1

Engineering White Paper
Centera Monitoring and Reporting

ControlCenter 5.2 SP3
CentraStar 3.1
Abstract
This white paper describes the different tools and channels via which system administrators can monitor their
Centera storage systems and report on configuration, capacity and performance. Topics discussed are sensors
and alerts, monitoring channels, message formats, management user interfaces, report types, report generation
and monitoring related security. This paper complements the information available in the Centera Online
Help, P/N 300-002-547, and the Centera Quick Start Guide, P/N 300-002-546.
Published 2006
Copyright © 2006 EMC Corporation. All rights reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is
subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES
NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION
IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable
software license.
1
Table of Contents
Introduction.....................................................................................................................................8
Centera Sensor Framework...........................................................................................................9
Introduction ................................................................................................................................................. 9
Monitoring Sensors ................................................................................................................................... 10
CentraStar............................................................................................................................................. 10
SDK....................................................................................................................................................... 10
Monitoring and Alerting ...............................................................................................................11
Overview of Monitoring Channels ............................................................................................................. 11
Alert Event Follow-up ................................................................................................................................ 12
Scheduled Reporting....................................................................................................................18
Health Reporting ....................................................................................................................................... 18
Introduction ........................................................................................................................................... 18
Overview of Health Reporting Channels ............................................................................................... 18
Scheduled Reporting Using CLI Scripting ................................................................................................. 19
General ................................................................................................................................................. 19
Creating a script file .............................................................................................................................. 19
Creating a batch file .............................................................................................................................. 19
Windows ............................................................................................................................................... 20
Solaris ................................................................................................................................................... 21
Real-time Reporting......................................................................................................................22
Capacity Reporting ................................................................................................................................... 22
General ................................................................................................................................................. 22
Terms & Definitions............................................................................................................................... 23
Capacity Components Overview........................................................................................................... 24
Overview of Capacity Reporting Tool Coverage ................................................................................... 25
Raw Capacity Reporting ....................................................................................................................... 25
Available Capacity Reporting ................................................................................................................ 26
System Buffer 26
Regeneration buffer 27
Regeneration Buffer Alerting 29
Regeneration Buffer Reporting 30
Object Count Reporting......................................................................................................................... 30
Pool Capacity Reporting ....................................................................................................................... 31
Quota Management 31
Pool Capacity Calculation 31
Pool Capacity Reporting Limitations 33
Useable Capacity Reporting ................................................................................................................. 34
Relevant Reports Needed To Calculate Useable Capacity 34
Useable Capacity for Capacity Bound Systems 35
Useable Capacity for Object Bound Systems 35
2
Best Practices for Object Bound Systems 35
Meta data Driven Capacity Reporting ................................................................................................... 36
Performance Reporting ............................................................................................................................. 37
Performance Sensors ........................................................................................................................... 37
Overview of Performance Reporting Tool Coverage............................................................................. 37
Replication Reporting............................................................................................................................ 38
How Replication Works 38
Replication Statistics 38
Restore Reporting ................................................................................................................................. 39
How Restore Is Different From Replication 39
Restore Statistics 39
How is the Restore Queue Calculated 40
Garbage Collection Reporting............................................................................................................... 41
How Garbage Collection Works 41
Garbage Collection Statistics 41
Cluster to Cluster Synchronization Reporting ........................................................................................... 42
Monitoring and Health Message Channels, Formats and Mappings ......................................43
ConnectEMC............................................................................................................................................. 43
ConnectEMC Overview......................................................................................................................... 43
ConnectEMC Formats........................................................................................................................... 44
ConnectEMC Alert Mapping.................................................................................................................. 45
ConnectEMC Notification .......................................................................................................................... 46
ConnectEMC Notification Formats........................................................................................................ 46
SNMP........................................................................................................................................................ 46
SNMP Overview.................................................................................................................................... 46
Centera SNMP Functionality................................................................................................................. 47
SNMP Heartbeat trap 47
SNMP Alarm trap 47
SNMP Mapping..................................................................................................................................... 48
SNMP Limitations ................................................................................................................................. 48
Monitoring API (MoPI)............................................................................................................................... 49
MoPI Overview...................................................................................................................................... 49
MoPI Formats ....................................................................................................................................... 49
MoPI Mapping....................................................................................................................................... 50
EMC ControlCenter................................................................................................................................... 51
ECC Overview ...................................................................................................................................... 51
ECC Storage Agent For Centera Installation & Configuration............................................................... 51
Discovery data collection policy 51
Configuring the Storage Agent for Centera 51
ECC Mapping........................................................................................................................................ 52
Monitoring 52
Health Reporting 53
Monitoring and Reporting Tools .................................................................................................53
3
Centera Command Line Interface (CLI) .................................................................................................... 53
EMC ControlCenter 5.2............................................................................................................................. 54
ECC Console ........................................................................................................................................ 54
Pool Capacity 54
Profiles & Capabilities 54
ECC StorageScope............................................................................................................................... 56
Include Centera Clusters and Pools in StorageScope groups 56
Run standard point-in-time reports 58
Define and run customizable historic trending reports 60
Appendix A – ConnectEMC Formats. ............................................................................................i
ConnectEMC alert_email-1.0.dtd i
ConnectEMC alert_email-1.1.dtd i
ConnectEMC health_email-1.0.dtd ii
ConnectEMC health_email-1.1.dtd vii
ConnectEMC health_email-1.2.dtd x
ConnectEMC health_email-1.3.dtd xi
ConnectEMC health_email-1.4.dtd xi
Appendix B – ConnectEMC Alert Notification ...........................................................................xv
Appendix C – ConnectEMC Sample Health Notification .........................................................xvi
Appendix D – SNMP Formats ...................................................................................................xxiv
Alarm trap v2.0 xxiv
Appendix E – MoPI Formats ......................................................................................................xxv
MoPI alert-1.0.dtd xxv
MoPI discovery-1.1.dtd xxvi
MoPI discovery-1.2.dtd xxix
MoPI health-x.y.dtd xxx
4
List of Figures
Figure 1 – Sensor Framework ...........................................................................................................................9
Figure 2 – script.cli.............................................................................................................................................19
Figure 3 – CLI Script Mode Syntax .................................................................................................................19
Figure 4 – sample.bat .......................................................................................................................................20
Figure 5 – Windows Scheduled Tasks ...........................................................................................................20
Figure 6 – Windows Scheduled Task Details ................................................................................................20
Figure 7 – Windows Scheduled Task List ......................................................................................................21
Figure 8 – sample.sh.........................................................................................................................................21
Figure 9 – Crontab Entry ..................................................................................................................................21
Figure 10 – Capacity Components On Empty System.................................................................................24
Figure 11 – Capacity Components On Used System...................................................................................24
Figure 12 – Show Capacity Total ....................................................................................................................26
Figure 13 – Show Capacity Availability ..........................................................................................................30
Figure 14 – Pool Capacity Algorithm ..............................................................................................................32
Figure 15 – Pool Capacity Re-calculation Algorithm ....................................................................................33
Figure 16 – Show Capacity Total ....................................................................................................................34
Figure 17 – Show Pool Capacity .....................................................................................................................34
Figure 18 – Show Capacity Availability ..........................................................................................................35
Figure 19 – Replication .....................................................................................................................................38
Figure 20 – Restore Queue Calculation .........................................................................................................40
Figure 21 – OnAlert Configuration...................................................................................................................43
Figure 22 – SNMP Heartbeat Trap .................................................................................................................47
Figure 23 – SNMP Alarm Trap ........................................................................................................................47
Figure 24 – ECC 5.2 SP3 Cluster, Pool and Profiles ...................................................................................55
Figure 25 – Create a New Group in ECC Console.......................................................................................56
Figure 26 – ECC All History .............................................................................................................................56
Figure 27 – ECC History...................................................................................................................................57
Figure 28 – ECC Components.........................................................................................................................57
Figure 29 – CAS Report Overview..................................................................................................................58
Figure 30 – CAS Clusters by Group ...............................................................................................................58
Figure 31 – All CAS Clusters – Utilization......................................................................................................58
Figure 32 – All CAS Clusters – Configuration ...............................................................................................58
Figure 33 – CAS Cluster Group List ...............................................................................................................58
Figure 34 – CAS Storage Nodes .....................................................................................................................58
Figure 35 – CAS Pools by Group ....................................................................................................................59
5
Figure 36 – All CAS Pools ................................................................................................................................59
Figure 37 – CAS Pools Group List ..................................................................................................................59
Figure 38 – CAS Cluster Configuration – General........................................................................................59
Figure 39 – CAS Cluster Configuration - Groups..........................................................................................59
Figure 40 – CAS Cluster Configuration – Pools............................................................................................59
Figure 41 – Sample Historic Trend Report ....................................................................................................60
Figure 42 – STS Online Help – How to Create Custom Reports................................................................60
Figure 43 – Create a New Custom Report.....................................................................................................60
Figure 44 – Create a New Custom Report – Select Fields to Report ........................................................60
Figure 45 – Create a New Custom Report – Define Filters.........................................................................60
Figure 46 – Create a New Custom Report – Determine Period and Trend ..............................................60
Figure 47 – ConnectEMC alert_email-1.0.dtd ..................................................................................................i
Figure 48 – ConnectEMC alert_email-1.1.dtd ................................................................................................ ii
Figure 49 – ConnectEMC health_email-1.0.dtd ........................................................................................... vii
Figure 50 – ConnectEMC health_email-1.1.dtd .............................................................................................ix
Figure 51 – ConnectEMC health_email-1.2.dtd .............................................................................................xi
Figure 52 – ConnectEMC health_email-1.3.dtd .............................................................................................xi
Figure 53 – ConnectEMC health_email-1.4.dtd ...........................................................................................xiv
Figure 54 – ConnectEMC Alert Notification ...................................................................................................xv
Figure 55 – Alarm trap v2.0...........................................................................................................................xxiv
Figure 56 – MoPI alert-1.0.dtd .......................................................................................................................xxv
Figure 59 – MoPI discovery-1.1.dtd .......................................................................................................... xxviii
Figure 60 – MoPI discovery-1.2.dtd ..............................................................................................................xxx
6
List of Tables
Table 1 – Sensor List ........................................................................................................................................10
Table 2 – Monitoring Channels and Frequency ............................................................................................11
Table 3 – Centera Alert List..............................................................................................................................17
Table 4 – Health Report Channels and Frequency ......................................................................................18
Table 5 – Capacity Notations...........................................................................................................................22
Table 6 – Capacity Transformation .................................................................................................................22
Table 7 – Capacity Terms & Definitions .........................................................................................................23
Table 8 – Capacity Reporting Channels and Frequency .............................................................................25
Table 9 – Disk Level Raw Capacity ................................................................................................................25
Table 10 – Node Level Raw Capacity ............................................................................................................25
Table 11 – Node Level Available Capacity – CentraStar 3.1 – 30M Objects ...........................................27
Table 12 – Node Level Available Capacity – CentraStar 3.1 – 50M Objects ...........................................27
Table 13 – Recommended Regeneration Buffer Policy...............................................................................28
Table 14 – Node Level Available Capacity – CentraStar 3.1 – 50M Objects – 1 Disk Policy ................28
Table 15 – Node Level Available Capacity – CentraStar 3.1 – 50M Objects – 4 Disk Policy ................29
Table 16 – Object Count per C-Clip................................................................................................................30
Table 17 – Object Count Limit per Node (M).................................................................................................31
Table 18 – Pool Capacity Algorithm................................................................................................................32
Table 19 – Performance Sensors....................................................................................................................37
Table 20 – Performance Reporting Channels ...............................................................................................37
Table 21 – ConnectEMC Formats...................................................................................................................44
Table 22 – ConnectEMC Alert Mapping.........................................................................................................45
Table 23 – ConnectEMC Notification Formats ..............................................................................................46
Table 24 – SNMP Mapping ..............................................................................................................................48
Table 25 – MoPI Formats .................................................................................................................................49
Table 26 – MoPI Mapping ................................................................................................................................50
Table 27 – Supported MoPI Alert Formats ....................................................................................................52
Table 28 – ECC Mapping with MoPI 1.0 ........................................................................................................52
Table 29 – ECC Mapping with MoPI 2.0 ........................................................................................................52
Table 30 – Supported MoPI Health Formats .................................................................................................53
Table 31 – ECC Functionality Overview.........................................................................................................54
Table 32 – ECC Pool Capacity Reporting......................................................................................................54
Table 33 – ECC Profile Capabilities................................................................................................................54
7
Introduction
A key requirement for any enterprise storage platform is the ability for system administrators and systems
management software to monitor the health of the device and to produce and analyze health, capacity and
performance reports on a regular basis.
This document discusses how Centera implements this functionality in the following topics:
• Centera sensor framework: discusses the framework used within Centera to continuously monitor the
health of a Centera and its hard- and software components.
• Definition of monitoring events: in this section we provide an overview of the types of events and the
different threshold values at which they are triggered.
• Event follow-up: here we provide background information on how EMC Support and/or the system
administrator must respond to an event.
• Health reporting: complementing the monitoring information, EMC Support and/or the system
administrator can receive a daily or on-request health report, providing an overview of the main system
configuration settings as well as capacity and performance information.
• Audit Logging: with the introduction of management profiles in CentraStar 3.1, a Centera Cluster now
also provides information about configuration changes, service interventions and any authentication which
occurs on the cluster.
• Monitoring and health reporting channels supported: Centera supports a number of channels for
monitoring, including industry standards such as SNMP, and health reporting. This section provides a
detailed description of each supported channel.
• Message formats and mapping: this section provides a detailed overview of the different formats used
for monitoring and health reporting, as well as a mapping of Centera alerts to the different formats.
• Monitoring and reporting tools: provides an overview of the different options system administrators
have for reporting Centera configuration, capacity and performance.
8
Centera Sensor Framework
Introduction
Centera is a distributed storage system running the CentraStar operating system on a cluster of 4 to 128 nodes.
The monitoring subsystem is based on a distributed system of sensors or agents which monitor a specific hard-
or software component on an individual node.
Each sensor has two parts:
• An Inspector periodically fetches “values” from the component being monitored.
• The Evaluator1 then aggregates values according to the sensor configuration and determines whether an
alert needs to be generated:
When the threshold is exceeded, a ‘degradation’ alert is sent;
When the value falls back in the ‘normal’ range, an ‘improvement’ alert2 is sent.
Federation
Site
Cluster
Rack/BSU
Root Switches
Node/
Blade
Cube Switches
SW Component
Disk, NIC, CPU, ...
SW Module
Figure 1 – Sensor Framework
To avoid alert storms inherent in a distributed system, the sensor framework has two levels of sensors. The
node-level sensors only fire internal alert events, which are then trapped and taken as input to the higher level
cluster-level sensors.
Cluster-level sensors apply a set of filters and aggregation rules to evaluate alert events from the individual
nodes and decide whether to issue a cluster alert to the outside world. Any alert event triggered by a cluster-
level sensor is presented to the messaging subsystem which is responsible for pushing the alert events to the
different alerting channels supported.
1
Not all sensors have an Evaluator and hence do not trigger alert events – these sensors are used solely for reporting purposes and are not
within the scope of this paper.
2
Not all channels process improvement alerts; only the MoPI interface, and EMC ControlCenter process and report improvements.
9
Monitoring Sensors
CentraStar
CentraStar 3.1 has over 100 sensors actively monitoring the health, capacity and performance of a Centera
cluster. Of those sensors, 23 have one ore more evaluators defined which can trigger 55 distinct alert events.
The table below has the list of the alerting sensors in CentraStar v3.1.
Each sensor has the following attributes:
• The Sensor Name indicates which component of the Centera system is being monitored;
• The Frequency indicates the frequency with which the sensor’s Inspector checks the Centera component,
in minutes (m) or hours (h); and
• The Type of sensor: this classification exists for historical reasons – Type B and C sensors are used to
monitor a set of hardware related events which have not been fully migrated to the new sensor framework,
whereas Type A sensors actually use the full sensor framework as described above. In future releases, all
Type B and C sensors will be reformatted to Type A.
# Sensor Name Freq Type
1 Application.Main.ReplicationManager.Main.ParkedReplicationsClipCount 12h A
2 Application.Main.ReplicationManager.Main.ParkedRestoreClipCount 12h A
3 Application.Main.ReplicationManager.Main.ReplicationETA 24h A
4 Application.Main.ReplicationManager.Main.ReplicationSubstate 10m A
5 Application.Main.CCCManager.Main.RestoreState 15m A
6 Application.Main.SECComplianceAdapter.Main.ComplianceModeRevert 1m A
7 CapacityReportingManager.pool.<poolid>.capacity 15m A
8 Foundation.Cluster.PowerManager.Main.PowerRedundancyStatus 15m A
9 Foundation.Cluster.RegenerationManager.Main.ClusterClipIntegrity 1h A
10 Platform.Main.FPHealthCheck.Main.CPUTemperature 1h A
11 Platform.Main.FPHealthCheck.Main.VarPartitionUsedSpace 1h A
12 Server.Product.SuicidePreventionManager.Main.AvailableCapacityHardStopPercent 1h A
13 Server.Product.SuicidePreventionManager.Main.FreeObjectPercent 1h A
14 Server.Product.SuicidePreventionManager.Main.RegenerationBufferAlertPercent 1h A
15 Server.Product.SuicidePreventionManager.Main.RegenerationBufferHardStopPercent 1h A
16 Foundation.Cluster.RegenerationManager.Main.FailedRegenerations 1h A
17 Server.Product.UpgradeManager.Main.FailingNodeUpgrades 2m A
18 Hardware.Main.NIC.Main.ExternalNICFailureCount 1m B
19 Hardware.Main.NIC.Main.InternalNICFailureCount 1m B
20 Hardware.Main.Switch.Main.CubeSwitchFailureCount 1m B
21 Hardware.DiskOflline Real-time C
22 Hardware.NodeOflline Real-time C
23 Hardware.RootSwitchOflline Real-time C
Table 1 – Sensor List
SDK
In addition to the alerts triggered by CentraStar, there is also an alert issued by the SDK when no
communication can be established with a cluster. This alert is only issued via the MoPI alert stream.
10
Monitoring and Alerting
Overview of Monitoring Channels
• Each alert event3 triggered by Centera is available via the following monitoring channels, for all
Compliance models, provided they are enabled and configured on the cluster. For more information on
how to enable and setup these channels, please refer to the Centera Online Help, P/N 300-002-547.
Channel Alerts Real-time History

Supported
ConnectEMC ! !
ConnectEMC Notification ! !
Centera CLI
MoPI ! !
SNMP ! !
EMC Control Center ! ! !
Table 2 – Monitoring Channels and Frequency
• ConnectEMC4: If ConnectEMC is enabled alerts will automatically be sent to the EMC Customer Support
Center where it is determined if intervention by an EMC engineer is necessary. The system operator uses
the CLI to configure ConnectEMC and to view the ConnectEMC configuration.
• ConnectEMC Notification: after a message is sent to EMC, the system administrator (via the
ConnectEMC Recipient List) can receive an email notification with an HTML formatted copy of the
message. As of CentraStar 3.1, it is possible to receive email notifications without sending a message to
the EMC Customer Support Center.
• SNMP: The Simple Network Management Protocol (SNMP) is an Internet-standard protocol for managing
devices on IP networks. SNMP allows Centera to send alerts to storage management software. The system
operator uses the CLI to configure SNMP and to view the current state of the SNMP configuration.
• Monitoring API (MoPI): The Centera SDK has a possibility to receive alerts with the MoPI call
FPEventCallback_RegisterForAllEvents. Applications wishing to use this interface must have an access
profile defined with the Monitoring capability enabled. Please refer to the Centera API Reference Guide,
P/N 069001185 for more information on the MoPI interface.
• EMC ControlCenter: as of ECC v5.2, and CentraStar v2.2, EMC ControlCenter users can monitor one or
more Centera clusters in their storage environment. The EMC ControlCenter Storage Agent for Centera is
available free of charge and is qualified to run on the EMC ControlCenter server. To enable the storage
agent you need a license card with Part Number 100S-000549, which ships with each base cube
CNRRK8NG3 and can be ordered separately using CNR8NECCLIC. Please refer to the ControlCenter
Planning and Installation Guide Volume 1, P/N 300-000-684 for more information on how to configure
the EMC ControlCenter Storage Agent for Centera.
3
The SDK alert and improvement alerts are only available via the MoPI interface and in EMC ControlCenter.
4
ConnectEMC sends health reports in encrypted XML to the EMC Customer Service Center; it does this on a daily basis and after each
Alert message; this allows the service engineer to have a good understanding of the state of the cluster at time of the alert.
11
Alert Event Follow-up
The following table shows the full list of Centera alerts, a brief description of the problem reported and a
recommendation for resolution.
• Symptom Code uniquely determines an alert event (the first 5 numbers of the symptom code actually
identify the sensor triggering the alert).
• Threshold indicates the value at which the Evaluator will issue an alert event
• Severity indicates the severity level of the problem found; We distinguish the following severities:
OK (used for improvements only),
Notification,
Warning,
Error, and
Critical.
• The resolution discusses some recommended steps you can take to identify and resolve the problem.
NOTE: If the resolution indicates “EMC Support has been notified of the problem and a case will automatically be
opened”, this assumes ConnectEMC is enabled and the email alert was successfully sent to EMC. If ConnectEMC is not
enabled, you must contact the EMC Service Center yourself!
Symptom Code Sensor Name Threshold (Severity)

Problem Description Resolution
1.1.1.1.01.01 ParkedReplicationsClipCount > 10000 (Warning)
1.1.1.1.01.02 > 25000 (Error)
1.1.1.1.01.03 > 50000 (Critical)
The replication parking on 1 or more of the nodes in the EMC Support has been notified of the problem
cluster contains more than 10000/25000/50000 entries. This is and a case will automatically be opened.
typically due to connectivity issues between this cluster and
Check the replication parking using the CLI or
the replica, or authorization and capability issues on the
CV to identify the root cause of the problem.
replica. In order to prevent the replication parking to increase,
CentraStar has paused the replication process.
1.1.1.1.05.01 ParkedRestoreClipCount > 10000 (Warning)
1.1.1.1.05.02 > 25000 (Error)
1.1.1.1.05.03 > 50000 (Critical)
The restore parking on 1 or more of the nodes in the cluster EMC Support has been notified of the problem
contains more than 10000/25000/50000 entries. This is and a case will automatically be opened.
typically due to connectivity issues between this cluster and
Check the replication parking using the CLI or
the replica, or authorization and capability issues on the
CV to identify the root cause of the problem.
replica. In order to prevent the restore parking to increase,
CentraStar has paused the restore process.
12
1.1.1.1.03.01 ReplicationETA >= 24 h (Warning)
1.1.1.1.03.02 >= 48 h (Error)
1.1.1.1.03.03 >= 120 h (Critical)
The replication process on the cluster will probably take EMC Support has been notified of the problem
longer than 24/48/120 hours to process the entire replication and a case will automatically be opened.
queue, taking into account write and replication rates over the
past 24 hours. This might indicate a problem with replication
or a write activity that is higher than average.
1.1.1.1.20.01 ReplicationSubstate paused_clusterfull (Error)
CentraStar has paused the replication process on the cluster Check the capacity of the target cluster.
because the replica cluster has not enough capacity to service
the replication requests.
1.1.1.1.20.02 ReplicationSubstate paused_parking_overflow (Error)
The replication parking on this cluster contains more than Check the replication parking using the CLI or
10,000 entries on 1 or more nodes. This is typically due to CV to identify the root cause of the problem.
connectivity issues between this cluster and the replica, or
authorization and capability issues on the replica.
1.1.1.1.20.03 ReplicationSubstate paused_no_capability (Error)
The replica cluster has refused a replication request because Check if the required capabilities are enabled
the access profile used for replication does not have the on the target cluster.
correct capabilities.
1.1.1.1.20.04 ReplicationSubstate paused_authentication_failure (Error)
The replica cluster has refused a replication request because Check the access profile on the target cluster.
the access profile used for replication could not be
authenticated.
1.1.1.1.20.05 ReplicationSubstate paused_pool_not_found (Error)
The replica cluster has refused a replication request because a Check the pool definition on the target cluster.
pool being replicated could not be found.
1.1.1.1.20.06 ReplicationSubstate paused_user (Warning)
Replication has been paused by the user. Check why replication has been paused and
resume replication if possible.
1.1.1.1.20.07 ReplicationSubstate cancelled (Warning)
Replication has been cancelled by the user. Check why replication has been cancelled.
Please note that new writes are not being
added to the replication queue.
13
1.1.4.1.06.01 RestoreState paused_clusterfull (Error)
The target cluster does not have enough capacity to process Check the capacity of the target cluster.
the restore request.
1.1.4.1.06.02 RestoreState paused_parking_overflow (Error)
The restore parking on the cluster contains more than 10,000 Check the restore parking using the CLI or CV
entries on one or more nodes. to identify the root cause of the problem.
1.1.4.1.06.03 RestoreState paused_no_capability (Error)
The target cluster has refused the restore request because the Check if the required capabilities are enabled
access profile used for restore does not have the correct on the target cluster.
capabilities.
1.1.4.1.06.04 RestoreState paused_authentication_failure (Error)
The access profile used for the restore could not be Check the access profile on the target cluster.
authenticated.
1.1.4.1.06.05 RestoreState paused_pool_not_found (Error)
The target cluster has refused the restore request because the Check the pool definition on the target cluster.
pool for the C-Clip being restored was not found on the target
cluster.
1.1.4.1.06.06 RestoreState paused_user (Warning)
The restore has been paused by the user. Check why restore has been paused and
resume restore if possible.
1.1.4.1.06.07 RestoreState cancelled (Warning)
The restore has been cancelled. Check why restore has been cancelled.
1.1.2.1.02.01 ComplianceModeRevert Average <> Maximum (Critical)
One or more nodes in the cluster do not share the cluster EMC Support has been notified of the problem
compliance settings. and a case will automatically be opened.
2.1.2.1.01.01 ClusterClipIntegrity INCOMPLETE (Error)

CentraStar has determined that 1 or more C-Clips have EMC Support has been notified of the problem
become unavailable due to a combination of disks and/or and a case will automatically be opened.
nodes being offline.
14
4.1.1.1.01.01 CPUTemperature > 66 °C (Warning)
4.1.1.1.01.02 > 86 °C (Error)
CentraStar has determined that 1 or more nodes in the cluster EMC Support has been notified of the problem
have a running temperature higher than 66 °C/86 °C. This will and a case will automatically be opened.
reduce the performance of the affected nodes and if the
situation persists, the node will go offline.
4.1.1.1.02.01 VarPartitionUsedSpace > 88% (Error)
CentraStar has determined that 1 or more nodes in the cluster EMC Support has been notified of the
with an internal partition used by CentraStar have limited
problem and a case will automatically be
capacity.
opened.
5.2.2.1.03.01 AvailableCapacityHardStop Percent < 20% (Warning)
5.2.2.1.03.02 < 10% (Error)
5.2.2.1.03.03 <= 5 % (Critical)
This sensor will be active when no Regeneration Buffer has Contact your EMC sales representative to
been defined. The alerts are issued when available capacity order more capacity.
on at least 1 cube in the cluster is less than 20%/10%/5% of
For the Error and Critical levels, EMC Support
the total raw capacity. As there is no regeneration buffer
will be notified of the problem and a case will
defined, it will soon not be possible to write to the affected
automatically be opened.
cube(s) anymore and disk and node regenerations will not be
able to complete.
5.2.2.1.04.01 FreeObjectPercent <20 % (Warning)
5.2.2.1.04.02 <10 % (Error)
5.2.2.1.04.03 =0 % (Critical)
The free object count on the cluster is less than 20/10/0% of Contact your EMC sales representative to
the allowed object count. It will soon not be possible/is no order more capacity.
longer possible to write to the cluster anymore.
5.2.2.1.01.01 RegenerationBufferAlert Percent < 125 % (Notification)
5.2.2.1.01.02 < 100 % (Warning)
5.2.2.1.01.03 < 50 % (Error)
5.2.2.1.01.04 < 25% (Critical)
This sensor will be active when a Regeneration Buffer has Contact your EMC sales representative to
been defined with the Alert Only policy. The available order more capacity.
capacity on at least 1 cube in the cluster is 25% higher
than/equal to/50% of/25% of the configured regeneration
buffer. As the policy is set to Alert Only it is still possible to
write to the afftected cube(s).
15
5.2.2.1.02.01 RegenerationBufferHardStopPercent < 150% (Warning)
5.2.2.1.02.02 < 125% (Error)
5.2.2.1.02.03 <= 100% (Critical)
This sensor will be active when a Regeneration Buffer has Contact your EMC sales representative to
been defined with the Hard Stop policy. The available order more capacity.
capacity on at least 1 cube in the cluster is 50%/25% higher
than/less than the configured regeneration buffer. As the
policy is set to Hard Stop it will soon/is not possible to write
to the affected cube(s) anymore. Disk and node regenerations
will still be able to use the capacity reserved in the
regeneration buffer.
3.1.3.1.02.01 ExternalNICFailureCount > 1 (Error)
CentraStar has determined that 1 or more NICs used for data EMC Support has been notified of the problem
access by your application are not functioning correctly. This and a case will automatically be opened.
will reduce bandwidth available for data access by your
application.
3.1.3.1.01.01 InternalNICFailureCount 1 (Warning)

3.1.3.1.01.02 >1 (Error)
CentraStar has determined that 1 or more NICs used by the EMC Support has been notified of the problem
internal Centera network are not functioning correctly. and a case will automatically be opened.
3.1.4.1.01.01 CubeSwitchFailureCount = 1 (Error)
3.1.4.1.01.02 > 1 (Critical)
CentraStar has determined that 1 or more cube switches in the EMC Support has been notified of the problem
cluster are not functioning correctly. This will reduce the and a case will automatically be opened.
internal bandwidth of the cluster and represents a single point
of failure until addressed.
3.1.5.1.01.01 DiskOffline 1 (Warning)
There are 1 or more disks offline in the cluster. EMC Support has been notified of the problem
and a case will automatically be opened.
3.1.6.01.01.01 NodesOffline 1 (Warning)

There are 1 or more nodes offline in the cluster. EMC Support has been notified of the problem
16
3.1.4.1.02.01 RootSwitchFailures 1 (Warning)
There is 1 root switch in the cluster not functioning correctly. EMC Support has been notified of the problem
1.1.3.1.01.01 PoolReachingQuota >80% (Warning)
1.1.3.1.01.02 >90% (Error)
1.1.3.1.01.03 > 100% (Critical)
This alert indicates that one or more pools have almost used EMC Customer Support Center has been
up their capacity quota. The pool capacity is virtual capacity notified of the problem but a case will not be
and is not related to the physical capacity of a Centera. opened.
Reaching a pool quota generates an alert but it does not stop
an application from writing to the affected pool(s).
2.1.3.1.01.015 NoRedundantPower > 1 (Warning)
This alert indicates that 1 or more nodes on a cluster are not EMC Customer Support Center has been
receiving redundant power. notified of the problem but a case will not be
opened.
2.1.2.1.03.01 FailedRegenerations > 0 (Error)
At least one disk or node regeneration cannot complete or has EMC Support has been notified of the problem
failed. and a case will automatically be opened.
5.2.7.1.02.01 FailingNodeUpgrades > 0 (Error)
At least one node has failed an upgrade process. EMC Support has been notified of the problem
n/a SDK Alert No connection
The SDK is unable to establish a connection with Centera. Please verify the network connection between
the application server and Centera and the
availability of the Centera.
Table 3 – Centera Alert List
5
This alert is not part of the standard CentraStar build – please contact you EMC Service to enable it.
17
Scheduled Reporting
Health Reporting
Introduction
Complementing the alerting functionality, CentraStar also has the ability to generate health reports. These
health reports contain the following information:
• System configuration information;
• General status and health of the various hard- and software components;
• Capacity information for clusters, cubes, nodes and pools;
• Performance information;
• Cluster wide sensor output.
Overview of Health Reporting Channels

Health reports are available via different channels, for all Compliance models, provided they are enabled and
configured on the cluster. The following table provides an overview of which channels support this feature and
the frequency with which they are made available:
Channel Health Sent Daily6 Sent After On Request

Report Alert Event7
Supported
ConnectEMC ! ! !
ConnectEMC Notification ! ! !
Centera CLI ! !
MoPI ! !
SNMP
EMC Control Center ! !
Table 4 – Health Report Channels and Frequency
• ConnectEMC: If ConnectEMC is enabled, health reports will automatically be sent to the EMC Customer
Support Center where they are analyzed to spot potential issues before they actually occur; if needed, a
support call will be automatically opened. The system operator uses the CLI to configure ConnectEMC
and to view the ConnectEMC configuration. Please refer to the Centera Online Help, P/N 300-002-547
for more information on ConnectEMC.
• ConnectEMC Notification: after a message is sent to EMC, the system administrator (via the
ConnectEMC Recipient List) can receive an email notification with an HTML formatted copy of the
message. Please refer to the Centera Online Help, P/N 300-002-547 for more information on setting up
ConnectEMC.
6
This is the default behavior; the frequency can be changed by EMC Service or the EMC Control Center administrator.
7
If multiple monitoring events are generated within a 15 minute window, only 1 health report is sent. The information in the health
report allows the service engineer to have a good understanding of the state of the cluster at time of the alert.
18
• Monitoring API (MoPI): The Centera SDK has a possibility to receive health reports via the MoPI
FPMonitor_GetDiscovery call . Applications wishing to use this interface must have the Monitoring
capability enabled. Please refer to the System Centera Online Help, P/N 300-002-547 for information on
how to define monitoring-enabled access profiles for EMC ControlCenter and other applications which
want to use the MoPI interface, and refer to the Centera API Reference Guide, P/N 069001185 for more
information on the MoPI interface.
• Centera Command Line Interface (CLI): the system administrator can download the health report
which is sent via ConnectEMC in native XML format using the show report health <filename> command.
Please refer to the Centera Centera Online Help, P/N 300-002-547 for more information on the CLI.
• EMC ControlCenter: as of ECC v5.2, and CentraStar v2.2, EMC ControlCenter users can view
configuration settings and capacity information which is refreshed by default every 15 minutes. The EMC
ControlCenter Storage Agent for Centera is available free of charge and is qualified to run on the EMC
ControlCenter server. To enable the storage agent you need a license card with Part Number 100S-
000549, which ships with each base cube and can be ordered separately using CNR8NECCLIC. Please
refer to the ControlCenter Planning and Installation Guide Volume 1, P/N 300-000-684 for more
information on how to configure the EMC ControlCenter Storage Agent for Centera.
Scheduled Reporting Using CLI Scripting
General
The Centera Command Line Interface (CLI) which is part of the standard Centera toolset can be executed from
a script file and scheduled in both Windows (scheduled tasks) and Solaris (cron).
In order to make use of this feature, simply install the CLI on the server or workstation you wish to schedule the
script on, and create a script file for each job you wish to schedule.
Creating a script file

The script file is a simple text file which contains the commands and parameters to be executed. Each line
contains an entry for the CLI, so each command and each parameter requested by the CLI must be put on a
separate line. The following is an example of a script that shows a replication report, pauses replication and
shows the report again.
show replication detail
replication pause
yes
show replication detail
quit
Figure 2 – script.cli
Creating a batch file

The batch file is a platform specific file which launches the CLI and executes the script file. The general format
of the CLI command is as follows:
java -cp CenteraViewer.jar com.filepool.remote.cli.CLI -u USER -p PASSWORD -ip
CLUSTER_IP -script SCRIPT_FILE
Figure 3 – CLI Script Mode Syntax
The following parameters need to be specified:

• USER: the access profile that will be used to execute the script.
• PASSWORD: the secret associated with the access profile.
19
• CLUSTER_IP: the ip address of one of the cluster’s access nodes.
• SCRIPT_FILE: the file containing the CLI script to execute.
Note: Please make sure the CenteraViewer.jar file is located in the directory where you launch the script from – if not, you
will receive the following Java error: java.lang.NoClassDefFoundError: com/filepool/remote/cli/CLI.
Windows
The following batch file runs the script we created earlier and directs the output to a file.
@echo off
CLUSTER_IP -script %1 > %2
Figure 4 – sample.bat
Figure 5 – Windows Scheduled Tasks
The batch file can be scheduled in Windows using Start> Settings> Control Panel> Scheduled Tasks. Create a
new task by clicking on the Add Scheduled Task wizard icon, or by selecting File> New> Scheduled Task in
the window menu.
Figure 6 – Windows Scheduled Task Details
20
In the task details, provide the name of the batch file with the script and output file as parameters.
Figure 7 – Windows Scheduled Task List
In the Scheduled Tasks window, the task shows up with information about the time it will run next and about
the last run.
Solaris
On Solaris, one can use the cron utility that allows tasks to be automatically run in the background at regular
intervals by the cron daemon.
The following batch file runs the script we created earlier and directs the output to a file.
#!/bin/sh
CLUSTER_IP -script $1 > $2
Figure 8 – sample.sh
You must ensure that the batch file has at least read and execute permissions (chmod 555) and then you can
schedule it via the crontab utility by typing crontab -e. Add a line to the crontab file for the batch file we just
created.
* * * * * command to be executed
- - - - -
| | | | |
| | | | +----- day of week (1 - 7) (monday = 1)
| | | +------- month (1 - 12)
| | +--------- day of month (1 - 31)
| +----------- hour (0 - 23)
+------------- min (0 - 59)
Figure 9 – Crontab Entry
21
Real-time Reporting
Capacity Reporting
General
EMC has standardized capacity reporting using a 1,024 notation. Whereas CentraStar reported capacity either
in bytes or by aggregating bytes using the 1,000 notation, this has now been synchronized to 1,024 notation in
CentraStar 3.1. Centera ChargeBack Reporter and the ConnectEMC Notification HTML report already were
using the 1024 notation.
Notation < 3.1 3.1
8
CLI/CV 1,000 1,024
ConnectEMC XML Bytes Bytes
ConnectEMC HTML 1,024 1,024
MoPI Bytes Bytes
ChargeBack Reporter 1,024 1,024
EMC Control Center 1,024 1,024
Table 5 – Capacity Notations
The following table shows how to convert from 1000 to 1024 notation.
1,000 1,024 Difference
1 KB 0.97 KB 2%
1 MB 0.95 MB 5%
1 GB 0.93 GB 7%
1 TB 0.91 TB 9%
Table 6 – Capacity Transformation
When discussing capacity reporting, it is important to note that different types of users have different reporting
needs. In order to address these different use cases, Centera provides a number of types of capacity reports:
• Raw Capacity: this provides a view on the physical capacity present in the system, as reported by the
operating and file system layer.
• Available Capacity: this view reports how much of the free raw capacity as reported in the Raw Capacity
view is actually available by applications to store data and meta data. The remaining raw capacity is
reserved for system use (database work buffers, growth, failover, regeneration …).
• Useable Capacity: this view is currently not provided via the Centera tools, but this paper discusses how
you can derive information on how much new data can be written to Centera given the available capacity
and taking into account the protection scheme used.
• Pool Capacity: this is the application’s view on capacity and represents the amount of user data that is
currently stored in a certain pool.
• Meta data Driven Capacity: this view reports the amount of user data categorized and grouped by
information from the C-Clip’s meta data tags9.
8
Using a 3.1 CLI or CV will report 1024 notation, also when running against a cluster running a CentraStar version older than 3.1.
22
Terms & Definitions
Terms Definitions
Total Raw Capacity The total physical capacity of the cluster/cube/node or disk.
Free Raw Capacity The capacity that is free and available for storing data or for self healing operations
in case of disk or node failures or for database growth and failover.
Used Raw Capacity The capacity that is used or otherwise not available to store data; this includes the
capacity reserved as system resources, not assigned for storage or offline, and
capacity actually used to store user data and associated audit and meta data.
System Resources The capacity that is used by the CentraStar software and is never available for
storing data; includes work directories and swap space. This component reports
system resources for ALL nodes, including Spare and Offline nodes.
Spare Capacity The capacity that is available on nodes that do not have the storage role assigned.
This component does NOT include System Resources.
Offline Capacity The capacity that is temporarily unavailable due to reboots, offline nodes, or
hardware faults, including nodes which do not have the storage role assigned. This
capacity will be available as soon as the cause has been solved. This component
does NOT include System Resources.
Used Capacity The capacity that is in use to store data. This includes Protected User Data plus
Audit & Meta Data.
Protected User Data The capacity taken by user data, including CDFs, reflections and protected copies of
user files.
Audit & Meta Data The overhead capacity required to manage the stored data. This includes indexes,
databases, and internal queues.
System Buffer Allocated space that allows internal databases and indexes to safely grow and
failover. As the system is filled with user data, and the Audit & Meta Data capacity
increases, the capacity allocated to the System Buffer decreases.
Regeneration Buffer Space that is allocated for regeneration. Depending on the Regeneration Buffer
Policy, this allocation can be a soft (Alert Only) or hard (Hard Stop) allocation.
Available Capacity until Alert The amount of capacity available until the regeneration buffer is reached and an
alert is raised. Irrespective of the Regeneration Buffer Policy, this equals Free Raw
Capacity - System Buffer - Regeneration Buffer.
Available Capacity The amount of capacity available to write. If the Regeneration Buffer Policy is set
to Alert Only, this equals Free Raw Capacity - System Buffer. If the Regeneration
Buffer Policy is set to Hard Stop, this equals Free Raw Capacity - System Buffer -
Regeneration Buffer.
Pool Quota Current quota for the pool. This is the maximum amount of data that can be written
to the pool.
Pool Used Capacity Current pool capacity being used.
Pool Free Capacity Current available capacity until the quota is reached.
Pool C-Clip Count Number of C-Clips stored in the pool.
Pool User File Count Number of user files stored in the pool.
Total object count The number of objects that can be stored.
Used object count The total object capacity already used.
Free object count The total number of objects that can still be written to the cluster.
Table 7 – Capacity Terms & Definitions
9
For more information on this type of reporting, please consult the ChargeBack Reporter Online Help.
23
Capacity Components Overview
On a freshly installed system, some capacity is already claimed and used by CentraStar, while some free
capacity is reserved and not available for use by applications to store data.
System Resources
System Buffer
DB/Index/
Audit & Meta-data
File-system
Initialization
Available Capacity Until Alert
Legend
Used
Alert Hard Regen Buffer
Only Stop Free
Free & Available
Figure 10 – Capacity Components On Empty System
As applications write information to Centera, the internal databases and indexes, reported as Audit & Meta data,
grow and take up the capacity which was reserved as System Buffer. The actual data, CDF and Blobs are stored
in the remaining space.
Figure 11 – Capacity Components On Used System
24
Overview of Capacity Reporting Tool Coverage
Channel Raw Available Pool Meta data Historic
ConnectEMC ! ! !
ConnectEMC Notification ! ! !
Centera CLI ! ! !
MoPI ! ! !
SNMP
EMC Control Center ! ! ! !
ChargeBack Reporter !
Table 8 – Capacity Reporting Channels and Frequency
Raw Capacity Reporting

On each disk, CentraStar has a number of system partitions which are used to store the CentraStar code and
include some working directories and swap space. These partitions are reported as System Resources and are
not available for data storage. With Gen4 the swap space was increased to 1GB per drive, and the single data
partition per drive was split in 100 GB partitions. The following table show how the drives in different
hardware generations are partitioned.
GB (1024 notation) Gen2 (250) Gen3 (320) Gen4 (320) Gen4 (500)
Total Raw Capacity 233.7 301.7 301.7 465.7
System Resources 1.7 1.7 2.7 2.7
Data Partition 232.0 300.0 299.0 463.0
99.3% 99.5% 99.1% 99.4%
Table 9 – Disk Level Raw Capacity
On a newly installed system, the 4 disks in a node are pre-formatted and the file-system journal and internal
databases and indexes initialized. This capacity is reported as used in audit and meta-data. The table below
shows how much capacity is reported as Free Capacity on newly installed nodes of different hardware
generations.
Total Raw Capacity 934.6 1,206.6 1,206.6 1,862.6
Audit & Meta Data 0.3 0.3 0.3 0.3
Free Capacity 927.7 1,199.7 1,195.5 1,851.5
99.3% 99.4% 99.1% 99.4%
Table 10 – Node Level Raw Capacity
25
The following report, which is available via the CLI, shows a typical Raw Capacity Report for a cluster:
Number of nodes: 8
Number of nodes with storage role: 6
Total Raw Capacity: 10,309 GB (100%)
Used Raw Capacity: 7,719 GB (75%)
Free Raw Capacity: 2,610 GB (25%)
System Resources: 57 GB (1%)
Offline Capacity: 3,844 GB (37%)
Spare Capacity: 2,563 GB (25%)
Used Capacity: 1,255 GB (12%)
Audit & Meta data: 397 MB (0%)
Protected User Data: 1,255 GB (12%)
Used Object Count: 2 M
Figure 12 – Show Capacity Total
Available Capacity Reporting

Available capacity is defined as the amount of data that can be stored on a Centera, which corresponds to the
Free Raw Capacity available at installation, minus two buffers:
• The System Buffer is reserved for system use;
• The Regeneration Buffer is reserved for regeneration of data when disks and/or nodes fail.
Available capacity will be used to store actual user data, protection data and audit and meta data.
System Buffer
The system buffer is a dynamic buffer which allocates space to allow internal databases and indexes to safely
grow and failover. As the system is filled with user data, and the Audit & Meta Data capacity increases, the
capacity allocated to the system buffer decreases.
With CentraStar 3.0, the BlobIndex has been spread over the 4 available disks, and an additional reservation is
made to allow safe growth of the database up to 7.5 million objects per disk or 30 million per node. The
BlobIndex itself has grown to accommodate for the additional pool information. A system of soft reservation
has been introduced to ensure that volumes which hold one of the databases are not used for data storage until
all other volumes reach a certain threshold.
With the introduction of CentraStar 3.1, the maximum object count has been raised to 50 million per node for
Gen4 nodes. For Gen2 and Gen, the limit can also be increased to 50 million objects per node, but a disruptive
migration task needs to be executed. Please contact your service representative for more information.
26
These reservations result in the following node level available capacity for systems running CentraStar 3.1
supporting 30 million objects per node:
GB (1024 notation) Gen2 (250) Gen3 (320)
Total Raw Capacity 934.6 1,206.6
System Resources 6.6 6.6
Audit & Meta Data 0.2 0.2
System Buffer 139.0 139.0
Available Capacity 788.8 1,060.8
84% 88%
Object Limit 30M 30M
Table 11 – Node Level Available Capacity – CentraStar 3.1 – 30M Objects
These reservations result in the following node level available capacity for systems running CentraStar 3.1
supporting 50 million objects per node:
Total Raw Capacity 934.6 1,206.6 1,206.6 1,862.6
Audit & Meta Data 0.2 0.2 0.2 0.2
System Buffer 200.0 200.0 208.2 208.2
Available Capacity 727.8 999.8 987.4 1,643.4
78% 83% 82% 88%
Object Limit 50M 50M 50M 50M
Table 12 – Node Level Available Capacity – CentraStar 3.1 – 50M Objects
Regeneration buffer
The regeneration buffer is capacity set aside per full or partial Cube (8 to 32 nodes for Gen2-3 and 4 to 16 nodes
for Gen4) that allows Centera enough space to regenerate disks and nodes that go offline.
The System Administrator can use the CLI to specify the size of the regeneration buffer per cube as well as the
policy to be used when the free space in any cube falls below the regeneration buffer:
• Alert Only will alert the System Administrator but applications can continue writing until there is not more
available capacity;
• Hard Stop will alert the System Administrator and turn the affected cube in Read-Only mode.
27
The default and minimum recommended policy is set to 1 Disk per Cube with a hard-stop policy. We
recommend system administrators to set the following policy:
Smallest Cube Size in the Cluster Buffer Size (disks) Recommended Policy
4 1 Hard Stop
8 4 Alert
16 4 Alert
24 4 Alert
32 4 Alert
Table 13 – Recommended Regeneration Buffer Policy
Please note that due to the fact that two mirror copies of a CDF or any file written in CPM will be stored on
separate Mirror Groups (set of nodes powered by the same power rail), regeneration of a disk or node will only
use half of the available remaining disks. To ensure regeneration has enough capacity to work with, CentraStar
3.0 and 3.1 actually reserve twice the amount of capacity by using the following formula for each Cube:
<number of disks specified by System Admin> * <largest disk size in Cube> *2
This results in the following capacity reservation and cluster level available capacity for systems running
CentraStar 3.1 supporting 50 million objects per node with a minimum policy of 1 disk per cube:
GB (1024 notation)
Minimum Policy – 1 Disk Gen2 (250) Gen3 (320) Gen4 (320) Gen4 (500)
4 Node Total Raw Capacity 3,738.5 4,826.5 4,826.5 7,450.6
Regeneration Buffer 467.3 603.3 603.3 931.3
Available Capacity 2,444.0 3,396.0 3,346.4 5,642.4
65% 70% 69% 76%
72% 77% 76% 82%
74% 79% 78% 84%
75% 80% 79% 85%
Table 14 – Node Level Available Capacity – CentraStar 3.1 – 50M Objects – 1 Disk Policy
28
If the recommended 4 disk policy is used, this results in the following capacity reservation and cluster level
available capacity for systems running CentraStar 3.1 supporting 50 million objects per node:
GB (1024 notation)
Recommended Policy – 4 Disks Gen2 (250) Gen3 (320) Gen4 (320) Gen4 (500)
Regeneration Buffer 1,869.2 2,413.2 2,413.2 3,725.3
53% 58% 57% 63%
61% 66% 65% 72%
65% 70% 69% 76%
Table 15 – Node Level Available Capacity – CentraStar 3.1 – 50M Objects – 4 Disk Policy
Regeneration Buffer Alerting
As mentioned in section “Alert Event Follow-up”, there are three sensors which track the Available Capacity
and alert when any Cube’s Available Capacity falls under a certain threshold. There are three possible
scenario’s, each monitored by a specific sensor.
Regeneration Buffer = 0
In this case, an alert will be sent when Available Capacity on at least 1 Cube is less than 20% of the Total Raw
capacity for the Cube.
Regeneration Buffer > 0 and Policy = Alert Only
In this case, an alert will be sent when Available Capacity on at least 1 Cube in the cluster is 25% higher than
the configured regeneration buffer for that Cube.
Regeneration Buffer > 0 and Policy = Hard Stop
In this case, an alert will be sent when Available Capacity on at least 1 Cube in the cluster is 50% higher than
the configured regeneration buffer for that Cube.
Please note that for small regeneration buffer sizes, the first alert will be issued when the cluster is relatively full. It is
therefore recommended to set the regeneration buffer to at least 4 disks for any cluster over 4 nodes.
29
Regeneration Buffer Reporting
The following report, which is available via the CLI, shows a typical Available Capacity Report for a cluster,
with an Alert Only Policy:
Number of nodes: 8
System Buffer: 774 GB (7%)
Regeneration Buffer: 350 GB (3%)
Available Capacity: 7,937 GB (77%)
Total Object Count: 240 M (100%)
Used Object Count: 17 M (7%)
Free Object Count: 223 M (93%)
Figure 13 – Show Capacity Availability
Object Count Reporting

As with any file or object level storage device, there is a limit as to the number of objects can be stored on a
specific drive and node. This object limit is a combination of file system and database limitations, as well as
some limitations which are the result of having to iterate of billions of objects across the entire cluster in
process such as regeneration and garbage collection.
First of all, it is important to understand what constitutes an object on a Centera cluster. The following list
details the different type of objects that count against the object count limits within a Centera:
• CDF: a meta data file associated with each C-Clip written to Centera.
• EBR/LH metadata: a meta data file associated with a C-Clip which is under EBR or LH, containing audit
information about the event or litigation hold.
• User file/fragment (CPM): a full copy of a mirrored user file or a 100MB fragment of a mirrored user file
• User file/fragment (CPP): a 1/6 fragment of a user file or a 1/6 fragment of a 100MB fragment of a user
file.
• Reflection: a meta data file associated with a deleted C-Clip, containing audit information about the delete
– a reflection replaces the CDF and possibly the EBR/LH metadata file
The following table shows how a typical C-Clip containing a single user file (<100MB) generates objects over
its life-cycle.
CPM CPP
Total Delta Total Delta
Objects Objects
User file Written 2 +2 7 +7
CDF Written 4 +2 9 +2
Litigation Hold Set 6 +2 11 +2
C-Clip Deleted 4 -2 9 -2
Garbage Collection 2 -2 2 -7
Table 16 – Object Count per C-Clip
30
As to the object count limits reported and imposed by CentraStar, there is a qualified limit and a hard stop limit
imposed by the system. Filling a node beyond the qualified limit increases node and disk regeneration times,
and will slow down processes such as garbage collection, restore and query. When a node has reached the hard
stop limit, the node will not accept any new objects on the node. When the entire cluster is approaching the
hard stop limit, CentraStar will issue alerts at 80%, 90% and 100%.
The following table shows the limits in millions of objects per node for various CentraStar releases and
hardware generations:
CentraStar Hardware Qualified Hard Stop
2.4 Gen2/3 25 50
3.0 Gen2/3 25 50
3.0 Gen4 30 5010
3.1 Gen2/311 30 50
3.1 Gen4 50 50
Table 17 – Object Count Limit per Node (M)
Pool Capacity Reporting

Pool Capacity is defined and reported in terms of the front-end or application view, reporting the capacity and
number of C-Clips and User Files that different applications wrote to Centera, minus what they deleted.
Unlike Raw Capacity, Pool Capacity does not depend on the protection scheme being used nor on single
instancing, and it does not fluctuate as processes such as regeneration and garbage collection process files.
Each time a C-Clip is written to a pool, pool capacity is added; each time a C-Clip is deleted, capacity is
removed. The amount with which the pool capacity is adjusted is the sum of the size of all the User Files
embedded or referenced in the C-Clip12.
Quota Management
An important aspect of managing the capacity for any storage device shared by multiple applications is the
ability to set and monitor quota. CentraStar 3.0 allows the system administrator to define a quota at the pool
level and will issue an alert for any pool which is close to using all of its quota.
CentraStar does not provide a hard-stop policy, so it is up to the system administrator to determine whether or
not to allow the application to write more data to the pool. The system administrator can use the pool mask to
disable write access to the pool, or remove the write capability from the profile(s) which have the pool as their
home pool.
Pool Capacity Calculation
CentraStar uses two complementary processes to calculate pool capacity. The first one processes C-Clips ‘on-
the-fly’ as they are written and deleted from Centera. The second is used to initialize pool capacity on an
upgraded system or to re-calculate the capacity of pools after a migration.
10
If a cluster was installed or upgraded to CentraStar 3.0 prior to February 15, 2006, then the Hard Stop limit was set to 30M – please
contact EMC Support if you are getting object count alerts (Symptom codes 5.2.2.1.04.0x).
11
For Gen2 and Gen, the limit can also be increased to 50 million objects per node, but a disruptive migration task needs to be executed.
Please contact your service representative for more information.
12
The size of the actual CDF is not taken into account.
31
Real-time Capacity Calculation
As an application writes and deletes C-Clips from the pool(s) it uses, the access nodes processing the write and
delete calls adjust the pool capacity according to the algorithm in the table below:
Write Clip Delete Clip
C-Clip Count Add 1 Subtract 1
User File Count Add 1 for each User File referenced or Subtract 1 for each User File referenced
embedded in the C-Clip or embedded in the C-Clip
Pool Capacity Add the capacity for each User File Remove the capacity for each User File
referenced or embedded in the C-Clip referenced or embedded in the C-Clip
Statistic Home Pool Pool of C-Clip
Current Month Month C-Clip was written
Table 18 – Pool Capacity Algorithm
For each Pool, CentraStar does not just keep a running total, but it maintains a statistic per month, from which
the total is derived13. The monthly statistic allows us to re-calculate the statistics for a certain month (see next
section). The following figure depicts how the statistics are maintained and affected by write and delete
activity.
1
Today is April 10th
Application A
Writes a new C-Clip
Capacity Pool A
7 7
6
3 1
2
Delete C-Clips written 1 1 1
in Jan and Feb
Jan Feb Mar Apr Total Before After
1
Figure 14 – Pool Capacity Algorithm
In order to minimize the impact of this process on performance, these statistics are only persisted and
communicated to other nodes every five minutes.
13
These monthly values are not available in reporting.
32
Capacity Initializaton and Re-Calculation
The second process is a temporary process which is run after the upgrade to CentraStar 3.0 in order to initialize
pool capacity statistics for historic data in the default pool. It can also be run after migrating data from the
default pool.
The process iterates through all the C-Clips in a pool based on their write time-stamp and calculates the pool
capacity statistics per monthly time-slots, either for all pools and all time periods, or for a certain pool and for a
certain time period. When it has completed processing all the C-Clips written in a month, it replaces the
existing monthly statistics with the new information.
Figure 15 – Pool Capacity Re-calculation Algorithm
Using the CLI command show pool capacitytasks you can monitor the progress of this background task.
Pool Capacity Reporting Limitations
Pool capacity statistics are provided on a best-effort basis. The following limitations must be taken into account
when using pool capacity reports:
• As pool capacity statistics are not persisted real-time, the process is vulnerable to access nodes going
offline, and the pool capacity statistics will not be 100% accurate. If we assume access nodes go offline
once per month, which results in 12 periods of maximum 5 minutes where the statistics are not persisted,
this corresponds to an error rate of 0.01%.
• Please note that if the application is deleting C-Clips while the pool capacity is being initialized or re-
calculated, the deletes will not be reflected in the pool capacity.
• If a C-Clip is written multiple times, or different versions (via Clip Update) are stored of the same C-Clip
without deleting the original C-Clip, the pool capacity will be updated each time for the entire capacity
represented in the C-Clip.
33
• Finally, pool capacity does not reflect actual capacity taken on disk. Because of the protection scheme and
the CDF and database overhead, more raw capacity is being used than reported in pool capacity.
• Finally, pool capacity also does not take into account single instancing, which means less raw capacity will
be reported as used than what would be expected by multiplying the pool capacity with the protection
scheme factor (2x for CPM).
Useable Capacity Reporting

Useable capacity is defined as the amount of user data that can be stored on a Centera, which means it must not
include the protection data and audit and meta data. CentraStar currently does not report useable capacity, but
we can offer some general guidelines to determine the amount of user data that can still be stored on a Centera,
provided the usage patterns and protection scheme settings do not change.
Ultimately, a Centera is either capacity bound or object bound14. This depends on the average size of the user
objects that are stored and the protection scheme. A Centera with small files will typically be object bound, a
Centera with large files will be capacity bound. To find the actual Useable Capacity for a Centera, calculate
the value for both methods and take the smaller of the two results.
Relevant Reports Needed To Calculate Useable Capacity
In order to perform the calculations described below, you need to issue the following commands in the CLI and
note the capacity components marked in bold.
Number of nodes: 8

System Resources: 57 GB (1%)
Offline Capacity: 0 GB (0%)
Spare Capacity: 0 GB (0%)
Used Capacity: 1,255 GB (12%)
Audit & Meta data: 397 MB (0%)
Protected User Data: 1,255 GB (12%)
Used Object Count: 17 M
Figure 16 – Show Capacity Total

Capacity / Pool Quota Used Free C-Clips Files
Email Pool 500 GB 400 GB 100 GB 3,000 K 6,000 K
Document Pool 300 GB 250 GB 50 GB 100 K 100 K
Total 800 GB 650 GB 150 GB 3,100 K 6,100 K
Figure 17 – Show Pool Capacity
14
As any other storage device, Centera is limited in the amount of objects any single node can store. An object can be a CDF copy, a
blob copy, a blob parity fragment or a reflection.
34
Number of nodes: 8
System Buffer: 774 GB (7%)
Regeneration Buffer: 350 GB (3%)
Available Capacity: 7,937 GB (77%)
Total Object Count: 240 M (100%)
Used Object Count: 17 M (7%)
Free Object Count: 223 M (93%)
Figure 18 – Show Capacity Availability
Useable Capacity for Capacity Bound Systems

Average C-Clip Size = <Total Used|Pool> / <Total C-Clips|Pool>
= 650 GB / 3,100,000
= 209 KB
Average C-Clip Capacity = <Used Capacity|Raw> / <Total C-Clips|Pool>
= 1,255 GB / 3,100,000 = 405 KB
Useable C-Clips = <Available Capacity> / <Average C-Clip Capacity>
= 7,937 GB / 405 KB
= 19,605 K
Useable Capacity = <Useable C-Clips> * <Average C-Clip Size>
= 19,605 K * 209 KB
= 4,097 GB
Useable Capacity for Object Bound Systems

Average C-Clip Size = <Total Used|Pool> / <Total C-Clips|Pool>
= 650 GB / 3,100,000
= 209 KB
Average Objects/C-Clip = <Used Object Count> / <Total C-Clips|Pool>
= 17,000,000 / 3,100,000
= 5.5
Useable C-Clips = <Free Object Count> / <Average Objects/C-Clip>
= 223,000,000 / 5.5
= 40.6 M
Useable Capacity = <Useable C-Clips> * <Average C-Clip Size>
= 40.6 M * 209 KB
= 8,498 GB
Best Practices for Object Bound Systems
In order to increase storage efficiency, the system administrator and application owner have the following
options:
• The 2.3 SDK introduced embedded blobs, which automatically embeds small files in the CDF – this
reduces the used object count per C-Clip.
• Containerization (combining multiple small files in a single blob) is available in some partner applications
and can be a viable solution; containerization should be used with caution as it reduces the granularity of
assigning meta data and the possibility to delete and shred individual files.
35
Meta data Driven Capacity Reporting
With the introduction of Centera Seek and the ChargeBack Reporter application, Centera system administrators
can get a more granular view on capacity utilization based on any meta data present in the CDF. This allows
system administrators to meet specific storage chargeback needs such as reporting by organization (divisions,
business units, …) and/or by application or file type.
Capacity can be reported as actual bytes written (similar to Pool Capacity reporting) and space consumed
(calculated based on the protection scheme and threshold settings).
ChargeBack Reporter has a number of standard reports based on meta data created by CentraStar, and allows
the system administrator to create custom reports based on application specific meta data, meta data included in
the CDF by using environment variables on the application server, or using the new Profile Driven Meta data
feature in CentraStar 3.0.
Creating a new report consists of four simple steps:
• Define tags in the C-Clip Definition File (CDF) which you want to categorize data on; e.g. Tag
CustomerID == /ecml/eclipdescription/custom-meta/customerid;
• Define categories on which to report; e.g. Customer Acme == Tag CustomerID = 130532;
• Define reports using a hierarchical structure of categories;
• Define the schedule on which the report will be run.
Other features include scheduling of reports and export to XML files. For more information please refer to the
Centera ChargeBack Reporter Online Help.
36
Performance Reporting
Performance Sensors
CentraStar provides the following sensor based performance statistics; each is calculated every 15 minutes as an
hourly average:
Statistic Sensor Unit
Read Throughput Application.Main.CCCManager.Main.ReadThroughput@cluster C-Clips/s
Read Bandwidth Application.Main.CCCManager.Main.ReadBandwidth@cluster MB/s
Write Throughput Application.Main.CCCManager.Main.WriteThroughput@cluster C-Clips/s
Write Bandwidth Application.Main.CCCManager.Main.WriteBandwidth@cluster MB/s
Delete Throughput Application.Main.CCCManager.Main.DeleteThroughput@cluster C-Clips/s
Delete Bandwidth Application.Main.CCCManager.Main.DeleteBandwidth@cluster MB/s15
Replication Throughput Application.Main.ReplicationManager.Main.ReplicationClipThroughput@cluster C-Clips/s
Replication Bandwidth Application.Main.ReplicationManager.Main.ReplicationBandwidth@cluster KB/s
Restore Throughput Application.Main.RestoreManager.Main.RestoreClipThroughput@cluster C-Clips/s
Restore Bandwidth Application.Main.RestoreManager.Main.RestoreBandwidth@cluster KB/s
Incremental Garbage Application.Main.CCCManager.Main.IncGCBandwidth@cluster MB/s
Collection Sweep Bandwidth
Full Garbage Collection Application.Main.CCCManager.Main.FullGCBandwidth@cluster MB/s
Sweep Bandwidth
Table 19 – Performance Sensors
Overview of Performance Reporting Tool Coverage

Channel Read/Write Delete GC Replication Restore
ConnectEMC ! ! ! ! !
ConnectEMC Notification
Centera CLI ! !
MoPI ! ! ! ! !
SNMP
EMC Control Center ! ! ! ! !
ChargeBack Reporter
Table 20 – Performance Reporting Channels
15
As a Delete call issued by the SDK only affects the C-Clip, the bandwidth only represents the size of the CDF, not the size of the blobs
referenced by the CDF.
37
Replication Reporting
How Replication Works
$ !"#
%
&'
()
Figure 19 – Replication
The replication process is driven by a replication queue which is maintained on each node with the storage role.
The queue is filled with C-Clip ID’s as they are written to the node, by the parking retry process which retries
C-Clips every 24 hours (or on-demand), or by EMC Service personnel.
This queue is processed by the nodes with the access role on a node-level FIFO scheme, using the store
timestamp of the CDF, i.e. the timestamp which indicates when the C-Clip actually was stored on the cluster
(this could be the result of a write request, a replication from another cluster, or a restore).
Because ach C-Clip is entered on two separate replication queues (a direct result of a CDF always being
mirrored) duplicates must first be filtered out. In addition, if a file is larger than 1MB, the replication process
first does an ‘exist’ check on the replica to avoid sending large files over the wire.
There are by default 20 replication threads active on each node with the access role.
Replication Statistics
Replication performance is reported via a number of reports in the CLI. The following statistics are available
and consolidated at cluster level every minute:
• !Number of C-Clips to be replicated: shows how many unique C-Clips are currently in the queue,
waiting to be replicated.
• "Number of Blobs to be replicated: reports how many blobs or user files are represented in the C-Clips
in the replication queue. This statistic is based on the average number of user files per C-Clip of the last
1,000 C-Clips processed by replication.
• #Number of MB to be replicated: reports how many MB are represented in the C-Clips in the
replication queue, including the size of the CDF and the embedded Blobs. This statistic is based on the
average MB per C-Clip of the last 1,000 C-Clips processed by replication.
• $Number of Parked Entries: reports the number of C-Clips which did not replicate successfully and
were ‘parked’.
38
• &Replication Speed: provides an hourly average of the number of C-Clips which are successfully
replicated per second.
• 'Replication Speed: provides an hourly average of the capacity represented by C-Clips which are
successfully replicated per second, including the size of the CDF and the embedded Blobs, this includes
Blobs which were detected on the target and were not physically exported.
• %Ingest: provides an hourly average of the number of C-Clips which are added to the replication queue,
excluding C-Clips which are added by EMC Service via the service folder log.
• %Replication ETA: provides an estimate of the time it will take for the replication queue to be empty,
based on the average replication speed and ingest rate over the last 24 hours. If the ingest rate over the
past 24 hours was higher than the replication speed, a value of infinite is reported.
• )Replication Lag: provides an estimate of the time it will take for a C-Clip to be replicated if it is added
to the queue now, based on the current queue size and the average replication speed over the last 24 hours.
Restore Reporting
How Restore Is Different From Replication
A Restore process has a lot in common with Replication in that it also copies C-Clips from a source to a target
cluster, shares a lot of common software components, and also has 20 threads assigned per access node.
However, there are some key differences which have an impact on the way reporting is done.
The first major difference is the process trigger. Where replication is a continuous process, triggered by new
entries in the replication queue, restore is an on-demand process, triggered by a full or partial restore request
from the system administrator.
Another key difference is that replication processes C-Clips based on the Store-timestamp, i.e. the timestamp
which indicates when the C-Clip actually was stored on the cluster (this could be the result of a write request, a
replication from another cluster, or a restore), while a restore will process all C-Clips whose Write-timestamp
matches the restore period, irrespective of when they arrived on the Cluster. Each node processing the index
will also maintain a checkpoint which indicates the Write-timestamp of the last C-Clip which was successfully
restored.
To ensure that a restore starts restoring and protecting data as soon as possible, each storage node starts iterating
its internal indexes and pushing C-Clips of which the Write-timestamp matches the restore period to the nodes
with access roles. It does not do a full iteration to determine the queue. As a result, the size and capacity in the
restore queue is not know upfront, as is the case with replication, and only an estimate can be provided.
Finally, there is also a difference in the dependency restore has on clip-integrity and regenerations. Replication
is not impacted by these system statuses, but restore requires clip-integrity to be complete, i.e. all C-Clips are
fully protected, and no regenerations can be active. If either of these conditions is not met, restore will pause,
and whenever the conditions are met again, the restore process will auto-resume.
Restore Statistics
Restore performance is reported via a number of reports in the CLI. The following statistics are available:
• Estimated Number of C-Clips to be restored: provides an estimate of how many unique C-Clips are
currently waiting to be restored. This statistic is calculated at cluster level every 60 minutes.
• Restore Checkpoint: this reports the current checkpoint of the slowest node, i.e. all C-Clips written prior
to this date and time are successfully restored. This statistic is calculated at cluster level every 60 minutes.
• Number of Parked Entries: reports the number of C-Clips which did not restore successfully and were
‘parked’. This statistic is consolidated at cluster level every minute.
• Restore Speed: provides an hourly average of the number of C-Clips which are successfully restored per
second. This statistic is consolidated at cluster level every minute.
39
• Restore Speed: provides an hourly average of the capacity represented by C-Clips which are successfully
restored per second, including the size of the CDF and the embedded Blobs; this includes Blobs which
were detected on the target and were not physically exported. This statistic is consolidated at cluster level
every minute.
• Restore ETA: provides an estimate of the time it will take for the restore to be completed, based on the
average restore speed over the past 24 hours and the estimated size of the restore. This statistic is
calculated at cluster level every minute.
How is the Restore Queue Calculated
The basic assumption behind the calculation is that the write activity which occurred in the time period which is
being restored is distributed uniformly over time. Each hour, the next 1,000 C-Clips to be restored are queried
and based on the Write-timestamp of the first and last C-Clip, CentraStar determines the time slice these 1,000
C-Clips represent. Then this time slice is extrapolate to the entire period still to be restored which gives an
estimate of the remaining queue.
For example, if you issue a restore for all C-Clips written on January 31st 2004, the first estimate of the queue
size would be calculated as follows:
• Query the first 1,000 C-Clips written that day.
• Determine the Write-timestamp of the 1,000th C-Clip = 0:15 AM.
• 1,000 C-Clips were written in 15 minutes or .25 hours on the 31st of January 2004.
• Extrapolate the .25 hours into a full day or 24/.25 = 96 time slices of 15 minutes.
• Estimated queue size = 96 * 1,000 = 96,000 C-Clips.
So if the write activity during the restore period is evenly distributed, the estimated queue size will be 100%
accurate. If the distribution is highly variable however, the estimated queue size will vary as well.
Figure 20 – Restore Queue Calculation
40
Garbage Collection Reporting
How Garbage Collection Works
Garbage Collection (GC) is part of the Centera deletion process and has two distinct phases: Incremental GC
and Full GC. Each of these processes has a mark phase, which marks blobs for deletion, and a sweep phase
which actually releases capacity.
• Incremental GC is triggered by delete or purge operations issued by applications, and removes blobs that
were referenced by C-Clips that have been deleted or purged since the previous run. When nodes are
offline, Incremental GC continues but will not always be able to release all deleted capacity.
• Full GC continually runs as a background process on the cluster and processes any remaining blobs which
were not processed by Incremental GC.
Garbage Collection Statistics
CentraStar currently does not provide any estimates on how much capacity is waiting to be released or how
much time will be needed to process the backlog.
The following information is available via sensors in the ConnectEMC Health Report:
• Delete Throughput: provides an hourly average of the number of C-Clips which are successfully deleted
by applications per second. This statistic is consolidated at cluster level every 15 minutes.
• Delete Bandwidth: provides an hourly average of the capacity represented by the CDF of the C-Clips
which are successfully deleted per second. It includes capacity of the embedded blobs, but does not
include the size of the Blobs which were referenced by the C-Clip. This statistic is consolidated at cluster
level every 15 minutes.
• Incremental Garbage Collection Sweep Bandwidth: provides an hourly average of the capacity which
was actually freed up on disk per second by Incremental GC. This statistic is consolidated at cluster level
every 15 minutes.
• Full Garbage Collection Sweep Bandwidth: provides an hourly average of the capacity which was
actually freed up on disk per second by Full GC. This statistic is consolidated at cluster level every 15
minutes.
41
Cluster to Cluster Synchronization Reporting
When replicating or restoring pools and clusters, system administrators want to verify that all data is safely
replicated or restored and that the source and target cluster and/or pool are synchronized. There are a number of
ways this can be achieved, but the accuracy with which these will report varies.
The most efficient way to verify that two clusters, and their replicating pools, are synchronized is via virtual
pool reporting of the number of C-Clips contained in each pool. If you compare the C-Clip count in each pool
on the source and target cluster, they should provide a fairly accurate number, when you take into account the
backlog of C-Clips in the replication queue and parking lot as discussed in Replication Statistics, and the
limitations of pool capacity reporting described in Pool Capacity Reporting Limitations. Obviously, any C-Clip
which is in the replication parking lot has not been replicated successfully and the root cause for the failure
needs to be identified and resolved.
The most accurate way to tell if two clusters are in synch for a certain time period, is to launch a query on both
systems and compare, or otherwise use a tool like CSG Replication. This will give a 100% accurate report,
provided no delete activity for the reporting period is happening during the time it takes to run the reporting.
Finally, system administrators can simply check basic capacity reports and verify that the used raw capacity and
object counts on both clusters match. However, these reports do not provide the required accuracy because of a
number of reasons:
• Cluster might use a different protection scheme, threshold or fallback scheme, affecting storage efficiency.
• Cluster might use a different storage strategy or threshold affecting single instancing.
• Cluster will fall back on different protection schemes at various times.
• Garbage collection is cleaning up at different speeds.
• Regeneration occurrence and speed is totally independent.
• Extra copies because of interrupted self-healing.
• Difference in timing of single instancing.
42
Monitoring and Health Message Channels, Formats and
Mappings
ConnectEMC
ConnectEMC Overview
ConnectEMC requires an SMTP (Simple Mail Transfer Protocol) server to process the outgoing alerts and
health reports that are sent to EMC Support and to ConnectEMC recipients. This can be a customer provided
SMTP server, or it can be routed via an EMC OnAlert station acting as an SMTP server.
EMC OnAlert is an application that provides remote support functionality to networked EMC devices and
includes Automatic Error Reporting and Remote Technical Support. Refer to the EMC OnAlert Product Guide,
P/N 300-999-378 for more information on EMC OnAlert and how to install it. If an OnAlert station is used it
must run an SMTP server to accept the health reports and alert messages.
ConnectEMC also sends a daily health report via the same SMTP channel. This daily notification can be used
as a verification that the ConnectEMC channel is functioning
correctly.
Centera will send the email messages from any node with the
access role enabled via the customer network. As long as at
least 1 node with an access role is online, email messages will be
sent.
If the SMTP server is not available, all messages will be queued.
The messages on the queue are retried every 15 minutes up to
100 times. If a Cluster is not sending email home to EMC for
more then 24 hrs, a case is created in the EMC Support systems
to check what is going on.
Note that it is possible to configure a second SMTP server.
Figure 21 – OnAlert Configuration
43
ConnectEMC Formats
ConnectEMC uses an XML format for sending alert messages to EMC. The XML messages are encrypted and
uu-encoded using FIPs 140 compliant encryption standards with AES (Advanced Encryption Standard) 256-bit
strong encryption16 to meet the US Department of Defense standards for security. Once received at EMC,
messages are parsed and inserted in EMC’s call tracking infrastructure.
Version Alert Format Health Format
2.1 - -
2.2 - -
2.3 alert_email-1.0.dtd health-1.0.dtd
2.3 SP2/SP3 alert_email-1.0.dtd health-1.1.dtd
Table 21 – ConnectEMC Formats
For more information on the formats please refer to Appendix A – ConnectEMC Formats.
16
In situations where 256-bit EAS is not allowed, a 56-bit DES version is available.
44
ConnectEMC Alert Mapping
Message Item Alert Type A Alert Type B17 Alert Type C18
alertIdentification
Type SW HW HW
symptomCode See Table 3 See Table 3 See Table 3
formatVersion DTD version number DTD version number DTD version number
formatDTD DTD name DTD name DTD name
creationDateTime Time of message creation Time of message creation Time of message creation
clusterIdentification
Name Cluster name Cluster name Cluster name
serialNumber Cluster serial number Cluster serial number Cluster serial number
revisionNumber Empty Empty Empty
ClusterID Cluster ID Cluster ID Cluster ID
clusterDomain Cluster domain Cluster domain Cluster domain
activeSoftwareVersion Cluster software version Cluster software version Cluster software version
alertDescription
alertLevel NOTIFICATION| NOTIFICATION| NOTIFICATION|
WARNING|ERROR| WARNING|ERROR| WARNING|ERROR|
CRITICAL CRITICAL CRITICAL
alertDescription SensorID:NodeName: SensorID:NicName | ‘disk’ | ‘node’ |
NodeID:SensorValue SensorID:SwitchName ‘rootswitch’
Component ID of Node of sensor ‘node’ | ‘cabinet’ ‘node’ | ‘cabinet’ |
reporting ‘rootswitch’
componentId Name of node of sensor NodeID | CabinetID NodeName | CabinetID |
reporting RootSwitchName
subcomponent ‘sensor’ ‘NIC’ | ‘switch’ ‘disk’ | ‘node’ | <empty>
subComponentId SensorID NicName | SwitchName DiskName | NodeName
| <empty>
Table 22 – ConnectEMC Alert Mapping
17
B type alerts are triggered for External NIC Failure, Internal NIC Failure and Cube Switch Failure.
18
C Type Alerts are triggered for Offline Disk, Offline Node and Root Switch Failure.
45
ConnectEMC Notification
ConnectEMC Notification Formats

Since CentraStar v2.4, each alert message sent to EMC is re-formatted from XML to HMTL and sent to each
email address on the ConnectEMC recipients list. The addresses on that list and the from/reply address on these
emails can be set in the CLI.
CentraStar Version Alert Format Health Format
2.1 - -
2.2 - -
2.3 - -
2.4 Version 1.1 Version 1.2
Table 23 – ConnectEMC Notification Formats
All notification emails have the following subject “ConnectEMC notification from <serial number>”, where the
serial number which has been configured at installation time by EMC Service on the reporting cluster.
For more information on the formats please refer to Appendix C – ConnectEMC Sample Health Notification.
SNMP
SNMP Overview
The Simple Network Management Protocol (SNMP) is an Internet-standard protocol for managing devices on
IP networks. The Centera SNMP Agent runs on all nodes with the access role19 and proactively sends
messages called SNMP Traps to a network management station.
The SNMP implementation for Centera supports SNMP v2.0 and uses a proprietary EMC MIB20 within the
enterprises.emc.centera branch. The prefix in use for Centera in the SNMP OID tree is
enterprises.emc.centera.1.3.6.1.4.1.1139.5.
There are two traps defined for Centera:
• the heartbeat trap (enterprises.emc.centera.notificationTrap.trapNotification) conveys the general health of
Centera;
• the alarm trap (enterprises.emc.centera.notificationTrap.trapAlarmNotification) ) identifies an event on the
Centera cluster and is used to indicate a warning or a critical system failure.
19
At any point in time, only one node with the access role actually sends messages – this is called the principal node. The principal role
is automatically taken over by another node in case of node failure.
20
The Management Information Base (MIB), is a database of OIDs. An Object Identifier (OID) is an SNMP attribute with a given type
and value. An OID uniquely defines the attribute name which is both a unique name in the MIB and a unique dotted decimal string.
Each number in the dotted decimal string represents a unique attribute in the MIB which structures OIDs. The MIB is located on the
Customer Tools CD and is installed in C:\Program Files\EMC\Centera\2_4\SystemOperator\lib\centera.mib.
46
Centera SNMP Functionality
SNMP Heartbeat trap
The Heartbeat trap is sent on a regular basis. The interval is configurable via the CLI. The primary function of
the Heartbeat trap is to report the overall state of the system. The cause of a reported problem is not part of the
message. Separate alarm traps will be sent with more details about the actual problem.
The Heartbeat trap is called a trapNotification and it
can send three types of messages to the network
management station, each describing a different level
in problem severity:
• Informational: the Centera cluster is healthy,
• Warning: a non-critical error has occurred, and
• Severe: the Centera cluster is in a critical state.

Use the Heartbeat trap to quickly assess if there is a
problem. The absence of the Heartbeat signal can also
be an important indicator of a problem.
Figure 22 – SNMP Heartbeat Trap
SNMP Alarm trap
The Alarm trap is sent when a problem occurs on the Centera cluster. Unlike the Heartbeat trap, this trap will
only be sent when a problem occurs. There are two error levels in this trap:
• Informational: used for notification of low-
impact events.
• Warning: a non-critical error has occurred;
• Severe: the Centera cluster is in a critical state.
This trap is called the trapAlarmNotification. The
severity level reported by the Alarm trap equals the
most severe failure reported. The description field of
the Alarm trap contains a concise message stating the
nature of the problem. The Alarm trap allows the
network management station to perform a specific
action in response.
Figure 23 – SNMP Alarm Trap
For more information on the format of the SNMP trap format please refer to Appendix D – SNMP Formats.
47
SNMP Mapping
Item Alert Type A Alert Type B21 Alert Type C22
notifyTimestamp Time the trap was sent Time the trap was sent Time the trap was sent
notifyServer Cluster ID of the Centera Cluster ID of the Centera Cluster ID of the Centera
notifyDescription sensor_id:system_name: ‘NIC’ | ‘switch’ ‘disk’ | ‘node’ |
system_id:value ‘rootswitch’
notifySeverity Sensor Notification => Informational (1) | Informational (1) |
SNMP informational (1) warning (2) | severe (3) warning (2) | severe (3)
Sensor Warning
=> SNMP warning (2)
Sensor Error or Critical
=> SNMP severe (3)
Table 24 – SNMP Mapping
SNMP Limitations
SNMP also defines two commands to retrieve information from an SNMP enabled device, but Centera does not
functionally support them:
• the GET command queries the value of a single managed object; the SNMP agent on a Centera cluster
answers with a “no such OID” message to a GET command.
• the GETNEXT command implements a tree walk over a sub tree of managed objects; Centera responds
with an “End of MIB” message to a GETNEXT command.
Centera does not support SET commands; all configuration and active management must be done via the CLI.
The Centera SNMP implementation uses a single trap for all events, it does not have a separate OIB for each
symptom code and does not send resolution traps.
21
22
48
Monitoring API (MoPI)
MoPI Overview
Applications have the possibility to receive alerts by implementing the FPEventCallback_RegisterForAllEvents call and
get health reports via the FPMonitor_GetDiscovery call. Via these calls, an XML file with the relevant content is
made available to the application.
To function properly, applications must use an access profile with the Monitoring capability enabled. Please
refer to the Centera API Reference Guide, P/N 069001185 for more information on the MoPI interface.
MoPI Formats
CentraStar Version Alert Format Health Format
2.1 alert-1.0.dtd discovery-1.1.dtd
2.3 SP2/3 alert-2.0.dtd health-1.1.dtd
2.4 alert-2.0.dtd health-1.2.dtd
Table 25 – MoPI Formats
For more information on the formats please refer to Appendix E – MoPI Formats.
49
MoPI Mapping
Item Alert Type A Alert Type B23 Alert Type C24 SDK Alert25
Alert
Type degradation | degradation | degradation | degradation |
improvement improvement improvement improvement
Severity notification | warning | notification | warning | notification | warning | n/a
error | critical error | critical error | critical
Node
Systemid NodeID of the node NodeID27 | CabinetID NodeID | NodeID | ID NodeID
reporting the value26 of the node reporting
the value
Systemname Name of node of NodeName | NodeName | n/a
sensor reporting CabinetName NodeName | Name of
the node reporting the
value
Device
Type ‘sensor’ ‘NIC’ | ‘switch’ ‘disk’ | ‘node’ | ‘SDK’
‘rootswitch’
Name ‘node’ NICName | DiskName | ‘-10101 Connect
SwitchName NodeName | failed for <ip
RootSwitchName address>’
Value SensorID: <empty> <empty> n/a
NodeName:NodeID:S
ensorValue
Table 26 – MoPI Mapping
23
24
25
Only available in MoPI 1.0 format.
26
For Type A alerts, this is not the node where the problem occurred – all alerts are triggered by cluster level sensors which filter and
aggregate node level sensor alerts. For instance, when Node A triggers a node level alert for a full /var partition, this alert is processed
by a cluster level sensor which runs on the principal Node B, which then issues an alert. The MoPI message will contain the NodeID of
Node B.
27
For Type B and C alerts, this is the node where the problem occurred.
50
EMC ControlCenter
ECC Overview
The Storage Agent for Centera, part of the EMC ControlCenter 5.2 release, communicates with the EMC
Centera clusters in your environment via the Centera MoPI interface and allows you to view detailed
information about the clusters, including nodes, services, and application profiles.
The agent also monitors the cluster, receiving Centera alerts that notify you of cluster errors or when a specified
threshold is reached. When an alert is triggered, right-click it and select Alerts,Help for information about the
alert and how to respond.
ECC Storage Agent For Centera Installation & Configuration

The EMC ControlCenter Storage Agent for Centera is available free of charge and is qualified to run on the
EMC ControlCenter server. To enable the storage agent you need a license card with Part Number 100S-
000549, which ships with base cabinet CNRRK8NG3 and can be ordered separately using CNR8NECCLIC.
NOTE: For the agent to function properly, the access profile used by the agent must have the monitoring capability
enabled. Please refer to the Centera Online Help, P/N 300-002-547 for information on how to define an access profile.
NOTE: ECC 5.2 does not support health formats for CentraStar versions 2.3 SP1 and higher. EMC Service can downgrade
2.3 SPx/2.4 and 3.0 clusters to the discovery-1.2.dtd format to allow for backwards compatibility with ECC 5.2. This is
done using CV: Tools → Service → MoPI → Install 2.3 Template.
Discovery data collection policy
The Discovery data collection policy collects general information about EMC Centera clusters. After you install
and run the Storage Agent for Centera, and have discovered the objects to monitor, the policy automatically
runs and collects information about the systems once every 24 hours, by default. To change how often the
collection policy runs or at what time of day, change the schedule attached to the policy.
Configuring the Storage Agent for Centera
After installing and starting the agent, you must use the Discover Other Objects dialog box to discover the
Centera clusters you want to monitor.
• In the Console, click the Monitor task. The menu bar changes.
• Click Discover and select Other.
• The Discover Other Objects dialog box appears.
• Provide the authentication information for the cluster(s) you want to monitor.
For more information on how to configure the EMC ControlCenter Storage Agent for Centera, please refer to
the ControlCenter Planning and Installation Guide Volume 1, P/N 300-000-684.
51
ECC Mapping
ECC uses the MoPI interface for monitoring and reporting on Centera.
Monitoring
ECC\MoPI alert-1.0.dtd alert-2.0.dtd alert-2.1.dtd
CentraStar 2.2 2.3 & 2.4 3.0 & 3.1
5.2 !
5.2 SP3 ! ! !
Table 27 – Supported MoPI Alert Formats
From the MoPI alerts, ECC distinguishes 8 different ECC Alert messages:
• a generic Alert for all Type A alerts,
• 6 Type B and Type C alerts are treated as individual alert messages, and
• the SDK alert.
ECC Alert Mapping with MoPI 1.0

Item Alert Type A Alert Type B28 and C29 and SDK
Name n/a <device type>
Severity n/a <severity>
Message n/a Component <device type> - <device name> on node <node systemid>
Table 28 – ECC Mapping with MoPI 1.0
ECC Alert Mapping with MoPI 2.0

Item Alert Type A Alert Type B30 and C31 and SDK
Name <device type> <device type>
Severity <severity> <severity>
Message Component sensor - <device name> with value Component <device type> - <device name> on
<device value> on node <node systemname> node <node systemname> (<node systemid>)
(<node systemid>)
Table 29 – ECC Mapping with MoPI 2.0
28
29
30
31
52
Health Reporting
ECC\MoPI discovery- health-1.1.dtd health-1.2.dtd health-1.3.dtd health-1.4.dtd
1.2.dtd
CentraStar 2.2 & 2.3 2.3 SP2/3 2.4 3.0 3.1
5.2 !
5.2 SP3 ! ! ! !
Table 30 – Supported MoPI Health Formats
If the ECC version does not support the format of the CentraStar cluster in your environment, EMC Service can
revert the format to an older version to ensure compatibility.
Monitoring and Reporting Tools

Centera offers 2 tools to monitor your Centera infrastructure and to be able to report configuration settings,
general health and status of the various components, and capacity and performance information.
• Centera Command Line Interface (CLI).
• EMC ControlCenter 5.2.
NOTE: Although capacity information in the CLI and ECC comes from the same source, customers will see small
differences in the capacity numbers reported in both tools. The reason for this can be 2-fold:
Timing: the Agent polls Centera every 24 hours by default, whereas the CLI is real-time;
Notation: the CLI and CV prior to CentraStar 3.1 use 1,000 notation for calculating GB, while ECC uses 1,024 notation.
This discrepancy causes ECC information to be 7% lower than CLI when using GB.
Centera Command Line Interface (CLI)

The CLI is a Command Line Interface for system operators to manage and monitor a Centera. For more
information on how to install and use the CLI, please refer to Centera Online Help, P/N 300-002-547, and the
Centera Quick Start Guide, P/N 300-002-546.
53
EMC ControlCenter 5.2
ECC Console
The EMC ControlCenter Agent for Centera has been available since the introduction of ECC 5.2 and supports
clusters running CentraStar v2.2 and higher. The ECC Console reports the following information on Centera:
Function 5.2 5.2 SP3
Services ! !
Licenses ! !
Access Nodes ! !
Storage Nodes !
Cluster Settings & Capacity ! !
Pool Capacity !
Profiles & Capabilities ! !
Table 31 – ECC Functionality Overview
Pool Capacity
Although pools are not supported until CentraStar 3.0, ECC has had a notion of pools since the first version.
CentraStar versions that do not support pools report the cluster capacity for the default or cluster pool and use
the total capacity as quota. From CentraStar 3.0 onwards, the actual pool capacity and quotas are reported:
CentraStar ECC 5.2 5.2 SP3
2.2 Cluster capacity in default pool Cluster capacity in default pool
2.3 Cluster capacity in default pool Cluster capacity in default pool
2.4 Same as 2.3 Cluster capacity in cluster pool
3.0 Same as 2.3 Pool capacity for all pools
3.1 Same as 2.3 Same as 3.0
Table 32 – ECC Pool Capacity Reporting
Profiles & Capabilities
For CentraStar versions that do not support pools, the access rights are shown as granted to the default or cluster
pool. As of CentraStar 3.0, the actual pool and profile access rights are reported:
CentraStar ECC 5.2 5.2 SP3
2.2 Rights to Default pool Rights to Default pool
2.3 Rights to Default pool Rights to Default pool
2.4 Same as 2.3 Rights to Cluster pool
3.0 Same as 2.3 Rights to Home Pool only
3.1 Same as 2.3 Same as 3.0
Table 33 – ECC Profile Capabilities
54
Figure 24 – ECC 5.2 SP3 Cluster, Pool and Profiles
55
ECC StorageScope
With the introduction of ECC 5.2 SP3, the StorageScope module in ECC reports Centera capacity information
for clusters being monitored by ECC. ECC customers now have the ability to:
• Include Centera Clusters and Pools32 in StorageScope groups;
• Run standard point-in-time reports;
• Define and run customizable historic trending reports.
Include Centera Clusters and Pools in StorageScope groups
StorageScope allows the creation of

StorageScope groups that contain
Clusters and/or Pools.
Groups are defined in the ECC Console
by a simple right click in the navigation
tree.
Cluster and pools can be added by
copying and pasting or dragging and
dropping them from the Storage
Systems folder (where Centera Clusters
and Folders are located) into the Group
you just created.
By adding a Cluster or a Pool to a
group, the StorageScope user can then
include Centera capacity in the
Figure 25 – Create a New Group in ECC Console
following standard reports:
All
This will include all CAS Clusters and
Centera Pools in the user’s
infrastructure. Product column will be
populated as ‘CAS Cluster’ or ‘Centera
Pool’.
Group List
Centera Pools that belong to user
defined groups. Product column will be
populated as ‘CAS Cluster’ or ‘Centera
Pool’.
Figure 26 – ECC All History
32
Only for clusters running CentraStar 3.0.
56
This will include CAS related reports
that have been run.
All in Multiple Groups
This will include all Centera Clusters
and Centera Pools that belong to more
than one user-defined group. Product
column will be populated as ‘Centera
Cluster’ or ‘Centera Pool’.
All not in a Group
Centera Pools that do not belong to any
user-defined group.
Figure 27 – ECC History
Component
This will include Centera agents
installed in the user’s infrastructure.
Component Type column will be
populated as ‘Centera’. Agent
Description will be populated as
‘Storage Agent for Centera’.
Figure 28 – ECC Components
57
Run standard point-in-time reports
ECC 5.2 SP3 also provides a number of Centera specific reports. These can be found under the CAS Clusters
section:
Figure 29 – CAS Report Overview Figure 30 – CAS Clusters by Group
Figure 31 – All CAS Clusters – Utilization Figure 32 – All CAS Clusters – Configuration
Figure 33 – CAS Cluster Group List Figure 34 – CAS Storage Nodes
58
Figure 35 – CAS Pools by Group Figure 36 – All CAS Pools
Figure 37 – CAS Pools Group List Figure 38 – CAS Cluster Configuration – General
Figure 39 – CAS Cluster Configuration - Groups Figure 40 – CAS Cluster Configuration – Pools
59
Define and run customizable historic trending reports
Users can also define customer reports for historic capacity utilization and trending. Please use the
StorageScope Online Help for information on how to create such reports.
Figure 41 – Sample Historic Trend Report Figure 42 – STS Online Help – How to Create
Custom Reports
Figure 44 – Create a New Custom Report – Select

Figure 43 – Create a New Custom Report Fields to Report
Figure 45 – Create a New Custom Report – Define Figure 46 – Create a New Custom Report –
Filters Determine Period and Trend
60
Appendix A – ConnectEMC Formats.
ConnectEMC alert_email-1.0.dtd
<!ELEMENT alert (alertIdentification, clusterIdentification, alertDescription)>
<!ELEMENT alertIdentification EMPTY>
<!ATTLIST alertIdentification
type CDATA #REQUIRED
symptomCode CDATA #REQUIRED
formatVersion CDATA #REQUIRED
formatDTD CDATA #REQUIRED
creationDateTime CDATA #REQUIRED>
<!ELEMENT clusterIdentification EMPTY>
<!ATTLIST clusterIdentification
name CDATA #REQUIRED
serialNumber CDATA #REQUIRED
revisionNumber CDATA #REQUIRED
clusterID CDATA #REQUIRED
clusterDomain CDATA #REQUIRED>
<!ELEMENT alertDescription EMPTY>
<!ATTLIST alertDescription
alertLevel CDATA #REQUIRED
alertDescription CDATA #REQUIRED
component CDATA #REQUIRED
componentId CDATA #REQUIRED
subComponent CDATA #REQUIRED
subComponentId CDATA #REQUIRED>
Figure 47 – ConnectEMC alert_email-1.0.dtd
ConnectEMC alert_email-1.1.dtd 33
<!ELEMENT alert (alertIdentification, clusterIdentification, alertDescription)>
<!ELEMENT alertIdentification EMPTY>
<!ATTLIST alertIdentification
symptomCode CDATA #REQUIRED
clusterDomain CDATA #REQUIRED
activeSoftwareVersion CDATA #REQUIRED>
<!ELEMENT alertDescription EMPTY>
<!ATTLIST alertDescription
alertLevel CDATA #REQUIRED
33
A bold element or attribute indicates a change compared with the previous version.
i
alertDescription CDATA #REQUIRED
component CDATA #REQUIRED
componentId CDATA #REQUIRED
subComponent CDATA #REQUIRED
subComponentId CDATA #REQUIRED>
Figure 48 – ConnectEMC alert_email-1.1.dtd
ConnectEMC health_email-1.0.dtd
<!ELEMENT health (reportIdentification,cluster)>
<!ELEMENT reportIdentification EMPTY>
<!ATTLIST reportIdentification
sequenceNumber CDATA #REQUIRED
reportInterval CDATA #REQUIRED
<!ELEMENT cluster
(clusterIdentification,licenseList,clusterServiceSettings,hardwareComponents,clu
sterPoolList,clusterCapacity,sensors)>
clusterDomain CDATA #REQUIRED>
<!ELEMENT licenseList (license*)>
<!ELEMENT license EMPTY>
<!ATTLIST license
licenseKey CDATA #REQUIRED
name CDATA #REQUIRED>
<!ELEMENT clusterServiceSettings EMPTY>
<!ATTLIST clusterServiceSettings
siteID CDATA #REQUIRED
customerName CDATA #REQUIRED
location CDATA #REQUIRED
adminTimeZone CDATA #REQUIRED
adminContact CDATA #REQUIRED
adminEmail CDATA #REQUIRED
adminPhone CDATA #REQUIRED
modemPhone1 CDATA #REQUIRED
lastServiceVisitDateTime CDATA #REQUIRED
lastServiceVisitType CDATA #REQUIRED
lastServiceVisitReason CDATA #REQUIRED
serviceModel CDATA #REQUIRED
activeSoftwareVersion CDATA #REQUIRED
activationManager CDATA #REQUIRED
distributionManager CDATA #REQUIRED
regenerationBuffer CDATA #REQUIRED
regenerationBufferPolicy CDATA #REQUIRED
complianceMode CDATA #REQUIRED
defaultRetentionPeriod CDATA #REQUIRED
ii
replicationStatus CDATA #REQUIRED
replicationIP CDATA #REQUIRED
snmpStatus CDATA #REQUIRED
snmpCommunity CDATA #REQUIRED
snmpIP CDATA #REQUIRED
snmpPort CDATA #REQUIRED
snmpTrapInterval CDATA #REQUIRED
notification CDATA #REQUIRED
protectionScheme CDATA #REQUIRED
protectionThreshold CDATA #REQUIRED
protectionSchemeAlternate CDATA #REQUIRED
namingScheme CDATA #REQUIRED
namingSchemeMode CDATA #REQUIRED
namingSchemeThreshold CDATA #REQUIRED
garbageCollectionStatus CDATA #REQUIRED
shreddingStatus CDATA #REQUIRED>
<!ELEMENT hardwareComponents (cabinetList)>
<!ELEMENT cabinetList (cabinet*)>
<!ATTLIST cabinetList
count CDATA #REQUIRED>
<!ELEMENT cabinet
(cabinetIdentification,switchList,atsList,modemList,nodeList,cabinetCapacity)>
<!ELEMENT cabinetIdentification EMPTY>
<!ATTLIST cabinetIdentification
revisionNumber CDATA #REQUIRED>
<!ELEMENT switchList (switch*)>
<!ATTLIST switchList
count CDATA #REQUIRED
clusterPrefix CDATA #REQUIRED>
<!ELEMENT switch (switchIdentification,portList)>
<!ELEMENT switchIdentification EMPTY>
<!ATTLIST switchIdentification
macAddress CDATA #REQUIRED
firmWare CDATA #REQUIRED
rail CDATA #REQUIRED
description CDATA #REQUIRED
ipAddress CDATA #REQUIRED>
<!ELEMENT portList (port*)>
<!ATTLIST portList
<!ELEMENT port (portIdentification)>
<!ELEMENT portIdentification EMPTY>
<!ATTLIST portIdentification
portNumber CDATA #REQUIRED
speedSetting CDATA #REQUIRED
status CDATA #REQUIRED>
<!ELEMENT atsList (ats*)>
iii
<!ATTLIST atsList
<!ELEMENT ats (atsIdentification,atsHealth)>
<!ELEMENT atsIdentification EMPTY>
<!ATTLIST atsIdentification
redundantPower CDATA #REQUIRED
powerSource CDATA #REQUIRED>
<!ELEMENT atsHealth EMPTY>
<!ATTLIST atsHealth
<!ELEMENT modemList (modem*)>
<!ATTLIST modemList
<!ELEMENT modem (modemIdentification)>
<!ELEMENT modemIdentification EMPTY>
<!ATTLIST modemIdentification
revisionNumber CDATA #REQUIRED>
<!ELEMENT nodeList (node*)>
<!ATTLIST nodeList
<!ELEMENT node
(nodeIdentification,nodeHealth,nodeServiceSettings,cpuList,powerSupplyList,nicLi
st,driveList,nodeCapacity)>
<!ELEMENT nodeIdentification EMPTY>
<!ATTLIST nodeIdentification
model CDATA #REQUIRED
motherboard CDATA #REQUIRED
rail CDATA #REQUIRED
role CDATA #REQUIRED
systemID CDATA #REQUIRED>
<!ELEMENT nodeHealth EMPTY>
<!ATTLIST nodeHealth
maintenanceMode CDATA #REQUIRED
<!ELEMENT nodeServiceSettings EMPTY>
<!ATTLIST nodeServiceSettings
hotFixes CDATA #REQUIRED
platformHotFixes CDATA #REQUIRED
platformVersion CDATA #REQUIRED
softwareVersion CDATA #REQUIRED>
<!ELEMENT cpuList (cpu*)>
<!ATTLIST cpuList
<!ELEMENT cpu (cpuIdentification,cpuHealth)>
<!ELEMENT cpuIdentification EMPTY>
iv
<!ATTLIST cpuIdentification
serialNumber CDATA #REQUIRED>
<!ELEMENT cpuHealth EMPTY>
<!ATTLIST cpuHealth
caseTemperature CDATA #REQUIRED
cpuTemperature CDATA #REQUIRED>
<!ELEMENT powerSupplyList (powerSupply*)>
<!ATTLIST powerSupplyList
<!ELEMENT powerSupply (powerSupplyIdentification)>
<!ELEMENT powerSupplyIdentification EMPTY>
<!ATTLIST powerSupplyIdentification
<!ELEMENT nicList (nic*)>
<!ATTLIST nicList
<!ELEMENT nic (nicIdentification,nicHealth)>
<!ELEMENT nicIdentification EMPTY>
<!ATTLIST nicIdentification
config CDATA #REQUIRED
dnsIPAddress CDATA #REQUIRED
duplex CDATA #REQUIRED
ipAddress CDATA #REQUIRED
linkSpeed CDATA #REQUIRED
macAddress CDATA #REQUIRED
subNet CDATA #REQUIRED
type CDATA #REQUIRED>
<!ELEMENT nicHealth EMPTY>
<!ATTLIST nicHealth
<!ELEMENT driveList (drive*)>
<!ATTLIST driveList
<!ELEMENT drive (driveIdentification,driveHealth,driveCapacity)>
<!ELEMENT driveIdentification EMPTY>
<!ATTLIST driveIdentification
index CDATA #REQUIRED
model CDATA #REQUIRED
<!ELEMENT driveHealth EMPTY>
<!ATTLIST driveHealth
<!ELEMENT driveCapacity EMPTY>

<!ATTLIST driveCapacity
auditMeta data CDATA #REQUIRED
freeCapacity CDATA #REQUIRED
v
protectedUserData CDATA #REQUIRED
systemResources CDATA #REQUIRED
totalCapacity CDATA #REQUIRED
usedCapacity CDATA #REQUIRED
usedObjectCount CDATA #REQUIRED>
<!ELEMENT nodeCapacity EMPTY>
<!ATTLIST nodeCapacity
offlineCapacity CDATA #REQUIRED
<!ELEMENT cabinetCapacity EMPTY>
<!ATTLIST cabinetCapacity
spareCapacity CDATA #REQUIRED
<!ELEMENT clusterPoolList (clusterPool*)>
<!ATTLIST clusterPoolList
<!ELEMENT clusterPool (poolIdentification,profileList,poolCapacity)>
<!ELEMENT poolIdentification EMPTY>
<!ATTLIST poolIdentification
<!ELEMENT profileList (profile*)>
<!ATTLIST profileList
<!ELEMENT profile (profileIdentification,capabilityList)>
<!ELEMENT profileIdentification EMPTY>
<!ATTLIST profileIdentification
enabled CDATA #REQUIRED>
<!ELEMENT capabilityList (capability*)>
<!ATTLIST capabilityList
<!ELEMENT capability EMPTY>
<!ATTLIST capability
enabled CDATA #REQUIRED>
<!ELEMENT poolCapacity EMPTY>
<!ATTLIST poolCapacity
vi
<!ELEMENT clusterCapacity EMPTY>
<!ATTLIST clusterCapacity
<!ELEMENT sensors (clusterSensorList,nodeSensorList)>
<!ELEMENT clusterSensorList (sensor*)>
<!ATTLIST clusterSensorList
<!ELEMENT nodeSensorList (sensor*)>
<!ATTLIST nodeSensorList
<!ELEMENT sensor EMPTY>
<!ATTLIST sensor
id CDATA #REQUIRED
value CDATA #REQUIRED
status CDATA #REQUIRED
previousStatus CDATA #REQUIRED
timeStamp CDATA #REQUIRED
symptomCode CDATA #IMPLIED>
Figure 49 – ConnectEMC health_email-1.0.dtd
ConnectEMC health_email-1.1.dtd34
adminTimeZone CDATA #REQUIRED
lastServiceVisitDateTime CDATA #REQUIRED
lastServiceVisitType CDATA #REQUIRED
lastServiceVisitReason CDATA #REQUIRED
34
Only the differences with the previous version are shown and indicated in bold.
vii
serviceModel CDATA #REQUIRED
replicationOfDeletes CDATA #REQUIRED
replicationProfile CDATA #REQUIRED
<!ELEMENT driveCapacity EMPTY>
<!ATTLIST driveCapacity
usedObjectCount CDATA #REQUIRED
availableCapacity CDATA #REQUIRED
systemBuffer CDATA #REQUIRED>
freeObjectCount CDATA #REQUIRED
systemBuffer CDATA #REQUIRED
totalObjectCount CDATA #REQUIRED>
<!ELEMENT cabinetCapacity EMPTY>
<!ATTLIST cabinetCapacity
viii
availableCapacityUntilAlert CDATA #REQUIRED
totalObjectCount CDATA #REQUIRED
freeObjectCount CDATA #REQUIRED>
<!ELEMENT clusterCapacity EMPTY>
<!ATTLIST clusterCapacity
availableCapacityUntilAlert CDATA #REQUIRED
totalObjectCount CDATA #REQUIRED
freeObjectCount CDATA #REQUIRED>
<!ELEMENT cluster
(clusterIdentification,licenseList,clusterServiceSettings,hardwareComponents,poo
lList,profileList,clusterPoolList,clusterCapacity,sensors)>
35
ix
replicationOfDeletes CDATA #REQUIRED
replicationProfile CDATA #REQUIRED
replicationOfIncomingReplications CDATA #REQUIRED
replicationPools CDATA #REQUIRED
<!ELEMENT clusterPoolList (clusterPool*)>
<!ATTLIST clusterPoolList
<!ELEMENT poolList (pool*)>
<!ATTLIST poolList
<!ELEMENT clusterPool (poolIdentification,profileList,poolCapacity)>
<!ELEMENT Pool
(poolIdentification,profileMappingList,profileRefList,poolCapacity)>
<!ELEMENT poolIdentification EMPTY>
<!ATTLIST poolIdentification
id CDATA #REQUIRED>
<!ELEMENT profileMappingList (profileMapping*)>
<!ATTLIST profileMappingList
<!ELEMENT profileMapping EMPTY>
<!ATTLIST profileMapping
x
<!ELEMENT profileRefList (profileRef*)>
<!ATTLIST profileRefList
<!ELEMENT profileRef (capabilityList)>
<!ATTLIST profileRef
<!ELEMENT profileIdentification EMPTY>
<!ATTLIST profileIdentification
enabled CDATA #REQUIRED
homePool CDATA #REQUIRED>
usedClipCount CDATA #REQUIRED
quota CDATA #REQUIRED>
<!ELEMENT cluster
(clusterIdentification,licenseList,clusterServiceSettings,connectEMCSettings,pro
tectionSchemeList,hardwareComponents,poolList,profileList,registeredApplicationL
ist,clusterCapacity,sensors)>
siteID
customerName
location
adminContact
adminEmail
adminPhone
modemPhone1
modemPhone2
activeSoftwareVersion
activationManager
distributionManager
regenerationBuffer
regenerationBufferPolicy
36
37
xi
complianceMode
defaultRetentionPeriod
replicationStatus
replicationOfDeletes
replicationProfile
replicationIP
replicationOfIncomingReplications
replicationPools
snmpStatus
snmpCommunity
snmpIP
snmpPort
snmpTrapInterval
notification
protectionScheme
protectionThreshold
protectionSchemeAlternate
namingScheme
namingSchemeMode
namingSchemeThreshold
garbageCollectionStatus
garbageCollectionStatusIncr
garbageCollectionStatusFull
garbageCollectionIncrCheckPoint
shreddingStatus
regenerationTimeout
downDelay
locationStrategy
rangesImplementation
advancedRetentionManagement
restrictedManagementIP>
<!ELEMENT connectEMCSettings EMPTY>

<!ATTLIST connectEMCSettings
status
lastReportSent
lastReportNumber
replyAddress
fromAddress
emailEMC
encryptEmailEMC
interval
recipients
lastGenerationStatus
smtpServers>
<!ELEMENT protectionSchemeList (protectionScheme*)>

<!ATTLIST protectionSchemeList
count>
<!ELEMENT protectionScheme EMPTY>

<!ATTLIST protectionScheme
name
xii
writeSize
dataSize
storageSize
fallback>
<!ELEMENT nodeHealth EMPTY>

<!ATTLIST nodeHealth
complianceMode
maintenanceMode
status
upTime
locked>

auditMetaData
freeCapacity
offlineCapacity
spareCapacity
protectedUserData
systemResources
totalCapacity
usedCapacity
usedObjectCount
availableCapacity
freeObjectCount
systemBuffer
totalObjectCount>
<!ELEMENT pool (poolIdentification, profileMappingList, profileRefList,

poolCapacity, poolRetention)>
<!ELEMENT poolRetention EMPTY>

<!ATTLIST poolRetention
default
enforceRetention
fixedMinimum
fixedMaximum
variableMinimum
variableMaximum>
<!ELEMENT registeredApplicationList (application*)>

<!ATTLIST registeredApplicationList
count>
<!ELEMENT application EMPTY>

<!ATTLIST application
applicationName
applicationVersion
profile
os
sdkVersion
hostname
dateOfFirstAuthentication
xiii
dateOfLastAuthentication
numberOfSuccessfulAuthentications>
xiv
Appendix B – ConnectEMC Alert Notification
Centera Alert Report
Version 1.1
Alert Identification
Type: Hardware
Symptom Code: 3.1.3.1.01.01
Format Version: 1.1
Format DTD: alert_email-1.1.dtd
Creation Date and Time: 02-01-2005 07:54:02.562 UTC
Cluster Identification
Name: cluster138
Serial Number: APM25431700200
Revision Number: no data available
Cluster ID: c3cd501c-1xx1-11b2-b513-d826a75388b0
Cluster Domain: company.com
Alert Description
Alert Level: WARNING
Alert Description:
$Hardware.NIC.Main.InternalNICFailureCount@$CLUSTER:eth0
Component: node
Component ID: c001n05
Sub-Component: nic
Sub-Component ID: eth0
Figure 54 – ConnectEMC Alert Notification
xv
Appendix C – ConnectEMC Sample Health Notification
Centera Health Report
Sequence Number # 455 Report Format Version 1.4
Report Identification
Type: health
Format DTD: health-1.4.dtd
Creation Date and Time: 01-27-2006 05:02:42.995 UTC
ConnectEMC Settings
Status: Enabled
Report Interval: 4 hrs 0 min 0 sec
Recipients: bdgcua@bebr3fp2.corp.emc.com
Reply Address: bdgcua@bebr3fp2.corp.emc.com
From Address:
SMTP Servers: 10.68.5.5, not configured
Email EMC:
Email EMC Encrypted:
Cluster Identification
Name: cube202
Serial Number: cube202
ClusterID: 2935708c-1dd2-11b2-a171-b2dde14cfc5f
ClusterDomain: localdomain
Cluster Service Settings
Site ID:
xvi
Location: not configured
Admin Contact: not configured
Admin Email: not configured
Admin Phone: not configured
Active Software Version: 3.1.0-934-650-12114
Activation Manager: 20d4cd20-1dd2-11b2-b7ed-8deef5e20500
Distribution Manager: 20d4cd20-1dd2-11b2-b7ed-8deef5e20500
Regeneration Buffer: 1 disk(s)
Regeneration Buffer Policy: HardStop
Compliance Mode: Basic
Restricted Management IP:
Advanced Retention Management: No
Replication Status: Disabled
ReplicationOfDeletes: Disabled
Replication Profile: Anonymous
Replication IP:
SNMP Status: Disabled
SNMP Community: public
SNMP IP Address:
SNMP Port: 162
SNMP Trap Interval: 60
Protection Scheme(s): R61,M2
Protection Threshold(s): 122880,0
Naming Scheme: Performance
Naming Scheme Mode: Full
Naming Scheme Threshold: 262144
xvii
Garbage Collection Status (Full): Enabled
Garbage Collection Status (Incr): Enabled
Shredding Status: Disabled
Audit Retention: 31536000000
Audit Syslog target:
Hardware Components
Cabinet Information
Avail
Protected
Total Sys Offline Used Audit Free Spare Available Cap System Regeneratio
Name User
Capacity Resources Capacity Capacity MetaData Capacity Capacity Capacity Until Buffer Buffer
Data
Alert
1 8943 GB 83 GB 1384 GB 5037 GB 489 GB 4549 GB 2438 GB 0 GB 748 GB 748 GB 1131 GB 559 GB
Cabinet 1
Switch Information
Cluster Prefix:10.255
Name Type MAC Addr Serial # Rail Description IP Addr

c001sw0 Cabinet 00:00:cd:20:f2:f8 61512075 0 Allied Telesyn AT-9924T-EMC version 2.6.6-00 11-Nov-2004 10.255.1.61
c001sw1 Cabinet 00:00:cd:20:f2:f4 61553618 1 Allied Telesyn AT-9924T-EMC version 2.6.6-00 11-Nov-2004 10.255.1.62
Port Information
c001sw0
Port Number Type Status

17 trunk Up
18 trunk Up
19 uplink Down
20 uplink Down
21 uplink Down
22 uplink Down
23 uplink Down
c001sw1
Port Number Type Status

17 trunk Up
xviii
18 trunk Up
19 uplink Down
20 uplink Down
21 uplink Down
22 uplink Down
23 uplink Down
ATS Information
Redundant Power
Type
Power Source
ats 0 0
Node Information
Node Serial # Sys ID Model Role Rail
c001n01 NNG00050301332 20d4cd20-1dd2-11b2-b7ed-8deef5e20500 100-580-006 A03 blc,access,storage B
c001n02 NNG00050301330 e9682e90-1dd1-11b2-9f61-bef3ff834670 100-580-006 A03 blc,access,storage A
c001n03 NNG00050301329 2219d860-1dd2-11b2-951b-ae0176eff945 100-580-006 A03 blc,storage B
c001n04 NNG00050301328 21d3c294-1dd2-11b2-8c9b-e1e4f3a1abc7 100-580-006 A03 blc,storage A
c001n05 NNG00050301326 1cf41a9e-1dd2-11b2-aa5c-bad01d28db90 100-580-006 A03 blc,storage B
c001n06 NNG00050301318 16e4c644-1dd2-11b2-92b4-da4d813e1862 100-580-006 A03 blc,storage A
c001n07 NNG00050301319 15c44514-1dd2-11b2-a171-b2dde14cfc5f 100-580-006 A03 blc,storage B
c001n08 NNG00044900027 e9274448-1dd1-11b2-b0d6-e74a97a399f2 100-580-006 A02 blc,storage A
Node Health and Service Settings

Node Node Status Maintenance Mode Compliance Mode Software Version
c001n01 online basic 3.1.0-934-650-12114
c001n02 online basic 3.1.0-934-650-12114
c001n03 offline 3.1.0-934-650-12114
c001n04 online basic 3.1.0-934-650-12114
c001n05 online basic 3.1.0-934-650-12114
c001n06 online basic 3.1.0-934-650-12114
c001n07 online basic 3.1.0-934-650-12114
c001n08 online basic 3.1.0-934-650-12114
Node Capacity
Protected Used Total Free
Total Sys Offline Used Audit System Available Free
Node User Obj Obj Obj
Capacity Resources Capacity Capacity MetaData Buffer Capacity Capacity
Data Count Count Count
c001n01 1118 GB 10 GB 0 GB 698 GB 67 GB 631 GB 173 GB 236 GB 409 GB 40282468 50000000 9717532
xix
c001n04 1118 GB 10 GB 0 GB 841 GB 84 GB 757 GB 164 GB 103 GB 267 GB 51173097 50000000 -1173097
c001n08 1118 GB 10 GB 0 GB 835 GB 79 GB 756 GB 166 GB 106 GB 272 GB 50151311 50000000 -151311
CPU Information
CPU Temp Case Temp
Node
(Celsius) (Celsius)
c001n01
c001n02
c001n03
c001n04 51 39
c001n05 50 39
c001n06 49 40
c001n07 50 40
c001n08 47 36
NIC Information
Node Name Config DNS IP Addr Duplex IP Addr Link Speed MAC Addr Subnet Status
c001n01 eth0 10.255.1.1 00:E0:81:64:25:62 enabled
eth1 10.255.1.1 00:E0:81:64:25:63 enabled
eth2 DHCP 152.62.69.48 F 10.69.137.21 100 00:E0:81:64:25:60 255.255.255.0 enabled
c001n02 eth0 10.255.1.2 00:E0:81:64:24:4E enabled
eth1 10.255.1.2 00:E0:81:64:24:4F enabled
eth2 DHCP 152.62.69.48 F 10.69.137.22 100 00:E0:81:64:24:4C 255.255.255.0 enabled
c001n04 eth0 10.255.1.4 00:E0:81:64:24:8A enabled
eth1 10.255.1.4 00:E0:81:64:24:8B enabled
eth2 00:E0:81:64:24:88 disabled
c001n05 eth0 10.255.1.5 00:E0:81:64:24:A2 enabled
eth1 10.255.1.5 00:E0:81:64:24:A3 enabled
eth2 00:E0:81:64:24:A0 disabled
xx
c001n06 eth0 10.255.1.6 00:E0:81:64:25:7E enabled
eth1 10.255.1.6 00:E0:81:64:25:7F enabled
eth2 00:E0:81:64:25:7C disabled
c001n07 eth0 10.255.1.7 00:E0:81:64:25:C2 enabled
eth1 10.255.1.7 00:E0:81:64:25:C3 enabled
eth2 00:E0:81:64:25:C0 disabled
c001n08 eth0 10.255.1.8 00:E0:81:63:9E:64 enabled
eth1 10.255.1.8 00:E0:81:63:9E:65 enabled
eth2 00:E0:81:63:9E:62 disabled
Drive Information
Protected Used
Total Sys Used Audit Free System Available
Node Index Status Mode User Obj
Capacity Resources Capacity MetaData Capacity Buffer Capacity
Data Count
c001n01 /dev/cstardm:0 enabled 279 GB 3 GB 185 GB 28 GB 156 GB 92 GB 56 GB 37 GB 9952624
/dev/cstardn:1 enabled 279 GB 3 GB 178 GB 14 GB 164 GB 99 GB 23 GB 75 GB 10640531
/dev/cstardo:2 enabled 279 GB 3 GB 168 GB 13 GB 155 GB 109 GB 23 GB 85 GB 9863145
/dev/cstardp:3 enabled 279 GB 3 GB 167 GB 13 GB 155 GB 109 GB 71 GB 39 GB 9826168
xxi
/dev/cstardp:3 disabled 279 GB 3 GB 0 GB 0 GB 0 GB 0 GB 0 GB 0 GB 0
Pools
Pool Capacity
Pool Used Capacity Used Obj Count Used Clip Count Quota Free Capacity
AuditArchive 0 GB 0 14484 Undefined Undefined
ConsoleArchive 0 GB 0 0 Undefined Undefined
default 739 GB 6473593 6473593 Undefined Undefined
cluster 739 GB 6473593 6488077 Undefined Undefined
Pool Retention
Pool Enforce Retention Default Fixed Min Fixed Max Var Min Var Max
AuditArchive No 0 Infinite Infinite 0 Infinite
ConsoleArchive No 0 Infinite Infinite 0 Infinite
default No 0 Infinite Infinite 0 Infinite
xxii
cluster No 0 Infinite Infinite 0 Infinite
Pool Access
Pool Profile read write retention-hold delete privileged-delete clip-enumeration exist clip-copy
ConsoleArchive console Yes Yes No Yes No Yes No Yes
default anonymous Yes Yes No Yes No Yes Yes Yes
Profiles
Home profile-
Profile Enabled monitor audit replication compliance accesscontrol configuration
Pool metadata
console ConsoleArchive Yes No Yes No No No No No
anonymous default Yes No Yes No No No No No
admin default Yes No Yes No No No No No
Cluster Capacity
Total Capacity: 8943 GB
System Resources: 83 GB
Offline Capacity: 1384 GB
Used Capacity: 5037 GB
Audit MetaData: 489 GB
Protected User Data: 4549 GB
Spare Capacity: 0 GB
Free Capacity: 2438 GB
Available Capacity: 748 GB
Available Capacity Until Alert: 748 GB
Used Object Count: 291048609
Total Object Count: 400000000
Free Object Count: 108951391
System Buffer: 1131 GB
Regeneration Buffer: 559 GB
xxiii
Appendix D – SNMP Formats
Alarm trap v2.0
trapAlarmNotification NOTIFICATION-TYPE
OBJECTS {notifyTimestamp,
notifyServer,
notifyDescription,
notifySeverity}
STATUS current
DESCRIPTION
"A trap indicating a problem on the Centera cluster.
The description can be used to describe the nature of the
change."
::= { notificationTrap 2 }
notifyTimestamp OBJECT-TYPE
SYNTAX Counter32
MAX-ACCESS not-accessible
STATUS current
DESCRIPTION
"The timestamp of the notification."
::= { genericNotify 1 }
notifyServer OBJECT-TYPE
SYNTAX OCTET STRING
STATUS current
DESCRIPTION
"The cluster id."
notifyDescription OBJECT-TYPE
SYNTAX OCTET STRING
STATUS current
DESCRIPTION
"A complete description of the event."
notifySeverity OBJECT-TYPE
SYNTAX INTEGER { informational (1),
warning (2),
severe (3)}
STATUS current
DESCRIPTION
"The severity level of the event, indicated by an integer
number in the range from 1 to 3."
Figure 55 – Alarm trap v2.0
xxiv
Appendix E – MoPI Formats
MoPI alert-1.0.dtd
<!ELEMENT alert (node, device)>
<!ATTLIST alert
type (degradation|improvement) #REQUIRED>
<!ELEMENT node EMPTY>
<!ATTLIST node
systemid CDATA #REQUIRED>
<!ELEMENT device EMPTY>
<!ATTLIST device
type (node|disk|switch|rootswitch|nic|sdk) #REQUIRED>
<!ATTLIST device
Figure 56 – MoPI alert-1.0.dtd
MoPI alert-2.0.dtd 38
<!ELEMENT alert (failure)>
<!ATTLIST alert
type (degradation|improvement) #REQUIRED
severity (ok|notification|warning|error|critical
#REQUIRED>
<!ELEMENT failure (node, device)>
<!ELEMENT node EMPTY>
<!ATTLIST node
systemid CDATA #REQUIRED
systemname CDATA #REQUIRED>
<!ELEMENT device EMPTY>
<!ATTLIST device
type (node|disk|switch|rootswitch|nic|sdk|sensor)
#REQUIRED
value CDATA #OPTIONAL>
MoPI alert-2.1.dtd 39
<!ATTLIST alert
type (degradation|improvement) #REQUIRED
severity (ok|notification|warning|error|critical) #REQUIRED
symptomcode CDATA #REQUIRED
timestamp CDATA #REQUIRED
clusterid CDATA #REQUIRED
formatDTD CDATA #REQUIRED>
38
39
xxv
MoPI discovery-1.1.dtd
<!ELEMENT applicationcontext ( client, securityprofile ) >
<!ELEMENT applicationcontexts ( applicationcontext* ) >
<!ELEMENT ats EMPTY >
<!ATTLIST ats powersource CDATA #REQUIRED >
<!ATTLIST ats status CDATA #REQUIRED >
<!ELEMENT cabinet ( ats?, cubeswitches, nodes ) >
<!ATTLIST cabinet id CDATA #REQUIRED >
<!ATTLIST cabinet availablerawcapacity CDATA #REQUIRED >
<!ATTLIST cabinet offlinerawcapacity CDATA #REQUIRED >
<!ATTLIST cabinet totalrawcapacity CDATA #REQUIRED >
<!ATTLIST cabinet usedrawcapacity CDATA #REQUIRED >
<!ELEMENT cabinets ( cabinet* ) >
<!ELEMENT capabilities ( capability* ) >
<!ELEMENT capability EMPTY >
<!ATTLIST capability enabled CDATA #REQUIRED >
<!ATTLIST capability name CDATA #REQUIRED >
<!ELEMENT client EMPTY >
<!ATTLIST client ip CDATA #IMPLIED >
<!ELEMENT cluster ( cabinets, rootswitches, services, pools, licenses) >
<!ATTLIST cluster availablerawcapacity CDATA #REQUIRED >
<!ATTLIST cluster clusterid CDATA #REQUIRED >
<!ATTLIST cluster compliancemode CDATA #REQUIRED >
<!ATTLIST cluster contactemail CDATA #REQUIRED >
<!ATTLIST cluster contactname CDATA #REQUIRED >
<!ATTLIST cluster groomingvisit CDATA #REQUIRED >
<!ATTLIST cluster location CDATA #REQUIRED >
<!ATTLIST cluster name CDATA #REQUIRED >
<!ATTLIST cluster offlinerawcapacity CDATA #REQUIRED >
<!ATTLIST cluster protectionschemes CDATA #REQUIRED >
<!ATTLIST cluster serial CDATA #REQUIRED >
<!ATTLIST cluster serviceid CDATA #REQUIRED >
<!ATTLIST cluster serviceinfo CDATA #REQUIRED >
<!ATTLIST cluster siteid CDATA #REQUIRED >
<!ATTLIST cluster softwareversion CDATA #REQUIRED >
<!ATTLIST cluster totalrawcapacity CDATA #REQUIRED >
<!ATTLIST cluster usedrawcapacity CDATA #REQUIRED >
<!ELEMENT cubeswitch EMPTY >
<!ATTLIST cubeswitch description CDATA #REQUIRED >
<!ATTLIST cubeswitch ip CDATA #REQUIRED >
<!ATTLIST cubeswitch mac CDATA #REQUIRED >
<!ATTLIST cubeswitch name CDATA #REQUIRED >
<!ATTLIST cubeswitch rail CDATA #REQUIRED >
<!ATTLIST cubeswitch serial CDATA #REQUIRED >
<!ATTLIST cubeswitch status CDATA #REQUIRED >
<!ELEMENT cubeswitches ( cubeswitch* ) >
<!ELEMENT discovery ( format?, cluster? ) >
<!ELEMENT format EMPTY >
<!ATTLIST format version CDATA #REQUIRED >
<!ELEMENT license EMPTY >
<!ATTLIST license key CDATA #REQUIRED >
<!ELEMENT licenses ( license* ) >
<!ELEMENT nic EMPTY >
xxvi
<!ATTLIST nic config CDATA #IMPLIED >
<!ATTLIST nic dnsip CDATA #IMPLIED >
<!ATTLIST nic duplex CDATA #IMPLIED >
<!ATTLIST nic ip CDATA #IMPLIED >
<!ATTLIST nic linkspeed CDATA #IMPLIED >
<!ATTLIST nic mac CDATA #IMPLIED >
<!ATTLIST nic name CDATA #REQUIRED >
<!ATTLIST nic status CDATA #REQUIRED >
<!ATTLIST nic subnet CDATA #IMPLIED >
<!ELEMENT nics ( nic* ) >
<!ELEMENT node ( nics, volumes ) >
<!ATTLIST node downtime CDATA #REQUIRED >
<!ATTLIST node hardwareversion CDATA #REQUIRED >
<!ATTLIST node name CDATA #REQUIRED >
<!ATTLIST node rail CDATA #REQUIRED >
<!ATTLIST node roles CDATA #REQUIRED >
<!ATTLIST node softwareversion CDATA #REQUIRED >
<!ATTLIST node status CDATA #REQUIRED >
<!ATTLIST node systemid CDATA #REQUIRED >
<!ATTLIST node totalrawcapacity CDATA #REQUIRED >
<!ATTLIST node usedrawcapacity CDATA #REQUIRED >
<!ELEMENT nodes ( node* ) >
<!ELEMENT pool ( applicationcontexts ) >
<!ATTLIST pool name CDATA #REQUIRED >
<!ATTLIST pool totalrawcapacity CDATA #REQUIRED >
<!ATTLIST pool usedrawcapacity CDATA #REQUIRED >
<!ELEMENT pools ( pool* ) >
<!ELEMENT rootswitch EMPTY >
<!ATTLIST rootswitch ip CDATA #REQUIRED >
<!ATTLIST rootswitch side CDATA #REQUIRED >
<!ATTLIST rootswitch status CDATA #REQUIRED >
<!ELEMENT rootswitches ( rootswitch* ) >
<!ELEMENT securityprofile ( capabilities ) >
<!ATTLIST securityprofile enabled CDATA #REQUIRED >
<!ATTLIST securityprofile name CDATA #REQUIRED >
<!ELEMENT servicecontentprotectiontransformation EMPTY >
<!ATTLIST servicecontentprotectiontransformation name CDATA #REQUIRED >
<!ATTLIST servicecontentprotectiontransformation scheme CDATA #REQUIRED>
<!ATTLIST servicecontentprotectiontransformation status CDATA #REQUIRED>
<!ATTLIST servicecontentprotectiontransformation threshold CDATA #REQUIRED >
<!ATTLIST servicecontentprotectiontransformation version CDATA #IMPLIED>
<!ELEMENT servicegarbagecollection EMPTY >
<!ATTLIST servicegarbagecollection name CDATA #REQUIRED >
<!ATTLIST servicegarbagecollection status CDATA #REQUIRED >
<!ATTLIST servicegarbagecollection version CDATA #IMPLIED >
<!ELEMENT serviceorganicregeneration EMPTY >
<!ATTLIST serviceorganicregeneration name CDATA #REQUIRED >
<!ATTLIST serviceorganicregeneration status CDATA #REQUIRED >
<!ATTLIST serviceorganicregeneration version CDATA #IMPLIED >
<!ELEMENT serviceperformanceregeneration EMPTY >
<!ATTLIST serviceperformanceregeneration name CDATA #REQUIRED >
<!ATTLIST serviceperformanceregeneration status CDATA #REQUIRED >
<!ATTLIST serviceperformanceregeneration version CDATA #IMPLIED >
xxvii
<!ELEMENT servicequery EMPTY >
<!ATTLIST servicequery name NMTOKEN #REQUIRED >
<!ATTLIST servicequery status CDATA #REQUIRED >
<!ATTLIST servicequery version CDATA #IMPLIED >
<!ELEMENT servicereplication EMPTY >
<!ATTLIST servicereplication ip CDATA #REQUIRED >
<!ATTLIST servicereplication name NMTOKEN #REQUIRED >
<!ATTLIST servicereplication status CDATA #REQUIRED >
<!ATTLIST servicereplication version CDATA #IMPLIED >
<!ELEMENT servicerestore EMPTY >
<!ATTLIST servicerestore ip CDATA #REQUIRED >
<!ATTLIST servicerestore name NMTOKEN #REQUIRED >
<!ATTLIST servicerestore status CDATA #REQUIRED >
<!ATTLIST servicerestore version CDATA #IMPLIED >
<!ELEMENT services ( servicegarbagecollection*,
servicecontentprotectiontransformation*, servicereplication*, servicerestore*,
servicequery*, serviceorganicregeneration*, serviceperformanceregeneration*,
serviceshredding*, servicesnmp* ) >
<!ELEMENT serviceshredding EMPTY >
<!ATTLIST serviceshredding name NMTOKEN #REQUIRED >
<!ATTLIST serviceshredding status CDATA #REQUIRED >
<!ATTLIST serviceshredding version CDATA #IMPLIED >
<!ELEMENT servicesnmp EMPTY >
<!ATTLIST servicesnmp communityname CDATA #REQUIRED >
<!ATTLIST servicesnmp ip CDATA #REQUIRED >
<!ATTLIST servicesnmp name NMTOKEN #REQUIRED >
<!ATTLIST servicesnmp port CDATA #REQUIRED >
<!ATTLIST servicesnmp status CDATA #REQUIRED >
<!ATTLIST servicesnmp trapinterval CDATA #REQUIRED >
<!ATTLIST servicesnmp version CDATA #IMPLIED >
<!ELEMENT volume EMPTY >
<!ATTLIST volume index CDATA #REQUIRED >
<!ATTLIST volume status CDATA #REQUIRED >
<!ATTLIST volume totalrawcapacity CDATA #IMPLIED >
<!ELEMENT volumes ( volume* ) >
Figure 59 – MoPI discovery-1.1.dtd
xxviii
MoPI discovery-1.2.dtd40
<!ELEMENT cabinet ( ats?, cubeswitches, nodes ) >
<!ATTLIST cabinet id CDATA #REQUIRED >
<!ATTLIST cabinet availablerawcapacity CDATA #REQUIRED >
<!ATTLIST cabinet offlinerawcapacity CDATA #REQUIRED >
<!ATTLIST cabinet totalrawcapacity CDATA #REQUIRED >
<!ATTLIST cabinet usedcapacity CDATA #REQUIRED >
<!ATTLIST cabinet sparecapacity CDATA #REQUIRED >
<!ATTLIST cabinet freecapacity CDATA #REQUIRED >
<!ATTLIST cabinet auditmeta data CDATA #REQUIRED >
<!ATTLIST cabinet offlinecapacity CDATA #REQUIRED >
<!ATTLIST cabinet systemresources CDATA #REQUIRED >
<!ATTLIST cabinet usedobjectcount CDATA #REQUIRED >
<!ELEMENT cluster ( cabinets, rootswitches, services, pools, licenses )>
<!ATTLIST cluster availablerawcapacity CDATA #REQUIRED >
<!ATTLIST cluster clusterid CDATA #REQUIRED >
<!ATTLIST cluster compliancemode CDATA #REQUIRED >
<!ATTLIST cluster contactemail CDATA #REQUIRED >
<!ATTLIST cluster contactname CDATA #REQUIRED >
<!ATTLIST cluster groomingvisit CDATA #REQUIRED >
<!ATTLIST cluster location CDATA #REQUIRED >
<!ATTLIST cluster name CDATA #REQUIRED >
<!ATTLIST cluster offlinerawcapacity CDATA #REQUIRED >
<!ATTLIST cluster protectionschemes CDATA #REQUIRED >
<!ATTLIST cluster serial CDATA #REQUIRED >
<!ATTLIST cluster serviceid CDATA #REQUIRED >
<!ATTLIST cluster serviceinfo CDATA #REQUIRED >
<!ATTLIST cluster siteid CDATA #REQUIRED >
<!ATTLIST cluster softwareversion CDATA #REQUIRED >
<!ATTLIST cluster totalrawcapacity CDATA #REQUIRED >
<!ATTLIST cluster usedcapacity CDATA #REQUIRED >
<!ATTLIST cluster sparecapacity CDATA #REQUIRED >
<!ATTLIST cluster freecapacity CDATA #REQUIRED >
<!ATTLIST cluster auditmeta data CDATA #REQUIRED >
<!ATTLIST cluster offlinecapacity CDATA #REQUIRED >
<!ATTLIST cluster systemresources CDATA #REQUIRED >
<!ATTLIST cluster usedobjectcount CDATA #REQUIRED >
<!ELEMENT node ( nics, volumes ) >
<!ATTLIST node downtime CDATA #REQUIRED >
<!ATTLIST node hardwareversion CDATA #REQUIRED >
<!ATTLIST node name CDATA #REQUIRED >
<!ATTLIST node rail CDATA #REQUIRED >
<!ATTLIST node roles CDATA #REQUIRED >
<!ATTLIST node softwareversion CDATA #REQUIRED >
<!ATTLIST node status CDATA #REQUIRED >
<!ATTLIST node systemid CDATA #REQUIRED >
<!ATTLIST node totalrawcapacity CDATA #REQUIRED >
<!ATTLIST node usedcapacity CDATA #REQUIRED >
<!ATTLIST node freecapacity CDATA #REQUIRED >
40
xxix
<!ATTLIST node auditmeta data CDATA #REQUIRED >
<!ATTLIST node offlinecapacity CDATA #REQUIRED >
<!ATTLIST node systemresources CDATA #REQUIRED >
<!ATTLIST node usedobjectcount CDATA #REQUIRED >
<!ELEMENT pool ( applicationcontexts ) >
<!ATTLIST pool name CDATA #REQUIRED >
<!ATTLIST pool totalrawcapacity CDATA #REQUIRED >
<!ATTLIST pool usedcapacity CDATA #REQUIRED >
<!ATTLIST pool sparecapacity CDATA #REQUIRED >
<!ATTLIST pool freecapacity CDATA #REQUIRED >
<!ATTLIST pool auditmeta data CDATA #REQUIRED >
<!ATTLIST pool offlinecapacity CDATA #REQUIRED >
<!ATTLIST pool systemresources CDATA #REQUIRED >
<!ATTLIST pool usedobjectcount CDATA #REQUIRED >
Figure 60 – MoPI discovery-1.2.dtd
MoPI health-x.y.dtd
As of CentraStar 2.3, the MoPI and ConnectEMC health reports have been synchronized. Please see
xxx
ConnectEMC Formats.
xxxi

Centera Monitoring and Reporting in CentraStar 3.1-1

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Centera Monitoring and Reporting in CentraStar 3.1-1

Uploaded by

Copyright:

Available Formats

Engineering White Paper

Centera Monitoring and Reporting

Figure 1 – Sensor Framework

Table 1 – Sensor List

Channel Alerts Real-time History

Symptom Code Sensor Name Threshold (Severity)

2.1.2.1.01.01 ClusterClipIntegrity INCOMPLETE (Error)

3.1.3.1.01.01 InternalNICFailureCount 1 (Warning)

3.1.6.01.01.01 NodesOffline 1 (Warning)

Overview of Health Reporting Channels

Channel Health Sent Daily6 Sent After On Request

Scheduled Reporting Using CLI Scripting

Creating a script file

Creating a batch file

Figure 3 – CLI Script Mode Syntax

The following parameters need to be specified:

Figure 5 – Windows Scheduled Tasks

Figure 6 – Windows Scheduled Task Details

Figure 7 – Windows Scheduled Task List

Figure 9 – Crontab Entry

Available Capacity Until Alert

Free & Available

Figure 10 – Capacity Components On Empty System

Figure 11 – Capacity Components On Used System

Raw Capacity Reporting

Figure 12 – Show Capacity Total

Available Capacity Reporting

Regeneration Buffer Alerting

Regeneration Buffer > 0 and Policy = Alert Only

Regeneration Buffer > 0 and Policy = Hard Stop

Figure 13 – Show Capacity Availability

Object Count Reporting

Pool Capacity Reporting

Pool Capacity Calculation

Today is April 10th

Figure 14 – Pool Capacity Algorithm

Figure 15 – Pool Capacity Re-calculation Algorithm

Pool Capacity Reporting Limitations

Useable Capacity Reporting

Relevant Reports Needed To Calculate Useable Capacity

Used Raw Capacity: 1,308 GB (13%)

Figure 16 – Show Capacity Total

Figure 17 – Show Pool Capacity

Figure 18 – Show Capacity Availability

Useable Capacity for Capacity Bound Systems

Useable Capacity for Object Bound Systems

Best Practices for Object Bound Systems

Table 19 – Performance Sensors

Overview of Performance Reporting Tool Coverage

How is the Restore Queue Calculated

Figure 20 – Restore Queue Calculation

Garbage Collection Statistics

Figure 21 – OnAlert Configuration

ConnectEMC Notification Formats

• Warning: a non-critical error has occurred, and

• Severe: the Centera cluster is in a critical state.

Figure 22 – SNMP Heartbeat Trap

SNMP Alarm trap

Figure 23 – SNMP Alarm Trap

ECC Storage Agent For Centera Installation & Configuration

Discovery data collection policy

Configuring the Storage Agent for Centera

ECC Alert Mapping with MoPI 1.0

ECC Alert Mapping with MoPI 2.0