Utility Service R3.2 - Veritas Cluster Server Deployment Guide

HP Enterprise Services
Utility Service R3.2 - Veritas Cluster Server

Deployment Guide
Version#1.0 – Draft
24 Oct 2011
HP RESTRICTED
HP Enterprise Services
Contents
1 Guide Document Control ........................................................................ 1
1.1 Document Approvers .......................................................................................... 1
1.2 Amendment Record ............................................................................................ 1
1.3 Related Documentation ....................................................................................... 1
2 Introduction .......................................................................................... 3
2.1 Purpose ............................................................................................................ 3
2.2 Audience ........................................................................................................... 3
2.3 Terminology and Acronyms ................................................................................. 3
3 Capability Overview ............................................................................... 6
4 Prerequisites ......................................................................................... 7
4.1 Infrastructure Dependencies for Solution ............................................................... 7
4.1.1 Physical Dependencies .................................................................................. 7
4.1.2 Functional Dependencies ............................................................................... 7
4.1.3 Software Dependencies ................................................................................. 8
4.1.4 Process Dependencies ................................................................................... 9
5 Security ............................................................................................... 11
5.1 Security Roles .................................................................................................. 11
5.2 Security Implementation ................................................................................... 11
5.2.1 Legal and Regulatory Compliance ................................................................. 11
5.2.2 Vulnerability Scan ....................................................................................... 11
6 Preparation .......................................................................................... 12
6.1 Preparation Roles ............................................................................................. 12
6.1.1 Role Definitions .......................................................................................... 12
6.1.2 Role Ownership .......................................................................................... 12
6.2 Preparation Tasks ............................................................................................. 13
6.2.1 Server Deployment Prepare Tasks ................................................................ 13
6.2.2 Storage Deployment Prepare Tasks .............................................................. 13
6.2.3 Network Deployment Prepare Tasks .............................................................. 13
6.2.4 Software Deployment Preparation Tasks ........................................................ 14
7 Implementation Considerations ........................................................... 15

7.1 Sizing Considerations ........................................................................................ 15
7.2 Networking Considerations ................................................................................ 15
7.3 Considerations for Applications Required for Capability.......................................... 15
7.4 Process Considerations...................................................................................... 15
7.5 Additional Considerations .................................................................................. 15
7.5.1 Veritas Cluster Server ................................................................................. 15
8 Configuration Standards ...................................................................... 16
Version#1.0 – Draft 24 Oct 2011 |i|

HP RESTRICTED
© 2009 HEWLETT-PACKARD DEVELOPMENT COMPANY, LP.
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Contents
8.1 OS version and VCS.......................................................................................... 16

8.2 Cluster nodes hardware .................................................................................... 16
9 Implementation ................................................................................... 17
9.1 Veritas Cluster Server Implementation ................................................................ 17
9.1.1 Overview ................................................................................................... 17
9.1.2 Implementation Roles ................................................................................. 17
9.1.3 Implementation Process Diagram ................................................................. 17
9.1.4 Implementation Activities ............................................................................ 18
9.1.5 Transformation Services .............................................................................. 68
9.1.6 Event Monitoring Activation ......................................................................... 68
9.1.7 Reporting Activation ................................................................................... 69
9.1.8 Cost Recovery Activation ............................................................................. 69
9.1.9 Pre-Acceptance Testing ............................................................................... 69
9.1.10 Customer Acceptance Testing ................................................................... 70
9.1.11 Operations Turnover ................................................................................ 70
9.1.12 Backing Off ............................................................................................ 70
9.1.13 De-implementation.................................................................................. 70
9.1.14 Completion ............................................................................................. 70
10 Other Deployment Considerations ....................................................... 71
11 Contacts .............................................................................................. 72
Version#1.0 – Draft 24 Oct 2011 | ii |

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Figures
Figures
Figure 1 - VCS cluster components .................................................................................. 6
Figure 2 - Implementation Process Diagram.................................................................... 18
Version#1.0 – Draft 24 Oct 2011 | iii |

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Guide Document Control
1 Guide Document Control

1.1 Document Approvers
The individuals who reviewed, verified, and approved the content of this document based on
the requirements of the defined services during this document’s creation are listed in the table
below:
Approver’s Name Approver’s Group’s Name Content Approved

Stefan Landwehrmann Manager Cloud Service Engineering 24 Oct 2011
1.2 Amendment Record

The following Change History log contains a record of changes made to this document:
Published / Version Author Section /

Revised Date # (optional) Nature of Change
19 Oct 2011 1.0 Pavol Masarovic, Initial Draft
Jennifer Vargas
1.3 Related Documentation

These documents are required. Ensure you have access to these documents when using this
document.
Document Name Short description of required document

Technical Reference Guide for Utility Services Foundational Technical Architecture -
Utility Services Foundational 3.2 Technical Reference Guide
Utility Services 3.2 Foundational http://enhanced1.sharepoint.hp.com/teams/Application-
Deployment Guide Services/projects/usr32/Project%20Documents/3-
%20Engineering/Utility_Services_Foundational_Technica
l_Architecture_Deployment_Guide.docx
Utility Services 3.2 Foundational http://enhanced1.sharepoint.hp.com/teams/Application-
Operation Guide Services/projects/usr32/Project%20Documents/3-
%20Engineering/Utility_Services_Foundational_Technica
l_Architecture_Operation_Guide.docx
Statement of Work (SOW) A detailed, client-facing description the work to be done
as a result of purchasing an offering/service associated
with the Capability being described in this guide.
Version#1.0 – Draft 24 Oct 2011 |1|

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Guide Document Control
Document Name Short description of required document

Utility Services Foundational http://enhanced1.sharepoint.hp.com/teams/Application-
Technical Architecture - Technical Services/projects/usr32/Project%20Documents/3-
Reference Guide %20Engineering/Utility_Services_Foundational_Technica
l_Architecture_Technical_Reference_Guide.doc
Utility Services Tier 4 Data Center The guideline for tier 4 data center Ethernet and storage
Network Interlink Sizing Guidance network interlink sizing.
Utility Services Tier 4 Data Center Network Interlink
Sizing Guidance
Utility Service Technical This document provides guidelines for server Removal
Architecture – from Production
Deployment Guide Appendix – http://enhanced1.sharepoint.hp.com/teams/AISPC/Publi
UNIX Decommission Process shedDocuments/Utility%20Services%20R3.1/UtilityServi
ce_UnixDecomissionProcess.docx
Utility Services Cabling and In detail it describes how to server rack, cable
VLANs connectivity and IP address, VLAN allocation.
These documents will be published along with the other
R3.2 docs/appendixes.
http://enhanced1.sharepoint.hp.com/teams/Application-
Services/projects/usr32/Project%20Documents/3-
%20Engineering/UtilityService-
Cabling%20and%20vlans.xlsx
Utility Services 3.2 Hardware - Provides detailed bill of materials for hardware and
Software Bill of Materials software components used in Utility Services.
http://enhanced1.sharepoint.hp.com/teams/Application-
Services/projects/usr32/Project%20Documents/3-
%20Engineering/Utility%20Services%20Hardware%20-
%20Software%20Bill%20of%20Materials.xls
Veritas Dynamic Multi-Pathing Installation Guide for Veritas Dynamic Multi-Pathing
https://sort.symantec.com/public/documents/dmp/5.1sp
1/linux/productguides/pdf/dmp_install_51sp1_lin.pdf
Veritas Cluster Server Installation Cluster Server Installation document
Guide https://sort.symantec.com/public/documents/vcs/5.1sp
1/linux/productguides/pdf/vcs_install_51sp1_lin.pdf

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Introduction
2 Introduction
This section describes the purpose and audience of, and the terminology relating to, this
document.
2.1 Purpose
This guide provides detailed information regarding how to configure and implement the
services and Capability functions defined within the Technical Reference Guide. In most cases,
following the guidelines in this document would result in the creation of client instance-specific
configuration information and procedures typically used by Production Change Management
and operations personnel to make additions and changes to the production environment for an
instance of the service.
This document is specifically focused on deploying Veritas Cluster Server (VCS) 5.1 SP1 and
Veritas Storage Foundation (VSF) on Red Hat 5.6 Enterprise Server Linux.
The implementation assumes that infrastructure is already in place. The scope of this
document is to cover all installation and configuration aspects of VCS and VSF running on
Utility Services platform and Tier 4 setup.
This deployment guide is part of a group of related documents for the Veritas Cluster Service
implementation offering. It identifies, from a business / management perspective, everything
that must be done before, during, and after solution deployment to ensure success.
2.2 Audience
The primary audiences for this Capability Guide are Applied Engineering and Operations, who
will use the guide as a deployment in conjunction with the Technical Reference Guide and
Operation Guide.
Secondary audiences for this guide are Sales, Solution Development including Solution and
Portfolio Architects, and Production Change Management organizations, or those who need to
understand the technical aspects of the Capability.
The VCS and OS skill sets are listed below and it is expected the lead OS administrator is at
least an Intermediate OS administrator, and has prior experience with Symantec VERITAS
Cluster Server (VCS), Storage Foundation and storage configuration.
Veritas Cluster Service and Storage Foundation skill sets
• Familiar with Veritas Cluster Service for Linux
• Familiar with Veritas Storage Foundation for Linux
• Familiar with RHEL 5.6 x86-64
2.3 Terminology and Acronyms

This section defines the terminology and common acronyms used in this document.

HP RESTRICTED
Term/Acronym Definition
Capability A capability is a defined set of competencies (a combination of
skills, processes, tools, technologies, and experiences)
required to deliver projects and services, specifically to
produce a defined set of delivery outputs.
A capability enables the delivery of one or more offerings and
is managed strategically at a global level.
From an organizational perspective, the term capability is an
Enterprise Services organizational entity responsible for
managing a related set of capabilities and the service offerings
most closely related to those capabilities.
HP Enterprise The HP Enterprise Services sales or demand catalog formerly known

Services Catalog as the AIM Catalog.
HP Enterprise Options under a Feature in the HP Enterprise Services Catalog that
Services Catalog are selected during the purchasing cycle. These often correlate with
Attribute Functional Components in the Capability.
HP Enterprise A modifier of an Attribute that is used to tailor the specific of an
Services Catalog Attribute’s services. May impact pricing.
Context
HP Enterprise Lowest level saleable item with a price. May represent a grouping of
Services Catalog other features. - Each feature is owned by only one offering or
Feature solution. - Owner is responsible for the management of the feature’s
lifecycle… (See HP Enterprise Services Lexicon for complete
description) These often correlate with Functional Components in the
Capability.
HP Enterprise Typically a grouping of features assembled to simplify the ordering
Services Catalog process in the HP Enterprise Services Catalog.
Package
Functional Part or a characteristic of a capability such as people (skill sets),
Component processes, and assets/technology (hardware/software) that may be
core to the capability or may be optional. The utilization/activation of
the Functional Component by Delivery will depend on client’s
purchase selections made during the purchase cycle as determined
by the HP Enterprise Services Catalog rules.
Service Offering Market-facing grouping of related features that addresses a class of
business needs that can be industry-independent or industry-specific.
Managed by a single service line, however may include multiple
service line features. Addresses client needs recognized in a
particular market sub-segment.
Access Control A security service that provides protection of system resources
Facility against unauthorized access. The two basic mechanisms for
implementing this service are access control lists and tickets.
Accountability A system’s ability to maintain a secure log of any changes that may
affect security.
VCS Veritas Cluster Server
SF Storage Foundation
VxVM Veritas Volume Manager
VxFS Veritas File System
Cluster A logical grouping of servers (nodes) providing failover capability

HP RESTRICTED
Term/Acronym Definition
I/O Fencing I/O fencing protects the data on shared disks when nodes in a cluster
detect a change in the cluster membership that indicates a split-brain
condition.
DC Data Center
CP Server Coordinator Point Server

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Capability Overview
3 Capability Overview
This section describes the solution(s); what the solution accomplishes the major solution
components, how the solution relates to or complements other solutions and how the
elements are affected by this implementation.
Veritas™Cluster Server by Symantec is a high-availability solution for applications and services

configured in a cluster. Veritas Cluster Server (VCS) monitors systems and application
services, and restarts services when hardware or software fails.
Veritas Cluster Server is high-available clustering solution for Utility Services Tier 3 and Tier 4
on Red hat Linux. It is based on two or more node clusters in two different cross Data Centers
with storage arrays in each DC and one coordinator point server in a third DC that is used for
cluster I/O fencing. I/O fencing protects the data on shared disks when nodes in a cluster
detect a change in the cluster membership that indicates a split-brain condition.
The fencing operation determines the following:
• The nodes that must retain access to the shared storage
• The nodes that must be ejected from the cluster
Failover clusters are used in the service offerings (tier 3 and tier 4) to provide enhanced
application up-time and failure tolerance. The details of implementation will vary by offering.
Below the major components diagram:
Figure 1 - VCS cluster components

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Prerequisites
4 Prerequisites
This section includes all prerequisite actions, software/firmware levels, and dependencies
required before the implementation of this solution.
4.1 Infrastructure Dependencies for Solution

This section identifies the prerequisites and co-requisites for this solution implementation.
4.1.1 Physical Dependencies

This section describes the physical assets needed to implement this solution.
4.1.1.1 Common Physical Elements

Common physical hardware and facility is the prerequisite of Utility Service deployment, which
includes:
• Highly reliable, physically secure HP Data Center
• Power supply supports up to 10KW each rack
4.1.1.2 Shared Management Server

HP shared management servers which are not dedicated for Utility Service are dependencies
of Utility Service Foundational Architecture:
• HP OO Remote Access Service (RAS) Server Deployment
• Enterprise Security Manager Deployment
• OVCM Configuration Server Deployment
4.1.1.3 OMC Network

OMCnet infrastructure is built on top of the HP Intranet. Utility Service environment is
dependent on the HP Outsourcing Services OMCnet connectivity for daily delivery activities
and monitoring by the Utility Service standard toolset as referenced in Section Utility Services
Standard Reference Architecture Tools in the Utility Services Foundational Technical
Architecture - Technical Reference Guide . The URL is available at Related Documentation
section.
4.1.2 Functional Dependencies

This section describes the functional dependencies that will be required during the
implementation of this solution. Refer to the Roles and Responsibilities section of the Technical
Reference Guide for role definitions. The URL to locate Utility Services Foundational Technical
Architecture - Technical Reference Guide is available at Related Documentation section.
Support organizations involved in the delivery of this service are:

• Network Tower: Management of OMC Net, Utility Services networking equipment, and
connection to the customer LANs.
• Windows Tower: Management of Windows-based systems, ProLiant hardware, blade
enclosures, and Windows-based management systems.
• Unix Tower: Management of HP-UX and Linux-based systems, ProLiant hardware running
Linux, Integrity hardware, blade enclosures, Linux-based management systems, and
HP-UX-based management systems.
• Storage Tower: Management of storage including storage arrays and Fibre Channel
infrastructure.

HP RESTRICTED
• Backup and Restore Tower: Management of backup and restore.

• Remote Management Center (RMC): Providing monitoring infrastructure and automated
event handling.
• Remote Service Center: Providing first and second line support.
• Global Facilities: Provides data center facilities, including managing space assignments,
power, cooling, and security.
• GSOC for security compliance
• Governance
Main Roles in the Delivery

• Technical Owner
• Delivery Lead
• Service Delivery Manager
• Implementation Manager
• Project Engineers
The solution is based on the HP Standard Reference Architecture (SRA). Key SRA components
are:
• Central HP managed Openview monitoring stations
• Central HP managed Workflow system
• Enterprise System List (ESL)
• Radio for inventory collection and software distribution
• Enterprise Root Model (ERM) to manage HP owned administrative accounts
• OMC infrastructure to connect in a secure manner to the customer servers. This includes
management of Active Cards / Active keys.
• VPN connectivity from home office to the HP network for weekend work and standby.
4.1.3 Software Dependencies

This section describes the software assets required to implement this solution.
4.1.3.1 Tools
Tools used in Utility Service solution are described in Utility Standard Reference Architecture
Tools section of the Utility Services Foundational Technical Architecture –Technical Reference
Guide. The URL to locate is available at Related Documentation section.
4.1.3.2 License
OS License can be provided by HP or by the Customer as defined in contract. There are also
VCS and SF licenses required for each cluster in defined environment.
4.1.3.3 Veritas Cluster Server

Supported Linux Operating Systems for VCS 5.1 SP1
• Red Hat Enterprise Linux 5 (RHEL 5) with Update 3 (2.6.18-128.el5 kernel) or later on
AMD Opteron or Intel Xeon EM64T (x86_64)
• SUSE Linux Enterprise Server 10 (SLES 10) with SP2 (2.6.16.60-0.21 kernel) on AMD
Opteron or Intel Xeon EM64T (x86_64)

HP RESTRICTED
• SUSE Linux Enterprise Server 11 (SLES 11) (2.6.27.19-5 kernel) on AMD Opteron or Intel
Xeon EM64T (x86_64)
• Oracle Enterprise Linux (OEL 5) with Update 3 (2.6.18-128.el5 kernel) or later on AMD
Opteron or Intel Xeon EM64T (x86_64)
Required Linux RPMs for VCS
Lists the RPMs that VCS requires for a given Linux operating system
Operating System Required RPMs

compat-libgcc-296-2.96-138.i386.rpm
RHEL 5
compat-libstdc++-33-3.2.3-61.i386.rpm
compat-libstdc++-296-2.96-138.i386.rpm
glibc-2.5-24.i686.rpm
libgcc-4.1.2-42.el5.i386.rpm
libstdc++-3.4.6-10.i386.rpm
compat-libstdc++-33-3.2.3-61.x86_64.rpm
glibc-2.5-24.x86_64.rpm
glibc-common-2.5-24.x86_64.rpm
libgcc-4.1.2-42.el5.x86_64.rpm
libstdc++-3.4.6-10.x86_64.rpm
java-1.4.2-gcj-compat-1.4.2.0-40jpp.115.noarch.rpm
compat-32bit-2006.1.25-11.2.x86_64.rpm
SLES 10
glibc-32bit-2.4-31.54.x86_64.rpm
compat-2006.1.25-11.2.x86_64.rpm
compat-libstdc++-5.0.7-22.2.x86_64.rpm
glibc-2.4-31.54.x86_64.rpm
libgcc-4.1.2_20070115-0.21.x86_64.rpm
libstdc++-4.1.2_20070115-0.21.x86_64.rpm
glibc-2.9-13.2
SLES 11
glibc-32bit-2.9-13.2
libgcc43-4.3.3_20081022-11.18
libgcc43-32bit-4.3.3_20081022-11.18
libstdc++43-4.3.3_20081022-11.18
libstdc++43-32bit-4.3.3_20081022-11.18
Hardware
– VCS need at least two dedicated NICs for heartbeat in additional to one NIC for
public; you can also configure aggregated interfaces.
– VCS need disks/SAN support SCSI-3 persistent reservation if you plan to use VCS
I/O fencing feature
• You must have three coordinator disks
• Each of the coordinator disks must use a physically separate disk or LUN
• The coordinator disks must exist in a disk group
4.1.4 Process Dependencies

4.1.4.1 Service Management Processes
The deployment operations need to apply ITIL best practices including problem management,
change management, and configuration management. For more information, go to:
http://enhanced1.sharepoint.hp.com/teams/GD-ITSM/default.aspx

HP RESTRICTED
4.1.4.2 Implementation Processes

Not Applicable.
Version#1.0 – Draft 24 Oct 2011 | 10 |

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Security
5 Security
This section describes the overall security strategy for this solution. Security for the Utility
Services Foundational Technical Architecture is detailed in the Utility Service Security Plan.
5.1 Security Roles

This section identifies the roles required to implement the security strategy for this solution.
Descriptions of the roles can be found in the Technical Reference Guide.
5.2 Security Implementation

Utility Services are built on a common shared infrastructure. This requires a common set of
controls to meet the base security objectives. These objectives, controls, and processes are
described in the Utility Service Security Guide.
5.2.1 Legal and Regulatory Compliance

This offering has been designed to satisfy HP Enterprise Security Policies and Standards
(ESPS) which may or may not satisfy client security requirements. Customer specific
deployments for risk adverse customers may require a security assessment including the
potential for the cost impact of additional security controls, if required. All relevant statutory,
regulatory, and contractual requirements will be explicitly defined and documented in the
client specific Statement of Work (SOW), The URL to locate SOW is available at Related
Documentation section.
5.2.2 Vulnerability Scan

Pre-install scans are performed on newly configured servers to identify known vulnerabilities
before production.
This network vulnerability scan is done external to the server and requires no software or
agent to be installed on the end server. Work with the Information Security team to request
an initial network vulnerability scan of the servers once built but prior to deployment. The first
pre-install scan is free. Any additional scans will be charged. To request a pre-install scan, go
to the network assessment team page select the Pre-Installation Scan Request Form.
Choose Scan Request Form in the left navigation. Review/address the scan findings and
update the Implementation Plan Builder (IPB) which can be accessed from the same URL.
Update for system admin responsible findings.
For the detailed pre-install scans processes, go to the Pre-install Scan Process page.
The request for pre-install scans must be received five (5) business days before it is needed,
to avoid charges.

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Preparation
6 Preparation
6.1 Preparation Roles
This section identifies the roles that will be needed to perform the preparation for the
implementation of this solution. Refer to the Roles and Responsibilities section of the Utility
Services Foundational Technical Architecture - Technical Reference Guide for definitions. The
URL is available at Related Documentation section.
6.1.1 Role Definitions

The following table describes the roles that are required for the delivery of the defined
services. Note that detailed functions of each role as they relate to this Capability are
contained in the Utility Service Foundational Technical Architecture Deployment Guide and
Operation Guides. The URLs are available at Related Documentation section.
Table 1: Role Definitions
Name of Role Short Definition of Role Special skills/training

System Engineer & Maintaining OS environment Windows, Unix, Linux
Administrator
Storage Engineers & Maintaining HP Storage for SAP HP Storage
Administrator Servers
Backup Engineer & Perform backup and restore HP Data Protector
Administrator activity
Monitoring Engineers & Monitoring System performance HP Openview
Administrator and availability
Security Engineer & Hardening OS servers, HP Security standards and
Administrator SDN Architecture
Firewall Engineer & Maintaining rules for access to a Client compartment, NAT,
Administrator client compartment and high availability
6.1.2 Role Ownership

Owners of the roles required for the delivery of the defined services are listed in the following
table:
Table 2: Role Ownership
Name of Role Owning Group & Leader Job Codes

Name
System Engineer & TBD TBD
Administrator
Storage Engineers & TBD TBD
Administrator
Backup Engineer & TBD TBD
Administrator
Monitoring Engineers & TBD TBD
Administrator
Network Engineer & TBD TBD

HP RESTRICTED
Name of Role Owning Group & Leader Job Codes

Name
Administrator
Security Engineer & TBD TBD
Administrator
Firewall Engineer & TBD TBD
Administrator
6.2 Preparation Tasks

This section describes the tasks required to prepare for the implementation of this solution.
6.2.1 Server Deployment Prepare Tasks

6.2.1.1 Utility Services Cabling and VLAN Preparation
1. Rack Elevation: All devices will be racked following the guidelines in Utility Services
Foundation Architecture. A site specific equipment rack plan must be prepared before
starting implementation. A detailed sample charge is available in the Rack Elevation sheet
in Utility Services 3.1 Cabling and VLAN document. The URL to locate Utility Services 3.1
Cabling and VLAN document is available at Related Documentation section.
2. VLAN and IP scope: The Utility Services design calls for several VLANs are associated
with different IP subnets. IP subnet ranges for the site must be acquired and documented
prior to implementation. A sample plan is included as VLAN Key sheet in Utility Services
R3.2 Cabling and VLANs document. The URL is available at Related Documentation
section.
3. Cabling: Cabling in a new data center is complex. A detailed cabling plan for each device,
such as for Flex-10 VCE uplinks, must be defined prior to installation. For detailed
information about required cabling, refer the sheet Cabling template in Utility Services 3.2
Cabling and VLANs documents. The URL to locate Utility Services 3.2 Cabling and VLAN
document is available at Related Documentation section.
6.2.1.2 Production Rack ports

The Production Rack Ports document provides detailed information about local area network
(LAN) and storage area network (SAN) connections in the Utility Services environment. This
information is used in planning the amount of site structured cable needed, purchasing the
correct number and type of patch cables, and interconnecting standard Utility Services racks.
For more information, refer Utility Services R3.2 Production Rack Port document. The URL to
locate Utility Services 3.2 Cabling and VLANs document is available at Related Documentation
section.
6.2.2 Storage Deployment Prepare Tasks

6.2.3 Network Deployment Prepare Tasks

The following documents must be read carefully before deployment:

• Utility Services Foundational Technical Architecture - Technical Reference Guide, The URL
is available at Related Documentation section.

HP RESTRICTED
• Utility Services Hardware - Software Bill of Materials - The URL is available at Related
• Procurve/Cisco Productions Manual
• ProCurve Support Page
• Cisco Support Page
6.2.4 Software Deployment Preparation Tasks

The following packages are required from Symantec to deploy Veritas Cluster Server and
Symantec Storage Foundation.
• Veritas Dynamic Multi-Pathing (DMP)
• Veritas Cluster Server (VCS)
• Veritas Storage Foundation (SF)
• Veritas Storage Foundation HA (SFHA)
URL for product download

http://www.symantec.com/business/products/trialware.jsp?pcid=pcat_business_cont&pvid=2
03_1

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Implementation Considerations
7 Implementation Considerations
7.1 Sizing Considerations
This section describes the sizing issues that must be addressed in order to ensure a successful
implementation.
Sizing for the cross-site network to support tier4 is discussed in Tier 4 Data Center Network
Interlink Sizing Guidance. The URL to locate Tier 4 Data Center Network Interlink Sizing
Guidance document is available at Related Documentation section.
7.2 Networking Considerations

This section describes the networking issues that must be addressed in order to ensure a
successful implementation.
7.3 Considerations for Applications Required for Capability

This section describes any issues that must be addressed in conjunction with Applications
required for the Capability in order to ensure a successful implementation.
For more information on Management server capacity requirement, refer Management Servers
section in Utility Services Foundational Technical Architecture –Technical Reference Guide. The
URL is available at Related Documentation section.
7.4 Process Considerations

This section outlines considerations regarding the processes associated with the Capability.
These processes include those necessary for implementing and operating the capability as well
as processes required by an agent or end-user to perform daily work utilizing the capability.
Refer to the Utility Services Foundational Technical Architecture –Technical Reference Guide
for the list of processes for this Capability. The URL is available at Related Documentation
section.
7.5 Additional Considerations

This section outlines other considerations not covered in the application, sizing, network, or
process considerations sections.
7.5.1 Veritas Cluster Server

Veritas Cluster Server ID numbers and cluster names couldn’t be the same in particular
environment and needs to be tracked on SharePoint for each customer.
I/O fencing should be used for each offering as a standard to protect data integrity. In case
that customer will not use it the risk acceptance document should be signed for this. Commented [PM1]: Link to document which should
cover this

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Configuration Standards
8 Configuration Standards
This section consolidates all configuration standards. These are then referenced, as
appropriate, in the Technical Reference Guide.
8.1 OS version and VCS

The Tier4 solution for Utility Services 3.2 is based in Red Hat Enterprise Linux Server 5.6. This
has been used to build the cluster nodes and the CP server.
The cluster software used is Veritas Cluster Service and Storage Foundation 5.1 SP1.
8.2 Cluster nodes hardware

Utility Services offer cost and performance server hardware building, which includes ISS
Small/Medium/Large/Extra Large (only physical servers not virtual). For configuration
information, refer Server Architecture section in Utility Services Foundational Technical
Architecture - Technical Reference Guide. The URL is available at Related Documentation
section.
This Tier4 solution uses ISS Medium equipment for the cluster nodes and ISS virtual
equipment in the CP Server.

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Implementation
9 Implementation
9.1 Veritas Cluster Server Implementation
9.1.1 Overview
This section describes the roles, processes, tools, hardware and software used to deliver the
services defined by this feature/attribute.
9.1.2 Implementation Roles

This section identifies the roles that will be required to perform the implementation tasks. The
role descriptions can be found in the Utility Services Foundational Technical Architecture -
Technical Reference Guide. The URL is available at Related Documentation section.
Table 3: Roles for Implementation
Name of Role Short Definition of Role Special skills/training

System Engineer & Maintaining OS environment Windows, Unix, Linux
Administrator
Monitoring Engineers Monitoring System performance and HP Openview
& Administrator availability
Network Engineer & Maintaining network configuration Client compartments,
Administrator Server connectivity, VPN
access, NAT & managed
routers
9.1.3 Implementation Process Diagram

The diagram below shows the relationship between the various components of the
implementation process.

HP RESTRICTED
Figure 2 - Implementation Process Diagram
9.1.4 Implementation Activities

This section describes the activities necessary to implement this solution(s) into either a test
or production environment.
9.1.4.1 Exclude non-LVM disks from multipath

All multipath devices except for BootLUN should be removed from multipathd control.
In /etc/multipath.conf set the exclude rules using wwid option in sections blacklist{} and
blacklist_exceptions{}

HP RESTRICTED
Remove all entries but BootLUN from /etc/multipath.bindings (if more than BootLUN is
allocated from SAN).
Refresh the initrd image to ensure the fixed multipath options are used during system boot.
When done, reload multipathd to apply changes.

HP RESTRICTED
9.1.4.2 Veritas Storage Foundation Installation

Run installer from the distribution source.
# tar -xvzf VRTS_SF_HA_Solutions_5.1_SP1_RHEL.tar.gz

# cd dvd1-redhatlinux/rhel5_x86_64
# ./installer
Select I) Install the product

HP RESTRICTED
Select 3) Veritas Storage Foundation
Accept the License terms:

HP RESTRICTED
Select 2) Install recommended Veritas Storage Foundation rpms
Enter the node names separated by spaces (node1 node2):
Select 1) Setup ssh between the nodes

HP RESTRICTED
Continue with the installation process…, press [Enter] to continue

HP RESTRICTED
Select 2) Enable keyless licensing and complete system licensing later (keyless mode
has been used during PoC)
The valid license key should be entered in production environment either from HP or from
customer, depending on the contract.

HP RESTRICTED
Select 2) SF Enterprise option
Do not enable VVR and choose n

HP RESTRICTED
Review the logs as required located in /opt/VRTS/install/logs.
9.1.4.3 Removing OS disks from VxVM

After installation the system lists all disks, but some of them are showing incorrect status.
There are two disks belonging to system that have to be removed from the system.

HP RESTRICTED
Device ‘cciss/c0d0’ is local disk being used as swap device.
The other device with status ‘online invalid’ is the BootLUN.
To verify compare output from multipath and VxVM, disk WWN and device files must be the
same.

HP RESTRICTED
The SAN boot disk should be renamed to BootLUN to keep the consistency, and along with
local SmartArray device should be excluded from VxVM.

HP RESTRICTED
9.1.4.4 Setting OS variables

MANPATH and PATH have to be updated
# vi /etc/man.config
Add the following entry in the MANPATH section:
MANPATH /opt/VRTS/man
# vi /etc/profile
Add the following entry after the “Path manipulation” section:
PATH=$PATH:/opt/VRTS/bin

HP RESTRICTED
9.1.4.5 Enabling new disks in VxVM

Disks should be renamed from auto-generated DMP names to ‘meaningful’ names, e.g.
W01_ora_01.
For disks, plexes, service groups and other cluster naming follow VCS naming standards
SymantecStorageFoundationHA_NamingStandars

HP RESTRICTED
Initialize disks to be used in VxVM
Note: DMP nodes must have the same name on both cluster nodes
For further setup and configuration of cluster system group and resources refer to Utility
Services R3.2 VCS Operations Guide.

HP RESTRICTED
9.1.4.6 System preparation for VCS

Generate password-less ssh keys on one system and add it to authorized keys. The same keys
should be then transferred to the secondary node and also added to authorized keys.
Execute ssh-keygen and enter root password
Insert the generated key in authorized keys file cat id_rsa.pub >> authorized_keys
9.1.4.7 Veritas Cluster Server Installation
# tar -xvzf VRTS_SF_HA_Solutions_5.1_SP1_RHEL.tar.gz

# cd dvd1-redhatlinux/rhel5_x86_64
# ./installer

HP RESTRICTED
Choose I) Install a product
Choose 2) Veritas Cluster Server (VCS)

HP RESTRICTED
Confirm the End User License agreement by select y
Select 3) Install all Veritas Cluster Server rpms

HP RESTRICTED
Ener the node names separated with space docnlvcs001 wycnlvcs001

HP RESTRICTED
Press Enter to continue
Choose n in dialog for installing additional licenses

HP RESTRICTED
The valid license key should be entered in production environment either from HP or from
customer, depending on the contract.

HP RESTRICTED
9.1.4.7.1 Configure cluster

Run /opt/VRTS/install/installvcs -configure
Enter the node names separated with space docnlvcs001 wycnlvcs001

HP RESTRICTED
Select n by asking to configure I/O Fencing in enabled mode

HP RESTRICTED
Enter the cluster name here CNLVCS001

HP RESTRICTED
Enter cluster unique ID 0
Select 2) Configure heartbeat links using LLT over UDP
Insert NIC, IP address and UDP port for cluster heartbeat links and low latency link

HP RESTRICTED
Select y for continue with heartbeat configuration and confirm the cluster verification by y

HP RESTRICTED
Select y for configure the Virtual IP
Enter the NIC for virtual IP, Virtual IP and NetMask for that IP

HP RESTRICTED
Confirm the information inserted in previous step by selecting y
Select n to skip configuration of VCS to use Symantec Security Services

HP RESTRICTED
Select n to skip configuration of SMTP notification
For adding VCS users select n to leave the default user=admin and password=password and n
to skip additional user creation
Select y to confirm user verification information

HP RESTRICTED
Select y to stop all VCS processes

HP RESTRICTED

HP RESTRICTED
9.1.4.8 Veritas Coordination Point Cluster Server installation

Note: Networking layout for Coordination point (CP) server has to be revisited; proof-of-
concept installation is made using 2 networks – management and backup
Choose I) Install a product
Choose 2) Veritas Cluster Server (VCS) and confirm End user license agreement by
selecting y

HP RESTRICTED
Select 3) Install all Veritas Cluster Server rpms, enter the node name wycvlcp001

HP RESTRICTED

HP RESTRICTED
Select n for Global Cluster Option and y for configure VCS cluster on wycvlcp001
Select n for configure I/O Fencing in enabled mode

HP RESTRICTED
Enter VCSCP001 as unique cluster name and 65535 as unique cluster ID

HP RESTRICTED
Select 2) Configure heartbeat links using LLT over UDP
Insert NIC, IP address and UDP port for cluster heartbeat link and low latency link

HP RESTRICTED
Confirm the information inserted in previous step by selecting y
Select n for configure the Virtual IP

HP RESTRICTED
Select n to skip configuration of VCS to use Symantec Security Services

HP RESTRICTED
9.1.4.8.1 Veritas Coordination Point Server configuration

Select [1] Configure Coordination Point Server on single node VCS system

HP RESTRICTED
Enter name of CP Server cp001 , virtual IP on which CP server process should depend
15.141.111.248, default port 14250, secure communication between CP server and
application clusters select n and Enter for default path of database /etc/VRTScps/db

HP RESTRICTED
Choose eth0 as network interface for virtual IP 15.141.111.248, select y to add NetworkHosts
and 255.255.252.0 as netmask for virtual IP.
9.1.4.9 Configure I/O fencing
The ssh-keys from every node have to be added to the CP server (cp001.centrica.hpecs.net)
prior to start this configuration.
# /opt/VRTS/install/installvcs -fencing

HP RESTRICTED
Select y to configure I/O fencing
Select option 1) Configure CP client based fencing

HP RESTRICTED
Choose y for support SCSI3 PR and press Enter to continue
Enter 3 for number of CP including CP servers and disks, 2 for number of disks

HP RESTRICTED
Enter FQDN of CP server cp001.centrica.hpecs.net, dmp as fencing mechanism for the disk
and enter both disks (W02_lck_01 and D02_lck_01) from the list and Enter continute

HP RESTRICTED
Enter disk group name for coordinating disk vxfencoorddg and confirm with y all above
settings

HP RESTRICTED

HP RESTRICTED
Confirm with y Client cluster verification
Select y to configure CP Agent

HP RESTRICTED
Enter vxfen as service group for CP Agent

HP RESTRICTED
Check cluster with hastatus -summary, hagrp -resources vxfen, hares –state
coordpoint, vxfenadm -d

HP RESTRICTED
9.1.4.10 Adding system groups and resources

Add cluster groups AppSG with following commands
haconf -makerw
hagrp -add AppSG
hagrp -modify AppSG SystemList docnlvcs001 1 wycnlvcs001 2
hagrp -autoenable AppSG -sys docnlvcs001
haconf -dump -makero

HP RESTRICTED
Add resource groups AppSG with following commands
haconf -makerw
hares -add AppDG DiskGroup AppSG
hares -modify AppDG Enabled 1
hares -modify AppDG DiskGroup AppDG
hares -modify AppDG StartVolumes 1
haconf -dump -makero
For further setup and configuration of cluster system group and resources refer to Utility
Services R3.2 VCS Operations Guide.
9.1.5 Transformation Services

For transformation services, see Infrastructure Rationalization services Capability offering.
9.1.6 Event Monitoring Activation

This section describes the monitoring strategy for this solution, identifies the individual
components needed to enable the strategy, and lists the steps necessary to activate
monitoring components.
For monitoring of the VxVM use VOLMON module of UXMON. And for cluster monitoring there
is specific module in UXMON called VCMON. It’s used to monitor resource group status and
switching in a Veritas Cluster system. The main functionalities are to monitor:
• resource group switching enabled

• resource group running (where / at all)
• nodes active

HP RESTRICTED
For details refer to UXMON 02.01.03 User Guide for OMU8 on the link
http://enhanced1.sharepoint.hp.com/teams/GDCC-
OM/Solution%20Repository/UXMON/UXMON_0201/Release%20Document%20Library/UXMON-
02.01.03_OMU8_User_Guide.pdf
Turning monitoring to production is part of release to production acceptance activities, More
detail info refer
http://teams2.sharepoint.hp.com/teams/OBRTPA/RTPA%20Documentation/Forms/AllItems.as
px?View=%7b62F37A60%2d8DEE%2d4FBF%2d88AF%2dE54BC657AECC%7d
9.1.7 Reporting Activation

This section describes the reporting strategy and identifies the reporting features as well as
describes the steps necessary to activate the reporting strategy.
Once server is in production, OVPA and OVSC should send information to CRDW, which is used
by reporting tool, Utility Service production reports can get from below sites:
https://hook.grenoble.hp.com/
https://os-customer-
reporting.emea.hp.com/businessobjects/enterprise115/desktoplaunch/InfoView/main/main.do
?objId=0
http://custom-reports.tools.emea.hp.com/
If the server is not in report, deliver lead should open a ticket with reporting tool support
team.
9.1.8 Cost Recovery Activation

This section describes the volume capture system strategy and identifies the volume capture
elements that need to be produced, how they are captured, and how they are supplied to a
volume capture system.
Include references to information that is in the configuration standards section or in vendor

documentation.
For Applications, this section should reference Applications Delivery standards on time capture,
internal billing, and so on. Details are required if this Capability requires any special
considerations that are not part of the Applications Delivery standards. Consult the Capability
Manager with any questions or to clarify existing standards.
Once servers are in production in ESL, Utility service starts the cross charge
9.1.9 Pre-Acceptance Testing

This section identifies the elements that need to be tested to ensure the solution has been
implemented properly.
Pre-Acceptance Testing is part of Release To Production Acceptance (RTPA) activities, More

detail info refer RTPA checklist at
http://teams2.sharepoint.hp.com/teams/OBRTPA/RTPA%20Documentation/Forms/AllItems.as
px?View=%7b62F37A60%2d8DEE%2d4FBF%2d88AF%2dE54BC657AECC%7d

HP RESTRICTED
The Release to production check is part of the BTR process - RTPA checks. The BTR process
can be found at http://irockweb.gre.hp.com/btr/.
9.1.10 Customer Acceptance Testing

Customer Acceptance testing should be coordinated by the Project Manager.
This section describes the elements involved in Client acceptance testing for this solution.
Detail acceptance testing will be provided by customer.
9.1.11 Operations Turnover

This section describes the elements that must be in place for successful turnover of the
solution to operations for ongoing maintenance and support.
To start operation, RTPA has to be followed and training have to be provided to Level 1, 2 and
3 teams, detail of RTPA process refer http://intranet.hp.com/eds/ww/wg/ito-do/Pages/IO-
RTPAPROCESS.aspx
9.1.12 Backing Off

Sometimes unforeseen factors arise and make it necessary to return a system to its original
state. This section describes the steps necessary to back out the implementation and return
the target environment to its original state.
Backing off is in implementation phase. Utility Service customer can use the same process for
requesting new servers to request Backing off, the percent of backing off depend on the
contact.
9.1.13 De-implementation
Certain factors may materialize that would call for the de-implementation of this solution.
Factors such as Client disengagement, end-of-life for offering, or upgrading to something else.
The DOS and SOW should be referenced to check for any special requirements the Client may
have when returning data to them, including details on the format of the returned data, and
particular requirements regarding the end of the service.
For Server with Windows, Linux or VMware OS installed, the de-implementation process
includes two parts: physical server decommission and virtual machine decommission, detail
info refer Utility Service server decommission process, The URL is available at Related
9.1.14 Completion
This section describes any tasks that are required after successful implementation of this
solution. This section includes the following:
There are no completion tasks.

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Other Deployment Considerations
10 Other Deployment Considerations

This section describes any other activities associated with deploying the solution and making it
ready for production use.

HP RESTRICTED
Utility Service R3.2 - Veritas Cluster Server Deployment Guide Contacts
11 Contacts
This section identifies document contacts for technical changes, business issues, and so forth.
Contact Name Group’s Name E-Mail

Stefan ECS Cloud Service Engineering stefan.landwehrmann@hp.com
Landwehrmann Manager
Cullen Johnson ECS Cloud Service Engineering cullen.johnson@hp.com
Pavol Masarovic ECS Cloud Service Engineering pavol.masarovic@hp.com
Jennifer Vargas ECS Cloud Service Engineering jvargas@hp.com
Andrew Duro ECS Cloud Service Engineering - andrew.duro@hp.com

Project Manager

HP RESTRICTED

Utility Service R3.2 - Veritas Cluster Server Deployment Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Utility Service R3.2 - Veritas Cluster Server Deployment Guide

Uploaded by

Copyright:

Available Formats

HP Enterprise Services

Utility Service R3.2 - Veritas Cluster Server

3 Capability Overview ............................................................................... 6

7 Implementation Considerations ........................................................... 15

8 Configuration Standards ...................................................................... 16

Version#1.0 – Draft 24 Oct 2011 |i|

8.1 OS version and VCS.......................................................................................... 16

10 Other Deployment Considerations ....................................................... 71

Version#1.0 – Draft 24 Oct 2011 | ii |

Version#1.0 – Draft 24 Oct 2011 | iii |

1 Guide Document Control

Approver’s Name Approver’s Group’s Name Content Approved

1.2 Amendment Record

Published / Version Author Section /

1.3 Related Documentation

Document Name Short description of required document

Version#1.0 – Draft 24 Oct 2011 |1|

Document Name Short description of required document

Version#1.0 – Draft 24 Oct 2011 |2|

Veritas Cluster Service and Storage Foundation skill sets

• Familiar with Veritas Cluster Service for Linux

• Familiar with Veritas Storage Foundation for Linux

• Familiar with RHEL 5.6 x86-64

2.3 Terminology and Acronyms

Version#1.0 – Draft 24 Oct 2011 |3|

HP Enterprise The HP Enterprise Services sales or demand catalog formerly known

Version#1.0 – Draft 24 Oct 2011 |4|

Version#1.0 – Draft 24 Oct 2011 |5|

Veritas™Cluster Server by Symantec is a high-availability solution for applications and services

Below the major components diagram:

Figure 1 - VCS cluster components

Version#1.0 – Draft 24 Oct 2011 |6|

4.1 Infrastructure Dependencies for Solution

4.1.1 Physical Dependencies

4.1.1.1 Common Physical Elements

4.1.1.2 Shared Management Server

4.1.1.3 OMC Network

4.1.2 Functional Dependencies

Support organizations involved in the delivery of this service are:

Version#1.0 – Draft 24 Oct 2011 |7|

• Backup and Restore Tower: Management of backup and restore.

Main Roles in the Delivery

4.1.3 Software Dependencies

4.1.3.3 Veritas Cluster Server

Version#1.0 – Draft 24 Oct 2011 |8|

Required Linux RPMs for VCS

Operating System Required RPMs

4.1.4 Process Dependencies

Version#1.0 – Draft 24 Oct 2011 |9|

4.1.4.2 Implementation Processes

Version#1.0 – Draft 24 Oct 2011 | 10 |

5.1 Security Roles

5.2 Security Implementation

5.2.1 Legal and Regulatory Compliance

5.2.2 Vulnerability Scan

Version#1.0 – Draft 24 Oct 2011 | 11 |

6.1.1 Role Definitions

Table 1: Role Definitions

Name of Role Short Definition of Role Special skills/training

6.1.2 Role Ownership

Table 2: Role Ownership

Name of Role Owning Group & Leader Job Codes

Version#1.0 – Draft 24 Oct 2011 | 12 |