You are on page 1of 35

RUN BOOK AUTOMATION

WITH HP OPERATIONS ORCHESTRATION


04.10.2011

Stefan Zahrer

Agenda

Why introduce Run Book Automation


Planning
Tool selection and appraisal
Experience so far
RBA Design Framework
Summary

Titel der Prsentation und Name des Redners

30.09.2011

WHY INTRODUCE RUN BOOK AUTOMATION

RBA implementation project objective

1. Main objective: Improve RUN efficiency in Infrastructure Operations reduce cost


2. Develop a RBA platform for all global data centers
3. Establish a RBA service

Quality criteria

Improve provisioning speed and quality

Make automation available to all services

Ensure reusability of flows and low maintenance effort

Quickly deliver financial benefits

Oh, almost forgot: Make this happen in a infrastructure that was


build over the last 10 years without automation in mind

Titel der Prsentation und Name des Redners

30.09.2011

Vision: Fully automated provisioning no hands-on tasks

Manual Provisioning

A
u
t
o
m
a
t
i
o
n

Rapid Provisioning
High Standardization
Up-to-date Documentation
Reliable Quality No Human Errors
Low Process Costs
No Workload Buildup
Less Operational Management

Industrialized Provisioning
Fixed Service Offering
Self-service Portal with service request dialog
Detailed execution logs
Reliable lead time and quality
Titel der Prsentation und Name des Redners

30.09.2011

PLANNING

Planning

2010

2011

2012

Establish RBA Service

POC for
tool selection
Pilot
Virtual Windows Server

Management
Decision

Other Flows and Automations


(HP SM Interface, etc.)

Review

RBA Design Framework Iterations

More Automations in
Munich

Automations for
global
Datacenters

Automations in the making


Installation of Managed virtual Linux Server
Installation of Oracle Databases
Promote AD Member Server to Domain
Controller
Automation for HVD data store administration
(later reuse for Netapp Storage London)

Titel der Prsentation und Name des Redners

30.09.2011

TOOL SELECTION AND APPRAISAL

HP OO wins in proof of concept

3 tools in POC
5 automation tasks to
demonstrate capabilities
Only HP OO can complete all 5
tasks within a set time

Category

HP OO

Accomplishment of mandatory workflows

++

++

Accomplishment of optional workflow

++

Skills of personnel (Number of personnel)

++

++

+++

++

++

Number of pre-assembled connectors and potential


workflow development effort

Overall quality of results


Pricing

HP OO tool appraisal after nine month of flow


customizing
In general very happy with HP OO tool!!!
Easy to use
Flow customizing more efficient than classical scripting
Program control in graphical flows is easy to understand
HP OO Central Web Portal
More functionality for OO Central user input dialog
Calendar for date selection, set default values, radio button, etc.
Fix one or two bugs OO Central
HP OO Studio features to improve
Flow synchronization and staging is complex
Replace file system based repositories with database repository with staging
functionality
Coordination of shared flow customizing is difficult
Improve check-in/check-out functionality
Feature to synchronize system accounts between repositories
More transparency on flow permissions, better handling
Non-functional
Further development internet community and knowledge base
Optimize support reaction times
Experienced HP OO experts in short supply
Accelerator Packs and Integrations are like coding
no des
simple
Titelexamples
der Prsentationbut
und Name
Redners drag&drop
30.09.2011

10

EXPERIENCE SO FAR

People

People like their jobs sometimes not readily offering information/support


For automation we need to now how things are done
but also why they are done this way understanding
both takes time
Automation requires cross-service collaboration
Initial concern about automation going berserk
In the beginning people were critical my job is to difficult for automation

Automation without peoples support may be possible but effort explodes

Stefan Zahrer

30.09.2011

12

Technology

Infrastructure is complex very, very complex indeed


Ask for picture of the architecture discover its in
peoples heads
Work on production systems as early as possible lab
systems are not the same
Architecture must be designed with automation in mind

Automation can be driver for standardization

Stefan Zahrer

30.09.2011

13

Run Book Automation is organizational change for all


IT functions
Automation Benefits
High Standardization
Fixed Service Offering
Fast and reliable
provisioning

Develop
People

Develop
Standards
Application Development
Identical processes for all
projects
Develop Self-Service
provisioning know-how
Standardize architecture to
a higher level

Infrastructure Operations
Design overarching
processes and consistent
infrastructure
Cross-service collaboration
Work on architecture, not in
run operations
Share responsibility for
automated service delivery

Develop
Business
Model

Outsourcing Provider
Collaborate in automation
design, build and release
Include automation in
continuous service
improvement process
Support operators to
become experts with crossservice knowledge
Develop Qualification

Technical Infrastructure
Standardization on all
levels (Storage, Server,
OS, DB, )
Administration designed for
automation (tool-based
decisions, etc.)

Offshoring
More advanced
administration tasks
Better understanding of
infrastructure required
Participation in automation
design

Documentation
Documentation in
automation tool
Consistent and reliable
Focus on technical
administration processes
Titel der Prsentation und Name des Redners

30.09.2011

14

RBA DESIGN FRAMEWORK

The Gordian Knot how to ensure reusability and


maintainability
HP OO flows execute functions in infrastructure systems based on organizational processes
Automation Flow
Automation Flow

Automation Flow
Automation Flow

Infrastructure

Process sequence and technical operations are closely interwoven

What happens if things were to change?


In the infrastructure: new software version, feature changes, new systems,
In the organization: Changes to processes, new CMDB attributes,
Titel der Prsentation und Name des Redners

30.09.2011

16

Major topics of RBA Design Framework

Flow Authoring

ITIL Integration

Start

Start of scheduled Controller Flow


Select and Start UserInterface-Flow in HP OO
Central

Check if respective Change is canceled

Input in User-InterfaceFlow of HP OO
Canceled?

System Call to I3S


(Create/Save Change)

User Interface (UI)


Controller (CO)
Utility (UT)

Flow Architecture
Flow Call Relationships
Naming Conventions
Authorization Model

Optional: Subcontroller (SC)


Action Plan (AP)

Yes

Abort
Report in Private Database

No
No

CI still exists in I3S


Change?

Change Management
Incident Management
CMDB Maintenance
Downtime Management
Freeze times, veto,

System Call to I3S to


verify Change
Relevance

No
Yes

Check if running in
Downtime Window
Mode (Flow Input)

Change Relevant
Flag set?

Yes
Yes

Check if Downtime is
still valid
Input Risk- and Impact-Parameters
No
Yes

Read Input Values from Private


Database

Time is agreed

Execution Time?

Do Work on CI

Downtime Window

Input Parameters for


Time is agreed (Start
Time/Finish Time)
Input Parameters for Downtime
Request

Update Data in Private DB

Document in I3S

Fetch Downtimes via I3S for first


task

Start immediately after aproval

Exited with Error?

Create Incident

yes

no

Check for last CI


Possibility that
downtime window
is not available

yes

Close Task
no

Submit Change

Exit

Writing of Variables in Private


Database

End of User Interface Flow

Aproval Process

First Automated Task (Status assigned)

I3S creates XML-File

Infrastructure Interface (II)

Scheduled File Check picking up XML-File


Start of Scheduling Flow
- Acceptance of I3S Task
- Searching for Downtime of every CI in I3S
- Scheduling of Controller Flow for every CI at defined
Downtime or execute immediately

Private Database

Flow Staging and Deployment

Advanced logging / reporting


Documentation of user input
Infrastructure Topology Model

Flow Repositories
Flow Release Process
Transactional Data
Persistence Table

Server Name
Table
Logical
Logging

Master Data
Key Value Store

Infrastructure Topology
Model

Titel der Prsentation und Name des Redners

30.09.2011

17

SUMMARY

Summary

Tools are (finally) ready for large style automation


Vendors starting to include interfaces for automation
in their products
Automation of existing infrastructure is possible
to large extend
Long time risk in using tools without
architecture model in place

Implementation effort is significant


Financial benefit/ROI depends on number of runs

Stefan Zahrer

30.09.2011

19

Questions

Stefan Zahrer

30.09.2011

20

BACKUP

Flow Architecture separates process control and


system control

User Interface (UI)

Utility (UT)

Controller (CO)

Creates GUI for users


Populates selection lists
Verifies user input
Checks user permissions
Controls processing order
of tasks

Optional: Subcontroller (SC)

Combines tasks on different


systems to reduce complexity
of Controllers

Action Plan (AP)

Combines functions to execute


a complex task on one system

Infrastructure Interface (II)

Executes a single function on


one infrastructure system

Titel der Prsentation und Name des Redners

30.09.2011

22

Characteristics of flow types

Flow Type

Complexity

Reusability

Modify and Test for


change to

User Interface

Very High

Very Low

- User input
Medium
- Infrastructure topology
- Organizational Process
Can be tested without affecting
infrastructure

Controller

High

Very Low

Organizational Process

High

Subcontroller

Medium

Low

Technical Process

High

Action Plan

Medium

High

Infrastructure systems and


applications

Low

Infrastructure Interface Low

High

Infrastructure systems and


applications

Low

Utility

High

New HP OO versions only

Low

Low

Titel der Prsentation und Name des Redners

Modify/Test
Effort

30.09.2011

23

Rules for flow calls limit complexity

User Interface (UI)


Scheduler

Persistence
Table

Utility (UT)

Controller (CO)

Optional: Subcontroller (SC)

Action Plan (AP)


Execute Configuration Changes

Action Plan (AP)

Infrastructure Interface (II)

Infrastructure Interface (II)


Execute Configuration Changes

Gather information
Create i3S Changes
Logging

Private DB
Logging, etc.

Infrastructure Systems

Titel der Prsentation und Name des Redners

30.09.2011

24

Private Database enhances HP OO standard features

Transactional Data
IP-NameTable
Persistence Table

Server Name Mgmt.


more to come

Flow run specific variables

Logical Logging

Reporting of success/failure

Detailed reporting for services

Documentation of user input

Handover variables for scheduled runs

Basis for result reporting

Master Data
Key Value Store
Global Constants

Infrastructure
Topology Model

Infrastructure Topology Model

Dependencies in infrastructure (Locations,


Networks, AD Entities, SW Packages, order,
preferences, etc.)

Supports guided user input

Reduces flow complexity

Titel der Prsentation und Name des Redners

30.09.2011

25

Infrastructure Topology Model stores information in a


structured easy to maintain form
RBA_Entities_RBA_Entities_RBA_Entities_Extended_Attributes
ID
int
<pk>
Entitiy_Entitiy_Entity_ID
int
<fk>
Name
varchar(512)
Value
varchar(512)
Description
varchar(512)

Main purposes
Enhance User Input flows
Content of list boxes, default selections
Limit selections to valid combinations

RBA_Entities_RBA_Entities_RBA_Entities
ID
int
<pk>
Entity_ID
int
<fk1>
Entity_Entity_ID
int
<fk2>
Description varchar(512)

RBA_Types
ID
int
<pk>
Type
varchar(512)
Description
varchar(512)

RBA_Entities
ID
int
<pk>
Type_ID int
<fk>
Name
varchar(512)
Description
varchar(512)

RBA_Entities_Extended_Attributes
ID
int
<pk>
Entity_ID int
<fk>
Name
varchar(512)
Value
varchar(512)
Description
varchar(512)

Dynamic selections in Controller flows based on


System architecture, location, usage,
Preferences (e.g. prefer one ESX Cluster
whenever possible)

Control software installation flows


Software versions/editions
Installation order
Required and optional packages

Staging data from DEV to PRD

RBA_Entities_RBA_Entities
ID
int
<pk>
Parent_Entity_ID
int
<fk2>
Child_Entity_IDint
<fk1>
Description
varchar(512)

RBA_Entities_RBA_Entities_Extended_Attributes
ID
int
<pk>
Entitiy_Entity_ID
int
<fk>
Name
varchar(512)
Value
varchar(512)
Description varchar(512)

No hard encoded values in flows

Reduce flow complexity data sets defined by SQL queries

Less need to change to flows edit database or SQL statement only

Database changes are immediately available in all affected flows


Titel der Prsentation und Name des Redners

30.09.2011

26

Interaction between flows and Private DB

User Interface Flows


Dialog Step 1
SQL

SQL

Dialog Step 2

Execution Flows
Summary
Step

SQL

Controller Flow
HP OO
Scheduler

SQL

Private DB
Logical
Logging
Table

Persistence
Table

Infrastructure Topology Model

Titel der Prsentation und Name des Redners

30.09.2011

27

Dependencies virt. Windows Installation simple view

Vmware
Network
Label

AD
Domains

Network
DNS
Domains

Vmware
Datastore
Suffix

Network
Segments

Vmware
Cluster

Vmware
Datastore
Exclude

Sites

Vmware
Datacenter

Vmware
vCenter

Location
Name

Hosting
Site

Windows
SW Depot

Windows
Server
Packages

Windows
Server
Role

Usage

Cost
Center

Backup
Environment

Type

Vmware
Resource
Pool

Windows
Server OS
Edition
Job Mgmt
Server
Windows
Server
Image

Hostname
Prefix

Windows
Server OS
Version

Windows
Server
Add.
Groups

Owner
Group

Titel der Prsentation und Name des Redners

30.09.2011

28

USER EXPERIENCE AND REPORTING

Input dialog with list boxes supports the user input


and avoids typos
Step 1

Step 2

Step 5: Input Summary

Step by Step
Input

Most input is selected


from list boxes
Comments support user

Titel der Prsentation und Name des Redners

30.09.2011

30

Only valid combinations can be selected less room


for errors
Step 1

Step 2

Limited
selections
Infrastructure
Topology Model

Cost Center and description


from database

Titel der Prsentation und Name des Redners

30.09.2011

31

Detailed logical reporting of all changes to


infrastructure
UUID = 7a18cc03-5cba-4d2b-9eab-1b594f3b38bc
Timestamp
RunID
7/21/2011 2:27:50 PM -30
7/21/2011 2:27:50 PM -30
7/21/2011 2:27:51 PM -30
7/21/2011 2:27:51 PM -30
7/21/2011 2:27:51 PM -30
7/21/2011 2:27:51 PM -30
7/21/2011 2:27:51 PM -30
7/21/2011 2:27:51 PM -30
7/21/2011 2:27:51 PM -30

ChangeID
C1234567
C1234567
C1234567
C1234567
C1234567
C1234567
C1234567
C1234567
C1234567

Requester
n059307
n059307
n059307
n059307
n059307
n059307
n059307
n059307
n059307

CI

mucs801126
mucs801126
mucs801126
mucs801126
mucs801126
mucs801126
mucs801126

Status
Start
Start
Success
Start
Success
Success
Start
Start
VM not found

Flow
CO.windows.vm.create
AP.infoblox.getNewHostname
AP.infoblox.getNewHostname
AP.infoblox.getIpByName
AP.infoblox.getIpByName
AP.dtm.addDownTimeWindow
SC.vmware4.vm.deployHost
AP.vmware4.vm.getDetails
AP.vmware4.vm.getDetails

7/21/2011 2:27:51 PM -30

C1234567

n059307

mucs801126

Start

AP.vmware4.customizationspec.modify

7/21/2011 2:27:52 PM -30

C1234567

n059307

mucs801126

Success

AP.vmware4.customizationspec.modify

7/21/2011 2:27:52 PM -30

C1234567

n059307

mucs801126

Start

AP.vmware4.datastore.getLargest

7/21/2011 2:27:54 PM -30

C1234567

n059307

mucs801126

Success

AP.vmware4.datastore.getLargest

7/21/2011 2:27:54 PM -30


7/21/2011 2:27:55 PM -30

C1234567
C1234567

n059307
n059307

mucs801126
mucs801126

Start
Success

AP.vmware4.vm.deploy
AP.vmware4.vm.deploy

7/21/2011 2:27:55 PM -30

C1234567

n059307

mucs801126

Start

AP.vmware4.task.waitUntilFinished

7/21/2011 2:46:34 PM -30

C1234567

n059307

mucs801126

Success

AP.vmware4.task.waitUntilFinished

7/21/2011 2:46:34 PM -30

C1234567

n059307

mucs801126

Start

AP.vmware4.vm.setCpuAndMemorySize

7/21/2011 2:46:49 PM -30

C1234567

n059307

mucs801126

Success

AP.vmware4.vm.setCpuAndMemorySize

7/21/2011 2:46:49 PM -30


7/21/2011 2:46:59 PM -30
7/21/2011 3:05:18 PM -30
7/21/2011 3:05:18 PM -30

C1234567
C1234567
C1234567
C1234567

n059307
n059307
n059307
n059307

mucs801126
mucs801126
mucs801126
mucs801126

Start
Success
Failure
Failure

AP.vmware4.vm.setPowerState
AP.vmware4.vm.setPowerState
SC.vmware4.vm.deployHost
CO.windows.vm.create

Comment
Start of VM Create

Start of VM Deploy

Windows Deployment failed!


ServerDeploy failed!

Titel der Prsentation und Name des Redners

30.09.2011

32

HP OO executes functions on infrastructure


components
HP OO flows

Infoblox
Network

DTM

i3S Change

OVO
Monitoring

planned
SCCM

HP OO

i3S CMDB

$Univers
Job Mgmt

VMware
vSphere

i3S Incident

SCOM
Monitoring

Netapp
Storage & Fabric

AD
Directory Service
HP SA

Networker
Backup
Windows / Linux
OS

DPPM

Reuse for other datacenters requires standards


Similar infrastructure systems and architecture
Management and administration of systems designed to support automation
Common ITIL management tools and processes
Titel der Prsentation und Name des Redners

30.09.2011

33

Status flow customizing

Topic

Scope
Released for Munich DC

Installation of managed virtual Windows servers


Users: IT Operations

mucs- and mums-servers in most common networks


Not covered: Add new server to DTM Tool
Add new server to Networker DB

Reserve IP Address and IP Name for mucs/-mums-Servers


Users: dedicated Team members

Permissions currently limited to four networks/VLANs

Create/revert/delete VMware Windows snapshots


Users: BA Service

Limited usage, make available to DIPRO/AO.NET?


Planned

Installation of virt. W2S Servers


Users: DMG, AI
Installation of Managed virtual Linux Server
For SAP virtualization 50 70 Linux installations required from November
Users: Linux Service, SAP Basis
Installation of Oracle Databases
Users: Oracle Service
Promote AD Member Server to Domain Controller
Users: AD Service
Automation for HVD data store administration (later reuse for Netapp Storage London)
Users: Storage Service, Windows Service
Automation of simple changes to $Universe backup jobs
Users: Job Mgmt. Service, SAP Basis

Titel der Prsentation und Name des Redners

30.09.2011

34

THANK YOU

You might also like