You are on page 1of 12

Mokum is the only full-time Oracle virtualization

integrator with the expertise to help you virtualize your


Production, Test and DR Oracle workloads.

sales@mokumsolutions.com
Copyright 2015 Mokum Solutions, Inc. All rights reserved.
Distribution of the Oracle Cloud Cookbook or derivative of the work in any form
is prohibited unless prior permission is obtained from the Copyright holder.
About Mokum Solutions, Inc.
Founded in March 2011, Mokum Solutions, Inc. specializes in the implementation,
delivery and support of Oracle technologies in private and public clouds. Mokum
corporate headquarters are located in San Francisco, CA http://mokumsolutions.com
or call 1 415 252 9164
About the Author
The author of the Oracle Cloud Cookbook is none other than the owner of
Mokum Solutions, Inc., Roddy Rodstein. Roddy is one of the most respected
Oracle Cloud Computing experts, having designed and managed many of the
worlds largest and most complex Oracle private clouds. Before establishing
Mokum in March 2011, Roddy spent three years at Oracle on the Oracle VM
and Oracle Linux team designing and supporting Oracle's largest and most
complex customer environments. Before Oracle, Roddy spent six years at Citrix,
designing and supporting Citrix's largest and most complex customer environments,
Including Oracle's. With Mr. Rodsteins rich background and knowledge, there
can be no better resource for revealing the Oracle Cloud recipe.
Audience
The Oracle Cloud Cookbook is a comprehensive, field tested reference design that
guides you through each step to move to your Oracle software portfolio to an elastic
Oracle cloud using the Oracle VM product line, Oracle Linux, Oracle Engineered
Systems managed by Oracle Enterprise Manager 12c, with total control over Oracle
processor licensing.

http://mokumsolutions.com

Last update: 02/13/2015


This chapter of the Oracle Cloud Cookbook presents Mokum's Oracle Cloud reference design. The Oracle Cloud reference designs encompass the software, hardware, storage, network,
orchestration and management components required to deploy a scalable, secure, and supportable internal or external Oracle cloud.

Table of Contents
The Oracle Cloud Reference Design Introduction
The Oracle Cloud Reference Design Implementation Overview
The Oracle Cloud Reference Design Support Infrastructure
Oracle Cloud Architectural Design
Oracle VM Hardware Architecture
Oracle VM Server Pool Design
Oracle VM Security Standards
Oracle VM Manager Security Controls
Oracle VM Server Security Controls
Virtual Machine Operating System Standards
Oracle VM Disaster Recovery
Oracle VM Application Integration
Change Log
Share on Twitter Share on Linkedin

The Oracle Cloud Reference Design Introduction


The Oracle Cloud reference design is a eld-tested best-practice standard, designed with simplicity, reproducibility, usability, scalability, supportability and security. The Oracle Cloud reference designs represent a complete Oracle Cloud
standard that can be leveraged as a vanilla solution or modied to more accurately reect organization-specic needs. The Oracle Cloud reference design includes the following categories:

Cloud Delivery Model

Infrastructure as a Service

Converged Infrastructure

Management, Orchestration, and Analytics

Virtual Machines

Oracle Enterprise Manager


Oracle VM Manager
Open Source
OpenStack | Puppet | Katello | ManageIQ | GrayLog2

Oracle VM for x86


x86 64 Servers
Storage & Network Services

IaaS

Note: A detailed explanation of each category and solution in the Oracle Cloud reference design is presented in the architectural overview section.

The Oracle Cloud Reference Design Implementation Overview


The Oracle Cloud reference design provides a well dened starting point for each Oracle Cloud implementation. It also serves as a baseline upon which all solution additions, revisions, and tools
will be based. As such, there is an increasing value to Oracle Cloud reference design in keeping implementations as close to the reference design as possible.
Prior to implementing an Oracle Cloud, its important that an infrastructure assessment (IA) and gap analysis (GA) be performed. During the IA/GA, the architecture of the solution will match the
customers business needs while maintaining the integrity of the Oracle Cloud reference design. Implementation and support will follow the analysis phase after careful consideration has been
given to any specic design modications that deviate from the Oracle Cloud reference design.
This document outlines the decision points necessary for implementing the Oracle Cloud reference design. For decisions that rely on preexisting factors or specic organizational needs, the
appropriate best practice will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on
organizational needs, existing architecture, and budget resource availability.
The Oracle Cloud reference design is designed to be scalable and resilient for ease of implementation, high availability, and ease of maintenance for internal and external Oracle clouds. The
complete solution is made up of multiple architectural components that work together to provide exibility and options for self-service Infrastructure as a Service delivery with broad network access,
resource pooling, elasticity, measured service, high availability, security and ease of maintenance. Infrastructure as a Service is the capability to provision and deliver fundamental computing resources as a service to
the consumer (consumer = end users). The Oracle Cloud reference design outlines the decision points necessary for implementing an Oracle VM cloud infrastructure to deliver self-service Infrastructure as a Service using
pre-congured virtual machine templates from the Oracle Enterprise Manager Cloud Control 12c self service portal.

The next Figure shows a high-level overview of the Oracle Cloud reference design components.

The Oracle Cloud reference design isolates Oracle VM server pools into the following four security domains:
Controlled: A controlled security domain is used to restrict access between security domains. A controlled security domain could contain groups of users with their network equipment or a
demilitarized zone (DMZ).
Uncontrolled: An uncontrolled security domain refers to any network not in control of an organization, such as the Internet.
Restricted: A restricted security domain can represent an organizations production, test and development networks. Access is restricted to authorized personnel, and there is no direct
access from the Internet.
Secured: A secured security domain is a network that is only accessible to a small group of highly trusted users, such as administrators and auditors.
Note: The classication of security domains is very similar to data classications. FIPS PUB 199 is the Standards for Security Categorization of Federal Information and Information Systems. FIPS
PUB 199 can be used to determine the security category of systems and within which security domain systems should reside.

The Oracle Cloud Reference Design Support Infrastructure


Support is an integral part of the Oracle Cloud reference design and includes a combination of Oracle support agreements and on-site and o-site support from the implementing party.
Administrators will have several options for support, including live assistance, phone support, and forums.
This table outlines the decision points for the support infrastructure for the Oracle Cloud reference design. For decisions that rely on preexisting factors or specic organizational needs, the
appropriate best practice will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on
organizational needs, existing architecture, and budget resource availability.
Decision Point

Decision

Justication

Mokum Solutions, Inc. +1 415 252-9164

4 of 14

http://mokumsolutions.com

Oracle Support Agreements

Oracle Support Agreements for the Oracle technologies will be active and
up to date.

Support is an integral part of every successful IT project. Oracle support


agreements are necessary to be able to create and manage service requests
as well as to be able to receive software patches and updates from Oracle
Enterprise Manager and My Oracle Support.

On-site and O-site support

On-site and o-site support from the implementing party will be used for
maintenance, site reviews, upgrades, and security audits.

On-site and o-site support from the implementing party for problem
resolution, system maintenance, site reviews, upgrades, and security audits
augments the Oracle support agreement and internal IT operations sta.

Oracle Cloud Architectural Design


The following sections provides the decision matrices for the Oracle Cloud reference design. Implementers of the Oracle Cloud reference design can use the decision matrices as quick reference
guide to identify settings and conguration decisions to be implemented in the environment. These decisions should be carefully analyzed during a gap analysis phase.

Oracle VM Hardware Architecture


The server hardware for your Oracle VM environment is a critical component in the success of your Oracle cloud project. The rst step in selecting an Oracle VM hardware platform is to size the
server hardware, followed by calculating the total number of servers required to be in each Oracle VM server pool. The formula to calculate Oracle VM server sizing is: The total aggregate virtual
machine CPU, RAM and Storage requirements plus your N+x availability requirements provides the total server count along with the hardware requirements.
Oracle VM server sizing is calculated by adding the aggregate CPU, RAM and storage requirements for all of the virtual machines that could run on an Oracle VM server, and then selecting server
hardware with ample CPU, RAM and storage resources. Once the server hardware has been selected, the number of servers in a server pool is calculated by selecting enough servers to support
the aggregate CPU, RAM and storage requirements of all of the virtual machines within a server pool, including the number of additional servers for availability, i.e. HA, Live Migration and
Distributed Resource Scheduling (DRS). Oracle VM server pools that use HA, Live Migration and DRS must have excess CPU and RAM capacity for hardware failures and virtual machine
migrations. The number of network interfaces for an Oracle VM server is determined by the network switch VLAN setup and the total number of Oracle VM management network ports, and the
virtual machine network ports for your environment.
Oracle VM server can be installed on an x86 64 bit server with up to 900 CPU cores or threads, up to 6 TB of RAM, with no limit on the number of network ports. Oracle VM server can be installed
on as little as a 3.2 G partition or disk. Since Oracle VM server can be installed on as little as 3.2 GB of disk, many customers use small ash storage modules or boot from SAN to reduce costs and
complexity.
Tip: I have had the opportunity to support and benchmark Oracle VM server installations on slow single 4 GB SSD Drives (18 MB/second Read Transfer Rate,17 MB/second Write Transfer Rate)
as well as Oracle VM server installations using local 7k, 10k and 15k disks. The read and write performance from either type of Oracle VM server installation disk on the virtual machine repositiry
storage (SAN, NFS, iSCSI, or local disk) from the Oracle VM server and the virtual machines was identical. The disk speed from the Oracle VM server installation does not aect the virtual machine
repositiry storage (SAN, NFS, iSCSI, or local disk) read and write performance.
The next table shows the maximum number of CPUs, RAM and NICs for Oracle VM server release 3.2.x, and 3.3.x.
Item

3.2 Maximums

3.3 Maximums

CPU Cores or Threads

160

900

RAM

4 TB

6 TB

NICs

40

No limit

Oracle VM Server CPU, RAM and storage hardware sizing is calculated by determining the total number of virtual machines CPU, RAM, and storage (I/O and disk) requirements per Oracle VM
server. For example, if a single virtual machine with 16 CPUs, 128 GB RAM, 1 TB of disk space with 1500 IOPS will run on one Oracle VM server, the Oracle VM server hardware should have at
least 16 CPU cores or threads, 130 GB RAM, 1 TB of disk space and the ability to support 1500 IOPS with local or remote storage. If two virtual machines each with 16 CPUs, 128 GB RAM, 1 TB of
disk space with 1500 IOPS will run on one Oracle VM server, the Oracle VM server hardware must have at least 32 CPU cores or threads, 300G RAM, 2 TB of disk space and the ability to support
3000 IOPS with local or remote storage.
A single Oracle VM 3.2.x server can support up to 160 CPU cores or threads, 4 TB of memory with local or remote storage. An Oracle VM server with 4 TB of RAM and 160 CPU cores or threads
could allocate the majority of the 4 TB of RAM and more than 160 CPU cores or threads to running virtual machines. Oracle VM server supports CPU oversubscription. CPU oversubscription
means that an Oracle VM server with 160 CPU cores could overallocate the total number of CPU cores to virtual machines. Oracle VM server does not support memory oversubscription, which
means that an Oracle VM server with 4 TB of RAM cannot overallocate RAM to virtual machines. By default, each Oracle VM server reserves 512 MB of RAM for Oracle VM server (dom0). The
average memory overhead for each running virtual machine on an Oracle VM server is approximately 20 MB plus 1% of each virtual machine' memory allocation. The remaining RAM can be
allocated to virtual machines.
A best practice is to avoid oversubscribing CPU-bound workloads such as the Oracle Database. CPU oversubscription with CPU-bound workloads negatively aects the performance and
availability of an Oracle VM server along with all of the virtual machines running on the server. CPU oversubscription for non-CPU-bound workloads, such as Oracle Fusion Middleware products,
is highly recommended. It is common to oversubscribe CPU cores 3-to-1 with non-CPU-bound workloads. For example, one CPU core could be allocated to 3 virtual CPUs for non-CPU-bound
workloads without a performance penalty.
Note: Virtual machines cannot aggregate CPU and memory resources from more than one Oracle VM server. That is, a virtual machine consumes resources only from the Oracle VM server where
the virtual machine is running.
Oracle VM has two high-availability features, HA and Live Migration. Oracle VM HA and Live Migration along with Distributed Resource Scheduling (DRS) must be considered to calculate the
total number of servers required to respond to hardware failures and virtual machine migrations.
The next Figure shows Oracle VM server pool designed with excess CPU and RAM capacity to be able to use HA, DRS and Live Migration. Excess CPU and RAM capacity is a requirement for HA,
DRS and Live Migration.
This image shows an
Oracle VM server pool
with excess capacity able
to use HA, Live Migration
and DRS.

This image shows an


Oracle VM server
pool responding to a HA
event, with DRS and/or
Live Migration moving
running virtual machines.

This image shows an


Oracle VM server
pool migrating running
virtual machines using DRS
and/or Live Migration.

Oracle VM HA automatically restarts virtual machines when an Oracle VM pool member fails or restarts. Oracle VM HA minimizes unplanned downtime by restarting virtual machines when an
Oracle VM server fails or restarts. Live Migration is used to eliminate planned downtime by migrating running virtual machines from one Oracle VM pool member to another during a maintenance
event, for example, for repairs or an upgrade. DRS is an Oracle VM feature which provides policy based real-time utilization monitoring of Oracle VM servers with the goal to distribute virtual
machine loads across a server pool. DRS migrates virtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM servers. Both HA, Live Migration and DRS require a server
pool with at least three servers with excess CPU and RAM capacity to be able to run and migrate virtual machines across the server the pool even if one Oracle VM servers fails.

Mokum Solutions, Inc. +1 415 252-9164

5 of 14

http://mokumsolutions.com
Tip: There is a known limitation with OCFS2 two node cluster and network failures that cause the node with the higher node number to self-fence. For example, with a two node Oracle VM server
pool, if one node has a network failure that triggers a HA event, both Oracle VM server will reboot. A best practice is to use a minimum of three Oracle VM servers for a server pool to eliminate
the two node OCFS2 limitation.
Oracle VM HA monitors the status of each server pool member using a network and storage heartbeat. If a server pool member fails to update or respond to network and/or storage heartbeats
due to hardware failure, the server pool member is fenced from the pool, promptly reboots, then all HA-enabled virtual machines are restarted on a live node in the pool. Oracle VM does not
support memory oversubscription, which means that an Oracle VM server pool must have sucient RAM capacity to be able to respond to a hardware failure using HA, or to support virtual
machine migrations.
The Oracle VM Live Migration and DRS move running virtual machines between server pool members across a LAN without loss of availability. Live Migration and DRS have two primary use
cases. The rst use case is to eliminate planned downtime by Live Migrating running virtual machines from one server pool member to another during planned maintenance events. The second
use case is to use DRS policies to load balance running virtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM servers. Since Oracle VM does not support memory
oversubscription, an Oracle VM server pool must have available RAM capacity to be able to migrate virtual machines between servers.
DRS is an Oracle VM feature which provides policy based real-time utilization monitoring of Oracle VM servers with the goal to distribute virtual machine loads across a server pool. DRS migrates
virtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM servers.
The exact number of network interfaces for an Oracle VM server is determined by the network switch VLAN setup and the number of Oracle VM management and virtual machine network ports.
Oracle VM supports both 802.1Q trunk port VLANs as well as port based VLANs, with Linux bonding Modes 1 (Active-Backup), 4 (802.3ad) and 6 (Adaptive load balancing). 802.1q trunk ports can
have two or more VLANs per port, in contrast to port based VLANS that are limited to one VLAN per port or port channel. 802.1Q uses fewer network switch ports and fewer Oracle VM server
NICs compared to port based VLANs that require a dedicated switch port and NIC per network. A network switch VLAN conguration must rst be selected to be able to calculate the exact
number of network switch ports and NICs for your Oracle VM servers.
Oracle VM uses a total of ve discrete networks for the Oracle VM server management functions; server management, cluster heartbeat, live migration, storage (only for NFS and iSCSI) and
virtual machines. Each Oracle VM server pool should have a discrete network for each of the ve aforementioned server management networks, as well as a discrete network for each virtual
machine network. For example, an Oracle VM Server on a 1-gigabit copper network with NFS or iSCSI storage could easily use 12 or more bonded NICs with access ports just for the server
management networks and one virtual machine network. In contrast to the latter 1-gigabit copper network example, an Oracle VM Server on a 10-gigabit ber network using 802.1q trunk ports
with NFS or iSCSI storage could easily use up to 4 bonded ports just for the server management and 2 bonded ports for the virtual machine networks.
Tip: In an clustered Oracle VM server pool, the loss of network connectivity for the Oracle VM cluster heartbeat network will causes a HA event. When a HA event occurs, the Oracle VM server
that loses cluster heartbeat connectivity is fenced from the server pool and reboots, then all HA-enabled guests are restarted on a live Oracle VM pool member.
Prior to implementing an Oracle Cloud, its important that an infrastructure assessment (IA) and gap analysis (GA) be performed. During the IA/GA, the hardware specications will be matched to
the customers business needs.
This table outlines the decision points for the for Oracle VM for x86 server hardware. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice
will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs,
existing architecture, and budget resource availability.
Decision Point

Decision

Justication

Certication

The server hardware must be jointly supported by the hardware vendor and
Oracle.

Only jointly supported hardware product receive vendor support when


problems occur and service tickets are created. The server hardware must
be jointly supported by the hardware vendor and Oracle.

Note: The following link is the Oracle' hardware certication page.


http://linux.oracle.com/pls/apex/f?p=117:1:5773793518142288::NO:RP::
CPU

Server hardware will be ordered with two socket Intel or AMD multiple-core The Maximum Number of CPU cores or threads an Oracle VM server can
CPUs for small and medium workloads and four socket multiple-core CPUs
support is 160. Oracle VM server maps a virtual CPU to a hardware thread
for large CPU-bound workloads.
on a CPU core in a CPU socket.
Oracle VM Server supports CPU oversubscription. CPU oversubscription
allows an Oracle VM Server with 160 CPU cores to overallocate the total
number of CPU cores to virtual machines. For example, a server with an
Intel Xeon processor 5600-series CPU with hyperthreading can have up to
six cores and twelve threads per socket. A two socket server with an Intel
Xeon processor 5600-series CPU could allocate twenty four virtual CPUs
without oversubscribing the physical CPUs.
CPU-bound workloads, such as Oracle Databases, should not be on Oracle
VM Servers with oversubscribed CPUs.

RAM

Server hardware will be ordered with the maximum amount of physical


memory.
Note: Oracle VM Server supports up to 4TB of RAM.

Oracle VM Server does not support memory oversubscription. For example,


an Oracle VM Server with 1TB of RAM cannot overallocate RAM to virtual
machines. By default, each Oracle VM Server reserves 512MB of RAM for
dom0. The average memory overhead for each running guest on a dom0 is
approximately 20MB plus 1% of the guests memory size. The remaining
physical RAM can be allocated to guests.
An Oracle VM Server in a server pool with Live Migration, DRS, DPM and/or
HA must have excess RAM capacity to accept virtual machines from a Live
Migration, DRS, DPM and/or HA operation. Oracle VM pool members
without available RAM can not support Live Migration, DRS, DPM and/or
HA. Having available RAM on each server provides exibility in terms of
adding new virtual machines to the server pool, and to allow Live Migration,
DRS, DPM and/or HA within a server pool.

Storage

Unless the Oracle VM server is booting from SAN, redundant SSD internal
hard drives are recomended.
Virtual machine image and conguration les are hosted on shared SAN,
iSCSI, or NFS repositories.

Oracle VM Server requires only 3 GB of local storage for the entire Oracle
VM Server installation. The design goal for Oracle VM is to support multiple
node Oracle VM Server pools with shared bre channel SAN, iSCSI and/or
NFS storage.
Oracle VM supports local storage without HA or Live Migration. With local
storage, the OCFS2 virtual machine le system must be on a dedicated non
SAS hard dirve. For example, a partition on same disk as Oracle VM server
installation is not supported. Local SAS storage for virtual machines is not
supported.

Network Interface Cards

A minimum of one Ethernet network interface (NIC) card is required just to


install Oracle VM Server, although at least four or more 10G NICs is
strongly recommended. NIC bonding with port-based VLANs
and/or 802.1Q tag-based VLANs are supported and congured post Oracle
VM Server installation with Oracle VM Manager or Enterprise Manager.
Oracle VM 3.0.1 through 3.1.1 supports two NIC ports per network
bond, and a total of ve network bonds per Oracle VM Server.
Oracle VM 3.2.x supports four NIC ports per network bond, and a total
of ten network bonds per Oracle VM Server.
Oracle VM 3.3.x supports an unlimited number of NICs, and bonds.
The exact number of network interfaces for an Oracle VM Server entirely
depends on your organizations business requirements, server hardware,
and network and storage infrastructure. For example, there are no NIC
limitations with a Cisco UCS hardware, in contract to legacy hardware with
physical NICs. Cisco UCS supports provisioning as many HA enabled vNICs
as necessary to meet the most demanding Oracle VM network
requirements, in contrast to legacy hardware that could require up to 6 10G
NICs, or 12 or more 1G ports. It is hard to succeed without a plan. Plan your
Oracle VM project in advance before ordering hardware, and deploying

Both 802.3AD NIC bonds, port-based VLANs and/or 802.1Q tag-based


VLANs are supported and congured post Oracle VM Server installation
with Oracle VM Manager. Network redundancy, i.e. 802.3AD NIC bonding
doubles the number of required NICs.
Oracle VM uses a total of ve discrete networks; Server Management,
Cluster Heartbeat, live Migration, Storage and Virtual Machines. All ve
networks can be supported using one or more 802.1Q tag-based VLANs (2
NICs) or using up to ve 802.3AD bond (10 NICs).
Each Oracle VM server pool should have a discrete network for the Server
Management, Cluster Heartbeat, Live Migration, Storage and Virtual
Machines. Isolating the Server Management, Cluster Heartbeat, Live
Migration and Storage networks protect the server pool from unexpected
server reboots by eliminating OCSF2 heartbeat interruptions that could
cause a pool member to loose network connectivity, fence from the pool and
reboot.
Each Oracle VM Server will be assigned a unique IP address on the Server
Management, Cluster Heartbeat, live Migration and Storage network.

Mokum Solutions, Inc. +1 415 252-9164

6 of 14

http://mokumsolutions.com

Oracle VM.
NAME

Rate(bit/s) Rate(byte/s)

Gigabit Ethernet
1 Gbit/s 125 MB/s
10 Gigabit Ethernet 10 Gbit/s 1.25 GB/s
Inniband DDR
16 Gbit/s
2 GB/s
Tip: One thing to consider is NIC rmware levels between bonded internal
NIC ports and PCI NIC ports. Consider only bonding internal NICs with
internal NICs and PCI NICs with PCI NICs.
Host Bus Adapter Cards

SAN Storage: At least 2 Host Bus Adapter Cards (HBAs).


NAME
4GFC
8GFC
10GFC
16GFC
20GFC

2 HBAs are used to eliminate a single point of failure.

Line-Rate Throughput MBps


4.25
8.5
10.52
14.025
21.04

800
1600
2550
3200
5100

Oracle VM Server Pool Design


Oracle VM uses the concept of a "server pool" to group together and centrally manage one or more server pools with up to 32 Oracle VM servers. If more than one location exists, Oracle VM
server pools may be dispersed to dierent locations. Oracle VM Manager with Oracle Enterprise Manager 12c provide a single point of administration for one or more dispersed Oracle VM server
pools.
Oracle VM server pools can accommodates organization-specic needs, i.e., Oracle technology license management (hard and soft partitioning) , defense in depth, the principle of least privilege,
compartmentalization of information, security domains and dierent applications and their performance, authentication, and security requirements.
The next Figure shows a high-level overview of how server pools can be used to implement security domains, defense in depth, the principle of least privilege and compartmentalization of
information.

This table outlines the decision points for an Oracle VM server pool. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice will be discovered
in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existing architecture,
and budget resource availability.
Decision Point

Decision

Justication

Oracle VM Server Pool Design

Prior to implementing an Oracle Cloud, its important that an infrastructure


assessment (IA) and gap analysis (GA) be performed. During the IA/GA, the
architecture of the solution will be matched to the customers business
needs.

Server pool design is a strategic, architectural security decision. Server


pools can be used to control Oracle licensing costs (hard and soft
partitioning) and as a way to implement security domains, defense in depth,
the principle of least privilege and compartmentalization of information.

Oracle VM Manager

The Oracle VM Manager installer provides two installation options. Oracle


VM 3.0.1 up to 3.1.1 oers a Demo or Production installation. Oracle VM
3.2.1 and above oers a Simple or Custom installation.

For large environments (>33 hosts), the Oracle VM Manager Database


repository should be on dedicated virtual or physical servers. If your Oracle
VM environment starts out small and scales out, make sure to have a plan to
scale up Oracle VM Manager with more RAM and CPUs and scale out the
Oracle VM Manager Database repository on dedicated virtual or physical
servers with RAC.

Oracle VM Manager will be installed in Production, Simple or Custom mode


on a dedicated physical or virtual server. Production and Custom mode uses
a local or external Oracle 11g Enterprise or RAC database on a dedicated
physical or virtual server. Simple mode uses a local MySQL database.
The Oracle VM Manager Database repository will not be shared with other
production or test databases on the same server.
The Oracle Enterprise Manager Agent and the Virtualization plug-in will be
installed to enable Oracle Enterprise Manager integration.
Monitoring and Alerting

The Oracle VM product family; Oracle VM Server, Oracle VM Manager,


virtual machines, Oracle VM Templates and Assemblies can be managed and
monitored with Oracle VM Manager and Oracle Enterprise Manager 12c.
Unlike Oracle VM 2.x, which could only be managed by Oracle VM Manager
or Oracle Enterprise Manager, not both, Oracle VM 3 and above can be
managed simultaneously by Oracle VM Manager along with Oracle
Enterprise Manager 12c Cloud Control.

For the Oracle VM Manager Database repository, scaling out means moving
from a single server Database to a multi node RAC cluster. An important
consideration when scaling out an Oracle VM Manager environment is to
determine if the underlying hardware where the Oracle VM Manager
Database repository runs is capable to transition to RAC. If the hardware is
not capable to transition to RAC, it is possible to move and/or export the
Oracle VM Manager Database repository to a dierent system with more
resources.
When things go wrong within an Oracle VM server pool, being able to
quickly determine the root cause of an issue can eliminate or reduce down
time. The most eective way to identify problems with an Oracle VM server
pool is to analyze the Oracle VM Manager, the Oracle VM Servers, and the
virtual machines performance statistics, and log les using Oracle
Enterprise Manager, SNMP based monitoring solutions, and a central log
host.

Oracle VM Manager is a stand-alone management solution for Oracle VM,


with limited monitoring and alerting functionality. Oracle VM is a
default Oracle Enterprise Manager 12c feature that provides Infrastructure
as a Service (IaaS), Database as a Service (DaaS), Platform as a Service
(PaaS) and Testing as a Service (TaaS) provisioning with a self-service
portal. Oracle VM should be enabled in Cloud Control by installing an
Oracle Management Agent with the Virtualization plug-in on a managed
Linux target with Oracle VM Manager. Once Oracle VM is enabled in Cloud
Control, Oracle VM Manager, Oracle VM Servers, and all the virtual
machines can be managed, and setup with performance monitoring proles
and alerts that can be used for root cause and statistical analysis.
A central log host should be congured to capture the Oracle VM Server,
the Oracle VM Manager, and the virtual machine operating systems log
les.
Network Time Protocol (NTP)

With Oracle VM, accurate time is essential to maintain system stability due
to time-sensitive cluster transactions between Oracle VM Servers. Without
accurate time, Oracle VM clusters can be brought to a complete standstill.

With Oracle VM, accurate time is essential to maintain system stability due
to time-sensitive cluster transactions between Oracle VM Servers. Without
accurate time, Oracle VM clusters can be brought to a complete standstill.

By default, Oracle VM Servers (up to Release 3.1.1) that are discovered by


A best practice is to set-up the Oracle VM Manager hosts as the upstream
Oracle VM Manager are congured to use the Oracle VM Manager host as
NTP time host to synchronize with upstream Coordinated Universal Time
the upstream NTP time host. A best practice is to set-up the Oracle VM
(UTC) sources as well as provide time services to Oracle VM Servers.
Manager hosts as the upstream NTP time host to synchronize with upstream
Coordinated Universal Time (UTC) sources as well as provide time services
to Oracle VM Servers.

Mokum Solutions, Inc. +1 415 252-9164

7 of 14

http://mokumsolutions.com

Oracle Linux and Red Hat Enterprise Linux ship with a default /etc/ntp.conf
le that points to three of Red Hat's upstream public UTC time sources. A
best practice is to have two internal NTP servers on your local network to
provide time services for internal systems and devices. Using internal time
servers normalizes system event time-stamps across the Enterprise as well
as reduces NTP Internet bandwidth usage.
Oracle VM Server Agent Roles

Oracle VM Manager facilitates centralized management of server pools and


their resources using an agent-based architecture. When an Oracle VM
server is added to a server pool, up to three Oracle VM agent roles can be
enabled. There are a total of three Oracle VM agent roles; 1) Master Server,
2) Utility Server and 3) VM Server. When an Oracle VM server is added to a
server pool, it can be assigned one, two, or all three of the agent roles.
Master Server
The Master Server is the principal server pool role within a server
pool. The Master Server is the server that communicates with Oracle
VM Manager. The Master Server dispatches commands received from
Oracle VM Manager to other servers within a server pool. There can
be only one Master Server in a server pool at any instant. The Virtual
IP feature is a mandatory server pool property that detect the loss of
the server pool master agent and responds with automatic failover to
the rst pool member that can lock the pool le systsm. The server
pool Virtual IP feature removes the single point of failure (SPOF) for
the server pool master agent role.

Master Server
By default each clustered server pool has one Master Server with the
Virtual IP feature enabled.
Utility Server
The Utility Server role is responsible for I/O-intensive operations such as
virtual machine creation and removal, as well for as creating, deleting,
modifying, copying and moving virtual machine les. Enabling the Utility
Server agent role with the VM Server role on the same Oracle VM server
may negatively aect running virtual machines during Utility Server
operations. Server pools that are not static and support the self service
portal in Oracle Enterprise Manager Cloud Control 12c should have one or
more dedicated utility servers to isolate the impact of I/O intensive
operations to Utility Servers.
VM Server
Unless a server pool is static, VM Servers should only have the VM Server
role enabled to be able to dedicate CPU, RAM and I/O resources to running
virtual machines, eliminating the eect of Utility Server operations.

Utility Server
The Utility Server role is responsible for I/O-intensive operations such
as virtual machine creation and removal, as well for as creating,
deleting, modifying, copying and moving virtual machine les.
The Master Server dispatches operations to Utility Servers. There can
be one or more Utility Servers in a server pool. When there are
multiple Utility Servers in a pool, the server Master Server will select
the least loaded utility server to conduct a task.
Tip: Oracle VM environments that dynamically grow should have one
or more dedicated Utility Servers to isolate I/O jobs to Utility Servers.
For example, an I/O intensive that runs on an Oracle VM Server with
the VM Server and Utility Server role will impact the performance of
all of the virtual machines running on the Oracle VM Server.
VM Server
Servers with the VM Server role are responsible for allocating CPU,
memory, and disk resources to the virtual machines in a server pool.
There can be one up to 32 VM Servers in a server pool.
Storage

Back-end storage
Each Oracle VM server pool uses one dedicated OCFS2 12G mount point
(OCFS2 or NFS) for the server pool's cluster congurations and one or more
shared OCFS2 or NFS repositories to host virtual machine conguration
les and images.
Front-end storage
The virtual machine layer is where the storage is presented to virtual
machines as either a at le (UUID.img), as RAW disks (LUN), or as a
combination of at les and RAW disks.

An Oracle VM storage solution consists of three distinct layers. Each layer


has its own unique requirements, congurations, dependencies and
features.
The rst layer is the storage array, which is referred to as back-end
storage. Oracle VM supports Fibre Channel and iSCSI SAN and NFS
back-end storage.
The second layer is the server layer consisting of the Oracle VM Server's
Device-Mapper Multipath congurations and the shared Oracle Cluster File
System 2 (OCFS2) or NFS virtual machine le system.
Note: OCFS does not factor disk space exhaustion including space for
virtual machine les as well as volume metadata. OCFS2 metadata can
consume over 6% of an OCFS2 volumes free disk space.
The third layer is the guest front-end storage consisting of multiple guest
storage (le and RAW) and driver options. RAW disks have the best
performance of the two front-end storage storage options. In most cases,
RAW disks are the best option for high I/O workloads like Oracle Databases.

Networks

Each Oracle VM server pool will have isolated Oracle VM management


networks and isolated virtual machine networks.
Oracle VM uses a total of ve discrete networks; Server Management,
Cluster Heartbeat, live Migration, Storage and Virtual Machines.
The exact number of network interfaces for an Oracle VM Server entirely
depends on your organizations business requirements and network and
storage infrastructure capabilities. For example, an Oracle VM Server with
four 10G NICs, congured with two 802.1Q bonds could support the most
demanding network and storage requirements, with only four 10G NICs. By
contrast, an Oracle VM Server using access ports/port-based VLANs
or 802.1Q tag-based VLANS on a 1G copper network, could easily use the
maximum number of supported NIC ports (<= 3.1.1 = 10 ports, >= 3.2 =
40 ports) to meet the minimum network requirements.

Each Oracle VM server pool should have a discrete network for the Server
Management, Cluster Heartbeat, live Migration, Storage and Virtual
Machines. Isolating the Server Management, Cluster Heartbeat, live
Migration and Storage networks protect the server pool from unexpected
server reboots by eliminating OCSF2 heartbeat interruptions that cause
pool members to fence from the pool and reboot.
Each Oracle VM Server will be assigned a unique IP address on the Server
Management, Cluster Heartbeat, live Migration and Storage network.
Note: The heartbeat trac is TCP on port 7777. Each Oracle VM server in
a pool must be able to communicate to all of the pool members over TCP on
port 7777.

RAM

The server pool must be designed with excess RAM capacity to


accommodate the memory requirements of virtual machines that could
migrate or start on any pool member.

Oracle VM server does not support memory oversubscription, which means


that an Oracle VM server cannot accept a DRS, Live Migration or HA
requests unless the server has available RAM for the virtual machines.
Having excess RAM on each Oracle VM server provides exibility in terms
of adding new virtual machines to the server pool, and to allow DRS, Live
Migration and HA to operate within a server pool.

NUMA

Contemporary CPUs from Intel and AMD have NUMA architectures. NUMA
stands for Non-Uniform Memory Access. With NUMA each physical CPU
(pCPU) will be assigned its own local memory. An assigned processormemory pair is called a NUMA node. Local memory access from CPUs on
the same socket will have signicantly lower latency than remote memory
access from CPUs on a dierent socket.

If your supporting virtual machines with more vCPUs than its NUMA node, disable NUMA. For
example, Xen NUMA aware scheduling will place a virtual machine with 32 vCPUs on a single
NUMA node, even if the node does not have 32 cores or threads.

Oracle VM supports NUMA using a Xen feature called NUMA aware


scheduling. NUMA aware scheduling will assign a virtual machine's vCPUs
(virtual CPUs) to a NUMA node as a NUMA client. If a virtual machine has
multiple vCPUs, the NUMA scheduler will always assign the virtual
machine's vCPUs to a single NUMA node to maintain memory locality. For
example, an Oracle Database virtual machine with 32 vCPUs allocated to a
single NUMA node with 20 threads would be oversubscribed. CPU-bound

Mokum Solutions, Inc. +1 415 252-9164

8 of 14

http://mokumsolutions.com

workloads, such as Oracle Databases, should not be on Oracle VM Servers


with oversubscribed CPUs.

Oracle VM Security Standards


The security controls used to secure Oracle VM are similar to the security controls used to protect your existing physical and virtual IT resources. As with physical and virtual IT resources,
securing Oracle VM is dependent on the security posture of each of its components, from the design, hardware, hypervisor, network, and storage to the virtual machine operating systems and
installed applications. In short, if the organization has a security policy for virtualization, networking, storage, operating systems and applications, the security policies could be applied to Oracle
VM.
Security controls should be employed using industry standard frameworks and standards in the context of the organization's Enterprise Architecture (EA). Organizations turn to their Enterprise
Architecture to understand how Oracle VM ts within their information system. An Enterprise Architecture is articulated in diagrams and written policies that dene organizational standards and
best practices to plan, build, run, and monitor technologies, including Oracle VM.
Enterprise Architecture has well dened principles and processes and an approach that generates a comprehensive, layered policy infrastructure used to communicate managements goals,
instructions, procedures, and response to laws and regulatory mandates. A policy infrastructure consists of written tier 1, tier 2, and tier 3 policies that encompass people, systems, data, and
information. Policies are broken down into high level policies and lower level standards, procedures, baselines, and guidelines.
Oracle VM policies typically fall within the layered policy infrastructure of the platform architecture domain. Platform architecture policies are the foundation used to manage the entire lifecycle
of an Oracle VM environment.
This table outlines the decision points for Oracle VM Manager security controls. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice will be
discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existing
architecture, and budget resource availability.

Oracle VM Manager Security Controls


Decision Point

Decision

Oracle VM Manager and DMZs

The Oracle VM Manager application was not designed to be an Internet


The Oracle VM Manager application was not designed to be an Internet
facing application. If Internet access is a requirement for Oracle VM
facing application.
Manager, VPN access should be used to access the Oracle VM Manager GUI.

Justication

Network Time Protocol (NTP)

With Oracle VM, accurate time is essential to maintain system stability due
to time-sensitive cluster transactions between Oracle VM Servers. Without
accurate time, Oracle VM clusters can be brought to a complete standstill.

With Oracle VM, accurate time is essential to maintain system stability due
to time-sensitive cluster transactions between Oracle VM Servers. Without
accurate time, Oracle VM clusters can be brought to a complete standstill.

By default, Oracle VM Servers (up to Release 3.1.1) that are discovered by


A best practice is to set-up the Oracle VM Manager hosts as the upstream
Oracle VM Manager are congured to use the Oracle VM Manager host as
NTP time host to synchronize with upstream Coordinated Universal Time
the upstream NTP time host. A best practice is to set-up the Oracle VM
(UTC) sources as well as provide time services to Oracle VM Servers.
Manager hosts as the upstream NTP time host to synchronize with upstream
Coordinated Universal Time (UTC) sources as well as provide time services
to Oracle VM Servers.
Oracle Linux and Red Hat Enterprise Linux ship with a default /etc/ntp.conf
le that points to three of Red Hat's upstream public UTC time sources. A
best practice is to have two internal NTP servers on your local network to
provide time services for internal systems and devices. Using internal time
servers normalizes system event time-stamps across the Enterprise as well
as reduces NTP Internet bandwidth usage.
Virtual Machine Console Access

Oracle VM uses the RAS proxy (Remote Access Service) java applet to proxy
virtual machine console trac from Oracle VM Manager to the
administrator's Client PC. An Oracle VM Manager administrative account is
a requirement to access a virtual machine's console. Any rewall between
Oracle VM Manager and the administrator's Client PC conecting to a virtual
machine console must have TCP port 15901 open for the RAS proxy.
Oracle VM Manager does not support role based access control. All
administrative users with access to the Oracle VM Manager GUI have root
administrative access to all of the objects managed by Oracle VM Manager,
including all of the virtual machine consoles. If an Oracle VM Manager
account is not an option for a user, for example for DBAs, Opertaions. or
application administators, Oracle VM role based access control can be
congured using Enterprise Manager Cloud Control. With Cloud Control,
each object managed by Oracle VM Manager can be congured with role
based access control, including each virtual machine console.

Host rewall

Host rewall failed connection


logging

All Oracle VM administrative users have root access to all of the objects
managed by Oracle VM Manager. Virtual machine end users such as DBAs
and application administrators should only have access to thier virtual
machines, not root access to all of the objects managed by Oracle VM
Manager. End user access to virtual machines can be controled using
Enterprise Manager Cloud Control. With Cloud Control, each object
managed by Oracle VM Manager can be congured with role based access
control, including each virtual machine console.

The iptables service should be enabled on each Oracle VM Manager host


using a ruleset managed in /etc/syscong/iptables. In order to use Oracle
Host rewalls, for example iptables, are a fundamental part of information
VM Manager, the Core API and the Oracle Management Agent with iptables,
security that protect hosts from attacks and intrusions.
it is necessary to open tcp ports 7001, 7002, tcp-54321 or tcps-54322, 15901
and 3872 as well as UDP 123.
Iptables failed connection logging should be enabled on each Oracle VM
Manager host.

Failed connect logging is a fundamental part of information security that


allows detection of attacks and intrusions.

The following two lines will be added prior to the last REJECT line in the
/etc/syscong/iptables le:
-A RH-Firewall-1-INPUT -m limit --limit 15/minute -j LOG
--log-prex "FW Drop:"
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-hostprohibited
Systems administrators should access the Oracle VM Manager host
with non-root individual user accounts and use sudo to perform selected
administrative tasks. Sudo stands for either "substitute user do" or "super
user do".
Root ssh access should be disabled on the Oracle VM Manager host. Sudo
should should be used to congure ne-grained permissions to allow
administrative users to perform selected administrative tasks with logging.
Root ssh access and sudo

Disables Root Access:


To disable root ssh access, edit the default /etc/ssh/sshd_cong le and
uncomment the
the #PermitRootLogin yes line and change the yes to no; that is,
PermitRootLogin no. Next, restart the sshd service by typing service
sshd restart to enable the change.

By default, Oracle Linux permit ssh access using the root super user
account.
One of the most important security measure that can be taken with Oracle
VM is to prevent unauthorized access to the root user account by disabling
root ssh access. A best practice is to only provision non-root individual user
accounts that can be audited, disabled, expired and managed using sudo.
Note: All sudo user access will be tracked and logged in the /var/log/secure
le.

The visudo command is used to edit the /etc/sudoers le. Consult the
sudoers man page for sudo conguration details.
Pre and post SSH login banners should be congured on each Oracle VM
Manager host.
SSH login banners

Pre-login banner:
Edit the /etc/ssh/sshd_cong and add the following directive:

To be able to successfully prosecute individuals who improperly use a


computer, the computer must have a warning banner displayed at all access
points.
SSH login banners presents a denitive warning or disclaimer to all users

Mokum Solutions, Inc. +1 415 252-9164

9 of 14

http://mokumsolutions.com

Banner /etc/banner.net
Next, create the /etc/banner.net le and add your login banner text, i.e.
This system is restricted to authorized access only. All activities on this
system are recorded and logged. Unauthorized access will be fully
investigated and reported to the appropriate law enforcement agencies.
Once the le has been created and the banner text is added and saved,
restart the sshd by typing:
# service sshd restart

that wish to access your systems using SSH. SSH login banners should
clarify which types of activities are illegal as well as advise legitimate users
of their obligations relating to the acceptable use of the system.

Post login banner:


Edit /etc/motd and add your login banner text, i.e.
This system is restricted to authorized access only. All activities on this
system are recorded and logged. Unauthorized access will be fully
investigated and reported to the appropriate law enforcement agencies.
Once the le has been edited and saved, restart the sshd by typing:
# service sshd restart
Central log host

A central log host should be used to log all user logins and iptables
connection failures.

Centralized logging for user logins and iptables connection failures


simplies security management for the detection of attacks and intrusions.

This table outlines the decision points for Oracle VM Server security controls. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice will be
discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existing
architecture, and budget resource availability.

Oracle VM Server Security Controls


Decision Point

Decision

Oracle VM Server and DMZs

Oracle VM Servers hosting Internet facing virtual machines can be placed in Oracle VM Servers in a DMZ should be restricted from inbound and
a DMZ without connectivity to the Internet or internal network segments to outbound Internet connectivity to reduce the attack surface.
reduce the attack surface. TCP/8899 is necessary to and from the Oracle VM
Servers to Oracle VM Manager to enable centralized management using
Oracle VM Manager.

Justication

Build Process

Before any Oracle VM Servers are placed on the production network, a


standard build processes should be executed to ensure that all Oracle VM
Servers are installed, congured and maintained in a manner that prevents
unauthorized access, unauthorized use and disruptions in service.

An Oracle VM Server build document provides employees with an approved


procedure to install and congure Oracle VM Server. An Oracle VM Server
build document is used with other IT infrastructure policies to address
interoperability and security of Oracle VM in the context of the entire
information system.

Patch Management

A key component of a successful Oracle VM deployment is acquiring and


vetting new releases, patches and updates for production systems. New
Oracle VM releases, patches and updates must be researched to identify
which release, patches and updates are applicable to your environment.
Newly released versions, patches and updates should be vetted before being
deployed into production. A best practice is to run the latest stable release
of Oracle VM.

A patch management program is an integral component of an organization's


information security program used to mitigate the risk from security
vulnerabilities (bugs) that are inherent in all operating systems and
applications.

Oracle VM Servers should be congured to use local custom yum


repositories. Local yum repositories with point-in-time static channel for
each supported Oracle VM release ensures all like Oracle VM server are
patched in a consistent manner across the organization.
All patches should be regression tested in the lab environment before they
are deployed on production systems. High-priority patches, security xes,
and upgrades will be applied as needed in accordance with <Company
Name>s Change Management Policy.

A key component of patch management is acquiring and vetting patches for


production systems. Patches must be researched to identify which patches,
security xes, and updates are applicable to your environment. Newly
released patches, security updates, and application updates will be tested
before being deployed in to production using time stamped local custom
repositories.
Pre- and post-production audits will be conducted in accordance with
<Company Name>s Change Management Policy to validate conguration
and patch compliance.

All production systems should undergo security audits in accordance with


<Company Name>s Change Management Policy to validate conguration
and patch compliance.
Host rewall

The iptables service will be enabled on each Oracle VM server using the
default policy and ruleset in /etc/syscong/iptables.

Host rewalls, for example iptables, are a fundamental part of information


security that protect hosts from attacks and intrusions.

Host rewall failed connection


logging

Iptables failed connection logging will be enabled on the Oracle VM


Manager host and each Oracle VM server.

Failed connect logging is a fundamental part of information security that


allows detection of attacks and intrusions.

The following two lines will be added prior to the last REJECT line in the
/etc/syscong/iptables le:
-A RH-Firewall-1-INPUT -m limit --limit 15/minute -j LOG
--log-prex "FW Drop:"
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-hostprohibited
Systems administrators should access Oracle VM Servers with non-root
individual user accounts and use sudo to perform selected administrative
tasks. Sudo stands for either "substitute user do" or "super user do".
Root ssh access should be disabled on the each Oracle VM servers. Sudo
should should be used to congure ne-grained permissions to allow
administrative users to perform selected administrative tasks with logging.

Root ssh access and sudo

Disables Root Access:


To disable root ssh access, edit the default /etc/ssh/sshd_cong le and
uncomment the
the #PermitRootLogin yes line and change the yes to no; that is,
PermitRootLogin no. Next, restart the sshd service by typing service
sshd restart to enable the change.

By default, Oracle VM Server permit ssh access using the root super user
account.
One of the most important security measure that can be taken with Oracle
VM is to prevent unauthorized access to the root user account by disabling
root ssh access. A best practice is to only provision non-root individual user
accounts, that can be audited, disabled, expired and managed using sudo.
Note: All sudo user access will be tracked and logged in the /var/log/secure
le.

Note: To enable sudo on Oracle VM Servers, it is neccessary to install


the ovs-support-tools meta-package that includes sudo.
The visudo command is used to edit the /etc/sudoers le. Consult the
sudoers man page for conguration details.
Pre and post SSH login banners should be congured on each Oracle VM
Manager host and Oracle VM Server.

SSH login banners

Pre-login banner:
Edit the /etc/ssh/sshd_cong and add the following directive:
Banner /etc/banner.net
Next, create the /etc/banner.net le and add your login banner text, i.e.

To be able to successfully prosecute individuals who improperly use a


computer, the computer must have a warning banner displayed at all access
points.
SSH login banners presents a denitive warning or disclaimer to all users
that wish to access your systems using SSH. SSH login banners should
clarify which types of activities are illegal as well as advise legitimate users
of their obligations relating to the acceptable use of the system.

Mokum Solutions, Inc. +1 415 252-9164

10 of 14

http://mokumsolutions.com

This system is restricted to authorized access only. All activities on this


system are recorded and logged. Unauthorized access will be fully
investigated and reported to the appropriate law enforcement agencies.
Once the le has been created and the banner text is added and saved,
restart the sshd by typing:
# service sshd restart
Post login banner:
Edit /etc/motd and add your login banner text, i.e.
This system is restricted to authorized access only. All activities on this
system are recorded and logged. Unauthorized access will be fully
investigated and reported to the appropriate law enforcement agencies.
Once the le has been edited and saved, restart the sshd by typing:
# service sshd restart
Note: By default Oracle VM Server's /etc/motd le displays the following
warning message: Warning: making manual modications in the
management domain
might cause inconsistencies between Oracle VM Manager and the server.
Rootkit prevention and monitoring

Wikipedia describes a rootkit as A rootkit is software that enables


continued privileged access to a computer while actively hiding its presence
from administrators by subverting standard operating system functionality
or other applications.

Monitoring the hypervisor (Oracle VM Server) for rootkits is fundamental


part of information security used to detect rootkits to prevent attacks and
intrusions. Each Oracle VM Server should have a rootkit prevention solution
in place, such as chkrootkit, that monitors the host for rootkits.

A Hypervisor (Oracle VM Server) may be one of the most sensitive operating


systems in the data center because it controls the hardware as well as all of
the virtual machines on it. If the hypervisor is compromised direct access to
the hardware and all of the virtual machines is possible, and other code
could be monitored and controlled by the attacker.
Central log host

A central log host will be used to log all user logins and iptables connection
failures.

Centralized logging for user logins and iptables connection failures


simplies security management for the detection of attacks and intrusions.

Virtual Machine Operating System Standards


This table outlines the decision points for the for virtual machine operating systems hosted on Oracle VM. For decisions that rely on preexisting factors or specic organizational needs, the
appropriate best practice will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on
organizational needs, existing architecture, and budget resource availability.
Decision Point

Decision

Virtual Machine Operating Systems

Standardizing on a small number of virtual machine operating systems


A small number of virtual machine operating systems should be be used. For
streamlines operations and increases the level of le duplication on
example, stadardizing on Oracle Linux 6 latest, in contract to supporting
production and archival virtual machine data. This design reduces
Oracle Linux 5U2, 5U3, 5U4, 5U5, 5U6, 5U6, 5U8, 5U9, 5U10, 6, 6U1, 6U2,
complexity and increases operational eciency by limiting the number of
6U3, 6U4, etc...
supported operating systems.

Justication

Virtual Machine Operating System


Versioning

In accordance with <Company Name>s Application Software Policy and


Application Software Standards, applications will determine the operating
system type and version.

Virtual Machine Operating System


Deployments

All new virtual machine operating systems will be deployed using a virtual
machine template in accordance with <Company Name>s Server Policy,
and Server Security Policy.

Linux virtual machines will be congured to use local custom yum


repositories.

Patch Management

All patches will be regression tested in the lab environment before they are
deployed on production systems.
High-priority patches, security xes, and application upgrades updates will
be applied as needed in accordance with <Company Name>s Change
Management Policy.

Each application has an operating system support matrix that lists the
supported operating systems, patch levels, and software prerequisites.
In accordance with <Company Name>s Application Software Policy and
Application Software Standards, applications will determine the operating
system type and version.
A virtual machine template is a self-contained, precongured virtual
machine with an operating system and optionally an application installed in
accordance with <Company Name>s Server Policy, Server Security Policy,
and Operating System Installation Guidelines.
Each time a new virtual machine is deployed using a virtual machine
template, <Company Name>s standards are applied to each new virtual
machine.
A key component of patch management is acquiring and vetting patches for
production systems. Patches must be researched to identify which patches,
security xes, and application updates are applicable to your environment.
Newly released patches, security updates, and application updates will be
tested before being deployed in to production using time stamped local
custom repositories.

Noncritical xes will be applied on a Quarterly basis in accordance with


<Company Name>s Change Management Policy.

Local yum repositories will be maintained for patch testing and production
using a point-in-time static channel for each supported operating system to
ensure all like operating systems are patched in a consistent manner across
the organization.

All production systems will undergo security audits in accordance with


<Company Name>s Change Management Policy to validate conguration
and patch compliance.

Pre- and post-production audits will be conducted in accordance with


<Company Name>s Change Management Policy to validate conguration
and patch compliance.

Oracle VM Disaster Recovery


An Oracle VM disaster recovery architecture includes the design and process to maintain business continuity following a disastrous event aecting the availability of an organization's primary
site. Failover to a disaster recovery site is prompted by the results of a disaster assessment. The failover process is the restoration of the primary site's services at the disaster recovery site.
Note: Disaster recovery requirements are calculated using Service-level Agreements (SLA), Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs) objectives. SLA, RPO and RTO
objectives and budget inuence the disaster recovery architecture and design.
Oracle VM uses the concept of a server pool to group together and manage one or more clustered Oracle VM servers. Once an Oracle VM server pool is created, the physical and virtual resources
are managed within the boundary of the server pool. Physical resources include server hardware, networks, storage, infrastructure services (DNS, NTP, LDAP, HTTP, etc..), operating system
installation media and administrative accounts. The virtual resources include virtual disks, virtual network interfaces, and virtual machine conguration les. For example, an Oracle VM
environment with multiple server pools located in one or more sites could be managed from a single Oracle VM Manager instance with each server pool's resources isolated to their respected
server pool. An Oracle VM server pool's resources from one site can be replicated and restored to another site for disaster recovery.
Restoration of the primary site's services at a disaster recovery site requires a replica of the primary site's physical and virtual resources at the disaster recovery site. A disaster recovery site
hosts a replica of the primary site's Oracle VM physical and virtual resources, i.e. server hardware, networks, storage, infrastructure services, virtual disks, and virtual machine conguration les.
The failover process involves restoring the primary sites virtual machines at the disaster recovery site, then systematically starting the virtual machines and services.
Note: Oracle VM Servers are not backed up and restored at the DR site. The time required to backup and restore an Oracle VM Server is signicantly greater then a PXE boot kickstart
installation.
A disaster recovery site can be a warm failover site waiting idle to respond to a disastrous occurrence, or part of a multi-site high availability design. A multi-site design uses excess capacity with
application high availability to mirror services across sites to handle the lose of one or more sites.

Mokum Solutions, Inc. +1 415 252-9164

11 of 14

http://mokumsolutions.com

The next Figure shows a warm Oracle VM failover site waiting idle to respond to a disastrous occurrence.

The next Figure shows a warm Oracle VM failover site responding to a disastrous occurrence and running the primary sites services.

The next Figure shows a multi-site Oracle VM design with application high availability solutions to mirror services across sites as well as excess capacity to handle the lose of one or more sites.

Virtual machines that are restored at a disaster recovery site expect the same networks, storage, and infrastructure services as in the primary site. In the event that the disaster recovery site has
dierent networks, storage, and infrastructure services, the properties of each virtual machines would need to be edited to use the new networks, storage and infrastructure services before
services can be restored.
The virtual machine operating systems are typically installed in virtual disks that are actually at les hosted on shared OCFS2 or NFS repositories. RAW disks such as ASM Disks, Log and
Archive Files, etc.. are presented to the virtual machines from the Oracle VM Servers as local devices. Each virtual machine's virtual network interface card(s) (vNIC) are connected to one or
more discrete networks using Xen bridges that are managed and presented to the virtual machines by the Oracle VM pool members. Virtual disks and virtual network interface card(s) allocations
are managed using Oracle VM Manager and/or Oracle Enterprise Manager with the congurations saved in each virtual machines vm.cfg le.
The virtual machine vm.cfg les, virtual disk images and RAW disks (ASM disks) can be replicated between sites using storage array replication and/or mirroring solutions. Rsync is an option if an
array does not have replication and/or mirroring functionality.
As soon as the replicated storage repositories are available, the failover process for a warm recovery site starts with the installation of Oracle VM Manager with the runInstall.sh --uuid option
using the primary sites Oracle VM Manager UUID. An Oracle VM Manager --uuid installation allows Oracle VM Manager to use the primary site' replicated repositories with the virtual machines.
Tip: The Oracle VM Manager UUID is listed in the .cong le on the Oracle VM Manager host in the /u01/app/oracle/ovm-manager-3/ directory as well as in each server pool' .ovsrepo le in the

Mokum Solutions, Inc. +1 415 252-9164

12 of 14

http://mokumsolutions.com
pool le system.
The next example shows the content of the .cong le with the UUID in bold.
# cat /u01/app/oracle/ovm-manager-3/.cong
DBHOST=localhost
SID=orcl
LSNR=1521
APEX=None
OVSSCHEMA=ovs
WLSADMIN=weblogic
OVSADMIN=admin
COREPORT=54321
UUID=0004fb00000100009edfaa0f93184f44
BUILDID=3.0.3.126
The next example shows the content of the .ovsrepo le with the UUID in bold.
# cat .ovsrepo
OVS_REPO_UUID=0004fb0000030000554308a6997a6b2f
OVS_REPO_MGR_UUID=0004fb00000100009edfaa0f93184f44
OVS_REPO_VERSION=3.0
This table outlines the decision points for an Oracle VM disaster recovery solution. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice will
be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existing
architecture, and budget resource availability.
Decision Point

Decision

Justication

Disaster Recovery Design

Prior to implementing an Oracle VM Disaster Recovery solution, its


important that an infrastructure assessment (IA) and gap analysis (GA) be
performed. During the IA/GA, the architecture of the solution will be
matched to the customers SLA, Recovery Time Objectives (RTOs) and
Recovery Point Objectives (RPOs) objectives.

Implementing a Disaster Recovery is a strategic decision. Disaster recovery


requirements are calculated using SLA, Recovery Time Objectives (RTOs)
and Recovery Point Objectives (RPOs) objectives. SLA, RPO and RTO
objectives and budget inuence the disaster recovery architecture and
design.

Oracle VM Manager

Oracle VM Manager will be installed in Production mode using the


As soon as the replicated storage repositories are available, the failover
runInstall.sh --uuid option with the primary site's Oracle VM Manager UUID. process for a warm recovery site starts with the installation of Oracle VM
Manager with the runInstall.sh --uuid option using the primary sites Oracle
Oracle VM Manager will be hosted on a dedicated physical server using an
VM Manager UUID. An Oracle VM Manager --uuid installation allows Oracle
external or local Oracle 11g Standard, Enterprise or RAC database.
VM Manager to use the primary site' replicated repositories and virtual
machines.
Once Oracle Enterprise Manager is restored, the Oracle Enterprise
Manager Agent and Virtualization plug-in will be installed to enable Oracle
The Oracle VM Manager UUID is listed in the .cong le on the Oracle
Enterprise Manager integration.
VM Manager host in the /u01/app/oracle/ovm-manager-3/ directory as well
as in each server pool' .ovsrepo le in the pool le system.

Oracle VM Server Builds

Oracle VM Servers will be installed using an automated build process.

Oracle VM servers are installed using an automated PXE boot conguration


to ensure that each server has a consistent installation conguration.

Oracle VM Server Backups

Oracle VM Servers will not backed up at the primary site and restored at
the DR site.

The time required to backup and restore an Oracle VM Server is


signicantly greater then an automated PXE boot kickstart installation.
Oracle VM servers are installed using an automated PXE boot conguration
to ensure that each server has a consistent installation conguration.

Storage

A replica of the primary site's repositories with the virtual machine


resources and RAW disks will be hosted at the disaster recovery site.

As soon as the replicated storage repositories and RAW disks are available,
the failover process for a warm recovery site starts with the installation of
Oracle VM Manager with the runInstall.sh --uuid option using the primary
sites Oracle VM Manager UUID. An Oracle VM Manager --uuid installation
allows Oracle VM Manager to use the primary site' replicated repositories
and virtual machines.
Virtual machines that are restored at a disaster recovery site expect the
same storage as in the primary site. In the event that the disaster recovery
site has dierent storage each virtual machine would need to be recreated
or edited to use the new storage before services can be restored.

Networks

A replica of the primary site's Oracle VM networks will be maintained at the


disaster recovery site.

Virtual machines that are restored at a disaster recovery site expect the
same networks as in the primary site. In the event that the disaster recovery
site has dierent networks each virtual machine would need to be edited to
use the new networks before services can be restored.

Infrastructure Services

A replica of the primary site's infrastructure services will be maintained at


the disaster recovery site.

Virtual machines that are restored at a disaster recovery site expect the
same infrastructure services as in the primary site. In the event that the
disaster recovery site has dierent infrastructure services, each virtual
machine operating system would need to be edited to use the new
infrastructure services before services can be restored.

Oracle VM Application Integration


This table outlines the decision points for an Oracle VM hosted application integration. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice
will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs,
existing architecture, and budget resource availability.
Decision Point
Application Support

Application Requirements and


Dependencies

Decision

Justication

Applications must be supported by the independent software vendor (ISV)


on the latest version of Oracle VM to be included in the Oracle VM
environment.

Only applications that are supported by Oracle on Oracle VM should be


hosted on Oracle VM. Only applications with ISV support for the latest
version of Oracle VM can be deployed and supported by <Company Name>,
the ISV, and Oracle on Oracle VM.

Applications will be analyzed for requirements and dependencies and tested


in accordance with <Company Name>'s Software Installation Standards.

Applications will be analyzed for requirements and dependencies and tested


to ensure compliance with ISV specications.

Applications should be installed and packaged using Oracle VM Template


Builder or Oracle Enterprise Manager.
Application Installation, Packaging,
and Distribution

Applications that are installed and packaged using Oracle VM Template


Builder will be deployed as Oracle VM templates.

Application installations that are packaged in Oracle VM templates or


deployed using Oracle Enterprise Manager have a consistent installation
conguration.

Applications that are installed using Oracle Enterprise Manager will be


installed on Oracle VM templates.

Application sunsetting

Applications will be sunsetted in accordance with <Company Name>'s


Hardware and Software Sunset Policy

Applications that have reached the end of their life cycle and are no longer
supported by a vendor will be given a sunset date. The sunset date is when
the product is scheduled to be removed from production.
Sunsetting applications that have reached the end of their life cycle results
in better customer service and reduced costs.

Mokum Solutions, Inc. +1 415 252-9164

13 of 14

http://mokumsolutions.com

All patches will be regression tested in the lab environment before they are
deployed on production systems in accordance with <Company Name>s
Change Management Policy.
Patch Management

Noncritical xes will be applied on a Quarterly basis in accordance with


<Company Name>s Change Management Policy.

A key component of patch management is acquiring and vetting patches for


production systems. Patches must be researched to identify which patches,
security xes, and application updates are applicable to your environment.
Newly released patches, security updates, and application updates will be
tested before being deployed in to production using time stamped local
custom repositories.

All production systems will undergo security audits in accordance with


<Company Name>s Change Management Policy to validate conguration
and patch compliance.

Pre- and post-production audits will be conducted in accordance with


<Company Name>s Change Management Policy to validate conguration
and patch compliance.

Change Log
Revision

Change Description

Updated By

Date

2.0

Document Creation

Roddy Rodstein

09/30/13

2.1

Content Refresh

Roddy Rodstein

01/14/14

2.2

NUMA Design Details

Roddy Rodstein

02/13/15

Mokum Solutions, Inc. +1 415 252-9164

14 of 14