OPENSTACK FOR VSPECIALISTS

MODULE 5: OPENSTACK COMPUTE SERVICE (NOVA)

EMC CONFIDENTIAL—INTERNAL USE ONLY 1

MODULE 5: OPENSTACK COMPUTE (NOVA)

Upon completion of this module, you should be able

to:
• Understand the concepts of the OpenStack compute
service
• Explain how Nova schedules compute resources
• Diagram the principal components of Nova and
explain their functions

EMC CONFIDENTIAL—INTERNAL USE ONLY 2

BASIC CONCEPTS: INSTANCES
Ironic
• Nova’s principal function is providing HTTP
compute ‘instances’ on demand, drawing
on a fleet of compute hosts nova-
compute
• Instances are usually VM’s, but could also
be LXC containers or even bare metal libvirt
servers, depending on Nova compute
driver configuration
VM Container
– Bare metal provided via Ironic project
– Nova containers != Magnum containers KVM LXC
• Cloud admins can set instance resource
quotas per project
– E.g. instance count, vCPUs, RAM

EMC CONFIDENTIAL—INTERNAL USE ONLY 3

WHAT HYPERVISORS ARE IN USE?

• KVM is the predominant hypervisor deployed

• QEMU is quick emulator often used with KVM
• Features available in compute service depend on
hypervisor(s) in use
• Hypervisor support matrix shows available
features, with mandatory featureset including
initial EC2 operation set

Source: OpenStack User Survey, October 2015

EMC CONFIDENTIAL—INTERNAL USE ONLY 4

HYPERVISOR SUPPORT MATRIX

• See the webpage for

full hypervisor data

EMC CONFIDENTIAL—INTERNAL USE ONLY 5

BASIC CONCEPTS: VM FLAVORS
• Flavors are resource templates for VMs, specifying name and resource
allocation (e.g. RAM allocation, vCPU allocation, boot ephemeral disk size, secondary
ephemeral disk size)

• Cloud admins can create and manage flavors

– Flavors can be public, or private with projects requiring explicit access
– Control access via nova flavor-access-add <flavor> <tenant_id>
– Can define metadata (extra_specs) on flavors, influencing scheduling

EMC CONFIDENTIAL—INTERNAL USE ONLY 6

BASIC CONCEPTS: VM IMAGES
• Images are bootable copies of the entire contents of a storage device
including partitions, boot sector, FAT, and operating system installation
– May also include application installations on top of the OS

• Images stored and retrieved via the Glance service API

• Common server boot options are from image or bootable volume
• Nova supports creating new images from snapshots of existing VM’s
– nova image-create [--show] [--poll] <server> <name>

• Labs use tiny Linux image called CirrOS, designed as test cloud guest
with minimal footprint

EMC CONFIDENTIAL—INTERNAL USE ONLY 7

 BASIC CONCEPTS: VM KEYPAIRS
• Nova has integrated management of
keypairs for SSH access to instances
• Keypair consists of public key and
private key
– Public key injected into instance at
boot via cloud-init
– Private key saved locally to .pem file
and used with SSH client for root
authentication to server

• Keypairs can be created and managed

or imported from existing public key
• Store private keys in a safe place!

EMC CONFIDENTIAL—INTERNAL USE ONLY 8

BASIC CONCEPTS: VM STORAGE
• VM’s can be associated with storage of two
types
– Ephemeral storage is managed by Nova and
Ephemeral persists for the life of the instance. It is
Cinder
Volume destroyed when the VM is deleted. Presented
as a filesystem to the VM.
– Block storage provided by Cinder is persistent
independent from the instance. Storage
volumes(s) are presented as block devices.

VM • Default boot disk for an instance is ephemeral,

unless using bootable volume
KVM
• Ephemeral could be DAS on compute node, or
shared storage
– KVM instances in /var/lib/nova/instances
Compute Node
EMC CONFIDENTIAL—INTERNAL USE ONLY 9
BASIC CONCEPTS: VM NETWORKING
• VM’s also need networking to get anything
done, outside of simple console access (via
nova-novncproxy)
eth0
• Nova supports two major modes of
networking: nova-network and Neutron

External • Nova-network: used but legacy, under threat

network of deprecation (scalability, resiliency, not
VIF really self-service- cloud admins manage all
VM networks)

KVM • Neutron networking: Networking aaS, allowing

tenants to model complex networks
• This course focuses on Neutron and ignores
Compute Node nova-networking
EMC CONFIDENTIAL—INTERNAL USE ONLY 10
BASIC CONCEPTS: VM IP ADDRESSES
• Under Neutron, a VM can have multiple
Local External interfaces on multiple user-specified networks
domain network
• The Neutron DHCP agent will assign a ‘fixed’
or private IP to each interface on a network
• In addition, a user may assign a floating IP to
Private Floating IP: an interface from an available pool on a
IP: 192.168.2.8
10.0.12.8
network
VIF
• Typically, private IPs used for communication
VM between VM’s in the same L2 domain provided
by a virtual switch
KVM
• Floating IPs used for accessing an instance
from a public network, usually, via routed
traffic
Compute Node
EMC CONFIDENTIAL—INTERNAL USE ONLY 11
BASIC CONCEPT: VM SECURITY GROUPS
• Every instance in Nova is associated with at
least one project security group (default if
none specified, called ‘default’)
• When using Neutron networking, Neutron
actually controls the security groups, but
management is also proxied through the Nova
API
– /etc/nova/nova.conf: security_group_api=neutron

• Security group is a collection of rules dictating

what traffic can ingress and egress a port
• Default security group drops all ingress traffic
and allows all egress (outbound) traffic
– User can change rules in project’s default group
– User can add other security groups

EMC CONFIDENTIAL—INTERNAL USE ONLY 12

BASIC CONCEPTS: RESOURCE MODEL
• Regions: Separate Nova deployments
Region with independent API endpoints, which
may use unified or federated Identity
Cell
AZ AZ AZ • Cells: Optional segmentation of
Host Host Host Host Host Host compute into autonomous pools of
resources under single API. Each cell
Host Host Host Host Host Host
has own Nova deployment, excepting
Host Host Host Host Host Host nova-api service. Two-level scheduling
of requests to cell and then to host.
Cell • Availability Zones: Public logical
AZ AZ AZ separation of resources within Nova
Host Host Host Host Host Host deployment for isolation or redundancy.
Host Host Host Host Host Host • Host Aggregates: Logical grouping of
Host Host Host Host Host Host hosts in Nova by admins with common
features for scheduling purposes.
Host
EMC CONFIDENTIAL—INTERNAL USE ONLY Aggregate 13
BASIC CONCEPT: INSTANCE METADATA
• Nova provides RESTful API for retrieving instance metadata (e.g. SSH
keys, hostname, user data) on a magic URL - http://169.254.169.254
• Supports a native OpenStack API and an EC2-compatible API
• API can be explored from an instance’s command line via curl:

EMC CONFIDENTIAL—INTERNAL USE ONLY 14

KEY SERVICE OPERATIONS
$ openstack flavor create …
• Nova API defines a huge number of $ openstack flavor list…
operations, but functionality can depend
on your deployment (e.g. VM evacuation, $ openstack keypair create…
migration) $ openstack keypair list …
• CRUD for VM flavors, flavor access lists
$ openstack server list …
(CUD for admin only)
$ openstack server create …
• CRD for instance keypairs $ openstack server reboot …
$ openstack server delete …
• CRUD on instances $ openstack server pause …
• Instance run and power state controls $ openstack server unpause
– Pausing stores state of VM in RAM $ openstack server suspend…
– Suspending stores state of VM on disk $ openstack server resume …

EMC CONFIDENTIAL—INTERNAL USE ONLY 15

KEY SERVICE OPERATIONS
• Instance VNC console access – $ openstack server console…
out of band (useful!)
$ openstack server image create…
• Create image from running
server (stored via Glance) $ openstack add volume …
$ openstack remove volume…
• Associate/disassociate Cinder
volumes from instances $ openstack add security group…
• Associate/dissociate security $ openstack remove security group…
groups with instances (via
Neutron)

EMC CONFIDENTIAL—INTERNAL USE ONLY 16

KEY SERVICE OPERATIONS: ADMINS
• Admins can use the CLI to check the available hypervisor hosts,
what VM’s the hosts are running
– $ nova hypervisor-list (or openstack hypervisor list)
– $ nova hypervisor-stats <server> (openstack hypervisor show)
– $ nova hypervisor-server <server>

• Admins can also use the CLI to manage host aggregates

– $ nova aggregate-*
– $ openstack help aggregate

• Set and manage project resource quotas

– $ nova quota-*
– $ openstack help quota

EMC CONFIDENTIAL—INTERNAL USE ONLY 17

 NOVA ARCHITECTURE
Horizon
• nova-api can expose endpoints for
HTTP HTTP/VNC Nova and EC2 compatible APIs, along
nova- with metadata API
nova-api novncprox
y • Diagram omits VM networking services
nova-
nova-
compute
consoleaut
h driver

libvirt
Nova nova-
DB scheduler

VM
nova-
nova- conductor
KVM
cert
SQL
AMQP
Nova queues
Control Node Nova queues Compute Node
EMC CONFIDENTIAL—INTERNAL USE ONLY 18
 MAIN COMPUTE SERVICES
Horizon • Involved in VM instance creation
HTTP HTTP/VNC workflow examined previously
nova-
nova-api novncprox • nova-conductor prevents direct DB
y connections from compute hosts
nova-
nova-
compute
consoleaut
h driver

libvirt
Nova nova-
DB scheduler

VM
nova-
nova- conductor
KVM
cert
SQL
AMQP
Nova queues
Control Node Nova queues Compute Node
EMC CONFIDENTIAL—INTERNAL USE ONLY 19
SERVICE ARCHITECTURE: SCHEDULER
• Scheduler is configured with a series of filters in
nova.conf to determine the optimal host to place
a new VM (scheduler_default_filters)
• Custom filters can be registered by deployer
• RAM and CPU filters affected by global and
aggregate-level oversubscription ratios
nova-
scheduler
– cpu_allocation_ratio=16.0
– ram_allocation_ratio=1.5

• Large number of stock filters available to check

Nova
resources, affinity, flavor type, etc.
DB
• Cells introduce extra layer of scheduling, as
request is directed to one cell’s scheduler
EMC CONFIDENTIAL—INTERNAL USE ONLY 20
SCHEDULING EXAMPLE
• Scheduler applies a
series of filters to
identify set of candidate
hosts for new instance
• Candidate list is then
weighed and sorted to
determine order for
request dispatching
• Filter and weigher
configuration key for
achieving performance
and resource utilization
requirements

Source: OpenStack Configuration Reference - Kilo

EMC CONFIDENTIAL—INTERNAL USE ONLY 21
VM STATES - INSTANCE CREATION

Source: OpenStack Nova Developer Documentation

EMC CONFIDENTIAL—INTERNAL USE ONLY 22
VM STATES – FULLER VIEW

Source: OpenStack Nova Developer Documentation

EMC CONFIDENTIAL—INTERNAL USE ONLY 23
SCALING WITH CELLS
• Cells allow scale in distributed fashion and
management of heterogeneous resources API Cell
nova-api database
• Cells organized in tree structure
AMQP
nova-cells
Broker
• Each cell has its own database, message
queue, and set of services except for
nova-api

• Extra nova-cells service handles inter-cell

Child Cells
communication and scheduling nova-cells
nova- AMQP
network Broker

• VM requests scheduled to cell first, then nova- nova-

database
scheduler compute
scheduled to host in selected cell, in a
two-level procedure

EMC CONFIDENTIAL—INTERNAL USE ONLY 24

 VNC CONSOLE SERVICES (NoVNC)
Horizon • Novncproxy service provides web interface to
HTTP HTTP/VNC connect to VNC proxy for VM instance
nova-
• Token generated when console requested and
nova-api novncprox
cached by nova-consoleauth for later checking
y

nova-
nova-
compute
consoleaut
h driver

libvirt
Nova nova-
DB scheduler

VM
nova-
nova- conductor
KVM
cert
SQL
AMQP
Nova queues
Control Node Nova queues Compute Node
EMC CONFIDENTIAL—INTERNAL USE ONLY 25
NoVNC PROCESS

Source: OpenStack Cloud Adminstrator Guide

EMC CONFIDENTIAL—INTERNAL USE ONLY 26
NOVA APIs and CLIs

• Nova API service can expose both OpenStack and

EC2 API versions
• CLI tools like python-novaclient and python-
openstackclient operate exclusively against the
OpenStack Nova API
• euca2ools package can be used to exercise the EC2
API
– Usage requires EC2 credentials obtained from Keystone or Horizon

EMC CONFIDENTIAL—INTERNAL USE ONLY 27

WHERE ARE MY INSTANCES, REALLY?
• By default on a compute node, Nova uses a local filesystem
location to store instance images and log files
– /var/lib/nova/instances

• For each instance on the compute node, there will be a

subdirectory labeled with the instance ID of each instance on
the host:
– /var/lib/nova/instances/b10a100c-0e7b-4418-8c47-eb9a3655d982

• The instance subdirectory contains the console log, libvirt config

file, and ephemeral disk files
– # ls b10a100c-0e7b-4418-8c47-eb9a3655d982/
console.log disk disk.info libvirt.xml

EMC CONFIDENTIAL—INTERNAL USE ONLY 28