2/1/2023

Chapter 5: Live Virtual Machine Migration

Introduction
• Physical cluster is a collection of servers interconnected by a physical
network
• Virtual clusters are built with VMs installed at distributed servers from
one or more physical clusters

1
 2/1/2023

Introduction
Properties of VM in Virtual Cluster
• Multiple VMs running with different OSes can be deployed on the same physical node
• The failure of any physical nodes may disable some VMs installed on the failing nodes
But the failure of VMs will not pull down the host system
• The purpose of using VMs is to consolidate multiple functionalities on the same
server.
• This will greatly enhance server utilization

Introduction
Properties of VM in Virtual Cluster
• Each VM can be installed on a remote server or replicated on multiple servers
belonging to the same or different physical clusters

• The boundary of a virtual cluster can change as VM nodes are added, removed, or
migrated dynamically over time

2
 2/1/2023

What is Live VM Migration?

• Move a VM from one physical machine to another while its applicants continue
to execute during migration
• One of the most important features provided by modern VM technologies
• Allows to move an OS instance to another physical node without interrupting any
hosted services on the migrating OS
• Allows workloads of one node to transfer to another node
• Usually involves
– Migrating memory state
– Migrating CPU state
– Optionally, migrating virtual disk state

Why Live VM Migration?

– Load balancing
– Fault tolerance
– Power management
– System maintenance

3
 2/1/2023

Why Live VM Migration?

Load balancing
• Required when the load is considerably unbalanced and impending downtime
often require simultaneous VM (s) migration
• Most of existing techniques try to migrate VMs from lightly loaded and heavily
PMs to moderately used PMs while keeping number migrations to a minimum

Why Live VM Migration?

Load balancing
The load balancing is required due to the huge number of heterogeneous input tasks with
heterogeneous resource requirement. VM manager verified the resource availability of the system for a
given set of input tasks.
• If the set of tasks can run with the available active VMs, then
the VM manager sent those tasks to the task scheduler.
• Otherwise, the VM manager creates the required VMs in the
host where resource availability satisfies.

The task scheduler act as a load balancer where the

mapping will be done among tasks and VMs based on the
resource requirement of the tasks.

4
 2/1/2023

Why Live VM Migration?

Fault tolerance
• Fault is an another challenge to guarantee the critical service availability
and reliability
• Failures should be anticipated and proactively handled, to minimize failure impacts on
the application execution and system performance

System maintenance
• A physical system required to be upgraded and serviced
– So all VM’s of that physical server must be moved to an alternate server for
maintenance and services are available to users without interruption

Live VM Migration
Power Management
• Switching the idle mode server to either sleep mode or off mode based on
resource demands
– Leads to great energy saving because idle mode server consumes 70% of their peak power

• Consolidate the running VM’s to fewer active hosts leads to great energy saving
– So dynamic allocation of VM’s to few active servers as much as possible
• Therefore, VM live migration is a good technique for cloud power efficiency

5
 2/1/2023

Live VM Migration
Components in live VM migration
• At the time of live VM migration, it is essential to know about what to
migrate or which content must be migrated
– Memory content
– Storage content

What is migrated?
• CPU context of VM, contents of main memory
• Narrow interface, easier than process migration
• Disk: assume NAS (network attached storage) that is accessible from both hosts, or local
disk is mirrored
• a network-accessible storage system, such as storage area network (SAN) or network
attached storage (NAS), is employed
• Only memory and CPU status needs to be transferred from the source node to the target
node. We do not consider migrating disk data
• Network: Assume both hosts on same LAN
• The migrated VM carries IP address, advertise new MAC address to IP mapping via ARP reply
• All protocol state, including any open sockets
• Migrate MAC address, let switches learn new MAC location
• Network packets redirected to new location (with transient losses)

6
 2/1/2023

What is migrated?
• I/O devices are provisioned at target
• Virtual I/O devices easier to migrate, direct device assignment of physical devices to
VMs (device passthrough) makes migration harder

Live VM Migration Techniques

Memory Content
• Memory modules that need to be moved under the process of migration
• VM Configured memory: The amount of actual physical memory that is given to guest VM by the
hypervisor
– The guest VM used this memory as their own physical memory
• VM Used Memory: Currently and frequently accessed through VM OS and all running processes.
These memory pages keep track by the guest VM
• Application Requested Memory: The amount of memory required for running an application and
it is allocated by guest VM OS
– It is not necessary that the requested memory is within the physical memory, it may be in disk storage
when all the VM configured memory is in use

7
 2/1/2023

Live VM Migration
Live VM Migration Steps
Start migration
• Makes preparations for the migration
– The migrating VM and the destination host
Transfer memory
• Send VM’s memory to the destination node
– All of the memory data is transferred in the first round
• The migration controller recopies the memory data which is changed in the last round
– These steps keep iterating until the dirty portion of the memory is small enough to handle
the final copy

Live VM Migration Techniques

Live VM Migration Steps
Suspend the VM and copy the last portion of the data
• The migrating VM’s execution is suspended
– When the last round’s memory data is transferred
• Non memory data such as CPU and network states is sent as well
• During this step, the VM is stopped and its applications will no longer run
– Service unavailable time is called the downtime of migration
 As short as possible so that it can be negligible to users

8
 2/1/2023

Live VM Migration Techniques

Live VM Migration Steps
Commit and activate the new host
• The VM reloads the states and recovers the execution of programs
the service provided by this VM continues
• The network connection is redirected to the new VM
• Clear the dependency to the source host
• Remove the original VM from the source host

Live VM Migration Techniques

What is Live VM Migration?

Migration Manager
At source: maps the guest VM’s memory and execution state
Transfer VM’s pages to the target migration manager over TCP connection
At Destination: the migration manager restores the VM state and resumes execution

Paper 1 : Live Migration of Virtual Machines

Auther: Christopher Clark, Keir Fraser, Steven Hand, Jacob
Gorm Hansen
Post-Copy Live Migration of Virtual MachinesMichael R. Hines,
Umesh Deshpande, and Kartik Gopalan

9
 2/1/2023

Steps to Migrate a VM
• Broad steps in any migration technique: Suppose we are migrating a
VM from host A to host B
1. Setup target host B, reserve resources for the VM
2. Push phase: push some memory of VM from A to B
3. Stop-and-copy: stop the VM at A, copy CPU context, and some memory
4. Pull phase: Start VM at host B, pull any further memory required from A
5. Clean up state from host A, migration complete
• Total migration time: time for steps 2,3,4
• Service downtime: time for step 3
• Other metrics: impact on application performance, network bandwidth consumed,
total pages transferred

Types of Live VM Migration Techniques

Pure stop-and-copy:
• VM stopped at source
• All state transferred to target
• VM restarted at target
• Too much downtime to be classified as “live” migration
Pure Demand Paging:
• VM stopped at source
• Transfer minimal execution context to target
• PC, Registers, non-pageable memory
• VM restarted at target
• Pull memory contents from source when needed
• Smaller downtime

10
 2/1/2023

Types of Live VM Migration Techniques

Pre-copy
• Most state is transferred in the push phase, followed by a
brief stop-and-copy phase
• Don’t stop VM at source
• Let the VM continue to run

• Copy VM’s physical memory contents to target over multiple iterations

• First iteration  copy all pages
• Each subsequent iteration  copy pages that were dirtied by the VM during the previous
iteration

• Stop-and-copy when number of dirt pages is small enough

• But what if the number of dirt pages never converges to a small enough number?
• After a fixed number of iterations, give up and stop-and-copy

Types of Live VM Migration Techniques

Pre-copy:
• Basic steps of VM migration follow in pre-
copy technique

11
 2/1/2023

Types of Live VM Migration Techniques

Pre-copy:

Types of Live VM Migration Techniques

Post-copy:
• VM stopped
• Bare minimum state required to run the VM is transferred to the target host (CPU
state)
• Start VM as the target, but without its memory
• Remaining state is pulled on demand while the VM is running at the new location
• Demand paging over network
• Actively pushing from source
• Advantage:
• Each page transferred over the network only once

12
 2/1/2023

Types of Live VM Migration Techniques

Post-copy: Basic steps of VM migration follow in post-copy technique

Types of Live VM Migration Techniques

Hybrid pre/post-copy
1. Perform one more rounds of live pre-copy rounds
2. Pause VM and transfer execution state
3. Use post-copy to transfer any remaining dirty pages from source

13
 2/1/2023

Types of Live VM Migration Techniques

• Most of the hypervisors support live migration but the implementation of live
migration with a little or no consideration towards its security
• Hence live migration might be susceptible to range of attacks from Denial-of-Service (DoS) attacks
to Man-In-The-Middle (MITM) attacks

• During the migration, data can be tampered or sniffed easily as it is not encrypted
• Thus compromising confidentiality and integrity of migrating data

• These security threats in live VM migration discourages many sectors, such as financial,
medical, and government, from taking advantage of VM live migration
• Hence, security is the critical challenge that needs examination to provide secure live
VM migration

14