TA26

I/O Virtualization (IOV) For Dummies

Mallik Mahalingam
Staff Engineer

Rich Brunner
Chief Platform Architect VMware, Inc.
This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Motivation
Software based I/O Virtualization provides
I/O Sharing, Consolidation, Security, Isolation, Mobility Simplified management Transparent Teaming and Failover

High I/O performance is desirable for enterprise class applications Challenge: How to achieve I/O performance without compromising virtualization benefits?

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Outline
VMware I/O Virtualization Architecture Hardware/Software Assists for Virtualized I/O Passthrough I/O

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

I/O Virtualization Architecture

Guest OS
Guest Device Driver

I/O Virtualization architecture consists of

Guest driver Virtual device Communication mechanism between virtual device and virtualization stack Virtualization I/O stack Physical device driver Real device

Device Emulation

I/O Stack Physical Device Driver

Physical Device
This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

I/O Virtualization Architecture (contd.)

Guest OS
Guest Device Driver

Device Emulation

Virtual device
model a real device
e.g., Intel e1000, LSI mptscsi

I/O Stack Physical Device Driver

model a simple virtualization friendly device

e.g., VMware vmxnet

Physical Device
This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

I/O Virtualization Architecture (contd.)

Guest OS
Guest Device Driver

Device Emulation

I/O Stack Physical Device Driver

Virtualization I/O stack

translates guest I/O addresses to host addresses handles inter VM communication multiplexes I/O requests from/to the physical device provides enterprise-class I/O features

Physical Device

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

I/O Virtualization Implementations

Virtualized I/O
Hosted or Split
Guest OS
Guest Device Driver Host OS/Dom0/ Parent Domain Virtual Device (VMM) Virtual Device I/O Stack Physical Device Driver Virtual Device I/O Stack Physical Device Driver Device Manager

Passthrough I/O
Guest OS
Guest Device Driver

Hypervisor Direct
Guest OS
Guest Device Driver

VMware Workstation, VMware Server, VMware ESX Server Microsoft Viridian & Virtual Server, Xen

VMware ESX Server (storage and network)

A Future Option

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Benefits of Virtualized I/O

Emulated I/O decouples virtual HW from real HW. Transparent move of VM between different physical machine.
Suspend/Resume Live VM Migration
Transparent to VMs (OS, Apps, Virtual Devices) Guest state is unaffected
Example: VMs open TCP connections to remain intact

VM to Virtual Device linking does not change. Physical device and Virtual Device association can change
Can be dynamic e.g Active NIC in a team of NICs
This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Improving Virtualized I/O

Guest OS
Guest Device Driver

Accelerate guest code

HW support for CPU and MMU virtualization Support for improving device virtualization
e.g. virtual interrupt delivery

Device Emulation

I/O Stack Physical Device Driver

Simplified device models Para-virtualized drivers

Physical Device

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Improving Virtualized I/O (contd.)

Guest OS
Guest Device Driver

Acceleration of virtualization stack Take advantage of multiple cores MSI and MSI-X Interrupts Dedicated DMA engine for memory-memory copy pNIC Acceleration NetQueue: Multiple receive queues (MAC address/VLAN) and interrupt per queue Map receive buffers to guest memory avoids a copy

Device Emulation

I/O Stack Physical Device Driver

Physical Device
This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Passthrough I/O
Guest drives device directly
Use case: I/O Appliances, High performance VMs
Guest OS
Device Driver

Guest OS
Device Driver

Guest OS
Device Driver

Requires:
I/O MMU for DMA Address Translation and protection (Intel VT-d, AMD I/O MMU) Partitionable I/O device for sharing (PCISIG IOV SR/MR specification)

Virtualization Layer

Device Manager

I/O MMU

VF

VF

VF

I/O Device

PF

PF = Physical Function, VF = Virtual Function

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Issues with Passthrough I/O

Guest Transparent VMotion
No simple way to checkpoint Passthrough device state

Hardware independence
Need a driver for every Passthrough device that VM sees HW errata/workarounds requires driver upgrade

Memory over-commitment
No visibility of DMAs to guest memory with Passthrough I/O

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Solutions for Passthrough I/O

VMotion
Guest Transparent
Virtualization Friendly Passthrough (vmxnet3) Device provides an hypervisor interface to save and restore state
Works only for Homogenous HW

Guest Aware
fail-over software combined with device Hot plug mechanism 2 Phase operation for VMotion (Ugly!)

Memory over-commitment Device and IOMMU co-operation to enable demand paging Define and develop APIs dynamic pinning/unpinning of DMA guest buffers Integrate control interface with Virtual Switch Hardware based packet queuing and control via virtual switch

VMware wants to work with IHVs to help define these hardware assists.
This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Device Class Directions

NIC
Physical Driver Acceleration for Emulated Model Best candidate for Heterogeneous Pass-thru

iSCSI Software HBA

Same as NIC

Fiber Channel and SAS HBA

Physical Driver Acceleration for Emulated Model Still researching Pass-thru solutions

Graphics
Better OpenGL support can help Emulated Model Poster-child for single-device Fixed Pass-thru HW not yet defined for multiple logical contexts Long, long way from Heterogeneous Mobile Pass-thru

This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.