Logical Domains (LDoms)
Liam Merwick LDoms Developer Sun Microsystems – SPARC Platform S/W Group




• • • • • • •
Page: 2

Background Features Architecture Components Future Roadmap Configuration Examples Q&A





Page: 3

USE IMPROVE EVANGELIZE Consolidation • Conventional Unix server utilisation is 7 to 15% • Data center compression > Goal: Increase utilisation of equipment 10% Utilisation 20% Time 10% Page: 4 .

File Server Web Server Mail Server Calendar Database Web Server Server SunRay Database Server App Server App OS Server Multiple OSes Single OS Trend to flexibility Trend to isolation Dynamic System Domains SPARC Logical Domains Solaris xVM VMware ESX Microsoft Virtual Server Page: 5 Solaris Containers (Zones + SRM) Solaris Containers for Linux Applications Solaris Trusted Extensions Solaris Resource Manager (SRM) .USE IMPROVE EVANGELIZE Solutions from Sun Hard Partitions App Server Database Identity Server Virtual Machines OS Virtualisation Resource Mgmt.

USE IMPROVE EVANGELIZE Virtualisation for SPARC platforms • New “sun4v” architecture Solaris X update (genunix) sun4u code US-Z CPU code SPARC hypervisor Operating System Solaris X (genunix) Solaris X (sun4v) sun4v interface CPU “Z” SPARC CPU Platform Page: 6 .

USE IMPROVE EVANGELIZE Virtual Machine for SPARC • Thin software layer between OS and platform hardware • Hypervisor + sun4v interface • Virtualises machine HW and isolates OS from register-level • Delivered with platform not OS • Not itself an OS sun4v virtual machine User App User App Solaris OpenBoot Hypervisor SPARC hardware User App stable interface “sun4v” Page: 7 .

with a dynamically reconfigurable sub- set of machine resources. and its own independent OS > Protection & isolation via SPARC hardware and LDoms Hypervisor firmware LDom A LDom B LDom C LDom D OS Environment of choice LDoms Hypervisor CPU CPU CPU CPU CPU Memory I/O Linux FreeBSD CPU CPU Memory CPU Memory Platform Hardware Page: 8 Memory I/O .USE IMPROVE EVANGELIZE Logical Domains Technology • Virtualisation and partitioning of machine resources > Each domain is a full virtual machine.


MAC & IP addresses Start. administrators own disks own network interfaces. tuning parameters own user accounts.USE IMPROVE EVANGELIZE Fundamentals • Each virtual machine should appear as an entirely independent machine > > > > > own kernel. patches. stop and reboot independently of each other • Paravirtualised OS Page: 10 .

USE IMPROVE EVANGELIZE Features • Add and remove cpus while OS is running • Protection and isolation via SPARC hardware and LDoms Hypervisor firmware • Service domains for I/O > direct I/O access Page: 11 .

started and stopped independently > Without requiring a power-cycle of machine Page: 12 .USE IMPROVE EVANGELIZE Features – UltraSPARC-T1 specific • Up to 32 LDoms per box > CPU threads individually assignable to different LDoms • Memory assignable to with an 8KB granularity • Two PCI busses independently assignable to the same or different LDoms. • Guest Domains can be configured.

USE IMPROVE EVANGELIZE Features – UltraSPARC-T2 specific • 64 threads => up to 64 domains > more domains per box than with UltraSPARC-T1 • On-chip NIU assignable to any domain. Page: 13 .

USE IMPROVE EVANGELIZE Architecture Page: 14 .

USE IMPROVE EVANGELIZE Architecture • • • • • Logical Domains overview Hypervisor LDoms Manager Virtualised I/O Dynamic Reconfiguration (DR) Page: 15 .

USE IMPROVE EVANGELIZE Hypervisor Support • Hypervisor software is responsible for maintaining separation between domains > Using extensions built into a sun4v CPU • Also provides Logical Domain Channels (LDCs) so that domains can communicate with each other > Mechanism by which domains can be virtually networked with each other. or provide services to each other Page: 16 .

USE IMPROVE EVANGELIZE LDoms Manager • One Manager per host HV > Controls Hypervisor and all its LDoms • Exposes control interface via CLI • Maps Logical Domains to physical resources > Constraint engine > Heuristic binding of LDoms to resources > Assists with performance optimisation > Assists in event of failures / blacklisting Page: 17 .

USE IMPROVE EVANGELIZE Direct I/O • Traditional model > Existing drivers and devices Logical Domain App App App continue to work Logical Domain owns PCI root and tree Device Driver /pci@B/qlc@6 Nexus Driver /pci@B Privileged Hyper Privileged Hypervisor I/O Bridge Virtual Nexus I/F I/O MMU PCI Root Hardware PCI-Express Page: 18 .

USE IMPROVE EVANGELIZE Virtualised I/O Logical Domain A App App App Virtual Device Service Nexus Driver /pci@B App Service Domain Device Driver /pci@B/qlc@6 Privileged Virtual Device Driver Hyper Privileged Hypervisor Virtual Nexus I/F Domain Channel I/O MMU Hardware PCI R oot I/O Bridge PCI B Page: 19 .

USE IMPROVE EVANGELIZE Virtual (Block) Disk device Logical Domain 1 App App App App Logical Domain 2 App App App App Logical Domain 3 App App App App Service Domain vDisk Server vDisk Server vDisk client Driver vDisk client driver vDisk client driver DeviceDriver Hypervisor Virtual SAN 2 Virtual SAN 1 I/O Bridge FC-AL I/F Page: 20 .

0/24 I/O Bridge Gb Ether I/F Page: 21 .168.USE IMPROVE EVANGELIZE Virtual Ethernet device Logical Domain 1 App App App App Logical Domain 2 App App App App Logical Domain 3 App App App App Service Domain V-Ether Switch V-Ether Switch V-Ether Driver V-Ether Driver V-Ether Driver DeviceDriver Hypervisor Virtual LAN 2: 63.24/16 Virtual LAN 1: 192.

USE IMPROVE EVANGELIZE Dynamic Reconfiguration (DR) • Ability to dynamically grow or shrink compute capacity of an LDom on demand • No need to re-boot Solaris • Simply add / remove: > CPUs > Memory (future) > I/O (future) • Improve utilisation by balancing resources between LDoms Page: 22 .

USE IMPROVE EVANGELIZE LDoms Manager Logical Domain 1 App App App App Logical Domain 2 App App App Logical Domain 3 App App App Service Domain LDoms Manager OS DR Stack OS DR Stack OS DR Stack Hypervisor HV DR Stack Page: 23 .

USE IMPROVE EVANGELIZE Example: vCPU reconfiguration Logical Domain 1 App App Logical Domain 1 App App Logical Domain 2 App App V Hypervisor V P Hypervisor V P P V V P P Example command line operations: % ldm remove-vcpu 1 Logical_Domain_1 % ldm add-vcpu 1 Logical_Domain_2 Page: 24 .

USE IMPROVE EVANGELIZE Components Page: 25 .

6.USE IMPROVE EVANGELIZE Components – Hardware & Software • Hardware > UltraSPARC-T1 (Niagara) family of CMT processors > T1000 Netra CP3060 T6300 > T2000 Netra T2000 > UltraSPARC-T2 (Niagara2) family of CMT processors > T5120 T5220 T6320 • Operating System > OpenSolaris (build 70+ recommended) > Solaris 10 11/06 (U3) or later > Linux 2.23 Page: 26 .

0.4.6 or later > 6.available as Free download) > 1.x and 7.1 (Solaris package .0 (For use with 6.x now available • LDoms Manager Software > 1.4.5.x firmware) • OS can and will be released/updated independently of Firmware/LDoms Manager Page: 27 .USE IMPROVE EVANGELIZE Components – Hardware & Software • Firmware > 6.0.

USE IMPROVE EVANGELIZE Future Roadmap Page: 28 .

0.1 Page: 29 .USE IMPROVE EVANGELIZE Latest Release: LDoms 1.0. Service and IO domains > Without bringing down guest domains • Support for new platforms • Updates to Firmware and LDoms Manager package • Upgrade to Solaris 10 8/07 (U4) recommended > Independent of LDoms 1.1 • Rebooting of Control.

USE IMPROVE EVANGELIZE Key Features on Roadmap • Warm Migration > Suspend and resume • Live Migration • Hybrid I/O • Dynamic Reconfiguration > Virtual and Physical I/O > Memory • Booting/Installing from DVD/ISO • Using ZFS/SVM volumes Page: 30 .

USE IMPROVE EVANGELIZE Configuration Examples Page: 31 .

USE IMPROVE EVANGELIZE Configuring a machine to run LDoms • Make resources available (remove from control domain) primary#> ldm set-mau 2 primary primary#> ldm set-vcpu 8 primary primary#> ldm set-mem 8g primary • Configure services primary#> ldm add-vcc port-range=5000-5100 primary-vcc0 primary primary#> ldm add-vds primary-vds0 primary primary#> ldm add-vsw net-dev=e1000g0 primary-vsw0 primary Page: 32 .

USE IMPROVE EVANGELIZE Adding another domain (I) • Add disk device primary#> ldm add-vdsdev /dev/dsk/c1t0d0s2 vol1@primary-vds0 • Adding resources primary#> ldm create domain1 primary#> ldm add-mem 2g domain1 primary#> ldm add-vcpu 4 domain1 primary#> ldm add-vdisk vdisk1 vol1@primary-vds0 domain1 primary#> ldm add-vnet vnet1 primary-vsw0 domain1 • Show domain constraints primary#> ldm list -l domain1 Page: 33 .

USE IMPROVE EVANGELIZE Adding another domain (II) • Booting the newly created domain primary#> ldm bind domain1 primary#> ldm start domain1 • Connect to the console primary#> telnet localhost 5000 Page: 34 .

USE IMPROVE EVANGELIZE LDoms Community • Newly Created community • Website > Page: 35 .org/os/communites/ldoms • Alias > ldoms-discuss@opensolaris.opensolaris.

xml Page: 36 IMPROVE EVANGELIZE More information • BigAdmin Portal > • Sun Blueprint > Beginners Guide to LDoms: Understanding and Deploying Logical Domains > http://www.pdf • Training Course > Solaris 10 Logical Domains Administration (SA-345-S10) > .USE IMPROVE EVANGELIZE Liam Merwick Sun Microsystems Liam.Merwick@sun.

Sign up to vote on this title
UsefulNot useful