USE

IMPROVE

EVANGELIZE

Logical Domains (LDoms)

Liam Merwick LDoms Developer Sun Microsystems SPARC Platform S/W Group

USE

IMPROVE

EVANGELIZE

Contents

Page: 2

Background Features Architecture Components Future Roadmap Configuration Examples Q&A

USE

IMPROVE

EVANGELIZE

Background

Page: 3

USE

IMPROVE

EVANGELIZE

Consolidation
Conventional Unix server utilisation is 7 to 15% Data center compression
> Goal: Increase utilisation of equipment

10%

Utilisation

20%

Time
10%
Page: 4

USE

IMPROVE

EVANGELIZE

Solutions from Sun

Hard Partitions
App Server Database Identity Server

Virtual Machines OS Virtualisation Resource Mgmt.

File Server Web Server Mail Server Calendar Database Web Server Server SunRay Database Server App Server

App OS Server

Multiple OSes Single OS Trend to flexibility Trend to isolation

Dynamic System Domains SPARC Logical Domains Solaris xVM VMware ESX Microsoft Virtual Server
Page: 5

Solaris Containers (Zones + SRM) Solaris Containers for Linux Applications Solaris Trusted Extensions

Solaris Resource Manager (SRM)

USE

IMPROVE

EVANGELIZE

Virtualisation for SPARC platforms

New sun4v architecture
Solaris X update (genunix) sun4u code
US-Z CPU code SPARC hypervisor

Operating System

Solaris X (genunix)

Solaris X (sun4v)

sun4v interface

CPU Z

SPARC CPU

Platform
Page: 6

USE

IMPROVE

EVANGELIZE

Virtual Machine for SPARC

Thin software layer between OS and platform hardware Hypervisor + sun4v interface
Virtualises machine HW and isolates OS from register-level Delivered with platform not OS Not itself an OS
sun4v virtual machine User App User App Solaris OpenBoot Hypervisor SPARC hardware User App

stable interface sun4v

Page: 7

USE

IMPROVE

EVANGELIZE

Logical Domains Technology

Virtualisation and partitioning of machine resources
> Each domain is a full virtual machine, with a dynamically reconfigurable sub-

set of machine resources, and its own independent OS > Protection & isolation via SPARC hardware and LDoms Hypervisor firmware

LDom A

LDom B

LDom C

LDom D

OS Environment of choice LDoms Hypervisor

CPU CPU CPU CPU CPU Memory I/O

Linux

FreeBSD

CPU CPU Memory

CPU Memory

Platform Hardware
Page: 8

Memory I/O

USE

IMPROVE

EVANGELIZE

Features

Page: 9

USE

IMPROVE

EVANGELIZE

Fundamentals
Each virtual machine should appear as an entirely independent machine
> > > > >

own kernel, patches, tuning parameters own user accounts, administrators own disks own network interfaces, MAC & IP addresses Start, stop and reboot independently of each other

Paravirtualised OS

Page: 10

USE

IMPROVE

EVANGELIZE

Features
Add and remove cpus while OS is running Protection and isolation via SPARC hardware and LDoms Hypervisor firmware Service domains for I/O
> direct I/O access

Page: 11

USE

IMPROVE

EVANGELIZE

Features UltraSPARC-T1 specific

Up to 32 LDoms per box
> CPU threads individually assignable to different LDoms

Memory assignable to with an 8KB granularity Two PCI busses independently assignable to the same or different LDoms. Guest Domains can be configured, started and stopped independently
> Without requiring a power-cycle of machine

Page: 12

USE

IMPROVE

EVANGELIZE

Features UltraSPARC-T2 specific

64 threads => up to 64 domains
> more domains per box than with UltraSPARC-T1

On-chip NIU assignable to any domain.

Page: 13

USE

IMPROVE

EVANGELIZE

Architecture

Page: 14

USE

IMPROVE

EVANGELIZE

Architecture
Logical Domains overview Hypervisor LDoms Manager Virtualised I/O Dynamic Reconfiguration (DR)

Page: 15

USE

IMPROVE

EVANGELIZE

Hypervisor Support
Hypervisor software is responsible for maintaining separation between domains
> Using extensions built into a sun4v CPU

Also provides Logical Domain Channels (LDCs) so that domains can communicate with each other
> Mechanism by which domains can be virtually networked

with each other, or provide services to each other

Page: 16

USE

IMPROVE

EVANGELIZE

LDoms Manager
One Manager per host HV
> Controls Hypervisor and all its LDoms

Exposes control interface via CLI Maps Logical Domains to physical resources
> Constraint engine > Heuristic binding of LDoms to resources > Assists with performance optimisation > Assists in event of failures / blacklisting

Page: 17

USE

IMPROVE

EVANGELIZE

Direct I/O
Traditional model
> Existing drivers and devices

Logical Domain
App App App

continue to work

Logical Domain owns PCI root and tree

Device Driver /pci@B/qlc@6 Nexus Driver /pci@B

Privileged

Hyper Privileged

Hypervisor I/O Bridge

Virtual Nexus I/F

I/O MMU PCI Root

Hardware

PCI-Express

Page: 18

USE

IMPROVE

EVANGELIZE

Virtualised I/O
Logical Domain A
App App App Virtual Device Service Nexus Driver /pci@B App

Service Domain
Device Driver /pci@B/qlc@6

Privileged

Virtual Device Driver

Hyper Privileged

Hypervisor

Virtual Nexus I/F

Domain Channel
I/O MMU

Hardware
PCI R oot

I/O Bridge

PCI B
Page: 19

USE

IMPROVE

EVANGELIZE

Virtual (Block) Disk device

Logical Domain 1
App App App App

Logical Domain 2
App App App App

Logical Domain 3
App App App App

Service Domain
vDisk Server vDisk Server

vDisk client Driver

vDisk client driver

vDisk client driver

DeviceDriver

Hypervisor

Virtual SAN 2 Virtual SAN 1

I/O Bridge
FC-AL I/F

Page: 20

USE

IMPROVE

EVANGELIZE

Virtual Ethernet device

Logical Domain 1
App App App App

Logical Domain 2
App App App App

Logical Domain 3
App App App App

Service Domain
V-Ether Switch V-Ether Switch

V-Ether Driver

V-Ether Driver

V-Ether Driver

DeviceDriver

Hypervisor

Virtual LAN 2: 63.24/16 Virtual LAN 1: 192.168.0/24

I/O Bridge

Gb Ether I/F

Page: 21

USE

IMPROVE

EVANGELIZE

Dynamic Reconfiguration (DR)

Ability to dynamically grow or shrink compute capacity of an LDom on demand No need to re-boot Solaris Simply add / remove:
> CPUs > Memory (future) > I/O (future)

Improve utilisation by balancing resources between LDoms

Page: 22

USE

IMPROVE

EVANGELIZE

LDoms Manager
Logical Domain 1
App App App App

Logical Domain 2
App App App

Logical Domain 3
App App App

Service Domain
LDoms Manager

OS DR Stack

OS DR Stack

OS DR Stack

Hypervisor

HV DR Stack

Page: 23

USE

IMPROVE

EVANGELIZE

Example: vCPU reconfiguration

Logical Domain 1
App App

Logical Domain 1
App App

Logical Domain 2
App App

V
Hypervisor

V P
Hypervisor

V P P

V P

Example command line operations: % ldm remove-vcpu 1 Logical_Domain_1 % ldm add-vcpu 1 Logical_Domain_2
Page: 24

USE

IMPROVE

EVANGELIZE

Components

Page: 25

USE

IMPROVE

EVANGELIZE

Components Hardware & Software

Hardware
> UltraSPARC-T1 (Niagara) family of CMT processors > T1000 Netra CP3060 T6300 > T2000 Netra T2000 > UltraSPARC-T2 (Niagara2) family of CMT processors > T5120 T5220 T6320

Operating System
> OpenSolaris (build 70+ recommended) > Solaris 10 11/06 (U3) or later > Linux 2.6.23
Page: 26

USE

IMPROVE

EVANGELIZE

Components Hardware & Software

Firmware
> 6.4.6 or later > 6.5.x and 7.0.x now available

LDoms Manager Software

> 1.0.1 (Solaris package - available as Free download) > 1.0 (For use with 6.4.x firmware)

OS can and will be released/updated independently of Firmware/LDoms Manager

Page: 27

USE

IMPROVE

EVANGELIZE

Future Roadmap

Page: 28

USE

IMPROVE

EVANGELIZE

Latest Release: LDoms 1.0.1

Rebooting of Control, Service and IO domains > Without bringing down guest domains Support for new platforms Updates to Firmware and LDoms Manager package Upgrade to Solaris 10 8/07 (U4) recommended > Independent of LDoms 1.0.1

Page: 29

USE

IMPROVE

EVANGELIZE

Key Features on Roadmap

Warm Migration > Suspend and resume Live Migration Hybrid I/O Dynamic Reconfiguration > Virtual and Physical I/O > Memory Booting/Installing from DVD/ISO Using ZFS/SVM volumes

Page: 30

USE

IMPROVE

EVANGELIZE

Configuration Examples

Page: 31

USE

IMPROVE

EVANGELIZE

Configuring a machine to run LDoms

Make resources available (remove from control domain) primary#> ldm set-mau 2 primary primary#> ldm set-vcpu 8 primary primary#> ldm set-mem 8g primary Configure services primary#> ldm add-vcc port-range=5000-5100 primary-vcc0 primary primary#> ldm add-vds primary-vds0 primary primary#> ldm add-vsw net-dev=e1000g0 primary-vsw0 primary

Page: 32

USE

IMPROVE

EVANGELIZE

Adding another domain (I)

Add disk device
primary#> ldm add-vdsdev /dev/dsk/c1t0d0s2 vol1@primary-vds0

Adding resources
primary#> ldm create domain1 primary#> ldm add-mem 2g domain1 primary#> ldm add-vcpu 4 domain1 primary#> ldm add-vdisk vdisk1 vol1@primary-vds0 domain1 primary#> ldm add-vnet vnet1 primary-vsw0 domain1

Show domain constraints

primary#> ldm list -l domain1
Page: 33

USE

IMPROVE

EVANGELIZE

Adding another domain (II)

Booting the newly created domain primary#> ldm bind domain1 primary#> ldm start domain1 Connect to the console primary#> telnet localhost 5000

Page: 34

USE

IMPROVE

EVANGELIZE

LDoms Community
Newly Created community Website > http://www.opensolaris.org/os/communites/ldoms Alias > ldoms-discuss@opensolaris.org

Page: 35

USE

IMPROVE

EVANGELIZE

More information
BigAdmin Portal > http://www.sun.com/bigadmin/hubs/ldoms/ Sun Blueprint > Beginners Guide to LDoms: Understanding and Deploying Logical Domains > http://www.sun.com/blueprints/0207/820-0832.pdf Training Course > Solaris 10 Logical Domains Administration (SA-345-S10) > http://uk.sun.com/training/catalog/courses/SA-345-S10.xml
Page: 36

USE

IMPROVE

EVANGELIZE

Liam Merwick Sun Microsystems Liam.Merwick@sun.com http://blogs.sun.com/merwick