WSV313

Architecting Private Clouds

Using Windows Server 2012
Michael Pascoe
Managing Director – Olikka
@OlikkaTech
Architecting Private Clouds
Using Windows Server 2012
Michael Pascoe
Managing Director – Olikka
@OlikkaTech
Architecting Private Clouds
Using Windows Server 2012
Michael Pascoe
Managing Director – Olikka
@OlikkaTech
Architecting Win Server 2012

for Private Clouds

Michael Pascoe
Managing Director – Olikka
@OlikkaTech
partner

Olikka - Gold Virtualisation

& Management
- Gold Desktop
Michael Pascoe
Managing Director - Silver Server
@OlikkaTech
Olikka is a specialist systems integrator

We are experts in
Infrastructure Management and Virtualization

We specialise in the
Design and Deployment of
Microsoft and Citrix software

We focus on two areas

Private Cloud and Windows Client
Agenda

Windows Server 2012 - The cloud infrastructure

Considerations in building clouds
Workloads
Networking
Storage
Resiliency
Example scenarios and configurations
Windows Server 2012 is Cloud Optimized

Multi-Tenant High Scale & Low Manageable

Clouds Cost Datacenters & Extensible
Windows Server 2012 : Summary of Cloud Enabling Features

High Scale
Multi-Tenant & Low Cost Manageable
Clouds Datacenters & Extensible

• Network Virtualization Compute

• Policy-Based Isolation
• QoS
• Performance Metrics Network • PowerShell
• Live & Storage • Hyper-V Extensible Switch
Migrations
• Cross-Premise Storage
Connectivity
Windows Server 2012 : Summary of Cloud Enabling Features

Multi-Tenant Manageable
Clouds & Extensible

High Scale & Low Cost Datacenters

Compute Network Storage

• Larger hosts • Datacenter Bridging (DCB) • Hyper-V over SMB

• Large VMs on 10GbE Networks • Copy offload (ODX)
• Higher VM density • Hardware offloads • Storage Spaces
• Large clusters • SR-IOV • Storage Thin Provisioning
• RSC / RSS • Synthetic Fibre Channel
• RDMA
• NIC Teaming
Scope of this
talk…
DDC
Reference
Architecture

http://aka.ms/bzm1rl
OK. I’m In!
How do I build my Cloud Infrastructure?
Requirements Influence Choices
Cost

High
Availability Scalability
Workloads

Networking

Storage

Resiliency
Performance Reliability

Security
Primary Considerations

Workloads

Networking

Storage

Resiliency
Primary Considerations

Workloads

Networking

Storage

Resiliency
 Workloads
Understanding Workloads
Do you Manage Servers or Apps?
Cloud Aware? Will apps be cloud ready or aware?
IaaS clouds fully supports legacy apps

Requirements of Each Workload?

2 sockets servers usually offer best ROI
Performance Requirements
App networking patterns - need SR-IOV?
Mix of servers to serve different workloads?

Will the Workloads Play Nice?

Trusted? Isolation between workloads
QoS policies
Primary Considerations

Workloads

Networking

Storage

Resiliency
 Networking
Primary Network Considerations

Isolation of Traffic Flows

Type of Infrastructure

Hardware Offload
 Networking
Typical Hyper-V Server Traffic Flows

Virtual Machines traffic

Cluster traffic
Storage traffic / CSV
Live Migrations
Management
 Workloads

Traffic Isolation / Separation Approaches

How many NICs should be in the server?
1. Separate NICs for different traffic flows
- WS2008 R2 best practice
2. Separate “Datacenter” traffic from “Tenant” Traffic
- Server 2012 common practice
3. Run everything through the virtual switch, one physical network
- Now possible with server 2012

Use Port ACLs, QoS, DCB & VM QoS

to enforce isolation and performance guarantees
 Networking

InfiniBand vs. 10GbE vs. 1GbE

10 Gb Ethernet
Great performance on familiar fabric
RDMA optional (for SMB 3.0 file access)
Offers Quality of Service (DCB) & Flexible bandwidth
10 GbE
allocation  
New offloads
…But physical switch ports more expensive
InfiniBand (32 Gb and 56 Gb) InfiniBand
Very high performance, low latency  
RDMA included (for SMB 3.0 file access)
…But Network management different than Ethernet
1 GbE
1 Gb Ethernet  

Adequate performance for many workloads

 Networking

Hardware Offloads for Scalability & Performance

Windows Server 2012 supports many Hyper-V Server

NIC features including: VM 1 VM n

Migration

Manage
Storage
• HW QoS via Datacenter Bridging (DCB)    

Live
   

• Remote Direct Memory Access (RDMA)    

Hyper-V
• Receive Segment Coalescing (RSC) Extensible
Switch
• Receive Side Scaling (RSS)  

• Virtual Machine Queue (VMQ) IPsecTO

RSS

• Guest IPsec Task Offload (IPsecTO) RDMA RSS RSC

VMQ SRIOV

• Single Root I/O Virtualization (SR-IOV) DCB DCB

     

NOTE: Offloads require capable HW, Windows Server 2012

drivers and potentially BIOS support (e.g. SR-IOV)
 Networking

Selecting the right NIC Offloads *

Some Performance offloads bypass
Windows functionality
Hyper-V Extensible Switch
Hyper-V Network Virtualization
NIC Teaming Flexibility Raw RDMA
RSS and Scalability Performance SR-IOV
VMQ
IPsecTO

* Per NIC constraints

Primary Considerations

Workloads

Networking

Storage

Resiliency
 Storage

Storage Selection Considerations

Cost / Performance
Block vs. File
Manageability
Vendor preference
Existing Investments
Approach to scaling
 Storage

Storage Scaling Approaches

Compute & Storage Scale Independently Compute & Storage Scale Together
Compute When? Compute/Storage
• Grow capacity on two axis When?
• Compute-bound VMs • Very low cost
Storage • Storage offloading • Disk-bound VMs
Compute/Storage

Hyper-V Servers Hyper-V Servers Hyper-V Servers Hyper-V Server

10 GbE / RDMA

File Servers
iSCSI / FC
SAS / iSCSI / FC
SAN Controller
Optional SAS
Controller
 Storage

Storage Scaling Approaches

Compute & Storage Scale Independently Compute & Storage Scale Together
Compute When? Compute/Storage
• Grow capacity on two axis When?
• Compute-bound VMs • Very low cost
Storage • Storage offloading • Disk-bound VMs
Compute/Storage

Hyper-V Servers Hyper-V Servers Hyper-V Servers Hyper-V Server

10 GbE / RDMA

File Servers
iSCSI / FC
SAS / iSCSI / FC
SAN Controller
Optional SAS
Controller
demo

Storage Spaces are

Awesome!
Primary Considerations
Workloads

Networking

Storage

Resiliency
What happens when hardware fails?
 Resiliency

Resiliency Approaches
Infrastructure Resiliency
VMs not designed to handle failures, H/A at server
level, Failover clustering as another layer of
protection, high-end servers, redundant power and
network gear
App-Level Resiliency
VMs designed to handle failures (e.g. Guest
Clustering) or downtime acceptable.
Lower End Industry Standard Server, single
infrastructure

Cluster Application-controlled failover

Hyper-V Failover Clustering / Guest clustering

VMVMs VMVMs VMs VMs

VMs VMs VM VM
Parent Parent Parent VMs Parent VMs
s s s s

               
       

Switch Switch
Switch
Switch
Building Your Cloud Infrastructure: Scenario Examples
1: Non-converged Enterprise Configuration

2: Converged Datacenter with File Server Storage

3: Converged Datacenter without Dedicated Storage Nodes

Optimized for Windows Server 2012
Validated at Microsoft’s Enterprise Engineering Center (EEC)
Step by Step guidance & PowerShell Script Samples
Available
http://technet.microsoft.com/en-us/library/hh831441
http://
 1: Non-converged Enterprise Configuration

Hyper-V Server
Easiest transition Path
• Physically isolated networks
VM 1 VM 2 • Regular SANs
• Similar to Windows Server 2008
Migration
   

Manage
Storage

R2 guidance

Cluster
   
Live

• Backward compatible design,

   
Hyper-V taking advantage of 10GbE, SR-
Extensible IOV, LBFO, RSS, RSC, VMQ, ODX
Switch

HBA / 1GbE /
10GbE
1 GbE 1 GbE 1 GbE 10 GbE + SR-IOV
 
         
FC / iSCSI

SAN
 1: Non-converged Enterprise Configuration

Hyper-V Server
Easiest transition Path
• Physically isolated networks
VM 1 VM 2 • Regular SANs
• Similar to Windows Server 2008
Migration
   

Manage
Storage

R2 guidance

Cluster
   
Live

• Backward compatible design,

   
Hyper-V Ext. Switch taking advantage of 10GbE, SR-
IOV, LBFO, RSS, RSC, DCB, VMQ,
NIC Teaming ODX
RSS DCB
10GbE 1GbE /
1 GbE 1 GbE 1 GbE 10 GbE + SR-IOV
 
         
FCoE / iSCSI

ODX
SAN
Scale & Resiliency for storage access
 1: Non-converged Enterprise Configuration

Hyper-V Server
Easiest transition Path
• Physically isolated networks
VM 1 VM 2 • Regular SANs
• Similar to Windows Server 2008
Migration
   

Manage
Storage

R2 guidance

Cluster
   
Live

• Backward compatible design,

   
Hyper-V Ext. Switch taking advantage of 10GbE, SR-
IOV, LBFO, RSS, RSC, VMQ, ODX
NIC Teaming
VMQ
HBA / 1GbE /
10GbE
1 GbE 1 GbE 1 GbE 10 GbE + SR-IOV
   
         
FC / iSCSI

SAN
Scale & Resiliency for VMs
 1: Non-converged Enterprise Configuration

Hyper-V Server
Easiest transition Path
• Physically isolated networks
VM 1 VM 2 • Regular SANs
• Similar to Windows Server 2008
Migration
   

Manage
Storage

R2 guidance

Cluster
   
Live

QoS
• Backward compatible design,
   
Hyper-V Ext. Switch taking advantage of 10GbE, SR-
Switch Extensions IOV, LBFO, RSS, RSC, VMQ, ODX
Network Virt.

HBA / 1GbE /
10GbE
1 GbE 1 GbE 1 GbE 10 GbE + SR-IOV
 
         
FC / iSCSI

SAN
Flexible & Extensible
 1: Non-converged Enterprise Configuration

Hyper-V Server
Easiest transition Path
• Physically isolated networks
VM 1 VM 2 • Regular SANs
• Similar to Windows Server 2008
Migration
   

Manage
Storage

R2 guidance

Cluster
   
Live

QoS
• Backward compatible design,
   
Hyper-V Ext. Switch taking advantage of 10GbE, SR-
Switch Extensions IOV, LBFO, RSS, RSC, VMQ, ODX
Network Virt.
SRIOV
HBA / 1GbE /
10GbE
1 GbE 1 GbE 1 GbE 10 GbE + SR-IOV
 
         
FC / iSCSI

Very High Scale VMs w/ SRIOV

SAN But, there are tradeoffs!
1: Non-converged Enterprise Configuration
Storage Scaling Approaches
Compute & Storage Scale Independently Compute & Storage Scale Together

Hyper-V Servers Hyper-V Servers Hyper-V Servers Hyper-V Server

10 GbE / RDMA

File Servers
iSCSI / FC
SAS / iSCSI / FC
SAN Controller
Optional SAS
Controller
 2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
VM 1 VM n • 10GbE Network(s)

Migration
Manage
•

Cluster /
File Server for VM storage
Storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
    existing FC/iSCSI SANs or
Hyper-V JBODs+Spaces
Extensible
Switch
  • Highlighted features: 10GbE
w/DCB, QoS, LBFO, Hyper-V over
SMB, Spaces

10GbE 10GbE
HBA 1/10GbE
(RDMA) (RDMA)
(Optional)
       
Datacenter Network Tenants Network
FC/SAS
iSCSI

SAN / JBODs
2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
VM 1 VM n • 10GbE Network(s)

Migration
Manage
•

Cluster /
File Server for VM storage
Storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
    existing FC/iSCSI SANs or
SMB File Share Hyper-V JBODs+Spaces
Extensible
Scale out FS Switch
  • Highlighted features: 10GbE
CSV w/DCB, QoS, LBFO, Hyper-V over
Space SMB, Spaces
Pool
10GbE 10GbE
SAS 1/10GbE
(RDMA) (RDMA)
       
Datacenter Network Tenants Network
SAS

Storage Spaces lowers cost of storage

JBODs
2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
VM 1 VM n • 10GbE Network(s)

Migration
Manage
•

Cluster /
File Server for VM storage
Storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
    existing FC/iSCSI SANs or
Hyper-V JBODs+Spaces
Extensible
Switch
  • Highlighted features: 10GbE
SMB File Share w/DCB, QoS, LBFO, Hyper-V over
Scale out FS SMB, Spaces
CSV
Clustered 10GbE 10GbE
1/10GbE
RAID (RDMA) (RDMA)
       
Datacenter Network Tenants Network
SAS

Using hardware RAID controllers

JBODs (Cluster In a Box, And more…)
2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
VM 1 VM n • 10GbE Network(s)

Migration
Manage
•

Cluster /
File Server for VM storage
Storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
    existing FC/iSCSI SANs or
Hyper-V JBODs+Spaces
Extensible
SMB File Share Switch
  • Highlighted features: 10GbE
Scale out FS w/DCB, QoS, LBFO, Hyper-V over
SMB, Spaces
CSV
LUN
10GbE 10GbE
HBA 1/10GbE
(RDMA) (RDMA)
       
Datacenter Network Tenants Network
FC

Also possible : Front-end the SAN with an SMB Share

SAN
2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
VM 1 VM n • 10GbE Network(s)

Migration
Manage
•

Cluster /
File Server for VM storage
Storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
    existing FC/iSCSI SANs or
Hyper-V JBODs+Spaces
Extensible
Switch
  • Highlighted features: 10GbE
w/DCB, QoS, LBFO, Hyper-V over
OS QoS OS QoS SMB, Spaces
DCB DCB
10GbE 10GbE
SAS 1/10GbE
(RDMA) (RDMA)
       
Datacenter Network Tenants Network
SAS

Using Qos & DCB

JBODs
2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
QoS VM 1 VM n • 10GbE Network(s)
New-NetQosPolicy

Migration
Manage
•

Cluster /
File Server for VM storage
Storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
Switch QoS Switch QoS
existing FC/iSCSI SANs or
   
Set-VMNetworkAdapter Hyper-V JBODs+Spaces
Extensible
Switch
DCB   • Highlighted features: 10GbE
Add-WindowsFeature Data-Center-Bridging w/DCB, QoS, LBFO, Hyper-V over
OS QoS OS QoS SMB, Spaces
Import-Module DcbQos
Enable-NetAdapterQos
DCB DCB
Enable-NetQosFlowControl
10GbE 10GbE
SAS 1/10GbE
New-NetQoSTrafficClass (RDMA)
(RDMA)
       
Datacenter Network Tenants Network
SAS

Using Qos & DCB

JBODs
 2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
VM 1 VM n • 10GbE Network(s)

Migration
Manage

Cluster /
•
Storage
File Server for VM storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
    existing FC/iSCSI SANs or
Hyper-V JBODs+Spaces
Extensible
Switch
NIC Teaming   • Highlighted features: 10GbE
NIC Teaming
w/DCB, QoS, LBFO, Hyper-V over
OS QoS OS QoS SMB, Spaces
DCB DCB
• Note: LBFO & RDMA can’t
10GbE 10GbE 10GbE
SAS
(RDMA) (RDMA)
10GbE
(RDMA) (RDMA)
1/10GbE coexist on same NICs.
             
Tenants Network
SAS
Datacenter Network

Use of NIC Teaming & Qos/DCB

SAN / JBODs
 2: Converged Datacenter with File Server Storage

“Green Field”
File Server Hyper-V Server
VM 1 VM n • 10GbE Network(s)

Migration
Manage
•

Cluster /
File Server for VM storage
Storage

Manage
Storage
Cluster

   

Live
    • Actual storage may be an
    existing FC/iSCSI SANs or
Hyper-V JBODs+Spaces
Extensible
Switch
• SMB Multichannel   • Highlighted features: 10GbE
• 2 Cluster Networks
•
w/DCB, QoS, LBFO, Hyper-V over
2 Networks for LM
• Multiple Mgmt IPs SMB, Spaces
• QoS + DCB

10GbE 10GbE 10GbE 10GbE

SAS (RDMA) 1/10GbE
(RDMA) (RDMA) (RDMA)
             
Tenants Network
SAS
Datacenter Network

RDMA with two Datacenter NICs

SAN / JBODs
2: Converged Datacenter with File Server Storage
Storage Scaling Approaches
Compute & Storage Scale Independently Compute & Storage Scale Together

Hyper-V Servers Hyper-V Servers Hyper-V Servers Hyper-V Server

10 GbE / RDMA

File Servers
iSCSI / FC
SAS / iSCSI / FC
SAN Controller
Optional SAS
Controller
demo

Migrating VM workload to
non-clustered host
3: Converged Datacenter without Dedicated Storage Nodes
Converged Compute +
Hyper-V Server Storage:
VM 1 VM n

Migration
Cluster /
• 10GbE Network(s), fully

Storage

Manage
   

Live
    converged, routing all traffic
    through the switch
 
Hyper-V
  Extensible
  Switch • Shared JBODs, scale compute
 
and storage together

• Highlighted features:
Storage Spaces on the Hyper-
SAS 10GbE V nodes, Hyper-V Switch
    ports for VM and Host traffic,
SAS QoS policies.
Converged Network

JBODs
3: Converged Datacenter without Dedicated Storage Nodes
Converged Compute +
Hyper-V Server Storage:
VM 1 VM n

Migration
Cluster /
• 10GbE Network(s), fully

Storage

Manage
   

Live
    converged, routing all traffic
    through the switch
 
Hyper-V
VM ACLs,
Extensible
 
NICs   QoS
Switch • Shared JBODs, scale compute
 
and storage together

• Highlighted features:
Storage Spaces on the Hyper-
SAS 10GbE V nodes, Hyper-V Switch
    ports for VM and Host traffic,
SAS QoS policies.
Converged Network

JBODs VM NICs, with ACLs + QoS on the Switch

3: Converged Datacenter without Dedicated Storage Nodes
Converged Compute +
Hyper-V Server Storage:
VM 1 VM n

Migration
Cluster /
• 10GbE Network(s), fully

Storage

Manage
   

Live
    converged, routing all traffic
    through the switch
 
Hyper-V
  Extensible
  Switch • Shared JBODs, scale compute
 
and storage together

• Highlighted features:
Storage Spaces on the Hyper-
SAS 10GbE V nodes, Hyper-V Switch
    ports for VM and Host traffic,
SAS VLANs QoS policies.
Converged Network
Cluster/Storage
Live Migration
JBODs Setup VLANs to separate traffic flows
Management
Tenants
3: Converged Datacenter without Dedicated Storage Nodes
Converged Compute +
Hyper-V Server Storage:
VM 1 VM n

Migration
Cluster /
• 10GbE Network(s), fully

Storage

Manage
   

Live
    converged, routing all traffic
    through the switch
 
Hyper-V
  Extensible
  Switch • Shared JBODs, scale compute
 
CSV and storage together
Space
• Highlighted features:
Pool
Storage Spaces on the Hyper-
SAS 10GbE V nodes, Hyper-V Switch
    ports for VM and Host traffic,
SAS QoS policies.
Converged Network

JBODs Spaces and CSV directly on the Hyper-V Servers

3: Converged Datacenter without Dedicated Storage Nodes
Converged Compute +
Hyper-V Server Storage:
VM 1 VM n

Migration
Cluster /
• 10GbE Network(s), fully

Storage

Manage
   

Live
    converged, routing all traffic
    through the switch
 
Hyper-V
  Extensible
Switch • Shared JBODs, scale compute
 
CSV and storage together
Space
• Highlighted features:
Pool
10GbE
Storage Spaces on the Hyper-
SAS RDMA
1/10GbE V nodes, Hyper-V Switch
      ports for VM and Host traffic,
SAS
CSV Traffic
QoS policies.
Converged Network

JBODs Add RDMA NICs for fast CSV Redirection

3: Converged Datacenter without Dedicated Storage Nodes
Storage Scaling Approaches
Compute & Storage Scale Independently Compute & Storage Scale Together

Hyper-V Servers Hyper-V Servers Hyper-V Servers Hyper-V Server

10 GbE / RDMA

File Servers
iSCSI / FC
SAS / iSCSI / FC
SAN Controller
Optional SAS
Controller
4: DAS, Non-Clustered Configuration

Hyper-V Server Very Low Cost (Huge Scale

VM 1 VM n
Clouds)

Migration

Manage
   

Live
   

   
• 10GbE Network(s), fully
 
Hyper-V converged
Extensible
  Switch
• No clusters, no H/A, local
  DAS
• For cloud-ready applications
DAS
• Highlighted feature: Live
migration w/o clusters
 

Converged Network
4: DAS, Non-Clustered Configuration
Storage Scaling Approaches
Compute & Storage Scale Independently Compute & Storage Scale Together

Hyper-V Servers Hyper-V Servers Hyper-V Servers Hyper-V Server

10 GbE / RDMA

File Servers
iSCSI / FC
SAS / iSCSI / FC
SAN Controller
Optional SAS
Controller
demo
Migrate from SMB storage
to local
Primary Considerations

Workloads

Networking

Storage

Resiliency
Windows Server 2012 is Cloud Optimized

Multi-Tenant High Scale & Low Manageable

Clouds Cost Datacenters & Extensible
Related Content
Breakout Sessions
WSV313 Diving into SMB 3.0
VIR316 Comparing Private Cloud Capabilities
VIR314 Hyper-V Live Migration and Live Storage Migration
WSV316 File System Enhancements: ReFS and Storage Spaces
WSV321 NIC Teaming and Multichannel
TechEd 2012; WSV313 Solutions
WSV325 File and Storage Services Management

Find Me Later At…

Twitter @OlikkaTech
Blog http://bolikka.com
 © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the
part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.