Windows 2012

Fail-over Clustering
Geo Paul/ March 15, 2013

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Impact Areas

• Scalability & Total Cost of Ownership • Quorum Model Improvements

(TCO) • High Availability Roles
• Tighter Integration with Hyper-V • Cluster Storage
• Patching Failover Clusters • Cluster Networking
• Active Directory Interoperability • Resource Hosting Subsystem (RHS)
• Cluster Validation Process • Usability Improvement

2 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Scalability & TCO

Windows 2008 & Windows 2008 R2 Windows 2012

16 nodes 64 nodes

1000 Virtual Machines 4000 Virtual Machines

384 Virtual Workloads per node 1024 Virtual workloads per node

Enterprise & Data center Edition Standard & Data enter edition

3 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Tighter Integration with Hyper-V
•Infrastructure as a Service (IaaS) for Cloud scenarios (Private, Public, Hybrid
– Failover Clustering is considered the Infrastructure as a Service (IaaS) piece for Cloud scenarios (Private or Public)
•Virtual Machine failover prioritization
–priorities can be configured to control the order in which a virtual machine fails over or starts
•Enhanced Failover Placement
– Intelligence is built into the cluster service that determines the resources available on each candidate node before making placement
decisions
•Virtual Machine Mobility scenarios
– More than one simultaneous Live Migration
– Storage Migration and Hyper-V Replica
•Application Monitoring
– Solution to monitor applications running in a virtual machine
– Provide automatic corrective action such as restarting the VM, restarting a service within VM, or even migrating it to another node
• Guest Clustering improvements
– iSCSI-shared storage limitation removed
4
– Virtual Fibre Channel adapters (up to four virtual HBAs can be assigned to a single Guest)
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Patching Fail-over cluster

• Patching a cluster is an intensive, manual process requiring a good deal of planning and coordination
• Cluster-Aware Updating (CAU) feature provides inbox, automated process for patching a cluster
– Can be run outside of the cluster
– Can be run as part of the CAU Role implemented in a cluster
– Totally hands-off once a CAU run is started
• Works seamlessly with WU/MU or with WSUS
• If cluster is self-updating, it must have CAU role installed
• Foundational Technologies:
– Failover Clustering feature
– Windows Update Agent (WUA)
– Windows PowerShell 3.0
– Windows Management Instrumentation (WMIv2)
– Windows Remote Management (WinRM)

5 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Cluster-Aware Updating – Functional Description

• CAU is cluster-aware, completely automated solution

• Extensible architecture
– Works with customer installers or plug-ins (
– Can be made to provide updates such as BIOS, NIC drivers, HBA drivers, etc…
• High-level process description

a) CAU run is initiated g) Node is rebooted if needed

b) CAU scans for and downloads applicable updates h) Process continues until node is completely
to each cluster node patched
c) CAU verifies cluster has quorum i) Node is brought out of Maintenance Mode
d) Pre-update custom scripts are run if needed j) Post-update scripts are run as needed
e) Roles are moved off node to be patched and node k) CAU moves on to the next node
is placed in Maintenance Mode l) Cluster patching completes and reports are
f) Updates are installed on nodes one at a time generated and placed on each node
6 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Installing CAU

• CAU role is installed using the CAU UI or the PowerShell cmdlet Add-CauClusterRole
• CAU role is not visible in Failover Cluster Manager. Run PowerShell cmdlet Get-CauClusterRole to
determine which node is the owning node
• Installed as part of the Failover Clustering feature

Dependencies
• Failover Cluster management tools
• .NET 4.x Framework installed
• Remote Management enabled on all cluster
nodes
• PowerShell Remoting enabled on each cluster
node (Enable-PSRemoting )
• Remote Shutdown firewall rule enabled on each
cluster node

7 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Active Directory Interoperability

• Computer Objects created in AD since Windows Server 2008

– CNO (Cluster Name Object) and VCO (Virtual Computer Object)
– Deletion of cluster COs, especially the CNO, is a common issue
• Prevent accidental deletion of CNO
– Requires Deny ACE be manually set beforehand in case of custom containers
• Repair option for VCO COs
• Intelligent placement of COs associated with a cluster
• Can specify a DN when creating a cluster
• AD-less bootstrapping of a cluster
• Cluster refresh of LastLogonTimeStamp
• RODC support

8 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Active Directory Integration

• Cluster service continues to use local system account (CLIUSR)

• Cluster common identity is the CNO (Cluster Name Object)
• Cluster Computer Objects still placed in Computers container
– Can be moved
– Can be pre-staged
– Can use LDAP Distinguished Name (DN) when creating a cluster
• Example: CN=Contoso-FC1,OU=Clusters,DC=Contoso,DC=Com
– CNO is placed in same container as cluster nodes
– VCO are placed in same container as CNO
• When using custom container, ensure CNO has right to Create Computer Objects
• CNO is now protected from accidental deletion
• Enable Active Directory Recycle Bin
• CNO rotates password and VCO passwords based on domain policy

9 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Domain-less cluster bootstrapping

• Legacy cluster required communication with a Domain Controller (PDC) before the cluster service could
start
• Virtualizing Domain Controllers in Hyper-V Failover Clusters were initially problematic as cluster could not
contact a DC to start the cluster service and bring resources online
• In WS2012, Failover Clusters no longer require DC connectivity to start. Uses local identity (CLIUSR) and
NTLM authentication
• NTLM using local account is fall back to Kerberos
– Local shared common identify is CLIUSR (local account) which uses NTLM
– Password stored in cluster database
– Password is rotated every 30 days
• Cluster Authorization Manager (CAM) is part of the cluster service architecture
– CAM uses local identity to control access to CSVv2 volumes

10 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Failover Clusters and Read-Only Domain Controllers

• Ability to deploy clusters to Branch Offices and DMZs

• Under normal conditions, clusters need R-W Domain Controllers
• Deploying Failover Clusters in an RODC environment (assumptions)
– Healthy AD infrastructure exists at a central site
– Forest functional level is Windows 2003 or higher
– AD Schema is updated to at least Windows Server 2008 R2 level
– Windows Server 2008 R2 ADPrep tool has been run in target domain
– At least one R_W DC is running Windows Server 2008 R2
– WAN connectivity is needed during password related operations and during cluster creation process and
whenever a new CAP is created
– All nodes in the same cluster must be physically co-located and directly connected to the same RODC

11 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Cluster Validation Process

• Storage validation process streamlined so it runs faster

• Verification of CSV requirements
• Can validate a single LUN
• Multi-site aware. Tests exclude replica LUNs
• Hyper-V tests included when Hyper-V role is installed
• Binary verification of hotfixes across all nodes if discrepancies are found with respect to installed hotfixes

12 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Failover Cluster Validation
• Series of tests that validates the hardware configuration to determine suitability for clustering
• Validation can be done before or after creating a cluster
– Recommended to do it before and after
• Cluster Configuration tests are not run before
• New cluster validation tests
– Hyper-V tests
• Only if Hyper-V role is installed on at least one node in the cluster
• Additional tests if Hyper-V Replica support is enabled
• Tests run against the virtual machines hosted in the cluster
• Failures in Hyper-V test does not mean cluster is not supported
– Storage test for Storage Spaces and Pools
– CVS tests included in Storage tests
– Can test a single disk using PowerShell (Test-Cluster –Disk), e.g: Test-Cluster -Disk "Cluster Disk 1"

Powershell :
Test-Cluster -Node node1.testlab.com,node2.testlab.com
13
Test-Cluster -Cluster testcluster
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Quorum Model Enhancements
•Dynamic Quorum model - new in Windows Server 2012
– Cluster manages the number of votes required to keep functioning
– Allows for cluster survival when >50% of the nodes are down
– Enabled by default
– Works with all Quorum models except Disk Only
•Implementation of Node Vote Weight
– Controls which nodes have a vote in determining quorum
– Useful in multi-site cluster scenarios
– Works with all Quorum models except Disk Only

14 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Dynamic Quorum model
• Implemented to provide higher availability within a cluster by continuously monitoring the quorum
model and making adjustments based on the active cluster membership
• This model allows a cluster to continue to function when less than 50% of the nodes are active
• Dynamic Quorum model is enabled by default when a cluster is created
• Using the Get-Cluster PowerShell cmdlet, the property DynamicQuorum is set = 1
• Dynamic Quorum can be manipulated using PowerShell or the Configure Cluster Quorum Wizard
• When a node leaves the cluster (Node is shut down), the node adjusts its own weight as part of the
shutdown of the cluster service and quorum is re-calculated by the remaining nodes

15 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Configuring Quorum Model in Windows Server 2012
• Use Configure Cluster Quorum Wizard in Failover Cluster Manager
– Use Typical Settings (Default and Recommended)
• Uses logic implemented in Windows Server 2008 (+R2)
– Add or Change Quorum Witness
• Allows for manual selection of a witness resource
– Configure Disk Witness (even # of nodes)
– Configure File Share Witness (multi-site clusters)
– Configure No Witness (odd # of nodes)
– Advanced Quorum Configuration and Witness Selection
• Configure Node Weights
– All Nodes
– Select Nodes
– No Nodes (witness disk is required and cannot fail)
• Use Set-ClusterQuorum PowerShell cmdlet to set the quorum model
• Use Get-ClusterQuorum PowerShell cmdlet to list the quorum details
16 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Node weighting
• The concept of node weighting is primarily implemented in a multi-site cluster scenario
• Node weights used to control the vote calculation process.
• Nodes are assigned a weight=0 if they will not be included in the quorum calculation by Quorum Manager
• Typical configuration is nodes at the Primary site have votes but nodes at DR site may not
– Determining factor is whether or not automatic or manual failover will be executed
• Does not work with “Disk Only Quorum model”
• Cluster Node Weights can be assigned using the Configure Cluster Quorum Wizard and by selecting the
option for Advanced Quorum Configuration and Witness Selection

17 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HA Roles
• Cluster-Aware Updating (CAU) – Not listed in “Configure Role
section”
• DFS Name Server
• DHCP Server
• Distributed Transaction Coordinator (DTC) Deprecated Roles:
• File Server (File Server Role)
Highly Available Print Server (Spooler)
– Scale out file server
Remote Desktop Connection Broker Roles
– Generic Application\Script\Service
• Hyper-V Replica Broker (Hyper-V Role)
• iSCSI Target Server (part of File Server Role)
• iSNS Server (iSNS Server Feature)
• Message Queuing (Message Queuing Services Feature)
• Other Server
• Virtual Machine (Hyper-V role)
• WINS Server (WINS Server Feature)
18 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
• Task Scheduler (created in a hidden group, not visible in FCM)
Cluster Storage
• Cluster Shared Volumes version 2 (CSVv2)
– High speed CSV I/O redirection
• Integration with SMB multi-channel and SMB Direct resulting in zero downtime for connected client s
– Multi-subnet support (e.g. Multi-site cluster configurations)
– CSV diagnostic Logs (Application and Services Logs\Microsoft\Windows\)
• FailoverClustering-CsvFlt
• FailoverClustering-CsvFs
• Improved Persistent Reservation algorithm which is less demanding on storage arrays
– PR keys are now unique to each node
• Increased logging to the cluster log during physical disk online process
• Integration with Storage Spaces feature (Storage pool)
– Provide highly available Simple and Mirrored spaces (Parity not supported)
• iSCSI Software Target integration
• Maximizing file system availability - Failover Clusters take advantage of the new file system check and
repair functionality (CHKDSK)
19 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Cluster Shared Volumes (CSV) Version 2.0
• CSV first introduced in Windows Server 2008 R2
• CSV provides the following:
 Distributed file access
 Concurrent access to the CSV volume from any node in the cluster
 Physical location of the data is transparent to the application
 In case of Hyper-V, multiple VMs can share a common CSV volume
 Live Migration of VMs is faster because storage ownership does not have to change
• CSV is compatible with any block level storage attached using FC, iSCSI, or SAS
• CSV requires the NTFS file system residing on either an MBR or GPT disk volume
• CSV file system CsvFs (CsvFs.sys) is mounted on top of NTFS volumes
• CSV in WS2012 supports arbitrary workloads as evidenced by supporting SMB shares in the Scale-Out File
Services Role
– CSV takes advantage of SMBv3 features like multi-channel and Direct Access (RDMA)
• Right-click on a disk to add it to the CSV Namespace
• Get-ClusterSharedVolume PowerShell cmdlet is ued to get CSV information
20 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CSV requirements
• Client for Microsoft Networks and File and Printer Sharing protocols enabled
• NTFS formatted partitions are required. ReFS is not supported in this release
• NTLM authentication with a Domain Controller is no longer required. NTLM authentication between cluster
nodes is used
• Cluster validation process verifies basic requirements like protocol bindings and SMB share access to the
CSV default share

21 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
WS2012 CSV Improvements

• CSV Namespace enabled by default

• Integration of CSV with more workloads (e.g. Scale-Out File Server)
• Improved Backup and Restore of CSV volumes
• Improved performance with block level I/O redirection and Direct I/O for more scenarios
• Support for Storage Spaces and Pools (iSCSI Target Server not supported)
• CSV scales to 64-Node clusters
• CSV works with SANs, FCoE, iSCSI, SAS arrays
• Zero downtime for planned and unplanned failovers of SMBv3 shares
• Multi-subnet support (e.g. multi-site clusters)
• No external authentication dependencies. Can operate without authenticating with DC.

22 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Cluster Networking

• NetFT virtual adapter driver updated to NDIS 6.3

• CSV is now supported on multi-subnet clusters
• Increased resiliency for underlying network failures
– Prevent nodes from being removed from cluster membership due to
transient network failures
– Improved regroup algorithm
• NetFT provides additional information in the cluster log
(LogLevel=5)
– Cluster parameter RouteHistoryLength (Default = 10)
• Automatic detection of iSCSI networks
• NetFT MAC address regeneration on conflict
– Allows OS to be sysprep’d with Failover Cluster feature installed
• SMB shares accessible via CAP, IP address, and DNS CNAME

23 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
 Resource Hosting Subsystem (RHS)

• RHS resource reattach process

– Reduces impact of RHS deadlocks on other resources
• Isolation of core cluster resources
– Any resource supported in clusres.dll runs in a dedicated core RHS
process
– Physical disk resources all run in a dedicated ‘storage’ RHS process
– 3rd Party resources run in a dedicated RHS process
• RHS Bugcheck on deadlock
– Requires a registry key be set
– “HKLM\Software\Microsoft\Windows NT\CurrentVersion\Failover
Clusters\DebugBreakOnDeadlock, REG DWORD = 3)” will force kernel
mode memory dump
• Better eventing (Event ID 1254) when resources reach terminal
state (failure thresholds) and failovers will no longer be
attempted
24 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Usability Improvement

• Failover Cluster Manager interface

– Multi-select items in the UI and execute an action such as Online, Offline, Live Migrate, etc…
– Move Core Resource and Available Storage groups in the UI
– Automated node draining - which moves all resource groups off a node and distributes them to other cluster nodes based on
configured policies and settings.
– Use of filtering
• Server Manager
– Failover Cluster Manager and Cluster-Aware Updating UI are available in the Menu bar under Tools
• Improved data retention in cluster log
• Ability to generate a cluster log using local time
• Ability to connect to SMB shares using IP address and CNAME
• Hyper-V Manager blocks the management of HA VMs
• More detailed information for cluster errors in FCM

25 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Scale-Out File Services

• Takes advantage of :
– New client access method by way of a new cluster resource type (Distributed Network Name (DNN))
– Cluster Shared Volumes v2 (CSVv2)
– SMBv3.0 . Cient SMB connections being distributed across all nodes in the cluster
• Supports SMB and NFS shares
• Provides for Continuous Availability using Transparent Failover.
• Hyper-V and SQL Server takes benefit of this feature

26 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Installing Failover Cluster Feature

• Not much has changed here

– Server Manager “Add and Remove Roles” Wizard
– PowerShell using Server Manager module (Install-WindowsFeature)
• Importance of –IncludeManagementTools

• Deprecated functionality
– Failover Cluster Automation Server
– Failover Cluster Command Interface (cluster.exe CLI)

27 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Creating a Failover Cluster

• Create a cluster using Failover Cluster Manager or PowerShell (New-Cluster)

• Can use a Distinguished Name (DN) format to place CNO in specific AD container
– Example: CN=Contoso-FC1,OU=Clusters,DC=Contoso,DC=Com
• Cluster objects follow cluster nodes in AD containers
– Exclusive of pre-staging
• Capability of adding all storage into the cluster

Powershell:
• New-Cluster -Name testcluster -Node node1.testlab.com,node2.testlab.com -StaticAddress 192.168.20.150

• If DHCP is used and you do not want all storage added by default:
New-Cluster -Name testcluster -Node node1.testlab.com,node2.testlab.com -IgnoreNetwork -NoStorage

28 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Failover Cluster Logging and Reporting

• Eventing model carries forward from Windows Server 2008

– More channels in WS2012
• Failover Cluster logging enabled with feature installation
• Cluster logging channels under Application and Services Logs\Microsoft\Windows
– Modify Event Viewer to show Analytic and Debug Logs
• Failover Cluster has six channels and Cluster-Aware Updating (CAU) has two
• Logs are stored %systemroot%\System32\winevt\logs
• Logs that are manually enabled by a user (Diagnostic and Analytic) show up under Event Trace Sessions
and\or Startup Event Trace Sessions in Performance Monitor
• Events can also be monitored in Server Manager
– Events Tile.
– Queries can be created and saved
• Cluster events can also be viewed in FCM and custom queries can be created

29 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Failover Cluster Log

• Primary log used for troubleshooting clusters

• Troublesome in the past due to ‘gaps’ in the log
• In WS2012, the cluster log is now a circular analytic log that persists through reboots of a cluster node thus
greatly reducing the possibility of ‘gaps’ in the log
• Default cluster log size is 300 MB and default cluster logging level is 3
– These can be changed using PowerShell
• Cluster log can now be saved in local time
– In the past it has been GMT
• Cluster log by default is deposited in %systemroot%\Cluster\Reports directory

30 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Powershell commands

• Install feature: Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools

• Test-Cluster -Node node1.testlab.com,node2.testlab.com
• To import fail-over cluster module: import-module failoverclusters
• To get a list of all Failover Clustering cmdlets: get-command –module failoverclusters
• create new clusterr: New-Cluster -Name <cluster name> -Node <node1>,<node2> -StaticAddress <ip
address>
• To list cluster groups: Get-ClusterGroup
• Bring group online: Start-ClusterGroup <groupname>
• Bring group offline: Stop-ClusterGroup <groupname>
• To list resources: Get-ClusterResource
• Bring resource online: Start-ClusterResource "<resourcename>"
• Bring resource offline: Stop-ClusterResource "<resourcename>"
• To fixquorum, equivalent to net start “clussvc /forcequorum” command: Start-ClusterNode –FixQuorum
• To stop cluster: stop-cluster <cluster nmae>
31 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Powershell commands continued..
• To start cluster:
start-cluster <node name>
start-clusternode <node name> -cluster <cluster name>
• To list resource type: Get-ClusterResourcetype
• Failover cluster role cmdlets
Add-CauclusterRole (Part of CAU PS Module)
Add-ClusterFileServerRole
Add-ClusterGenericApplicationRole
Add-ClusterGenericScriptRole
Add-ClusterGenericServiceRole
Add-ClusteriSCSITargetServerRole
Add-ClusterScaleOutFileServerRole
Add-ClusterServerRole
Add-ClusterVirtualMachineRole
32 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Additional Resources

• Mapping cluster.exe commands to Powershell

http://technet.microsoft.com/en-us/library/ee619744(v=ws.10).aspx
• Failover Cluster PowerShell Cmdlets
http://technet.microsoft.com/library/hh847239.aspx
http://technet.microsoft.com/en-us/library/ee461009.aspx
• Cluster-Aware Updating (CAU) PowerShell Cmdlets
http://technet.microsoft.com/library/hh847221.aspx
• CAU Scenario Overview
http://technet.microsoft.com/library/hh831694.aspx
• TechNet Information on Migrating a Clustered DHCP Server
http://technet.microsoft.com/en-us/library/ee460952(v=WS.10).aspx

33 © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.