Emr Na c04702054 1

Technical white paper
vSphere 6.0 with HP ProLiant Gen9

Servers, OneView, 3PAR, Cisco Nexus
5600 and Brocade 6510 Deployment
Guide
Updated: 4/30/2015
Hongjun Ma, HP DCA

vSphere 6.0 with HP ProLiant Gen9 Servers, OneView, 3PAR, Cisco Nexus 5600 and Brocade 6510 Deployment Guide
Table of contents
Introduction ............................................................................................................................................................................................... 3
Testing Physical Topology ..................................................................................................................................................................... 4
Network Configuration ........................................................................................................................................................................... 6
Storage Fabric Configuration ..............................................................................................................................................................14
OneView Uplink Configuration Summary .........................................................................................................................................20
OneView Server Profile and Volume Configuration .......................................................................................................................21
vSphere 6.0 Network and Storage Validation .................................................................................................................................25
Conclusion ...............................................................................................................................................................................................27
Page 2
Introduction
This document describes the deployment procedures for building a comprehensive infrastructure that uses technologies from HP,
VMware, Cisco, and Brocade using vSphere 6.0 and HP ProLiant Gen9 servers. The document is an extension of the original
OneView Gen8 vSphere5.5 Deployment Guide, which includes some initial configuration tasks not specified here.
The hardware and software for vSphere 6.0 deployment testing include the following:
Infrastructure Products Version

Components
Blade server H/W HP BladeSystem C7000 Chassis with ProLiant BL460 Gen9 servers SPP2014.11
Blade server I/O HP Virtual Connect FlexFabric-20/40 F8 Module for c-Class BladeSystem 4.31
Storage HP 3PAR StoreServ 7400 2-node system 3.2.1(MU2)P12
Networking Cisco Nexus 5672UP Switches NX-OS 7.0(1)N1(1)
SAN Fabric Brocade 6510 Switches FOS: v7.0.2c
Management S/W Products Version
Virtual Machine VMware vCenter and vSphere 6.0
Management
Infrastructure HP OneView infrastructure management virtual appliance 1.20
Management
Storage 3PAR StoreServ Management Console 2.0
Management
Although this is the default base design, each of the components can be scaled to support specific business requirements. For
example, different quantities or combinations of blades and chassis can be deployed to increase compute capacity; additional disk
shelves can be deployed to improve I/O capacity and throughput; and special hardware or software features can be added to
introduce new features.
Page 3
Testing Topology
HP BladeSystem C7000 chassis is loaded with HP ProLiant BL460c Gen9 blades. All blade servers I/Os are provided by Virtual
Connect FlexFabric 20/40-F8 interconnect module.
HP 3PAR StoreServ 7400 provides server storage for VMware datastore. 3PAR StoreServ not only provides Tier-1 storage
performance and features like Adaptive Optimization but also is integrated with HP OneView infrastructure management platform
for easy of provisioning.
Ethernet networking consists of a pair of Cisco Nexus 5672UP switches. Nexus 5600 series are upgraded models over Nexus
5500 switches. The same vPC configuration guidelines with HP Virtual Connect modules discussed in this paper also apply to
other Cisco Nexus product lines include Cisco Nexus 9000, Nexus 7000, Nexus 6000 and Nexus 5500. We choose Nexus 5600
platform because Cisco Nexus 5500 are widely deployed and connected with Virtual Connect in Data Center. We wanted to
validate the latest Virtual Connect FlexFabric-20/40 F8 module continues to inter-operate with Nexus 5600/5500. Virtual Connect
can also work with other data center switches from HP Network, Brocade, Arista, Juniper, Extreme, etc.
Future OneView release will support FCOE uplink from Virtual Connect to Nexus switches. 40G uplink testing of Virtual Connect
FlexFabric-20/40 F8 module has been covered in another separate technical white paper.
SAN fabric consists of a pair of Brocade 6510 switches. The 6510 switch is the latest Brocade Gen 5 Fibre Channel product. It
offers 48x16G FC ports. We tested 8G FC speed because Virtual Connect and 3PAR StoreServ support up to 8G FC interface. We
Page 4
choose Brocade because of its leadership position in data center Fibre Channel fabric market. Meanwhile, HP OneView has built-in
integration with Brocade Network Advisor to automatically provision SAN fabric for automatic zoning between servers and Fibre
Channel storage array. HP OneView and Virtual Connect module can also work with Cisco MDS FC switches.
Server infrastructure management is provided by HP OneView software. HP OneView is HP converged infrastructure platform
managing server provisioning, monitoring, LAN and SAN networking configuration tasks.
HP OneView integrates functions of all traditional HP management software such as Onboard Administrator, Virtual Connect
Manager, Virtual Connect Enterprise Manager and Insight Control software. Beyond it, it also has built-in Brocade SAN Fabric as
well as HP 3PAR StoreServ storage provisioning integration.
Page 5
Network Configuration
The two Nexus 5672 UP switches are set up as vPC cluster. When deploying Virtual Connect with Nexus switches vPC technology,
typically one Virtual Connect module will have one link connecting to switch-1 and another link connecting to switch-2. On Switch
side, users should config one vPC port-channel on both switches for the vPC switches communicating with this Virtual Connect
module. Users should follow the same above cabling and configuration guidelines for the second Virtual Connect module.
In summary, each Nexus switch should have two vPC port-channels, the first one is for communication with the first virtual
connect module and the second one is for communication with the second virtual connect module.
In our testing,
 Eth1/2 on both Nexus 5672 switches are connected with the first VC FF-20/40 F8 module in I/O bay 1 and the two
Eth1/2 are bundled under vPC port-channel 101.
 Eth1/3 on both Nexus 5672 switches are connected with the second VC FF-20/40 F8 module in I/O bay 2 and the two
Eth1/3 are bundled under vPC port-channel 102.
Note: The most common mistakes when configuring Virtual Connect with Nexus vPC are a) not following the above cabling
guidelines and b) try to config one port-channel bundling four links together across Nexus and Virtual Connect. Virtual
Connect does NOT support LACP across two modules so two vPC port-channels should be created.
Users can verify cabling between Virtual Connect and Nexus by using “show lldp neighbor” command.
The following CLI captures verified the first VC FF-20/40 F8 module X5 port is connected with Nexus 5672-01 Eth1/2 and its X6
port connected with Nexus 5672-02 Eth1/2. The second VC FF-20/40 F8 module X5 port is connected with Nexus 5672-01 Eth
1/3 and its X6 port connected with Nexus 5672-02 Eth1/3.
Page 6
DIA-5672-01# show lldp neighbors

Device ID Local Intf Hold-time Capability Port ID
...
VC2040F8-TW2338001W Eth1/2 120 X5
VC2040F8-TW2338000M Eth1/3 120 X5
...
DIA-5672-02# show lldp neigh

Device ID Local Intf Hold-time Capability Port ID
...
VC2040F8-TW2338001W Eth1/2 120 X6
VC2040F8-TW2338000M Eth1/3 120 X6
...
When configuring Nexus port VLAN trunking with HP Virtual Connect modules, there are several things to note:
 Treat virtual connect ports as regular host trunk ports. Virtual Connect does NOT use STP to communicate with
upstream switches. For control signaling, Virtual Connect only uses standard LLDP and LACP with upstream switches for
neighbor discovery and port bundling.
 Typically users will configure trunk ports on Nexus switch when connecting with Virtual Connect in the context of
hypervisor environment.
 Users should config STP edge port trunk for ports connecting with Virtual Connect. This is to prevent switches STP
blocking Virtual Connect uplinks for up to 30 seconds upon its connectivity restoration in the event of module upgrade
and link failure.
 Optionally, one VLAN could be selected as native VLAN for the trunk but OneView Uplink configuration should also mark
this VLAN as native.
 Optionally, users can enable BPDU guard under the interfaces or enable them as global default setting to safeguard
switching domain from any devices in the server edge.
The following capture shows the vPC port-channel 101 configuration on both Nexus 5672 switches to connect with the first
Virtual Connect Module.
DIA-5672-01# show run int po101 membership
interface port-channel101
switchport mode trunk
switchport trunk allowed vlan 10,160-161,201
spanning-tree port type edge trunk
spanning-tree bpduguard enable
speed 10000
vpc 101
interface Ethernet1/2
channel-group 101 mode active
speed 10000
vpc 101
Page 7
The following capture shows the vPC port-channel 102 configuration on both Nexus 5672 switches to connect with the second
Virtual Connect Module.
speed 10000
vpc 102
speed 10000
vpc 102
The following capture shows the vPC status on both nexus 5672 switches.
DIA-5672-01# show vpc
...
vPC Peer-link status

---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up 1,10,40,160-164,201
vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
101 Po101 up success success 10,160-161,
201
201
DIA-5672-02# show vpc

...
vPC Peer-link status

---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up 1,10,40,160-164,201
vPC status
Page 8
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
201
The following capture verifies the switches received 0 BPDU packets from Virtual Connect.
DIA-5672-01# show spanning-tree int po101 detail
...
Port 4196 (port-channel101, vPC) of VLAN0160 is designated forwarding
Port path cost 1, Port priority 128, Port Identifier 128.4196
Designated root has priority 160, address 002a.6a13.c3c1
Designated bridge has priority 61600, address 002a.6ae0.a541
Designated port id is 128.4196, designated path cost 3
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
The port type is edge by port type edge trunk configuration
Link type is point-to-point by default
Bpdu guard is enabled
BPDU: sent 0, received 0
Users can verify Virtual Connect physical connectivity to Nexus switches in OneView “Interconnect” section. The following captures
module serial number listed in Nexus show lldp neighbor commands.
Page 9
The following capture verified Virtual Connect uplink port Nexus LLDP neighbor information.
Page 10
The following capture shows the first VC FF-20/40 F8 module LACP uplink configuration corresponding to vPC port-channel 101
on both switches.
Page 11
The following capture shows the second VC FF-20/40 F8 module LACP uplink configuration corresponding to vPC port-channel
102 on both switches.
Page 12
Page 13
Storage Fabric Configuration
SAN Fabric consists of a couple of Brocade 6510 switches running Fabric OS. On HP VC FF-20/40 F8 modules, X7 and X8 for each
module are configured as native FC ports with 8G FC transceivers plugged in. The first VC module is connected with 6510-1 switch
for SAN Fabric-A access. The second VC module is connected with 6510-2 switch for SAN Fabric-B access. 3PAR 7400 port
0:2:3(Node1:Slot2:Port3) and 1:2:3(Node2:Slot2:Port3) are connected with 6510-1 for SAN Fabric-A access while port 0:2:4 and
1:2:4 are connected with 6510-2 for SAN Fabric-B access.
The above 3PAR mirrored cabling between 7400 paired nodes can support 3PAR Persistent Ports feature, which can provide
transparent traffic switch over to server workload upon the node failure/upgrade.
SAN Fabric switches need to provision the following tasks:
 Turn on NPIV for ports connecting with Virtual Connect and 3PAR 7400
 Optionally set up zone alias
 Zoning manually or automatically (through Brocade Network Advisor integration with OneView)
Page 14
Step by Step configuration of Brocade manual zoning and Brocade Network Advisor(BNA) auto zoning is covered in OneView Gen8
vSphere5.5 Deployment Guide . In this setup, BNA auto zoning with OneView was used for quick FC fabric provisioning.
DIA-BSAN-01:FID3:admin> zoneshow
...
Effective configuration:
...
zone: Gen9vSphere6Blade6_vhba0
10:00:ce:16:0f:90:00:2c
20:23:00:02:ac:00:ad:ec
21:23:00:02:ac:00:ad:ec
10:00:ce:16:0f:90:00:30
20:23:00:02:ac:00:ad:ec
21:23:00:02:ac:00:ad:ec
DIA-BSAN-02:FID128:admin> zoneshow
...
Effective configuration:
...
10:00:ce:16:0f:90:00:2e
21:24:00:02:ac:00:ad:ec
20:24:00:02:ac:00:ad:ec
10:00:ce:16:0f:90:00:32
20:24:00:02:ac:00:ad:ec
21:24:00:02:ac:00:ad:ec
On OneView side, the uplink connectivity and login status could be viewed in Interconnect section for two Virtual Connect modules.
In OneView, two FC networks were created, one for Fabric A going up to Brocade 6510-1 and one for Fabric B going up to Brocade
6510-2. The two FC networks were mapped to Brocade Network Advisor (BNA) fabrics discovered.
Page 15
Page 16
Two FC uplink sets were created in Logical Interconnect section carrying the two FC networks created with the physical ports
assigned to them.
Page 17
In order for end to end storage volume and FC fabric provisioning to work, 3PAR system was added into OneView with its physical
host ports mapped to the configured OneView FC networks.
Page 18
3PAR port PWWN addresses can be verified through 3PAR CLI or GUI Management Console.
DIA-3PAR-01 cli% showport | grep 0:2:3 1:2:3 0:2:4 1:2:4

N:S:P Mode State ----Node_WWN---- -Port_WWN/HW_Addr- Type Protocol Label Partner FailoverState
0:2:3 target ready 2FF70002AC00ADEC 20230002AC00ADEC host FC - 1:2:3 none
Brocade Network Advisor was added into SAN Manager section.
Page 19
OneView Uplink Configuration Summary

The following capture shows OneView LAN and SAN uplink configurations for VC FF-20/40 F8 module.
VC FF-20/80 F8 in I/O Bay 1:

 X5, X6: 10G Ethernet ports using “VC-1-Uplink” Ethernet Uplink Set to connect with Nexus 5672-1
 X7, X8: 8G FC ports using “VSAN-01” FC Uplink Set to connect with Brocade 6510-1
 X9, X10: 2x20G internal horizontal stacking links between VC modules.
VC FF-20/80 F8 in I/O Bay 1:
 X5, X6: 10G Ethernet ports using “VC-2-Uplink” Ethernet Uplink Set to connect with Nexus 5672-2
 X7, X8: 8G FC ports using “VSAN-02” FC Uplink Set to connect with Brocade 6510-2
 X9, X10: 2x20G internal horizontal stacking links between VC modules.
Page 20
OneView Server Profile and Volume Configuration

Gen9 server profile connection details are listed below. The servers were equipped with Broadcom 20G CNA network adapter
carrying FCOE traffic along with Ethernet trunk traffic.
The Ethernet trunk to the hosts carried mgmt., vmotion, IP storage, NSX VTEP vlans. The network set Maximum Bandwidth
parameter was increased to 20G from 10G for Broadcom 20G CNA adapter.
Page 21
Two virtual volumes on 3PAR were created inside OneView console for server profile assignments.
Page 22
In OneView server profile, SAN storage was provisioned for vSphere shared storage using previous created virtual volumes.
After server profiles were applied, Brocade Network Advisor will provision 6510 switches for auto-zoning. 3PAR will also have
vSphere hosts configured automatically from OneView connection.
Page 23
In 3PAR StoreServ management console, hosts to volumes mapping can be viewed in Mapped view.
Page 24
vSphere 6.0 Network and Storage Validation

vSphere 6.0 hosts should see two 20G NIC presented along with two vmhba adapters.
The two LUNs presented through OneView server profile are ready for VMFS provisioning.
Network traffic were generated from centos virtual machines to prove that Broadcom 630 adapter on ProLiant Gen9 servers
could carry up to 17Gbps of throughput per port simultaneously.
Page 25
With virtual machines generating storage traffic about 5Gbps of throughput per SAN fabric, networking traffic can keep at around
12Gbps of throughput. It proves that FlexFabric Broadcom 630 adapter can transfer total of 17Gbps of aggregate LAN and SAN
per port.
Page 26
Conclusion
This paper is a technical deployment document that should allow the end reader to successfully deploy and deliver a
comprehensive infrastructure of network, compute, and storage with a mix of vendor technologies.
It also validates the successful deployment of vSphere 6.0 with HP Proliant Gen9 servers.
Page 27

Emr Na c04702054 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Emr Na c04702054 1

Uploaded by

Copyright:

Available Formats

Technical white paper

vSphere 6.0 with HP ProLiant Gen9

Hongjun Ma, HP DCA

Infrastructure Products Version

DIA-5672-01# show lldp neighbors

DIA-5672-02# show lldp neigh

DIA-5672-01# show run int po101 membership

DIA-5672-02# show run int po101 membership

channel-group 101 mode active

DIA-5672-01# show run int po102 membership

DIA-5672-02# show run int po102 membership

vPC Peer-link status

DIA-5672-02# show vpc

vPC Peer-link status

DIA-5672-01# show spanning-tree int po101 detail

Storage Fabric Configuration

DIA-3PAR-01 cli% showport | grep 0:2:3 1:2:3 0:2:4 1:2:4

Brocade Network Advisor was added into SAN Manager section.

OneView Uplink Configuration Summary

VC FF-20/80 F8 in I/O Bay 1:

OneView Server Profile and Volume Configuration

vSphere 6.0 Network and Storage Validation

You might also like