Sfs Deployment With Vxrail 7.0 PDF

Dell EMC Networking SmartFabric Services
Deployment with VxRail 7.0

August 2020
Deployment Guide
Abstract
In this guide, SmartFabric Services (SFS) is used to deploy a new leaf-
spine fabric for a new VxRail cluster. SFS automatically reconfigures
the fabric with user-specified VLANs during VxRail cluster deployment.
The SFS-enabled leaf-spine topology is connected to the data center's
existing network using Layer 2 or Layer 3 uplinks.
Dell Networking Infrastructure Solutions

The information in this publication is provided as is. Dell Inc. makes no representations or warranties of any kind with respect
to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular
purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2020 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other
trademarks are trademarks of Dell Inc. or its subsidiaries. Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks
of Intel Corporation in the U.S. and/or other countries. Other trademarks may be trademarks of their respective owners.
Published in the USA 04/20 Deployment Guide H18479.
Dell Inc. believes the information in this document is accurate as of its publication date. The information is subject to change
without notice.
2 Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0

Deployment Guide
Contents
Chapter 1 Introduction 6
Purpose of this guide ................................................................................................7
Dell Technologies .....................................................................................................7
VxRail ........................................................................................................................7
SmartFabric Services ...............................................................................................7
SmartFabric Services with VxRail ............................................................................9
OpenManage Network Integration............................................................................9
Typographical conventions .....................................................................................10
Chapter 2 Hardware Overview 11

Supported switches.................................................................................................12
Hardware used in this guide ...................................................................................12
Chapter 3 Topology 14
Overview .................................................................................................................15
Production topology with SmartFabric Services.....................................................15
Production topology connection details ..................................................................16
OOB management topology ...................................................................................18
OOB management connection details....................................................................19
Chapter 4 Deployment Planning 20

Minimum requirements ...........................................................................................21
Unsupported environments.....................................................................................21
Unsupported features .............................................................................................21
Production topology deployment options ...............................................................21
Uplink options..........................................................................................................22
External switches ....................................................................................................23
VLANs and IP addresses........................................................................................23
VxRail network configuration table .........................................................................25
DNS server records ................................................................................................27
Chapter 5 Configure the First Leaf Switch Pair 28

Cabling ....................................................................................................................29
Configure leaf switch OOB management interfaces ..............................................29
Enable SmartFabric ................................................................................................30
Connect to the SmartFabric GUI ............................................................................31
Update fabric and switch names ............................................................................32
Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0 3

Deployment Guide
Configure L2 uplinks to the external network .........................................................33
Configure L3 routed uplinks to the external network..............................................43
Configure a jump host port .....................................................................................64
Chapter 6 Deploy VxRail 70

Prerequisites ...........................................................................................................71
Initial VxRail cluster deployment steps ...................................................................71
Additional configuration steps for L3 uplinks ..........................................................77
Validate and build VxRail cluster ............................................................................80
Chapter 7 Expand to Multirack 85

Expand SmartFabric and VxRail cluster to multirack .............................................86
Verify preferred master setting before fabric expansion ........................................87
Configure management settings for new switches ................................................89
Add switches to SmartFabric ..................................................................................90
Connect to the SmartFabric GUI ............................................................................91
Configure additional rack and switch names ..........................................................92
Configure leaf switch addresses for L3 uplinks ......................................................92
Add a VxRail node to the cluster ............................................................................94
Chapter 8 Deploy and Configure OMNI 102

Deploy OMNI VM ..................................................................................................103
Configure OMNI ....................................................................................................107
OMNI plugin ..........................................................................................................114
Connect to OMNI Appliance Management user interface ...................................114
Connect to OMNI in vCenter ................................................................................116
Chapter 9 Use OMNI for Post-deployment Configuration 118

Connect a non-VxRail server to a leaf pair ..........................................................119
Add a new port group to the SmartFabric ............................................................125
Move L3 uplinks to spine switches .......................................................................127
Appendix A Validated Components 139

General .................................................................................................................140
Dell EMC PowerSwitch systems ..........................................................................140
VxRail P570 nodes ...............................................................................................140
VxRail appliance software ....................................................................................141
OMNI software ......................................................................................................141
Appendix B CLI Commands 142

Switch CLI validation commands..........................................................................143
Return to Full Switch mode...................................................................................150

Deployment Guide
Appendix C Cisco Nexus External Switch Configuration Example 152
Configure external Nexus switches for L3 routed connections ............................153
Configure external Nexus switches for L2 connections .......................................164
Validated Nexus switches .....................................................................................171
Appendix D Support and Feedback 172

Technical resources ..............................................................................................173
Fabric Design Center ............................................................................................173
Feedback and technical support...........................................................................173

Deployment Guide
Chapter 1 Introduction
This chapter presents the following topics:
Purpose of this guide .............................................................................................7
Dell Technologies ...................................................................................................7
VxRail .......................................................................................................................7
SmartFabric Services .............................................................................................7
SmartFabric Services with VxRail ........................................................................9
OpenManage Network Integration ........................................................................9
Typographical conventions .................................................................................10

Deployment Guide
Purpose of this guide
This guide demonstrates the deployment of a leaf-spine fabric using SmartFabric Services
and shows how SmartFabric Services simplifies the deployment of a new VxRail cluster.
This guide also covers connecting the leaf-spine topology to the existing data center
network, expanding the Smart Fabric and VxRail cluster from a single rack to multiple
racks, and how to use Open Manage Network Integration (OMNI) for Day-2 operations.
Dell Technologies
Our vision at Dell Technologies is to be the essential technology company for the data
era. Dell ensures modernization for today’s applications and for the emerging cloud-native
world.
Dell is committed to disrupting the fundamental economics of the market with an open
strategy that gives you the freedom of choice for networking operating systems and top-
tier merchant silicon. Our strategy enables business transformations that maximize the
benefits of collaborative software and standards-based hardware, including lowered costs,
flexibility, freedom, and security. Dell provides further customer enablement through
validated deployment guides that demonstrate these benefits while maintaining a high
standard of quality, consistency, and support.
VxRail
VxRail is at the forefront of a fundamental shift in IT infrastructure consumption – away
from application-specific, “build-your-own” infrastructure and toward virtualized, general-
purpose, engineered systems. Dell Technologies and VMware have embraced this shift
with the VxRail hyperconverged appliance. VxRail has a simple, scale-out architecture
that uses VMware vSphere and VMware vSAN to provide server virtualization and
software-defined storage.
SmartFabric Services
Dell EMC SmartFabric OS10 includes SmartFabric Services (SFS). With SFS, customers
can quickly and easily deploy and automate data center networking fabrics.
SFS SFS has two personalities:

Personalities
• VxRail Layer 2 (L2) Single Rack personality – the original (legacy) SFS
personality, and automates the configuration of a single pair of ToR (or leaf)
switches for VxRail clusters
• Layer 3 (L3) Fabric personality – the new SFS personality available as of

OS10.5.0.5, automates configuration of a leaf-spine fabric

Deployment Guide
VxRail L2 Single Rack personality
Note: For new single rack and multirack SFS deployments, Dell requires using the L3 Fabric
personality instead of the VxRail L2 Single Rack personality.
The VxRail L2 Single Rack personality is the original SFS personality. It is enabled by
running a Python script in the OS10 Linux shell.
Currently, this personality is limited to a single rack and cannot be expanded to a
multirack deployment. If switches with this personality enabled are upgraded, they will
continue to operate with the VxRail L2 Single Rack personality.
Note: The VxRail L2 Single Rack personality is not covered in this deployment guide. It is covered
in the VMware Integration for VxRail Fabric Automation SmartFabric User Guide, Release 1.1.
See the Dell EMC VxRail Support Matrix (account required), and the SmartFabric OS10 Solutions
(HCI, Storage, MX) Support Matrix for supported software versions.
L3 Fabric personality
Note: Dell requires using the L3 Fabric personality for new SFS deployments. All examples in this
guide use this personality. Unless otherwise specified, statements in this guide regarding
SmartFabric behavior and features are applicable to the L3 Fabric personality only.
Note: The L3 personality provides the option of deploying a VxRail cluster in a single rack but also
provides the option to extend to multi-rack in the future.
The new L3 Fabric Personality was introduced in OS10.5.0.5. It allows users to deploy
SmartFabric Services in a single rack and expand to multirack as business needs evolve.
The SFS L3 Fabric personality automatically builds an L3 leaf-spine fabric. This enables
faster time to production for hyperconverged and private cloud environments while being
fully interoperable with existing data center infrastructure.
Figure 1. From a network of boxes to a networked fabric
The SFS L3 Fabric build process is as follows:

Deployment Guide
1. Rack and cable leaf and spine switches.
2. Enable SFS from the OS10 CLI with the smartfabric l3fabric enable
role [options] command.
a. Specify the role (leaf or spine)
b. Specify Virtual Link Trunking interconnect (VLTi) ports on leafs
3. Switches boot in SmartFabric mode.
4. Switches discover each other using LLDP.
5. Switches elect one of the leaf nodes as the SmartFabric master.
6. Leaf and spine connections are established using private IP addresses and
external Border Gateway Protocol (eBGP).
7. Leaf Nodes are configured as hardware VTEPs for the infrastructure network
overlay using BGP EVPN.
Spine 1 Spine 2
Layer 3 Network Fabric

eBGP/ECMP/EVPN/VXLAN
Leaf 1A Leaf 1B
Leaf1B Leaf 2A Leaf 2B Leaf nA Leaf nB
VLTi VLTi VLTi

Rack 1 Rack 2 Rack n
Figure 2. SFS Layer 3 leaf-spine fabric
SmartFabric Services with VxRail

With SmartFabric Services, switches are automatically configured during VxRail
deployment. When additional VxRail Nodes are connected to the SmartFabric, the fabric
identifies them as VxRail Nodes and automatically onboards them to the required
networks.
OpenManage Network Integration

OpenManage Network Integration (OMNI) enables configuration and management of Dell
EMC PowerSwitch systems running Dell EMC SmartFabric OS10 within VMware vCenter.
With OMNI, networks created in vCenter are automatically configured in the fabric.
The following tasks are done in the OMNI plug-in in vCenter:
• View the leaf-spine topology

• View switch status
• Configure server-facing interfaces and port channels
• Configure uplinks to external networks

Deployment Guide
• Create networks
• Configure routing
• Upgrade SmartFabric OS10
Typographical conventions
The CLI and GUI examples in this document use the following conventions:
Monospace Text CLI examples
Underlined Monospace Text CLI examples that wrap the page
Italic Monospace Text Variables in CLI examples
Bold Monospace Text Commands entered at the CLI prompt, or to

highlight information in CLI output
Bold text GUI elements and information entered in the GUI

Deployment Guide
Chapter 2 Hardware Overview
Supported switches ..............................................................................................12
Hardware used in this guide ................................................................................12

Deployment Guide
Supported switches
Only the Dell EMC PowerSwitch systems listed in Table 1 are supported with SFS in leaf
or spine roles. SFS does not run on third-party switches or other Dell EMC PowerSwitch
models.
To use the SFS features detailed in this guide, switches must be running SmartFabric
OS10.5.0.7P3 or a later version specified in the SmartFabric OS10 Solutions (HCI,
Storage, MX) Support Matrix.
Table 1. Supported switches
VxRail node
Dell EMC PowerSwitch model Typical role
connectivity options
S4112F/T-ON, S4128F/T-ON, S4148F/T-ON Leaf 10 GbE
S5212F-ON, S5224F-ON, S5248F-ON, S5296F-ON Leaf 10/25 GbE
S5232F-ON, Z9264F-ON Spine See the Note below
Note: The roles shown are recommended, with the exception that Z9264F-ON is supported as a
spine only. S5232F-ON may be used as a leaf with ports connected to VxRail nodes broken out to
10 GbE or 25 GbE. VxRail nodes do not currently support 100 GbE NICs for VxRail system traffic.
Any combination of the leaf and spine switches listed in Table 1 may be used with the
exception that leaf switches must be deployed in pairs. Each leaf switch in the pair must
be the same model due to VLT requirements.
SFS supports up to 20 switches in the fabric. This scales to nine racks if two spines and
18 leafs are used.
Hardware used in this guide

This section briefly describes the hardware used to validate the deployment examples in
this guide. Appendix A contains a detailed listing of hardware and software versions used.
All supported leaf and spine switches are listed in Table 1.
Dell EMC The Dell EMC PowerSwitch S5248F-ON is a 1-Rack Unit (1U), multilayer switch with 48x
PowerSwitch 25 GbE, 4x 100 GbE, and 2x 200 GbE ports. This guide uses two S5248F-ON switches in
S5248F-ON each rack as leaf switches.
Figure 3. Dell EMC PowerSwitch S5248F-ON

Deployment Guide
Dell EMC The Dell EMC PowerSwitch Z9264F-ON is a 2U aggregation/spine switch with 64x 100
PowerSwitch GbE ports. This guide uses two Z9264F-ON switches as spine switches.
Z9264F-ON
Figure 4. Dell EMC PowerSwitch Z9264F-ON
Dell EMC The Dell EMC PowerSwitch S3048-ON is a 1U switch with 48x 1 GbE BASE-T ports and
PowerSwitch 4x 10 GbE SFP+ ports. This guide uses one S3048-ON switch in each rack for out-of-
S3048-ON band (OOB) management traffic. This includes connections to VxRail node iDRAC ports
and dedicated switch management ports.
Figure 5. Dell EMC PowerSwitch S3048-ON
Dell EMC VxRail Dell EMC VxRail P, V, S, E, and G Series nodes are built on 14th generation (14G)
14G nodes PowerEdge servers. VxRail P, V, and S Series nodes have a 2U form factor, as shown in
Figure 6.
Figure 6. Dell EMC VxRail 2-RU node (P, V, and S Series)
Note: The deployment example in this guide builds a VxRail cluster with four VxRail nodes. VxRail
supports cluster sizes up to 64 nodes. With SFS, VxRail clusters must have a minimum of three
nodes. Two-node VxRail clusters are not currently supported.
VxRail node VxRail nodes support various combinations of network adapters. See the Dell EMC VxRail
network Network Planning Guide for network connectivity options by node type and the Dell EMC
adapters VxRail Support Matrix (account required) for supported network adapters.
Note: For VxRail node connections to the leaf switches listed in Table 1, use supported 10 GbE or
25 GbE network adapters only. 1 GbE network adapters are not supported for VxRail node to
SFS-enabled leaf switch connections.
Each VxRail node also includes an integrated Dell Remote Access Card (iDRAC) for out-
of-band management.

Deployment Guide
Chapter 3 Topology
Overview ................................................................................................................15
Production topology with SmartFabric Services ..............................................15
Production topology connection details ...........................................................16
OOB management topology ................................................................................18
OOB management connection details ...............................................................19

Deployment Guide
Overview
The topology is divided into two major parts:
• Production
• Out-of-band (OOB) management
The production topology contains redundant components and is used for all mission-
critical and end-user network traffic. The OOB management network is an isolated
network for remote management of hardware.
Production topology with SmartFabric Services

The production topology uses a leaf-spine fabric for performance and scalability.
SmartFabric Services (SFS) automates the deployment of this fabric.
SmartFabric
External Spine1 Spine2

network
Leaf1A Leaf1B Leaf2A Leaf2B

Existing DNS &
NTP services VLTi VLTi
VxRail node 1 VxRail node 4
VxRail node 2
VxRail node 3
Rack 1 Rack 2
Layer 3 connections
Layer 2 connections
VLTi connections
Layer 2 or Layer 3 uplink connections
Figure 7. SmartFabric topology with connections to VxRail nodes and external network
Note: The deployment examples in this guide use two network adapter ports per VxRail node, as
shown in Figure 7. See the Dell EMC VxRail Network Planning Guide for VxRail node connectivity
options.

Deployment Guide
With SFS, two leaf switches are used in each rack for redundancy and performance. A
Virtual-Link Trunking interconnect (VLTi) connects each pair of leaf switches. Every leaf
switch has an L3 uplink to every spine switch. Equal-Cost MultiPath routing (ECMP) is
leveraged to use all available bandwidth on the leaf-spine connections.
SFS uses BGP-EVPN to stretch L2 networks across the L3 leaf-spine fabric. This allows
for the scalability of L3 networks with the VM mobility benefits of an L2 network. For
example, a VM can be migrated from one rack to another without the need to change its
IP address and gateway information.
The example in this guide builds the SmartFabric shown in Figure 7 in two stages:
1. The first stage is a single rack deployment. Leaf switches 1A and 1B are deployed
in Rack 1 without spine switches, and a two-leaf fabric is created using SFS. The
fabric is connected to the external network using either L2 or L3 uplinks. The
external network is typically a preexisting network in the data center. Three VxRail
nodes are connected to the two leaf switches, and a three-node VxRail cluster is
deployed.
2. In the second stage, two spine switches are added and connected to leaf
switches 1A and 1B. Leaf switches 2A and 2B are added in Rack 2 and are also
connected to the spine switches. The fabric is expanded to include the two spines
and two additional leafs using SFS. A fourth VxRail node is added in Rack 2 and
joined to the existing VxRail cluster.
Note: Single and multirack deployment options are discussed in Chapter 4 - Deployment
Planning.
Production topology connection details

Production network connections for this deployment example are shown in Figure 8. Each
S5200 series leaf switch has one connection to each spine switch, and each VxRail node
has one connection to each leaf. Connections from leafs to spines are 100 GbE, and
connections from leafs to VxRail nodes are 25 GbE.
Note: If S4100 series leaf switches are used (not shown), connections to VxRail nodes are
10 GbE. S4100 series switches have 100 GbE ports available for connections to spines.

Deployment Guide
Stack ID
Z9264F-ON Reset
Spine 1
Stack ID
Z9264F-ON Reset
Spine 2
S5248F-ON Stack ID
Leaf 1A
VLTi
S5248F-ON
Stack ID
Leaf 1B
VxRail P570
node
GRN=10G
GRN=10G
ACT/ LNK B
ACT/ LNK A
25 GbE Layer 2 link

100 GbE Layer 3 link (Spine 1)
100 GbE Layer 3 link (Spine 2)
200 GbE VLTi link
Figure 8. Production network connection details
Note: In this example, the two QSFP28-DD double density ports (2x 100 GbE interfaces per
physical port), available on S5248F-ON switches, are used to create a 400 GbE VLTi. This
requires QSFP28-DD DAC cables or optics. On switches without QSFP28-DD ports, QSFP28
(100 GbE) or QSFP+ (40 GbE) ports are typically used for VLTi connections. The VLTi
synchronizes L2 and L3 control-plane information across the two nodes. The VLTi is used for data
traffic only when there is a link failure that requires the VLTi to reach the destination. Dell
Technologies recommends using at least two physical ports on each switch for the VLTi for
redundancy and to provide additional bandwidth if there is a failure.

Deployment Guide
OOB management topology
The out-of-band (OOB) management network is an isolated network for remote
management of hardware. This includes VxRail nodes, servers, switches, storage arrays,
and rack power distribution units (PDUs) using their dedicated management ports.
For OOB management network connections, one S3048-ON switch is installed in each
rack, as shown in Figure 9.
OOB
Management
Network Core
Rack 1 Rack 2 Rack 3 Rack n

S3048-ON S3048-ON S3048-ON S3048-ON
Lea f Switch 1A Lea f Switch 2A Lea f Switch 3A Lea f Switch nA
Lea f Switch 1B Lea f Switch 2B Lea f Switch 3B Lea f Switch nB
VxRail Nod e Spin e S wi tch 1 VxRail Nod e VxRail Nod e
VxRail Nod e Spin e S wi tch 2 VxRail Nod e VxRail Nod e
PowerE dge PowerE dge

VxRail Nod e VxRail Nod e
Ser ver Ser ver
PowerE dge PowerE dge
Storage A rray
Ser ver Ser ver
Figure 9. OOB management network connections
The OOB management network also enables connections to the PowerSwitch SFS GUI. It
also enables switch console access using SSH, and VxRail node console access using
the iDRAC. This network is also used to carry heartbeat messages between switches
configured as VLT peers, and for OpenManage Network Integration (OMNI) to
communicate with the SFS master switch.
Note: OOB management switches are not part of the SmartFabric.
Note: This guide covers the equipment shown in Racks 1 and 2. Other devices and racks shown
in Figure 9 are for demonstration purposes only.
Four 10 GbE SFP+ ports are available on each S3048-ON for use as uplinks to the OOB
management network core.
1 GbE BASE-T ports on each S3048-ON are connected downstream to hardware

management ports on each device in the rack. This includes the VxRail node iDRAC ports
and switch management ports. Management ports on other devices, such as PowerEdge
server iDRAC ports, storage array management ports, and rack PDU management ports,
are also connected to this network.

Deployment Guide
OOB management switch configuration is not detailed in this guide. The S3048-ON can
function as an OOB management switch with its OS10 factory default configuration. By
default, all ports are in switchport mode, in VLAN 1, administratively up, and rapid per-
VLAN spanning tree plus (RPVST+) is enabled.
Note: At a minimum, Dell Technologies recommends changing the admin password to a complex
password during the first login.
Note: For reference, devices on the OOB Management network in this guide use the
100.67.0.0/16 IP address block. These addresses are examples only. Use IP addresses suitable
for your environment.
OOB management connection details

Figure 10 shows how switch management ports and VxRail node iDRAC ports connect to
the OOB management switch in each rack.
OOB Management
network core
S3048-ON 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
OOB management
switch
S5248F-ON
Leaf 1A
S5248F-ON
Leaf 1B
Stack ID
Z9264F-ON Reset
Spine 1
Stack ID
Z9264F-ON Reset
Spine 2
VxRail P570
node
GRN=10G
GRN=10G
ACT/ LNK B
ACT/ LNK A
iDRAC
1GbE link
10GbE link
Figure 10. OOB management network connection details

Deployment Guide
Chapter 4 Deployment Planning
Minimum requirements ........................................................................................21
Unsupported environments .................................................................................21
Unsupported features ..........................................................................................21
Production topology deployment options .........................................................21
Uplink options .......................................................................................................22
External switches..................................................................................................23
VLANs and IP addresses .....................................................................................23
VxRail network configuration table ....................................................................25
DNS server records ..............................................................................................27

Deployment Guide
Minimum requirements
Minimum requirements for VxRail 7.0 deployments with SFS include:
• Three VxRail nodes running VxRail appliance software version 7.0 or a later as
specified in the SmartFabric OS10 Solutions (HCI, Storage, MX) Support Matrix.
• VxRail nodes must meet the hardware and software requirements listed in the
Dell EMC VxRail Support Matrix.
• Onboard NICs in VxRail nodes must be 10 GbE or 25 GbE.
• Two Dell EMC PowerSwitch units as listed in Table 1 deployed as leaf switches.
Each leaf switch in the pair must be the same model due to VLT requirements.
• Dell EMC PowerSwitch units must be running SmartFabric OS10.5.0.7P3 or a
later version as specified in the SmartFabric OS10 Solutions (HCI, Storage, MX)
Support Matrix.
• One 1 GbE BASE-T, also referred to as 1000BASE-T, switch for OOB
management connections. Dell Technologies recommends using one
PowerSwitch S3048-ON per rack.
• One DNS server which can be an existing DNS server that is reachable on the
network with host records added for this deployment. The example DNS host
records used in this guide are shown in Table 5.
Unsupported environments
SFS does not currently support the following environments:
• vSAN stretched clusters

• VMware Cloud Foundation (VCF)
• NSX
• VxRail L3 Everywhere
Unsupported features
SFS does not currently support the following features:
• Multiple VRF tenants

• Route policies or Access Control Lists (ACLs)
• OSPF or routing protocols other than eBGP
• Multicast routing protocols
• Networking features not covered in the SmartFabric Services for OpenManage
Network Integration User Guide, Release 1.3. This guide is available on the Dell
EMC OpenManage Network Integration for VMware vCenter website.
Production topology deployment options

Deployment Guide
Deployment options for deploying SFS for VxRail include:
• Single rack deployment – A two-leaf SmartFabric is deployed in a single rack.

The VxRail nodes that are in the same rack, are connected to two leaf switches,
and a VxRail cluster is built. This is covered in Chapter 5 and Chapter 6 of this
guide.
• Expand single rack deployment to multirack – The two-leaf SmartFabric is

expanded to multirack by adding spine switches to connect the racks and two leaf
switches per rack. VxRail nodes in the additional racks are connected to the
additional SmartFabric leaf switches and are joined to the existing VxRail cluster.
This is covered in Chapter 7 of this guide.
• Multirack deployment – A multirack SmartFabric with spines and two leaf

switches per rack is deployed. VxRail nodes are installed in multiple racks and
connected to the SmartFabric leaf switches in each rack. A VxRail cluster is built
using VxRail nodes in multiple racks.
Uplink options
SFS uplink options to external network switches include:
• L2 uplinks from a leaf pair

• L3 uplinks from a leaf pair
• L3 uplinks from spines
If there is no preference for the uplink type, L2 uplink configuration has fewer steps than
L3 uplink configuration. Uplinks from spines must be L3.
Note: Dell Technologies recommends using uplinks from a leaf pair as a best practice. The leafs
are referred to as border leafs. VxRail nodes and other servers in the rack may be connected to
border leafs in the same manner as other leafs in the SmartFabric.
L2 uplink If an L2 uplink is used, determine the VLAN ID to use for VxRail external management,
planning and if ports in the uplink will be tagged or untagged. Typically, this will be the same VLAN
used for DNS and NTP services on the existing network, as shown in the example in this
guide. Optionally, traffic may be routed from the external switch to the DNS/NTP servers.
The L2 uplink may be an LACP or static LAG. If L2 uplinks connect to a pair of Dell EMC
PowerSwitch systems, Dell Technologies recommends using LACP with VLT per the
example in this guide.
L2 uplink configuration is covered in detail in the Configure L2 uplinks to the external

network section of this guide.
Note: With L2 uplinks, all routing into and out of the SmartFabric is done on external switches.
L3 uplink SFS supports using L3 routed or L3 VLAN uplinks.

planning
With L3 routed uplinks, each physical link is a point-to-point IP network. With L3 VLAN, all
uplinks are in a LAG, and an IP address is assigned to the VLAN containing the LAG. This

Deployment Guide
guide provides examples using L3 routed uplinks. L3 VLAN examples are beyond the
scope of this guide.
Point-to-point IP networks and addresses must be planned for each physical link in the L3
uplink.
Each leaf switch in the SmartFabric needs an IP address on the External Management
VLAN. An anycast gateway address on the same VLAN is also specified. This is the
virtual router/anycast gateway address shared by all leafs in the SmartFabric.
SmartFabric supports routing using eBGP or static routes. eBGP and static routing
examples are both provided in this guide.
Note: SFS does not support other routing protocols.
If eBGP is used, ASNs and router IDs must be determined for the external switches.
These are automatically configured on all switches in the SmartFabric.
Note: SFS uses ASNs 65011 for leafs, and 65012 for spines. If these ASNs conflict with your
environment, they may be changed in the SFS GUI under 5. Edit Default Fabric Settings.
L3 uplink configuration is covered in detail in the Configure L3 routed uplinks to the

external network section of this guide.
External switches
External switches must have available ports for connections from the existing network to
the SFS border leafs (or spines if applicable). For redundancy, Dell Technologies
recommends two external switches with at least two links per switch to the SmartFabric.
Use enough connections to provide sufficient bandwidth for the traffic anticipated across
these links. If using Dell EMC PowerSwitch systems as external switches, Dell
Technologies recommends configuring them as VLT peers, as shown in the examples in
this guide.
Note: This guide provides external switch configuration examples for Dell EMC PowerSwitch
systems. Cisco Nexus switch configuration examples are provided in Appendix C.
VLANs and IP addresses

VLANs and IP addresses used for VxRail node traffic must be planned before VxRail
deployment can begin. VxRail node traffic is divided into six or more VLANs, as shown in
Table 2.
Table 2. VLANs used by VxRail
VLAN Purpose
VxRail Cluster Build Used to build the VxRail cluster. SFS automatically creates this VLAN
and names it SFS Client Management.
VxRail Internal Used for VxRail node discovery. SFS automatically creates this VLAN
Management and names it SFS Client Control.

Deployment Guide
VLAN Purpose
VxRail External User-specified VLAN for VxRail Manager, ESXi, vCenter Server, NTP,
Management DNS, and vRealize Log Insight traffic
vMotion User-specified VLAN for Virtual machine (VM) migration traffic
vSAN User-specified VLAN for distributed storage traffic
VM networks User-specified VLAN or VLANs as required for VM data traffic
Note: All VLANs in Table 2 share the physical connections shown in Figure 8 in this deployment.
VLAN IDs and network addresses planned for this deployment example are shown in
Table 3.
Table 3. VLAN IDs and network addresses
VLAN ID Description Network
4091* SFS Client Management/VxRail cluster build 192.168.10.0/24 (default)
3939* SFS Client Control/VxRail Internal Management IPv6 multicast
1811 External Management 172.18.11.0/24
1812 vMotion 172.18.12.0/24
1813 vSAN 172.18.13.0/24
1814 VM Network A 172.18.14.0/24
1815 VM Network B 172.18.15.0/24

*Automatically created VLANs
Note: SFS automatically creates VLANs 4091 and 3939. VLANs 1811 through 1815 and their
network IP addresses are user-defined and are examples only. In SmartFabric mode, VLANs 2
through 3999, excluding 3939, are available for use. VLANs 4091 and 3939 may be changed from
their defaults in the SFS GUI under 5. Edit Default Fabric Settings. VLAN 3939 is also a VxRail
default VLAN. If VLAN 3939 is changed in the SFS GUI, you must also change it to match in
VxRail per the VxRail documentation.
Note: VLANs 4000 through 4094 are reserved for SFS. For more information about the reserved
VLANs, see the SmartFabric Services for OpenManage Network Integration User Guide, Release
1.3. The guide is available on the Dell EMC OpenManage Network Integration for VMware
vCenter website.
Note: SFS uses the 172.16.0.0/16, and 172.30.0.0/16 IP address blocks internally for the leaf-
spine network configuration. If these networks conflict with your environment, these default IP
addresses blocks may be changed in the SFS GUI under 5. Edit Default Fabric Settings.
In SmartFabric mode, each VLAN in Table 3 is automatically placed in a VXLAN virtual

network with a Virtual Network Identifier (VNI) that matches the VLAN ID. VLAN 4091 is in
virtual network 4091, VLAN 1811 is in virtual network 1811, and so on.
The show virtual network command is used to view virtual networks, VLANs, and
port-VLAN assignments. This command is covered in more detail later in this guide.

Deployment Guide
VxRail network configuration table
The VxRail network configuration table is used as a reference during VxRail deployment.
The values used for the deployment examples in this guide are shown in the right column.
The IP addresses and VLAN IDs are from Table 3.
Table 4. VxRail network configuration table
Category Description Values used in this

guide
System Globals NTP servers ntp.dell.lab*
NIC Configuration 2x 10 GbE or 2x 25 GbE, 4x 10 GbE 2x 10 GbE or 2x 25 GbE
Management Top Level Domain Domain dell.lab
ESXi Hosts ESXi hostname prefix vxrail
Separator None
Iterator Num 0x
Offset 1
Suffix
ESXi host starting IP address 172.18.11.101
ESXi host ending IP address 172.18.11.164
vCenter Server (Leave Join existing vCenter server unchecked

blank if Customer-
supplied) vCenter Server hostname vcenter01
vCenter Server IP address 172.18.11.62
Join existing vCenter vCenter Server hostname (FQDN)

Server (Leave blank if
VxRail VC) vCenter Server username
vCenter Server password
vCenter Server management username
vCenter Server management password
Data center name
Cluster name
Join existing vCenter FQDN or IP address

Single Sign-On domain
(Leave blank if VxRail HTTPS port
VC)
Single Sign-On administrator username
Single Sign-On administrator password
VxRail Manager VxRail hostname vxmgr01
VxRail IP address 172.18.11.72
Networking Management Subnet mask 255.255.255.0
Management Gateway 172.18.11.254

Deployment Guide
Category Description Values used in this
guide
Management Network VLAN ID 1811
DNS Servers 172.18.11.50

(In L2 uplink example) or
172.19.11.50
(In L3 uplink example)**
vCenter Server Network Static Binding
vMotion Starting address for IP pool 172.18.12.101
Ending address for IP pool 172.18.12.164
Subnet mask 255.255.255.0
VLAN ID 1812
vSAN Starting address for IP pool 172.18.13.101
Ending address for IP pool 172.18.13.164
Subnet mask 255.255.255.0
VLAN ID 1813
VM … (unlimited number) VM Network name and VLAN ID VM_Network_A, 1814

Networks
VM Network name and VLAN ID VM_Network_B, 1815
Solutions Logging None, vRealize Log Insight, Syslog server vRealize Log Insight
vRealize Log Insight hostname vxinsight
vRealize Log Insight IP address 172.18.11.55
Accounts Virtual appliance Use the same credentials for all virtual checked
accounts accounts
vCenter Server management account administrator

username
Shared password for all non-ESXi password

accounts
VxRail Manager Service Username: "mystic" password

Account
ESXi Accounts Use the same credentials for all hosts checked
Root username root
Root password password
Management username administrator
Management password password

*An NTP server is recommended but not required. If an NTP server is not provided,
VxRail uses the time that is set on VxRail node 1.
**In the L2 uplink example in this guide, the DNS/NTP servers on the existing network are
on the same External Management VLAN, 1811, as the VxRail nodes. IP addresses on
this network use the 172.18.11.0/24 address block. In the L3 uplink example, the

Deployment Guide
DNS/NTP servers are on a different VLAN, 1911, with IP addresses in the 172.19.11.0/24
address block. VLAN 1911 represents a pre-existing management VLAN and is used only
on the external switches in the L3 uplink example.
This deployment example uses four VxRail nodes. However, host IP address pools are
configured for the Management, vMotion, and vSAN VLANs using addresses in the .101
through .164 range, as shown in Table 4. This is optional but removes the need to
configure new IP addresses if new hosts are added to the cluster.
Note: VxRail supports cluster sizes up to 64 nodes.
DNS server records

VxRail nodes must be able to reach a correctly configured DNS server during and after
VxRail deployment. The DNS server must include forward and reverse lookup entries for
ESXi hosts, vCenter, the Platform Services Controller, VxRail Manager, and vRealize Log
Insight.
Add forward and reverse lookup records on the DNS server using the hostnames and IP
addresses used in your deployment. The DNS entries for the deployment examples in this
guide are listed in Table 5.
Table 5. DNS hostnames and IP addresses
Hostname IP address
vxrail01.dell.lab 172.18.11.101
vcenter01.dell.lab 172.18.11.62
vxmgr01.dell.lab 172.18.11.72
vxinsight.dell.lab 172.18.11.55
omni.dell.lab 172.18.11.56
ntp.dell.lab In L2 uplink example - 172.18.11.51

In L3 uplink example - 172.19.11.51
In the L2 uplink example in this guide, the DNS server address is 172.18.11.50. In the L3
uplink example, the DNS server address is 172.19.11.50.

Deployment Guide
Chapter 5 Configure the First Leaf Switch
Pair
Cabling ...................................................................................................................29
Configure leaf switch OOB management interfaces ........................................29
Enable SmartFabric ..............................................................................................30
Connect to the SmartFabric GUI .........................................................................31
Update fabric and switch names.........................................................................32
Configure L2 uplinks to the external network ...................................................33
Configure L3 routed uplinks to the external network ......................................43
Configure a jump host port .................................................................................64

Deployment Guide
Cabling
Cable the switches and VxRail nodes, as shown in Figure 11, and power on all devices.
Leaf1A Leaf1B
VLTi
VxRail node 1
VxRail node 2
VxRail node 3
Rack 1
Figure 11. VxRail node and leaf switch connections
For connection details, see Figure 8. Also, make OOB management connections, as
shown in Figure 10.
Configure leaf switch OOB management interfaces

An IP address is configured on the OOB management interface of each switch. This
interface is used to access the SFS GUI, and it is also used as the VLT backup link. It also
enables console access using SSH as an option to the serial console.
A management route is also configured if routing is used on the OOB management
network.
Note: Configure a unique OOB management IP address on each switch. The IP addresses shown
are examples only. Use IP addresses suitable for your environment. The management route
should not be 0.0.0.0/0, or this may interfere with the data network’s default route. Use a specific
destination prefix, as shown.
This is done on each switch as follows:
OS10# configure terminal

OS10(config)# interface mgmt 1/1/1
OS10(conf-if-ma-1/1/1)# no ip address dhcp
OS10(conf-if-ma-1/1/1)# ip address 100.67.127.28/24
OS10(conf-if-ma-1/1/1)# no shutdown
OS10(conf-if-ma-1/1/1)# exit
OS10(config)# management route 100.67.0.0/16 100.67.127.254
OS10(config)# end
OS10# write memory

Deployment Guide
Note: If % Error: ZTD is in progress(configuration is locked) is preventing entry
into configuration mode, enter the command ztd cancel to proceed.
Other global settings may also be configured here, such as ip name-server and ntp
server if used by the switch. These settings are not required for the deployment example
in this guide. The hostname of the switch may be configured at the CLI or in the SFS GUI.
In this guide, the SFS GUI is used.
Enable SmartFabric
SmartFabric
Leaf1A Leaf1B
VLTi
Figure 12. First pair of leaf switches in SmartFabric mode
Caution: The following commands delete the existing switch configuration. Switch management
settings such as management IP address, management route, hostname, NTP server, and IP
name server are retained.
Ensure the physical VLTi connections are made between leaf pairs before proceeding.
Note: This example uses the two QSFP28 2x100Gb DD ports, interfaces 1/1/49-1/1/52, for the
VLTi connections on each S5248F-ON leaf.
To put the first pair of leaf switches in SmartFabric mode and configure them as VLT
peers, run the following commands on each switch:
OS10(config)# smartfabric l3fabric enable role LEAF vlti ethernet
1/1/49-1/1/52
Reboot to change the personality? [yes/no]:y
The configuration is applied, and the switches reload.

To verify switches are in SmartFabric mode, run the following command on each switch:
OS10# show switch-operating-mode
Switch-Operating-Mode : Smart Fabric Mode
Note: For more information, see SmartFabric Services for OpenManage Network Integration User
Guide, Release 1.3. The guide is available on the Dell EMC OpenManage Network Integration for
VMware vCenter web site. For additional SmartFabric CLI commands, see the SmartFabric
Services chapter of the Dell EMC SmartFabric OS10 User Guide Release 10.5.0.

Deployment Guide
Connect to the SmartFabric GUI
1. From a workstation with access to the OOB management network, use a browser
to connect to the management IP address of either leaf switch by going to
https://switch_mgmt_ip_address.
2. Log in as admin.
Note: After reloading the switches, it takes about 2 minutes after the login prompt displays at the
switch CLI for SFS to come up and for the GUI to be fully functional.
Note: The SFS GUI supports Chrome, Firefox, and Edge browsers. Languages other than English
are not currently supported.
All GUI configuration is done on the SFS master switch. If you connect to an SFS switch
that is not the master, a link to the master is provided. This is outlined in red in Figure 13.
Figure 13. Connected to switch that is not the master
3. If applicable, click the link provided to go to the master switch, and log in as
admin.
Note: The IPv4 address of the SFS master may also be determined by running show
smartfabric cluster from the CLI of any switch in the SmartFabric. The master is always a
leaf switch, never a spine. Only one leaf switch in the SmartFabric will have ROLE set to MASTER.
The remaining leafs will have ROLE set to BACKUP.
When connected to the SFS master switch, the GUI appears, as shown in Figure 14.
Figure 14. Connected to SFS master switch

Deployment Guide
4. Hover over each switch (or node) and the VLTi link to view additional information,
as shown in Figure 15 and Figure 16.
Figure 15. Node details
Figure 16. VLTi link details
Update fabric and switch names

The fabric, rack, and switch names may be changed from their default settings as follows:
1. On the SFS GUI Home page, click 1. Update Default Fabric, Switch Names
and Descriptions to open the Set Fabric and Switch Name window.
2. On the Network Fabric page, update the fabric Name (optional) and Description
(optional) and click NEXT.
Note: The Network Fabric ID is automatically set to 100 and cannot be changed. All directly
connected switches in SmartFabric mode join this fabric.
3. On the Racks page, update the Name (recommended) and Description

(optional) of the rack. In this example, the rack name is set to Rack 1, as shown
in Figure 17.

Deployment Guide
Figure 17. Rack name changed to Rack 1
4. Click NEXT.
5. On the Switches page, update the Name (recommended, if not previously
configured from the CLI) and Description (optional) of the switches. Hostnames
are set to S5248F-Leaf1A and S5248F-Leaf1B in this example.
Figure 18. Switch name configuration page
6. Click FINISH to apply the settings.
Configure L2 uplinks to the external network

Uplinks to the existing network may be configured as L2, L3 routed, or L3 VLAN. This
section covers L2 uplinks.
Note: If L3 uplinks are used, proceed to the Configure L3 routed uplinks to the external network
section. If there is no preference for the uplink type, consider using an L2 uplink configuration as it
has fewer steps than L3 uplink configuration.
The switches are cabled as shown in Figure 19. When L2 uplink configuration is complete,
Leaf1A and Leaf1B will connect with a VLT port channel to a switch pair named External-
A and External-B. In this example, an existing DNS/NTP server also connects to the
external switches using a VLT port channel. All VLT port channels use LACP in this guide.
Note: DNS and NTP servers do not have to connect in this manner as long as they are reachable
on the network.

Deployment Guide
All ports on the four switches shown in Figure 19 are in the External Management VLAN,
1811.
VLTi
External-A External-B
33 21 22 33 21 22
VLT
SmartFabric
VLT
53 54 53 54
Existing Leaf1A Leaf1B
3 2 1 1 2 3
DNS/NTP server
VLTi
L2 Connections
(All in VLAN 1811) VxRail node 1
VLTi connections
VxRail node 2
VxRail node 3
Rack 1
Figure 19. L2 uplinks to the external network
Configure L2 Note: Any ports available on the leaf switches may be used as uplinks, provided they are
uplinks in SFS compatible with the corresponding ports on the external switches. If leaf switch uplink ports will not
use their native speeds, the interfaces must be first broken out to the correct speed before the
uplinks are created. This is done using the Breakout Switch Ports option on the SFS GUI home
page. A breakout example is shown in the Change the port-group speed in the SFS GUI section of
this guide.
L2 uplinks to the external network are configured as follows:

1. On the SFS GUI home page, click 2. Create Uplink for External Network
Connectivity.
2. On the Uplink Details page, select Layer 2. Enter a Name and optionally a
Description.

Deployment Guide
Figure 20. Uplink details
3. Click NEXT.
4. On the Port Configuration page, select the uplink ports used on each leaf switch
and set the LAG Mode to LACP or Static. In this example, 100 GbE ports 1/1/53-
1/1/54 are used on each switch, and the LAG mode is set to LACP.
Note: Be sure to configure the corresponding ports on the external switches with the same LAG
mode. External switch configuration examples using LACP are provided in the Configure external
switches for L2 connections section of this guide.
Figure 21. Uplink port configuration
5. Click NEXT.
VxRail Manager must be able to contact a DNS server to resolve hostnames during
deployment. The External Management VLAN is created to enable this, and the uplinks
are added to it as follows:
6. On the Network Configuration page, click ADD NETWORK.

Deployment Guide
Figure 22. Network configuration page
7. In the dialog box that opens, provide a Name, Description (optional), and a
VLAN ID for the External Management network. In this example, VLAN ID 1811
from Table 3 is used.
Figure 23. Network details
8. Click OK.
9. Next to Tagged Networks, select the External Management VLAN created
above, ExtMgmt-1811. Use the arrow button to move it to the box on the right, as
outlined in red in Figure 24. This makes the uplinks tagged members of the
External Management VLAN.

Deployment Guide
Figure 24. Uplink ports tagged in the External Management Network
10. Leave the box next to UnTagged Network set to None.

11. If networks automatically created through vCenter integration are to be extended
on this uplink, select Yes. Otherwise, select No. Yes is used in this example.
Note: Networks created through vCenter integration include the External Management, VSAN,
vMotion, and VM Networks created during VxRail deployment. It also includes networks added
through OMNI post-deployment.

After uplink configuration, the SFS GUI Home page appears, as shown in Figure 25.
Figure 25. SFS Home page after uplinks configured
The show smartfabric uplinks command may be used at the leaf switch CLI to
view configured interfaces and networks on the uplink.

Deployment Guide
Note: The command output shown in the following command is for Leaf1A. The output for
Leaf1B is the same.
S5248F-Leaf1A# show smartfabric uplinks

----------------------------------------------------------
Name : L2-to-external-network
Description :
ID : 8ca32653-854c-4347-af94-e6afaa136c3a
Media Type : ETHERNET
Native Vlan : 0
Untagged-network :
Networks : network-1811
Configured-Interfaces : 68X00Q2:ethernet1/1/54,
690ZZP2:ethernet1/1/53, 68X00Q2:ethernet1/1/53,
690ZZP2:ethernet1/1/54
Configure This section shows example configurations for both external switches for L2 connections
external to the SmartFabric.
switches for L2
connections Note: The external switches used in this example are Dell EMC PowerSwitch systems. If the
external switches are Cisco Nexus, see Appendix C.
Note: This is only an example. Modify your external switch configuration as needed for your
network.
General settings
Configure the hostname, OOB management IP address, and OOB management route as
shown.
configure terminal configure terminal
hostname External-A hostname External-B
interface mgmt1/1/1 interface mgmt1/1/1

no ip address no ip address
ip address 100.67.127.37/24 ip address 100.67.127.36/24
no shutdown no shutdown
management route 100.67.0.0/16 management route 100.67.0.0/16

100.67.127.254 100.67.127.254
Configure VLANs
Create the External Management VLAN. If traffic will be routed from the external switches
to other external networks, assign a unique IP address on each switch and configure
VRRP to provide gateway redundancy. Set the VRRP priority. The switch with the highest

Deployment Guide
priority value becomes the master VRRP router. Assign the same virtual address to both
switches.
interface vlan1811 interface vlan1811
description External_Mgmt description External_Mgmt
vrrp-group 11 vrrp-group 11
priority 150 priority 100
virtual-address 172.18.11.254 virtual-address 172.18.11.254
Configure interfaces
Configure the interfaces for connections to the SFS leaf switches. Interfaces 1/1/21 and
1/1/22 are configured in VLT port channel 100 in this example. Port-channel 100 is set as
an LACP port channel with the channel-group 100 mode active command.
Use the switchport mode trunk command to enable the port channel to carry traffic
for multiple VLANs. Configure the port channel as tagged on VLAN 1811 (the External
Management VLAN).
Optionally, allow the forwarding of jumbo frames with the MTU 9216 command.
In this example, interface 1/1/33 on each external switch is configured in VLT port channel
1 for connections to the DNS/NTP server. Port-channel 1 is set as an LACP port channel
with the channel-group 1 mode active command.
Configure ports directly connected to nodes, servers, or other endpoints as STP edge
ports. As a best practice, flow control settings remain at their factory defaults as shown.
interface port-channel100 interface port-channel100
description "To Leaf1A/1B" description "To Leaf1A/1B"
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan switchport trunk allowed vlan
1811 1811
mtu 9216 mtu 9216
vlt-port-channel 100 vlt-port-channel 100
interface range ethernet1/1/21- interface range ethernet1/1/21-

1/1/22 1/1/22
description "To Leaf1A/1B" description "To Leaf1A/1B"
no switchport no switchport
channel-group 100 mode active channel-group 100 mode active
mtu 9216 mtu 9216

description "To DNS/NTP" description "To DNS/NTP"

Deployment Guide
switchport access vlan 1811 switchport access vlan 1811
spanning-tree port type edge spanning-tree port type edge
interface ethernet1/1/33 interface ethernet1/1/33

flowcontrol receive on flowcontrol receive on
flowcontrol transmit off flowcontrol transmit off
Configure VLT
This example uses interfaces 1/1/29 and 1/1/31 for the VLTi. Remove each interface from
L2 mode with the no switchport command.
Create the VLT domain. The backup destination is the OOB management IP address of
the VLT peer switch. Configure the interfaces used as the VLTi with the discovery-
interface command.
As a best practice, use the vlt-mac command to manually configure the same VLT MAC
address on both the VLT peer switches. This improves VLT convergence time when a
switch is reloaded.
Caution: Be sure the VLT MAC address is the same on both switches to avoid any unpredictable
behavior.
If you do not configure a VLT MAC address, the MAC address of the primary peer is used
as the VLT MAC address on both switches.
Note: For more information about VLT, see the Dell EMC SmartFabric OS10 User Guide on the
Dell EMC Networking OS10 Info Hub.
When the configuration is complete, exit configuration mode and save the configuration
with the end and write memory commands.

Deployment Guide
interface range interface range
ethernet1/1/29,1/1/31 ethernet1/1/29,1/1/31
description VLTi description VLTi
vlt-domain 255 vlt-domain 255

backup destination 100.67.127.36 backup destination 100.67.127.37
discovery-interface discovery-interface
vlt-mac 00:00:01:02:03:20 vlt-mac 00:00:01:02:03:20
end end
write memory write memory
Validation Once the uplink interfaces have been configured on the external switches and in the SFS
GUI, additional validation is done using the switch CLI.
Show command output on External-A

Note: The command output shown in the following commands is for the External-A switch. The
output for External-B is similar.
Run the show vlan command to verify ports are correctly assigned to the External
Management VLAN. Port channel 100 connects to the SFS leaf switches and is a tagged
member of the same VLAN configured on the SmartFabric uplinks (VLAN 1811). It is
tagged because it is also tagged on the SmartFabric leaf switches. The DNS/NTP server
is connected on port channel 1, which is an access member of VLAN 1811 in this
example.
External-A# show vlan
Codes: * - Default VLAN, M - Management VLAN, R - Remote Port Mirroring VLANs,
@ – Attached to Virtual Network, P - Primary, C - Community, I - Isolated
Q: A - Access (Untagged), T - Tagged
NUM Status Description Q Ports
* 1 Active A Eth1/1/1-1/1/20,1/1/23-/1/28,1/1/30,1/1/34
A Po100,1000
1811 Active External_Mgmt T Po100,1000
A Po1
4094 Active T Po1000
The show port channel summary command confirms port channel 100 connected to
the leaf switches is up and active. Port channel 1000 is the VLTi, and port channel 1 is
connected to the DNS/NTP server.

Deployment Guide
External-A# show port-channel summary
Flags: D - Down I - member up but inactive P - member up and active
U - Up (port-channel) F - Fallback Activated
--------------------------------------------------------------------------------
Group Port-Channel Type Protocol Member Ports
--------------------------------------------------------------------------------
1 port-channel1 (U) Eth DYNAMIC 1/1/33(P)
100 port-channel100 (U) Eth DYNAMIC 1/1/21(P) 1/1/22(P)
1000 port-channel1000 (U) Eth STATIC 1/1/29(P) 1/1/31(P)
Show command output on Leaf1A

Note: The command output shown in the following commands is for Leaf1A. The output for
Leaf1B is similar.
With SFS, port channel numbers are automatically assigned as they are created. Port
channel 1 is the uplink connected to the external switches and is up and active. Port
channel 1000 is reserved for the VLTi.
S5248F-Leaf1A# show port-channel summary

--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
1000 port-channel1000 (U) Eth STATIC 1/1/49(P) 1/1/50(P) 1/1/51(P)
1/1/52(P)
The L2 uplink, port channel 1 in this example, is added as a tagged member of VLAN
1811. This is verified at the CLI using the show virtual-network command as
follows:
S5248F-Leaf1A# show virtual-network
Codes: DP - MAC-learn Dataplane, CP - MAC-learn Controlplane, UUD - Unknown-
Unicast-Drop
Un-tagged VLAN: 4080
Virtual Network: 1811
VLTi-VLAN: 1811
Members:
VLAN 1811: port-channel1, port-channel1000
VxLAN Virtual Network Identifier: 1811
Source Interface: loopback2(172.30.0.0)
Remote-VTEPs (flood-list):

Description: In-band SmartFabric Services discovery network
VLTi-VLAN: 3939

Deployment Guide
Members:
VLAN 3939: port-channel1000, ethernet1/1/1, ethernet1/1/2, ethernet1/1/3

Description: Default untagged network for client onboarding
VLTi-VLAN: 4091
Members:
Untagged: ethernet1/1/1, ethernet1/1/2, ethernet1/1/3
VLAN 4091: port-channel1000
Configure L3 routed uplinks to the external network

Uplinks to the existing network may be configured as L2, L3 routed, or L3 VLAN. This
section covers L3 routed uplinks.
Note: If L2 uplinks were configured in the preceding section, skip this section and go to Configure
a jump host port.
Note: L3 VLAN uplink configuration is beyond the scope of this guide.
Connections, port numbers, and networks used for external management are shown in
Figure 26. The External Management VLAN is VLAN 1911 on the external switches and is
VLAN 1811 on the SmartFabric switches.

Deployment Guide
VLTi
33 21 22 33 21 22
VLT po1
SmartFabric
53 54 53 54
Existing DNS & Leaf1A Leaf1B
3 2 1 1 2 3
NTP server
VLTi
DNS & NTP server in external switch

VLAN 1911, 172.19.11.0/24 network
L3 uplink connections VxRail node 1

L2 connections
VLTi connections
VxRail node 2
VxRail node 3
Rack 1
VxRail nodes in SmartFabric VLAN 1811,
172.18.11.0/24 network
Figure 26. L3 routed uplinks to the external network
Point-to-point IP The point-to-point links used in this deployment are labeled A-E in Figure 27.
networks
E
33 21 22 33 21 22
VLTi
A B C D
VLT po1
SmartFabric
53 54 53 54
NTP services VLTi
Rack 1
L3 uplink connections
L2 connections
VLTi connections
Figure 27. Point to Point connections
Each L3 uplink is a separate, point-to-point IP network. Table 6 details the links labeled in
Figure 27. The IP addresses in the table are used in the switch configuration examples.

Deployment Guide
Table 6. L3 routed uplink IP addresses
Link Source IP Destination Destination IP

Source switch Network
label address switch address
A External-A 192.168.1.0 Leaf1A 192.168.1.1 192.168.1.0/31
B External-A 192.168.1.2 Leaf1B 192.168.1.3 192.168.1.2/31
C External-B 192.168.2.0 Leaf1A 192.168.2.1 192.168.2.0/31
D External-B 192.168.2.2 Leaf1B 192.168.2.3 192.168.2.2/31
E External-A 192.168.3.20 External-B 192.168.3.21 192.168.3.20/31
BGP Example This section covers the L3 routed uplink configuration with BGP.
Note: If BGP is not used, go to the Static route example.
BGP ASNs and router IDs

Figure 28 shows the autonomous system numbers (ASNs) and router IDs used for the
external switches and SFS leaf switches in this example. External switches share a
common ASN, and all SFS leaf switches share a common ASN.
AS 65101
10.0.2.1/32 10.0.2.2/32
33 21 22 33 21 22
VLTi
VLT po1
SmartFabric
53 54 53 54
NTP services
AS 65011 VLTi
Rack 1
L2 connections
VLTi connections
Figure 28. BGP ASNs and router IDs
Note: Using private ASNs in the data center is a best practice. Private, 2-byte ASNs range from
64512 through 65534.
In this example, ASN 65101 is used on both external switches. SFS leaf switches use
ASN 65011 by default for all leafs in the fabric.
Note: If L3 uplinks are connected from SFS spine switches, the spine switches use ASN 65012 by
default.

Deployment Guide
The IP addresses shown on the external network switches in Figure 28 are loopback
addresses used as BGP router IDs. On the SmartFabric switches, BGP router IDs are
automatically configured from the SFS default private subnet address block,
172.16.0.0/16.
Note: SFS default ASNs and IP address blocks may be changed by going to 5. Edit Default
Fabric Settings in the SFS GUI.
Configure L3 routed uplinks with BGP in SFS

Table 7 shows the values entered in the SFS GUI to configure the L3 uplinks for this
example. The steps below the table are run once for each uplink using the values in the
table.
Table 7. L3 uplink configuration details with BGP
Leaf1A-to- Leaf1A-to- Leaf1B-to- Leaf1B-to-

Field name
External-A External-B External-A External-B
Uplink Type L3 Routed L3 Routed L3 Routed L3 Routed
Uplink Name Leaf1A-to- Leaf1A-to- Leaf1B-to- Leaf1B-to-

Switch Group Leaf Leaf Leaf Leaf
Rack Rack 1 Rack 1 Rack 1 Rack 1
Leaf Switch Leaf1A Leaf1A Leaf1B Leaf1B
Interface Ethernet 1/1/53 Ethernet 1/1/54 Ethernet 1/1/53 Ethernet 1/1/54
Network Name Leaf1A-to-ExtA Leaf1A-to-ExtB Leaf1B-to-ExtA Leaf1B-to-ExtB
IPv4 Address 192.168.1.1 192.168.2.1 192.168.1.3 192.168.2.3
Prefix Length 31 31 31 31
Routing Protocol eBGP eBGP eBGP eBGP
Profile Name eBGP-Leaf1A- eBGP-Leaf1A- eBGP-Leaf1B- eBGP-Leaf1B-

to-ExtA to-ExtB to-ExtA to-ExtB
Peer IPv4 Address 192.168.1.0 192.168.2.0 192.168.1.2 192.168.2.2
Remote ASN 65101 65101 65101 65101
Note: Any ports available on the leaf switches may be used as uplinks, provided they are
compatible with the corresponding ports on the external switches. If leaf switch uplink ports will not
uplinks are created. This is done using the 1. Breakout Switch Ports option on the SFS GUI
home page. A breakout example is shown in the Change the port-group speed in the SFS GUI
section of this guide.

Deployment Guide
To configure L3 routed uplinks with BGP, do the following using the data from Table 7:
1. In the SFS GUI, select 2. Create Uplink for External Network Connectivity.
Figure 29. SFS GUI Home Page
2. On the Uplink Details page:

a. Set Uplink Connectivity to Layer 3.
b. Leave Network Type set to L3 Routed.
c. Enter a unique Name and, optionally, a Description.
Figure 30. Uplink Details
3. Click NEXT.

Deployment Guide
4. On the Port Configuration page:
a. Leave Switch Group set to Leaf.
b. Next to Rack, select the rack that contains the switches with the uplinks. In
this example, Rack 1 is selected.
c. Next to Leaf Switch, select the first leaf, Leaf1A in this example.
d. Next to Configured Interface, select the first interface. In this example,
100 GbE interface 1/1/53 is selected.
Figure 31. Port Configuration
5. Click NEXT.

Deployment Guide
6. On the Network Configuration page:
a. Enter a unique Name and, optionally, a Description.
b. Enter the Interface IP Address and Prefix length.
c. Select the Routing Protocol, eBGP.
d. Enter a unique Profile Name.
e. Enter the Peer Interface IP Address and Remote ASN.
Figure 32. Network Configuration page with BGP
7. Click FINISH.
Repeat the steps in this section for the remaining three uplinks using the data from Table
7.
After uplink configuration, the SFS GUI Home page appears, as shown in Figure 33.
Figure 33. SFS Home page after uplinks configured

Deployment Guide
Individual uplinks created are visible on the Uplinks tab of the SFS GUI, as shown in
Figure 34.
Figure 34. SFS L3 uplinks created
Static route This section shows L3 routed uplink configuration with a static route.
example
Note: If BGP is used instead of a static route, continue to Configure external switches for L3
connections.
Note: Currently, only one static route per L3 uplink is allowed. If multiple routes are needed, use a
default route, 0.0.0.0/0, as the destination network, or add additional uplinks for specific networks.
Support for multiple static routes per L3 uplink is planned for a future release.
Configure L3 uplinks with a static route in SFS

Table 8 shows the values entered in the SFS GUI to configure the L3 uplinks for this
example. The steps below the table are run once for each uplink using the values from the
table.
Table 8. L3 uplink configuration details with a static route

Field name
Uplink Type L3 Routed L3 Routed L3 Routed L3 Routed
Uplink Name Leaf1A-to- Leaf1A-to- Leaf1B-to- Leaf1B-to-

Switch Group Leaf Leaf Leaf Leaf
Rack Rack 1 Rack 1 Rack 1 Rack 1
Leaf Switch Leaf1A Leaf1A Leaf1B Leaf1B
Network Name Leaf1A-to-ExtA Leaf1A-to-ExtB Leaf1B-to-ExtA Leaf1B-to-ExtB
IPv4 Address 192.168.1.1 192.168.2.1 192.168.1.3 192.168.2.3
Routing Protocol Static Route Static Route Static Route Static Route
Policy Name Leaf1A-to-ExtA Leaf1A-to-ExtB Leaf1B-to-ExtA Leaf1B-to-ExtB
Network Address 172.19.11.0 172.19.11.0 172.19.11.0 172.19.11.0

Deployment Guide
Field name
Next Hop IP 192.168.1.0 192.168.2.0 192.168.1.2 192.168.2.2

Address
Note: Any ports available on the leaf switches may be used as uplinks, provided they are
compatible with the corresponding ports on the external switches. If leaf switch uplink ports will not
uplinks are created. This is done using the 1. Breakout Switch Ports option on the SFS GUI
home page. A breakout example is shown in the Change the port-group speed in the SFS GUI
section of this guide.
To configure L3 routed uplinks with a static route, do the following:
1. In the SFS GUI, select 2. Create Uplink for External Network Connectivity.
Figure 35. SFS GUI Home Page

a. Set Uplink Connectivity to Layer 3.
b. Leave Network Type set to L3 Routed.
c. Enter a unique Name and, optionally, a Description.
Figure 36. Uplink Details

Deployment Guide
3. Click NEXT.
a. Leave Switch Group set to Leaf.
b. Next to Racks, select the rack that contains the uplink switches. In this
example, Rack 1 is selected.
c. Next to Leaf Switches, select the first leaf, Leaf1A in this example.
d. Next to Configured Interfaces, select the first interface. In this example,
100 GbE interface 1/1/53 is selected.
Figure 37. Port Configuration
5. Click NEXT.

Deployment Guide
a. Enter a unique Name and, optionally, a Description.
b. Enter the Interface IP Address and Prefix length.
c. Leave Routing Protocol set to Static Route.
d. Enter a unique Policy Name.
e. Enter the destination Network Address and Prefix Length. This is the
external management network, 172.19.11.0/24, in this example.
f. Enter the Next Hop IP Address. This is the IP address of the connected
interface on the external switch.
Figure 38. Network Configuration page with a static route
7. Click FINISH.
8.
After uplink configuration, the SFS GUI Home page appears as shown in Figure 39.
Figure 39. SFS Home page after uplink configuration

Deployment Guide
Individual uplinks created are visible on the Uplinks tab of the SFS GUI, as shown in
Figure 40.
Figure 40. SFS L3 uplinks created
Configure This section shows example configurations for both external switches for L3 routed
external connections to the SmartFabric.
switches for L3
connections Note: The external switches used in this example are Dell EMC PowerSwitch systems. If the
external switches are Cisco Nexus, see Appendix C.
Note: This is only an example. Modify your external switch configuration as needed for your
network.
General settings
Configure the hostname, OOB management IP address, and management route.
hostname External-A hostname External-B
interface mgmt1/1/1 interface mgmt1/1/1

no ip address no ip address
management route 100.67.0.0.0/16 management route 100.67.0.0/16

100.67.127.254 100.67.127.254
Configure VLANs
VLAN 1911 represents a preexisting management VLAN on the external network. DNS
and NTP services are located on this VLAN. Assign a unique IP address to the VLAN on
each switch.
Configure VRRP to provide gateway redundancy. Set the VRRP priority. The switch with
the highest priority value becomes the master VRRP router. Assign the same virtual
address to both switches.

Deployment Guide
vrrp-group 19 vrrp-group 19
priority 150 priority 100
virtual-address 172.19.11.254 virtual-address 172.19.11.254
Configure the interfaces for connections to the SFS switches. Ports 1/1/21 and 1/1/22 are
configured as L3 interfaces. The IP addresses used are from Table 6. Optionally, allow
the forwarding of jumbo frames with the MTU 9216 command. As a best practice, flow
control settings remain at their factory defaults as shown.
In this example, VLT port channel 1 connects to the DNS/NTP server. It is on VLAN 1911,
which represents the preexisting management VLAN, and the port channel is configured
as a spanning tree edge port.
Interface 1/1/33 on each external switch is configured in VLT port channel 1 for
connections to the DNS/NTP server. Port-channel 1 is set as an LACP port channel with
the channel-group 1 mode active command.

Deployment Guide
description Leaf1A description Leaf1A
mtu 9216 mtu 9216

description Leaf1B description Leaf1B
mtu 9216 mtu 9216

switchport access vlan 1911 switchport access vlan 1911
spanning-tree port type edge spanning-tree port type edge

Configure VLT
This example uses interfaces 1/1/29 and 1/1/31 for the VLTi. Remove each interface from
L2 mode with the no switchport command. As a best practice, flow control settings
remain at their factory defaults, as shown.
Create the VLT domain. The backup destination is the OOB management IP address of
the VLT peer switch. Configure the interfaces used as the VLTi with the discovery-
interface command.
As a best practice, use the vlt-mac command to manually configure the same VLT MAC
address on both the VLT peer switches. This improves VLT convergence time when a
switch is reloaded.
Caution: Be sure the VLT MAC address is the same on both switches to avoid any unpredictable
behavior.

Deployment Guide
If you do not configure a VLT MAC address, the MAC address of the primary peer is used
as the VLT MAC address on both switches.
Note: For more information about VLT, see the Dell EMC SmartFabric OS10 User Guide on the
Dell EMC Networking OS10 Info Hub.
interface range interface range
description VLTi description VLTi
no switchport no shutdown
no shutdown no switchport
vlt-domain 255 vlt-domain 255

backup destination 100.67.127.36 backup destination 100.67.127.37
discovery-interface discovery-interface
vlt-mac 00:00:01:02:03:20 vlt-mac 00:00:01:02:03:20
Configure BGP
Note: If BGP is not used, go to the Configure static routes section.
Configure a loopback interface to use for the BGP router ID.
Configure the BGP ASN with the router bgp command. The external switches share
the same ASN. Use the address that was set for interface loopback0 as the router ID.
Use the address-family ipv4 unicast and redistribute connected

commands to redistribute IPv4 routes from physically connected interfaces.
Configure the neighbor IP addresses and ASNs.
VLAN 4000 is used for the iBGP connection between the external switches. VLAN4000 IP
addresses are configured per Table 6.

Deployment Guide
interface loopback0 interface loopback0
description router_ID description router_ID
router bgp 65101 router bgp 65101

router-id 10.0.2.1 router-id 10.0.2.2
address-family ipv4 unicast address-family ipv4 unicast

redistribute connected redistribute connected
neighbor 192.168.1.1 neighbor 192.168.2.1

remote-as 65011 remote-as 65011



description iBGP description iBGP
end end
Configure static routes

Note: If BGP is used, skip this section and go to Validate BGP example.
Configure two routes to the external management network. This is 172.18.11.0/24, one to
the connected IP address of Leaf1A, and one to Leaf1B.

Deployment Guide
ip route 172.18.11.0/24 ip route 172.18.11.0/24
192.168.1.1 192.168.2.1
ip route 172.18.11.0/24 ip route 172.18.11.0/24

192.168.1.3 192.168.2.3
end end
Validate BGP Note: This section shows validation commands for the BGP example. If static routes are used,
example skip this section and go to Validate static route example.
Now that the uplink interfaces are configured on the external switches and on the SFS leaf
switches, connectivity can be verified using the switch CLI.
Show command output on External-A (BGP example)

Run the show ip bgp summary command to verify that BGP is up for each neighbor.
When BGP is up, uptime is shown in the Up/Down column. The neighbors for External-A
shown in the output below are Leaf1A, Leaf1B, and External-B.
External-A# show ip bgp summary

BGP router identifier 10.0.2.1 local AS number 65101
Neighbor AS MsgRcvd MsgSent Up/Down State/Pfx
192.168.1.1 65011 1327 1316 19:09:00 4
192.168.1.3 65011 1325 1324 19:09:00 4
192.168.3.21 65101 1319 1315 19:01:18 5
Run the show ip interface brief command to verify connected interfaces are up,
and IP addresses are configured correctly. VLAN 1911 is the external management VLAN
that contains the DNS/NTP server. VLAN 4094 and port channel 1000 are automatically
configured for the VLTi.
Note: Unused interfaces have been removed from the output for brevity.
External-A# show ip interface brief

Interface Name IP-Address OK Method Status Protocol
================================================================================
Ethernet 1/1/21 192.168.1.0/31 YES manual up up
Ethernet 1/1/29 unassigned YES unset up up
Management 1/1/1 100.67.127.37/24 YES manual up up
Vlan 1 unassigned YES unset up up
Vlan 1911 172.19.11.252/24 YES manual up up

Deployment Guide
Port-channel 1 unassigned YES unset up up
Loopback 0 10.0.2.1/32 YES manual up up
The show ip route command output for the External-A switch appears as shown. No
BGP routes from the SFS fabric are learned at this stage of deployment. Interfaces 1/1/21
and 1/1/22 are connected to the SFS leaf switches.
External-A# show ip route

Codes: C - connected
S - static
B - BGP, IN - internal BGP, EX - external BGP
O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
E2 - OSPF external type 2, * - candidate default,
+ - summary route, > - non-active route
Gateway of last resort is not set
Destination Gateway Dist Last Change
----------------------------------------------------------------------------------
C 10.0.2.1/32 via 10.0.2.1 loopback0 0/0 00:39:19
B IN 10.0.2.2/32 via 192.168.3.21 200/0 00:31:38
C 172.19.11.0/24 via 172.19.11.252 vlan1911 0/0 00:44:00
C 192.168.1.0/31 via 192.168.1.0 ethernet1/1/21 0/0 01:44:44
C 192.168.1.2/31 via 192.168.1.2 ethernet1/1/22 0/0 01:40:50
B IN 192.168.2.0/31 via 192.168.3.21 200/0 00:31:38
B IN 192.168.2.2/31 via 192.168.3.21 200/0 00:31:38
C 192.168.3.20/31 via 192.168.3.20 vlan4000 0/0 00:31:51
Show command output on Leaf1A (BGP example)

Leaf1B is similar.
When BGP is up, uptime is shown in the Up/Down column. The neighbors for Leaf1A
shown in the output below are Leaf1B, External-A, and External-B.
S5248F-Leaf1A# show ip bgp summary

172.16.0.0 65011 13 16 00:06:59 8
192.168.1.0 65101 12 14 00:07:30 8
192.168.2.0 65101 8 9 00:04:14 8

Deployment Guide
and IP addresses are configured correctly.
In the output below, interfaces 1/1/1-1/1/3 are connected to the VxRail nodes, 1/1/49-
1/1/52 are the VLTi, and 1/1/53-1/1/54 are the uplinks to the external switches. SFS uses
VLAN 4090, Loopback 1, and Loopback 2 internally. VLAN 4094 and port channel 1000
are automatically configured for the VLTi.
S5248F-Leaf1A# show ip interface brief

================================================================================
Virtual-network 3939 unassigned YES unset up up
Run the show ip route command to verify routes to the external management VLAN,
172.19.11.0/24, have been learned using BGP from the external switches. In this
example, two routes to 172.19.11.0/24 are learned, one using each external switch.
S5248F-Leaf1A# show ip route

S - static
----------------------------------------------------------------------------------
B EX 10.0.2.1/32 via 192.168.1.0 20/0 00:43:16
via 192.168.2.0
B EX 10.0.2.2/32 via 192.168.1.0 20/0 00:43:16
via 192.168.2.0

Deployment Guide
C 172.16.0.0/31 via 172.16.0.1 vlan4090 0/0 02:19:46
C 172.16.128.0/32 via 172.16.128.0 loopback1 0/0 02:20:07
B IN 172.16.128.1/32 via 172.16.0.0 200/0 02:19:44
B EX 172.19.11.0/24 via 192.168.1.0 20/0 00:43:32
via 192.168.2.0
C 172.30.0.0/32 via 172.30.0.0 loopback2 0/0 02:20:07
C 192.168.1.0/31 via 192.168.1.1 ethernet1/1/53 0/0 01:12:49
B IN 192.168.1.2/31 via 172.16.0.0 200/0 01:09:12
C 192.168.2.0/31 via 192.168.2.1 ethernet1/1/54 0/0 01:10:18
B IN 192.168.2.2/31 via 172.16.0.0 200/0 01:07:51
B EX 192.168.3.20/31 via 192.168.1.0 20/0 00:43:21
via 192.168.2.0
Validate static Note: This section shows validation commands for the static route example. If BGP was used,
route example skip this section and go to Configure a jump host port.
Once the uplink interfaces have been configured on the external switches and in the SFS
GUI, connectivity can be verified using the switch CLI.
Show command output on External-A (static route example)

and IP addresses are configured correctly. In the output below, interface 1/1/33 and port
channel 1 connect to the DNS/NTP server. 1/1/21-1/1/22 are the links to the SFS leaf
switches, and 1/1/29 and 1/1/31 are the VLTi links. VLAN 4094 and port channel 1000 are
automatically configured for the VLTi.

================================================================================
Run the show ip route command to verify static routes to the external management
VLAN, 172.18.11.0/24, are properly configured.

Deployment Guide
S - static
----------------------------------------------------------------------------------
S 172.18.11.0/24 via 192.168.1.1 ethernet1/1/21 1/0 3 days 23:35:18
via 192.168.1.3 ethernet1/1/22
C 172.19.11.0/24 via 172.19.11.252 vlan1911 0/0 3 days 23:26:55
C 192.168.1.0/31 via 192.168.1.0 ethernet1/1/21 0/0 21:58:31
C 192.168.1.2/31 via 192.168.1.2 ethernet1/1/22 0/0 21:58:33
Show command output on Leaf1A (static route example)

Leaf1B is similar.
1/1/52 are the VLTi links, and 1/1/53-1/1/54 are the uplinks to the external switches.


Deployment Guide
Run the show ip route command to verify static routes to the external management
VLAN, 172.19.11.0/24, are correctly configured.
Note: Since BGP is used by SFS to exchange routes within the fabric, some BGP routes appear
in the output.

S - static
----------------------------------------------------------------------------------
C 172.16.0.0/31 via 172.16.0.1 vlan4090 0/0 02:09:34
C 172.16.0.0/31 via 172.16.0.1 vlan4090 0/0 00:40:43

B IN 172.16.128.0/32 via 172.16.0.0 200/0 00:40:42
C 172.16.128.1/32 via 172.16.128.1 loopback1 0/0 00:40:50
S 172.19.11.0/24 via 192.168.1.0 ethernet1/1/53 1/0 00:37:51
via 192.168.2.0 ethernet1/1/54
C 172.30.0.0/32 via 172.30.0.0 loopback2 0/0 00:40:50
C 192.168.1.0/31 via 192.168.1.1 ethernet1/1/53 0/0 00:37:56
B IN 192.168.1.2/31 via 172.16.0.0 200/0 00:35:52
C 192.168.2.0/31 via 192.168.2.1 ethernet1/1/54 0/0 00:36:57
B IN 192.168.2.2/31 via 172.16.0.0 200/0 00:34:51
Configure a jump host port

VxRail Manager is used for VxRail deployments. The VxRail Manager VM automatically
runs on the master VxRail node, which is the node with the lowest VxRail serial number.
Note: Before VxRail deployment, the VxRail Manager is accessible on an untagged port on the
SFS Client Management VLAN (VLAN 4091 by default). The default IP address is
192.168.10.200.
VxRail Manager is accessed by connecting a laptop computer or a jump host directly to

any available leaf switch port, as shown in Figure 41.

Deployment Guide
Leaf1A Leaf1B
9 3 2 1 1 2 3
VLTi
Jump host or
Laptop VxRail node 1
VxRail node 2
VxRail node 3
Rack 1
Figure 41. Jump host connected leaf switch for VxRail deployment
This section covers the configuration of a leaf switch port for connection to a jump host or
laptop computer (referred to only as a jump host for the remainder of this guide).
Change native If the jump host has a 1 GbE or 10 GbE NIC, and it is connected to a 25 GbE port on an
port speed on S5200 series switch, the switch port used must be changed from its native 25 GbE speed
S5200 series to 10 GbE for the port to come up.
switches
If the jump host has a 1 GbE or 10 GbE NIC, and is connected to a 10 GbE port on an
S4100 series leaf switch, or has a 25 GbE NIC and connects to an S5200 series leaf
switch, leave the port at its native speed, skip this section, and go to Configure the jump
host interface.
Note: When in 10 GbE mode, an S5200 series switch port will autonegotiate to 1 GbE when
connected to a 1 GbE NIC. To connect a jump host with a 1 GbE BASE-T or 10 GbE BASE-T port
to an SFP port on the leaf switch, use a Dell EMC supported SFP-1G-T or SFP-10G-T adapter.
Determine the port group

Determine the port group containing the port that the jump host will use. This is done by
referencing the table below for S5200 series switches, or by running the show port-
group command from the leaf switch CLI. For example, if the jump host is connected to
port 1/1/9, it is in port group 1/1/3.

Deployment Guide
Table 9. S5200 series port-group numbers and speeds
Non-native
Port Non-
Native physical Native logical
group native Applicable switches
interface name speed interface
number speed
name
Eth 1/1/1-1/1/4 1/1/1 25g-4x 10g-4x Eth 1/1/x:1 S5212/S5224/S5248/S5296

Eth 1/1/13-1/1/16 1/1/4 25g-4x 10g-4x Eth 1/1/x:1 S5224/S5248/S5296
Eth 1/1/17-1/1/20 1/1/5 25g-4x 10g-4x Eth 1/1/x:1 S5224/S5248/S5296
Eth 1/1/21-1/1/24 1/1/6 25g-4x 10g-4x Eth 1/1/x:1 S5224/S5248/S5296
Eth 1/1/25-1/1/28 1/1/7 25g-4x 10g-4x Eth 1/1/x:1 S5248/S5296
Eth 1/1/29-1/1/32 1/1/8 25g-4x 10g-4x Eth 1/1/x:1 S5248/S5296
Eth 1/1/33-1/1/36 1/1/9 25g-4x 10g-4x Eth 1/1/x:1 S5248/S5296
Eth 1/1/37-1/1/40 1/1/10 25g-4x 10g-4x Eth 1/1/x:1 S5248/S5296
Eth 1/1/41-1/1/44 1/1/11 25g-4x 10g-4x Eth 1/1/x:1 S5248/S5296
Eth 1/1/45-1/1/48 1/1/12 25g-4x 10g-4x Eth 1/1/x:1 S5248/S5296
Eth 1/1/49-1/1/52 1/1/13 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/53-1/1/56 1/1/14 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/57-1/1/60 1/1/15 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/61-1/1/64 1/1/16 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/65-1/1/68 1/1/17 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/69-1/1/72 1/1/18 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/73-1/1/76 1/1/19 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/77-1/1/80 1/1/20 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/81-1/1/84 1/1/21 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/85-1/1/88 1/1/22 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/89-1/1/92 1/1/23 25g-4x 10g-4x Eth 1/1/x:1 S5296
Eth 1/1/93-1/1/96 1/1/24 25g-4x 10g-4x Eth 1/1/x:1 S5296
Note: Changing the speed is done for all ports in the port group. In this example, setting port
group 1/1/3 to 10g-4x changes ports 1/1/9-1/1/12 to 10 GbE, and the ports are renamed 1/1/9:1-
1/1/12:1.
Change the port-group speed in the SFS GUI

In this section, port group 1/1/3 is changed from its 4x25 G native speed to 4x10 G to
accommodate a jump host connected to port 1/1/9 on S5248F-Leaf1A.
1. On the SFS GUI Home page, click 1. Breakout Switch Ports.

2. Select the Rack, Switch, and Port group. Set Breakout Capabilities to
4X10GEFixedFormFactor, as shown in Figure 42.

Deployment Guide
Figure 42. Configure switch ports
3. Click OK to apply the setting.

Configure the The jump host interface is configured as follows:
jump host
1. On the SFS GUI Home page, click 2. Configure Jump Host.
interface
2. In the Configure Jump Host window:
a. Enter a Name, such as Jump Host 1.
b. (Optional) Enter a Description
c. Select the Rack, for example, Rack 1.
d. Select the Switch, for example, Leaf1A.
e. Select the Configured Interface that the jump host uses. In this example, it
is 1/1/9:1.
Note: Port 1/1/9 was automatically renamed to 1/1/9:1 when its port group was changed from its
native setting of 4x 25 GbE to 4x 10 GbE.
f. Next to Untagged Network, leave the network set to

Client_Management_Network, as shown in Figure 43.

Deployment Guide
Figure 43. Configure jump host
3. Click OK to apply the settings.

The jump host interface is added as an untagged member of the
Client_Management_Network (VLAN 4091 by default).
To validate, run the show virtual-network command at the Leaf1A switch CLI.
Note: The output below is with an L3 uplink. If an L2 uplink to the external network is configured,
the External Management Network, VLAN 1811 in this guide, also appears in the command output
along with the L2 uplink port channel as a member. An output is shown in the Show command
output on Leaf1A section of the L2 uplink configuration chapter.
VLAN 3939 contains port channel 1000 (the VLTi) and the three interfaces connected to
the VxRail nodes, 1/1/1-1/1/3. VLAN 4091 contains port channel 1000, the three
interfaces connected to the VxRail nodes, and the jump host port, ethernet 1/1/9:1.
Note: The jump host port is only configured on one of the leaf switches.

Codes: DP - MAC-learn Dataplane, CP - MAC-learn Controlplane, UUD
- Unknown-Unicast-Drop
VLTi-VLAN: 3939
Members:
VLAN 3939: port-channel1000, ethernet1/1/1, ethernet1/1/2,
ethernet1/1/3

Deployment Guide

VLTi-VLAN: 4091
Members:
Untagged: ethernet1/1/1, ethernet1/1/2, ethernet1/1/3,
ethernet1/1/9:1
Configure the By default, the initial VxRail Manager IP address is 192.168.10.200/24, and it is in VLAN
jump host IP 4091. After the initial configuration, the VxRail Manager address changes to its new
addresses address on the External Management VLAN (VLAN 1811 in this example). The new
VxRail Manager address used in this guide is 172.18.11.72/24 per the planning data in
Table 4.
During installation, the jump host must be able to reach both the initial and new VxRail
Manager addresses, so two addresses are configured on its network adapter, one for
each network.
The IP addresses are configured on the jump host NIC in this example as follows:
• 192.168.10.201/24, to communicate with the initial VxRail Manager address,

192.168.10.200/24
• 172.18.11.201/24, to communicate with the new VxRail Manager address,
172.18.11.72/24
Note: Both addresses may be configured simultaneously if the network adapter supports it, or in
sequence if required. During VxRail deployment, the jump host port on the switch is automatically
moved from VLAN 4091 to the External Management VLAN, along with VxRail Manager. Since
the jump host port on the switch is untagged in VLAN 4091, and will also be untagged in the
External Management VLAN, no VLAN information is configured on the jump host NIC.
Once the jump host has been configured with an IP address on the 192.168.10.0/24
network, verify the jump host can communicate with VxRail Manager by pinging
192.168.10.200 from the jump host.
Figure 44. Jump host successfully pings VxRail manager

Deployment Guide
Chapter 6 Deploy VxRail
Prerequisites .........................................................................................................71
Initial VxRail cluster deployment steps .............................................................71
Additional configuration steps for L3 uplinks...................................................77
Validate and build VxRail cluster ........................................................................80

Deployment Guide
Prerequisites
Initial VxRail cluster deployment steps

In this chapter, the three VxRail nodes in Rack 1 are deployed as a 3-node cluster. VLANs
specified during VxRail deployment are automatically configured in the SmartFabric.
1. In a browser on the jump host, go to https://192.168.10.200 to connect to VxRail
Manager. The Welcome to VxRail screen displays, as shown in Figure 45.
Figure 45. VxRail welcome screen
2. Click GET STARTED on the welcome screen.

3. On the EULA page, review the terms provided, and if you agree, click ACCEPT.
4. On the Expected Dell EMC switch Clusters page, the SmartFabric cluster is
discovered, as shown in Figure 46.

Deployment Guide
Figure 46. Dell EMC SmartFabric switch cluster found
5. Check the box next to I confirm I want to configure the listed Dell EMC switch
clusters and click NEXT.
6. In the Dell EMC Switch Password Change window, set the REST_USER
password.
Note: The REST_USER is an account used by VxRail and OMNI to configure the switches.
Figure 47. Set REST_USER password

Deployment Guide
7. Click NEXT. A success message displays.
Figure 48. REST_USER password changed
8. Click NEXT.
9. The VxRail cluster type page displays. Select VxRail cluster.
Figure 49. VxRail cluster selection
Note: 2-node VxRail clusters are not currently supported with SFS.
10. Click NEXT.

Deployment Guide
Wait for all VxRail nodes in the rack to be discovered. Discovery may take about 5
minutes. The three VxRail nodes in Rack 1 are discovered, as shown in Figure 50.
Figure 50. VxRail nodes discovered
11. When all nodes in the rack have been discovered, check the box next to I
confirm I want to configure the listed VxRail nodes and click NEXT.
12. On the How would you like to configure VxRail screen, select your preferred
configuration method, Step-by-step, or Configuration file. Either configuration
method may be used.
Note: A JSON-formatted configuration file may be used if you have saved one from a previous
installation using the same versions of VxRail and SmartFabric OS10, or if you have been
provided one from your sales representative. If you do not have a configuration file, select Step-
by-step.

Deployment Guide
Figure 51. Select the configuration method
13. Click NEXT.
Note: Step-by-step VxRail configuration screens are not in this guide, but are provided in the
VxRail Appliance Installation Procedures Guide that is available on Dell Technologies Solve
Online (account required). The values used in the VxRail configuration for this example are
entered using the planning data provided in Table 4.
14. When done with the configuration screens, you have the option to click REVIEW
FIRST to review your selections, as shown in Figure 52.
Figure 52. Switch Configuration screen
15. When done reviewing, click CONFIGURE SWITCH.
Note: VxRail Manager uses multicast DNS (mDNS) on the SFS Client Control network to discover
the SFS master switch virtual IP address.

Deployment Guide
When the switch configuration is complete, the Review and validate screen displays, as
shown in Figure 53.
CAUTION: Do not press the VALIDATE button at this time.
Figure 53. Review and validate screen
Note: Before proceeding, be sure to keep the browser window that is displaying the screen shown
in Figure 53 open on the jump host.
Optionally, switch configuration can be verified at the CLI with the show virtual-
network command.
The command output below shows the External Management, vMotion, vSAN, and VM
network VLANs (VLANs 1811 through 1815 from Table 4) specified on the VxRail
deployment screens have been automatically configured on the leaf switches. The VxRail
node ports, interfaces 1/1/1-1/1/3, and the VLTi port channel, port channel 1000, are
members of all VLANs. The jump host port, interface 1/1/9:1, is still a member of VLAN
4091.
Note: The output below is with an L3 uplink. If an L2 uplink is configured, the uplink port channel
also appears as a member of VLANs 1811 through 1815.

Unicast-Drop
VLTi-VLAN: 1811
Members:

VLTi-VLAN: 1812
Members:

Deployment Guide

VLTi-VLAN: 1813
Members:

VLTi-VLAN: 1814
Members:

VLTi-VLAN: 1815
Members:

VLTi-VLAN: 3939
Members:

VLTi-VLAN: 4091
Members:
Untagged: ethernet1/1/1, ethernet1/1/2, ethernet1/1/3, ethernet1/1/9:1
Additional configuration steps for L3 uplinks

Deployment Guide
Note: If using L2 uplinks, skip this section and go to Validate and build VxRail cluster.
Traffic on the External Management network, VLAN 1811, must be able to reach the DNS
server on the external network during VxRail deployment. To accomplish this with L3
uplinks, an IP address is assigned to each leaf switch on virtual network 1811. An anycast
gateway address shared by all leafs is also configured on the same network.
Since this is on virtual network 1811, available IP addresses in the 172.18.11.0/24

address block are used per the planning data in Table 3.
Table 10. Leaf switch External Management network IP addresses and anycast gateway
Item IP address/prefix
Leaf1A IP address 172.18.11.253/24
Leaf1B IP address 172.18.11.252/24
Gateway IP address 172.18.11.254/24
Note: If present, additional leaf switches in the fabric will also need one IP address per leaf on this
network.
The IP addresses and gateway are configured as follows:
1. From a workstation, launch the SFS GUI.

2. On the SFS GUI Home page, select 3. Update Network Configuration. The
window opens, as shown in Figure 54.
Figure 54. Update network configuration window
a. Next to Network, select the External Management network, Management

Network 1811, from the drop-down list.
b. Next to Enable IP Address, select IPv4.
Note: When IPv4 is selected, additional fields appear, as shown in Figure 55.

Deployment Guide
c. Next to Interface IP Addresses, enter an interface IP address for each leaf
switch in the SmartFabric per Table 10. Click the blue button to add IP
address entry fields.
Note: If you plan to expand the fabric, additional leaf switches will also need IP addresses on this
network, with one IP address per leaf. This is covered in Expand SmartFabric and VxRail cluster
to multi-rack.
d. Enter the Prefix Length for the IP addresses and a Gateway IP Address.
These values are from Table 10.
When complete, the Update Network Configuration window appears, as shown in Figure
55.
3. Click OK.
BGP Validation Note: If static routes are used, proceed to the Validate and build VxRail cluster section. (Static
route validation was done earlier in the Validate static route example section of this guide).
If BGP is used on the uplinks, ensure the external switches have learned the routes to the
VxRail External Management network, 172.18.11.0/24 in this example, to reach the VxRail
nodes and VxRail Manager. This is done with the show ip route command. The BGP-
discovered route to 172.18.11.0/24 is shown in bold in the output below.

Deployment Guide
Note: The command output shown is for the External-A switch. The output for External-B is
similar. BGP verification from the leaf switches was done in Show command output on Leaf1A
(BGP example).
Note: Command output from a Cisco Nexus switch is shown in Appendix C: BGP validation on
N9K-External-A during VxRail deployment.

S - static
Destination Gateway Dist/Metric Change
----------------------------------------------------------------------------------
C 10.0.2.1/32 via 10.0.2.1 loopback0 0/0 18:36:55
B IN 10.0.2.2/32 via 192.168.3.21 200/0 18:16:18
C 172.19.11.0/24 via 172.19.11.252 vlan1911 0/0 18:29:16
B EX 172.18.11.0/24 via 192.168.1.1 20/0 16:02:33
via 192.168.1.3
C 192.168.1.0/31 via 192.168.1.0 ethernet1/1/21 0/0 21:10:53
C 192.168.1.2/31 via 192.168.1.2 ethernet1/1/22 0/0 18:36:56
B IN 192.168.2.0/31 via 192.168.3.21 200/0 21:10:51
B IN 192.168.2.2/31 via 192.168.3.21 200/0 18:16:18
C 192.168.3.20/31 via 192.168.3.20 vlan4000 0/0 18:29:12
Validate and build VxRail cluster

1. Return to the Review and validate screen on the jump host.
Figure 56. Review and validate screen
2. Click VALIDATE.

Deployment Guide
Note: VxRail Manager must be able to reach the DNS server for validation to succeed. Expand
Show history to view the validation status. Validation may take 5 to 10 minutes.
When validation is complete, a message indicating whether the configuration has passed
or failed displays, as shown in Figure 57.
Note: If validation failed, address the items that failed and validate again.
Figure 57. Validation has passed
Note: Once validation passes, Dell Technologies recommends clicking the DOWNLOAD JSON
button to save an offline copy of your VxRail settings.
3. Click BUILD VXRAIL. The screen appears as shown in Figure 58.
Figure 58. New VxRail Manager IP address - start configuration
The switch port connected to the jump host has been automatically moved from VLAN
4091 to the External Management VLAN, VLAN 1811, on the leaf switch to enable it to
reach VxRail Manager on the new network.
Optionally, you can run the show virtual network command on the leaf switch that
the jump host is connected to verify the change. In the output below, the jump host port
1/1/9:1 is now untagged in VLAN 1811.

Deployment Guide
Note: Virtual networks 1812 through 1815 and 3939 have been removed from the output below for
brevity. The output below is with an L3 uplink. If an L2 uplink is configured, the uplink port channel
also appears as a member of VLAN 1811.

Unicast-Drop
VLTi-VLAN: 1811
Members:
Untagged: ethernet1/1/9:1

VLTi-VLAN: 4091
Members:
Note: Ensure the jump host NIC has an IP address on the new network, 172.18.11.0/24 in this
example, before proceeding with the next step. The jump host port on the leaf switch is untagged,
so do not configure a VLAN ID on the jump host NIC.
4. Click the START CONFIGURATION button shown in Figure 58. You are
redirected to the new VxRail Manager IP address in the browser, and VxRail
deployment starts, as shown in Figure 59.

Deployment Guide
Figure 59. VxRail deployment in progress
Deployment takes about 1 hour. When VxRail is successfully deployed, the Hooray!
message displays, as shown in Figure 60.
Figure 60. VxRail deployment successful
5. Click MANAGE VXRAIL to manage the cluster.
Note: The jump host or workstation used must be able to reach the DNS server to resolve the
hostname of the vCenter server for the connection to succeed.
6. Log in using your SSO credentials. In this example, the username is

administrator@vsphere.local. You are redirected to the vCenter server.

Deployment Guide
Note: If prompted, launch the vSphere Client (HTML5). The older Flash-based vSphere Web
Client (Flex) is deprecated and is not used in this guide.
The Hosts and Clusters page of the vSphere Client appears, as shown in Figure 61.
Figure 61. Newly created VxRail cluster
Caution: Review any warnings that may appear in the vSphere Client.

Deployment Guide
Chapter 7 Expand to Multirack
Expand SmartFabric and VxRail cluster to multirack ......................................86
Verify preferred master setting before fabric expansion .................................87
Configure management settings for new switches ..........................................89
Add switches to SmartFabric ..............................................................................90
Connect to the SmartFabric GUI .........................................................................91
Configure additional rack and switch names ....................................................92
Configure leaf switch addresses for L3 uplinks ...............................................92
Add a VxRail node to the cluster ........................................................................94

Deployment Guide
Expand SmartFabric and VxRail cluster to multirack
CAUTION: Do not connect any new switches in SmartFabric mode to the fabric until the preferred
master settings are validated on the existing and new switches. This is covered in the Verify
preferred master setting before fabric expansion section of this chapter.
In this chapter, the SmartFabric is expanded to a second rack with the addition of Spine1,
Spine2, Leaf2A, and Leaf2B. After the fabric is expanded, VxRail node 4 in Rack 2 is
added to the existing VxRail cluster.
Note: SFS supports up to 20 switches in the fabric. This scales to nine racks if two spines and 18
leafs are used. Any combination of leafs and spines may be used with the exception that leaf
switches must be deployed in pairs, and at least two spines are used. VxRail does not support
single spine deployments.
SmartFabric

network

Existing DNS &
VxRail node 2
VxRail node 3
Rack 1 Rack 2
Layer 3 connections
Layer 2 connections
VLTi connections
Layer 2 or Layer 3 connections
Figure 62. SmartFabric and VxRail cluster expansion

Deployment Guide
Verify preferred master setting before fabric expansion
During fabric expansion, the newly added switches may come up and form a fabric among
themselves and elect a master before they are connected to the existing fabric. When the
new fabric merges with the running fabric, it is possible for the master switch from the new
leaf switches to overwrite the configuration in the existing fabric. It is critical to ensure a
pair of leaf nodes in the existing fabric are configured to be the "preferred master" before
expanding the fabric.
When you create an uplink to the external network using the SFS GUI or OMNI, the
preferred master is automatically set on all leaf switches in the fabric at that time.
Note: Spine switches are never elected SmartFabric master or preferred master switches.
Check preferred Before connecting additional leaf switches to the fabric, verify the preferred master is set
master status on on at least one pair of leaf switches currently in the SmartFabric. This is done by running
existing leafs the show smartfabric cluster command on each leaf in the existing fabric.
The output for at least one pair of leaf switches in the SmartFabric must show
PREFERRED-MASTER is set to true. The following commands and output are from
Leaf1A and Leaf1B:
S5248F-Leaf1A# show smartfabric cluster

----------------------------------------------------------
CLUSTER DOMAIN ID : 100
VIP : fde2:53ba:e9a0:cccc:0:5eff:fe00:1100
ROLE : BACKUP
SERVICE-TAG : 690ZZP2
MASTER-IPV4 : 100.67.127.27
PREFERRED-MASTER : true
----------------------------------------------------------
S5248F-Leaf1B# show smartfabric cluster

----------------------------------------------------------
CLUSTER DOMAIN ID : 100
VIP : fde2:53ba:e9a0:cccc:0:5eff:fe00:1100
ROLE : MASTER
SERVICE-TAG : 68X00Q2
MASTER-IPV4 : 100.67.127.27
PREFERRED-MASTER : true
----------------------------------------------------------
For the example in this guide, there are only two leaf switches in the SmartFabric at this
stage of deployment. However, if there were additional leaf switches in the SmartFabric
when the uplink was created, they would also show PREFERRED-MASTER is set to true.
If the fabric was previously expanded after the uplink was created, the added leafs would
not have PREFERRED-MASTER set to true. This is allowed as long as PREFERRED-
MASTER is set to true on at least one pair of leaf switches currently in the SmartFabric.

Deployment Guide
Create an uplink Note: This section only applies if there are no leaf switch pairs in the SmartFabric with
if needed PREFERRED-MASTER set to true. If this does not apply, proceed to the Check preferred
master status on new leafs section.
If leaf switch pairs in the existing SmartFabric do not show PREFERRED-MASTER set to
true, create an uplink by following the instructions in the Configure L2 uplinks to the
external network or Configure L3 routed uplinks to the external network sections. After the
uplink is created, return to the preceding section and check the preferred master setting
again.
If you are using a demo or lab environment without the need for an uplink, create a
temporary uplink to set all of the leaf switches that are currently in the SmartFabric, to be
the preferred master.
Note: Physical port connections are not required to create this temporary uplink.
Create a temporary uplink as follows:
1. On the SFS GUI Home page, select 2. Create Uplink for External Network
Connectivity.
a. Next to Uplink Connectivity, leave Layer 2 selected.
b. Enter a Name, such as temp.
3. Click NEXT.
a. Next to Racks, select any rack.
b. Next to Configured Interfaces, select an available interface on either
switch.
Note: You cannot use this interface for other purposes until you delete the uplink.
c. Leave the LAG mode set to LACP.

5. Click NEXT > FINISH.
After the uplink is created, verify all leaf switches in the SmartFabric show PREFERRED-
MASTER is set to true.
To make the interface used in the temporary uplink available for other purposes, you can
delete the uplink without affecting the preferred master setting by performing the following
steps:
1. On the SFS GUI Uplinks page, select the uplink by name, temp in this example.
2. Click DELETE > OK.
The port used for the temporary uplink is now available.

Deployment Guide
Check preferred Run the show smartfabric cluster command on each leaf switch to be added to
master status on the SmartFabric and ensure none show PREFERRED-MASTER set to true. These
new leafs switches should still be in their factory default mode, Full Switch mode, and the command
and output should appear as follows on each switch:
OS10# show smartfabric cluster
----------------------------------------------------------
CLUSTER DOMAIN ID :
VIP : unknown
ROLE : unknown
SERVICE-TAG : unknown
MASTER-IPV4 :
PREFERRED-MASTER :
----------------------------------------------------------
If any leaf switch to be added to the SmartFabric shows PREFERRED-MASTER is set to

true, the switch configuration should be cleared. This is done by taking each affected
leaf switch out of SmartFabric mode and returning to Full Switch mode with the following
commands:

OS10(config)# no smartfabric l3fabric
After the switch reloads, run show smartfabric cluster again on each affected leaf
switch to confirm PREFERRED-MASTER is no longer set to true.
Note: New switches will be placed in SmartFabric mode in the Add switches to SmartFabric
section of this chapter.
Configure management settings for new switches

A unique IP address is configured on the OOB management interface of each switch to be
added to the SmartFabric. A management route is also configured if routing is used on the
OOB management network.
Note: Configure a unique OOB management IP address for each switch. The management route
should not be 0.0.0.0/0, as it may interfere with the data network’s default route. Use a specific
destination prefix, as shown in the example below.
Run the following command on each switch to be added to the SmartFabric:

OS10(config)# interface mgmt 1/1/1
OS10(conf-if-ma-1/1/1)# no ip address dhcp
OS10(conf-if-ma-1/1/1)# ip address 100.67.127.26/24
OS10(conf-if-ma-1/1/1)# no shutdown
OS10(conf-if-ma-1/1/1)# exit
OS10(config)# management route 100.67.0.0/16 100.67.127.254

Deployment Guide
OS10(config)# end
OS10# write memory
Note: If % Error: ZTD is in progress(configuration is locked) is

preventing entry into configuration mode, enter the command ztd cancel to proceed.
Other global settings may also be configured here, such as ip name-server and ntp
server if used by the switch. These settings are not required for the deployment example
in this guide. The hostname of the switch may be configured at the CLI or in the SFS GUI.
In this guide, the SFS GUI is used.
Add switches to SmartFabric

Caution: Do not connect any new switches in SmartFabric mode to the fabric until the preferred
master settings are validated on the existing and new leaf switches. This was covered earlier in
the Verify preferred master setting before fabric expansion section of this guide.
In this section, the two spine switches and two leaf switches are added to the
SmartFabric. These are Spine1, Spine2, Leaf2A, and Leaf2B, as shown in Figure 62.
Cable the switches as shown in Figure 62. Connection details are shown in Figure 8. Also,
make OOB management connections, as shown in Figure 10.
Caution: The following commands delete the existing switch configuration. Switch management
settings such as management IP address, management route, hostname, NTP server, and IP
name server are retained.
Spines The following commands are run on Spine1 and Spine2. This puts the switches in
SmartFabric mode as spines.

OS10(config)# smartfabric l3fabric enable role SPINE
The configuration is applied, and the switch reloads. Repeat on the second spine switch.
Leafs The following commands are run on Leaf2A and Leaf2B. This puts the switches in
SmartFabric mode as leafs and configures them as VLT peers.
Note: This example uses the two QSFP28 2x100Gb DD ports, Ethernet 1/1/49-1/1/52, for the
VLTi connections on each leaf.

OS10(config)# smartfabric l3fabric enable role LEAF vlti ethernet
1/1/49-1/1/52

Deployment Guide
The configuration is applied, and the switch reloads. Repeat on the second leaf switch.
(Optional) To verify a leaf or spine switch is in SmartFabric mode, run the following
command:

Switch-Operating-Mode : Smart Fabric Mode
Connect to the SmartFabric GUI

1. From a workstation with access to the OOB management network, use a browser
to connect to the management IP address of any switch in the fabric by going to
https://switch_mgmt_ip_address.
2. Log in as admin.
Note: After reloading a switch in SmartFabric mode, it takes about 2 minutes after the login
prompt displays at the switch CLI for SFS to come up and for the GUI to be fully functional. If you
connect to a switch in the fabric that is not the master, a link to the master is provided in the GUI.
The IPv4 address of the master may also be determined by running show smartfabric
cluster from the CLI of any switch in the SmartFabric.
Once logged into the master, the SFS GUI Home page shows the leaf-spine topology and
configured uplinks. For the example used in this guide, it appears as shown in Figure 63.
Figure 63. SFS GUI Home page
Note: Since hostnames have not been configured on the four additional switches, each appears
with its default hostname, OS10. Hostnames for the additional switches are configured in the next
section.

Deployment Guide
Configure additional rack and switch names
On the SFS GUI Home page, click 1. Update Default Fabric, Switch Names and
Descriptions. This opens the Set Fabric and Switch Name window.
1. On the Network Fabric page, update the Name (optional) and Description
(optional) of the fabric and click NEXT.
Note: The Network Fabric ID is automatically set to 100 and cannot be changed. All directly
connected switches in SmartFabric mode join this fabric.
2. On the Racks page, the second rack appears. Update the Name (recommended)
and Description (optional) of the second rack, as shown in Figure 64.
Figure 64. The second rack renamed Rack 2
3. Click NEXT.
4. On the Switches page, the additional switches appear along with their service
tags, roles, and models. Update the Names (recommended) and Descriptions
(optional) of the newly added switches. The four additional switches with updated
names are outlined in red in Figure 65.
Figure 65. Switch name configuration page
Configure leaf switch addresses for L3 uplinks

Note: If using L2 uplinks to the external network, skip this section and go to Add a VxRail node to
the cluster.

Deployment Guide
Traffic on the External Management network, VLAN 1811, must be able to reach the
external network. To accomplish this with L3 uplinks, an IP address on virtual network
1811 is assigned to each leaf switch in the SmartFabric.
IP addresses are configured for the new leafs added to the SmartFabric, Leaf2A and
Leaf2B. The examples used in this guide are shown in Table 11.
Note: Existing leaf IP addresses and the gateway IP address were configured during VxRail
cluster deployment in the Additional configuration steps for L3 uplinks section of this guide.
Table 11. Leaf switch External Management network IP addresses and anycast gateway
Item IP address/prefix Status
Leaf1A IP address 172.18.11.253/24 Previously configured
Leaf1B IP address 172.18.11.252/24 Previously configured
Leaf2A IP address 172.18.11.251/24 To be configured
Leaf2B IP address 172.18.11.250/24 To be configured
Gateway IP address 172.18.11.254/24 Previously configured
This is done as follows:
1. At a workstation, launch the SFS GUI.

2. On the SFS GUI Home page, select 3. Update Network Configuration. The
window opens, as shown in Figure 66.
3. Next to Networks, select the External Management network, Management

Network 1811, from the drop-down list.
Note: When Management Network 1811, is selected, additional fields appear, as shown in
Figure 67.
a. Next to Interface IP Addresses, two IP addresses configured earlier are

listed for the existing leaf switches, Leaf1A and Leaf1B. Use the blue
button to add an IP address for each leaf switch added to the SmartFabric.
The additional addresses are for Leaf2A and Leaf2B per Table 11 in this
example.

Deployment Guide
b. Leave the Prefix Length and Gateway IP Address at the settings
previously configured.
When complete, the Update Network Configuration window appears, as shown in
Figure 67.
4. Click OK to apply the settings.
Add a VxRail node to the cluster

For this example, VxRail node 4 is connected to the fabric. It is in Rack 2 and connected
to Leaf2A and Leaf2B, as shown in Figure 62.
Ensure forward and reverse lookup records have been added to the DNS server for each
new node to be added to the cluster. In this example, the node FQDN is vxrail04.dell.lab,
and its IP address is 172.18.11.104. Figure 68 shows validation commands run from a
Microsoft Windows-based system with connectivity to the DNS server.
Figure 68. Forward and reverse lookup commands

Deployment Guide
Use the vSphere Client to add the VxRail node as follows:
1. Connect to the vCenter in a browser and launch the vSphere Client.

2. In the vSphere Client, right-click the cluster and select VxRail > Add VxRail
Hosts.
3. After a brief scan, the fourth VxRail node is discovered, outlined in red in Figure
69.
Figure 69. VxRail node discovered
4. Click ADD HOST.

5. In the Discovered Hosts window, check the box next to the service tag, as
shown in Figure 70.
Figure 70. Discovered Hosts window
6. Click NEXT.

Deployment Guide
7. In the User Credentials window, enter the vCenter and switch REST_USER
credentials, as shown in Figure 71.
Figure 71. User Credentials window
8. Click NEXT.
9. In the Host IP Address window, address pools configured during cluster

installation are displayed. Addresses may be added to each pool if necessary,
however, it is not necessary in this example.
Figure 72. Host IP Address window
10. Click NEXT.

Deployment Guide
11. In the ESXi Credentials window, enter ESXi Root and Management user
credentials for the new node. Scroll down and verify the ESXi IP address and
hostname are correct under Host Configurations.
Figure 73. ESXi Credentials window
12. Click NEXT.

13. In the Maintenance Mode window, check the box if the host should be put in
maintenance mode. This is not done in this example.
Figure 74. Maintenance Mode window
14. Click NEXT.

Deployment Guide
15. Review the settings in the Validate window. If no changes are needed, click
VALIDATE. Validation may take 3 to 5 minutes.
Figure 75. Validate window
16. A Success message displays when validation has passed, as shown in Figure
76.
Figure 76. Validation successful
17. Click FINISH.

The Host expansion is in progress. Health monitoring is currently disabled during
this task message displays and a progress bar displays under Status. Both are outlined
in red in Figure 77.
Note: The host expansion process may take 10 to 15 minutes to complete.

Deployment Guide
Figure 77. Host expansion in progress
18. When complete, the fourth VxRail node appears in the cluster, as shown in Figure
78.
Figure 78. Fourth VxRail node added to the cluster
CAUTION: Review any warnings that may appear in the vSphere Client.
Optionally, verify the interface connected to the new VxRail node has been automatically
added to the VxRail networks on the leaf switches. To verify the connection, run the show
virtual network command. In this example, the new VxRail node is connected to
interface 1/1/1 on Leaf2A and Leaf2B.
Note: The command output shown in the following command is for Leaf2A. The output for Leaf2B
is the same.
The output below confirms the interface connected to the new VxRail node, ethernet1/1/1,
has been automatically placed in all VxRail virtual networks/VLANs.

Unicast-Drop
VLTi-VLAN: 1811
Members:

Deployment Guide
VLAN 1811: port-channel1000, ethernet1/1/1
Remote-VTEPs (flood-list): 172.30.0.0(CP)

VLTi-VLAN: 1812
Members:

VLTi-VLAN: 1813
Members:

VLTi-VLAN: 1814
Members:

VLTi-VLAN: 1815
Members:

VLTi-VLAN: 3939
Members:

VLTi-VLAN: 4091
Members:
Deployment Guide
Untagged: ethernet1/1/1
Deployment Guide
Chapter 8 Deploy and Configure OMNI
Deploy OMNI VM .................................................................................................103
Configure OMNI...................................................................................................107
OMNI plugin .........................................................................................................114
Connect to OMNI Appliance Management user interface ..............................114
Connect to OMNI in vCenter ..............................................................................116
Deployment Guide
Deploy OMNI VM
The OMNI VM is available for download from the Dell EMC OpenManage Network
Integration for VMware vCenter website. Download OMNI-version#.zip and extract the
OMNI-version#.ova file to a location accessible from the vSphere client.
Note: VxRail 7.0 supports OMNI 1.3.24 or a later version specified in the SmartFabric OS10
Solutions (HCI, Storage, MX) Support Matrix.
1. To deploy the OMNI VM, launch the vSphere Client and go to Hosts and
Clusters.
2. Right-click the VxRail cluster and select Deploy OVF Template.
Figure 79. Deploy OVF template
3. On the Select an OVF template page, point to the location of the OMNI .ova file
and click NEXT.
Deployment Guide
4. Enter a Virtual machine name and select a location for the OMNI VM. The
default location is used in this example, as shown in Figure 80.
Figure 80. OMNI VM name and folder
5. Click NEXT.
6. On the Select a compute resource page, ensure the VxRail cluster is selected,
and Compatibility checks succeeded displays at the bottom of the page.
Figure 81. VxRail cluster selected
7. Click NEXT.
Deployment Guide
8. The Review details page displays, as shown in Figure 82.
Figure 82. Review details page
9. Click NEXT.
10. On the License agreements page, review the terms provided. If you agree,
check the I accept all license agreements box and click NEXT.
11. On the Select storage page, select the vSAN datastore. Ensure Compatibility
checks succeeded displays at the bottom of the page.
Figure 83. VxRail VSAN datastore selected
12. Click NEXT.
Deployment Guide
13. On the Select networks page, change the two destination networks to match the
two source networks, as shown in Figure 84.
Figure 84. Select networks
14. Click NEXT.

15. On the Ready to complete page, review the settings.
Figure 85. Review settings
16. Click FINISH to deploy the OMNI VM.
Note: OMNI VM installation may take 2 to 3 minutes.
When complete, the OMNI VM appears under the VxRail cluster, as shown in Figure 86.
Deployment Guide
Figure 86. OMNI VM deployed
Configure OMNI
Note: Before proceeding, determine an IP address on the External Management VLAN and a
hostname for the OMNI VM. Add this information to the DNS server. In this example, the
hostname is omni.dell.lab, and the address is 172.18.11.56/24.
1. In the vSphere Client, power on the OMNI VM and launch the web console for the
VM.
2. Log in with the default OMNI VM username, admin, and password, admin. The
first time you log in, follow the prompts to change the admin password.
3. The OMNI Menu displays, as shown in Figure 87.
Figure 87. OMNI menu
Deployment Guide
4. Select 0. Full setup. The Network Manager terminal user interface (TUI)
displays.
Figure 88. Network Manager TUI
Note: In the TUI, use the Tab and Arrow keys to navigate and the Enter key to select.
5. Select Edit a connection > Wired connection 1.
Figure 89. Wired connection 1 selected
6. In the Edit Connection window, make the following settings:

Change the profile name to external management.
Note: The Device field is automatically populated with a MAC address and (ens160).
Set the IPv4 Configuration to Manual.

Next to IPv4 Configuration, select Show to expand the additional fields.
Enter the OMNI VM IP Address/prefix, Gateway, DNS server, and Search
domain. The values used for this example are shown in Figure 90.
Deployment Guide
Note: The example in Figure 90 is with L2 uplinks, so the DNS server has an IP address on the
same network as the OMNI VM, 172.18.11.50 in this example. If L3 uplinks are used, the DNS
server will be on a different network. For the L3 uplink examples in this guide, the DNS server IP
address is 172.19.11.50.
Set the IPv6 Configuration to Ignore.

Default values are used for the remaining settings. When complete, the Edit Connection
window appears as shown in Figure 90.
Figure 90. External management connection settings
7. Select OK. The Ethernet connections list displays.
Deployment Guide
8. Select Wired connection 2.
Figure 91. Wired connection 2 selected
9. In the Edit Connection window, make the following settings:

Change the profile name to internal management.
Note: The Device field is automatically populated with a MAC address and (ens192).
Set IPv4 Configuration to Disabled.

Set IPv6 Configuration to Link-Local.
Next to IPv6 Configuration, select Show to expand the IPv6 configuration
fields.
Next to Routing, select Edit > Add.
Set the Destination/Prefix to fde1:53ba:e9a0:cccc::/64. Leave Next Hop
empty and Metric at 0.
The Edit Connection window appears, as shown in Figure 92.
Deployment Guide
Figure 92. Internal management connection settings
Note: Only part of the Destination/Prefix field is visible on the screen. Be sure it is set to
fde1:53ba:e9a0:cccc::/64.
10. Select OK > OK > Back to return to the Network Manager TUI menu.
11. On the Network Manager TUI menu, select Activate a connection. The
connection activation window displays, as shown in Figure 93.
Figure 93. Activate/Deactivate connections window
Note: When active, connection names have an asterisk (*) next to them.
Deployment Guide
12. Deactivate both connections as follows:
Select external management > Deactivate.
Select internal management > Deactivate.
13. Activate both connections as follows:
Select external management > Activate.
Select internal management > Activate.
14. Select Back to return to the Network Manager TUI menu.
15. On the Network Manager TUI menu, select Set system hostname, as shown in
Figure 94.
Figure 94. Set system hostname selected
16. Change the Hostname from its default setting to the new hostname,
omni.dell.lab, in this example.
Figure 95. Set hostname to omni.dell.lab
17. Click OK > OK > Quit.

18. The NTP Server IP/Hostname: prompt displays. At the prompt, specify the NTP
server, ntp.dell.lab in this example, as shown in Figure 96. If an NTP server is
not used, press Enter to skip.
19. At the Install SSL Certificates from remote server [y]? prompt, enter n.
Deployment Guide
Note: OMNI certificate installation is outside the scope of this guide. Certificates can be imported
later by selecting 5. Password/SSL configuration menu from the OMNI Menu. Follow the
instructions provided in SmartFabric Services for OpenManage Network Integration User Guide,
Release 1.3. The guide is available on the Dell EMC OpenManage Network Integration for
VMware vCenter website.
Figure 96. Specify NTP server
20. Press Enter to return to the OMNI menu.

21. On the OMNI menu, select 4. Register/Update OMNI vSphere client plugin
with vCenter.
22. Enter the requested information as prompted. The following values are used in
this example:
OMNI IP/FQDN to use for registration: omni.dell.lab
vCenter server FQDN: vcenter01.dell.lab
vCenter server username: administrator@vsphere.local
vCenter server password: password
After the vCenter registration information is entered, the screen appears as shown in
Figure 97.
Figure 97. OMNI plugin vCenter registration input and output
23. Ensure no error messages are displayed, and press Enter when prompted to
return to the OMNI menu.
24. At the OMNI menu, select 9. Logout.
This completes the OMNI VM console configuration.
Deployment Guide
OMNI plugin
Note: You do not have to log out and log back into the vSphere Client for the plugin notification to
appear.
In the vSphere Client, an OMNI plugin banner appears at the top of the screen, outlined
in red in the figure below. Click the REFRESH BROWSER button that appears in the
banner.
Figure 98. OMNI plugin notification
Note: If there are other messages present, such as a license warning, the message shown in the
figure above may be located behind the other messages. When there are multiple messages,
there are "<" and ">" icons present to the left of the banner to cycle through the messages.
Connect to OMNI Appliance Management user interface

1. Open a browser session, go to https://OMNI_IP/omni with the IP address or
hostname of the OMNI VM that is configured during the initial setup. In this
example, https://172.18.11.56/omni is used.
2. Enter the credentials for OMNI VM appliance administrator.
Figure 99. OMNI Appliance Management User Interface
Deployment Guide
3. Click Services on the left menu.
Figure 100. Services menu
Note: The Services menu displays the list of OMNI management and SmartFabric vCenter
automation services running on the OMNI appliance.
4. To start the SmartFabric vCenter automation service, select the service as shown
and click START.
Figure 101. Start fabric automation services
Deployment Guide
5. The system displays a start service success message.
Figure 102. Start service success message
Connect to OMNI in vCenter

1. In the vSphere Client, select Menu > OpenManage Network Integration, as
shown in Figure 103.
Figure 103. Access OMNI
Deployment Guide
2. In the left pane, click Home. In the right pane, click the radio button next to the
service instance as shown.
Figure 104. OMNI Home page
3. Click EDIT.
4. In the Edit a Service Instance window, enter the REST_USER Password that is
configured on the SmartFabric switches.
Note: The REST_USER password was set during VxRail deployment, as shown in Figure 47.
Figure 105. Enter REST_USER password
5. Click SUBMIT.
A Success message displays in green, as shown in Figure 106.
Figure 106. REST_USER authentication success
The OMNI vCenter plugin configuration is complete.
Note: After OMNI is deployed, use OMNI for switch configuration instead of the SFS GUI. The
SFS GUI is intended for initial deployment only.
Deployment Guide
Chapter 9 Use OMNI for Post-deployment
Configuration
Connect a non-VxRail server to a leaf pair ......................................................119
Add a new port group to the SmartFabric .......................................................125
Move L3 uplinks to spine switches ..................................................................127
Deployment Guide
This chapter demonstrates different post-deployment switch configuration tasks that are
done with OMNI as needed.
Connect a non-VxRail server to a leaf pair

Non-VxRail devices are onboarded to the SmartFabric using the OMNI plugin in vCenter.
In this example, a standard PowerEdge Server in Rack 2 is connected to the fabric, as
shown in Figure 107.
SmartFabric

network

Existing DNS & 3 2 1 1 2 3 5 1 1 5
VxRail node 2 PowerEdge server
VxRail node 3
Rack 1 Rack 2
Layer 3 connections
Layer 2 connections
VLTi connections
Layer 2 or Layer 3 connections
Figure 107. Non-VxRail device connected to SmartFabric
Add a server to The PowerEdge server to be added is connected to Leaf2A and Leaf2B in this example,
the fabric as shown in Figure 107.
Breakout switch ports using OMNI

If the server uses a 1 GbE or 10 GbE NIC, and it is connected to 25 GbE ports on S5200
series switches, the corresponding leaf switch ports used must be changed from 25 GbE
to 10 GbE before proceeding. To breakout switch ports on S5248F-Leaf2A to 10 GbE, use
the following steps.
1. In the vSphere Client, select Menu > OpenManage Network Integration.

2. In the left pane, click Service Instance. In the right menu, select Switches.
Deployment Guide
Figure 108. Server Interfaces
3. Choose the first leaf switch, S5248F-Leaf2A. Click the drop-down icon to expand
the contents. Click the drop-down icon next to Breakout Ports & Jump Port.
4. Choose the port group to break out. In this example, the PowerEdge server is
connected to port 5, therefore, port-group1/1/2 is selected.
Note: See Table 9 to map the S5200 ports to port groups.
5. Click BREAKOUT PORT.
6. In the Mode drop down box, select the breakout mode. In this example,
4x10GEFixedFormFactor is selected.
7. Click SUBMIT.
Deployment Guide
Repeat the above steps to breakout port group 1/1/2 on switch S5248F-Leaf2B. After the
wanted switch port is broken out to 10 GbE, you can continue to add the server to the
fabric.
In this example, the server's two connected ports are configured as an LACP NIC team in
the operating system running on the PowerEdge server. An IP address is assigned to the
NIC team on VM_Network_A, 172.18.14.0/24.
Note: NIC teaming and IP address configuration procedures on the PowerEdge server depend on
the operating system used and are not covered in this guide.
To onboard the server to the SmartFabric using OMNI, do the following:
1. In the vSphere Client, select Menu > OpenManage Network Integration.

2. In the left pane, select the Service Instance.
3. In the right pane, select Server Interface. The automatically configured VxRail
node interfaces are listed with their MAC addresses used as the Server Interface
IDs, as shown in Figure 111.
Deployment Guide
4. Click CREATE. The Create Server Interface Profile window appears, as shown
in Figure 112.
5. In the Create Server Interface Profile window that opens, do the following
6. Enter a name for the Server Interface ID, for example, LACP-NIC-Team-1.
Next to Server Profile, select New Server Profile.
Enter a Server Profile ID, for example, PowerEdge-Server-1.
Next to Server Profile Bonding Type, select AutoDetect from the drop-
down list.
Next to Untagged Network, select a network for this connection from the
drop-down list. In this example, it is VM_Network_A 1814.
Next to Static Onboarding Option, select Yes.
Next to NIC Bonding, select Enable.
Select the Leaf Node and the Interface that is connected to the server. In
this example, Leaf2A and ethernet 1/1/5:1 are selected.
Leave the Routing Protocol option set to None.
Deployment Guide
Figure 112. Create a server interface profile
Click CREATE.
Repeat steps 4 through 6 above for the second connection to the PowerEdge server using
the values in Table 12.
Table 12. Values for second PowerEdge Server connection
Field Value
Server interface ID LACP-NIC-Team-2
Server Profile Existing Server Profile
Server Profile ID PowerEdge-Server-1
Untagged Network VM_Network_A 1814
Static Onboarding option Yes
NIC Bonding Enable
Leaf Node Leaf2B
Interface ethernet 1/1/5:1
Routing Protocol None
Deployment Guide
When complete, the Server Interface Profile page appears, as shown in Figure 113.
Figure 113. Server interface profiles created
Validation Validation may be done at a leaf switch CLI with the following commands:
show port-channel summary

The show port-channel summary command output shows the newly created port
channel, port channel 1, on interface 1/1/5:1. (P) indicates the member is up and active.
Note: In the output below, port channels 96 through 97 are SFS-configured uplinks to the spines,
and port channel 1000 is the VLTi.

--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
1 port-channel1 (U) Eth DYNAMIC 1/1/5:1(P)
96 port-channel96 (U) Eth STATIC 1/1/55(P)
1/1/52(P)
show virtual-network
The show virtual-network output shows that port channel 1, containing the
interface to the PowerEdge server, has been added as an untagged member of virtual
network 1814.
Note: Other virtual networks have been removed from the output for brevity.
Deployment Guide
Unicast-Drop
VLTi-VLAN: 1814
Members:
Untagged: port-channel1
Ping host from onboarded server

The configuration may also be verified by pinging a host on the same virtual network as
the newly onboarded server. In this case, the target host is a VM running on a VxRail
node, is on VM_Network_A, VLAN 1814, and is at 172.18.14.1.
Figure 114. Added PowerEdge server pings a VM
Add a new port group to the SmartFabric

With the OMNI plugin installed, port groups (VLANs) created in the vSphere Client are
automatically propagated to the SmartFabric switches and associated with the VxRail
node interfaces.
In this example, a port group named Application is created in the vSphere Client as
follows:
1. In the vSphere Client, select Menu > Networking.

2. In the left pane, right-click the vDS named VMware HCIA Distributed Switch
and select Distributed Port Group > New Distributed Port Group.
3. In the New Distributed Port Group window, set the Name to Application and
click NEXT.
4. Set VLAN type to VLAN, the VLAN ID to 30, and click NEXT.
5. Click FINISH.
The VMware HCIA Distributed Switch > Configure > Settings > Topology screen
appears as shown in Figure 115.
Deployment Guide
Figure 115. Application port group created on VLAN 30
The port group created in the vSphere client is propagated to the SFS switches as a
virtual network, and the VxRail node interfaces are automatically added to the virtual
network.
Note: Interfaces for statically onboarded devices, such as the non-VxRail PowerEdge server
added in the previous section, are not automatically added to new virtual networks.
Validation This is validated with the show virtual-network command on the leaf switches.
Note: In the two command output examples that follow, other virtual networks have been removed
from the output for brevity. See Figure 107 for the VxRail node locations in the topology.
The output below from Leaf1A shows that virtual network 30 was created, and the three
VxRail node interfaces, ethernet1/1/1-1/1/3, are members of the virtual network. The
output from Leaf1B, not shown, is the same.

Unicast-Drop
Virtual Network: 30
VLTi-VLAN: 30
Members:
VLAN 30: port-channel1, port-channel1000, ethernet1/1/1, ethernet1/1/2,
ethernet1/1/3
Note: The output above is from a topology using an L2 uplink from Leaf1A and Leaf1B to the
external network. Port-channel1 is the L2 uplink in this example. Port-channel 1000 is always the
VLTi.
The output below from Leaf2A shows that virtual network 30 was created, and the single
VxRail node interface, ethernet1/1/1, is a member of the virtual network. The output from
Leaf2B, not shown, is the same.
Deployment Guide
Unicast-Drop
Virtual Network: 30
VLTi-VLAN: 30
Members:
Move L3 uplinks to spine switches

Note: L2 uplinks are not applicable to this section. Only L3 uplinks are supported from the spine
switches. L3 uplink configuration from leaf switches is covered in Configure L3 routed uplinks to
the external network.
L3 uplinks to the external network can be moved from the leaf layer to the spine layer if
desired. This section covers the migration process.
Figure 116 shows the existing L3 uplinks from the leafs to the external switches in purple.
The new connections from the spines to the external network are shown in yellow. In this
example, 40 GbE connections are used from the external switches to the SmartFabric
switches.
Note: In SmartFabric mode, the auto breakout feature will automatically break out the 100 GbE
interfaces on the Spine switches to 40 GbE if connected to 40 GbE ports on the external switches
and Dell EMC supported cables and optics are used.
Deployment Guide
VLTi
21 22 27 28 21 22 27 28
27 28 27 28
Spine1 Spine2
Existing DNS & NTP

services 53 54 53 54 SmartFabric
Leaf1A Leaf1B
VLTi
L3 Leaf existing
connections
L3 Spine new
connections
SFS Connections
VLTi connections
Existing connections
Figure 116. Existing and new L3 uplink connections to external switches
BGP ASNs and Figure 117 shows the autonomous system numbers (ASNs) and router IDs used for the
router IDs external switches. The external switches share a common ASN. SmartFabric leafs share
a common ASN, and SmartFabric spines share a common ASN as shown.
Deployment Guide
ASN 65101
VLTi
10.0.2.1 10.0.2.2
ASN 65012
Spine1 Spine2
Existing DNS & NTP

services
Leaf1A Leaf1B
VLTi
ASN 65011
Existing L3 uplinks
New L3 uplinks
SFS Connections
VLTi connections
In this example, ASN 65101 is used on both external switches. By default, all SmartFabric
leaf switches use 65011, and all SmartFabric Spine switches use 65012.
The IP addresses shown in Figure 117 are loopback addresses used as BGP router IDs
on the external network switches. Router IDs on the SFS switches are automatically
configured from the SFS default private subnet address block, 172.16.0.0/16.
Point-to-point IP The existing point-to-point links used in this deployment are labeled A-D, and the new
networks point-to-point links are labeled E-H in Figure 118.
Deployment Guide
VLTi
E F G H
A B C D Spine1 Spine2
Existing DNS & NTP

services SmartFabric
Leaf1A Leaf1B
VLTi
L3 Leaf existing
connections
L3 Spine new
connections
SFS Connections
VLTi connections
Figure 118. Point-to-point connections
Each L3 uplink is a separate, point-to-point IP network. Table 13 details the links labeled
in Figure 118. The IP addresses in the table are used in the switch configuration
examples.
Table 13. L3 uplink IP addresses

A External-A 192.168.1.0 Leaf1A 192.168.1.1 192.168.1.0/31
B External-A 192.168.1.2 Leaf1B 192.168.1.3 192.168.1.2/31
C External-B 192.168.2.0 Leaf1A 192.168.2.1 192.168.2.0/31
D External-B 192.168.2.2 Leaf1B 192.168.2.3 192.168.2.2/31
E External-A 192.168.1.4 Spine1 192.168.1.5 192.168.1.4/31
F External-A 192.168.1.6 Spine2 192.168.1.7 192.168.1.6/31
G External-B 192.168.2.4 Spine1 192.168.2.5 192.168.2.4/31
H External-B 192.168.2.6 Spine2 192.168.2.7 192.168.2.6/31
Configure L3 Table 14 shows the values entered in the OMNI plug-in in vCenter for this example. The
uplinks in OMNI following steps are run once for each uplink using the values from the table.
Deployment Guide
Table 14. Uplink configuration details
Link label E F G H
L3 Type L3 Routed L3 Routed L3 Routed L3 Routed
Uplink Name Spine01-to- Spine02-to- Spine01-to- Spine02-to-

ExternalA ExternalA ExternalB ExternalB
Switch Group Spine Spine Spine Spine
Domain Fabric 1 Fabric 1 Fabric 1 Fabric 1
Spine Switch Spine01 Spine02 Spine01 Spine02
Network SPN01-to-ExtA SPN02-to-ExtA SPN01-to-ExtB SPN02-to-ExtB

Name
IPv4 Address 192.168.1.5 192.168.1.7 192.168.2.5 192.168.2.7
Protocol eBGP eBGP eBGP eBGP
Router ID 10.0.1.1 10.0.1.2 10.0.1.1 10.0.1.2
Policy ID/ eBGP-SPN01-to- eBGP-SPN02-to- eBGP-SPN01-to- eBGP-SPN02-to-

Routing ExtA ExtA ExtB ExtB
Policy Name
Peer IPv4 192.168.1.4 192.168.1.6 192.168.2.4 192.168.2.6

Address
Remote ASN 65101 65101 65101 65101
In the OMNI plugin in vCenter, select the Service Instance in the left pane. In the right
pane, click the Uplink tab. The existing uplinks from the leaf switches are displayed, as
shown.
Deployment Guide
Figure 119. OMNI Uplink tab
To create uplinks from the spine switches, do the following:
1. Click CREATE.
Set the Uplink Port Type to L3.
Set the L3 Type to L3 Routed Interface.
Enter a unique Name and, optionally, a Description.
Figure 120. Uplink Details page
3. Click NEXT.
Set the Switch Group to Spine.
The Domain is set to the Fabric ID by default.
Deployment Guide
Select the Node, Z9264F-Spine1.
Select the appropriate interface, ethernet 1/1/27, in this example.
Figure 121. Port Configuration page
5. Click NEXT.
Set the Network Profile Name, SPN01-to-ExtA.
Set the Prefix Length, 31, and IP Address, 192.168.1.5.
Select eBGP as the Routing Protocol.
Set the Policy ID and Routing Policy Name, eBGP-SPN01-to-ExtA.
Note: Policy ID and Routing Policy Name are text fields and do not have to match. However, the
fields must be unique for each link, as shown in Table 14.
Set the Remote IP Address, 192.168.1.4.

Set the Remote ASN, 65101.
Figure 122. Network Configuration page
Deployment Guide
Figure 123. Network Configuration page continued
7. Click FINISH.
14.
Once all four uplinks are configured, they are visible on the Uplink tab of the OMNI plugin.
The new uplinks are outlined in red in Figure 124.
Figure 124. Uplinks from spines created
Deployment Guide
Configure The external switches start at their configuration shown in Chapter 5 - Configure external
external switches for L3 connections. This section covers the configuration changes for both
switches external switches.
Configure the interfaces for L3 connections to the SmartFabric spine switches. Interfaces
1/1/27 and 1/1/28 are used in this example.
Enter configuration mode with the configure terminal command.
For each interface, add a description and take the port out of L2 mode with the no
switchport command. Assign an IP address to each interface per Table 13.
Optionally, allow the forwarding of jumbo frames with the MTU 9216 command. As a best
practice, flow control settings remain at their factory defaults as shown.

description Spine1 description Spine1
mtu 9216 mtu 9216

description Spine2 description Spine2
mtu 9216 mtu 9216
Configure BGP
Enter BGP configuration mode with the router bgp 65101 command. The external
switches share the same ASN. For the router ID, use the address that was set for
interface loopback0 in Chapter 5 - Configure external switches for L3 connections.
Configure the neighbor addresses per Table 13.
Configure the neighbor ASN. By default, all SmartFabric spine switches use 65012.
Deployment Guide
router bgp 65101 router bgp 65101
router-id 10.0.2.1 router-id 10.0.2.2


end end
Validation Once the uplink interfaces are configured on the external switches and in the SFS GUI,
connectivity can be verified using the switch CLI.
Show command output on External-A

When BGP is up, uptime is shown in the Up/Down column. The neighbors for External-A
shown in the output below are Leaf1A, Leaf1B, Spine1, Spine2, and External-B.
Note: Leaf1A and Leaf1B can be disconnected from the external switches after validation is
complete.
External-A# show ip bgp summary

192.168.1.1 65011 3407 3404 07:10:58 8
192.168.1.3 65011 3409 3389 07:10:10 8
192.168.1.5 65012 76 66 00:13:36 6
192.168.1.7 65012 59 45 00:09:28 6
192.168.3.21 65101 5013 5019 3d:00:20:57 6
Run the show ip interface brief command to verify connected interfaces are up
In the output below, interfaces 1/1/27 and 1/1/28 are connected to the spine switches.

================================================================================
Deployment Guide
Show command output on Spine1

Note: The command output shown in the following commands is for Spine1. The output for
Spine2 is similar.
When BGP is up, uptime is shown in the Up/Down column. The neighbors for Spine1
shown in the output below are the four leaf switches (with automatically configured IP
addresses in the 172.16.0.0/16 address range), and External-A and External-B.
Z9264F-Spine1# show ip bgp summary

172.16.0.2 65011 103 110 01:19:12 23
172.16.0.4 65011 105 111 01:18:43 23
172.16.0.6 65011 126 112 01:18:24 8
172.16.0.16 65011 123 112 01:18:20 8
192.168.1.4 65101 22 29 00:15:59 15
192.168.2.4 65101 16 17 00:10:51 15
In the output below, interfaces 1/1/27 and 1/1/28 are connected to the external switches.
Z9264F-Spine1# show ip interface brief

================================================================================
Deployment Guide
Remove uplinks After the new uplinks from the spines are validated, the original uplinks from the leafs may
from leafs be removed as follows:
1. In the OMNI plugin in vCenter, select the Service Instance in the left pane. In the
right pane, click the Uplink tab. The existing uplinks are displayed.
2. Remove the first uplink from a leaf switch to an external switch by selecting the
link followed by DELETE. Repeat for the remaining three leaf switch uplinks.
3. Remove the physical connections from the leaf switches to the external switches.
Deployment Guide
Appendix A Validated Components
This appendix presents the following topics:
General .................................................................................................................140
Dell EMC PowerSwitch systems .......................................................................140
VxRail P570 nodes ..............................................................................................140
VxRail appliance software .................................................................................141
OMNI software .....................................................................................................141
Deployment Guide
General
The following tables include hardware, software, and firmware that was used to configure
and validate the examples in this guide.
Note: For more information about supported components and versions, see the Dell EMC VxRail
Support Matrix (account required).
Note: Switches validated for the Cisco Nexus examples are in Appendix C.
Dell EMC PowerSwitch systems

Table 15. Switches and operating systems
Operating system
Qty Item
version
2 Dell EMC PowerSwitch Z9264F-ON spine switches 10.5.0.7P3.751
4 Dell EMC PowerSwitch S5248F-ON leaf switches 10.5.0.7P3.751
2 Dell EMC PowerSwitch S3048-ON OOB management switches 10.5.0.7P3.751
2 Dell EMC PowerSwitch Z9100-ON series external switches 10.5.0.7P3.751
VxRail P570 nodes

A cluster of four VxRail P570 nodes was used to validate the examples in this guide. The
nodes were each configured per Table 16.
Table 16. VxRail P570 node components
Qty per node Item Firmware version
2 Intel Xeon Gold 6136 CPU @ 3.00 GHz, 12 cores
12 16 GB DDR4 DIMMs (192 GB total)
2 240 GB SAS SSD
8 1.2 TB SAS HDD
1 Dell HBA330 Storage Controller 16.17.00.05
1 Boot Optimized Storage Solution (BOSS) Controller 2.5.13.3022
1 Broadcom 57414 rNDC – 2x 25 GbE SFP28 ports 21.60.22.11
- BIOS 2.6.4
- CPLD 1.1.3
- iDRAC with Lifecycle Controller 4.10.10.10
- Backplane Expander 2.46
Deployment Guide
VxRail appliance software
The examples in this guide were validated using VxRail 7.0 appliance software. The
software consists of the component versions shown in Table 17.
Table 17. VxRail appliance software component versions
Item Version
VxRail Manager 7.0.000-26451670
VMware ESXi 7.0.0 Build 15843807
VMware vCenter Server Appliance 7.0.0.10100
VMware vSAN 7.0
VMware vRealize Log Insight 4.8.0 Build13036238
OMNI software
OMNI software used in this guide is shown in Table 18.
Table 18. OMNI software
Item Version
OMNI 1.3.24
Deployment Guide
Appendix B CLI Commands
Switch CLI validation commands .....................................................................143
Return to Full Switch mode ...............................................................................150
Deployment Guide
Switch CLI validation commands
This section provides a list of the most common commands and their output for the
examples used in this guide.
General show version

commands Leaf and spine switches must be running a supported version of SmartFabric OS10. Run
the show version command to check the operating system version. SmartFabric OS10
is available on Dell Digital Locker (account required).
OS10# show version

Dell EMC Networking OS10 Enterprise
Copyright (c) 1999-2020 by Dell Inc. All Rights Reserved.
OS Version: 10.5.0.7P3
Build Version: 10.5.0.7P3.751
Note: See the SmartFabric OS10 release notes for upgrade instructions.
show license status

Run the command show license status to verify license installation. The License
Type: field should indicate PERPETUAL. If an evaluation license is installed, licenses
purchased from Dell are available for download on Dell Digital Locker. Installation
instructions are provided in the Dell EMC SmartFabric OS10 User Guide on Dell EMC
Networking OS10 Info Hub.
OS10# show license status
System Information
---------------------------------------------------------
Vendor Name : Dell EMC
Product Name : S5248F-ON
Hardware Version : A01
Platform Name : x86_64-dellemc_s5248f_c3538-r0
PPID : CN046MRJCES0089K0015
Service Tag : 690ZZP2
Product Base :
Product Serial Number:
Product Part Number :
License Details
----------------
Software : OS10-Enterprise
Version : 10.5.0.7P3
License Type : PERPETUAL
License Duration: Unlimited
License Status : Active
License location: /mnt/license/690ZZP2.lic
---------------------------------------------------------
Note: If SmartFabric OS10 was factory installed, a perpetual license is already on the switch.
Deployment Guide
show interface status
The show interface status | grep up command is used to verify required
interfaces are up, and links are established at their appropriate speeds. In this example,
ports 1/1/1-1/1/3 are VxRail nodes, port 1/1/9:1 is a jump host, ports 1/1/49-1/1/52 are the
VLTi, ports 1/1/53-1/1/54 are uplinks to the external network, and ports 1/1/55-1/1/56 are
uplinks to the spines.
S5248F-Leaf1A# show interface status | grep up

Port Description Status Speed Duplex Mode Vlan Tagged-Vlans
Eth 1/1/1 up 25G full T -
Eth 1/1/9:1 up 10G full T -
Eth 1/1/49 up 100G full -
show port-channel summary

The show port-channel summary command is used to view port channel numbers,
interfaces used, and status. Port channel 1 (number may vary) is the L2 uplink. Port
channels 96 through 97 (numbers may vary) are automatically created spine uplinks. The
VLTi is automatically configured as a static LAG using port channel 1000. Ports 1/1/53
and 1/1/54 are port channel members, and (P) indicates each is up and active.

--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
1/1/52(P)
show virtual network

The show virtual network command is used to view virtual networks, VLANs, and
interfaces assigned to each VLAN. Port channel 1 (number may vary) is the L2 uplink if
configured. Port channel 1000 is the VLTi. Interfaces 1/1/1-1/1/3 are connected to VxRail
nodes. Interface 1/1/9:1, shown under VLAN 1811, is the jump host port.
Note: The jump host port was automatically moved from VLAN 4091 to VLAN 1811 during VxRail
deployment.
Deployment Guide
Unicast-Drop
VLTi-VLAN: 1811
Members:
ethernet1/1/3

VLTi-VLAN: 1812
Members:
ethernet1/1/3

VLTi-VLAN: 1813
Members:
ethernet1/1/3

VLTi-VLAN: 1814
Members:
ethernet1/1/3

VLTi-VLAN: 1815
Members:
ethernet1/1/3
Deployment Guide
VLTi-VLAN: 3939
Members:

VLTi-VLAN: 4091
Members:
show lldp neighbors

The show lldp neighbors command is useful for identifying connected equipment.
Interfaces 1/1/1-3 are connected to the Broadcom adapter in each VxRail node, 1/1/53-54
are connected to the external network, and 1/1/55-56 are connected to the spines.
Interface 1/1/9:1 is connected to the jump host, and mgmt1/1/1 is connected to the OOB
management switch.
S5248F-Leaf1A# show lldp neighbors

Loc PortID Rem Host Name Rem Port Id Rem Chassis Id
----------------------------------------------------------------------------------
ethernet1/1/1 Broadcom Adv. Dua... 00:0a:f7:b6:15:d0 00:0a:f7:b6:15:d0
ethernet1/1/1 vxrail03.dell.lab 00:0a:f7:b6:15:d0 vmnic0
ethernet1/1/2 Broadcom Adv. Dua... 00:0a:f7:b6:1a:80 00:0a:f7:b6:1a:80
ethernet1/1/2 vxrail01.dell.lab 00:0a:f7:b6:1a:80 vmnic0
ethernet1/1/3 Broadcom Adv. Dua... 00:0a:f7:b6:1c:10 00:0a:f7:b6:1c:10
ethernet1/1/3 vxrail02.dell.lab 00:0a:f7:b6:1c:10 vmnic0
ethernet1/1/49 S5248F-Leaf1B ethernet1/1/49 3c:2c:30:10:41:00
ethernet1/1/53 External-A ethernet1/1/21 4c:d9:8f:d8:32:8e
ethernet1/1/54 External-B ethernet1/1/21 4c:d9:8f:d8:38:0e
ethernet1/1/55 Z9264F-Spine1 ethernet1/1/1 20:04:0f:05:c0:97
ethernet1/1/56 Z9264F-Spine2 ethernet1/1/1 50:9a:4c:d5:50:70
ethernet1/1/9:1 Not Advertised 00:10:18:d6:b3:e0 00:10:18:d6:b3:e0
mgmt1/1/1 R127-OOBmgmt Gi1/0/23 f8:b1:56:19:cb:87
Note: If an entry is not shown for the jump host, port 1/1/9:1 in the above example, the jump host
may be connected to the other leaf switch, or the NIC on the jump host does not have LLDP
enabled. This is not required and may be ignored.
Deployment Guide
show ip interface brief
1/1/52 are the VLTi, and 1/1/53-1/1/54 are L3 uplinks to the external switches. VLAN
4090, Loopback 1, and Loopback 2 are automatically configured by SFS. VLAN 4090 is
used for iBGP, Loopback 1 is the router ID, and Loopback 2 is the VTEP IP address.

show ip bgp summary

shown in the output below are Leaf1B, External-A, and External-B.

172.16.0.0 65011 13 16 00:06:59 8
192.168.1.0 65101 12 14 00:07:30 8
192.168.2.0 65101 8 9 00:04:14 8
show ip route
With L3 uplinks, the show ip route command is used to ensure the leaf switches have
routes to the external network, 172.19.11.0/24 in this example, to reach the DNS server.
This BGP-discovered route is shown in bold in the output below.

S - static
Deployment Guide
Destination Gateway Dist/Metric Change
----------------------------------------------------------------------------------
B EX 10.0.2.1/32 via 192.168.1.0 20/0 21:07:08
via 192.168.2.0
B EX 10.0.2.2/32 via 192.168.1.0 20/0 21:07:08
via 192.168.2.0
C 172.16.0.0/31 via 172.16.0.1 vlan4090 0/0 21:10:36
C 172.16.128.0/32 via 172.16.128.0 loopback1 0/0 21:10:39
B IN 172.16.128.1/32 via 172.16.0.0 200/0 21:09:53
B EX 172.19.11.0/24 via 192.168.1.0 20/0 21:07:08
via 192.168.2.0
C 172.18.11.0/24 via 172.18.11.250 virtual-network1811 0/0 16:02:54
C 172.30.0.0/32 via 172.30.0.0 loopback2 0/0 21:10:39
C 192.168.1.0/31 via 192.168.1.1 ethernet1/1/53 0/0 21:10:41
B IN 192.168.1.2/31 via 172.16.0.0 200/0 21:09:53
C 192.168.2.0/31 via 192.168.2.1 ethernet1/1/54 0/0 21:07:14
B IN 192.168.2.2/31 via 172.16.0.0 200/0 21:09:53
B EX 192.168.3.20/31 via 192.168.1.0 20/0 21:07:08
via 192.168.2.0
VLT commands show vlt domain_id

This command is used to validate the VLT configuration status. In SmartFabric mode, the
VLT domain ID is 255. The Role for one switch in the VLT pair is primary, and its peer
switch (not shown) is assigned the secondary role. The VLTi Link Status and VLT
Peer Status must both be up.
S5248F-Leaf1A# show vlt 255

Domain ID : 255
Unit ID : 2
Role : primary
Version : 2.3
Local System MAC address : 3c:2c:30:10:36:00
Role priority : 32768
VLT MAC address : 3c:2c:30:10:36:00
IP address : fda5:74c8:b79e:1::2
Delay-Restore timer : 90 seconds
Peer-Routing : Disabled
Peer-Routing-Timeout timer : 0 seconds
VLTi Link Status
port-channel1000 : up
VLT Peer Unit ID System MAC Address Status IP Address Version

----------------------------------------------------------------------------------
1 3c:2c:30:10:41:00 up fda5:74c8:b79e:1::1 2.3
Deployment Guide
show vlt domain_id backup-link
This command is used to verify VLT peers are communicating on the backup link over the
OOB management network. The Destination is the management IP address of the
peer. The Peer Heartbeat status must be Up.
S5248F-Leaf1A# show vlt 255 backup-link

VLT Backup Link
------------------------
Destination : 100.67.127.27
Peer Heartbeat status : Up
Heartbeat interval : 30
Heartbeat timeout : 90
Destination VRF : default
show vlt domain_id mismatch

This command highlights any potential configuration issues between VLT peers. All items
must indicate No mismatch.
S5248F-Leaf1A# show vlt 255 mismatch

VLT-MAC mismatch:
No mismatch
Peer-routing mismatch:
No mismatch
VLAN mismatch:
No mismatch
VLT VLAN mismatch:

No mismatch
VLT Virtual Network Mismatch:

Virtual Network Name Mismatch:
No mismatch
Virtual Network VLTi-VLAN Mismatch:

No mismatch
Virtual Network Mode Mismatch:

No mismatch
Virtual Network Tagged Interfaces Mismatch:

No mismatch
Virtual Network Untagged Interfaces Mismatch:

No mismatch
Virtual Network VNI Mismatch:

No mismatch
Deployment Guide
Virtual Network Remote-VTEP Mismatch:
No mismatch
Virtual Network anycast ip Mismatch:

No mismatch
Virtual Network anycast mac Mismatch:

No mismatch
EVPN Mismatch:
EVPN Mode Mismatch:
No mismatch
EVPN EVI Mismatch:

No mismatch
NVE Mismatch:
No mismatch
DHCP Snooping Mismatch:
Global Snooping Configuration Mismatch

------------------------------------------------------------
Codes: SE - Static Entry Mismatch
DT - DAI Trust Mismatch
ST - Snooping Trust Mismatch
SAV - Source-Address-Validation Mismatch
ARP - ARP Inspection Mismatch
VS - VLAN Snooping Mismatch
Interface Interface Snooping Configuration Mismatch
---------------------------------------------------------------------
Multicast routing mismatches:
Global status:
Parameter VRF Local Peer
---------------------------------------------------------------------
No mismatch
Vlan status IPv4 IPv6

VlanId Local Peer Local Peer
--------------------------------------------------------
No mismatch
Return to Full Switch mode

CAUTION: The following command deletes the existing switch configuration.
To delete the existing switch configuration and go from SmartFabric mode back to Full
Switch mode, run the following commands on each switch:
Deployment Guide
S5248F-Leaf1A# configure terminal
S5248F-Leaf1A(config)# no smartfabric l3fabric
The switch reboots into Full Switch mode. The mode can be verified with the following
command:

Switch-Operating-Mode : Full Switch Mode
Deployment Guide
Appendix C Cisco Nexus External Switch
Configuration Example
Configure external Nexus switches for L3 routed connections ...................153
Configure external Nexus switches for L2 connections ................................164
Validated Nexus switches ..................................................................................171
Deployment Guide
Configure external Nexus switches for L3 routed connections
SmartFabric uplinks may be connected to external Cisco Nexus switches. This appendix
includes a Cisco Nexus 9000 switch configuration example for L3 routed connections to
SmartFabric leaf switches.
Note: L3 routed uplinks on the SmartFabric leaf switches are configured per the Configure L3
routed uplinks with BGP in SFS section of this guide.
Connections, port numbers, and networks used for external management in this example
are shown in Figure 125. The External Management VLAN is VLAN 1911 on the external
Nexus switches, and is VLAN 1811 on the SmartFabric switches.
Peer Link
N9K-External-A N9K-External-B
1 49 50 1 49 50
vPC
SmartFabric
53 54 53 54
3 2 1 1 2 3
DNS/NTP server
VLTi
DNS & NTP server in external switch

VLAN 1911, 172.19.11.0/24 network
L3 uplink connections VxRail node 1

L2 connections
VLTi connections
VxRail node 2
VxRail node 3
Rack 1
VxRail nodes in SmartFabric VLAN 1811,
172.18.11.0/24 network
Figure 125. L3 routed uplinks from SmartFabric to external Nexus switches
In this example, an existing DNS/NTP server connects to the Nexus switches using a vPC
270B
in VLAN 1911.
Note: DNS and NTP servers do not have to connect in the manner shown if they are reachable on
381B
the network.
Point-to-point IP The L3 point-to-point links used in this example are labeled A-D in Figure 126.
271B
networks
Deployment Guide
Pee r Li nk
1 49 50 1 49 50
A B C D
vPC
SmartFabric
53 54 53 54
DNS/NTP server VLTi
Rack 1
L2 connections
VLTi connections
Figure 126. Point to Point connections
310B
Each L3 uplink is a separate, point-to-point IP network. Table 19 details the links labeled
27B
in Figure 126. The IP addresses in the table are used in the switch configuration
examples.
Table 19. L3 routed uplink IP addresses

3864B

124B 126B 127B 128B

125B 129B
A
103B N9K-External-A
104B 192.168.1.0
105B Leaf1A
106B 192.168.1.1
107B 192.168.1.0/31
108B
B
109B N9K-External-A
10B 192.168.1.2
1B Leaf1B
12B 192.168.1.3
13B 192.168.1.2/31
14B
C
15B N9K-External-B
16B 192.168.2.0
17B Leaf1A
18B 192.168.2.1
19B 192.168.2.0/31
120B
D
12B N9K-External-B
12B 192.168.2.2
123B Leaf1B
124B 192.168.2.3
125B 192.168.2.2/31
126B
Deployment Guide
BGP ASNs and Figure 127 shows the ASNs and router IDs used for the external Nexus switches and SFS
router IDs leaf switches in this example. External switches share a common ASN, and all SFS leaf
switches share a common ASN.
AS 65101
Pee r Li nk
10.0.2.1/32 10.0.2.2/32
1 49 50 1 49 50
vPC
SmartFabric
53 54 53 54
NTP services
AS 65011 VLTi
Rack 1
L2 connections
VLTi connections
31B
In this example, ASN 65101 is used on both Nexus external switches. SFS leaf switches
274B
use ASN 65011 by default for all leafs in the fabric.
Note: If L3 uplinks are connected from SFS spine switches, the spine switches use ASN 65012 by
382B
default.
The IP addresses shown on the external network switches in Figure 127 are loopback
275B
addresses used as BGP router IDs. On the SmartFabric switches, BGP router IDs are
automatically configured from the SFS default private subnet address block,
172.16.0.0/16.
Note: SFS default ASNs and IP address blocks may be changed by going to 5. Edit Default
38B
Fabric Settings in the SFS GUI.
Note: All of the Nexus switch configuration commands used to validate this topology are shown in
384B
the sections that follow. The Nexus switches were reset to their default configuration settings
using the write erase command before running the configuration commands below. This is
only an example. Modify your external switch configuration as needed for your environment.
General settings Enable the following features: interface-vlan, lacp, vrrp, vpc, bgp, lldp.
276B
Configure the hostname, OOB management IP address on VRF management, and the
VRF management route as shown.
Deployment Guide
N9K-External-A
120B N9K-External-B
12B
configure terminal
168B configure terminal
170B
feature
1687B interface-vlan feature
170B interface-vlan
feature
168B lacp feature
1702B lacp
feature
1689B vrrp feature
1703B vrrp
feature
1690B vpc feature
1704B vpc
feature
169B bgp feature
1705B bgp
feature
1692B lldp feature
1706B lldp
hostname N9K-External-A
1693B hostname N9K-External-B
170B
interface mgmt 0
1694B interface mgmt 0
1708B
ip address 100.67.127.30/24
1695B ip address 100.67.127.29/24
1709B
vrf member management

169B vrf member management
170B
no shutdown
1697B no shutdown
17B
vrf context management

1698B vrf context management
172B
ip route 100.67.0.0/16
169B ip route 100.67.0.0/16
173B
100.67.127.254 100.67.127.254
Configure the VLAN 1911 represents a preexisting management VLAN on the external network. DNS
27B
External and NTP services are located on this VLAN. Optionally, enable jumbo frames with the mtu
Management 9216 command. Assign a unique IP address to the VLAN on each switch.
VLAN
Configure VRRP to provide gateway redundancy and assign the same virtual address to
278B
both switches.
N9K-External-A
12B N9K-External-B
123B
vlan 1911
174B vlan 1911
1725B
name ExtMgmt
175B name ExtMgmt
1726B
no shutdown
176B no shutdown
172B
interface Vlan1911
17B interface Vlan1911
1728B
description ExtMgmt
178B description ExtMgmt
1729B
no shutdown
179B no shutdown
1730B
mtu 9216
1720B mtu 9216
173B
ip address 172.19.11.252/24
172B ip address 172.19.11.253/24
1732B
vrrp 11
172B vrrp 11
173B
address 172.19.11.254
1723B address 172.19.11.254
1734B
no shutdown
1724B no shutdown
1735B
Configure the Create the vPC domain. The peer-keepalive destination is the OOB management IP
279B
vPC domain and address of the vPC peer switch.

peer link
Configure a port channel to use as the vPC peer link. Put the port channel in trunk mode
280B
and allow the default and External Management VLANs, 1 and 1911 respectively.
Deployment Guide
Configure the interfaces to use in the vPC peer link. Put the interfaces in trunk mode and
281B
allow the default and External Management VLANs, 1 and 1911 respectively. Add the
interfaces to the peer link port channel.
N9K-External-A
124B N9K-External-B
125B
vpc domain 129

1736B vpc domain 129
1753B
role priority 1
173B role priority 65535
1754B
peer-keepalive destination
1738B peer-keepalive destination
175B
100.67.127.29 100.67.127.30
interface port-channel 1000

1739B interface port-channel 1000
1756B
description "Peer-Link to
1740B description "Peer-Link to
175B
External-B" External-A"
switchport
174B switchport
1758B
switchport mode trunk

1742B switchport mode trunk
1759B
switchport trunk allowed vlan

1743B switchport trunk allowed vlan
1760B
1,1911 1,1911
vpc peer-link
174B vpc peer-link
176B
no shutdown
1745B no shutdown
1762B
interface ethernet 1/51-52

1746B interface ethernet 1/51-52
1763B
description "Link to External-B"

174B description "Link to External-A"
1764B
switchport
1748B switchport
1765B

176B

176B
1,1911 1,1911
channel-group 1000 mode active
175B channel-group 1000 mode active
1768B
no shutdown
1752B no shutdown
1769B
Configure Configure the interfaces for connections to the SFS switches. Ports 1/49 and 1/50 are
28B
interfaces configured as L3 interfaces. The IP addresses used are from Table 19. Optionally, allow
the forwarding of jumbo frames using the mtu 9216 command.
Create port channel 1. In this example, port channel 1 connects to the DNS/NTP server. It
283B
is on VLAN 1911, which represents the preexisting management VLAN. Add the port
channel to vPC 1.
Interface 1/1 on each external switch is connected to the DNS/NTP server. Each interface
284B
is added to VLAN 1911 and port-channel 1. Port-channel 1 is set as an LACP port-

channel with the channel-group 1 mode active command.
Deployment Guide
N9K-External-A
126B N9K-External-B
127B
interface ethernet 1/49

170B interface ethernet 1/49
1796B
description Leaf1A
17B description Leaf1A
179B
no shutdown
172B no shutdown
1798B
no switchport
173B no switchport
179B
mtu 9216
174B mtu 9216
180B
ip address 192.168.1.0/31
175B ip address 192.168.2.0/31
180B

1802B
description Leaf1B
17B description Leaf1B
1803B
no shutdown
178B no shutdown
1804B
no switchport
179B no switchport
1805B
mtu 9216
1780B mtu 9216
1806B
ip address 192.168.1.2/31
178B ip address 192.168.2.2/31
1807B
180B
description "vPC to DNS/NTP"

1783B description "vPC to DNS/NTP"
1809B
switchport
1784B switchport
180B
switchport mode access

1785B switchport mode access
18B
switchport access vlan 1911

1786B switchport access vlan 1911
182B
vpc 1
178B vpc 1
183B
no shutdown
178B no shutdown
184B

185B
description "Link to DNS/NTP"

1790B description "Link to DNS/NTP"
186B
switchport
179B switchport
187B

18B

189B

1820B
no shutdown
1795B no shutdown
182B
Configure BGP Configure a loopback interface to use for the BGP router ID.
285B
Allow BGP to distribute routes with the route-map allow permit command.
286B
Configure the BGP ASN with the router bgp command. The external switches share
287B
the same ASN. Use the address that was set for interface loopback0 as the router ID.
Use the address-family ipv4 unicast and redistribute direct route-map

379B
allow commands to redistribute IPv4 routes from physically connected interfaces.

Use the maximum-paths 2 command to configure the maximum number of paths that
28B
BGP adds to the route table for equal-cost multipath load balancing.
Specify the neighbor IP addresses and ASNs. Configure an address family for each
289B
neighbor.
290B
with the end and copy running-config startup-config commands.
Deployment Guide
External-A
128B External-B
129B
interface loopback0
182B interface loopback0
1840B
description router_ID
1823B description router_ID
184B
no shutdown
1824B no shutdown
1842B
ip address 10.0.2.1/32
1825B ip address 10.0.2.2/32
1843B
route-map allow permit 10

1826B route-map allow permit 10
184B
router bgp 65101

1827B router bgp 65101
1845B
router-id 10.0.2.1
182B router-id 10.0.2.2
1846B
address-family ipv4 unicast

1829B address-family ipv4 unicast
1847B
redistribute direct route-map

1830B redistribute direct route-map
184B
allow allow
maximum-paths 2
183B maximum-paths 2
1849B
neighbor 192.168.1.1 remote-as

1832B neighbor 192.168.2.1 remote-as
1850B
65011 65011
185B
no shutdown
1834B no shutdown
1852B
neighbor 192.168.1.3 remote-as

1835B neighbor 192.168.2.3 remote-as
1853B
65011 65011
1854B
no shutdown
1837B no shutdown
185B
end
183B end
1856B
copy running-config startup-

1839B copy running-config startup-
1857B
config config
Validate L3 After the uplink interfaces are configured on the Nexus external switches and on the SFS
291B
connections to leaf switches, connectivity can be verified using the switch CLI.
Cisco Nexus
switches Show command output on N9K-External-A
Note: The command output shown in the following commands is for the N9K-External-A switch.
The output for N9K-External-B is similar.
29B
When BGP is up, uptime is shown in the Up/Down column. The neighbors for N9K-
External-A shown in the output below are Leaf1A and Leaf1B.
N9K-External-A# show ip bgp summary

2691B
BGP summary information for VRF default, address family IPv4 Unicast
269B
BGP router identifier 10.0.2.1, local AS number 65101

2693B
BGP table version is 15, IPv4 Unicast config peers 2, capable peers 2
2694B
7 network entries and 14 paths using 2296 bytes of memory

2695B
BGP attribute entries [2/312], BGP AS path entries [1/6]

269B
BGP community entries [0/0], BGP clusterlist entries [0/0]

2697B
Neighbor
2698B V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
Deployment Guide
192.168.1.1
269B 4 65011 2912 2529 15 0 0 1d18h 5
192.168.1.3
270B 4 65011 2907 2529 15 0 0 1d18h 5
Run the show ip interface brief command to verify IP addresses are configured
380B
correctly. VLAN 1911 is the external management VLAN that contains the DNS/NTP
server. Loopback 0 is the router ID, and interfaces 1/49-1/50 are connected to the SFS
leaf switches.
N9K-External-A# show ip interface brief
2701B
IP Interface Status for VRF "default"(1)

270B
Interface
2703B IP Address Interface Status
Vlan1911
2704B 172.19.11.252 protocol-up/link-up/admin-up
Lo0
Eth1/49
Eth1/50
The show ip route command output for the N9K-External-A switch appears as shown.
293B
Note: The 172.18.11.0/24 External Management network has not yet been configured on the SFS
fabric, so it is not learned using BGP at this stage of deployment.
N9K-External-A# show ip route

2708B
IP Route Table for VRF "default"

2709B
'*' denotes best ucast next-hop

2710B
'**' denotes best mcast next-hop

271B
'[x/y]' denotes [preference/metric]

271B
'%<string>' in via output denotes VRF <string>

2713B
10.0.2.1/32, ubest/mbest: 2/0, attached

2714B
*via 10.0.2.1, Lo0, [0/0], 18:53:33, local

2715B
*via 10.0.2.1, Lo0, [0/0], 18:53:33, direct

2716B

271B
*via 172.19.11.252, Vlan1911, [0/0], 18:52:51, direct

2718B

2719B
*via 172.19.11.252, Vlan1911, [0/0], 18:52:51, local

270B

271B
*via 172.19.11.254, Vlan1911, [0/0], 18:52:51, vrrp_engine

27B

273B
*via 192.168.1.0, Eth1/49, [0/0], 00:00:09, direct

274B

275B
*via 192.168.1.0, Eth1/49, [0/0], 00:00:09, local

276B

27B
*via 192.168.1.2, Eth1/50, [0/0], 18:53:35, direct

278B

279B
*via 192.168.1.2, Eth1/50, [0/0], 18:53:35, local

2730B
192.168.2.0/31, ubest/mbest: 2/0

2731B
*via 192.168.1.1, [20/0], 00:00:05, bgp-65101, external, tag

273B 65011
273B 65011
192.168.2.2/31, ubest/mbest: 2/0
2734B

2735B 65011
2736B 65011
Deployment Guide
Show command output on Leaf1A
386B
Leaf1B is similar.
294B
shown in the output below are Leaf1B, N9K-External-A, and N9K-External-B.

273B

2738B
Neighbor
2739B AS MsgRcvd MsgSent Up/Down State/Pfx
172.16.0.1
2740B 65011 3222 3240 1d:22:14:58 8
192.168.1.0
2741B 65101 2794 3231 1d:18:29:11 4
192.168.2.0
274B 65101 2795 3226 1d:18:26:04 4
381B

382B
1/1/52 are the VLTi, and 1/1/53-1/1/54 are the uplinks to the external switches. VLAN
4090, Loopback 1, and Loopback 2 are used internally by SFS. VLAN 4094 and port
channel 1000 are automatically configured for the VLTi.
387B

2743B
Interface Name
274B IP-Address OK Method Status Protocol
================================================================================
2745B
Ethernet 1/1/1
2746B unassigned YES unset up up
Ethernet 1/1/2
Ethernet 1/1/3
Ethernet 1/1/49
Ethernet 1/1/50
Ethernet 1/1/51
Ethernet 1/1/52
Ethernet 1/1/53
2753B 192.168.1.1/31 YES manual up up
Ethernet 1/1/54
2754B 192.168.2.1/31 YES manual up up
Management 1/1/1
275B 100.67.127.28/24 YES manual up up
Vlan 4000
Vlan 4090
275B 172.16.0.1/31 YES manual up up
Vlan 4094
Port-channel 1000
Loopback 1
2760B 172.16.128.0/32 YES manual up up
Loopback 2
2761B 172.30.0.0/32 YES manual up up
Virtual-network 3939
Run the show ip route command to verify routes to the External Management VLAN,
295B
172.19.11.0/24, have been learned using BGP from the Nexus switches. In this example,
two routes to 172.19.11.0/24 are learned, one using each Nexus switch. The routs are
shown in bold in the output below.
Deployment Guide
2763B
2764B
S - static
2765B

276B

276B

2768B

2769B

270B

271B
Destination
27B Gateway Dist Last Change
----------------------------------------------------------------------------------
273B
B EX 10.0.2.1/32
274B via 192.168.1.0 20/0 00:43:16
via 192.168.2.0
275B
B EX 10.0.2.2/32
276B via 192.168.1.0 20/0 00:43:16
via 192.168.2.0
27B
C 172.16.0.0/31
278B via 172.16.0.1 vlan4090 0/0 02:19:46
C 172.16.128.0/32
279B via 172.16.128.0 loopback1 0/0 02:20:07
B IN 172.16.128.1/32
2780B via 172.16.0.0 200/0 02:19:44
B EX 172.19.11.0/24
2781B via 192.168.1.0 20/0 00:43:32
via 192.168.2.0
278B
C 172.30.0.0/32
2783B via 172.30.0.0 loopback2 0/0 02:20:07
C 192.168.1.0/31
2784B via 192.168.1.1 ethernet1/1/53 0/0 01:12:49
B IN 192.168.1.2/31
2785B via 172.16.0.0 200/0 01:09:12
C 192.168.2.0/31
2786B via 192.168.2.1 ethernet1/1/54 0/0 01:10:18
B IN 192.168.2.2/31
278B via 172.16.0.0 200/0 01:07:51
To continue deployment, go to the Configure a jump host port section of this guide.
296B
BGP validation on N9K-External-A during VxRail deployment

During VxRail deployment, virtual networks are automatically configured on the
297B
SmartFabric leaf switches. IP addresses are then manually assigned to each leaf switch
on the External Management network, 172.18.11.0/24 in this guide, as shown in the
Additional configuration steps for L3 uplinks section.
Once the items above are done, run the show ip route command on the external
298B
Nexus switches to verify routes to the External Management network, 172.18.11.0/24,

have been learned using BGP from the SmartFabric leaf switches. These are shown in
bold in the output below.
Note: The following command output is for the N9K-External-A switch. The output for N9K-
38B
External-B is similar.
278B
Deployment Guide
N9K-External-A# show ip route
IP Route Table for VRF "default"
2789B
'*' denotes best ucast next-hop

2790B
'**' denotes best mcast next-hop

2791B
'[x/y]' denotes [preference/metric]

279B
'%<string>' in via output denotes VRF <string>

2793B

2794B
*via 10.0.2.1, Lo0, [0/0], 18:53:33, local

2795B
*via 10.0.2.1, Lo0, [0/0], 18:53:33, direct

2796B

279B
*via 172.19.11.252, Vlan1911, [0/0], 18:52:51, direct

2798B

279B
*via 172.19.11.252, Vlan1911, [0/0], 18:52:51, local

280B

2801B
*via 172.19.11.254, Vlan1911, [0/0], 18:52:51, vrrp_engine

280B
172.18.11.0/24, ubest/mbest: 2/0

2803B

2804B 65011
2805B 65011
2806B
*via 192.168.1.0, Eth1/49, [0/0], 00:00:09, direct

2807B

280B
*via 192.168.1.0, Eth1/49, [0/0], 00:00:09, local

2809B

2810B
*via 192.168.1.2, Eth1/50, [0/0], 18:53:35, direct

281B

281B
*via 192.168.1.2, Eth1/50, [0/0], 18:53:35, local

2813B
192.168.2.0/31, ubest/mbest: 2/0

2814B

2815B 65011
2816B 65011
192.168.2.2/31, ubest/mbest: 2/0
2817B

281B 65011
2819B 65011
To continue deployment, go to the Validate and build VxRail cluster section of this guide.
29B
Deployment Guide
Configure external Nexus switches for L2 connections
The external Nexus and SmartFabric leaf switches are cabled as shown in Figure 128 and
38B
are powered on. When L2 uplink configuration is complete, Leaf1A and Leaf1B connect
with a VLT port channel to a virtual PortChannel (vPC) on the external Nexus switches. In
this example, an existing DNS/NTP server also connects to the Nexus switches using a
vPC.
Pee r Li nk
1 49 50 1 49 50
VLT/vPC
SmartFabric
vPC
53 54 53 54
3 2 1 1 2 3
DNS/NTP server
VLTi
L2 Connections
(All in VLAN 1811) VxRail node 1
VLTi/Peer Link
connections
VxRail node 2
VxRail node 3
Rack 1
Figure 128. L2 uplinks to external Nexus 9000 switches
312B
Note: DNS and NTP servers do not have to connect in this manner if they are reachable on the
389B
network.
All ports on the four switches shown in Figure 128 are in the External Management VLAN,
30B
1811, in this example.
Note: All Nexus switch configuration commands used to validate this topology are shown in the
3840B
sections that follow. These are only examples. Modify your Nexus external switch configuration as
needed for your environment.
General settings Enable the following features: interface-vlan, lacp, vrrp, vpc, lldp.
301B
Configure the hostname, OOB management IP address on VRF management, and the
VRF management route as shown.
Note: Nexus spanning tree settings are at their factory defaults in this example. You may
3841B
configure spanning tree on the Nexus switches as needed for your environment. On Dell leaf
switches in SmartFabric mode, spanning tree is disabled on L2 uplinks. See Dell EMC Networking
SmartFabric Services Deployment with VxRail for more information.
Deployment Guide
N9K-External-A
1230B N9K-External-B
123B
configure terminal
185B configure terminal
187B
feature
1859B interface-vlan feature
1872B interface-vlan
feature
1860B lacp feature
1873B lacp
feature
186B vrrp feature
1874B vrrp
feature
1862B vpc feature
1875B vpc
feature
1863B lldp feature
1876B lldp
hostname N9K-External-A
1864B hostname N9K-External-B
187B
interface mgmt 0
1865B interface mgmt 0
187B
ip address 100.67.127.30/24
186B ip address 100.67.127.29/24
1879B
vrf member management

1867B vrf member management
180B
no shutdown
186B no shutdown
18B
vrf context management

1869B vrf context management
182B
ip route 100.67.0.0/16
1870B ip route 100.67.0.0/16
183B
100.67.127.254 100.67.127.254
Configure the VLAN 1811 represents a preexisting management VLAN on the external network. DNS
302B
External and NTP services are located on this VLAN. Optionally, enable jumbo frames with the mtu
Management 9216 command.
VLAN
If traffic will be routed from the external switches to other external networks, assign a
30B
unique IP address on each switch and configure VRRP to provide gateway redundancy.
Assign the same virtual address to both switches.
N9K-External-A
123B N9K-External-B
123B
vlan 1811
184B vlan 1811
1895B
name ExtMgmt
185B name ExtMgmt
1896B
no shutdown
186B no shutdown
1897B
interface Vlan1811
187B interface Vlan1811
189B
description ExtMgmt
18B description ExtMgmt
189B
no shutdown
189B no shutdown
190B
mtu 9216
1890B mtu 9216
190B
ip address 172.18.11.252/24
189B ip address 172.18.11.253/24
1902B
vrrp 11
1892B vrrp 11
1903B
address 172.18.11.254
1893B address 172.18.11.254
1904B
no shutdown
1894B no shutdown
1905B
Configure the Create the vPC domain. The peer-keepalive destination is the OOB management IP
304B
vPC domain and address of the vPC peer switch.

peer link
Configure a port channel to use as the vPC peer link. Put the port channel in trunk mode
305B
and allow the default and External Management VLANs, 1 and 1811 respectively.
Configure the interfaces to use in the vPC peer link. Put the interfaces in trunk mode and
306B
allow the default and External Management VLANs, 1 and 1811 respectively. Add the
Deployment Guide
interfaces to the peer link port channel. Port-channel 1000 is set as an LACP port-channel
with the channel-group 1000 mode active command.
N9K-External-A
1235B
vpc domain 129

1906B vpc domain 129
1923B
role priority 1
1907B role priority 65535
1924B
peer-keepalive destination
1908B peer-keepalive destination
1925B
100.67.127.29 100.67.127.30

1926B
description "Peer-Link to
190B description "Peer-Link to
1927B
External-B" External-A"
switchport
19B switchport
1928B

192B

1930B
1,1811 1,1811
vpc peer-link
194B vpc peer-link
193B
no shutdown
195B no shutdown
1932B

193B
description "Link to External-B"

197B description "Link to External-A"
1934B
switchport
198B switchport
1935B

1936B

1937B
1,1811 1,1811
1938B
no shutdown
192B no shutdown
193B
Configure
329B
Configure the interfaces for connections to the SFS leaf switches. Interfaces 1/49 and
307B
interfaces 1/50 are configured in vPC 100 in this example. Port-channel 100 is set as an LACP port-
channel with the channel-group 100 mode active command.
Use the switchport mode trunk command to enable the port-channel to carry traffic
308B
for multiple VLANs. Allow VLAN 1811 (the External Management VLAN).
Optionally, allow the forwarding of jumbo frames with the mtu 9216 command.
309B
In this example, interface 1/1 on each external switch is configured in vPC 1 for
310B
connections to the DNS/NTP server. Port-channel 1 is set as an LACP port-channel with

the channel-group 1 mode active command.
31B
with the end and copy running-config startup-config commands.
Deployment Guide
N9K-External-A
1237B

1972B
description "vPC to Leaf1A/1B"

194B description "vPC to Leaf1A/1B"
1973B
switchport
1942B switchport
1974B

1975B

1976B
1811 1811
vpc 100
1945B vpc 100
197B
mtu 9216
1946B mtu 9216
1978B
no shutdown
1947B no shutdown
197B

1980B
description "Link to Leaf1A/1B"

194B description "Link to Leaf1A/1B"
198B
switchport
1950B switchport
1982B

1983B

1984B
1811 1811
mtu 9216
1953B mtu 9216
1985B

1986B
no shutdown
195B no shutdown
1987B
198B
description "vPC to DNS/NTP"

1957B description "vPC to DNS/NTP"
198B
switchport
1958B switchport
190B

19B

192B
vpc 1
196B vpc 1
193B
no shutdown
1962B no shutdown
194B

195B
description "Link to DNS/NTP"

1964B description "Link to DNS/NTP"
196B
switchport
1965B switchport
197B

198B

19B

20B
no shutdown
196B no shutdown
201B
end
1970B end
20B
copy running-config startup-

197B copy running-config startup-
203B
config config
Validation Once the uplink interfaces have been configured in the SFS GUI and on the external
312B
Nexus switches, connectivity can be verified using the switch CLI.
Show command Note: The command output shown in the following commands is for Leaf1A. The output for
3842B
output on Leaf1A Leaf1B is similar.
With SFS, port channel numbers are automatically assigned as they are created. In this
31B
example, port channel 1 is the uplink connected to the Nexus switches. It has two
members that are both up and active. Port channel 1000 is reserved for the VLTi.
Deployment Guide
280B
Flags:
281B D - Down I - member up but inactive P - member up and active
U - Up (port-channel)
28B F - Fallback Activated
--------------------------------------------------------------------------------
283B
Group Port-Channel
284B Type Protocol Member Ports
--------------------------------------------------------------------------------
285B
1
286B port-channel1 (U) Eth DYNAMIC 1/1/53(P) 1/1/54(P)
1000 port-channel1000 (U)
287B Eth STATIC 1/1/49(P) 1/1/50(P) 1/1/51(P)
1/1/52(P)
28B
The L2 uplink, port channel 1 in this example, is a tagged member of VLAN 1811. This is
384B
verified at the CLI using the show virtual-network command as follows:

289B

2830B
Unicast-Drop
2831B

283B
VLTi-VLAN: 1811
283B
Members:
2834B
2835B

2836B
ethernet1/1/3
2837B

283B
2839B

2840B
VLTi-VLAN: 1812
2841B
Members:
284B

2843B
ethernet1/1/3
284B

2845B
2846B

2847B
VLTi-VLAN: 1813
284B
Members:
2849B

2850B
ethernet1/1/3
2851B

285B
2853B

2854B
VLTi-VLAN: 1814
285B
Members:
2856B

2857B
ethernet1/1/3
Deployment Guide
285B

2859B
2860B

2861B
VLTi-VLAN: 1815
286B
Members: 2863B

2864B
ethernet1/1/3
2865B

286B
2867B

286B

2869B
VLTi-VLAN: 3939
2870B
Members: 2871B

287B

2873B

2874B
2875B

2876B

287B
VLTi-VLAN: 4091
287B
Members: 2879B

280B

281B

28B

283B
284B
Use the show vlt 255 vlt-port-detail to verify the status of VLT ports. Port
385B
channel 1 is the L2 uplink to the Nexus switches. The output shows information for both
VLT peer switches. An asterisk (*) denotes the local switch. In this case, Leaf1A is VLT
unit 1, and Leaf1B is VLT unit 2.
S5248F-Leaf1A# show vlt 255 vlt-port-detail
285B
vlt-port-channel ID : 1
286B
VLT Unit ID
287B Port-Channel Status Configured ports Active ports
-------------------------------------------------------------------------------
28B
* 1
289B port-channel1 up 2 2
2 2890B port-channel1 up 2 2
Show command Note: The command output shown in the following commands is for the N9K-External-A switch.
384B
output on N9K- The output for N9K-External-B is similar.

External-A
Deployment Guide
The show port-channel summary command confirms port channels are up. Po1
386B
connects to the DNS/NTP server, Po100 connects to the SFS leaf switches, and Po1000
is the peer link.
N9K-External-A# show port-channel summary
2891B
Flags: D - Down
289B P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
2893B
s - Suspended
2894B r - Module-removed
S - Switched
2895B R - Routed
U - Up (port-channel)
2896B
p - Up in delay-lacp mode (member)

2897B
M - Not in use. Min-links not met

289B
--------------------------------------------------------------------------------
289B
Group Port-
290B Type Protocol Member Ports
Channel
2901B
--------------------------------------------------------------------------------
290B
1
2903B Po1(SU) Eth LACP Eth1/1(P)
100
2904B Po100(SU) Eth LACP Eth1/49(P) Eth1/50(P)
1000 Po1000(SU) Eth
2905B LACP Eth1/51(P) Eth1/52(P)
Run the show vlan command to verify ports are correctly assigned to the External
387B
Management VLAN (VLAN 1811). Po1 connects to the DNS/NTP server, Po100 connects
to the SFS leaf switches, and Po1000 is the peer link.
N9K-External-A# show vlan
2906B
VLAN
2907B Name Status Ports
----
2908B -------------------------------- --------- -------------------------------
1
290B default active Po1000, Eth1/51, Eth1/52
1811
2910B ExtMgmt active Po1, Po100, Po1000, Eth1/49
Eth1/50, Eth1/51, Eth1/52
291B
VLAN
291B Type Vlan-mode
----
2913B ----- ----------
1
2914B enet CE
1811
2915B enet CE
Remote SPAN VLANs

2916B
-------------------------------------------------------------------------------
2917B
Primary
2918B Secondary Type Ports
-------
291B --------- --------------- -------------------------------------------
Run the show vpc command to verify all vpc connections are up. In this example,
314B
Po1000 is the peer link, Po1 connects to the DNS/NTP server, and Po100 connects to the
SFS leaf switches.
N9K-External-A# show vpc

290B
Legend:
291B
(*) - local vPC is down, forwarding via vPC peer-link

29B
vPC domain id
293B : 129
Deployment Guide
Peer status
294B : peer adjacency formed ok
vPC keep-alive status
295B : peer is alive
Configuration consistency status
296B : success
Per-vlan consistency status
297B : success
Type-2 consistency status
298B : success
vPC role
29B : primary
Number of vPCs configured
2930B : 2
Peer Gateway
2931B : Disabled
Dual-active excluded VLANs
293B : -
Graceful Consistency Check
293B : Enabled
Auto-recovery status
2934B : Disabled
Delay-restore status
2935B : Timer is off.(timeout = 30s)
Delay-restore SVI status
2936B : Timer is off.(timeout = 10s)
vPC Peer-link status

2937B
---------------------------------------------------------------------
2938B
id
293B Port Status Active vlans
--
2940B---- ------ --------------------------------------------------
1
2941BPo1000 up 1,1811
vPC status
294B
----------------------------------------------------------------------
2943B
id
294B Port Status Consistency Reason Active vlans
--
2945B---- ------ ----------- ------ ------------
1
2946BPo1 up success success 1811
100
2947B Po100 up success success 1811
Note To continue deployment, go to Configure a jump host port.

384B
Validated Nexus switches

The Cisco Nexus switches in the table below were used to validate the example in this
appendix.
Table 20. External Cisco Nexus switches

3865B
Qty
1238B Item
1239B Operating system version
1240B
2
127B Cisco Nexus 93180YC-EX switches
128B 7.0(3)I4(2)
129B
Note: Other validated components are listed in Appendix A.

3845B
Deployment Guide
Appendix D Support and Feedback

40B
Technical resources ...........................................................................................173
Fabric Design Center..........................................................................................173

405B
Feedback and technical support.......................................................................173

406B
Deployment Guide
Technical resources
Dell EMC Networking Info Hub
316B
Dell EMC Networking OS10 Info Hub

317B
Dell EMC SmartFabric OS10 User Guide Release 10.5.0
SmartFabric OS10 Solutions (HCI, Storage, MX) Support Matrix

318B
Dell EMC PowerSwitch S3048-ON Documentation

319B
Dell EMC PowerSwitch S5248F-ON Documentation

320B
Dell EMC PowerSwitch Z9264F-ON Documentation

321B
Dell EMC Networking Transceivers and Cables

32B
Dell EMC OpenManage Network Integration for VMware vCenter – This site includes
32B
OMNI software and the SmartFabric Services for OpenManage Network Integration User
Guide, Release 1.3
Dell EMC OS10 SmartFabric Services FAQ

324B
Dell EMC VxRail Network Planning Guide

325B
Dell EMC VxRail Support Matrix (account required)

326B
Dell Technologies Solve Online (account required)

327B
Dell EMC VxRail support and documentation (account required)

328B
VxRail Documentation Quick Reference List (account required)

329B
Dell EMC Networking SmartFabric Services Deployment with VxRail 4.7

30B
Fabric Design Center

The Dell EMC Fabric Design Center (FDC) is a cloud-based application that automates
31B
the planning, design, and deployment of network fabrics that power Dell EMC compute,
storage, and hyperconverged infrastructure solutions. The FDC is ideal for turnkey
solutions and automation based on validated deployment guides.
FDC allows design customization and flexibility to go beyond validated deployment

32B
guides. For additional information, go to the Dell EMC Fabric Design Center.
Feedback and technical support

We encourage readers to provide feedback on the quality and usefulness of this
38B
publication by sending an email to Dell_Networking_Solutions@Dell.com.

For technical support, go to http://www.dell.com/support.
3B
Deployment Guide

Sfs Deployment With Vxrail 7.0 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sfs Deployment With Vxrail 7.0 PDF

Uploaded by

Copyright:

Available Formats

Dell EMC Networking SmartFabric Services

Deployment with VxRail 7.0

Dell Networking Infrastructure Solutions

2 Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0

Chapter 2 Hardware Overview 11

Chapter 4 Deployment Planning 20

Chapter 5 Configure the First Leaf Switch Pair 28

Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0 3

Chapter 6 Deploy VxRail 70

Chapter 7 Expand to Multirack 85

Chapter 8 Deploy and Configure OMNI 102

Chapter 9 Use OMNI for Post-deployment Configuration 118

Appendix A Validated Components 139

Appendix B CLI Commands 142

4 Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0

Appendix D Support and Feedback 172

Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0 5

This chapter presents the following topics:

Purpose of this guide .............................................................................................7

Dell Technologies ...................................................................................................7

SmartFabric Services .............................................................................................7

SmartFabric Services with VxRail ........................................................................9

OpenManage Network Integration ........................................................................9

Typographical conventions .................................................................................10

6 Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0

SFS SFS has two personalities:

• Layer 3 (L3) Fabric personality – the new SFS personality available as of

Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0 7

Figure 1. From a network of boxes to a networked fabric

The SFS L3 Fabric build process is as follows:

8 Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0

Layer 3 Network Fabric

VLTi VLTi VLTi

SmartFabric Services with VxRail

OpenManage Network Integration

The following tasks are done in the OMNI plug-in in vCenter:

• View the leaf-spine topology

Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0 9

Monospace Text CLI examples

Underlined Monospace Text CLI examples that wrap the page

Italic Monospace Text Variables in CLI examples

Bold Monospace Text Commands entered at the CLI prompt, or to

Bold text GUI elements and information entered in the GUI

10 Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0

This chapter presents the following topics:

Supported switches ..............................................................................................12

Hardware used in this guide ................................................................................12

Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0 11

Table 1. Supported switches

S4112F/T-ON, S4128F/T-ON, S4148F/T-ON Leaf 10 GbE

S5212F-ON, S5224F-ON, S5248F-ON, S5296F-ON Leaf 10/25 GbE

S5232F-ON, Z9264F-ON Spine See the Note below

Hardware used in this guide

Figure 3. Dell EMC PowerSwitch S5248F-ON

12 Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0

Figure 4. Dell EMC PowerSwitch Z9264F-ON

Figure 5. Dell EMC PowerSwitch S3048-ON

Figure 6. Dell EMC VxRail 2-RU node (P, V, and S Series)

Dell EMC Networking SmartFabric Services Deployment with VxRail 7.0 13

This chapter presents the following topics:

Production topology with SmartFabric Services ..............................................15

Production topology connection details ...........................................................16

OOB management topology ................................................................................18