NimbelStorage NTS-2001-I Student Guide R5 Feb2017

NTS-2001-I: Nimble
Introduction and Administration
Course Revision 5
Introductions
 Name
 Employer
 Job function
 Data storage experience
 Hands on experience with Nimble products?
 What do you hope to get out of this course?
© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-4

Course Topics
 Product Introduction  Backup and Disaster Recovery

» Module 1: AF-Series Introduction » Module 8: Introduction to Nimble Storage
» Module 2: CS-Series Introduction Snapshots
» Module 3: NimbleOS WebUI Introduction » Module 9: Introduction to Nimble Storage
» Module 4: Introduction to Customer Replication
Support and InfoSight
 Exam completion – required for course
 Product Utilization completion
» Module 5: Pre-Installation, Racking, and
Cabling
» Module 6: Initial Configuration of the
Unified Flash Fabric Overview
» Module 7: Working with Nimble Storage
Volumes

Documentation
Documentation location
1. Log into InfoSight  Categories

2. Click Resources and then » Release Notes
Documentation » Product User Guides
» Solution and Integration Guides
» Support and Knowledge Base Articles
 Refine Searches By:

» Document type
» Nimble Software and Solutions
» NimbleOS Version
» Integration
» Platform

Units of Measure Defined
Raw versus Usable versus Effective Capacity
AF-Series arrays using

deduplication will see
Subtract additional savings.
capacity
for Add storage
Raw RAID Triple capacity due
to inline Effective
Capacity Parity,
Usable compression Capacity
& system
Capacity (typical 30%
reserves
to 75%)
Effective: 49.5TB
Raw: 48TB Usable: 33TB
(assuming 50% compression)

TiB Vs. TB
collateral utilizes
TB and TiB
Nimble OS uses TiB
Tebibyte (TiB) and terabyte (TB) are related
 TB is base 10 (i.e., 1 TB = 1,000,000,000,000 bytes)
 Ti, or tebi, represents multiplication by 10244. So the math

using Ti is more accurate:
» 1 tebibyte = 240 bytes or 1,099,511,627,776 bytes or more commonly
stated as 1024 gigabytes

Module 01:
The All Flash Array (AF-Series)

Module 01 Objectives
At the end of this module, you will be able to:

 Describe the AF-Series hardware
 Understand the AF-Series Write Operations
» Deduplication
» Compression
» Sequential stripe writes
» Triple+ Parity with Integrated Spare
» Integrated spare rebuild
» Quick RAID rebuild
» SmartSecure Encryption
 Understand AF-Series Read Operations
» Changed block operations
 Scale-to-Fit with the AF-Series

Product Line Introduction
AF-Series: a product family of the Unified Flash Fabric
All Flash Array Family: AF-Series
AF9000
300,000 IOPS
AF7000
230,000 IOPS
AF5000
120,000 IOPS
AF3000
50,000 IOPS IOPS based on a 70%

AF1000
Read and 30% write
workload
35,000 IOPS
www.nimblestorage.com/technology-products/all-flash-array-specifications/

AF-Series Hardware Tour
What’s in an AF-Series Array?
Back: Dual Power
Supplies (AC and DC
available)
Front: SSD Drives Back: Dual Controllers

(CPU, Network)
Controller Head Shelf

Expansion Shelves (optional)

AF-Series Chassis Front View
 4U Chassis
 48 SSD Drive capacity
 2 Banks of 24 x 3.5” SSD drives
 Bank A – SSD Drives 1 -24
 Bank B – SSD Drives 25 - 48 Slot 21 Slot 22 Slot 23 Slot 24
Slot 17
Slot 13
Slot 9
Slot 5
Slot 1 Slot 2 Slot 3 Slot 4

AF-Series Chassis Front View / Dual-Flash Carrier (DFC)
Power On LED
Slot 17 Heartbeat
NIC1/2 LED
Slot 13
Slot 9 Power Fault Over
Slot 5 Temperature
DFC latch
Bank B Ejector latch
Bank B
Bank A
Bank A Ejector latch

AF-Series Dual-Flash Carrier (DFC)
 Hot swappable base carrier hosting two banks.

 Hot swappable SSD carrier provides tool-less
drive installation.
 LEDs on each SSD Carrier and on the Base
Carrier
Presence & Drive Fault LED

Activity Slot Fault
Base Carrier
Release
SSDCarrier
Release

AF-Series Hardware Tour - Back
1
5
3
4
6 7
1. Power Supply 3. Fans 5. Data Networks 7. SAS Ports
2. Power Intel 4. Management Network 6. KVM/Serial Port 8. Controller
Controller Detail View (AF-Series and Newer CS-Series)

Mid-Chassis Fans

AF-Series Spares / FRUs (Controller Head Shelf)
Or
Spares Kit
Controller
(NICs/HBAs not included) Power Supply
Additional Spares
AF-Series SSD
NIC / HBA
Cables

AF-Series Write Operations
CASL Architecture
8K 4K 4K
ack
Mirrored
1) Writes are sent by a variety of
applications in variable block sizes
NVDIMM (Active CTRL) NVDIMM (Standby CTRL)
2) CASL places incoming writes into the
8K 4K 4K 8K 4K 4K
active controllers NVRAM
3) CASL mirrors the active controllers
NVRAM to the standby controllers DRAM
NVRAM
4) CASL acknowledges the write
CS-Series AF-Series

CASL Architecture
8K 4K 4K
ack
Mirrored
5) Blocks are copied into DRAM
a) What happens next depends of
type of array
8K 4K 4K 8K 4K 4K
b) All-Flash Array
1. Variable block deduplication
is applied DRAM
2. Variable block compression 8K 4K 4K
Variable Block Variable Block
is applied Deduplication Compression
Sequential Stripe 10MB
3. Blocks are formed into a
10MB sequential stripe
write Index
4. Blocks written to SSD and
are indexed in DRAM
CS-Series AF-Series
Index

Mirrored
DRAM
Variable Block
Deduplication
AF-Series
Index
AF-Series: Deduplication
Deduplication Primer
Inline Deduplication
Vs.
Source / Server Post Process
Deduplication
Target / Storage Array
Inline Deduplication:
• Dedupe applied as the writes are processed
Post Process Deduplication:

• Writes the data as-is and uses a background operation to remove duplicates

Process Review
Incoming writes Batched in NVRAM/NVDIMM
Batch Flush
Variable Block Deduplication

• Secured by strong SHA2-256 encryption
Variable Block Compression
Full Stripe Writes

Deduplication Block Sizes
Variable Block Dedupe Compression
Fixed Block Dedupe

Target / Storage Array
Variable Block Level Dedupe:

• Unit of duplicate detection is volume block which depends on the performance policy.
• Requires less metadata by adapting the detection size to the application block size, thereby
improving efficiency and speed.
Application Aware:
DB VDI DB Exchange
VDI Exchange VDI VDI

Fingerprint Matching or Byte Comparison?
With most forms of deduplication,

performance is adversely affected Fingerprint Index
in some way.
Deduplication
Performance • Better than block-by-block
• Secured by strong SHA2-256 encryption

Dedupe Isn’t Free
 Fingerprinting content costs CPU

 Identifying duplicates costs:
» memory
» I/O (flash reads)
» CPU
 Tracking content costs storage
» Flash content/FDR
 Not every workload benefits!

 These costs are true of dedupe in general, not our implementation.

Deduplication Global, Domain Level, or Off?
On – On –
Off
Globally Domain/Application
• Per Application Category Eliminate the

• Per Storage Pool deduplication
• Per Volume “tax”
Make informed decisions about whether to use

deduplication or not and what workloads or volumes
would be best served by deduplication.
InfoSight
Predictive Analytics
Deduplication Management – AFA only
Global Deduplication
Deduplication at Domain/Application Level

Application Category Management – AFA only
 Application Categories are introduced with

Deduplication and are found as part of the
performance policy
 Application Category:
» A data structure in the WebUI that limits the scope
of duplication on volumes attached to an
Application Policy
» There are 8 fixed application Categories

Nimble Differentiators – More about App Category
App Category VDI Database
VDI-Policy-1 VDI-Policy-2 VDI-Policy-3 VDI-Policy-4 DB-Policy-1

Perf Policy BS=4K BS=8K BS=4K BS=8K BS=32K
Dedupe=ON Dedupe=ON Dedupe=ON Dedupe=OFF Dedupe=OFF
Volume V1 V2 V3 V4 V5 V6 V7 V8 V10 V11
V9
Like colored volumes
denote block sharing Dedupe = OFF

Viewing Data Reduction Savings
Monitor >> Space
Data Reduction Savings
5.2TB | 5.93X
Clones 0 B 1.0X
Deduplication 3.79 TB 2.55X
Compression 1.4 TB 2.33X
Deduplication Savings
Savings Ratio Participating
Volumes
3.79 TB 2.55X 18 of 18 vols
0B 1.0X 1 of 1 vols
Mirrored
DRAM

Deduplication Compression
AF-Series
Index
AF/CS-Series: Compression
What you need to know about Lempel-Ziv 4
 Compression performed in-line

» Applied after deduplication, if deduplication is enabled
 LZ4 is a fast lossless compression algorithm
» Provides compression speeds of 300 MB/s per CPU core
» Provides a fast decoder that provides speeds up to and beyond 1GB/s per
CPU core
• It can reach RAM speed limits on multi-core systems.
© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 40 1-40

Viewing Data Reduction Savings
Monitor >> Space
Data Reduction Savings
5.2TB | 5.93X
Clones 0 B 1.0X
Deduplication 3.79 TB 2.55X
Compression 1.4 TB 2.33X
Compression Savings
Savings Ratio
1.4 TB 2.33X
0B 1.0X

Application Compression with Nimble
*March 2016

Mirrored
DRAM
Sequential
Variable Block Variable Block Stripe 10MB
Index
AF-Series
Index
Sequential Stripe Write

Common File Systems– Write in place file system (EMC, EQL)
Pros
•Simple to implement, long
history
•Good sequential read

performance without cache
Cons
•Poor random write performance
•Slow, high overhead compression

Common File Systems – Hole filling (WAFL, ZFS)
WAFL – Write Anywhere File Layout

ZFS – Copy on write transactional model
Pros
•Good random write performance until
disk fills up
•More efficient redirect-on-write

snapshots
Cons
•Performance degrades over time
•Slow, high overhead compression

Nimble Storage File System – Always write full stripes
 Good AND consistent write

performance
 Very efficient snapshots
 Fast inline compression
 Efficient flash utilization,
long flash life
 Ground up design
 Enables variable block size
 Uses a sweeping process to ensure full
stripe write space

Background Sweeping
• Data blocks are indexed as they are written

• Over time the deletion of snapshots and data leaves stale data blocks
• Sweeping removes stale blocks and forms new stripe writes with the
remaining active blocks

Mirrored
DRAM
Sequential
Index
AF-Series
Triple+ Parity with

Integrated Spare Index
AF-Series: Triple+ Parity With Integrated Spare

Triple+ Parity With Integrated Spare
Triple+ Parity RAID Stripe

D1 D2 D3 P Q R
 Triple Parity: 3 drives
worth of parity information
is written

Intra-Drive Parity: Data Layout
Triple+ Parity RAID Stripe

 Every chunk in a stripe consists of Data, D1 D2 D3 P Q R
Metadata (checksums, sequence number
etc…), and Parity (XOR of Data)
 This parity can be used to recover 1 sector
corruption or IO error, when there is no
redundancy in the stripe (3 drive failure)
 This recovery can be done independent of the
condition of other drives: can recover from 1- Data
sector corruption on every drive read in the Chunk
stripe. (1 per drive)
 Note: Intra-Drive Parity is also available in CS- Metadata
Series starting with Nimble OS 3.X. Chances Parity
of data recovery in HDD is lower than SSD
due to errors/corruptions in HDD are often
larger.
Integrated Spare: Data, Parity, and Spare layout
AFA Triple+ Parity

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
D D D D D D D D D D D D D D D D D D D D P Q R S
D D D D D D D D D D D D D D D D D D D P Q R S D
D D D D D D D D D D D D D D D D D D P Q R S D D Tolerates Simultaneous Failure of Any 3 SSDs
D D D D D D D D D D D D D D D D D P Q R S D D D
D D D D D D D D D D D D D D D D P Q R S D D D D
D D D D D D D D D D D D D D D P Q R S D D D D D
D D D D D D D D D D D D D D P Q R S D D D D D D
D D D D D D D D D D D D D P Q R S D D D D D D D
D D D D D D D D D D D D P Q R S D D D D D D D D
D D D D D D D D D D D P Q R S D D D D D D D D D
D D D D D D D D D D P Q R S D D D D D D D D D D Built in virtual spare allows 4th failure
D D D D D D D D D P Q R S D D D D D D D D D D D
D D D D D D D D P Q R S D D D D D D D D D D D D
D D D D D D D P Q R S D D D D D D D D D D D D D
D D D D D D P Q R S D D D D D D D D D D D D D D
D D D D D P Q R S D D D D D D D D D D D D D D D
D D D D P Q R S D D D D D D D D D D D D D D D D
D D D P Q R S D D D D D D D D D D D D D D D D D
D D P Q R S D D D D D D D D D D D D D D D D D D Intra-drive parity fixes sector loss in single read
D P Q R S D D D D D D D D D D D D D D D D D D D
P Q R S D D D D D D D D D D D D D D D D D D D D
Q R S D D D D D D D D D D D D D D D D D D D D P
R S D D D D D D D D D D D D D D D D D D D D P Q
S D D D D D D D D D D D D D D D D D D D D P Q R
Data Quick RAID rebuild
Metadata
Parity
Mirrored
DRAM
Sequential
Index
AF-Series
Index
Integrated Spare: Rebuild Operation

Integrated Spare RAID : Single Drive Failure Scenario
Failed
Disk
Disk 0 Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8 Disk 9 Disk 10
D D D D D D D P Q R D
D D D D D D P Q R D D
D D D D D P Q R D D D
Spare Chunk
D D D D P Q R P D D D
D D D P Q R Q D D D D
D D P Q R R D D D D D
D P Q R S D D D D D D
P Q R D D D D D D D D
Q R D D D D D D D D P
R D D D D D D D D P Q
D D D D D D D D P Q R
 Generate data/parity belonging to failed drive

 Write data to spare chunk

Mirrored
DRAM
Sequential
Index
AF-Series
Index
Quick RAID Rebuild

Quick Raid Rebuild: Motivation & Goal
Long rebuilds create frustration when:

 User pulls a drive out by mistake
 Drive is not completely failed, but needs to be reseated for various reasons
 S/W stack needs to power cycle the drive for recovery purposes
 S/W stack needs to reset interposer for recovery purposes that may lead HBA link timeout
Goal:
 Enable the NOS to take action to help reset or recover an unhealthy drive.
 See whether there is any way the RAID system can bypass the painful long rebuild or reduce its
impact to minimum

Quick Raid Rebuild: Impact to end-user
 When a drive is mistakenly taken out, it can be reinserted immediately without

lengthy rebuild time
 No timeout period for quick rebuild eligible (!)
 Rebuild is essentially incremental in some scenarios
 The feature is ready on both AFA and CS in the NimbleOS 3.X release
Notes:
 Not applicable when replacing a failed drive
 “Incremental” rebuild, subject to certain conditions
© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 59 1-59

Mirrored
DRAM
Sequential
Index
AF-Series
Index
SmartSecure Encryption
SmartSecure Software Based Encryption
What is SmartSecure?
Encryption that:
 Ensures the secrecy of data “at rest”
» Uses AES-256-XTS cipher for cryptographic protection of data
» FIPS 140-2 level 1 certified *****
 Protects against threat vectors Hello Hello

» Theft of entire storage array
» Theft of disks (HDDs or SSDs)
 Is transparent to host access to encrypted volumes
» Data is encrypted at write time when entering the array
» Data is decrypted at read time when exiting the array

Why Deploy SmartSecure?

 Theft or disposal of drives or an entire array does not expose data
» Data Center physical security breach
» Shipping or transporting an array to an affiliate site or new location
» RMA of a failed component
 Ability to irrecoverably shred (destroy) data at a volume level
» Volume keys marked for deletion when an encrypted volume is deleted
 Requirements for data to be encrypted on disk
» Government, compliance, customer, etc.
 Ensures data secrecy on replication streams over a WAN

Key Data Points

 No additional cost
» Familiar Nimble Storage licensing/pricing model
 Supported on all Nimble Storage arrays running NimbleOS 2.3 and higher
» Leverages the Intel AES-NI instruction set
• Best performance on later model CS series arrays
– CS235, CS300, CS500, CS700
 Supports encryption of data on both HDDs and SSDs

 Preserves compression feature
» Data is compressed prior to encryption
 Easy to administer & manage

Administration >> Security >> Encryption

Optional
SmartSecure Software Based Encryption Content
Technical Data Points

 The Passphrase…
» In “Secure Mode” the passphrase must be entered after an array restart or power on…
• In “Secure Mode”, encrypted volumes remain offline and cannot be accessed until the passphrase is
entered
• If the passphrase is not available, the data in encrypted volume is lost forever
» In “Available Mode” the passphrase may be required…
• If controllers are being swapped during an upgrade process
• In the rare scenario involving NVRAM loss
 Do not lose the passphrase!

» The array user must keep track of and maintain the passphrase
» The passphrase is never stored on disk or SSD within the Nimble Storage array
» The passphrase is not transmitted to Nimble Storage Technical Support
» The passphrase is not copied into Email Alerts, SNMP, or Syslog

Technical Data Points

 Volume Keys
» The encrypted state of a volume is defined at volume creation time
• Encrypted volumes cannot be “converted” to unencrypted volumes
• Unencrypted volumes cannot be “converted” to encrypted volumes
» New encrypted volumes each get a new encryption key
• 256 bits in length
• Encrypted by the master key using AES-256-KeyWrap
• Stored in the key table
» Cloned encrypted volumes are also encrypted
• Clones get their own new volume key
• But also have access to their ancestor’s key to read shared blocks

AF-Series Read Operations
CASL Architecture
Mirrored
1) Read from NVDIMM
2) If not found, check DRAM
3) If not found, read from SSD using the
index for a quick lookup 1 8K 4K 4K 8K 4K 4K
• If found, validate checksum,

decompress, dedupe, and return 2 DRAM
data
Index
3
AF-Series
Index

Changed block operations review
Compression Performance Comparison During Changed Block
Operation
Fixed block Architecture CASL Variable Blocks

8 blocks grouped Individual blocks
& compressed 1 2 3 4 5 6 7 8
compressed and coalesced
Group placed into into stripe
N fixed size slots 1 12 2 33 4 5 56 67 87 8
Updated data block
Entire group read & compressed and coalesced
uncompressed into new stripe
Block updated with
new data
New group compressed &
re-written
Other Array Manufacturers Nimble Storage

Compression Performance Comparison For A Changed Block
Fixed block Architecture CASL Variable Blocks

8 blocks grouped Individual blocks
& compressed compressed and coalesced
Group placed into Cost of fixed block architecture relative to CASL: into stripe
N fixed size slots 1. Additional blocks read from disk
2. Additional CPU cycles for decompression & Updated data block
Entire group read & recompression of all N blocks compressed and coalesced
uncompressed 3. Additional M-1 blocks written to disk into new stripe
Block updated with
new data
New group compressed &
re-written
Other Array Manufacturers Nimble Storage

Scale-to-Fit with the AF-Series
Nimble Storage AF-Series Scale-to-Fit: Scale Out
4 Node AF9000
Cluster
AF9000
300K IOPS Up to 1.4M IOPS
8180TB Capacity
AF7000
230K IOPS
IOPS based on
a 70% Read
and 30% write
workload AF5000
120K IOPS
SCALE UP
AF3000
50K IOPS
SCALE DEEP
AF3000 and 5000 + 1 AFS2 AF7000 + 2 AFS2 AF9000 + 2 AFS2

AF-Series Scale Out
SCALE DEEP

Scale Deep with the AFS2 Expansion Shelf
The AF-Series AFS2 Expansion Shelf

AFS2 Front View

AFS2 Bezel Panel
Power Button
Power On LED Heartbeat LED

On=PowerOK Always On
Off=No Power
AFS2 shelves have the same
front panel LEDs, except the SAS
port LEDs replace the NIC LEDs
SAS-out SAS-in
On=Link On=Link
Off=No Link Off=No Link
Power Fault Over Temperature

Off=No Fault Off=OK
or No PSU On=System OverTemp
On=PSU Fault, Blink=At least 2 fans failed
No AC, or Internal Fault

AFS2 Rear View
Path A Path B
SAS IN port
Expander FW Status
Green-OK
Red-FW Failed to Load SAS Link Status
Green-12Gbps Red-Fault
Orange-6Gbps Off-No link

AFS2 Connections – AF3000/AF5000
 One AFS2 is supported.

 Connect P1 SAS-out from the
head to SAS-in of AFS2
– P2 would work just as well but P1
is preferred for consistency
SAS expansion port (p1) SAS expansion port (p1)
SAS IN port SAS IN port

AFS2 Connections – AF7000/AF9000
 Support up to 2 AFS2.
 First AFS2 - Use P1
 Second AFS2 – Use P2
SAS
– Do not daisy chain AFS2 expansion
port (p2)
AFS2 shelves ship with 1 meter cables. If

you do not order a 3 meter cable you will
need to install one AFS2 shelf above the
controller shelf and one below.

AFS2 Web GUI
 When a AFS2 shelf is

added, it will need to be
activated using the
Activate button.
 Once activated the shelf

will be added to the
storage pool.
Once and expansion shelf

is activated, it cannot be
removed from the solution.

Expanding Existing System
1 Fill Expansion Shelf until 2 Then balance capacity between them

capacity utilization
matches the control
head
50% Capacity 50% Capacity
Expansion Shelf Controller Shelf

Power On/Off Order
 On  Off
» Power expansion shelves first, then the – Power off the controller shelf and then
controller shelves the expansion shelves
Controller Shelf
Expansion Shelf

AF9000
300K IOPS
AF7000
IOPS based on
230K IOPS AF-Series Scale Up
a 70% Read
and 30% write
workload AF5000
120K IOPS
SCALE UP
AF3000
50K IOPS

Scale Up with Model Upgrades
Model Upgrades

AF-Series Controller Upgrades
 To ensure the most accurate information regarding controller upgrades for the AF-
Series array download the Nimble All Flash Array Configuration Support Matrix from
InfoSight.

Upgrading Controllers
1 Determine which controller is currently in Standby mode.

• In WebUI – Manage Arrays >> Select individual array
• In CLI – use command ctrl --list

2 Label and disconnect all cables running to the Standby controller.
Management
ETH2

3 Remove the Standby controller.

• Loosen the controller thumb screw.
• Pull the controller release lever out and down.
• Pull the controller straight out.

4 Remove the onboard USB stick from the old controller and install it into the new controller.
During controller upgrade, you must remove the USB stick from the existing controller and
install it into the new controller. Failure to perform this step will prevent the new controller
from coming online.

5 Install new controller and reconnect all cables.
6 Verify the controller powers up and is in standby mode.

7 Perform a failover to the new controller and ensure that the new controller is in active mode.
• In WebUI – Manage Arrays >> Select individual array >> click “Make Active”
• In CLI – use command failover

8 Confirm the non-upgraded controller is in standby mode and repeat steps 2 – 7 for the opposite
controller:
2. Label and disconnect all cables running to the Standby controller.
3. Remove the Standby controller
4. Remove the onboard USB stick from the old controller and install it into the new controller.
5. Install new controller and reconnect all cables.
6. Verify the controller powers up and is in standby mode.
7. Perform a failover to the new controller and ensure that the new controller is in active mode.

9 Verify that the model number in the WebUI or CLI reflects the new model number.

4 Node AF9000
Cluster
Up to 1.4M IOPS
8180TB Capacity
AF-Series Scale Out

Scale Out with Clustered Arrays
4 Node AF9000  Cluster up to 4 of any

Cluster Nimble Storage arrays
AF9000
125K IOPS Up to 1.4M IOPS
8180TB Capacity
 Simple configuration
 Automated host connection
AF7000 management
270K IOPS
 Dynamic load-balancing and
automated capacity rebalancing
AF5000
140K IOPS
 Linear performance
scaling
SCALE UP
AF3000,
60K IOPS
 Automatic data
SCALE DEEP
migrations

Keep Applications Running – No Disruptions!
 Add or remove arrays

non-disruptively
» Automatically rebalances data
 Non-disruptively perform a technology

refresh
» Migrates data to upgraded hardware Repurpose older
array for Disaster
Recovery
Exchange + SQL Server VDI VDI VDI

Scale-Out Management - Groups
 Group: Manage arrays from a single

console
» 1-4 arrays per group
» Add and remove arrays on the fly
» Uses a single management IP
• Get status and view performance and capacity
reports
• Create and manage storage pools and volumes

Scale-Out Pools
 Pools
Nimble Scale-Out Group
» Simplify management
Array A1* Array A2 Array A3 Array A4
» Easy to grow or shrink
• Array is used in only 1 pool at a time L
» Can migrate HW (live) Pool Pool
Nimble Scale-Out : Finance-1

 How To Use Array A1* Array A2 Array A3 Array A4
» Use to scale out apps Pooling by L
Application
» Use to simplify allocation to SharePoint VDI
departments
Nimble Scale-Out: Remote Office-1
Array A1* Array A2 Array A3 Array A4
*As of OS 3.x, multi-array pools, group merges, etc are Pooling by L
supported on both FC and iSCSI Organization
The initial FC release does not support volume moves, Sales IT/Back Office Apps
striped pools and group merge.
Scale Out – Managing Hosts & Paths
Nimble Connection Mgr (NCM)

• Automatic MPIO configuration and path management
• Eliminates manual connection setup to individual arrays
Nimble Connection
Manager or NCS/PSP
plug-in for VMware
NIC 1 NIC 2
Switch 1 Switch 2
NCM Nimble Scale-Out Group

Module 02:
The Adaptive Flash Array (CS-Series)
Course Revision 5a

 Describe the CSx000 hardware
 Understand the CSx000 Write Operations
 Understand CSx000 Read Operations
 Scale-to-Fit with the CSx000
© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5a 2-2

Product Line Introduction
CSx000: a product family of the Unified Flash Fabric
Adaptive Flash Product Portfolio – Mapping
CS7000
• 4U24
CS5000 • Up to 230K IOPS
• 21TB to 882TB raw
CS3000 • 4U24 • 6 Expansion shelves
• Up to 120K IOPS • 1 AFS
• 4U24 • 21TB to 882TB raw
CS1000/H • Up to 50K IOPS • 6 Expansion shelves
• 4U24 • 21TB to 882TB raw • 1 AFS
• Up to 35K IOPS • 6 Expansion shelves
• 11TB to 882TB raw • 1 AFS
• 6 Expansion shelves CS700
• 1 AFS
• 3U16
• 125K IOPS
CS500 • 12TB to 612TB raw
• 3U16 • 6 Expansion shelves
CS300 • 100K IOPS • 1 AFS (25.6TB max)
CS235 • 12TB to 612TB raw
• 3U16
• 6 Expansion shelves
• 30K IOPS
CS215 • 3U16 • 12TB to 612TB raw
• 1 AFS (25.6TB max)
CS210 • 15K IOPS • 6 Expansion shelves
• 3U16 • 24TB to 612TB raw
• 3U16 • 15K IOPS • 1 AFS (25.6TB max)
• 3x 6TB Expansion shelves
• 15K IOPS • 12TB to 282GB raw • No AFS
• 8TB to 98TB raw • 3x 6TB Expansion shelves
• 1x 6TB Expansion shelves • No AFS, no FC
• No AFS, no FC
Portfolio Consolidation
Adaptive Flash Array: CSx000
CS7000
Up to
230,000 IOPS
CS5000
Up to
120,000 IOPS
CS3000
Up to
50,000 IOPS
CS1000 IOPS based on a 70%
CS1000H Read and 30% write
workload
Up to
35,000 IOPS
www.nimblestorage.com/technology-products/all-flash-array-specifications/
CSx000 Hardware Tour
What’s in an CSx000 Array? Similar components as an AF-Series
Back: Dual Power
Supplies (AC and DC
available)
Front: SSD Drives Front: HDD Drives Back: Dual Controllers

(CPU, Network)
Controller Head Shelf

Expansion Shelves (optional)

Drive Layout – CS1000, CS3000, CS5000, CS7000
 4U 24 Chassis  HDDs: 18 + 3 RAID

 24 x 3.5” Slots carry 21x HDDs + 3x DFCs » New Nimble-branded HDD carriers
 DFCs : minimum of 3 SSDs in Bank A
» Bank B available for cache upgrades
Slots 21-24
Slots 17-20
Slots 13-16
Slots 9-12
Slots 5-8
Slots 1-4
21X Disk Carriers
© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5a DC latch 2-8
Drive Layout – CS1000, CS3000, CS5000, CS7000
Slots 21-24
Slots 17-20 Power On

Heartbeat
Slots 13-16 NIC1/2
Power Over
Slots 9-12
Fault Temperature
Slots 5-8
Slots 1-4
DFC latch
Cache: 3x Dual Flash Carriers
Bank B Bank A latch
Bank A Bank A latch

Drive Layout – CS1000H: A half populated CS1000
First 11 HDDs
referred to as
CS1000H
Cache:
2x Dual Flash Carriers Fully
populated
CS1000H is
referred to as
CS1000FP
once upgraded
The WebUI will only display CS1000. To identify a CS1000H or CS1000FP look at the
controller shelf capacity or navigate to Manage >> Array >> [Select array] and view the visual
representation.

CS-Series Write Operations
CS-Series Write Operations
AF-Series features shared with the CS-Series

 Compression
 Stripe writes Variable Block
Compression
 Encryption
 Sweeping Variable Block
Sequential Stripe Write
Encryption
Sweeping

CS-Series Write Operation
8K 4K 4K
ack
Mirrored
5) Blocks are copied into DRAM
type of array
8K 4K 4K 8K 4K 4K
b) All Flash Array
1. …
c) Hybrid Flash Array DRAM
1. Variable block compression 8K 4K 4K
Variable Block
is applied Compression
Sequential Stripe 4.5MB
2. Blocks are formed into a
4.5MB sequential stripe
write
Index
3. Sequential stripe is written
to hard disk drives
CS-Series
4. Cache worthy data and any
data destined for pinned
volumes is also sent to SSD
5. Blocks are indexed
Index

CS-Series: Triple+ Parity RAID
CS-Series Triple+ Parity
 Utilizes the same Intra-Parity as the AF-Series

 Left synchronous rotation – first two parities (P & Q) are rotational and the
third parity (R) is non-rotational.
 Supports the loss of three disks.
Stripe 0 D1 D2 D3 P Q R
Stripe 1 D2 D3 P Q D1 R
The system will shutdown if
Stripe 2 D3 P Q D1 D2 R there are three disk failures
prior to any one of those failed
Stripe 3 P Q D1 D2 D3 R disks being rebuilt.
Stripe 4 Q D1 D2 D3 P R
Stripe 5 D1 D2 D3 P Q R
CS-Series Read Operations
CASL Architecture
Mirrored
1) Read from NVDIMM
2) If not found, check DRAM
3) If not found, read from SSD
• If found, validate checksum, 1 8K 4K 4K 8K 4K 4K
decompress, and return data

4) If not found, read from disk using the 2 DRAM
index on the SSD for a quick lookup
Variable Block
• If found, validate checksum, Compression
decompress, and return data
5) And, if cache-worthy, write to cache
3 4
The CS-Series arrays have the CS-Series
same changed block benefit
demonstrated in the AF-Series
arrays. 5
Index

AF-Series and CS-Series Architecture Review
AF-Series/CS-Series Write Operation Comparison
8K 4K 4K
ack
Mirrored
Blocks are copied into DRAM
type of array
8K 4K 4K 8K 4K 4K
DRAM
8K 4K 4K
AF-Series Sequential Stripe 10MB
CS-Series
Sequential Stripe 4.5MB

Index
Index
CS-Series AF-Series
Triple+ Parity with

Triple+ Parity Integrated Spare
Index
Index

Scale-to-Fit with the CSx000
Nimble Storage CSx000 Scale-to-Fit
CS7000 4 Node
230K IOPS CS7000
AFS Cluster
Up to 25.6TB
of flash Up to 920K IOPS
Multi-PB Capacity
CS5000
IOPS based on 120K IOPS
70% Read and
30% Write
workload CS3000
50K IOPS
SCALE UP
CS1000 /
CS1000H
35K IOPS
SCALE DEEP
Add expansion shelves to meet storage growth

CSx000 Scale Deep
SCALE DEEP
Add expansion shelves to meet storage growth

Scale Deep with the ES2 Expansion Shelf
The ES2 Expansion Shelf

Drive Layout – ES2
 4U24 Chassis
 24 x 3.5” Slots carry 21x HDDs + 3x DFCs
 HDDs: 18 + 3 RAID
» New Nimble-branded HDD carriers
 DFCs : Bank A pre-configured with 3 SSDs
» Bank B available for cache upgrades

CSx000 Scale Deep
Start with what is needed…
Scale deep as needs change
CSx000 supports up to 6 ES2 expansion shelves
CSx000 CSx000 CSx000 CSx000 CSx000 CSx000 CSx000

1X ES2 2X ES2 3X ES2 4X ES2 5X ES2 6X ES2

ES2 Rear View – Same as AFS2
SAS
Out port
SAS In
Expander FW Status
port
Green-OK
Red-FW Failed to Load SAS Link Status
Green-12Gbps Red-Fault
Orange-6Gbps Off-No link

CSx000 Expansion Cabling
CSx000 ES2
Controller Shelf Expansion Shelf
ES2 ES2
Expansion Shelf Expansion Shelf
ES2 ES2
ES2
Expansion Shelf

Activating a New Shelf (under Manage >> Arrays >> [select one array]
Activated
Once and expansion shelf is activated, it cannot be removed from the solution.

CS7000
230K IOPS
AFS
Up to 25.6TB
of flash
CS5000
IOPS based on
50% Read and
120K IOPS CSx000 Scale Up
50% Write
workload CS3000
50K IOPS
SCALE UP
CS1000 /
CS1000H
35K IOPS

Do you need to expand cache on your CS-Series array?
Nimble OS >> Monitor >> Performance
InfoSight >> Manage >>

Assets >> Cache tab

CSx000 Cache Upgrade
Increase On-Board Flash Cache

CSx000 Cache Upgrade
Populate bank B of all expansion shelves with

additional cache
Bank B

Adding / Removing SSDs
When an SSD is removed… When an SSD is added…
Cache Pool Cache Pool
… the Cache Pool shrinks by the size of … the Cache Pool grows by the size of
the removed SSD the added SSD
All-Flash Shelf
 SSDs can be ordered in 4-packs

» CSx000 supports up to twelve 4-packs for a total of 48 SSD
» CS1000H supports up to six 4-packs for a total of 24 SSD

CSx000 Expansion Cabling
CSx000 All-Flash Shelf

Controller Shelf
ES2 ES2
ES2 ES2
ES2 ES2

CSx000 Controller Upgrades
 Non-disruptive controller upgrades within CSx000 controller family

 Cache upgrades may be required to meet MFR, Memory to Flash Ratio
 Timeless Storage supported at launch
Upgrade Upgrade Upgrade

controllers controllers controllers
to CS3000 to CS5000 to CS7000
CS1000 CS3000 CS5000 CS7000

CS1000H “Half-pop” Controller Upgrades
 11TB must be upgraded to 22TB “Full Pop” before upgrading to CS3000

 Capacity and controller upgrades are non-disruptive
 No further controller upgrades beyond CS3000 are supported
 Timeless Storage supported at launch
Capacity Controller
upgrade to upgrade to
full-pop CS3000
C1000H-11T C1000H-22T C3000H-22T
The WebUI will only display CS1000 or CS3000. To identify a CS1000H-11T, CS1000-22T, or
CS3000H-22T look at the controller shelf capacity or navigate to Manage >> Array >> [Select
array] and view the visual representation.
Nimble Storage CS-Series Scale-to-Fit
4 Node
CS7000
CSx000 Scale out Cluster
Up to 1.4M IOPS
 Same capabilities as the AF-Series Multi-PB Capacity
» Add or remove arrays non-disruptively

» Automatically rebalances data
» Non-disruptively perform a technology refresh
» Migrates data to upgraded hardware
» 1-4 arrays per group
» Add and remove arrays on the fly
» Uses a single management IP
» Mix and match:
• CSx00
• CSx000
• AFx000

Module 03:
NimbleOS WebUI Introduction
Course Revision 5

 Access and navigate the NimbleOS WebUI
 Understand user administration
 Describe how to use Microsoft’s Active Directory with user administration
 Understand how to monitor the array with the WebUI
 Understand how to update NimbleOS

Nimble OS WebUI Tour
Accessing the WebUI Interface
https://{arrays management IP address}
https://10.206.9.110

WebUI Tour
Capacity
Events
Performance

WebUI Tour

User Administration
User Administration
Administration >> Security >> Users and Groups

 RBAC = Role Based Access Control
» Multiple Users
» Pre-defined Roles
» Default Username: admin
» Support for Microsoft
Active Directory

Roles
 Administrator (Default “admin” user)

» Complete access and security capabilities – no
limits.
 PowerUser
» Same as Administrator but without the privilege to
modify security settings. Eg., no user account
creation and password changes.
 Operator
» Same as PowerUser but without the privilege to
delete data and operations that are rarely
performed. Eg., volume delete.
 Guest
» Read Only privilege except for security settings.
See Appendix A of the Nimble Storage User Guide for a complete listing of capabilities by command

User Management
 Usernames can’t be any of the system users like “root”

 SSH key based password-less login is supported for all users in the Administrator role
 All user accounts except for built-in Administrator can be enabled/disabled and removed

Joining Microsoft Active Directory
1 Ensure AD has an existing

group for use with the array.
Create a new AD group if

>
needed. New >> Group



>
> Provide the group name, set

the scope, and group type.



>
> Provide the group name, set

the scope, and group type.
> Add group members.

2 Navigate to Microsft Active Directory in the WebUI

3 Complete all fields
Click Join 4

5 Test the connection

Assigning Users/Groups to Microsoft Active Directory
1 Navigate to Users and Groups in the WebUI

2 Click Add and then select User or Group

3 Provide name, desired role, description (optional) and desired inactivity timeout value

Microsoft Active Directory
3 Test the login

Introduction to Monitoring With the WebUI
Monitor >> Space
Review Select what

overall data to
system usage display
Open Review data

InfoSight for reduction
predictive savings
analytics of
space
utilization
Set timeframe
Display
and pause
usage,
the capture
savings, or
for analysis
both
Breakdown of
usage and
savings by
application
category
Monitor >> Performance
Pause data Select

stream timeframe
and volumes
to display
View performance
metrics for selected
timeframe on selected
volume(s)
Launch InfoSight
CS-Series – monitor to view a
cache hit/miss. If miss breakdown of
is high, upgrade cache latency factors

Monitor >> Interfaces
Pause data
Select
stream
timeframe to
display
View performance
metrics for each
interface

Monitor >> Replication
 Monitor replication throughput for

both send and receive

Monitor >> Connections
 View the initiator and number of connections for each volume Mouse
over
connection
number to
view
Mouse
connection
over
addresses
volume to
view
details

Monitor >> Audit Log
 View activity of each WebUI user
Filter audit log

Manage >> Arrays >> [Select Array]
 Graphical display of array health

CSx000
AF-Series

Events
 View list of system events and full event description
Filter event log

Updating NimbleOS
Controllers – Nimble OS Upgrade
One-click, zero-downtime Nimble OS upgrades

 Before you begin:
– Check your current version
– Obtain the most recent version
– Check system health

NimbleOS Upgrade Process
1. Load new firmware to standby

2. Reboot standby to run new rev.
Firmware
3. Load new firmware to other controller
4. Reboot active to activate new rev. – causes failover and the
standby becomes active
Controller A Controller B
Active
Standby Standby
Active
Failover
Module 04:
Introduction to Customer Support & InfoSight
Course Revision 5

 Present Nimble Support’s mission statement
 Understand the InfoSight Customer portal
 Describe support coverage and logistics

Introduction to Support
Customer Support Mission Statement
Our Mission
Maintain a maniacal focus on providing the
industry’s most enviable Customer Support
 It’s not just a Customer Support mission
 Its part of Nimble’s DNA across the board
 We monitor, assess and evolve as fast as necessary

Why Do We Do Things This Way?
 We believe users should expect data to be available always

and instantly, regardless of media type.
 We believe in lowering risk and accelerating business
outcomes.
 We believe solutions should be easy to consume.
 We believe solutions should not have architectural dead ends

Consider These Issues…
 All your hosts are sporadically disconnecting from VMware.

Nobody can figure out why.
 A VM host experiences high latency. But only after a certain
time.
 An application has abnormally slow I/O speeds, regardless of
how fast the storage it’s attached to.
 A server exhibits unstable behavior unpredictably.
 App data growth is much faster than anticipated

Consider These Issues…
 All your hosts are sporadically disconnecting from VMware.

Nobody can figure out why.
 A VM host experiences high latency. But only after a certain
time.
How Are You
 An application Solving Such
has abnormally slowProblems
I/O speeds,Today?
regardless of
how fast the storage it’s attached to.
 A server exhibits unstable behavior unpredictably.
 App data growth is much faster than anticipated

InfoSight Customer Portal
InfoSight Engine: Data Collection
Comprehensive Telemetry
Comprehensive Telemetry
• Log
 Log filesfiles
• Sensors (>30Million per day!)
 Sensors (>30Million per day!)
• Health status
 Health status
5-minute Heartbeats
• Array <------> Nimble Support
InfoSight
• Proactive Monitoring
Engine
>30M sensor values per array per day

Collecting sensory data
Reads vs Writes Temperature
Network Statistics
CPU Utilization
Snapshot status Write Latency
Statistics
Write IOPS
Compression
Cache Utilization Read IOPS

PS Voltages
Replication status
Fan Speeds
Read Latency
Data block size Block Sizes
Random vs Sequential IOPS

Preventing Problems to Avoid Downtime
 9 out of 10 problems are automatically detected

 Rapid root-cause analysis cuts hours of
troubleshooting
 Correlated analysis enables visibility across
the application-to-storage stack
 No level-1 and level-2 support pain Measured
uptime
InfoSight predicts and prevents issues across the application-to-storage stack,

enabling you to avoid disruption and time wasted dealing with infrastructure problems

Real-time monitoring & alerts
Instant Email Alerting

 Easy and flexible setup
 System alerts to Nimble Support and the
customer
» All warning level and alerts
» Proactive disk monitoring
» System utilization
» Protection (Snapshot) monitoring
From: <customer>
Sent: Monday, June 20, 2011 2:03 PM
 Automated case created To: alerts
Subject: Nimble Alert on <array name> - WARNING: System temperature is warm
 Nimble support proactively Time: Mon Jun 20 14:03:01 2011
contacts customer
Type: 12323
Id: 76501
 Ability to ‘remote’ in for real- Message: current temperature for sensor bp-temp1 at backplane left side is 44 C
time troubleshooting
Array name: <name>
Serial: <AA..>
Version: 1.0.6.1-9650-opt
Visibility Beyond Storage
InfoSight VMVision gives

visibility up to the VM layer
• Determine VM latency
factors: storage, host
or network
• Take corrective action
on noisy neighbor VMs
• Reclaim space from
underutilized VMs
InfoSight VMVision pinpoints VM-related issues
Predict Future Needs and Simplify Planning
 Accurately forecasts future capacity,

performance and bandwidth needs
 Prescriptive guidance ensures optimal
long-term performance
 Predicts performance hotspots and
tells you how to avoid them
 Eliminates planning guesswork
Leverage predictive analytics to identify future needs and potential hot-spots specific to
your environment, with prescriptive guidance to ensure optimal long-term performance

Visualize Intuitively
• Executive dashboards give you

peace-of-mind that everything is
running perfectly
• Correlated visualization provides a

use-case view of what’s happening
across the “applications-to-storage”
stack
• Manage your global storage

environment from anywhere
Complete visibility through the cloud to all information you need to maintain a resilient
environment and ensure smooth operations

InfoSight: A Customer Portal and Much More
Blacklisting and
Dynamic Upgrade
Paths
Software & Takeover Fault Behind-the-Scenes

Analysis Operational Monitoring
Support Aggregate
Automation Studies

Proactive Wellness: Powered By InfoSight Data Sciences
Proactive Support Prevents Problems
of the cases automatically

> 93% opened by Nimble
field proven availability

across all GA software
99.999% releases
of support cases auto-

86% closed
by Nimble

Support Coverage and Logistics
How to get level 3 support – the traditional method
Call Fill out verbal Caller motives

Level 1
Support questionnaire questioned
Answer more Wait on hold

Provide
questions, provide Level 2 or wait for a
logs
logs. Again. call back
Wait on hold
Caller motives Welcome to level 3
or wait for a
questioned support
call back
Level 3 support as easy as 1-2-3
nimblestorage
1 Pick up the phone 2 Dial 1-877-364-6253 3 Level 3 support

Global Support Centers – Seamless coverage
EMEA
Reading, UK
AMERICA’S HQ
San Jose, Ca RTP Japan
Raleigh, NC Tokyo
APAC
Local toll free numbers Singapore
land in any of 5 Centers
around the globe seamlessly
7 days a week, 365 days a year.
Note: Japan install base is handled by TSC.

Support for SmartStack
 Nimble SmartStack Integration Support

• Support.SmartStack@nimblestorage.com or Option 7 from 1-877-3Nimble
• Coordinate third-party vendor support
• No additional charge
 Cisco Solution Support for Critical Infrastructure

• Cisco will offer issue management and solution expertise
• Will coordinate with Nimble and other vendors for support
• Fee-based option on top of product support contract
 All configurations validated and supported by participating vendor partners

• Product Certifications
• Hardware Compatibility Lists
 Backed by TSAnet as global alliance framework that enables fast-track

technical support cooperation between several partners
• SLA’s are required for each vendor and cases are opened by customer

Service Logistics Regional Capabilities
 Next Business Day – “NBD”  4 Hour Parts Delivery  4 Hour Parts Replacement - Onsite Engineer
• United States
• United States
• Canada
• Canada • United States
• Europe
• Europe • Canada
• Australia
• Australia • Europe
• New Zealand
• New Zealand • Australia
• Bermuda
• Bermuda • New Zealand
• China
• China (Major Cities) • China (Major Metros)
• Hong Kong
• Hong Kong • Hong Kong
• Indonesia
• Indonesia • Indonesia
• Israel
• Israel • Israel (Tel Aviv)
• Kenya
• Thailand • Thailand
• Thailand
• United Arab Emirates • United Arab Emirates
• Nigeria
• Singapore • Singapore
• United Arab Emirates
• Malaysia • Malaysia
• Singapore
• Philippines • Philippines
• Malaysia
• South Korea • South Korea
• Philippines
• India (Major Cities) • India (Major Cities)
• South Korea
• South Africa (Johannesburg) • South Africa (Johannesburg)
• India
• Taiwan • Taiwan
• South Africa
• Vietnam (Major Cities) • Vietnam (Major Cities)
• Taiwan
• Vietnam
How do we decide on new locations?
• We look at sales/partner coverage in the area
• Anticipate a minimum of 5 installed systems in the area within next 12 months
• 10 to 14 days to open a new Depot in most locations around the world
Module 05:
Pre-Installation, Racking, and Cabling
Course Revision 5

 Use the pre-installation workbook
 Understand rack mounting principles
 Cable an array
 Understand networking and fibre channel concepts

Pre-Installation Workbook
Pre-Install Checklist
 Collect all necessary data to perform an installation

 Organized in the same order that you will be entering in the data
 Can be left with the customer

Pre-Install Checklist

Racking
Use the Quick Start Guide
Download Quick Start Guides from InfoSight before starting

Racking
AF = 4U CSx000 = 4U
AFS2 = 4U AFS = 4U
ES2 = 4U

Racking
Precaution
 Ensure there is at least 24” of cable slack
at the back of the array or shelf.
» Allow enough room to slide the array or
expansion shelf out 12” from the front of the
rack in order to replace a center chassis fan.

Cabling
Power
Connect one to
commercial power and
one to backup power.

Networking Terminology
 Interface Pairs
»Controller A eth1 & Controller B eth1
»IP addresses float between
Interface Pairs
Ethernet Ports
eth5 eth6
eth1 eth2 eth3 eth4 SAS Out Ports
When cabling to switches ensure you cable even

ports to one switch and odd ports to the other.

Fibre Channel Ports
fc5 fc9 fc10 fc6
eth1 eth2 fc1 fc2 SAS Out Ports

Networking Concepts
Understanding IPs
 Management Network
» The Array Management IP address
» The 2 Controller Diagnostic IP addresses
 Data Network
» The Data IP addresses
 Discovery Addresses
» iSCSI Discovery Address

Discovery Addresses
 Typical iSCSI Discovery

iSCSI
Send Targets
Discovery
» Returns a list of Targets from the Array

» Makes it easy to find Volumes from the Host
Targets
Volume1
Volume2
Volume3
 Nimble iSCSI Discovery (per Subnet)

» Acts as a load balancer (Automatic Mode) Connect to Data IPs
» Redirects to a Data IP address to be used to connect to the array Discovery
.20 .21
(172.16.0.20) .22
• Used to virtualize connections to each array in the group 2 times
• Used by the array to distribute/balance hosts across several data IPs
» Need a Discovery for each Data Subnet
Data Addresses
• Discovery cannot redirect across subnet boundaries 172.16.0.21
Redirects & Returns 172.16.0.22
from Array:
172.16.0.21 & 17.16.0.22

Typical Subnets
One Shared Network Two Dedicated Networks Advanced Networks

Management Subnet Management Subnet
• Array Management Address • Array Management Address
• Acts as Discovery IP • Acts as Discovery IP
• 2 Controller Diag Addresses • 2 Controller Diag Addresses
Management • Replication (by default, 2.x) • Replication (by default, 2.x)
• Array Management Address
• Acts as Discovery IP
• 2 Controller Diag Addresses
• Replication (by default, 2.x)
Data Subnet Data Subnet Data Subnet

Data • All Data Addresses • 1 Set (Data) • 1 Set (Data)
• All Data Addresses • Discovery IP • 1 Discovery IP • 1 Discovery IP
Note: in 2.x, every array in the group must have access to each subnet
iSCSI Switch Selection Guidelines
Network Design
 Best if dedicated, redundant iSCSI network
 Otherwise, use VLANs to keep iSCSI traffic separate
Switch Attributes
 Good Quality Layer 2 or Layer 3 Managed
» Stacked preferred, but be aware of issues around stacking such as what happens when the
master switch in the stack fails and upgrading of switch firmware may require an outage
» ISLs at least, concern is under or over specifying the total bandwidth required
» ISLs – may want to use IP Address Zones to prevent iSCSI traffic from crossing the ISL
 Support for Jumbo Frames (with Flow Control) is Desirable
 Non-Blocking Backplane
» Bandwidth of the backplane >= (# of ports) * (bi-directional port speed)

Optimizing Switch Links: IP Address Zones
 Challenge
» How to minimize Inter-Switch Links for traffic? 172.16.0.81 172.16.0.82
eth1 eth2
» Especially if connections are automatic? Discovery redirects to Discovery redirects to
172.16.0.21 172.16.0.22
Single Data Subnet
 Answer: IP Address Zones
» Discovers Data Address via Discovery IP
» Uses Host IP address to select Data Address Avoid
» Examples: Traffic?
• Even/ Odd: Odd vs Even #s kept together 172.16.0.21 172.16.0.22
(Data IP) (Data IP)
• Bisect: Low vs High #s are kept together
• Default: Single (no division)
Using Odd/Even zone .21 Management
.22
Subnet
172.16.0.20 .20
(Discovery)

Best Practices
Practice Notes
Do not use Spanning Tree Protocol Do not use STP on switch ports that connect to iSCSI initiators or the
(STP) or use PortFast Nimble storage array network interfaces.
Configure flow control on each switch Configure Flow Control on each switch port that handles iSCSI
port connections. If your application server is using a software iSCSI initiator
and NIC combination to handle iSCSI traffic, you must also enable Flow
Control on the NICs to obtain the performance benefit.
Disable unicast storm control Disable unicast storm control on each switch that handles iSCSI traffic.
However, the use of broadcast and multicast storm control is
encouraged.
Use jumbo frames when applicable You must have jumbo frames enabled from end to end for them to work
correctly
Testing Network connectivity Use the Ping command to test network connectivity and to help
determine if jumbo frames are enabled across the network, example:
vmkping -d -s 8972 x.x.x.x

Fibre Channel Concepts
Fibre Channel Components
Dual Fabric Design Array (Target)
Configure one FC port on

controller A to FC Switch A and
one FC port on Controller A to FC
Switch B
Configure one FC port on 2 3 4 2 3 4
controller B to FC Switch B and CISCO UCS
ID
6296 UP CISCO UCS
ID
6296 UP
one FC port on Controller B to FC

STAT 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 STAT 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
FC Switch A FC Switch B
Switch A
All FC ports must be in the same
S
A
S
S
A
PCIe3
PCIe4
PCIe1
CIMC
PORT 1
PORT 0
S
S
A
PCIe5
PCIe 2
zone
PSU
2
650 W AC
1 2
PSU
1 
M
Server (Initiator)

Fibre Channel Config Steps
1. Verify Fabric/Zoning
» Single Initiator
» Single Target
2. Configure Initiator Group
» Server port
» Volume
3. Create volume and link to the initiator group
4. Verify connection from volume to assigned initiators

Module 06:
Array Initialization & Setup
Course Revision 5

 Perform an array initialization
 Initialize an array configuration
 Understand port and firewall considerations
 Understand SNMP setup
 Understand SYSLOG use
 Understand NimbleOS’s event log

Array Initialization
Before You Begin
Important: The computer used to initially configure the array must be on the same
physical subnet as the Nimble array, or have direct (non-routed) access to it.
• Ensure Adobe Flash Player is installed

Prerequisites
 Before launching the NWT:
» Set a static IP: Set your IP address to the same subnet that your array management IP address
will be on.
» Have your array controllers A & B correctly cabled to your switch fabric per the previous
drawings.
» Complete all your switch configurations for Flow Control, Jumbo Frames, Spanning tree,
Unicast, etc.
» Install the Nimble Windows Toolkit (NWT) on the Laptop or Server you are using for the
installation.

Nimble Windows Toolkit Installation
 NWT 3.3 includes

» Nimble Set-Up Manager (NSM)
• Available via InfoSight
» Nimble VSS Requestor
» Nimble VSS Provider
» Nimble Connection Manager
(NCM)*
» Nimble Connection Service
(NCS)*
» NimbleDSM*
* Windows Server 2008 R2 or 2012 /

MPIO
Reboot Required after Install

NWT– Nimble Array Setup Manager
Shows
• Array(s)
• Model
• NOS Version Number
Steps
• Select Array to Setup
AF-000917 All Flash Array
• Click Next Button
• The next Popup screen informs
you that Setup Manager is going AF-000917
to connect using the IP address

indicated in the Popup

End User Agreement
Accept the End User

Agreement by clicking in
the checkbox and the
“Proceed” button

3.x Arrays, Select “New Group” or “Add Array”
Option 1: New Group

» Configure first array
» Mgmt IP, etc.
» See next step
AF-000917
All Flash Array
Option 2: Add To Group
» Inherits group’s identity
& network
» See Multi-Array Section
for example

Enter the array name

• Make it useful such as
• Row and rack number AF-000917
• Data center & app
Set your management IP address Subnet

mask & Default Gateway
Enter and confirm your array password
Click “Finish”

You should get this screen

after a few seconds.
Click “Continue”, your default

browser window will be
opened and directed to the
management IP. If it does
not, open a browser and
point it to the management
IP address

Array Initialization Using the CLI
Array Initialization
The steps to initialize an

array using the CLI are:
1. Connect to the array

using the provided USB
dongle and a terminal
program
2. Login to the Array
3. Run setup and use the

installation workbook
to answer presented
questions

Initial Array Configuration
Initial Array Configuration
1 Launch the WebUI and Login

Configure Subnets
2 Click “Add Data Subnet” to add a subnet for your data network or networks

Adding Data Networks
3 If you have additional data networks you want to add, click the “Add Data Subnet” again.

Interface Assignments
4 Assign Interfaces to desired ports

Domain & Time Configuration
5 Configure the domain and DNS server(s)
6 Configure time zone and NTP Server
Next add the domain, DNS, Time Zone &

NTP server information
Configure Support Information
7 Configure email alerts, auto support, and HTTP proxy
The ‘From’ address should identify the array
Add one or more destination

addresses
Add your SMTP Server information

NOTE:
Additional SMTP
Server settings are
available after Check “Send AutoSupport …”
initial
configuration
Select Alert Type
Click ‘Finish’ to complete setup
Setup Complete

AutoSupport Setup and Testing
1 Navigate to AutoSupport/HTTP Proxy
Administration >> Alerts & Monitoring >>
AutoSupport/HTTP Proxy

AutoSupport Setup
2 Enable AutoSupport and test
Check – Send AutoSupport data to Nimble Storage
Click – Test AutoSupport Settings

AutoSupport Setup
Once the “Test

AutoSupport
Settings” have
completed a green
or red dot will be
displayed indicating
the test status.

AutoSupport Setup
Clicking on the dropdown

will reveal all the tests
that were performed.
If one or more tests

failed, you will see a
“red” dot to the left of the
test that failed.
Send the AutoSupport to Nimble Support

AutoSupport Setup

Setup Email Alerts
1 Navigate to Email Alerts
Administration >> Alerts & Monitoring
Email Alerts

Setup Email Alerts
2 Complete all fields and test
Add your SMTP Server information
Select Alert Type
Click “Test”
Setup Email Alerts
3 After testing Email Alerts, click the “Save” button to save the settings

Controller Failover Check
1 Navigate to Manage >> Arrays
Manage
Arrays

Click the Array 2


4 Confirm the message

5 Verify the Standby Controller is now set to “Active” and that all connections are good.

Port and Firewall Considerations
Incoming Server Ports
To use: Open local For local IP addresses: Notes:

port:
SSH 22 Array management IP
HTTP 80 Array IP
GUI (HTTPS) 443 Array management IP HTTP (port 80) communication is
redirected to HTTPS
iSCSI 3260 Discovery and data IP Needed for data access

SNMP 4290 SNMP daemon
GUI charts, NPM 4210 Array management IP
Control 4211 Array management IP

Replication (data) 4213 Array management IP
See Appendix B in Nimble Storage Installation and Configuration Guide 2.2 for complete listing
Outgoing Server Ports
Port # Service Protocol IP Address
N/A Manual SCP of cores to Nimble SCP support.nimblestorage.com
N/A AutoSupport, heartbeat HTTPS nsdiag.nimblestorage.com
N/A alerts* HTTPS nsalerts.nimblestorage.com
N/A software downloads HTTPS update.nimblestorage.com
2222 secure tunnel SSH hogan.nimblestorage.com
25/UDP & TCP** SMTP SMTP SMTP server IP
*An array sends alerts through HTTPS POST back to Nimble Support if AutoSupport is enabled
**Default (configurable)
SNMP
Setup SNMP
 SNMP versions 1, 2 and 2c
 Configure through the NimbleOS GUI (Administration > SNMP) or CLI snmp command
 Nimble Storage arrays send traps but do not receive them
 UDP traps are supported
Arrays use the alert level setting for email alerts to determine the
events that are sent as SNMP traps
SYSLOG
SYSLOG
 Nimble Storage arrays use the standard SYSLOG protocol
 Support for Red Hat Enterprise Server and Splunk implementations of SYSLOG
 Nimble Storage SYSLOG stores

» records of administrator manipulation of the array
» history of alerts
» array issues
 SYSLOG files can be shipped from an array to a centralized, remote server

Event Logs
System Event Monitoring
 Events are displayed on the Events page of the NimbleOS GUI

» Access by clicking “Events” located on the main NimbleOS menu selection

LAB 1
Tasks:
 Launch Nimble Setup Manager (NSM)
 Recall key configuration steps
 Launch array GUI and configure basic array parameters including autosupport
Time to complete: 20 Minutes

LAB 1b (Optional)
Tasks:
 Utilize the CLI to complete initial setup
Time to complete: 15 Minutes – this is an optional lab

Module 07:
Working with Nimble Storage Volumes
Course Revision 5

 Understand basic volume concepts
 Understand performance policies and how to create one
 Understand access control and initiator groups and how to create them
 Understand protection templates and how to create one
 Understand volume collections and how to create them
 Understand volume pinning
 Create a volume using the WebUI

Volume Concepts
Volumes Overview
Physical storage
resource
Volume Logical storage

resource
Triple+ Parity
Storage Pool

Thin Provisioning
Triple Parity
RAID or Triple Consumed Space
Volume
Parity+
Space from the pool is

consumed as data is
written

Volume Reserves
Triple Parity
Volume
Volume Reserve
RAID or Triple
Parity+
A reservation reserves a
guaranteed minimum amount
of physical space from the
pool for a volume
Volume Quotas
Volume Reserve
Volume Quota
Volume
Pool
A quota sets the amount of a volume that can be consumed before an alert is sent and
writes are disallowed.
Snapshots
Snapped Volume
Snapshot Reserve – An accounting for a
set amount of space that will be
guaranteed available for the snapshot.
Snapshot Reserve
Snapshot Quota – An accounting for the
Snapshot Quota
total amount of space a snapshot can
consume.

Performance Policies
What is a Performance Policy?
 A performance policy is used to define parameters for volumes assigned to the

policy:
» Pre-configured policies can
be used or a custom policy
can be created.
» NOTE: AF-Series arrays
performance policies include
a column for deduplication
AF-Series Performance
Policy list

Custom Performance Policies
 The following parameters are set when building a custom performance policy:
» Application Category
» Storage Block Size
» Compression On/Off
» Caching On/Off
» Space Management Parameters
» Deduplication Enabled/Disabled
- AFA only

Access Control / Initiator Groups
Access Control
 Access control – which hosts have access to a volume based on an initiator

group
 Best Practice: Always limit access to a host initiator group
 NOTE: Allow multiple initiator access – for use with clusters, not MPIO
Fibre Channel iSCSI

Initiator Groups
 A set of host initiators that can be assigned access to a specified volume via Access
Control.
 Can be created at volume creation or as a separate task
» Manage >> Initiator Groups

Initiator Groups
 The parameters within an Initiator Group will depend on the protocol - iSCSI or Fibre Channel
iSCSI Initiator Group Fibre Channel Initiator Group
NOTE: An IQN can only be assigned to one initiator group

iSCSI Initiator Groups
• Initiator Groups are applied to:

• Volumes
• Volumes+Snapshots
• Snapshots only
• Can limit to specific subnets
• Supports up t0 255 subnets

iSCSI Initiator Group Target Subnets
 Limiting Subnets
» Only targets on specified subnets can be
accessed by an Initiator Group

Fibre Channel Initiator Groups
• Initiator Groups are applied to:

• Volumes
• Volumes+Snapshots
• Snapshots only
• Limit based on World Wide Port

Name, WWPN
• Use of a Fibre Channel fabric
alias is supported which then
automatically completes the
WWPN

Volume Collections
Volume Collection
 A grouping of volumes that share snapshot/replication

schedules
» Manage >> Protection >> Volume Collections
» Created via a blank schedule or Protection Template
» All volumes in a group will be snapped and replicated as a
group
 Best practice: Create a volume collection for each
application
Volume Collection
» Oracle Database and log files
 Ensure you do not create overlapping schedules

Creating A Volume Collection
 Name the Volume Collection
 Using a Protection template (for

Schedules)
» Or create a custom schedule

Volume Collection >> App Synchronization
• App flushes/quiesces I/O

while we take a snapshot
and then unfreezes
• VMFS consistent snapshots
• SQL consistent snapshots
• Exchange consistent
snapshots
• SQL/Exchange uses MS
VSS framework and
requires NPM on the
Application Host – more
later
Add Schedules
 Sets Snapshot Schedule

» Include Retention Policy
» App Synchronization
» Replication
 Create Multiple Custom Schedules

» Examples
• Hourly
• Daily
• Weekly
» Do not create overlapping schedules

Volume Pinning – CS-Series Only
Volume Pinning – CS-Series Only
 Manage >> Volumes >> [Selected

Volume] >> Edit >> Performance
 Pins the entire volume in cache
» Writes are still held on disk
 Should only be used on workloads that
require peak performance
Cache is a shared resource. Pinning a

volume may impact the performance of
other volumes.

Creating a Volume Via the WebUI
View of Volume Wizard
 Same wizard is used to create both Fibre Fibre Channel Volume

Channel or iSCSI
 Creates protected volume in 4 easy steps
» General Properties
» Space
» Protection
» Performance
 Build volume components separately or as
needed
» Initiator Groups
• Fibre channel and iSCSI dependent
» Custom Performance Policies
» Volume Collections
• Custom Protection Templates
iSCSI Volume

Volume Create Wizard: Step 1 – Performance & Access
 Volume Name
» Helps to include host and app in
volume name
 Performance Policy
» Use existing policy (based upon
app type)
 Set Access Control

» Best to include at least 1 initiator group
» NOTE: Multiple initiator access is for

apps such as clustering, not MPIO.

Volume Wizard: Step #2 – Size & Space
 Size
» Set the volume size seen by the
application
Click to
 Thresholds expand
» Best to use the defaults
» Setting Reserve > 0 reduces thin
provisioning

Volume Wizard: Step #3 - Protection
 Protection
» Best to use a volume collection for all
of your volumes
 Protection Schedules
» Displays Protection Schedule
associated with selected volume
collection

Volume Wizard: Step #4 - Performance
 Caching
» Normal – cache all hot data
» Pinned – cache entire volume

Viewing Volume and Replica Usage
Volume
Filter
Volume Volume
Summary Usage Key
 Under “Manage > Volumes”

» Displays all pertinent volume data

Nimble Windows Toolkit & Nimble Connection Manager
Nimble Windows Tools
 Download NWT from InfoSight

 Prerequisites must be installed prior to NWT or the installation will stop
» .NET Framework 4.5.2 or above
» Microsoft PowerShell 3.0 (required to run Nimble PowerShell Cmdlets)
» MPIO enabled
» Windows Server hotfixes must be applied (the installation will inform you which hotfixes are
needed)
 What is installed:
» Nimble Setup Manager (NSM)
» Nimble Connection Manager & Connection Service (iSCSI)
» Nimble VSS Requestor/VSS Hardware Provider (iSCSI & Fibre Channel)
» Nimble Device-Specific Module (DSM) for MPIO
» Nimble Diagnostic Utility
» Microsoft Windows PowerShell module
» Nimble Hotfix Monitor Service

Nimble Connection Manager (NCM)
 Simplifies making and maintaining the optimal number of iSCSI connections

between host and array
 Requires that the “Automatic iSCSI Connection Method” be used
» Found in the Network and Configuration Group settings
 NCM performs the following tasks:
» Gathers interface, subnet, and volume information
» Calculates the optimal number of connections
» Determines which host network interfaces are in the same subnets as the array network
interfaces
» Attempts to make the optimal number of connections
» Displays the connections and all necessary information so users can easily verify connections
are optimal

Nimble Connection Manager for VMware
 Install onto each ESX host

 Prerequisites
» ESXi 5.x or 6.x with Enterprise or Enterprise Plus
» vCenter Server 5.x or 6.x
» Root access to ESX host(s)
» 15 minute window when ESXi host can be offline
 Install using vSphere Update Manager or ESXCLI
• Use the correct package for your version of ESXi (NCM for ESX 5.x or NCM for 6.x)
 NCM for vSphere 6.0 or higher required to support VVols on iSCSI arrays
 What is installed:
» Nimble Connection Service (NCS)
» Nimble Path Selection Plugin (PSP)
Note A fresh installation of NCM on ESXi 5.x does not require a

reboot. However, new installations of NCM on ESXi 6.0 do
require a reboot.
Nimble Connection Manager for Linux
 Linux NCM
» RHEL OS versions 6.5, 6.7, 7.0, 7.1
» Ensure that connection redundancy is always maintained to the Nimble array.
» Manage multipath connections (at the I/O region level) to volumes striped across multiple
arrays.
» Configure block device level settings for optimal performance.
» Automatically manage iSCSI and multipath configuration.
 Prerequisites
» sg3_utils and sg3_utils-libs
» device-mapper-multipath
» iscsi-initiator-utils (for iSCSI deployments)
 Linux NCM is bundled into a single executable binary called ncm_installer

LAB 2
Tasks:
 Build a volume
» Configure an initiator and an initiator group
» Create a volume collection
» Configure protection schedules
Time to complete: 15 minutes

LAB 3 (Optional)
Tasks:
 Connect to a host
» Launch NCM
» Configure the Windows host via NCM
» Connect and examine the new volume
» Prepare and mount the volume on a Windows host
Time to complete: 15 minutes – this lab is optional.

Module 08:
Introduction to Nimble Storage SmartSnap
Course Revision 5

 Describe how Nimble snapshots work
 Understand snapshot scheduling
» Need to add RPO RTO definitions
» Define Change rate
 Understand recovering from a snapshot
» Clones Zero Copy Clones

Understanding Snapshots
Snapshots
What is a Snapshot?
Snapped Data
New Data (non-snapped)

COW Snapshots
Changed Block
Snapped Data
New Data (non-snapped)
Snapshot Reserve

ROW Snapshots
Changed Block
Snapped Data
New Data +
Changed Blocks

File and Snapshot Lifecycle 09:00
filename A B C D State of data at 09:00 filename 4-block file created

A B C D

filename A B C D State of data at 09:00 filename 4-block file created

Snap10 A B C D 10:00 snapshot A B C D
10 snap!

filename A B’ C D State of data at 10:20 filename 4-block file created

Snap10 A B C D 10:00 snapshot A B C D B’
10 snap!
If block B is changed, the original state
can be recovered by rolling back to the
snap taken at 10:00

File and Snapshot Lifecycle 11
filename A B’ C D State of data at 11:00 filename 4-block file created

Snap10 A B C D 10:00 snapshot A B C D B’
Snap11 A B’ C D 11:00 snapshot
10 11
10 snap!
The next snap taken captures the
change made to block B
Any snapshots can be used to recover from without the loss of snapshots taken before or
after the snapshot being recovered from.

Basic Terms
Things to consider and DR Terms
RPO – Recovery Point Objective

• defined by business continuity planning. It is the maximum targeted period in which
data might be lost from an IT service due to a major incident. The RPO gives systems
designers a limit to work to.

RTO – Recovery Time Objective

• is the targeted duration of time and a service level within which a business process
must be restored after a disaster (or disruption) in order to avoid unacceptable
consequences associated with a break in business continuity.

Synchronous Replication –
• the process of copying data over a storage area network (SAN), local area network
(LAN) or wide area network (WAN) so there are multiple up-to-date copies of the data.
RPO = Zero
Zero Data Loss

Asynchronous Replication –
• The write is considered complete as soon as local storage acknowledges it. Remote
storage is updated, but with some lag. Asynchronous Replication is a schedule based
event. Lost Data ≠ 0

Change Rate –
• Is the amount of data that is changed/modified in a given period of time
• Note: the higher the change rate, the more bandwidth you may need to ensure
RPO/RTO can be met.
Changing Data
A A’ A’’ A’’’
Time

Understand Snapshot Scheduling
Volume Collections and Schedules
 Snapshot Schedules
» Determined by Volume Collections which
include the following parameters:
• Schedule name
• How often the snapshot should be taken
• Timing of the snapshot
• Which days to run the snapshot on
• Number of snapshots to retain

Taking a manual snapshot
1. Manage >> Volumes >> [Select desired volume]

2. Click “Take Snapshot”

3. Enter a name for the snapshot and select desired Status and Writability settings.
4. Click OK to immediately take a snapshot

5. Click the Snapshot tab to view the snapshot.

Snapshot Status
Manage >> Protection >> Volume Collections >> [Select a volume collection]

“New Data” is not Snapshot Capacity
This is not the capacity being retained by a snapshot

This is the “new data” since the prior snapshot.
There is only one place to see snapshot capacity
- It is for ALL snapshots of the volume, not broken out by individual snapshot

Understand Recovering from a Snapshot
Recovery Scenarios
 Recovery from local snapshots

» Single volume, volume collection
» Replacing entire volume
 Testing my DR site without interrupting replication

» Use of clones

Zero Copy Clone
 Snapshots are ROW

» Snapped data is held as a single dataset
» New writes are directed to available space in the storage pool
 Zero copy clone
» Allows a volume to be created for online use based on a snapshot
» Any changed data is handled like a ROW snapshot
» Occupies no additional space until new data is written or changed
Pointers

Recovery Scenarios – Recovery from local snapshots
Clone the snapshot (creates a first-class volume)
1 3

LAB 4
Tasks:
 Recover from a Snapshot
» Simulate a data loss event
» Create a zero-copy clone
» Connect to the clone and recover data
» Disconnect and delete the clone

Module 09:
Introduction to Nimble Storage
SmartReplicate
Course Revision 5

 Describe basic replication concepts
 Understand SmartReplicate components
 Explain how replication works
 Configure Replication
 Understand the replication considerations
 Perform SmartReplicate Disaster Recovery

Replication Introduction
Enterprise Level Data Protection, Efficiency & Security
 Options, flexibility & effortless management with all-inclusive packaging
SmartReplicate:
• Efficient (thin, block diffs+ data
reduction)
• WAN optimized
• Secure (AES-256bit encryption)
• No license required
CS700
DR
AF9000
Production

Data Protection & Copy Data Management
Others Nimble Storage
D2D + Replication Snapshots + Replication
Tier 1 Tier 2 Fast, cost-effective hybrid storage
Primary 9:00
9:15
9:30
9:45
 No backup window
 Rapid local recovery
Backup  Cost-effective, simple DR
Tier 3
(+Dedupe)
9:00
Disaster Tier 3 10:00
Recovery (+Dedupe) 11:00
12:00
 Instantaneous zero-
Space 9:00 copy clones (e.g., for
efficient 10:00 test and dev instances)
11:00
clones 12:00

Integrated Data Protection Efficiency
Advanced flash endurance

management
Application consistent backups Backup, DR and archival

• Protect more frequently • DR
1/2 thedown
costtoand
5 minfootprint
RPO of eMLC flash
• Recover rapidly from online backups • Retain backups for months cost-effectively
• Cost-optimized Test/Dev with cloning

Replication Use Cases
Primary to DR Many to 1 (ROBO) Reciprocal

9:00 AM
10:00 AM
11:00 AM
9:00 AM 9:00 AM
12:00 PM
9:00 AM 9:00 AM 10:00 AM 10:00 AM
9:00 AM
10:00 AM 10:00 AM 11:00 AM 11:00 AM
10:00 AM
11:00 AM 11:00 AM 12:00 PM 12:00 PM
11:00 AM
12:00 PM 12:00 PM
12:00 PM
9:00 AM 9:00 AM 9:00 AM

10:00 AM 9:00 AM 9:00 AM 10:00 AM 10:00 AM
11:00 AM 10:00 AM 10:00 AM 11:00 AM 11:00 AM
12:00 PM 11:00 AM 11:00 AM 12:00 PM 12:00 PM
12:00 PM 9:00 AM 12:00 PM
10:00 AM
11:00 AM
12:00 PM
Flexible, efficient WAN-optimized replication facilitates rapid recovery

at another site in the event of a disaster
Replication Components
Software Components
 Partner:
» Identifies a Nimble array that will replicate to and/or from
 Snapshot Schedule:
» Attribute of a volume collection
» Details when to snapshot and replicate and to which partner (one or more of
these per volume collection)
 Throttle:
» Provides the ability to limit replication transmit bandwidth

Partner
 Identifies a Nimble array that can replicate to and/or from this array
 Must be created on upstream and downstream arrays
 Attributes:
» Name: must match group name
» Hostname: must match array’s management IP address
» Secret: shared secret between partners
 Connected: successfully established communications
» Management process re-affirms 1/minute
» Test function performs this on demand
 Synchronized: successfully replicated configuration, updated as needed and every
4 hours

Partner (cont’d)
 Pause/Resume:
» Terminate all in-progress replications inbound or outbound, to/from this
partner do not allow new ones to start until Resume
» Persists across restarts
 Test (button in GUI):
» Perform basic connectivity test
• Management process Controller A to B and B to A
• Data transfer process Controller A to B and B to A
 Throttles:
» Limit transmit bandwidth to this partner
» Scheduling parameters include days, at time, until time
» Existence is mutually exclusive with array throttles (a system can contain
array-wide throttles or partner-wide throttles, but not both)
Replication Partner Notes
• Replication happens on
Management IP
(default)
• Custom option available
IP/Subnet e.g. 10G interface
• You can have many

replication partners
• You can pause

replication by partner
but NOT by Volume
Collection or schedule

Replication Schedules
• Replication configured using

Volume Collection schedule
attributes
• Different Schedules in the

same Collection must
replicate to the same partner
• Calculate your change rate

and bandwidth – can you get it
all done??!!!

Replication QOS – Bandwidth Limit
• Support Multiple QOS

Policies
• Applies to Partner
• Can define a Global QOS

for all partners – Under
Manage Replication
Partner

How Replication Works
Introduction
Replication creates copies of volumes on a separate Nimble array

» Primarily for the purpose of off-site backup and disaster recovery
 Transfers
» Asynchronous / triggered by snapshots
» Transfers compressed snapshot deltas
 Controlled by two processes:

» Management (scheduling)
» Data transfer
 Topologies supported: 1:1, N:1, bi-directional (reciprocal)
 Replica volume can be brought online instantaneously

How Replication Works – The Basics
1. Create a replication partnership

2. Define replication schedule
3. At first replication the entire volume is
Network
copied to the replica partner

Replica Snapshot 4. Subsequent replicas contain only
changes that have occurred

Setup Replication Partner
1. Navigate to: Manage >> Protection >> Replication Partners >> New Replication Partner
2. Enter the partner arrays name.

» Review the notes included
3. Type in a description (Optional)
4. Enter the Hostname or IP address
5. Enter a shared secret

Replication Partnership over Data Subnet
 Manage -> Protection -> Replication Partners -> New Replication Partner
 Using Data Networks

» Can now specify replication over data
networks
» Data networks must be reachable/routable
» Initial connection will still go through
management network
 Pools
» Destination replicas will be created by
default in the specified pool

Volume Collection Schedules
Snapshots and replication are configured via Volume Collection Schedules
 Groups related volumes into a set that is snapshotted and replicated as a unit
 Contains one or more Snapshot Schedules that specify:
» When to take snapshots
» To/from replication partner
» Which snapshots to replicate
» How many snapshots to retain locally
» How many snapshots to retain on the replica
» Alert threshold
 Created on upstream array, automatically replicated to downstream

Volume Collection (cont’d)
 Replicated as configuration data along with all snapshot schedules that define a
downstream partner
» Sent to downstream partner as changes are made (transformed on
downstream, i.e. “Replicate To”  “Replicate From”
» Volumes created in offline state downstream as needed
» Clones created downstream only if parent snapshot exists
 Partner considered “synchronized” only if all relevant configuration is successfully

replicated (volcolls, schedules, volume creation)

Viewing Replication Status
Replication Status
 Replication status:
» Completed: Replication to partner is completed.
» Pending: Replication to partner not yet started (pending completion of prior snapcoll)
» In-progress: Replication in progress and status shows amount of progress
» N/A: Upstream: non-replicable, Downstream: always shows this status

Replication Considerations
Snapshot Collection
 Creation of replicable snapcoll triggers its replication to start

» I.E. Replicate every # of snapshots
• Counter starts at creation of the schedule and does not reset
 Must replicate in the order they are created

 Replication deferred if volume collection busy replicating prior snapcoll
 Replication will not proceed unless partner is synchronized
 Replicable snapcoll cannot be removed by user unless replication to the partner is
paused

Replication & Deduplication Notes
 Replication works between any combo of dedupe & non-dedupe
 Data is rehydrated on  Data is deduplicated (if

transmission from source applicable) on the destination
Replication

Let’s talk about Keys
 There are two types of Encryption Keys

» A Volume Key
• When Encryption is turned on and there is NO Dedupe
» A Domain Key
• When Encryption is turned on and there is (YES) Dedupe
NO Dedupe YES Dedupe

Uses Volume Key Uses Domain Key

Replicating between 3.x Arrays using Dedupe and Encryption
 No encryption and enabled deduplication on both ends – no changes to

workflow for replication
Concurrent deduplication and
encryption on the same volume is
unsupported on software
releases prior to 3.4.0.0.
No Encryption
No Dedupe B
 No encryption and deduplication enabled on both sides – no changes to

workflow for replication
No Encryption
Yes Dedupe B
 Encryption on and no deduplication on both sides – no changes to workflow for

replication
Yes Encryption
No Dedupe B
Replicating caveats
 Encryption enable on both ends and deduplication on upstream only – data is

un-deduped and sent encrypted with a volume key to the downstream array.
Data is stored encrypted on the destination array.
Will un-dedupe & change

Domain encryption to
Volume Encryption
Yes Encryption Yes Encryption Volume Encryption

Yes Dedupe No Dedupe No Dedupe

Replicating caveats
 Encryption enabled on both ends and deduplication in only on the downstream

array - the data is sent over the wire encrypted using the upstream key, and
then on downstream array data is unencrypted and then re-encrypted using
the downstream domain key.
Encrypted
Yes Encryption
keeping Yes
Re-encrypted
Encryptionusing
Upstream
No Dedupe
domain key Downstream
Yes Dedupe
domain key

Replicating between 3.x to 2.x Arrays
 If the destination is not trusted (NO Dedupe Not using Encryption but supports encryption).
» In this case, the data is decrypted using upstream domain key, un-deduped and re-encrypted using the
upstream volume key, the data is then stored downstream as is
Re-Encrypt
Will decrypt
Un-dedupe
before sending
Stored as is
No Encryption
Yes Encryption No Dedupe
No Dedupe
Yes Dedupe Yes Encryption with Upstream
Volume Encryption key
SmartReplicate Disaster Recovery
General DR Operations
 Handover
» Graceful transition between two site (no data loss)
» Examples:
• Handover to DR site for non-DR situations
• Handover from DR site when recovering from a DR situation
 Promote
» Making the DR site primary with the data it has (possible data loss)
» Example:
• Promote a DR site during a disaster
 Demote
» Clearing ownership status from a former source
» Example:
• Production system comes back up after promotion to DR site.

Volume Ownership
 Schedules and volumes have a notion of “ownership”

» On downstream array, replicated objects are owned by upstream array and cannot be directly
modified

Replication Concepts - Promote
Upstream Array Downstream

Temp Upstream Array Array)
(Orig. Downstream
10AM Snap Snap 10AM
11AM Snap Snap 11AM

Promote
11:30AM

Promote
Upstream Array Temp Upstream (Orig. Downstream Array)
When you promote a downstream replication partner,
the system:
1. Suspends the replication relationship associated with the volume
collection.
2. Give ownership of volumes to the downstream array.
10AM Snap
3. Creates a second (local) instance of the volume collection and Snap 10AM
assumes ownership.
4. Clears “Replicate From”
11AM Snap Snap 11AM
5. Brings the most recently replicated snapshots online as volumes.
The contents of the newly available volumes are then consistent Promote
11:30AM
with the last replicated snapshots.
6. Begin taking snapshots per defined schedules
Only use promote if the upstream array is no longer

available.
Upstream(Orig.
Temp Downstream Array Upstream Array) Temp Upstream (Orig. Downstream Array)
10AM Snap Snap 10AM
11AM Snap Snap 11AM

Promote
11:30AM
Reconfigure role - downstream
Snap 12PM

Replication Conceptsm – Promote
Handover
Temp Downstream (Orig. Upstream Array) Temp Upstream (Orig. Downstream Array)
When you Perform a handover the system will:
1. Take all associated volumes offline
2. Takes a snapshot of all associated volumes
3. Replicates these snapshots to a downstream replication
partner
10AM Snap Snap 10AM
4. Transfers ownership of the volume collection to the
partner
5. BringsSnap
11AM the newly replicated volumes online Snap 11AM
6. Reverses replication roles/direction Promote
11:30AM
Snap 12PM
Handover
© 2014 Nimble
Replication Concepts - Handover
Upstream(Orig.
Temp Downstream Array Upstream Array) Downstream
Temp Upstream Array Array)
(Orig. Downstream
Reverse roles
10AM Snap Snap 10AM
11AM Snap Snap 11AM

Promote
11:30AM Snap
Snap
Automatic Snap Snap 12PM

Taken Before
Restore Handover Snap
Demote
 Demote:
»Offlines volumes
»Relinquishes ownership of volcoll objects
»Stop taking local snapshots
 Example:
»After an outage and promote away from an Upstream system;
Upstream System Comes back on-line:
• Prepares a system to become new downstream partner (to manually re-
establish reverse replication)
• or Prepares a system for Fail-Back (handover to the original Upstream
or Production System)
• Looks for common snapshot as a starting point to replicate from
Recovery Scenarios – Testing at DR site (still replicating)
1. Go to downstream replica
2. Clone the snapshot (create a first class volume)
3. Add/adjust ACLs on the volume
4. Mount the volume
5. Interrogate/Test the data and applications (via Windows,
ESX, etc.)
6. Unmount the volume
7. Delete the cloned volume

Recovery Scenarios – Full Disaster Recovery
 Failover to DR site
1. Promote downstream volume collections at DR site
2. Add/adjust ACLs on the volumes
3. Mount volumes to application servers (Windows/ESX)
4. Start production environment at DR site

Recovery Scenarios – Failback after Disaster
 Failback to Primary site

1. Install new array and configure as downstream partner
2. Allow replication of volumes while still running at DR site
3. Gracefully shutdown apps at DR site
4. Perform Handover to primary site
5. Start production environment at primary site

LAB 5
Tasks:
 Setup partner replication
» Configure the upstream array
» Configure the downstream array
» Test the connection stats

NimbelStorage NTS-2001-I Student Guide R5 Feb2017

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NimbelStorage NTS-2001-I Student Guide R5 Feb2017

Uploaded by

Copyright:

Available Formats

NTS-2001-I: Nimble

Introduction and Administration

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-4

 Product Introduction  Backup and Disaster Recovery

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-5

1. Log into InfoSight  Categories

 Refine Searches By:

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-7

AF-Series arrays using

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-9

Tebibyte (TiB) and terabyte (TB) are related

 TB is base 10 (i.e., 1 TB = 1,000,000,000,000 bytes)

 Ti, or tebi, represents multiplication by 10244. So the math

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-10

The All Flash Array (AF-Series)

At the end of this module, you will be able to:

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-12

50,000 IOPS IOPS based on a 70%

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-14

Front: SSD Drives Back: Dual Controllers

Controller Head Shelf

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-16

Slot 1 Slot 2 Slot 3 Slot 4

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-17

Slot 21 Slot 22 Slot 23 Slot 24

Bank B Ejector latch

Bank A Ejector latch

 Hot swappable base carrier hosting two banks.

Presence & Drive Fault LED

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-19

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-21

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-22

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-23

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-25

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-26

NVDIMM (Active CTRL) NVDIMM (Standby CTRL)

Target / Storage Array

Post Process Deduplication:

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-28

Incoming writes Batched in NVRAM/NVDIMM

Variable Block Deduplication

Variable Block Compression

Full Stripe Writes

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-29

Variable Block Dedupe Compression

Fixed Block Dedupe

Variable Block Level Dedupe:

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-30

With most forms of deduplication,

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-31

 Fingerprinting content costs CPU

 Not every workload benefits!

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-33

• Per Application Category Eliminate the

Make informed decisions about whether to use

Deduplication at Domain/Application Level

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-35

 Application Categories are introduced with

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-36

App Category VDI Database

VDI-Policy-1 VDI-Policy-2 VDI-Policy-3 VDI-Policy-4 DB-Policy-1

Volume V1 V2 V3 V4 V5 V6 V7 V8 V10 V11

© 2016 NIMBLE STORAGE | CONFIDENTIAL: DO NOT DISTRIBUTE Course Revision 5 1-37