ECS - ECS Capacity Expansion Procedures-Add Disk(s)

You might also like

Download as pdf or txt
Download as pdf or txt
You are on page 1of 20

ECS ™ Procedure Generator

Solution for Validating your engagement

Topic
ECS Capacity Expansion Procedures
Selections
Which capacity expansion activity will you be performing?: Add Disk(s)
Select Disk Activity: EX500 Storage Disk Expansion

Generated: October 25, 2022 8:44 AM GMT

REPORT PROBLEMS

If you find any errors in this procedure or have comments regarding this application, send email to
SolVeFeedback@dell.com

Copyright © 2022 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell
EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be
trademarks of their respective owners.

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of
any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable
software license.

This document may contain certain words that are not consistent with Dell's current language guidelines.
Dell plans to update the document over subsequent future releases to revise these words accordingly.

This document may contain language from third party content that is not under Dell's control and is not
consistent with Dell's current guidelines for Dell's own content. When such third party content is updated
by the relevant third parties, this document will be revised accordingly.

Publication Date: October, 2022

Dell Technologies Confidential Information version: 2.3.6.114

Page 1 of 20
Contents
Preliminary Activity Tasks .......................................................................................................3
Read, understand, and perform these tasks.................................................................................................3

Notes, cautions, and warnings................................................................................................4


Revision history.......................................................................................................................4
Disk Capacity Expansion EX500 Overview.............................................................................4
Storage disk expansion planning for Project Manager............................................................4
Storage disk expansion planning for onsite personnel............................................................4
EX500 upgrade paths .............................................................................................................5
ECS EX500 disk expansion configurations, upgrade options, and rules ................................5
Disk Capacity Expansion Rules ..............................................................................................6
Disk Capacity Expansion configurations .................................................................................6
Update ECS VDC license based on disk capacity added .......................................................8
Presite Tasks ................................................................................................................................................8
Tools .............................................................................................................................................................9
Procedure .....................................................................................................................................................9

Physically install expansion disks .........................................................................................13


Expand storage disk for object software ...............................................................................15
Check ECS rack health with xDoctor ....................................................................................18
Validate ECS UI ....................................................................................................................20
Post Deployment Actions Required for Defective Disks .......................................................20

Dell Technologies Confidential Information version: 2.3.6.114

Page 2 of 20
Preliminary Activity Tasks
This section may contain tasks that you must complete before performing this procedure.

Read, understand, and perform these tasks


1. Table 1 lists tasks, cautions, warnings, notes, and/or knowledgebase (KB) solutions that you need to
be aware of before performing this activity. Read, understand, and when necessary perform any
tasks contained in this table and any tasks contained in any associated knowledgebase solution.

Table 1 List of cautions, warnings, notes, and/or KB solutions related to this activity

2. This is a link to the top trending service topics. These topics may or not be related to this activity.
This is merely a proactive attempt to make you aware of any KB articles that may be associated with
this product.

Note: There may not be any top trending service topics for this product at any given time.

ECS Top Service Topics

Dell Technologies Confidential Information version: 2.3.6.114

Page 3 of 20
Notes, cautions, and warnings
NOTE: A NOTE indicates important information that helps you make better use of your product.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells
you how to avoid the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

Revision history
Table 1. Revision history

Revision Date Description of change

September Rev1.0. Initial release of document for EX500 release.


2019

June 2021 Rev 1.1 Updates for 16 TB support on ECS version 3.5 and 2 TB and 4 TB support on
ECS version 3.7.

July 2022 Rev 1.2 Updates for S5248F-ON Switch qualification.

September Rev 1.3 Updates for 20TB/192GB memory support.


2022

Disk Capacity Expansion EX500 Overview


This document provides details on EX500 storage disk capacity expansion.

Storage disk expansion planning for Project Manager


The project manager must plan for and carry out certain aspects of the storage disk expansion.

• EX500 upgrade paths


• EX500 disk expansion configurations, upgrade options, and rules
• Update ECS VDC license based on disk capacity added

Storage disk expansion planning for onsite personnel


The onsite personnel must plan for and carry out certain aspects of the storage disk expansion.

• Pre-site requirements
• Validate disk expansion kits
• Locate the racks to be expanded

Dell Technologies Confidential Information version: 2.3.6.114

Page 4 of 20
• Connect service laptop to the ECS racks to be expanded
• Execute xDoctor to check health of ECS racks to be expanded
• Hardware—Add expansion disks to all chassis in racks
• Software—Expand and Verify Storage Disks in racks
• Validate ECS health using xDoctor of the expanded racks
• Ensure that the customer validates ECS UI

Upon completion of these requirements the procedure is complete.

EX500 upgrade paths


Learn about the EX500 ECS appliance upgrade paths.

If there are five or more nodes in the system (VDC) with the same disk size and total node disk capacity,
the minimum node expansion is only one. This upgrade is possible when the expansion node has the
same disk size and the total disk capacity of the existing nodes. For example, you have an EX500
appliance with five nodes containing 24x8TB disks (192TB). You can add a single node with 24x8TB
disks or a total disk capacity of 192TB.
If you are adding nodes that have different disk sizes or node capacity than the nodes currently in the
system, you must add a minimum of five nodes at a time. For example, you have an EX500 appliance
with five nodes containing 24x8TB drives. If you want to add nodes of different drive size (12TB or
16TB), you must add five or more nodes, at a time.

NOTE: All drives within a node must be of the same drive size, but there can be nodes of differing
drive sizes within a rack.

• Disk upgrade kits are available in 12-disk increments.


• EX500 upgrades: 12 to 24 disks.
• To add disk capacity to an EX500, disk upgrade kits per node must be ordered based on the
existing disk capacity 8 TB, 12 TB , and16 TB.
• 2 TB, 4 TB or 20 TB disk capacity is available on ECS 3.7 and later.
• All empty drive slots must be filled with a disk filler.

Table 1. EX500 Disk Upgrade Kit Lists the disk upgrade kit details.

Disk Upgrade Kit Quantity Upgrade Result Per Server

12-disk upgrade 1 EX500 - 12 to 24 disk upgrade

ECS EX500 disk expansion configurations, upgrade options, and


rules
Learn about ECS 500 appliance configurations, disk capacity expansion options, and rules
configurations.
The EX500 supports disk configurations, 12 or 24, 3.5" disks per node.

Dell Technologies Confidential Information version: 2.3.6.114

Page 5 of 20
Disk Capacity Expansion Rules
• Rack/Multi-rack VDC:
o All EX500 nodes in a rack with the same disk size must have the same number of disks
(12 or 24).
o EX500 nodes in a single / multi-rack VDC can have different disk sizes and qualities as
long as there is a minimum of 5 nodes with the same disk size and quality.
o Expand from the bottom chassis to the top chassis.
• Node/Server:
o EX500 does not support mixing of disk capacities within a server.
o All empty slots must contain fillers.

Disk Capacity Expansion configurations


Learn about the EX500 rack chassis minimum and maximum stack-up and disk slot layout for supported
configurations.

Table 1. EX500 Supported Disk Capacity Expansion Configurations

Node or Rack Minimim Nodes Disk Enclosures Disks in Disks in Gross Storage
or Maximum per Node Enclosure Enclosure capacity
(BP2 Back) (BP1 Front

Node minimum Not 2 12 0 2 TB = 24 TB


applicable
4 TB = 48 TB
8 TB = 96 TB
12 TB = 144 TB
16 TB = 192 TB
20 TB = 240 TB

Node maximum Not 2 12 12 2 TB = 48 TB


applicable
4 TB = 96 TB
8 TB = 192 TB
12 TB = 288 TB
16 TB = 384 TB
20 TB = 480 TB

Rack minimum 5 10 12 0 2 TB = 120 TB


4 TB = 240 TB
8 TB = 480 TB

Dell Technologies Confidential Information version: 2.3.6.114

Page 6 of 20
12 TB = 720 TB
16 TB = 960 TB
20 TB = 1.20
PB

Rack maximum 16 32 12 12 2 TB = 768 TB


4 TB = 1.53 PB
8 TB = 3.07 PB
12 TB = 4.60
PB
16 TB = 6.14
PB
20 TB = 7.68
PB

The following graphic represents the recommended racking. The grapic below depicts 0U PDU.
Figure 1. EX500 minimum and maximum configurations

Dell Technologies Confidential Information version: 2.3.6.114

Page 7 of 20
Update ECS VDC license based on disk capacity added
The VDC license may require updating by Dell EMC Software Licensing Central. The Project Manager
should review the current VDC license to determine if additional storage capacity is required to existing
VDC license post disk capacity expansion.

Presite Tasks
Learn about the tasks you must carry out before you arrive at the customer site.

About this task

Dell Technologies Confidential Information version: 2.3.6.114

Page 8 of 20
The VDC license may require updating using Software Licensing Central. If additional storage capacity is
required to be added to existing the VDC License, the Project Manager informs the customer that
existing license must be regenerated and reapplied to VDC.

Steps

1. Determine the service login user and password (default) required for this engagement. See ECS
Service User Access document in ECS Solve > ECS How to Procedures for default information.
If the CLI user or password has been changed from the default, contact the customer for details.
2. Obtain from Project Manager the Configuration Guide detailing disk capacity expansion
information required to successfully perform the capacity expansion engagement. Verify that the
following details are included:
o Disk expansion option kits, quantities and capacity (2 TB, 4 TB, 8 TB, 12 TB, 16 TB, or
20 TB).
o Rack PSNT serial numbers to be expanded.
o Quantity of disks to be added per EX500 Chassis per rack.
o Virtual Data Center (VDC) disk capacity anticipated post disk capacity expansion.
3. Contact the Dell EMC Project Manager to validate material has arrived at the customer site and
implementation schedule.

Tools
Ensure that you have the following tools to complete the procedure:

Steps

1. Service laptop
2. 25' Ethernet cable
3. ESD gloves or ESD wristband

Procedure
Learn about the steps required to update ECS VDC license, based on disk capacity added.

Steps

1. Validate the disk expansion kits:


1. Locate and validate disk expansion kits and quantities based on the information and
Configuration Guide provided by the Project Manager.
2. If any material is missing contact the Project Manager before continuing with this
procedure.
2. Locate the ECS racks:
1. Open the back door of the ECS racks. The PSNT tag is located in the top middle of the
rack.
2. Validate the PSNT Serial Number against the configuration implementation information
provided. Determine if the rack is in the disk capacity add engagement, if so based on
the configuration details go to next task; otherwise, close the door and move to next
rack.

Dell Technologies Confidential Information version: 2.3.6.114

Page 9 of 20
3. Connect the service laptop to the ECS appliance:
Access to private network IP addresses (192.168.219.1 to 16 and 192.168.219.101 to 116) are
limited to the nodes connected in the rack backend 1/10/25GbE fox management switch.
Private.4 (NAN) network IP add.resses (169.254.x.x) of all nodes in all racks in the ECS Virtual
Data Center (VDC) are accessible from any node in the ECS VDC once you SSH in to a node
using a private IP address (192.168.219.x)
Access to public network IP addresses for all ECS racks are available once you SSH to one of
the ECS nodes if security lockdown is not enabled.
1. Connect your service laptop to the VDC:

Table 1. Service Laptop VDC Connection Options

Option Description

If the cabinet contains a Open the service shelf and connect the red network cable to the service
service shelf with a red laptop. The red cable connects to port 34 on the fox switch. The fox switch
network cable: is the bottom back-end switch in a dual switch configuration.

If the cabinet does not If the cabinet does not contain a service shelf with a red network cable.
contain a service shelf with
a red network cable:

If you want to connect a Locate port 36 on the fox switch. The fox switch is the bottom back-end
service laptop to the back of switch in a dual switch configuration. Port 36 has a 1GB SFP that you can
the rack: connect your service laptop to with a Cat6 cable.

Figure 1. Fox switch

1 - Port 34 for service tray connection

2 - Port 36 for connection from rear

2. Set the network interface on the laptop to the static address 192.168.219.99, subnet
mask 255.255.255.0, with no gateway required.
3. Use the ping command to verify that the temporary network between the laptop and
rack private management network is functioning.
If 192.168.219.1 does not answer, try 192.168.219.2. If neither responds, verify the
laptop IP or subnet mask, network connection, and switch port connection. If the service
laptop is connected to the Dell VPN, ping to 192.168.219.x does not return a response.
For example:

Dell Technologies Confidential Information version: 2.3.6.114

Page 10 of 20
C:\>ping 192.168.219.1
Pinging 192.168.219.1 with 32 bytes of data:
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.219.1:


Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

4. Verify that xDoctor is installed and is at the latest version across the ECS systems:
1. If you are not connected to the VDC, establish a secure shell (SSH) session using
PuTTy.
2. Authenticate with service credentials. See ECS Service User Access document in ECS
Solve >ECS >How to Procedures for default information.
3. Connect to node 1 of the rack (192.168.219.1). If node 1 is not available, connect to
node 2 (192.168.219.2).
4. Run the following command to check the xDoctor version: sudo xdoctor --
sysversion
In the following example, the xDoctor version is uniform on all nodes.

# sudo xdoctor --sysversion


xDoctor Uniform on all nodes: 4.6-46

In the following example, the xDoctor version is not uniform on all nodes. xDoctor
automatically updates the node which is not uniform.

# sudo -i xdoctor --sysversion


xDoctor Not Uniform on all nodes:
[4.6-46] -> [168.254.1.1, '168.254.1.2’, '168.254.1.3’]
[4.6-45] -> ['168.254.1.4’]

5. If the installed version of xDoctor listed in the above step is not the latest version as
documented in the ECS xDoctor Users Guide available in ECS SolVe, see the section in
the guide on upgrading or reinstalling xDoctor.
6. If all nodes have the latest version, then go to next step in this procedure.
5. Use xDoctor to check the ECS rack health.
1. Run the following command to log in to the current Rack Master: ssh master.rack
2. Launch xDoctor and perform a Full Diagnosis Suite using the system scope (default).
For example:

# sudo xdoctor
2018-11-12 19:42:44,421: xDoctor_4.6-46 - INFO: Initializing xDoctor v4.6-46
...

Dell Technologies Confidential Information version: 2.3.6.114

Page 11 of 20
2018-11-12 19:42:45,058: xDoctor_4.6-46 - INFO: Removing orphaned session -
session_1542051003.670
2018-11-12 19:42:45,059: xDoctor_4.6-46 - INFO: Removing orphaned session -
session_1542051684
2018-11-12 19:42:45,060: xDoctor_4.6-46 - INFO: Starting xDoctor
session_1542051764.325 ... (SYSTEM)
2018-11-12 19:42:45,060: xDoctor_4.6-46 - INFO: Master Control Check ...
2018-11-12 19:42:45,135: xDoctor_4.6-46 - INFO: xDoctor Composition - Full
Diagnostic Suite for ECS
2018-11-12 19:42:45,136: xDoctor_4.6-46 - INFO: Session limited to 0:40:00
2018-11-12 19:42:45,423: xDoctor_4.6-46 - INFO: -------------------------------
------------
2018-11-12 19:42:45,424: xDoctor_4.6-46 - INFO: ECS Version: 3.2 SP2 Patch 1 -
3.2.2.1
2018-11-12 19:42:45,424: xDoctor_4.6-46 - INFO: -------------------------------
------------
2018-11-12 19:42:45,432: xDoctor_4.6-46 - INFO: xDoctor Pre Features
2018-11-12 19:42:45,433: xDoctor_4.6-46 - INFO: Cron Activation
2018-11-12 19:42:45,433: xDoctor_4.6-46 - INFO: xDoctor already active ...
2018-11-12 19:42:45,433: xDoctor_4.6-46 - INFO: --------------------
2018-11-12 19:42:45,533: xDoctor_4.6-46 - INFO: Validating System Version ...
2018-11-12 19:42:46,159: xDoctor_4.6-46 - INFO: |- xDoctor version is sealed to
4.6-46
2018-11-12 19:42:46,159: xDoctor_4.6-46 - INFO: |- System version is sealed to
3.2.2.0-1960.7545e90.40
2018-11-12 19:42:46,159: xDoctor_4.6-46 - INFO: Distributing xDoctor session
files ...
2018-11-12 19:42:46,399: xDoctor_4.6-46 - INFO: Collecting data on designated
nodes, please stand by ... (update every 5 to 30 seconds)
2018-11-12 19:42:46,400: xDoctor_4.6-46 - INFO: Collection Limit: 0:32:00,
Pacemaker Limit: 900 sec
2018-11-12 19:42:51,406: xDoctor_4.6-46 - INFO: Collecting data ... at 0:00:05
2018-11-12 19:43:01,407: xDoctor_4.6-46 - INFO: Collecting data ... at 0:00:15
2018-11-12 19:43:16,422: xDoctor_4.6-46 - INFO: Collecting data ... at 0:00:30

3. Run the following command to determine the report archive for the xDoctor session run
in the previous step: sudo xdoctor -r | grep -a1 Latest For example:

# sudo xdoctor -r | grep -a1

Latest
Latest Report:
xdoctor -r -a 2015-10-27_183001

4. Use the output from the command in step 5.c. above to view the latest xDoctor report.
Add the -WEC option to display only "Warning, Error, and Critical" events. For example:
sudo xdoctor -r -a < archive date_time > -WEC
The following example shows a clean report with no events:

# sudo xdoctor -r -a 2015-10-27_183001 -WEC

Dell Technologies Confidential Information version: 2.3.6.114

Page 12 of 20
Displaying xDoctor Report (2015-10-27_183001) Filter:['CRITICAL', 'ERROR',
'WARNING'] ...

The following example shows a report with an error:

# sudo xdoctor -r -a 2015-10-27_210554 -WEC

Displaying xDoctor Report (2015-10-27_210554) Filter:['CRITICAL', 'ERROR',


'WARNING'] ...

Timestamp = 2015-10-27_210554
Category = health
Source = fcli
Severity = ERROR
Message = Object Main Service not Healthy
Extra = 10.241.172.46
RAP = RAP014
Solution = 204179

5. Resolve any Warning, Errors, or Critical events that the report may return before
continuing to the next step, unless associated with the acknowledged failure. All xDoctor
reported Warning, Error, and Critical events must be resolved before proceeding.
Contact ECS Remote Support for assistance as required.
6. Expand the EX500 node in order of rack and node.
For each rack to be expanded, start at the EX500 bottom-most node to be expanded, and
proceed one node at a time to completion, with the top-most node in the rack to be expanded.

Physically install expansion disks


When expanding disks on an EX500, ensure that you follow the plan that is provided to you by the
Project Manager on what nodes to expand with the disk expansion kits provided.
Prerequisites

• Ensure ESD precautions (wristband or gloves) are in place before installing disks.
• Identify the disks to be replaced and locations:
Steps

4. If there are disk fillers in the slots where the new drives are going to be added, remove them.

Dell Technologies Confidential Information version: 2.3.6.114

Page 13 of 20
Figure 1. EX500 Front View Hard Drive Bay

5. Insert disks one at a time into the hard drive slot 0 through 11 in disk enclosure bay 1 until the
carrier connects with the backplane.

NOTE: Do not force the hard drive into the slot and backplane connections. The backplane can be
permanently damaged. Slowly and carefully insert the drive into the slot until the cam lever
engages. Ensure that the cam lever on the carrier engages properly.

6. With the disk carrier latch fully open, align the module with the guides and gently insert the disk
into the slot until the carrier handle begins to close.
7. Close the hard drive carrier handle to lock the hard drive in place.
8. Continue until all the expansion kits (drives) for the nodes have been added.
9. Verify that there are no disk fault LED (amber) OFF on any of the added disks.
a.If any added disk fault LEDs (amber) OFF are lit, reseat the disk.
b.Document any fault that is not resolved and any fault LED failures in the table in Post
Deployment Actions Required for Defective Disks.
10.Repeat steps 1-6 for any additional nodes in the rack.
11.Verify that the expansion storage disks are physically detected in the chassis and rack:
a.Run the following command:
viprexec -i 'cs_hal list disks | grep -i total'
Where total is the number of disks expected based on initial disks + disks added per
chassis. In the following example of 12 disks being added to an existing 12 disk total=24

#admin~> viprexec -i 'cs_hal list disks | grep -i total'

Output from host : 192.168.219.1


total: 24

Output from host : 192.168.219.2


total: 24

Output from host : 192.168.219.3


total: 24

Output from host : 192.168.219.4


total: 24

Output from host : 192.168.219.5


total: 23 << Example of “1” disk which is not detected (DEAD or FAULT)

b.If the result of all chassis Totals are as expected, go to Expand storage disk for object
software.
c. Document failures (IP address and count of missing disks) in the table in "Postexpansion
actions that are required if disks are defective," then continue the procedure to expand
disks.

Dell Technologies Confidential Information version: 2.3.6.114

Page 14 of 20
Expand storage disk for object software
Steps

1. Assign the expanded disks to the object application.


Service = "Unallocated" are the disks which have been physically added and are to be
expanded. Run:
sudo -i fcli disks expand
For example, expanding from 12 to 24 disks in a 5 node/chassis rack:

#admin~> sudo -i fcli disks expand

AGENT ID HOSTNAME SERVICE DISK TYPE GOOD BAD


SUSPECT STAGING
bb497c28-5f7b-40bb-919a-8039b177bd75 provo-green object-main 12000GB HDD 12 0
0 0
bb497c28-5f7b-40bb-919a-8039b177bd75 provo-green Unallocated 12000GB HDD 15 0
0 0
e234090d-c343-4c03-8a1a-b7963f42e99e sandy-green object-main 12000GB HDD 12 0
0 0
e234090d-c343-4c03-8a1a-b7963f42e99e sandy-green Unallocated 12000GB HDD 15 0
0 0
d966c42f-6388-4435-b74a-1e8e34afdf83 orem-green object-main 12000GB HDD 12 0
0 0
d966c42f-6388-4435-b74a-1e8e34afdf83 orem-green Unallocated 12000GB HDD 15 0
0 0
e435d8ac-c7b7-48c1-bca1-b37d5c6ad5b3 ogden-green object-main 12000GB HDD 12 0
0 0
e435d8ac-c7b7-48c1-bca1-b37d5c6ad5b3 ogden-green Unallocated 12000GB HDD 15 0
0 0
bb72eba0-7db3-4652-885a-43a22fb3caa6 layton-green object-main 12000GB HDD 12 0
0 0
bb72eba0-7db3-4652-885a-43a22fb3caa6 layton-green Unallocated 12000GB HDD 14 0
0 0

Requesting expand disks ....


Assigning 15 unallocated disks on node bb497c28-5f7b-40bb-919a-8039b177bd75 to the
application
.
.
.
Assigning 15 unallocated disks on node bb72eba0-7db3-4652-885a-43a22fb3caa6 to the
application
Expand disks completed

2. Monitor preparation from service unallocated to object-main. Run:


watch 'sudo -i fcli disks list'
Verify all unallocated disks transition to object-main status. It may take up to 15-30 minutes
for all disks to transition from Unallocated status to object-main status in the rack
depending on the quantity added per chassis.

Dell Technologies Confidential Information version: 2.3.6.114

Page 15 of 20
Initial display may show most added disks in unallocated state. For example:

#admin~> watch 'sudo -i fcli disks list'

Every 2.0s: sudo -i fcli disks list Wed Nov 14 20:43:41 2018

AGENT ID HOSTNAME SERVICE DISK TYPE GOOD BAD


SUSPECT STAGING
bb497c28-5f7b-40bb-919a-8039b177bd75 provo-green object-main 12000GB HDD 24 0
0 0
bb497c28-5f7b-40bb-919a-8039b177bd75 provo-green Unallocated 12000GB HDD 15 0
0 0
e234090d-c343-4c03-8a1a-b7963f42e99e sandy-green object-main 12000GB HDD 24 0
0 0
e234090d-c343-4c03-8a1a-b7963f42e99e sandy-green Unallocated 12000GB HDD 15 0
0 0
d966c42f-6388-4435-b74a-1e8e34afdf83 orem-green object-main 12000GB HDD 24 0
0 0
d966c42f-6388-4435-b74a-1e8e34afdf83 orem-green Unallocated 12000GB HDD 15 0
0 0
e435d8ac-c7b7-48c1-bca1-b37d5c6ad5b3 ogden-green object-main 12000GB HDD 24 0
0 0
e435d8ac-c7b7-48c1-bca1-b37d5c6ad5b3 ogden-green Unallocated 12000GB HDD 15 0
0 0
bb72eba0-7db3-4652-885a-43a22fb3caa6 layton-green object-main 12000GB HDD 24 0
0 0
bb72eba0-7db3-4652-885a-43a22fb3caa6 layton-green Unallocated 12000GB HDD 15 0
0 0

The following example shows disks transitioning from unallocated state to object-main:

Every 2.0s: sudo -i fcli disks list Wed Nov 14 20:50:41 2018

AGENT ID HOSTNAME SERVICE DISK TYPE GOOD BAD


SUSPECT STAGING
bb497c28-5f7b-40bb-919a-8039b177bd75 provo-green object-main 12000GB HDD 50 0
0 0
bb497c28-5f7b-40bb-919a-8039b177bd75 provo-green Unallocated 12000GB HDD 10 0
0 0
e234090d-c343-4c03-8a1a-b7963f42e99e sandy-green object-main 12000GB HDD 49 0
0 0
e234090d-c343-4c03-8a1a-b7963f42e99e sandy-green Unallocated 12000GB HDD 11 0
0 0
d966c42f-6388-4435-b74a-1e8e34afdf83 orem-green object-main 12000GB HDD 51 0
0 0
d966c42f-6388-4435-b74a-1e8e34afdf83 orem-green Unallocated 12000GB HDD 9 0
0 0
e435d8ac-c7b7-48c1-bca1-b37d5c6ad5b3 ogden-green object-main 12000GB HDD 55 0
0 0
e435d8ac-c7b7-48c1-bca1-b37d5c6ad5b3 ogden-green Unallocated 12000GB HDD 5 0
0 0

Dell Technologies Confidential Information version: 2.3.6.114

Page 16 of 20
bb72eba0-7db3-4652-885a-43a22fb3caa6 layton-green object-main 12000GB HDD 52 0
0 0
bb72eba0-7db3-4652-885a-43a22fb3caa6 layton-green Unallocated 12000GB HDD 8 0
0 0

The following example shows all disks transitioning from unallocated state to object-main:

Every 2.0s: sudo -i fcli disks list Wed Nov 14 21:05:41 2018

AGENT ID HOSTNAME SERVICE DISK TYPE GOOD BAD


SUSPECT STAGING
bb497c28-5f7b-40bb-919a-8039b177bd75 provo-green object-main 12000GB HDD 24 0
0 0
e234090d-c343-4c03-8a1a-b7963f42e99e sandy-green object-main 12000GB HDD 24 0
0 0
d966c42f-6388-4435-b74a-1e8e34afdf83 orem-green object-main 12000GB HDD 24 0
0 0
e435d8ac-c7b7-48c1-bca1-b37d5c6ad5b3 ogden-green object-main 12000GB HDD 24 0
0 0
bb72eba0-7db3-4652-885a-43a22fb3caa6 layton-green object-main 12000GB HDD 24 0
0 0

3. Press Ctrl+C to exit the watch command.


4. Document failure(s) (Hostname and quantity of "unallocated disks) in the table in Post
Deployment Actions Required for Defective Disks, and continue the procedure to expand disks.
5. Validate that the new disks are mounted inside of the object-main container. Run:
viprexec 'dockobj df | grep dae -c'
The value for each node display should match the total for "object-main - GOOD disks
displayed in step 2.

#admin~> viprexec -i 'dockobj df | grep dae -c'

Output from host : 192.168.219.1


24

Output from host : 192.168.219.2


24

Output from host : 192.168.219.3


24

Output from host : 192.168.219.4


24

Output from host : 192.168.219.5


23 << Example of “1” disk which is not mounted

Output from host : 192.168.219.6


24

Dell Technologies Confidential Information version: 2.3.6.114

Page 17 of 20
6. Document failure(s) (IP address and count of missing disks) in the table in Post Deployment
Actions Required for Defective Disks.

Check ECS rack health with xDoctor


Steps

1. Launch xDoctor and perform a Full Diagnosis Suite using the default setting system scope. Run:
sudo xdoctor
For example:

# sudo xdoctor

2018-11-12 19:42:44,421: xDoctor_4.6-46 - INFO: Initializing xDoctor v4.6-46 ...


2018-11-12 19:42:45,058: xDoctor_4.6-46 - INFO: Removing orphaned session -
session_1542051003.670
2018-11-12 19:42:45,059: xDoctor_4.6-46 - INFO: Removing orphaned session -
session_1542051684
2018-11-12 19:42:45,060: xDoctor_4.6-46 - INFO: Starting xDoctor
session_1542051764.325 ... (SYSTEM)
2018-11-12 19:42:45,060: xDoctor_4.6-46 - INFO: Master Control Check ...
2018-11-12 19:42:45,135: xDoctor_4.6-46 - INFO: xDoctor Composition - Full Diagnostic
Suite for ECS
2018-11-12 19:42:45,136: xDoctor_4.6-46 - INFO: Session limited to 0:40:00
2018-11-12 19:42:45,423: xDoctor_4.6-46 - INFO: --------------------------------------
-----
2018-11-12 19:42:45,424: xDoctor_4.6-46 - INFO: ECS Version: 3.2 SP2 Patch 1 - 3.2.2.1
2018-11-12 19:42:45,424: xDoctor_4.6-46 - INFO: --------------------------------------
-----
2018-11-12 19:42:45,432: xDoctor_4.6-46 - INFO: xDoctor Pre Features
2018-11-12 19:42:45,433: xDoctor_4.6-46 - INFO: Cron Activation
2018-11-12 19:42:45,433: xDoctor_4.6-46 - INFO: xDoctor already active ...
2018-11-12 19:42:45,433: xDoctor_4.6-46 - INFO: --------------------
2018-11-12 19:42:45,533: xDoctor_4.6-46 - INFO: Validating System Version ...
2018-11-12 19:42:46,159: xDoctor_4.6-46 - INFO: |- xDoctor version is sealed to 4.6-46
2018-11-12 19:42:46,159: xDoctor_4.6-46 - INFO: |- System version is sealed to
3.2.2.0-1960.7545e90.40
2018-11-12 19:42:46,159: xDoctor_4.6-46 - INFO: Distributing xDoctor session files ...
2018-11-12 19:42:46,399: xDoctor_4.6-46 - INFO: Collecting data on designated nodes,
please stand by ... (update every 5 to 30 seconds)
2018-11-12 19:42:46,400: xDoctor_4.6-46 - INFO: Collection Limit: 0:32:00, Pacemaker
Limit: 900 sec
2018-11-12 19:42:51,406: xDoctor_4.6-46 - INFO: Collecting data ... at 0:00:05
2018-11-12 19:43:01,407: xDoctor_4.6-46 - INFO: Collecting data ... at 0:00:15
2018-11-12 19:43:16,422: xDoctor_4.6-46 - INFO: Collecting data ... at 0:00:30

2. Determine the report archive for the xDoctor session executed in step 1. Run:
sudo xdoctor -r | grep -a1 Latest
For example:

Dell Technologies Confidential Information version: 2.3.6.114

Page 18 of 20
# sudo xdoctor -r | grep -a1Latest
Latest Report:
xdoctor -r -a 2015-10-27_183001

3. Use the output from step 2 to view the latest xDoctor report. Add the -WEC option to display only
Warning, Error and Critical events.
sudo xdoctor -r -a < archive date_time > -WEC
The following example shows a clean report with no events:

# sudo xdoctor -r -a 2015-10-27_183001 -WEC

Displaying xDoctor Report (2015-10-27_183001) Filter:['CRITICAL', 'ERROR', 'WARNING']


...

The following example shows a report with an error:

Displaying xDoctor Report (2015-10-27_210554) Filter:['CRITICAL', 'ERROR', 'WARNING']


...

Timestamp = 2015-10-27_210554
Category = health
Source = fcli
Severity = ERROR
Message = Object Main Service not Healthy
Extra = 10.241.172.46
RAP = RAP014
Solution = 204179

4. If the report returns any Warning, Errors, or Critical events, resolve those events, unless
associated with the acknowledged failure, before you continue with step 5.
5. End the SSH session. Run:
# logout
6. Disconnect the service laptop from the fox switch.
7.
o If this is a single-rack system, go to Validate ECS UI. You have completed storage disk
expansion.
o If this is a multi-rack system, for all racks in the VDC:
d.Repeat Update ECS VDC license based on disk capacity added starting at step 2.
e.Repeat Physically install expansion disks.
f. Repeat Expand storage disk for object software.
g.Repeat Check ECS rack health with xDoctor.
Results

You have completed storage disk expansion for all racks.

Dell Technologies Confidential Information version: 2.3.6.114

Page 19 of 20
Next steps

After you complete storage disk expansion for all the racks in the VDC for which you plan to carry out
storage disk expansion, go to Validate ECS UI.

Validate ECS UI
When all disk expansion additions are successfully completed, the customer must validate that the
correct number of disks and capacity appears in the ECS UI portal.
Steps

1. Open browser to ECS UI portal on one of the nodes in the VDC which had disks expanded and
navigate to Dashboard.
2. In Node & Disks and Capacity Utilization panels, verify that the Disks quantity and Total Capacity
display the expected value.

Post Deployment Actions Required for Defective Disks


Use the table in this section to track any defective disk drives during disk deployment.

If installed disk is defective and detected as FAILED upon deployment, see the ECS SSD Replacement
Guide.
If ECS does not detect a disk; that is, if a disk is not listed under ECS UI Manage >Maintenance, contact
ECS Remote Support for follow-up.

Rack PSNT SN number Node IP or hostname/Chassis number in Rack Failure Comment

Example: Example: Example:


APM00144307335 192.168.219.1 / 1 Disk not listed in GUI
Disk Failed

Dell Technologies Confidential Information version: 2.3.6.114

Page 20 of 20

You might also like