Node Upgrade Guide

Isilon
NL400

Upgrade a node
May 2017

l Upgrading nodes.....................................................................................................2
l Preparing for the upgrade.......................................................................................3
l Performing the upgrade......................................................................................... 11
l Reviewing the upgrade......................................................................................... 26
l Roll back a failed upgrade..................................................................................... 28
l Where to go for support....................................................................................... 29
Node Upgrade Guide

Upgrading nodes
You can upgrade the drives, memory, and NICs in most EMC Isilon nodes.
The process to upgrade nodes in a cluster consists of three stages.
Preparing for the upgrade
An upgrade procedure requires significant planning and preparation. Steps in the
process, such as smartfailing drives from multiple nodes, can take several weeks to
complete. Before the actual maintenance window, you must also obtain a hardware
upgrade package for each node you plan to upgrade. It is important that you
understand this procedure and communicate its full impact to everyone who might be
affected.
Performing the upgrade
After you prepare the cluster for work, you must upgrade the nodes during the agreed
upon maintenance window. A maintenance window is necessary, because full cluster
availability during the upgrade procedure is not guaranteed. You must adhere to all
EMC Isilon safety and quality guidelines as you perform each step of the procedures
for removing and installing node components.
Reviewing the upgrade
After nodes are upgraded, you must review the status of the cluster and confirm that
all configuration changes were successful and that all new storage pools are
established and healthy.

Downgrading a node
You can downgrade configure-to-order (CTO) components in a node.
The procedure to perform a downgrade is exactly the same as the procedure to
perform an upgrade. Follow the steps to prepare, perform, and review all changes to
the node configuration.

Working with clusters in SmartLock compliance mode

Clusters running in SmartLock compliance mode require a sudo prefix to run root
commands.
If a cluster is running in SmartLock compliance mode, root access is disabled on the
cluster. Because of this, you can run some commands only through the sudo program.
Prefixing a command with sudo enables you to run commands that require root
access. For example, if you do not have root access, the following command fails:

isi drivefirmware status

However, if you are on the sudoers list, the following command succeeds:

sudo isi drivefirmware status

Compliance mode commands that require changes beyond the sudo prefix are noted
in the procedure steps.
For more information on the sudo program and compliance mode commands, see the
OneFS CLI Administration Guide.

2 NL400 Node Upgrade Guide

 Upgrade a node

Preparing for the upgrade

Before you upgrade nodes in a cluster, open a support case, prepare the environment,
and obtain a hardware upgrade package.

Schedule an upgrade maintenance window

When you perform an upgrade, the procedure will interrupt the cluster workflow and
performance might be affected. Schedule a maintenance window to avoid unexpected
disruptions.
The cluster will remain online during the upgrade procedure, but full cluster availability
cannot be guaranteed. Nodes will be rebooted one at a time during the process, and
will also be removed and added to storage pools.
Procedure
1. Schedule a maintenance window based on the number of nodes that you will
upgrade.
The window should allow for up to one hour per node.

Request a hardware upgrade package

You must request hardware upgrade kits, and a hardware upgrade file, from the Isilon
Verification team.
For hardware upgrades, in addition to the physical hardware upgrade kit for each
node, you will also need a customized upgrade package file with a specific serial
number .
Procedure
1. Do one of the following:
l Run the following command to view a current diagnostic log for the cluster:
isi_gather_info
l Run the following command to obtain upgrade receipts from the upgraded
nodes: isi_inventory_tool --display --itemType=receipt
Record the system output.
2. Gather the following cluster information:
l Sales order number
l Cluster name
l Node numbers
l Node serial numbers
l OneFS version

3. Request a download link for the hardware upgrade package from the Isilon
Verification Team by sending the information you gathered in previous steps to
cto.upgrades.validation@emc.com.
Requests for download links can take approximately 7 business days. These
requests must be made before you begin the upgrade process.

Preparing for the upgrade 3

Node Upgrade Guide

Note

EMC Isilon Technical Support cannot help with the creation or correction of
hardware upgrade packages, or changes to receipts. For these requests,
contact the Isilon Verification Team.

Download the hardware upgrade package

Before you upgrade your hardware, obtain a hardware upgrade package from your
Isilon Account Team. The hardware update package updates the configure-to-order
(CTO) and as-built information on the node, then forwards the updated information to
Isilon Technical Support.
Procedure
1. Obtain a hardware upgrade package from your Isilon Account Team.
2. Note the name of the hardware upgrade package. You will use the name for
other commands.
Package names follow this convention:
IsiHW_Package_ [order number] _ [node serial number] _ [date and time
stamp] .tgz
For example,
IsiHw_Package_ORD108024_SX2003011184736_201507072125.tgz

3. Place the hardware upgrade package on the cluster through a network drop, or
by asking someone at the cluster site to place the package for you. If neither of
these options is available to you, contact Isilon Technical Support for
assistance.

Install the hardware upgrade package on the node

You must place an upgrade package on each node that is receiving an upgrade, unpack
the package, and then install it.
Procedure
1. Take the upgrade package that was provided by your Isilon Account Team and
place it on the node that the package was created for.
Use a network drop or have the customer place the package for you.

2. Unpack the update by typing the following command:

tar xzvf IsiHW_Package_<order-number>_<node-serial-
number>_<date-time-stamp>.tgz

3. Change directories to where the package was placed.

4. Install the update package. Depending on your version of OneFS, type one of
the following commands:
OneFS 8.0 or later
isi upgrade patches install<order-number>_<node-serial-
number>_<date-time-stamp>.tar
Earlier than OneFS 8.0
isi pkg install --forced_local IsiHW_Package_<order-
number>_<node-serial-number>_<date-time-stamp>.tar

4 NL400 Node Upgrade Guide

 Upgrade a node

As the package installs, the following messages appear:

Preparing to install the package...

Checking the package for installation...
Installing the package
Committing the installation...
Package is committed.

If the node does not perform as expected, contact Isilon Technical Support.

5. Repeat these steps for each node that you are upgrading.

Note

Your Isilon Account Team will generate an individual upgrade package for each
node you are upgrading. Make sure to install the correct upgrade package on
each node.

Review the upgrade receipt

Verify that the receipt file in the hardware upgrade package is consistent with the
defined upgrade path and the hardware order.
Procedure
1. Change directory to the location of the upgrade package by running the
following command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

2. Display the receipt file by running the following command:

./isi_cto_upgrade --overview

Note

If the cluster is in SmartLock compliance mode, display the receipt file by

running the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --overview

Note

You can execute this command in advance of the upgrade engagement. This
command will not make changes to the node.

The command output will confirm verification of the package integrity, node
configure-to-order (CTO) capability, chassis VPD, and the node serial number.
The output will also display the receipt that was generated before the upgrade
next to the receipt that was generated after the upgrade.

Request a hardware upgrade package 5

Node Upgrade Guide

Note

You might see some noncritical error messages as part of the output. If you are
not sure whether it is safe to continue, contact EMC Isilon Technical Support.

3. Compare the command output of the receipt that was generated before the
upgrade to the command output of the receipt that was generated after the
upgrade.
The content of the receipt will be similar to the following example:

<receipt
receiptFileName="30301230_SX4001013160081.xml"
receiptFormatVersion="1.2" date="2014-01-23T01:40:03Z">
<bom>
<sbb qty="1" desc="SBB, BASE, DDR, X400" part="610-0005-03"/>
<sbb qty="1" desc="SBB, 3.5" 66TB HDD / 1.2TB SSD EMLC"
part="611-0075-02"/>
<sbb qty="1" desc="SBB, Memory, 48GB DDR3, 12x4G"
part="612-0006-01"/>
<sbb qty="1" desc="SBB, 10GbE, Dual-port, SFP+, without Optics"
part="613-0002-01"/>
</bom>
</receipt>

Results
If the receipt from before the upgrade is not consistent with the receipt from after the
upgrade, or if neither are consistent with the defined upgrade path and the hardware
order, contact your Account Team and request a corrected upgrade package.
If you have issues that are not related to the receipt, contact EMC Isilon Technical
Support.

Review storage pools configuration

As you upgrade nodes in a cluster, OneFS will organize the upgraded nodes into a new
storage pool.
After you upgrade a node, OneFS recognizes the new node configuration and creates
a new node pool for the upgraded node. As long as there are fewer than three nodes in
the new node pool, OneFS will only read data from that node, not write to it. After you
upgrade a third node and there are three nodes in the new node pool, OneFS will write
data to those nodes.
You might experience degraded performance as nodes are reprovisioned to new node
pools, but data on the cluster will remain available.
Before you begin an upgrade engagement, review the cluster's storage pool
configuration to determine which policies you will need to change after the upgrade is
complete.
For more information about storage pools, see the OneFS CLI Administration Guide.

Adding new nodes during an upgrade

If you are adding new nodes to a cluster as part of a hardware upgrade engagement,
review the configuration of the nodes to determine when you should add them to the
cluster.
If the nodes that you are adding to the cluster have the same hardware configuration
as the existing nodes prior to the upgrade, add the new nodes to the cluster before
you upgrade.

6 NL400 Node Upgrade Guide

 Upgrade a node

If the nodes that you are adding to the cluster have the hardware configuration that
the existing nodes will have following the upgrade, add the new nodes to the cluster
only after you upgrade the existing nodes and confirm that the cluster is healthy.
Following these guidelines will prevent OneFS from creating new node pools before
the hardware upgrade is completed, which will reduce the complexity of the
environment.

Review scheduled jobs

Review scheduled jobs that might cause performance issues during the upgrade.
Pause any running jobs to minimize cluster performance issues during the upgrade
process.
If you are upgrading drives, pause all jobs before you smartfail the drives, to enable the
smartfail process to complete as quickly as possible.
You can restart the jobs after the upgrade is complete.

Confirm SSD compatibility

If you are installing a new solid state drive (SSD) into a cluster, you must confirm that
the drive you have onsite is compatible with the cluster.
See the Isilon Supportability and Compatibility Guide to determine whether the drive
you have is compatible with the cluster.

Gather logs
Before you begin any maintenance on a cluster, gather cluster logs.
You must collect cluster logs before all maintenance procedures. Cluster logs provide
snapshots of the cluster, which you can review to make sure that maintenance is
successful.
Procedure
1. Open a secure shell (SSH) connection to any node in the cluster and log in.
2. Gather cluster logs by running the following command:
isi_gather_info

Review the current hardware status

View the status and configuration information for memory, drives, and NICs that are
currently installed.
Capture this pre-upgrade information to review after the upgrade is complete.
Procedure
1. Change directory to the location of the upgrade package by running the
following command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

2. View the current hardware configuration by running the following command:

./isi_cto_upgrade --overview

Review scheduled jobs 7

Node Upgrade Guide

Note

If the cluster is in SmartLock compliance mode, view the current hardware

configuration by running the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --overview

3. Confirm that the current amount of installed RAM matches the hardware
upgrade receipt, and that all DIMMs are healthy.
4. Confirm that the installed drives match the hardware upgrade receipt, and that
there are no failed drives.
5. Confirm that the installed network interfaces match the hardware upgrade
receipt, and that all NICs are healthy.

Open a support case

Open a support case for the customer prior to the upgrade engagement.
Procedure
1. Open a new support case with EMC Isilon Technical Support.
An open support case will save time if an issue arises during the upgrade
procedure.
2. Attach your engagement plan and any other supporting documentation to the
support case.
Your engagement plan should include expected changes to storage pools and
rollback options, if you encounter an issue.

Smartfail drives to be upgraded

If you are upgrading drives as part of an upgrade engagement, smartfail the drives in
advance.
Before you begin

Note

You should not arrive for a drive upgrade engagement until all the drives that you are
going to upgrade, in all nodes, are smartfailed.

Before you smartfail multiple drives, check the amount of free space available on the
cluster. Follow best practices related to available space and protection levels on the
cluster.
Refer to the following documentation for more information:
l Best practice guide for maintaining enough free space on Isilon clusters and pools
l How to determine if an isilon cluster is in a window of risk for data loss
If you are upgrading to SSDs, smartfail drives in the lowest drive bay numbers. For
example, if you are adding two SSDs to each node, smartfail the drives in bays one and
two.

8 NL400 Node Upgrade Guide

 Upgrade a node

Figure 1 Front drive bays

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24
GHE-000066

Figure 2 Back drive bays

25 26 27 28

29 30 31 32

33 34 35 36
GHE-000067

Smartfail all the drives that will be upgraded, one node at a time. Wait for the smartfail
process to complete on one node, then proceed to the next node.
Procedure
1. Open a secure shell (SSH) connection to the node that contains the drive you
want to remove. Type the command:
ssh <cluster-name>-<node-number>

2. Confirm the node number and bay number of the drive you want to remove.
Depending on your version of OneFS, type one of the following commands:
OneFS 8.0 or later
isi devices drive list

Earlier than OneFS 8.0

isi devices

Note

Note the node number and bay number of the drive you want to remove. If the
drive you need to remove has already been smartfailed automatically, you can
skip the next step. A smartfailed drive will display a status of REPLACE or, if the
node was restarted after the smartfail, a status of EMPTY. A smartfailed SED
drive might display a status of ERASE. This status also indicates the node is
ready to be replaced.

3. Using the node number and bay number of the drive, smartfail the drive by
typing one of the following commands:
OneFS 8.0 or later
isi devices drive smartfail <bay-number> --node-lnn <node-
number>
Earlier than OneFS 8.0
isi devices -a smartfail -d <node-number>:bay<bay-number>

Smartfail drives to be upgraded 9

Node Upgrade Guide

If the drive does not respond to the smartfail command, do not remove it from
the node. Contact Isilon Technical Support.

4. Confirm that the drive was smartfailed by repeating step two.

CAUTION

Do not replace the drive until you have confirmed that the smartfail
process is complete.

Results
Repeat this procedure for every drive, in every node that you are upgrading.
The length of time that it takes for the smartfail process to complete depends on the
size of the cluster, drive capacity, and the cluster workflow.

Install a drive support package

For clusters running OneFS 7.1.1 and later, install the latest drive support package to
update your drive firmware to the latest supported revision.
Before you begin
See the Considerations for installing the latest drive support package section before you
begin the installation.
Procedure
1. Go to the EMC Support page that lists all the available versions of the drive
support package.
2. Click the latest version of the drive support package and download the file.

Note

See the Considerations for installing the latest drive support package section in
order to select the appropriate variant of the package. If you are unable to
download the package, contact EMC Isilon Technical Support for assistance.

3. Open a secure shell (SSH) connection to any node in the cluster and log in.
4. Create or check for the availability of the directory structure /ifs/data/
Isilon_Support/dsp.
5. Copy the downloaded file to the dsp directory through SCP, FTP, SMB, NFS,
or any other supported data-access protocols.
6. Unpack the file by running the tar command.
For example, based on the variant selected for the drive support package,
unpack the package by running one of the following commands:

tar -zxvf Drive_Support_<version>.tgz

tar –zxvf Drive_Support_<version>_No_SSD.tgz

7. Install the package by running the isi_dsp_install command.

10 NL400 Node Upgrade Guide

 Upgrade a node

For example, based on the variant selected for the drive support package, install
the package by running one of the following commands:

isi_dsp_install Drive_Support_<version>.tar
isi_dsp_install Drive_Support_<version>_No_SSD.tar

Note

l You must run the isi_dsp_install command to install the drive support
package. Do not use the isi pkg command.
l Running isi_dsp_install will install the drive support package on the
entire cluster.
l The installation process takes care of installing all the necessary files from
the drive support package followed by the uninstallation of the package. You
do not need to delete the package after its installation or prior to installing a
later version.

Reboot the cluster

With explicit permission from the customer, you can reboot the cluster to make sure
the status of the cluster is healthy.
Rebooting the cluster can flush out unexpected hardware alerts.

Note

Although rebooting the cluster can help avoid certain issues, taking this step might not
be possible in your environment. And, although this step is recommended, it is
optional.

Procedure
1. Reboot the cluster according to EMC Isilon best practices for restarting entire
clusters.

Performing the upgrade

Upgrade nodes with one or more new components.

CAUTION

Perform the upgrade procedure on only one node at a time. Working on multiple
nodes in parallel might lower the protection level of the cluster, put data at risk,
and lead to the interruption of client workflows.

Reboot the cluster 11

Node Upgrade Guide

Run the pre-upgrade script

After the hardware upgrade package is installed on the node, you must run the pre-
upgrade script.
Before you begin
If the hardware upgrade package is not installed, follow the procedure for installing a
hardware package on a node before you begin this procedure.
Procedure
1. Change to the correct directory by running the following command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

The directory name includes the package info.

2. To run the pre-upgrade script, run the following command:

./isi_cto_upgrade --pre

Note

If the cluster is in SmartLock compliance mode, run the pre-upgrade script by

running the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --pre

Messages appear as the script confirms the following information:

a. CTO capability
b. Upgrade order number
c. Node serial number
d. Current node hardware configuration
The command output is similar to the following example:

Verifying package integrity

Package integrity is good
Beginning pre-installation actions
Node CTO capability verified
Upgrade is for Order 30548476, Node S/N SN4002313370143
Chassis VPD verified
Node S/N verified
Node CTO configuration verified
Importing SoftAVL (./Soft-AVL_201412020001.xml)
Success
Importing local /etc/gconfig/probe_config.0.gc (/etc/gconfig/
probe_config.0.gc)
Success
Importing local /etc/gconfig/probe_config.1.gc (/etc/gconfig/
probe_config.1.gc)
Success
Importing local /etc/gconfig/probe_config.gc (/etc/gconfig/
probe_config.gc)

12 NL400 Node Upgrade Guide

 Upgrade a node

Success
...

Note
The upgrade script displays a list of missing and found components. The missing
component list contains the new parts that you will add to the node. The found
component list contains the parts that you will remove. You must accept this
message to continue. You might see non-critical error messages as part of the
output. If you do not feel that it is safe to continue, contact EMC Isilon
Technical Support.

3. Once the pre-upgrade script concludes, you are prompted to power down the
node:

Ready to power down the node? (y/n)

4. Type y to power down the node.

Upgrading a drive
If you are upgrading drives as part of an upgrade engagement, remove the smartfailed
drives.
Remove the old drives only. Do not install the new drives until you have powered the
node back up.

Access the drives

You can access drives by removing the panels that cover the drive bays.
Procedure
1. To access drives at the front of the node, remove the front panel.
There is a release button on each side of the front panel. To remove the panel,
press in the panel release buttons while pulling the panel away from the node.

1. Front panel

Upgrading a drive 13
Node Upgrade Guide

2. To access drives at the back of the node, remove the rear EMI shield.
There is a vertical handle at each end of the shield. To remove the shield, press
the handles toward the center of the shield while pulling away from the node.

1. Rear EMI shield

Remove the drive to be upgraded

Drives are secured in the node. You must release the locking handle before removing a
drive.
Before you begin
Only remove the drives that you smartfailed in preparation for the upgrade. If you are
upgrading to SSDs, the smartfailed drives should be in the lowest drive bay numbers.
For example, if you are adding two SSDs to each node, the smartfailed drives should
be in bays one and two.
Figure 3 Front drive bays

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24
GHE-000066

14 NL400 Node Upgrade Guide

 Upgrade a node

Figure 4 Back drive bays

25 26 27 28

29 30 31 32

33 34 35 36
GHE-000067

Procedure
1. To remove a drive, pull the locking handle on the drive toward you.
The drive releases from the node.

1. Locking handle 2. Drive bay

Upgrading a NIC or DIMMs

If you are upgrading a NIC or DIMMs as part of an upgrade engagement, you will need
to access the inside of the node.
If you are only upgrading drives, skip ahead to powering up the node. Do not
disconnect and open the node.

Slide the node out of the rack

Slide the node away from the rack to access the contents of the node.
Procedure
1. Label the InfiniBand, ethernet, and power cables connected to the back of the
node to ensure that they are reconnected correctly.

Upgrading a NIC or DIMMs 15

Node Upgrade Guide

2. Disconnect all cables from the back of the node.

Note

If there are transceivers connected to the end of your IB or ethernet cables,

make sure to remove them with the cables. If you are using fiber ethernet
cables, you will need to disconnect the cable from the transceiver, then remove
the transceiver from the node.

3. Remove the node front panel.

4. Remove the retaining screws that secure the node to the rack cabinet.
5. Slide the node from the rack cabinet to fully extend the slide rails and provide
clear access to the node. Do not remove the node from the slide rails.

DANGER

Slide the node out from the rack slowly. Do not extend the rails completely
until you confirm that the node is latched and safely secured to the rails.

Remove the node top panel

You remove the top panel to gain access to the contents of the node.

WARNING

Properly ground yourself to prevent electrostatic discharge from damaging the

node. For example, attach an ESD strap to your wrist and the node chassis.

Procedure
1. Loosen the captive screw that secures the node top panel.
2. Slide the top panel toward the rear of the node, and then lift the top panel to
access the node interior.

Remove the cross bracket

Removing the cross bracket provides clear access to the inside of the node.
Procedure
1. Locate the cross bracket within the node.

16 NL400 Node Upgrade Guide

 Upgrade a node

Figure 5 Cross bracket

2. Remove the cross bracket by pressing on the side of the node chassis where
the cross bracket is connected. Unhook the cross bracket from the chassis,
then lift straight up to unhook the other side of the bracket.

Remove the air baffle

In order to gain full access to the internal components of the node, you must remove
the air baffle.
Procedure
1. Locate the air baffle within the node.

Upgrading a NIC or DIMMs 17

Node Upgrade Guide

Figure 6 Air baffle

2. Raise the front end of the air baffle, unhook the tabs at the back end of the
baffle, and then lift the baffle out of the node.

Upgrading DIMMs
If you are upgrading DIMMs as part of an upgrade engagement, remove the old
DIMMs and replace them with the new DIMMs.
If you are installing a different amount of DIMMs than you are removing, make sure
that the new DIMMs are installed in the same color slots and in the same bank from
which the old DIMMs were removed.

Remove the DIMM to be upgraded

You must remove the DIMM that you are upgrading from the node.
Procedure
1. Press down on the DIMM locking arm on each side of the DIMM to release the
DIMM from the slot.

18 NL400 Node Upgrade Guide

 Upgrade a node

CAUTION

If you are replacing a DIMM in slot P2 DIMM 3A or P2 DIMM 3B, remove the
network interface card (NIC) to allow enough space to remove the DIMM
without damaging the NIC.

1. DIMM

Install the new DIMM

You must install the new DIMM in the slot from which you removed the old DIMM.
Procedure
1. Remove the new DIMM from the antistatic package.
2. Locate the open slot that the old DIMM was removed from. Align the notch in
the DIMM with the tab of the open slot and press down firmly on both ends of
the DIMM until the two arms lock into place, securing the DIMM.

Note

Install the new DIMM in the empty slot that used to hold the old DIMM. A DIMM
that is installed in another open slot runs the risk of not being recognized by the
system.

Upgrading a NIC or DIMMs 19

Node Upgrade Guide

Upgrading a NIC
If you are upgrading the NIC as part of an upgrade engagement, remove the NIC and
replace it with the new card.

Remove the NIC to be upgraded

You must remove the NIC that you are upgrading from the node.
Procedure
1. If necessary, disconnect any cables from the NIC.
2. Remove the mounting screw that secures the NIC to the node.
3. Remove the NIC from the node.

1. Mounting screw 2. NIC

20 NL400 Node Upgrade Guide

 Upgrade a node

Install the new NIC

After the original NIC is removed, you can install the new NIC.
Procedure
1. Insert the new NIC in the empty card slot from which you removed the original
NIC. Check to ensure that the card is fully seated in the motherboard
connector.
2. Secure the card to the back panel of the node with the mounting screw.
3. Reconnect any cables that you removed from the card.

Install new transceivers and cables

Install new front-end cables and transceivers if necessary for the new NIC.
Procedure
1. Locate front-end cables at the back of the node.
2. If the upgraded NIC requires new cables, replace the front-end cables with the
new cables.
3. Remove all dust covers and plugs from the new transceiver.
4. Insert the cable connector into the transceiver until it clicks.
Transceivers are keyed to ensure proper connection to the cable. If the
transceiver does not connect easily to the cable, check to be sure that you are
aligning the transceiver correctly.
5. Repeat these steps to attach the new transceiver to other front end cables as
necessary.

Install the air baffle

You must replace the air baffle by inserting the baffle tabs into slots in the chassis.
Procedure
1. Hook the tabs on the back end of the air baffle into the metal slots at the back
of the node.
2. Lower the front of the air baffle back into its original position within the node.

Install the cross bracket

You must install the cross bracket by hooking it to the bracket holes on the interior of
the node and then snapping the other end into the chassis wall.

WARNING

The cross bracket sits directly above the boot drives. Use caution when installing
the cross bracket so that the boot drives are not dislodged or damaged.

Install the node top panel

You must secure the top panel onto the node.
Procedure
1. Place the top panel on the node so that the front edge of the top panel is about
one inch behind the drive bays, and then slide the top panel forward into place.

Upgrading a NIC or DIMMs 21

Node Upgrade Guide

WARNING

The chassis intrusion switch can be damaged if the top panel is slid too far
back on the node.

2. Tighten the captive top panel screw to secure the top panel to the node.

Return the node to the rack

Return the node to the rack after all work is complete.
Procedure
1. Slide the node back into the rack cabinet.

WARNING

Slide the node slowly so you do not slam the node into the rack and damage
the node.

2. Reconnect the ethernet, InfiniBand, and power cables to the back of the node.
3. Secure the node to the rack cabinet.
4. Replace the node front panel.

Connect the cables to the node

You must connect all the power and network cables to the back of the node.
Procedure
1. Locate the labels on all the power and network cables to ensure that they are
connected correctly.

Power up the node

Power up the node by pressing the power button on the back panel.
Procedure
1. Power up the node by pressing the power button on the back panel of the node.
It is located just left of center, toward the upper part of the back panel.

Confirm a healthy node

Confirm that the node has powered up successfully.
Procedure
1. Review the node power-up for errors.
Do not continue until the node has successfully booted to the command
prompt.

Note

If errors are present following a reboot, do not continue. Contact EMC Isilon
Technical Support.

22 NL400 Node Upgrade Guide

 Upgrade a node

Install the new drive

You install a drive in a node by inserting it into an empty drive bay.
Procedure
1. With the locking handle on the drive open, insert the drive into the empty drive
bay by sliding the drive along the bay rails until it stops.
Do not force the drive into the drive bay. Forcing the drive into the drive bay
could damage both the drive and the drive bay.
2. Hold the drive in place and gently push the locking handle down against the end
of the drive to secure, or seat, the drive in the node.
Drives that are not fully seated will not be recognized when the node is started,
and a red light will appear above the drive. To avoid this problem, run your
finger across all the installed drives to ensure that they are all seated evenly.
3. Connect the drive to the node using the node number and bay number where
you just inserted the drive. Depending on your version of OneFS, run one of the
following commands:
OneFS 8.1 or later
OneFS will automatically add the drive. If ARR (Automatic Replacement
Recognition) has been disabled, add the drive with the OneFS 8.0
command.

OneFS 8.0
isi devices drive add <bay-number> --node-lnn <node-number>

Earlier than OneFS 8.0

isi devices -a add -d <node-number>:<bay-number>

The add command connects the drive to the node.

If the add command fails, make sure that the drive is formatted by running one
of the following commands:
OneFS 8.0 or later
isi devices format <bay-number> --node-lnn <node-number>

Earlier than OneFS 8.0

isi devices -a format -d <node-number>:<bay-number>

The format command introduces the drive to the cluster.

If the drive does not respond to either of these commands, do not remove the
drive from the node. Contact Isilon Technical Support.

4. After the new drive has been installed, replace the front panel on the node by
aligning the panel with the front of the node and pressing the panel until it clicks
onto the node.
If you replaced a drive in the rear of the node, replace the EMI shield.

Install the new drive 23

Node Upgrade Guide

Run the post-upgrade script

Once you install the new hardware in the node, you must verify and report the
upgrade by running the post-upgrade script.
Procedure
1. To change to the correct directory, use the information in the name of your
hardware package to run the following command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

2. Verify installation of the hardware by running the following command:

./isi_cto_upgrade --post

Note

If the cluster is in SmartLock compliance mode, verify installation of the

hardware by running the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --post

As the script confirms the upgrade, a number of status messages appear.

If an FTP connection is available, the script sends an updated ABR to Isilon
Technical Support.

Remove the upgrade package from the node

After you run all the scripts, it is safe to remove the upgrade package.
Procedure
1. Change out of the hardware package directory by typing the following
command:
cd /

2. To delete the upgrade package from a node, use the information in the name of
your hardware package type the following command:
isi pkg delete --forced_local IsiHW_Package_ <order-number> _
<node-serial-number> _ <date-time-stamp>

Update drive firmware

After you install a new drive in a node, update drive firmware.
This procedure explains how to update drive firmware on clusters running OneFS 7.1.1
or later. If your cluster is running an earlier version of OneFS, you must download and
install the latest drive firmware package. For more information, see the latest drive
firmware package release notes available on https://support.emc.com/.

24 NL400 Node Upgrade Guide

 Upgrade a node

Note

Do not restart or power off nodes while drive firmware is being updated on the cluster.

Procedure
1. Open a secure shell (SSH) connection to any node in the cluster and log in.
2. Depending on your version of OneFS, run one of the following commands to
update the drive firmware for your cluster:
OneFS 8.0 or later
To update the drive firmware for your entire cluster, run the following
command:
isi devices drive firmware update start all --node-lnn
all
To update the drive firmware for a specific node only, run the following
command:
isi devices drive firmware update start all --node-lnn
<node-number>
OneFS 7.1.1 - OneFS 8.0
For OneFS versions between 7.1.1 - 8.0 you will need to run the following
command on each node that requires drive firmware:
isi devices -a fwupdate

CAUTION

You must wait for one node to finish updating before you initiate an update
on the next node. To confirm that a node has finished updating, run the
following command:
isi devices -d <node-number>
A drive that is still updating will display a status of FWUPDATE.

Updating the drive firmware of a single drive takes approximately 15 seconds,

depending on the drive model. OneFS updates drives sequentially.

Verify a drive firmware update

After you update the drive firmware in a node, confirm that the firmware is updated
properly and that the affected drives are operating correctly.
Procedure
1. Ensure that no drive firmware updates are currently in progress by running one
of the following commands:
OneFS 8.0 or later

isi devices drive firmware update list

Verify a drive firmware update 25

Node Upgrade Guide

Earlier than OneFS 8.0

isi devices

If a drive is currently being updated, [FW_UPDATE] appears in the status

column.
2. Verify that all drives have been updated by running the following command:
OneFS 8.0 or later

isi devices drive firmware list

Earlier than OneFS 8.0

isi drivefirmware status

If all drives have been updated, the Desired FW column is empty.

3. Verify that all affected drives are operating in a healthy state by running the
following command:
OneFS 8.0 or later

isi devices drive list --node-lnn all

Earlier than OneFS 8.0

isi devices

If a drive is operating in a healthy state, [HEALTHY] appears in the status

column.

Reviewing the upgrade

After you upgrade the required nodes, review the status of the cluster and the
configuration of all storage pools.

Confirm new node configuration

Confirm that the nodes were upgraded successfully by viewing the current hardware
configuration.
Procedure
1. View the current hardware configuration, including installed RAM, by running
the following command:
isi_hw_status -iIc

26 NL400 Node Upgrade Guide

 Upgrade a node

2. View the installed drives by running the following command:

isi_radish -q

3. View the installed network interfaces by running the following command:

ifconfig | grep flags

4. View the current storage pools by running the following command:

isi status -d

Note

After you upgrade all the nodes in a node pool, the nodes should become part of
a new pool. It is possible that the old node pool will remain visible with no nodes
in it. If you see this issue, contact EMC Isilon Technical Support for assistance.

5. View the hardware as-built record by running the following command:

isi_inventory_tool --display --itemType=asbuilt

6. Confirm that the receipt was installed by running the following command:
isi_inventory_tool --display --itemType=receipt

7. Verify and upload the as-built record by running the following command:
isi_cto_update --abr

This command sends the as-built record report through SupportIQ.

Results
The output of these commands should match the new configure-to-order (CTO)
configuration.

Review SmartPools policies

Following an upgrade, it is recommended that you review SmartPools settings.
Procedure
1. Review all the SmartPools policies to ensure that they accommodate the new
cluster configuration.
For instructions on how to view and adjust file pool policies, see the OneFS CLI
Administration Guide.

Review switch settings

Following a NIC upgrade, we recommend that you review switch settings.
Procedure
1. Review all switch settings to ensure that they accommodate the new NICs.
For instructions on how to configure a switch, refer to the manufacturer's
documentation.

Gather logs
After you complete maintenance on a cluster, gather cluster logs.
You must collect cluster logs after all maintenance. Cluster logs provide snapshots of
the cluster that you can review to make sure that maintenance is successful.

Review SmartPools policies 27

Node Upgrade Guide

Procedure
1. Gather cluster logs by typing the command:
isi_gather_info

Update the install database

After all work is complete, update the install database.
Procedure
1. Browse to the EMC Product Registration and Install base Maintenance
service portal, at: http://emc.force.com/createPSCcase.
2. Select the Product Registration and Install Base Maintenance
option.
3. To open the form, select the IB Status Change option.
4. Complete the form with the applicable information.
5. To submit the form, click Submit.

Roll back a failed upgrade

If you encounter an issue during an upgrade engagement, you can roll back the
upgrade to the original hardware configuration.
If you need to roll back an upgrade, contact EMC Isilon Technical Support for
assistance.

CAUTION

You can attempt to roll back an upgrade only if the upgrade package is still
installed. If you have already uninstalled the upgrade package, you cannot roll
back the upgrade.

Procedure
1. Change directory to the location of the upgrade package by running the
following command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

2. Run the following command:

./isi_cto_upgrade --rollback

Note

If the cluster is in SmartLock compliance mode, run the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --rollback

28 NL400 Node Upgrade Guide

 Upgrade a node

The command output is similar to the following example:

Verifying package integrity

Package integrity is good
Node CTO capability verified
Chassis VPD verified
Node S/N verified
Importing receipt (./rollback_receipt.xml)
Success
Rollback receipt imported
HAL re-initialized
Unprovisioning drives, will get reprovisioned on reboot
...

At the end of the command output, you are prompted to power down the node.

3. Power down the node and replace the original hardware.

Where to go for support

Contact EMC Isilon Technical Support for any questions about EMC Isilon products.

Online Support Live Chat Create a Service Request

Telephone l United States: 1-800-SVC-4EMC (800-782-4362)
Support
l Canada: 800-543-4782
l Worldwide: +1-508-497-7901
l Local phone numbers for a specific country are available at
EMC Customer Support Centers

Help with Online For questions specific to EMC Online Support registration or
Support access, email support@emc.com.
Isilon Info Hubs For the list of Isilon info hubs, see the page on the EMC Isilon
Community Network, EMC Isilon Community Network. Isilon info
hubs organize Isilon documentation, videos, blogs, and user-
contributed content into topic areas, making it easy to find
content about subjects that interest you.

Support for IsilonSD Edge

If you are running a free version of IsilonSD Edge, community support is available
through the EMC Isilon Community Network. However, if you have purchased one or
more licenses of IsilonSD Edge, you can contact EMC Isilon Technical Support for
assistance, provided you have a valid support contract for the product.

Where to go for support 29

Node Upgrade Guide

Copyright © 2017 EMC Corporation All rights reserved.

Published May 2017

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH
RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION
REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
Published in the USA.

30 NL400 Node Upgrade Guide