Ifs Pub Upgrade Planning and Process Guide en Us

PowerScale OneFS
Upgrade Planning and Process Guide

8.1.0.0 - 9.5.0.0
January 2023
Notes, cautions, and warnings
NOTE: A NOTE indicates important information that helps you make better use of your product.
CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid
the problem.
WARNING: A WARNING indicates a potential for property damage, personal injury, or death.
© 2013 - 2023 Dell Inc. or its subsidiaries. All rights reserved. Dell Technologies, Dell, and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners.
Contents
Chapter 1: Introduction to this guide............................................................................................. 6

About this guide................................................................................................................................................................... 6
Where to get help................................................................................................................................................................6
Additional options for getting help............................................................................................................................ 6
Chapter 2: Planning an upgrade.....................................................................................................7

Upgrade planning overview ..............................................................................................................................................7
Planning an upgrade - summary................................................................................................................................. 7
Prerequisites for completing an upgrade....................................................................................................................... 7
Review required documentation...................................................................................................................................... 8
Types of OneFS upgrades................................................................................................................................................. 8
Parallel upgrades............................................................................................................................................................ 8
Rolling upgrades.............................................................................................................................................................9
Simultaneous upgrades................................................................................................................................................ 9
Verify the upgrade path..................................................................................................................................................... 9
Upgrade Paths for supported versions of OneFS................................................................................................ 10
Check supportability and compatibility requirements............................................................................................... 10
Assess upgrade impact..................................................................................................................................................... 10
Consider upgrade limitations...................................................................................................................................... 11
Review feature changes and known issues............................................................................................................ 11
Plan an upgrade schedule................................................................................................................................................. 11
Chapter 3: Completing pre-upgrade tasks....................................................................................12

Pre-upgrade process - Overview................................................................................................................................... 12
Pre-upgrade - Summary............................................................................................................................................. 12
Collect cluster information...............................................................................................................................................13
Review and save information about cluster status .............................................................................................13
Gathering cluster logs................................................................................................................................................. 13
Check cluster readiness....................................................................................................................................................14
Check hardware health - web administration interface......................................................................................14
Check hardware health - CLI.................................................................................................................................... 14
Check the available free space................................................................................................................................. 15
Resolve outstanding events and errors.................................................................................................................. 16
Verify configurations and settings................................................................................................................................. 17
Upgrading in a Cloudpools environment................................................................................................................. 17
Preserve the Kerberos keytab file............................................................................................................................17
Install a supported version of DataIQ or InsightIQ............................................................................................... 17
Download the OneFS installation bundle file............................................................................................................... 17
Upgrade compatibility check utility................................................................................................................................18
Run the upgrade compatibility check utility...........................................................................................................18
Upgrade compatibility check summary................................................................................................................... 19
Reconfigure unsupported SMB settings................................................................................................................ 19
On-Cluster Analysis tool.................................................................................................................................................. 20
HealthCheck tool...............................................................................................................................................................20
Contents 3
Backup data........................................................................................................................................................................20
SyncIQ backup............................................................................................................................................................. 20
NDMP backup.............................................................................................................................................................. 20
Back up custom settings............................................................................................................................................ 21
Complete or stop jobs in progress................................................................................................................................. 21
Complete system jobs................................................................................................................................................ 22
Update Node Firmware Package...................................................................................................................................22
Update Drive Support Package..................................................................................................................................... 22
Configure IPMI ports........................................................................................................................................................ 22
Disable SupportAssist or Secure Remote Services (SRS)...................................................................................... 22
Chapter 4: Performing the OneFS upgrade.................................................................................. 23

Upgrade process overview............................................................................................................................................. 23
Upgrade - Summary....................................................................................................................................................23
Completing a parallel upgrade of OneFS..................................................................................................................... 23
Performing a parallel upgrade using the web administration interface.......................................................... 24
Performing a parallel upgrade using the CLI......................................................................................................... 24
Client connections during parallel upgrades......................................................................................................... 25
Completing a rolling upgrade of OneFS....................................................................................................................... 26
Perform a rolling upgrade using the web administration interface................................................................. 26
Perform a rolling upgrade using the CLI................................................................................................................ 27
Configuration changes during rolling upgrades....................................................................................................28
Client connections during rolling upgrades........................................................................................................... 28
Completing a simultaneous upgrade of OneFS.......................................................................................................... 28
Perform a simultaneous upgrade using the web administration interface.................................................... 28
Perform a simultaneous upgrade using the CLI................................................................................................... 29
Adding a node to the cluster while an upgrade is in progress................................................................................29
Committing an upgrade of OneFS................................................................................................................................ 29
Commit an upgrade using the web interface....................................................................................................... 30
Commit an upgrade using the CLI........................................................................................................................... 30
Rolling back an upgrade of OneFS................................................................................................................................30
Roll back an upgrade using the web interface..................................................................................................... 30
Roll back an upgrade using the CLI.........................................................................................................................30
Verify the OneFS installation using the CLI.................................................................................................................31
Chapter 5: Completing post-upgrade tasks................................................................................. 32

Post-upgrade process - Overview................................................................................................................................ 32
Post-upgrade - Summary.......................................................................................................................................... 32
Allow the Upgrade job to run..........................................................................................................................................32
Verify operational status using the CLI........................................................................................................................32
Re-establish user privileges............................................................................................................................................ 34
Check client connections, and test the workflow.....................................................................................................34
Verify Kerberos migration................................................................................................................................................34
Verify Kerberos migration using the web administration interface................................................................. 34
Verify Kerberos migration using the CLI................................................................................................................35
Reapply custom settings................................................................................................................................................. 35
Test custom scripts.......................................................................................................................................................... 35
Reinstall Aspera................................................................................................................................................................. 35
Install the latest roll-up patch........................................................................................................................................ 35
4 Contents
Chapter 6: Troubleshooting your upgrade....................................................................................37
Troubleshooting an upgrade........................................................................................................................................... 37
Appendix A: Features in OneFS Releases..................................................................................... 38

OneFS 9.5.0.0.................................................................................................................................................................... 38
New features................................................................................................................................................................ 38
OneFS 9.4.0.0..................................................................................................................................................................... 41
New features.................................................................................................................................................................41
OneFS 9.3.0.0.................................................................................................................................................................... 42
New features................................................................................................................................................................ 43
OneFS 9.2.1.0..................................................................................................................................................................... 44
New features................................................................................................................................................................ 44
OneFS 9.2.0.0.................................................................................................................................................................... 45
New features................................................................................................................................................................ 45
OneFS 9.1.0.0..................................................................................................................................................................... 46
Modifications and enhancements in OneFS 9.1.0.0............................................................................................ 46
OneFS 9.0.0.0.................................................................................................................................................................... 47
Changes in OneFS 9.0.0.0........................................................................................................................................ 48
Appendix B: Known Issues in OneFS Releases..............................................................................49

OneFS 9.5.0.0.................................................................................................................................................................... 49
Known Issues................................................................................................................................................................49
OneFS 9.4.0.0.....................................................................................................................................................................51
Known Issues.................................................................................................................................................................51
OneFS 9.3.0.0.................................................................................................................................................................... 52
Known Issues................................................................................................................................................................52
OneFS 9.2.1.0..................................................................................................................................................................... 54
Known Issues................................................................................................................................................................54
OneFS 9.2.0.0.................................................................................................................................................................... 56
Known issues................................................................................................................................................................56
OneFS 9.1.0.0..................................................................................................................................................................... 57
Known issues in OneFS 9.1.0.0................................................................................................................................ 58
OneFS 9.0.0.0.................................................................................................................................................................... 62
File system.................................................................................................................................................................... 62
Hardware....................................................................................................................................................................... 62
Networking....................................................................................................................................................................62
Protocols....................................................................................................................................................................... 63
Security known issues................................................................................................................................................ 64
SmartPools....................................................................................................................................................................64
SMB................................................................................................................................................................................ 64
Upgrades and Installation.......................................................................................................................................... 65
OneFS Web UI ............................................................................................................................................................ 65
Contents 5
1
Introduction to this guide
Topics:
• About this guide
• Where to get help
About this guide

This guide provides important information and steps that you must review and follow when upgrading your OneFS 8.1.2.x
through 9.x clusters.
NOTE: For upgrades from OneFS 8.1.1.x and earlier, please contact your account team.
Where to get help

The Dell Technologies Support site (https://www.dell.com/support) contains important information about products and
services including drivers, installation packages, product documentation, knowledge base articles, and advisories.
A valid support contract and account might be required to access all the available information about a specific Dell Technologies
product or service.
Additional options for getting help

This section contains resources for getting answers to questions about PowerScale products.
Dell Technologies Support ● https://www.dell.com/support/incidents-online/en-us/contactus/product/

isilon-onefs
Telephone support ● United States: 1-800-SVC-4EMC (1-800-782-4362)
● Canada: 1-800-543-4782
● Worldwide: 1-508-497-7901
● Local phone numbers for a specific country or region are available at https://
www.dell.com/support/incidents-online/en-us/contactus/product/isilon-onefs.
PowerScale OneFS Documentation Info ● https://www.dell.com/support/kbdoc/en-us/000152189/powerscale-onefs-info-
Hubs hubs
Dell Community Board for self-help ● https://www.dell.com/community
6 Introduction to this guide

2
Planning an upgrade
Topics:
• Upgrade planning overview
• Prerequisites for completing an upgrade
• Review required documentation
• Types of OneFS upgrades
• Verify the upgrade path
• Check supportability and compatibility requirements
• Assess upgrade impact
• Plan an upgrade schedule
Upgrade planning overview

This chapter provides information to help you create an upgrade plan. Your upgrade plan helps you estimate the time that it
takes to complete tasks and determine who must complete them.
Create an upgrade plan by evaluating how the upgrade affects users and workflow, assessing the impact of the new version of
OneFS on your cluster, and analyzing upgrade risks.
NOTE: To provide you with the best user experience, Dell Technologies recommends using Dell Remote Proactive Services
for upgrade assistance. The services team is available to perform OneFS pre-upgrade health checks and can complete the
OneFS upgrade remotely for most customers. Contact your Account Team for more information.
Planning an upgrade - summary

The following is a summary of steps to perform during the upgrade planning phase:
1. Review required documentation.
2. Determine the type of upgrade.
3. Verify the upgrade path.
4. Check supportability and compatibility.
5. Assess upgrade impact.
6. Schedule the upgrade.
7. (Optional) Upgrade a test cluster.
Prerequisites for completing an upgrade

Permissions and interfaces
To complete the tasks described in this guide, you must be able to log in to the cluster as the root user through the following
interfaces:
● OneFS command-line interface
● OneFS web administration interface
NOTE: If you are not logged into the OneFS cluster with root privileges, you might not be able to run all of the commands in
this guide.
Planning an upgrade 7
Review required documentation
Reviewing the documentation in this list helps you to understand the upgrade process and the impact the upgrade could have on
your workflow. Some links require a support login.
Required documentation
● Current PowerScale Software Releases
Check on current OneFS releases, release dates, and which version has the Target Code status.
● PowerScale OneFS Release Notes and Maintenance Release Notes
Read the OneFS release notes for each version between your current version and your target version for information about
new features, changes, resolved issues, and known issues.
NOTE: The new features and known issues for all supported OneFS versions have been compiled into this guide in the
following appendices: Features; Known Issues.
● PowerScale OneFS Current Patches

Review patches that have been released for the version of OneFS to which you are upgrading.
● PowerScale Supportability and Compatibility Guide
Confirm that your PowerScale software and PowerScale hardware is compatible with the version of OneFS to which you are
upgrading.
Other documentation
● OneFS Upgrade Process Troubleshooting guide
Review this troubleshooting guide if you encounter issues during your upgrade.
● OneFS Technical and Security Advisories
Determine whether any PowerScale Technical Advisories or Security Advisories have been issued for the version of OneFS
to which you are upgrading.
Types of OneFS upgrades

Depending on the version of OneFS installed on your cluster, there are different types of upgrades available: parallel upgrades,
rolling upgrades, or simultaneous upgrades.
NOTE: See the Verify the upgrade path topic in this guide for information about which types of upgrades are supported
between OneFS versions.
Parallel upgrades
Dell Technologies recommends using the parallel upgrade option when upgrading any size cluster running OneFS 8.2.2 and later.
Parallel upgrades require a smaller maintenance window than rolling upgrades, and do not require the interruption of service like
simultaneous upgrades.
A parallel upgrade installs the new operating system on a subset of nodes and restarts that subset of nodes at the same time.
Each subset of nodes attempts to make a reservation for their turn to upgrade until all nodes are upgraded. Node subsets and
reservations are based on diskpool and node availability.
During a parallel upgrade, node subsets that are not being upgraded remain online and can continue serving clients. However,
clients that are connected to a restarting node are disconnected and reconnected. How the client connection behaves when
a node is restarted depends on several factors including client type, client configuration (mount type, timeout settings), IP
allocation method, and how the client connected to the cluster. In OneFS 9.2.0.0 and later, client connection behavior is
managed by the disruption manager settings.
8 Planning an upgrade
Rolling upgrades
A rolling upgrade installs the new operating system and restarts each node individually in the OneFS cluster so that only one
node is offline at a time. A rolling upgrade takes longer to complete than a simultaneous upgrade. During a rolling upgrade, nodes
that are not currently being upgraded remain online and can continue serving clients. However, clients that are connected to
a restarting node are disconnected and reconnected. How the client connection behaves when a node is restarted depends
on several factors including client type, client configuration (mount type, timeout settings), IP allocation method, and how the
client connected to the cluster.
Simultaneous upgrades
A simultaneous upgrade installs the new operating system and restarts all nodes in the OneFS cluster at the same time.
Simultaneous upgrades are faster than rolling upgrades but require a temporary interruption of service during the upgrade
process. All client connections to the cluster must be terminated before completing the upgrade and data is inaccessible until
the installation of the new OneFS operating system is complete and the cluster is back online.
Verify the upgrade path

Verify that the current version of OneFS on your cluster can be upgraded to the target version.
All upgrades follow one of these scenarios:
● Upgrade to the patched target OneFS version
● Upgrade to the target OneFS version, and then add patches
● Upgrade to an intermediate OneFS version, upgrade to the target OneFS version, and then add patches
To view which version of OneFS is running on all nodes in your cluster, run the following command:
isi_for_array uname -r
Find your version of OneFS in the supported OneFS Upgrade Paths matrix and confirm which versions of OneFS are supported
upgrade paths.
Upgrade Paths for supported versions of OneFS
The following table can be used to determine the supported upgrade path from the current version of OneFS on your cluster to
a target version of OneFS. The upgrade paths table displays the supported upgrade paths from major OneFS version to major
OneFS version, but does not include patch information.
Key:
● "O" : Rolling and Simultaneous upgrades available
● "=" : Parallel, Rolling, and Simultaneous upgrades available
● "x" : Unsupported upgrade path
Table 1. OneFS Upgrade Paths

Upgrade to target version
Upgrade 8. 8. 8. 8. 9. 9. 9. 9. 9. 9. 9.
from 1. 2. 2. 2. 0. 1. 2. 2. 3. 4. 5.
current 3 0 1 2 0 0 0 1 0 0 0
version
8.1.2 O O O O O O O O x x x
8.1.3 x O O O O x x x x x
8.2.0 O O O O O O O O O
8.2.1 O O O O O O O O
8.2.2 = = = = = = =
9.0.0 = = = = = =
9.1.0 = = = = =
9.2.0 = = = =
9.2.1 = = =
9.3.0 = =
9.4.0 =
9.5.0
NOTE: These supported upgrade paths do not guarantee bug fix and feature parity. For more information about using
patches to achieve bug fix and feature parity, contact your account team.
NOTE: If you are upgrading from OneFS 8.1.x and using Mellanox switches, it is recommended that you first upgrade to
OneFS 8.2.2 before upgrading to the OneFS 9.x family. For more information, see the Flexnet unable to assign IP addresses
KB article.
NOTE: If you are upgrading from an earlier version of OneFS, contact your account team for assistance.
Check supportability and compatibility requirements

Review the PowerScale Supportability and Compatibility Guide to confirm that your current hardware components, software
components, and protocol versions are compatible with the version of OneFS to which you are upgrading.
For information about OneFS compatibility with Hadoop, see the Hadoop - Info Hub page.
Assess upgrade impact

Consider all the areas of your environment that might be affected by upgrading to a new version and plan a strategy for
performing the tasks that must be completed and addressing issues that might occur. Assessing the impact of the upgrade on
your environment, clients, and performance helps ensure that the upgrade does not disrupt your cluster, workflow, or users.
10 Planning an upgrade
NOTE: If you have a performance-sensitive workload, PowerScale recommends that you consult your Sales Engineer for
assistance during the pre-upgrade planning process.
Consider upgrade limitations

If the upgrade cannot be completed for any reason—for example, if there is insufficient space on the cluster or if the upgrade
process detects a stalled drive—the system will revert to the current version and the upgrade will be cancelled. Preparing your
cluster as recommended in this guide will help you to avoid situations that might result in a cancelled upgrade.
NOTE: In OneFS 8.2.0 and newer, you have the ability to pause and resume the upgrade process in order to resolve
blocking issues.
Review feature changes and known issues

Familiarize yourself with new, modified, and deprecated features as well as known issues in the target version of OneFS to
which you are upgrading.
Functionality changes and new features in the target version might impact the performance, configuration, or functionality of
your cluster after upgrading. If the target version contains an issue that affects your environment, you should consider waiting
to upgrade until after the issue is resolved in a later version of OneFS.
See the Release Notes for a summary of new features, feature changes, and known issues in the target version of OneFS. It
is recommended to also review the release notes for each OneFS release between your current OneFS version and the target
version.
NOTE: The new features and known issues for all supported OneFS versions have been compiled into this guide in the
following appendices: Features; Known Issues.
Plan an upgrade schedule

An upgrade schedule can help ensure that the upgrade goes smoothly. The schedule should consider all the factors that go into
an upgrade and estimate how long each stage of the upgrade process might take.
The upgrade process begins with ensuring that your OneFS cluster is ready to be upgraded. It is recommended that you set
aside two weeks to check the cluster health and resolve software and hardware issues before performing the upgrade itself.
Upgrade The maintenance window should encompass the pre-upgrade, upgrade, and post-upgrade phases.
maintenance Estimate the time that it takes to run the upgrade considering cluster size and upgrade type (parallel,
window rolling, or simultaneous). Schedule time to inform users when the upgrade will take place and that client
connections might be slow, file access might be affected, and clients might be disconnected. A best
practice is to upgrade the cluster during an off-hours maintenance window.
Schedule time for node and drive health checks and replacement of bad hardware. Include time to update
configurations and settings that are not supported in the new version.
Estimate the time that it takes to back up your data, considering cluster size, number of files, types of
files, and file size. Also include time to collect information about the cluster such as status, logs, and
settings.
If performing a parallel or rolling upgrade, consider whether you will configure client connection drain
times, which will extend the required maintenance window, but lower the impact on client connections.
Build in time to let the upgrade jobs run to completion and to reestablish permissions and connections.
Schedule time or extend the maintenance window to accommodate post-upgrade tasks such as
reconfiguring custom settings, updating scripts to reflect command and functionality changes in the
upgrade version, and potential troubleshooting.
(Optional) If available, upgrading a test cluster with the same current version of OneFS before you upgrade your
Upgrade a test production cluster can expose issues that could slow down or prevent the upgrade of your production
cluster system.
After you upgrade a test cluster, verify that the cluster is operational and validate key workflows on the
test cluster by simulating how administrators, users, and applications interact with the system.
3
Completing pre-upgrade tasks
Topics:
• Pre-upgrade process - Overview
• Collect cluster information
• Check cluster readiness
• Verify configurations and settings
• Download the OneFS installation bundle file
• Upgrade compatibility check utility
• On-Cluster Analysis tool
• HealthCheck tool
• Backup data
• Complete or stop jobs in progress
• Update Node Firmware Package
• Update Drive Support Package
• Configure IPMI ports
• Disable SupportAssist or Secure Remote Services (SRS)
Pre-upgrade process - Overview

Performing the pre-upgrade tasks described in this guide helps to ensure that the OneFS cluster hardware, software modules,
configuration, features, and file system do not have preexisting issues that could adversely affect the upgrade process.
Performing the pre-upgrade tasks also ensures that important cluster data is collected and any custom configuration settings
are recorded.
Pre-upgrade - Summary
The following is a summary of steps to perform during the pre-upgrade phase:
1. Collect cluster information.
a. Collect cluster status.
b. Gather cluster logs.
c. Check cluster hardware health.
d. Check cluster available space.
2. Resolve events and errors.
3. Verify cluster configuration.
a. Preserve the Kerberos keytab file.
b. Install DataIQ or InsightIQ.
4. Download the OneFS installation file.
5. Run the upgrade compatibility check utility.
a. Reconfigure unsupported SMB settings.
6. Run the On-Cluster Analysis tool.
7. Run the HealthCheck tool.
8. Back-up cluster data.
9. Complete system jobs.
10. Update Node Firmware Package.
11. Update Drive Support Package.
12. Configure IPMI ports.
13. Disable SupportAssist.
12 Completing pre-upgrade tasks

Collect cluster information
Before you upgrade, collect and record key information about your OneFS cluster and how it is configured.
Review and save information about cluster status

Before you upgrade, run the isi status command to get the status of your cluster, events, and jobs then save the
information to a file.
Run the isi status > /ifs/data/isi_status_output command to save the output of the isi status command
to a file named isi_status_output in the /ifs/data directory.
NOTE: Run the isi status command to view the status of the cluster, events, and jobs without saving the output to a
file.
Gathering cluster logs

You can gather cluster logs and send the logs to Dell Technologies Support for analysis. Cluster logs can be sent automatically or
manually through the cluster command-line and web administration interfaces.
NOTE: Your cluster must be connected to the internet to be able to send log files directly. In newer versions of OneFS,
you must also have remote support and SRS enabled. If your cluster does not have an internet connection or if your upload
has failed, you can copy the log file from the cluster and upload the log file with an FTP client to an FTP server. For more
information about how to gather and send cluster log files, including the command parameters for configuring how logs are
uploaded to Dell Technologies Support, refer to the administration guide for your version of OneFS.
Gather cluster logs - Web UI

You must have root access to run log gathering commands.
1. To gather the log files, go to Cluster Management > Diagnostics.
2. Click Start Gather to open the Start Gather window.
3. Configure the log gather and click Start Gather.
For information about configuring the log gather, see the administration guide for your version of OneFS.
After the log gathering process is complete, a link to the file that the process generates appears under Filename in the
Archived Info Manager section.
Gather cluster logs - CLI

You must have root access to run log gathering commands.
To gather the log files in OneFS 9.0.0.0 and earlier, run the following command:
isi_gather_info
To gather the log files in OneFS 9.1.0.0 and later, run the following command:
isi diagnostics gather start
The files generated during the log gathering process are stored on the cluster in the /ifs/data/Isilon_Support/pkg
directory.
Completing pre-upgrade tasks 13

Check cluster readiness
Completing pre-upgrade tasks such as checking the health of the hardware components on the cluster, ensuring that storage
space requirements are met, and managing outstanding cluster events and errors are recommended for a successful upgrade.
Check hardware health - web administration interface

Use the OneFS web administration interface to evaluate the health of the clusters hardware components. Log in as root user
and perform the following steps on your cluster:
1. Click Cluster Management > Hardware Configuration.
2. In the Nodes tab, check that each node has a healthy status.
3. In the Drives tab, check that each drive has a healthy status.
4. If any correctable or fatal errors are reported, do not continue the upgrade. If the errors cannot be resolved, contact Dell
Technologies Support
NOTE: Dell Networking switches running Dell Network Operating System (DNOS) version 10.5.0.6C1 or earlier, that are
used for back-end traffic and configured as a single switch (also known as a flat switch, or a Top of Rack (TOR) switch)
might be affected by a certificate expiration issue. For more information, including a list of affected switches and a
workaround, see DTA 190714.
Check hardware health - CLI

Run the following commands from the OneFS command-line interface as root user to evaluate the health of the cluster's
hardware components and the status of job engine jobs.
1. To return information about cluster health and check for jobs or devices that report a status of ATTENTION, SMARTFAIL,
or DOWN.
isi status -v
2. To check for drives that do not report a status of HEALTHY, L3, or JOURNAL.
isi_for_array -s 'isi devices drive list | egrep -v "HEALTHY|L3|JOURNAL"'
3. To check the mirror status of the boot drives on each node.
isi_for_array -s 'gmirror status'
NOTE: If a drive is degraded, do not continue with the upgrade until the issue is resolved.
4. If the cluster has an InfiniBand network, confirm whether a node has been assigned the OpenSM (subnet manager) main
role.
isi_for_array -s 'ps -auwwwx | grep opensm' | grep master
Confirm that the output displays only one node in the cluster as the main (opensm). The output should be similar to the
following:
node-2: root 1610 0.0 2.3 436292 384672 ?? Ss 20May15 97:31.63

opensm: 0x00151b00007a671b main (opensm)
NOTE:
If the command does not return output that includes a node in the cluster as the main (opensm), then another device
has assumed the OpenSM main role. In that case, confirm that cables from another cluster are not connected to this
cluster's primary or secondary InfiniBand switch, and then run the command again. For more information about this
OpenSM issue, see KB 192527.

NOTE: For information regarding the Number of OpenSM Masters does not match the number of
InfiniBand Interfaces in use error, see KB Article 167821.
NOTE: If you are using the Dell S4112F-ON, Dell S4148F-ON, Dell Z9100-ON, or Dell Z9264-ON as back-end network
switches, see DTA 190714 for a workaround to a known upgrade issue.
Check the available free space

Ensure that the minimum available-space requirements for the cluster, nodes, node pools, and critical directories are met before
you upgrade. Do not continue with the upgrade if the minimum available-space requirements are not met.
Available space requirements for clusters, nodes, node pools, and

directories
The cluster, nodes, node pools, and several critical directories in the file system must meet available space requirements prior to
upgrading OneFS. If the available space for any of these items falls below the minimum requirement, make more space available
before upgrading. Otherwise, the upgrade process will fail and might not return an error indicating available space as the cause.
Item Minimum Requirement Task

Cluster The cluster cannot be more than Make more available space.
90 percent capacity.
Node Each node cannot be more than Make more available space.
92 percent capacity.
Node pool Each node pool cannot be more Make more available space.
than 90 percent capacity.
root partition (/) The root partition cannot be more If this directory is at or near the minimum available-
than 97 percent capacity. space requirement, see the following resources for
steps to address the issue:
● Knowledge base article 464118.
● Event ID 100010003, The /(root) partition is near
capacity.
/ifs The /ifs directory cannot be If this directory is at or near the minimum available-
more than 90 percent capacity. space requirement, see the following resources for
● Event ID 100010004, The cluster's /ifs partition
is near capacity.
/var The /var partition cannot be If this directory is at or near the minimum available-
more than 90 percent capacity. space requirement, see the following resources for
● Event ID 100010001, The /var partition is near
capacity.
/var/crash The /var/crash directory If this directory is at or near the minimum available-
cannot be more than 90 percent space requirement, see the following resources for
capacity. steps to address the issue:
● Event ID 100010002, The /var/crash partition is
near capacity.

Confirm used and available space
● To confirm how much space is being used on the cluster and on each node, run the following command:
isi stat
● To confirm how much space is being used in each node pool, run the following command:
isi stat -p
● To confirm how much space is being used by critical directories on the cluster, run the following command:
isi_for_array -s 'df -h'
The isi_for_array output is similar to the following for each node in the cluster:
virtual-demo-3: Filesystem Size Used Avail Capacity Mounted on

--------------------------------------------------------------------
virtual-demo-3: /dev/imdd0a 495M 418M 38M 92% /
virtual-demo-3: devfs 1.0K 1.0K 0B 100% /dev
virtual-demo-3: /dev/imdd1a 495M 6.7M 449M 1% /var
virtual-demo-3: /dev/imdd2a 496M 5.9M 451M 1% /var/crash
virtual-demo-3: OneFS 40G 283M 20G 1% /ifs
virtual-demo-2: Filesystem Size Used Avail Capacity Mounted on

--------------------------------------------------------------------
virtual-demo-2: /dev/imdd0a 495M 417M 38M 92% /
virtual-demo-2: devfs 1.0K 1.0K 0B 100% /dev
virtual-demo-2: /dev/imdd1a 495M 6.1M 449M 1% /var
virtual-demo-2: /dev/imdd2a 496M 5.9M 451M 1% /var/crash
virtual-demo-2: OneFS 40G 283M 20G 1% /ifs
Resolve outstanding events and errors

Before you begin your upgrade, resolve any outstanding critical events, errors, and failures, as these issues can disrupt the
OneFS upgrade process.
1. Run the following command to view all events on the cluster:
isi event groups list --sort=severity
● If the command returns any critical errors, check the log files in the following directories for more information:
○ /var/log
○ /var/log/messages
○ /var/crash
● NOTE: If any log files contain messages about a dynamic sector recovery (DSR) failure or a Data Integrity (IDI)
failure, contact PowerScale before you upgrade.
2. Cancel non-critical events before upgrading to prevent a recurrence of notifications that you know to be harmless.
If you have critical events that you are unable to resolve, contact support before upgrading.

Verify configurations and settings
Confirm that your OneFS configurations for CloudPools, NIC aggregation, Kerberos, InsightIQ, and SMB are supported by the
target version of OneFS to which your are upgrading.
Upgrading in a Cloudpools environment

If you're upgrading a OneFS 8.x cluster to OneFS 8.2.x. or later with a Cloudpools environment, there are additional items to
consider. For more information on pre-upgrade and upgrade steps, see KB article 000020736
Preserve the Kerberos keytab file

If Kerberos authentication is configured on the cluster, upgrading OneFS might delete the local /etc/<name>.keytab file
that is on each node. Choose one of the following methods to preserve the keytab file during the upgrade:
● Copy the keytab file to the /ifs directory where it can be accessed by all nodes on the cluster instead of saved locally on
each node.
● Add the following text to the /etc/mcp/override/user_preserve_files.xml file of each node on the cluster
where <name> is the name of the keytab file:
<?xml version="1.0" encoding="UTF-8"?>

<user_preserve>
<files>
<file name="etc/allow_unsupported_boot" recursive="no">
</file>
...
...
<file name="etc/<name>.keytab" recursive="no" />
</files>
</user_preserve>
Install a supported version of DataIQ or InsightIQ

Before you upgrade OneFS, confirm the version of DataIQ or InsightIQ you are running is compatible with the target version of
OneFS to which you are upgrading. If the versions are not compatible, upgrade DataIQ or InsightIQ before upgrading OneFS.
See the PowerScale Supportability and Compatibility Guide for OneFS compatibility information.
See the DataIQ - info hub for documents and content related to DataIQ, including release notes, installation guides, user guides,
and troubleshooting guides.
See the InsightIQ - Info hub for documents and content related to InsightIQ.
NOTE: If you are upgrading from OneFS 8.1.x or earlier to OneFS 8.2.x or later, a maximum of 99 FSA reports will be
retained after the upgrade. For more information, see KB article 540000.
Download the OneFS installation bundle file

1. From the OneFS Downloads page on the Online Support site, under Browse Product, select your target version of OneFS.
2. Download the installation bundle file for the target version of OneFS to which you are upgrading.
NOTE: It is recommended that you use a OneFS installation bundle file that includes the latest patches. If the
installation bundle is used, you can skip the patch installation task in the post-upgrade steps. Contact your support
representative for access to an installation file that does not include patches.
NOTE: If you use the web administration interface to upgrade from OneFS 9.3.0.x or earlier, to OneFS 9.4.0.0 or later,
you must first change the file extension from .isi to .tar.gz. (example: change OneFS_v9.4.0.0_Install.isi to
OneFS_v9.4.0.0.Install.tar.gz)

3. Open a secure shell (SSH) connection to any node in the cluster and log in using the root account.
4. Move the installation file that you downloaded into the /ifs/data directory on the cluster you want to upgrade.
5. Validate the integrity of the installation file.
For OneFS 9.4.0.0 and later, use the following instructions:
a. To import the package into the OneFS Catalog, run the isi upgrade catalog import /ifs/data/
<filename>.isi command.
b. To verify the package, run the isi upgrade catalog verify --file /ifs/data/<filename>.isi
command.
For OneFS 9.3.0.0 and earlier, use the following instructions:
a. Optional: On the OneFS downloads page, click Checksum Values and record the SHA-256 checksum value displayed.
b. Optional: In OneFS CLI, run the following command where <installation-file-name> is the name of the downloaded
installation file:
● sha256 /ifs/data/<installation-file-name>
c. Optional: Compare the SHA-256 checksum value that you recorded from the downloads page on the Online Support site
to the checksum value returned from the SHA-256 command. If the values do not match, re-download the installation
file.
6. (OneFS 9.3.0.0 and earlier) To unzip the downloaded tgz file, run the following command, where <installation-file-name> is
the name of the downloaded tgz file:
gzip -d /ifs/data/<installation-file-name>
NOTE: Clusters running OneFS 9.4.0.0 and later are not required to unzip .tgz files.
7. (OneFS 9.3.0.0 and earlier) To unbundle the .tar file, run the following command where <installation-file-name> is the name
of the OneFS install tar file:
tar -vxf /ifs/data/<installation-file-name> /ifs/data/
NOTE: Clusters running OneFS 9.4.0.0 and later are not required to unbundle .tar files.
8. (OneFS 9.3.0.0 and earlier) Validate the manifest file (example: OneFS_v9.3.0.0_Install.tar.gz). See the Checking Manifests
section of the PowerScale OneFS Security Configuration Guide for more information.
NOTE: Clusters running OneFS 9.4.0.0 and later use the OneFS Catalog to verify packages.
Upgrade compatibility check utility

The upgrade compatibility check utility examines cluster settings, activities, and statuses to verify whether your cluster is
compatible with the target version of OneFS to which you are upgrading.
The upgrade compatibility check utility is included in each OneFS installation package. You can run the utility alone or as part of
the upgrade process.
Run the upgrade compatibility check utility

To confirm that your OneFS cluster is compatible with the target version that you are upgrading to, run the upgrade
compatibility check utility.
NOTE: The upgrade compatibility check utility is included in each OneFS installation package, which must be downloaded
and accessible in an /ifs directory.
1. Open a secure shell (SSH) connection to any node in the cluster and log in to the cluster with the root account.
2. Start the upgrade compatibility check utility by running the following command, where <install-image-path> is the file path
of the upgrade installation file.
isi upgrade cluster assess <install-image-path>

NOTE: The upgrade compatibility check utility might take several minutes to run. If the utility returns errors, resolve the
errors before continuing with the upgrade. Warnings are informational and do not prevent an upgrade.
3. View the results of the upgrade compatibility check by running the following command:
isi_upgrade_logs -a
Upgrade compatibility check summary

The upgrade compatibility check utility examines the cluster and returns warnings or errors if something is not compatible with
the version of OneFS to which you are upgrading. The upgrade compatibility check utility examines the following areas:
Check name Description Recommendation

Disk load Checks the cluster usage level and Disconnect all client connections and
returns a warning if the disk load is stop all system jobs before upgrading.
greater than 50%.
Free Space Checks cluster free space and returns Free up space in the necessary partition
a warning if capacity thresholds are before upgrading.
exceeded for the following partitions:
● /
● /var
● /ifs
● Node pools
Drive stalls Checks the health of the drives in the Smartfail and resolve the stalled drive
cluster. issue before upgrading.
Smartfail operation status Determines if a Smartfail operation is Wait for the Smartfail operation to
running on the cluster. complete before upgrading.
IntegrityScan job status Determines if an IntegrityScan job is Wait for the job to complete before
running on the cluster. upgrading.
Unresolved critical alerts Checks for any critical alerts on the Resolve all critical alerts before
cluster. upgrading.
Unsupported SMB configuration Checks for unsupported SMB settings Reconfigure any unsupported SMB
on the cluster. settings before upgrading.
SMB access zone association Checks if any SMB shares are associated Resolve issues where the share paths
with multiple access zones where the overlap before upgrading. The upgrade
share paths overlap. will succeed, but you can't create new
SMB shares until the overlap issue is
resolved.
Kerberos keytab Checks if the Kerberos keytab file will be You must configure the Kerberos
preserved after the upgrade. settings to preserve the keytab file
before upgrading.
HDFS Kerberos keytab Checks if the HDFS Kerberos keytab file You must configure the Kerberos
will be preserved after the upgrade. settings to preserve the keytab file
before upgrading.
Reconfigure unsupported SMB settings

If the SMB settings on the cluster are not supported by the target version of OneFS to which you are upgrading, the upgrade
might fail. The upgrade compatibility check utility confirms whether your current SMB settings are supported.
If the upgrade compatibility check utility detects unsupported SMB settings, remove or modify the unsupported SMB settings
through the command-line interface or web administration interface before you upgrade.
NOTE: SMB1 is disabled during the upgrade process which might cause some clients to have authentication issues.

On-Cluster Analysis tool
The PowerScale On-Cluster Analysis (OCA) tool analyzes the health status of a running cluster and helps you prepare for
an upgrade. The OCA tool analyzes your cluster and displays any issues that it encounters, a summary of the issue, and any
associated KB articles to help you resolve the issue. The OCA tool also has a feature that provides you with an estimated
upgrade plan based on the analysis.
For more information about the OCA tool, please see the KB article: How to run the On-Cluster Analysis tool
HealthCheck tool
The HealthCheck tool enables you to evaluate the status of specific software and hardware components of your cluster and
your cluster environment. For more information about the HealthCheck tool, see this site: HealthCheck Info Hub
Backup data
It is recommended to backup your cluster data immediately before you upgrade. Schedule sufficient time for the back up to
complete before the upgrade window.
SyncIQ backup
SyncIQ is one option you can use to backup your OneFS cluster. SyncIQ creates and references snapshots to replicate a
consistent point-in-time image of a root directory.
For more information about backing up your OneFS cluster, see the OneFS CLI Administration Guide or the OneFS Web
Administration Guide for your version of OneFS.
NOTE: If you are upgrading your cluster from OneFS 8.1.0.x or earlier to OneFS 8.1.1.x, 8.1.2.x, or 8.1.3.x, and your cluster
is in Compliance mode, you must ensure that all SyncIQ partners are on the same code and patch level OneFS before
restarting SyncIQ backups, or the backups fail. This issue is resolved in OneFS 8.2.0 and later.
NOTE: If you are upgrading your cluster to OneFS 8.2.0 through 9.2.1.0, are using SyncIQ, and have non-networked
(NANON) nodes on your source cluster, SyncIQ jobs might fail after the upgrade. For more information including a
workaround, see the KB article SyncIQ jobs failing with NANON after upgrade
NDMP backup
Other OneFS cluster backup options include using the Network Data Management Protocol (NDMP).
From a backup server, you can perform both NDMP three-way backup and NDMP two-way backup processes between a cluster
and backup devices such as tape devices, media servers, and virtual tape libraries (VTLs).
See the OneFS Web Administration Guide or the OneFS CLI Administration Guide for information about backing up data using
NDMP.
Complete NDMP backups

Before you upgrade, you must wait for Network Data Management Protocol (NDMP) backups to finish so you have saved
copies of your data.
NOTE: If you cannot wait for NDMP backups to finish, stop the active NDMP backups 30– 60 minutes before the upgrade,
as the NDMP backup process requires this additional time to come to a stop.
For instructions on how to stop backup jobs, see the documentation for your backup application.
NOTE: If you are upgrading from OneFS version 8.0.x or 8.1.x to OneFS 8.2.x, you must disable the NDMP service before
performing the upgrade.

Back up custom settings
Most settings are preserved during a OneFS upgrade. However, documenting and backing up custom settings enables you to
reapply any settings that are not preserved during the upgrade process.
Document and back up the following custom settings:
Setting Description Recommendation

SMB audit If you have an audit log directory After upgrading, you must reconfigure SMB audit logging. See File
logging in /ifs, you might have custom System Auditing with the Common Event Enabler (CEE) for more
SMB logging settings configured. information about SMB audit logging.
Passwords for After you upgrade, you might Make a list of the local accounts and their passwords before you
local user have to reset the passwords upgrade.
accounts of the local user accounts that
you configured on the cluster.
Other users should be prepared to
reset the passwords of their local
accounts after the upgrade.
sysctl If you changed the default PowerScale does not recommend modifying sysctl parameters
parameters value assigned to one or unless you are instructed to do so by Dell Technologies Support. If
more sysctl parameters you must modify a sysctl parameter, configure the parameter in
by editing the /etc/mcp/ the /etc/mcp/override/sysctl.conf file to ensure that the
override/sysctl.conf file, change is preserved when you upgrade a node or a cluster.
the etc/mcp/templates
Before you upgrade, document your custom sysctl parameters and
directory, or the /etc/local/
back up the /etc/mcp/override/sysctl.conf, the etc/mcp/
sysctl.conf file, you might
templates directory, and the /etc/local/sysctl.conf files.
need to reset the parameter after
you upgrade. For more information about making sysctl changes persist through
upgrades, see KB article 000102543 and KB article 000083411.
If you modified a sysctl
parameter by editing another
file—for example, the /etc/
sysctl.conf file—the change
will not be preserved during the
upgrade.
Aspera You must reinstall and reconfigure Before you upgrade, back up the Aspera configuration files in the
Aspera after you upgrade. following directories:
NOTE: Aspera is no longer ● /ifs/.ifsvar/aspera/etc/
supported In OneFS 8.2.0 and ● /ifs/.ifsvar/aspera/www/
later. ● /usr/local/aspera/var/aspera-prepost
To determine which version of Aspera is compatible with the version
of OneFS to which you are upgrading, see the Supportability and
Compatibility Guide, and then go to the Aspera website to download
the Aspera install files.
Cron jobs Cron jobs settings that were Document and back up custom cron job settings or configure them
not configured in the /etc/mcp/ in the /etc/mcp/override/crontab.smbtime file before you
override/crontab.smbtime upgrade.
file are not preserved during an
upgrade. After you upgrade, you might have to modify a cron job to
accommodate changes to OneFS commands.
Complete or stop jobs in progress

You should ensure that there are no jobs running on your OneFS cluster before beginning the upgrade. Wait for jobs to fully
complete or stop jobs before upgrading.

Complete system jobs
Ensure that no system jobs are running during the upgrade by allowing system jobs to finish before the upgrade starts, or by
cancelling them. OneFS performs system jobs through a service that runs in the background, and if any system jobs are running
during the upgrade, the upgrade process might fail.
1. To check for running system jobs, run the following command, and make a note of the job ID for any jobs that you want to
cancel:
isi job status
2. To cancel a job, run the following command where <job_id> is the ID of the job you want to cancel:
isi job jobs cancel <job_id>
NOTE: Do not cancel the Upgrade, FlexProtect, FlexProtectLin, or IntegrityScan jobs. If any of these four system jobs
are running, you cannot continue with the upgrade. If an Upgrade, FlexProtect, FlexProtectLin, or IntegrityScan system
job takes longer than expected to complete, contact Dell Technologies Support.
NOTE: Sync polices and jobs must be canceled or paused in order for the upgrade to complete successfully.
Update Node Firmware Package

Before upgrading your OneFS cluster, ensure that your nodes are running the most recent version of firmware found in the
Node Firmware Package.
For additional information on updating node firmware, see OneFS Node Firmware Package, available on the Online Support site.
NOTE: In OneFS 9.2.0.0 and later, you can choose to update the Node Firmware Package during a parallel upgrade. If you
choose to perform the Node Firmware update during an upgrade, delay this step until the Performing a Parallel Upgrade
chapter.
Update Drive Support Package

Before upgrading your OneFS cluster, ensure that your drives are running the most recent version of firmware found in the
Drive Support Package.
For more information, see the OneFS Drive Support Package, available on the Online Support site.
Configure IPMI ports

If you have enabled Intelligent Platform Management Interface (IPMI) ports, it is recommended that you change the IPMI
port IP configuration to static in the BIOS settings for each affected node before the upgrade. If you change the IPMI port
configuration during the upgrade process, and your workflow requires the IPMI ports be enabled, you must manually re-enable
the ports after the upgrade process is complete.
NOTE: Use of IPMI ports is supported in OneFS version 8.2.2 and later.
Disable SupportAssist or Secure Remote Services

(SRS)
SupportAssist and SRS allow remote support access to your cluster.
It is recommended that you disable SupportAssist or SRS before you upgrade your OneFS cluster.

4
Performing the OneFS upgrade
Topics:
• Upgrade process overview
• Completing a parallel upgrade of OneFS
• Completing a rolling upgrade of OneFS
• Completing a simultaneous upgrade of OneFS
• Adding a node to the cluster while an upgrade is in progress
• Committing an upgrade of OneFS
• Rolling back an upgrade of OneFS
• Verify the OneFS installation using the CLI
Upgrade process overview

The OneFS upgrade process consists of downloading the OneFS installation image, starting the upgrade, and verifying that the
upgrade is complete.
The following upgrade types are available:
● Parallel upgrade (for OneFS 8.2.2 and later)
● Rolling upgrade
● Simultaneous upgrade
A OneFS installation image is required to upgrade your cluster. See the Download the OneFS image section for more
information.
NOTE: You can upgrade OneFS using the command-line interface or the web administration interface.
Upgrade - Summary
The following is a summary of steps to perform during the upgrade phase:
1. Perform the upgrade.
2. Commit the upgrade.
3. Verify the upgrade.
Completing a parallel upgrade of OneFS

If you perform a parallel upgrade, subsets of nodes within the OneFS cluster are restarted in an undetermined order. During
a parallel upgrade, client connections to the restarting subset of nodes are disconnected, but other subsets of nodes remain
available for client connection.
NOTE: The parallel upgrade feature is available in OneFS version 8.2.2 and later. Upgrades where the source cluster is on
OneFS 8.2.1 or earlier must use the rolling or simultaneous option.
Performing the OneFS upgrade 23

Performing a parallel upgrade using the web administration
interface
You can use the web administration interface to upgrade the version of OneFS that is running on your cluster.
Perform the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the
new version of OneFS.
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click Upgrade OneFS.
4. Enter the file path location of the upgrade image to install. The file path must be accessible in an /ifs directory.
5. (Optional) In OneFS 9.2.0.0 and later, to update node firmware during the upgrade, enter the file path location of the node
firmware package.
6. In the Upgrade Type drop-down list, select Parallel Upgrade.
7. (Optional) To skip optional pre-upgrade checks related to compatibility, select Skip optional pre-upgrade checks.
8. (Optional) In OneFS 9.2.0.0 and later, to manage disruption to SMB client connections, follow these steps:
a. Check Enable disruption manager
b. (Optional) Check Wait Forever to wait for all SMB clients to disconnect from a node before upgrading.
c. Enter the Drain time out to specify the amount of time OneFS waits for clients to manually disconnect before the node
reboots and upgrades.
d. Enter the CELOG time out to specify the amount of time OneFS waits for clients to disconnect before alerting the
OneFS administrator. This allows the administrator to work with the connected clients before rebooting the node.
9. Click Start Upgrade to begin the upgrade.
The cluster might display several confirmation messages. Confirm each message to continue the upgrade process.
NOTE: In 9.2.0.0 and later, you can use the upgrade status screen to monitor the progress of your upgrade, view
connected clients, and delay draining clients from specific nodes.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the
cluster is accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will
return to normal. At this stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of
OneFS. Some new features in the upgrade might not be available until the upgrade is committed.
Performing a parallel upgrade using the CLI

You can use the command-line interface to upgrade the version of OneFS that is running on your cluster.
Perform the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the
1. Open a secure shell (SSH) connection on any node in the cluster and log in with the root account.
2. In OneFS 9.2.0.0 and later, you can manage the amount of time OneFS waits to disconnect client connections using the isi
upgrade cluster drain command and sub-commands.
3. To perform a parallel upgrade, run the following command, where <install-image-path> is the file path of the upgrade install
image. The file path must be accessible in an /ifs directory.
isi upgrade cluster start --parallel <install-image-path>
In OneFS 9.2.0.0 and later, you can include the firmware upgrade with your OneFS upgrade. To perform a parallel upgrade
that includes a firmware upgrade, run the following command where <install-image-path> is the file path of the upgrade
install image and <firmware-path> is the file path of the firmware package.
isi upgrade cluster start --parallel <install-image-path> --fw-pkg <firmware-path>
NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web administration
interface.
24 Performing the OneFS upgrade

4. You can specify the following parallel upgrade options:
Options Description
Nodes to Upgrade specific nodes with the --nodes <integer_range_list> option.
select for
Specify the nodes in their upgrade order as a comma-separated list (for example, --nodes 7,3,2,5) or as
upgrade
a dash-separated range (for example, --nodes 1-7) of logical node numbers (LNNs).
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is
sent to confirm that the upgrade is making progress. Do not leave the cluster in a partially upgraded
state for a prolonged period. Some new features in the upgrade might not be available until all the nodes
in the cluster have been upgraded and the upgrade is committed. Refer to the release notes for the
OneFS version that you are upgrading to for information about features that require all the nodes to be
upgraded.
OneFS 9.2.0.0 and later ignores the node restart order. Instead, OneFS restarts any node that is ready, as long as only one
node is restarting at a time.
The following example for OneFS 8.2.2 through OneFS 9.1.0.x starts a parallel upgrade on nodes 7,3,2,5, in that order:
isi upgrade cluster start --parallel <install-image-path> --nodes 7,3,2,5
OneFS 8.2.1 and earlier does not support parallel upgrades.

5. To add nodes to a current upgrade, run the following command, where <nodes> is the list of LNNs:
isi upgrade add-nodes <nodes>
6. To add any remaining nodes to the upgrade, run the following command:
isi upgrade add-remaining-nodes
Client connections during parallel upgrades

Parallel upgrades allow users to access data before, during, and after the upgrade. However, as groups of nodes are upgraded
and restarted, users may experience brief pauses in the time it takes to complete a read or write operation.
If the cluster is configured for dynamic IP allocation, client connections to restarted nodes are automatically reconnected. If
a client is reconnected to a node that has not yet been upgraded and restarted, the client might be required to reestablish a
connection to the cluster more than once.
The following table describes the expected client behavior when a node is restarted on a cluster that is configured for dynamic
IP allocation:
Client Expected client behavior

SMB2 Client transitions from the restarting node to a new node, but the client is disrupted.
NOTE: In OneFS 9.3.0.0 and later, client transitions from the restarting node to a new node
without disruption.
SMB3 Client transitions from the restarting node to a new node without disruption.
NFSv2 and NFSv3 Client transitions from the restarting node to a new node without disruption.
NFSv4 Client transitions from the restarting node to a new node without disruption. Clients use NFSv4
failover support.
NOTE: For more information about NFS settings, see article 457328, Best practices for NFS client settings.

Completing a rolling upgrade of OneFS
If you perform a rolling upgrade, the OneFS cluster nodes are restarted one at a time. During a rolling upgrade, client
connections to the restarting node are disconnected, but other nodes remain available for client connection.
NOTE: When you perform a rolling upgrade between OneFS releases, you can upgrade all the nodes in the cluster, or you
can select specific nodes to upgrade. If you upgrade only some of the nodes, the remaining nodes in the cluster are not
upgraded but can be upgraded later in the upgrade process. The upgrade is not complete until all nodes are upgraded, and
only one upgrade can be in progress at a time. You cannot upgrade some nodes to one version of OneFS and then upgrade
another group of nodes to a different version of OneFS.
Perform a rolling upgrade using the web administration interface

You can use the web administration interface to upgrade the version of OneFS that is running on the cluster.
You must download the OneFS installation image OneFS installation image. The installation image is used to run compatibility
checks. You must download the image and run the checks before upgrading. You must resolve any compatibility issues found by
the checks before upgrading.
4. Browse to the location of the installation image that you want to install. The file path must be accessible in an /ifs
directory. Then click Select.
5. In the Upgrade Type list, select Rolling Upgrade.
6. Under Nodes to Upgrade, specify an upgrade option:
● To upgrade all the nodes in the cluster, click All nodes.
● To upgrade specific nodes, click Specify nodes and reboot order.
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is sent to confirm
that the upgrade is making progress. Do not leave the cluster in a partially upgraded state for a prolonged period.
Some new features in the upgrade might not be available until all the nodes in the cluster have been upgraded
and the upgrade is committed. Refer to the release notes for the OneFS version that you are upgrading to for
information about features that require all the nodes to be upgraded.
7. If you are upgrading specific nodes, select the node or nodes in the Nodes Available for Upgrade list, and then click Add.
The selected node or nodes are added to the Nodes Selected for Upgrade section. To arrange the nodes in the preferred
restart order, click the arrows next to the nodes in the Nodes Selected for Upgrade section.
NOTE: In OneFS 9.2.0.0 and later, the restart order is ignored. Instead, OneFS restarts any node that is ready, as long
as only one node is restarting at a time. This process allows for quicker rolling upgrades.
8. Optional: To skip the pre-upgrade checks for compatibility issues, select Skip optional pre-upgrade checks.
NOTE: It is recommended that you run the optional pre-upgrade checks. Before starting an upgrade, OneFS checks that
your cluster is healthy enough to complete the upgrade process. Some of the pre-upgrade checks are mandatory, and
will be performed even if you choose to skip the optional checks. All pre-upgrade checks contribute to a safer upgrade.
9. Click Start Upgrade.

NOTE: In 9.2.0.0 and later, you can use the upgrade status screen to monitor the progress of your upgrade and view
connected clients.

Perform a rolling upgrade using the CLI
You can use the command-line interface or the web administration interface to upgrade the version of OneFS that is running on
the cluster.
Review the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the
Command-line Interface instructions:
1. Open a secure shell (SSH) connection on any node in the cluster and log in with the root account.
2. To perform a rolling upgrade, run the following command, where <install-image-path> is the file path of the upgrade install
image. The file path must be accessible in an /ifs directory.
For OneFS 8.2.2 and later:
isi upgrade cluster start --rolling <install-image-path>
For OneFS 8.2.1 and earlier:
isi upgrade cluster start <install-image-path>
interface.
3. Optional: You can specify the following rolling upgrade options:
Options Description
Nodes to Upgrade specific nodes with the --nodes <integer_range_list> option.
select for
Specify the nodes in their upgrade order as a comma-separated list (for example, --nodes 7,3,2,5) or as
upgrade
a dash-separated range (for example, --nodes 1-7) of logical node numbers (LNNs).
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is
sent to confirm that the upgrade is making progress. Do not leave the cluster in a partially upgraded
state for a prolonged period. Some new features in the upgrade might not be available until all the nodes
in the cluster have been upgraded and the upgrade is committed. Refer to the release notes for the
OneFS version that you are upgrading to for information about features that require all the nodes to be
upgraded.
OneFS 9.2.0.0 and later ignores the node restart order. Instead, OneFS restarts any node that is ready, as long as only one
node is restarting at a time.
The following example for OneFS 8.2.2 through OneFS 9.1.0.x starts a rolling upgrade on nodes 7,3,2,5, in that order:
isi upgrade cluster start --rolling <install-image-path> --nodes 7,3,2,5
The following example for OneFS 8.2.1 and earlier starts a rolling upgrade on nodes 7,3,2,5 in that order:
isi upgrade cluster start <install-image-path> --nodes 7,3,2,5
4. Optional: To add nodes to a current upgrade, run the following command, where <nodes> is the list of LNNs:
isi upgrade add-nodes <nodes>
5. Optional: To add any remaining nodes to the upgrade, run the following command:
isi upgrade add-remaining-nodes

Configuration changes during rolling upgrades

You can continue to manage data and modify some cluster configurations during a rolling upgrade. For example, you can modify
SMB shares and NFS exports. It is recommended that you make configuration changes from the node with the highest devid.
The node with the highest devid is the last node to be upgraded and helps you avoid being disconnected during the upgrade
process.
NOTE: Attempts to enable or configure settings on an upgraded node fail until the upgrade is complete. New OneFS
features and software modules are not functional until all nodes in the cluster are upgraded.
Client connections during rolling upgrades

Rolling upgrades allow users to access data before, during, and after the upgrade. However, as nodes are upgraded and
restarted, users may experience brief pauses in the time it takes to complete a read or write operation.
If the cluster is configured for dynamic IP allocation, client connections to restarted nodes are automatically reconnected. If
a client is reconnected to a node that has not yet been upgraded and restarted, the client might be required to reestablish a
The following table describes the expected client behavior when a node is restarted on a cluster that is configured for dynamic
IP allocation:
Client Expected client behavior

SMB2 Client transitions from the restarting node to a new node, but the client is disrupted.
NOTE: In OneFS 9.3.0.0 and later, client transitions from the restarting node to a new node
without disruption.
SMB3 Client transitions from the restarting node to a new node without disruption.
NFSv2 and NFSv3 Client transitions from the restarting node to a new node without disruption.
NFSv4 Client transitions from the restarting node to a new node without disruption. Clients use NFSv4
failover support.
NOTE: If a client is reconnected to a node that has not yet been upgraded, the client might be required to reestablish a
NOTE: For more information about NFS settings, see article 457328, Best practices for NFS client settings.
Completing a simultaneous upgrade of OneFS

If you run a simultaneous upgrade, the OneFS cluster is upgraded simultaneously and then all of the nodes in the cluster are
restarted simultaneously. During a simultaneous upgrade, the entire cluster is down and unavailable for client connections.
Perform a simultaneous upgrade using the web administration

interface
You can use the web administration interface to upgrade the OneFS version that is running on a cluster.
4. Browse to the location of the installation image that you want to install. The file path must be accessible in an /ifs
directory. Then click Select.
5. In the Upgrade Type list, select Simultaneous Upgrade.

6. (Optional) To skip the pre-upgrade checks for compatibility issues, select Skip optional pre-upgrade checks.
NOTE: It is recommended that you run the optional pre-upgrade checks. Before starting an upgrade, OneFS checks that
your cluster is healthy enough to complete the upgrade process. Some of the pre-upgrade checks are mandatory, and
will be performed even if you choose to skip the optional checks. All pre-upgrade checks contribute to a safer upgrade.
7. Click Start Upgrade.

Perform a simultaneous upgrade using the CLI

You can use the command-line interface to upgrade the version of OneFS that is running on the cluster.
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster and log in with the root account.
2. To perform a simultaneous upgrade, run the following command, where <install-image-path> is the file path of the upgrade
install image. The file path must be accessible in an /ifs directory.
isi upgrade cluster start --simultaneous <install-image-path>
interface.
Adding a node to the cluster while an upgrade is in

progress
In some circumstances, you might need to add a node to the cluster while an upgrade is in progress and before the upgrade has
been committed.
To add a node during an upgrade, at least one node in the cluster must have been successfully upgraded to the target version
of OneFS. When a new node is added to the cluster during an upgrade, the system images the new node with the previously
committed version of OneFS. Then as the cluster upgrade continues, the new node is upgraded to the target version of OneFS.
If the cluster upgrade is stopped and rolled back, all the nodes in the cluster are returned to the previously committed version of
OneFS.
NOTE: Before adding new hardware to the cluster, first confirm that the hardware is compatible with the current version of
OneFS that is installed and the target version of OneFS that you are upgrading to. See the PowerScale Supportability and
Compatibility Guide for more information.
Committing an upgrade of OneFS

You must commit an upgrade to complete the upgrade process. Once you commit the upgrade, you cannot roll back to the
previous version of OneFS.
New features in the target version of OneFS are not available until the upgrade has been committed.

Commit an upgrade using the web interface
You can use the web interface to commit an upgrade of OneFS. Once you commit the upgrade, you cannot roll back to the
previous version of OneFS.
3. Click the Commit Upgrade to OneFS <version> button.
The cluster might display several confirmation messages. Confirm each message to continue the commit process.
Commit an upgrade using the CLI

You can use the command-line interface to commit an upgrade of OneFS. Once you commit the upgrade, you cannot roll back to
the previous version of OneFS.
2. To commit the upgrade, run the following command:
isi upgrade cluster commit
Rolling back an upgrade of OneFS

The upgrade rollback feature allows you to stop the upgrade that is in progress and restore all the upgraded nodes to the
previous committed version of OneFS.
NOTE: You can roll back an upgrade only if the upgrade process is still active and the upgrade has not been committed. A
committed upgrade cannot be rolled back.
Roll back an upgrade using the web interface

You can use the web administration interface to stop an upgrade and return to the previous committed version of OneFS.
NOTE: The rollback process must restart all the upgraded nodes simultaneously. This will temporarily disrupt cluster
services and data availability.
3. Click the Roll Back to OneFS <version> button. In OneFS 9.2.0.0 and later, click the Halt and roll back current upgrade
button.
The cluster might display several confirmation messages. Confirm each message to continue the rollback process.
The cluster displays the rollback progress.
Roll back an upgrade using the CLI

Rolling back stops an upgrade and returns the cluster to the previous committed version of OneFS.
NOTE: The rollback process must restart all the upgraded nodes simultaneously. This will temporarily disrupt cluster
services and data availability.
Command-Line Interface instructions:
2. To roll back the upgrade, run the following command:
isi upgrade cluster rollback

Verify the OneFS installation using the CLI
After you install OneFS, verify that the installation was successful.
1. Confirm that the health of all the nodes in the cluster is OK by running the following command:
isi stat
2. Remove the installation files from the /ifs/data directory by running the following command where
<installation_file_name> is the name of the installation file:
rm /ifs/data/<installation_file_name>
3. Collect information about the cluster in OneFS 9.0.0.0 and earlier, by running the following command:
isi_gather_info
Collect information about the cluster in OneFS 9.1.0.0 and later, by running the following command:
isi diagnostics gather start

5
Completing post-upgrade tasks
Topics:
• Post-upgrade process - Overview
• Allow the Upgrade job to run
• Verify operational status using the CLI
• Re-establish user privileges
• Check client connections, and test the workflow
• Verify Kerberos migration
• Reapply custom settings
• Test custom scripts
• Reinstall Aspera
• Install the latest roll-up patch
Post-upgrade process - Overview

After an upgrade, you should perform restoration and change management tasks to ensure that your cluster performs and
behaves as expected. It is recommended that you build time into your upgrade plan to re-establish custom settings and
privileges, and re-enable connections and features. You should also make time to modify settings for new and changed features.
Post-upgrade - Summary
The following is a summary of steps to perform during the post-upgrade phase:
1. Allow upgrade jobs to run.
2. Verify operational status.
3. Re-establish user privileges.
4. Restore client connections and workflows.
5. Verify Kerberos migration.
6. Restore custom settings.
7. Test custom scripts.
8. (Optional) Install latest patch.
Allow the Upgrade job to run

After an upgrade, an important job titled Upgrade that upgrades on-disk data structures, might continue to run for a while. The
Upgrade job must be allowed to run to completion.
Although the cluster remains accessible while the Upgrade job is running, the job might temporarily decrease the cluster's
performance.
NOTE: For more information about the Upgrade job, see KB Article 194551.
Verify operational status using the CLI

You can run a series of CLI commands as root to help verify that the PowerScale cluster is working correctly after an upgrade.
If you find an unresolvable issue, contact Dell Technologies Support.
32 Completing post-upgrade tasks

1. Check the new version number of the cluster:
isi_for_array -s uname -a
2. View the status of the cluster and ensure all the nodes are operational:
isi status
3. Check the devices in the nodes to validate the status of the drives:
isi_for_array -s "isi devices list"
4. Check the status of jobs and resume any paused jobs:
isi job status
isi job jobs resume <jobID>
5. Review the list of events, and address any critical events:
isi event groups list --sort=severity
6. To verify network connectivity and SmartConnect functionality, ping all internal and external interfaces on the cluster.
7. Verify the network interfaces:
isi network interfaces list
8. Verify the subnets:
isi network subnets list --verbose
9. Verify the pools:
isi network pools list --verbose
10. To check for issues, review the log files on the cluster:
isi_upgrade_logs
cat /var/log/messages
11. Check the input and output:
isi statistics system
12. Check the global SMB settings:
isi smb settings global view
13. Check the status of the node firmware to ensure it is consistent across nodes:
For OneFS 9.0.0 and later:
isi_upgrade_logs --get-fw-report
Completing post-upgrade tasks 33

For OneFS 8.2.2 and earlier:
isi upgrade firmware devices
14. Ensure that all the licenses carried over and remain up to date:
isi license licenses list
15. Check the status of the authentication providers to ensure that they remain active:
isi auth status --verbose
16. Review the list of SyncIQ jobs:
isi sync jobs list
17. Check the SyncIQ job reports:
isi sync reports list
18. Review the list of the scheduled snapshots:
isi snapshot schedules list
19. If you use SupportAssist or SRS, confirm that the service is reenabled.
Re-establish user privileges

After you upgrade, re-establish user privileges and roles. You can log in to the cluster through SSH as root or though an
administrator account if that role has been assigned to any users.
NOTE: If the system administered the user roles through RBAC before the upgrade, any custom roles that existed are still
in place. However, if the privileges assigned to built-in roles have changed in the new version, the users who are assigned to
those built-in roles will have those new privileges.
See the OneFS CLI Administration Guide or the OneFS Web Administration Guide for more information.
Check client connections, and test the workflow

After the OneFS upgrade, check client connections and confirm that all users, clients, and applications can access the cluster.
Test your workflows to ensure that they function correctly.
NOTE: If you are using NDMP backups on your cluster, re-enable the NDMP service and test that it's working correctly.
Verify Kerberos migration

If you are using Kerberos authentication, you must verify that the Kerberos providers and settings have been migrated
successfully.
Verify Kerberos migration using the web administration interface

Verify that Kerberos authentication providers, and settings are correct:
1. Click Access > Authentication Providers > Kerberos Provider.
2. In the Kerberos Realms, Kerberos Domains, and Kerberos Providers tables, verify that the Kerberos providers are
correct.

3. In the Kerberos Settings area, verify that the Kerberos authentication settings are correct.
Verify Kerberos migration using the CLI

Verify that Kerberos authentication providers, and settings are correctly represented in the output of each of these commands:
● isi auth krb5 realm list
● isi auth krb5 domain list
● isi auth krb5 spn list
● isi auth settings krb5 view
Reapply custom settings

Some custom settings might not have been preserved during the upgrade. Reapply the custom settings that you backed up and
recorded when you performed the pre-upgrade tasks.
The custom settings include:
● SMB audit logging
● Passwords for local user accounts
● Changes to system controls
● Aspera
● Cron jobs
● Certificates
● Static routes
Test custom scripts

Test any custom scripts you have implemented to confirm they continue to work. If command syntax changes were made in the
target version of OneFS, some custom scripts might not work as intended.
NOTE: It is recommended to ensure that your custom scripts work on a test cluster before implementing them on a
production system.
Reinstall Aspera
If you were running Aspera for PowerScale before you upgraded your cluster, you must reinstall Aspera after you upgrade.
Aspera is not supported in OneFS 8.2.0 and later.
1. Run the following command to verify that the upgrade process enabled the Aspera Central service and the Aspera node
daemon:
isi services
2. If the Aspera service is not running properly, you must download and re-install a compatible version of Aspera from the
Aspera Enterprise Server website and verify that the Aspera Central service is enabled.
● For information about which versions of Aspera are compatible with your version of OneFS, see the PowerScale
Supportability and Compatibility Guide.
Install the latest roll-up patch

Install the latest patch for the target version of OneFS to which you upgraded.
Confirm if your version of OneFS requires a patch. Most OneFS images already include the latest patch. For more information,
check the OneFS image download, or contact your account team.
1. See PowerScale OneFS Current Patches to view a list of patches that are available for your version of OneFS.
Completing post-upgrade tasks 35

2. Download and install the latest roll-up patch using the instructions found in the README.txt file included with the patch
download.

6
Troubleshooting your upgrade
Topics:
• Troubleshooting an upgrade
Troubleshooting an upgrade
If you experience problems with your upgrade, try the following:
● Check the upgrade logs and review for errors.
● Read through the Customer Troubleshooting Guide: OneFS Upgrade Process Flowchart for information that is related to
upgrade failures and error messages.
● Search for OneFS upgrade information within the knowledge base on the PowerScale support site.
● Contact your Dell Technologies Support representative.
Troubleshooting your upgrade 37

A
Features in OneFS Releases
This appendix lists OneFS features, enhancements, and improvements introduced by OneFS version. This information is pulled
directly from the Release Notes for each OneFS version.
Topics:
• OneFS 9.5.0.0
• OneFS 9.4.0.0
• OneFS 9.3.0.0
• OneFS 9.2.1.0
• OneFS 9.2.0.0
• OneFS 9.1.0.0
• OneFS 9.0.0.0
OneFS 9.5.0.0
Full release notes available at: https://dl.dell.com/content/manual58071479-powerscale-onefs-9-5-0-0-release-notes.pdf
New features
This section summarizes the new features in OneFS 9.5.0.0.
SupportAssist SupportAssist is the new remote connectivity system that connects your OneFS cluster with Dell
introduced as Technologies Support. For more information, see the SupportAssist section in the OneFS Administration
the new way guide.
to connect with
Dell Technologies
Support.
Host-based The firewall controls inbound traffic on the front-end network. The firewall comes with predefined default
firewall added to policies that protect the OneFS default ports. You can modify the default policies and create custom
OneFS. policies. New Web UI options and isi network firewall commands are available.
Account policy Security restrictions can be configured to enforce a delay for subsequent logins after a failed
restrictions have administrative login. You can also disable inactive local accounts after a specified number of days, and
been updated. limit the number of active sessions a user can have on a node.
Expanded Password complexity is configurable for each local provider, and password policies can be configured for
password policy each access zone.
configuration.
Support If the Active Directory domain requires encryption, the redirector will enable and use encryption.
for Active
Directory domain
encryption
added.
TLS options You can configure PKI communication between the OneFS cluster and an LDAP server to use TLS. Enable
added for LDAP TLS and configure options in the isi auth ldap create or isi auth ldap modify commands.
communications. Options define levels of verification during the TLS handshake and include revocation checking against
Online Certificate Status Protocol (OCSP) URIs.
38 Features in OneFS Releases

Secure transfer The FTPS protocol support has been implemented for secure transfer of log files which protects cluster
of log files configuration logs from interception.
added.
NFS persisted The NFS locks and waiters functionality are updated in the Web UI and CLI commands with new listing
locks and waiters options.
listings updated.
Expanded The OneFS web interface now includes the ability to import and manage your certificates. For more
certificate information, see the Certificates section of the OneFS Web Administration Guide.
management in
the web
interface.
Read Improved read performance for NFS over RDMA.

performance has
been improved.
Smart QoS Protocol operations limits added on workloads.

(partitioned
performance)
workload limits
updated.
HealthCheck Removed the HealthCheck definition package list from the output of the patch CLI isi upgrade
definition patch list.
packages have
been enhanced.
OneFS PAPI Security command added to configure authorization and authenticity of peripherals (typically attached
command added using USB connections to the node's hardware).
to enable or
disable USB port
connectivity.
Command to You can disable all USB ports across the cluster with an option that was added to the isi security
disable all USB settings modify command. The STIG hardening profile disables all USB ports across the cluster.
peripheral ports
is added.
Configurable Set a maximum threshold limit that stops moving files from source to target node pools or tier.
option added
for setting a
threshold limit of
SmartPools.
SyncIQ updated SyncIQ target location files are updated to display the same create time (c-time) as the file in the source
file create time in location. This update takes effect as files are synced.
target location.
Support for the Support for the Nvidia Connectx-6 Ethernet adapter has been added to OneFS in version 9.5.0.0.
Nvidia ConnectX-
6 Ethernet
Adapter added.
Remove earlier Once a version of a software component (like Python or SQLite) is installed, OneFS removes any earlier
versions of versions of that software component.
software
components.
The Python The Python version has been updated from 2.7.x to 3.8.x in OneFS.
version has been
updated.
Features in OneFS Releases 39

The Python The Python library urllib3 was updated to version 1.26.9.
urllib3 library has
been updated.
The libxml2 The libxml2 library was updated to version 2.9.14.

library has been
updated.
Support added OneFS 9.5.0.0 supports CURL 7.83.0.

for newer
versions of
CURL.
Expanded A new IPv6 configuration option enables or disables IPv6 on the cluster. Other options include several
support for IPv6 duplicate address detection (DAD) settings, IPv6 automatic configuration, local link generation, and
on OneFS. processing of ICMPv6 redirect messages. IPv6 configuration was changed to use the isi network
external modify, isi network pools create, and isi network pools modify
commands.
OneFS auditing System audits collect platform events, such as shutdowns and reboots, and account related events, such
includes a new as password changes.
system audit
topic.
Syslog The syslog forwarding feature in the auditing module forwards logs directly to the configured remote
forwarding in the servers. The local collection that occurred in previous releases is eliminated.
audit module is
improved.
TLS options You can configure syslog forwarding to use TLS. You can configure each audit topic (configuration,
added to syslog protocol, and system) separately for TLS. Each audit topic.
forwarding in the
audit module.
Root password The system audit topic captures root password changes. This feature is always on and cannot be disabled.
changes are
audited.
Restricted CLI The Restricted CLI is a new login shell. All commands that are issued in the Restricted CLI are logged.
provides an No file access is permitted. For information about the Restricted CLI, see the PowerScale OneFS 9.5.0.0
audited and Security Configuration Guide.
limited CLI.
The bash The bash shell is updated to version 5.1.16.
shell has been
updated.
Rekey feature You can reencrypt the OneFS keystores without interrupting processing. There are separate reencrypting
added to the key commands for the cluster key store and the self-encrypted drive (SED) key stores.
store manager.
Single sign-on The WebUI includes a wizard for enabling single sign-ons.
capability added
to the WebUI.
Apache changes The OneFS HTTP data paths and control paths are separated. The two paths use separate ports.
have been added. Configuration parameters in the isi http settings modify command can set Apache directives
that control session timeouts.
FIPS 140-2 The STIG hardening profile implements the FIPS 140-2 cryptographic algorithms and disables services
compliance that do not comply with the standard. In nonhardening mode, you can enable FIPS mode to use the
for hardening cryptographic algorithms.
clusters.
Improved The workflows for applying a hardening profile and returning to system defaults are simplified.
workflows in
the Hardening
Module.

Hardening A profile report lists all rules in the profile and indicates whether the cluster complies with each rule. The
Module includes verbose option shows, for each rule, where the rule is configured, the expected value for compliance, and
new profile the current value on the cluster.
reports.
STIG hardening Many new rules are added to the STIG profile. The new rules configure a cluster to comply with the
profile complies United States Federal Government APL.
with the
Approved
Products List
(APL).
See the PowerScale OneFS support pages on www.dell.com/support for detailed user documentation.
OneFS 9.4.0.0
New features
Support for QLC Support for QLC SED Non-FIPS drives has been added to OneFS in version 9.4.0.9.
SED Non-FIPS NOTE: Your cluster must be on OneFS 9.4.0.9 or later to access this feature.
drives
Performance OneFS 9.4.0.0 includes numerous performance enhancements.
enhancements
New Datamover OneFS 9.4.0.0 adds the Datamover data transfer engine, enabling you to transfer data between
data transfer PowerScale clusters and object-at-scale cloud systems. Datamover ensures that copies of your data are
engine consistent among PowerScale clusters and/or cloud platforms, and allows you to control the frequency of
data transfers at scheduled times using policies. You can transfer data at the directory level and exclude
specific files and subdirectories from being transferred.
The embedded Datamover feature provides data replication for file and object deployments on-premises
or in the cloud, enables file-to-file transfers between PowerScale clusters using RPC, and file-to-object
copy transfers to S3 (ECS, AWS) and Azure cloud systems. Datamover provides data protection, data
repurposing (copy), and data archiving, with flexible push/pull data transfers between systems. For
details, see the OneFS 9.4.0.0 CLI Administration Guide.
Rolling The rolling migration from InfiniBand to Ethernet begins with Generation 6 clusters in OneFS 9.4.0.0. You
InfiniBand should replace any InfiniBand switches with Ethernet switches. Contact your Dell representative for more
to Ethernet information.
migration
isi_gather_in The isi_gather_info HTTP upload method is deprecated in OneFS 9.4.0.0. The HTTP upload
fo: HTTP upload method will be removed entirely in a release later in 2022. The alternative upload methods are ESRS
method (preferred) and FTP.
deprecation In the OneFS release following 9.4.0.0, the names of the HTTP upload options will change to emphasize
that they are plain text:
● isi_gather_info: --http will be renamed to --http-insecure
● isi diagnostics gather: --http-upload will be renamed to --http-insecure-upload
Long filename OneFS 9.4.0.0 increases the maximum supported filename length to 1024 bytes. The maximum path
support length is increased to 4096 bytes.
Enhancements OneFS SmartConnect supports viewing status information for nodes in a network pool with the CLI
to SmartConnect command isi network pools status. See the OneFS 9.4.0.0 CLI Administration Guide and
diagnostics OneFS9.4.0.0 CLI Command Reference for details.
Disable A security improvement provides granularity for enabling or disabling HTTP services. The new isi
nonessential services http modify command enables or disables the following services individually: restful
HTTP services access namespace (RAN), PAPI , WebUI, and remote management API. For more information, see the
OneFS 9.4.0.0 Security Configuration Guide.

OneFS fails to a A security improvement in STIG hardening mode ensures that OneFS fails to a secure state. If OneFS fails
secure state to start, fails to shut down properly, or otherwise encounters an unexpected catastrophic event while in
hardening mode, the system reboots and displays the login prompt. The kernel debugger is disabled in
hardening mode.
Signed Dell distributes all upgrade packages, patches, service packs, or application components with digitally
certificates signed certificates. OneFS verifies the certificates before installation and prevents installation if the
required and verification fails. A new catalog internal to OneFS stores upgrade packages and associated files such
verified for as readmes, manifests, and certificates. Administrators can manage the packages, access the readme
upgrades files, and verify components with new isi upgrade catalog commands. See the OneFS 9.4.0.0 CLI
Administration Guide and OneFS9.4.0.0 CLI Command Reference for details.
Automatic and OneFS administrators can monitor security anomalies across a OneFS cluster and trigger a specified
on-demand action. By default, a cluster in STIG hardening mode runs security checks once per month and on all
security checks reboots except the first one. Administrators can run security checks on-demand on specified nodes or
across the cluster. The on-demand security checks are available with new isi security check
commands or APIs. See the OneFS 9.4.0.0 Security Configuration Guide for more information.
Enable Administrators can enable FIPS 140-2 compliance on a OneFS cluster without applying the STIG hardening
FIPS 140-2 profile. For more information, see the OneFS 9.4.0.0 Security Configuration Guide.
independent of
STIG hardening
Hardened HTTP The default HTTP response templates are changed in this release to prevent sniffing and block cross-site
response headers scripting.For details, see the OneFS 9.4.0.0 Security Configuration Guide.
Expanded OneFS 9.4.0.0 supports UEFI secure boot on the following PowerScale nodes:
support for UEFI ● A2000, A300, A3000
secure boot ● H700, H7000
● B100
● F200, F600, F900
● P100
For more information, see the OneFS 9.4.0.0 Security Configuration Guide.
Revised internal To support more modern workloads, especially with CloudPools and SyncIQ, the OneFS internal Keystore
Keystore Manager is revised for better performance. There are no user-facing impacts from this change.
Manager
Enhancements OneFS 9.4.0.0 dynamically detects whether a node is in a Not All Nodes on Network (NANON) situation
to NANON and applies the appropriate configuration changes. Use the isi status command to check the status
configuration of nodes in the cluster. By default, the status is updated every 5 minutes.
support
Tag fields for OneFS 9.4.0.0 allows tagging or labeling quotas to enumerate directories associated with customers for
quotas in APEX telemetry and showback purposes.
consoles
Improved SED OneFS 9.4.0.0 optimizes self-encrypted drive (SED) deployment to ensure efficient deployment.
deployment
Inline OneFS 9.4.0.0 has inline deduplication enabled by default.
deduplication
enabled by
default
SRS Secure Remote Support (SRS) will be decommissioned over the next few years. This will impact all
decommision versions of SRS across all versions of OneFS. The replacement, SupportAssist, will be included in the
coming soon OneFS 9.5.0.0 release before SRS decommission begins.
OneFS 9.3.0.0
Full release notes available at: https://dl.dell.com/content/manual64485109

New features
Support for OneFS supports basic NFSv4.1 and NFSv4.2.

NFSv4.1 and
NFSv4.2
Writable OneFS 9.3.0.0 supports creating and managing modifiable copies of snapshots. Writable snapshots enable
snapshots you to test data recovery scenarios without having to create a copy of an entire dataset. You can create
writable snapshots from a source snapshot, and then run tests against the writable snapshot to verify the
data. You manage writable snapshots using the command-line interface (CLI) or API endpoints.
Additions to S3 OneFS S3 support improvements include:
protocol support ● Amazon Web Services chunked encoding for streaming data upload.
● Non-slash delimiter support for Get Bucket.
● Improvements to the create and delete bucket functionality.
Support for PowerScale OneFS 9.3.0.0 adds support for Dell Technologies APEX file storage services (File Services).
APEX File Storage administrators and application owners can use File Services to perform self-service cluster
Storage Services data management tasks. Service providers manage the infrastructure, availability, and performance of
self-service PowerScale clusters.
administration Storage administrators and application owners can perform self-service cluster data management tasks,
including:
● Managing folders and the file hierarchy structure
● Monitoring SMB shares, NFS exports, and HDFS access
● Managing storage pools policies
● Monitoring quotas
● Monitoring snapshots
● Viewing reports
● Managing users
Enhancements OneFS 9.3.0.0 provides granular privileges. Each functional area of cluster administration is associated
to role-based with a top-level or parent privilege. Each parent privilege can have one or more subprivileges, which can
access control: also have subprivileges. Granular privileges enable you to control the specific actions that a user can
granular perform within a cluster administration area in a detailed way. Administrators can grant the following
privileges types of privileges:
● No permission (-): The user cannot access the functional area . (New)
● Read (r): The user can view the functional area, for example, viewing SyncIQ policies.
● Execute (x): The user can read and run jobs that are associated with the functional area. Create and
update permissions are allowed only for specific URIs. For example, execute permission is configured
to view and run SyncIQ policies. (New)
● Write (w): The user has full access to the functional area.
See the OneFS 9.3.0.0 CLI Administration Guide or OneFS 9.3.0.0 Web Administration Guide for privileges
details. See the OneFS API Reference Guide or the API Marketplace for API details.
Upgrade: OneFS supports transparent SMB node reboots during the upgrade process.
transparent SMB
disconnection
and reconnection
on node reboots
Support OneFS 9.3.0.0 and later releases support cluster-wide session management. A session cookie for a OneFS
for cluster API or Web UI session applies across all nodes in the cluster with no need to reconnect if the session
coherent session moves from one node to another. See the PowerScale OneFS 9.3.0.0 API Reference Guide for details.
management
Long filename 9.3.0.0 increases the maximum supported filename length to 1024 bytes. The maximum path length is
support increased to 4096 bytes.
Support for OneFS 9.3.0.0 adds support for HDFS ACL. Ranger ACL continues to be supported.
HDFS ACL

PowerScale The PowerScale Performance Accelerator (PSPA) node provides added performance to clusters that
Performance consist of nodes that are CPU bound.
Accelerator 1U
node support
PowerScale The PowerScale Backup Accelerator (PSBA) node provides two-way NDMP backup and recovery from a
Backup OneFS cluster. Offloading NDMP workloads from the primary storage nodes provides a seamless way to
Accelerator 1U accelerate tape backups. PSBA is supported for Generation 5 and Generation 6 and later clusters.
node support
Support for UEFI OneFS adds support for UEFI Secure Boot for the A2000, securing the platform during the platform boot
secure boot and OneFS loading phases. Storage administrators can opt in to enable the feature in the A2000 BIOS.
See the PowerScale OneFS Security Configuration Guide for details and instructions.
Default OneFS 9.3.0.0 enhances file system security by setting the default access permissions for /ifs
permissions and /ifs/data to 755.
changes for /ifs
and /ifs/data
Performance OneFS offers improved read/write performance for inline data operations.
improvements
for inline read/
write operations
rsync deprecated The rsync tool is deprecated in this release. On-node alternatives for local copies are cp, tar, and pax.
For remote copies, a user can enable an /ifs export and copy directly.
9.3.0.0 NDMP backups that are created with 9.3.0.0 work only with 9.3.0.0 and later releases.
NDMP backup
compatibility
OneFS 9.2.1.0
Full release notes available at: https://dl.dell.com/content/docu104030
New features
InfiniBand OneFS continues to support backend InfiniBand connections.

support
Support for OneFS supports new 4U nodes that provide support for inline software data compression and data
Archive and deduplication:
Hybrid nodes ● Archive nodes:
○ A300: high performance and accessibility in a standard chassis
○ A3000: high performace and density in a deep chassis
● Hybrid nodes:
○ H700: maximum performance for demanding workloads in a standard chassis
○ H7000: high performance, high capacity in a deep chassis
Inline software data compression is 3:1 depending on the workload and the dataset.
Hardware OneFS supports Generation 5 and Generation 6 hardware, as well as hybrid nodes H700 and H7000 and
interoperability archive nodes A300 and A3000, including interoperability among the generations and new nodes.
WARNING: Do not install the A300, A3000 (archive nodes), H700, or H7000 (hybrid nodes)
into existing Generation 6 chassis installations. The archive and hybrid nodes must only be
installed in the chassis provided from the factory. The higher powered archive and hybrid
nodes can cause a fuse to open on the Generation 6 chassis midplane, which then requires a
chassis replacement.

Expanded Added support for cryptographic modules: TPM (Trusted Platform Module)+SED and TCM+SED.
support for
Archive and
Hybrid nodes
SSH ciphers OneFS no longer supports the 3des-cbc cipher.
OneFS 9.2.0.0
New features
Support OneFS 9.2.0.0 supports backing up and exporting the current configuration of a PowerScale cluster
for cluster for test and auditing purposes. See the 9.2.0.0 PowerScale OneFS CLI Administration Guide or Web
configuration Administration Guide for details.
export/import
Permission In OneFS 9.2.0.0, the system default umask setting is changed to 077 and the hardening STIG profile
changes rules are greatly simplified. Applying and reverting the STIG profile is faster.
New RBAC The ISI_PRIV_RESTRICTED_AUTH privilege enables granting a subset of rights to allow an administrator
privilege: to run user and group management actions, such as finding and listing users, setting user passwords,
ISI_PRIV_RESTR unlocking accounts, and adding and removing users and groups. The ISI_PRIV_RESTRICTED_AUTH
ICTED_AUTH privilege is intended for help or service desk types of administrators who do not need the full
ISI_PRIV_AUTH set of privileges. Administrators with the ISI_PRIV_RESTRICTED_AUTH privilege cannot
modify access level privileges, such as root level access to a cluster. Administrators with the restricted
auth privilege can only modify users and groups that have the same or less privilege.
External key This enhancement enables you to offload the SED data encryption key to an external KMIP-compliant Key
management for Management Server. See the 9.2.0.0 PowerScale OneFS CLI Administration Guide or Web Administration
self-encrypted Guidefor details.
drives (SEDs)
FIPS compliance PowerScale OneFS replaced the FIPS 140-2 cryptographic module from OpenSSL with a private label
validation module that provides updated FIPS compliance.
OpenSSL update The OpenSSL version has been updated to OpenSSL 1.0.2u.
IPv6 compliance 9.2.0.0 PowerScale OneFS is IPv6/USGv6 compliant.
RDMA support The NFS over RDMA feature (NFSoRDMA) is supported for NFSv3. This enhancement adds another
for NFS front-end network transport communication mechanism between the client and OneFS node. See the
9.2.0.0 PowerScale OneFS CLI Administration Guide or Web Administration Guide for details.
S3 enhancement The S3 protocol on OneFS is enhanced with consistent ETAG functionality. Two new options, use-
md5-for-etag and validate-content-md5 were added. See the 9.2.0.0 PowerScale OneFS CLI
Administration Guide or Web Administration Guide for details.
HTTP basic HTTP basic authentication is disabled by default. To enable HTTP basic authentication from the WebUI:
authentication 1. Log in to the WebUI as a user with administrative privileges.
is disabled by 2. Click Protocols > HTTP settings.
default
3. In the Authentication settings section, click the Authentication drop-down and choose the HTTP
authentication type.
4. Save changes.
See the 9.2.0.0 PowerScale OneFS CLI Administration Guide or Web Administration Guide for details.
Improvements OneFS 9.2.0.0 includes enhancements to storage efficiency, quota statistics, and cluster statistics
to storage tracking to improve the accuracy of data reduction and storage efficiency reporting.
efficiency
reporting

For existing OneFS systems, upgrading to 9.2.0.0 includes an upgrade to a new inode version. In upgrade
scenarios, the statistics tracking and reporting enhancements are inoperative until after non-disruptive
upgrade commit. After commit:
● All version 6 and 7 inodes are upgraded to version 8.
● Existing quotas are upgraded to include the new accounting statistics.
Data reduction reporting is available after inode and quota upgrades are complete.
NOTE: The statistics presented in the previous version of OneFS are maintained and the new
statistics are not available until the ugprades complete.
See the Upgrade Planning and Process Guide and CLI Administration Guide for more information.
Improved Starting with upgrades from PowerScale 9.2.0.0 to later releases, parallel upgrades are non-disruptive.
OneFS upgrade When a node takes its upgrade "turn", new client connections are directed away from that node. The
experience node waits for all its SMB clients to disconnect before it reboots. This same process occurs for cluster
reboots beginning with 9.2.0.0. See the Upgrade Planning and Process Guide for more information.
Changes to In OneFS 8.2.0, references to the FEC link aggregation mode were replaced with loadbalance link
the FEC aggregation mode. Since OneFS 8.0.0, FEC link aggregation has been an alias to loadbalance link
link aggregation aggregation mode. While the name has changed, the underlying mode has not. No configuration changes
mode are necessary.
F900 support for F900 all-flash nodes support inline software data compression (3:1 depending on the workload and the
inline software dataset), and data deduplication with OneFS 9.2.0.0. F900 nodes are 2U models that require a minimum
data compression cluster size of three nodes. Clusters can be expanded to a maximum of 252 nodes in single node
in OneFS 9.2.0.0 increments. The F900 node uses direct-attached NVMe (Non-Volatile Memory Express) SSDs for fast
data access, with integrated parallelism.
The network The network names ipv6 and internal are special purpose names reserved for use by OneFS. Using
names ipv6 ipv6 or internal for your network names is no longer allowed.
and internal
are reserved by
OneFS
New and changed There are new and changed OneFS API endpoints for 9.2.0.0. For details, see the PowerScale OneFS API
OneFS RESTful Reference Guide or the PowerScale OneFS RESTful API documentation on the Dell API Marketplace.
API endpoints
New events and There are new OneFS events and alerts for 9.2.0.0. For details, see the PowerScale OneFS Event
alerts Reference Guide.
SmartLock As of OneFS 8.2.0, SyncIQ requires that there is no retention policy set on the target Write Once
compliance mode Read Many (WORM) domain. The source WORM domain can have a retention policy.
consideration for
SyncIQ
OneFS 9.1.0.0
Modifications and enhancements in OneFS 9.1.0.0

Functional area Feature description Summary of benefits
Protocols: Open Support for Open Stack Swift will be removed in a future OneFS release. Use See article 542999 for
Stack Swift the S3 protocol instead. more information.
Supportability The OneFS version string format is updated to provide details about a release Details included in the
build for support purposes. The version string has the following components: version string assist
Support and Engineering
<major>.<minor>.<service>.<patch> staff when evaluating
(<type> <identifier>, <build>, <datetime>, <details>) issues. OneFS handles the
old and new version string

Functional area Feature description Summary of benefits
● <major>: major release family, delivers significant new functionality formats seamlessly; there
● <minor>: delivers incremental functionality for the major release family is no action required. You
● <service>: delivers quality improvements but no new functionality can still use the isi
● <patch>: delivers a small number of precision fixes for specific issues version command to
view the complete version
● additional information about the release, contained in parentheses:
string.
○ <type>: Internal, Beta, Release, Patch, Hotfix
○ <identifier>: an optional identifier to further specify release contents
○ <build>: the Engineering-defined build number
○ <datetime>: timestamp for the build: YYYY-MM-DD HH:MM:SS
○ <details>: a list of the installed patch names
For example, a version string for 9.1.0.0 might be:
Isilon OneFS 9.1.0.0 (Release, Build

B_DECCAN_039(RELEASE), 2020-07-05 11:07:40,
0x90100B000000027
Supportability: SNAP_RESERVE_FULL, notifies the user when snapshot reserve space is The reserve space
CELOG nearly full. The threshold at which this event is triggered is now configurable, threshold can be modified.
and the reserve space threshold can now be changed, or customized in the CLI
and WebUI.
Supportability OneFS provides in-product license activation.
Supportability Upgrading OneFS is simplified with a bundle that includes the upgrade version Provides a smoother,
and Upgrade of OneFS and all applicable, necessary roll-up patches. less disruptive upgrade
experience.
Upgrade Upgrade performance improvements. Improves efficiency and
reduces latency during
upgrade.
Backup and OneFS adds multi-stream capability to the NDMP Backup Restart Extension Improves backup
Recovery (BRE). efficiency.
Antivirus OneFS adds support for CAVA-based antivirus solutions. Enables supporting a
variety of CAVA Antivirus
vendors.
CloudPools CloudPools performance improvements. Improved performance,
network bandwidth
utilization, and cost
savings.
CloudPools ECS2 (Virtustream) is no longer a supported CloudPools target.
Security: STIG The default behavior to redirect incoming communication from HTTP port 80 to Prevents use of all
HTTPS port 8080 is disabled. unnecessary and/or
non-secure functions,
ports, protocols, and/or
services.
Hardware OneFS 9.1.0.0 and later includes InfiniBand back-end network support for Expands back-end
PowerScale F200 and F600 nodes. support for the F200 and
F600 nodes.
OneFS 9.0.0.0

Changes in OneFS 9.0.0.0
Isilon is now Isilon OneFS is now PowerScale OneFS. The brand change is reflected in the WebUI. The CLI isi
PowerScale commands and OneFS API are not affected by this change, though output strings are updated with the
PowerScale brand.
Basic S3 protocol OneFS 9.0.0.0 adds basic S3 data access protocol support as a multi-protocol option.
support in OneFS
Support for up to OneFS 9.0.0.0 supports Leaf-Spine networks for up to 252 nodes.
252 nodes in a
OneFS cluster
PowerScale F200 The PowerScale F200 and F600 nodes are all-flash, 1U models that require a minimum cluster size of
and F600 nodes three nodes. Clusters can be expanded to a maximum of 252 nodes in single node increments.
OneFS 9.0.0.0 supports inline software data compression and data deduplication on the F200 and F600
nodes. Inline data deduplication and software-enabled inline compression are enabled by default on F200
and F600 nodes.
The F600 node uses direct-attached NVMe (Non-Volatile Memory Express) SSDs for fast data access,
with built-in parallelism.
Security On new installations of OneFS 9.0.0.0 and later, all protocols are disabled by default. You must enable
enhancement: any protocols that you plan to use. Use the following command to enable a protocol using the CLI, where
protocols are <protocol> is one of smb, nfs, hdfs, ftp, http, https, or s3.
disabled by
default isi services <protocol> enable
Enhancements to OneFS 9.0.0.0 enables configuring and managing the HealthCheck tool using the OneFS 9.0.0.0 Web
the HealthCheck Administration UI. Use the WebUI to configure HealthCheck runs, manage notifications, review reports,
tool and more.
Support for When OneFS CloudIQ is enabled, telemetry data is sent to Dell Technologies using Secure Remote
enabling and Services (SRS). Data includes system logs and system configuration data. The data is used to build alerts
disabling CloudIQ and performance metrics to improve product diagnostics and issue resolution. CloudIQ is enabled by
on a per-cluster default. If you do not want to send telemetry data to Dell Technologies, you can disable CloudIQ using the
basis OneFS 9.0.0.0 Web Administration UI.
Support for OneFS 9.0.0.0 provides a Web Administration UI for managing pools of compatible node types. You can
managing node change only those pools that do not have hard restrictions.
pools and
compatible node
types in the Web
UI
OneFS Ranger The OneFS Ranger plugin is enhanced to support NANON for Hadoop.
plugin support
for NANON
New events and There are new OneFS events and alerts for 9.0.0.0. For details, see the PowerScale OneFS 9.0.0.0 Event
alerts Reference Guide.
Changes to Protocols in OneFS 9.0.0.0

265942 As an SNMP security enhancement, all versions of SNMP are disabled by default. To enable SNMP,
you must change the snmpv2 community string or the snmpv3 auth and priv passwords from their
defaults.

B
Known Issues in OneFS Releases
This appendix lists OneFS known issues found by OneFS version. This information is pulled directly from the Release Notes for
each OneFS version.
Topics:
• OneFS 9.5.0.0
• OneFS 9.4.0.0
• OneFS 9.3.0.0
• OneFS 9.2.1.0
• OneFS 9.2.0.0
• OneFS 9.1.0.0
• OneFS 9.0.0.0
OneFS 9.5.0.0
Known Issues
This section describes issues that are known to affect this release of OneFS.
For information about issues known to affect other OneFS releases, log in to the PowerScale OneFS Product Page and view the
available OneFS Release Notes.
Datamover/SmartSync
PSCALE-133734 The Datamover to Cloud services copy job can pause with the message:
"CL_SOCKET_RETRY; Socket not ready for send/recv: error"
Workaround: Restart the Datamover service and resume the copy job.
PSCALE-132042 The Datamover service can get stuck in the running state when attempting to copy an object to a Cloud
service with a path name longer than 702 characters. Contact Dell Support to terminate the Datamover
daemon and restart it.
PSCALE-129725 This known issue has two parts:
1. If you cancel a job within a minute of that job entering the pause state, Datamover may not honor the
job cancel request. The issue is the timing of the job cache refresh on the Datamover node.
Workaround: If the job is not canceled within a couple of minutes after sending the job cancel request,
send another job cancel request.
2. If the remote account for a copy job is not reachable and you send a request to pause the job, the
pause request will not be honored while the account is not reachable.
Workaround: Resolve the account connectivity issue and re-send the pause request
Events, Alerts, and Cluster Monitoring

PSCALE-135179 Fixed an issue in alert management, where the data from an alert channel would not be removed when
the alert channel was deleted.
Known Issues in OneFS Releases 49

PSCALE-129736 Fixed an issue where CELOG would stop generating new events or sending critical notification alerts if
the CELOG database was too full.
PSCALE-125690 OneFS does not display an error if a dynamically allocated network pool is used in SRS configuration.
Workaround: SRS configuration requires statically allocated network pools. Use a statically allocated
network pool for your SRS configuration.
File System
PSCALE-154245 If the cluster has custom FTP or FTPS settings for isi_gather_info, and if the cluster loses quorum, or if
the /ifs partition becomes read-only, log uploads may fail.
Workaround:
Specify the --noconfig option when initiating the log gather. Example:
isi_gather_info –-noconfig
Or provide your unique FTP/FTPS credentials with the --noconfig option. Example:
isi_gather_info –-ftp –ftp-user <username> --ftp-host <ftp-host> --ftp-path
<path> --noconfig
PSCALE-153229 When a file policy name exceeds 199 bytes, the job engine enters an infinite core dump loop.
Workaround: Avoid creating file policy names that exceed 199 bytes.
PSCALE-128902 Changes in cluster configuration such as adding or removing nodes or disk pools becoming full, can result
in the creation of many small shadow stores. Most of these shadow stores are empty and will be removed
by the ShadowStoreDelete job. In some cases, the result can be a large number of shadow stores
added to a set of stale SINs and, rarer, a node panic.
Workaround: Run the following command daily:
# isi_for_array sysctl efs.sfm.shadow.sin_cache_inval=1
This will avoid node panics, though the cluster will continue creating lots of small shadow stores.
Networking
PSCALE-162900 In rare circumstances, SmartConnect's internal caches can get out of sync with the FlexNet configuration
and have differing views of what interfaces and IPs belong in each network pool. The easiest way to view
this is by running the `isi network pool status <pool id> --show-all` command and noticing discrepancies
with the network pool configuration. The work around is restart SmartConnect on any affected nodes.
PSCALE-161675 On clusters that contain a large number of dynamic network pools, it is possible for a failover event
(such as a node reboot) to trigger more internal system events than SmartConnect can keep up with.
This can lead to high CPU usage in isi_smartconnect_d and isi_flexnet_d. It could also cause a crash
of isi_smartconnect_d. Outside of reducing the number of dynamic network pools, there is currently no
workaround.
Protocols
PSCALE-161316 When concurrent session limits feature is enabled, there is a potential for temporary user lockout if
cluster time is changed or the JWT signing keys are rotated. User access resumes when a previous
session expires.
PSCALE-160907 With API authentication mode configured to require SSO, user's without
ISI_PRIV_LOGIN_PAPI_BYPASS_MFA cannot authenticate to the WebUI using username and password
login form. Instead sign-in with SSO option.
PSCALE-160679 If STIG hardening profile is disabled, all NFS exports will have their security flavors reset from Kerberos
5 Privacy (krb5p) to the system default of Unix. If Linux clients are configured to use krb5p, they will
experience DU until they are configured to either use Unix (auth_sys) authentication, or the exports are
all re-configured to use krb5p.
50 Known Issues in OneFS Releases

Workaround: Modify each export:isi nfs exports modify <export id> --zone=<your zone> --security-
flavors=krb5pOrRe-enable STIG hardening profile with:isi hardening apply --profile=STIG.
Security
PSCALE-162981 If you applied the STIG hardening profile and disabled the STIG hardening profile, SSO MFA remains
enabled. This setting may prevent accounts without the ISI_PRIV_LOGIN_PAPI_BYPASS_MFA from
accessing the WebUI or PAPI.
Workaround:
To disable this setting, log on to the CLI as root user and run the following command:
isi_gconfig -t jwt-config authentication_mode=0
PSCALE-142898 ICAP reports are not deleted automatically after expiring.

Workaround
Use a longer expiration period.
OneFS 9.4.0.0
Known Issues
Datamover/SmartSync
PSCALE-133734 The Datamover to Cloud services copy job can pause with the message:
"CL_SOCKET_RETRY; Socket not ready for send/recv: error"
Workaround: Restart the Datamover service and resume the copy job.
PSCALE-132042 The Datamover service can get stuck in the running state when attempting to copy an object to a Cloud
service with a path name longer than 702 characters. Contact Dell Support to terminate the Datamover
daemon and restart it.
PSCALE-129725 This known issue has two parts:
1. If you cancel a job within a minute of that job entering the pause state, Datamover may not honor the
job cancel request. The issue is the timing of the job cache refresh on the Datamover node.
Workaround: If the job is not canceled within a couple of minutes after sending the job cancel request,
send another job cancel request.
2. If the remote account for a copy job is not reachable and you send a request to pause the job, the
pause request will not be honored while the account is not reachable.
Workaround: Resolve the account connectivity issue and re-send the pause request
Events, Alerts, and Cluster Monitoring

PSCALE-126275, If you enable SRS against a gateway that is using unicode characters in the subject of the TLS certificate,
PSCALE-129890 it might cause an error. If this issue occurs, SRS will not connect to the gateway.

Workaround: Replace the TLS certificate on the SRS gateway with one containing only valid ascii
characters.
PSCALE-125690 OneFS does not display an error if a dynamically allocated network pool is used in SRS configuration.
Workaround: SRS configuration requires statically allocated network pools. Use a statically allocated
network pool for your SRS configuration.
File system
PSCALE-128902 Changes in cluster configuration such as adding or removing nodes or disk pools becoming full, can result
in the creation of many small shadow stores. Most of these shadow stores are empty and will be removed
by the ShadowStoreDelete job. In some cases, the result can be a large number of shadow stores
added to a set of stale SINs and, rarer, a node panic.
Workaround: Run the following command daily:
# isi_for_array sysctl efs.sfm.shadow.sin_cache_inval=1
This will avoid node panics, though the cluster will continue creating lots of small shadow stores.
Networking
PSCALE-132973, If you have a OneFS cluster with over 100 nodes, and are using redundant back-end Int-B Infiniband
PSCALE-117317, switches, and you encounter an issue that causes your back-end switches to reboot, nodes can split from
PSCALE-119564 the OneFS cluster. If this issue happens across a large number of nodes on the same cluster, it might
result in data unavailability on that cluster for a few seconds at a time.
Workaround: OneFS should retry adding the nodes to the cluster automatically. Once the nodes are
added to the cluster, the issue will be resolved.
SyncIQ
PSCALE-117552 SyncIQ is unable to rename a target directory if its parent directory has a "default-directory" quota set. If
there is a target subdirectory quota, SyncIQ provides a "not supported" message.
OneFS 9.3.0.0
Full release notes available at: https://dl.dell.com/content/manual64485109
Known Issues
Hardware
PSCALE-116185 It is possible for FIPS SED drives to become unavailable after repeated suspend/re-add operations.
PSCALE-117511 An NVDIMM device may encounter a critical error, possibly resulting in system performance degradation.
The operating system and/or applications may fail as a result. There may be a message similar to the
following:
Multi-bit memory errors detected on a memory device at location(s)

(slot).

A Critical event was detected on the Non-Volatile Dual In-line Memory
Module (NVDIMM) device in the slot DIMM_A7.
. where slot is the location of the affected NVDIMM.

Recommended action: Remove and reinstall the NVDIMM device. If the issue persists, contact the service
provider. For information about removing and reinstalling the NVDIMM, see the system Owner's Manual
on the support site.
PSCALE-117317 Multiple nodes can split after reloading the Int-B Infiniband switch.
Networking
PSCALE-113065 Nodes A300, A3000, H700, and H7000 ship with BMC LAN access disabled. Because of a firmware
issue, the following commands to enable access may not work, and you may not be able to change
allocation-type from static to dhcp or from dhcp to static:
isi ipmi settings modify --enabled=true --alloation-type dhcp -f

isi ipmi settings modify --enabled=true --allocation-type static-f
Workaround: Reset BMC by issuing the command:
ipmitool mc reset cold
on the affected node.
Protocols
Placeholder Placeholder.
Security
PSCALE-116997 Reverting security hardening can fail with the message:
API service is unavailable or restarting.
Supportability
PSCALE-119865 In rare circumstances, running isi stats heat on a very large cluster with tens of thousands of files
accessed can time out.
Workaround: Run isi stats heat –numeric to get statistics on global inodes rather than file
names.
PSCALE-60739 In rare circumstances, there can be a performance issue when there are more than a thousand snapshot
queries using SNMP.
Upgrades
PSCALE-119827 When upgrading a hardened cluster to OneFS 9.3.0.0, the upgrade might fail if a previous upgrade
attempt was rolled back.
Workaround: If upgrading from OneFS 9.2.x, install the latest patch before upgrading to OneFS 9.3.0.0.
If upgrading from OneFS 9.0.x and 9.1.x, and you encounter an upgrade failure, run the isi hardening
apply --profile=STIG command, then run the upgrade to OneFS 9.3.0.0 again. If upgrading from
OneFS 8.2.2 and earlier, it is recommended to upgrade to OneFS 9.0.0.0 first, before upgrading to OneFS
9.3.0.0.

PSCALE-111132 When upgrading to OneFS 9.3.0.0, and if a treewalk job is running in the job engine before the upgrade is
started, or is queued before all nodes have finished the upgrade, the treewalk job might fail. This is more
likely if the job pauses and then resumes.
Workaround: If the job doesn't restart itself, manually restart it once all nodes have finished the upgrade
to OneFS 9.3.0.0.
PSCALE-107219 When starting an upgrade, it must be possible to create a hard link to the file being used as the source.
That prevents you from starting upgrades from files that are part of a writable snap, a WORM compliance
domain, or other portions of the OneFS file system where hard links are limited.
Workaround: Copy the source image file to an unrestricted part of the file system before starting the
upgrade
PSCALE-61121 Do not enable NFSv4.1 during the upgrade process. Enable NFSv4.1 only after the upgrade to OneFS
9.3.0.0 is complete and committed. Enabling NFSv4.1 during the upgrade proess can cause unexpected
behavior, in particular if failover occurs while the cluster is running in mixed mode (not all servers in the
cluster have completed upgrade), or if the cluster upgrade is rolled back rather than being committed.
In both situations, the NFSv4.1 mounts may become inaccessible, and manual intervention may be
necessary on the clients with those mounts in order to recover.
OneFS 9.2.1.0
Known Issues
For information about issues known to affect other OneFS releases, log in to PowerScale Technical Support and view the
available OneFS Release Notes
Antivirus
PSCALE-21576 If you use isi antivirus cava servers modify --server-name=<name> --server-
uri=<uri> to configure multiple CAVA server URIs and the interval between configurations is short,
the CAVA URI may not be configured correctly.
Workaround: Restart the lwavscan daemon.
PSCALE-23191 CAVA jobs always start on the first node in the cluster. If Node 1 fails or splits from the cluster, no
scheduled CAVA jobs can start until Node 1 rejoins the cluster. Running jobs are not affected.
Workaround: The issue resolves itself after Node 1 rejoins the cluster.
File system
PSCALE-48358 The following warning message can appear when the backup of multi TB files or data sets fails:
cannot allocate memory
Workaround: Reduce the number of NDMP read threads.

PSCALE-22253 After you upgrade to 9.2.0.0 or 9.2.1.0, a CELOG event can appear that warns about under protected
node pools, even though no hardware has changed. The warning appears because enhancements to
estimations of the reliability of some suggested protections policies in OneFS 9.2.0.0 or 9.2.1.0 can result
in different suggested protections.
Workaround: If there was no warning before the upgrade, you can safely ignore this event. Any new
suggested protection policies have the same estimated reliability and storage efficiency, so no further
action is required.

Networking
PSCALE-113065 Nodes A300, A3000, H700, and H7000 ship with BMC LAN access disabled. Because of a firmware
issue, the following commands to enable access may not work, and you may not be able to change
allocation-type from static to dhcp or from dhcp to static:
isi ipmi settings modify --enabled=true --alloation-type dhcp -f

isi ipmi settings modify --enabled=true --allocation-type static-f
Workaround: Reset BMC by issuing the command:
ipmitool mc reset cold
on the affected node.

PSCALE-55830 If a node fails to mount the /ifs directory, the following error message can appear:
Error from drive_d: 61: Connection refused
Workaround: Reboot the node in question.

PSCALE-55676 In a multi-interface, same-subnet network configuration, Linux clients using the NFS over RDMA protocol
can sometimes incur Address Resolution Protocol (ARP) flux and the NFS mount will fail.
Workaround: See the Red Hat article How to connect two network interfaces on the same subnet for a
description of the problem and recommended solutions.
PSCALE-23063 The following warning message can appear repeatedly on the console:
Console spam - WARNING: Intel (R) Network Connections are quality tested
using Intel (R) Ethernet Optics.
Workaround: The message is harmless. You can safely ignore it.
NFS
PSCALE-57485 In rare circumstances, NFSoRDMA may not work after a node reboot.
Workaround: Restart the NFS service.
PSCALE-37664 If the sunrpc.rdma_slot_table_entries parameter is set to 128 or higher, the cluster might experience
write performance slowness.
Workaround: Do not set the sunrpc.rdma_slot_table_entries parameter to 128 or higher.
Security
PSCALE-60478 Upgrade from OneFS 8.1.2 to 9.2.0.0 or 9.2.1.0 does not preserve the HTTP basic auth setting.
Workaround: Before you upgrade, install the latest 8.1.2 RUP.
Upgrades
PSCALE-25579 If you are upgrading to OneFS 9.2.0.0 or later, CELOG maintenance mode is no longer supported. If you
are upgrading to OneFS 9.2.0.0 or later, suppressed mode might cause the upgrade to fail.
Workaround: Take your cluster out of maintenance mode before upgrading to OneFS 9.2.0.0 or later. Do
not put any nodes into suppressed mode until the upgrade has been committed.
PSCALE-22265 When upgrading OneFS, if your cluster is displaying critical hardware events, using the isi event bulk
--ignore or isi event bulk --resolve commands might not ignore or resolve the events and cause the
upgrade to fail.
No workaround.

OneFS 9.2.0.0
Known issues
For information about issues known to affect other OneFS releases, log in to Dell Technologies Support and view the available
OneFS Release Notes.
Antivirus
PSCALE-21576 If you use isi antivirus cava servers modify --server-name=<name> --server-
uri=<uri> to configure multiple CAVA server URIs and the interval between configurations is short,
the CAVA URI may not be configured correctly.
Workaround: Restart the lwavscan daemon.
PSCALE-23191 CAVA jobs always start on the first node in the cluster. If Node 1 fails or splits from the cluster, no
scheduled CAVA jobs can start until Node 1 rejoins the cluster. Running jobs are not affected.
Workaround: The issue resolves itself after Node 1 rejoins the cluster.
PSCALE-56419 The CAVA anti-virus server encounters a connectivity issue in an IPv6 environment.
Workaround: Use CAVA only in an IPv4 environment.
File system
PSCALE-48358 The following warning message can appear when the backup of multi TB files or data sets fails:
cannot allocate memory
Workaround: Reduce the number of NDMP read threads.

PSCALE-22253 After you upgrade to 9.2.0.0 or 9.2.1.0, a CELOG event can appear that warns about under protected
node pools, even though no hardware has changed. The warning appears because enhancements to
estimations of the reliability of some suggested protections policies in OneFS 9.2.0.0 or 9.2.1.0 can result
in different suggested protections.
Workaround: If there was no warning before the upgrade, you can safely ignore this event. Any new
suggested protection policies have the same estimated reliability and storage efficiency, so no further
action is required.
FTP and HTTP

PSCALE-48379 If permissions are set appropriately, OneFS files and directories can be accessed from a web browser
using HTTP or HTTPS. To allow your users to access files and directories from a web browser, ensure
that the files and directories have permissions set that allow HTTP access.
Networking
PSCALE-55830 If a node fails to mount the /ifs directory, the following error message can appear:
Error from drive_d: 61: Connection refused
Workaround: Reboot the node in question.

PSCALE-55676 In a multi-interface, same-subnet network configuration, Linux clients using the NFS over RDMA protocol
can sometimes incur Address Resolution Protocol (ARP) flux and the NFS mount will fail.
Workaround: See the Red Hat article How to connect two network interfaces on the same subnet for a
description of the problem and recommended solutions.
PSCALE-23063 The following warning message can appear repeatedly on the console:
Console spam - WARNING: Intel (R) Network Connections are quality tested
Workaround: The message is harmless. You can safely ignore it.
NFS
PSCALE-57485 In rare circumstances, NFSoRDMA may not work after a node reboot.
Workaround: Restart the NFS service.
PSCALE-37664 If the sunrpc.rdma_slot_table_entries parameter is set to 128 or higher, the cluster might experience
write performance slowness.
Workaround: Do not set the sunrpc.rdma_slot_table_entries parameter to 128 or higher.
Security known issues

PSCALE-60478 Upgrade from OneFS 8.1.2 to 9.2.0.0 or 9.2.1.0 does not preserve the HTTP basic auth setting.
Workaround: Before you upgrade, install the latest 8.1.2 RUP.
Upgrades
PSCALE-59530 When upgrading the cluster to OneFS 9.2.0.0, some eventgroups that occurred before or during the
upgrade will show up as having an "Unknown" cause. Some common events include reboot, synciq and
external_network, but can also include avscan, filesystem, hwmon, overheating, powersupply, physmem,
reboot-fail, storage_transport, synciq, windwos_auth, windows_idmap, and windows_networking. If new
events occur after these "Unknown" eventgroups are resolved, they will show up with the correct
message.
Workaround: The situation resolves itself after upgrade after the "Unknown" events are set to
"Resolved".
PSCALE-25579 If you are upgrading to OneFS 9.2.0.0 or later, CELOG maintenance mode is no longer supported. If you
are upgrading to OneFS 9.2.0.0 or later, suppressed mode might cause the upgrade to fail.
Workaround: Take your cluster out of maintenance mode before upgrading to OneFS 9.2.0.0 or later. Do
not put any nodes into suppressed mode until the upgrade has been committed.
PSCALE-22265 When upgrading OneFS, if your cluster is displaying critical hardware events, using the isi event bulk
--ignore or isi event bulk --resolve commands might not ignore or resolve the events and cause the
upgrade to fail.
No workaround.
OneFS 9.1.0.0

Known issues in OneFS 9.1.0.0
Issue ID Functional area Description Workaround
PSCALE-23231 SERC: Hardening If a STIG hardening profile is applied to a cluster, then If you have to add
during an upgrade, joining new nodes may fail when nodes to a hardened
they upgrade. cluster, either rollback the
upgrade or complete the
upgrade by committing it.
PSCALE-22815 Protocols: Antivirus If you configure CAVA server(s) that do not work and Run the isi
then enable CAVA, OneFS raises the CELOG event antivirus cava
SW_AVSCAN_CAVA_ALL_SERVERS_OFFLINE, but status command on
clears the event immediately. the node that reported
the event and confirm
whether or not the
event has actually been
resolved.
PSCALE-22198 Network: Flexnet Due to a rare race condition, it is possible for changes Retrying the operation
to the Flexnet configuration to race, and cause should succeed.
neither to take. When this occurs, the following error
message may appear:
Error writing /ifs/.ifsvar/modules/flexnet/
flx_config.xml
PSCALE-20905 Protocols: Antivirus SMB may become inaccessible if you enable the To avoid this issue,
CAVA service by running the command isi use isi antivirus
service cava enable. cava settings
modify --service-
To resolve this issue, run the following commands:
enabled=yes to enable
1. isi antivirus cava settings modify the CAVA service.
--service-enabled=no
2. isi antivirus cava settings modify
--revert-ip-pool
3. isi zone zones delete AvVendor
--ip-pool=groupnet0.subnet0.pool1
--service-enabled=yes
PSCALE-18859 Protocols: Antivirus It is possible for the CAVA daemon (lwavscan) to No action is required
crash while shutting down if there are outstanding when this occurs.
scan requests in its request queue. This generates a
core file, but does not affect the subsequent restart
of the daemon or normal operation of antivirus
scanning afterwards.
PSCALE-18830 FreeBSD: Kernel When rebooting a node, it is possible for NFS to If you encounter this
survive quorum loss, which might prevent a new issue, you must manually
instance of NFS from starting up on that node. If this stop NFS on that node.
issue occurs, var/log/messages will display the Stopping NFS will allow a
following message: new instance of NFS to
start.
service nfs already open by pid
.
PSCALE-18691 Protocols: NFS If an NFS client tries to connect to a cluster using the Users should avoid
Access Zone called AvVendor, the NFS daemon will connecting to the cluster
fail. with the AvVendor
Access Zone. When
CAVA antivirus is enabled,
the Access Zone called
AvVendor is created

strictly to be used by
the CEE/CAVA servers to
connect to the cluster.
PSCALE-17399 Supportability: The user remotesupport is disabled by default. To log into the
Remote Connectivity This is expected behavior. cluster with the
remotesupport user,
you must enable the
remotesupport user
and set a password for
that user.
PSCALE-17205 Protocols: HTTP If some OneFS network settings are modified, it The administrative user
is possible that an administrator user's session can should log in again.
pre-emptively expire. This is because OneFS must
regenerate the WebUI configuration.
PSCALE-17090 Upgrade and During an upgrade to a OneFS 9.1.0.0 or later build To avoid this issue,
Installation that includes a pre-installed patch, if a user enters enter the command isi
the command isi upgrade patches list/ upgrade patches
view on a node that hasn't been upgraded yet, it list/view on a node
might cause PAPI to fail on that node. that has already been
upgraded.
PSCALE-16216 Network During system boot, a message similar to the You can safely ignore this
following can appear: message.
WARNING: Intel (R) Network

Connections are quality tested
Using untested modules is not
supported and may cause unstable
operation or damage to the module
or the adapter. Intel Corporation
is not responsible for any harm
caused by using untested modules.
PSCALE-15733 Upgrade and During an upgrade from OneFS 9.0.0.0 to OneFS If you encounter this
Installation 9.1.0.0, a node might enter a state that keeps the issue, you must manually
node from rebooting. If this issue occurs, it might reboot any affected
cause the upgrade to stall. nodes after other nodes
in the cluster have
finished upgrading.
PSCALE-15298 Protocols: Antivirus The isi antivirus cava status Review your antivirus
-v command and isi statistics software's logs.
query current --format json --keys
node.antivirus.statscommand output does
not count the number of CAVA jobs under Scan Fails
in the scan report. The count is provided only for
on-access scan.
PSCALE-14506 Protocols: Antivirus If you create a CAVA job with one non-existing path
and multiple valid paths, the job never starts. You can
find the following message in the lwavscand.log.
Startup of Job <job>

failed
The lwavscand.log does not provide information

about why the job fails.
PSCALE-14486 Protocols: Antivirus If you create CAVA jobs with non-existing paths, To avoid this issue,
you may find information about the job failure in the remove or update the
lwavscand.log on node 1 every 15 seconds. non-existing path.

PSCALE-13471 Protocols: Antivirus If node1 is separated from the cluster, none of To avoid this issue, you
the CAVA jobs started. CAVA server offline events can restart the antivirus
are raised even though after running the isi process on all nodes.
antivirus cava status command, the system
status appears as "RUNNING."
PSCALE-13389 Network: Flexnet Under rare circumstances, isi_flexnet_d can fail Restart
to assign IPs when it regains quorum when an IP isi_flexnet_d.
movement happens shortly before the quorum regain.
The workaround is to restart isi_flexnet_d
273443, Upgrade and If you have Drive Support Package (DSP) v1.33
PSCALE-12634 Installation installed on your cluster and upgrade to OneFS
9.0.0.0, the system checks performed after upgrade
might fail.
273087, Protocols: Antivirus Under certain workloads, the CAVA antivirus scan You must install a patch
PSCALE-11729 requests are not balanced among the CEE/CAVA to avoid this issue.
servers connected to the node. Some CEE/CAVA
servers may handle the majority of requests, while
others handle few or no requests.
272919 ,PSCALE-93 Protocols:S3 If you use the S3 protocol with SmartConnect, and To avoid this issue, set
06 a zone's base domain is not set using the isi s3 the base domain using
zone settings modify command, the client may the isi s3 zone
receive signature mismatch and other errors. settings modify
command.
272018 Protocols:S3 If a multipart upload is performed on a directory
for which the bucket owner does not have access
permission, the bucket owner cannot stop the upload
or list parts.
271215, Protocols:S3 When you try to list multipart uploads using
PSCALE-8902 Delimiter='/', groups are not broken into common
prefixes.
271214, Protocols:S3 When listing multipart uploads, the Delimiter field
PSCALE-8890 is silently ignored, and the slash character is always
used as a delimiter.
271084, Protocols:S3 The S3 protocol does not include packet header sizes
PSCALE-13347 in bandwidth usage statistics.
271027 Protocols:S3 Etag changes on the SyncIQ target cluster.
270819 Upgrade and In rare circumstances, if /var is full during an To avoid this issue,
Installation upgrade, the upgrade can fail. If the upgrade fails, delete /var/ifs/upgrade/
a file on /var may become corrupted. Even after you AgentPersistent.db on all
make space on /var, the corrupted file will prevent affected nodes, then run
the upgrade from continuing. isi upgrade retry-
last-action.
268952 Protocols:S3 If Bucket Directory Create Mode for a zone is To avoid this issue,
configured without the execute permission, clients ensure to add the execute
cannot access the bucket. permission when changing
the Bucket Directory
Create Mode using the
isi s3 settings
zone command.
267883 , Protocols:S3 The S3 protocol does not support a file name which is
PSCALE-8852 not represented as a valid UTF-8 string. As a result,
the following error might appear:
400(InvalidURI): Path cannot be

encoded as UTF-8 string.

267549 Protocols:S3 Administrators can delete S3 buckets that are not
empty. This behavior differs from the S3 protocol,
which does not allow non-empty bucket deletion.
OneFS handles bucket deletion the same way that
SMB share and NFS export deletion are handled.
When you delete an S3 bucket through OneFS, the
bucket path is deleted but the directory it referenced
still exists. If you create a new bucket with the same
path as the bucket that was deleted, the directory
that the previous bucket referenced will be accessible
again through the new bucket.
267404 Hardware After you smart fail a drive, remove the drive from
the system, and then reinsert the drive, false power
supply events may occur. This is due to a known
firmware issue.
266840, Protocols:S3 If S3 bucket operations like Put-Bucket and Delete-
PSCALE-8756 Bucket go through the protocol layer, they are not
audited.
266764 Protocols:S3 If you modify or delete an S3 bucket immediately
after creating it, the operation may fail because the
previous bucket configuration has not propagated
yet. Retry the modification or deletion. The operation
will eventually succeed
264794, Protocols:S3 encoding-type=url for GET Bucket (List Objects) is
PSCALE-8898 not supported.
264538, Protocols:S3 The S3 protocol does not support symlink. If you try
PSCALE-8897 to access the symlink file or directory, the following
error message is displayed:
InvalidObjectState (403)
264415, Protocols:S3 The S3 protocol does not validate the Content-Md5

PSCALE-8872 request header. To avoid calculating Md5, a synthetic
etag is created when Content-Md5 is not supplied in
the request.
263910 Upgrade and If you perform a parallel upgrade from OneFS 8.2.2 to To avoid this issue,
Installation OneFS 9.0.0.0, you might encounter an issue where restart the parallel
all nodes on a network appear to be unavailable. If upgrade and use the --
this issue occurs, the upgrade fails. skip-optional option.
263655, Protocols:S3 While creating or modifying a bucket from the OneFS
PSCALE-8855 web administration interface, you can add the same
access control list (ACL) multiple times to the
bucket.
263571, Protocols:S3 The S3 protocol does not support aws-chunked
PSCALE-8870 (streaming upload) content-encoding.
262949, Protocols:S3 If you try to remove a bucket through PAPI
PSCALE-8833 when the S3 service is disabled, the bucket gets
deleted. But, the temporary directory (.isi_s3_xxx)
under the bucket root directory and part directories
(.isi_s3_parts_xxx) under the target directory does
not get deleted. You must remove these directories
manually.
261596 Upgrade and If you perform a rolling upgrade of your OneFS No workaround. You
Installation cluster, and you have dynamic IPs configured, must manually move the
SmartConnect might not allocate them properly after dynamic IPs after the
the upgrade. upgrade.

212754 Protocols: Antivirus The AVSCAN policy impact level is not passed to the
Job Engine, meaning that no priority can be given to
the AVSCAN jobs over other Job Engine-driven jobs.
OneFS 9.0.0.0
File system
263645 In rare circumstances, when inline deduplication is enabled on large clusters, the performance of nodes
that have a mirror of a certain shadow store reference count block may become limited.
Workaround: Inline dedupe can be enabled on large clusters but you should disable the
efs.sfm.inline_dedupe.do_read sysctl. Use the following command to disable the sysctl:
isi_for_array sysctl efs.sfm.inline_dedupe.do_read=0
Hardware
267404 After you smartfail a drive, remove the drive from the system, and then reinsert the drive, false power
supply events may occur. This is due to a known firmware issue which will be addressed in a June, 2020
update.
Networking
272321 A rare race condition in the InfiniBand driver or a faulty InfiniBand card can cause a node to fail with a
panic message:
Fatal trap 12: page fault while in kernel mode
Followed by a panic stack such as the following:
Stack: --------------------------------------------------
kernel:trap_fatal+0xa2
kernel:trap_pfault+0x2fd
kernel:trap+0x61d
kernel:ib_create_ah+0x9
kernel:cm_alloc_msg+0x33
kernel:ib_send_cm_dreq+0xbe
kernel:rdma_disconnect+0x149
kernel:sdp_output_reset+0xa4
kernel:sdp_shutdown_task+0x96
kernel:taskqueue_run_locked+0xe5
kernel:taskqueue_thread_loop+0xa7
kernel:fork_exit+0x74
Workaround: Rebooting the node will resolve the issue if the issue was caused by the rare race
condition. If rebooting the node does not resolve the issue, a faulty InfiniBand card may be the cause.
Contact Dell Technologies Supportto get help with replacing the card on the node.

Protocols
272944 If the OneFS cluster is configured with dynamic IPs and the S3 protocol, it is possible for a rare race
case to occur that can make the S3 process unresponsive. This situation may arise if multiple connections
occur simultaneously to the same dynamic IP after it fails to move to a different node within the specified
time period.
272910 In rare cases, you can encounter a situation in which a node splits but does not join the group, or a node
becomes hung while attempting to reboot. To check whether your node is in this state, run the following
command:
ps -uax | grep lw
If lwsmd is the only service listed and has not logged any messages to /var/log/lwsmd.log since
it stopped services, it is possible that your node is in this state. To resolve the situation, kill the lwsmd
process.
272919 If you use the S3 protocol with SmartConnect, and a zone's base domain is not set using the isi s3
zone settings modify command, the client may receive signature mismatch and other errors.
Workaround: To avoid this issue, ensure to set the base domain using the isi s3 zone settings
modify command.
272713 If ESRS with telemetry is enabled and NFS is not enabled, the isi_esrs_d logs will show an error when
gathering telemetry data for vfs.nlm and nfs/log-level. You can ignore these errors.
272430 An Active Directory username specified during the S3 secret key generation does not give a user access
to S3 unless it is specified in the canonical format of DOMAIN\user, where DOMAIN is uppercase, and
user is lowercase.
Additionally, an Active Directory username specified in the ACL of a bucket does not give that user access
unless it is specified in the same canonical format.
272287 On certain occasions, the S3 service associated with the cluster's health conditions may not be ready
to accept new connections and requests for a brief period. As a result, you may experience data
unavailability (DU).
272018 If a multipart upload is performed on a directory for which the bucket owner does not have access
permission, the bucket owner cannot stop the upload or list parts.
271215 When you try to list multipart uploads using Delimiter='/', groups are not broken into common prefixes.
271214 When listing multipart uploads, the Delimiter field is silently ignored, and the slash character is always
used as a delimiter.
271084 The S3 protocol does not include packet header sizes in bandwidth usage statistics.
271027 Etag changes on the SyncIQ target cluster.
268952 If Bucket Directory Create Mode for a zone is configured without the execute permission, clients cannot
access the bucket.
Workaround: To avoid this issue, ensure to add the execute permission when changing the Bucket
Directory Create Mode using the isi s3 settings zone command.
267883 The S3 protocol does not support a file name which is not represented as a valid UTF-8 string. As a result,
the following error might appear:
400(InvalidURI): Path cannot be encoded as UTF-8 string.
267549 Administrators can delete S3 buckets that are not empty. This behavior differs from the S3 protocol,
which does not allow non-empty bucket deletion. OneFS handles bucket deletion the same way that SMB
share and NFS export deletion are handled. When you delete an S3 bucket through OneFS, the bucket
path is deleted but the directory it referenced still exists. If you create a new bucket with the same path
as the bucket that was deleted, the directory that the previous bucket referenced will be accessible again
through the new bucket.
266840 If S3 bucket operations like Put-Bucket and Delete-Bucket go through the protocol layer, they are not
audited.

266764 If you modify or delete an S3 bucket immediately after creating it, the operation may fail because the
previous bucket configuration has not propagated yet. Retry the modification or deletion. The operation
will eventually succeed.
264794 encoding-type=url for GET Bucket (List Objects) is not supported.
264538 The S3 protocol does not support symlink. If you try to access the symlink file or directory, the following
error message is displayed:
InvalidObjectState (403)
264415 The S3 protocol does not validate the Content-Md5 request header. To avoid calculating Md5, a
synthetic etag is created when Content-Md5 is not supplied in the request.
263655 While creating or modifying a bucket from the OneFS web administration interface, you can add the same
access control list (ACL) multiple times to the bucket.
263571 The S3 protocol does not support aws-chunked (streaming upload) content-encoding.
262949 If you try to remove a bucket through PAPI when the S3 service is disabled, the bucket gets deleted.
But, the temporary directory (.isi_s3_xxx) under the bucket root directory and part directories
(.isi_s3_parts_xxx) under the target directory does not get deleted. You must remove these
directories manually.
Security known issues

267949 If you are running the S3 service and change any of its ports, you must manually restart the S3 service for
changes to take effect.
Workaround:
To restart the S3 service, log on as root and run the following commands:
isi services s3 disable

isi services s3 enable
SmartPools
271386 When adding A100 nodes to a cluster or when forcing reprovisioning of A100 nodes, it is possible that a
CELOG event similar to the following appears:
The identified node is not provisioned.
When a node is not provisioned, that means that it is not associated with a node pool. You cannot write to
a node that is not provisioned.
Workaround: Attempt to move the unprovisioned A100 nodes into the same node pool by using
compatibilities. See the OneFS Web Administration Guide or OneFS CLI Administration Guide for details.
SMB
267941 If a client connects to an SMB3-CA share using a local account, automatic migration during a planned
maintenance event may not happen correctly. This can increase the length of an I/O pause, although it
will not lead to a data unavailable condition.
Workaround: To avoid this issue, ensure that the cluster has been joined to the same Active Directory
domains as the SMB3-CA clients.

Upgrades and Installation
PSCALE-13838 If your OneFS 9.0.0.0 cluster includes PowerScale F200 or PowerScale F600 nodes, and you have SRS v2
installed, the SRS service might not work properly. (PSCALE-13838)
Workaround: Upgrade to SRS v3 before enabling the SRS service on clusters that include the
PowerScale F200 or PowerScale F600 nodes.
273443 If you have Drive Support Package (DSP) v1.33 installed on your cluster and upgrade to OneFS 9.0.0.0,
the system checks performed after upgrade might fail.
No workaround.
271388 If you perform a simultaneous node firmware upgrade on a OneFS cluster, an isi_papi_d error might cause
the firmware upgrade to fail.
Workaround: Run the isi upgrade retry-last-action command to resolve the issue.
271316 If you upgrade from OneFS 8.1.3.0 or earlier to OneFS 8.2.x or later, and commit the upgrade, any new
CloudPool recall jobs will be paused until the smartlink-upgrade job completes.
Workaround: After the smartlink-upgrade job completes, all CloudPool recall jobs will resume.
271063 If you upgrade a hardened OneFS cluster using the simultaneous upgrade option, the upgrade might fail.
Workaround: Uninstall the cluster hardening, or use the rolling upgrade option.
270819 In rare circumstances, if /var is full during an upgrade, the upgrade can fail. If the upgrade fails, a file
on /var may become corrupted. Even after you make space on /var, the corrupted file will prevent the
upgrade from continuing.
Workaround: Delete /var/ifs/upgrade/AgentPersistent.db on all affected nodes, then run
isi upgrade retry-last-action.
270747 If you perform a parallel upgrade on your OneFS cluster, a hook error might occur on a node causing the
upgrade to fail. If this issue occurs, you will see the following error in the upgrade log: Failure during
upgrade status gather.
Workaround: Run the isi upgrade retry-last-action command on the node that had the error.
The lnn of the failed node can be found in the following location in the upgrade log:
HookDataMigrationUpgrade|Failed:7
268043 If you perform a rollback of a cluster upgrade and, while the cluster is archiving, issue an isi_for_array
hostname command, it might cause the rollback process to stall.
Workaround: If you encounter this issue, extend the isi_for_array timeout by running the following
command: isi_for_array --rpc-timeout 180
263910 If you perform a parallel upgrade from OneFS 8.2.2 to OneFS 9.0.0.0, you might encounter an issue where
all nodes on a network appear to be unavailable. If this issue occurs, the upgrade fails.
Workaround: Use the --skip-optional option when you restart the parallel upgrade.
261596 If you perform a rolling upgrade of your OneFS cluster, and you have dynamic IPs configured,
SmartConnect might not allocate them properly after the upgrade.
Workaround: Manually move the dynamic IPs after the upgrade.
OneFS Web UI
272790 The WebUI does not search hidden directories. For example, if you place directories or files in a hidden
directory such as /ifs/.secretdir and then browse for them using the WebUI, you will not be able to
find them.
Workaround: Manually add the proper path, or do not place directories or files in hidden directories.
271291 In some cases, the WebUI does not display complete IPv6 addresses.
Workaround: To view the complete IPv6 addresses for the internal network, go to the Network
Configuration > Internal tab. To view the complete IPv6 addresses for the external network, go to
the Network Configuration > External tab and click View Pool Details.

268970 The OneFS WebUI may not list all existing snapshots.
Workaround: To get a complete list, use the CLI command isi snapshot snapshots list .

Ifs Pub Upgrade Planning and Process Guide en Us

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ifs Pub Upgrade Planning and Process Guide en Us

Uploaded by

Copyright:

Available Formats

PowerScale OneFS

Upgrade Planning and Process Guide

Chapter 1: Introduction to this guide............................................................................................. 6

Chapter 2: Planning an upgrade.....................................................................................................7

Chapter 3: Completing pre-upgrade tasks....................................................................................12

Chapter 4: Performing the OneFS upgrade.................................................................................. 23

Chapter 5: Completing post-upgrade tasks................................................................................. 32

Appendix A: Features in OneFS Releases..................................................................................... 38

Appendix B: Known Issues in OneFS Releases..............................................................................49

About this guide

Where to get help

Additional options for getting help

Dell Technologies Support ● https://www.dell.com/support/incidents-online/en-us/contactus/product/

6 Introduction to this guide

Upgrade planning overview

Planning an upgrade - summary

Prerequisites for completing an upgrade

● PowerScale OneFS Current Patches

Types of OneFS upgrades

Verify the upgrade path

Table 1. OneFS Upgrade Paths

Check supportability and compatibility requirements

Assess upgrade impact

Consider upgrade limitations

Review feature changes and known issues

Plan an upgrade schedule

Pre-upgrade process - Overview

12 Completing pre-upgrade tasks

Review and save information about cluster status

Gathering cluster logs

Gather cluster logs - Web UI

Gather cluster logs - CLI

isi diagnostics gather start

Completing pre-upgrade tasks 13

Check hardware health - web administration interface

Check hardware health - CLI

isi_for_array -s 'isi devices drive list | egrep -v "HEALTHY|L3|JOURNAL"'

3. To check the mirror status of the boot drives on each node.

isi_for_array -s 'gmirror status'

isi_for_array -s 'ps -auwwwx | grep opensm' | grep master

node-2: root 1610 0.0 2.3 436292 384672 ?? Ss 20May15 97:31.63

14 Completing pre-upgrade tasks

Check the available free space

Available space requirements for clusters, nodes, node pools, and

Item Minimum Requirement Task

Completing pre-upgrade tasks 15

isi_for_array -s 'df -h'

virtual-demo-3: Filesystem Size Used Avail Capacity Mounted on

virtual-demo-2: Filesystem Size Used Avail Capacity Mounted on

Resolve outstanding events and errors

isi event groups list --sort=severity

16 Completing pre-upgrade tasks

Upgrading in a Cloudpools environment

Preserve the Kerberos keytab file

<?xml version="1.0" encoding="UTF-8"?>

Install a supported version of DataIQ or InsightIQ

Download the OneFS installation bundle file

Completing pre-upgrade tasks 17

tar -vxf /ifs/data/<installation-file-name> /ifs/data/

Upgrade compatibility check utility

Run the upgrade compatibility check utility

18 Completing pre-upgrade tasks

Upgrade compatibility check summary