VxRail ™ SolVe Generator

Solution for Validating your engagement

Topic
VxRail Software Upgrade Procedures
Selections
Select the VxRail Appliance Model: VxRail E560/E560F
Select the Installed VxRail Software Version: v4.7.526
Select the Target VxRail Software Version: v7.0.372
Is VMware Horizon VDI Installed?: No
Select your current VxRail Configuration: VxRail Cluster - non-vSAN Stretched Cluster

Generated: August 5, 2022 10:40 PM GMT

REPORT PROBLEMS

If you find any errors in this procedure or have comments regarding this application, send email to
SolVeFeedback@dell.com

Copyright © 2022 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell
EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be
trademarks of their respective owners.

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of
any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable
software license.

This document may contain certain words that are not consistent with Dell's current language guidelines.
Dell plans to update the document over subsequent future releases to revise these words accordingly.

This document may contain language from third party content that is not under Dell's control and is not
consistent with Dell's current guidelines for Dell's own content. When such third party content is updated
by the relevant third parties, this document will be revised accordingly.

Publication Date: August, 2022

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 1 of 19
 Contents
Preliminary Activity Tasks .......................................................................................................3
Read, understand, and perform these tasks.................................................................................................3

Data-in-Place upgrade solution from v4.7.300 or later to v7.0.3xx .........................................4

Summary:......................................................................................................................................................4
Prerequisites: ................................................................................................................................................4
Materials Required:.......................................................................................................................................5
Procedure: ....................................................................................................................................................5
VxRail Cluster Upgrade ................................................................................................................................6
Task 1: Disk Space Expansion (Applicable for releases earlier than v4.7.301 and release
v4.7.400 only) 6
Task 2: [Optional] Configure alarm definition for email notification ...............................................9
Task 3: Take a snapshot of all the service VMs............................................................................9
Task 4: Upgrade from v4.7.30x/4xx/5xx to v7.0.3xx .....................................................................9
Task 5: Remove snapshots of the service Virtual Machines.......................................................15
Task 6: Mystic Service account password change (Optional).....................................................15
Appendix A: Get the correct controller key value..............................................................17
Troubleshooting ....................................................................................................................19

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 2 of 19
Preliminary Activity Tasks
This section may contain tasks that you must complete before performing this procedure.

Read, understand, and perform these tasks

1. Table 1 lists tasks, cautions, warnings, notes, and/or knowledgebase (KB) solutions that you need to
be aware of before performing this activity. Read, understand, and when necessary perform any
tasks contained in this table and any tasks contained in any associated knowledgebase solution.

Table 1 List of cautions, warnings, notes, and/or KB solutions related to this activity

523973: VxRail: BOSS Card not detected in iDRAC or M.2 device reports failure causing volume
degraded

88055: SSH access may fail post upgrade to ESXi 7.0 Update 3d

• VxRail 7.0.320 and later releases contain resolutions for two of the VMware vSAN issues detailed in
VMware Proactive Customer Advisory issued in June 2022: KB 88870
https://kb.vmware.com/s/article/88870. One issue is not yet resolved by VMware and requires the
workaround described in KB 88870. The Dell VxRail: VxRail Response to VMware Proactive
Customer Advisory for June 2022 contains more information https://www.dell.com/support/kbdoc/EN-
US/000201082.

2. This is a link to the top trending service topics. These topics may or not be related to this activity.
This is merely a proactive attempt to make you aware of any KB articles that may be associated with
this product.

Note: There may not be any top trending service topics for this product at any given time.

VxRail Appliance Top Service Topics

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 3 of 19
IMPORTANT: Please confirm whether a customer is using RecoverPoint for Virtual Machines (RP4VM)
in the VxRail cluster, prior to upgrading VxRail. RP4VM must be upgraded prior to upgrading VxRail to
prevent a DRU/DU issue on the VxRail cluster!

Note: It is highly advisable to monitor node reboots through IDRAC/BMC interfaces to detect boot related
issues.

Data-in-Place upgrade solution from v4.7.300 or later to v7.0.3xx

Summary:
This procedure describes the steps required to perform a data-in-place upgrade on a VxRail from
v4.7.300 or later to v7.0.3xx.
Notes:
• During an LCM upgrade from VxRail v4.x or 7.0.x to a later 7.0.x release, VMware VDS upgrade
resets multicast filtering mode to the default, enabling IGMP/MLD snooping on the VDS. IGMP/MLD
snooping can potentially affect Microsoft Network Load Balancer (NLB) applications. If your
environment is configured with Microsoft NLB for VxRail-hosted applications, the virtual IP will not be
reachable until you manually disable IGMP/MLD snooping. See KB 000196307 for more detailed
information.
• Starting from 7.0.370, VxRail Manager uses subnets and IP address for RKE2 (Rancher's next-
generation Kubernetes distribution) network. If an existing IP is accessible for VxRail Manager and
overlaps with any of the RKE2 subnets or IP address, an IP conflict or a routing issue can occur. See
KB 000197988 for detailed information to avoid such issue before upgrading.
• Upgrades from VxRail versions earlier than 4.7.410 might fail on ESXi hosts with secure boot
enabled. See KB 000529655 for details and workaround.
• vSphere 7.0 license keys are required for a cluster to run on vSphere 7.0. The LCM precheck will not
stop you from upgrading without 7.0 license keys. Without them, the nodes will run on 60-day
evaluation license keys. Go to my.vmware.com to determine whether your clusters need 7.0 license
keys and acquire them there if applicable.
• Upgrading from VxRail software v4.7.300: If you applied any workaround from KB 537948 to mitigate
the Excessive Hardware health alarms being triggered for "Sensor -1 type" on ESXi hosts running
vSphere 6.7 U3 issue, see KB 000023633 to learn whether you need to reverse any workarounds
after completing the v7.0.3xx upgrade.
• If the VxRail Manager SSL certificate has been changed, see KB 000158410 (How to prevent
vCenter plug-in issue caused by VxRail Manager SSL certificate mismatch) before proceeding with
the upgrade.

Prerequisites:
• Before starting the upgrade, reconfigure the Witness virtual SCSI controller to use a Paravirtual SCSI
controller. For more details, see VMware KB solution kb.vmware.com/s/article/76948 vSAN Health
Service - Stretched cluster - Witness appliance upgrade to vSphere 7.0 or higher with caution
(76948).
• The Cluster will lose its embedded vSAN license after the upgrade. See
kb.vmware.com/s/article/79361 for a workaround.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 4 of 19
 • If the VxRail Cluster is joining an External vCenter, check the VxRail and External vCenter
Interoperability Matrix KB 000157682 to ensure that the External vCenter is running a supported
version.
• All customer-supplied vCenter configurations must be upgraded to vCenter 7.0 U1 before upgrading
to VxRail 7.0.3xx. See KB 000181307 and follow the detailed procedure to update vCenter settings.
• Ensure that VxRail health monitoring is disabled before upgrading the external vCenter to avoid false
alerts.

Materials Required:
Ensure that the following passwords are available:
• VxRail Manager Root
• vCenter Administrator
• vCenter Root
• PSC Root
• ESXi Node Root
• iDRAC/BMC (aka Baseboard Management Controller (if configured)
VxVerify detects issues which could cause complications or failures during VxRail LCM upgrades.
Download the VxVerify tool per the instructions in KB 000021527.
A temporary IP address on the same subnet needed to upgrade vCenter Server (not required if using
external vCenter).
There is no need to load the software composite file locally when performing an Internet connected
upgrade. The upgrade file automatically downloads from Dell Technologies.
If you will not have Internet connection, download the file from dell.com/support/ and make it locally
available.
Upgrade Composite Bundle File name (Choose one of the following options):

Note: From the Composite bundle download page, record the checksum values under File Name
section, and validate them after the bundle had been downloaded in your local client.

1. Full bundle for Internal or External vCenter

• VXRAIL_COMPOSITE-7.0.3xx-xxx_for_4.7.x.zip
2. Slim bundle for External vCenter only: This composite file does not include the Internal vCenter files,
which reduce the overall file size.
• VXRAIL_COMPOSITE-SLIM-7.0.3xx-xxx_for_4.7.x.zip

Procedure:
Note: Screenshots in this document might differ depending on the VxRail version you are running.

1. [ ] Run the VxVerify tool. Instructions are available in KB 000021527.

2. [ ] Log in to the vCenter Server using HTLM5 Client with administrative privileges.
3. [ ] Ensure that the VxRail is in a healthy state. Address any issues found before starting the
upgrade process.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 5 of 19
 a. Go to Home > Hosts and Clusters > VxRail-Cluster.
b. Select Monitor > Issues and Alarms > All issues.
c. Verify the VxRail cluster health status.
4. [ ] Confirm that the VxRail is currently running v4.7.300 or later.

a. Go to Home > Hosts and Clusters > VxRail-Cluster.

b. Click Configure > VxRail > System.

Figure 1 Review the version.

VxRail Cluster Upgrade

Task 1: Disk Space Expansion (Applicable for releases earlier than v4.7.301 and release
v4.7.400 only)
Note: For releases earlier than v4.7.301 and release v4.7.400 only, there is not enough disk space in
the VxRail Manager /data/store2 to perform an upgrade to v7.0.3xx. The bundle is larger than the
available space, therefore the disk_space_expansion.py script must be run to add additional space to
/data/store2.

1. [ ] Download the script disk_space_expansion.py from dell.com/support/.

2. [ ] Use FileZilla to upload the disk_space_expansion.py to the VxRail Manager VM. Open FileZilla
and enter the following information:
Host: <enter_vxrail_manager_ip_address>
Username: mystic
Password: <password>
Port: 22

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 6 of 19
 Figure 2 Upload the disk_space_expansion.py.

3. [ ] Click Quickconnect to connect to the VxRail Manager VM. The VxRail Manager default directory
is /home/mystic.
4. [ ] Upload the disk_space_expansion.py file to the /home/mystic directory. Once the upload has
completed, close FileZilla.
5. [ ] Using PuTTy or other SSH client, establish an SSH session to the VxRail Manager VM using the
following permissions:
Host Name (or IP address): <enter_vxrail_manager_ip_address>
Port: 22
Login: mystic
Password: <password>
6. [ ] Switch to root user account.
su <enter>
Password: <password>
7. [ ] Run the ls -l command to verify that the file has been uploaded.

Figure 3 Disk space expansion.py file was uploaded.

8. [ ] Run the df -Th command. Note in this example that the /data/store2 size is approximately 14G,
which is not large enough to accommodate the upgrade bundle.

Figure 4 Datastore size

9. [ ] Run the script with the following command. It will display the message Config logic volume
succeeded and Running script succeeded if the script ran successfully.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 7 of 19
 python disk_space_expansion.py -c <vcenter_ip> -u <vcenter_usr> -p
<vcenter_pwd> [-s] [-k <controller_key>]
For example:
python disk_space_expansion.py -c 10.xx.xx.xx -u
administrator@vsphere.local -p Passw0rd!

Figure 5 Script ran successfully.

Note: The -p option is optional. If it is not provided, the script prompts you to enter the vCenter
password in interaction mode.

For example:
python disk_space_expansion.py -c <vc_ip> -u <vc_usr> [-s] [-k
<controller_key>] VC password:'<vc_pwd>'

Note: The -s (skip taking snapshot) option is optional. If it is not provided, the script will take a
snapshot of the VxRail Manager VM automatically.

For example:
python disk_space_expansion.py -c 10.xx.xx.xx -u
administrator@vsphere.local -p Passw0rd! -s

Note: The -k (specify a controller key value) option is optional. If the disk expansion fails due to a
controller key error, use this option to provide a controller key value when adding disk space to the
VM.

For example:
python disk_space_expansion.py -c 10.xx.xx.xx -u
administrator@vsphere.local -p Passw0rd! -k 1000

Note: Go to Appendix A to obtain the correct controller key value if the script fails.

10. [ ] After the disk space expansion has completed, run the df -Th command to verify the /data/store2
has been extended. It should be approximately 77G. Once the disk space expansion has completed,
close the SSH session.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 8 of 19
 Figure 6 Datastore extended.

Task 2: [Optional] Configure alarm definition for email notification

To enable email notification during LCM upgrade, see KB 000019243.
Email notification is sent out during the following stages:
• When the cluster starts to perform the LCM upgrade.
• For each node:
• When the node starts the upgrade
• When the node completes the upgrade
• During any node upgrade failure
• When the cluster completes LCM upgrade

Task 3: Take a snapshot of all the service VMs

• VxRail Manager VM
• VMware vCenter Server VM (if using embedded)
• VMware vCenter Server Platform Services Controller VM (if using embedded)
• VMware vRealize Log Insight VM (if using VxRail deployed syslog)
• Secure Remote Services (SRS) VE VM (if call home is enabled and using VxRail Manager deployed
SRS)

1. [ ] Go to Hosts and Clusters.

2. [ ] Take a snapshot of a service VM. Right-click VxRail Manager > Snapshots > Take Snapshot.
3. [ ] Enter a snapshot name and click OK.
4. [ ] Repeat Steps 2 and 3 for each of the remaining service VMs.

Task 4: Upgrade from v4.7.30x/4xx/5xx to v7.0.3xx

1. [ ] Click VxRail Cluster > Configure > VxRail > Updates.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 9 of 19
 Figure 7 Configure VxRail updates.

2. [ ] If the cluster is not Internet-enabled, skip to Step 6 to perform the LOCAL UPDATES option.
3. [ ] If connected to the Internet, select INTERNET UPDATES to automatically get the required file.
4. [ ] If the support account is not already set, you must enter your Support username and password
and click Login. You will then be prompted to select INTERNET UPDATES again.
5. [ ] The required file will begin to upload. When completed, skip to Step 10.
6. [ ] If the cluster is not Internet-enabled, click LOCAL UPDATES.
7. [ ] Click BROWSE.
8. [ ] In open file dialog, select the upgrade composite package file that was previously downloaded
and saved locally, and then click Open.

Figure 8 Select the upgrade composite package.

9. [ ] The upload process begins.

10. [ ] Monitor the upload progress in the UI. DO NOT navigate away from upload page until the
uploading process is complete.
11. [ ] After a short delay, extraction and pre-checking will begin automatically.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 10 of 19
 Figure 9 Extracting bundles and prechecking progress.

12. [ ] [Optional] Run a precheck and identify issues before you apply the update. If upgrading from a
release earlier than 4.7.510, skip to Step 13.

a. Click PRE-CHECK.
b. Click RUN PRE-CHECK.
c. Enter the usernames, passwords, and temporary IP settings as prompted. Input fields differ based
on whether you are using an external vCenter or internal vCenter. Click VALIDATE to continue.
d. Review the credentials and click CONFIRM.
e. Monitor the precheck process.
f. Click CLOSE PRE-CHECK.
13. [ ] Click CONTINUE UPDATE to start the upgrade. If upgrading from a release earlier than 4.7.510,
click CONTINUE.
14. [ ] Choose option A to immediately begin the update or option B to schedule the update for a later
date and time:
Option A: Select Update Now and click NEXT.

Figure 10 Update now.

Option B: Select Schedule Update, enter the date and time for the update to begin and click
NEXT.
15. [ ] Enter the usernames, passwords, and temporary IP settings as prompted. Input fields will differ
depending on whether you are using an external vCenter or internal vCenter. Click NEXT to continue.
16. [ ] After you have entered all the fields (multiple screens), click FINISH to start the update process.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 11 of 19
17. [ ] Monitor the upgrade progress on the screen.
Notes:
• If you are logged out of the VxRail Manager UI, you can relaunch it by selecting VxRail Cluster >
Configure > VxRail > Updates.
• To monitor the progress after VxRail Manager had been upgraded, log out of vCenter and then log in
again to redeploy the VxRail plug-in. The progress shows automatically.
18. [ ] When the upgrade completes, log in to vCenter Server using HTLM5 Client with
administrator@vsphere.local and the vCenter password.

Figure 11 vCenter Log in screen

19. [ ] From the Home tab, select Hosts and Clusters and select the VxRail Cluster. Under the
Configure tab, expand the VxRail selection and choose Updates to verify the VxRail component
versions.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 12 of 19
 Figure 12 Upgrade complete.

20. [ ] Check compliance status. Select CREATE NEW REPORT to create a compliance drift report.
21. [ ] Click CREATE.
22. [ ] Click View the report.
23. [ ] Review the report to ensure that there are no noncompliant components. Click the left arrow to
expand the details of a component.

Figure 13 VxRail Compliance Drift Report

24. [ ] Ensure that the VxRail is in a healthy state.

a. Go to Home > Hosts and Clusters > VxRail-Cluster.

b. Select Monitor > Issues and Alarms > All Issues.
c. Verify the VxRail cluster health status.
25. [ ] If SRS is enabled in your VxRail, check the SRS status.

a. Go to Home > Host and Clusters > VxRail-Cluster.

b. Select Configure > VxRail > Support.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 13 of 19
 Figure 14 Support page.

c. In the Support section, check the last call home time.

Figure 15 Check Last heartbeat.

d. Open in browser: https://<SRS VE IP>:9443. Login with admin username and password. Check
whether SRS VE connectivity status is connected.

Figure 16 Check the connectivity status.

e. If not connected, perform the following steps to correct the status.

a. Go to Home > Hosts and Clusters > VxRail-Cluster.

b. Select Configure > VxRail > Support.
c. In the SRS section, click DELETE.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 14 of 19
 d. After successful deletion, click EDIT to add it again.

Task 5: Remove snapshots of the service Virtual Machines

Once the upgrade has completed successfully, remove all the snapshots created in the “Take a snapshot
of the service Virtual Machines” section to prevent vSAN capacity utilization long term:
• VxRail Manager VM
• VMware vCenter Server VM (if using embedded)
• VMware vCenter Server Platform Services Controller VM (if using embedded)
• VMware vRealize Log Insight VM (if using VxRail deployed syslog)
• SRS VE VM (if call home is enabled and using VxRail Manager deployed SRS)
1. [ ] Go to Hosts and Clusters > VxRail-Cluster.
2. [ ] Remove the snapshot of a service VM. Right-click VxRail Manager > Snapshots > Manage
Snapshots.
3. [ ] Select the snapshot and click Delete.
4. [ ] Click OK to remove the snapshot.
5. [ ] Repeat Steps 2 through 4 for each of the remaining service VMs.

Task 6: Mystic Service account password change (Optional)

Starting with release of v4.7.x, you can change the Mystic Service account.

1. [ ] Log in to vCenter Server using HTML5 Client with administrative privileges.

2. [ ] Go to Home > Host and Clusters > VxRail-Cluster > VxRail Manager.
3. [ ] Under the Summary tab, click Launch Web Console.
4. [ ] Log in with root username and password.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 15 of 19
 Figure 17 VxRail Manager console.

5. [ ] Run the following command to update the password:

# passwd mystic
6. [ ] Enter the new password as prompted.
7. [ ] Type logout to log out of VxRail Manager console.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 16 of 19
Appendix A: Get the correct controller key value
1. [ ] Connect to https://<vCenterServer IP Address>/mob and log in using administrator@domain
credentials.

Figure 18 Log in to vCenter.

2. [ ] Click content to access the MOB properties table. The MOB properties table has three columns
labeled NAME, TYPE, and VALUE.

Figure 19 MOB properties table.

3. [ ] Locate property with NAME rootFolder and TYPE ManagedObjectReference:Folder and click on
the link for (Datacenters) VALUE.

Figure 20 MOB root folder for data center

4. [ ] Locate the property with NAME ChildEntity and TYPE ManagedObjectReference:ManagedEntity[].

Click on the link for the VxRail-Datacenter VALUE.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 17 of 19
 Figure 21

5. [ ] Locate the property with NAME datastore and TYPE ManagedObjectReference:Datastore[] for the
VxRail-Virtual-SAN-datastore-uuid VALUE.

Figure 22 Sample MoRef ID for vSAN Database

6. [ ] Click on datastore-morefid link just to the left of VxRail-Virtual-SAN-datastore-uuid.

7. [ ] Locate the property with NAME vm and TYPE ManagedObject Reference:VirtualMachine[].
Record the MoRef ID value of the VxRail Manager entity labeled VxRail Manager.

Figure 23 Sample MoRef ID for VxRail Manager

8. [ ] Click on VxRail Manager-morefid link just to the left of VxRail Manager.

9. [ ] Locate the property with NAME config and TYPE VirtualMachineConfigInfo. Click on the link for the
config VALUE.

Figure 24 Sample MoRef ID for config

10. [ ] Locate the property with NAME hardware and TYPE VirtualHardware. Click on the link for the
hardware VALUE.

Figure 25 Sample MoRef ID for hardware

11. [ ] Locate the property with NAME device and TYPE VirtualDevice[]. Record the value of
VirtualLsiLogicController.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 18 of 19
 Figure 26 VirtualLsiLogicController value.

Troubleshooting
If vCenter upgrade fails, you can revert vCenter and PSC. See KB 000157660 for more details and
instructions.

Dell Technologies Confidential Information- for internal use only version: 4.8.1.6

Page 19 of 19