Nutanix Field Advisory #96

Release Date: 19 August 2021

Last Update: 30 August 2021

Title:
Nutanix clusters may experience instability and a rare data inconsistency issue during an upgrade to AOS
6.0.1

Products Affected:
AOS 6.0.1

Severity:
High

Description:
Nutanix has identified an issue where clusters may experience instability and potentially data inconsistency on
rare occasions after upgrading to AOS 6.0.1.

Identification:
To determine whether your AOS 6.0.1 cluster is impacted, please use the following steps.

1. Check the AOS version of the cluster:

a. Log on to Prism Element

b. Click on the username (on the top right) > About Nutanix

Tel: 1 (855) 688-2649 Ext 3

©2021 Nutanix, Inc. All Rights Reserved.
 Alternatively, refer to KB-1005 for further details on how to check the AOS version.

2. For AOS 6.0.1 clusters, to determine if the cluster is impacted, log in to any CVM on the cluster and run
the following commands:

a. nutanix@cvm:~$ allssh "zgrep -E '^F2021.*Found

nullified|^F2021.*orig_merged_vblock' /home/nutanix/data/logs/stargate*FATAL*"

Example Output:
================== XXXXXXXXXX =================
Log file created at: XXXXXXXX
Running on machine: cvm
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu[Z] threadid file:line] msg
F20210623 14:53:59.004729Z 4579 vdisk_micro_merged_vblock_fixer_op.cc:504]
QFATAL NFF Check failed: orig_merged_vblock vdisk_id=63740138
operation_id=16559 shard_idx=-1 block1=8787 num=1 8787

b. nutanix@cvm:~$ allssh "zgrep -E '^F2021.*nullified vblock'

/home/nutanix/data/logs/stargate*FATAL*"

Example Output:
================== XXXXXXXXXX =================
Log file created at: XXXXXXXX
Running on machine: cvm
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu[Z] threadid file:line] msg
F20210623 01:00:24.658257Z 1144 vdisk_micro_read_op.cc:659] vdisk_id=63740552
operation_id=5074 shard_idx=-1 Found nullified vblock entry for vblock
...Truncated...

Tel: 1 (855) 688-2649 Ext 3

©2021 Nutanix, Inc. All Rights Reserved.
 c. nutanix@cvm:~$ allssh "zgrep -E '^F2021.*Attempting to copy nullified vblock'
/home/nutanix/data/logs/curator.*"

Example Output:
================== XXXXXXXXXX =================
Log file created at: XXXXXXXX
Running on machine: cvm
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu[Z] threadid file:line] msg
F20210623 14:39:58.190263Z 5461 vdisk_block_number_reduce_task.cc:2183]
Attempting to copy nullified vblock entry for block 671 from vdisk 9540 into
descendant vdisk 9564 donor_block_info.incomplete_block=0

3. If any of the commands return an output as highlighted above, and the cluster is running AOS 6.0.1, the
cluster is impacted. Clusters running 6.0.1 that do not observe the output highlighted above are not
impacted.

Workaround:
Customers that are impacted should immediately contact Nutanix Support at https://portal.nutanix.com for
assistance and reference KB-11924.

Resolution:
AOS 6.0.1.1 (STS) or later contains the fix for this issue.

Revision History:
Revision Date Comments
1.0 19-Aug-2021 Initial release
2.0 30-Aug-2021 Final release

If you have questions, please open a case with Nutanix Support at https://portal.nutanix.com, or by calling
Support at the phone numbers on the website https://www.nutanix.com/support.

Thank you for being a Nutanix customer.

Nutanix Worldwide Support

Tel: 1 (855) 688-2649 Ext 3

©2021 Nutanix, Inc. All Rights Reserved.