2023/5/26 21:48 VNX/CLARiiON: How to gather the necessary information for a VNX performance analysis (User Correctable) | Dell

mance analysis (User Correctable) | Dell Singapore

Article Number: 000040959 📠 Print

VNX/CLARiiON: How to gather the necessary information for a VNX

performance analysis (User Correctable)
Summary: Configuration and log files that must be obtained for a Performance Analysis on all types of
CLARiiON and VNX Block arrays.

Audience Level: Partners

Article Content

Symptoms

Response times for host applications have increased or are unacceptably high.

Once the problem period has been established, obtain the basic information diagnostic files that must cover the period when the problems occur:

A detailed problem description about which Servers, or LUNs, had performance problems and when these occurred.

SPCollects for both Storage Processors, which were run after the problem occurred (so that their logs would include the time of the incident). If Mirrorview is in use, then the

same data will need to be collected from both arrays and which array is the Source and which is the Destination array.

Unisphere (or Navisphere) Analyzer NAR/NAZ files, which cover the whole period when the problem was occurring. NAR/NAZ files will need to be collected from both arrays

if Mirrorview is configured.

FASTdata for VNX2 arrays, KBA 476626, How to gather Pool skew data for EMC Support.

Host Grabs and Switch logs may also be required.

Poor read and write performance and high response times for connected hosts.

Performance problems on a VNX or CLARiiON array, which requires performance analysis.

Performance issues when using Mirrorview

Gathering Navisphere Analyzer .NAR files and configuration information for a CLARiiON or VNX performance case.

Information and data files that must be obtained for a Performance Analysis on all types of CLARiiON arrays.

How to gather the necessary information for a CLARiiON or VNX performance analysis.

What logs do I collect for CLARiiON or VNX performance issues?

For Unity, refer to KBA 491175, How to gather the necessary information for a Unity performance analysis

Cause

Performance Service Requests (SR) will not be fully investigated until the SPCollects and NAR/NAZ files and a detailed description of the symptoms has been provided. The

SPCollects have the configuration, hardware, and fault information and the NAR/NAZ files have the performance data for the system array. The details of the performance issue

symptoms is needed so that the engineer will know which period needs to be analyzed in detail and the LUNs that had the noticeable performance impact. Without this detail,

only the general performance statistics will be analyzed to look at issues affecting all LUNs. When Mirrorview is configured, spcollects and NAR/NAZ files from both arrays will be

needed. When Mirrorview is used, the destination array can be the cause for performance issues and will need to examined.

Dell EMC Support Services investigate performance cases to identify issues that are affecting customer sites. Performance tuning (to achieve peak performance for a host or

application) and reconfiguration is a task for a Storage Administrator or a Solutions Architect so this will not be handled in the Service Request. Performance issues can have a

number of causes including hardware faults, forced flushing (caused by write cache being 100% full), head contention (several busy LUNs sharing the same drives), the

storage array being at its maximum utilization, Mirrorview configuration issues or network issues.

Service requests that do not involve an actual problem, such as requesting a report of the current level of workload on an array, are outside of the charter of Support. For self-help or

those needing help to determine the level of workload on an array can use the LiveOptics service. Please see the following article for detailed information about the service:

KB 515691 Dell EMC Unity/VNX KB: How to use LiveOptics for a general health-check, storage assessment, and/or host/environment
sizing (User Correctable)

https://www.dell.com/support/kbdoc/en-sg/article/lkbprint?ArticleNumber=000040959&AccessLevel=40&Lang=en 1/6
2023/5/26 21:48 VNX/CLARiiON: How to gather the necessary information for a VNX performance analysis (User Correctable) | Dell Singapore

Resolution

For a video demonstration, refer the attachments section.

Problem Description

For each event, Technical Support needs, as a minimum, to know the following in the description of the performance problem:

The time and date (with the time zone GMT/UTC difference)

The server names or CLARiiON Storage Groups that own the LUNs

The LUNs that the affected application was using at that time

Is it reading or writing performance that is suffering?

The detailed description is critical to finding the issues seen by the hosts and without this only a general analysis of the performance data can be done, to look for issues affecting

all LUNs.

SPCollects

The output files from running SPCollect on both SPs, will be necessary to show the detailed array configuration and any events which occurred on the array which may have had a

performance impact. If the array is running VNX of 05.32.000.5.206 or above, then also uploading the archived ktraces would be very useful for diagnosing performance issues,

which are no longer in occurring.

To gather SPCollects, see Unisphere Help article "Generating and transferring SP log files" or:

See Dell EMC KBA 332841, Gathering SPCollect files on a VNX Series array, for instructions on how to obtain SPCollects on a VNX. FASTdata should also be included for

cases that are escalated to Level 2 Support or Engineering, KBA 476626, How to gather Pool skew data for EMC Support.

See Dell EMC KBA 334497, How to gather CLARiiON SPCollects using Unisphere, for instructions on how to obtain SPCollects on a CLARiiON running Unisphere.

When file based storage is part of the performance issue, the configuration and log files from the Data Movers would be needed too. Refer to KBA 466642, Celerra: Important

information required to troubleshoot EMC NAS Service Requests.

NAR/NAZ files

NAR and NAZ files contain the same data, but only NAR files can be opened in Navisphere/Unisphere Analyzer. If the CLARiiON has an Analyzer license ('-NavisphereAnalyzer' or '-

UnisphereAnalyzer' in the software list), then NAR files will be produced; otherwise data logging produces NAZ files. NAZ files are for use only by Dell EMC support to assist with

break/fix performance diagnostics.

Analyzer .NAR or .NAZ files, should be gathered, covering the period when the issue occurred. If the problem occurs throughout the day, include NAR/NAZ files, supply a total period

of approximately 24 hours (for a day when the issues occurred). Uploading more NAR files than are needed can allow support to get more background information to an issue and

see if it has been going on for some time, but usually a performance analysis will concentrate on a single NAR/NAZ file containing the most relevant data.

To gather the performance data it is first necessary to access the Analyzer menu and the method for this differs depending on the version of Unisphere or Navisphere:

For Navisphere (CLARiiONs running Release 29 or lower), select the Tools drop-down menu (at the top of the Navisphere Window), then select Analyzer. A submenu will then

appear with the Navisphere Analyzer options.

For Unisphere (CLARiiON CX4 running Release 30), go to the System List view and select the CLARiiON you wish to monitor. Highlight the Monitoring Tab and select

Analyzer, which will bring up a page with all of the available Analyzer options (which will depend on whether the Analyzer enabler is installed).

For VNX Unisphere, after selecting the VNX system, click on the System Tab. Click on Monitoring and Alerts, followed by Statistics for Block. This will bring up a page with all

of the available Analyzer options (which will depend on whether the Analyzer enabler is installed). In other words, the folders under the taskbar at the top should be listed as:

This_VNX > System > Monitoring and Alerts > Statistics for Block.

https://www.dell.com/support/kbdoc/en-sg/article/lkbprint?ArticleNumber=000040959&AccessLevel=40&Lang=en 2/6
2023/5/26 21:48 VNX/CLARiiON: How to gather the necessary information for a VNX performance analysis (User Correctable) | Dell Singapore

Before starting the data logging, the archive interval should be set appropriately:

Click on the 'Performance Data Logging' option, in the Analyzer menu (see above) to check the current settings or to change them.

Before making any changes it is best to stop the data logging first (if it is already running), which will close the current file.

Check that Periodic Archiving is enabled, and if not enable it. This is critical to not delete previously archived NAR/NAZ files.

A typical NAR/NAZ file will contain 155 polling intervals of data. Therefore, if the archive polling interval was 120 seconds, there would be 5 hours and 10 minutes of data in

each file. Once each file is full, it will be closed and a new one started. Data Logging must run for at least 11 data polling intervals to create a new file. If you stop Data

Logging before the 11 intervals have passed and the NAR file will contain data from the last, complete NAR file. These can later be merged together, by Dell EMC support if

needed, in order to produce graphs covering a longer period, but only upload the original files to EMC Online Support for a performance analysis, i.e. not any altered or merged

files.

The archive interval is set to 300 seconds by default. Having a higher polling interval can reduce the impact of data logging on the SP Utilization for CX4 and older arrays, but

will reduce the amount of detail in each NAR file, see , KBA 330218, How to reduce High Storage Processor Utilization or Slow Unisphere response time

On VNX, SP Utilization levels are less likely to be an issue and too high a polling interval can miss details about heavily loaded drives in a Pool. Therefore with the VNX, it is

recommended to have the archive interval set at 60 seconds for testing and no more than 300 seconds for regular monitoring.

It is usually recommended to also set the real-time interval to the same value as the archive interval. The impact of Analyzer on the SP Utilization (which is typically around

1% of additional utilization) would be dictated by the lower of these two values.

The results recorded will be averages of the performance over archive intervals. Therefore if the archive polling interval was set at 600 seconds, performance problems

which only lasted a few minutes, would be hard to diagnose.

For FLARE release 22 and earlier, you will need to be in Engineering Mode to change the Archive Interval, KBA 330603, How do I use Navisphere or Unisphere UI to collect

NAR or NAZ files.

To change the archive interval using the CLI, use the following command:

naviseccli ~ analyzer -set -narinterval 120 (where ~ would be where the arguments -h <IP address>, -user <username> etc. may need to included)

To start data logging:

Click the 'Performance Data Logging' option, in the Navisphere Analyzer menu (see above). In the data logging window that opens, click the Start button.

Alternatively use the following CLI command:

naviseccli ~ analyzer -start (where ~ would include the SP IP address and the Navisphere login)

A NAR/NAZ file will not be available to download while it is still open for data logging. Therefore the list of NAR files will only include the files which have been completed

(normally happens when there are 155 polling intervals contained in them), not the file currently being logged .

To obtain the latest data, there is a 'Create New' button in the 'Retrieve Archive' window. Alternatively the data logging can be stopped and then started again. Either

option will close the current NAR/NAZ file and then start a new one (although you may need to wait for a minute or so for the current file to be closed).

The following CLI command will start a new NAR/NAZ file and make the latest NAR/NAZ file available to download:

naviseccli ~ analyzer -archive -new (where ~ would include the SP IP address and the Navisphere login)

To retrieve the completed archives (after they have had time to gather sufficient information and the performance issues were occurring during this period):

Click 'Retrieve Archive' from the Analyzer menu (also referred to as 'Statistics for Block').

Make sure the correct CLARiiON/VNX is selected in the top drop-down box.

https://www.dell.com/support/kbdoc/en-sg/article/lkbprint?ArticleNumber=000040959&AccessLevel=40&Lang=en 3/6
2023/5/26 21:48 VNX/CLARiiON: How to gather the necessary information for a VNX performance analysis (User Correctable) | Dell Singapore

Select SP A.

Select the relevant archives and click Retrieve.

Repeat for SP B (although usually the same data is present in the the NAR/NAZ files from either SP).

The path for saving the files can be changed, but the default path is set in the 'Customize Charts' window (using the Archives tab), on the Analyzer menu.

Alternatively use the following CLI command to retrieve all the archives from one SP: naviseccli -h <SP IP> analyzer -archive -all

If you have remote access to the SP, via EMC Remote or Remotely Anywhere, then the NAR/NAZ files can be transferred directly from C:\EMC\Archives folder. (Flare

release 24 and higher)

See KBA 330603, How do I use Navisphere or Unisphere UI to collect NAR or NAZ files, for further information on using Navisphere/Unisphere to collect NAR/NAZ files.

For Release 22 and earlier, see KBA 327561, How do I enable Navisphere Analyzer monitoring in FLARE Release 22 and earlier.

Do not use the 'naviseccli ~ analyzer -archiveretrieve' in Release 24 or higher. Instead use 'naviseccli ~ analyzer -archive <-all or -file filenames> - see KBA Link Error

366073, Multiple NAR or NAZ files having duplicate data and file sizes, when using the naviseccli archiveretrieve command

For complete listing and instructions of using the naviseccli Analyzer command on VNX refer to KBA 502186, Collect and Retrieve NAR Files From a File Only VNX Using naviseccli

The NAR/NAZ file title contains the date and time (in GMT) of the last data point logged. It is important to stress that the Navisphere files have to be for the period when the

problems were actually occurring.

Upload files to EMC Online Support in the same form that they were produced on the SP. Do not merge NAR files and do not decrypt NAZ files. Any changes to the files can

introduce errors, so it is important that the original files are supplied.

A guide to running Navisphere Analyzer can be found by going to 'Help' in Navisphere. Select 'Help topics', followed by 'Analyzing storage-system performance using

Analyzer'

A guide to using Naviseccli for retrieving NAR files can be found on EMC Online Support, titled: 'EMC Navisphere Analyzer Command Line Interface (CLI)'.

For creating a script on a host to automate the creation and collection of archive files, see KBA 330717, How do I automate collecting .NAR files

Host Grab

Grab or EMC Report output files may be needed, if the issue is unique to specific hosts.

Select this link to EMC Support which will search for the Grab utilities for different OS types.

Download the utility and the README file. Follow the README file to run the utility for the host operating system chosen.

Alternatively, many of these grabs can also be downloaded from ftp://ftp.emc.com/pub/emcgrab/

Switch logs

These may be needed, especially if the performance problem only exists on certain paths. Please refer to the following Dell EMC Knowledgebase Articles for the model of the switch

being used:

Brocade: KBA 322041, What logs and when to collect them from a Brocade B-Series switch, during normal - and troubleshooting circumstances

Cisco: KBA 304027, How to collect logs from a Cisco MDS 9000 Family Switch using the CLI, or

KBA 321979, How to collect logs from a Cisco MDS 9000 Family Switch using the GUI

McDATA: KBA Link Error 326166, What logs can you collect from a standalone McDATA switch or

KBA Link Error 319780, What logs need to be collected when troubleshooting issues with Connectrix MP-1620M, MP-2640M or Nishan 3300 switches

For a video demonstration, please see the attachment below.

Additional Information
https://www.dell.com/support/kbdoc/en-sg/article/lkbprint?ArticleNumber=000040959&AccessLevel=40&Lang=en 4/6
2023/5/26 21:48 VNX/CLARiiON: How to gather the necessary information for a VNX performance analysis (User Correctable) | Dell Singapore

Analyzer Enabler

Release 24 and later stores the old archives but encrypts them, NAZ file, if the latest enabler is not present. Therefore, Release 24 does not require an enabler to gather performance

information. However, the NAZ files are only for use by Dell EMC Support and cannot be opened in Navisphere or Unisphere.

Arrays running any Release of FLARE from 14 to 22 cannot gather performance information unless a Navisphere Analyzer Enabler is loaded.

The enabler will appear as the following in Navisphere (right-clicking the CLARiiON array -> Properties -> Software):

-NavisphereAnalyzer
or
-UnisphereAnalyzer

For an intermittent problem in FLARE pre-Release 24, it is possible to script the archive retrieval to store the archives, see KBA 330717, How do I automate collecting .NAR files

If no recent NAR/NAZ files have been produced and the CLARiiON is running Release 24 or higher core software, then the data logging has not been started. To start Data Logging:

1. Go to the Analyzer menu bar in Navisphere and select Data Logging.

2. Check that Periodic Archiving is enabled, and if not enable it. This will create files every few hours, rather than putting all the data into a single file.
3. Click Start to begin the data logging. By Default this will run for 7 days before automatically stopping again. This is the "Stop Automatically After" parameter.

Alternatively, the performance logging can be prepared with the following naviseccli commands for both SP:

1. naviseccli -h <SP A IP> setstats -on

2. naviseccli -h <SP B IP> setstats -on
3. naviseccli -h <SP A IP> analyzer -set -default
4. naviseccli -h <SP A IP> analyzer -set -periodicarchiving 1

The logging can then be started on both SPs with the following command, and the logging will keep running for 7 days (unless it is manually stopped again):

naviseccli -h <SP A IP> analyzer -start

Dell EMC Remote Support has a break/fix character and we approach each performance SR from that perspective. If we find that the hardware and software is operating as

designed (health check) and we can find nothing broken, we will still take the next step and attempt to identify the cause. So while we can check over the logs to look for evidence of

a fault or inadvisable configuration, we do not produce detailed performance reports in these cases. If the cause is determined to be configuration related (such as spindle overload

or an under provisioned raid group), we will attempt to identify what the possible causes are, but making configuration recommendations or performance tuning is outside of the

scope of Dell EMC Remote Support's charter. In this case, we will contact your Site Support Managers to let them know that this particular issue may require additional assistance.

If you require help with resolving the configuration issue(s), then Consultants can be engaged through your Site Support Managers.

Note: NAZ files

To analyze performance using a NAZ file, it will need to be decrypted and converted to a NAR file, by Dell EMC Support. NAZ files are for use for support purposes only and Dell EMC

does not decrypt NAZ files for use by the customer. If the customer needs NAR files, rather than NAZ, then a Unisphere Analyzer license/enabler is required to be installed on the

storage array.

Partner Notes

For listing of performance issues and their associated KBAs see KBA 473729, How do I find information about Navisphere/Unisphere Analyzer or performance issues

For the archived ktraces, see KBA 304869, How to gather persistent ktrace logs stored on Storage Processor (SP)

Note: When recreating an issue for analysis, keep in mind the following:

Do keep note of the times that performance issues happened on the hosts, so that these can be cross referenced against logs and graphs from the array.

Do not reboot any SP or restart management servers, without first stopping data logging (and waiting at least 60 seconds). This will close the current files, so that no data is

lost when the processes are restarted.

Do run RBA traces, if possible (as described in KBA 470187, How to gather RBA traces for performance analysis by Engineering), but NAR files and SPCollects will still be

needed, which cover this same period. Archived ktrace and FASTdata files should also be gathered.

Do reset the switch port counters in advance, so that only relevant SAN events will be listed against the ports.

https://www.dell.com/support/kbdoc/en-sg/article/lkbprint?ArticleNumber=000040959&AccessLevel=40&Lang=en 5/6
2023/5/26 21:48 VNX/CLARiiON: How to gather the necessary information for a VNX performance analysis (User Correctable) | Dell Singapore

If the issue in Celerra or VNX-File related you must follow the below Knowledge Base articles;

KBA 335378, How to Triage VNX-File/Celerra NAS performance issues

KBA 336004, How to address/approach Unified File and Celerra Performance issues

KBA 465335, Online analysis of VNX-File/Celera NAS performance issues

Note: For procedures managing Performance SRs refer to KBA 500810, Managing Unified Storage Division (USD) Performance Analysis Service Requests

Note: Performance Support Request Form

Provide the performance request document to the customer, to gather details of the performance issue.

Please note this form can be filled out by Dell-EMC personnel during a phone conversation with the customer, if emailing/replying is not suitable.

All the symptom information listed above should be included in the form's performance problem description.

Note: Dell-EMC Engineering may also request RBA trace files. See KBA 470187, How to gather RBA traces for performance analysis by Engineering, for details on how to run these.

Attachments

🗏 Media_emc161922_csp_en_US_1.mp4

Article Properties

Affected Product
VNX2 Series

Product
Celerra NS, CLARiiON CX3 Ultrascale Series, CLARiiON CX4 Series, VNX1 Series, VNX2 Series, VNX5100, VNX5200, VNX5300,
VNX5400, VNX5500, VNX5600, VNX5700, VNX5800, VNX7500, VNX7600, VNX8000

Last Published Date

14 Mar 2021

Version
3

Article Type
Solution

https://www.dell.com/support/kbdoc/en-sg/article/lkbprint?ArticleNumber=000040959&AccessLevel=40&Lang=en 6/6