SN Category

Pre
1 Access Permission

2 Remote Access

3 Opening Email

3.a PSA Task

4 Check CPG config in advance

4.a CPG

5 Logs Capture

6 System Check

7 Validate previous activities are

completed

7.a HW Rebalance

8 OS

9 OS

10 DO License
11 Dedup

11.A Dedup and compression

 Dedup and compression
11.B
*Performance checks*

12 Dedup

12.a Dedup (TDVV3)

13 Free Space Availability

14 AO
15 Active Tasks

16 Drive Readiness

17 Node Pair Addition

18.a PD Health

18.b Health Check

18.c LD, Chunklets Check

18.d CPU Utilization

18.e VLUN Utilization

18.f PD Utilization

18.e Verify if delack is increasing in past week

18.h Cache Memory Utilization

19 Compactcpg on Fully provisioned VV's

20 Flash cache check

21 CPG & VV's

Exe
Note :- Only one instance

1 Dry run on entire system

1.a Dry run on a CPG Level

2 Tunesys execution

3 Task Validation

4 Update Customer /Requestor and Case

Owner

Pos
 1 Task Completion status

2 Validation of Task Completion

2.a

Id CagePos Type State Total OK Fail Free Uninit Unavail Fail OK

0 0:00:00 FC normal 819 726 0 41 0 0 0 0

3 Dedup

4 Resume Tasks
5 Uninstall Scripts

Update Customer /Requestor and Case

Owner
6

Addition

1 Alternate Commands

Cancel T
1 Cancel Tunesys
 Tunesys / Data Rebalancing
Pre Checks
Validate if we have permission from the customer to execute tunesys. If not please ask for permission. CHECK SSI.

Check if remote access is available via RDA/NGDC and confirm with the customer if this can be performed remotely or v

Send an opening email in pre-defined standard format

Create a Task for tunesys in PSA and update it on day to day basis. This can be closed post completion of final data rebal

Check if CPG Parameter modification is a PRE-REQ in CSR Technical Write up. Perform analysis from Sales upgrade, show
the activity.

Validate if previously on any trip / phase if CPG parameters were changed to meet the requirement for a data movemen

Ensure to save session logs via putty

Validate the Storeserv s/n, to make sure we are working on correct system. CLI # showsys -d

Validate all the trips / phases are completed by checking the HW configuration from Config file or from the system again
Also please check if there was a revised config approval if any.

Ensure that HW is evenly balanced across the node pairt such as having same no of drives / cages behind each node pair
data rebalancing phase.

If the HPE 3PAR OS is 3.1.1.GA

DO NOT RUN TUNESYS IT WILL CAUSE AN OUTAGE

If the HPE 3PAR OS is 3.2.2 and above, tunesys starts automatically after HW Upgrade (post admithw). Please verify if th
CLI#showtask -active or CLI#showtask.

For any version of 3.1.1 OS Family, tunesys requires DO License. Please verify using command: CLI# showlicense
From HPE OS 3.1.2 onwards, tunesys does not require DO License.
Note: DO NOT RUN TUNEVV ON DEDUP VOLUMES IF THERE ARE GREATER THAN 16 TDVV IN A CPG ON SYSTEM BELOW
OUTAGE.

On DEDUP ENABLED ARRAY, install below scripts as per below simplified table pruor to tunesys execution

Ex to identify dedup volumes from CPG:- ----- Inserv CLI # showcpg -----
--------------------(MB)---------------------
----Volumes---- -Usage- ------ Usr ------ ---- Snp ---- ---- Adm ----
Id Name Domain Warn% VVs TPVVs TDVVs Usr Snp Total Used Total Used Total Used

1 A13PAR01_CPG_SNAP_03 - - 41 35 5 7 40 6026240 6026240 166912 128000 49152 40192

2 A23PAR01_CPG_VV_01_closed - - 14 2 11 12 2 512640 512640 64896 2048 57344 40192
FIX FOR THIS IS AVAILABLE IN 331MU1 AND 322 MU6 (FUTURE CODE)

Validate if the system has dedup and compression enabled by running the command # showvv -s
Validate version of TDVV by running the command # showcpg -d

If the HPE 3PAR OS is below 3.2.2x (7K/ 8K/9K/20K system)

For Tunesys/Tuneld /compactpcg tasks initiate the task without disabling the Dedup or Compression GC.
# tunenodech approach is recommended
For Tunevv operation on a DECO volumes, Dedup and Compression GC needs to be disabled.
If the HPE 3PAR OS is above 3.3.1x ( 8K/9K/20K system)
Ensure to disable dedup and compression GC for any tuning operation (version 3 TDVV)
NOTE: 1. Please engage ESCALS's if tunevv operation needs to be performed on subjected VV's
2. Dedup GC and Compression GC can be disabled only for 8-10 hrs. Ensure to set REMINDERS to enable GC, else
3. For Version 2 TDVV's, tunesys task can be run without disabling compression and dedup. For Tunevv operation
As a best practice, we need to run tunes on DECO, TDVV, Compression Volumes during the Low IO Period (8-10 hour win
NOTE: Communicate to customer not to run IO intense activity such as backup etc.

Validate the historic data.

Example:
1. cli # srstatcpu -btsecs 2019-09-23 -hourly
Time Secs User% Sys% Idle% Intr/s CtxtSw/s
2019-09-23 00:00:00 CDT 1569214800 0.9 1.2 97.9 91587.4 81181.4
2019-09-23 01:00:00 CDT 1569218400 0.9 1.1 88.0 91754.6 82173.3
2019-09-23 02:00:00 CDT 1569222000 0.9 1.1 76.0 94815.3 91263.6
2019-09-23 03:00:00 CDT 1569225600 1.0 2.3 98.7 140549.5 138923.3
2019-09-23 04:00:00 CDT 1569229200 0.9 7.8 91.2 185524.6 144532.3
2. cli # srstatcmp -btsecs 2019-09-23 -hourly
--------------------count/sec-------------------- --hit%--- ------dack/sec-
Time Secs rhit whit r w r+w lockblk r w NL FC SSD150
2019-09-23 00:00:00 CDT 1569214800 40713.2 823.3 48947.1 8088.3 57035.3 0.1 83.2 10.2 0.0 0.0 0.0
2019-09-23 01:00:00 CDT 1569218400 33720.5 842.6 40305.1 9110.7 49415.7 0.1 83.7 9.2 0.0 0.0 0.0
2019-09-23 02:00:00 CDT 1569222000 27350.1 1868.9 32051.7 8308.9 40360.6 0.0 85.3 22.5 0.0 0.0 0.0
2019-09-23 03:00:00 CDT 1569225600 44743.2 1177.2 53419.1 11846.6 65265.8 0.0 83.8 9.9 0.0 0.0 0.0
2019-09-23 04:00:00 CDT 1569229200 105840.9 1500.5 120263.5 21186.3 141449.8 0.0 88.0 7.1 0.0 0.0 0.0
3.cli # srstatcmp -btsecs 2019-09-23 -hourly
---------IO/s--------- ----------KBytes/s---------- ---Svct ms---- ---IOSz
Time Secs Rd Wr Tot Rd Wr Tot Rd Wr Tot Rd
2019-09-23 00:00:00 CDT 1569214800 4785.0 2675.4 7460.4 748620.5 97247.4 845867.9 0.99 0.47 0.81 156.5
2019-09-23 01:00:00 CDT 1569218400 6301.4 2467.6 8769.0 598485.7 116553.4 715039.1 0.76 1.07 0.85 95.0
2019-09-23 02:00:00 CDT 1569222000 4781.2 3619.3 8400.5 477149.4 83964.6 561114.0 1.07 0.47 0.81 99.8
2019-09-23 03:00:00 CDT 1569225600 9503.0 4101.5 13604.5 748294.6 102011.7 850306.4 0.69 0.48 0.62 78.7
2019-09-23 04:00:00 CDT 1569229200 15547.6 5472.8 21020.4 1582068.2 266287.9 1848356.1 0.98 0.78 0.93 101.8

If the HPE 3PAR OS is 3.2.1 GA / 3.2.1 MU1 / 3.2.1 MU2

DO NOT RUN TUNESYS ON DEDUP ENABLED ARRAY AND IT MAY CAUSE AN OUTAGE. RECOMMENDED OS IS 3.2.1EMU3

If the HPE 3PAR OS is 331GA or EGA

DO NOT EXECUTE TUNEVV / TUNESYS / TUNELD on TDDV3 / DDS VOLUME AS THE NEW SA LD'S CREATED AS DESTINATIO
WHICH MEANS OTHER VV'S CAN USE THIS SPACE AND MAY CORRUPT THE INTERNAL STRUCTURE OF TDVV3 SA SPACE LD

For TDVV3 (3.3.1.x onwards), #showcpg -d <CPG name> will display the TDVV version, sample output below. Please ensu
TDVV2, you may not have the version # displayed

Check the free space available in the array using command showspace. Tunesys task requires certain amount of free spa
total free space in each tier. The probability of tunesys failing is high when the array is over 85% utilized. The array over
AO should be suspended before Data Movement Activity, take permission from customer before execution.
Please check if the Adaptive Optimization is licensed and configured by Issuing CLI # showaocfg and
If AO Policies are defined (showaocfg Output), suspend all scheduled tasks using command
CLI# setsched -suspend_all
Note:- Any scheduled tasks that involves region moves has to be suspended [mandatory].
Issue command CLI # showtask -active and wait for any running task to complete including AO Task.
It is OK to have remote copy sync running as an active task.
Please execute tunesys only after ensuring chunklet initialization is complete on newly added drives. Issue command
CLI# showpd -c to verify (Un-initialize chunklet count should be 0)

If the activity includes addition of node pair + cages + PD's behind the new node pair only, Please check the free space o
or above, we need to call out to the customer that once node pair is added, a  temporary CPG should be created behind
space behind existing node pair prior to tunesys execution.
Command to check the free space behind node pair Tier wise:   cli# showpd -c -p -devtype <FC/NL/SSD> -nd 0,1 
Caution!!! Failing to check this may lead to an array outage
Template: "Dear Customer - Since the current (Array or Tier) utilization is 90% and above post HW upgrade please create
some VV's to free up space behind existing node pair.
Reason: If one of the node pair runs out of space, new LD creation may not be possible and CPG may go to a failed state

Ensure that all drives are in normal state. Data evacuation should not be performed when the drives are in degraded / f
verify.
CLI# showpd -failed -degraded ==> None to be listed as failed or degraded
CLI# showpd -s ==> There are two status, The main state and detailed state and both should be normal.
Sample Output below with normal status
----- showpd -s -----
Id CagePos Type -State- -Detailed_State-
0 0:0:0 FC normal normal
1 0:0:1 FC normal normal

Sample Output below that requires attention

----- showpd -s -----
Id CagePos Type -State- -Detailed_State-
0 0:0:0 FC normal Servicing
1 0:0:1 FC normal Servicing ==> If the status is servicing, engage break fix team to fix this issue.

Perform Healthcheck on the array and ensure that array is healthy with no show stoppers prior to execution of tunesys
(Send details to swd-3par-gss-ssc@hpe.com )

If healthcheck reports LD's are degraded Ex:- From ha cage to ha mag or LD's chunklets on a remote disk, this may get re
available free space.
Please reach out to Escal team for further clarification.

CPU Idle must be above 50%

CLI# statcpu -t -d 30 -iter 1
Sample Output below
node,cpu user sys idle intr/s ctxt/s
0 total 1 4 96 25577 29308
1 total 2 4 95 26057 29661
2 total 0 3 97 24064 26481
3 total 0 3 97 24805 28164
Check the average service time of VLUN and it should not be over 25ms.
CLI# statvlun
Sample Output below
cli% statvlun -ni -iter 1
13:21:17 05/11/2016 r/w I/O per second KBytes per sec Svt ms IO
Lun VVname Host Port Cur Avg Max Cur Avg Max Cur Avg Cur

23 3PAR-Vmware-Test-LUN23 3par-esx10 0:1:1 t 1466 1466 1466 93544 93544 93544 5.75 5.75 63.8
24 3PAR-Vmware-Test-LUN24 3par-esx10 1:1:2 t 1435 1435 1435 91461 91461 91461 5.46 5.46 63.7

If the service time is over 25ms, this is an indication of host latency and tunesys should not be initiated.

Check the performance of the PD prior to the execution of DO. Verify by issuing the below command
CLI# statpd -devinfo -iter 1
Sample Output below
13:14:52 05/11/2016 r/w I/O per second KBytes per sec Svt ms IOSz KB Idle %
ID Port Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen Cur Avg
0 1:0:1 t 84 84 84 7366 7366 7366 7.73 7.73 87.9 87.9 1 60 60 ==> FC
1 0:0:1 t 47 47 47 1690 1690 1690 5.22 5.22 3 6.2 36.2 0 85 85 ==> FC
- NL PDs maximum IOPs is 75 @ 8K reads
- FC 10K PDs maximum IOPs is 150 @ 8K reads
- FC 15K PDs maximum IOPs is 200 @ 8K reads

If the IOPS is more than the above mentioned threshold, please check with Escal and notify to the customer that the arr
read/write performance.
NOTE:- Please ensure to capture session logs

Using historical data on STaTS, verify data from 7 days perform.db file whether delack counter has increased. This would
assist customer reach support to address the performance issue.

Check the historical performance data reports for cache memeory.

CLI# statcmp
Run this command with an interval of 3-5 mins and compare the DelAck output under p
if DelAck number is increasing. Contact CSR Team/Escal.
Sample Output below

Page Statistics
------------CfcDirty------------ --------------CfcMax--------------- ---------
Node FC NL SSD_150KRPM SSD_100KRPM FC NL SSD_150KRPM SSD_100KRPM FC N
0 962 8 0 4969 110400 7200 0 81600 0 162960
1 2126 26 0 5252 110400 7200 0 81600 0 175501
2 1153 104 0 6386 110400 7200 0 81600 0 208356
3 897 14 0 6182 110400 7200 0 81600 0 192514
Running compactcpg with >=8TB fully provisioned VV can result in sysmgr hang.
This issue can be experienced when a compactcpg is initiated for a CPG which holds a fully provisioned VV sized >= 8TB

Due to an issue in code (variable overflow) sysmgr will start spinning in an infinite loop with the lock still outstanding. As a resu
makes the array un-manageable.

Fix: Patch P92 (322 EMU4) | 322 MU6 | 331 MU1 onwards

Command to check: cli# showvv -cpg <CPG NAME>

Under provision type it will be listed as Full, if the FPVV size is more than 8TB, please hold free space reclamation until P92 for

Check if the system has Flash cache and if SSD's were added recently.
If yes, please inform customer to take a maintenance window to delete the existing Flash cache configuration and recre
tunesys cannot tune flash cacheLD's. If customer is not aware of this procedure, engage breakfix team by calling 1800 63
customer & support team and get them in conference. And this has to be documented in an e-mail which includes the cu
Ex to identify flash cache LD's using CLI # showflashcache

Issue command CLI#showcpg to identify the number of CPG's and their virtual volumes. CLI# Showcpg -sdg com
CPG configuration.
Execution
Note :- Only one instance of tunesys can be run at a time.
Perform a dry run on the array to assess the no of tunes suggested.
Issue Command: CLI # tunesys -dr

Note:- If dry run does not suggest any tunes, inform customer that we will do the analysis and execute suitable tune com
*** Keep customer informed ***

Perform a dry run on a particular CPG to assess the no of tunes suggested.

Issue Command: CLI # tunesys -dr -cpg <CPG NAME>

Note:- If dry run does not suggest any tunes, inform customer that we will do the analysis and execute suitable tune com
*** Keep customer informed ***

Depending on the context above, execute tunesys

CLI # tunesys
CLI # tunesys -cpg <CPG NAME> (Use this command only when you want to run tune on a specific CPG.)

Tunesys runs as an active task and can be verified using below commands.
CLI # showtask -active
CLI # showtask -d <tunesys task ID#>
Sample Output below:
cli% showtask -active
Id Type Name Status Phase Step -------StartTime-------- -FinishTime- -Priority- -Use
26006 system_tuning tunesys active 2/3 102400/15126528 2016-05-10 10:36:59 AEST - n/a 3par
26007 background_command tunenodech active 2/2 100/7402 2016-05-10 10:42:23 AEST - n/a 3par

Share the tunesys task ID and inform that we are monitoring this task remotely and add the below note to the e-mail th

Please share the overall PD utilization of the array to compare pre and post tunesys changes with an existing template

Note: If the system is licensed and configured for Remote Copy and when the tunesys task is active and if a fail over occu
CLI# canceltask <TASK ID> (for more details refer cancel tunesys task sectio

Post Checks
 Ensure that tunesys is completed by verifying the task details especially, the no of tunes performed and no of tunes succ
CLI # showtask -active (Tunesys task should not be listed if its already completed)
CLI # showtask (Make a note of the tunesys task ID and ensure its in done status)
CLI # showtask -d
Copy showpd -c output to an Excel sheet and covert this data from text to columns. Add a new Column (PD Utilization) a
Chunklets * 100
Filter the Drive type along with different Capacity and check the PD Utilization percentage. It should not be more than a
course of action.
al OK Fail Free Uninit Unavail Fail OK Fail Free Uninit Fail PD Utiliz
9 726 0 41 0 0 0 0 0 52 0 0 88.6446

If the HPE 3PAR OS is 3.2.1 and above version that has dedup enabled, we have to run a Dedup Garbage Collection scrip
Dedup Script and Procedure
Ex to identify dedup volumes from CPG:- ----- Inserv CLI # showcpg -----
--------------------(MB)---------------------
----Volumes---- -Usage- ------ Usr ------ ---- Snp ---- ---- Adm ----
Id Name Domain Warn% VVs TPVVs TDVVs Usr Snp Total Used Total Used Total Used

1 A13PAR01_CPG_SNAP_03 - - 41 35 5 7 40 6026240 6026240 166912 128000 49152 40192

2 A23PAR01_CPG_VV_01_closed - - 14 2 11 12 2 512640 512640 64896 2048 57344 40192

Post completion of the tunesys activity, resume all scheduled tasks. CLI#showsched -resume_all
Uninstall (relevant scripts) and remove all scripts post execution of the activity
Inform the customer / HPE whoever is the point of contact that this CSR is completed and we will go ahead and close the
Team PDL 3par.sps.csr@hpe.com

Additional Commands

If tunesys does not balance the data even after task completion, the alternate option is to run the below command prov
If the dry run is suggesting more tunes, safely proceed and execute the below alternate command. Consult CSR Team / E
CLI # tunesys –nodepct 1 – chunkpct 1 (All OS versions)
CLI # tunesys -tunenodech –chunkpct 1 (From 3.2.1 and above OS)
CLI # tunenodech -node <node ID> (From 3.2.1 and above OS)
CLI # tunesys -tunenodech -devtype <FC/SSD/NL> (From 3.2.2 and above)
CLI # tunenodech -devtype FC -node 0 (From 3.2.1 and above).
Tunenodech has good success rate if run individually on nodes than if ran within th

Cancel Tunesys Task

Identify the tunesys task ID by using the command CLI# showtask -active
Use CLI# canceltask <Task ID> to cancel tunesys task.

NOTE:- Post task cancellation verify status using the command CLI# showtask -active and if there are any activ
subtasks initiated by tunesys (maintask).
Sample Below.
root@1111111-1 Fri May 1 10:27:22:/common/support# showtask -active
Id Type Name Status Phase Step -------StartTime-------- -FinishT
26006 system_tuning tunesys active 2/3 2149376/15126528 2016-05-10 10:36:59 AEST -
26209 move_regions tuneld: tp-1-sd-0.86 active 1/1 933/1024 2016-05-12 10:11:42 AEST -
root@1111111-1Thu May 1 10:42:02:/common/support# canceltask 26006
Are you sure you want to cancel task 26006?
select q=quit y=yes n=no: y
root@1111111-1 Thu May 1 10:42:27:/common/support# showtask -active
Id Type Name Status Phase Step -------StartTime-------- -FinishTime- -Pr

26208 background_command tuneld active 1/1 0/1 2016-05-12 10:11:39 AEST - n/a
26209 move_regions tuneld: tp-1-sd-0.86 active 1/1 966/1024 2016-05-12 10:11:42 AEST - n/a
root@1111111-1 Thu May 12 10:44:46:/common/support# showtask -d 26006
Id Type Name Status Phase Step -------StartTime-------- -------FinishTime------- -Priority-
26006 system_tuning tunesys cancelled --- --- 2016-05-10 10:36:59 AEST 2016-05-12 10:43:29 AEST n/a

Note:- Initiate Insplore for analysis *** Keep Customer Informed ***
cklist
Links

Tunesys Opening Template

CSR Sharepoint

CSR Sharepoint
Run the script disable_dedup_GC to disable/enable the dedup GC using
U21 kit.

Run the below commands for disable/enable compr GC:

To disable compr GC:
tcli -e "kvar set -n disable_compr_gc -v 1"

restore compr GC:

tcli -e "kvar set -n disable_compr_gc -v 0"
Validate the idle%,delack,IOPS,Svct and suggest the best time frame to
initiate the tunes based on observations

Advisory

https://www.storefrontremote.com
Dedup Script & Procedure Download
Procedure to capture logs
 Task status

NA

NA

NA

NA
NA
NA

NA
NA

NA

NA

NA

NA
NA

NA
NA

NA

NA

NA

NA
NA

NA

NA

NA

NA
NA
NA
Comments for Pending and NA