Article Number : 000082055

Dell EMC Unity/VNXe: How to clear replication hung tasks on

Unity/VNXe (Dell EMC Correctable)
Summary : This KB is to describe the procedure to clear the replication hung tasks.

Audience Level: Internal

Article Content

Symptoms

Unable to delete replication session due to hung tasks

Unable to delete replication session from GUI or CLI

Cause

The delete fails because replication session was in progress when destination was removed. We would need
to manually clean up this session otherwise the replication deletion or creation could be failed.

Resolution

ATTENTION: These commands require Root access which means this is a L2 procedure only.

Warning (read carefully):

If synchronous replications sessions are configured there's a high chance for this command to cause a
Memory Leak (panic), which could lead to an outage. Please refer to KB 000537958
Please see alternative in Restricted Notes section
Make sure you read TSA 113 - "repcli.x task" command may cause memory leak and SP panic
Verify running tasks by (on the primary SP only, otherwise, it will display blank):
/EMC/CEM/bin/clariiontool/repcli.x task list
For the task id that is running, look for Complete = No and note the ID, mover values

To abort the task:

/EMC/CEM/bin/clariiontool/repcli.x task abort <task_id> -mover <mover>
For a full syntax help, type:
/EMC/CEM/bin/clariiontool/repcli.x

Clean the hung tasks from DB:

Note:
1, Check if there are hung tasks/sessions on both Source and Destination system;
2, Check for all system VDMs and User VDMs;
3, The vdm_id in the following commands can be checked out from mount_vdm.sh list ;
4, The following .server_config commands only work on VDMs with replication configured.

Version: 6 24-May-2023 1/4

 Login to primary SP:
====
Step 1: Execute the following to see the list of hung tasks in testDB:

For SPA
/nas/bin/.server_config SVDM_A -v "testDB1 vdmid=<vdm_id> display"

Use the following script to check all VDMs running on SVDM_A:

for i in `mount_vdm.sh -list |grep _A | cut -d "|" -f 1` ; do echo /nas/bin/.server_config SVDM_A -v testDB1
vdmid=$i display ; /nas/bin/.server_config SVDM_A -v "testDB1 vdmid=$i display" ; done

For SPB
/nas/bin/.server_config SVDM_B -v "testDB1 vdmid=<vdm_id> display"

Use the following script to check all VDMs running on SVDM_B:

for i in `mount_vdm.sh -list |grep _B | cut -d "|" -f 1` ; do echo /nas/bin/.server_config SVDM_B -v testDB1
vdmid=$i display ; /nas/bin/.server_config SVDM_B -v "testDB1 vdmid=$i display" ; done

Step 2: Delete all the Keys shown in previous commands:

/nas/bin/.server_config SVDM_x -v "testDB1 vdmid=<vdm_id> delete key=xxx"

Where SVDM_x is SVDM_A or SVDM_B, xxx is the Key from the output in #1

Example for hung task of KEY= CKM001943018090000_0_1198613:8591133205_Initiator on SPA and vdmid

7, each key needs to be deleted separately
/nas/bin/.server_config SVDM_A -v "testDB1 vdmid=7 delete key=
CKM001943018090000_0_1198613:8591133205_Initiator"

Step 3: Execute the following to check the hung tasks in cmdQueueDb:

For SPA
/nas/bin/.server_config SVDM_A -v "cmdQueueDbTest1 vdmid=<vdm_id> display"

Use the following script to check all VDMs running on SVDM_A:

for i in `mount_vdm.sh -list |grep _A | cut -d "|" -f 1` ; do echo /nas/bin/.server_config SVDM_A -v
cmdQueueDbTest1 vdmid=$i display ; /nas/bin/.server_config SVDM_A -v "cmdQueueDbTest1 vdmid=$i
display" ; done

For SPB
/nas/bin/.server_config SVDM_B -v "cmdQueueDbTest1 vdmid=<vdm_id> display"

Use the following script to check all VDMs running on SVDM_B:

for i in `mount_vdm.sh -list |grep _B | cut -d "|" -f 1` ; do echo /nas/bin/.server_config SVDM_B -v
cmdQueueDbTest1 vdmid=$i display ; /nas/bin/.server_config SVDM_B -v "cmdQueueDbTest1 vdmid=$i
display" ; done

Step 4: Delete all the Keys shown in previous commands:

Version: 6 24-May-2023 2/4

 /nas/bin/.server_config SVDM_x -v "cmdQueueDbTest1 vdmid=<vdm_id> delete key=xxx"

Where SVDM_x is SVDM_A or SVDM_B, xxx is the Key from the output in #3

Example for hung task of

KEY=103079215111_CKM00194301809_0000_103079215143_CKM00193002457_0000 on SPA and vdmid
7, each key needs to be deleted separately
/nas/bin/.server_config SVDM_A -v "cmdQueueDbTest1 vdmid=7 delete
key=103079215111_CKM00194301809_0000_103079215143_CKM00193002457_0000"

Step 5: Check and remove the tasks found in /nas/tools/_console**

a) cd /nas/tools
ln -s console _console
./_console
taskmgr.dump

b) If there are replication tasks listed here, destroy them.

taskmgr.destroy 787548 <<<<< This is an example of a task ID

Step 6: Reboot SP to clear tasks from DART memory

Step 7: Double check that there are no hanging tasks anymore following steps 1 and 3.

Please make sure there is no hung task on BOTH testDB1 and cmdQueueDbTest1, also check BOTH PR and
DR site.

Internal Notes

Reach out to Replication SME if you have any questions

Please escalate to Escalation Engineering and refer to AR #727446 requested by EE

Article Properties

Affected Product

Dell EMC Unity Family

Last Published Date

12/14/2022 7:31 PM

Version: 6 24-May-2023 3/4

 Version

Article Type

Solution

Version: 6 24-May-2023 4/4