5/9/2019 Product Support Portal - Article Welcome: Inder RATHORE

(http://nokia.com)
Portal Profile | Logout (/portal/logout)
(https://market.alcatel-
lucent.com/myprofile)

Welcome to the

Product Support Portal (PSP)

PSP Home (/portal/)

VDP Appliance is nearly full or full. Recover space.

Average rating: based on feedback from one user
Information >

Problem Description
VDP:[002] The VDP storage is nearly full.
Or
VDP:[003] The VDP storage is full.

Diagnostic Steps
The alarms itself are self-explanatory.
Check Filesystem status in vdp.
root@vdp:~/ (mailto:root@vdp:~/)#: df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 32G 6.2G 24G 21% /

udev 7.8G 144K 7.8G 1% /dev

tmpfs 7.8G 0 7.8G 0% /dev/shm

/dev/sda1 128M 37M 85M 31% /boot

/dev/sda7 1.5G 199M 1.2G 15% /var

/dev/sda9 138G 13G 118G 10% /space

/dev/sdc1 1.0T 867G 157G 85% /data01

/dev/sdd1 1.0T 861G 163G 85% /data02

/dev/sdb1 1.0T 864G 160G 85% /data03

The difference between file system space as above, and reported by vdp as in the screenshot should not be much. Big difference means that
https://ps-kb.app.alcatel-lucent.com/portal/search/article/view/21008121
daily VDP maintenance is not able to recover space efficiently. 1/7
 Solution
5/9/2019 Product Support Portal - Article

The solution to this problem is to run Garbage collection manually. It may have to be done multiple times to claim more space.
Step1: Delete old backup starting from the oldest date. It has to be done from VDP restore tab as shown below. This step will free raw space in
the filesystem.

Step2: Check that maintenance scheduler is not running.

root@vdp:~/ (mailto:root@vdp:~/)#: status.dpn

Thu Apr 12 08:40:16 ART 2018 [vdp] Thu Apr 12 11:40:16 2018 UTC (Initialized Mon Dec 19 14:58:33 2016 UTC)

Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent F
ull and Stripe Status by Disk

0.0 10.105.44.135 7.2.80-118 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 2.99 14485 81137692 43.8% 43%(onl:2
598) 43%(onl:2599) 43%(onl:2597)

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1436902363@00:50:56:B3:BA:D7

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)

System-Status: ok

Access-Status: full

Last checkpoint: cp.20180412110235 finished Thu Apr 12 08:05:27 2018 after 02m 52s (OK)

Last GC: finished Thu Apr 12 08:02:35 2018 after 01m 59s >> recovered 0.00 KB (MSG_ERR_DISKFULL)

Hfscheck in progress: started Thu Apr 12 08:05:27 2018 >> checked 1704 of 6487 stripes (hfscheck)

Maintenance windows scheduler capacity profile is active.

The maintenance window is currently running.

Currently running task(s): hfscheck

Next backup window start time: Thu Apr 12 17:00:00 2018 ART

Next maintenance window start time: Fri Apr 13 08:00:00 2018 ART

The above bold line shows that it is running. If running, it is better to wait for it to finish.
Also check the status of maintenance scheduler, and backup scheduler.
root@vdp:~/#: dpnctl status

https://ps-kb.app.alcatel-lucent.com/portal/search/article/view/21008121 2/7
5/9/2019 Product Support Portal - Article
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

Step3: Disable all backup jobs in VDP GUI. Wait for all backup jobs to finish if some are running. In NetAct operating documentation,
NetAct Administration > NetAct Administration Operating Procedures > Administering Backups > Taking and Restoring Disk-based Backups >
Taking and restoring backups of virtual machines > Enabling or disabling VDP backup jobs
Step4: Stop Maintenance service, and backup scheduler.
root@vdp:~/ (mailto:root@vdp:~/)#: dpnctl stop maint

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: Suspending maintenance windows scheduler...

root@vdp:~/ (mailto:root@vdp:~/)#: dpnctl stop sched

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: Suspending backup scheduler...

dpnctl: INFO: Backup scheduler suspended.

Step5: Check services status that they are stopped which are highlighted in bold.
root@vdp:~/ (mailto:root@vdp:~/)#: dpnctl status

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: gsan status: up

dpnctl: INFO: MCS status: up.

dpnctl: INFO: emt status: up.

dpnctl: INFO: Backup scheduler status: down.

dpnctl: INFO: axionfs status: down.

dpnctl: INFO: Maintenance windows scheduler status: suspended.

dpnctl: INFO: Unattended startup status: enabled.

dpnctl: INFO: avinstaller status: up.

dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

root@vdp:~/ (mailto:root@vdp:~/)#: status.dpn

https://ps-kb.app.alcatel-lucent.com/portal/search/article/view/21008121 3/7
5/9/2019 Product Support Portal - Article
Thu Apr 12 08:14:48 ART 2018 [vdp] Thu Apr 12 11:14:48 2018 UTC (Initialized Tue Jul 14 19:32:43 2015 UTC)

Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent F
ull and Stripe Status by Disk

0.0 10.105.39.7 7.2.80-129 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 0.85 15354 76059877 52.4% 52%(onl:2
440) 52%(onl:2436) 52%(onl:2441)

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1436902363@00:50:56:B3:BA:D7

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)

System-Status: ok

Access-Status: full

No checkpoint yet

Last GC: finished Thu Apr 12 07:45:59 2018 after 01h 03m >> recovered 201.29 GB (OK)

No hfscheck yet

Maintenance windows scheduler capacity profile is active.

WARNING: Scheduler is STOPPED.

Next backup window start time: Thu Apr 12 20:00:00 2018 ART

Next maintenance window start time: Fri Apr 13 08:00:00 2018 ART

Step6: Run Garbage Collection

root@vdp:~/ (mailto:root@vdp:~/)#: avmaint garbagecollect --timeout=7200--ava

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<garbage-collect-params

gccount="0"

gcmincount="0"

refcheck="true"

killsecs="0"

maxtime="3600"

maxpass="100"

throttle="0"

usehistory="false"

limitadjust="0"

orphansfirst="false"/>

Note: timeout=<how many seconds should GC run>. Keep it sufficient, approximately 2 hours is required because default GC run time is 3600
seconds.

Step7: Confirm that GC has started.

root@vdp:~/ (mailto:root@vdp:~/)#: status.dpn

https://ps-kb.app.alcatel-lucent.com/portal/search/article/view/21008121 4/7
5/9/2019 Product Support Portal - Article
Thu Apr 12 08:57:12 ART 2018 [vdp] Thu Apr 12 11:57:12 2018 UTC (Initialized Tue Jul 14 19:32:43 2015 UTC)

Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent F
ull and Stripe Status by Disk

0.0 10.105.39.7 7.2.80-129 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 3.95 15795 76060910 52.4% 52%(onl:2
440) 52%(onl:2436) 52%(onl:2441)

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1436902363@00:50:56:B3:BA:D7

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)

System-Status: ok

Access-Status: full

No checkpoint yet

GC in progress: started Thu Apr 12 08:56:21 2018 >> recovered 0.00 KB (so far)

No hfscheck yet

Maintenance windows scheduler capacity profile is active.

WARNING: Scheduler is STOPPED.

Next backup window start time: Thu Apr 12 20:00:00 2018 ART

Next maintenance window start time: Fri Apr 13 08:00:00 2018 ART

Step8: After GC is finished. You can check the space recovered using the same command as above. It can also be confirmed with VDP
configuration window in GUI.
root@vdp:~/ (mailto:root@vdp:~/)#: status.dpn

https://ps-kb.app.alcatel-lucent.com/portal/search/article/view/21008121 5/7
5/9/2019 Product Support Portal - Article
Thu Apr 12 08:14:48 ART 2018 [vdp] Thu Apr 12 11:14:48 2018 UTC (Initialized Tue Jul 14 19:32:43 2015 UTC)

Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent F
ull and Stripe Status by Disk

0.0 10.105.39.7 7.2.80-129 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 0.85 15354 76059877 52.4% 52%(onl:2
440) 52%(onl:2436) 52%(onl:2441)

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1436902363@00:50:56:B3:BA:D7

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)

System-Status: ok

Access-Status: full

No checkpoint yet

Last GC: finished Thu Apr 12 07:45:59 2018 after 01h 03m >> recovered 201.29 GB (OK)

No hfscheck yet

Maintenance windows scheduler capacity profile is active.

WARNING: Scheduler is STOPPED.

Next backup window start time: Thu Apr 12 20:00:00 2018 ART

Next maintenance window start time: Fri Apr 13 08:00:00 2018 ART

Step9: Repeat GC few times by repeating steps 6-8.

Step10: Enable the stopped services.
root@rc5vdp:~/ (mailto:root@rc5vdp:~/)#: dpnctl start maint

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: Resuming maintenance windows scheduler...
dpnctl: INFO: maintenance windows scheduler resumed.

root@rc5vdp:~/ (mailto:root@rc5vdp:~/)#: dpnctl start sched

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.

Check that both the above services are enabled.

root@vdp:~/#: dpnctl status

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

Step11: Enable backup jobs in VDP GUI in vcenter. In NetAct operating documentation,
NetAct Administration > NetAct Administration Operating Procedures > Administering Backups > Taking and Restoring Disk-based Backups >
Taking and restoring backups of virtual machines > Enabling or disabling VDP backup jobs

Workaround
https://ps-kb.app.alcatel-lucent.com/portal/search/article/view/21008121 6/7
 Cause
5/9/2019 Product Support Portal - Article

Information for Internal Audiences

Tested on :
Customer Name : Claro Argentina
NetAct : rc8
NetAct Version : 17.8

Applicable Products & Releases

NetAct NetAct 17.2
NetAct 17.8

Linked Content
Articles Attachments Videos

There are no linked articles

Comments
Rate this article: Leave a Comment

Date Type Comment Commenter Reason for Closing Status

No data available in table

PSP Home

Need more help? Feel free to contact us (mailto:psp.admin@nokia.com)

© Nokia 2018 All rights reserved
Terms of use (https://www.nokia.com/en_int/terms) Privacy (https://www.nokia.com/en_int/privacy)
Cookie policy (https://www.nokia.com/en_int/cookies) V3.7.14 [2019-02-20]   

https://ps-kb.app.alcatel-lucent.com/portal/search/article/view/21008121 7/7