Related Sap Note No.

391

Archive Stuck Situation and Troubleshooting

Archive Stuck Situation Symptom: When in SAP R/3, archive stuck situation arise, R3 stops responding, no user will be able to login to the system, update process will be hang up, all users who are logged in system will also be hang.

Physical Symptom: At os level /oracle/<sid>/oraarch or /oracle/<sid>/saparch directory will be full of archive log files. There will not be free space available in above volumes.

Situation when archive stuck can arise: When there is big client copy is under progress or big background batch input job is running or any major database update related process is running, that time archive log will be generated rapidly. In such situation if archive log backup is not taken at regular interval, then the archive volume will be filled up and archive stuck situation arise.

Action to be taken when archive stuck situation arise

For Unix System:

1. Login with ora<sid> user in system

2. Find out the location of archive log files. Generally it is /oracle/<SID>/oraarch or saparch volume.

How to find out the location of archive log file You will find <SID>arch1_xxxx.dbf file in archive directory (For Example: SRQarch1_2833.dbf file is reside in /oracle/SRQ/oraarch volume)

3. Check for how much free space is available in oraarch directory #bdf | grep oraarch

/dev/vg12/oraarch 87785472 87785472 0 100% /oracle/SRQ/oraarch

In above example oraarch directory is 100% utilized. We need to move files from oraarch volume to other directory.

4. Immediately create ticket with concern database backup team to take the archive log backup of system for until further notice.

5. Find out the free volume in system where you can move the archive log file. The volume must be from sapdata volume, where you can copy the archive log files.

How to find out free volume in system: srqadm@hg20449:/ > bdf | grep sapdata Filesystem kbytes used avail %used Mounted on /dev/vg11/sapdata9 52428800 40963976 11375328 78% /oracle/SRQ/sapdata9 /dev/vg11/sapdata8 52428800 40963976 11375328 78% /oracle/SRQ/sapdata8 /dev/vg11/sapdata7 52428800 40963976 11375328 78% /oracle/SRQ/sapdata7 /dev/vg11/sapdata6 52428800 40963976 11375328 78% /oracle/SRQ/sapdata6 /dev/vg11/sapdata5 57671680 55300216 2353008 96% /oracle/SRQ/sapdata5 /dev/vg11/sapdata4 52428800 40963976 11375328 78% /oracle/SRQ/sapdata4 /dev/vg11/sapdata3 52428800 40963976 11375328 78% /oracle/SRQ/sapdata3 /dev/vg10/sapdat20 52428800 51204024 1215280 98% /oracle/SRQ/sapdata20 /dev/vg11/sapdata2 52428800 40963976 11375328 78% /oracle/SRQ/sapdata2 /dev/vg10/sapdat19 52428800 40963976 11375328 78% /oracle/SRQ/sapdata19 /dev/vg10/sapdat18 52428800 40963976 11375328 78% /oracle/SRQ/sapdata18 /dev/vg10/sapdat17 52428800 40963976 11375328 78% /oracle/SRQ/sapdata17

/dev/vg11/sapdat16 104857600 75781696 28848816 72% /oracle/SRQ/sapdata16 /dev/vg11/sapdat15 104857600 83973664 20720856 80% /oracle/SRQ/sapdata15 /dev/vg11/sapdat14 52428800 40963976 11375328 78% /oracle/SRQ/sapdata14 /dev/vg11/sapdat13 57671680 40964136 16577088 71% /oracle/SRQ/sapdata13 /dev/vg11/sapdat12 52428800 42486840 9864360 81% /oracle/SRQ/sapdata12 /dev/vg11/sapdat11 57671680 56324184 1337040 98% /oracle/SRQ/sapdata11 /dev/vg11/sapdat10 52428800 50180024 2231280 96% /oracle/SRQ/sapdata10 /dev/vg11/sapdata1 52428800 43020008 9335360 82% /oracle/SRQ/sapdata1

In above output /oracle/SRQ/sapdata16 is having the maximum free space available upto 28 Gb, here you can move upto 24 Gb of archive log files from /oracle/SRQ/oraarch volume.

6. Create one directory with name movedarchfiles into /oracle/SRQ/sapdata16 directory

#cd /oracle/SRQ/sapdata16 #mkdir movedarchfiles

7. Copy *.dbf file from /oracle/SRQ/oraarch directory to the /oracle/SRQ/sapdata16/movedarchfiles directory

#cd /oracle/SRQ/sapdata16/movedarchfiles #/oracle/SRQ/sapdata16/movedarchfiles > mv /oracle/SRQ/oraarch/*.dbf

Caution: Do not move all *.dbf file together, it will fill up the sapdata volume if the total size of all dbf file is greater than available size in sapdata volume, sapdata volume will be 100% utilized. Only move 15 to 20 dbf file to the sapdata volume in one single action. Note: Only move the file upto that level by which oraarch directory usage comes below threshold volume.

8. After moving archive log file to the above volume, go to /oracle/SRQ/saparch volume and rename the arch<SID>.log file to arch<SID>.log.old file as per following. Note: arch<SID>.log file can be in saparch or oraarch directory.

#cd /oracle/SRQ/saparch # /oracle/SRQ/saparch > mv arch<SID>.log arch<SID>.log.old

9. Keep an eye on archive log backup process and be in touch with database backup team in order to take continuous archive log backup.

10. Once archive directory usage come down to 20% to 30%, start moving the archive files from the /oracle/SRQ/sapdata16/movedarchfiles to /oracle/SRQ/oraarch directory with following command

# mv /oracle/SRQ/sapdata16/movedarchfiles/*.dbf /oracle/SRQ/oraarch/.

Note: oraarch directory should note be utilized above threshold value after moving the archive log file, if it is so, then only move some files and then get archive log backup and then once again move some files.

11. After moving archive log file to the above volume, go to /oracle/SRQ/saparch volume and rename the arch<SID>.log file to arch<SID>.log.old1 file as per following.

Note: This action is required at each and every movement action of archive log files. arch<SID>.log file can be in saparch or oraarch directory.

# /oracle/SRQ/saparch > mv arch<SID>.log arch<SID>.log.old1

12. Login at R3.

13. After successful completion of step 1 to 12, there should not be single archive file in movedarchfiles directory and oraarch directory must be below threshold value. 14. After successful following step 1 to 13, delete the movedarchfiles directory from sapdata volume

#cd /oracle/SRQ/sapdata16 #rm /oracle/SRQ/sapdata16/movedarchfiles

Process: 1. When you get alert for archive stuck situation, update the alert with all details 2. When you get CSS for archive stuck situation, update the CSS with all details.