Joint SHARE/GUIDE Technical Conference

Washington, D.C.

3035: Advanced HSM: Recovering

DFSMShsm CDSs, SDSPs and Tapes

Project: MVS Storage Management (MVSS) Project

Session Number: 3035
Presented by: Stan Kissinger
IBM Corporation
9000 S. Rita Road
Tucson, AZ 85744
kissingr@us.ibm.com

ABSTRACT
Everyone knows that DFSMShsm is the place to go to get your
user data recovered in the event of a loss or catastrophe, but
what happens when HSM managed data is lost? During this
session the speaker will present the procedures and
techniques necessary to recover the HSM Control Data Sets
(CDS's), the Small Data Set Packing Data Sets (SDSP's), and
HSM owned tapes.

Permission is granted to SHARE to publish this presentation paper in the SHARE proceedings.
IBM retains the right to distribute copies of this presentation to whomever it chooses.

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 1

 The following are trademarks of the
IBM Corporation

IBM

DFSMSdss

DFSMShsm

DFSMS/MVS

DFSORT

International Business Machines

Corporation provides this information "as is"
without warranty of any kind, either express
or implied, including, but not limited to, the
implied warranties of merchantability or
fitness for a particular purpose.

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 2

 Agenda

Preparing for data loss

How DFSMShsm locates its data

Sample recovery procedures

ML1 data set deleted - SDSP

ML1 data set deleted - non-SDSP

ML2 data set deleted

Backup data set deleted

ML2 or backup tape overwritten or rewound

Standard CDS Recovery

CDS Recovery with no good Journal

Recap

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 3

 Be Prepared

Backup user data sets

Dump ML1 volumes weekly

Export SDSP data sets daily

TAPECOPY ML2 and Backup tapes

DFSMShsm r1.4.0 supports tape duplexing

Keep plenty of CDS and Journal backups

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 4

 Order this book!

DFSMS/MVS Version 1 Release 4 IBM

Managing Data Availability

in a System-Managed
Storage Environment

IBM

IBM
IBM
IBM

IBM
SC26-4928-03

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 5

 Get to know your auditor

AUDIT MEDIACONTROLS

Rebuilds missing/incorrect CDS records from data on

DFSMShsm-owned media.

Audit requires "proof beyond reasonable doubt", not

just "preponderance of evidence."

Missing CDS records created only for migrated data.

Must be cataloged to MIGRAT to be considered valid.

Single-threaded operation. Can take a long time to

complete.

Important APAR -- OW24651

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 6

 Understand how DFSMShsm locates
migration and backup data

Requirements for access to migration

and backup data

CDS records identify migration or backup volser

CDS records identify location of data on volser

Catalog shows migrated data as residing on

pseudo-volser MIGRAT

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 7

 Locating data on DASD

MIG001
CDS
V TOC

DS N 1

Backup and migration copies are not

cataloged, identified only in CDS records

Migration - mprefix.HMIG.Tssmmhh.user1.user2.xyddd

Backup - bprefix.BACK.Tssmmhh.user1.user2.xyddd

Data accessed through VTOC of volume

identified in CDS records

Data sets in SDSPs do not get generated

name until migrated to ML2

Data sets exist as records in SDSP KSDS

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 8

 Locating data on tape

Header TM dsn1 dsn2 dsn3 TM Trailer TMTM

VOL1 HDR1 HDR2 DATA DATA DATA DATA EOF1 EOF2

FBID 1 2 3 4 5 6 7 8 9

Backup and ML2 tapes written in

single-file format

ML2 - mprefix.HMIGTAPE.DATASET
Backup - bprefix.BACKTAPE.DATASET

Data written in 16K blocks

DFSMShsm CDS records save starting FBID

Each user data set starts a new 16K block

Individual user data sets may be

represented by multiple 16K blocks

Internal control information identifies end of data set

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 9

 What's your point?

Tape positioning

RECYCLE, RECALL and RECOVER processing

issue POINT macro to position to starting FBID

POINT can read past EOF mark on tape when in

high-speed mode

High-speed mode used only if POINT beyond certain

number of sectors

Important APAR -- OW32375

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 10

 Disaster Strikes!

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 11

 Step 1 - Assess the situation

How much is it going to cost to retrieve the

lost data?

Is it cheaper to recreate the lost data than to retrieve it?

What alternate sources exist for the data?

Backups, TAPECOPY alternate tapes, dump copies, etc.

What is the nature of the data loss?

CDS records missing, data missing, both?

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 12

 Example 1 -- ML1 data set deleted
SDSP

HDELETE issued or data set deleted after

having been RECALLed

Determine ML1 volser data set resided on

Use DFSMShsm Log, Migration Activity Log, backup copy

of MCDS or Journal from time of RECALL/HDELETE

Determine if SDSP still contain the records

for the data set

Use FIXCDS D dsn to display MCDFNSCR flag

If records remain in SDSP

(MCDFNSCR=ON)
HSEND FIXCDS D dsn PATCH(6 BITS(1...1...))

DEFINE NONVSAM(NAME(dsn) VOLUMES(MIGRAT)

DEVICE(d/t))

HRECALL dsn

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 13

 Example 1 -- ML1 data set deleted
SDSP (cont.)

If records not in SDSP (MCDFNSCR=OFF

or no MCD exists)
Recover SDSP from backup/dump to new name

REPRO from backup SDSP to active SDSP using the

FROMKEY(dsn) and TOKEY(dsn) parameters

HSEND FIXCDS D dsn

ADDMIGRATEDDATASET(ml1vol) if no MCD currently
exists

HSEND FIXCDS D dsn PATCH(6 BITS(1...1...))

DEFINE NONVSAM(NAME(dsn) VOLUMES(MIGRAT)

DEVICE(d/t))

HRECALL dsn

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 14

 Example 2 -- ML1 data set deleted
non-SDSP

HDELETE issued or data set deleted after

having been RECALLed

Determine ML1 volser data set resided on

and name of migration copy

Use backup copy of MCDS or Journal from time of

RECALL/HDELETE

Restore migration copy from dump of ML1

Use DFSMSdss RESTORE for migration copy name

Recreate catalog and CDS records

DEFINE NONVSAM(NAME(dsn) VOLUMES(MIGRAT)
DEVICE(d/t))

HSEND AUDIT MEDIACONTROLS VOLUMES(ml1vol)

FIX ODS(xxx)

HRECALL dsn

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 15

 Example 3 -- ML2 data set deleted

HDELETE issued or data set deleted after

having been RECALLed

Determine ML2 volser data set resided on

Use DFSMShsm Log, Migration Activity Log, backup

copy of MCDS or Journal from time of
RECALL/HDELETE

Recreate catalog and CDS records

DEFINE NONVSAM(NAME(dsn) VOLUMES(MIGRAT)
DEVICE(d/t))

HSEND AUDIT MEDIACONTROLS VOLUMES(ml2vol)

FIX ODS(xxx)

HRECALL dsn

See Case 9 in Managing Data Availability

manual for details

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 16

 Example 4 -- Backup data set deleted

Data set deleted, desired backup version rolled

off, HBDELETEd or EXPIREBVd

AUDIT MEDIACONTROLS will not recreate

missing CDS records for backup tapes, lacks
"proof beyond reasonable doubt".

Determine tape backup version resided on

Use DFSMShsm Log, Backup Activity Log, backup copy

of BCDS or Journal from time backup version lost

Plant additional evidence

IEBGENER backup tape to new tape with
DSN=mprefix.HMIGTAPE.DATASET

ADDVOL new tape as ML2

DELVOL MARKFULL planted ML2 tape to prevent reuse

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 17

 Example 4 -- Backup data set deleted
(cont.)

Recreate catalog and CDS records

DEFINE NONVSAM(NAME(dsn) VOLUMES(MIGRAT)
DEVICE(d/t))

HSEND AUDIT MEDIACONTROLS VOLUMES(ml2vol) FIX

ODS(xxx)

HRECALL dsn

FIXCDS V ml2vol DELETE

FIXCDS T L2-ml2vol-0000 DELETE

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 18

 Overwritten Tape

Header TM dsnx TM Trailer TMTMdsn2 dsn3 TM Trailer TMTM

VOL1 HDR1 HDR2 DATA EOF1 EOF2 TA DATA EOF1 EOF2

Other Application Data Old HSM Data

Tape partially overwritten by another

application

Previously written DFSMShsm data overwritten

End-of-file precedes remaining DFSMShsm data

Error indications

Gets abend813-04 when being read

Data lost

Data set 1 overwritten entirely

Data set 2 partially overwritten

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 19

 Rewound Tape

dsn7b dsn8 dsn9 TM Trailer TMTM dsn5 dsn6 dsn7a

DATA DATA DATA DATA EOF1 EOF2 TA DATA DATA DATA

Newer HSM Data Older HSM Data

Tape rewound while being written to by

DFSMShsm

Seen when tape drives shared across systems and

VARY ONLINE issued for device from another system

Error indications

Gets abend237-0C when being written to

Gets abendA13-14 when being read

Data lost

Tape labels, data sets 1, 2, 3 and 4 overwritten entirely

Data set 5 partially overwritten
Data set 7 split

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 20

 Example 5 -- ML2 or Backup tape
overwritten or rewound

Some data has been lost, need to retrieve what

valid data remains.

Do not run AUDIT MEDIACONTROLS against

these tapes, as it will invalidate all data beyond
EOF mark.

DELVOL MARKFULL damaged tape

Prevent further data from being overwritten

Copy damaged tape volume (optional)

Damaged tape will be returned to scratch, may want

newer data written to tape

Reinitialize damaged tape

Need to create correct labels

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 21

 Example 5 -- ML2 or Backup tape
overwritten or rewound (cont.)

Delete first data set on tape if spanned from

previous tape
Use LIST command to identify spanning condition

Use DELETE or BDELETE command to delete dataset

RECYCLE problem tape

Use RECYCLE to move remaining valid data to another

tape

Identify lost data sets

Use sample ICETOOL jobs in Managing Data Availability

manual to identify data sets not moved by RECYCLE
and to build appropriate recovery commands

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 22

 Example 5 -- ML2 or Backup tape
overwritten or rewound (cont.)

RECYCLE FORCE damaged tape

Use RECYCLE FORCE to delete CDS records for lost
data sets and damaged volume

Deal with lost data sets

ML2 - issue RECOVER commands for lost data sets

Backup - Issue BACKDS commands for lost data sets

See Cases 6 & 7 in Managing Data

Availability manual for details

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 23

 Example 6 -- Standard CDS Recovery

IDCAMS EXAMINE shows Index damage,

duplicate records, etc. Journal records complete.

Two CDS recovery procedures provided

UPDATEC

Enhanced CDS Recovery using JCL

Important APAR -- OW34069

Enhanced CDS Recovery Sort JCL modified

in DFSMShsm r1.4.0

Must use modified JCL if any system running r1.4.0

See DFSMShsm Storage Administration

Guide / DFHSM System Programmer's
Guide for details

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 24

 Example 7 -- Broken CDS, No Journal

IDCAMS EXAMINE shows Index damage,

duplicate records, etc. Journal records missing.

Stop DFSMShsm on all systems

IDCAMS REPRO the Data component

REPRO IDS(cds_data_component) ODS(your_copy)

REPRO IDS(cds_cluster) ODS(your_copy) FADDR(0)

Sort by key to eliminate duplicate records

Reload CDS

See Case1 in Managing Data Availability

manual for details

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 25

 Conclusion

Be Prepared

Evaluate the situation before taking any

action

Know your tools

Practice CDS Recovery

REMEMBER -- It's only ones and

zeroes!

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 26

 References
Managing Data Availability in a System-Managed Storage
Environment (SC264928)

DFSMShsm Storage Administration Reference (SH211075)

DFHSM System Programmer's Command Reference

(SH350083)

DFSMShsm Storage Administration Guide (SH211076)

DFHSM System Programmer's Guide (SH350085)

DFSMShsm Diagnosis Reference (LY279608)

DFHSM Diagnosis Reference for Data Areas and Control

Blocks (LY350105)

DFSMShsm Diagnosis Guide (LY279607)

DFHSM Diagnosis Guide (LY350098)

Visit our web site

http://www.storage.ibm.com/software/sms/hsm/hsmhome.htm

Joint SHARE/GUIDE - 3035 (c) Copyright IBM Corporation, 1998 27