Home Chapter 11a - Maintaining MVS Contents

The Web Version of this chapter is split into 2 pages - this is page 1 - page contents are as follows:

Section 11.1 - Introduction

Section 11.2 - When and How to put on Maintenance and Upgrades
11.2.1 The Issues
11.2.2 Option A - PUT tapes and product tapes
11.2.3 Option B - CBIPO and CBPDO
Section 11.3 - Cloning and the MVS Maintenance Process
11.3.1 The Issues
11.3.2 Restructuring your MVS System
11.3.3 SYSRES Backups
11.3.4 The Options

Home Contents Next part of chapter Next Section Top of Page

11.1 - Introduction
Once you have created a production MVS system, you will have to modify it from time to time. This may include
putting on fixes to correct and prevent problems, and upgrading components of the system to provide new functions
required by your users. This chapter deals with some key questions raised by the need to update your system:

* when should you put on maintenance and upgrades, and which of IBM's service delivery options should you use?

* how should you structure and "clone" your system to make the maintenance process as painless and effective as
possible?

11.2 When and How to put on Maintenance and Upgrades

11.2.1 The Issues

IBM provides a bewildering array of offerings for the provision of service, service information, and product upgrades,
including:

* PUT tapes - tapes of all PTFs for your product profile, sent out automatically around once a month. These also
include "cover letters" describing key PTFs and the problems they resolve, and HOLDDATA describing PTFs on
previous PUT tapes which were in error

* CBIPOs - installation tapes integrating a whole range of products with stable service up to a given PUT level. These
are created in response to a specific customer order

* CBPDOs - installation tapes integrating PTFs for your product profile from several PUT tapes, and additional
products if required. Again, these are put together in response to a specific customer order
* electronic download of PTFs from DIAL-IBM (known as IBMLINK in the U.S.)

 * delivery of individual PTFs on tape on request

* PSP tapes - tapes offering information about high-impact problems relevant to your product profile

* product tapes - individual products ordered by a customer

* cumulative service tapes - tapes including service from a series of PUT tapes for a single product or group of
products (often supplied along with a product tape to bring the product up to a reasonably current level)

* problem records on INFO/MVS and DIAL-IBM (IBMLINK), describing individual problems and their resolving
PTFs

Given this array of sources of information, fixes, and product upgrades, which ones should you use and when? To some
extent, the answer will depend on your circumstances and personal preferences. But let's start with a few certainties:

* all IBM service and product upgrades are packaged in SMP/E format and should always be applied using SMP/E

* when you are experiencing a problem with a significant impact on your users and IBM informs you that there is a
PTF or a series of PTFs which will correct it, you should apply them - this is known as corrective service.

* when you are installing or upgrading a product, you should obtain the relevant PSP information. If this tells you there
are any HIPER (high impact and pervasive) problems outstanding for the product you are installing you should apply
the fixes for these.

* whenever you receive a PUT tape, you should RECEIVE the HOLDDATA from it - this will prevent you
inadvertently applying fixes which are known to be in error.

Beyond these few certainties, you will have to apply your judgement. Broadly speaking, there are two options, though
you can "mix and match" these options when it suits you. The options are:

* keep reasonably up to date with your preventative maintenance by applying the fixes from PUT tapes regularly, and
upgrade products using product tapes and cumulative service tapes

OR

* ignore all preventative service except at product installation time, and bring your whole system up to date in a "big
bang" by installing a CBIPO or CBPDO on a less frequent basis - say once a year.

The rest of this section discusses these two options in detail.

Home Contents Previous Section Next Section Top of Page

11.2.2 Option A - PUT tapes and product tapes

Few systems programmers would advocate installing all the service on every PUT tape as soon as it arrived, but quite a
few do seem to favour installing PUT service on a regular basis (there's a fascinating selection of comments on this
question in the "NASCOM Highlights" section of September 1990's issue of "Technical Support"). Typically, they
install the service from 2 or 3 PUT tapes every 3 months, but avoid putting on the service from the most recent couple
of tapes.
The reason for avoiding the most recent service is simple - if it has errors in it and you install it, you will experience
problems. If, on the other hand, you wait for a while before putting on the service, other installations will find the
errors, and the following PUT tapes will include HOLDDATA for the PTFs in error. As long as you observe the advice
above and RECEIVE all HOLDDATA from PUT tapes as soon as you get them, you will have the benefit of other
users' experience when you install the service from slightly older PUT tapes, as SMP/E will automatically HOLD the
PTFs in error as a result of the HOLDDATA from the more recent tapes.

When these systems programmers wish to install a new product or upgrade a product, they typically order a product
tape from IBM and a cumulative service tape to go with it, which will bring the product up to the most recent PUT
level. As with the PUT tapes themselves, they will avoid installing cumulative service right up to the current PUT level.
Instead they will bring the product up to the level of service of, say, the last PUT tape but two, and avoid putting on the
more recent service until they have received HOLDDATA from the next two PUT tapes (once you have added a
product to your profile, future service for it should be automatically included in your PUT tapes).

If you decide to adopt this approach, you will have to go through the following processes whenever you install the
service from a group of PUT tapes:

* RECEIVE the SYSMODs from the tapes (you should already have RECEIVEd the HOLDDATA from them and
from any subsequent tapes you have on-site)

* Ask IBM for any PSP information relating to these PUT tapes

* Obtain any additional fixes for HIPER problems indicated by the PSP information

* Perform an APPLY CHECK to show you what fixes will be applied, and more importantly, what fixes will be HELD
and why

* Investigate any SYSTEM type HOLDs; often these are merely informational and you can release them without any ill
effect (either by specifying BYPASS(HOLDSYSTEM(...)) when you come to APPLY your service, which will release
all SYSTEM type HOLDs of the specified types, or by coding up ++RELEASE statements for the PTFs concerned and
RECEIVEing these)

* APPLY the service

* Bring up a test system with these fixes on, and test it thoroughly, paying particular attention to those products and
system functions to which you have applied service

* Migrate your updated system into production (see the second part of this chapter for discussion of this process)

When you wish to install or upgrade a product using a product tape and cumulative service tape, you will need to go
through a similar series of processes:

* RECEIVE, APPLY, and ACCEPT the base product

* RECEIVE the cumulative service

* Research the cumulative service (i.e. the same steps as required for a PUT tape between RECEIVE and APPLY)

* APPLY the service

* Test the new/upgraded product thoroughly

* Migrate into production

There are three main advantages of this approach:

* by keeping reasonably up to date with service, you should in theory prevent some avoidable problems

* if you are reasonably up to date with service and you do experience problems, the "chain" of PTFs and prerequisite
PTFs required to fix them should be short

* by keeping all products up to a current level without reinstalling the whole system, it avoids regularly repeating much
of the "customisation" and "restructuring" activity required to migrate a CBIPO system into production

However, there are some serious disadvantages:

* the PTF research phase of the process is time consuming and tedious, particularly if you have to repeat it every few
months - and there is therefore a temptation to skimp on it, which is risky!

* the number of problems prevented by "preventative service" is often less than the number of new problems it
introduces, even if you use HOLDDATA as effectively as possible.

* there is a dangerous temptation when putting on regular service to regard this as a "minor change", assume that
SMP/E HOLDs will prevent any problems, and neglect the requirement for thorough testing before going into
production. On the other hand, if you do test thorougly every time, this will be a time consuming exercise, both for you
and for the other people you will have to involve

* upgrading core products such as MVS itself and JES2 is more complex using a product tape than using a CBIPO

* many sites have a large selection of IBM program products, and most of these require an upgrade every couple of
years - so if you are going to keep up to date by this process you are going to have a full time job just upgrading
program products

You can, of course, deal with this last two problems by combining the PUT tape approach with the CBIPO/CBPDO
process, rather than with the product tape approach - or use product tapes for some upgrades and CBIPO/CBPDO for
others. Personally, however, I prefer to ignore PUT tapes and product tapes altogether (except as sources of
HOLDDATA and corrective fixes), and do all my maintenance and upgrades using option B.

Home Contents Previous Section Next Section Top of Page

11.2.3 Option B - CBIPO and CBPDO

The alternative approach to keeping up to date with PUT tapes and product tapes is to bring your system up to a stable
current level less frequently using CBIPO and/or CBPDO.

Installing an MVS CBIPO involves rebuilding your entire system from scratch and reinstalling almost every IBM
product you have got (other than CICS, NCP, and IMS/DB2). The CBIPO process is discussed in detail in chapter 7
above; but for the sake of comparison with the PUT/product tape approach, we will repeat a few key points here:

* the CBIPO comes with integrated service which has been researched by IBM and brought up to a relatively stable
level

* because IBM has already done PTF research or your order, you should not have to do as much as on a PUT tape
installation

* the CBIPO process steps you through all the jobs needed to build a "base" copy of your MVS system and all the other
products included in your order, but once this is complete you must customise your system to make it functionally
equivalent to your existing production system and restructure it as discussed in the second part of this chapter to
provide a suitable environment for future maintenance

Installing a CBPDO is much simpler. The simplest type of CBPDO is known as a "service only CBPDO". This merely
integrates service from several PUT tapes (up to two years worth) with additional fixes indicated by IBM PTF research.
IBM automatically start building the CBPDO from the point to which your last CBIPO or CBPDO brought your
system, unless you request otherwise. As with a CBIPO, you must still request and follow up PSP information, but
because IBM have already integrated the results of their own PTF research, less research should be required than for a
PUT tape, and the level of service represented by the CBPDO should be more stable than that on a recent PUT tape.
CBPDO therefore has all the advantages of CBIPO - relatively little PTF research and a relatively stable level of
service - without the disadvantage of having to rebuild your system from scratch.

You may also order a CBPDO which combines the service found on a "service only" CBPDO with additional/upgraded
products. This provides an easy way of installing additional products, or upgrading a group of products, at the same
time as putting on preventative service. The CBPDO installation process where new/upgraded products are involved is:

* RECEIVE the products and service on the CBPDO

* Order the PSP information and any additional fixes indicated by it

* APPLY the products and service

* Perform any customisation required for the new/upgraded products

* Test the new products thoroughly and any functions which appear to be affected by service

* Migrate to production

CBPDO therefore provides a method of upgrading your maintenance level and installing new/upgraded products with
less work overall than using PUT tapes and product tapes.

MVS and JES upgrades, however, are really too complex to do using CBPDO, so you should order a CBIPO to do one
of these. And MVS and JES upgrades are becoming ever more frequent - to the point where you are likely to need a
major upgrade every year. So many sites simplify their upgrade workload still further by installing all product upgrades
and new products as part of an annual MVS upgrade. The next logical step is to cut out preventative maintenance in
between CBIPOs and bundle that in with your annual CBIPO as well. I've followed that line for many years with few
problems. The advantages are clear:

* each upgrade is a major upgrade and so is thoroughly tested before implementation

* each upgrade takes you to a relatively stable level

* no unnecessary fixes are put on which can lead to unpredictable problems when the fix turns out to be in error

* product upgrades and preventative maintenance take you a couple of months each year, leaving the rest of the year
free for more varied and productive work

* time spent on PTF research is minimised

The disadvantages mirror the advantages of Option A:

* if you do have problems, you could have to apply a long chain of PTFs from a series of PUT tapes to fix them
* a CBIPO or CBPDO usually represents a less recent level of service than a PUT tape you could have obtained at the
same time, because (a) it may have been built up to three months ago; and (b) chains of PTFs in error are excluded at
build time.

* a major effort is required to recustomise and restructure your system every year

If these problems worry you, you can always take an intermediate approach. For example, install a CBIPO every 1-2
years, but use CBPDO every 6 months to keep up to date with preventative maintenance and install any new products
or upgrades which cannot be put off until the next CBIPO.

The suitability of option B also depends on how likely you are to need recent service to be applied to your system. If
yours is a "leading edge" site - one which regularly introduces new hardware or software begore it has been widely
proven on other user sites - you are more likely to need recent fixes to deal with the problems this is likely to cause. In
this situation, you might find it necessary to keep up to relatively recent service levels by adopting a variant of Option
A.

Home Contents Previous Section Next Section Top of Page

11.3 Cloning and the MVS Maintenance Process

11.3.1 The Issues

Cloning is an essential stage in the process of implementing maintenance on an MVS system. In order to test fixes and
upgrades before they go into production, the systems programmer must first apply them to a testing version of the MVS
system. Once testing has been completed successfully, the maintenance must be migrated onto the production version
of the system. Either at the stage of creating the test system or at the stage of migrating into production, this process
almost always involves "cloning" - copying one version of the system to create another, functionally identical version.

Your approach to cloning and applying maintenance can be defined by a set of policies and procedures addressing
questions like:

* into which set of libraries will maintenance be installed?

* how will the maintenance be tested?

* how will maintenance be migrated into the production libraries?

* how will you regress back to the previous level if you have problems with the maintenance?

IBM offers little advice on these questions, so over the years systems programmers have evolved a broad selection of
answers. This section describes three of the options I have seen in use, with the object of making clear the principles
involved and recommending a preferred technique.

Home Contents Previous Section Next Section Top of Page

11.3.2 Restructuring your MVS System

All the options involve some degree of "restructuring" your system after you have installed a CBIPO (or built your
system in some other way), in order to simplify the process of migrating between the "live" and "target" versions of
your system. The CBIPO process places all MVS target libraries on the SYSRES pack (assuming that you combine all
the "logical" SYSRES packs on a single physical one, as you should); and this is also the normal configuration for
other installation processes. However, some of the datasets placed on the SYSRES by CBIPO cause difficulties for
most cloning strategies. The process of restructuring your system therefore typically includes a "tidy up" of the
SYSRES pack.

Although there are varying approaches to what should be left on the SYSRES pack and what should be taken off, all
the options I discuss below use a similar approach: only target libraries which are updated by SMP/E (and SMP/E
alone) should be left on the SYSRES pack. The only exception to this rule is the SMP/E target zone - this is the only
dataset whose location varies between the different options, and in some of these it resides on the SYSRES. Thus,
datasets such as SYS1.LOGREC, SYS1.DUMPnn datasets, ISF.HASPINDX, and SYS1.BRODCAST, which are
updated by MVS itself or by other system components should be moved elsewhere. Datasets such as SYS1.PROCLIB,
SYS1.PARMLIB, and SYS1.UADS which are updated by system support staff should also be moved off the SYSRES
pack. By moving these datasets off the SYSRES pack, you ensure that:

* if you copy one version of the SYSRES pack over the top of another you do not overwrite data placed in these
datasets by the system or the systems programmers

* you can IPL off any one of several alternative SYSRES packs and still use the same copy of these datasets

As at MVS/ESA 3.1.3, the datasets mentioned above are the only ones placed on the SYSRES pack by CBIPO which
you need to move off, but you should be able to check for yourself which datasets need moving by identifying those
datasets updated by the system itself or by staff working on the system. The master catalog pack is a very suitable
alternative location for all of the datasets listed above.

Restructuring your system may also involve setting up different SMP/E target zones corresponding to the live and
target versions of your system. In addition, in order to be able to IPL your system at will from any one of two or more
SYSRES packs, you must ensure that all MVS datasets on the SYSRES packs are cataloged with indirect volume
references (i.e. as being on DEVTYPE(0000) and VOLUME(******)).

Home Contents Previous Section End of Page Top of Page

11.3.3 SYSRES Backups

Another issue we need to address is the question of SYSRES backups. Again, I feel there is a definite "best" approach
to this, and I have assumed this approach in all three options discussed below. This is to back up the maintenance
SYSRES pack after you have put on maintenance and tested it thoroughly, but before migrating your maintenance into
production. The associated SMP/E target zone should be backed up along with the MVS target libraries, so that if you
need to restore the MVS libraries you can also recreate an SMP/E environment which is in step with them. These
backups should be kept on a GDG with a fairly large cycle (I keep 10 generations), so that you always have copies of a
series of versions of your SYSRES pack, as it was when it was put into production at each maintenance upgrade.
Option A actually integrates this backup into the migration process, which is a useful way of ensuring it gets done, and
this could also be done for the other options.

11.3.4 The Options

Given that you need two SYSRES packs to make your system maintainable, there are two possible approaches to using
these:

* designate one the permanent live SYSRES and the other the permanent maintenance SYSRES

 OR

* alternate the usage of the two packs (i.e. "flip-flop" the packs)
The flip-flop approach creates the problem of how to keep the SMP/E target zone(s) in step with the target volumes
themselves, and there are at least two approaches to dealing with this problem too.

These permutations give us the three options which will be discussed below:

* Option A - Stable SYSRES configuration, in which one pack is always the live SYSRES, another is always the target
SYSRES, and there is only one SMP/E target zone, which relates to the target SYSRES pack.

* Option B - SYSRES flip-flop with two SMP/E target zones, where both the new target SYSRES and the new target
SMP/E zone are rebuilt from the old target SYSRES and SMP/E zone at the end of each maintenance cycle.

* Option C - SYSRES flip-flop with one SMP/E target zone, where the new target SYSRES is rebuilt from the old
target SYSRES at the end of each cycle, but the target zone is never rebuilt.

As you will see, my preference is for option A, primarily because it avoids any possible confusion over which is the
current live SYSRES pack and which the current target pack. Such confusion could lead an operator to IPL the target
pack on your production machine, or a systems programmer to apply maintenance to the live pack, with potentially
disastrous consequences in either case. Option B also suffers from the considerable overhead involved in rebuilding the
SMP/E target zone.

I should point out that the three options selected are slightly arbitrary combinations of various factors, and you should
be able to combine these factors in different ways to get other viable combinations. My selection of combinations is
justified only on the grounds that I've seen each of them working in practice.

Next part of chapter

Home Contents Previous Section Top of Page
This page last changed: 5 July 1998.